diff --git a/README.md b/README.md index b0f64dc..1719ab8 100644 --- a/README.md +++ b/README.md @@ -8,6 +8,8 @@ An example implementation of AWX on single node K3s using AWX Operator, with eas - Fixed (configurable) passwords for AWX and PostgreSQL - Fixed (configurable) versions of AWX and PostgreSQL +**If you want to view the guide for the specific version of AWX Operator, switch the page to the desired tag instead of the `main` branch.** + ## Table of Contents @@ -27,17 +29,17 @@ An example implementation of AWX on single node K3s using AWX Operator, with eas - Tested on: - CentOS Stream 8 (Minimal) - - K3s v1.23.8+k3s2 + - K3s v1.24.3+k3s1 - Products that will be deployed: - - AWX Operator 0.25.0 + - AWX Operator 0.26.0 - AWX 21.3.0 - - PostgreSQL 12 + - PostgreSQL 13 ## References - [K3s - Lightweight Kubernetes](https://rancher.com/docs/k3s/latest/en/) - [INSTALL.md on ansible/awx](https://github.com/ansible/awx/blob/21.3.0/INSTALL.md) @21.3.0 -- [README.md on ansible/awx-operator](https://github.com/ansible/awx-operator/blob/0.25.0/README.md) @0.25.0 +- [README.md on ansible/awx-operator](https://github.com/ansible/awx-operator/blob/0.26.0/README.md) @0.26.0 ## Requirements @@ -55,10 +57,15 @@ An example implementation of AWX on single node K3s using AWX Operator, with eas ### Prepare CentOS Stream 8 host -Disable Firewalld. This is [recommended by K3s](https://rancher.com/docs/k3s/latest/en/advanced/#additional-preparation-for-red-hat-centos-enterprise-linux). +Disable Firewalld and nm-cloud-setup if enabled. This is [recommended by K3s](https://rancher.com/docs/k3s/latest/en/advanced/#additional-preparation-for-red-hat-centos-enterprise-linux). ```bash +# Disable Firewalld sudo systemctl disable firewalld --now + +# Disable nm-cloud-setup if exists and enabled +systemctl disable nm-cloud-setup.service nm-cloud-setup.timer +reboot ``` Install required packages to deploy AWX Operator and AWX. @@ -83,7 +90,7 @@ Install specified version of AWX Operator. Note that this procedure is applicabl cd ~ git clone https://github.com/ansible/awx-operator.git cd awx-operator -git checkout 0.25.0 +git checkout 0.26.0 ``` Export the name of the namespace where you want to deploy AWX Operator as the environment variable `NAMESPACE` and run `make deploy`. The default namespace is `awx`. @@ -114,10 +121,13 @@ replicaset.apps/awx-operator-controller-manager-68d787cfbd 1 1 Clone this repository and change directory. +If you want to use files suitable for the specific version of AWX Operator, [refer tags in this repository](https://github.com/kurokobo/awx-on-k3s/tags) and specify desired tag in `git checkout`. + ```bash cd ~ git clone https://github.com/kurokobo/awx-on-k3s.git cd awx-on-k3s +git checkout 0.26.0 ``` Generate a Self-Signed certificate. Note that IP address can't be specified. If you want to use a certificate from public ACME CA such as Let's Encrypt or ZeroSSL instead of Self-Signed certificate, follow the guide on [πŸ“ **Use SSL Certificate from Public ACME CA**](acme) first and come back to this step when done. @@ -146,7 +156,7 @@ Modify two `password`s in `base/kustomization.yaml`. Note that the `password` un - name: awx-postgres-configuration type: Opaque literals: - - host=awx-postgres + - host=awx-postgres-13 - port=5432 - database=awx - username=awx @@ -163,9 +173,9 @@ Modify two `password`s in `base/kustomization.yaml`. Note that the `password` un Prepare directories for Persistent Volumes defined in `base/pv.yaml`. These directories will be used to store your databases and project files. Note that the size of the PVs and PVCs are specified in some of the files in this repository, but since their backends are `hostPath`, its value is just like a label and there is no actual capacity limitation. ```bash -sudo mkdir -p /data/postgres +sudo mkdir -p /data/postgres-13 sudo mkdir -p /data/projects -sudo chmod 755 /data/postgres +sudo chmod 755 /data/postgres-13 sudo chown 1000:0 /data/projects ``` @@ -190,7 +200,7 @@ $ kubectl -n awx logs -f deployments/awx-operator-controller-manager -c awx-mana ... ----- Ansible Task Status Event StdOut (awx.ansible.com/v1beta1, Kind=AWX, awx/awx) ----- PLAY RECAP ********************************************************************* -localhost : ok=67 changed=0 unreachable=0 failed=0 skipped=44 rescued=0 ignored=0 +localhost : ok=71 changed=0 unreachable=0 failed=0 skipped=48 rescued=0 ignored=0 ``` Required objects has been deployed next to AWX Operator in `awx` namespace. @@ -202,12 +212,12 @@ awx.awx.ansible.com/awx 4m17s NAME READY STATUS RESTARTS AGE pod/awx-operator-controller-manager-68d787cfbd-j6k7z 2/2 Running 0 7m43s -pod/awx-postgres-0 1/1 Running 0 4m6s +pod/awx-postgres-13-0 1/1 Running 0 4m6s pod/awx-84d5c45999-h7xm4 4/4 Running 0 3m59s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/awx-operator-controller-manager-metrics-service ClusterIP 10.43.134.67 8443/TCP 7m43s -service/awx-postgres ClusterIP None 5432/TCP 4m6s +service/awx-postgres-13 ClusterIP None 5432/TCP 4m6s service/awx-service ClusterIP 10.43.232.137 80/TCP 4m NAME READY UP-TO-DATE AVAILABLE AGE @@ -218,21 +228,18 @@ NAME DESIRED CURRENT replicaset.apps/awx-operator-controller-manager-68d787cfbd 1 1 1 7m43s replicaset.apps/awx-84d5c45999 1 1 1 3m59s -NAME READY AGE -statefulset.apps/awx-postgres 1/1 4m6s +NAME READY AGE +statefulset.apps/awx-postgres-13 1/1 4m6s NAME CLASS HOSTS ADDRESS PORTS AGE ingress.networking.k8s.io/awx-ingress awx.example.com 192.168.0.100 80, 443 4m NAME TYPE DATA AGE -secret/default-token-6tp55 kubernetes.io/service-account-token 3 7m43s -secret/awx-operator-controller-manager-token-sz6wq kubernetes.io/service-account-token 3 7m43s secret/awx-admin-password Opaque 1 4m17s secret/awx-postgres-configuration Opaque 6 4m17s secret/awx-secret-tls kubernetes.io/tls 2 4m17s secret/redhat-operators-pull-secret kubernetes.io/dockerconfigjson 1 4m17s secret/awx-app-credentials Opaque 3 4m2s -secret/awx-token-jfndh kubernetes.io/service-account-token 3 4m2s secret/awx-secret-key Opaque 1 4m13s secret/awx-broadcast-websocket Opaque 1 4m9s ``` diff --git a/acme/README.md b/acme/README.md index 95c5e88..a736bab 100644 --- a/acme/README.md +++ b/acme/README.md @@ -42,7 +42,7 @@ This guide does not provide any information how to configure Azure, other DNS se Deploy cert-manager first. ```bash -kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.8.2/cert-manager.yaml +kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.9.1/cert-manager.yaml ``` Ensure the pods in `cert-manager` namespace are running. diff --git a/backup/ansible/README.md b/backup/ansible/README.md index 4be4ba3..841ac45 100644 --- a/backup/ansible/README.md +++ b/backup/ansible/README.md @@ -32,7 +32,7 @@ An example simple playbook for Ansible is also provided in this repository. This | - | - | - | | `awxbackup_namespace` | The name of the NameSpace where the `AWXBackup` resource will be created. | `awx` | | `awxbackup_name` | The name of the `AWXBackup` resource. Dynamically generated using execution time by default. | `awxbackup-{{ lookup('pipe', 'date +%Y-%m-%d-%H-%M-%S') }}` | -| `awxbackup_spec` | The `spec` of the `AWXBackup` resource. Refer [official documentation](https://github.com/ansible/awx-operator/tree/0.25.0/roles/backup) for acceptable fields. | `deployment_name: awx`
`backup_pvc: awx-backup-claim`
`clean_backup_on_delete: true` | +| `awxbackup_spec` | The `spec` of the `AWXBackup` resource. Refer [official documentation](https://github.com/ansible/awx-operator/tree/0.26.0/roles/backup) for acceptable fields. | `deployment_name: awx`
`backup_pvc: awx-backup-claim`
`clean_backup_on_delete: true` | | `awxbackup_timeout` | Time to wait for backup to complete, in seconds. If exceeded, the playbook will fail. | `600` | | `awxbackup_keep_days` | Number of days to keep `AWXBackup` resources. `AWXBackup` resources older than this value will be deleted by this playbook. Set `0` to keep forever. | `30` | @@ -58,6 +58,11 @@ rolebinding.rbac.authorization.k8s.io/awx-backup created Obtain the API Token which required to authenticate the Kubernetes API. This token will be used later. ```bash +# Kubernetes 1.24 or later +$ kubectl -n awx create token awx-backup --duration=87600h +eyJhbGciOiJSUzI...hcGsPI5MzmaMHQvw + +# Kubernetes 1.23 or earlier $ SECRET=$(kubectl -n ${NAMESPACE} get sa awx-backup -o jsonpath='{.secrets[0].name}') $ kubectl -n ${NAMESPACE} get secret ${SECRET} -o jsonpath='{.data.token}' | base64 -d eyJhbGciOiJSUzI...hcGsPI5MzmaMHQvw diff --git a/base/awx.yaml b/base/awx.yaml index ac18e55..93f03b1 100644 --- a/base/awx.yaml +++ b/base/awx.yaml @@ -5,8 +5,8 @@ metadata: name: awx spec: # These parameters are designed for use with: - # - AWX Operator: 0.25.0 - # https://github.com/ansible/awx-operator/blob/0.25.0/README.md + # - AWX Operator: 0.26.0 + # https://github.com/ansible/awx-operator/blob/0.26.0/README.md # - AWX: 21.3.0 # https://github.com/ansible/awx/blob/21.3.0/INSTALL.md @@ -27,6 +27,8 @@ spec: projects_persistence: true projects_existing_claim: awx-projects-claim + postgres_init_container_resource_requirements: {} + postgres_resource_requirements: {} web_resource_requirements: {} task_resource_requirements: {} ee_resource_requirements: {} diff --git a/base/kustomization.yaml b/base/kustomization.yaml index 348fa99..bcbcce0 100644 --- a/base/kustomization.yaml +++ b/base/kustomization.yaml @@ -16,7 +16,7 @@ secretGenerator: - name: awx-postgres-configuration type: Opaque literals: - - host=awx-postgres + - host=awx-postgres-13 - port=5432 - database=awx - username=awx @@ -28,6 +28,14 @@ secretGenerator: literals: - password=Ansible123! + # If you want to specify SECRET_KEY for your AWX manually, uncomment following lines and change the value. + # Refer AAC documentation for detail about SECRET_KEY. + # https://docs.ansible.com/automation-controller/latest/html/administration/secret_handling.html + #- name: awx-secret-key + # type: Opaque + # literals: + # - secret_key=MySuperSecureSecretKey123! + # Temporary workaround to suppress errors from AWX Operator 0.21.0. # Refer: https://github.com/kurokobo/awx-on-k3s/issues/74 - name: redhat-operators-pull-secret diff --git a/base/pv.yaml b/base/pv.yaml index a2d7411..bc3c4c0 100644 --- a/base/pv.yaml +++ b/base/pv.yaml @@ -2,7 +2,7 @@ apiVersion: v1 kind: PersistentVolume metadata: - name: awx-postgres-volume + name: awx-postgres-13-volume spec: accessModes: - ReadWriteOnce @@ -11,7 +11,7 @@ spec: storage: 8Gi storageClassName: awx-postgres-volume hostPath: - path: /data/postgres + path: /data/postgres-13 --- apiVersion: v1 diff --git a/galaxy/README.md b/galaxy/README.md index 381dff0..0c10997 100644 --- a/galaxy/README.md +++ b/galaxy/README.md @@ -298,7 +298,7 @@ Modify two `password`s in `pulp/kustomization.yaml`. Prepare directories for Persistent Volumes defined in `pulp/pv.yaml`. ```bash -sudo mkdir -p /data/galaxy/postgres +sudo mkdir -p /data/galaxy/postgres-13 sudo mkdir -p /data/galaxy/redis sudo mkdir -p /data/galaxy/pulp ``` diff --git a/galaxy/pulp/pv.yaml b/galaxy/pulp/pv.yaml index a679c6c..3fddc6b 100644 --- a/galaxy/pulp/pv.yaml +++ b/galaxy/pulp/pv.yaml @@ -11,7 +11,7 @@ spec: storage: 8Gi storageClassName: galaxy-postgres-volume hostPath: - path: /data/galaxy/postgres + path: /data/galaxy/postgres-13 --- apiVersion: v1 diff --git a/git/deployment.yaml b/git/deployment.yaml index 6d90ae6..1ddac80 100644 --- a/git/deployment.yaml +++ b/git/deployment.yaml @@ -17,7 +17,7 @@ spec: spec: containers: - name: git - image: gitea/gitea:1.16 + image: gitea/gitea:1.17 ports: - name: gitea-http containerPort: 3000 diff --git a/restore/README.md b/restore/README.md index a99ee27..9c58ef5 100644 --- a/restore/README.md +++ b/restore/README.md @@ -27,19 +27,19 @@ If your AWX instance is running, it is recommended that it be deleted along with ```bash # Delete AWX resource, PVC, and PV kubectl -n awx delete awx awx -kubectl -n awx delete pvc postgres-awx-postgres-0 -kubectl delete pv awx-postgres-volume +kubectl -n awx delete pvc postgres-13-awx-postgres-13-0 +kubectl delete pv awx-postgres-13-volume # Delete any data in the PV -sudo rm -rf /data/postgres +sudo rm -rf /data/postgres-13 ``` Then prepare directories for your PVs. `/data/projects` is required if you are restoring the entire AWX to a new environment. ```bash -sudo mkdir -p /data/postgres +sudo mkdir -p /data/postgres-13 sudo mkdir -p /data/projects -sudo chmod 755 /data/postgres +sudo chmod 755 /data/postgres-13 sudo chown 1000:0 /data/projects ``` @@ -67,7 +67,6 @@ If you want to restore from AWXBackup object, specify its name in `restore/awxre ```yaml ... # Parameters to restore from AWXBackup object - backup_pvc_namespace: awx backup_name: awxbackup-2021-06-06 πŸ‘ˆπŸ‘ˆπŸ‘ˆ ... ``` @@ -77,7 +76,6 @@ If the AWXBackup object no longer exists, place the backup files and specify the ```yaml ... # Parameters to restore from existing files on PVC (without AWXBackup object) - backup_pvc_namespace: awx backup_pvc: awx-backup-claim πŸ‘ˆπŸ‘ˆπŸ‘ˆ backup_dir: /backups/tower-openshift-backup-2021-06-06-10:51:49 πŸ‘ˆπŸ‘ˆπŸ‘ˆ ... @@ -102,7 +100,7 @@ $ kubectl -n awx logs -f deployments/awx-operator-controller-manager -c awx-mana ... ----- Ansible Task Status Event StdOut (awx.ansible.com/v1beta1, Kind=AWX, awx/awx) ----- PLAY RECAP ********************************************************************* -localhost : ok=69 changed=0 unreachable=0 failed=0 skipped=42 rescued=0 ignored=0 +localhost : ok=73 changed=0 unreachable=0 failed=0 skipped=46 rescued=0 ignored=0 ``` This will create AWXRestore object in the namespace, and now your AWX is restored. diff --git a/restore/awxrestore.yaml b/restore/awxrestore.yaml index bdacf66..f364441 100644 --- a/restore/awxrestore.yaml +++ b/restore/awxrestore.yaml @@ -8,11 +8,9 @@ spec: deployment_name: awx # Parameters to restore from AWXBackup object - #backup_pvc_namespace: awx #backup_name: awxbackup-2021-06-06 # Parameters to restore from existing files on PVC (without AWXBackup object) - #backup_pvc_namespace: awx #backup_pvc: awx-backup-claim #backup_dir: /backups/tower-openshift-backup-2021-06-06-10:51:49 diff --git a/restore/pv.yaml b/restore/pv.yaml index 6732957..051aa2a 100644 --- a/restore/pv.yaml +++ b/restore/pv.yaml @@ -2,7 +2,7 @@ apiVersion: v1 kind: PersistentVolume metadata: - name: awx-postgres-volume + name: awx-postgres-13-volume spec: accessModes: - ReadWriteOnce @@ -11,7 +11,7 @@ spec: storage: 8Gi storageClassName: awx-postgres-volume hostPath: - path: /data/postgres + path: /data/postgres-13 --- apiVersion: v1 diff --git a/tips/dockerhub-rate-limit.md b/tips/dockerhub-rate-limit.md index 037e751..6223868 100644 --- a/tips/dockerhub-rate-limit.md +++ b/tips/dockerhub-rate-limit.md @@ -4,13 +4,13 @@ If your Pod for PostgreSQL is in `ErrImagePull` and its `Events` shows following events, this is due to [the Rate Limit on Docker Hub](https://docs.docker.com/docker-hub/download-rate-limit/). ```bash -$ kubectl -n awx describe pod awx-postgres-0 +$ kubectl -n awx describe pod awx-postgres-13-0 ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- ... - Warning Failed 2s kubelet Failed to pull image "postgres:12": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/postgres:12": failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/postgres/manifests/sha256:...: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit + Warning Failed 2s kubelet Failed to pull image "postgres:13": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/postgres:13": failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/postgres/manifests/sha256:...: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit ... ``` diff --git a/tips/external-db.md b/tips/external-db.md index 8841259..d33baa5 100644 --- a/tips/external-db.md +++ b/tips/external-db.md @@ -25,7 +25,7 @@ version: "3" services: postgres: - image: postgres:12 + image: postgres:13 ports: - 5432:5432 restart: always @@ -87,14 +87,14 @@ Note that the `type=unmanaged` is the important configuration to use external da ### Modify `base/pv.yaml` -Comment out following unnecessary lines which related to `awx-postgres-volume` in `base/pv.yaml`. +Comment out following unnecessary lines which related to `awx-postgres-13-volume` in `base/pv.yaml`. ```yaml # --- πŸ‘ˆπŸ‘ˆπŸ‘ˆ # apiVersion: v1 πŸ‘ˆπŸ‘ˆπŸ‘ˆ # kind: PersistentVolume πŸ‘ˆπŸ‘ˆπŸ‘ˆ # metadata: πŸ‘ˆπŸ‘ˆπŸ‘ˆ -# name: awx-postgres-volume πŸ‘ˆπŸ‘ˆπŸ‘ˆ +# name: awx-postgres-13-volume πŸ‘ˆπŸ‘ˆπŸ‘ˆ # spec: πŸ‘ˆπŸ‘ˆπŸ‘ˆ # accessModes: πŸ‘ˆπŸ‘ˆπŸ‘ˆ # - ReadWriteOnce πŸ‘ˆπŸ‘ˆπŸ‘ˆ @@ -103,7 +103,7 @@ Comment out following unnecessary lines which related to `awx-postgres-volume` i # storage: 8Gi πŸ‘ˆπŸ‘ˆπŸ‘ˆ # storageClassName: awx-postgres-volume πŸ‘ˆπŸ‘ˆπŸ‘ˆ # hostPath: πŸ‘ˆπŸ‘ˆπŸ‘ˆ -# path: /data/postgres πŸ‘ˆπŸ‘ˆπŸ‘ˆ +# path: /data/postgres-13 πŸ‘ˆπŸ‘ˆπŸ‘ˆ --- apiVersion: v1 @@ -115,7 +115,7 @@ metadata: ### Prepare directories -You do not need to create the `/data/postgres` directory that the main guide instructs you to create. +You do not need to create the `/data/postgres-13` directory that the main guide instructs you to create. ## The next steps diff --git a/tips/https-redirection.md b/tips/https-redirection.md index 3ca49fb..edaa243 100644 --- a/tips/https-redirection.md +++ b/tips/https-redirection.md @@ -74,7 +74,7 @@ namespace/awx unchanged secret/awx-admin-password unchanged secret/awx-postgres-configuration unchanged secret/awx-secret-tls configured -persistentvolume/awx-postgres-volume unchanged +persistentvolume/awx-postgres-13-volume unchanged persistentvolume/awx-projects-volume unchanged persistentvolumeclaim/awx-projects-claim unchanged awx.awx.ansible.com/awx configured πŸ‘ˆπŸ‘ˆπŸ‘ˆ diff --git a/tips/troubleshooting.md b/tips/troubleshooting.md index 2439c26..913a78c 100644 --- a/tips/troubleshooting.md +++ b/tips/troubleshooting.md @@ -44,7 +44,7 @@ If the Pods are working properly, its `STATUS` are `Running`. If your Pods are n $ kubectl -n awx get pod NAME READY STATUS RESTARTS AGE awx-operator-controller-manager-68d787cfbd-j6k7z 2/2 Running 0 7m43s -awx-postgres-0 1/1 Running 0 4m6s +awx-postgres-13-0 1/1 Running 0 4m6s awx-84d5c45999-h7xm4 0/4 Pending 0 3m59s ``` @@ -101,7 +101,7 @@ For AWX Operator and AWX, specifically, the following commands are helpful. - `kubectl -n awx logs -f deployment/awx -c awx-ee` - `kubectl -n awx logs -f deployment/awx -c redis` - Logs of PostgreSQL - - `kubectl -n awx logs -f statefulset/awx-postgres` + - `kubectl -n awx logs -f statefulset/awx-postgres-13` ### Reveal "censored" output in the AWX Operator's log @@ -133,15 +133,15 @@ spec: If your Pod for PostgreSQL is in `ErrImagePull` and its `Events` shows following events, this is due to [the rate limit on Docker Hub](https://docs.docker.com/docker-hub/download-rate-limit/). ```bash -$ kubectl -n awx describe pod awx-postgres-0 +$ kubectl -n awx describe pod awx-postgres-13-0 ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- - Normal Pulling 9s kubelet Pulling image "postgres:12" - Warning Failed 2s kubelet Failed to pull image "postgres:12": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/postgres:12": failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/postgres/manifests/sha256:505d023f030cdea84a42d580c2a4a0e17bbb3e91c30b2aea9c02f2dfb10325ba: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit + Normal Pulling 9s kubelet Pulling image "postgres:13" + Warning Failed 2s kubelet Failed to pull image "postgres:13": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/postgres:13": failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/postgres/manifests/sha256:...: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit Warning Failed 2s kubelet Error: ErrImagePull - Normal BackOff 1s kubelet Back-off pulling image "postgres:12" + Normal BackOff 1s kubelet Back-off pulling image "postgres:13" Warning Failed 1s kubelet Error: ImagePullBackOff ``` @@ -179,7 +179,7 @@ Typical solutions are one of the following: ee_resource_requirements: {} πŸ‘ˆπŸ‘ˆπŸ‘ˆ ``` - - You can specify more specific value for each containers. Refer [official documentation](https://github.com/ansible/awx-operator/blob/0.25.0/README.md#containers-resource-requirements) for details. + - You can specify more specific value for each containers. Refer [official documentation](https://github.com/ansible/awx-operator/blob/0.26.0/README.md#containers-resource-requirements) for details. - In this way you can run AWX with fewer resources, but you may encounter performance issues. ### The Pod is `Pending` with "1 pod has unbound immediate PersistentVolumeClaims." event @@ -199,9 +199,9 @@ Check the `STATUS` of your PVs and ensure your PVs doesn't have `Available` or ` ```bash $ kubectl get pv -NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE -awx-projects-volume 2Gi RWO Retain Released awx/awx-projects-claim awx-projects-volume 17h -awx-postgres-volume 8Gi RWO Retain Released awx/postgres-awx-postgres-0 awx-postgres-volume 17h +NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE +awx-projects-volume 2Gi RWO Retain Released awx/awx-projects-claim awx-projects-volume 17h +awx-postgres-13-volume 8Gi RWO Retain Released awx/postgres-13-awx-postgres-13-0 awx-postgres-volume 17h ``` Probably this is the second (or more) time to deploy AWX for you. These PVs which have `Released` state are tied to your old (and probably no longer exists now) PVCs you created in the past. @@ -259,8 +259,11 @@ This problem occurs when the AWX pod and the PostgreSQL pod cannot communicate p To solve this, check or try the following: -- Ensure your PostgreSQL (typically the Pod named `awx-postgres-0`) is in `Running` state. +- Ensure your PostgreSQL (typically the Pod named `awx-postgres-0` or `awx-postgres-13-0`) is in `Running` state. +- Ensure `host` under `awx-postgres-configuration` in `base/kustomizaton.yaml` has correct value. + - Specify `awx-postgres` for AWX Operator 0.25.0 or earlier, `awx-postgres-13` for `0.26.0`. - Ensure your `firewalld`, `ufw` or any kind of firewall has been disabled on your K3s host. +- Ensure your `nm-cloud-setup` service on your K3s host is disabled if exists. - Ensure your `awx-postgres-configuration` has correct values, especially if you're using external PostgreSQL. - Uninstall K3s and install it again. @@ -272,33 +275,33 @@ In this situation, your Pod for PostgreSQL is in `CrashLoopBackOff` state and it $ kubectl -n awx get pod NAME READY STATUS RESTARTS AGE awx-operator-controller-manager-68d787cfbd-j6k7z 2/2 Running 0 7m43s -awx-postgres-0 1/1 CrashLoopBackOff 3 4m6s +awx-postgres-13-0 1/1 CrashLoopBackOff 3 4m6s awx-84d5c45999-h7xm4 4/4 Running 0 3m59s $ kubectl -n awx logs statefulset/awx-postgres mkdir: cannot create directory '/var/lib/postgresql/data': Permission denied ``` -You should check the permissions and the owner of directories where used as PV on your K3s host. If you followed my guide, it would be `/data/postgres`. There is additional `data` directory created by K3s under `/data/postgres`. +You should check the permissions and the owner of directories where used as PV on your K3s host. If you followed my guide, it would be `/data/postgres-13`. There is additional `data` directory created by K3s under `/data/postgres-13`. ```bash -$ ls -ld /data/postgres /data/postgres/data -drwxr-xr-x. 2 root root 18 Aug 20 10:09 /data/postgres -drwxr-xr-x. 3 root root 20 Aug 20 10:09 /data/postgres/data +$ ls -ld /data/postgres-13 /data/postgres-13/data +drwxr-xr-x. 2 root root 18 Aug 20 10:09 /data/postgres-13 +drwxr-xr-x. 3 root root 20 Aug 20 10:09 /data/postgres-13/data ``` In my environment, `755` and `root:root` (`0:0`) works correctly. So you can try: ```bash -sudo chmod 755 /data/postgres /data/postgres/data -sudo chown 0:0 /data/postgres /data/postgres/data +sudo chmod 755 /data/postgres-13 /data/postgres-13/data +sudo chown 0:0 /data/postgres-13 /data/postgres-13/data ``` Or, you can also try `999:0` as owner/group for the directory. ```bash -sudo chmod 755 /data/postgres /data/postgres/data -sudo chown 999:0 /data/postgres /data/postgres/data +sudo chmod 755 /data/postgres-13 /data/postgres-13/data +sudo chown 999:0 /data/postgres-13 /data/postgres-13/data ``` `999` is [the UID of the `postgres` user which used in the container](https://github.com/docker-library/postgres/blob/master/12/bullseye/Dockerfile#L23). diff --git a/tips/upgrade-operator.md b/tips/upgrade-operator.md index 99d9e2a..31029b1 100644 --- a/tips/upgrade-operator.md +++ b/tips/upgrade-operator.md @@ -29,7 +29,34 @@ Refer [πŸ“README: Backing up using AWX Operator](../README.md#backing-up-using- ## πŸ“ Upgrade from `0.14.0` or later (e.g. from `0.14.0` to `0.15.0`) -If you are using AWX Operator `0.14.0` or later and want to upgrade to newer version, simply, deploy the new version of AWX Operator to the same namespace where the old AWX Operator is running. +If you are using AWX Operator `0.14.0` or later and want to upgrade to newer version, basically upgrade is done by deploying the new version of AWX Operator to the same namespace where the old AWX Operator is running. + +Note that only when upgrading **from `0.25.0` or earlier to `0.26.0` or later**, since the bundled PostgreSQL version will be changed to 13, so the following additional tasks are required. + +```bash +# Required only when upgrading from 0.25.0 or earlier to 0.26.0 or later +sudo mkdir -p /data/postgres-13 +sudo chmod 755 /data/postgres-13 +cat < pv-postgres-13.yaml +--- +apiVersion: v1 +kind: PersistentVolume +metadata: + name: awx-postgres-13-volume +spec: + accessModes: + - ReadWriteOnce + persistentVolumeReclaimPolicy: Retain + capacity: + storage: 8Gi + storageClassName: awx-postgres-volume + hostPath: + path: /data/postgres-13 +EOF +kubectl apply -f pv-postgres-13.yaml +``` + +To upgrade your AWX Operator, perform following steps. ```bash # Prepare required files @@ -61,6 +88,14 @@ PLAY RECAP ********************************************************************* localhost : ok=56 changed=0 unreachable=0 failed=0 skipped=35 rescued=0 ignored=0 ``` +If your AWX Operator has upgraded from `0.25.0` or earlier to `0.26.0` or later, old PV for PostgreSQL 12 can be removed since new AWX is running with new PV for PostgreSQL 13. + +```bash +# Recommended only when upgraded from 0.25.0 or earlier to 0.26.0 or later +kubectl delete pv awx-postgres-volume +sudo rm -rf /data/postgres +``` + ## πŸ“ Upgrade from `0.13.0` (e.g. from `0.13.0` to `0.14.0`) If you are using AWX Operator `0.13.0` and want to upgrade to newer version, you should consider the big changes in AWX Operator in `0.14.0`. [As described in the documentation](https://github.com/ansible/awx-operator/blob/0.14.0/README.md#v0140), in `0.14.0`, AWX Operator changed from cluster scope to namespace scope. Also, the Operator SDK `1.x` is used. @@ -179,17 +214,12 @@ During the AWX upgrade, a rollout of the Deployment resource will be performed a For this reason, if we do not have enough free resources on our K3s node, we can manually delete the old AWX instance beforehand in order to free up resources. Note that the rollout history will be lost with this step. ```bash -kubectl -n awx delete deployment awx -``` +$ kubectl -n awx delete deployment awx +deployment.apps "awx" deleted -Ensure that it is not the `awx` resource that should be deleted, but the `deployment` resource. If we accidentally delete the `awx` resource or any Secrets, we will not be able to upgrade successfully. - -After a few minutes of waiting, our AWX Operator will successfully launch the new Deployment and the Pod for AWX. - -```bash $ kubectl -n awx get all -NAME READY STATUS RESTARTS AGE -pod/awx-postgres-0 1/1 Running 0 8m57s +NAME READY STATUS RESTARTS AGE +pod/awx-postgres-13-0 1/1 Running 0 8m57s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/awx-postgres ClusterIP None 5432/TCP 8m57s @@ -198,3 +228,7 @@ service/awx-service ClusterIP 10.43.248.150 80/TCP 8m51 NAME READY AGE statefulset.apps/awx-postgres 1/1 8m58s ``` + +Ensure that it is not the `awx` resource that should be deleted, but the `deployment` resource. If we accidentally delete the `awx` resource or any Secrets, we will not be able to upgrade successfully. + +After a few minutes of waiting, our AWX Operator will successfully launch the new Deployment and the Pod for AWX. diff --git a/tips/version-mapping.md b/tips/version-mapping.md index 1bda13c..20ac864 100644 --- a/tips/version-mapping.md +++ b/tips/version-mapping.md @@ -10,6 +10,7 @@ The table below maps the AWX Operator versions and bundled AWX versions. | AWX Operator | AWX | | - | - | +| 0.26.0 | 21.4.0 | | 0.25.0 | 21.3.0 | | 0.24.0 | 21.3.0 | | 0.23.0 | 21.2.0 |