awx-on-k3s/tips/upgrade-operator.md

Upgrade AWX Operator and AWX

This guide provides the procedure for the following three types of upgrading AWX Operator.

• Upgrade from 0.14.0 or later (e.g. from 0.14.0 to 0.15.0)
• Upgrade from 0.13.0 (e.g. from 0.13.0 to 0.14.0)
• Upgrade from 0.12.0 or earlier (e.g. from 0.12.0 to 0.13.0)

Note that once you upgrade AWX Operator, your AWX will also be upgraded automatically to the version bundled with the upgraded AWX Operator as shown in 📝Version Mapping for AWX Operator and AWX.

There is image_version parameter for AWX resource to change which image will be used, but it appears that using a version of AWX other than the one bundled with the AWX Operator is currently not supported. Conversely, if you want to upgrade AWX, you need to plan to upgrade AWX Operator first.

Table of Contents

• ✅ Take a backup of the old AWX instance
• 📝 Upgrade from 0.14.0 or later (e.g. from 0.14.0 to 0.15.0)
  • 📝 Procedure
• 📝 Upgrade from 0.13.0 (e.g. from 0.13.0 to 0.14.0)
  • 📝 Procedure
• 📝 Upgrade from 0.12.0 or earlier (e.g. from 0.12.0 to 0.13.0)
• ❓ Troubleshooting
  • New Pod gets stuck in Pending state

✅ Take a backup of the old AWX instance

Before performing the upgrade, make sure that you have a backup of your old AWX.

Refer 📝README: Backing up using AWX Operator to take backup using AWX Operator.

📝 Upgrade from 0.14.0 or later (e.g. from 0.14.0 to 0.15.0)

If you are using AWX Operator 0.14.0 or later and want to upgrade to newer version, basically upgrade is done by deploying the new version of AWX Operator to the same namespace where the old AWX Operator is running.

Warning

If you are planning to upgrade AWX Operator from 2.0.0 to 2.0.1 or later, note that the extra_volumes and extra_volumes in base/awx.yaml for 2.0.0 as a workaround for specific issue causes failure of upgrading.

To avoid this, follow these steps before upgrading AWX Operator. Steps 1 and 2 can also be achieved by kubectl -n awx edit awx awx.

1. Remove the definition of the volume and volume mount that named awx-projects-web in extra_volumes and web_extra_volume_mounts in your base/awx.yaml.
   • If there are no other volumes or volume mounts, you can remove whole extra_volumes and web_extra_volume_mounts.
2. Apply modified base/awx.yaml by kubectl apply -k base
3. Wait for deployment for AWX to be completed

Once your AWX has been deployed without volume awx-projects-web, your AWX can be safely upgraded. Proceed to the next step.

Warning

If you are planning to upgrade AWX Operator from 0.25.0 or earlier to 0.26.0 or later, note that since the bundled PostgreSQL version will be changed to 13, so the following additional tasks are required.

# Required only when upgrading from 0.25.0 or earlier to 0.26.0 or later
sudo mkdir -p /data/postgres-13
sudo chmod 755 /data/postgres-13
cat <<EOF > pv-postgres-13.yaml
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: awx-postgres-13-volume
spec:
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  capacity:
    storage: 8Gi
  storageClassName: awx-postgres-volume
  hostPath:
    path: /data/postgres-13
EOF
kubectl apply -f pv-postgres-13.yaml

📝 Procedure

To upgrade your AWX Operator, perform following steps. If your target version is earlier than 2.3.0, switch this page to the desired tag instead of the main branch.

# Prepare required files
cd ~
git clone https://github.com/kurokobo/awx-on-k3s.git
cd awx-on-k3s
git checkout 2.3.0  # Checkout the version to upgrade to

# Deploy AWX Operator
kubectl apply -k operator

This will upgrade the AWX Operator first, after that, AWX will be also upgraded as well.

To monitor the progress of the deployment, check the logs of deployments/awx-operator-controller-manager:

kubectl -n awx logs -f deployments/awx-operator-controller-manager -c awx-manager

When the deployment completes successfully, the logs end with:

$ kubectl -n awx logs -f deployments/awx-operator-controller-manager -c awx-manager
...
----- Ansible Task Status Event StdOut (awx.ansible.com/v1beta1, Kind=AWX, awx/awx) -----
PLAY RECAP *********************************************************************
localhost                  : ok=56   changed=0    unreachable=0    failed=0    skipped=35   rescued=0    ignored=0

If your AWX Operator has upgraded from 0.25.0 or earlier to 0.26.0 or later, old PV for PostgreSQL 12 can be removed since new AWX is running with new PV for PostgreSQL 13.

# Recommended only when upgraded from 0.25.0 or earlier to 0.26.0 or later
kubectl delete pv awx-postgres-volume
sudo rm -rf /data/postgres

📝 Upgrade from 0.13.0 (e.g. from 0.13.0 to 0.14.0)

If you are using AWX Operator 0.13.0 and want to upgrade to newer version, you should consider the big changes in AWX Operator in 0.14.0. As described in the documentation, in 0.14.0, AWX Operator changed from cluster scope to namespace scope. Also, the Operator SDK 1.x is used.

This means that upgrading from 0.13.0 to 0.14.0 or later requires a bit of finesse, such as cleaning the old AWX Operator. If you are using 0.12.0 or earlier and want to upgrade to 0.14.0 or later, I recommend you to upgrade to 0.13.0 first and then come back to here to avoid unintended issue.

In this guide, for example, perform upgrading from 0.13.0 to 0.14.0. The AWX Operator 0.13.0 or earlier resides in the default namespace by default and the related AWX instance resides in the awx namespace, as described in this repository. After the upgrade, everything related to the AWX Operator 0.14.0 will reside in the awx namespace.

Phase	AWX Operator	AWX Instance
Before Upgrade	0.13.0 in default namespace	19.3.0 in awx namespace
After Upgrade	0.14.0 in awx namespace	19.4.0 in awx namespace

📝 Procedure

To upgrade AWX Operator, remove the old AWX Operator that is running in the default namespace first. In addition, remove Service Account, Cluster Role, and Cluster Role Binding that are required for old AWX Operator to work.

kubectl -n default delete deployment awx-operator
kubectl -n default delete serviceaccount awx-operator
kubectl -n default delete clusterrolebinding awx-operator
kubectl -n default delete clusterrole awx-operator

Since we removed only old AWX Operator, the old CRDs are still exist. Therefore, the old awx resource which means old AWX instance is still running in the awx namespace.

Finally, deploy the new AWX Operator to the awx namespace.

# Prepare required files
cd ~
git clone https://github.com/ansible/awx-operator.git
cd awx-operator
git checkout 0.14.0  # Checkout the version to upgrade to

# Deploy AWX Operator
export NAMESPACE=awx  # Specify the namespace where the old AWX instance exists
make deploy

This will update the CRDs in the cluster and create the required Service Account, Roles, etc. in the awx namespace. Also, AWX Operator will start working. Once AWX Operator is up and running, it will start rolling out a new version of the AWX instance automatically based on the old awx resource definition.

To monitor the progress of the deployment, check the logs of deployments/awx-operator-controller-manager:

kubectl -n awx logs -f deployments/awx-operator-controller-manager -c awx-manager

When the deployment completes successfully, the logs end with:

$ kubectl -n awx logs -f deployments/awx-operator-controller-manager -c awx-manager
...
----- Ansible Task Status Event StdOut (awx.ansible.com/v1beta1, Kind=AWX, awx/awx) -----
PLAY RECAP *********************************************************************
localhost                  : ok=56   changed=0    unreachable=0    failed=0    skipped=35   rescued=0    ignored=0

📝 Upgrade from 0.12.0 or earlier (e.g. from 0.12.0 to 0.13.0)

If you are using 0.12.0 or earlier and want to upgrade to newer version, simply, deploy the new version of AWX Operator. This procedure can be applicable for upgrading to up to 0.13.0. If you want to upgrade to 0.14.0 or later, I recommend you to upgrade to 0.13.0 by following this procedure first and then perform upgrading to 0.14.0 or later.

# Specify the version to upgrade to in the URL
kubectl apply -f https://raw.githubusercontent.com/ansible/awx-operator/0.13.0/deploy/awx-operator.yaml

This will upgrade the AWX Operator first, after that, AWX will be also upgraded as well.

To monitor the progress of the deployment, check the logs of deployment/awx-operator:

kubectl logs -f deployment/awx-operator

When the deployment completes successfully, the logs end with:

$ kubectl logs -f deployment/awx-operator
...
--------------------------- Ansible Task Status Event StdOut  -----------------
PLAY RECAP *********************************************************************
localhost                  : ok=54   changed=0    unreachable=0    failed=0    skipped=37   rescued=0    ignored=0 
-------------------------------------------------------------------------------

❓ Troubleshooting

Some hints for when you got stuck during upgrade.

New Pod gets stuck in Pending state

If the K3s node does not have enough free resources to deploy a new AWX instance, the new Pod for AWX gets stuck in Pending state.

$ kubectl -n awx get pods
NAME                                               READY   STATUS    RESTARTS   AGE
awx-7d74496d7d-d66dw                               4/4     Running   0          19d
awx-84d5c45999-55gb4                               0/4     Pending   0          10s     👈👈👈

Try running kubectl -n awx describe pod <Pod Name> and check the Events section at the end for the cause.

$ kubectl -n awx describe pod awx-84d5c45999-55gb4
...
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  106s  default-scheduler  0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory.     👈👈👈
  Warning  FailedScheduling  105s  default-scheduler  0/1 nodes are available: 1 Insufficient cpu, 1 Insufficient memory.     👈👈👈

This means that the node does not have enough CPU or memory resources to start the Pod.

During the AWX upgrade, a rollout of the Deployment resource will be performed and temporarily two AWX Pods will be running. This means that the required Resource Requests for CPU and memory will be doubled.

For this reason, if we do not have enough free resources on our K3s node, we can manually delete the old AWX instance beforehand in order to free up resources. Note that the rollout history will be lost with this step.

$ kubectl -n awx delete deployment awx
deployment.apps "awx" deleted

$ kubectl -n awx get all
NAME                    READY   STATUS    RESTARTS   AGE
pod/awx-postgres-13-0   1/1     Running   0          8m57s

NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/awx-postgres   ClusterIP   None            <none>        5432/TCP   8m57s
service/awx-service    ClusterIP   10.43.248.150   <none>        80/TCP     8m51s

NAME                            READY   AGE
statefulset.apps/awx-postgres   1/1     8m58s

Ensure that it is not the awx resource that should be deleted, but the deployment resource. If we accidentally delete the awx resource or any Secrets, we will not be able to upgrade successfully.

After a few minutes of waiting, our AWX Operator will successfully launch the new Deployment and the Pod for AWX.