awx-on-k3s/backup/ansible/README.md

Appendix: Back up AWX using Ansible

An example simple playbook for Ansible is also provided in this repository. This can be used with ansible-playbook, ansible-runner, and AWX. It can be also used with the scheduling feature on AWX too.

Table of Contents

• Requirements
• Variables
• Preparation
  • Prepare Service Account and API Token
  • Prepare Backup Storage
• Use with Ansible
• Use with Ansible Runner
• Use with AWX

Requirements

• Ansible collections
  • kubernetes.core
• Pip modules
  • Refer the kubernetes.core.k8s module documentation

Variables

This example playbook is designed to allow you to customize your backup with variables.

Variables	Description	Default
awxbackup_namespace	The name of the NameSpace where the AWXBackup resource will be created.	awx
awxbackup_name	The name of the AWXBackup resource. Dynamically generated using execution time by default.	awxbackup-{{ lookup('pipe', 'date +%Y-%m-%d-%H-%M-%S') }}
awxbackup_spec	The spec of the AWXBackup resource. Refer official documentation for acceptable fields.	deployment_name: awx backup_pvc: awx-backup-claim clean_backup_on_delete: true
awxbackup_timeout	Time to wait for backup to complete, in seconds. If exceeded, the playbook will fail.	600
awxbackup_keep_days	Number of days to keep AWXBackup resources. AWXBackup resources older than this value will be deleted by this playbook. Set 0 to keep forever.	30

Note that this playbook enables clean_backup_on_delete by default that only works with AWX Operator 0.24.0 and later. This option makes that your actual backup data in your PVC is deleted at the same time the AWXBackup resource is deleted. You can disable this feature by explicitly specifying clean_backup_on_delete: false. Refer the official documentation for detail.

Preparation

Prepare Service Account and API Token

Create a Service Account, Role, and RoleBinding to manage the AWXBackup resource.

# Specify NameSpace where your AWXBackup resources will be created.
$ NAMESPACE=awx
$ kubectl -n ${NAMESPACE} apply -f rbac/sa.yaml
serviceaccount/awx-backup created
role.rbac.authorization.k8s.io/awx-backup created
rolebinding.rbac.authorization.k8s.io/awx-backup created

Prepare Backup Storage

Since you have complete control over spec of AWXBackup via awxbackup_spec variables, whether or not this step is required depends on your environment. Check the official documentation and prepare as needed.

If your AWX was deployed by referring the main guide on this repository, preparing backup storage by following the basic backup guide is good starting point.

Use with Ansible

Export required environment variables.

export K8S_AUTH_HOST="https://<Your K3s Host>:6443/"
export K8S_AUTH_VERIFY_SSL=no

Obtain and export the API Token which required to authenticate the Kubernetes API.

# For Kubernetes 1.24 or later:
# This will generate new token that valid for 1 hours.
# Of course you can modify the value since "1h" is just an example.
$ export K8S_AUTH_API_KEY=$(kubectl -n awx create token awx-backup --duration=1h)

# For Kubernetes 1.23 or earlier:
# Obtain and decode token from Secret that automatically generated for the Service Account.
$ SECRET=$(kubectl -n ${NAMESPACE} get sa awx-backup -o jsonpath='{.secrets[0].name}')
$ export K8S_AUTH_API_KEY=$(kubectl -n ${NAMESPACE} get secret ${SECRET} -o jsonpath='{.data.token}' | base64 -d)

# Modify variables using "-e" as needed
ansible-playbook project/backup.yml \
  -e awxbackup_spec="{'deployment_name':'awx','backup_pvc':'awx-backup-claim','clean_backup_on_delete':'true'}" \
  -e keep_days=90

Use with Ansible Runner

Refer the guide for Ansible Runner for the basic usage.

Modify following files as needed. Note that the EE quay.io/ansible/awx-ee:latest contains required modules and collections by default.

• 📝env/settings: Configure your Execution Environment
• 📝env/envvars: Specify your K3s host and API Token
• 📝env/extravars: Modify variables

Then execute Ansible Runner.

ansible-runner run . -p backup.yml

Use with AWX

This playbook can also be run through Job Templates on AWX. Of course Schedules can be set up in the Job Template to obtain periodic backup.

This is the way to make the backup of the AWX itself where the Job Template for the backup is configured.

In this case, the PostgreSQL will be dumped while the job is running, so complete logs of the job itself is not part of the backup. Therefore, after restoration, the last backup job will be shown as failed since the AWX can't determine the result of the job, but this can be safely ignored.

1. Add new Container Group to make the API token usable inside the EE.

   • Enable Customize pod specification and put following YAML string. serviceAccountName and automountServiceAccountToken are important to make the API token usable inside the EE.

     apiVersion: v1
     kind: Pod
     metadata:
       namespace: awx
     spec:
       serviceAccountName: awx-backup
       automountServiceAccountToken: true
       containers:
         - image: quay.io/ansible/awx-ee:latest
           name: worker
           args:
             - ansible-runner
             - worker
             - '--private-data-dir=/runner'
           resources:
             requests:
               cpu: 250m
               memory: 100Mi
     

2. Add new Project including the playbook.

   • You can specify this repository (https://github.com/kurokobo/awx-on-k3s.git) directly, but use with caution. The playbook in this repository is subject to change without notice. You can use Tag or Commit to fix the version to be used.

3. Add new Job Template which use the playbook.

   • Select appropriate Execution Environment. The default AWX EE (latest) (quay.io/ansible/awx-ee:latest) contains required collections and modules by default, so it's good for the first choice.
   • Select your backup.yml as Playbook.
   • Specify Variables as needed.
   • Select your Container Group created in the above step as Instance Group.

4. (Optional) Add new Schedules for periodic backup.

If you want to make the backup of another AWX on different namespace of different cluster, create new Credential with OpenShift or Kubernetes API Bearer Token type instead of Container Group and then specify the Credential in the Job Template. To obtain the API token, refer "Use with Ansible" section.