mirror of
https://github.com/Expand-sys/awx-on-k3s
synced 2025-12-15 21:42:15 +11:00
docs: add guide to trust custom ca for jobs
This commit is contained in:
kurokobo
parent
ace278ef8b
commit
579eabab10
1 changed files with 213 additions and 6 deletions
|
|
@ -6,17 +6,31 @@ If your AWX has to trust custom Certificate Authority, you can pass the CA certi
|
|||
- Use private Git repository via SSL, without ignoring SSL verification.
|
||||
- Use LDAPS to authenticate users.
|
||||
|
||||
Refer [the official documentation](https://github.com/ansible/awx-operator#trusting-a-custom-certificate-authority) for more information.
|
||||
Refer to [the official documentation](https://github.com/ansible/awx-operator#trusting-a-custom-certificate-authority) for more information.
|
||||
|
||||
Note that if you want to trust custom CA for jobs (running playbooks or accessing inventory sources), a different approach is required. Refer to [Appendix: Trust custom CA for jobs](#appendix-trust-custom-ca-for-jobs).
|
||||
|
||||
<!-- omit in toc -->
|
||||
## Table of Contents
|
||||
|
||||
- [Overview](#overview)
|
||||
- [Prepare required CA certificatess](#prepare-required-ca-certificatess)
|
||||
- [Prepare required CA certificates](#prepare-required-ca-certificates)
|
||||
- [Modify `base/kustomization.yaml`](#modify-basekustomizationyaml)
|
||||
- [Modify `base/awx.yaml`](#modify-baseawxyaml)
|
||||
- [Apply configuration](#apply-configuration)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
- [Appendix: Trust custom CA for jobs](#appendix-trust-custom-ca-for-jobs)
|
||||
- [Method 1: Use Container Group](#method-1-use-container-group)
|
||||
- [Create Secret](#create-secret)
|
||||
- [Create Container Group on AWX](#create-container-group-on-awx)
|
||||
- [Specify Container Group](#specify-container-group)
|
||||
- [Method 2: Mount host filesystem](#method-2-mount-host-filesystem)
|
||||
- [Update certificate store](#update-certificate-store)
|
||||
- [Modify options on AWX](#modify-options-on-awx)
|
||||
- [Add environment variables to jobs](#add-environment-variables-to-jobs)
|
||||
- [Method 3: Add certificate file to EE image](#method-3-add-certificate-file-to-ee-image)
|
||||
- [Add files to the existing image](#add-files-to-the-existing-image)
|
||||
- [Build new EE using Ansible Builder](#build-new-ee-using-ansible-builder)
|
||||
|
||||
## Overview
|
||||
|
||||
|
|
@ -32,11 +46,9 @@ There are two kinds of certificate, one is used to trust LDAP server, and the ot
|
|||
| `ldap_cacert_secret` | `ldap-ca.crt` | `awx-web` | `/etc/openldap/certs/ldap-ca.crt` |
|
||||
| `bundle_cacert_secret` | `bundle-ca.crt` | `awx-web`, `awx-task`, and `awx-ee` | `/etc/pki/ca-trust/source/anchors/bundle-ca.crt` |
|
||||
|
||||
Note that the `awx-ee` container is used to run management jobs only, not EE which runs your playbooks. If the EE running your playbook needs a certificate, you will need to [customize the pod specification](../containergroup).
|
||||
Note that the `awx-ee` container is used to run management jobs only, this is not EE which runs your playbooks. If the EE running your playbook needs certificates, refer [Appendix: Trust custom CA for jobs](#appendix-trust-custom-ca-for-jobs) on the bottom.
|
||||
|
||||
If you want to mount the certificate to the additional containers in AWX pod or the additional path other than above, you shoud add extra volumes and extra mounts using `extra_volumes` and `_extra_volume_mounts` field, but this is not covered in this guide. Refer to [the official documentation](https://github.com/ansible/awx-operator#custom-volume-and-volume-mount-options).
|
||||
|
||||
## Prepare required CA certificatess
|
||||
## Prepare required CA certificates
|
||||
|
||||
Place your certificates under `base` directory.
|
||||
|
||||
|
|
@ -197,3 +209,198 @@ SSL-Session:
|
|||
Verify return code: 0 (ok) 👈👈👈 Ensure there is no verification error
|
||||
...
|
||||
```
|
||||
|
||||
## Appendix: Trust custom CA for jobs
|
||||
|
||||
The `bundle_cacert_secret` stated above does not add any certificates to automation job pods, so it can't be used to trust custom CA to run jobs e.g. running playbooks, accessing inventory sources.
|
||||
|
||||
To trust custom CA for your jobs, there are several solutions depend on your modules or plugins that used in your jobs.
|
||||
|
||||
- Update system certificate store with custom CA certificate on automation job pods.
|
||||
- Place custom CA certificates on automation job pods and pass its path through environment variable or parameters for modules or plugins.
|
||||
|
||||
This guide provides some ways as examples to achieve above solutions by invoking `update-ca-trust` and adding environment variable `REQUESTS_CA_BUNDLE` for `requests` python module (since `requests` does not refer system certificate store, custom CA certificates should be passed through this environment variable).
|
||||
|
||||
<!-- no toc -->
|
||||
- [Method 1: Use Container Group](#method-1-use-container-group)
|
||||
- Add required certificates and environment variable on the automation job pods by mounting it using [Container Group](../containergroup) with custom pod specification.
|
||||
- [Method 2: Mount host filesystem](#method-2-mount-host-filesystem)
|
||||
- Add required certificates and environment variable on the automation job pods by mounting host filesystem.
|
||||
- [Method 3: Add certificate file to EE image](#method-3-add-certificate-file-to-ee-image)
|
||||
- Add required certificates and environment variable on the automation job pods by adding it to the EE image.
|
||||
|
||||
In this example, the value of the environment variable `REQUESTS_CA_BUNDLE` is specified as `/etc/pki/tls/cert.pem`. This file is a symbolic link to `/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` that updated by `update-ca-trust` by combining and includes all CA certificates in the system certificate store into one file.
|
||||
|
||||
### Method 1: Use Container Group
|
||||
|
||||
This method can be used for standard jobs, such as the case that some modules in your playbook require custom CA certificates.
|
||||
|
||||
Note that this method can't be used for accessing inventory sources since current AWX does not support specifying Container Group for inventory sync.
|
||||
|
||||
#### Create Secret
|
||||
|
||||
Prepare CA certificate file with PEM format (refer to [Prepare required CA certificates](#prepare-required-ca-certificates)) first, then create Secret with the certificate.
|
||||
|
||||
```bash
|
||||
kubectl -n awx create secret generic awx-custom-job-certs \
|
||||
--from-file=job-ca.crt=cacert.pem
|
||||
```
|
||||
|
||||
#### Create Container Group on AWX
|
||||
|
||||
Create new Container Group by `Administration` > `Instance Group` > `Add` > `Add container group` on AWX, then enable `Customize pod specification` and put following specification.
|
||||
|
||||
This specification supports both jobs that refers system certificate store and the store for `requests`.
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
namespace: awx
|
||||
spec:
|
||||
serviceAccountName: default
|
||||
automountServiceAccountToken: false
|
||||
initContainers:
|
||||
- name: init
|
||||
image: quay.io/ansible/awx-ee:latest
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- |
|
||||
mkdir -p /etc/pki/ca-trust/extracted/{java,pem,openssl,edk2}
|
||||
update-ca-trust
|
||||
volumeMounts:
|
||||
- name: ca-trust-extracted
|
||||
mountPath: /etc/pki/ca-trust/extracted
|
||||
- name: awx-custom-job-certs
|
||||
mountPath: /etc/pki/ca-trust/source/anchors/job-ca.crt
|
||||
subPath: job-ca.crt
|
||||
readOnly: true
|
||||
containers:
|
||||
- image: quay.io/ansible/awx-ee:latest
|
||||
name: worker
|
||||
args:
|
||||
- ansible-runner
|
||||
- worker
|
||||
- '--private-data-dir=/runner'
|
||||
env:
|
||||
- name: REQUESTS_CA_BUNDLE
|
||||
value: /etc/pki/tls/cert.pem
|
||||
resources:
|
||||
requests:
|
||||
cpu: 250m
|
||||
memory: 100Mi
|
||||
volumeMounts:
|
||||
- name: ca-trust-extracted
|
||||
mountPath: /etc/pki/ca-trust/extracted
|
||||
- name: awx-custom-job-certs
|
||||
mountPath: /etc/pki/ca-trust/source/anchors/job-ca.crt
|
||||
subPath: job-ca.crt
|
||||
readOnly: true
|
||||
volumes:
|
||||
- name: ca-trust-extracted
|
||||
emptyDir: {}
|
||||
- name: awx-custom-job-certs
|
||||
secret:
|
||||
secretName: awx-custom-job-certs
|
||||
items:
|
||||
- key: job-ca.crt
|
||||
path: job-ca.crt
|
||||
```
|
||||
|
||||
This specification has following changes from the default one:
|
||||
|
||||
- Init container (`initContainers`) is added to invoke `update-ca-trust` to update certificate store using CA certificate that mounted through Secret resource.
|
||||
- Worker container mounts certificate store that updated by init container (`volumeMounts`).
|
||||
- Worker container has environment variable `REQUESTS_CA_BUNDLE` (`env`) that has path for updated certificate file.
|
||||
- Volumes (`volumes`) are added to achieve above changes.
|
||||
|
||||
If you just want to place CA certificates somewhere on automation job pods and `update-ca-trust` is not required for you, you can ignore `initContainers` and `ca-trust-extracted` (under `volumes` and `volumeMounts`), and change `mountPath` for `awx-custom-job-certs`.
|
||||
|
||||
#### Specify Container Group
|
||||
|
||||
The Container Group can be specified as `Instance Groups` for the Job Template.
|
||||
|
||||
### Method 2: Mount host filesystem
|
||||
|
||||
This method can be used for both standard jobs and inventory sync.
|
||||
|
||||
Note that this method changes the global configuration of your K3s host and AWX, so whole system and other applications on your K3s host, and all automation job pods running on the AWX are also affected.
|
||||
|
||||
#### Update certificate store
|
||||
|
||||
Prepare CA certificate file with PEM format (refer to [Prepare required CA certificates](#prepare-required-ca-certificates)) first, then place the certificate file under `/etc/pki/ca-trust/source/anchors` on your K3s host, and invoke `update-ca-trust`.
|
||||
|
||||
```bash
|
||||
$ ls -l /etc/pki/ca-trust/source/anchors
|
||||
total 4
|
||||
-rw-r--r--. 1 root root 1489 Apr 27 22:49 cacert.pem
|
||||
|
||||
$ sudo update-ca-trust
|
||||
```
|
||||
|
||||
If your Kubernetes cluster has multiple nodes, this step have to be invoked on all nodes where your automation job pods will be launched on.
|
||||
|
||||
If you just want to place CA certificates somewhere on automation job pods and `update-ca-trust` is not required for you, you can ignore `update-ca-trust`.
|
||||
|
||||
#### Modify options on AWX
|
||||
|
||||
On `Settings` > `Jobs settings` > `Edit` page, modify following two options.
|
||||
|
||||
- Enable `Expose host paths for Container Groups`.
|
||||
- Ensure `"/etc/pki/ca-trust:/etc/pki/ca-trust:O"` is specified in `Paths to expose to isolated jobs`.
|
||||
|
||||
#### Add environment variables to jobs
|
||||
|
||||
To add environment variables to your jobs, on `Settings` > `Jobs settings` > `Edit` page, modify `Extra Environment Variables` with following value. If the environment variable is not required for you, this step can be ignored.
|
||||
|
||||
```json
|
||||
{
|
||||
"REQUESTS_CA_BUNDLE": "/etc/pki/tls/cert.pem"
|
||||
}
|
||||
```
|
||||
|
||||
### Method 3: Add certificate file to EE image
|
||||
|
||||
This method can be used for both standard jobs and inventory sync, and can be achieved by following two ways.
|
||||
|
||||
- [Add files to the existing image](#add-files-to-the-existing-image)
|
||||
- [Build new EE using Ansible Builder](#build-new-ee-using-ansible-builder)
|
||||
|
||||
#### Add files to the existing image
|
||||
|
||||
Prepare CA certificate file with PEM format (refer to [Prepare required CA certificates](#prepare-required-ca-certificates)), then you can add the certificate to existing image using following Dockerfile.
|
||||
|
||||
This example shows the way to add local certificate file (`cacert.pem`) and environment variable to `quay.io/ansible/awx-ee:latest`.
|
||||
|
||||
```dockerfile
|
||||
FROM quay.io/ansible/awx-ee:latest
|
||||
|
||||
USER root
|
||||
COPY cacert.pem /etc/pki/ca-trust/source/anchors/cacert.pem
|
||||
RUN update-ca-trust
|
||||
ENV REQUESTS_CA_BUNDLE=/etc/pki/tls/cert.pem
|
||||
|
||||
USER 1000
|
||||
```
|
||||
|
||||
Build and push image to container registry, then define new EE on AWX, and specify it for your jobs.
|
||||
|
||||
#### Build new EE using Ansible Builder
|
||||
|
||||
Prepare CA certificate file with PEM format (refer to [Prepare required CA certificates](#prepare-required-ca-certificates)), then you can add `additional_build_files` and `additional_build_steps.append_final` to your `execution-environment.yml` to add the certificate and environment variable to your EE.
|
||||
|
||||
```yaml
|
||||
...
|
||||
additional_build_files:
|
||||
- src: certs/cacert.pem
|
||||
dest: certs
|
||||
|
||||
additional_build_steps:
|
||||
append_final:
|
||||
- COPY _build/certs/cacert.pem /etc/pki/ca-trust/source/anchors/cacert.pem
|
||||
- RUN update-ca-trust
|
||||
- ENV REQUESTS_CA_BUNDLE=/etc/pki/tls/cert.pem
|
||||
```
|
||||
|
||||
Build and push image to container registry, then define new EE on AWX, and specify it for your jobs.
|
||||
|
|
|
|||
Loading…
Reference in a new issue