expand/awx-on-k3s
1
0
Fork 0
mirror of https://github.com/Expand-sys/awx-on-k3s synced 2025-12-15 21:42:15 +11:00
Code Issues Projects Releases Packages Wiki Activity Actions

feat: add files to deploy private container registry on k3s

Browse source
This commit is contained in:
kurokobo 2021-07-16 12:21:38 -04:00
parent 0601e18594
commit e26aae3f5c
10 changed files with 414 additions and 17 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

43
README.md
View file

@ -1,3 +1,4 @@
<!-- omit in toc -->
# AWX on Single Node K3s # AWX on Single Node K3s
An example implementation of AWX on single node K3s using AWX Operator, with easy-to-use simplified configuration with ownership of data and passwords. An example implementation of AWX on single node K3s using AWX Operator, with easy-to-use simplified configuration with ownership of data and passwords.
@ -7,25 +8,25 @@ An example implementation of AWX on single node K3s using AWX Operator, with eas
- Fixed (configurable) passwords for AWX and PostgreSQL - Fixed (configurable) passwords for AWX and PostgreSQL
- Fixed (configurable) versions of AWX and PostgreSQL - Fixed (configurable) versions of AWX and PostgreSQL
<!-- omit in toc -->
## Table of Contents ## Table of Contents
- [AWX on Single Node K3s](#awx-on-single-node-k3s) - [Environment](#environment)
- [Table of Contents](#table-of-contents) - [References](#references)
- [Environment](#environment) - [Procedure](#procedure)
- [References](#references) - [Prepare CentOS 8 host](#prepare-centos-8-host)
- [Procedure](#procedure) - [Install K3s](#install-k3s)
- [Prepare CentOS 8 host](#prepare-centos-8-host) - [Install AWX Operator](#install-awx-operator)
- [Install K3s](#install-k3s) - [Prepare required files](#prepare-required-files)
- [Install AWX Operator](#install-awx-operator) - [Deploy AWX](#deploy-awx)
- [Prepare required files](#prepare-required-files) - [Backing up and Restoring using AWX Operator](#backing-up-and-restoring-using-awx-operator)
- [Deploy AWX](#deploy-awx) - [Backing up using AWX Operator](#backing-up-using-awx-operator)
- [Backing up and Restoring using AWX Operator](#backing-up-and-restoring-using-awx-operator) - [Prepare for Backup](#prepare-for-backup)
- [Backing up using AWX Operator](#backing-up-using-awx-operator) - [Invoke Manual Backup](#invoke-manual-backup)
- [Prepare for Backup](#prepare-for-backup) - [Restoring using AWX Operator](#restoring-using-awx-operator)
- [Invoke Manual Backup](#invoke-manual-backup) - [Prepare for Restore](#prepare-for-restore)
- [Restoring using AWX Operator](#restoring-using-awx-operator) - [Invoke Manual Restore](#invoke-manual-restore)
- [Prepare for Restore](#prepare-for-restore) - [Deploy Private Container Registry](#deploy-private-container-registry)
- [Invoke Manual Restore](#invoke-manual-restore)
## Environment ## Environment
@ -325,3 +326,11 @@ Then restore the Secret for TLS manually (or create newly using original certifi
```bash ```bash
kubectl apply -f awx-secret-tls.yaml kubectl apply -f awx-secret-tls.yaml
``` ```
## Deploy Private Container Registry
To use Execution Environments in AWX (AWX-EE), we have to push the container image built with `ansible-builder` to the container registry.
If we don't want to push our container images to Docker Hub or other cloud services, we can deploy a private container registry on K3s.
See [📝 `registry/README.md`](registry/README.md) for instructions.

230
registry/README.md Normal file
View file

@ -0,0 +1,230 @@
<!-- omit in toc -->
# Deploy Private Container Registry
Deploying your private container registry on your K3s to use with AWX.
<!-- omit in toc -->
## Table of Contents
- [Procedure](#procedure)
- [Prepare required files](#prepare-required-files)
- [Deploy Private Container Registry](#deploy-private-container-registry)
- [Quick Testing](#quick-testing)
- [Testing with Docker](#testing-with-docker)
- [Digging into the Registry](#digging-into-the-registry)
- [(Optional) Use as Private Container Registry for K3s](#optional-use-as-private-container-registry-for-k3s)
- [Procedure](#procedure-1)
- [Testing](#testing)
## Procedure
### Prepare required files
Generate a Self-Signed Certificate. Note that IP address can't be specified.
```bash
REGISTRY_HOST="registry.example.com"
openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -out ./registry/tls.crt -keyout ./registry/tls.key -subj "/CN=${REGISTRY_HOST}/O=${REGISTRY_HOST}" -addext "subjectAltName = DNS:${REGISTRY_HOST}"
```
Modify `hosts` and `host` in `registry/ingress.yaml`.
```yaml
...
- hosts:
- registry.example.com 👈👈👈
secretName: registry-secret-tls
rules:
- host: registry.example.com 👈👈👈
...
```
Generate `htpasswd` string by your own username and password to use as the user for the container registry.
```bash
$ kubectl run htpasswd -it --restart=Never --image httpd:2.4 --rm -- htpasswd -nbB reguser Registry123!
reguser:$2y$05$VLMvcWCPF0VUuHi0BXBz7eoXGZ6KRl1gataiqTXz4DdSVIXGloKiq
pod "htpasswd" deleted
```
Replace `htpasswd` in `registry/configmap.yaml` with your own `htpasswd` string that generated above.
```yaml
...
htpasswd: |-
reguser:$2y$05$VLMvcWCPF0VUuHi0BXBz7eoXGZ6KRl1gataiqTXz4DdSVIXGloKiq 👈👈👈
```
Prepare directories for Persistent Volumes defined in `registry/pv.yaml`.
```bash
sudo mkdir -p /data/registry
```
### Deploy Private Container Registry
Deploy Private Container Registry.
```bash
kubectl apply -k registry
```
Required resources has been deployed in `registry` namespace.
```bash
$ kubectl get all -n registry
NAME READY STATUS RESTARTS AGE
pod/registry-5b4f874b77-9gb64 1/1 Running 0 27s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/registry-service ClusterIP 10.43.50.156 <none> 5000/TCP 28s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/registry 1/1 1 1 27s
NAME DESIRED CURRENT READY AGE
replicaset.apps/registry-5b4f874b77 1 1 1 27s
```
Now your container registry can be used through `registry.example.com` or the hostname you specified.
## Quick Testing
### Testing with Docker
Add your registry as an insecure registry and restart Docker daemon.
```bash
sudo tee /etc/docker/daemon.json <<EOF
{
"insecure-registries" : ["registry.example.com"]
}
EOF
sudo systemctl restart docker
```
Log in to your container registry.
```bash
$ docker login registry.example.com
Username: reguser
Password:
WARNING! Your password will be stored unencrypted in /home/********/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
```
Now you can push/pull the image to/from your container registry.
```bash
# Pull from docker.io
docker pull docker.io/docker/whalesay:latest
# Tag as your own image on your private container registry
docker tag docker.io/docker/whalesay:latest registry.example.com/reguser/whalesay:latest
# Push your own image to your private container registry
docker push registry.example.com/reguser/whalesay:latest
```
```bash
# Remove local images
docker image rm docker.io/docker/whalesay:latest
docker image rm registry.example.com/reguser/whalesay:latest
# Pull the image from your private container registry
docker pull registry.example.com/reguser/whalesay:latest
```
```bash
$ docker run -it --rm registry.example.com/reguser/whalesay:latest cowsay hoge
______
< hoge >
------
\
\
\
## .
## ## ## ==
## ## ## ## ===
/""""""""""""""""___/ ===
~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ / ===- ~~~
\______ o __/
\ \ __/
\____\______/
```
### Digging into the Registry
There is an useful CLI tool called [**reg**](https://github.com/genuinetools/reg) to dig into the container registry.
```bash
# Install reg
sudo curl -fSL https://github.com/genuinetools/reg/releases/download/v0.16.1/reg-linux-amd64 -o /usr/local/bin/reg
sudo chmod +x /usr/local/bin/reg
# List repositories and tags in the container registry
reg ls -k registry.example.com
reg tags -k registry.example.com/reguser/whalesay
# Delete tags on the registry
reg rm -k registry.example.com/reguser/whalesay:latest
```
## (Optional) Use as Private Container Registry for K3s
Optionally, this registry can also be registered as a private container registry for K3s.
### Procedure
To achieve this, create a `registries.yaml` and restart K3s.
```bash
sudo tee /etc/rancher/k3s/registries.yaml <<EOF
configs:
registry.example.com:
auth:
username: reguser
password: Registry123!
tls:
insecure_skip_verify: true
EOF
# The K3s service can be safely restarted without affecting the running resources
sudo systemctl restart k3s
```
If this is successfully applied, you can check the applied configuration in the `config.registry` section of the following command.
```bash
sudo /usr/local/bin/crictl info
# With jq
sudo /usr/local/bin/crictl info | jq .config.registry
```
### Testing
You can launch your Pod using an image from a private repository that requires authentication.
```bash
$ kubectl run whalesay -it --restart=Never --image registry.example.com/reguser/whalesay:latest --rm -- cowsay hoge
______
< hoge >
------
\
\
\
## .
## ## ## ==
## ## ## ## ===
/""""""""""""""""___/ ===
~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ / ===- ~~~
\______ o __/
\ \ __/
\____\______/
pod "whalesay" deleted
```

35
registry/configmap.yaml Normal file
View file

@ -0,0 +1,35 @@
---
apiVersion: v1
kind: ConfigMap
metadata:
name: registry-configmap
labels:
app: registry
data:
config.yml: |-
version: 0.1
log:
fields:
service: registry
storage:
cache:
blobdescriptor: inmemory
filesystem:
rootdirectory: /var/lib/registry
delete:
enabled: true
http:
addr: :5000
headers:
X-Content-Type-Options: [nosniff]
auth:
htpasswd:
realm: basic-realm
path: /etc/docker/registry/htpasswd
health:
storagedriver:
enabled: true
interval: 10s
threshold: 3
htpasswd: |-
reguser:$2y$05$VLMvcWCPF0VUuHi0BXBz7eoXGZ6KRl1gataiqTXz4DdSVIXGloKiq

36
registry/deployment.yaml Normal file
View file

@ -0,0 +1,36 @@
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: registry
labels:
app: registry
spec:
replicas: 1
selector:
matchLabels:
app: registry
template:
metadata:
labels:
app: registry
spec:
containers:
- name: registry
image: registry:2.7
ports:
- containerPort: 5000
protocol: TCP
volumeMounts:
- name: registry-volume
mountPath: /var/lib/registry
- name: registry-config
mountPath: /etc/docker/registry
volumes:
- name: registry-volume
persistentVolumeClaim:
claimName: registry-claim
- name: registry-config
configMap:
defaultMode: 420
name: registry-configmap

21
registry/ingress.yaml Normal file
View file

@ -0,0 +1,21 @@
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: registry-ingress
spec:
tls:
- hosts:
- registry.example.com
secretName: registry-secret-tls
rules:
- host: registry.example.com
http:
paths:
- path: /
pathType: ImplementationSpecific
backend:
service:
name: registry-service
port:
number: 5000

23
registry/kustomization.yaml Normal file
View file

@ -0,0 +1,23 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: registry
generatorOptions:
disableNameSuffixHash: true
secretGenerator:
- name: registry-secret-tls
type: kubernetes.io/tls
files:
- tls.crt
- tls.key
resources:
- namespace.yaml
- pv.yaml
- pvc.yaml
- ingress.yaml
- service.yaml
- configmap.yaml
- deployment.yaml

5
registry/namespace.yaml Normal file
View file

@ -0,0 +1,5 @@
---
apiVersion: v1
kind: Namespace
metadata:
name: registry

14
registry/pv.yaml Normal file
View file

@ -0,0 +1,14 @@
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: registry-volume
spec:
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
capacity:
storage: 5Gi
storageClassName: registry-volume
hostPath:
path: /data/registry

13
registry/pvc.yaml Normal file
View file

@ -0,0 +1,13 @@
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: registry-claim
spec:
accessModes:
- ReadWriteOnce
volumeMode: Filesystem
resources:
requests:
storage: 5Gi
storageClassName: registry-volume

11
registry/service.yaml Normal file
View file

@ -0,0 +1,11 @@
---
apiVersion: v1
kind: Service
metadata:
name: registry-service
spec:
ports:
- protocol: TCP
port: 5000
selector:
app: registry