feat: add files to deploy private container registry on k3s

README.md

@@ -1,3 +1,4 @@
+<!-- omit in toc -->
 # AWX on Single Node K3s
 
 An example implementation of AWX on single node K3s using AWX Operator, with easy-to-use simplified configuration with ownership of data and passwords.
@@ -7,25 +8,25 @@ An example implementation of AWX on single node K3s using AWX Operator, with eas
 - Fixed (configurable) passwords for AWX and PostgreSQL
 - Fixed (configurable) versions of AWX and PostgreSQL
 
+<!-- omit in toc -->
 ## Table of Contents
 
-- [AWX on Single Node K3s](#awx-on-single-node-k3s)
-  - [Table of Contents](#table-of-contents)
-  - [Environment](#environment)
-  - [References](#references)
-  - [Procedure](#procedure)
+- [Environment](#environment)
+- [References](#references)
+- [Procedure](#procedure)
   - [Prepare CentOS 8 host](#prepare-centos-8-host)
   - [Install K3s](#install-k3s)
   - [Install AWX Operator](#install-awx-operator)
   - [Prepare required files](#prepare-required-files)
   - [Deploy AWX](#deploy-awx)
-  - [Backing up and Restoring using AWX Operator](#backing-up-and-restoring-using-awx-operator)
+- [Backing up and Restoring using AWX Operator](#backing-up-and-restoring-using-awx-operator)
   - [Backing up using AWX Operator](#backing-up-using-awx-operator)
     - [Prepare for Backup](#prepare-for-backup)
     - [Invoke Manual Backup](#invoke-manual-backup)
   - [Restoring using AWX Operator](#restoring-using-awx-operator)
     - [Prepare for Restore](#prepare-for-restore)
     - [Invoke Manual Restore](#invoke-manual-restore)
+- [Deploy Private Container Registry](#deploy-private-container-registry)
 
 ## Environment
 
@@ -325,3 +326,11 @@ Then restore the Secret for TLS manually (or create newly using original certifi
 ```bash
 kubectl apply -f awx-secret-tls.yaml
 ```
+
+## Deploy Private Container Registry
+
+To use Execution Environments in AWX (AWX-EE), we have to push the container image built with `ansible-builder` to the container registry.
+
+If we don't want to push our container images to Docker Hub or other cloud services, we can deploy a private container registry on K3s.
+
+See [📝 `registry/README.md`](registry/README.md) for instructions.

registry/README.md

@@ -0,0 +1,230 @@
+<!-- omit in toc -->
+# Deploy Private Container Registry
+
+Deploying your private container registry on your K3s to use with AWX.
+
+<!-- omit in toc -->
+## Table of Contents
+
+- [Procedure](#procedure)
+  - [Prepare required files](#prepare-required-files)
+  - [Deploy Private Container Registry](#deploy-private-container-registry)
+- [Quick Testing](#quick-testing)
+  - [Testing with Docker](#testing-with-docker)
+  - [Digging into the Registry](#digging-into-the-registry)
+- [(Optional) Use as Private Container Registry for K3s](#optional-use-as-private-container-registry-for-k3s)
+  - [Procedure](#procedure-1)
+  - [Testing](#testing)
+
+## Procedure
+
+### Prepare required files
+
+Generate a Self-Signed Certificate. Note that IP address can't be specified.
+
+```bash
+REGISTRY_HOST="registry.example.com"
+openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -out ./registry/tls.crt -keyout ./registry/tls.key -subj "/CN=${REGISTRY_HOST}/O=${REGISTRY_HOST}" -addext "subjectAltName = DNS:${REGISTRY_HOST}"
+```
+
+Modify `hosts` and `host` in `registry/ingress.yaml`.
+
+```yaml
+...
+    - hosts:
+        - registry.example.com     👈👈👈
+      secretName: registry-secret-tls
+  rules:
+    - host: registry.example.com     👈👈👈
+...
+```
+
+Generate `htpasswd` string by your own username and password to use as the user for the container registry.
+
+```bash
+$ kubectl run htpasswd -it --restart=Never --image httpd:2.4 --rm -- htpasswd -nbB reguser Registry123!
+reguser:$2y$05$VLMvcWCPF0VUuHi0BXBz7eoXGZ6KRl1gataiqTXz4DdSVIXGloKiq
+
+pod "htpasswd" deleted
+```
+
+Replace `htpasswd` in `registry/configmap.yaml` with your own `htpasswd` string that generated above.
+
+```yaml
+...
+  htpasswd: |-
+    reguser:$2y$05$VLMvcWCPF0VUuHi0BXBz7eoXGZ6KRl1gataiqTXz4DdSVIXGloKiq     👈👈👈
+```
+
+Prepare directories for Persistent Volumes defined in `registry/pv.yaml`.
+
+```bash
+sudo mkdir -p /data/registry
+```
+
+### Deploy Private Container Registry
+
+Deploy Private Container Registry.
+
+```bash
+kubectl apply -k registry
+```
+
+Required resources has been deployed in `registry` namespace.
+
+```bash
+$ kubectl get all -n registry
+NAME                            READY   STATUS    RESTARTS   AGE
+pod/registry-5b4f874b77-9gb64   1/1     Running   0          27s
+
+NAME                       TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
+service/registry-service   ClusterIP   10.43.50.156   <none>        5000/TCP   28s
+
+NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
+deployment.apps/registry   1/1     1            1           27s
+
+NAME                                  DESIRED   CURRENT   READY   AGE
+replicaset.apps/registry-5b4f874b77   1         1         1       27s
+```
+
+Now your container registry can be used through `registry.example.com` or the hostname you specified.
+
+## Quick Testing
+
+### Testing with Docker
+
+Add your registry as an insecure registry and restart Docker daemon.
+
+```bash
+sudo tee /etc/docker/daemon.json <<EOF
+{
+  "insecure-registries" : ["registry.example.com"]
+}
+EOF
+sudo systemctl restart docker
+```
+
+Log in to your container registry.
+
+```bash
+$ docker login registry.example.com
+Username: reguser
+Password: 
+WARNING! Your password will be stored unencrypted in /home/********/.docker/config.json.
+Configure a credential helper to remove this warning. See
+https://docs.docker.com/engine/reference/commandline/login/#credentials-store
+
+Login Succeeded
+```
+
+Now you can push/pull the image to/from your container registry.
+
+```bash
+# Pull from docker.io
+docker pull docker.io/docker/whalesay:latest
+
+# Tag as your own image on your private container registry
+docker tag docker.io/docker/whalesay:latest registry.example.com/reguser/whalesay:latest
+
+# Push your own image to your private container registry
+docker push registry.example.com/reguser/whalesay:latest
+```
+
+```bash
+# Remove local images
+docker image rm docker.io/docker/whalesay:latest
+docker image rm registry.example.com/reguser/whalesay:latest
+
+# Pull the image from your private container registry
+docker pull registry.example.com/reguser/whalesay:latest
+```
+
+```bash
+$ docker run -it --rm registry.example.com/reguser/whalesay:latest cowsay hoge
+ ______ 
+< hoge >
+ ------ 
+    \
+     \
+      \     
+                    ##        .            
+              ## ## ##       ==            
+           ## ## ## ##      ===            
+       /""""""""""""""""___/ ===        
+  ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~   
+       \______ o          __/            
+        \    \        __/             
+          \____\______/   
+```
+
+### Digging into the Registry
+
+There is an useful CLI tool called [**reg**](https://github.com/genuinetools/reg) to dig into the container registry.
+
+```bash
+# Install reg
+sudo curl -fSL https://github.com/genuinetools/reg/releases/download/v0.16.1/reg-linux-amd64 -o /usr/local/bin/reg
+sudo chmod +x /usr/local/bin/reg
+
+# List repositories and tags in the container registry
+reg ls -k registry.example.com
+reg tags -k registry.example.com/reguser/whalesay
+
+# Delete tags on the registry
+reg rm -k registry.example.com/reguser/whalesay:latest
+```
+
+## (Optional) Use as Private Container Registry for K3s
+
+Optionally, this registry can also be registered as a private container registry for K3s.
+
+### Procedure
+
+To achieve this, create a `registries.yaml` and restart K3s.
+
+```bash
+sudo tee /etc/rancher/k3s/registries.yaml <<EOF
+configs:
+  registry.example.com:
+    auth:
+      username: reguser
+      password: Registry123!
+    tls:
+      insecure_skip_verify: true
+EOF
+
+# The K3s service can be safely restarted without affecting the running resources
+sudo systemctl restart k3s
+```
+
+If this is successfully applied, you can check the applied configuration in the `config.registry` section of the following command.
+
+```bash
+sudo /usr/local/bin/crictl info
+
+# With jq
+sudo /usr/local/bin/crictl info | jq .config.registry
+```
+
+### Testing
+
+You can launch your Pod using an image from a private repository that requires authentication.
+
+```bash
+$ kubectl run whalesay -it --restart=Never --image registry.example.com/reguser/whalesay:latest --rm -- cowsay hoge
+ ______ 
+< hoge >
+ ------ 
+    \
+     \
+      \     
+                    ##        .            
+              ## ## ##       ==            
+           ## ## ## ##      ===            
+       /""""""""""""""""___/ ===        
+  ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~   
+       \______ o          __/            
+        \    \        __/             
+          \____\______/   
+pod "whalesay" deleted
+```

registry/configmap.yaml

@@ -0,0 +1,35 @@
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: registry-configmap
+  labels:
+    app: registry
+data:
+  config.yml: |-
+    version: 0.1
+    log:
+      fields:
+        service: registry
+    storage:
+      cache:
+        blobdescriptor: inmemory
+      filesystem:
+        rootdirectory: /var/lib/registry
+      delete:
+        enabled: true
+    http:
+      addr: :5000
+      headers:
+        X-Content-Type-Options: [nosniff]
+    auth:
+      htpasswd:
+        realm: basic-realm
+        path: /etc/docker/registry/htpasswd
+    health:
+      storagedriver:
+        enabled: true
+        interval: 10s
+        threshold: 3
+  htpasswd: |-
+    reguser:$2y$05$VLMvcWCPF0VUuHi0BXBz7eoXGZ6KRl1gataiqTXz4DdSVIXGloKiq

registry/deployment.yaml

@@ -0,0 +1,36 @@
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: registry
+  labels:
+    app: registry
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: registry
+  template:
+    metadata:
+      labels:
+        app: registry
+    spec:
+      containers:
+        - name: registry
+          image: registry:2.7
+          ports:
+            - containerPort: 5000
+              protocol: TCP
+          volumeMounts:
+            - name: registry-volume
+              mountPath: /var/lib/registry
+            - name: registry-config
+              mountPath: /etc/docker/registry
+      volumes:
+        - name: registry-volume
+          persistentVolumeClaim:
+            claimName: registry-claim
+        - name: registry-config
+          configMap:
+            defaultMode: 420
+            name: registry-configmap

registry/ingress.yaml

@@ -0,0 +1,21 @@
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: registry-ingress
+spec:
+  tls:
+    - hosts:
+        - registry.example.com
+      secretName: registry-secret-tls
+  rules:
+    - host: registry.example.com
+      http:
+        paths:
+          - path: /
+            pathType: ImplementationSpecific
+            backend:
+              service:
+                name: registry-service
+                port:
+                  number: 5000

registry/kustomization.yaml

@@ -0,0 +1,23 @@
+---
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: registry
+
+generatorOptions:
+  disableNameSuffixHash: true
+
+secretGenerator:
+  - name: registry-secret-tls
+    type: kubernetes.io/tls
+    files:
+      - tls.crt
+      - tls.key
+
+resources:
+  - namespace.yaml
+  - pv.yaml
+  - pvc.yaml
+  - ingress.yaml
+  - service.yaml
+  - configmap.yaml
+  - deployment.yaml

registry/namespace.yaml

@@ -0,0 +1,5 @@
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: registry

registry/pv.yaml

@@ -0,0 +1,14 @@
+---
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  name: registry-volume
+spec:
+  accessModes:
+    - ReadWriteOnce
+  persistentVolumeReclaimPolicy: Retain
+  capacity:
+    storage: 5Gi
+  storageClassName: registry-volume
+  hostPath:
+    path: /data/registry

registry/pvc.yaml

@@ -0,0 +1,13 @@
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: registry-claim
+spec:
+  accessModes:
+    - ReadWriteOnce
+  volumeMode: Filesystem
+  resources:
+    requests:
+      storage: 5Gi
+  storageClassName: registry-volume

registry/service.yaml

@@ -0,0 +1,11 @@
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: registry-service
+spec:
+  ports:
+    - protocol: TCP
+      port: 5000
+  selector:
+    app: registry