awx-on-k3s/registry

Deploy Private Container Registry

Deploying your private container registry on your K3s to use with AWX.

Table of Contents

• Procedure
  • Prepare required files
  • Deploy Private Container Registry
• Quick Testing
  • Testing with Docker
  • Digging into the Registry
• Use as Private Container Registry for AWX or K3s
  • Procedure
  • Testing

Procedure

Prepare required files

Generate a Self-Signed Certificate. Note that IP address can't be specified.

REGISTRY_HOST="registry.example.com"
openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -out ./registry/tls.crt -keyout ./registry/tls.key -subj "/CN=${REGISTRY_HOST}/O=${REGISTRY_HOST}" -addext "subjectAltName = DNS:${REGISTRY_HOST}"

Modify hosts and host in registry/ingress.yaml.

...
    - hosts:
        - registry.example.com     👈👈👈
      secretName: registry-secret-tls
  rules:
    - host: registry.example.com     👈👈👈
...

Generate htpasswd string by your own username and password to use as the user for the container registry.

$ kubectl run htpasswd -it --restart=Never --image httpd:2.4 --rm -- htpasswd -nbB reguser Registry123!
reguser:$2y$05$VLMvcWCPF0VUuHi0BXBz7eoXGZ6KRl1gataiqTXz4DdSVIXGloKiq

pod "htpasswd" deleted

Replace htpasswd in registry/configmap.yaml with your own htpasswd string that generated above.

...
  htpasswd: |-
    reguser:$2y$05$VLMvcWCPF0VUuHi0BXBz7eoXGZ6KRl1gataiqTXz4DdSVIXGloKiq     👈👈👈    

Prepare directories for Persistent Volumes defined in registry/pv.yaml.

sudo mkdir -p /data/registry

Deploy Private Container Registry

Deploy Private Container Registry.

kubectl apply -k registry

Required resources has been deployed in registry namespace.

$ kubectl get all -n registry
NAME                            READY   STATUS    RESTARTS   AGE
pod/registry-5b4f874b77-9gb64   1/1     Running   0          27s

NAME                       TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/registry-service   ClusterIP   10.43.50.156   <none>        5000/TCP   28s

NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/registry   1/1     1            1           27s

NAME                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/registry-5b4f874b77   1         1         1       27s

Now your container registry can be used through registry.example.com or the hostname you specified.

Quick Testing

Testing with Docker

Add your registry as an insecure registry and restart Docker daemon.

sudo tee /etc/docker/daemon.json <<EOF
{
  "insecure-registries" : ["registry.example.com"]
}
EOF
sudo systemctl restart docker

Log in to your container registry.

$ docker login registry.example.com
Username: reguser
Password: 
WARNING! Your password will be stored unencrypted in /home/********/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

Now you can push/pull the image to/from your container registry.

# Pull from docker.io
docker pull docker.io/docker/whalesay:latest

# Tag as your own image on your private container registry
docker tag docker.io/docker/whalesay:latest registry.example.com/reguser/whalesay:latest

# Push your own image to your private container registry
docker push registry.example.com/reguser/whalesay:latest

# Remove local images
docker image rm docker.io/docker/whalesay:latest
docker image rm registry.example.com/reguser/whalesay:latest

# Pull the image from your private container registry
docker pull registry.example.com/reguser/whalesay:latest

$ docker run -it --rm registry.example.com/reguser/whalesay:latest cowsay hoge
 ______ 
< hoge >
 ------ 
    \
     \
      \     
                    ##        .            
              ## ## ##       ==            
           ## ## ## ##      ===            
       /""""""""""""""""___/ ===        
  ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~   
       \______ o          __/            
        \    \        __/             
          \____\______/   

Digging into the Registry

There is an useful CLI tool called reg to dig into the container registry.

# Install reg
sudo curl -fSL https://github.com/genuinetools/reg/releases/download/v0.16.1/reg-linux-amd64 -o /usr/local/bin/reg
sudo chmod +x /usr/local/bin/reg

# List repositories and tags in the container registry
reg ls -k registry.example.com
reg tags -k registry.example.com/reguser/whalesay

# Delete tags on the registry
reg rm -k registry.example.com/reguser/whalesay:latest

Use as Private Container Registry for AWX or K3s

This registry can be used not only as a registry to store Execution Environment for AWX, but also as a private registry for K3s.

Procedure

To achieve this, create a registries.yaml and restart K3s.

Note that required imagePullSecrets will be automatically created by AWX once you register valid Credential for your registry on AWX. Therefore, the auth section is only necessary if Kubernetes pulls the image directly without AWX, as in the following Testing procedure.

The tls section is required to disable SSL Verification as the endpoint is HTTPS with a Self-Signed Certificate.

sudo tee /etc/rancher/k3s/registries.yaml <<EOF
configs:
  registry.example.com:
    auth:
      username: reguser
      password: Registry123!
    tls:
      insecure_skip_verify: true
EOF

# The K3s service can be safely restarted without affecting the running resources
sudo systemctl restart k3s

If this is successfully applied, you can check the applied configuration in the config.registry section of the following command.

sudo /usr/local/bin/crictl info

# With jq
sudo /usr/local/bin/crictl info | jq .config.registry

If you want Kubernetes to be able to pull images directly from this private registry, alternatively you can also manually create imagePullSecrets for the Pod instead of writing your credentials in auth in registries.yaml.

Testing

You can launch your Pod using an image from a private repository that requires authentication.

$ kubectl run whalesay -it --restart=Never --image registry.example.com/reguser/whalesay:latest --rm -- cowsay hoge
 ______ 
< hoge >
 ------ 
    \
     \
      \     
                    ##        .            
              ## ## ##       ==            
           ## ## ## ##      ===            
       /""""""""""""""""___/ ===        
  ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~   
       \______ o          __/            
        \    \        __/             
          \____\______/   
pod "whalesay" deleted