awx-on-k3s/registry/README.md

<!-- omit in toc -->
# Deploy Private Container Registry

Deploying your private container registry on your K3s to use with AWX.

<!-- omit in toc -->
## Table of Contents

- [Procedure](#procedure)
  - [Prepare required files](#prepare-required-files)
  - [Deploy Private Container Registry](#deploy-private-container-registry)
- [Quick Testing](#quick-testing)
  - [Testing with Docker](#testing-with-docker)
  - [Digging into the Registry](#digging-into-the-registry)
- [(Optional) Use as Private Container Registry for K3s](#optional-use-as-private-container-registry-for-k3s)
  - [Procedure](#procedure-1)
  - [Testing](#testing)

## Procedure

### Prepare required files

Generate a Self-Signed Certificate. Note that IP address can't be specified.

```bash
REGISTRY_HOST="registry.example.com"
openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -out ./registry/tls.crt -keyout ./registry/tls.key -subj "/CN=${REGISTRY_HOST}/O=${REGISTRY_HOST}" -addext "subjectAltName = DNS:${REGISTRY_HOST}"
```

Modify `hosts` and `host` in `registry/ingress.yaml`.

```yaml
...
    - hosts:
        - registry.example.com     👈👈👈
      secretName: registry-secret-tls
  rules:
    - host: registry.example.com     👈👈👈
...
```

Generate `htpasswd` string by your own username and password to use as the user for the container registry.

```bash
$ kubectl run htpasswd -it --restart=Never --image httpd:2.4 --rm -- htpasswd -nbB reguser Registry123!
reguser:$2y$05$VLMvcWCPF0VUuHi0BXBz7eoXGZ6KRl1gataiqTXz4DdSVIXGloKiq

pod "htpasswd" deleted
```

Replace `htpasswd` in `registry/configmap.yaml` with your own `htpasswd` string that generated above.

```yaml
...
  htpasswd: |-
    reguser:$2y$05$VLMvcWCPF0VUuHi0BXBz7eoXGZ6KRl1gataiqTXz4DdSVIXGloKiq     👈👈👈    
```

Prepare directories for Persistent Volumes defined in `registry/pv.yaml`.

```bash
sudo mkdir -p /data/registry
```

### Deploy Private Container Registry

Deploy Private Container Registry.

```bash
kubectl apply -k registry
```

Required resources has been deployed in `registry` namespace.

```bash
$ kubectl get all -n registry
NAME                            READY   STATUS    RESTARTS   AGE
pod/registry-5b4f874b77-9gb64   1/1     Running   0          27s

NAME                       TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/registry-service   ClusterIP   10.43.50.156   <none>        5000/TCP   28s

NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/registry   1/1     1            1           27s

NAME                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/registry-5b4f874b77   1         1         1       27s
```

Now your container registry can be used through `registry.example.com` or the hostname you specified.

## Quick Testing

### Testing with Docker

Add your registry as an insecure registry and restart Docker daemon.

```bash
sudo tee /etc/docker/daemon.json <<EOF
{
  "insecure-registries" : ["registry.example.com"]
}
EOF
sudo systemctl restart docker
```

Log in to your container registry.

```bash
$ docker login registry.example.com
Username: reguser
Password: 
WARNING! Your password will be stored unencrypted in /home/********/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
```

Now you can push/pull the image to/from your container registry.

```bash
# Pull from docker.io
docker pull docker.io/docker/whalesay:latest

# Tag as your own image on your private container registry
docker tag docker.io/docker/whalesay:latest registry.example.com/reguser/whalesay:latest

# Push your own image to your private container registry
docker push registry.example.com/reguser/whalesay:latest
```

```bash
# Remove local images
docker image rm docker.io/docker/whalesay:latest
docker image rm registry.example.com/reguser/whalesay:latest

# Pull the image from your private container registry
docker pull registry.example.com/reguser/whalesay:latest
```

```bash
$ docker run -it --rm registry.example.com/reguser/whalesay:latest cowsay hoge
 ______ 
< hoge >
 ------ 
    \
     \
      \     
                    ##        .            
              ## ## ##       ==            
           ## ## ## ##      ===            
       /""""""""""""""""___/ ===        
  ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~   
       \______ o          __/            
        \    \        __/             
          \____\______/   
```

### Digging into the Registry

There is an useful CLI tool called [**reg**](https://github.com/genuinetools/reg) to dig into the container registry.

```bash
# Install reg
sudo curl -fSL https://github.com/genuinetools/reg/releases/download/v0.16.1/reg-linux-amd64 -o /usr/local/bin/reg
sudo chmod +x /usr/local/bin/reg

# List repositories and tags in the container registry
reg ls -k registry.example.com
reg tags -k registry.example.com/reguser/whalesay

# Delete tags on the registry
reg rm -k registry.example.com/reguser/whalesay:latest
```

## (Optional) Use as Private Container Registry for K3s

Optionally, this registry can also be registered as a private container registry for K3s.

### Procedure

To achieve this, create a `registries.yaml` and restart K3s.

```bash
sudo tee /etc/rancher/k3s/registries.yaml <<EOF
configs:
  registry.example.com:
    auth:
      username: reguser
      password: Registry123!
    tls:
      insecure_skip_verify: true
EOF

# The K3s service can be safely restarted without affecting the running resources
sudo systemctl restart k3s
```

If this is successfully applied, you can check the applied configuration in the `config.registry` section of the following command.

```bash
sudo /usr/local/bin/crictl info

# With jq
sudo /usr/local/bin/crictl info | jq .config.registry
```

### Testing

You can launch your Pod using an image from a private repository that requires authentication.

```bash
$ kubectl run whalesay -it --restart=Never --image registry.example.com/reguser/whalesay:latest --rm -- cowsay hoge
 ______ 
< hoge >
 ------ 
    \
     \
      \     
                    ##        .            
              ## ## ##       ==            
           ## ## ## ##      ===            
       /""""""""""""""""___/ ===        
  ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~   
       \______ o          __/            
        \    \        __/             
          \____\______/   
pod "whalesay" deleted
```