feat: add instructions for private git repo, builder, and runner

.gitignore

@@ -1,2 +1,3 @@
+debug
 *.crt
 *.key

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2021 kurokobo
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -26,7 +26,13 @@ An example implementation of AWX on single node K3s using AWX Operator, with eas
   - [Restoring using AWX Operator](#restoring-using-awx-operator)
     - [Prepare for Restore](#prepare-for-restore)
     - [Invoke Manual Restore](#invoke-manual-restore)
+- [Deploy Private Git Repository](#deploy-private-git-repository)
 - [Deploy Private Container Registry](#deploy-private-container-registry)
+- [Use Ansible Builder](#use-ansible-builder)
+- [Use Ansible Runner](#use-ansible-runner)
+- [Additional Configuration for AWX](#additional-configuration-for-awx)
+  - [Configure AWX to use Git Repository with Self-Signed Certificate](#configure-awx-to-use-git-repository-with-self-signed-certificate)
+  - [Expose your /etc/hosts to K3s](#expose-your-etchosts-to-k3s)
 
 ## Environment
 
@@ -327,10 +333,82 @@ Then restore the Secret for TLS manually (or create newly using original certifi
 kubectl apply -f awx-secret-tls.yaml
 ```
 
+## Deploy Private Git Repository
+
+To use AWX with SCM, this repository includes the manifests to deploy [Gitea](https://gitea.io/en-us/).
+
+See [📝`git/README.md`](git) for instructions.
+
 ## Deploy Private Container Registry
 
 To use Execution Environments in AWX (AWX-EE), we have to push the container image built with `ansible-builder` to the container registry.
 
 If we don't want to push our container images to Docker Hub or other cloud services, we can deploy a private container registry on K3s.
 
-See [📝 `registry/README.md`](registry/README.md) for instructions.
+See [📝`registry/README.md`](registry) for instructions.
+
+## Use Ansible Builder
+
+See [📝`builder/README.md`](builder) for instructions.
+
+## Use Ansible Runner
+
+See [📝`runner/README.md`](runner) for instructions.
+
+## Additional Configuration for AWX
+
+### Configure AWX to use Git Repository with Self-Signed Certificate
+
+1. Add Credentials for SCM
+2. Allow Self-Signed Certificate such as this Gitea
+   - Open `Settings` > `Jobs settings` in AWX
+   - Press `Edit` and scroll down to `Extra Environment Variables`, then add `"GIT_SSL_NO_VERIFY": "True"` in `{}`
+   - Press `Save`
+
+### Expose your /etc/hosts to K3s
+
+If we don't have a DNS server and are using `/etc/hosts`, we will need to do some additional tasks to get the Pods on K3s to resolve names according to `/etc/hosts`.
+
+This is necessary for AWX to resolve the hostname for your Private Git Repository or pull images from the Container Registry.
+
+One easy way to do this is to use `dnsmasq`.
+
+1. Add entries to `/etc/hosts` on your K3s host. Note that the IP addresses have to be replaced with your K3s host's one.
+
+   ```bash
+   sudo echo "192.168.0.100 awx.example.com" >> /etc/hosts
+   sudo echo "192.168.0.100 registry.example.com" >> /etc/hosts
+   sudo echo "192.168.0.100 git.example.com" >> /etc/hosts
+   ```
+
+2. Install and start `dnsmasq` with default configuration.
+
+   ```bash
+   sudo dnf install dnsmasq
+   sudo systemctl enable dnsmasq --now
+   ```
+
+3. Create new `resolv.conf` to use K3s. Note that the IP addresses have to be replaced with your K3s host's one.
+
+   ```bash
+   sudo echo "nameserver 192.168.0.100" > /etc/rancher/k3s/resolv.conf
+   ```
+
+4. Add `--resolv-conf /etc/rancher/k3s/resolv.conf` as an argument for `k3s server` command.
+
+   ```bash
+   $ cat /etc/systemd/system/k3s.service
+   ...
+   ExecStart=/usr/local/bin/k3s \
+       server \
+           '--write-kubeconfig-mode' \
+           '644' \
+           '--resolv-conf' \     👈👈👈
+           '/etc/rancher/k3s/resolv.conf' \     👈👈👈
+   ```
+
+5. Restart K3s.
+
+   ```bash
+   sudo systemctl restart k3s
+   ```

builder/.gitignore

@@ -0,0 +1 @@
+context

builder/README.md

@@ -0,0 +1,96 @@
+# Examples of Ansible Builder
+
+- [ansible/ansible-builder](https://github.com/ansible/ansible-builder)
+- [Introduction — ansible-builder documentation](https://ansible-builder.readthedocs.io/en/latest/index.html)
+
+## Environment in This Example
+
+- CentOS 8.2
+- Python 3.9
+- Docker 20.10.7
+
+## Install
+
+```bash
+python3 -m pip install ansible-builder
+```
+
+## Prepare Required Files
+
+`execution-environment.yml` is required file to build Execution Environment.
+
+The base image can be chosen from the tags from [http://quay.io/ansible/ansible-runner](https://quay.io/repository/ansible/ansible-runner?tab=tags).
+
+## Build Execution Environment
+
+`ansible-builder build` command builds Execution Environment as a container image according to the definition in `execution-environment.yml`.
+
+```bash
+ansible-builder build --tag registry.example.com/ansible/ee:2.10-custom --container-runtime docker --verbosity 3
+```
+
+```bash
+$ ansible-builder build --tag registry.example.com/ansible/ee:2.10-custom --container-runtime docker --verbosity 3
+Ansible Builder is building your execution environment image, "registry.example.com/ansible/ee:2.10-custom".
+File context/_build/requirements.yml will be created.
+File context/_build/requirements.txt will be created.
+File context/_build/bindep.txt will be created.
+File context/_build/ansible.cfg will be created.
+Rewriting Containerfile to capture collection requirements
+Running command:
+  docker build -f context/Dockerfile -t registry.example.com/ansible/ee:2.10-custom context
+Sending build context to Docker daemon   7.68kB
+Step 1/25 : ARG EE_BASE_IMAGE=quay.io/ansible/ansible-runner:stable-2.10-devel
+Step 2/25 : ARG EE_BUILDER_IMAGE=quay.io/ansible/ansible-builder:latest
+Step 3/25 : FROM $EE_BASE_IMAGE as galaxy
+...
+Removing intermediate container a083001a665a
+ ---> 050cf7076379
+Successfully built 050cf7076379
+Successfully tagged registry.example.com/ansible/ee:2.10-custom
+
+Complete! The build context can be found at: /home/********/awx-on-k3s/builder/context
+```
+
+```bash
+$ docker image ls
+REPOSITORY                              TAG                 IMAGE ID       CREATED         SIZE
+registry.example.com/ansible/ee         2.10-custom         6fb343319a80   2 minutes ago   871MB
+```
+
+Now you can push this image to your [private container registry](../registry/README.md) to use as Execution Environment on AWX.
+
+```bash
+$ docker push registry.example.com/ansible/ee:2.10-custom
+The push refers to repository [registry.example.com/ansible/ee]
+...
+2.10-custom: digest: sha256:0138445c58253c733f2e255b618469d9f61337901c13e3be6412984fd835ad55 size: 3880
+```
+
+The `Dockerfile` generated for the build will be saved under the `context` directory.
+
+```bash
+$ cat context/Dockerfile
+ARG EE_BASE_IMAGE=quay.io/ansible/ansible-runner:stable-2.10-devel
+ARG EE_BUILDER_IMAGE=quay.io/ansible/ansible-builder:latest
+
+FROM $EE_BASE_IMAGE as galaxy
+ARG ANSIBLE_GALAXY_CLI_COLLECTION_OPTS=
+USER root
+...
+```
+
+## Create Context Files
+
+`ansible-builder create` command only creates context files like `Dockerfile`, it does not invoke building the image.
+
+```bash
+$ ansible-builder create --verbosity 3
+Ansible Builder is generating your execution environment build context.
+File context/_build/requirements.yml will be created.
+File context/_build/requirements.txt will be created.
+File context/_build/bindep.txt will be created.
+File context/_build/ansible.cfg will be created.
+Rewriting Containerfile to capture collection requirements
+Complete! The build context can be found at: /home/********/awx-on-k3s/builder/context
+```

builder/ansible.cfg

@@ -0,0 +1,2 @@
+[defaults]
+host_key_checking = False

builder/bindep.txt

@@ -0,0 +1 @@
+bind-utils

builder/execution-environment.yml

@@ -0,0 +1,20 @@
+---
+version: 1
+
+build_arg_defaults:
+  EE_BASE_IMAGE: quay.io/ansible/ansible-runner:stable-2.10-devel
+
+ansible_config: ansible.cfg
+
+dependencies:
+  galaxy: requirements.yml
+  python: requirements.txt
+  system: bindep.txt
+
+additional_build_steps:
+  prepend:
+    - RUN whoami
+    - RUN cat /etc/os-release
+  append:
+    - RUN echo This is a post-install command!
+    - RUN ls -la /etc

builder/requirements.txt

@@ -0,0 +1 @@
+example-pypi-package

builder/requirements.yml

@@ -0,0 +1,5 @@
+---
+collections:
+  - name: community.general
+    version: 3.3.2
+    source: https://galaxy.ansible.com

git/README.md

@@ -0,0 +1,76 @@
+<!-- omit in toc -->
+# Deploy Private Git Repository using Gitea
+
+Deploying your private Git repository using [Gitea](https://gitea.io/en-us/) to use AWX with the playbooks on SCM.
+
+Note that this sample manifest does not include any databases, so the SQLite3 has to be selected as `Database Type` for Gitea.
+
+<!-- omit in toc -->
+## Table of Contents
+
+- [Procedure](#procedure)
+  - [Prepare required files](#prepare-required-files)
+  - [Deploy Private Git Repository](#deploy-private-git-repository)
+
+## Procedure
+
+### Prepare required files
+
+Generate a Self-Signed Certificate. Note that IP address can't be specified.
+
+```bash
+GIT_HOST="git.example.com"
+openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -out ./git/tls.crt -keyout ./git/tls.key -subj "/CN=${GIT_HOST}/O=${GIT_HOST}" -addext "subjectAltName = DNS:${GIT_HOST}"
+```
+
+Modify `hosts` and `host` in `git/ingress.yaml`.
+
+```yaml
+...
+    - hosts:
+        - git.example.com     👈👈👈
+      secretName: git-secret-tls
+  rules:
+    - host: git.example.com     👈👈👈
+...
+```
+
+Prepare directories for Persistent Volumes defined in `git/pv.yaml`.
+
+```bash
+sudo mkdir -p /data/git
+```
+
+### Deploy Private Git Repository
+
+Deploy Private Git Repository.
+
+```bash
+kubectl apply -k git
+```
+
+Required resources has been deployed in `git` namespace.
+
+```bash
+$ kubectl get all -n git
+NAME                       READY   STATUS    RESTARTS   AGE
+pod/git-576868dc5b-z7z55   1/1     Running   0          31s
+
+NAME                  TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)           AGE
+service/git-service   ClusterIP   10.43.192.81   <none>        3000/TCP,22/TCP   31s
+
+NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
+deployment.apps/git   1/1     1            1           31s
+
+NAME                             DESIRED   CURRENT   READY   AGE
+replicaset.apps/git-576868dc5b   1         1         1       31s
+```
+
+Now your Git repository is accesible through `https://git.example.com/` or the hostname you specified. Visit the URL and follow the installation wizard.
+
+Note that this sample manifest does not include any databases, so the SQLite3 has to be selected as `Database Type` for Gitea.
+
+| Configration   | Recommemded Value                                        |
+| -------------- | -------------------------------------------------------- |
+| Database Type  | `SQLite3`                                                |
+| Gitea Base URL | `https://git.example.com/` or the hostname you specified |

git/deployment.yaml

@@ -0,0 +1,32 @@
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: git
+  labels:
+    app: git
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: git
+  template:
+    metadata:
+      labels:
+        app: git
+    spec:
+      containers:
+        - name: git
+          image: gitea/gitea:1.14
+          ports:
+            - name: gitea-http
+              containerPort: 3000
+            - name: gitea-ssh
+              containerPort: 22
+          volumeMounts:
+            - name: git-volume
+              mountPath: /data
+      volumes:
+        - name: git-volume
+          persistentVolumeClaim:
+            claimName: git-claim

git/ingress.yaml

@@ -0,0 +1,21 @@
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: git-ingress
+spec:
+  tls:
+    - hosts:
+        - git.example.com
+      secretName: git-secret-tls
+  rules:
+    - host: git.example.com
+      http:
+        paths:
+          - path: /
+            pathType: ImplementationSpecific
+            backend:
+              service:
+                name: git-service
+                port:
+                  number: 3000

git/kustomization.yaml

@@ -0,0 +1,22 @@
+---
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: git
+
+generatorOptions:
+  disableNameSuffixHash: true
+
+secretGenerator:
+  - name: git-secret-tls
+    type: kubernetes.io/tls
+    files:
+      - tls.crt
+      - tls.key
+
+resources:
+  - namespace.yaml
+  - pv.yaml
+  - pvc.yaml
+  - ingress.yaml
+  - service.yaml
+  - deployment.yaml

git/namespace.yaml

@@ -0,0 +1,5 @@
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: git

git/pv.yaml

@@ -0,0 +1,14 @@
+---
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  name: git-volume
+spec:
+  accessModes:
+    - ReadWriteOnce
+  persistentVolumeReclaimPolicy: Retain
+  capacity:
+    storage: 5Gi
+  storageClassName: git-volume
+  hostPath:
+    path: /data/git

git/pvc.yaml

@@ -0,0 +1,13 @@
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: git-claim
+spec:
+  accessModes:
+    - ReadWriteOnce
+  volumeMode: Filesystem
+  resources:
+    requests:
+      storage: 2Gi
+  storageClassName: git-volume

git/service.yaml

@@ -0,0 +1,15 @@
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: git-service
+spec:
+  ports:
+    - name: gitea-http
+      protocol: TCP
+      port: 3000
+    - name: gitea-ssh
+      protocol: TCP
+      port: 22
+  selector:
+    app: git

registry/README.md

@@ -12,7 +12,7 @@ Deploying your private container registry on your K3s to use with AWX.
 - [Quick Testing](#quick-testing)
   - [Testing with Docker](#testing-with-docker)
   - [Digging into the Registry](#digging-into-the-registry)
-- [(Optional) Use as Private Container Registry for K3s](#optional-use-as-private-container-registry-for-k3s)
+- [Use as Private Container Registry for K3s](#use-as-private-container-registry-for-k3s)
   - [Procedure](#procedure-1)
   - [Testing](#testing)
 
@@ -174,9 +174,9 @@ reg tags -k registry.example.com/reguser/whalesay
 reg rm -k registry.example.com/reguser/whalesay:latest
 ```
 
-## (Optional) Use as Private Container Registry for K3s
+## Use as Private Container Registry for K3s
 
-Optionally, this registry can also be registered as a private container registry for K3s.
+This registry can also be registered as a private container registry for K3s. This is required procedure to use this registry to store images for AWX Execution Environment.
 
 ### Procedure
 

runner/.gitignore

@@ -0,0 +1 @@
+artifacts

runner/README.md

@@ -0,0 +1,39 @@
+# Examples of Ansible Runner
+
+- [ansible/ansible-runner](https://github.com/ansible/ansible-runner)
+- [Ansible Runner — ansible-runner 2.0.1 documentation](https://ansible-runner.readthedocs.io/en/stable/index.html)
+
+## Environment in This Example
+
+- CentOS 8.2
+- Python 3.9
+- Docker 20.10.7
+
+## Install
+
+```bash
+python3 -m pip install ansible-runner
+```
+
+## Prepare Required Files
+
+Refer [`projects/demo.yml`](projects/demo.yml) and [`env/settings`](env/settings).
+
+Your own Execution Environment can be used using `container_image` in [`env/settings`](env/settings).
+
+## Invoke Ansible Runner
+
+The sample playbook will help you figure out the differences between Execution Environments, such as Ansible version, `pip list`, etc.
+
+```bash
+ansible-runner run . -p demo.yml
+```
+
+The settings can also be given directly as arguments like `--container-image` without using `env/settings`.
+
+## Tips
+
+- If `process_isolation` is set to `false` as default, `ansible-runner` will just work as a wrapper for the local `ansible-playbook` command.
+- If `process_isolation` is set to `true`, `connection: local` points to the container itself, so the playbook cannot affect `localhost` which means the container host. To avoid this, you need to SSH to local host without using `connection: local`. [Related issues is here](https://github.com/ansible/ansible-runner/issues/752).
+- `container_image` defaults to `quay.io/ansible/ansible-runner:devel`. You can check available tags for the public Execution Environment at [`quay.io/ansible/ansible-runner`](https://quay.io/repository/ansible/ansible-runner?tab=tags)
+- The `process_isolation_show_paths` described in the documentation does not work with Docker and Podman. Instead, you can use `container_volume_mounts` as described in [`env/settings`](env/settings), but this is not documented.

runner/env/settings

@@ -0,0 +1,9 @@
+---
+process_isolation: true
+process_isolation_executable: docker
+container_image: registry.example.com/ansible/ee:2.10-custom
+
+# Specify additional bind-mount points to an isolated Docker or Podman container.
+# Note that this is UNDOCUMENTED option.
+#container_volume_mounts:
+#  - /etc/sample:/etc/sample

runner/project/demo.yml

@@ -0,0 +1,38 @@
+---
+- hosts: localhost
+  connection: local
+
+  tasks:
+    - name: Ensure that the host is reachable
+      ansible.builtin.ping:
+
+    - name: Print variables for debugging
+      ansible.builtin.debug:
+        var: data
+      vars:
+        data:
+          ansible_playbook_python: "{{ ansible_playbook_python }}"
+          ansible_python_version: "{{ ansible_python_version }}"
+          ansible_python.executable: "{{ ansible_python.executable }}"
+          ansible_version.full: "{{ ansible_version.full }}"
+
+    - name: Invoke debug commands
+      ansible.builtin.command: "{{ item }}"
+      changed_when: false
+      failed_when: false
+      loop:
+        - hostname
+        - whoami
+        - pwd
+        - python3 -m pip list
+        - ansible-galaxy collection list -p .
+      register: command_results
+    - ansible.builtin.debug:
+        msg: "{{ item.stdout_lines }}"
+      loop: "{{ command_results.results }}"
+      loop_control:
+        label: "{{ item.cmd }}"
+
+    # - name: Simply pause for a while
+    #   ansible.builtin.pause:
+    #     minutes: 10