feat: add workaround for rate limit on docker hub

README.md

@@ -438,4 +438,5 @@ kubectl apply -f awx-secret-tls.yaml
   - [📝Uninstall deployed resouces](tips/uninstall.md)
   - [📝Deploy older version of AWX Operator](tips/deploy-older-operator.md)
   - [📝Upgrade AWX Operator and AWX](tips/upgrade-operator.md)
+  - [📝Workaround for the rate limit on Docker Hub](tips/dockerhub-rate-limit.md)
   - [📝Troubleshooting Guide](tips/troubleshooting.md)

registry/README.md

@@ -210,7 +210,7 @@ sudo /usr/local/bin/crictl info
 sudo /usr/local/bin/crictl info | jq .config.registry
 ```
 
-If you want Kubernetes to be able to pull images directly from this private registry, alternatively you can also manually create `imagePullSecrets` for the Pod instead of writing your credentials in `auth` in `registries.yaml`.
+If you want Kubernetes to be able to pull images directly from this private registry, alternatively you can also manually create `imagePullSecrets` for the Pod instead of writing your credentials in `auth` in `registries.yaml`. [Another guide about rate limiting on Docker Hub](../tips/dockerhub-rate-limit.md) explains how to use `ImagePullSecrets`.
 
 ### Testing
 

tips/README.md

@@ -6,4 +6,5 @@
 - [📝Uninstall deployed resouces](uninstall.md)
 - [📝Deploy older version of AWX Operator](deploy-older-operator.md)
 - [📝Upgrade AWX Operator and AWX](upgrade-operator.md)
+- [📝Workaround for the rate limit on Docker Hub](dockerhub-rate-limit.md)
 - [📝Troubleshooting Guide](troubleshooting.md)

tips/dockerhub-rate-limit.md

@@ -0,0 +1,158 @@
+<!-- omit in toc -->
+# Workaround for the rate limit on Docker Hub
+
+If your Pod for PostgreSQL is in `ErrImagePull` and its `Events` shows following events, this is due to [the Rate Limit on Docker Hub](https://docs.docker.com/docker-hub/download-rate-limit/).
+
+```bash
+$ kubectl -n awx describe pod awx-postgres-0
+...
+Events:
+  Type     Reason            Age   From               Message
+  ----     ------            ----  ----               -------
+  ...
+  Warning  Failed            2s    kubelet            Failed to pull image "postgres:12": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/postgres:12": failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/postgres/manifests/sha256:...: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
+  ...
+```
+
+If you just follow the steps in this repository to deploy you AWX, your pull request to Docker Hub will be identified as a free, anonymous account. Therefore, you will be limited to 200 requests in 6 hours. The message "429 Too Many Requests" indicates that the limit has been exceeded. To solve this, you should pass your Docker Hub credentials to the Pod as ImagePullSecrets.
+
+<!-- omit in toc -->
+## Table of Contents
+
+- [Procedure](#procedure)
+  - [Create `base/config.json`](#create-baseconfigjson)
+  - [Modify `base/kustomization.yaml`](#modify-basekustomizationyaml)
+  - [Modify `base/awx.yaml`](#modify-baseawxyaml)
+  - [Next Step](#next-step)
+- [Appendix: Create Secret for ImagePullSecrets by Hand](#appendix-create-secret-for-imagepullsecrets-by-hand)
+- [Appendix: Modify ImagePullSecrets for the Specific Service Account](#appendix-modify-imagepullsecrets-for-the-specific-service-account)
+- [Appendix: Apply Docker Hub Credential at Cluster Level](#appendix-apply-docker-hub-credential-at-cluster-level)
+
+## Procedure
+
+There are several ways to achieve this, but this guide will show you an example of using Kustomize to save all your configuration as code. In addition to [the main guide](../), a few additional files need to be modified.
+
+### Create `base/config.json`
+
+First, prepare JSON file named `config.json` under your `base` directory.
+
+```bash
+$ DOCKERHUB_USERNAME=<Your Username for Docker Hub>
+$ DOCKERHUB_PASSWORD=<Your Password or Personal Access Token for Docker Hub>
+$ DOCKERHUB_AUTH=$(echo -n "${DOCKERHUB_USERNAME}:${DOCKERHUB_PASSWORD}" | base64)
+$ cat <<EOF > base/config.json
+{
+    "auths": {
+        "docker.io": {
+            "auth": "${DOCKERHUB_AUTH}"
+        }
+    }
+}
+EOF
+```
+
+Ensure your `config.json` includes base64-encoded string as following. Note that this base64-encoded string includes your password in plain text and easily revealed by `echo "<Encoded String>" | base64 --decode` command.
+
+```bash
+$ cat base/config.json
+{
+    "auths": {
+        "docker.io": {
+            "auth": "ZXhhbXBsZS...MtdG9rZW4K"
+        }
+    }
+}
+```
+
+### Modify `base/kustomization.yaml`
+
+Then, add following four lines to under `secretGenerator` in `base/kustomization.yaml`.
+
+```yaml
+...
+secretGenerator:
+  ...
+  - name: awx-registry-secret     👈👈👈
+    type: kubernetes.io/dockerconfigjson     👈👈👈
+    files:     👈👈👈
+      - .dockerconfigjson=config.json     👈👈👈
+  ...
+resources:
+  ...
+```
+
+### Modify `base/awx.yaml`
+
+Finally, add following line to `base/awx.yaml`.
+
+```yaml
+...
+spec:
+  ...
+  image_pull_secret: awx-registry-secret     👈👈👈
+  ...
+```
+
+### Next Step
+
+Now everything is ready to deploy, go back to [the main guide (`README.md`)](../) and run `kubectl apply -k base`.
+
+## Appendix: Create Secret for ImagePullSecrets by Hand
+
+If you want to create Secret manually instead of using Kustomize, you can choose alternative way. If you don't have `awx` namespace, create it first by `kubectl create ns awx`.
+
+- **Method 1**: Create Secret by using existing `~/.docker/config.json`
+
+  ```bash
+  # Log in to Docker Hub via docker command first,
+  $ docker login
+
+  # Then create Secret by specifying ~/.docker/config.json which generated by Docker
+  $ kubectl -n awx create secret generic awx-registry-secret \
+      --from-file=.dockerconfigjson=${HOME}/.docker/config.json \
+      --type=kubernetes.io/dockerconfigjson
+  ```
+
+- **Method 2**: Create Secret by specifying username and password manually
+
+  ```bash
+  # Create Secret by specifying username and password manually
+  $ DOCKERHUB_USERNAME=<Your Username for Docker Hub>
+  $ DOCKERHUB_PASSWORD=<Your Password or Personal Access Token for Docker Hub>
+  $ kubectl -n awx create secret docker-registry awx-registry-secret \
+      --docker-server=docker.io \
+      --docker-username=${DOCKERHUB_USERNAME} \
+      --docker-password=${DOCKERHUB_PASSWORD}
+  ```
+
+## Appendix: Modify ImagePullSecrets for the Specific Service Account
+
+Once create new namespace, the default Service Account named `default` is also created by default. This `default` Service Account is used when no Service Account has been specified for the Pod, and you can modify ImagePullSecrets which used by default for this `default` Service Account too. This can be applied to [Private Git Repository](../git) and [Private Container Registry](../registry) included in this repository. Additionally, this can also be applied to the Pod for PostgreSQL created by AWX Operator, since the Service Account is not specified.
+
+First, you should create Secret for the Service Account by referring [Appendix: Create Secret for ImagePullSecrets by Hand](#appendix-create-secret-for-imagepullsecrets-by-hand). Note that the namespace and the name of the Secret in the command should be changed to suit your environment.
+
+Then patch the `default` service account.
+
+```bash
+kubectl -n <Your Namespace> patch serviceaccount default -p '{"imagePullSecrets": [{"name": "<Your Secret>"}]}'
+```
+
+## Appendix: Apply Docker Hub Credential at Cluster Level
+
+You can also change the entire K3s configuration so that the specific credential for Docker Hub is always used, regardless of the namespace. Create `/etc/rancher/k3s/registries.yaml` and restart K3s service as follows.
+
+```bash
+# Create /etc/rancher/k3s/registries.yaml including your credential
+$ DOCKERHUB_USERNAME=<Your Username for Docker Hub>
+$ DOCKERHUB_PASSWORD=<Your Password or Personal Access Token for Docker Hub>
+$ sudo tee /etc/rancher/k3s/registries.yaml <<EOF
+configs:
+  registry-1.docker.io:
+    auth:
+      username: ${DOCKERHUB_USERNAME}
+      password: ${DOCKERHUB_PASSWORD}
+EOF
+
+# Then restart K3s. The K3s service can be safely restarted without affecting the running resources
+$ sudo systemctl restart k3s
+```

tips/troubleshooting.md

@@ -10,6 +10,7 @@ Some hints and guides for when you got stuck during deployment and daily use of
   - [First Step: Investigate your Situation](#first-step-investigate-your-situation)
     - [Investigate Status and Events of the Pods](#investigate-status-and-events-of-the-pods)
     - [Investigate Logs of the Containers inside the Pods](#investigate-logs-of-the-containers-inside-the-pods)
+  - [The Pod is `ErrImagePull` with "429 Too Many Requests"](#the-pod-is-errimagepull-with-429-too-many-requests)
   - [The Pod is `Pending` with "1 Insufficient cpu, 1 Insufficient memory." event](#the-pod-is-pending-with-1-insufficient-cpu-1-insufficient-memory-event)
   - [The Pod is `Pending` with "1 pod has unbound immediate PersistentVolumeClaims." event](#the-pod-is-pending-with-1-pod-has-unbound-immediate-persistentvolumeclaims-event)
   - [The Pod is `Running` but stucked with "[wait-for-migrations] Waiting for database migrations..." message](#the-pod-is-running-but-stucked-with-wait-for-migrations-waiting-for-database-migrations-message)
@@ -101,6 +102,27 @@ For AWX Operator and AWX, specifically, the following commands are helpful.
 - Logs of PostgreSQL
   - `kubectl -n awx logs -f statefulset/awx-postgres`
 
+### The Pod is `ErrImagePull` with "429 Too Many Requests"
+
+If your Pod for PostgreSQL is in `ErrImagePull` and its `Events` shows following events, this is due to [the rate limit on Docker Hub](https://docs.docker.com/docker-hub/download-rate-limit/).
+
+```bash
+$ kubectl -n awx describe pod awx-postgres-0
+...
+Events:
+  Type     Reason            Age   From               Message
+  ----     ------            ----  ----               -------
+  Normal   Pulling           9s    kubelet            Pulling image "postgres:12"
+  Warning  Failed            2s    kubelet            Failed to pull image "postgres:12": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/postgres:12": failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/postgres/manifests/sha256:505d023f030cdea84a42d580c2a4a0e17bbb3e91c30b2aea9c02f2dfb10325ba: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
+  Warning  Failed            2s    kubelet            Error: ErrImagePull
+  Normal   BackOff           1s    kubelet            Back-off pulling image "postgres:12"
+  Warning  Failed            1s    kubelet            Error: ImagePullBackOff
+```
+
+If you follow the steps in this repository to deploy you AWX, your pull request to Docker Hub will be identified as a free, anonymous account. Therefore, you will be limited to 200 requests in 6 hours. The message "429 Too Many Requests" indicates that it has been exceeded.
+
+To solve this, you can simply wait until the limit is freeed up, or [consider giving your Docker Hub credentials to K3s by follwing the guide on this page](dockerhub-rate-limit.md).
+
 ### The Pod is `Pending` with "1 Insufficient cpu, 1 Insufficient memory." event
 
 If your Pod is in `Pending` state and its `Events` shows following events, the reason is that the node does not have enough CPU and memory to start the Pod. By default AWX requires at least 2 CPUs and 4 GB RAM. In addition more resources are required to run K3s and the OS itself.