diff --git a/README.md b/README.md index 5570706..a58ba2c 100644 --- a/README.md +++ b/README.md @@ -438,4 +438,5 @@ kubectl apply -f awx-secret-tls.yaml - [📝Uninstall deployed resouces](tips/uninstall.md) - [📝Deploy older version of AWX Operator](tips/deploy-older-operator.md) - [📝Upgrade AWX Operator and AWX](tips/upgrade-operator.md) + - [📝Workaround for the rate limit on Docker Hub](tips/dockerhub-rate-limit.md) - [📝Troubleshooting Guide](tips/troubleshooting.md) diff --git a/registry/README.md b/registry/README.md index d99a3d5..98b13dd 100644 --- a/registry/README.md +++ b/registry/README.md @@ -210,7 +210,7 @@ sudo /usr/local/bin/crictl info sudo /usr/local/bin/crictl info | jq .config.registry ``` -If you want Kubernetes to be able to pull images directly from this private registry, alternatively you can also manually create `imagePullSecrets` for the Pod instead of writing your credentials in `auth` in `registries.yaml`. +If you want Kubernetes to be able to pull images directly from this private registry, alternatively you can also manually create `imagePullSecrets` for the Pod instead of writing your credentials in `auth` in `registries.yaml`. [Another guide about rate limiting on Docker Hub](../tips/dockerhub-rate-limit.md) explains how to use `ImagePullSecrets`. ### Testing diff --git a/tips/README.md b/tips/README.md index 893b2cc..fc9be84 100644 --- a/tips/README.md +++ b/tips/README.md @@ -6,4 +6,5 @@ - [📝Uninstall deployed resouces](uninstall.md) - [📝Deploy older version of AWX Operator](deploy-older-operator.md) - [📝Upgrade AWX Operator and AWX](upgrade-operator.md) +- [📝Workaround for the rate limit on Docker Hub](dockerhub-rate-limit.md) - [📝Troubleshooting Guide](troubleshooting.md) diff --git a/tips/dockerhub-rate-limit.md b/tips/dockerhub-rate-limit.md new file mode 100644 index 0000000..037e751 --- /dev/null +++ b/tips/dockerhub-rate-limit.md @@ -0,0 +1,158 @@ + +# Workaround for the rate limit on Docker Hub + +If your Pod for PostgreSQL is in `ErrImagePull` and its `Events` shows following events, this is due to [the Rate Limit on Docker Hub](https://docs.docker.com/docker-hub/download-rate-limit/). + +```bash +$ kubectl -n awx describe pod awx-postgres-0 +... +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + ... + Warning Failed 2s kubelet Failed to pull image "postgres:12": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/postgres:12": failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/postgres/manifests/sha256:...: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit + ... +``` + +If you just follow the steps in this repository to deploy you AWX, your pull request to Docker Hub will be identified as a free, anonymous account. Therefore, you will be limited to 200 requests in 6 hours. The message "429 Too Many Requests" indicates that the limit has been exceeded. To solve this, you should pass your Docker Hub credentials to the Pod as ImagePullSecrets. + + +## Table of Contents + +- [Procedure](#procedure) + - [Create `base/config.json`](#create-baseconfigjson) + - [Modify `base/kustomization.yaml`](#modify-basekustomizationyaml) + - [Modify `base/awx.yaml`](#modify-baseawxyaml) + - [Next Step](#next-step) +- [Appendix: Create Secret for ImagePullSecrets by Hand](#appendix-create-secret-for-imagepullsecrets-by-hand) +- [Appendix: Modify ImagePullSecrets for the Specific Service Account](#appendix-modify-imagepullsecrets-for-the-specific-service-account) +- [Appendix: Apply Docker Hub Credential at Cluster Level](#appendix-apply-docker-hub-credential-at-cluster-level) + +## Procedure + +There are several ways to achieve this, but this guide will show you an example of using Kustomize to save all your configuration as code. In addition to [the main guide](../), a few additional files need to be modified. + +### Create `base/config.json` + +First, prepare JSON file named `config.json` under your `base` directory. + +```bash +$ DOCKERHUB_USERNAME= +$ DOCKERHUB_PASSWORD= +$ DOCKERHUB_AUTH=$(echo -n "${DOCKERHUB_USERNAME}:${DOCKERHUB_PASSWORD}" | base64) +$ cat < base/config.json +{ + "auths": { + "docker.io": { + "auth": "${DOCKERHUB_AUTH}" + } + } +} +EOF +``` + +Ensure your `config.json` includes base64-encoded string as following. Note that this base64-encoded string includes your password in plain text and easily revealed by `echo "" | base64 --decode` command. + +```bash +$ cat base/config.json +{ + "auths": { + "docker.io": { + "auth": "ZXhhbXBsZS...MtdG9rZW4K" + } + } +} +``` + +### Modify `base/kustomization.yaml` + +Then, add following four lines to under `secretGenerator` in `base/kustomization.yaml`. + +```yaml +... +secretGenerator: + ... + - name: awx-registry-secret 👈👈👈 + type: kubernetes.io/dockerconfigjson 👈👈👈 + files: 👈👈👈 + - .dockerconfigjson=config.json 👈👈👈 + ... +resources: + ... +``` + +### Modify `base/awx.yaml` + +Finally, add following line to `base/awx.yaml`. + +```yaml +... +spec: + ... + image_pull_secret: awx-registry-secret 👈👈👈 + ... +``` + +### Next Step + +Now everything is ready to deploy, go back to [the main guide (`README.md`)](../) and run `kubectl apply -k base`. + +## Appendix: Create Secret for ImagePullSecrets by Hand + +If you want to create Secret manually instead of using Kustomize, you can choose alternative way. If you don't have `awx` namespace, create it first by `kubectl create ns awx`. + +- **Method 1**: Create Secret by using existing `~/.docker/config.json` + + ```bash + # Log in to Docker Hub via docker command first, + $ docker login + + # Then create Secret by specifying ~/.docker/config.json which generated by Docker + $ kubectl -n awx create secret generic awx-registry-secret \ + --from-file=.dockerconfigjson=${HOME}/.docker/config.json \ + --type=kubernetes.io/dockerconfigjson + ``` + +- **Method 2**: Create Secret by specifying username and password manually + + ```bash + # Create Secret by specifying username and password manually + $ DOCKERHUB_USERNAME= + $ DOCKERHUB_PASSWORD= + $ kubectl -n awx create secret docker-registry awx-registry-secret \ + --docker-server=docker.io \ + --docker-username=${DOCKERHUB_USERNAME} \ + --docker-password=${DOCKERHUB_PASSWORD} + ``` + +## Appendix: Modify ImagePullSecrets for the Specific Service Account + +Once create new namespace, the default Service Account named `default` is also created by default. This `default` Service Account is used when no Service Account has been specified for the Pod, and you can modify ImagePullSecrets which used by default for this `default` Service Account too. This can be applied to [Private Git Repository](../git) and [Private Container Registry](../registry) included in this repository. Additionally, this can also be applied to the Pod for PostgreSQL created by AWX Operator, since the Service Account is not specified. + +First, you should create Secret for the Service Account by referring [Appendix: Create Secret for ImagePullSecrets by Hand](#appendix-create-secret-for-imagepullsecrets-by-hand). Note that the namespace and the name of the Secret in the command should be changed to suit your environment. + +Then patch the `default` service account. + +```bash +kubectl -n patch serviceaccount default -p '{"imagePullSecrets": [{"name": ""}]}' +``` + +## Appendix: Apply Docker Hub Credential at Cluster Level + +You can also change the entire K3s configuration so that the specific credential for Docker Hub is always used, regardless of the namespace. Create `/etc/rancher/k3s/registries.yaml` and restart K3s service as follows. + +```bash +# Create /etc/rancher/k3s/registries.yaml including your credential +$ DOCKERHUB_USERNAME= +$ DOCKERHUB_PASSWORD= +$ sudo tee /etc/rancher/k3s/registries.yaml <