mirror of
https://github.com/Expand-sys/awx-on-k3s
synced 2025-12-15 21:42:15 +11:00
Merge pull request #36 from kurokobo/imagepullsecret
feat: add workaround for rate limit on docker hub
This commit is contained in:
kurokobo
GitHub
commit
3cb691db10
5 changed files with 183 additions and 1 deletions
|
|
@ -438,4 +438,5 @@ kubectl apply -f awx-secret-tls.yaml
|
|||
- [📝Uninstall deployed resouces](tips/uninstall.md)
|
||||
- [📝Deploy older version of AWX Operator](tips/deploy-older-operator.md)
|
||||
- [📝Upgrade AWX Operator and AWX](tips/upgrade-operator.md)
|
||||
- [📝Workaround for the rate limit on Docker Hub](tips/dockerhub-rate-limit.md)
|
||||
- [📝Troubleshooting Guide](tips/troubleshooting.md)
|
||||
|
|
|
|||
|
|
@ -210,7 +210,7 @@ sudo /usr/local/bin/crictl info
|
|||
sudo /usr/local/bin/crictl info | jq .config.registry
|
||||
```
|
||||
|
||||
If you want Kubernetes to be able to pull images directly from this private registry, alternatively you can also manually create `imagePullSecrets` for the Pod instead of writing your credentials in `auth` in `registries.yaml`.
|
||||
If you want Kubernetes to be able to pull images directly from this private registry, alternatively you can also manually create `imagePullSecrets` for the Pod instead of writing your credentials in `auth` in `registries.yaml`. [Another guide about rate limiting on Docker Hub](../tips/dockerhub-rate-limit.md) explains how to use `ImagePullSecrets`.
|
||||
|
||||
### Testing
|
||||
|
||||
|
|
|
|||
|
|
@ -6,4 +6,5 @@
|
|||
- [📝Uninstall deployed resouces](uninstall.md)
|
||||
- [📝Deploy older version of AWX Operator](deploy-older-operator.md)
|
||||
- [📝Upgrade AWX Operator and AWX](upgrade-operator.md)
|
||||
- [📝Workaround for the rate limit on Docker Hub](dockerhub-rate-limit.md)
|
||||
- [📝Troubleshooting Guide](troubleshooting.md)
|
||||
|
|
|
|||
158
tips/dockerhub-rate-limit.md
Normal file
158
tips/dockerhub-rate-limit.md
Normal file
|
|
@ -0,0 +1,158 @@
|
|||
<!-- omit in toc -->
|
||||
# Workaround for the rate limit on Docker Hub
|
||||
|
||||
If your Pod for PostgreSQL is in `ErrImagePull` and its `Events` shows following events, this is due to [the Rate Limit on Docker Hub](https://docs.docker.com/docker-hub/download-rate-limit/).
|
||||
|
||||
```bash
|
||||
$ kubectl -n awx describe pod awx-postgres-0
|
||||
...
|
||||
Events:
|
||||
Type Reason Age From Message
|
||||
---- ------ ---- ---- -------
|
||||
...
|
||||
Warning Failed 2s kubelet Failed to pull image "postgres:12": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/postgres:12": failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/postgres/manifests/sha256:...: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
|
||||
...
|
||||
```
|
||||
|
||||
If you just follow the steps in this repository to deploy you AWX, your pull request to Docker Hub will be identified as a free, anonymous account. Therefore, you will be limited to 200 requests in 6 hours. The message "429 Too Many Requests" indicates that the limit has been exceeded. To solve this, you should pass your Docker Hub credentials to the Pod as ImagePullSecrets.
|
||||
|
||||
<!-- omit in toc -->
|
||||
## Table of Contents
|
||||
|
||||
- [Procedure](#procedure)
|
||||
- [Create `base/config.json`](#create-baseconfigjson)
|
||||
- [Modify `base/kustomization.yaml`](#modify-basekustomizationyaml)
|
||||
- [Modify `base/awx.yaml`](#modify-baseawxyaml)
|
||||
- [Next Step](#next-step)
|
||||
- [Appendix: Create Secret for ImagePullSecrets by Hand](#appendix-create-secret-for-imagepullsecrets-by-hand)
|
||||
- [Appendix: Modify ImagePullSecrets for the Specific Service Account](#appendix-modify-imagepullsecrets-for-the-specific-service-account)
|
||||
- [Appendix: Apply Docker Hub Credential at Cluster Level](#appendix-apply-docker-hub-credential-at-cluster-level)
|
||||
|
||||
## Procedure
|
||||
|
||||
There are several ways to achieve this, but this guide will show you an example of using Kustomize to save all your configuration as code. In addition to [the main guide](../), a few additional files need to be modified.
|
||||
|
||||
### Create `base/config.json`
|
||||
|
||||
First, prepare JSON file named `config.json` under your `base` directory.
|
||||
|
||||
```bash
|
||||
$ DOCKERHUB_USERNAME=<Your Username for Docker Hub>
|
||||
$ DOCKERHUB_PASSWORD=<Your Password or Personal Access Token for Docker Hub>
|
||||
$ DOCKERHUB_AUTH=$(echo -n "${DOCKERHUB_USERNAME}:${DOCKERHUB_PASSWORD}" | base64)
|
||||
$ cat <<EOF > base/config.json
|
||||
{
|
||||
"auths": {
|
||||
"docker.io": {
|
||||
"auth": "${DOCKERHUB_AUTH}"
|
||||
}
|
||||
}
|
||||
}
|
||||
EOF
|
||||
```
|
||||
|
||||
Ensure your `config.json` includes base64-encoded string as following. Note that this base64-encoded string includes your password in plain text and easily revealed by `echo "<Encoded String>" | base64 --decode` command.
|
||||
|
||||
```bash
|
||||
$ cat base/config.json
|
||||
{
|
||||
"auths": {
|
||||
"docker.io": {
|
||||
"auth": "ZXhhbXBsZS...MtdG9rZW4K"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Modify `base/kustomization.yaml`
|
||||
|
||||
Then, add following four lines to under `secretGenerator` in `base/kustomization.yaml`.
|
||||
|
||||
```yaml
|
||||
...
|
||||
secretGenerator:
|
||||
...
|
||||
- name: awx-registry-secret 👈👈👈
|
||||
type: kubernetes.io/dockerconfigjson 👈👈👈
|
||||
files: 👈👈👈
|
||||
- .dockerconfigjson=config.json 👈👈👈
|
||||
...
|
||||
resources:
|
||||
...
|
||||
```
|
||||
|
||||
### Modify `base/awx.yaml`
|
||||
|
||||
Finally, add following line to `base/awx.yaml`.
|
||||
|
||||
```yaml
|
||||
...
|
||||
spec:
|
||||
...
|
||||
image_pull_secret: awx-registry-secret 👈👈👈
|
||||
...
|
||||
```
|
||||
|
||||
### Next Step
|
||||
|
||||
Now everything is ready to deploy, go back to [the main guide (`README.md`)](../) and run `kubectl apply -k base`.
|
||||
|
||||
## Appendix: Create Secret for ImagePullSecrets by Hand
|
||||
|
||||
If you want to create Secret manually instead of using Kustomize, you can choose alternative way. If you don't have `awx` namespace, create it first by `kubectl create ns awx`.
|
||||
|
||||
- **Method 1**: Create Secret by using existing `~/.docker/config.json`
|
||||
|
||||
```bash
|
||||
# Log in to Docker Hub via docker command first,
|
||||
$ docker login
|
||||
|
||||
# Then create Secret by specifying ~/.docker/config.json which generated by Docker
|
||||
$ kubectl -n awx create secret generic awx-registry-secret \
|
||||
--from-file=.dockerconfigjson=${HOME}/.docker/config.json \
|
||||
--type=kubernetes.io/dockerconfigjson
|
||||
```
|
||||
|
||||
- **Method 2**: Create Secret by specifying username and password manually
|
||||
|
||||
```bash
|
||||
# Create Secret by specifying username and password manually
|
||||
$ DOCKERHUB_USERNAME=<Your Username for Docker Hub>
|
||||
$ DOCKERHUB_PASSWORD=<Your Password or Personal Access Token for Docker Hub>
|
||||
$ kubectl -n awx create secret docker-registry awx-registry-secret \
|
||||
--docker-server=docker.io \
|
||||
--docker-username=${DOCKERHUB_USERNAME} \
|
||||
--docker-password=${DOCKERHUB_PASSWORD}
|
||||
```
|
||||
|
||||
## Appendix: Modify ImagePullSecrets for the Specific Service Account
|
||||
|
||||
Once create new namespace, the default Service Account named `default` is also created by default. This `default` Service Account is used when no Service Account has been specified for the Pod, and you can modify ImagePullSecrets which used by default for this `default` Service Account too. This can be applied to [Private Git Repository](../git) and [Private Container Registry](../registry) included in this repository. Additionally, this can also be applied to the Pod for PostgreSQL created by AWX Operator, since the Service Account is not specified.
|
||||
|
||||
First, you should create Secret for the Service Account by referring [Appendix: Create Secret for ImagePullSecrets by Hand](#appendix-create-secret-for-imagepullsecrets-by-hand). Note that the namespace and the name of the Secret in the command should be changed to suit your environment.
|
||||
|
||||
Then patch the `default` service account.
|
||||
|
||||
```bash
|
||||
kubectl -n <Your Namespace> patch serviceaccount default -p '{"imagePullSecrets": [{"name": "<Your Secret>"}]}'
|
||||
```
|
||||
|
||||
## Appendix: Apply Docker Hub Credential at Cluster Level
|
||||
|
||||
You can also change the entire K3s configuration so that the specific credential for Docker Hub is always used, regardless of the namespace. Create `/etc/rancher/k3s/registries.yaml` and restart K3s service as follows.
|
||||
|
||||
```bash
|
||||
# Create /etc/rancher/k3s/registries.yaml including your credential
|
||||
$ DOCKERHUB_USERNAME=<Your Username for Docker Hub>
|
||||
$ DOCKERHUB_PASSWORD=<Your Password or Personal Access Token for Docker Hub>
|
||||
$ sudo tee /etc/rancher/k3s/registries.yaml <<EOF
|
||||
configs:
|
||||
registry-1.docker.io:
|
||||
auth:
|
||||
username: ${DOCKERHUB_USERNAME}
|
||||
password: ${DOCKERHUB_PASSWORD}
|
||||
EOF
|
||||
|
||||
# Then restart K3s. The K3s service can be safely restarted without affecting the running resources
|
||||
$ sudo systemctl restart k3s
|
||||
```
|
||||
|
|
@ -10,6 +10,7 @@ Some hints and guides for when you got stuck during deployment and daily use of
|
|||
- [First Step: Investigate your Situation](#first-step-investigate-your-situation)
|
||||
- [Investigate Status and Events of the Pods](#investigate-status-and-events-of-the-pods)
|
||||
- [Investigate Logs of the Containers inside the Pods](#investigate-logs-of-the-containers-inside-the-pods)
|
||||
- [The Pod is `ErrImagePull` with "429 Too Many Requests"](#the-pod-is-errimagepull-with-429-too-many-requests)
|
||||
- [The Pod is `Pending` with "1 Insufficient cpu, 1 Insufficient memory." event](#the-pod-is-pending-with-1-insufficient-cpu-1-insufficient-memory-event)
|
||||
- [The Pod is `Pending` with "1 pod has unbound immediate PersistentVolumeClaims." event](#the-pod-is-pending-with-1-pod-has-unbound-immediate-persistentvolumeclaims-event)
|
||||
- [The Pod is `Running` but stucked with "[wait-for-migrations] Waiting for database migrations..." message](#the-pod-is-running-but-stucked-with-wait-for-migrations-waiting-for-database-migrations-message)
|
||||
|
|
@ -101,6 +102,27 @@ For AWX Operator and AWX, specifically, the following commands are helpful.
|
|||
- Logs of PostgreSQL
|
||||
- `kubectl -n awx logs -f statefulset/awx-postgres`
|
||||
|
||||
### The Pod is `ErrImagePull` with "429 Too Many Requests"
|
||||
|
||||
If your Pod for PostgreSQL is in `ErrImagePull` and its `Events` shows following events, this is due to [the rate limit on Docker Hub](https://docs.docker.com/docker-hub/download-rate-limit/).
|
||||
|
||||
```bash
|
||||
$ kubectl -n awx describe pod awx-postgres-0
|
||||
...
|
||||
Events:
|
||||
Type Reason Age From Message
|
||||
---- ------ ---- ---- -------
|
||||
Normal Pulling 9s kubelet Pulling image "postgres:12"
|
||||
Warning Failed 2s kubelet Failed to pull image "postgres:12": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/postgres:12": failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/postgres/manifests/sha256:505d023f030cdea84a42d580c2a4a0e17bbb3e91c30b2aea9c02f2dfb10325ba: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
|
||||
Warning Failed 2s kubelet Error: ErrImagePull
|
||||
Normal BackOff 1s kubelet Back-off pulling image "postgres:12"
|
||||
Warning Failed 1s kubelet Error: ImagePullBackOff
|
||||
```
|
||||
|
||||
If you follow the steps in this repository to deploy you AWX, your pull request to Docker Hub will be identified as a free, anonymous account. Therefore, you will be limited to 200 requests in 6 hours. The message "429 Too Many Requests" indicates that it has been exceeded.
|
||||
|
||||
To solve this, you can simply wait until the limit is freeed up, or [consider giving your Docker Hub credentials to K3s by follwing the guide on this page](dockerhub-rate-limit.md).
|
||||
|
||||
### The Pod is `Pending` with "1 Insufficient cpu, 1 Insufficient memory." event
|
||||
|
||||
If your Pod is in `Pending` state and its `Events` shows following events, the reason is that the node does not have enough CPU and memory to start the Pod. By default AWX requires at least 2 CPUs and 4 GB RAM. In addition more resources are required to run K3s and the OS itself.
|
||||
|
|
|
|||
Loading…
Reference in a new issue