Trying to use dockerRegistryMirror due to Docker Rate Limiting

[JamesAskew]

My runners are scaling in and out really nicely, but as a result, I am now getting hit with some Docker rate limiting. To get around this I want to use Amazon's public ecr gallery. I've set the value of dockerRegistryMirror in my values-override.yaml file and deployed a new revision. But images are still being pulled from docker.io.

Has anyone managed to get a registry mirror working that could give me some tips?

I've read all of the discussion and issues which seem relevant, but nothing has helped me.

Here are my helm overrides:

dockerRegistryMirror: "https://public.ecr.aws"
image:
  repository: "summerwind/actions-runner-controller"
  actionsRunnerRepositoryAndTag: "summerwind/actions-runner:latest"
  dindSidecarRepositoryAndTag: "docker:dind"
  pullPolicy: Always
  # The default image-pull secrets name for self-hosted runner container.
  # It's added to spec.ImagePullSecrets of self-hosted runner pods.
  actionsRunnerImagePullSecrets: []

Inside the controller pod, I can see the following arguments:

--metrics-addr=127.0.0.1:8080 --enable-leader-election --port=9443 --sync-period=1m --default-scale-down-delay=10m --docker-image=docker:dind --runner-image=summerwind/actions-runner@sha256:af356cd14f621b98194da116b33d331af45191dc0ab07d708769290793494581 --docker-registry-mirror=https://public.ecr.aws

But regardless the image is always from docker.io

Any helps or tips would be greatly appreciated.

[toast-gear]

Any of docker configuration options that are at a container file system level require you use the dind runner image

[toast-gear]

Is it possible for the runner image to come from the mirror? I can see the mirror is being set, but I am not sure what it is actually being used for.

Ah I see, the config you have configured is referring to your runner's docker daemon, not your k8s node. What's your driver for wanting to pull the runner image itself from a mirror? You can configure the pull policy and the image details from your runner*.yaml manifests.

My runners are scaling in and out really nicely, but as a result, I am now getting hit with some Docker rate limiting.

Are these rate limits really from the runner image itself? Are they not from images your runner's are consuming? If your pull policy is set to IfNotPresent then the image should just be getting pulled once per node and then consumed locally.

[JamesAskew]

What's your driver for wanting to pull the runner image itself from a mirror?

It's quite possible, I've made a bad assumption.

If your pull policy is set to IfNotPresent then the image should just be getting pulled once per node and then consumed locally

I was using IfNotPresent, I changed it to Always yesterday as I wanted to do some tests. Even with it set to IfNotPresent though, I would see image pull failures when scaling out. The runner would be in an Image Pull Back Off state complaining about docker rate limits. This was happening before an action had been allocated to the runner. This is what started me down the path of the mirror.

Looking at the controller and runner logs wasn't really showing me much, I couldn't see where the images were coming from. I had to rely on the labels on the pod.

I'll do some more testing now with the changes above.

Really appreciate the help and the extra information!

[toast-gear]

I was using IfNotPresent, I changed it to Always yesterday as I wanted to do some tests. Even with it set to IfNotPresent though, I would see image pull failures when scaling out. The runner would be in an Image Pull Back Off state complaining about docker rate limits. This was happening before an action had been allocated to the runner. This is what started me down the path of the mirror.

can you confirm where you were configuring this? We document what each values.yaml option actually does in the chart's root README.md here

[JamesAskew]

Yeah, I was setting it in my helm overrides file. image.pullPolicy - Although knowing what I know now I wonder if it actually needs to be setting in the Kubernetes resource file?

[toast-gear]

Yeah, I was setting it in my helm overrides file. image.pullPolicy

yeh this controls the pull policy of the controller image, not the runner images

Although knowing what I know now I wonder if it actually needs to be setting in the Kubernetes resource file?

yup, you got it!