automatically adjust pr envs resource request

[ITViking]

What does this PR do?

This adds a cronjob, which will run every 6 hours. It will take approximately pr-envs*1'ish minute to complete.
The job will adjust the redis, varnish, nginx and cli deployments to create pods with a set resource request, fitting for pr environments.

This is in effect now

Should this be tested by the reviewer and how?

Any specific requests for how the PR should be reviewed?

What are the relevant tickets?

https://reload.atlassian.net/jira/software/c/projects/DDFDRIFT/boards/464?selectedIssue=DDFDRIFT-264

[ath88]

Does the output here reveal if the deployment was changed or not? If so - you could skip the wait, if the resources had already been set.

[ITViking]

that is a good point, will have to look into that

[ITViking]

I left a comment, but there's more pressing matters to get at than this at the moment

[ath88]

I like your approach. Could you turn it into a helm chart instead, so the relationship between the resources is clearly marked for someone inspecting the cluster?

[hypesystem]

Looks very reasonable to me! Have you tested that it works in some limited capacity? Or are you awaiting approval before testing?

I have one concern which I will see if I can help with a solution for.

[hypesystem]

Pin to kubernetes version we know works with our setup.

kubectl may update or we may update cluster to create inconsistency unknowingly - so it's better to be able to control it.

[hypesystem]

Hmm, I think my take on this is that it is potentially creating a lot of work on the Kubernetes API server if we patch everything every minute?

Could we find a way to directly query if the deployments have their memory limits set correctly before trying to set them?

Looking into this, will report back if I find a good approach.

[hypesystem]

Beginning of something:

kubectl get deploy -n $ns -o json | jq '.items[] | { name: .metadata.name, resources: .spec.template.spec.containers[0].resources }'

produces a list of JSON objects each with a name and the resource spec. Next up, picking those that do not match the desired memory limit and request.

[hypesystem]

kubectl get deploy -n $ns -o json | jq '.items[] | [.metadata.name, .spec.template.spec.containers[0].resources.limits.memory, .spec.template.spec.containers[0].resources.requests.memory] | join(",")' --raw-output

This outputs a list of the format:

<deploymentname>,<memorylimit>,<memoryrequest>

To get a particular list of deployments, we can append this to the command (selecting nginx|redis|varnish|cli followed by memory indications):

 | grep -E '^(nginx|redis|varnish|cli),([0-9]+[A-Za-z]+|),([0-9]+[A-Za-z]+|)$'

If we save the output of the above as variable spec get the individual fields like this:

deployment_name=`echo $spec | cut -f1 -d,`
current_memory_limit=`echo $spec | cut -f2 -d,`
current_memory_request=`echo $spec | cut -f3 -d,`

Now we can decide to only patch if the limit or request differ from the current state. Much fewer requests.

When everything is as it should be it means we get the following requests every execution (6 hours):

• 1 GET namespaces
• 1 GET deployments per namespace
• (and then up to 4 patch calls per namespace, but for most sites 0)

Whereas before, when we naively patched, we would get:

• 1 GET namespaces
• 4 PATCH calls per namespace

[ITViking]

This has been tested and has been live since before christmas (december? can't remeber, but for a good while now)

[hypesystem]

@ITViking Then I think this is good to merge - we can look at potentially using my research at a later point if we think it is too resource intensive. The minute long break between each namespace is probably helping us though 😄