expose metrics securely

[ffromani]

Consolidating sparse musings about the topic in one place:the preferr

We want to re-enable the exporting of prometheus metrics securely. This affects both the operator proper and RTE.

• controller-runtime gained a super nice feature to do this automatically. We can't consume it till the oldest kube version we support is 1.28.
• we would STRONGLY prefer a consistent solution all across the supported versions (e.g. no controller-runtime against head and another solution for older branches)
• the preferred approach is thus injecting a sidecar container with kube-rbac-proxy
• the proxy image path should be overridable somehow if possible.
• for RTE, the operator should learn introspecting its own manifest the image spec of the proxy and use the same image for RTE sidecars. This image should NOT be user-overridable
• for the sidecar manifests and related, we can create a pkg/metrics package possibly with pkg/metrics/manifests/yaml or somthing like that
• enabling full prometheus support in CI is not necessary for this effort
• e2e testing is however a must; a simple curl against the proxy can be sufficient
• we may want to create a custom kube-rbac-proxy imager ONLY FOR CI TESTING if this makes our life easier