External Certificate Manager

[katheris]

This proposal aims to allow Strimzi users to use an external certificate manager, specifically cert-manager, to manage certificates.

Related to strimzi/strimzi-kafka-operator#929

[Frawless]

Guess this is because we want to keep for self-signed certs our current way and not to add another option that will mostly add just support burden?

[katheris]

This is because there are lots of different issuers that work with cert-manager. So rather than Strimzi having to actively support all the different types, I've proposed that the user creates the Issuer or ClusterIssuer and handles supplying a Secret with the trusted certificates for the issuer they have chosen. That way Strimzi can work with any cert-manager issuer integrations.

[PaulRMellor]

Will we need to provide any guidelines on conventions in the docs?

[Frawless]

I think we just need to mention in in the docs properly as users could be confused from different projects where integration of CM works without creating any Issuer (afaiu operator creates self-sign Issuer when it is not created by users).

[tinaselenge]

issuing cluster certificates (e.g for Kafka etc) - I wonder if it would be useful to include the secret names for these certificates as an example.

[tinaselenge]

Suggested change

	### Issuing certificates
	### Issuing end-entity certificates

[tinaselenge]

Thanks for the proposal Kate, it looks good to me.

[scholzj]

I left some nits. But the main thing missing seems to be how you plan to roll out the trust to the new CAs. There is no reference to strimzi.io/ca-key-generation or to rotation of private keys, so it is not clear how do you expect to handle it.

[scholzj]

I don't think this really helps because the CA will be anyway bootstrapped as self-signed as it is today in most cases, and there is not much we can do about it.

[scholzj]

Should the type separation happen already at the clusterCa level? It would seem more logical to me.

[katheris]

@scholzj Do you mean like this?

spec:
  clusterCa:
    validityDays: <integer> # notBefore=now, notAfter=now + validityDays
    generateCertificateAuthority: <boolean>
    generateSecretOwnerReference: <boolean>
    renewalDays: <integer> # days before notAfter when we should start renewal
    certificateExpirationPolicy: <renew-certificate|replace-key>
    certificateIssuerType: <internal|cert-manager.io> # (1)
    certManagerIssuerRef: # (2)
      name: <string>
      kind: <Issuer|ClusterIssuer>
      group: <string> # cert-manager.io by default

[scholzj]

Yeah. Although I would maybe call it just type instead of certificateIssuerType. It would also default to internal when not set (I hope that can be implemented in the Java api classes).

I think that would create a better abstraction as not all the fields in the CA configuration might be applicable to all issuer types.

[scholzj]

Maybe we should instead (or next to this) describe the expectation of how the Secret should look like? IIRC, trust-manager creates a Secret will all CAs bundled into a single file? Is that supported / expected?

[fvaleri]

I agree that this needs some clarification. Maybe a sequence diagram with the PKI generation process including user, CO, trust-manager and cert-manager would also help.

[scholzj]

I think we should again aim here for a separation of the Strimzi used Secrets and the user-provided Secrets. I.e. the user should provide it in some custom secret and we should copy it ourself into <CLUSTER_NAME>-cluster-ca-cert or <CLUSTER_NAME>-clients-ca-cert if needed.

[scholzj]

• Does this work with the trust-manager suggested earlier?
• Can we work around it and manage the generation ourself? E.g. based on the hash of the user-provided certificate detect changes and bump the generation ourself?
• What is the impact on the strimzi.io/ca-key-generation given we now do not have the private key secret?

[scholzj]

We do not control that Secret, so we should likely not annotate it / rely on its annotation. We need to cary it on the internal Secrets maybe?

[scholzj]

Maybe we should make it clear that this is where we control the CN / SANs of the certificates?

[scholzj]

I guess UO and the Clients CA deserves more attention here? Will the UO issue the Certificate resources? How will it keep the certificate Secrets? Or will type: tls-external be used here?

[scholzj]

Not sure I follow the need to annotate the Secrets here. Normally, during the reconciliation:

• You take the hash of the certiicate
• Use the Hash to annotate the Pod in the Deployment / StrimziPodSet
• Either Kubernetes or Strimzi takes care of rolling the pod based on the Pod annotations being different

[scholzj]

I think this part is really important - (keeping the old CA cert in the Secret still) - and should be made more prominent? We should also cover the point when it should be removed.

[MichaelMorrisEst]

I think I would be very very useful to have this implementation behind an interface and support a mechanism for loading alternative implementations for other external certificate managers. This would allow users to integrate with other external certificate managers

[MichaelMorrisEst]

The issuerRef element looks to be cert-manager specific rather than something relevant to any external cert manager which could become an issue when supporting other external managers (as mentioned above).
Perhaps the certificateIssuer should instead have a certManager specific sub-element, and add different similar elements in the future that are specific to other certificate managers, i.e. something like:

clusterCa:
    certificateIssuer:
        certManager:
            issuerRef:
                name: <string>
                kind: <Issuer|ClusterIssuer>
                group: <string> # cert-manager.io by default
        someOtherManager: <-- future addition -->
            managerSpecificConfig:
                ...
        oneOf:
        - properties
            certManager{}
            someOtherManager{}

Or alternatively just allow a map of values to be specified, but that would be less user friendly

[fvaleri]

2. The property certificateIssuer.issuerRef will only be used by Strimzi if certificateIssuer.type is set to cert-manager.io.

From the above phrase it looks like the intention is to make issuerRef cert-maanger specific.

[ppatierno]

we are talking about generating the CA certificate right? Can we make it explicit here?

[PaulRMellor]

Any potential issues with time-outs and retries need to be mentioned here?

[ppatierno]

which CAs are we talking about here? Aren't the cluster CA and clients CA being generate via cert-manager (it relates to my previous question I guess) ... I am confused :-/

[PaulRMellor]

If the secret is referenced in the Issuer or ClusterIssuer, should we create it first?

[PaulRMellor]

Looks good. I made a few suggested changes to wording for clarity and readability. There are also a couple of questions for further clarification.

[PaulRMellor]

Suggested change

	Strimzi will be updated to allow users to specify that certificates should be issued by an external certificate manager, rather than issued by the Cluster Operator.
	Strimzi will be updated to allow users to specify that certificates should be issued by an external certificate manager instead of the Cluster Operator.

[PaulRMellor]

Suggested change

	* Strimzi will not be responsible for installing cert-manager, but we will document the versions of cert-manager that we have tested with.
	* Strimzi will not be responsible for installing cert-manager, but we will document the supported versions of cert-manager that we have tested with.

[PaulRMellor]

Will we need to provide any guidelines on conventions in the docs?

[PaulRMellor]

Any potential issues with time-outs and retries need to be mentioned here?

[PaulRMellor]

If the secret is referenced in the Issuer or ClusterIssuer, should we create it first?

[PaulRMellor]

Suggested change

	Similar to today Strimzi will use the notion of a "generation" to determine whether to roll the cluster to pick up changes in either the `<CLUSTER_NAME>-cluster-ca-cert` or `<CLUSTER_NAME>-clients-ca-cert`.
	When the user creates the CA cert Secret they must add the `strimzi.io/ca-cert-generation` annotation to it.
	If the user updates the Secret to change the certificates included they must increment the annotation to inform Strimzi it has changed.
	Similar to today Strimzi will put the annotation on the pods (Kafka, ZooKeeper etc) to be able to spot when the generation has been changed.
	Strimzi will use the current process to determine whether to roll the cluster to pick up changes in the `<CLUSTER_NAME>-cluster-ca-cert` or `<CLUSTER_NAME>-clients-ca-cert` secrets.
	When a user creates the CA certificate secret, they must add the `strimzi.io/ca-cert-generation` annotation.
	Strimzi adds this annotation to the pods (Kafka, ZooKeeper, etc.) and uses it to detect when the secrets have changed.
	If the user updates the secret to change the certificates, they must increment the annotation to inform Strimzi of the change.

[PaulRMellor]

do we need to consider maintenance time windows at all?

[PaulRMellor]

Suggested change

	This feature will be optional and not disabled by default.
	This feature will be optional and enabled by default.

[PaulRMellor]

What about a new cluster?

[PaulRMellor]

Suggested change

	To revert to user managed CAs the user will:
	To revert to user managed CAs the user must:

[PaulRMellor]

Maybe we should?

[katheris]

My feeling was that it wouldn't be a common scenario for customers to move from a mechanism where they have more control, to a mechanism where they have less control.

[scholzj]

I'm not sure it is even supported to go from custom CA to Strimzi CA. At least I do not remember anyone trying it / testing it. In the long term, I think the idea would be also to get rid of the internal CA management - so not sure this would be a issue from my point of view.