Skip to content

Enhance the generation and handling of keystore and truststore #312

Open
0 of 3 issues completed
Open

Description

@leochr

There are various ways to inject keystore and truststore for the application. In some cases, some conflicts could prevent the application from working as expected. Re-evaluate the generation and handling of keystore and truststore and improve as possible.

Open Liberty runtime also supports providing certificates as env var to establish trust. Switch to this mechanism if possible: https://openliberty.io/docs/latest/reference/feature/transportSecurity-1.0.html#_provide_certificates_from_an_environment_variable_or_a_file

Activity

  1. self-assigned this
    on Aug 4, 2022
  2. removed their assignment
    on Aug 24, 2022
  3. leochr commented on Nov 22, 2023

    @leochr
    MemberAuthor

    @idlewis please add your analysis of truststore handling in Liberty container here. Thank you

  4. idlewis commented on Nov 22, 2023

    @idlewis
    Member

    I think I can see two issues with the trust store handling in docker-server.sh

    1. The script uses 'keytool -import' to add certificates to the default trust store in 2 situations (ca.crt exists in /etc/x509/certs or SEC_IMPORT_K8S_CERTS=true). If there is an existing trust store, this command will fail, as the passwords won't match
    2. If a trust store exists in the default location, then the script will add its own config to config overrides. If there was an existing trust store before the script ran, this config will cause a password mismatch, and an error on Liberty startup.

    The fix for 2 is pretty simple (only add config if docker-server.sh created the truststore)

    For 1, we need to decide what to do in the case of an existing trust store.

  5. justin-fleming commented on Feb 22, 2024

    @justin-fleming
    Member

    Summary

    The docker-server.sh script shipped with Liberty does not import .crt files that are missing a newline at the end of the file in the /var/run/secrets/kubernetes.io/serviceaccount dir.

    The problem is on this line where we are cating *.crt and attempting to iterate over the certs to import them: https://github.com/OpenLiberty/ci.docker/blob/main/releases/latest/full/helpers/runtime/docker-server.sh#L62

    Steps to recreate

    Setup:

    • Given 2 cert files: thing1.crt and thing2.crt
    • I simulated the customers 'bad' certs by doing the following:
      mkdir badcerts
      printf "%s" "$(< thing1.crt)" > badcerts/thing1-bad.crt
      printf "%s" "$(< thing2.crt)" > badcerts/thing2-bad.crt
      
      This removes the newline at the end of the files:
      # diff thing1.crt badcerts/thing1-bad.crt 
      19c19
      < -----END CERTIFICATE-----
      ---
      > -----END CERTIFICATE-----
      \ No newline at end of file
      

    Without newlines at the end of the cert files, no truststore is generated (even though the certs are mounted):

    • Created a configmap with the bad certs:

      oc create configmap --from-file cert1.crt=badcerts/thing1-bad.crt --from-file cert2.crt=badcerts/thing2-bad.crt certs
      
    • Created this deployment:

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: liberty-deployment
        namespace: cert-test
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: liberty-test
        template:
          metadata:
            labels:
              app: liberty-test
            namespace: liberty-test
          spec:
            containers:
            - name: liberty
              image: image-registry.openshift-image-registry.svc:5000/cert-test/ol-test:1.0
              ports:
              - containerPort: 9080
              imagePullPolicy: Always
              volumeMounts:
              - name: cert-volume
                mountPath: /var/run/secrets/kubernetes.io/serviceaccount
              env:
              - name: SEC_IMPORT_K8S_CERTS
                value: 'true'
            volumes:
              - name: cert-volume
                configMap:
                  name: certs
      
    • Check details in the running pod:

      # oc rsh liberty-deployment-86df686575-8kc72 
      sh-4.4$ ls -la /output/resources/security/
      total 4
      drwxrwx---. 2 default root   21 Feb 15 19:56 .
      drwxrwx---. 1 default root   22 Feb 15 19:56 ..
      -rw-rwx---. 1 default root 2650 Feb 15 19:56 key.p12
      
      sh-4.4$ ls -la /var/run/secrets/kubernetes.io/serviceaccount/
      total 0
      drwxrwsrwx. 3 root 1000680000 94 Feb 22 17:21 .
      drwxr-xr-x. 3 root root       60 Feb 22 17:21 ..
      drwxr-sr-x. 2 root 1000680000 40 Feb 22 17:21 ..2024_02_22_17_21_26.2791951379
      lrwxrwxrwx. 1 root 1000680000 16 Feb 22 17:21 cert1.crt -> ..data/cert1.crt
      lrwxrwxrwx. 1 root 1000680000 16 Feb 22 17:21 cert2.crt -> ..data/cert2.crt
      lrwxrwxrwx. 1 root 1000680000 32 Feb 22 17:21 ..data -> ..2024_02_22_17_21_26.2791951379
      
      sh-4.4$ cat *.crt
      -----BEGIN CERTIFICATE-----
      ...
      cert-data-1
      ...
      -----END CERTIFICATE----------BEGIN CERTIFICATE-----
      ...
      cert-data-2
      ...
      -----END CERTIFICATE-----sh-4.4$
      

    With newlines, the certs are imported into the truststore:

    • Delete & recreate the configmap with the 'good' certs and restart the app pod:

      oc delete cm certs
      oc create configmap --from-file cert1.crt=thing1.crt --from-file cert2.crt=thing2.crt certs
      oc delete po -l 'app=liberty-test'
      
    • Check details in the new pod:

      # oc rsh liberty-deployment-86df686575-s42f7
      sh-4.4$ ls -la /output/resources/security/
      total 8
      drwxrwx---. 1 default    root   23 Feb 22 17:35 .
      drwxrwx---. 1 default    root   22 Feb 15 19:56 ..
      -rw-rwx---. 1 default    root 2650 Feb 15 19:56 key.p12
      -rw-r--r--. 1 1000680000 root 2054 Feb 22 17:35 trust.p12
      sh-4.4$ cat /var/run/secrets/kubernetes.io/serviceaccount/*.crt
      -----BEGIN CERTIFICATE-----
      ...
      cert-data-1
      ...
      -----END CERTIFICATE-----
      -----BEGIN CERTIFICATE-----
      ...
      cert-data-2
      ...
      -----END CERTIFICATE-----
      sh-4.4$
      
  6. changed the title [-]Evaluate the generation and handling of keystore and truststore and improve as possible[/-] [+]Enhance the generation and handling of keystore and truststore[/+] on May 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions