Repository navigation
Add did:key node IDs and certificate-free member/user identities - #8602
Amaury Chamayou (achamayou) wants to merge 16 commits into
Conversation
Keep SHA-256 defaults and digest behavior unchanged while supporting canonical public-key IDs throughout enrollment, authentication, node lifecycle, and SDK tooling. Preserve legacy enrollment during CCF 7 rolling upgrades and cover mixed-format lifecycle and recovery. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
Disk-based DID recovery and encoded node requests remain broken, and legacy enrollment adds quadratic table scanning.
3 open issues
What changed in this PR
Adds opt-in public-key identities across CCF while retaining SHA-256 defaults. This supports same-key certificate replacement, not the digest agility requested in #2419.
Changes:
- Adds canonical
did:keyderivation, enrollment, and authentication. - Integrates identity formats into joining, recovery, signing tools, and governance paths.
- Updates schemas, migration documentation, and regression coverage.
Custom instructions used
.github/instructions/reviewing.instructions.md
| File | Description |
|---|---|
| tests/lts_compatibility.py | Checks legacy enrollment during upgrades. |
| tests/infra/remote.py | Propagates node identity format. |
| tests/infra/openapi.py | Decodes schema-validation path parameters. |
| tests/infra/node.py | Supports format-aware node identities. |
| tests/infra/network.py | Configures member and user formats. |
| tests/infra/member.py | Supports DID signing and recovery paths. |
| tests/infra/crypto.py | Adds identity derivation helpers. |
| tests/infra/consortium.py | Passes enrollment formats. |
| tests/infra/clients.py | Supports DID COSE identities. |
| tests/governance.py | Adds DID lifecycle and recovery scenarios. |
| tests/config.jinja | Emits node identity configuration. |
| src/node/startup_inputs.h | Passes genesis member formats. |
| src/node/rpc/test/node_frontend_test.cpp | Tests DID configuration and joining. |
| src/node/rpc/test/internal_tables_access_test.cpp | Tests enrollment conflict handling. |
| src/node/rpc/test/frontend_test.cpp | Tests DID certificate authentication. |
| src/node/rpc/serialization.h | Serializes join identity format. |
| src/node/rpc/self_cert_auth.h | Matches the local node's format. |
| src/node/rpc/node_frontend.h | Handles DID joining and duplicates. |
| src/node/rpc/node_call_types.h | Adds join format selection. |
| src/node/node_state.h | Derives and confirms node IDs. |
| src/node/internal_tables_access.h | Enforces enrollment format constraints. |
| src/node/gov/handlers/helpers.h | Decodes and validates identity paths. |
| src/node/gov/handlers/acks.h | Decodes member action paths. |
| src/node/gov/gov_endpoint_registry.h | Updates identity schemas. |
| src/js/extensions/ccf/converters.cpp | Exposes native identity converters. |
| src/endpoints/test/endpoint_registry.cpp | Tests encoded DID action paths. |
| src/endpoints/authentication/cose_auth.cpp | Checks enrolled DID key binding. |
| src/endpoints/authentication/cert_auth.cpp | Adds DID certificate lookup. |
| src/enclave/enclave.h | Passes configured node format. |
| src/crypto/test/crypto.cpp | Tests DID encoding and key equality. |
| src/crypto/did_key.cpp | Implements canonical DID conversion. |
| samples/constitutions/default/actions.js | Adds DID enrollment safeguards. |
| python/utils/submit_recovery_share.sh | Supports DID recovery signing. |
| python/tests/test_identity.py | Tests SDK identity derivation. |
| python/tests/test_cose_sign.py | Tests both signing formats. |
| python/src/ccf/identity.py | Adds identity helpers and CLI. |
| python/src/ccf/cose.py | Adds signing format selection. |
| js/ccf-app/test/polyfill.test.ts | Tests identity polyfills. |
| js/ccf-app/src/polyfill.ts | Implements local identity conversion. |
| js/ccf-app/src/global.ts | Declares identity APIs. |
| include/ccf/service/tables/users.h | Adds user enrollment format. |
| include/ccf/service/tables/nodes.h | Adds format-aware node derivation. |
| include/ccf/service/tables/members.h | Adds member enrollment format. |
| include/ccf/node/configuration.h | Adds startup format options. |
| include/ccf/identity_id_format.h | Defines supported formats. |
| include/ccf/entity_id.h | Validates and describes DID IDs. |
| include/ccf/crypto/did_key.h | Declares DID conversion APIs. |
| include/ccf/crypto/cert_id.h | Adds certificate identity derivation. |
| doc/use_apps/issue_commands.rst | Documents DID signing options. |
| doc/schemas/node_openapi.json | Allows DID node IDs. |
| doc/schemas/gov/2024-07-01/gov.json | Updates stable governance schemas. |
| doc/schemas/gov/2023-06-01-preview/gov.json | Updates preview governance schemas. |
| doc/schemas/gov_openapi.json | Updates latest governance schemas. |
| doc/operations/configuration.rst | Documents startup identity settings. |
| doc/host_config_schema/host_config.json | Describes format configuration fields. |
| doc/governance/open_network.rst | Documents DID user enrollment. |
| doc/governance/adding_member.rst | Documents DID member workflows. |
| doc/build_apps/auth/cert.rst | Explains replacement-certificate authentication. |
| doc/architecture/cryptography.rst | Defines identity and migration semantics. |
| cmake/crypto.cmake | Includes DID implementation in builds. |
| CHANGELOG.md | Announces opt-in identity support. |
Review effort: Balanced
Use a one-byte identity format enum and explicit QuickJS boolean conversions to satisfy Virtual A's clang-tidy checks. Normalize DER-imported RSA public keys through JWK before canonical PKCS#1 re-encoding. This preserves DID encoding while fixing the Node 24 failure in modules_test and programmability_and_jwt. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Preserve enrolled member formats and recovery roles during disk-based recovery. Avoid scanning legacy certificate rosters and decode node identity path parameters, with native and end-to-end regression coverage. Allow certificate DID results to move into optional returns, fixing the required clang-tidy CI failure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…errors - set_member no longer no-ops for an already-enrolled did:key member: it replaces the stored certificate, encryption key and member data and resets the member to Accepted, exactly like legacy members. - Join requests whose caller key cannot be identified in the requested ID format now fail with 400 instead of 500; DID derivation failures in cert/node/self authentication policies fall through to generic reasons rather than leaking parser error text. - Clarify in docs that enrolled-certificate validity does not bound access for DID identities. - Add JS unit, native and e2e regression coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Refresh against main at 9cdc86c without rewriting published history. Adapt the unsupported-key join regression to the updated certificate helper's separate validity arguments, and move the DID release note and SDK version to the next patch, 7.0.24. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…59d-f93e-40fd-9f2f-a0b4f11e15c0
Store public keys without identity certificates, authenticate DID COSE requests directly, and migrate legacy identities atomically in ordered governance proposals. Update genesis, SDK/CLI, recovery tooling, schemas, regression coverage, and the RST migration guide. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 264fbdd6871f4086ab3af9b5484907c3
Retain legacy certificate migration while generating and recovering fresh DID identities without certificates. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 264fbdd6871f4086ab3af9b5484907c3
Integrate current COSE dependencies, crypto support, and documentation checks while preserving explicit DID signing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 264fbdd6871f4086ab3af9b5484907c3
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 264fbdd6871f4086ab3af9b5484907c3
Eddy Ashton (eddyashton)
left a comment
There was a problem hiding this comment.
Comments from initial read.
| CCF supports compressed P-256, P-384, and P-521 keys and PKCS#1 RSA public keys between 2048 and 16384 bits. | ||
| Existing authentication and signing algorithm restrictions still apply. | ||
| Only bare, canonical IDs are accepted; DID URLs with fragments, queries, or paths are not identity IDs. | ||
| When inserting an ID into an HTTP path, percent-encode its colons as ``%3A`` rather than encoding the endpoint's literal action suffix. |
There was a problem hiding this comment.
This is surprising! I thought dids were deliberately "URL-safe" such that they didn't need encoding? Is this a flaw in our parser?
There was a problem hiding this comment.
That's a good point, I will check again.
| .map((action) => action.args?.[`${role}_id`]), | ||
| ); | ||
| if (!isDid) { | ||
| if (typeof ccf.certificatesHaveSamePublicKey !== "function") { |
There was a problem hiding this comment.
Why are we checking for this function (legitimate to avoid calling it on old nodes!), when we don't even call it?
| // Validation sees pre-proposal state; application checks the actual tables. | ||
| const role = tableName === "public:ccf.gov.members.certs" ? "member" : "user"; | ||
| const removedIds = new Set( | ||
| previousActions | ||
| .filter((action) => action.name === `remove_${role}`) | ||
| .map((action) => action.args?.[`${role}_id`]), | ||
| ); |
There was a problem hiding this comment.
I don't like this bit. I don't like calling this function (which does validation) again during application. No need to do that! Also we shouldn't be constraining to previous actions; if they eventually write to different tables, then the application order doesn't matter. I don't like the action-at-a-distance reconstruction of another action name (remove_*).
I think we could get the same result by modifying the existing (idempotent!) set_* actions to do all of this. The replacement story is that you say set_user: did_key=..., and internally that does some validation and dedup (removes equivalent legacy users). Or maybe, as discussed, we decide that validation is too expensive and remove it. But weird to require some usually-independent remove_* actions in the same proposal.
| member_id = "([^/]+)" | ||
| opt_query = r"(\?.*)?" | ||
|
|
||
| if match := re.match( | ||
| f"^/gov/members/state-digests/{hex_id}:update{opt_query}$", | ||
| f"^/gov/members/state-digests/{member_id}:update{opt_query}$", |
There was a problem hiding this comment.
Does this regex work? I guess because we call match.
|
2 more open comments, since I'm not exactly sure where they go:
|
Stop rejecting enrollment of a public key that is already enrolled under the other ID format, in both the default constitution and genesis/C++ enrollment. Operators are responsible for managing duplicates. The constitution no longer inspects earlier actions in a proposal, and set_member/set_user apply no longer re-run validation. Remove the now-unused ccf.certificatesHaveSamePublicKey JS API and ccf::crypto::certs_have_same_public_key. Reject base64url (u) multibase did:key IDs with an explicit error and document that only base58-btc (z) is supported. Replace "canonical" with the specification's "deterministic" wording, with links to the did:key identifier syntax. Rework the migration guide around add-then-remove, and clarify how TLS client certificate authentication maps to did:key enrollments. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Validating a large RSA public key takes close to a second, and was reachable without authentication: governance read endpoints validated the key in a did:key path ID, and TLS client certificate authentication validated the key of any unknown caller. Only check the syntax of an ID which is used to look up an identity, and derive the did:key of a TLS caller, an enrolled COSE signer or a node without validating its key. Keys are still validated when a member or user is enrolled. Encode base58 ten digits per bignum division. Validate a decoded did:key as parsed from its SubjectPublicKeyInfo, as validating the key built from its raw encoding does not reject an invalid RSA modulus. Stop rejecting a node which joins again with the same key under a different ID format, as node keys are not reused. Document that P-521 and RSA keys use multicodec codes which the did:key specification does not list, and restore constitution coverage for runtimes without did:key support. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
Disk recovery omits the DID vote and recovery share for keys with both certificate and DID enrollments, potentially leaving migration recovery below required thresholds despite all private keys being present.
1 open issue
1 resolved since last review
Review effort: Balanced
| if service_id not in members: | ||
| id_format = "did:key" | ||
| service_id = infra.crypto.did_key_from_private_key( | ||
| slurp_file(os.path.join(self.common_dir, f)) | ||
| ) |
Resolve conflicts in python/src/ccf/cose.py and doc/operations/configuration.rst. create_cose_sign1_prepare keeps the certificate or did_key identity selection, and uses main's signing-algorithm digest and algorithm name for both. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Python SDK and the JS polyfill accepted RSA keys that a service refuses to enroll. Apply the deterministic part of the service's validation to RSA keys: an odd exponent greater than one, and an odd modulus with no prime factor below 752. The service additionally rejects a modulus which is prime or a power of a prime. When recovering a network from a saved common directory, the test infrastructure rebuilt a single member per key, so a key enrolled under both ID formats lost its second identity. Rebuild every enrollment, and cover a member enrolled under both formats in the did:key recovery test. Match a hex ID or a percent-encoded did:key explicitly when deriving COSE headers from a governance request path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🔵 Needs a closer look
Security-critical changes across identity, authentication, and recovery require human review, with current-head live-upgrade and extended ledger-recovery compatibility still unvalidated.
1 open issue
1 resolved since last review
Review effort: Balanced


Motivation
Remove the certificate dependency from member/user enrollment and COSE authentication, and use IDs containing public keys rather than hard-coded SHA-256 fingerprints.
Related to #2419; existing signature, receipt, and attestation digest algorithms are unchanged.
Implementation summary
did_key, withoutcertorid_format, and store DER public keys in newpublic_keysgovernance tables instead of identity certificates.did:keyenrollments as independent identities:set_memberandset_useronly validate their own arguments, and CCF does not detect a public key enrolled under both ID formats.z)did:keyIDs that are the deterministic encoding of a P-256, P-384, P-521, or 2048 to 16384-bit RSA key; the base64url (u) form is rejected.kid, nodes) are checked or derived without that validation.did:keynode IDs, legacy certificate enrollment, and TLS client-certificate compatibility: a client certificate carrying an enrolled key authenticates as thatdid:keyidentity.Safety and compatibility
Legacy certificate identities retain their SHA-256 IDs and existing behavior.
did:keyIDs embed supported public keys, have no certificate validity period, and are authorized and revoked explicitly through governance.They are enrolled identities, not aliases or authorization grants.
Operators are responsible for duplicate keys.
A key enrolled under both ID formats in one role is two identities: two members with a vote each and, if both are recovery members, a recovery share each.
Removing one enrollment does not revoke the other.
With both enrolled, TLS client authentication using the enrolled certificate resolves to the legacy ID.
Validating a 16384-bit RSA public key takes about 0.8 s, so it only runs on governance-authenticated enrollment.
Measured on a local node, an unauthenticated governance lookup of such a
did:keytakes under 1 ms, and an unknown TLS caller presenting such a certificate costs about 3 ms per request, compared with 0.6 ms for an EC certificate.P-521 and RSA keys use multicodec codes that the
did:keyspecification does not list, so other implementations may not accept those IDs.COSE verification uses the existing verifier and algorithms; the Python SDK signs with ECDSA only.
A node that joins again with a key that has already joined receives the existing node's response, whichever ID format it requests; node keys are not reused.
A newly enrolled member starts
Acceptedand must acknowledge the service state; the recovery threshold check still applies when the legacy member is removed.There is no backfill or automatic transfer of historical metadata or recovery state.
Upgrade every node before opting in to
did:keyidentities.The legacy constitution path does not use the new runtime functions, so it remains compatible with older nodes during rolling upgrades.
Validation
At the current head, after merging the latest
main:unit.governance_test(including the legacy-migration anddid:keygenesis scenarios),e2e_logging, andschema_testend-to-end tests.did:keysupport.scripts/ci-checks.shwithout auto-fix.Not rerun since enrollments were made independent: the live upgrade from released CCF 7.0.23, and
lts_compatibility.Full extended compatibility was not passing before that change.
With
-DLONG_TESTS=ON,cd build && ./tests.sh --output-on-failure -R '^lts_compatibility$' --no-tests=errorreproduced an existing failure while released CCF 6.0.28 initiates private recovery (HTTP 500), before that ledger-recovery sequence reaches the current build.Remaining extended ledger-compatibility cases and required GitHub CI checks must pass before merge.
Custom instructions used
Applied
.github/copilot-instructions.md,.github/instructions/reviewing.instructions.md,.github/instructions/changelog.instructions.md,.github/CONTRIBUTING.md, and thetestingandformatting-and-lintingskills.