Skip to content

Add did:key node IDs and certificate-free member/user identities - #8602

Open
Amaury Chamayou (achamayou) wants to merge 16 commits into
microsoft:mainfrom
achamayou:achamayou-issue-2419-support-more-digest-algorithms-dc20ad
Open

Amaury Chamayou (achamayou) wants to merge 16 commits into
microsoft:mainfrom
achamayou:achamayou-issue-2419-support-more-digest-algorithms-dc20ad

Conversation

@achamayou

@achamayou Amaury Chamayou (achamayou) commented Oct 19, 2026 •

Copy link
Copy Markdown
Member

Motivation

Remove the certificate dependency from member/user enrollment and COSE authentication, and use IDs containing public keys rather than hard-coded SHA-256 fingerprints.
Related to #2419; existing signature, receipt, and attestation digest algorithms are unchanged.

Implementation summary

  • Enroll members and users directly with did_key, without cert or id_format, and store DER public keys in new public_keys governance tables instead of identity certificates.
  • Authenticate COSE requests against enrolled public keys; support certificate-free genesis members, Python SDK/CLI signing, and recovery-share submission.
  • Treat certificate and did:key enrollments as independent identities: set_member and set_user only validate their own arguments, and CCF does not detect a public key enrolled under both ID formats.
  • Accept only base58-btc (z) did:key IDs that are the deterministic encoding of a P-256, P-384, P-521, or 2048 to 16384-bit RSA key; the base64url (u) form is rejected.
  • Validate the public key when a member or user is enrolled. IDs used to look up an identity (path parameters, TLS callers, COSE kid, nodes) are checked or derived without that validation.
  • Preserve opt-in did:key node IDs, legacy certificate enrollment, and TLS client-certificate compatibility: a client certificate carrying an enrolled key authenticates as that did:key identity.
  • Update caller types and API/configuration schemas, and document the certificate-free workflow and add-then-remove migration in the migration guide.

Safety and compatibility

Legacy certificate identities retain their SHA-256 IDs and existing behavior.
did:key IDs embed supported public keys, have no certificate validity period, and are authorized and revoked explicitly through governance.
They are enrolled identities, not aliases or authorization grants.

Operators are responsible for duplicate keys.
A key enrolled under both ID formats in one role is two identities: two members with a vote each and, if both are recovery members, a recovery share each.
Removing one enrollment does not revoke the other.
With both enrolled, TLS client authentication using the enrolled certificate resolves to the legacy ID.

Validating a 16384-bit RSA public key takes about 0.8 s, so it only runs on governance-authenticated enrollment.
Measured on a local node, an unauthenticated governance lookup of such a did:key takes under 1 ms, and an unknown TLS caller presenting such a certificate costs about 3 ms per request, compared with 0.6 ms for an EC certificate.

P-521 and RSA keys use multicodec codes that the did:key specification does not list, so other implementations may not accept those IDs.
COSE verification uses the existing verifier and algorithms; the Python SDK signs with ECDSA only.
A node that joins again with a key that has already joined receives the existing node's response, whichever ID format it requests; node keys are not reused.

A newly enrolled member starts Accepted and must acknowledge the service state; the recovery threshold check still applies when the legacy member is removed.
There is no backfill or automatic transfer of historical metadata or recovery state.
Upgrade every node before opting in to did:key identities.
The legacy constitution path does not use the new runtime functions, so it remains compatible with older nodes during rolling upgrades.

Validation

At the current head, after merging the latest main:

  • All 66 tests labelled unit.
  • The governance_test (including the legacy-migration and did:key genesis scenarios), e2e_logging, and schema_test end-to-end tests.
  • The complete Python SDK suite: 726 tests.
  • JavaScript build and 80 tests, including coexisting enrollments and a runtime without did:key support.
  • Full warning-as-error Sphinx build and scripts/ci-checks.sh without auto-fix.

Not rerun since enrollments were made independent: the live upgrade from released CCF 7.0.23, and lts_compatibility.
Full extended compatibility was not passing before that change.
With -DLONG_TESTS=ON, cd build && ./tests.sh --output-on-failure -R '^lts_compatibility$' --no-tests=error reproduced an existing failure while released CCF 6.0.28 initiates private recovery (HTTP 500), before that ledger-recovery sequence reaches the current build.
Remaining extended ledger-compatibility cases and required GitHub CI checks must pass before merge.

Custom instructions used

Applied .github/copilot-instructions.md, .github/instructions/reviewing.instructions.md, .github/instructions/changelog.instructions.md, .github/CONTRIBUTING.md, and the testing and formatting-and-linting skills.

Keep SHA-256 defaults and digest behavior unchanged while supporting canonical public-key IDs throughout enrollment, authentication, node lifecycle, and SDK tooling. Preserve legacy enrollment during CCF 7 rolling upgrades and cover mixed-format lifecycle and recovery.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@achamayou
Amaury Chamayou (achamayou) requested a review from a team as a code owner October 19, 2026 17:20
Copilot AI balanced review requested due to automatic review settings October 19, 2026 17:20
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Disk-based DID recovery and encoded node requests remain broken, and legacy enrollment adds quadratic table scanning.

3 open issues
What changed in this PR

Adds opt-in public-key identities across CCF while retaining SHA-256 defaults. This supports same-key certificate replacement, not the digest agility requested in #2419.

Changes:

  • Adds canonical did:key derivation, enrollment, and authentication.
  • Integrates identity formats into joining, recovery, signing tools, and governance paths.
  • Updates schemas, migration documentation, and regression coverage.

Custom instructions used

  • .github/instructions/reviewing.instructions.md
File Description
tests/​lts_compatibility.py Checks legacy enrollment during upgrades.
tests/​infra/​remote.py Propagates node identity format.
tests/​infra/​openapi.py Decodes schema-validation path parameters.
tests/​infra/​node.py Supports format-aware node identities.
tests/​infra/​network.py Configures member and user formats.
tests/​infra/​member.py Supports DID signing and recovery paths.
tests/​infra/​crypto.py Adds identity derivation helpers.
tests/​infra/​consortium.py Passes enrollment formats.
tests/​infra/​clients.py Supports DID COSE identities.
tests/​governance.py Adds DID lifecycle and recovery scenarios.
tests/​config.jinja Emits node identity configuration.
src/​node/​startup_inputs.h Passes genesis member formats.
src/​node/​rpc/​test/​node_frontend_test.cpp Tests DID configuration and joining.
src/​node/​rpc/​test/​internal_tables_access_test.cpp Tests enrollment conflict handling.
src/​node/​rpc/​test/​frontend_test.cpp Tests DID certificate authentication.
src/​node/​rpc/​serialization.h Serializes join identity format.
src/​node/​rpc/​self_cert_auth.h Matches the local node's format.
src/​node/​rpc/​node_frontend.h Handles DID joining and duplicates.
src/​node/​rpc/​node_call_types.h Adds join format selection.
src/​node/​node_state.h Derives and confirms node IDs.
src/​node/​internal_tables_access.h Enforces enrollment format constraints.
src/​node/​gov/​handlers/​helpers.h Decodes and validates identity paths.
src/​node/​gov/​handlers/​acks.h Decodes member action paths.
src/​node/​gov/​gov_endpoint_registry.h Updates identity schemas.
src/​js/​extensions/​ccf/​converters.cpp Exposes native identity converters.
src/​endpoints/​test/​endpoint_registry.cpp Tests encoded DID action paths.
src/​endpoints/​authentication/​cose_auth.cpp Checks enrolled DID key binding.
src/​endpoints/​authentication/​cert_auth.cpp Adds DID certificate lookup.
src/​enclave/​enclave.h Passes configured node format.
src/​crypto/​test/​crypto.cpp Tests DID encoding and key equality.
src/​crypto/​did_key.cpp Implements canonical DID conversion.
samples/​constitutions/​default/​actions.js Adds DID enrollment safeguards.
python/​utils/​submit_recovery_share.sh Supports DID recovery signing.
python/​tests/​test_identity.py Tests SDK identity derivation.
python/​tests/​test_cose_sign.py Tests both signing formats.
python/​src/​ccf/​identity.py Adds identity helpers and CLI.
python/​src/​ccf/​cose.py Adds signing format selection.
js/​ccf-app/​test/​polyfill.test.ts Tests identity polyfills.
js/​ccf-app/​src/​polyfill.ts Implements local identity conversion.
js/​ccf-app/​src/​global.ts Declares identity APIs.
include/​ccf/​service/​tables/​users.h Adds user enrollment format.
include/​ccf/​service/​tables/​nodes.h Adds format-aware node derivation.
include/​ccf/​service/​tables/​members.h Adds member enrollment format.
include/​ccf/​node/​configuration.h Adds startup format options.
include/​ccf/​identity_id_format.h Defines supported formats.
include/​ccf/​entity_id.h Validates and describes DID IDs.
include/​ccf/​crypto/​did_key.h Declares DID conversion APIs.
include/​ccf/​crypto/​cert_id.h Adds certificate identity derivation.
doc/​use_apps/​issue_commands.rst Documents DID signing options.
doc/​schemas/​node_openapi.json Allows DID node IDs.
doc/​schemas/​gov/​2024-07-01/​gov.json Updates stable governance schemas.
doc/​schemas/​gov/​2023-06-01-preview/​gov.json Updates preview governance schemas.
doc/​schemas/​gov_openapi.json Updates latest governance schemas.
doc/​operations/​configuration.rst Documents startup identity settings.
doc/​host_config_schema/​host_config.json Describes format configuration fields.
doc/​governance/​open_network.rst Documents DID user enrollment.
doc/​governance/​adding_member.rst Documents DID member workflows.
doc/​build_apps/​auth/​cert.rst Explains replacement-certificate authentication.
doc/​architecture/​cryptography.rst Defines identity and migration semantics.
cmake/​crypto.cmake Includes DID implementation in builds.
CHANGELOG.md Announces opt-in identity support.

Review effort: Balanced

Comment thread tests/infra/member.py Resolved
Comment thread src/node/rpc/node_frontend.h Resolved
Use a one-byte identity format enum and explicit QuickJS boolean conversions to satisfy Virtual A's clang-tidy checks.

Normalize DER-imported RSA public keys through JWK before canonical PKCS#1 re-encoding. This preserves DID encoding while fixing the Node 24 failure in modules_test and programmability_and_jwt.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Preserve enrolled member formats and recovery roles during disk-based recovery. Avoid scanning legacy certificate rosters and decode node identity path parameters, with native and end-to-end regression coverage.

Allow certificate DID results to move into optional returns, fixing the required clang-tidy CI failure.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…errors

- set_member no longer no-ops for an already-enrolled did:key member: it
  replaces the stored certificate, encryption key and member data and
  resets the member to Accepted, exactly like legacy members.
- Join requests whose caller key cannot be identified in the requested
  ID format now fail with 400 instead of 500; DID derivation failures in
  cert/node/self authentication policies fall through to generic reasons
  rather than leaking parser error text.
- Clarify in docs that enrolled-certificate validity does not bound
  access for DID identities.
- Add JS unit, native and e2e regression coverage.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Refresh against main at 9cdc86c without rewriting published history. Adapt the unsupported-key join regression to the updated certificate helper's separate validity arguments, and move the DID release note and SDK version to the next patch, 7.0.24.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Store public keys without identity certificates, authenticate DID COSE requests directly, and migrate legacy identities atomically in ordered governance proposals. Update genesis, SDK/CLI, recovery tooling, schemas, regression coverage, and the RST migration guide.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 264fbdd6871f4086ab3af9b5484907c3
@achamayou Amaury Chamayou (achamayou) changed the title Add opt-in did:key identities for nodes, members, and users Add did:key node IDs and certificate-free member/user identities Nov 8, 2026
Retain legacy certificate migration while generating and recovering fresh DID identities without certificates.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 264fbdd6871f4086ab3af9b5484907c3
Integrate current COSE dependencies, crypto support, and documentation checks while preserving explicit DID signing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 264fbdd6871f4086ab3af9b5484907c3
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 264fbdd6871f4086ab3af9b5484907c3
@achamayou
Amaury Chamayou (achamayou) marked this pull request as ready for review November 9, 2026 18:16
Copilot AI balanced review requested due to automatic review settings November 9, 2026 18:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Batch validation copies the full action prefix for every validator, causing quadratic copying and allocation even when no migration is requested.

1 open issue
3 resolved since last review

Review effort: Balanced

@eddyashton Eddy Ashton (eddyashton) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comments from initial read.

Comment thread doc/architecture/cryptography.rst Resolved
Comment thread doc/architecture/cryptography.rst Outdated
CCF supports compressed P-256, P-384, and P-521 keys and PKCS#1 RSA public keys between 2048 and 16384 bits.
Existing authentication and signing algorithm restrictions still apply.
Only bare, canonical IDs are accepted; DID URLs with fragments, queries, or paths are not identity IDs.
When inserting an ID into an HTTP path, percent-encode its colons as ``%3A`` rather than encoding the endpoint's literal action suffix.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is surprising! I thought dids were deliberately "URL-safe" such that they didn't need encoding? Is this a flaw in our parser?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good point, I will check again.

Comment thread doc/architecture/cryptography.rst Resolved
Comment thread doc/governance/open_network.rst Resolved
Comment thread python/src/ccf/identity.py Resolved
.map((action) => action.args?.[`${role}_id`]),
);
if (!isDid) {
if (typeof ccf.certificatesHaveSamePublicKey !== "function") {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we checking for this function (legitimate to avoid calling it on old nodes!), when we don't even call it?

Comment on lines +276 to +282
// Validation sees pre-proposal state; application checks the actual tables.
const role = tableName === "public:ccf.gov.members.certs" ? "member" : "user";
const removedIds = new Set(
previousActions
.filter((action) => action.name === `remove_${role}`)
.map((action) => action.args?.[`${role}_id`]),
);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't like this bit. I don't like calling this function (which does validation) again during application. No need to do that! Also we shouldn't be constraining to previous actions; if they eventually write to different tables, then the application order doesn't matter. I don't like the action-at-a-distance reconstruction of another action name (remove_*).

I think we could get the same result by modifying the existing (idempotent!) set_* actions to do all of this. The replacement story is that you say set_user: did_key=..., and internally that does some validation and dedup (removes equivalent legacy users). Or maybe, as discussed, we decide that validation is too expensive and remove it. But weird to require some usually-independent remove_* actions in the same proposal.

Comment thread tests/infra/clients.py Outdated
Comment on lines +379 to +383
member_id = "([^/]+)"
opt_query = r"(\?.*)?"

if match := re.match(
f"^/gov/members/state-digests/{hex_id}:update{opt_query}$",
f"^/gov/members/state-digests/{member_id}:update{opt_query}$",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this regex work? I guess because we call match.

@eddyashton

Copy link
Copy Markdown
Member

2 more open comments, since I'm not exactly sure where they go:

  • The docs say "canonical" did-key in multiple places. I don't find "canonical" at all in the linked spec. What is a canonical (or non-canonical) did-key?
  • We claim to support did-key, but seems we actually require the base-58 encoding. From this bit of spec, the prefix is either did:key:z for base-58-btc encoding, or did:key:u for base-64-url-no-pad. I think it's fine for us to always produce a base-58 (as every example in the did-key spec does), but should we accept/parse keys that come as base-64? I sort of can't believe that did:key has no test vectors for this, despite being clear in the Identifier Syntax sectionthat u/base64url is accepted: mb-value := (z[a-km-zA-HJ-NP-Z1-9]+|u[A-Za-z0-9_-]+)

Stop rejecting enrollment of a public key that is already enrolled under
the other ID format, in both the default constitution and genesis/C++
enrollment. Operators are responsible for managing duplicates. The
constitution no longer inspects earlier actions in a proposal, and
set_member/set_user apply no longer re-run validation.

Remove the now-unused ccf.certificatesHaveSamePublicKey JS API and
ccf::crypto::certs_have_same_public_key.

Reject base64url (u) multibase did:key IDs with an explicit error and
document that only base58-btc (z) is supported. Replace "canonical"
with the specification's "deterministic" wording, with links to the
did:key identifier syntax.

Rework the migration guide around add-then-remove, and clarify how TLS
client certificate authentication maps to did:key enrollments.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Validating a large RSA public key takes close to a second, and was
reachable without authentication: governance read endpoints validated
the key in a did:key path ID, and TLS client certificate authentication
validated the key of any unknown caller.

Only check the syntax of an ID which is used to look up an identity, and
derive the did:key of a TLS caller, an enrolled COSE signer or a node
without validating its key. Keys are still validated when a member or
user is enrolled. Encode base58 ten digits per bignum division.

Validate a decoded did:key as parsed from its SubjectPublicKeyInfo, as
validating the key built from its raw encoding does not reject an
invalid RSA modulus.

Stop rejecting a node which joins again with the same key under a
different ID format, as node keys are not reused. Document that P-521
and RSA keys use multicodec codes which the did:key specification does
not list, and restore constitution coverage for runtimes without
did:key support.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 16, 2026 09:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Disk recovery omits the DID vote and recovery share for keys with both certificate and DID enrollments, potentially leaving migration recovery below required thresholds despite all private keys being present.

1 open issue
1 resolved since last review

Review effort: Balanced

Comment thread tests/infra/network.py Outdated
Comment on lines +790 to +794
if service_id not in members:
id_format = "did:key"
service_id = infra.crypto.did_key_from_private_key(
slurp_file(os.path.join(self.common_dir, f))
)
Resolve conflicts in python/src/ccf/cose.py and
doc/operations/configuration.rst. create_cose_sign1_prepare keeps the
certificate or did_key identity selection, and uses main's
signing-algorithm digest and algorithm name for both.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 16, 2026 10:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Recovery reconstructs only the legacy identity when both legacy and DID enrollments exist, omitting votes or shares needed to complete recovery despite all required keys being available.

2 open issues

Review effort: Balanced

Comment thread js/ccf-app/src/polyfill.ts Resolved
The Python SDK and the JS polyfill accepted RSA keys that a service
refuses to enroll. Apply the deterministic part of the service's
validation to RSA keys: an odd exponent greater than one, and an odd
modulus with no prime factor below 752. The service additionally
rejects a modulus which is prime or a power of a prime.

When recovering a network from a saved common directory, the test
infrastructure rebuilt a single member per key, so a key enrolled under
both ID formats lost its second identity. Rebuild every enrollment, and
cover a member enrolled under both formats in the did:key recovery test.

Match a hex ID or a percent-encoded did:key explicitly when deriving
COSE headers from a governance request path.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 16, 2026 15:03

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Security-critical changes across identity, authentication, and recovery require human review, with current-head live-upgrade and extended ledger-recovery compatibility still unvalidated.

1 open issue
1 resolved since last review

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants