Skip to content

Add core TypeScript submodules: cron, retry, rate-limit, and crypto - #5903

Open
bradleyshep wants to merge 40 commits into
masterfrom
bradley/submodules-core
Open

bradleyshep wants to merge 40 commits into
masterfrom
bradley/submodules-core

Conversation

@bradleyshep

Copy link
Copy Markdown
Contributor

Description of Changes

Adds reusable scheduling, retry, rate-limit, and cryptographic helpers for SpacetimeDB TypeScript modules.

  • Cron: calendar and interval jobs with typed arguments, time zones, and run history.
  • Retry: scheduled attempts, exponential backoff, and attempt history.
  • Rate-limit: fixed-window limits with configuration and administrative controls.
  • Crypto: hashing, encoding, and webhook-signature helpers.
  • Shared example UI/server support, package documentation, tests, and examples.
  • Registers the packages in the pnpm workspace and existing lint/build commands. Adds a submodule CI job for type checks, tests, and builds.

This is the prerequisite branch for the remaining submodule groups. It is based on master commit 3653d2ed4.

Example screenshots

Existing example screenshots from #5823:

Cron

Cron example

Rate Limit

Rate Limit example

API and ABI breaking changes

No existing SpacetimeDB API or ABI is changed. This adds new TypeScript package APIs that need review before merge.

Rollback safety impact

n/a. This adds opt-in TypeScript packages and examples; it does not change existing server storage formats.

Expected complexity level and risk

3/5

The packages are opt-in and do not change existing server behavior. Complexity is in scheduling, retry recovery, and cron's direct use of the internal spacetime:sys@2.0 host ABI. Review failure recovery and repeated execution for applications that use these packages.

Testing

Verified locally on bradley/submodules-core:

  • Frozen-lockfile install for the root, SDK, and present submodule workspaces.
  • Type checks for the added packages.
  • Existing test scripts for the added packages and examples.
  • Build scripts for the added packages and examples.
  • Regenerate client bindings and verify no tracked changes.
  • Lint and formatting checks for the added packages and examples.
  • Verify workspace dependencies are present in this branch.
  • Review the public APIs, documentation, and cron host-ABI dependency.
  • Run local scheduling/recovery smoke tests against a running SpacetimeDB instance.

Commands used for this group:

pnpm -r -F "./spacetime-cron-ts/**" -F "./spacetime-retry-ts/**" -F "./spacetime-rate-limit-ts/**" -F "./spacetime-crypto-ts/**" -F "./spacetime-submodule-shared-ts/**" run typecheck
pnpm -r -F "./spacetime-cron-ts/**" -F "./spacetime-retry-ts/**" -F "./spacetime-rate-limit-ts/**" -F "./spacetime-crypto-ts/**" -F "./spacetime-submodule-shared-ts/**" run test
pnpm -r -F "./spacetime-cron-ts/**" -F "./spacetime-retry-ts/**" -F "./spacetime-rate-limit-ts/**" -F "./spacetime-crypto-ts/**" -F "./spacetime-submodule-shared-ts/**" run build
pnpm -r -F "./spacetime-cron-ts/**" -F "./spacetime-retry-ts/**" -F "./spacetime-rate-limit-ts/**" -F "./spacetime-crypto-ts/**" -F "./spacetime-submodule-shared-ts/**" run lint

The TypeScript SDK was built first. Live deployment and provider tests were not run during split validation. Existing workspace peer-dependency warnings remain.

@bradleyshep bradleyshep mentioned this pull request Sep 9, 2026
5 of 7 tasks
@bradleyshep
bradleyshep requested a review from aasoni September 10, 2026 12:56

@clockwork-tien clockwork-tien left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are places where we are using snake_case instead of camelCase, worth checking thoroughly to ensure consistency. I have also left additional comments inline.

Comment thread spacetime-crypto-ts/package.json Outdated
Comment thread spacetime-rate-limit-ts/example/spacetimedb/src/index.ts Outdated
Comment thread spacetime-rate-limit-ts/example/spacetimedb/src/index.ts Outdated
Comment thread spacetime-rate-limit-ts/example/spacetimedb/src/index.ts Outdated
Comment thread spacetime-rate-limit-ts/example/spacetimedb/src/index.ts Outdated
Comment thread spacetime-retry-ts/src/submodule.ts Outdated
Comment thread spacetime-retry-ts/README.md Outdated
Comment thread spacetime-cron-ts/README.md Outdated
Comment thread spacetime-crypto-ts/scripts/test-vectors.ts Outdated
Comment thread spacetime-retry-ts/src/submodule.ts Outdated

@clockwork-tien clockwork-tien left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed a lot of the public surface is only public because the settings are passed in on every call instead of being encapsulated by the library. I would be interested to hear if there is reasoning behind this.

The example writes its own wrapper in consumer code to call the rate limiter which should be handled internally by the library. Here buildRateLimitKey is exported only so a caller can build the key, and scope is passed in twice. With the current design the caller has to build the argument, so the builder has to be exported.

function consumeAction(
  tx: Tx,
  scope: string,
  actorKey: string,
  limit: number,
  windowSeconds: number,
  cost = 1,
) {
  return rateLimit.consumeRateLimit(tx.as.rateLimit, {
    key: rateLimit.buildRateLimitKey(scope, actorKey),
    scope,
    limit,
    windowSeconds,
    cost,
  });
}

const result = consumeAction(tx, TAP_SCOPE, key, tapLimit, TAP_WINDOW_SECONDS);

Possible improvements, one way is a client(config) per package:

const tapLimiter = rateLimit.client({ scope: TAP_SCOPE, windowSeconds: TAP_WINDOW_SECONDS });

const result = tapLimiter.consume(tx.as.rateLimit, { key, limit: tapLimit });

A few more points:

  • Worth exporting the error codes, seems there are quite a few of them, e.g. rate_limit.not_authorized, since renaming one breaks callers silently
  • Worth having a setup function per package with consistent naming and signature, client(config) when the module loads and install(ctx) in init (resend.install rather than resend.installResend), instead of
spacetimeCron(sdk) + createCron(jobs, opts) // needs both
createRetrySubmodule(deps, handlers, auth?)
installRateLimit(ctx)   
installRateLimitState(ctx, opts?)
installResend(ctx)
installRetry(ctx)                  

Comment thread spacetime-rate-limit-ts/src/submodule.ts Outdated
Comment thread spacetime-rate-limit-ts/src/submodule.ts Outdated
Comment thread spacetime-rate-limit-ts/src/submodule.ts Outdated
Comment thread spacetime-rate-limit-ts/src/limit.ts Outdated
Comment thread spacetime-rate-limit-ts/src/limit.ts Outdated
Comment thread spacetime-crypto-ts/src/vendors.ts Outdated
Comment thread spacetime-rate-limit-ts/src/submodule/schema.ts Outdated
Comment thread spacetime-rate-limit-ts/src/submodule/operations.ts Outdated
Comment thread spacetime-crypto-ts/src/index.ts Outdated
Comment thread spacetime-crypto-ts/src/timing.ts Outdated
Configure fixed policies once, expose stable errors, keep installers idempotent, and support admin revocation. Remove internal sweep exports and align init naming. Verify packed consumers with the default TypeScript template.
Drop the install options, the standalone install and sweep exports, and helpers exported only for tests. runSweep and resetBuckets share one maxRows validation.
client({ handlers }) imports the SDK directly and returns the tables,
install, submit, views, and reducers. Contexts are typed against the
retry tables instead of unknown, and thrown codes come from an exported
errors object.

retryHandler no longer defines a property on the caller's type builder,
so one builder can back several handlers. The custom auth policy is
replaced by submit(), which host reducers call after their own checks.

History is written once per attempt with its final status, pruned by
removing the oldest row through the ranAt index, and served newest first
without sorting. The unused taskName and status indexes are removed,
admin rows store addedAt as a timestamp, and the task view returns the
newest tasks.
verifyStripeSignature, verifySvixSignature, and verifyGithubSignature
return { ok: true } or { ok: false, reason } with a code from the
exported errors object, so callers can tell a missing signature, a bad
or stale timestamp, an invalid Svix secret, and a mismatch apart.

Also remove the unused SHA256_INTERNAL_BLOCK_SIZE constant, make
timingSafeEqual compile under noUncheckedIndexedAccess and enable that
check, and drop allowImportingTsExtensions now that tests import
without extensions.
Limiters are configured once per scope and expose their policy and a peek read. Remove per-call limit overrides, the admin consume procedure, and the raw consume and key exports. Export isAdmin and requireAdmin for host operations. The example uses one limiter per upgrade tier, reads status through peek, and uses reactor.* codes for its own errors. Regenerate example bindings.
cron imports the SDK directly instead of receiving it as a parameter,
so tables and types come from the real SDK and the dynamic SDK casts
are gone. client({ jobs, ...options }) replaces spacetimeCron(sdk) and
createCron(jobs, opts) and returns tables, schedule, unschedule,
reconcileReducer, and publicViews. cronTable() stays the job
declaration.

cronReducer and cronProcedure infer the handler context from the host
schema, and registration fails to compile when the schema lacks
cron.tables. cron.tables is typed per job, so host code can read fire
tables without casts. Thrown codes come from an exported errors object.

Also set the unpublished package version to 0.1.0, describe Failed runs
accurately, drop decorative separators from the modules, and remove
allowImportingTsExtensions.
client({ tasks }) declares each task's argument type before schema().
retry.retryReducer(spacetimedb, handlers) registers the scheduled
reducer afterward, so every handler receives the host's reducer context
and its task's arguments with their real types, and the host schema must
include retry.tables. retryHandler is removed along with the fixture's
context cast.
Libraries that create tables, schemas, and enum builders with this package could not emit .d.ts files because the inferred types named modules outside the package exports (TS2742) or an unexported class with private members (TS4094). The types are exported by name only; runtime exports are unchanged.
cron, crypto, rate-limit, and retry build ESM JavaScript and .d.ts files
into dist with tsc (tsconfig.build.json). pnpm pack and publish apply
publishConfig, which points main, types, and exports at dist and keeps
the subpath names; files lists dist, LICENSE, and README.md. prepack
builds before copying the license. Inside the workspace the manifests
still point at src, so examples, fixture modules, and tests do not need
a build first.

Relative imports in src carry .js extensions so the emitted modules
resolve in Node as well as in bundlers. rate-limit drops
allowImportingTsExtensions like the other packages.
@bradleyshep

Copy link
Copy Markdown
Contributor Author

Changes since review

  • Every package now uses client(config) for setup and install(ctx) in init. Cron replaces spacetimeCron + createCron. Retry handlers are registered after schema() and receive the host's typed context.
  • Rate-limit policy lives only in the client, which exposes consume and peek. Configuring the same scope twice throws, and the last admin cannot be removed.
  • Crypto verifiers return { ok, reason }. Each package exports its error codes as an errors object.
  • Packages ship compiled dist on pack; the workspace still resolves to src. Emitting declarations required exporting a few type names from spacetimedb/server; there are no runtime changes to the SDK.

Scheduled procedures run in a pool of instances, so later occurrences can
execute while the recovery probe blocks and before the host is killed. The
fireCount bound counted those runs as replays. Assert on runs scheduled
before the restart and completed after the stop, and use a two-second
schedule so the pre-kill checks finish before the next occurrence.
An index scan in a transaction yields rows inserted by that transaction
first, so pruning after the insert deleted the new attempt instead of
the oldest one once history reached 1,000 rows.
Comment thread spacetime-rate-limit-ts/src/limit.ts
Comment thread spacetime-retry-ts/src/submodule.ts
Comment thread .github/workflows/ci.yml Outdated
Comment thread .github/workflows/ci.yml Outdated
Comment thread spacetime-rate-limit-ts/src/limit.ts
Comment thread spacetime-rate-limit-ts/src/limit.ts
Comment thread spacetime-submodule-shared-ts/package.json Outdated
Comment thread spacetime-cron-ts/README.md Outdated

@JulienLavocat JulienLavocat left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

README LGTM now!

@bradleyshep
bradleyshep enabled auto-merge October 7, 2026 14:40
# Description of Changes

Adds a Resend submodule for sending transactional email and tracking
delivery state in SpacetimeDB.

- Synchronous procedures for outbound Resend requests and email
operations.
- Webhook signature verification, payload validation, idempotent
ingestion, and replay.
- Private delivery state and administrative configuration.
- Package documentation, unit tests, and a dispatch example with
generated client bindings.

Builds on `bradley/submodules-core`. The package uses crypto; the
example also uses rate-limit and shared example support.

## Example screenshots

Existing example screenshots from #5823:

### Resend

![Resend
example](https://github.com/user-attachments/assets/d13860fb-e4e9-481d-88a1-8f38c6a494ce)

# API and ABI breaking changes

No existing SpacetimeDB API or ABI is changed. This adds new TypeScript
package APIs that need review before merge.

# Rollback safety impact

n/a. This adds opt-in TypeScript packages and examples; it does not
change existing server storage formats.

# Expected complexity level and risk

**2/5**

An isolated, opt-in Resend integration that does not change existing
server behavior. For applications that use it, review delivery
idempotency, webhook verification, and access to provider credentials.

# Testing

Verified locally on `bradley/submodules-email`:

- [x] Frozen-lockfile install for the root, SDK, and present submodule
workspaces.
- [x] Type checks for the added packages.
- [x] Existing test scripts for the added packages and examples.
- [x] Build scripts for the added packages and examples.
- [x] Regenerate client bindings and verify no tracked changes.
- [x] Lint and formatting checks for the added packages and examples.
- [x] Verify workspace dependencies are present in this branch.
- [ ] Review the public email API, authorization, and webhook handling.
- [ ] Run the Resend smoke test with credentials and a local SpacetimeDB
instance.

Commands used for this group:

```sh
pnpm -r -F "./spacetime-resend-ts/**" run typecheck
pnpm -r -F "./spacetime-resend-ts/**" run test
pnpm -r -F "./spacetime-resend-ts/**" run build
pnpm -r -F "./spacetime-resend-ts/**" run lint
```

The TypeScript SDK was built first. Live deployment and provider tests
were not run during split validation. Existing workspace peer-dependency
warnings remain.
Comment thread .github/workflows/ci.yml

@cloutiertyler cloutiertyler left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Except for the one comment, this LGTM

@bradleyshep
bradleyshep disabled auto-merge October 9, 2026 20:48
@bradleyshep
bradleyshep added this pull request to the merge queue Oct 10, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 10, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants