Standalone processors are recompiled from scratch on every boot (~4s each, no compilation cache)

[devarismeroxa]

Summary

conduit run recompiles every .wasm under --processors.path on every boot. There is no wazero compilation cache anywhere in the repo — grep -rn "CompilationCache" returns nothing outside design text. With wazevo (the default on amd64/arm64) a 19 MB module costs ~3.8s, so a user with five installed processors pays ~19s of pure recompilation on every start, and it recurs on every restart, every crash-loop iteration, and every container start.

Found while investigating make test runtime (#2861); the test-side symptom is fixed there, this is the production half and it is the more valuable one.

Path

pkg/conduit/runtime.go:519   proc_standalone.NewRegistry
  -> standalone/registry.go:115  reloadPlugins
     -> loadPlugins: iterate every non-directory entry under --processors.path
        -> standalone/registry.go:273  r.runtime.CompileModule(ctx, wasmBytes)

newRuntime is wazero.NewRuntime with the default RuntimeConfig — no WithCompilationCache. The default processors path is <basePath>/processors (pkg/conduit/config.go:327), which is exactly where conduit processor-plugins install writes artifacts, so this hits anyone using standalone processors at all.

Measurements

Independently reproduced by two sessions, against the real 18.4 MB chaos fixture, no -race, M-series arm64:

configuration	run 1	run 2	run 3
compiler, no cache (what ships today)	3.811s	3.810s	—
interpreter	0.265s	0.264s	—
compiler + in-memory wazero.NewCompilationCache()	3.736s	3.811s	3.801s
compiler + NewCompilationCacheWithDir	3.793s	0.111s	0.112s

Two non-obvious results worth recording:

• An in-memory CompilationCache gives zero reuse. CompiledModule.Close() evicts the entry, so the obvious fix does nothing. Anyone attempting this should not stop when the in-memory version fails to help.
• Only NewCompilationCacheWithDir survives module close and process exit — 34x on compile alone, ~50x measured across the full boot path.

Why this needs a design doc rather than a patch

It introduces on-disk state Conduit does not have today, so at minimum:

• Location. A new directory, or under the existing state dir? Interaction with --processors.path, containers, and read-only root filesystems.
• Invalidation. Keyed on module bytes, wazero version, GOARCH, and Conduit version. A stale entry after a wazero bump would execute code compiled by a different compiler version — this is the part that needs care.
• Config surface. On by default? A --processors.compilation-cache.* family? An opt-out for reproducibility-sensitive deployments.
• Permissions and multi-tenancy. A shared cache directory is a code-execution surface; two Conduit instances as different users must not be able to poison each other.
• Failure mode. A corrupt or unwritable cache must degrade to today's behavior, never fail startup.

Architecture note

This only bites where wazevo is active. NewRuntimeConfig() is engineKindAuto and selects the compiler only when platform.CompilerSupported() — amd64/arm64 on the mainstream OSes. On linux/386 (which this repo has a required build check for), production already runs the interpreter at ~0.26s and there is nothing to cache.

Suggested acceptance

• Second and subsequent boots with unchanged processor binaries skip compilation, demonstrated with a timing test.
• A wazero version bump, a Conduit version bump, or a changed module invalidates the entry.
• An unwritable or corrupt cache directory logs and falls back to compiling, never fails startup.