Repository navigation
Description
(via Claude)
Description
Build-time shader compilation (FXC for D2D1, DXC for D3D12) currently runs inside the ForAttributeWithMetadataName transform callbacks in D2DPixelShaderDescriptorGenerator and ComputeShaderDescriptorGenerator. Roslyn's incremental driver invokes transform callbacks sequentially on one thread, so all shader compilations in a project are fully serialized, and their cost adds up linearly with shader body complexity.
I'm proposing moving the compilation out of the transform and into a dedicated pipeline node that compiles all shaders in a project in parallel, warming the existing DynamicCache, with a Combine-ordered join node folding results back into the per-shader models.
Rationale
Paint.NET has several projects with a lot of nontrivial D2D1 pixel shaders, and the serial compilation dominates their build times. Profiling the compiler server during a warm rebuild of the largest one shows Roslyn's own parse/bind/emit phases bursting across many cores, separated by a long plateau pinned at exactly ~1.00 core — and stubbing all shader Execute() bodies removes the plateau, i.e. the plateau is the FXC time.
Everything needed for parallelism is already in place:
HlslBytecodeInfoKeyis a pure content key (HLSL source + profile + options), so compilations are independent.DynamicCache<TKey, TValue>is already thread-safe (ConcurrentDictionary-backed; a same-key race is benign — both compile, one result is discarded).D3DCompilein d3dcompiler_47 is documented thread-safe and stateless, and the DXC wrapper already uses a[ThreadStatic]per-thread compiler instance.
Nothing invokes any of it concurrently today.
Measured effect in Paint.NET (warm --no-incremental Release x64 rebuilds, second of two runs, stock 3.2.0 vs. a locally packed build of the proposed change):
| Project (shader-heavy) | 3.2.0 (serial) | parallel | delta |
|---|---|---|---|
| Effects.Gpu (51 shaders) | 21.4s | 13.5s | -7.9s (-37%) |
| Windows.Direct2D1.Managed (51 shader files) | 12.6s | 9.0s | -3.6s (-29%) |
| Windows.Framework (546 mostly small shaders) | 11.2s | 7.6s | -3.6s (-32%) |
Expected wall time for the compile phase goes from sum of all shader compile times to roughly max(slowest shader, total / cores).
Proposed API
None — this is an internal change to the source generators. No public API surface is affected, and the generated code is byte-for-byte identical (verified across Paint.NET's ~660 shaders).
Sketch of the design
// Transform stops at building HlslBytecodeInfoKey (no compilation), and captures the
// info needed to synthesize compile diagnostics later (by value, no symbols/trees).
// One node compiles everything in parallel into the existing memoizing cache:
IncrementalValueProvider<bool> hlslBytecodeCache =
shaderInfoWithNoHlslBytecode
.Select(static (item, _) => item.HlslInfoKey)
.Collect()
.Select(static (keys, token) =>
{
HlslBytecodeSyntaxProcessor.CompileAllInParallel(keys, token); // Parallel.ForEach over distinct keys
return true;
});
// Per-shader join, ordered after the warm-up by the Combine edge; pure cache hit:
IncrementalValuesProvider<D2D1ShaderInfo> shaderInfo =
shaderInfoWithNoHlslBytecode
.Combine(hlslBytecodeCache)
.Select(static (pair, token) => /* fold in bytecode + synthesize deferred diagnostics */);The ordering is enforced by the dataflow graph itself (the join's input depends on the warm-up node's output), not by assumptions about driver scheduling. DynamicCache is untouched.
One consequence: the two compile-related diagnostics (GetInfoDiagnostics, GetDoublePrecisionSupportDiagnostics) can no longer be created in the transform, since they need the type symbol and symbols must not escape the transform. The location/name info is instead captured by value (file path + TextSpan + LinePositionSpan) into the model, and the diagnostics are synthesized in the join using Location.Create(path, span, lineSpan). Capturing by value (rather than holding a SyntaxTree reference) keeps transform outputs equatable across unrelated edits, so all the existing incrementality guarantees (and tests) hold unchanged.
An alternative design (kicking off Task.Run per shader in the transform and joining downstream) squeezes out a little extra overlap with the transpile phase, but relies on emergent driver scheduling, forces the cache to hold Task<HlslBytecodeInfo>, and leaves orphaned task lifetimes to manage — the explicit collect/warm/join shape seemed much easier to review and reason about. Happy to discuss if you'd prefer that variant.
Drawbacks
- The compile diagnostics now use external-file locations (
Location.IsInSource == false). Error list/navigation behavior is unchanged in practice, and no code fixers target these diagnostics. - Any shader edit re-runs the warm-up node over all keys, but unchanged shaders are cache hits, so the re-run is effectively free.
[D2DPixelShaderSource](raw HLSL literals) still compiles inline in its own generator; it could get the same treatment later if desired.
Additional context
Separately from this (and composing with it): CLI builds construct a fresh GeneratorDriver every time, so DynamicCache never gets to reuse anything across command-line rebuilds and every build pays the full compile cost even when nothing changed. I'd like to follow up with a second proposal for an opt-in persistent on-disk bytecode cache keyed on the same content key, which would make steady-state rebuilds nearly free. Keeping that separate to keep this one small.
I have a PR ready implementing this for both descriptor generators.
Activity