Skip to content

Parallelize build-time shader compilation in the descriptor generators #931

Open

Description

@rickbrew

(via Claude)

Description

Build-time shader compilation (FXC for D2D1, DXC for D3D12) currently runs inside the ForAttributeWithMetadataName transform callbacks in D2DPixelShaderDescriptorGenerator and ComputeShaderDescriptorGenerator. Roslyn's incremental driver invokes transform callbacks sequentially on one thread, so all shader compilations in a project are fully serialized, and their cost adds up linearly with shader body complexity.

I'm proposing moving the compilation out of the transform and into a dedicated pipeline node that compiles all shaders in a project in parallel, warming the existing DynamicCache, with a Combine-ordered join node folding results back into the per-shader models.

Rationale

Paint.NET has several projects with a lot of nontrivial D2D1 pixel shaders, and the serial compilation dominates their build times. Profiling the compiler server during a warm rebuild of the largest one shows Roslyn's own parse/bind/emit phases bursting across many cores, separated by a long plateau pinned at exactly ~1.00 core — and stubbing all shader Execute() bodies removes the plateau, i.e. the plateau is the FXC time.

Everything needed for parallelism is already in place:

  • HlslBytecodeInfoKey is a pure content key (HLSL source + profile + options), so compilations are independent.
  • DynamicCache<TKey, TValue> is already thread-safe (ConcurrentDictionary-backed; a same-key race is benign — both compile, one result is discarded).
  • D3DCompile in d3dcompiler_47 is documented thread-safe and stateless, and the DXC wrapper already uses a [ThreadStatic] per-thread compiler instance.

Nothing invokes any of it concurrently today.

Measured effect in Paint.NET (warm --no-incremental Release x64 rebuilds, second of two runs, stock 3.2.0 vs. a locally packed build of the proposed change):

Project (shader-heavy) 3.2.0 (serial) parallel delta
Effects.Gpu (51 shaders) 21.4s 13.5s -7.9s (-37%)
Windows.Direct2D1.Managed (51 shader files) 12.6s 9.0s -3.6s (-29%)
Windows.Framework (546 mostly small shaders) 11.2s 7.6s -3.6s (-32%)

Expected wall time for the compile phase goes from sum of all shader compile times to roughly max(slowest shader, total / cores).

Proposed API

None — this is an internal change to the source generators. No public API surface is affected, and the generated code is byte-for-byte identical (verified across Paint.NET's ~660 shaders).

Sketch of the design

// Transform stops at building HlslBytecodeInfoKey (no compilation), and captures the
// info needed to synthesize compile diagnostics later (by value, no symbols/trees).

// One node compiles everything in parallel into the existing memoizing cache:
IncrementalValueProvider<bool> hlslBytecodeCache =
    shaderInfoWithNoHlslBytecode
    .Select(static (item, _) => item.HlslInfoKey)
    .Collect()
    .Select(static (keys, token) =>
    {
        HlslBytecodeSyntaxProcessor.CompileAllInParallel(keys, token); // Parallel.ForEach over distinct keys

        return true;
    });

// Per-shader join, ordered after the warm-up by the Combine edge; pure cache hit:
IncrementalValuesProvider<D2D1ShaderInfo> shaderInfo =
    shaderInfoWithNoHlslBytecode
    .Combine(hlslBytecodeCache)
    .Select(static (pair, token) => /* fold in bytecode + synthesize deferred diagnostics */);

The ordering is enforced by the dataflow graph itself (the join's input depends on the warm-up node's output), not by assumptions about driver scheduling. DynamicCache is untouched.

One consequence: the two compile-related diagnostics (GetInfoDiagnostics, GetDoublePrecisionSupportDiagnostics) can no longer be created in the transform, since they need the type symbol and symbols must not escape the transform. The location/name info is instead captured by value (file path + TextSpan + LinePositionSpan) into the model, and the diagnostics are synthesized in the join using Location.Create(path, span, lineSpan). Capturing by value (rather than holding a SyntaxTree reference) keeps transform outputs equatable across unrelated edits, so all the existing incrementality guarantees (and tests) hold unchanged.

An alternative design (kicking off Task.Run per shader in the transform and joining downstream) squeezes out a little extra overlap with the transpile phase, but relies on emergent driver scheduling, forces the cache to hold Task<HlslBytecodeInfo>, and leaves orphaned task lifetimes to manage — the explicit collect/warm/join shape seemed much easier to review and reason about. Happy to discuss if you'd prefer that variant.

Drawbacks

  • The compile diagnostics now use external-file locations (Location.IsInSource == false). Error list/navigation behavior is unchanged in practice, and no code fixers target these diagnostics.
  • Any shader edit re-runs the warm-up node over all keys, but unchanged shaders are cache hits, so the re-run is effectively free.
  • [D2DPixelShaderSource] (raw HLSL literals) still compiles inline in its own generator; it could get the same treatment later if desired.

Additional context

Separately from this (and composing with it): CLI builds construct a fresh GeneratorDriver every time, so DynamicCache never gets to reuse anything across command-line rebuilds and every build pays the full compile cost even when nothing changed. I'd like to follow up with a second proposal for an opt-in persistent on-disk bytecode cache keyed on the same content key, which would make steady-state rebuilds nearly free. Keeping that separate to keep this one small.

I have a PR ready implementing this for both descriptor generators.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions