Skip to content

feat(aot): graceful fallback to JIT when AOT compile fails - #86

Closed
cennn wants to merge 1 commit into
SandAI-org:mainfrom
cennn:feat/aot-jit-fallback
Closed

cennn wants to merge 1 commit into
SandAI-org:mainfrom
cennn:feat/aot-jit-fallback

Conversation

@cennn

@cennn cennn commented Oct 1, 2026

Copy link
Copy Markdown
Collaborator

Problem

When MAGI_COMPILE_AOT=1 is set globally, models using SimpleFSDP/DTensor fail AOT compilation with:

Attempted to read undefined local variable
File ".../simple_fsdp/builder.py", line 236, in forward
    state.aot_compile(*args, **kwargs)
File ".../aot_compile.py", line 143, in aot_compile_fullgraph
    raise e.with_traceback(None) from e.__cause__

This crashes the entire warmup process, preventing AOT from being used even for compatible models (e.g., VAE which has no FSDP).

Solution

Catch exceptions in the AOT compile path and gracefully fall back to the JIT bytecode capture path. This allows MAGI_COMPILE_AOT=1 to be set globally:

  • VAE (no FSDP): AOT compiles successfully → skip Dynamo trace on verify (~14s savings)
  • Base/SR (SimpleFSDP+DTensor): AOT fails → falls back to JIT → inductor cache hits from bake tar

Key implementation details:

  1. Catch any exception from state.aot_compile()
  2. Reset compile_config.aot = False to prevent further AOT attempts for this state
  3. Reset compiled_entry and call torch._dynamo.reset() to clear AOT-only guard_filter_fn
  4. Re-run _ensure_compiled() for JIT mode and proceed with bytecode capture

Context

Part of CI cold-start optimization effort (athena PR #1112). AOT serializes Dynamo FX graphs so verify can skip Dynamo tracing + RestartAnalysis entirely (~42s savings for compatible models).

When MAGI_COMPILE_AOT=1, models using SimpleFSDP/DTensor may fail AOT
compilation (e.g. "Attempted to read undefined local variable"). Instead
of crashing, catch the exception and fall back to the JIT bytecode
capture path. This allows AOT to be enabled globally while FSDP models
auto-degrade to JIT + inductor cache hits.
@cennn

cennn commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator Author

Superseded by #89 (rebased the still-useful parts onto current main). Closing this PR.

@cennn cennn closed this Oct 14, 2026
jiahy0825 pushed a commit that referenced this pull request Oct 15, 2026
…split graphs (#89)

* fix: cache key on graph+placements, AOT JIT fallback, skip FSDP overlap on split graphs

Unmerged remnants of #88 / #86 / #84 / #79, rebased onto current main:

- Cache key includes graph.code and DTensor placements. Changing SimpleFSDP
  fully_shard -> replicate used to reuse the old artifact (1-rank AllGathers
  served as DtoD copies).
- AOT compile failures fall back to JIT so MAGI_COMPILE_AOT=1 can stay
  process-wide (VAE AOT, SimpleFSDP DiT JIT).
- enable_fsdp no longer runs fullgraph overlap when disable_graph_split is
  False (VAE in a monolithic FSDP process). #79's restart-analysis half
  already landed as #75.
- Timing bench dim 128->1024 so launch jitter does not flake max/min < 1.2
  on B300/H100.

* fix(cache): hash input placements only, not graph.code

Restart-analysis retraces with different FX node names, so including graph.code
in the magi cache key made the second pass miss the first-pass directory
(test_assert_cache_hit_succeeds_with_restart_analysis / test_restart_analysis_cache
on pt212). DTensor fully_shard vs replicate is still distinguished by the input
placement signature.

* fix(cache): hash only DTensor placements so restart-analysis still hits

* fix(aot): do not swallow AssertionError/OOM when AOT falls back to JIT
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant