Compile Time Ref Prep

[willtebbutt]

Moves preparation of some Base.RefValues which are used on the reverse-pass to compile time.

The reason for doing this is to

1. ensure performance robustness -- the new line of code is literally just a :new instruction, with all other prep having been done beforehand. There is very little that can go wrong here. Conversely, we previously had calls to Mooncake.__make_ref inserted into the IR, which itself called Mooncake.zero_like_rdata_type and Mooncake.zero_like_rdata_from_type. While there are meant to always be performant, it's hard to ensure this 100% of the time. Now, these functions get called at compile-time, and insert their results directly into the IR. Consequently, even if the performance of these functions is poor, it won't show up at runtime.
2. make errors happen sooner: previously, if something went wrong in Mooncake.__make_ref, it likely wouldn't show up until the reverse-pass is run. Conversely, any error which happens should now happen during rule-compilation, which is substantially preferable.

TLDR: designs-away the possibility of one source of performance error, and surfaces other errors sooner.

Todo:

• add comment about bitstypes

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Files with missing lines	Coverage Δ
src/interpreter/s2s_reverse_mode_ad.jl	95.06% <100.00%> (+0.74%)	⬆️

[github-actions]

Performance Ratio:
Ratio of time to compute gradient and time to compute function.
Warning: results are very approximate! See here for more context.

┌────────────────────────────┬──────────┬─────────┬─────────────┬─────────┐
│                      Label │ Mooncake │  Zygote │ ReverseDiff │  Enzyme │
│                     String │   String │  String │      String │  String │
├────────────────────────────┼──────────┼─────────┼─────────────┼─────────┤
│                   sum_1000 │     79.1 │     1.0 │        5.61 │    8.11 │
│                  _sum_1000 │     6.63 │  1300.0 │        34.3 │    1.07 │
│               sum_sin_1000 │     2.29 │    1.68 │        10.6 │    1.98 │
│              _sum_sin_1000 │      2.6 │   249.0 │        13.1 │    2.43 │
│                   kron_sum │     66.7 │    3.67 │       230.0 │    10.7 │
│              kron_view_sum │     60.1 │    9.96 │       266.0 │   103.0 │
│      naive_map_sin_cos_exp │     2.47 │ missing │        7.03 │    2.33 │
│            map_sin_cos_exp │     2.71 │    1.51 │        5.99 │    2.95 │
│      broadcast_sin_cos_exp │     2.55 │    2.26 │        1.46 │    2.27 │
│                 simple_mlp │     4.79 │    3.03 │        6.53 │    3.27 │
│                     gp_lml │     4.49 │    2.28 │     missing │    2.23 │
│ turing_broadcast_benchmark │      3.2 │ missing │        25.2 │ missing │
│         large_single_block │     4.42 │  4280.0 │        31.1 │    2.18 │
└────────────────────────────┴──────────┴─────────┴─────────────┴─────────┘