improve type safety of std.zig.Ast #22397

Techatrix · 2025-01-03T09:11:46Z

This PR introduces named/distinct integer types to std.zig.Ast, improving type safety and aligning it more closely with other compiler data structures like ZIR.

At it's core, the changes can be summarized by the following:

Before

// Most fields and declarations have been redacted for simplicity.
source: [:0]const u8,
tokens: TokenList.Slice,
nodes: NodeList.Slice,
extra_data: []Node.Index,

pub const TokenIndex = u32;

pub const Node = struct {
    tag: Tag,
    main_token: TokenIndex,
    data: Data,

    pub const Index = u32;

    pub const Tag = enum {...};

    pub const Data = struct {
        lhs: Index,
        rhs: Index,
    };
};

After

// Most fields and declarations have been redacted for simplicity.
source: [:0]const u8,
tokens: TokenList.Slice,
nodes: NodeList.Slice,
extra_data: []u32,

/// Index into `tokens`, or null.
pub const TokenIndex = u32;

/// Index into `tokens`, or null.
pub const OptionalTokenIndex = enum(u32) {
    none = std.math.maxInt(u32),
    _,
};

/// Index into `extra_data`.
pub const ExtraIndex = enum(u32) { _ };

pub const Node = struct {
    tag: Tag,
    main_token: TokenIndex,
    data: Data,

    /// Index into `nodes`.
    pub const Index = enum(u32) { _ };

    /// Index into `nodes`, or null.
    pub const OptionalIndex = enum(u32) {
        none = std.math.maxInt(u32),
        _,
    };

    pub const Tag = enum {...};

    pub const Data = struct {
        lhs: Item,
        rhs: Item,

        pub const Item = union {
            node: Index,
            opt_node: OptionalIndex,
            token: TokenIndex,
            opt_token: OptionalTokenIndex,
            extra_index: ExtraIndex,
            @"for": For,
        };
    };
};

I initial changed TokenIndex to a named/distinct integer type as well but it made dealing with them unergonomic. This was especially noticable while porting lib/std/zig/render.zig because it frequently relied on relative token indicies.

The offset types (Node.Offset, TokenOffset, etc.) have been added for ZIR because it stores relative ast indicies as opposed to absolute indicies.

zig ast-check src/Sema.zig (ReleaseFast)

$ poop "build/stage4-fast/bin/zig ast-check src/Sema.zig" "build/stage4-ast-fast/bin/zig ast-check src/Sema.zig" -d 20000
Benchmark 1 (191 runs): build/stage4-fast/bin/zig ast-check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           105ms ± 3.36ms     103ms …  145ms          2 ( 1%)        0%
  peak_rss           77.6MB ± 43.8KB    77.3MB … 77.7MB         24 (13%)        0%
  cpu_cycles          345M  ± 5.05M      339M  …  359M           0 ( 0%)        0%
  instructions        523M  ± 5.98       523M  …  523M           1 ( 1%)        0%
  cache_references   1.46M  ± 61.0K     1.28M  … 1.60M           8 ( 4%)        0%
  cache_misses        334K  ± 29.7K      296K  …  411K           2 ( 1%)        0%
  branch_misses      4.87M  ± 2.98K     4.86M  … 4.88M           2 ( 1%)        0%
Benchmark 2 (190 runs): build/stage4-ast-fast/bin/zig ast-check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           105ms ± 1.71ms     103ms …  112ms          6 ( 3%)          +  0.4% ±  0.5%
  peak_rss           77.7MB ± 31.2KB    77.5MB … 77.7MB         13 ( 7%)          +  0.2% ±  0.0%
  cpu_cycles          348M  ± 5.45M      342M  …  365M           6 ( 3%)          +  0.8% ±  0.3%
  instructions        533M  ± 4.81       533M  …  533M           2 ( 1%)        💩+  1.8% ±  0.0%
  cache_references   1.46M  ± 58.3K     1.31M  … 1.62M          10 ( 5%)          +  0.0% ±  0.8%
  cache_misses        328K  ± 26.1K      293K  …  395K          32 (17%)          -  1.8% ±  1.7%
  branch_misses      4.82M  ± 2.97K     4.82M  … 4.83M           5 ( 3%)          -  0.9% ±  0.0%

zig fmt --check src/Sema.zig (ReleaseFast)

$ poop "build/stage4-fast/bin/zig fmt --check src/Sema.zig" "build/stage4-ast-fast/bin/zig fmt --check src/Sema.zig" -d 20000
Benchmark 1 (342 runs): build/stage4-fast/bin/zig fmt --check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time          58.5ms ± 1.59ms    56.8ms … 64.8ms          5 ( 1%)        0%
  peak_rss           68.0MB ± 73.3KB    67.8MB … 68.1MB          0 ( 0%)        0%
  cpu_cycles          177M  ± 4.71M      173M  …  198M          11 ( 3%)        0%
  instructions        353M  ± 4.89       353M  …  353M           9 ( 3%)        0%
  cache_references    619K  ± 27.7K      542K  …  679K          64 (19%)        0%
  cache_misses        277K  ± 22.9K      242K  …  348K          30 ( 9%)        0%
  branch_misses      1.32M  ± 1.85K     1.32M  … 1.33M           8 ( 2%)        0%
Benchmark 2 (336 runs): build/stage4-ast-fast/bin/zig fmt --check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time          59.5ms ± 1.71ms    57.6ms … 64.6ms          3 ( 1%)        💩+  1.8% ±  0.4%
  peak_rss           68.2MB ± 55.1KB    68.0MB … 68.2MB         69 (21%)          +  0.2% ±  0.0%
  cpu_cycles          181M  ± 5.11M      177M  …  198M           2 ( 1%)        💩+  2.5% ±  0.4%
  instructions        363M  ± 5.30       363M  …  363M           6 ( 2%)        💩+  2.8% ±  0.0%
  cache_references    630K  ± 27.0K      556K  …  688K          59 (18%)          +  1.7% ±  0.7%
  cache_misses        280K  ± 23.6K      245K  …  341K           0 ( 0%)          +  1.2% ±  1.3%
  branch_misses      1.31M  ± 1.82K     1.31M  … 1.32M           0 ( 0%)          -  0.9% ±  0.0%

zig ast-check src/Sema.zig (Debug)

$ poop "build/stage4/bin/zig ast-check src/Sema.zig" "build/stage4-ast/bin/zig ast-check src/Sema.zig" -d 20000
Benchmark 1 (33 runs): build/stage4/bin/zig ast-check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           610ms ± 11.5ms     602ms …  672ms          2 ( 6%)        0%
  peak_rss           84.4MB ± 58.3KB    84.3MB … 84.5MB          8 (24%)        0%
  cpu_cycles         2.28G  ± 4.31M     2.27G  … 2.29G           1 ( 3%)        0%
  instructions       3.73G  ± 7.31      3.73G  … 3.73G           1 ( 3%)        0%
  cache_references   11.3M  ±  282K     10.8M  … 11.9M           0 ( 0%)        0%
  cache_misses        579K  ± 40.9K      489K  …  631K           2 ( 6%)        0%
  branch_misses      5.81M  ± 11.9K     5.80M  … 5.86M           1 ( 3%)        0%
Benchmark 2 (33 runs): build/stage4-ast/bin/zig ast-check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           619ms ± 2.44ms     614ms …  623ms          0 ( 0%)          +  1.5% ±  0.7%
  peak_rss           86.2MB ± 31.8KB    86.0MB … 86.2MB          6 (18%)        💩+  2.0% ±  0.0%
  cpu_cycles         2.32G  ± 6.82M     2.31G  … 2.33G           0 ( 0%)        💩+  1.8% ±  0.1%
  instructions       3.86G  ± 6.49      3.86G  … 3.86G           0 ( 0%)        💩+  3.6% ±  0.0%
  cache_references   11.2M  ±  392K     10.6M  … 12.2M           1 ( 3%)          -  1.0% ±  1.5%
  cache_misses        585K  ± 35.9K      526K  …  650K           0 ( 0%)          +  1.1% ±  3.3%
  branch_misses      5.74M  ± 7.35K     5.73M  … 5.76M           2 ( 6%)        ⚡-  1.3% ±  0.1%

zig fmt --check src/Sema.zig (Debug)

$ poop "build/stage4/bin/zig fmt --check src/Sema.zig" "build/stage4-ast/bin/zig fmt --check src/Sema.zig" -d 20000
Benchmark 1 (62 runs): build/stage4/bin/zig fmt --check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           327ms ± 2.77ms     323ms …  336ms          5 ( 8%)        0%
  peak_rss           73.0MB ± 66.2KB    72.8MB … 73.1MB         14 (23%)        0%
  cpu_cycles         1.21G  ± 9.52M     1.19G  … 1.24G           6 (10%)        0%
  instructions       2.22G  ± 6.31      2.22G  … 2.22G           0 ( 0%)        0%
  cache_references   1.90M  ±  110K     1.71M  … 2.26M           2 ( 3%)        0%
  cache_misses        324K  ± 26.2K      283K  …  377K           0 ( 0%)        0%
  branch_misses      2.95M  ±  179K     2.74M  … 3.75M           3 ( 5%)        0%
Benchmark 2 (56 runs): build/stage4-ast/bin/zig fmt --check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           358ms ± 6.11ms     353ms …  399ms          2 ( 4%)        💩+  9.6% ±  0.5%
  peak_rss           75.3MB ± 32.0KB    75.2MB … 75.3MB         24 (43%)        💩+  3.1% ±  0.0%
  cpu_cycles         1.33G  ± 19.7M     1.31G  … 1.46G           2 ( 4%)        💩+  9.8% ±  0.5%
  instructions       2.54G  ± 8.03      2.54G  … 2.54G           1 ( 2%)        💩+ 14.5% ±  0.0%
  cache_references   1.94M  ±  164K     1.73M  … 2.80M           1 ( 2%)          +  1.9% ±  2.7%
  cache_misses        341K  ± 33.7K      302K  …  493K           1 ( 2%)        💩+  5.2% ±  3.4%
  branch_misses      2.99M  ±  150K     2.79M  … 3.52M           2 ( 4%)          +  1.3% ±  2.1%

mlugg · 2025-01-03T15:12:29Z

I initial changed TokenIndex to a named/distinct integer type as well but it made dealing with them unergonomic.

Did you try having an offset method?

fn offset(tok: TokenIndex, off: i32) TokenIndex {
    const new_tok = @as(i64, @intFromEnum(tok)) + off;
    return @enumFromInt(new_tok);
}

Regardless: this PR seems like a fantastic change. My main concern just from the PR description is that final performance data point -- strictly speaking, this change shouldn't impact performance at all. Do you have any idea why the last performance number is consistently so bad?

Techatrix · 2025-01-03T15:57:32Z

Did you try having an offset method?

Kind of. I had next, previous, shift and shiftBackwards methods. I don't have the performance data anymore, but before I undid the change to TokenIndex, zig fmt --check src/Sema.zig (Debug) Benchmark was at least a 30% performance regression. What would previously be simple add or subtract on a token index would require a function call and lib/std/zig/render.zig did a lot of them.

My main concern just from the PR description is that final performance data point -- strictly speaking, this change shouldn't impact performance at all. Do you have any idea why the last performance number is consistently so bad?

Here are some of the possibles reasons:

accessing the fields of nodes and tokens goes through a function call (tree.nodes.items(.data)[node] -> tree.nodeData(node))
@sizeOf(Ast.Node.Data) has increased from 8 to 16 in debug builds
accessing Ast.Node.Data.Item in a debug builds has a safety check on each field access

Removing the tokenTag and tokenStart functions by manually inlining them may help to improve performance (in debug builds).

RetroDev256 · 2025-01-03T17:53:41Z

If I remember correctly, wasn't null for optional indexes represented as zero? The PR writeup shows none = std.math.maxInt(u32), and I wonder if it could remain zero.

mlugg · 2025-01-03T18:12:50Z

maxInt(u32) is a much better value for .none than 0. With 0, you need to make sure it's not in a context where the first token/node is actually valid, or you need to reserve a dummy entry in the array. maxInt(u32) avoids this downside.

rohlem · 2025-01-03T18:44:09Z

zig fmt --check src/Sema.zig (Debug) Benchmark was at least a 30% performance regression. What would previously be simple add or subtract on a token index would require a function call [...]

accessing the fields of nodes and tokens goes through a function call (tree.nodes.items(.data)[node] -> tree.nodeData(node))

[...] manually inlining them may help to improve performance

All of these points seem like a perfectly valid use case for an inline fn / callconv(.@"inline") to me.
(Well, long-term the compiler's inlining heuristic should instead be improved, but that can be done later.)
Did that also not work / am I missing some reason the compiler inlining them w-/shouldn't be exactly equivalent?

mlugg · 2025-01-03T22:31:34Z

It might be worth seeing whether marking functions inline can improve debug performance -- if it does, I think this would be a valid use of inline.

However, I don't see a 10% regression in Debug performance for an already fairly fast subcommand as a huge deal, so this PR is fine as-is if you don't feel like looking into that.

Techatrix · 2025-01-04T12:30:34Z

Unfortunately, marking the functions as inline made the performance worse in debug builds.

zig ast-check src/Sema.zig (ReleaseFast)

poop "build/stage4-fast/bin/zig ast-check src/Sema.zig" \
     "build/stage4-ast-fast/bin/zig ast-check src/Sema.zig" \
     "build/stage4-ast-inline-fast/bin/zig ast-check src/Sema.zig" -d 30000

Benchmark 1 (288 runs): build/stage4-fast/bin/zig ast-check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           104ms ±  906us     102ms …  107ms          1 ( 0%)        0%
  peak_rss           77.6MB ± 66.0KB    77.4MB … 77.7MB         69 (24%)        0%
  cpu_cycles          340M  ± 2.74M      336M  …  351M           1 ( 0%)        0%
  instructions        523M  ± 7.68       523M  …  523M          22 ( 8%)        0%
  cache_references   1.42M  ± 31.3K     1.34M  … 1.52M           6 ( 2%)        0%
  cache_misses        348K  ± 32.0K      309K  …  402K           0 ( 0%)        0%
  branch_misses      4.83M  ± 2.67K     4.83M  … 4.84M          11 ( 4%)        0%
Benchmark 2 (283 runs): build/stage4-ast-fast/bin/zig ast-check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           106ms ±  813us     104ms …  108ms          0 ( 0%)        💩+  2.0% ±  0.1%
  peak_rss           77.6MB ± 49.4KB    77.4MB … 77.7MB         71 (25%)          -  0.1% ±  0.0%
  cpu_cycles          348M  ± 2.46M      344M  …  354M           0 ( 0%)        💩+  2.2% ±  0.1%
  instructions        533M  ± 5.64       533M  …  533M           9 ( 3%)        💩+  1.8% ±  0.0%
  cache_references   1.48M  ± 30.8K     1.41M  … 1.58M           2 ( 1%)        💩+  4.2% ±  0.4%
  cache_misses        340K  ± 29.1K      308K  …  395K           0 ( 0%)        ⚡-  2.5% ±  1.4%
  branch_misses      4.89M  ± 3.15K     4.88M  … 4.90M           1 ( 0%)        💩+  1.2% ±  0.0%
Benchmark 3 (284 runs): build/stage4-ast-inline-fast/bin/zig ast-check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           105ms ±  891us     104ms …  108ms          0 ( 0%)        💩+  1.6% ±  0.1%
  peak_rss           77.6MB ± 60.5KB    77.3MB … 77.7MB         86 (30%)          -  0.1% ±  0.0%
  cpu_cycles          346M  ± 2.54M      342M  …  353M           0 ( 0%)        💩+  1.8% ±  0.1%
  instructions        533M  ± 7.38       533M  …  533M          10 ( 4%)        💩+  1.9% ±  0.0%
  cache_references   1.55M  ± 42.0K     1.45M  … 1.70M           3 ( 1%)        💩+  8.9% ±  0.4%
  cache_misses        343K  ± 31.1K      308K  …  395K           0 ( 0%)          -  1.6% ±  1.5%
  branch_misses      4.84M  ± 2.91K     4.83M  … 4.85M           9 ( 3%)          +  0.2% ±  0.0%

zig fmt --check src/Sema.zig (ReleaseFast)

poop "build/stage4-fast/bin/zig fmt --check src/Sema.zig" \
     "build/stage4-ast-fast/bin/zig fmt --check src/Sema.zig" \
     "build/stage4-ast-inline-fast/bin/zig fmt --check src/Sema.zig" -d 30000

Benchmark 1 (509 runs): build/stage4-fast/bin/zig fmt --check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time          58.2ms ±  629us    57.5ms … 61.3ms         62 (12%)        0%
  peak_rss           68.0MB ± 64.6KB    67.8MB … 68.1MB        117 (23%)        0%
  cpu_cycles          175M  ± 2.03M      174M  …  186M          90 (18%)        0%
  instructions        353M  ± 4.94       353M  …  353M           8 ( 2%)        0%
  cache_references    617K  ± 6.73K      606K  …  644K          29 ( 6%)        0%
  cache_misses        274K  ± 21.5K      260K  …  338K          98 (19%)        0%
  branch_misses      1.31M  ± 1.61K     1.30M  … 1.32M          17 ( 3%)        0%
Benchmark 2 (499 runs): build/stage4-ast-fast/bin/zig fmt --check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time          59.5ms ±  715us    58.7ms … 63.6ms         25 ( 5%)        💩+  2.3% ±  0.1%
  peak_rss           68.2MB ± 40.2KB    68.0MB … 68.2MB         88 (18%)          +  0.2% ±  0.0%
  cpu_cycles          180M  ± 2.08M      179M  …  193M          66 (13%)        💩+  2.9% ±  0.1%
  instructions        363M  ± 5.74       363M  …  363M          17 ( 3%)        💩+  2.8% ±  0.0%
  cache_references    637K  ± 8.31K      618K  …  678K           5 ( 1%)        💩+  3.2% ±  0.2%
  cache_misses        275K  ± 22.6K      259K  …  338K          86 (17%)          +  0.7% ±  1.0%
  branch_misses      1.33M  ± 1.67K     1.32M  … 1.33M           6 ( 1%)        💩+  1.5% ±  0.0%
Benchmark 3 (500 runs): build/stage4-ast-inline-fast/bin/zig fmt --check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time          59.4ms ±  789us    58.3ms … 63.2ms          5 ( 1%)        💩+  2.1% ±  0.2%
  peak_rss           68.2MB ± 58.4KB    67.9MB … 68.2MB        102 (20%)          +  0.2% ±  0.0%
  cpu_cycles          179M  ± 2.69M      177M  …  195M           7 ( 1%)        💩+  2.4% ±  0.2%
  instructions        363M  ± 4.89       363M  …  363M           5 ( 1%)        💩+  2.8% ±  0.0%
  cache_references    631K  ± 9.62K      575K  …  671K           4 ( 1%)        💩+  2.2% ±  0.2%
  cache_misses        285K  ± 28.2K      247K  …  338K           0 ( 0%)        💩+  4.2% ±  1.1%
  branch_misses      1.31M  ± 1.56K     1.31M  … 1.32M           3 ( 1%)          +  0.2% ±  0.0%

zig ast-check src/Sema.zig (Debug)

poop "build/stage4/bin/zig ast-check src/Sema.zig" \
     "build/stage4-ast/bin/zig ast-check src/Sema.zig" \
     "build/stage4-ast-inline/bin/zig ast-check src/Sema.zig" -d 30000

Benchmark 1 (52 runs): build/stage4/bin/zig ast-check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           586ms ± 8.94ms     583ms …  648ms          1 ( 2%)        0%
  peak_rss           84.5MB ± 93.2KB    84.2MB … 84.6MB          1 ( 2%)        0%
  cpu_cycles         2.20G  ± 3.81M     2.19G  … 2.21G           0 ( 0%)        0%
  instructions       3.73G  ± 15.9      3.73G  … 3.73G           7 (13%)        0%
  cache_references   11.1M  ±  228K     10.7M  … 11.7M           0 ( 0%)        0%
  cache_misses        536K  ± 38.4K      495K  …  600K           0 ( 0%)        0%
  branch_misses      5.79M  ± 9.13K     5.78M  … 5.83M           2 ( 4%)        0%
Benchmark 2 (50 runs): build/stage4-ast/bin/zig ast-check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           603ms ± 3.12ms     601ms …  624ms          1 ( 2%)        💩+  2.9% ±  0.5%
  peak_rss           86.0MB ± 95.2KB    85.7MB … 86.1MB          1 ( 2%)        💩+  1.7% ±  0.0%
  cpu_cycles         2.27G  ± 2.94M     2.26G  … 2.27G           0 ( 0%)        💩+  3.1% ±  0.1%
  instructions       3.86G  ± 6.44      3.86G  … 3.86G           3 ( 6%)        💩+  3.6% ±  0.0%
  cache_references   10.8M  ±  258K     10.4M  … 11.5M           1 ( 2%)        ⚡-  2.1% ±  0.9%
  cache_misses        584K  ± 33.2K      522K  …  628K           0 ( 0%)        💩+  9.0% ±  2.6%
  branch_misses      5.68M  ± 7.37K     5.67M  … 5.72M           1 ( 2%)        ⚡-  2.0% ±  0.1%
Benchmark 3 (50 runs): build/stage4-ast-inline/bin/zig ast-check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           611ms ± 1.19ms     609ms …  616ms          1 ( 2%)        💩+  4.3% ±  0.4%
  peak_rss           86.5MB ± 79.9KB    86.3MB … 86.6MB          0 ( 0%)        💩+  2.3% ±  0.0%
  cpu_cycles         2.30G  ± 3.94M     2.29G  … 2.31G           1 ( 2%)        💩+  4.6% ±  0.1%
  instructions       3.88G  ± 6.90      3.88G  … 3.88G           2 ( 4%)        💩+  4.0% ±  0.0%
  cache_references   12.6M  ±  343K     12.1M  … 14.2M           1 ( 2%)        💩+ 14.0% ±  1.0%
  cache_misses        608K  ± 23.4K      521K  …  636K           3 ( 6%)        💩+ 13.4% ±  2.3%
  branch_misses      5.72M  ± 9.50K     5.71M  … 5.77M           2 ( 4%)        ⚡-  1.2% ±  0.1%

zig fmt --check src/Sema.zig (Debug)

poop "build/stage4/bin/zig fmt --check src/Sema.zig" \
     "build/stage4-ast/bin/zig fmt --check src/Sema.zig" \
     "build/stage4-ast-inline/bin/zig fmt --check src/Sema.zig" -d 30000

Benchmark 1 (93 runs): build/stage4/bin/zig fmt --check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           323ms ± 2.94ms     319ms …  332ms          0 ( 0%)        0%
  peak_rss           73.0MB ± 74.3KB    72.8MB … 73.1MB          6 ( 6%)        0%
  cpu_cycles         1.20G  ± 11.6M     1.17G  … 1.23G           0 ( 0%)        0%
  instructions       2.22G  ± 6.77      2.22G  … 2.22G           1 ( 1%)        0%
  cache_references   1.97M  ±  125K     1.72M  … 2.45M           2 ( 2%)        0%
  cache_misses        359K  ± 14.9K      300K  …  419K          13 (14%)        0%
  branch_misses      3.19M  ±  281K     2.71M  … 3.82M           0 ( 0%)        0%
Benchmark 2 (84 runs): build/stage4-ast/bin/zig fmt --check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           360ms ± 3.66ms     354ms …  372ms          4 ( 5%)        💩+ 11.2% ±  0.3%
  peak_rss           74.9MB ± 63.2KB    74.7MB … 74.9MB         18 (21%)        💩+  2.5% ±  0.0%
  cpu_cycles         1.33G  ± 13.2M     1.32G  … 1.38G           4 ( 5%)        💩+ 11.6% ±  0.3%
  instructions       2.54G  ± 11.9      2.54G  … 2.54G           7 ( 8%)        💩+ 14.5% ±  0.0%
  cache_references   1.79M  ± 92.2K     1.67M  … 2.20M           2 ( 2%)        ⚡-  9.2% ±  1.7%
  cache_misses        372K  ± 23.4K      316K  …  417K          11 (13%)        💩+  3.5% ±  1.6%
  branch_misses      3.21M  ±  386K     2.78M  … 4.44M           5 ( 6%)          +  0.5% ±  3.1%
Benchmark 3 (73 runs): build/stage4-ast-inline/bin/zig fmt --check src/Sema.zig
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           415ms ± 2.64ms     411ms …  422ms          1 ( 1%)        💩+ 28.5% ±  0.3%
  peak_rss           75.1MB ± 43.5KB    75.0MB … 75.2MB         13 (18%)        💩+  2.9% ±  0.0%
  cpu_cycles         1.55G  ± 10.00M    1.54G  … 1.58G           5 ( 7%)        💩+ 29.6% ±  0.3%
  instructions       2.60G  ± 8.63      2.60G  … 2.60G           5 ( 7%)        💩+ 17.1% ±  0.0%
  cache_references   2.86M  ±  167K     2.59M  … 3.45M           3 ( 4%)        💩+ 44.9% ±  2.2%
  cache_misses        363K  ± 23.8K      317K  …  392K           0 ( 0%)          +  1.0% ±  1.7%
  branch_misses      3.74M  ±  399K     3.13M  … 4.98M           3 ( 4%)        💩+ 17.2% ±  3.2%

Patch

diff --git a/lib/std/zig/Ast.zig b/lib/std/zig/Ast.zig
index 686a2e884d..28d62b7ad2 100644
--- a/lib/std/zig/Ast.zig
+++ b/lib/std/zig/Ast.zig
@@ -75,23 +75,23 @@ pub const OptionalTokenOffset = enum(i32) {
     }
 };
 
-pub fn tokenTag(tree: *const Ast, token_index: TokenIndex) Token.Tag {
+pub inline fn tokenTag(tree: *const Ast, token_index: TokenIndex) Token.Tag {
     return tree.tokens.items(.tag)[token_index];
 }
 
-pub fn tokenStart(tree: *const Ast, token_index: TokenIndex) ByteOffset {
+pub inline fn tokenStart(tree: *const Ast, token_index: TokenIndex) ByteOffset {
     return tree.tokens.items(.start)[token_index];
 }
 
-pub fn nodeTag(tree: *const Ast, node: Node.Index) Node.Tag {
+pub inline fn nodeTag(tree: *const Ast, node: Node.Index) Node.Tag {
     return tree.nodes.items(.tag)[@intFromEnum(node)];
 }
 
-pub fn nodeMainToken(tree: *const Ast, node: Node.Index) TokenIndex {
+pub inline fn nodeMainToken(tree: *const Ast, node: Node.Index) TokenIndex {
     return tree.nodes.items(.main_token)[@intFromEnum(node)];
 }
 
-pub fn nodeData(tree: *const Ast, node: Node.Index) Node.Data {
+pub inline fn nodeData(tree: *const Ast, node: Node.Index) Node.Data {
     return tree.nodes.items(.data)[@intFromEnum(node)];
 }
 
diff --git a/lib/std/zig/Parse.zig b/lib/std/zig/Parse.zig
index abbc9b0fc1..8762cdfd8e 100644
--- a/lib/std/zig/Parse.zig
+++ b/lib/std/zig/Parse.zig
@@ -11,23 +11,23 @@ nodes: Ast.NodeList,
 extra_data: std.ArrayListUnmanaged(u32),
 scratch: std.ArrayListUnmanaged(Node.Index),
 
-fn tokenTag(p: *const Parse, token_index: TokenIndex) Token.Tag {
+inline fn tokenTag(p: *const Parse, token_index: TokenIndex) Token.Tag {
     return p.tokens.items(.tag)[token_index];
 }
 
-fn tokenStart(p: *const Parse, token_index: TokenIndex) Ast.ByteOffset {
+inline fn tokenStart(p: *const Parse, token_index: TokenIndex) Ast.ByteOffset {
     return p.tokens.items(.start)[token_index];
 }
 
-fn nodeTag(p: *const Parse, node: Node.Index) Node.Tag {
+inline fn nodeTag(p: *const Parse, node: Node.Index) Node.Tag {
     return p.nodes.items(.tag)[@intFromEnum(node)];
 }
 
-fn nodeMainToken(p: *const Parse, node: Node.Index) TokenIndex {
+inline fn nodeMainToken(p: *const Parse, node: Node.Index) TokenIndex {
     return p.nodes.items(.main_token)[@intFromEnum(node)];
 }
 
-fn nodeData(p: *const Parse, node: Node.Index) Node.Data {
+inline fn nodeData(p: *const Parse, node: Node.Index) Node.Data {
     return p.nodes.items(.data)[@intFromEnum(node)];
 }

This function checks for various possibilities that are never produced by the parser. Given that lastToken is unsafe to call on an Ast with errors, I also removed code paths that would be reachable on an Ast with errors.

AstGen doesn't seem to report enough information to figure out if a func decl has a calling convention ast node. A calling convention could be present without it by marking the function as inline for example.

This commits adds the following distinct integer types to std.zig.Ast: - OptionalTokenIndex - TokenOffset - OptionalTokenOffset - Node.OptionalIndex - Node.Offset - Node.OptionalOffset The `Node.Index` type has also been converted to a distinct type while TokenIndex remains unchanged. `Ast.Node.Data.Item` has also been changed to a (untagged) union to provide safety checks.

The existing comment are incomplete, outdated and sometimes incorrect.

Techatrix force-pushed the type-safe-ast branch 2 times, most recently from d842369 to 80c0304 Compare January 3, 2025 12:55

Techatrix force-pushed the type-safe-ast branch from 80c0304 to cb91fda Compare January 4, 2025 12:30

Techatrix added 7 commits January 9, 2025 19:58

remove unreachable code paths from std.zig.Ast.lastToken

1c6626b

This function checks for various possibilities that are never produced by the parser. Given that lastToken is unsafe to call on an Ast with errors, I also removed code paths that would be reachable on an Ast with errors.

fix ZonGen error message when encountering an array access

58ac879

fix incorrect error source location on function calling convention

6b83db2

AstGen doesn't seem to report enough information to figure out if a func decl has a calling convention ast node. A calling convention could be present without it by marking the function as inline for example.

add a reference to ziglang#21690

b8f72af

add std.zig.Ast.blockStatements and std.zig.Ast.builtinCallParams

59b7cdc

update doc comments of std.zig.Ast.Node.Tag

2b23652

The existing comment are incomplete, outdated and sometimes incorrect.

Techatrix force-pushed the type-safe-ast branch from cb91fda to 2b23652 Compare January 9, 2025 19:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

improve type safety of std.zig.Ast #22397

improve type safety of std.zig.Ast #22397

Techatrix commented Jan 3, 2025

mlugg commented Jan 3, 2025

Techatrix commented Jan 3, 2025

RetroDev256 commented Jan 3, 2025

mlugg commented Jan 3, 2025 •

edited

Loading

rohlem commented Jan 3, 2025 •

edited

Loading

mlugg commented Jan 3, 2025

Techatrix commented Jan 4, 2025

improve type safety of std.zig.Ast #22397

Are you sure you want to change the base?

improve type safety of std.zig.Ast #22397

Conversation

Techatrix commented Jan 3, 2025

mlugg commented Jan 3, 2025

Techatrix commented Jan 3, 2025

RetroDev256 commented Jan 3, 2025

mlugg commented Jan 3, 2025 • edited Loading

rohlem commented Jan 3, 2025 • edited Loading

mlugg commented Jan 3, 2025

Techatrix commented Jan 4, 2025

mlugg commented Jan 3, 2025 •

edited

Loading

rohlem commented Jan 3, 2025 •

edited

Loading