Repository navigation
Compare rnix's parser with Nix's - #202
Open
philiptaron wants to merge 28 commits into
Open
philiptaron wants to merge 28 commits into
philiptaron wants to merge 28 commits into
Conversation
The start of a crate that compares rnix's parser with Nix's. render() writes a tree rnix parsed back as Nix with every expression in parentheses: if Nix parses that to the same expression as the source, rnix grouped the tokens like Nix does. Nix is then the only printer involved, so the comparison needs neither Nix's internals nor a copy of its AST printer. Like fuzz/, the crate is kept out of rnix's workspace and package.
Checker::check() parses an input with rnix and with nix-instantiate,
and, when both accept it, compares Nix's parse of the source with Nix's
parse of rnix's rendering. The outcome says whether both parsers agree,
which one rejects the input, or how the trees differ.
Nix looks variables up while it parses, so inputs are parsed inside
`with { };`. nix-instantiate runs without a store or nix.conf, from /
with a made-up HOME, so the results are the same on every machine.
The trees can match while a string's value doesn't: Nix reads escapes, carriage returns and indentation itself. With Checker::with_strings(), every string literal is also evaluated by Nix, with its interpolations replaced by a placeholder, and compared with the value rnix's Str::normalized_parts() gives it.
rnix-compat compares rnix's parse of every .nix file under the paths it is given, and of expressions given with -e, with Nix's, on one thread per CPU. It lists the inputs on which the parsers disagree and counts the outcomes, and exits with status 1 if rnix parses input that Nix accepts differently, so it can guard a corpus like nixpkgs.
cases.md lists inputs by the cause of the disagreement, with a short explanation of what Nix does: those found so far, from Nix's grammar, its tests, and generated inputs, plus inputs on which the parsers agree but which are easy to get wrong. tests/cases.rs records how each compares in cases.expect, so a fix shows up as a change to it.
Checker::check_file() reads a file and compares it, with an Unreadable outcome for a file that can't be read or isn't UTF-8 (all rnix parses). Outcome::is_agreement() says whether an outcome needs listing. The command uses both, and so can tests over directories.
test_data.expect records the inputs in rnix's test_data on which the parsers disagree: two pipe tests that Nix rejects. nix-lang.expect does the same for the language tests of the Nix release rnix follows, from $RNIX_COMPAT_NIX_SRC. When that release changes, the diff of the expectations shows what changed for rnix, new tests included.
nix develop .#compat puts nixVersions.nix_2_35 on PATH for compat/, and points RNIX_COMPAT_NIX_SRC at its source, for its language tests, and RNIX_COMPAT_NIXPKGS at the nixpkgs this flake locks, the largest corpus to compare.
compat/README.md states the policy (rnix follows CppNix's parser release by release, and tries first to parse what Nix accepts like Nix, then to reject what it rejects), how inputs are compared, what the outcomes mean, how to run the comparison, and what to do for a new Nix release. The release checklist refreshes compat's lockfile too.
</> isn't empty: Nix rejects it because a search path is names separated by /, which can't start or end with one.
Cargo reads .cargo/config.toml from the directory it runs in, so `cargo test --manifest-path compat/Cargo.toml` from rnix's root ran without CARGO_WORKSPACE_DIR, and expect-test looked for the .expect files relative to rnix's root. Absolute paths don't depend on that.
A NUL in a path ends the input to Nix too, a NUL in a comment is part of the comment to both parsers, and a NUL on its own is the end of the input, which Nix rejects. Say why Nix stops at a NUL, and that fixing the same in strings (NixOS/nix#12131, #12135) left it outside them.
Nix parses strings and comments holding any bytes; rnix, built on rowan, parses a &str. The case notation can't write such input, so the note points at the language test that shows it.
nix_files took any path that isn't a directory for a file, so a path that doesn't exist became an unreadable outcome, which isn't a failure: `rnix-compat --strings "$RNIX_COMPAT_NIXPKGS"` outside the dev shell compared nothing and exited 0. Now it's an error, with status 2.
The with { }; line in front of every input moved Nix's positions down a
line: parse-fail-dup-attrs-2.nix's duplicate was "at «stdin»:10:5",
where Nix's own parse-fail-dup-attrs-2.err.exp says 9:5. Errors now
match Nix's expected output, which comparing errors will need.
When Nix couldn't evaluate the string literals of an input, the outcome was a string-differs with an empty literal, which looks like rnix's fault. Like a rendering Nix rejects, it's a bug in compat, so it's now the outcome strings-rejected, which fails too.
Nix's --json writes them as \b and \f; compat wrote \u0008 and \u000c, so a string-differs report could show the same value twice in two ways that look different.
escape() left ⟨ alone, so the text ⟨LF⟩ and a line feed looked the same in reports. It writes ⟨ as ⟨U+27E8⟩ now, which unescape() reads.
From rnix's root, Cargo doesn't read compat/.cargo/config.toml, so UPDATE_EXPECT=1 can't update the expectations in tests/render.rs. And the rendering writes paths as they are, so a path rnix reads where Nix reads something else wouldn't show.
Nix's rendering of a parse prints 1, 1.0 and 1.0000001 all as 1, so the comparison of trees couldn't tell if rnix read an integer where Nix reads a float, or a float's value wrongly. Like strings, numbers are now evaluated by Nix and compared, by type and value, with what rnix's Integer::value and Float::value give: Nix's 1 == 1.0, so the types count too. A difference is the new outcome number-differs. As the check covers more than strings, --strings is now --values, with_strings() is with_values(), and strings-rejected is values-rejected.
…eadable ones A file that wasn't UTF-8 was unreadable, which isn't a failure, so a corpus run passed whatever was in it. Nix reads bytes, so compat now gives it the file anyway: if Nix accepts it, rnix, which only parses UTF-8, rejects it, a failure; if Nix rejects it too, both do. What's left of unreadable, a file that can't be read at all, fails a run now too. Nix's eval-fail-toJSON-non-utf-8.nix is rnix-rejects.
compat/cases.md lists input that Nix accepts and rnix parses wrongly, so following Nix's parser is a goal, not yet a fact.
If every call to Nix failed, because of a wrong RNIX_COMPAT_NIX_INSTANTIATE or a Nix without the dummy store, every input looked like one only rnix accepts, which isn't a failure, so a corpus run passed. Nix::new now checks that --version succeeds and that Nix parses 1 as 1, and fails otherwise.
json_string copied how builtins.toJSON escapes strings, and had already gotten backspace and form feed wrong. Nix now writes rnix's value too, from a Nix literal for it, as it does for numbers, so the two sides of a report always use the same escapes.
Besides rnix parsing input that Nix accepts differently, rnix-compat fails when compat itself does (render-rejected, values-rejected), and when a file can't be read, as the README says.
compat builds rnix from its checkout, so the claim holds as of a version, not for whichever rnix is compared.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary & Motivation
rnix now aims to parse like Nix (CppNix), following its parser release by release, starting with Nix 2.35, in two steps:
This adds
compat/, a crate outside rnix's workspace and package (likefuzz/) that compares the two parsers. The library doesn't change.For each input,
rnix-compatparses it with rnix and withnix-instantiate --parse. When both accept it, rnix's tree is written back as Nix with every expression in parentheses, and Nix parses that too: if Nix reads the source and the rendering as the same expression, rnix grouped the tokens like Nix does. Nix is the only printer involved, so nothing copies Nix's AST. Nix's rendering of a parse can't tell1,1.0and1.0000001apart, so with--values, Nix also evaluates every string and number literal, to check their values and types againstStr::normalized_parts,Integer::valueandFloat::value.nix develop .#compatpins the Nix release, andcompat/README.mdhas the details, including what to do when Nix releases.Results with Nix 2.35.3
#comment in two, and one isn't UTF-8, which rnix can't parse); 16 that rnix accepts (duplicate attributes and parameters, dynamic names, NUL in strings)test_datapipe_mixed.nixandpipe_mixed_math.nixare input Nix rejectscompat/cases.mdlists the inputs on which the parsers disagree, by cause, andcases.expectrecords their outcomes:0.a,1.5e,a/(b),a + !borf let { body = 1; }like Nix, and gets carriage returns and some escapes in strings wrong.f x: x,a |> b <| c,(a ], indented strings as attribute names, and numbers out of range.Further context
Next:
Related to #107.