Repository navigation
Parse what Nix accepts like Nix does - #203
Open
philiptaron wants to merge 14 commits into
Open
philiptaron wants to merge 14 commits into
philiptaron wants to merge 14 commits into
Conversation
This was referenced Oct 30, 2026
philiptaron
force-pushed
the
parse-like-nix
branch
2 times, most recently
from
October 31, 2026 01:13
f5361a2 to
e584e6a
Compare
Nix's comments are #[^\r\n]*. rnix's ran to the next line feed, so with carriage returns alone between lines, the lines after a comment were part of it: Nix reads `f # c<CR>x` as `f x`, rnix read `f`. This fixes Nix's parse-okay-crlf.nix and eval-fail-eol-2.nix too.
tests/check.rs used 0.a and 1.5e, which rnix is about to parse like Nix, as examples of rnix parsing input differently. A NUL, which rnix rejects on purpose, stays one.
Nix's float literals are [1-9][0-9]*\.[0-9]* or 0?\.[0-9]+, so 1. is a float but 0. isn't: Nix reads 0.a as selecting a from 0, and 0.e1 as selecting e1. rnix read a float 0. followed by a. Leading zeros keep working as before: 00.5 is 00 followed by .5.
Nix's float literals end with an optional [Ee][+-]?[0-9]+, so it reads 1.5e as the float 1.5 followed by the variable e. rnix made 1.5e an error token.
A Nix path needs a path character or an interpolation after each /, so a/(b), 2/"c" and d/[ ] are divisions. rnix took any / followed by something other than whitespace as part of a path, and rejected them.
Nix parses ! at the start of any operand: a + !b is a + (!b), and the operand of ! reaches as far as operators that bind more tightly than ! do, so a + !b + c is a + !(b + c). rnix only parsed ! at the start of an operand of an operator that binds more loosely, like //, and rejected a + !b and -!a.
In Nix's grammar, let { ... } is an expr_simple, like rec { ... }: it
can be an argument, an operand or a list element, and can be followed
by a select or or. rnix only parsed it where an expression starts, and
stopped after it, so it rejected f let { body = 1; } and attached the
or of e: let { } or to the function.
In a "..." string, Nix reads a carriage return, alone or before a line feed, as a line feed, while one escaped with a backslash stays a carriage return. Str::normalized_parts kept them, so a file with Windows line endings gave strings other values than in Nix. With this, rnix agrees with all of Nix's parse-okay-crlf.nix.
Nix's lexer makes each escape in an indented string ('''', ''$ and
''\x) a token of its own, and its parser strips the indentation token
by token: an escape counts as text when it measures the indentation,
but what the escape stands for is stripped like what's written, so the
spaces after an escaped line feed go, and only the last token can lose
a last line of spaces. Str::normalized_parts stripped the text before
unescaping it, so '' ''\n '' was "\n " instead of "\n".
normalized_parts now splits the text like Nix's lexer and follows its
parser's stripIndentation. In 78,369 generated indented and "..."
strings, it now gives every string the value Nix does; before this and
the fix for carriage returns, 3,254 differed.
Nix ends its input at a NUL, by accident it seems. Following it would mean ignoring whatever comes after one.
After an interpolation, Nix's lexer reads the rest of a path a / or a
name at a time, so ./${a}//b is a path. Before one, // can't be in a
path, and no path can end with /. rnix rejected // anywhere it wasn't
right before an interpolation. Found by generating inputs.
A ~ doesn't continue a path, so Nix reads ./${a}~/b as ./${a} applied
to ~/b. rnix's tokenizer ends the first path there too, but the parser
took any path token right after an interpolated path as more of it.
Found by generating inputs: in /${l}~/${a} or, the or attached to the
wrong expression.
rnix now parses all the input in cases.md that Nix accepts like Nix, but for a NUL, on purpose, and input that isn't UTF-8.
philiptaron
force-pushed
the
parse-like-nix
branch
from
October 31, 2026 02:33
e584e6a to
870e6a3
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary & Motivation
Stacked on #202. This is the first of its two goals: rnix parses input that Nix accepts to the same tree as Nix. It fixes every cause in
compat/cases.mdof rnix getting such input wrong, and two more that generated inputs found:comment-crf # c⟨CR⟩xf x: a carriage return ends a commentfzero-dot0.aafrom00.applied toafloat-exponent1.5e1.5followed byeslash-operanda/(b)not-operanda + !b + ca + !(b + c)legacy-letf let { body = 1; }let { … }is a simple expressionstring-cr"a⟨CR⟩⟨LF⟩b""a\nb""a\r\nb"indented-escape'' ''\n ''"\n", from stripping indentation token by token"\n "path-double-slash./${a}//bpath-then-home./${a}~/b./${a}applied to~/bEach fix is a commit of its own, with a test in rnix's own suite (in
test_data, or a unit test for carriage returns, which Git could rewrite in a file), and the change it makes tocompat/cases.expect.With Nix 2.35.3:
compat/cases.md: rnix parses every input that Nix accepts like Nix, but for a NUL outside a string, which rnix rejects on purpose (Nix stops reading at it), and input that isn't UTF-8, which rnix can't parse.parse-okay-crlf.nixandeval-fail-eol-2.nixnow agree. The 16 disagreements left are all over input that Nix rejects.Backwards-incompatible changes
Trees and string values change wherever they differed from Nix's:
0.ais now a select,1.5ean application, a comment no longer runs past a carriage return, andStr::normalized_partsgives other values for strings with carriage returns, or with escapes in their indentation. The API doesn't change.Further context
The changelog lists these fixes for the next release.