Skip to content

Support modern Python syntax (3.12-3.14): PEP 695, 701, 750, 758 - #1338

Open
CarterPerez-dev wants to merge 1 commit into
google:mainfrom
CarterPerez-dev:modern-python-syntax
Open

CarterPerez-dev wants to merge 1 commit into
google:mainfrom
CarterPerez-dev:modern-python-syntax

Conversation

@CarterPerez-dev

Copy link
Copy Markdown

YAPF 0.43.0 can't parse code that uses Python 3.12+ syntax. On PEP 695 generics, PEP 701 f-strings and PEP 750 t-strings it fails with IndexError: tuple index out of range, which hides the real ParseError. It also rewrites except* as except *. With this PR YAPF formats all Python syntax up to 3.14, whatever Python version it runs on. Output for code without the new syntax does not change, apart from the bug fixes listed below.

Fixes #1170, fixes #1136, fixes #1319, fixes #1194, fixes #1198, fixes #1215, fixes #1250, fixes #1266, fixes #1277, fixes #1290, fixes #1303, fixes #1304.

What changed

Tokenizer (_ylib2to3/pgen2/tokenize.py)

  • f-strings and t-strings with any prefix (f, rf, t, tr, any case) are scanned by a small state machine that follows the CPython 3.12+ lexer modes. That covers nested quotes of the same kind, nested replacement fields in format specs, {{/}}, \N{...}, comments and backslashes in replacement fields, and replacement fields that span lines. Each f-string or t-string is still emitted as a single STRING token, so the rest of YAPF keeps treating it as an atom and never changes it.
  • Identifiers containing characters that PEP 3131 allows but \w doesn't match (combining marks, ℘, ·).
  • await outside async def (await in a generator expression, top-level await) when the next token can only be an await operand, and async before a comment or a backslash continuation. Code that uses async/await as names is tokenized as before.
  • Form feeds in leading whitespace, and backslash-only lines in indentation, follow CPython's rules.

Grammar (Grammar.txt, parse.py)

  • typeparams on def and class; the type statement with type as a soft keyword; type parameter bounds, constraints and PEP 696 defaults; *Ts and **P.
  • except / except* with any number of unparenthesized types (PEP 758).
  • '*' test in subscripts (x[*a or b]) and testlist_star_expr after augmented assignment.
  • Soft keywords are only considered at the start of a statement. Without this, nested backtracking crashed on match(type, _), match[0], match[1] and similar.

Formatter

  • Type parameter lists are formatted like parameter lists, including the magic trailing comma, dedent-style closing brackets and named-assign splitting for defaults. A parameter list that follows a type parameter list is indented as if the type parameters were not there.
  • except*, 6 .real (removing the space produced invalid code), type type[T], and except A, B,:.
  • match match: and case case: were split into two lines.
  • Single-quoted f-strings with a newline in a replacement field are treated as multi-line strings.
  • FormatErrorMsg reports ParseError positions instead of raising IndexError.

Bugs found by formatting CPython's Lib/ and the black and ruff formatter test suites (each one crashed, produced invalid code, changed the AST, hung or was non-idempotent in 0.43.0)

  • Comments containing \f, \v, \x1c-\x1e, \x85, U+2028 or U+2029 were split by str.splitlines(). That broke indentation, hung the comment splicer, or changed the AST.
  • a; b # pylint: disable=line-too-long (and the other unformatted paths) glued a and b together, and so did a backslash continuation before the ;.
  • Statements after a ; were measured with the lengths of the earlier statements.
  • Several blank-line rules around comments were not idempotent.
  • A backslash before a dictionary value raised AssertionError under INDENT_DICTIONARY_VALUE.
  • RecursionError on deeply nested expressions that CPython accepts. On Python 3.12+ the recursion limit is raised while formatting.
  • A backslash right after a compound statement's colon (else: \) grew whitespace on every run. The backslash is now dropped and the body goes on its own line; a comment that followed it becomes a trailing comment of the header, which is where CPython reads it.
  • A backslash-only line before an indented line, or anywhere inside an indented block, crashed or produced invalid code.
  • await [](), await (yield x) and await ... outside async def crashed.
  • Splits that YAPF itself adds before | and & were treated as hand-written splits on the next run (the "retain the original split before a bitwise operator" heuristic), so long union annotations were not idempotent. When YAPF adds such a split, the line is solved again with that split pinned until the result is stable. In the corpora below that happened on 0.33% of the searched lines and never needed more than 3 searches.
  • A comment after a string continued with a backslash was positioned relative to the string's first line, and the spacing after such strings changed on lines that are not reformatted (yapf: disable, --lines).

Grammar cache: the pickle file name now includes a hash of Grammar.txt. Two installs with the same version string, for example a development checkout next to the release, no longer load each other's cached grammar.

Testing

  • The existing suite passes on Python 3.7 through 3.14 (3.7 in a container): 659 tests.
  • New golden tests in yapftests/reformatter_modern_syntax_test.py, with fixtures in yapftests/testdata/modern_syntax/ for each new syntax feature and for the continuation, await and union-split fixes. Every case is checked under pep8, google, facebook and a heavily customized style, and the test also asserts idempotency.
  • Differential test of the tokenizer against CPython 3.14's tokenize: every string, f-string, t-string and identifier span matches on 27,485 real files and on 50,000 generated nested f-strings and t-strings. The only other differences are 3 files where columns after a form feed are shifted by one, and 5 files that CPython itself rejects.
  • AST equivalence (ast.dump(ast.parse(src)) == ast.dump(ast.parse(formatted))) and idempotency on CPython 3.14's Lib/, Home Assistant core, mypy, pydantic, and black's and ruff's test fixtures (27,544 files), under the 4 styles above (110,176 runs; inputs that CPython rejects are excluded from both columns):
yapf 0.43.0 this PR
Formatted correctly 101,308 108,715
Crash 7,194 0
Output changes the AST or does not parse 141 0
Second run changes the output 53 1
Timeout 20 0

The one remaining non-idempotent result is ruff's expression/dict_comp.py under google, a comment on its own line between a dict key and its value. 0.43.0 fails on it in the same way.

  • Byte-for-byte comparison with 0.43.0 on every run that 0.43.0 formats correctly: 101,146 of 101,308 outputs are identical. The 162 that differ are all intended: 157 are except * becoming except*, 4 are a comment after a form feed inside a block that now gets the block's indentation instead of column 0, and 1 is a comment directly above an async def that is no longer separated from it by a blank line (the same as for def).
  • Negative controls: deliberately dropping a name from the output, bumping a number literal, or changing comments on every pass is caught by the AST and idempotency checks on 100% of the files where the change applies.

Other intended output changes

These shapes don't occur in any of the 27,544 files above outside of formatter test fixtures, but the behavior did change:

  • A backslash-only line at column 0 inside an indented block is treated as CPython does (a blank line), instead of being kept. 0.43.0 kept it only when the next statement was the last one in its block or at module level, and crashed or produced invalid code otherwise.
  • With join_multiple_lines, if x: \ followed by the body on the next line kept the backslash and indented the body twice in some cases; the backslash is now dropped there too.
  • await ( and await [ outside async def are formatted as await expressions, the way Python 3.7+ parses them, not as a call or subscript of a variable named await.

@google-cla

google-cla Bot commented Nov 2, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

Format all Python syntax up to 3.14, regardless of the Python version YAPF
runs on, without changing the output for code that does not use it.

Tokenizer: scan f-strings and t-strings with a state machine that follows
the CPython 3.12+ lexer modes and emit each one as a single STRING token.
Recognize identifiers with characters that PEP 3131 allows but \w does not
match, `await` and `async` outside `async def` where they can only be
keywords, and follow CPython's rules for form feeds and backslash-only
lines in indentation where the old tokenizer produced broken output.

Grammar: PEP 695 type parameters on def and class, the `type` statement,
PEP 696 defaults, PEP 758 unparenthesized except lists, `'*' test` in
subscripts and starred expressions after augmented assignment. Only treat
soft keywords as keywords at the start of a statement.

Formatter: format type parameter lists like parameter lists, keep
`except*` together, keep the space in `6 .real`, report parse errors with
their position instead of raising IndexError, and keep `match match:` on
one line.

Also fix bugs found by formatting CPython's Lib and the black and ruff
formatter test suites: comments containing unicode line breaks, statements
after `;` glued together on unformatted lines, lengths after semicolon
splits, non-idempotent blank lines around comments, a crash on a backslash
before a dictionary value, RecursionError on deeply nested expressions,
grammar cache collisions between installs with the same version, a
backslash after a compound statement's colon growing whitespace on every
run, backslash-only lines inside indented blocks, non-idempotent splits
before `|` and `&`, and spacing after strings continued with a backslash on
lines that are not reformatted.

Add golden tests for the new syntax under pep8, google, facebook and a
custom style.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment