Repository navigation
Support modern Python syntax (3.12-3.14): PEP 695, 701, 750, 758 - #1338
Open
CarterPerez-dev wants to merge 1 commit into
Open
CarterPerez-dev wants to merge 1 commit into
CarterPerez-dev wants to merge 1 commit into
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
CarterPerez-dev
force-pushed
the
modern-python-syntax
branch
from
November 2, 2026 18:23
e78fe51 to
598f942
Compare
Format all Python syntax up to 3.14, regardless of the Python version YAPF runs on, without changing the output for code that does not use it. Tokenizer: scan f-strings and t-strings with a state machine that follows the CPython 3.12+ lexer modes and emit each one as a single STRING token. Recognize identifiers with characters that PEP 3131 allows but \w does not match, `await` and `async` outside `async def` where they can only be keywords, and follow CPython's rules for form feeds and backslash-only lines in indentation where the old tokenizer produced broken output. Grammar: PEP 695 type parameters on def and class, the `type` statement, PEP 696 defaults, PEP 758 unparenthesized except lists, `'*' test` in subscripts and starred expressions after augmented assignment. Only treat soft keywords as keywords at the start of a statement. Formatter: format type parameter lists like parameter lists, keep `except*` together, keep the space in `6 .real`, report parse errors with their position instead of raising IndexError, and keep `match match:` on one line. Also fix bugs found by formatting CPython's Lib and the black and ruff formatter test suites: comments containing unicode line breaks, statements after `;` glued together on unformatted lines, lengths after semicolon splits, non-idempotent blank lines around comments, a crash on a backslash before a dictionary value, RecursionError on deeply nested expressions, grammar cache collisions between installs with the same version, a backslash after a compound statement's colon growing whitespace on every run, backslash-only lines inside indented blocks, non-idempotent splits before `|` and `&`, and spacing after strings continued with a backslash on lines that are not reformatted. Add golden tests for the new syntax under pep8, google, facebook and a custom style.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
YAPF 0.43.0 can't parse code that uses Python 3.12+ syntax. On PEP 695 generics, PEP 701 f-strings and PEP 750 t-strings it fails with
IndexError: tuple index out of range, which hides the realParseError. It also rewritesexcept*asexcept *. With this PR YAPF formats all Python syntax up to 3.14, whatever Python version it runs on. Output for code without the new syntax does not change, apart from the bug fixes listed below.Fixes #1170, fixes #1136, fixes #1319, fixes #1194, fixes #1198, fixes #1215, fixes #1250, fixes #1266, fixes #1277, fixes #1290, fixes #1303, fixes #1304.
What changed
Tokenizer (
_ylib2to3/pgen2/tokenize.py)f,rf,t,tr, any case) are scanned by a small state machine that follows the CPython 3.12+ lexer modes. That covers nested quotes of the same kind, nested replacement fields in format specs,{{/}},\N{...}, comments and backslashes in replacement fields, and replacement fields that span lines. Each f-string or t-string is still emitted as a singleSTRINGtoken, so the rest of YAPF keeps treating it as an atom and never changes it.\wdoesn't match (combining marks,℘,·).awaitoutsideasync def(awaitin a generator expression, top-level await) when the next token can only be an await operand, andasyncbefore a comment or a backslash continuation. Code that usesasync/awaitas names is tokenized as before.Grammar (
Grammar.txt,parse.py)typeparamsondefandclass; thetypestatement withtypeas a soft keyword; type parameter bounds, constraints and PEP 696 defaults;*Tsand**P.except/except*with any number of unparenthesized types (PEP 758).'*' testin subscripts (x[*a or b]) andtestlist_star_exprafter augmented assignment.match(type, _),match[0], match[1]and similar.Formatter
except*,6 .real(removing the space produced invalid code),type type[T], andexcept A, B,:.match match:andcase case:were split into two lines.FormatErrorMsgreportsParseErrorpositions instead of raisingIndexError.Bugs found by formatting CPython's
Lib/and the black and ruff formatter test suites (each one crashed, produced invalid code, changed the AST, hung or was non-idempotent in 0.43.0)\f,\v,\x1c-\x1e,\x85, U+2028 or U+2029 were split bystr.splitlines(). That broke indentation, hung the comment splicer, or changed the AST.a; b # pylint: disable=line-too-long(and the other unformatted paths) gluedaandbtogether, and so did a backslash continuation before the;.;were measured with the lengths of the earlier statements.AssertionErrorunderINDENT_DICTIONARY_VALUE.RecursionErroron deeply nested expressions that CPython accepts. On Python 3.12+ the recursion limit is raised while formatting.else: \) grew whitespace on every run. The backslash is now dropped and the body goes on its own line; a comment that followed it becomes a trailing comment of the header, which is where CPython reads it.await [](),await (yield x)andawait ...outsideasync defcrashed.|and&were treated as hand-written splits on the next run (the "retain the original split before a bitwise operator" heuristic), so long union annotations were not idempotent. When YAPF adds such a split, the line is solved again with that split pinned until the result is stable. In the corpora below that happened on 0.33% of the searched lines and never needed more than 3 searches.yapf: disable,--lines).Grammar cache: the pickle file name now includes a hash of
Grammar.txt. Two installs with the same version string, for example a development checkout next to the release, no longer load each other's cached grammar.Testing
yapftests/reformatter_modern_syntax_test.py, with fixtures inyapftests/testdata/modern_syntax/for each new syntax feature and for the continuation,awaitand union-split fixes. Every case is checked underpep8,google,facebookand a heavily customized style, and the test also asserts idempotency.tokenize: every string, f-string, t-string and identifier span matches on 27,485 real files and on 50,000 generated nested f-strings and t-strings. The only other differences are 3 files where columns after a form feed are shifted by one, and 5 files that CPython itself rejects.ast.dump(ast.parse(src)) == ast.dump(ast.parse(formatted))) and idempotency on CPython 3.14'sLib/, Home Assistant core, mypy, pydantic, and black's and ruff's test fixtures (27,544 files), under the 4 styles above (110,176 runs; inputs that CPython rejects are excluded from both columns):The one remaining non-idempotent result is ruff's
expression/dict_comp.pyundergoogle, a comment on its own line between a dict key and its value. 0.43.0 fails on it in the same way.except *becomingexcept*, 4 are a comment after a form feed inside a block that now gets the block's indentation instead of column 0, and 1 is a comment directly above anasync defthat is no longer separated from it by a blank line (the same as fordef).Other intended output changes
These shapes don't occur in any of the 27,544 files above outside of formatter test fixtures, but the behavior did change:
join_multiple_lines,if x: \followed by the body on the next line kept the backslash and indented the body twice in some cases; the backslash is now dropped there too.await (andawait [outsideasync defare formatted as await expressions, the way Python 3.7+ parses them, not as a call or subscript of a variable namedawait.