Branch ev3 engine - #468
Draft
proto-aiken-13 wants to merge 32 commits into
Draft
proto-aiken-13 wants to merge 32 commits into
proto-aiken-13 wants to merge 32 commits into
Conversation
- Moved error handling logic to a dedicated errors module, improving organization and maintainability.
Updated ev3 engine such that it no longer needs the sling client class. The fake sling client class has been left in for now for potential testing purposes - however the plugin and frontend will be taking care of the connection to the ev3
…ort and added an Ev3Evaluator within the conductor
Previously, EV3 device functions (ev3_motorA, ev3_colorSensorGetColor, etc.)
had no representation in the SVML compiler's primitive-function table, so any
call to one would fail to compile with "Primitive function X not implemented".
Sinter already implements these as VM-internal functions (CALLV/CALLTV,
indices 0-40, fixed by sinter's own C source), but nothing in the compile
pipeline ever emitted those opcodes.
This adds first-class "internal function" support to SVMLCompiler, parallel
to its existing primitive-function handling:
- SVMLCompiler now accepts an optional `internalFunctions: Map<string, number>`
(constructor / fromProgram / fromFunctionNode), threaded through closures so
nested functions can also resolve internal calls.
- CompilerAnnotation gains isInternal/internalIndex fields alongside the
existing isPrimitive/primitiveIndex.
- getTokenAnnotation checks internalFunctions before falling back to
PRIMITIVE_FUNCTIONS.
- emitFunctionCall emits CALLV/CALLTV (mirroring the existing CALLP/CALLTP
path) when a call resolves to an internal function.
- emitLoadSymbol/emitStoreSymbol extend their isPrimitive guards to also
cover isInternal (no first-class references/assignment, consistent with
how primitives are already handled).
This is purely additive: internalFunctions defaults to undefined, so every
existing caller (PySvmlEvaluator, PySvmlSinterEvaluator, PyCseEvaluator*)
is unaffected byte-for-byte.
EV3Engine.ts is the first consumer: it builds internalFunctions from
stdlib/ev3.ts's EV3_FUNCTIONS list (index = array position, matching
sinter's ev3_functions.c ordering exactly) and passes it into
SVMLCompiler.fromProgram. EV3Engine now runs the full pipeline - parse,
analyze (with the ev3 stdlib group added for name resolution), compile,
assemble - and returns a base64-encoded SVML binary, rather than
JSON.stringify(program) as before.
Adds Ev3ExecutionPlugin (src/conductor/plugins/), a thin IPlugin that wires
EV3Engine into the existing {type:'run'} / {type:'result'} channel protocol
consumed by the frontend's Ev3WebPlugin. Adds engines/ev3/entry.ts as the
worker bootstrap (new Conduit + registerPlugin), built via a dedicated
rollup config since this plugin is not a BasicEvaluator and doesn't fit the
generic initialise.ts/scripts/build.ts path used by other evaluators.
Removes the old EV3Engine.ts implementation that only produced
JSON-serialized SVML and never assembled or executed anything - it was
never a complete pipeline, only a compilation smoke test.
…/py-slang into branch-ev3-engine
…o branch-ev3-engine
… for it Pynter's target is Python (SICPy) §3 specifically (see pynter/README.md, updated separately in the pynter repo) -- it has no runtime notion of "chapter" to gate narrower/wider rules for §1/§2/§4, so it implements §3's semantics unconditionally. Enforce that pathway boundary instead of silently running other chapters with the wrong rules: - runCodePvmlDetailed (pvml-runner.ts) and generateNativePynterTestCases (tests/utils.ts) now throw a clear error for variant !== 3. - PyPvmlPynterEvaluator's hardcoded chapter (previously 4, inconsistent with Pynter's actual scope) is now 3. - Every native-Pynter test call site across the suite now declares §3. Most are still-valid §3 programs nominally written for another chapter (linked-list/stream/pairmutator/list/parser-stdlib), so only the declared variant changes. One exception: stdlib.test.ts's "Chapter 1 Builtins" miscTests asserts §1-specific restrictions (bool/ function excluded from ==/!=/ordering, list literals rejected) that are genuinely false at §3, not just untested there -- that sweep is removed rather than mislabeled. - operator-conformance-pynter.test.ts's chapter loop is now just [3], matching its now-single-chapter scope. Renumbered EQP/NEQP (added for is/is not's pointer-identity semantics, distinct from ==/!=' structural equality) from 89/90 down to 85/86, so PYNTER_OPCODE_MAX can become a single contiguous threshold (0x56) now that Pynter implements them -- the prior gap (85-88 unimplemented, 89-90 implemented) couldn't be expressed as one cutoff without also wrongly admitting the still-unimplemented FLOORDIVG/NEWITER/FOR_ITER. Verified via yarn pynter:report against a Pynter build with the matching VM-side changes (separate PR in source-academy/pynter): 86% pass rate (1042/1207) on the native-Pynter suite. Remaining failures trace to one cause -- the CSE machine's is/is-not logic and this suite's own is-operator test data still reflect the pre-revamp restricted-identity spec, not yet updated to match -- plus a handful of pre-existing, unrelated float-precision and Unicode-grapheme mismatches. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…o branch-ev3-engine
…by chapter
Two independent fixes toward reviving the PVML-compiler-to-PVMLInterpreter
browser pathway (compile Python to PVML bytecode, run it with the pure-TS
PVMLInterpreter -- no WASM, no native binary):
1. resolver.ts: `global x` inside a function, when `x` has no top-level
assignment anywhere in the module, resolved `x` all the way to the
absolute-root builtins/prelude environment instead of stopping one level
in at the module environment. PVMLCompiler's getTokenAnnotation treats
anything resolving to that root environment as an unimplemented primitive
function lookup, so `def f(): global x; x = 1` failed to compile
("Primitive function x not implemented"). The CSE machine never hit this,
since its dynamic Map-based global environment doesn't need static
pre-declaration for a brand-new global name at all -- this was a latent
bug in shared resolver code that only PVML's static-slot model exposed.
Fixed the loop's stop condition to match the same "one below root" test
visitGlobalStmt already uses for its own isModuleLevel check.
2. New EQG12/NEQG12/LTG12/GTG12/LEG12/GEG12 opcodes (91-96): `==`/`!=`/
ordering mean different things at §1/§2 (bool -- and for ==/!=, function --
operands rejected outright) vs §3/§4 (bool participates as the int it is).
Rather than have the interpreter ask "which chapter is this?" at runtime,
PVMLCompiler now takes a required `variant` parameter (threaded through
fromProgram/fromFunctionNode, mirroring how the CSE machine threads
`variant` as an explicit argument rather than context state) and picks
between the two opcode families in getCompareOpCode based on it. Neither
family needs Pynter support: they're both above PYNTER_OPCODE_MAX, and
the *.g.12 family specifically can never reach Pynter regardless, since
the native-Pynter pathway is permanently restricted to §3 (pvml-runner.ts).
All PVMLCompiler.fromProgram call sites updated for the new required
`variant` parameter (PyPvmlEvaluator: 4, PyPvmlPynterEvaluator: 3,
pvml-runner.ts: threaded from its own variant param, tests: explicit or
defaulted to 4 in generatePVMLTestCases/compiledEntryOpcodes). Added
`Global / Nonlocal` and chapter-gated-comparison test coverage to
pvml.test.ts -- previously zero coverage existed for either, since global/
nonlocal used to silently no-op and the chapter distinction didn't exist.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…pathway
PVMLCompiler gains a `useGlobalMap` mode ("REPL mode", off by default —
"Pynter mode" stays exactly as before): module-level names compile to new
LDGG/STGG opcodes, backed by PVMLInterpreter's `globalEnv`, a dynamically-
growable name-indexed Map, instead of the usual fixed-size-array module
environment. This is what makes incremental, multi-chunk evaluation
possible at all -- Python's global scope can gain brand new names at any
time (a REPL chunk defining something a later chunk uses, or `global x`
introducing a name with no top-level assignment anywhere), which a
statically-sized array can't accommodate without knowing every global up
front. Native Pynter needs none of this (prelude+script are always compiled
and run as one single-shot unit, so its whole global namespace is known
upfront) and keeps its existing fast slot-based path entirely unchanged --
LDGG/STGG are additive and opt-in, never emitted unless a caller asks for
`useGlobalMap`.
Closures needed to become portable to make this useful: PVMLClosure now
carries a direct reference to its compiled PVMLIR (`ir`), resolved once at
creation time (NEWC), rather than a numeric index re-resolved against
"whichever program happens to be running" at call time. Without this, a
closure stored in the persistent global environment by one chunk (e.g. a
prelude-defined function) couldn't be called from a later, separately-
compiled chunk, since its index would refer to a completely different
program's functions array. `functionIndex` is kept alongside `ir` for
debug/display only.
The resolver gains a parallel `moduleNames` constructor param (distinct
from the existing `preludeNames`, which seeds the *root* builtins
environment as primitives, not globals) so a later chunk's module
environment can be seeded with every name already bound by the prelude or
earlier chunks, resolving them as ordinary globals.
PyPvmlEvaluator adopts the same persistence pattern PyCseEvaluatorBase
already uses (see PyCseEvaluator.ts): one persistent global environment
survives across evaluateChunk() calls, and each stdlib group's SICPy
prelude is compiled and run into it exactly once, memoized the same way
(`once()`). Also now wires up every stdlib group (linkedList/list/
pairmutator/stream), not just misc/math -- so pair()/head()/streams/etc.
are finally reachable through this pathway, closing a gap noted in
README.md.
Verified end-to-end: a dedicated pvml-repl-persistence.test.ts exercises
useGlobalMap mode directly (cross-chunk variables, functions, `global`
from within a function, brand-new globals a fixed array couldn't have
pre-sized, nonlocal closures unaffected); PyPvmlEvaluator.test.ts drives
the real evaluator through a mocked conductor (persistence, prelude
loading, print() output forwarding, error reporting). Full suite (2563
tests) and native-Pynter parity report both still green -- the parity
report actually improved by one (global-keyword 6/9 -> 7/9), since the
prior commit's resolver fix benefits native Pynter too (it's unconditional,
not gated by useGlobalMap).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The compiler has emitted EQP/NEQP for is/is not since earlier this branch, but the interpreter never implemented them -- is/is not compiled fine but threw "Unimplemented opcode" the moment a program actually ran one. Same design as the paired Pynter (C) implementation: pure identity (===), no bool-as-int coercion (so 1 is True stays false, unlike ==), no structural recursion into arrays (so [1,2] is [1,2] is false even though [1,2] == [1,2] is true). JS already gives the right answer for scalars (no separate identity to model) and for object references (arrays/closures), so this is the same left === right / left !== right pair strictEqual/strictNotEqual already use for ==/!=, just without the bool coercion. Documented one genuine, separate limitation surfaced while adding real execution tests: PVML doesn't distinguish int from float at the *value* level (both compile to a plain JS number, unlike the CSE machine's bigint/number split), so `1 is 1.0` comes out true here where real Python says false -- a pre-existing representation gap, not something this change introduces or fixes. Also a `yarn format:ci` fixup in resolver.ts (line wrapping only, from an earlier commit's edit). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…sons Brings this pathway's own test helper up to parity with the other two (generateTestCases for CSE, generateNativePynterTestCases for native Pynter): it previously had no `groups` parameter at all (compiling every case through PVMLCompiler.fromProgram's bare internal fallback, hardcoded to `[misc, math]`), so it could never exercise anything from linkedList/list/pairmutator/stream. It now: - Accepts `groups`, defaulting to `[misc, math]` to preserve existing callers' behavior. - Compiles and runs each group's SICPy prelude once per test case (in useGlobalMap mode) before the case itself, into the same PVMLInterpreter globalEnv -- so prelude-defined functions like pair()/head() are callable, mirroring PyPvmlEvaluator's own prelude loading. - Properly resolves through analyzeWithEnvironments (chapter validators now actually apply -- NoListsValidator, NoIsOperatorValidator, etc. -- where before compilation bypassed them entirely). - Gains a PVML_SKIP_REASONS mechanism (mirroring NATIVE_PYNTER_SKIP_REASONS): complex numbers and parse()/tokenize() are labeled skips, not failures, since neither has any PVML compiler/stdlib- group wiring at all. Surfaced and fixed one real bug while validating this against pvml.test.ts itself: visitForStmt hardcoded a local-slot STLG store for the loop variable, bypassing emitStoreSymbol entirely. Harmless before useGlobalMap existed (everything used slots anyway), but in useGlobalMap mode a module-level `for` loop's target got stored via STLG while every read of it went through the new isGlobal path (LDGG) -- silently reading `undefined` instead of the value just stored. Now routed through emitStoreSymbol like every other assignment. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Refactored the Ev3Evaluator and the EV3Engine to support the refactored pvml-compiler.
…o branch-ev3-engine
…repr() Gives the browser PVML pathway CSE-machine-equivalent numeric data layout, deliberately going beyond native Pynter's narrow 32-bit-embedded capabilities (per this project's explicit split: Pynter stays restricted, the browser pathway gets full desktop power). Native Pynter itself is untouched. - Python `int` now compiles to a genuine bigint (LGCBI + a bigint constant pool on PVMLIR), fixing the `1 is 1.0` int/float conflation bug, with a `targetsPynter` compiler flag preserving the old int32/float32 encoding for native Pynter and the assembler's binary format (neither can carry arbitrary precision). - Arithmetic/comparison opcodes are now bigint-aware, sharing a new `numericCompare` helper with the CSE machine (extracted from its pyCompare) so both engines agree on cross-type int/float ordering. - range()/for-loops, abs(), round() (banker's rounding), max()/min(), is_number()/is_integer()/is_float() all preserve or correctly report int-vs-float now instead of collapsing both to `number`. - str()/repr() reuse the CSE machine's own formatting code directly (see cse-interop.ts) via a PVML-value -> CSE-Value converter, rather than reimplementing Python's float/string/list formatting rules. repr() gets its own primitive index (Pynter's unused `prompt` stub) since it differs from str() only in quoting a bare string argument, which a shared index couldn't distinguish. - Complex numbers are a new PVMLBoxType variant (reusing the existing engine-agnostic PyComplexNumber class directly), with a complex constant pool + LGCC, a new POWG opcode for `**` (not previously supported by PVML at all, for any type), and real()/imag()/complex() builtins. Ordering correctly rejects complex operands, matching Python. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…class/spread calls, parser group, remaining builtins Brings the PVML-in-browser pathway from a single hardcoded chapter-4 evaluator to genuine per-chapter infrastructure matching PyCseEvaluator1..4, plus the language-feature and stdlib gaps that were blocking full parity. - PyPvmlEvaluator1..4: chapter-selectable evaluators (variant + stdlib groups), wired into scripts/build.ts and conductor/index.ts exactly like the CSE evaluators. PyPvmlEvaluator kept as a deprecated PyPvmlEvaluator4 alias for existing callers. - Computed/first-class function calls (`f = abs; f(-5)`, `funcs[0]()`) via a shared dispatchCall() extracted from call(), reused by CALL/CALLT, CALLA/CALLTA, and invokeValue(). - *args: rest params on the definition side (PVMLIR.hasRestParam) and call-site spread (`f(*xs)`) via two new opcodes (CALLA/CALLTA) plus a compiler-internal `_concat_arrays` primitive — rejected with a clear error in targetsPynter mode, since native Pynter has no representation for variadic argument arrays. - Chapter-4 parser stdlib group: parse()/tokenize() reuse the CSE machine's own transform()/lexer directly; apply_in_underlying_python() is built on a new PVMLInterpreter.invokeValue(), which lets a primitive synchronously recurse into the interpreter's own step() loop to run a nested call to completion — no async/resumable-machine plumbing needed, since PVML's run()/step() is already a flat loop over currentFrame/callerFrame links. - Remaining builtins: 22 `math` functions + time_time with no native-Pynter equivalent (arbitrary-precision math_comb/factorial/gcd/isqrt/lcm/perm included), named math constants (math_pi etc., via a new PRIMITIVE_CONSTANTS mechanism for bare-value-not-callable primitives), arity() (closures via hasRestParam/numArgs, primitives via a new PRIMITIVE_MIN_ARGS table derived from the CSE machine's own minArgMap so it can't drift out of sync), and stream() (the lazy continuation reuses the same primitive with a new PVMLPrimitive.boundArgs field rather than a runtime-synthesized closure, which this interpreter can't create). - Bug fix found while porting: math_ceil/math_floor/math_trunc returned a float (unaryMath) instead of a genuine Python int/bigint (unaryMathToInt), unlike the CSE machine's own implementation. - `from X import Y` now compiles to a no-op instead of throwing, matching CSE (SICPy has no real module system). Verified: full jest suite passing with 0 regressions (2555 tests, up from 2462), tsc clean on both configs, lint/format clean, all four evaluator bundles built and smoke-tested through the actual @sourceacademy/conductor API (mocked IRunnerPlugin), and a native-Pynter parity diff against the pre-change baseline showing zero new failures. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…te README - Fix a real bug: PVMLCompiler.isTailCall was dead code (never set anywhere in the codebase's history), so CALLT/CALLTP/CALLTA were never emitted for any real compiled program — PVML had zero working tail-call optimization despite the interpreter's frame-reuse logic being fully implemented and tested. Adds compileTail(), the sole place that sets isTailCall = true, scoped narrowly to the direct value of a `return` statement (recursively through both branches of a Ternary) so it can never leak into a call that merely appears inside a tail-position expression (`g(x)` in `return f(g(x))`, or `f(x)` in `return f(x) + 1`) — verified both via PVMLInterpreter and, in a new opt-in test file, against the real native Pynter binary (native Pynter's own VM already correctly implemented CALLT/CALLTP; only the compiler needed fixing). - Add a third CLI engine, `--engine pvml-browser` (src/repl.ts), running PVML bytecode directly on PVMLInterpreter — no native binary, all four SICPy chapters — plus a PY_SLANG_ENGINE environment variable for the default engine. New runCodePvmlInterpreter/runCodePvmlInterpreterDetailed headless runners in pvml-runner.ts back it, mirroring PyPvmlEvaluatorBase's persistent-globalEnv prelude model. Along the way, fixed a real output bug: PVMLInterpreter's print() deliberately omits the trailing newline itself (unlike CSE's), so naively concatenating sendOutput chunks ran consecutive print() calls together. - Update README: the CLI section for all three engines and the new env var; the evaluator table for PyPvmlEvaluator1..4 (was still showing the single deprecated PyPvmlEvaluator) with an accurate, narrower description for PyPvmlPynterEvaluator; a new "Tail-call optimization" subsection; and "Running the test suite" clarified into its three actual tiers (CSE and PVML-in-browser both always run, native Pynter parity is opt-in) with a corrected, no-longer-stale claim about which stdlib groups the PVML compiler wires up by default. Verified: full jest suite passing with 0 regressions (2571 tests, up from 2564), tsc clean on both configs, lint/format clean, the built dist/repl.cjs CLI manually exercised for all three engines (including byte-identical output between cse and pvml-browser on the same program) and both env-var and --flag precedence, and a native-Pynter parity diff against the pre-change baseline showing zero new failures. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ternalFunctions support The svml->pvml/pynter rename (and a parallel, more mature rewrite of the compiler - cross-scope slot-collision fix, correct NEWA/list-literal stack handling, primitive-as-value via NEWCP) left EV3Engine.ts, Ev3Evaluator.ts, and pvml-compiler.ts out of sync with each other and with the real resolver.ts/tokenizer.ts APIs. This fixes all three and closes the gaps found along the way. pvml-compiler.ts: - Reintegrates VM-internal-function support (isInternal/internalIndex on CompilerAnnotation, CALLV/CALLTV emission, NEWCV for primitive-as-value) on top of the current, more evolved compiler - not a reapplication of the older svml-compiler.ts diff, which would have silently dropped the envBuilders/tokenAnnotations cross-scope threading and the NEWA/NEWCP fixes already present here. - internalFunctions is a new optional constructor/fromProgram parameter, defaulting to undefined; every existing caller that doesn't pass it is unaffected. - visitCallExpr's isPrimitiveCallee optimization is generalized to also cover isInternal (CALLV, like CALLP, dispatches by index directly and never consumes a loaded callee value). EV3Engine.ts / Ev3Evaluator.ts: - Replaced the nonexistent `analyzeWithEnvironments` import with the real Resolver API (`new Resolver(...)` + `.resolveEnvironments()` + `.errors`) - this function never existed in resolver.ts. - Fixed stale svml-era names/paths (SVMLCompiler -> PVMLCompiler, SINTER_OPCODE_MAX -> PYNTER_OPCODE_MAX, engines/svml -> engines/pvml). - Fixed resolver/tokenizer import paths to their current one-level-deeper locations (resolver/resolver.ts, tokenizer/tokenizer.ts). - Both now build their PVMLCompiler internalFunctions map from stdlib/ev3.ts's EV3_FUNCTIONS and pass the `ev3` stdlib group into the resolver, so ev3_* calls actually resolve and compile to CALLV instead of throwing "Primitive function ev3_motorA not implemented". - Ev3Evaluator.ts additionally: swapped its execution backend from a guessed initSinter/WASM approach to the real one, native-pynter.ts's runNativePynter (spawns the native pynter `runner` binary via child_process). This is Node-only - Ev3Evaluator cannot run inside a browser Web Worker, only in a Node context (py-slang's own REPL/CLI). Added `/// <reference types="node" />` to resolve `process`, since src/ is otherwise kept Node-free for browser-safety. ev3-engine.test.ts: - Extends the existing compile-level suite with coverage of every operator in the Python §3 spec's operator table (docs.sourceacademy.org/python/ python_3.pdf, "Dynamic Type Checking" section). - Explicitly scoped to compile-only verification: PVMLCompiler never inspects operand types (type checking is dynamic/runtime, enforced by the VM), and EV3Engine.execute() only compiles+assembles, never executes - so these tests confirm operator syntax compiles, not the table's result-type rules. - Documents two confirmed, currently-unimplemented pieces of the spec as expected failures rather than silently skipping them: complex number literals (visitComplexExpr throws unconditionally) and is/is not (getCompareOpCode has no case for TokenType.IS/ISNOT, despite being valid per the spec's BNF).
…pport + Ev3Engine optimization PRIMITIVE_FUNCTIONS dummy-index registration and the CALLP->CALLV rewrite pass are no longer reachable from anywhere in the pipeline: pvml-compiler.ts now accepts internalFunctions directly and emits CALLV/CALLTV at compile time, so there's nothing left for a post-compile rewrite to do. Confirmed via repo-wide search that nothing imports this file (grep -rn 'ev3-primitives|rewriteEv3PrimitiveCalls|EV3_DUMMY_OFFSET' src/)."
…/py-slang into branch-ev3-engine
Added the variantNumber parameter to be used within fromProgram
Member
|
This PR has major problems. It needs to be rebased to the latest main branch: There are many commits in here that should already have been merged. |
martin-henz
marked this pull request as draft
September 15, 2026 08:16
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
py-slang repository to test out the EV3 engine on py-slang as per the latest pynter specifications.
This repository should be run with the following version of the frontend and plugin:
Frontend: source-academy/frontend#4025
Plugin: source-academy/plugins#54