* Base resume context
* Fixes for api change
* Align codegen
* Align `frame.rs` to the api changes
* fix jit
* Use new oparg
* Fix doc
* let `ir` to decide exception depth
* Bytecode parity phase 3
Compiler changes:
- Emit TO_BOOL in and/or short-circuit evaluation (COPY+TO_BOOL+JUMP)
- Add module-level __conditional_annotations__ cell (PEP 649)
- Only set conditional annotations for AnnAssign, not function params
- Skip __classdict__ cell when future annotations are active
- Convert list literals to tuples in for-loop iterables
- Fix cell variable ordering: parameters first, then alphabetical
- Fix RESUME DEPTH1 flag for yield-from/await
- Don't propagate __classdict__/__conditional_annotations__ freevar
through regular functions — only annotation/type-param scopes
- Inline string compilation path
* Skip test_thread_safety in _test_multiprocessing
SIGSEGV in _finalizer_registry dict access under aggressive GC
and thread switching. Root cause is dict thread-safety in VM.
* Skip list→tuple optimization for async for; propagate future_annotations to nested scopes
Align variant ordering with BinaryOperator enum and
CPython's NB_* constants from opcode.h. Divmod is placed
last as it has no corresponding NB_* constant.
Pre-compute referent pointers once per object in step 3 and reuse
them in step 4 (BFS reachability). Previously, gc_get_referent_ptrs()
was called independently in both steps. If a dict's write lock state
changed between the two calls (e.g., held by another thread during
one traversal but not the other), the two traversals could return
different results. This caused live objects to be incorrectly
classified as unreachable and cleared by GC.
* use `prek` for unified linting
* Fix actionlint error
* Generate metadata when specific files change
* `check_redundant_patches.py` to accept glob path
* Test
* revert defective changes
* use `rustfmt` over `cargo fmt` for individual files
* debug reviewdog
* rustfmt
* Move comment to correct location
* defevtive fmt test
* Fail with reviewdog
* fix reviewdog perms
* Try to use present token
* without checks oerms
* put normal perms
* fmt
* Match CPython LOAD_SPECIAL stack semantics for with/async-with
LOAD_SPECIAL now pushes (callable, self_or_null) matching CPython's
CALL convention, instead of a single bound method:
- Function descriptors: push (func, self)
- Plain attributes: push (bound, NULL)
Updated all with-statement paths:
- Entry: add SWAP 3 after SWAP 2, remove PUSH_NULL before CALL 0
- Normal exit: remove PUSH_NULL before CALL 3
- Exception handler (WITH_EXCEPT_START): read exit_func at TOS-4
and self_or_null at TOS-3
- Suppress block: 3 POP_TOPs after POP_EXCEPT (was 2)
- FBlock exit (preserve_tos): SWAP 3 + SWAP 2 rotation
- UnwindAction::With: remove PUSH_NULL
Stack effects updated: LoadSpecial (2,1), WithExceptStart (7,6)
* Normalize LOAD_FAST_CHECK and JUMP_BACKWARD_NO_INTERRUPT
Add LOAD_FAST_CHECK → LOAD_FAST and JUMP_BACKWARD_NO_INTERRUPT →
JUMP_BACKWARD to opname normalization in dis_dump.py. These are
optimization variants with identical semantics.
* Add EXTENDED_ARG to SKIP_OPS, normalize LOAD_FAST_CHECK and JUMP_BACKWARD_NO_INTERRUPT
* Remove duplicate return-None when block already has return
Skip duplicate_end_returns for blocks that already end with
LOAD_CONST + RETURN_VALUE. Run DCE + unreachable elimination
after duplication to remove the now-unreachable original return
block.
* Improve __static_attributes__ collection accuracy
- Support tuple/list unpacking targets: (self.x, self.y) = val
- Skip @staticmethod and @classmethod decorated methods
- Use scan_target_for_attrs helper for recursive target scanning
* Use method mode for function-local import attribute calls
Function-local imports (scope is Local+IMPORTED) should use method
mode LOAD_ATTR like regular names, not plain mode. Only module/class
scope imports use plain LOAD_ATTR + PUSH_NULL.
* Optimize constant iterable before GET_ITER to LOAD_CONST tuple
Convert BUILD_LIST/SET 0 + LOAD_CONST + LIST_EXTEND/SET_UPDATE + GET_ITER
to just LOAD_CONST (tuple) + GET_ITER, matching CPython's optimization
for constant list/set literals in for-loop iterables.
Also fix is_name_imported to use method mode for function-local imports,
and improve __static_attributes__ accuracy (skip @classmethod/@staticmethod,
handle tuple/list unpacking targets).
* Fix cell variable ordering: parameters first, then alphabetical
CPython orders cell variables with parameter cells first (in
parameter definition order), then non-parameter cells sorted
alphabetically. Previously all cells were sorted alphabetically.
Also add for-loop iterable optimization: constant BUILD_LIST/SET
before GET_ITER is folded to just LOAD_CONST tuple.
* Emit COPY_FREE_VARS before MAKE_CELL matching CPython order
CPython emits COPY_FREE_VARS first, then MAKE_CELL instructions.
Previously RustPython emitted them in reverse order.
* Fix RESUME AfterYield encoding to match CPython 3.14 (value 5)
CPython 3.14 uses RESUME arg=5 for after-yield, not 1.
Also reorder COPY_FREE_VARS before MAKE_CELL and fix cell
variable ordering (parameters first, then alphabetical).
* Address code review feedback from #7481
- Set is_generator flag for generator expressions in scan_comprehension
- Fix posonlyargs priority in collect_static_attributes first param
- Add match statement support to scan_store_attrs
- Fix stale decorator stack comment
- Reorder NOP removal after fold_unary_negative for better collection folding
* Fold constant list/set/tuple literals in compiler
When all elements of a list/set/tuple literal are constants and
there are 3+ elements, fold them into a single constant:
- list: BUILD_LIST 0 + LOAD_CONST (tuple) + LIST_EXTEND 1
- set: BUILD_SET 0 + LOAD_CONST (tuple) + SET_UPDATE 1
- tuple: LOAD_CONST (tuple)
This matches CPython's compiler optimization and fixes the most
common bytecode difference (92/200 sampled files).
Also add bytecode comparison scripts (dis_dump.py, compare_bytecode.py)
for systematic parity tracking.
* Use BUILD_MAP 0 + MAP_ADD for large dicts (>= 16 pairs)
Match CPython's compiler behavior: dicts with 16+ key-value pairs
use BUILD_MAP 0 followed by MAP_ADD for each pair, instead of
pushing all keys/values on the stack and calling BUILD_MAP N.
* Fix clippy warnings and cargo fmt
* fix surrogate
* fix: flush stdout on interpreter shutdown matching CPython behavior
When stdout flush fails during shutdown, report the error via
run_unraisable and exit with code 120 (matching CPython's
Py_FinalizeEx). Skip flushing already-closed or None streams.
Stderr flush errors remain silently ignored per CPython behavior.
Fixes#5521
Signed-off-by: majiayu000 <1835304752@qq.com>
* refactor: replace magic number 120 with named constant EXITCODE_FLUSH_FAILURE
Address review feedback on PR #7503: improve readability by extracting
the CPython-compat exit code into a named constant.
Signed-off-by: majiayu000 <1835304752@qq.com>
---------
Signed-off-by: majiayu000 <1835304752@qq.com>
* Bytecode parity
Compiler changes:
- Remove PUSH_NULL from decorator cal
ls, use CALL 0
- Collect __static_attributes__ from self.xxx = patterns
- Sort __static_attributes__ alphabetically
- Move __classdict__ init before __doc__ in class prologue
- Fold unary negative constants
- Fold constant list/set literals (3+ elements)
- Use BUILD_MAP 0 + MAP_ADD for 16+ dict pairs
- Always run peephole optimizer for s
uperinstructions
- Emit RETURN_GENERATOR for generator
functions
- Add is_generator flag to SymbolTabl
e
* Fix formatting and collapsible_if clippy warnings in compile.rs
* Fix clippy, fold_unary_negative chaining, and generator line tracing
- Replace irrefutable if-let with let for ExceptHandler
- Remove folded UNARY_NEGATIVE instead of replacing with NOP,
enabling chained negation folding
- Initialize prev_line to def line for generators/coroutines
to suppress spurious LINE events from preamble instructions
- Remove expectedFailure markers for now-passing tests
* Fix JIT StoreFastStoreFast, format, and remove expectedFailure markers
- Add StoreFastStoreFast handling in JIT instructions
- Fix cargo fmt in frame.rs
- Remove 11 expectedFailure markers for async jump tests in
test_sys_settrace that now pass
* Fix peephole optimizer: use NOP replacement instead of remove()
Using remove() shifts instruction indices and corrupts subsequent
references, causing "pop stackref but null found" panics at runtime.
Replace folded/combined instructions with NOP instead, which are
cleaned up by the existing remove_nops pass.
* Revert peephole_optimize to use remove() for chaining support
NOP replacement broke chaining of peephole optimizations (e.g.
LOAD_CONST+TO_BOOL then LOAD_CONST+UNARY_NOT for 'not True').
The remove() approach is used by upstream and works correctly here;
fold_unary_negative keeps NOP replacement since it doesn't need chaining.
* Fix StoreFastStoreFast to handle NULL from LoadFastAndClear
StoreFast uses pop_value_opt() to allow NULL values from
LoadFastAndClear in inlined comprehension cleanup paths.
StoreFastStoreFast must do the same, otherwise the peephole
optimizer's fusion of two StoreFast instructions panics when
restoring unbound locals after an inlined comprehension.
* Handle EINTR retry in os.write() (PEP 475)
Add EINTR retry loop to os.write(), matching the existing
pattern in os.read() and os.readinto(). Remove the
expectedFailure marker from test_write in _test_eintr.py.
* Add atomic snapshot for dict/dict_keys in extract_elements
Add fast paths for dict and dict_keys types in
extract_elements_with, matching _list_extend() in CPython
Objects/listobject.c. Each branch takes an atomic snapshot
under a single read lock, preventing race conditions from
concurrent dict mutation without the GIL.
Remove expectedFailure from test_thread_safety.
* Cleanup matrix usage. enables jit on macos
* Pin some actions to commit hash
* Disable node cache
* Inline `CARGO_ARGS`
* Add `stdio` and `host_env` features
* Only upload to pages if not running on fork
* Emit TO_BOOL before conditional jumps, fix class/module prologue
- Emit TO_BOOL before POP_JUMP_IF_TRUE/FALSE in the general case
of compile_jump_if (Compare expressions excluded since they
already produce a bool)
- Module-level __doc__: use STORE_NAME instead of STORE_GLOBAL
- Class body __module__: use LOAD_NAME instead of LOAD_GLOBAL
- Class body: store __firstlineno__ before __doc__
* Emit MAKE_CELL and COPY_FREE_VARS before RESUME
Emit MAKE_CELL for each cell variable and COPY_FREE_VARS N for
free variables at the start of each code object, before RESUME.
These instructions are no-ops in the VM but align the bytecode
with CPython 3.14's output.
* Emit __static_attributes__ at end of class bodies
Store a tuple of attribute names (currently always empty) as
__static_attributes__ in the class namespace, matching CPython
3.14's class body epilogue. Attribute name collection from
self.xxx accesses is a follow-up task.
* Remove expectedFailure from DictProxyTests iter tests
test_iter_keys, test_iter_values, test_iter_items now pass
because class bodies emit __static_attributes__ and
__firstlineno__, matching the expected dict key set.
* Use 1-based stack indexing for LIST_EXTEND, SET_UPDATE, etc.
Switch LIST_APPEND, LIST_EXTEND, SET_ADD, SET_UPDATE, MAP_ADD
from 0-based to 1-based stack depth argument, matching CPython's
PEEK(oparg) convention. Adjust the VM to subtract 1 before
calling nth_value.
* Use plain LOAD_ATTR + PUSH_NULL for calls on imported names
When the call target is an attribute of an imported name (e.g.,
logging.getLogger()), use plain LOAD_ATTR (method_flag=0) with
a separate PUSH_NULL instead of method-mode LOAD_ATTR. This
matches CPython 3.14's behavior which avoids the method call
optimization for module attribute access.
* Duplicate return-None epilogue for fall-through blocks
When the last block in a code object is exactly LOAD_CONST None +
RETURN_VALUE (the implicit return), duplicate these instructions
into blocks that would otherwise fall through to it. This matches
CPython 3.14's behavior of giving each code path its own explicit
return instruction.
* Run cargo fmt on ir.rs
* Remove expectedFailure from test_intrinsic_1 in test_dis
* Emit TO_BOOL before conditional jumps for all expressions including Compare
* Add __classdict__ cell for classes with function definitions
Set needs_classdict=true for class scopes that contain function
definitions (def/async def), matching CPython 3.14's behavior for
PEP 649 deferred annotation support. Also restore the Compare
expression check in compile_jump_if to skip TO_BOOL for comparison
operations.
* Emit __classdictcell__ store in class body epilogue
Store the __classdict__ cell reference as __classdictcell__ in
the class namespace when the class has __classdict__ as a cell
variable. Uses LOAD_DEREF (RustPython separates cell vars from
fast locals unlike CPython's unified array).
* Always run DCE to remove dead code after terminal instructions
Run basic dead code elimination (truncating instructions after
RETURN_VALUE/RAISE/JUMP within blocks) at all optimization
levels, not just optimize > 0. CPython always removes this dead
code during assembly.
* Restrict LOAD_ATTR plain mode to module/class scope imports
Only use plain LOAD_ATTR + PUSH_NULL for imports at module or
class scope. Function-local imports use method call mode LOAD_ATTR,
matching CPython 3.14's behavior.
* Eliminate unreachable blocks after jump normalization
Split DCE into two phases: (1) within-block truncation after
terminal instructions (always runs), (2) whole-block elimination
for blocks only reachable via fall-through from terminal blocks
(runs after normalize_jumps when dead jump instructions exist).
* Fold BUILD_TUPLE 0 into LOAD_CONST empty tuple
Convert BUILD_TUPLE with size 0 to LOAD_CONST () during constant
folding, matching CPython's optimization for empty tuple literals.
* Handle __classcell__ and __classdictcell__ in type.__new__
- Remove __classcell__ from class dict after setting the cell value
- Add __classdictcell__ handling: set cell to class namespace dict,
then remove from class dict
- Register __classdictcell__ identifier
- Use LoadClosure instead of LoadDeref for __classdictcell__ emission
- Reorder MakeFunctionFlag bits to match CPython
- Run ruff format on scripts
* Revert __classdict__ cell and __classdictcell__ changes
The __classdict__ cell addition (for classes with function defs)
and __classdictcell__ store caused cell initialization failures
in importlib. These require deeper VM changes to properly support
the cell variable lifecycle. Reverted for stability.
* Fix unreachable block elimination with fixpoint reachability
Use fixpoint iteration to properly determine block reachability:
only mark jump targets of already-reachable blocks, preventing
orphaned blocks from falsely marking their targets as reachable.
Also add a final DCE pass after assembly NOP removal to catch
dead code created by normalize_jumps.
* Check enclosing scopes for IMPORTED flag in LOAD_ATTR mode
When deciding whether to use plain LOAD_ATTR for attribute calls,
check if the name is imported in any enclosing scope (not just
the current scope). This handles the common pattern where a module
is imported at module level but used inside functions.
* Add __classdict__ cell for classes with function definitions
Set needs_classdict=true when a class scope contains function
definitions (def/async def), matching CPython 3.14 which always
creates a __classdict__ cell for PEP 649 support in such classes.
* Store __classdictcell__ in class body epilogue
Store the __classdict__ cell reference as __classdictcell__ in
the class namespace using LoadClosure (which loads the cell
object itself, not the value inside). This matches CPython 3.14's
class body epilogue.
* Fix clippy collapsible_if warnings and cargo fmt
* Revert __classdict__ and __classdictcell__ changes (cause import failures)
* Revert type.__new__ __classcell__ removal and __classdictcell__ handling
Revert the class cell cleanup changes from e6975f973 that cause
import failures when frozen module bytecode is stale. The original
behavior (not removing __classcell__ from class dict) is restored.
* Re-add __classdict__ cell and __classdictcell__ store
Restore the __classdict__ cell for classes with function
definitions and __classdictcell__ store in class body epilogue.
Previous failure was caused by stale .pyc cache files containing
bytecode from an intermediate MakeFunctionFlag reorder attempt,
not by these changes themselves.
* Reorder MakeFunctionFlag to match CPython's SET_FUNCTION_ATTRIBUTE
Reorder discriminants: Defaults=0, KwOnlyDefaults=1, Annotations=2,
Closure=3, Annotate=4, TypeParams=5. This aligns the oparg values
with CPython 3.14's convention.
Note: after this change, stale .pyc cache files must be deleted
(find . -name '*.pyc' -delete) to avoid bytecode mismatch errors.
* Use CPython-compatible power-of-two encoding for SET_FUNCTION_ATTRIBUTE
Override From/TryFrom for MakeFunctionFlag to use power-of-two
values (1,2,4,8,16,32) matching CPython's SET_FUNCTION_ATTRIBUTE
oparg encoding, instead of sequential discriminants (0,1,2,3,4,5).
* Remove expectedFailure from test_elim_jump_after_return1 and test_no_jump_over_return_out_of_finally_block
* Remove __classcell__ and __classdictcell__ from class dict in type.__new__
* Remove expectedFailure from test___classcell___expected_behaviour, cargo fmt
* Handle MakeCell and CopyFreeVars as no-ops in JIT
These prologue instructions are handled at frame creation time
by the VM. The JIT operates on already-initialized frames, so
these can be safely skipped during compilation.
* Remove expectedFailure from test_load_fast_known_simple
* Restore expectedFailure for test_load_fast_known_simple
The test expects LOAD_FAST_BORROW_LOAD_FAST_BORROW superinstruction
which RustPython does not emit yet.
When an inlined comprehension's first iterator expression contains
nested scopes (such as a lambda), those scopes' sub_tables appear at the
current position in the parent's sub_table list. The previous code
spliced the comprehension's own child sub_tables (e.g. inner inlined
comprehensions) into that same position before compiling the iterator,
which shifted the iterator's sub_tables to wrong indices.
Move the splice after the first iterator is compiled so its sub_tables
are consumed at their original positions.
Fixes nested list comprehensions like:
```python
[[x for _, x in g] for _, g in itertools.groupby(..., lambda x: ...)]
```
Disclosure: I used AI to develop the patch though I was heavily
involved.
* Disallow instantiation of sys.getwindowsversion type
Add slot_new to PyWindowsVersion that raises TypeError,
matching sys.flags behavior.
* Remove incorrect WSAHOS errno constant
WSAHOS was hardcoded as an alias for WSAHOST_NOT_FOUND, but
CPython guards it with #ifdef WSAHOS which doesn't exist in
modern Windows SDK headers.
* Fix mmap resize to raise OSError instead of SystemError
* Fix CreateProcess with empty environment on Windows
Empty env dict produced a single null terminator, but
CreateProcessW requires a double null for a valid empty
environment block.
* Revert mmap resize error to SystemError and fix errno.rs formatting
mmap resize raises SystemError (not OSError) when mremap is unavailable,
matching CPython behavior. test_mmap catches SystemError to skip unsupported
resize operations.
* Fix named mmap resize to raise OSError and unmark test_sleep expectedFailure
Named mmap resize on Windows should raise OSError (not SystemError).
Remove expectedFailure mark from TimeEINTRTest.test_sleep as it now passes.
* Use expectedFailureIf for TimeEINTRTest.test_sleep on Linux
test_sleep passes on macOS but fails on Linux due to timing.
* Remove expectedFailure for TimeEINTRTest.test_sleep
test_sleep now passes on all platforms.
* Enable PEP 709 inlined comprehensions for function-like scopes
Activate the existing compile_inlined_comprehension() implementation
by fixing 6 bugs that prevented it from working:
- LoadFastAndClear: push NULL (not None) when slot is empty so
StoreFast can restore empty state after comprehension
- StoreFast: accept NULL from stack for the restore path
- sub_tables.remove(0) replaced with next_sub_table cursor to
match the pattern used elsewhere in the compiler
- in_inlined_comp flag moved from non-inlined to inlined path
- is_inlined_comprehension_context() now checks comp_inlined flag
and restricts inlining to function-like scopes
- comp_inlined set only when parent scope uses fastlocals
Symbol table analysis handles conflict detection:
- Nested scopes in comprehension → skip inlining
- Bound name conflicts with parent symbol → skip inlining
- Cross-comprehension reference conflicts → skip inlining
- Splice comprehension sub_tables into parent for nested scope tracking
* Add localspluskinds, unify DEREF to localsplus index
- Add CO_FAST_LOCAL/CELL/FREE/HIDDEN constants and
localspluskinds field to CodeObject for per-slot metadata
- Change DEREF instruction opargs from cell-relative indices
(NameIdx) to localsplus absolute indices (oparg::VarNum)
- Add fixup_deref_opargs pass in ir.rs to convert cell-relative
indices to localsplus indices after finalization
- Replace get_cell_name with get_localsplus_name in
InstrDisplayContext trait
- Update VM cell_ref/get_cell_contents/set_cell_contents to use
localsplus indices directly (no nlocals offset)
- Update function.rs cell2arg, super.rs __class__ lookup with
explicit nlocals offsets
* Fix clippy warnings, formatting, restore _opcode_metadata.py
Fix cast_possible_truncation, nonminimal_bool, collapsible_if,
manual_contains clippy lints. Restore _opcode_metadata.py to
upstream/main version (3.14 aligned).
Pre-copy closure cells in Frame::new for coroutine locals().
Handle raw values in merged cell slots during inlined comps.
Exclude async comprehensions from inlining path.
* Exclude async/await comprehensions from PEP 709 inlining in symboltable
Async comprehensions and comprehensions with await in the element
expression need their own coroutine scope and cannot be inlined.
The symboltable builder was not checking these conditions, causing
incorrect symbol scope resolution when an async comprehension was
nested inside an inlined comprehension (e.g. [[x async for x in g]
for j in items]).
* Add GetDescriptor for PyBoundMethod (return self)
CPython's method_descr_get always returns the bound method unchanged.
This preserves the original binding when __get__ is called on an
already-bound method (e.g. a.meth.__get__(b, B) still returns a).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Add constructor validation for PyBoundMethod
Reject non-callable functions and None instances, matching CPython's
method_new which checks PyCallable_Check(func) and instance != Py_None.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Fix PyBoundMethod __reduce__ to propagate errors
Previously swallowed errors from get_attr with .ok(), silently
returning None. Now propagates errors matching CPython's method_reduce.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Detect list mutation during sort even when list length is unchanged
* Use mutation counter instead of capacity check for sort mutation detection
The capacity heuristic missed mutations when `clear()` reset capacity to
0 via `mem::take`. An AtomicU32 counter on PyList, incremented in
`borrow_vec_mut()`, reliably detects all mutations during sort.
* Hold write guard during sort mutation counter reads
* Fix list mutation counter race in `borrow_vec_mut`
* Fix allow_threads and EINTR handling
- Wrap Windows SemLock acquire wait with allow_threads
- Retry nanosleep on EINTR with remaining time instead of
returning early
- Remove expectedFailure for test_sleep in _test_eintr.py
* Remove expectedFailureIfWindows for testHashComparisonOfMethods
