Commit Graph

102 Commits

Author SHA1 Message Date
Jeong, YunWon
9140ef583a Bytecode parity - boolop, comprehension, CFG passes (#7631)
* Bytecode parity - boolop, comprehension, CFG passes

- Flatten nested same-op BoolOp and add IfExpr to jump_if
- Simplify is_name_imported to module-level only
- Enable inlined comprehensions in module/class scope
- Add TweakInlinedComprehensionScopes with fast_hidden tracking
- Track fb_range in FBlockInfo for with-statement line info
- Add constant subscript folding and unary Not folding
- Add emit_return_const_no_location for implicit returns
- Use JumpNoInterrupt for ternary/except jumps
- Add STACK_USE_GUIDELINE threshold for collection building
- Reorder CFG pipeline: inline small blocks earlier, resolve
  line numbers before cold block extraction
- Add redirect_empty_unconditional_jump_targets,
  reorder_conditional_chain_and_jump_back_blocks,
  materialize_empty_conditional_exit_targets,
  duplicate_shared_jump_back_targets passes
- Add borrow deoptimization for multi-handler, named-except,
  protected conditional tail, and protected import joins
- Run folding/optimization passes twice around peephole

* Bytecode parity - slice augassign, async comp inline

- Augmented assignment with two-part slices uses BINARY_SLICE/STORE_SLICE
- Inline async comprehensions (remove await/async guards)
- Inlined comprehension cleanup jump uses JumpNoInterrupt
- Class firstlineno uses first decorator line when decorated
- Recurse into nested functions for static attribute collection
- Fold unary positive complex constants (+0.0j)
- Add deoptimize_borrow_for_folded_nonliteral_exprs pass
- Add inline_single_predecessor_artificial_expr_exit_blocks pass
- Skip shared artificial expr-stmt exit blocks in small-block inlining
- Mark folded boolop tail as folded_from_nonliteral_expr

* apply reviews
2026-04-21 00:10:00 +09:00
Jeong, YunWon
1f1be5e29e Align bytecode codegen structure with CPython 3.14 (#7588)
* Align bytecode codegen structure with CPython 3.14

* Bytecode parity - constant folding, annotation ordering, superinstruction alignment

- Add BoolOp constant folding with short-circuit semantics in compile_expression
- Add constant truthiness evaluation for assert statement optimization
- Disable const collection/boolop folding in starred unpack and assignment contexts
- Move annotation block generation after body with AnnotationsPlaceholder splicing
- Reorder insert_superinstructions to run before push_cold_blocks (matching flowgraph.c)
- Lower LOAD_CLOSURE after superinstructions to avoid false LOAD_FAST_LOAD_FAST
- Add ToBool before PopJumpIf in comparisons and chained compare cleanup blocks
- Unify annotation dict building to always use incremental BuildMap + StoreSubscr
- Add TrueDivide constant folding for integer operands
- Fold constant sets to Frozenset (not Tuple) in try_fold_constant_collection
- Add PyVmBag for frozenset constant materialization in code objects
- Add remove_redundant_const_pop_top_pairs pass and peephole const+branch folding
- Emit Nop for skipped constant expressions and constant-true asserts
- Preserve comprehension local ordering by source-order bound name collection
- Simplify annotation scanning in symboltable (remove simple-name gate)

* Fix CI regressions in marshal and fast-local ops

* impl more

* Align bytecode codegen with CPython structure

* Bytecode parity - comprehension/except scope ordering, load_fast_borrow fixes

- Reorder comprehension symbol-table walk so the outermost iterator
  registers its sub_tables in the enclosing scope before the comp
  scope, and rescan elt/ifs in CPython's order. Codegen peeks past the
  outermost iterator's nested scopes to find the comprehension table.
- For plain try/except, emit handler sub_tables before the else block
  so codegen's linear sub_table cursor stays aligned.
- Rename `collect_simple_annotations` to `collect_annotations` and
  evaluate non-simple annotations during __annotate__ compilation to
  preserve source-order side effects while keeping the simple-name
  index stable.
- Dedupe equivalent code constants in `arg_constant` and add a
  structural equality check on `CodeObject`.
- Disable LOAD_FAST_BORROW for the tail end block when a try has a
  bare `except:` clause, and have `new_block` inherit the flag from
  the current block.
- Remove `cfg!(debug_assertions)` guard around the
  `optimize_load_fast_borrow` start-depth check so mismatches are
  handled (return instead of assert) in release builds.
- Collapse nop-only blocks that precede a return epilogue and hoist
  the prior line number into the next real instruction so the
  line table matches.
- Unmark now-passing `test_consts_in_conditionals`,
  `test_load_fast_unknown_simple`,
  `test_load_fast_known_because_already_loaded`, and PEP 646 f3/f4
  annotation checks.

* Bytecode parity - try/except line tracking, assert 0 shape

- In `compile_try_except`, drop the leading Nop and set the end
  block's source range from the last orelse/body statement so line
  events after the try fall on the right line.
- Recognise constant-false asserts as the direct-raise shape (no
  ToBool/PopJumpIfFalse) and flip the test assertion accordingly.
- Extend `remove_redundant_nops_in_blocks` to also look through a
  trailing nop before a return-epilogue pair (LoadConst/ReturnValue
  or LoadSmallInt/ReturnValue) so the epilogue keeps the correct
  line number.
- Rename `preds` to `predecessor_blocks` in the LOAD_FAST_BORROW
  disable pass and add a test-only `debug_late_cfg_trace` helper.
- Regenerate the `nested_double_async_with` snapshot: the tail
  reference to `stop_exc` now emits LOAD_FAST instead of
  LOAD_FAST_BORROW.

* Bytecode parity - iter folding, break/continue line, cold inlining

- Fold a constant list iterable into a constant tuple in for-loop
  iterable position, matching the CPython optimizer, and strip a
  redundant LIST_TO_TUPLE immediately before GET_ITER in the IR
  peephole pass.
- Emit a Nop at the break/continue source range before unwinding
  so line events land on the break/continue statement instead of
  the following instruction.
- Drop `propagate_disable_load_fast_borrow`; the forward propagation
  was over-zealous and the per-block inheritance in `new_block` plus
  the bare-except marker are enough.
- Relax `inline_small_or_no_lineno_blocks` so small exit blocks at
  the tail of a cold block are always inlined, not just return
  epilogues.
- Add codegen tests covering the LIST_TO_TUPLE/GET_ITER peephole and
  the late-CFG trace helper for a for-loop list-literal iterable.
2026-04-18 09:19:11 +09:00
Shahar Naveh
2827eca293 Simplify Intruction.deopt() (#7615)
* Simplify `Instruction.deopt()`

* Adjust `scripts/generate_opcode_metadata.py`
2026-04-17 20:04:22 +09:00
Shahar Naveh
330aa08488 Macro for defining opcode & instruction enums (#7573)
* Macro for defining opcode & instruction enums

* Convert `Instruction` as well

* revert `#[repr(...)]` changes
2026-04-16 03:18:14 +09:00
Jeong, YunWon
d201c48e1c Bytecode parity - for-loop cleanup, return duplication, late jump threading fix (#7580)
- Use POP_TOP instead of POP_ITER for for-loop break/return cleanup
- Expand duplicate_end_returns to clone final return for jump predecessors
- Restrict late jump threading pass to unconditional jumps only
- Skip exception blocks in inline/reorder passes
- Simplify threaded_jump_instr NoInterrupt handling
2026-04-12 22:43:35 +09:00
Jeong, YunWon
eac45727d2 Bytecode parity - direct loop backedges (#7578) 2026-04-10 18:48:57 +09:00
Jeong, YunWon
a49ce5bf34 Bytecode parity - exception (#7557)
* Fix exception handling: except* chaining, finally cleanup, RERAISE

- Align except* bytecode chaining
- Fix exception state model and finally handler cleanup
- Fix RERAISE to only pop exception, preserve values below

* Port IR optimization passes from flowgraph.c

- BUILD_TUPLE n + UNPACK_SEQUENCE n elimination
- Dead store elimination within basic blocks
- apply_static_swaps for SWAP reduction

* Add bytecode comparison and disassembly dump scripts

- compare_bytecode.py: compare CPython vs RustPython bytecode output
- dis_dump.py: extract disassembly in normalized JSON format
2026-04-10 12:51:48 +09:00
Shahar Naveh
9282a870db Unify lint CI job (#7505)
* use `prek` for unified linting

* Fix actionlint error

* Generate metadata when specific files change

* `check_redundant_patches.py` to accept glob path

* Test

* revert defective changes

* use `rustfmt` over `cargo fmt` for individual files

* debug reviewdog

* rustfmt

* Move comment to correct location

* defevtive fmt test

* Fail with reviewdog

* fix reviewdog perms

* Try to use present token

* without checks oerms

* put normal perms

* fmt
2026-03-26 21:43:34 +09:00
Shahar Naveh
9a5de28b79 Align _opcode_metadata.py to 3.14.3 (#7456)
* Align `_opcode_metadata.py` to 3.14.3

* Unmark passing test

* Ensure python 3.14 runs on CI

* Update banner

* Fix `test__opcode.py`

* Adjust generate script

* Fix docs
2026-03-18 22:09:00 +09:00
Shahar Naveh
ab32c785ea Restructure CI with matrix approach and multi-feature support (#7376)
* `--features` flag can take multiple values

* Simplify CI job

* Remove bad default

* Enable jit on macOS
2026-03-07 20:28:26 +09:00
Shahar Naveh
798f9e80cd Update libc to 0.2.182 (#7247)
* Update to libc 0.2.182, refactor update script

* Fix typing
2026-03-04 19:57:53 +09:00
Jeong, YunWon
d0b5a5af50 [update_lib] fast date lookup for todo (#7299)
* [update_lib] fast date lookup for todo

* add deps

* Auto-format: ruff format

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-03-02 01:23:52 +09:00
Shahar Naveh
b05e74b78e Don't return frozenset for modules (#7276) 2026-03-01 09:43:34 +09:00
Shahar Naveh
f3cf86c8c2 More accurate test dependencies (#7261) 2026-02-28 22:11:07 +09:00
Shahar Naveh
e0eb6d5e8e Reapply "ast.NodeVisitor for import tracking" (#7241)
* Reapply "`ast.NodeVisitor` for import tracking (#7229)" (#7230)

This reverts commit a47572c89e.
2026-02-28 10:37:00 +09:00
CPython Developers
abc7d6dfe6 Update test_monitoring from CPython 3.14.3 2026-02-27 04:36:46 +09:00
Jeong, YunWon
a47572c89e Revert "ast.NodeVisitor for import tracking (#7229)" (#7230)
This reverts commit 804a7e42fc.
2026-02-27 03:55:13 +09:00
Shahar Naveh
804a7e42fc ast.NodeVisitor for import tracking (#7229) 2026-02-27 03:07:57 +09:00
Shahar Naveh
457d328294 update_lib test format fix + deps for atexit & eintr (#7227)
* update_lib deps for atexit & eintr

* Fix test format
2026-02-26 12:03:47 +09:00
Shahar Naveh
64088bb624 Add CI step for checking redundant test patches (#7126) 2026-02-15 20:39:57 +09:00
Jeong, YunWon
718e74d5d4 Update scripts/update_lib/cmd_quick.py 2026-02-13 15:20:19 +09:00
Jeong, YunWon
190cecd7c8 [update_lib] _pyio deps 2026-02-13 14:00:00 +09:00
Jeong, YunWon
c974b77127 [update_lib] fix hard_deps resolution and fix commit not to miss test data (#7058)
* [update_lib] fix hard_dep not to unclude other tests

* [update_lib] commit test data
2026-02-09 14:41:15 +09:00
Jeong, YunWon
d2a9937ad6 [update_lib] todo shows last updated date (#7053) 2026-02-09 09:39:01 +09:00
Jeong, YunWon
648223ade2 non-code migration 2026-02-05 00:21:34 +09:00
Jeong, YunWon
2f034130b7 [update_lib] auto-mark original contents recovery (#6960) 2026-02-02 16:03:26 +09:00
Jeong, YunWon
babc3c634f Make auto-mark output deterministic and fix blank line leak (#6957)
* Make auto-mark output deterministic and fix blank line leak

Sort set iteration in build_patches and dict iteration in
_iter_patch_lines Phase 2 so expectedFailure markers are
always added in alphabetical order.

Include preceding blank line in _method_removal_range so
removing a super-call override doesn't leave behind the
blank line that was added with the method.

* deps code

* auto-mark handles crash better

* Auto-format: ruff format

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-02-02 13:53:01 +09:00
Jeong, YunWon
68ebb61f7b fix: Include hard_deps in git commit (#6940)
Bug: update_lib quick ast was not committing _ast_unparse.py
because git_commit() only added lib_path and test_paths,
but not hard_deps.

Fixed by:
- Add hard_deps parameter to git_commit()
- Collect hard_deps from DEPENDENCIES in main()
- Add hard_deps to paths_to_add in git_commit()
2026-02-02 00:47:29 +09:00
Jeong, YunWon
153b0da60d automark add reasons to reason-less expectedFailures and handles parent better (#6936) 2026-02-01 22:58:07 +09:00
Jeong, YunWon
0546bb0410 more deps grouping (#6931) 2026-02-01 17:01:56 +09:00
ShaharNaveh
07ba834381 Update os constant to libc 0.2.180 & target cpython 3.14 2026-02-01 08:48:20 +09:00
github-actions[bot]
9338509a71 Auto-format: ruff format 2026-01-30 20:50:37 +00:00
Jeong, YunWon
922b644116 [update_lib] fix async detection 2026-01-31 00:59:29 +09:00
Jeong, YunWon
528d6573e0 [update_lib] Fix async func auto-mark (#6903) 2026-01-31 00:00:41 +09:00
Jeong, YunWon
d257d95362 set encodings for opcode_metadata scripts 2026-01-28 16:56:47 +09:00
Jeong, YunWon
6a3643cdde Improve deps output (#6874)
* Improve deps output: [x]/[ ] sync status, TODO counts

- Replace ambiguous [+] with [x] (synced) / [ ] (not synced)
- Add (TODO: n) suffix for test files with expectedFailure/skip markers

* Refactor update_lib: extract shared utilities
2026-01-26 18:35:35 +09:00
Jeong, YunWon
9071147db5 [update_lib] Apply test grouping to deps (#6867)
* quick resolve test path too

* Fix migrate resolve

* Remove test/ from deps data

* resolve test to lib
2026-01-24 18:40:57 +02:00
Jeong, YunWon
f35791ec64 Fix update_lib tests (#6866) 2026-01-24 22:47:03 +09:00
Jeong, YunWon
0cc8f63849 [update_lib] deps grouping (#6854) 2026-01-24 21:29:11 +09:00
Jeong, YunWon
691d2816f9 update_lib todo also shows test todo (#6859)
also tracking untracked files considered in  #6775
2026-01-24 16:04:36 +09:00
Noa
f842fbe25d Use std::fmt::from_fn (#6850)
New in 1.93 - replaces our ad-hoc equivalent types.
2026-01-24 09:21:45 +09:00
Jeong, YunWon
180c68e76b improve deps CI formatting and name resolution (#6847)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Improvements**
* Enhanced test module dependency tracking in automated checks to
properly identify and report test-related dependencies.
* Improved dependency resolution logic with better module name mapping
and deduplication for clearer dependency reporting.
* Refined formatting of dependency information and todo items for
improved readability.
* Dependency review comments now automatically refresh with the latest
information when updated.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-01-24 00:32:32 +09:00
Lee Dogeon
9b56aa5b60 [update_lib] show dependent tests in deps subcommand (#6828)
---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jeong YunWon <jeong@youknowone.org>
2026-01-23 08:13:16 +09:00
fanninpm
122fac9c53 Optimize CI cache usage (#6707)
* Make whats_left.py compile with same settings as before

* Add --no-default-features to scripts/whats_left.py

* Build RustPython with no default features

* Retrigger CI
2026-01-21 20:17:42 +09:00
Jeong, YunWon
d690b2d65e [update_lib] todo subcommand (#6823)
* [update_lib] todo

* better CI comment
2026-01-21 09:00:45 +09:00
Jeong, YunWon
58c804309b [update_lib] show deps (#6821)
* show_deps

* soft deps tree

* show deps CI
2026-01-21 01:25:30 +09:00
Jeong, YunWon
5ef8d48e5a update_lib hard dependency resolver (#6817)
* update_lib __init__ handling

* dependency

* Auto-format: ruff format

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-01-20 23:52:28 +09:00
github-actions[bot]
f1fd1e9121 Auto-format: ruff format 2026-01-20 11:38:26 +09:00
Jeong, YunWon
41cdc5cd5b BooleanOptionalAction 2026-01-20 11:38:26 +09:00
Lee Dogeon
876d3f5e80 Correct testcase 2026-01-20 11:38:26 +09:00