Compare commits

...

18 Commits

Author SHA1 Message Date
Jeong, YunWon
18657330c9 Drop old PyObjectRef outside type lock to prevent deadlock
Dropping values inside with_type_lock can trigger weakref callbacks,
which may access attributes (LOAD_ATTR specialization) and re-acquire
the non-reentrant type mutex, causing deadlock.

Return old values from lock closures so they drop after lock release.
2026-03-21 00:39:45 +09:00
Jeong, YunWon
ea2d66e799 type lock 2026-03-20 22:47:55 +09:00
Jeong, YunWon
38de7462c0 Fix Constants newtype usage in init_cleanup_code 2026-03-20 22:47:55 +09:00
Jeong, YunWon
cb2db07463 Extract datastack_frame_size_bytes_for_code, skip monitoring for init_cleanup frames, guard trace dispatch
- Extract datastack_frame_size_bytes_for_code as free function, use it
  to compute init_cleanup stack bytes instead of hardcoded constant
- Add monitoring_disabled_for_code to skip instrumentation for
  synthetic init_cleanup code object in RESUME and execute_instrumented
- Add is_trace_event guard so profile-only events skip trace_func dispatch
- Reformat core.rs (rustfmt)
2026-03-20 22:47:55 +09:00
Jeong, YunWon
76e6ece941 address review: check datastack space for extra_bytes, require CO_OPTIMIZED in vectorcall fast path 2026-03-20 22:47:55 +09:00
Jeong, YunWon
fb0dfa102c address review: invalidate init cache on type modification, add cspell words 2026-03-20 22:47:55 +09:00
Jeong, YunWon
9df4787aed Align call-init frame flow and spec cache atomic ordering 2026-03-20 22:47:55 +09:00
Jeong, YunWon
e19335e8f2 Tighten CALL_ALLOC_AND_ENTER_INIT stack-space guard 2026-03-20 22:47:55 +09:00
Jeong, YunWon
b3daabf169 Align type _spec_cache and latin1 singleton string paths 2026-03-20 22:47:55 +09:00
Jeong, YunWon
471fe551fa Align BINARY_OP_EXTEND with CPython descriptor cache model 2026-03-20 22:47:55 +09:00
Lee Dogeon
38f742aa8d Implement Hashable for PyBoundMethod (#7474) 2026-03-20 13:15:40 +00:00
김은빈
1b9dc4ef14 Detect list mutation during sort even when list length is unchanged (#7432)
* Detect list mutation during sort even when list length is unchanged

* Use mutation counter instead of capacity check for sort mutation detection

The capacity heuristic missed mutations when `clear()` reset capacity to
0 via `mem::take`. An AtomicU32 counter on PyList, incremented in
`borrow_vec_mut()`, reliably detects all mutations during sort.

* Hold write guard during sort mutation counter reads

* Fix list mutation counter race in `borrow_vec_mut`
2026-03-20 21:40:54 +09:00
Shahar Naveh
4141e74e14 Add zizmor CI (#7463) 2026-03-20 21:37:12 +09:00
Shahar Naveh
2ef77f82e1 Run CI tests with max processes (#7254)
* Run CI witb max cores

* Step for flaky MP tests

* Add `test_class` to flkay mp tests
2026-03-20 20:35:18 +09:00
Jeong, YunWon
8be5230a9b Fix allow_threads and EINTR handling (#7457)
* Fix allow_threads and EINTR handling

- Wrap Windows SemLock acquire wait with allow_threads
- Retry nanosleep on EINTR with remaining time instead of
  returning early
- Remove expectedFailure for test_sleep in _test_eintr.py

* Remove expectedFailureIfWindows for testHashComparisonOfMethods
2026-03-20 19:59:13 +09:00
dependabot[bot]
247044a805 Bump which from 8.0.0 to 8.0.2 (#7469)
Bumps [which](https://github.com/harryfei/which-rs) from 8.0.0 to 8.0.2.
- [Release notes](https://github.com/harryfei/which-rs/releases)
- [Changelog](https://github.com/harryfei/which-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/harryfei/which-rs/compare/8.0.0...8.0.2)

---
updated-dependencies:
- dependency-name: which
  dependency-version: 8.0.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-20 12:43:07 +09:00
Shahar Naveh
68cf736a9f Add cargo shear CI step (#7461)
* Add `cargo shear` CI step

* Remove some deps

* Move `ruff_python_parser` to dev deps

* Remove `thread_local` crate

* Cleanup `num-integer` usage

* Remove parser fro. wasm

* Remove windows-sys from venv launcher

* Update lock

* Resolve more

* Fix wasm

* Update lock
2026-03-20 12:42:51 +09:00
Shahar Naveh
53fa525fc9 Resume opcode to hold ResumeType (#7465) 2026-03-19 14:24:07 +00:00
26 changed files with 527 additions and 420 deletions

View File

@@ -61,6 +61,9 @@
"dedents",
"deduped",
"deoptimize",
"downcastable",
"downcasted",
"dumpable",
"emscripten",
"excs",
"interps",

View File

@@ -141,7 +141,12 @@ jobs:
- name: Install dependencies
uses: ./.github/actions/install-linux-deps
with: ${{ matrix.dependencies || fromJSON('{}') }}
# zizmor has an issue with dynamic `with`
# with: ${{ matrix.dependencies || fromJSON('{}') }}
with:
gcc-multilib: ${{ matrix.dependencies.gcc-multilib || false }}
musl-tools: ${{ matrix.dependencies.musl-tools || false }}
gcc-aarch64-linux-gnu: ${{ matrix.dependencies.gcc-aarch64-linux-gnu || false }}
- uses: dtolnay/rust-toolchain@stable
with:
@@ -180,76 +185,13 @@ jobs:
if: ${{ !contains(github.event.pull_request.labels.*.name, 'skip:ci') }}
env:
RUST_BACKTRACE: full
# PLATFORM_INDEPENDENT_TESTS are tests that do not depend on the underlying OS.
# They are currently only run on Linux to speed up the CI.
PLATFORM_INDEPENDENT_TESTS: >-
test__colorize
test_array
test_asyncgen
test_binop
test_bisect
test_bool
test_bytes
test_call
test_cmath
test_collections
test_complex
test_contains
test_copy
test_dataclasses
test_decimal
test_decorators
test_defaultdict
test_deque
test_dict
test_dictcomps
test_dictviews
test_dis
test_enumerate
test_exception_variations
test_float
test_fractions
test_genericalias
test_genericclass
test_grammar
test_range
test_index
test_int
test_int_literal
test_isinstance
test_iter
test_iterlen
test_itertools
test_json
test_keyword
test_keywordonlyarg
test_list
test_long
test_longexp
test_operator
test_ordered_dict
test_pep646_syntax
test_pow
test_raise
test_richcmp
test_scope
test_set
test_slice
test_sort
test_string
test_string_literals
test_strtod
test_structseq
test_subclassinit
test_super
test_syntax
test_tstring
test_tuple
test_unary
test_unpack
test_unpack_ex
test_weakref
test_yield_from
# Tests that can be flaky when running with multiple processes `-j 2`. We will use `-j 1` for these.
FLAKY_MP_TESTS: >-
test_class
test_eintr
test_multiprocessing_fork
test_multiprocessing_forkserver
test_multiprocessing_spawn
name: Run snippets and cpython tests
runs-on: ${{ matrix.os }}
strategy:
@@ -304,17 +246,23 @@ jobs:
run: python -m pip install -r requirements.txt && pytest -v
working-directory: ./extra_tests
- name: run cpython platform-independent tests
if: runner.os == 'Linux'
- name: Detect available cores
id: cores
shell: bash
run: |
target/release/rustpython -m test -j 1 -u all --slowest --fail-env-changed --timeout 600 -v ${{ env.PLATFORM_INDEPENDENT_TESTS }}
timeout-minutes: 45
cores=$(python -c 'print(__import__("os").process_cpu_count())')
echo "cores=${cores}" >> $GITHUB_OUTPUT
- name: Run CPython tests
run: |
target/release/rustpython -m test -j ${{ steps.cores.outputs.cores }} ${{ join(matrix.extra_test_args, ' ') }} --slowest --fail-env-changed --timeout 600 -v -x ${{ env.FLAKY_MP_TESTS }} ${{ join(matrix.skips, ' ') }}
timeout-minutes: ${{ matrix.timeout }}
env:
RUSTPYTHON_SKIP_ENV_POLLUTERS: true
- name: run cpython platform-dependent tests
- name: Run flaky MP CPython tests
run: |
target/release/rustpython -m test -j 1 ${{ join(matrix.extra_test_args, ' ') }} --slowest --fail-env-changed --timeout 600 -v -x ${{ env.PLATFORM_INDEPENDENT_TESTS }} ${{ join(matrix.skips, ' ') }}
target/release/rustpython -m test -j 1 ${{ join(matrix.extra_test_args, ' ') }} --slowest --fail-env-changed --timeout 600 -v ${{ env.FLAKY_MP_TESTS }}
timeout-minutes: ${{ matrix.timeout }}
env:
RUSTPYTHON_SKIP_ENV_POLLUTERS: true
@@ -557,3 +505,29 @@ jobs:
run: wasmer run --dir $(pwd) target/wasm32-wasip1/release/rustpython.wasm -- "$(pwd)/extra_tests/snippets/stdlib_random.py"
- name: run cpython unittest
run: wasmer run --dir $(pwd) target/wasm32-wasip1/release/rustpython.wasm -- "$(pwd)/Lib/test/test_int.py"
cargo-shear:
name: cargo shear
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: cargo-bins/cargo-binstall@1800853f2578f8c34492ec76154caef8e163fbca # v1.17.7
- run: cargo binstall --no-confirm cargo-shear
- run: cargo shear
security-lint:
runs-on: ubuntu-latest
permissions:
security-events: write
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- name: Run zizmor
uses: zizmorcore/zizmor-action@71321a20a9ded102f6e9ce5718a2fcec2c4f70d8 # v0.5.2

View File

@@ -12,7 +12,7 @@ on:
name: Periodic checks/tasks
env:
CARGO_ARGS: --no-default-features --features stdlib,importlib,encodings,ssl-rustls,jit
CARGO_ARGS: --no-default-features --features stdlib,importlib,stdio,encodings,ssl-rustls,jit,host_env
PYTHON_VERSION: "3.14.3"
jobs:
@@ -35,7 +35,7 @@ jobs:
python-version: ${{ env.PYTHON_VERSION }}
- run: sudo apt-get update && sudo apt-get -y install lcov
- name: Run cargo-llvm-cov with Rust tests.
run: cargo llvm-cov --no-report --workspace --exclude rustpython_wasm --exclude rustpython-compiler-source --exclude rustpython-venvlauncher --verbose --no-default-features --features stdlib,importlib,encodings,ssl-rustls,jit
run: cargo llvm-cov --no-report --workspace --exclude rustpython_wasm --exclude rustpython-compiler-source --exclude rustpython-venvlauncher --verbose --no-default-features --features stdlib,importlib,stdio,encodings,ssl-rustls,jit,host_env
- name: Run cargo-llvm-cov with Python snippets.
run: python scripts/cargo-llvm-cov.py
continue-on-error: true
@@ -48,7 +48,7 @@ jobs:
if: ${{ github.event_name != 'pull_request' }}
uses: codecov/codecov-action@v5
with:
file: ./codecov.lcov
files: ./codecov.lcov
testdata:
name: Collect regression test data
@@ -170,12 +170,12 @@ jobs:
- name: restructure generated files
run: |
cd ./target/criterion/reports
find -type d -name cpython | xargs rm -rf
find -type d -name rustpython | xargs rm -rf
find -mindepth 2 -maxdepth 2 -name violin.svg | xargs rm -rf
find -type f -not -name violin.svg | xargs rm -rf
for file in $(find -type f -name violin.svg); do mv $file $(echo $file | sed -E "s_\./([^/]+)/([^/]+)/violin\.svg_./\1/\2.svg_"); done
find -mindepth 2 -maxdepth 2 -type d | xargs rm -rf
find . -type d -name cpython -print0 | xargs -0 rm -rf
find . -type d -name rustpython -print0 | xargs -0 rm -rf
find . -mindepth 2 -maxdepth 2 -name violin.svg -print0 | xargs -0 rm -rf
find . -type f -not -name violin.svg -print0 | xargs -0 rm -rf
find . -type f -name violin.svg -exec sh -c 'for file; do mv "$file" "$(echo "$file" | sed -E "s_\./([^/]+)/([^/]+)/violin\.svg_./\1/\2.svg_")"; done' _ {} +
find . -mindepth 2 -maxdepth 2 -type d -print0 | xargs -0 rm -rf
cd ..
mv reports/* .
rmdir reports

55
Cargo.lock generated
View File

@@ -113,7 +113,7 @@ version = "1.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc"
dependencies = [
"windows-sys 0.60.2",
"windows-sys 0.61.2",
]
[[package]]
@@ -124,7 +124,7 @@ checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d"
dependencies = [
"anstyle",
"once_cell_polyfill",
"windows-sys 0.60.2",
"windows-sys 0.61.2",
]
[[package]]
@@ -1140,12 +1140,6 @@ dependencies = [
"regex",
]
[[package]]
name = "env_home"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c7f84e12ccf0a7ddc17a6c41c93326024c42920d7ee630d04950e6926645c0fe"
[[package]]
name = "env_logger"
version = "0.11.9"
@@ -1172,7 +1166,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb"
dependencies = [
"libc",
"windows-sys 0.52.0",
"windows-sys 0.61.2",
]
[[package]]
@@ -1670,7 +1664,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8cfc352a66ba903c23239ef51e809508b6fc2b0f90e3476ac7a9ff47e863ae95"
dependencies = [
"scopeguard",
"windows-sys 0.59.0",
"windows-sys 0.61.2",
]
[[package]]
@@ -2982,7 +2976,7 @@ dependencies = [
"errno",
"libc",
"linux-raw-sys",
"windows-sys 0.52.0",
"windows-sys 0.61.2",
]
[[package]]
@@ -3047,7 +3041,7 @@ dependencies = [
"security-framework",
"security-framework-sys",
"webpki-root-certs",
"windows-sys 0.52.0",
"windows-sys 0.61.2",
]
[[package]]
@@ -3299,7 +3293,6 @@ dependencies = [
"mt19937",
"nix 0.30.1",
"num-complex",
"num-integer",
"num-traits",
"num_enum",
"oid-registry",
@@ -3342,9 +3335,7 @@ dependencies = [
"unic-ucd-age",
"unic-ucd-bidi",
"unic-ucd-category",
"unic-ucd-ident",
"unicode-bidi-mirroring",
"unicode-casing",
"unicode_names2 2.0.0",
"uuid",
"webpki-roots",
@@ -3358,9 +3349,6 @@ dependencies = [
[[package]]
name = "rustpython-venvlauncher"
version = "0.5.0"
dependencies = [
"windows-sys 0.61.2",
]
[[package]]
name = "rustpython-vm"
@@ -3424,7 +3412,6 @@ dependencies = [
"strum",
"strum_macros",
"thiserror 2.0.18",
"thread_local",
"timsort",
"uname",
"unic-ucd-bidi",
@@ -3453,7 +3440,6 @@ version = "0.5.0"
dependencies = [
"console_error_panic_hook",
"js-sys",
"ruff_python_parser",
"rustpython-common",
"rustpython-pylib",
"rustpython-stdlib",
@@ -3747,7 +3733,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3a766e1110788c36f4fa1c2b71b387a7815aa65f88ce0229841826633d93723e"
dependencies = [
"libc",
"windows-sys 0.60.2",
"windows-sys 0.61.2",
]
[[package]]
@@ -3875,7 +3861,7 @@ dependencies = [
"getrandom 0.3.4",
"once_cell",
"rustix",
"windows-sys 0.52.0",
"windows-sys 0.61.2",
]
[[package]]
@@ -3944,15 +3930,6 @@ dependencies = [
"winapi",
]
[[package]]
name = "thread_local"
version = "1.1.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f60246a4944f24f6e018aa17cdeffb7818b76356965d03b07d6a9886e8962185"
dependencies = [
"cfg-if",
]
[[package]]
name = "time"
version = "0.3.47"
@@ -4473,13 +4450,11 @@ dependencies = [
[[package]]
name = "which"
version = "8.0.0"
version = "8.0.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d3fabb953106c3c8eea8306e4393700d7657561cb43122571b172bbfb7c7ba1d"
checksum = "81995fafaaaf6ae47a7d0cc83c67caf92aeb7e5331650ae6ff856f7c0c60c459"
dependencies = [
"env_home",
"rustix",
"winsafe",
"libc",
]
[[package]]
@@ -4520,7 +4495,7 @@ version = "0.1.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22"
dependencies = [
"windows-sys 0.52.0",
"windows-sys 0.61.2",
]
[[package]]
@@ -4835,12 +4810,6 @@ dependencies = [
"version_check",
]
[[package]]
name = "winsafe"
version = "0.0.19"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d135d17ab770252ad95e9a872d365cf3090e3be864a34ab46f48555993efc904"
[[package]]
name = "wit-bindgen"
version = "0.51.0"

View File

@@ -35,7 +35,6 @@ rustpython-compiler = { workspace = true }
rustpython-pylib = { workspace = true, optional = true }
rustpython-stdlib = { workspace = true, optional = true, features = ["compiler"] }
rustpython-vm = { workspace = true, features = ["compiler", "gc"] }
ruff_python_parser = { workspace = true }
cfg-if = { workspace = true }
log = { workspace = true }
@@ -56,6 +55,7 @@ rustyline = { workspace = true }
criterion = { workspace = true }
pyo3 = { version = "0.28.2", features = ["auto-initialize"] }
rustpython-stdlib = { workspace = true }
ruff_python_parser = { workspace = true }
[[bench]]
name = "execution"
@@ -169,7 +169,6 @@ ascii = "1.1"
bitflags = "2.11.0"
bitflagset = "0.0.3"
bstr = "1"
bytes = "1.11.1"
cfg-if = "1.0"
chrono = { version = "0.4.44", default-features = false, features = ["clock", "oldtime", "std"] }
constant_time_eq = "0.4"
@@ -216,7 +215,6 @@ strum = "0.27"
strum_macros = "0.28"
syn = "2"
thiserror = "2.0"
thread_local = "1.1.9"
unicode-casing = "0.1.1"
unic-char-property = "0.9.0"
unic-normal = "0.9.0"

View File

@@ -392,7 +392,6 @@ class SocketEINTRTest(EINTRBaseTest):
class TimeEINTRTest(EINTRBaseTest):
""" EINTR tests for the time module. """
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_sleep(self):
t0 = time.monotonic()
time.sleep(self.sleep_time)

View File

@@ -1,5 +1,6 @@
"Test the functionality of Python classes implementing operators."
import sys
import unittest
from test import support
from test.support import cpython_only, import_helper, script_helper
@@ -614,7 +615,7 @@ class ClassTests(unittest.TestCase):
with self.assertRaises(TypeError):
a >= b
@unittest.expectedFailureIfWindows("TODO: RUSTPYTHON; AssertionError: 1543448294720 != 1543448295392")
@unittest.skipIf(sys.platform == "win32", "TODO: RUSTPYTHON; flaky on Windows")
def testHashComparisonOfMethods(self):
# Test comparison and hash of methods
class A:

View File

@@ -153,7 +153,6 @@ class TestBase(unittest.TestCase):
class TestBugs(unittest.TestCase):
@unittest.skip("TODO: RUSTPYTHON; figure out how to detect sort mutation that doesn't change list length")
def test_bug453523(self):
# bug 453523 -- list.sort() crasher.
# If this fails, the most likely outcome is a core dump.
@@ -170,7 +169,6 @@ class TestBugs(unittest.TestCase):
L = [C() for i in range(50)]
self.assertRaises(ValueError, L.sort)
@unittest.expectedFailure # TODO: RUSTPYTHON; figure out how to detect sort mutation that doesn't change list length
def test_undetected_mutation(self):
# Python 2.4a1 did not always detect mutation
memorywaster = []

View File

@@ -6881,9 +6881,9 @@ impl Compiler {
self,
Instruction::Resume {
context: if is_await {
u32::from(bytecode::ResumeType::AfterAwait)
bytecode::ResumeType::AfterAwait
} else {
u32::from(bytecode::ResumeType::AfterYieldFrom)
bytecode::ResumeType::AfterYieldFrom
}
}
);
@@ -7055,7 +7055,7 @@ impl Compiler {
emit!(
self,
Instruction::Resume {
context: u32::from(bytecode::ResumeType::AfterYield)
context: bytecode::ResumeType::AfterYield
}
);
}
@@ -7277,7 +7277,7 @@ impl Compiler {
emit!(
compiler,
Instruction::Resume {
context: u32::from(bytecode::ResumeType::AfterYield)
context: bytecode::ResumeType::AfterYield
}
);
emit!(compiler, Instruction::PopTop);

View File

@@ -304,7 +304,7 @@ pub enum Instruction {
} = 120,
// CPython 3.14 RESUME (128)
Resume {
context: Arg<u32>,
context: Arg<oparg::ResumeType>,
} = 128,
// CPython 3.14 specialized opcodes (129-211)
BinaryOpAddFloat = 129, // Placeholder

View File

@@ -276,16 +276,47 @@ impl fmt::Display for ConvertValueOparg {
}
}
oparg_enum!(
/// Resume type for the RESUME instruction
#[derive(Copy, Clone, Debug, Hash, PartialEq, Eq)]
pub enum ResumeType {
AtFuncStart = 0,
AfterYield = 1,
AfterYieldFrom = 2,
AfterAwait = 3,
/// Resume type for the RESUME instruction
#[derive(Copy, Clone, Debug, Hash, PartialEq, Eq)]
pub enum ResumeType {
AtFuncStart,
AfterYield,
AfterYieldFrom,
AfterAwait,
Other(u32),
}
impl From<u32> for ResumeType {
fn from(value: u32) -> Self {
match value {
0 => Self::AtFuncStart,
1 => Self::AfterYield,
2 => Self::AfterYieldFrom,
3 => Self::AfterAwait,
_ => Self::Other(value),
}
}
);
}
impl From<ResumeType> for u32 {
fn from(typ: ResumeType) -> Self {
match typ {
ResumeType::AtFuncStart => 0,
ResumeType::AfterYield => 1,
ResumeType::AfterYieldFrom => 2,
ResumeType::AfterAwait => 3,
ResumeType::Other(v) => v,
}
}
}
impl core::fmt::Display for ResumeType {
fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
u32::from(*self).fmt(f)
}
}
impl OpArgType for ResumeType {}
pub type NameIdx = u32;
@@ -714,6 +745,7 @@ macro_rules! newtype_oparg {
self.0.fmt(f)
}
}
impl OpArgType for $name {}
}
}

View File

@@ -48,7 +48,6 @@ libc = { workspace = true }
nix = { workspace = true }
num-complex = { workspace = true }
malachite-bigint = { workspace = true }
num-integer = { workspace = true }
num-traits = { workspace = true }
num_enum = { workspace = true }
parking_lot = { workspace = true }
@@ -78,16 +77,12 @@ constant_time_eq = { workspace = true }
## unicode stuff
unicode_names2 = { workspace = true }
# TODO: use unic for this; needed for title case:
# https://github.com/RustPython/RustPython/pull/832#discussion_r275428939
unicode-casing = { workspace = true }
# update version all at the same time
unic-char-property = { workspace = true }
unic-normal = { workspace = true }
unic-ucd-bidi = { workspace = true }
unic-ucd-category = { workspace = true }
unic-ucd-age = { workspace = true }
unic-ucd-ident = { workspace = true }
ucd = "0.1.1"
unicode-bidi-mirroring = { workspace = true }

View File

@@ -193,7 +193,8 @@ mod _multiprocessing {
remaining.min(poll_ms)
};
let res = unsafe { WaitForSingleObjectEx(self.handle.as_raw(), wait_ms, 0) };
let handle = self.handle.as_raw();
let res = vm.allow_threads(|| unsafe { WaitForSingleObjectEx(handle, wait_ms, 0) });
match res {
WAIT_OBJECT_0 => {

View File

@@ -13,15 +13,5 @@ license.workspace = true
name = "venvlaunchert"
path = "src/main.rs"
[target.'cfg(windows)'.dependencies]
windows-sys = { workspace = true, features = [
"Win32_Foundation",
"Win32_System_Threading",
"Win32_System_Environment",
"Win32_Storage_FileSystem",
"Win32_System_Console",
"Win32_Security",
] }
[lints]
workspace = true

View File

@@ -72,7 +72,6 @@ static_assertions = { workspace = true }
strum = { workspace = true }
strum_macros = { workspace = true }
thiserror = { workspace = true }
thread_local = { workspace = true }
memchr = { workspace = true }
caseless = "0.2.2"

View File

@@ -5,6 +5,7 @@ use super::{
PyAsyncGen, PyCode, PyCoroutine, PyDictRef, PyGenerator, PyModule, PyStr, PyStrRef, PyTuple,
PyTupleRef, PyType,
};
use crate::common::hash::PyHash;
use crate::common::lock::PyMutex;
use crate::function::ArgMapping;
use crate::object::{PyAtomicRef, Traverse, TraverseFn};
@@ -17,7 +18,8 @@ use crate::{
function::{FuncArgs, OptionalArg, PyComparisonValue, PySetterValue},
scope::Scope,
types::{
Callable, Comparable, Constructor, GetAttr, GetDescriptor, PyComparisonOp, Representable,
Callable, Comparable, Constructor, GetAttr, GetDescriptor, Hashable, PyComparisonOp,
Representable,
},
};
use core::sync::atomic::{AtomicU32, Ordering::Relaxed};
@@ -1193,6 +1195,14 @@ impl Comparable for PyBoundMethod {
}
}
impl Hashable for PyBoundMethod {
fn hash(zelf: &Py<Self>, vm: &VirtualMachine) -> PyResult<PyHash> {
let self_hash = crate::common::hash::hash_object_id_raw(zelf.object.get_id());
let func_hash = zelf.function.hash(vm)?;
Ok(crate::common::hash::fix_sentinel(self_hash ^ func_hash))
}
}
impl GetAttr for PyBoundMethod {
fn getattro(zelf: &Py<Self>, name: &Py<PyStr>, vm: &VirtualMachine) -> PyResult {
let class_attr = vm
@@ -1248,7 +1258,7 @@ impl PyBoundMethod {
}
#[pyclass(
with(Callable, Comparable, GetAttr, Constructor, Representable),
with(Callable, Comparable, Hashable, GetAttr, Constructor, Representable),
flags(IMMUTABLETYPE, HAS_WEAKREF)
)]
impl PyBoundMethod {

View File

@@ -24,7 +24,7 @@ use core::cell::Cell;
use core::ops::{Neg, Not};
use core::ptr::NonNull;
use malachite_bigint::{BigInt, Sign};
use num_integer::Integer;
use num_integer::{ExtendedGcd, Integer};
use num_traits::{One, Pow, PrimInt, Signed, ToPrimitive, Zero};
#[pyclass(module = false, name = "int")]
@@ -414,7 +414,6 @@ impl PyInt {
if a.is_negative() { a + n } else { a }
}
fn inverse(a: BigInt, n: &BigInt) -> Option<BigInt> {
use num_integer::*;
let ExtendedGcd { gcd, x: c, .. } = a.extended_gcd(n);
if gcd.is_one() {
Some(normalize(c, n))

View File

@@ -30,11 +30,13 @@ use alloc::fmt;
use core::cell::Cell;
use core::ops::DerefMut;
use core::ptr::NonNull;
use core::sync::atomic::{AtomicU32, Ordering};
#[pyclass(module = false, name = "list", unhashable = true, traverse = "manual")]
#[derive(Default)]
pub struct PyList {
elements: PyRwLock<Vec<PyObjectRef>>,
mutation_counter: AtomicU32,
}
impl fmt::Debug for PyList {
@@ -48,6 +50,7 @@ impl From<Vec<PyObjectRef>> for PyList {
fn from(elements: Vec<PyObjectRef>) -> Self {
Self {
elements: PyRwLock::new(elements),
mutation_counter: AtomicU32::new(0),
}
}
}
@@ -135,7 +138,9 @@ impl PyList {
}
pub fn borrow_vec_mut(&self) -> PyRwLockWriteGuard<'_, Vec<PyObjectRef>> {
self.elements.write()
let guard = self.elements.write();
self.mutation_counter.fetch_add(1, Ordering::Relaxed);
guard
}
fn repeat(&self, n: isize, vm: &VirtualMachine) -> PyResult<PyRef<Self>> {
@@ -391,12 +396,21 @@ impl PyList {
// replace list contents with [] for duration of sort.
// this prevents keyfunc from messing with the list and makes it easy to
// check if it tries to append elements to it.
let mut elements = core::mem::take(self.borrow_vec_mut().deref_mut());
let (mut elements, version_before) = {
let mut guard = self.elements.write();
let version_before = self.mutation_counter.load(Ordering::Relaxed);
(core::mem::take(guard.deref_mut()), version_before)
};
let res = do_sort(vm, &mut elements, options.key, options.reverse);
core::mem::swap(self.borrow_vec_mut().deref_mut(), &mut elements);
let mutated = {
let mut guard = self.elements.write();
let mutated = self.mutation_counter.load(Ordering::Relaxed) != version_before;
core::mem::swap(guard.deref_mut(), &mut elements);
mutated
};
res?;
if !elements.is_empty() {
if mutated {
return Err(vm.new_value_error("list modified during sort"));
}

View File

@@ -276,7 +276,6 @@ pub struct TypeSpecializationCache {
pub init: PyAtomicRef<Option<PyFunction>>,
pub getitem: PyAtomicRef<Option<PyFunction>>,
pub getitem_version: AtomicU32,
// Serialize cache writes/invalidation similar to CPython's BEGIN_TYPE_LOCK.
write_lock: PyMutex<()>,
retired: PyRwLock<Vec<PyObjectRef>>,
}
@@ -302,9 +301,6 @@ impl TypeSpecializationCache {
#[inline]
fn swap_init(&self, new_init: Option<PyRef<PyFunction>>, vm: Option<&VirtualMachine>) {
if let Some(vm) = vm {
// Keep replaced refs alive for the currently executing frame, matching
// CPython-style "old pointer remains valid during ongoing execution"
// without accumulating global retired refs.
self.init.swap_to_temporary_refs(new_init, vm);
return;
}
@@ -329,8 +325,6 @@ impl TypeSpecializationCache {
#[inline]
fn invalidate_for_type_modified(&self) {
let _guard = self.write_lock.lock();
// _spec_cache contract: type modification invalidates all cached
// specialization functions.
self.swap_init(None, None);
self.swap_getitem(None, None);
}
@@ -457,9 +451,15 @@ fn is_subtype_with_mro(a_mro: &[PyTypeRef], a: &Py<PyType>, b: &Py<PyType>) -> b
}
impl PyType {
#[inline]
fn with_type_lock<R>(vm: &VirtualMachine, f: impl FnOnce() -> R) -> R {
let _guard = vm.state.type_mutex.lock();
f()
}
/// Assign a fresh version tag. Returns 0 if the version counter has been
/// exhausted, in which case no new cache entries can be created.
pub fn assign_version_tag(&self) -> u32 {
fn assign_version_tag_inner(&self) -> u32 {
let v = self.tp_version_tag.load(Ordering::Acquire);
if v != 0 {
return v;
@@ -467,7 +467,7 @@ impl PyType {
// Assign versions to all direct bases first (MRO invariant).
for base in self.bases.read().iter() {
if base.assign_version_tag() == 0 {
if base.assign_version_tag_inner() == 0 {
return 0;
}
}
@@ -487,8 +487,23 @@ impl PyType {
}
}
pub fn assign_version_tag(&self) -> u32 {
self.assign_version_tag_inner()
}
pub(crate) fn version_for_specialization(&self, vm: &VirtualMachine) -> u32 {
Self::with_type_lock(vm, || {
let version = self.tp_version_tag.load(Ordering::Acquire);
if version == 0 {
self.assign_version_tag_inner()
} else {
version
}
})
}
/// Invalidate this type's version tag and cascade to all subclasses.
pub fn modified(&self) {
fn modified_inner(&self) {
if let Some(ext) = self.heaptype_ext.as_ref() {
ext.specialization_cache.invalidate_for_type_modified();
}
@@ -505,11 +520,15 @@ impl PyType {
let subclasses = self.subclasses.read();
for weak_ref in subclasses.iter() {
if let Some(sub) = weak_ref.upgrade() {
sub.downcast_ref::<PyType>().unwrap().modified();
sub.downcast_ref::<PyType>().unwrap().modified_inner();
}
}
}
pub fn modified(&self) {
self.modified_inner();
}
pub fn new_simple_heap(
name: &str,
base: &Py<PyType>,
@@ -898,6 +917,74 @@ impl PyType {
self.find_name_in_mro(attr_name)
}
/// CPython-style `_PyType_LookupRefAndVersion` equivalent for interned names.
/// Returns the observed lookup result and the type version used for the lookup.
pub(crate) fn lookup_ref_and_version_interned(
&self,
name: &'static PyStrInterned,
vm: &VirtualMachine,
) -> (Option<PyObjectRef>, u32) {
let version = self.tp_version_tag.load(Ordering::Acquire);
if version != 0 {
let idx = type_cache_hash(version, name);
let entry = &TYPE_CACHE[idx];
let name_ptr = name as *const _ as *mut _;
loop {
let seq1 = entry.begin_read();
let entry_version = entry.version.load(Ordering::Acquire);
let type_version = self.tp_version_tag.load(Ordering::Acquire);
if entry_version != type_version
|| !core::ptr::eq(entry.name.load(Ordering::Relaxed), name_ptr)
{
break;
}
let ptr = entry.value.load(Ordering::Acquire);
if ptr.is_null() {
if entry.end_read(seq1) {
return (None, entry_version);
}
continue;
}
if let Some(cloned) = unsafe { PyObject::try_to_owned_from_ptr(ptr) } {
let same_ptr = core::ptr::eq(entry.value.load(Ordering::Relaxed), ptr);
if same_ptr && entry.end_read(seq1) {
return (Some(cloned), entry_version);
}
drop(cloned);
continue;
}
break;
}
}
Self::with_type_lock(vm, || {
let assigned = if self.tp_version_tag.load(Ordering::Acquire) == 0 {
self.assign_version_tag_inner()
} else {
self.tp_version_tag.load(Ordering::Acquire)
};
let result = self.find_name_in_mro_uncached(name);
if assigned != 0
&& !TYPE_CACHE_CLEARING.load(Ordering::Acquire)
&& self.tp_version_tag.load(Ordering::Acquire) == assigned
{
let idx = type_cache_hash(assigned, name);
let entry = &TYPE_CACHE[idx];
let name_ptr = name as *const _ as *mut _;
entry.begin_write();
entry.version.store(0, Ordering::Release);
let new_ptr = result.as_ref().map_or(core::ptr::null_mut(), |found| {
&**found as *const PyObject as *mut _
});
entry.value.store(new_ptr, Ordering::Relaxed);
entry.name.store(name_ptr, Ordering::Relaxed);
entry.version.store(assigned, Ordering::Release);
entry.end_write();
}
(result, assigned)
})
}
/// Cache __init__ for CALL_ALLOC_AND_ENTER_INIT specialization.
/// The cache is valid only when guarded by the type version check.
pub(crate) fn cache_init_for_specialization(
@@ -912,15 +999,17 @@ impl PyType {
if tp_version == 0 {
return false;
}
if self.tp_version_tag.load(Ordering::Acquire) != tp_version {
return false;
}
let _guard = ext.specialization_cache.write_lock.lock();
if self.tp_version_tag.load(Ordering::Acquire) != tp_version {
return false;
}
ext.specialization_cache.swap_init(Some(init), Some(vm));
true
Self::with_type_lock(vm, || {
if self.tp_version_tag.load(Ordering::Acquire) != tp_version {
return false;
}
let _guard = ext.specialization_cache.write_lock.lock();
if self.tp_version_tag.load(Ordering::Acquire) != tp_version {
return false;
}
ext.specialization_cache.swap_init(Some(init), Some(vm));
true
})
}
/// Read cached __init__ for CALL_ALLOC_AND_ENTER_INIT specialization.
@@ -954,26 +1043,27 @@ impl PyType {
if tp_version == 0 {
return false;
}
let _guard = ext.specialization_cache.write_lock.lock();
if self.tp_version_tag.load(Ordering::Acquire) != tp_version {
return false;
}
let func_version = getitem.get_version_for_current_state();
if func_version == 0 {
return false;
}
ext.specialization_cache
.swap_getitem(Some(getitem), Some(vm));
ext.specialization_cache
.getitem_version
.store(func_version, Ordering::Relaxed);
true
Self::with_type_lock(vm, || {
let _guard = ext.specialization_cache.write_lock.lock();
if self.tp_version_tag.load(Ordering::Acquire) != tp_version {
return false;
}
let func_version = getitem.get_version_for_current_state();
if func_version == 0 {
return false;
}
ext.specialization_cache
.getitem_version
.store(func_version, Ordering::Release);
ext.specialization_cache
.swap_getitem(Some(getitem), Some(vm));
true
})
}
/// Read cached __getitem__ for BINARY_OP_SUBSCR_GETITEM specialization.
pub(crate) fn get_cached_getitem_for_specialization(&self) -> Option<(PyRef<PyFunction>, u32)> {
let ext = self.heaptype_ext.as_ref()?;
// Match CPython check order: pointer (Acquire) then function version.
let getitem = ext
.specialization_cache
.getitem
@@ -981,7 +1071,7 @@ impl PyType {
let cached_version = ext
.specialization_cache
.getitem_version
.load(Ordering::Relaxed);
.load(Ordering::Acquire);
if cached_version == 0 {
return None;
}
@@ -1334,38 +1424,41 @@ impl PyType {
// // TODO: how to uniquely identify the subclasses to remove?
// }
*zelf.bases.write() = bases;
// Recursively update the mros of this class and all subclasses
fn update_mro_recursively(cls: &PyType, vm: &VirtualMachine) -> PyResult<()> {
let mut mro =
PyType::resolve_mro(&cls.bases.read()).map_err(|msg| vm.new_type_error(msg))?;
// Preserve self (mro[0]) when updating MRO
mro.insert(0, cls.mro.read()[0].to_owned());
*cls.mro.write() = mro;
for subclass in cls.subclasses.write().iter() {
let subclass = subclass.upgrade().unwrap();
let subclass: &Py<PyType> = subclass.downcast_ref().unwrap();
update_mro_recursively(subclass, vm)?;
Self::with_type_lock(vm, || {
*zelf.bases.write() = bases;
// Recursively update the mros of this class and all subclasses
fn update_mro_recursively(cls: &PyType, vm: &VirtualMachine) -> PyResult<()> {
let mut mro =
PyType::resolve_mro(&cls.bases.read()).map_err(|msg| vm.new_type_error(msg))?;
// Preserve self (mro[0]) when updating MRO
mro.insert(0, cls.mro.read()[0].to_owned());
*cls.mro.write() = mro;
for subclass in cls.subclasses.write().iter() {
let subclass = subclass.upgrade().unwrap();
let subclass: &Py<PyType> = subclass.downcast_ref().unwrap();
update_mro_recursively(subclass, vm)?;
}
Ok(())
}
update_mro_recursively(zelf, vm)?;
// Invalidate inline caches
zelf.modified_inner();
// TODO: do any old slots need to be cleaned up first?
zelf.init_slots(&vm.ctx);
// Register this type as a subclass of its new bases
let weakref_type = super::PyWeak::static_type();
for base in zelf.bases.read().iter() {
base.subclasses.write().push(
zelf.as_object()
.downgrade_with_weakref_typ_opt(None, weakref_type.to_owned())
.unwrap(),
);
}
Ok(())
}
update_mro_recursively(zelf, vm)?;
// Invalidate inline caches
zelf.modified();
// TODO: do any old slots need to be cleaned up first?
zelf.init_slots(&vm.ctx);
// Register this type as a subclass of its new bases
let weakref_type = super::PyWeak::static_type();
for base in zelf.bases.read().iter() {
base.subclasses.write().push(
zelf.as_object()
.downgrade_with_weakref_typ_opt(None, weakref_type.to_owned())
.unwrap(),
);
}
})?;
Ok(())
}
@@ -1457,20 +1550,31 @@ impl PyType {
)));
}
let mut attrs = self.attributes.write();
// First try __annotate__, in case that's been set explicitly
if let Some(annotate) = attrs.get(identifier!(vm, __annotate__)).cloned() {
let annotate_key = identifier!(vm, __annotate__);
let annotate_func_key = identifier!(vm, __annotate_func__);
let attrs = self.attributes.read();
if let Some(annotate) = attrs.get(annotate_key).cloned() {
return Ok(annotate);
}
// Then try __annotate_func__
if let Some(annotate) = attrs.get(identifier!(vm, __annotate_func__)).cloned() {
// TODO: Apply descriptor tp_descr_get if needed
if let Some(annotate) = attrs.get(annotate_func_key).cloned() {
return Ok(annotate);
}
// Set __annotate_func__ = None and return None
drop(attrs);
let none = vm.ctx.none();
attrs.insert(identifier!(vm, __annotate_func__), none.clone());
Ok(none)
let (result, _prev) = Self::with_type_lock(vm, || {
let mut attrs = self.attributes.write();
if let Some(annotate) = attrs.get(annotate_key).cloned() {
return (annotate, None);
}
if let Some(annotate) = attrs.get(annotate_func_key).cloned() {
return (annotate, None);
}
self.modified_inner();
let prev = attrs.insert(annotate_func_key, none.clone());
(none, prev)
});
Ok(result)
}
#[pygetset(setter)]
@@ -1493,20 +1597,27 @@ impl PyType {
return Err(vm.new_type_error("__annotate__ must be callable or None"));
}
let mut attrs = self.attributes.write();
// Clear cached annotations only when setting to a new callable
if !vm.is_none(&value) {
attrs.swap_remove(identifier!(vm, __annotations_cache__));
}
attrs.insert(identifier!(vm, __annotate_func__), value.clone());
let _prev_values = Self::with_type_lock(vm, || {
self.modified_inner();
let mut attrs = self.attributes.write();
let removed = if !vm.is_none(&value) {
attrs.swap_remove(identifier!(vm, __annotations_cache__))
} else {
None
};
let prev = attrs.insert(identifier!(vm, __annotate_func__), value);
(removed, prev)
});
Ok(())
}
#[pygetset]
fn __annotations__(&self, vm: &VirtualMachine) -> PyResult<PyObjectRef> {
let annotations_key = identifier!(vm, __annotations__);
let annotations_cache_key = identifier!(vm, __annotations_cache__);
let attrs = self.attributes.read();
if let Some(annotations) = attrs.get(identifier!(vm, __annotations__)).cloned() {
if let Some(annotations) = attrs.get(annotations_key).cloned() {
// Ignore the __annotations__ descriptor stored on type itself.
if !annotations.class().is(vm.ctx.types.getset_type) {
if vm.is_none(&annotations)
@@ -1521,8 +1632,7 @@ impl PyType {
)));
}
}
// Then try __annotations_cache__
if let Some(annotations) = attrs.get(identifier!(vm, __annotations_cache__)).cloned() {
if let Some(annotations) = attrs.get(annotations_cache_key).cloned() {
if vm.is_none(&annotations)
|| annotations.class().is(vm.ctx.types.dict_type)
|| self.slots.flags.has_feature(PyTypeFlags::HEAPTYPE)
@@ -1559,11 +1669,21 @@ impl PyType {
vm.ctx.new_dict().into()
};
// Cache the result in __annotations_cache__
self.attributes
.write()
.insert(identifier!(vm, __annotations_cache__), annotations.clone());
Ok(annotations)
let (result, _prev) = Self::with_type_lock(vm, || {
let mut attrs = self.attributes.write();
if let Some(existing) = attrs.get(annotations_key).cloned()
&& !existing.class().is(vm.ctx.types.getset_type)
{
return (existing, None);
}
if let Some(existing) = attrs.get(annotations_cache_key).cloned() {
return (existing, None);
}
self.modified_inner();
let prev = attrs.insert(annotations_cache_key, annotations.clone());
(annotations, prev)
});
Ok(result)
}
#[pygetset(setter)]
@@ -1579,43 +1699,43 @@ impl PyType {
)));
}
let mut attrs = self.attributes.write();
let has_annotations = attrs.contains_key(identifier!(vm, __annotations__));
let _prev_values = Self::with_type_lock(vm, || {
self.modified_inner();
let mut attrs = self.attributes.write();
let has_annotations = attrs.contains_key(identifier!(vm, __annotations__));
match value {
crate::function::PySetterValue::Assign(value) => {
// SET path: store the value (including None)
let key = if has_annotations {
identifier!(vm, __annotations__)
} else {
identifier!(vm, __annotations_cache__)
};
attrs.insert(key, value);
if has_annotations {
attrs.swap_remove(identifier!(vm, __annotations_cache__));
let mut prev = Vec::new();
match value {
crate::function::PySetterValue::Assign(value) => {
let key = if has_annotations {
identifier!(vm, __annotations__)
} else {
identifier!(vm, __annotations_cache__)
};
prev.extend(attrs.insert(key, value));
if has_annotations {
prev.extend(attrs.swap_remove(identifier!(vm, __annotations_cache__)));
}
}
crate::function::PySetterValue::Delete => {
let removed = if has_annotations {
attrs.swap_remove(identifier!(vm, __annotations__))
} else {
attrs.swap_remove(identifier!(vm, __annotations_cache__))
};
if removed.is_none() {
return Err(vm.new_attribute_error("__annotations__"));
}
prev.extend(removed);
if has_annotations {
prev.extend(attrs.swap_remove(identifier!(vm, __annotations_cache__)));
}
}
}
crate::function::PySetterValue::Delete => {
// DELETE path: remove the key
let removed = if has_annotations {
attrs
.swap_remove(identifier!(vm, __annotations__))
.is_some()
} else {
attrs
.swap_remove(identifier!(vm, __annotations_cache__))
.is_some()
};
if !removed {
return Err(vm.new_attribute_error("__annotations__"));
}
if has_annotations {
attrs.swap_remove(identifier!(vm, __annotations_cache__));
}
}
}
attrs.swap_remove(identifier!(vm, __annotate_func__));
attrs.swap_remove(identifier!(vm, __annotate__));
prev.extend(attrs.swap_remove(identifier!(vm, __annotate_func__)));
prev.extend(attrs.swap_remove(identifier!(vm, __annotate__)));
Ok(prev)
})?;
Ok(())
}
@@ -1648,9 +1768,13 @@ impl PyType {
#[pygetset(setter)]
fn set___module__(&self, value: PyObjectRef, vm: &VirtualMachine) -> PyResult<()> {
self.check_set_special_type_attr(identifier!(vm, __module__), vm)?;
let mut attributes = self.attributes.write();
attributes.swap_remove(identifier!(vm, __firstlineno__));
attributes.insert(identifier!(vm, __module__), value);
let _prev_values = Self::with_type_lock(vm, || {
self.modified_inner();
let mut attributes = self.attributes.write();
let removed = attributes.swap_remove(identifier!(vm, __firstlineno__));
let prev = attributes.insert(identifier!(vm, __module__), value);
(removed, prev)
});
Ok(())
}
@@ -1772,24 +1896,26 @@ impl PyType {
value: PySetterValue<PyTupleRef>,
vm: &VirtualMachine,
) -> PyResult<()> {
let key = identifier!(vm, __type_params__);
match value {
PySetterValue::Assign(ref val) => {
let key = identifier!(vm, __type_params__);
PySetterValue::Assign(val) => {
self.check_set_special_type_attr(key, vm)?;
self.modified();
self.attributes.write().insert(key, val.clone().into());
let _prev_value = Self::with_type_lock(vm, || {
self.modified_inner();
self.attributes.write().insert(key, val.into())
});
}
PySetterValue::Delete => {
// For delete, we still need to check if the type is immutable
if self.slots.flags.has_feature(PyTypeFlags::IMMUTABLETYPE) {
return Err(vm.new_type_error(format!(
"cannot delete '__type_params__' attribute of immutable type '{}'",
self.slot_name()
)));
}
let key = identifier!(vm, __type_params__);
self.modified();
self.attributes.write().shift_remove(&key);
let _prev_value = Self::with_type_lock(vm, || {
self.modified_inner();
self.attributes.write().shift_remove(&key)
});
}
}
Ok(())
@@ -2397,10 +2523,12 @@ impl Py<PyType> {
// Check if we can set this special type attribute
self.check_set_special_type_attr(identifier!(vm, __doc__), vm)?;
// Set the __doc__ in the type's dict
self.attributes
.write()
.insert(identifier!(vm, __doc__), value);
let _prev_value = PyType::with_type_lock(vm, || {
self.modified_inner();
self.attributes
.write()
.insert(identifier!(vm, __doc__), value)
});
Ok(())
}
@@ -2462,23 +2590,29 @@ impl SetAttr for PyType {
}
let assign = value.is_assign();
// Invalidate inline caches before modifying attributes.
// This ensures other threads see the version invalidation before
// any attribute changes, preventing use-after-free of cached descriptors.
zelf.modified();
// Drop old value OUTSIDE the type lock to avoid deadlock:
// dropping may trigger weakref callbacks → method calls →
// LOAD_ATTR specialization → version_for_specialization → type lock.
let _prev_value = Self::with_type_lock(vm, || {
// Invalidate inline caches before modifying attributes.
// This ensures other threads see the version invalidation before
// any attribute changes, preventing use-after-free of cached descriptors.
zelf.modified_inner();
if let PySetterValue::Assign(value) = value {
zelf.attributes.write().insert(attr_name, value);
} else {
let prev_value = zelf.attributes.write().shift_remove(attr_name); // TODO: swap_remove applicable?
if prev_value.is_none() {
return Err(vm.new_attribute_error(format!(
"type object '{}' has no attribute '{}'",
zelf.name(),
attr_name,
)));
if let PySetterValue::Assign(value) = value {
Ok(zelf.attributes.write().insert(attr_name, value))
} else {
let prev_value = zelf.attributes.write().shift_remove(attr_name); // TODO: swap_remove applicable?
if prev_value.is_none() {
return Err(vm.new_attribute_error(format!(
"type object '{}' has no attribute '{}'",
zelf.name(),
attr_name,
)));
}
Ok(prev_value)
}
}
})?;
if attr_name.as_wtf8().starts_with("__") && attr_name.as_wtf8().ends_with("__") {
if assign {

View File

@@ -1051,7 +1051,7 @@ struct ExecutingFrame<'a> {
}
#[inline]
fn specialization_compact_int_value(i: &PyInt, vm: &VirtualMachine) -> Option<isize> {
fn cpython_compact_int_value(i: &PyInt, vm: &VirtualMachine) -> Option<isize> {
// _PyLong_IsCompact(): a one-digit PyLong (base 2^30),
// i.e. abs(value) <= 2^30 - 1.
const CPYTHON_COMPACT_LONG_ABS_MAX: i64 = (1i64 << 30) - 1;
@@ -1066,7 +1066,7 @@ fn specialization_compact_int_value(i: &PyInt, vm: &VirtualMachine) -> Option<is
#[inline]
fn compact_int_from_obj(obj: &PyObject, vm: &VirtualMachine) -> Option<isize> {
obj.downcast_ref_if_exact::<PyInt>(vm)
.and_then(|i| specialization_compact_int_value(i, vm))
.and_then(|i| cpython_compact_int_value(i, vm))
}
#[inline]
@@ -1075,7 +1075,7 @@ fn exact_float_from_obj(obj: &PyObject, vm: &VirtualMachine) -> Option<f64> {
}
#[inline]
fn specialization_nonnegative_compact_index(i: &PyInt, vm: &VirtualMachine) -> Option<usize> {
fn cpython_nonnegative_compact_index(i: &PyInt, vm: &VirtualMachine) -> Option<usize> {
// _PyLong_IsNonNegativeCompact(): a single base-2^30 digit.
const CPYTHON_COMPACT_LONG_MAX: u64 = (1u64 << 30) - 1;
let v = i.try_to_primitive::<u64>(vm).ok()?;
@@ -3999,7 +3999,7 @@ impl ExecutingFrame<'_> {
let value = self.pop_value();
if let Some(list) = obj.downcast_ref_if_exact::<PyList>(vm)
&& let Some(int_idx) = idx.downcast_ref_if_exact::<PyInt>(vm)
&& let Some(i) = specialization_nonnegative_compact_index(int_idx, vm)
&& let Some(i) = cpython_nonnegative_compact_index(int_idx, vm)
{
let mut vec = list.borrow_vec_mut();
if i < vec.len() {
@@ -4099,7 +4099,7 @@ impl ExecutingFrame<'_> {
if let (Some(list), Some(idx)) = (
a.downcast_ref_if_exact::<PyList>(vm),
b.downcast_ref_if_exact::<PyInt>(vm),
) && let Some(i) = specialization_nonnegative_compact_index(idx, vm)
) && let Some(i) = cpython_nonnegative_compact_index(idx, vm)
{
let vec = list.borrow_vec();
if i < vec.len() {
@@ -4119,7 +4119,7 @@ impl ExecutingFrame<'_> {
if let (Some(tuple), Some(idx)) = (
a.downcast_ref_if_exact::<PyTuple>(vm),
b.downcast_ref_if_exact::<PyInt>(vm),
) && let Some(i) = specialization_nonnegative_compact_index(idx, vm)
) && let Some(i) = cpython_nonnegative_compact_index(idx, vm)
{
let elements = tuple.as_slice();
if i < elements.len() {
@@ -4161,7 +4161,7 @@ impl ExecutingFrame<'_> {
if let (Some(a_str), Some(b_int)) = (
a.downcast_ref_if_exact::<PyStr>(vm),
b.downcast_ref_if_exact::<PyInt>(vm),
) && let Some(i) = specialization_nonnegative_compact_index(b_int, vm)
) && let Some(i) = cpython_nonnegative_compact_index(b_int, vm)
&& let Ok(ch) = a_str.getitem_by_index(vm, i as isize)
&& ch.is_ascii()
{
@@ -4774,9 +4774,6 @@ impl ExecutingFrame<'_> {
&& let Some(init_func) = cls.get_cached_init_for_specialization(cached_version)
&& let Some(cls_alloc) = cls.slots.alloc.load()
{
// Match CPython's `code->co_framesize + _Py_InitCleanup.co_framesize`
// shape, using RustPython's datastack-backed frame size
// equivalent for the extra shim frame.
let init_cleanup_stack_bytes =
datastack_frame_size_bytes_for_code(&vm.ctx.init_cleanup_code)
.expect("_Py_InitCleanup shim is not a generator/coroutine");
@@ -4787,9 +4784,8 @@ impl ExecutingFrame<'_> {
) {
return self.execute_call_vectorcall(nargs, vm);
}
// CPython creates `_Py_InitCleanup` + `__init__` frames here.
// Keep the guard conservative and deopt when the effective
// recursion budget for those two frames is not available.
// Two frames are created: `_Py_InitCleanup` + `__init__`.
// Guard recursion limit accordingly and fall back.
if self.specialization_call_recursion_guard_with_extra_frames(vm, 1) {
return self.execute_call_vectorcall(nargs, vm);
}
@@ -5191,8 +5187,8 @@ impl ExecutingFrame<'_> {
a.downcast_ref_if_exact::<PyInt>(vm),
b.downcast_ref_if_exact::<PyInt>(vm),
) && let (Some(a_val), Some(b_val)) = (
specialization_compact_int_value(a_int, vm),
specialization_compact_int_value(b_int, vm),
cpython_compact_int_value(a_int, vm),
cpython_compact_int_value(b_int, vm),
) {
let op = self.compare_op_from_arg(arg);
let result = op.eval_ord(a_val.cmp(&b_val));
@@ -7320,10 +7316,7 @@ impl ExecutingFrame<'_> {
.load()
.is_some_and(|f| f as usize == PyBaseObject::getattro as *const () as usize);
if !is_default_getattro {
let mut type_version = cls.tp_version_tag.load(Acquire);
if type_version == 0 {
type_version = cls.assign_version_tag();
}
let type_version = cls.version_for_specialization(_vm);
if type_version != 0
&& !oparg.is_method()
&& !self.specialization_eval_frame_active(_vm)
@@ -7361,10 +7354,7 @@ impl ExecutingFrame<'_> {
}
// Get or assign type version
let mut type_version = cls.tp_version_tag.load(Acquire);
if type_version == 0 {
type_version = cls.assign_version_tag();
}
let type_version = cls.version_for_specialization(_vm);
if type_version == 0 {
// Version counter overflow — backoff to avoid re-attempting every execution
unsafe {
@@ -7584,10 +7574,7 @@ impl ExecutingFrame<'_> {
let owner_type = obj.downcast_ref::<PyType>().unwrap();
// Get or assign type version for the type object itself
let mut type_version = owner_type.tp_version_tag.load(Acquire);
if type_version == 0 {
type_version = owner_type.assign_version_tag();
}
let type_version = owner_type.version_for_specialization(_vm);
if type_version == 0 {
unsafe {
self.code.instructions.write_adaptive_counter(
@@ -7622,10 +7609,7 @@ impl ExecutingFrame<'_> {
}
let mut metaclass_version = 0;
if !mcl.slots.flags.has_feature(PyTypeFlags::IMMUTABLETYPE) {
metaclass_version = mcl.tp_version_tag.load(Acquire);
if metaclass_version == 0 {
metaclass_version = mcl.assign_version_tag();
}
metaclass_version = mcl.version_for_specialization(_vm);
if metaclass_version == 0 {
unsafe {
self.code.instructions.write_adaptive_counter(
@@ -7797,7 +7781,7 @@ impl ExecutingFrame<'_> {
bytecode::BinaryOperator::Subscr => {
let b_is_nonnegative_int = b
.downcast_ref_if_exact::<PyInt>(vm)
.is_some_and(|i| specialization_nonnegative_compact_index(i, vm).is_some());
.is_some_and(|i| cpython_nonnegative_compact_index(i, vm).is_some());
if a.downcast_ref_if_exact::<PyList>(vm).is_some() && b_is_nonnegative_int {
Some(Instruction::BinaryOpSubscrListInt)
} else if a.downcast_ref_if_exact::<PyTuple>(vm).is_some() && b_is_nonnegative_int {
@@ -7812,16 +7796,14 @@ impl ExecutingFrame<'_> {
Some(Instruction::BinaryOpSubscrListSlice)
} else {
let cls = a.class();
let (getitem, type_version) =
cls.lookup_ref_and_version_interned(identifier!(vm, __getitem__), vm);
if cls.slots.flags.has_feature(PyTypeFlags::HEAPTYPE)
&& !self.specialization_eval_frame_active(vm)
&& let Some(_getitem) = cls.get_attr(identifier!(vm, __getitem__))
&& let Some(_getitem) = getitem
&& let Some(func) = _getitem.downcast_ref_if_exact::<PyFunction>(vm)
&& func.can_specialize_call(2)
{
let mut type_version = cls.tp_version_tag.load(Acquire);
if type_version == 0 {
type_version = cls.assign_version_tag();
}
if type_version != 0 {
if cls.cache_getitem_for_specialization(
func.to_owned(),
@@ -8360,11 +8342,8 @@ impl ExecutingFrame<'_> {
&& cls_new_fn as usize == obj_new_fn as usize
&& cls_alloc_fn as usize == obj_alloc_fn as usize
{
let init = cls.get_attr(identifier!(vm, __init__));
let mut version = cls.tp_version_tag.load(Acquire);
if version == 0 {
version = cls.assign_version_tag();
}
let (init, version) =
cls.lookup_ref_and_version_interned(identifier!(vm, __init__), vm);
if version == 0 {
unsafe {
self.code.instructions.write_adaptive_counter(
@@ -8620,8 +8599,8 @@ impl ExecutingFrame<'_> {
a.downcast_ref_if_exact::<PyInt>(vm),
b.downcast_ref_if_exact::<PyInt>(vm),
) {
if specialization_compact_int_value(a_int, vm).is_some()
&& specialization_compact_int_value(b_int, vm).is_some()
if cpython_compact_int_value(a_int, vm).is_some()
&& cpython_compact_int_value(b_int, vm).is_some()
{
Some(Instruction::CompareOpInt)
} else {
@@ -8684,10 +8663,7 @@ impl ExecutingFrame<'_> {
&& cls.slots.as_sequence.length.load().is_none()
{
// Cache type version for ToBoolAlwaysTrue guard
let mut type_version = cls.tp_version_tag.load(Acquire);
if type_version == 0 {
type_version = cls.assign_version_tag();
}
let type_version = cls.version_for_specialization(vm);
if type_version != 0 {
unsafe {
self.code
@@ -8919,7 +8895,7 @@ impl ExecutingFrame<'_> {
idx.downcast_ref_if_exact::<PyInt>(vm),
) {
let list_len = list.borrow_vec().len();
if specialization_nonnegative_compact_index(int_idx, vm).is_some_and(|i| i < list_len) {
if cpython_nonnegative_compact_index(int_idx, vm).is_some_and(|i| i < list_len) {
Some(Instruction::StoreSubscrListInt)
} else {
None
@@ -9025,10 +9001,7 @@ impl ExecutingFrame<'_> {
}
// Get or assign type version
let mut type_version = cls.tp_version_tag.load(Acquire);
if type_version == 0 {
type_version = cls.assign_version_tag();
}
let type_version = cls.version_for_specialization(vm);
if type_version == 0 {
unsafe {
self.code.instructions.write_adaptive_counter(

View File

@@ -840,6 +840,7 @@ pub mod module {
reinit_mutex_after_fork(&vm.state.atexit_funcs);
reinit_mutex_after_fork(&vm.state.global_trace_func);
reinit_mutex_after_fork(&vm.state.global_profile_func);
reinit_mutex_after_fork(&vm.state.type_mutex);
reinit_mutex_after_fork(&vm.state.monitoring);
// PyGlobalState parking_lot::Mutex locks

View File

@@ -115,17 +115,30 @@ mod decl {
#[cfg(unix)]
{
// this is basically std::thread::sleep, but that catches interrupts and we don't want to;
let ts = nix::sys::time::TimeSpec::from(dur);
// Capture errno inside the closure: attach_thread (called by
// allow_threads on return) can clobber errno via syscalls.
let (res, err) = vm.allow_threads(|| {
let r = unsafe { libc::nanosleep(ts.as_ref(), core::ptr::null_mut()) };
(r, nix::Error::last_raw())
});
let interrupted = res == -1 && err == libc::EINTR;
if interrupted {
// Loop on nanosleep, recomputing the
// remaining timeout after each EINTR so that signals don't
// shorten the requested sleep duration.
use std::time::Instant;
let deadline = Instant::now() + dur;
loop {
let remaining = deadline.saturating_duration_since(Instant::now());
if remaining.is_zero() {
break;
}
let ts = nix::sys::time::TimeSpec::from(remaining);
let (res, err) = vm.allow_threads(|| {
let r = unsafe { libc::nanosleep(ts.as_ref(), core::ptr::null_mut()) };
(r, nix::Error::last_raw())
});
if res == 0 {
break;
}
if err != libc::EINTR {
return Err(
vm.new_os_error(format!("nanosleep: {}", nix::Error::from_raw(err)))
);
}
// EINTR: run signal handlers, then retry with remaining time
vm.check_signals()?;
}
}

View File

@@ -115,6 +115,7 @@ where
switch_interval: AtomicCell::new(0.005),
global_trace_func: PyMutex::default(),
global_profile_func: PyMutex::default(),
type_mutex: PyMutex::default(),
#[cfg(feature = "threading")]
main_thread_ident: AtomicCell::new(0),
#[cfg(feature = "threading")]

View File

@@ -599,6 +599,8 @@ pub struct PyGlobalState {
pub global_trace_func: PyMutex<Option<PyObjectRef>>,
/// Global profile function for all threads (set by sys._setprofileallthreads)
pub global_profile_func: PyMutex<Option<PyObjectRef>>,
/// Global type mutation/versioning mutex for CPython-style FT type operations.
pub type_mutex: PyMutex<()>,
/// Main thread identifier (pthread_self on Unix)
#[cfg(feature = "threading")]
pub main_thread_ident: AtomicCell<u64>,

View File

@@ -23,8 +23,6 @@ rustpython-stdlib = { workspace = true, default-features = false, optional = tru
# make sure no threading! otherwise wasm build will fail
rustpython-vm = { workspace = true, features = ["compiler", "encodings", "serde", "wasmbind"] }
ruff_python_parser = { workspace = true }
serde = { workspace = true }
wasm-bindgen = { workspace = true }
@@ -48,3 +46,6 @@ wasm-opt = false#["-O1"]
[lints]
workspace = true
[package.metadata.cargo-shear]
ignored = ["serde", "rustpython-common"]

View File

@@ -38,6 +38,16 @@ from functools import reduce
from unittest.runner import registerResult, result
def _get_method_dict(test):
"""Get the __dict__ of the underlying function for a test method.
Works for both bound methods (__func__.__dict__) and plain functions.
"""
method = getattr(test, test._testMethodName)
func = getattr(method, "__func__", method)
return func.__dict__
class TablePrinter(object):
# Modified from https://github.com/agramian/table-printer, same license as above
"Print a list of dicts as a table"
@@ -325,9 +335,7 @@ class CustomTextTestResult(result.TestResult):
self.stream.writeln("TEST SUITE: %s" % self.suite)
self.stream.writeln("Description: %s" % self.getSuiteDescription(test))
try:
name_override = getattr(test, test._testMethodName).__func__.__dict__[
"test_case_name"
]
name_override = _get_method_dict(test)["test_case_name"]
except:
name_override = None
self.case = name_override if name_override else self.case
@@ -345,7 +353,11 @@ class CustomTextTestResult(result.TestResult):
self.results["suites"][self.suite_number] = {
"name": self.suite,
"class": test.__class__.__name__,
"module": re.compile(".* \((.*)\)").match(str(test)).group(1),
"module": (
m.group(1)
if (m := re.compile(r".* \((.*)\)").match(str(test)))
else str(test)
),
"description": self.getSuiteDescription(test),
"cases": {},
"used_case_names": {},
@@ -380,34 +392,22 @@ class CustomTextTestResult(result.TestResult):
if "test_type" in getattr(
test, test._testMethodName
).__func__.__dict__ and set([s.lower() for s in self.test_types]) == set(
[
s.lower()
for s in getattr(test, test._testMethodName).__func__.__dict__[
"test_type"
]
]
[s.lower() for s in _get_method_dict(test)["test_type"]]
):
pass
else:
getattr(test, test._testMethodName).__func__.__dict__[
"__unittest_skip_why__"
] = 'Test run specified to only run tests of type "%s"' % ",".join(
self.test_types
_get_method_dict(test)["__unittest_skip_why__"] = (
'Test run specified to only run tests of type "%s"'
% ",".join(self.test_types)
)
getattr(test, test._testMethodName).__func__.__dict__[
"__unittest_skip__"
] = True
if "skip_device" in getattr(test, test._testMethodName).__func__.__dict__:
for device in getattr(test, test._testMethodName).__func__.__dict__[
"skip_device"
]:
_get_method_dict(test)["__unittest_skip__"] = True
if "skip_device" in _get_method_dict(test):
for device in _get_method_dict(test)["skip_device"]:
if self.config and device.lower() in self.config["device_name"].lower():
getattr(test, test._testMethodName).__func__.__dict__[
"__unittest_skip_why__"
] = "Test is marked to be skipped on %s" % device
getattr(test, test._testMethodName).__func__.__dict__[
"__unittest_skip__"
] = True
_get_method_dict(test)["__unittest_skip_why__"] = (
"Test is marked to be skipped on %s" % device
)
_get_method_dict(test)["__unittest_skip__"] = True
break
def stopTest(self, test):