Fix cron-ci failures: ctypes set_attr, missing features, __func__ AttributeError

- Use cls.set_attr() instead of cls.as_object().set_attr() in ctypes
  to ensure modified() is called and type cache stays valid
- Add host_env feature to cron-ci.yaml for frozen_origname_matches test
- Add stdio feature to cron-ci.yaml for encodings initialization
- Fix __func__ AttributeError in custom_text_test_runner.py

.cargo/config.toml

@@ -3,3 +3,6 @@ rustflags = "-C link-arg=/STACK:8000000"
 
 [target.'cfg(all(target_os = "windows", not(target_env = "msvc")))']
 rustflags = "-C link-args=-Wl,--stack,8000000"
+
+[target.wasm32-unknown-unknown]
+rustflags = ["--cfg=getrandom_backend=\"wasm_js\""]

.claude/commands/apple-container.md

@@ -0,0 +1,46 @@
+---
+allowed-tools: Bash(container *), Bash(cargo *), Read, Grep, Glob
+---
+
+# Run Tests in Linux Container (Apple `container` CLI)
+
+Run RustPython tests inside a Linux container using Apple's `container` CLI.
+**NEVER use Docker, Podman, or any other container runtime.** Only use the `container` command.
+
+## Arguments
+- `$ARGUMENTS`: Test command to run (e.g., `test_io`, `test_codecs -v`, `test_io -v -m "test_errors"`)
+
+## Prerequisites
+
+The `container` CLI is installed via `brew install container`.
+The dev image `rustpython-dev` is already built.
+
+## Steps
+
+1. **Check if the container is already running**
+   ```shell
+   container list 2>/dev/null | grep rustpython-test
+   ```
+
+2. **Start the container if not running**
+   ```shell
+   container run -d --name rustpython-test -m 8G -c 4 \
+       --mount type=bind,source=/Users/al03219714/Projects/RustPython3,target=/workspace \
+       -w /workspace rustpython-dev sleep infinity
+   ```
+
+3. **Run the test inside the container**
+   ```shell
+   container exec rustpython-test sh -c "cargo run --release -- -m test $ARGUMENTS"
+   ```
+
+4. **Report results**
+   - Show test summary (pass/fail counts, expected failures, unexpected successes)
+   - Highlight any new failures compared to macOS results if available
+   - Do NOT stop or remove the container after testing (keep it for reuse)
+
+## Notes
+- The workspace is bind-mounted, so local code changes are immediately available
+- Use `container exec rustpython-test sh -c "..."` for any command inside the container
+- To rebuild after code changes, run: `container exec rustpython-test sh -c "cargo build --release"`
+- To stop the container when done: `container rm -f rustpython-test`

.claude/commands/investigate-test-failure.md

@@ -0,0 +1,49 @@
+---
+allowed-tools: Bash(python3:*), Bash(cargo run:*), Read, Grep, Glob, Bash(git add:*), Bash(git commit:*), Bash(cargo fmt:*), Bash(git diff:*), Task
+---
+
+# Investigate Test Failure
+
+Investigate why a specific test is failing and determine if it can be fixed or needs an issue.
+
+## Arguments
+- `$ARGUMENTS`: Failed test identifier (e.g., `test_inspect.TestGetSourceBase.test_getsource_reload`)
+
+## Steps
+
+1. **Analyze failure cause**
+   - Read the test code
+   - Analyze failure message/traceback
+   - Check related RustPython code
+
+2. **Verify behavior in CPython**
+   - Run the test with `python3 -m unittest` to confirm expected behavior
+   - Document the expected output
+
+3. **Determine fix feasibility**
+   - **Simple fix** (import issues, small logic bugs): Fix code → Run `cargo fmt --all` → Pre-commit review → Commit
+   - **Complex fix** (major unimplemented features): Collect issue info and report to user
+
+   **Pre-commit review process**:
+   - Run `git diff` to see the changes
+   - Use Task tool with `general-purpose` subagent to review:
+     - Compare implementation against cpython/ source code
+     - Verify the fix aligns with CPython behavior
+     - Check for any missed edge cases
+   - Proceed to commit only after review passes
+
+4. **For complex issues - Collect issue information**
+   Following `.github/ISSUE_TEMPLATE/report-incompatibility.md` format:
+
+   - **Feature**: Description of missing/broken Python feature
+   - **Minimal reproduction code**: Smallest code that reproduces the issue
+   - **CPython behavior**: Result when running with python3
+   - **RustPython behavior**: Result when running with cargo run
+   - **Python Documentation link**: Link to relevant CPython docs
+
+   Report collected information to the user. Issue creation is done only upon user request.
+
+   Example issue creation command:
+   ```
+   gh issue create --template report-incompatibility.md --title "..." --body "..."
+   ```

.claude/commands/upgrade-pylib-next.md

@@ -0,0 +1,33 @@
+---
+allowed-tools: Skill(upgrade-pylib), Bash(gh pr list:*)
+---
+
+# Upgrade Next Python Library
+
+Find the next Python library module ready for upgrade and run `/upgrade-pylib` for it.
+
+## Current TODO Status
+
+!`cargo run --release -- scripts/update_lib todo 2>/dev/null`
+
+## Open Upgrade PRs
+
+!`gh pr list --search "Update in:title" --json number,title --template '{{range .}}#{{.number}} {{.title}}{{"\n"}}{{end}}'`
+
+## Instructions
+
+From the TODO list above, find modules matching these patterns (in priority order):
+
+1. `[ ] [no deps]` - Modules with no dependencies (can be upgraded immediately)
+2. `[ ] [0/n]` - Modules where all dependencies are already upgraded (e.g., `[0/3]`, `[0/5]`)
+
+These patterns indicate modules that are ready to upgrade without blocking dependencies.
+
+**Important**: Skip any modules that already have an open PR in the "Open Upgrade PRs" list above.
+
+**After identifying a suitable module**, run:
+```
+/upgrade-pylib <module_name>
+```
+
+If no modules match these criteria, inform the user that all eligible modules have dependencies that need to be upgraded first.

.claude/commands/upgrade-pylib.md

@@ -0,0 +1,157 @@
+---
+allowed-tools: Bash(git add:*), Bash(git commit:*), Bash(python3 scripts/update_lib quick:*), Bash(python3 scripts/update_lib auto-mark:*)
+---
+
+# Upgrade Python Library from CPython
+
+Upgrade a Python standard library module from CPython to RustPython.
+
+## Arguments
+- `$ARGUMENTS`: Library name to upgrade (e.g., `inspect`, `asyncio`, `json`)
+
+## Important: Report Tool Issues First
+
+If during the upgrade process you encounter any of the following issues with `scripts/update_lib`:
+- A feature that should be automated but isn't supported
+- A bug or unexpected behavior in the tool
+- Missing functionality that would make the upgrade easier
+
+**STOP the upgrade and report the issue first.** Describe:
+1. What you were trying to do
+    - Library name
+    - The full command executed (e.g. python scripts/update_lib quick cpython/Lib/$ARGUMENTS.py)
+2. What went wrong or what's missing
+3. Expected vs actual behavior
+
+This helps improve the tooling for future upgrades.
+
+## Steps
+
+1. **Run quick upgrade with update_lib**
+   - Run: `python3 scripts/update_lib quick $ARGUMENTS` (module name)
+   - Or: `python3 scripts/update_lib quick cpython/Lib/$ARGUMENTS.py` (library file path)
+   - Or: `python3 scripts/update_lib quick cpython/Lib/$ARGUMENTS/` (library directory path)
+   - This will:
+     - Copy library files (delete existing `Lib/$ARGUMENTS.py` or `Lib/$ARGUMENTS/`, then copy from `cpython/Lib/`)
+     - Patch test files preserving existing RustPython markers
+     - Run tests and auto-mark new test failures (not regressions)
+     - Remove `@unittest.expectedFailure` from tests that now pass
+     - Create a git commit with the changes
+   - **Handle warnings**: If you see warnings like `WARNING: TestCFoo does not exist in remote file`, it means the class structure changed and markers couldn't be transferred automatically. These need to be manually restored in step 2 or added in step 3.
+
+2. **Review git diff and restore RUSTPYTHON-specific changes**
+   - Run `git diff Lib/test/test_$ARGUMENTS` to review all changes
+   - **Only restore changes that have explicit `RUSTPYTHON` comments**. Look for:
+     - `# XXX: RUSTPYTHON` or `# XXX RUSTPYTHON` - Comments marking RustPython-specific code modifications
+     - `# TODO: RUSTPYTHON` - Comments marking tests that need work
+     - Code changes with inline `# ... RUSTPYTHON` comments
+   - **Do NOT restore other diff changes** - these are likely upstream CPython changes, not RustPython-specific modifications
+   - When restoring, preserve the original context and formatting
+
+3. **Investigate test failures with subagent**
+   - First, get dependent tests using the deps command:
+     ```
+     cargo run --release -- scripts/update_lib deps $ARGUMENTS
+     ```
+   - Look for the line `- [ ] $ARGUMENTS: test_xxx test_yyy ...` to get the direct dependent tests
+   - Run those tests to collect failures:
+     ```
+     cargo run --release -- -m test test_xxx test_yyy ... 2>&1 | grep -E "^(FAIL|ERROR):"
+     ```
+   - For example, if deps output shows `- [ ] linecache: test_bdb test_inspect test_linecache test_traceback test_zipimport`, run:
+     ```
+     cargo run --release -- -m test test_bdb test_inspect test_linecache test_traceback test_zipimport 2>&1 | grep -E "^(FAIL|ERROR):"
+     ```
+   - For each failure, use the Task tool with `general-purpose` subagent to investigate:
+     - Subagent should follow the `/investigate-test-failure` skill workflow
+     - Pass the failed test identifier as the argument (e.g., `test_inspect.TestGetSourceBase.test_getsource_reload`)
+   - If subagent can fix the issue easily: fix and commit
+   - If complex issue: subagent collects issue info and reports back (issue creation on user request only)
+   - Using subagent prevents context pollution in the main conversation
+
+4. **Mark remaining test failures with auto-mark**
+   - Run: `python3 scripts/update_lib auto-mark Lib/test/test_$ARGUMENTS.py --mark-failure`
+   - Or for directory: `python3 scripts/update_lib auto-mark Lib/test/test_$ARGUMENTS/ --mark-failure`
+   - This will:
+     - Run tests and mark ALL failing tests with `@unittest.expectedFailure`
+     - Remove `@unittest.expectedFailure` from tests that now pass
+   - **Note**: The `--mark-failure` flag marks all failures including regressions. Review the changes before committing.
+
+5. **Handle panics manually**
+   - If any tests cause panics/crashes (not just assertion failures), they need `@unittest.skip` instead:
+     ```python
+     @unittest.skip("TODO: RUSTPYTHON; panics with 'index out of bounds'")
+     def test_crashes(self):
+         ...
+     ```
+   - auto-mark cannot detect panics automatically - check the test output for crash messages
+
+6. **Handle class-specific failures**
+   - If a test fails only in the C implementation (TestCFoo) but passes in the Python implementation (TestPyFoo), or vice versa, move the marker to the specific subclass:
+     ```python
+     # Base class - no marker here
+     class TestFoo:
+         def test_something(self):
+             ...
+
+     class TestPyFoo(TestFoo, PyTest): pass
+
+     class TestCFoo(TestFoo, CTest):
+         # TODO: RUSTPYTHON
+         @unittest.expectedFailure
+         def test_something(self):
+             return super().test_something()
+     ```
+
+7. **Commit the test fixes**
+   - Run: `git add -u && git commit -m "Mark failing tests"`
+   - This creates a separate commit for the test markers added in steps 2-6
+
+## Example Usage
+```
+# Using module names (recommended)
+/upgrade-pylib inspect
+/upgrade-pylib json
+/upgrade-pylib asyncio
+
+# Using library paths (alternative)
+/upgrade-pylib cpython/Lib/inspect.py
+/upgrade-pylib cpython/Lib/json/
+```
+
+## Example: Restoring RUSTPYTHON changes
+
+When git diff shows removed RUSTPYTHON-specific code like:
+```diff
+-# XXX RUSTPYTHON: we don't import _json as fresh since...
+-cjson = import_helper.import_fresh_module('json') #, fresh=['_json'])
++cjson = import_helper.import_fresh_module('json', fresh=['_json'])
+```
+
+You should restore the RustPython version:
+```python
+# XXX RUSTPYTHON: we don't import _json as fresh since...
+cjson = import_helper.import_fresh_module('json') #, fresh=['_json'])
+```
+
+## Notes
+- The cpython/ directory should contain the CPython source that we're syncing from
+- `scripts/update_lib` package handles patching and auto-marking:
+  - `quick` - Combined patch + auto-mark (recommended)
+  - `migrate` - Only migrate (patch), no test running
+  - `auto-mark` - Only run tests and mark failures
+  - `copy-lib` - Copy library files (not tests)
+- The patching:
+  - Transfers `@unittest.expectedFailure` and `@unittest.skip` decorators with `TODO: RUSTPYTHON` markers
+  - Adds `import unittest # XXX: RUSTPYTHON` if needed for the decorators
+  - **Limitation**: If a class was restructured (e.g., method overrides removed), update_lib will warn and skip those markers
+- The smart auto-mark:
+  - Marks NEW test failures automatically (tests that didn't exist before)
+  - Does NOT mark regressions (existing tests that now fail) - these are warnings
+  - Removes `@unittest.expectedFailure` from tests that now pass
+- The script does NOT preserve all RustPython-specific changes - you must review `git diff` and restore them
+- Common RustPython markers to look for:
+  - `# XXX: RUSTPYTHON` or `# XXX RUSTPYTHON` - Inline comments for code modifications
+  - `# TODO: RUSTPYTHON` - Test skip/failure markers
+  - Any code with `RUSTPYTHON` in comments that was removed in the diff
+- **Important**: Not all changes in the git diff need to be restored. Only restore changes that have explicit `RUSTPYTHON` comments. Other changes are upstream CPython updates.

.claude/scripts/setup-env.sh

@@ -0,0 +1,50 @@
+#!/bin/bash
+# Claude Code web session startup script
+# Sets up the development environment for RustPython
+
+set -e
+
+cd /home/user/RustPython
+
+echo "=== RustPython dev environment setup ==="
+
+# 1. Ensure python3 points to 3.13+ (needed for scripts/update_lib)
+#    /usr/local/bin takes precedence over /usr/bin in PATH,
+#    so we update the symlink there directly.
+CURRENT_PY=$(python3 --version 2>&1 | grep -oP '\d+\.\d+')
+if [ "$(printf '%s\n' "3.13" "$CURRENT_PY" | sort -V | head -1)" != "3.13" ]; then
+    echo "Upgrading python3 default to 3.13..."
+    # Find best available Python >= 3.13
+    TARGET=""
+    for ver in python3.14 python3.13; do
+        if command -v "$ver" &>/dev/null; then
+            TARGET=$(command -v "$ver")
+            break
+        fi
+    done
+    if [ -n "$TARGET" ]; then
+        # Override /usr/local/bin/python3 if it exists and is outdated
+        if [ -e /usr/local/bin/python3 ]; then
+            sudo ln -sf "$TARGET" /usr/local/bin/python3
+        fi
+        # Also set /usr/bin via update-alternatives
+        sudo update-alternatives --install /usr/bin/python3 python3 "$TARGET" 3 2>/dev/null || true
+        sudo update-alternatives --set python3 "$TARGET" 2>/dev/null || true
+        echo "python3 now: $(python3 --version)"
+    else
+        echo "WARNING: No Python 3.13+ found. scripts/update_lib may not work."
+    fi
+else
+    echo "python3 already >= 3.13: $(python3 --version)"
+fi
+
+# 2. Clone CPython source if not present (needed for scripts/update_lib)
+if [ ! -d "cpython" ]; then
+    echo "Cloning CPython v3.14.3 (shallow)..."
+    git clone --depth 1 --branch v3.14.3 https://github.com/python/cpython.git cpython
+    echo "CPython source ready."
+else
+    echo "CPython source already present."
+fi
+
+echo "=== Setup complete ==="

.claude/settings.json

@@ -0,0 +1,15 @@
+{
+  "hooks": {
+    "SessionStart": [
+      {
+        "matcher": "",
+        "hooks": [
+          {
+            "type": "command",
+            "command": "bash .claude/scripts/setup-env.sh"
+          }
+        ]
+      }
+    ]
+  }
+}

.coderabbit.yml

@@ -0,0 +1,3 @@
+reviews:
+  path_filters:
+    - "!Lib/**"

.cspell.dict/cpython.txt

@@ -0,0 +1,233 @@
+ADDOP
+aftersign
+argdefs
+argtypes
+asdl
+asname
+attro
+augassign
+badcert
+badsyntax
+baseinfo
+basetype
+binop
+bltin
+boolop
+BUFMAX
+BUILDSTDLIB
+bxor
+byteswap
+cached_tsver
+cadata
+cafile
+calldepth
+callinfo
+callproc
+capath
+carg
+cellarg
+cellvar
+cellvars
+ceval
+cfield
+CLASSDEREF
+classdict
+cmpop
+codedepth
+CODEUNIT
+CONIN
+CONOUT
+constevaluator
+consti
+CONVFUNC
+convparam
+copyslot
+cpucount
+datastack
+defaultdict
+denom
+deopt
+dictbytype
+DICTFLAG
+dictoffset
+distpoint
+dynload
+elts
+eofs
+evalloop
+excepthandler
+exceptiontable
+fastlocal
+fastlocals
+fblock
+fblocks
+fdescr
+ffi_argtypes
+fielddesc
+fieldlist
+fileutils
+finalbody
+finalizers
+firsttraceable
+flowgraph
+formatfloat
+freelist
+freevar
+freevars
+fromlist
+getdict
+getfunc
+getiter
+getsets
+getslice
+globalgetvar
+HASARRAY
+HASBITFIELD
+HASPOINTER
+HASSTRUCT
+HASUNION
+heaptype
+hexdigit
+HIGHRES
+IFUNC
+IMMUTABLETYPE
+INCREF
+inlinedepth
+inplace
+ismine
+ISPOINTER
+iteminfo
+Itertool
+keeped
+kwnames
+kwonlyarg
+kwonlyargs
+lasti
+libffi
+linearise
+lineiterator
+linetable
+loadfast
+localsplus
+Lshift
+lsprof
+MAXBLOCKS
+maxdepth
+metavars
+miscompiles
+mult
+multibytecodec
+nameobj
+nameop
+ncells
+nconsts
+newargs
+newfree
+NEWLOCALS
+newsemlockobject
+nfrees
+nkwargs
+nkwelts
+nlocalsplus
+Nondescriptor
+noninteger
+nops
+noraise
+nseen
+NSIGNALS
+numer
+opname
+opnames
+orelse
+outparam
+outparm
+paramfunc
+parg
+pathconfig
+patma
+peepholer
+phcount
+platstdlib
+posonlyarg
+posonlyargs
+prec
+preinitialized
+pybuilddir
+pycore
+pyinner
+pydecimal
+Pyfunc
+pylifecycle
+pymain
+pyrepl
+PYTHONTRACEMALLOC
+PYTHONUTF8
+pythonw
+PYTHREAD_NAME
+releasebuffer
+repr
+resinfo
+Rshift
+SA_ONSTACK
+saveall
+scls
+setdict
+setfunc
+setprofileallthreads
+SETREF
+setresult
+setslice
+settraceallthreads
+SLOTDEFINED
+SMALLBUF
+SOABI
+SSLEOF
+stackdepth
+stackref
+staticbase
+stginfo
+storefast
+stringlib
+structseq
+subkwargs
+subparams
+subscr
+sval
+swappedbytes
+sysdict
+templatelib
+testconsole
+threadstate
+ticketer
+tmptype
+tok_oldval
+tstate
+tvars
+typeobject
+typeparam
+Typeparam
+typeparams
+typeslots
+unaryop
+uncollectable
+Unhandle
+unparse
+unparser
+untracking
+VARKEYWORDS
+varkwarg
+venvlauncher
+venvlaunchert
+venvw
+venvwlauncher
+venvwlaunchert
+wbits
+weakreflist
+weakrefobject
+webpki
+winconsoleio
+withitem
+withs
+worklist
+xstat
+XXPRIME

.cspell.dict/python-more.txt

@@ -0,0 +1,301 @@
+abiflags
+abstractmethods
+addcompare
+aenter
+aexit
+aiter
+altzone
+anext
+anextawaitable
+annotationlib
+appendleft
+argcount
+arrayiterator
+arraytype
+asend
+asyncgen
+athrow
+backslashreplace
+baserepl
+basicsize
+bdfl
+bigcharset
+bignum
+bivariant
+breakpointhook
+cformat
+chunksize
+classcell
+classmethods
+closefd
+closesocket
+codepoint
+codepoints
+codesize
+contextvar
+cpython
+cratio
+ctype
+ctypes
+dealloc
+debugbuild
+decompressor
+defaultaction
+descr
+dictcomp
+dictitems
+dictkeys
+dictview
+digestmod
+dllhandle
+docstring
+docstrings
+dunder
+endianness
+endpos
+eventmask
+excepthook
+exceptiongroup
+exitfuncs
+extendleft
+fastlocals
+fdel
+fedcba
+fget
+fileencoding
+fillchar
+fillvalue
+finallyhandler
+firstiter
+firstlineno
+fnctl
+frombytes
+fromhex
+fromunicode
+frozensets
+fset
+fspath
+fstring
+fstrings
+ftruncate
+genexpr
+genexpressions
+getargs
+getattro
+getcodesize
+getdefaultencoding
+getfilesystemencodeerrors
+getfilesystemencoding
+getformat
+getframe
+getframemodulename
+getnewargs
+getopt
+getpip
+getrandom
+getrecursionlimit
+getrefcount
+getsizeof
+getswitchinterval
+getweakref
+getweakrefcount
+getweakrefs
+getweakrefs
+getwindowsversion
+gmtoff
+groupdict
+groupindex
+hamt
+hostnames
+idfunc
+idiv
+idxs
+impls
+indexgroup
+infj
+inittab
+Inittab
+instancecheck
+instanceof
+interpchannels
+interpqueues
+irepeat
+isabstractmethod
+isbytes
+iscased
+isfinal
+istext
+itemiterator
+itemsize
+iternext
+keepends
+keyfunc
+keyiterator
+kwarg
+kwargs
+kwdefaults
+kwonlyargcount
+lastgroup
+lastindex
+linearization
+linearize
+listcomp
+longrange
+lvalue
+mappingproxy
+markupbase
+maskpri
+maxdigits
+MAXGROUPS
+MAXREPEAT
+maxsplit
+maxunicode
+memoryview
+memoryviewiterator
+metaclass
+metaclasses
+metatype
+mformat
+mro
+mros
+multiarch
+mymodule
+namereplace
+nanj
+nbytes
+ncallbacks
+ndigits
+ndim
+needsfree
+nldecoder
+nlocals
+NOARGS
+nonbytes
+Nonprintable
+onceregistry
+origname
+ospath
+pendingcr
+phello
+platlibdir
+popleft
+posixsubprocess
+posonly
+posonlyargcount
+prepending
+profilefunc
+pycache
+pycodecs
+pycs
+pydatetime
+pyexpat
+pyio
+pymain
+PYTHONAPI
+PYTHONBREAKPOINT
+PYTHONDEBUG
+PYTHONDONTWRITEBYTECODE
+PYTHONFAULTHANDLER
+PYTHONHASHSEED
+PYTHONHOME
+PYTHONINSPECT
+PYTHONINTMAXSTRDIGITS
+PYTHONIOENCODING
+PYTHONNODEBUGRANGES
+PYTHONNOUSERSITE
+PYTHONOPTIMIZE
+PYTHONPATH
+PYTHONPATH
+PYTHONSAFEPATH
+PYTHONUNBUFFERED
+PYTHONVERBOSE
+PYTHONWARNDEFAULTENCODING
+PYTHONWARNINGS
+pytraverse
+PYVENV
+qualname
+quotetabs
+radd
+rdiv
+rdivmod
+readall
+readbuffer
+reconstructor
+refcnt
+releaselevel
+reraised
+reverseitemiterator
+reverseiterator
+reversekeyiterator
+reversevalueiterator
+rfloordiv
+rlshift
+rmod
+rpow
+rrshift
+rsub
+rtruediv
+rvalue
+scproxy
+seennl
+setattro
+setcomp
+setprofileallthreads
+setrecursionlimit
+setswitchinterval
+settraceallthreads
+showwarnmsg
+signum
+sitebuiltins
+slotnames
+STACKLESS
+stacklevel
+stacksize
+startpos
+subclassable
+subclasscheck
+subclasshook
+subclassing
+suboffset
+suboffsets
+SUBPATTERN
+subpatterns
+sumprod
+surrogateescape
+surrogatepass
+sysconf
+sysconfigdata
+sysdict
+sysvars
+teedata
+thisclass
+titlecased
+tkapp
+tobytes
+tolist
+toreadonly
+TPFLAGS
+tracefunc
+unimportable
+unionable
+unraisablehook
+unsliceable
+urandom
+valueiterator
+vararg
+varargs
+varnames
+warningregistry
+warnmsg
+warnoptions
+warnopts
+weaklist
+weakproxy
+weakrefs
+weakrefset
+winver
+withdata
+xmlcharrefreplace
+xoptions
+xopts
+yieldfrom

.cspell.dict/rust-more.txt

@@ -0,0 +1,94 @@
+ahash
+arrayvec
+bidi
+biguint
+bindgen
+bitand
+bitflags
+bitflagset
+bitor
+bitvec
+bitxor
+bstr
+byteorder
+byteset
+caseless
+chrono
+consts
+cranelift
+cstring
+datelike
+deserializer
+deserializers
+fdiv
+flamescope
+flate2
+fract
+getres
+hasher
+hexf
+hexversion
+idents
+illumos
+ilog
+indexmap
+insta
+keccak
+lalrpop
+lexopt
+libc
+libcall
+libloading
+libz
+longlong
+Manually
+maplit
+memmap
+memmem
+metas
+modpow
+msvc
+muldiv
+nanos
+nonoverlapping
+objclass
+peekable
+pemfile
+powc
+powf
+powi
+prepended
+punct
+replacen
+retag
+rmatch
+rposition
+rsplitn
+rustc
+rustfmt
+rustls
+rustyline
+seedable
+seekfrom
+siphash
+siphasher
+splitn
+subsec
+thiserror
+timelike
+timsort
+trai
+ulonglong
+unic
+unistd
+unraw
+unsync
+wasip1
+wasip2
+wasmbind
+wasmer
+wasmtime
+widestring
+winapi
+winresource
+winsock

.cspell.dict/rustpython.txt

@@ -0,0 +1,32 @@
+cfgs
+miri
+py
+pyarg
+pyargs
+pyast
+pyattr
+pyclass
+pyclassmethod
+pyexception
+pyfunction
+pygetset
+pyimpl
+pylib
+pymath
+pymethod
+pymodule
+pyname
+pyobj
+pyobject
+pypayload
+pyref
+pyslot
+pystaticmethod
+pystone
+pystr
+pystruct
+pystructseq
+pytype
+rustix
+struc
+zelf

.cspell.json

@@ -1,288 +1,95 @@
 // See: https://github.com/streetsidesoftware/cspell/tree/master/packages/cspell
 {
   "version": "0.2",
+  "import": [
+    "@cspell/dict-en_us/cspell-ext.json",
+    // "@cspell/dict-cpp/cspell-ext.json",
+    "@cspell/dict-python/cspell-ext.json",
+    "@cspell/dict-rust/cspell-ext.json",
+    "@cspell/dict-win32/cspell-ext.json",
+    "@cspell/dict-shell/cspell-ext.json",
+  ],
+  "allowCompoundWords": true,
   // language - current active spelling language
   "language": "en",
   // dictionaries - list of the names of the dictionaries to use
   "dictionaries": [
+    "cpython",  // Sometimes keeping same terms with cpython is easy
+    "python-more",  // Python API terms not listed in python
+    "rust-more",  // Rust API terms not listed in rust
+    "rustpython",  // RustPython derive macros and internal terms
     "en_US",
     "softwareTerms",
     "c",
     "cpp",
     "python",
-    "python-custom",
     "rust",
-    "unix",
-    "posix",
-    "winapi"
+    "shell",
+    "win32"
   ],
   // dictionaryDefinitions - this list defines any custom dictionaries to use
-  "dictionaryDefinitions": [],
+  "dictionaryDefinitions": [
+    {
+      "name": "cpython",
+      "path": "./.cspell.dict/cpython.txt"
+    },
+    {
+      "name": "python-more",
+      "path": "./.cspell.dict/python-more.txt"
+    },
+    {
+      "name": "rust-more",
+      "path": "./.cspell.dict/rust-more.txt"
+    },
+    {
+      "name": "rustpython",
+      "path": "./.cspell.dict/rustpython.txt"
+    }
+  ],
   "ignorePaths": [
     "**/__pycache__/**",
+    "target/**",
     "Lib/**"
   ],
   // words - list of words to be always considered correct
+  // (compound words like pyarg, baseclass, microbenchmark are handled by allowCompoundWords)
   "words": [
-    // Rust
-    "ahash",
-    "bidi",
-    "biguint",
-    "bindgen",
-    "bitflags",
-    "bstr",
-    "byteorder",
-    "chrono",
-    "consts",
-    "cstring",
-    "flate2",
-    "fract",
-    "hasher",
-    "idents",
-    "indexmap",
-    "insta",
-    "keccak",
-    "lalrpop",
-    "libc",
-    "libz",
-    "longlong",
-    "Manually",
-    "maplit",
-    "memmap",
-    "metas",
-    "modpow",
-    "nanos",
-    "objclass",
-    "peekable",
-    "powc",
-    "powf",
-    "prepended",
-    "punct",
-    "replacen",
-    "rsplitn",
-    "rustc",
-    "rustfmt",
-    "seekfrom",
-    "splitn",
-    "subsec",
-    "timsort",
-    "trai",
-    "ulonglong",
-    "unic",
-    "unistd",
-    "winapi",
-    "winsock",
-    // Python
-    "abstractmethods",
-    "aiter",
-    "anext",
-    "arrayiterator",
-    "arraytype",
-    "asend",
-    "athrow",
-    "basicsize",
-    "cformat",
-    "classcell",
-    "closesocket",
-    "codepoint",
-    "codepoints",
-    "cpython",
-    "decompressor",
-    "defaultaction",
-    "descr",
-    "dictcomp",
-    "dictitems",
-    "dictkeys",
-    "dictview",
-    "docstring",
-    "docstrings",
-    "dunder",
-    "eventmask",
-    "fdel",
-    "fget",
-    "fileencoding",
-    "fillchar",
-    "finallyhandler",
-    "frombytes",
-    "fromhex",
-    "fromunicode",
-    "fset",
-    "fspath",
-    "fstring",
-    "fstrings",
-    "genexpr",
-    "getattro",
-    "getformat",
-    "getnewargs",
-    "getweakrefcount",
-    "getweakrefs",
-    "hostnames",
-    "idiv",
-    "impls",
-    "infj",
-    "instancecheck",
-    "instanceof",
-    "isabstractmethod",
-    "itemiterator",
-    "itemsize",
-    "iternext",
-    "keyiterator",
-    "kwarg",
-    "kwargs",
-    "linearization",
-    "linearize",
-    "listcomp",
-    "mappingproxy",
-    "maxsplit",
-    "memoryview",
-    "memoryviewiterator",
-    "metaclass",
-    "metaclasses",
-    "metatype",
-    "mro",
-    "mros",
-    "nanj",
-    "ndigits",
-    "ndim",
-    "nonbytes",
-    "origname",
-    "posixsubprocess",
-    "pyexpat",
-    "PYTHONDEBUG",
-    "PYTHONHOME",
-    "PYTHONINSPECT",
-    "PYTHONOPTIMIZE",
-    "PYTHONPATH",
-    "PYTHONPATH",
-    "PYTHONVERBOSE",
-    "PYTHONWARNINGS",
-    "qualname",
-    "radd",
-    "rdiv",
-    "rdivmod",
-    "reconstructor",
-    "reversevalueiterator",
-    "rfloordiv",
-    "rlshift",
-    "rmod",
-    "rpow",
-    "rrshift",
-    "rsub",
-    "rtruediv",
-    "scproxy",
-    "setattro",
-    "setcomp",
-    "showwarnmsg",
-    "warnmsg",
-    "stacklevel",
-    "subclasscheck",
-    "subclasshook",
-    "unionable",
-    "unraisablehook",
-    "valueiterator",
-    "vararg",
-    "varargs",
-    "varnames",
-    "warningregistry",
-    "warnopts",
-    "weakproxy",
-    "xopts",
-    // RustPython
-    "baseclass",
-    "Bytecode",
-    "cfgs",
-    "codegen",
+    "aiterable",
+    "alnum",
+    "coro",
     "dedentations",
     "dedents",
     "deduped",
+    "deoptimize",
+    "downcastable",
     "downcasted",
     "dumpable",
-    "GetSet",
-    "internable",
-    "makeunicodedata",
-    "miri",
-    "notrace",
-    "pyarg",
-    "pyarg",
-    "pyargs",
-    "PyAttr",
+    "emscripten",
+    "excs",
+    "interps",
+    "jitted",
+    "jitting",
+    "kwonly",
+    "lossily",
+    "mcache",
+    "oparg",
     "pyc",
-    "PyClass",
-    "PyClassMethod",
-    "PyException",
-    "PyFunction",
-    "pygetset",
-    "pyimpl",
-    "pymember",
-    "PyMethod",
-    "PyModule",
-    "pyname",
-    "pyobj",
-    "PyObject",
-    "pypayload",
-    "PyProperty",
-    "pyref",
-    "PyResult",
-    "pyslot",
-    "PyStaticMethod",
-    "pystr",
-    "pystruct",
-    "pystructseq",
-    "pytrace",
-    "reducelib",
-    "richcompare",
-    "RustPython",
-    "struc",
-    "tracebacks",
-    "typealiases",
-    "Unconstructible",
-    "unhashable",
-    "uninit",
+    "significand",
+    "summands",
     "unraisable",
     "wasi",
-    "zelf",
-    // cpython
-    "argtypes",
-    "asdl",
-    "asname",
-    "augassign",
-    "badsyntax",
-    "basetype",
-    "boolop",
-    "bxor",
-    "cellarg",
-    "cellvar",
-    "cellvars",
-    "cmpop",
-    "dictoffset",
-    "elts",
-    "excepthandler",
-    "finalbody",
-    "freevar",
-    "freevars",
-    "fromlist",
-    "heaptype",
-    "IMMUTABLETYPE",
-    "kwonlyarg",
-    "kwonlyargs",
-    "linearise",
-    "maxdepth",
-    "mult",
-    "nkwargs",
-    "orelse",
-    "patma",
-    "posonlyarg",
-    "posonlyargs",
-    "prec",
-    "stackdepth",
-    "unaryop",
-    "unparse",
-    "unparser",
-    "VARKEYWORDS",
-    "varkwarg",
-    "wbits",
-    "withitem",
-    "withs"
+    "weaked",
+    // unix
+    "posixshmem",
+    "shm",
+    "CLOEXEC",
+    "endgrent",
+    "gethrvtime",
+    "getrusage",
+    "sigaction",
+    "WRLCK",
+    // win32
+    "IFEXEC"
   ],
   // flagWords - list of words to be always considered incorrect
   "flagWords": [

.devcontainer/Dockerfile

@@ -0,0 +1,6 @@
+FROM rust:bullseye
+
+# Install clang
+RUN apt-get update \
+  && apt-get install -y clang \
+  && rm -rf /var/lib/apt/lists/*

.devcontainer/devcontainer.json

@@ -1,6 +1,25 @@
 {
-  "image": "mcr.microsoft.com/devcontainers/universal:2",
-  "features": {
-    "ghcr.io/devcontainers/features/rust:1": {}
-  }
+  "name": "Rust",
+  "build": {
+    "dockerfile": "Dockerfile"
+  },
+  "runArgs": ["--cap-add=SYS_PTRACE", "--security-opt", "seccomp=unconfined"],
+  "customizations": {
+    "vscode": {
+      "settings": {
+        "lldb.executable": "/usr/bin/lldb",
+        // VS Code don't watch files under ./target
+        "files.watcherExclude": {
+          "**/target/**": true
+        },
+        "extensions": [
+          "rust-lang.rust-analyzer",
+          "tamasfe.even-better-toml",
+          "vadimcn.vscode-lldb",
+          "mutantdino.resourcemonitor"
+        ]
+      }
+    }
+  },
+  "remoteUser": "vscode"
 }

.gemini/config.yaml

@@ -0,0 +1,2 @@
+ignore_patterns:
+  - "Lib/**"

.gitattributes

@@ -1,6 +1,70 @@
 Lib/**      linguist-vendored
-Cargo.lock  linguist-generated -merge
+Cargo.lock  linguist-generated
 *.snap      linguist-generated -merge
 vm/src/stdlib/ast/gen.rs    linguist-generated -merge
 Lib/*.py    text working-tree-encoding=UTF-8 eol=LF
 **/*.rs     text working-tree-encoding=UTF-8 eol=LF
+crates/rustpython_doc_db/src/*.inc.rs   linguist-generated=true
+
+# Binary data types
+*.aif binary
+*.aifc binary
+*.aiff binary
+*.au binary
+*.bmp binary
+*.exe binary
+*.icns binary
+*.gif binary
+*.ico binary
+*.jpg binary
+*.pck binary
+*.pdf binary
+*.png binary
+*.psd binary
+*.tar binary
+*.wav binary
+*.whl binary
+*.zip binary
+
+# Text files that should not be subject to eol conversion
+[attr]noeol -text
+
+Lib/test/cjkencodings/*                    noeol
+Lib/test/tokenizedata/coding20731.py       noeol
+Lib/test/decimaltestdata/*.decTest         noeol
+Lib/test/test_email/data/*.txt             noeol
+Lib/test/xmltestdata/*                     noeol
+
+# Shell scripts should have LF even on Windows because of Cygwin
+Lib/venv/scripts/common/activate text eol=lf
+Lib/venv/scripts/posix/* text eol=lf
+
+# CRLF files
+[attr]dos text eol=crlf
+
+# Language aware diff headers
+# https://tekin.co.uk/2020/10/better-git-diff-output-for-ruby-python-elixir-and-more
+# https://gist.github.com/tekin/12500956bd56784728e490d8cef9cb81
+*.css   diff=css
+*.html  diff=html
+*.py    diff=python
+*.md    diff=markdown
+
+# Generated files
+# https://github.com/github/linguist/blob/master/docs/overrides.md
+#
+# To always hide generated files in local diffs, mark them as binary:
+# $ git config diff.generated.binary true
+#
+[attr]generated linguist-generated=true diff=generated
+
+Lib/_opcode_metadata.py                             generated
+Lib/keyword.py                                      generated
+Lib/idlelib/help.html                               generated
+Lib/test/certdata/*.pem                             generated
+Lib/test/certdata/*.0                               generated
+Lib/test/levenshtein_examples.json                  generated
+Lib/test/test_stable_abi_ctypes.py                  generated
+Lib/token.py                                        generated
+
+.github/workflows/*.lock.yml linguist-generated=true merge=ours

.github/actions/install-linux-deps/action.yml

@@ -0,0 +1,49 @@
+# This action installs a few dependencies necessary to build RustPython on Linux.
+# It can be configured depending on which libraries are needed:
+#
+# ```
+# - uses: ./.github/actions/install-linux-deps
+#   with:
+#     gcc-multilib: true
+#     musl-tools: false
+# ```
+#
+# See the `inputs` section for all options and their defaults. Note that you must checkout the
+# repository before you can use this action.
+#
+# This action will only install dependencies when the current operating system is Linux. It will do
+# nothing on any other OS (macOS, Windows).
+
+name: Install Linux dependencies
+description: Installs the dependencies necessary to build RustPython on Linux.
+inputs:
+  gcc-multilib:
+    description: Install gcc-multilib (gcc-multilib)
+    required: false
+    default: "false"
+  musl-tools:
+    description: Install musl-tools (musl-tools)
+    required: false
+    default: "false"
+  gcc-aarch64-linux-gnu:
+    description: Install gcc-aarch64-linux-gnu (gcc-aarch64-linux-gnu)
+    required: false
+    default: "false"
+  clang:
+    description: Install clang (clang)
+    required: false
+    default: "false"
+runs:
+  using: composite
+  steps:
+    - name: Install Linux dependencies
+      shell: bash
+      if: ${{ runner.os == 'Linux' }}
+      run: >
+        sudo apt-get update
+
+        sudo apt-get install --no-install-recommends
+        ${{ fromJSON(inputs.gcc-multilib) && 'gcc-multilib' || '' }}
+        ${{ fromJSON(inputs.musl-tools) && 'musl-tools' || '' }}
+        ${{ fromJSON(inputs.clang) && 'clang' || '' }}
+        ${{ fromJSON(inputs.gcc-aarch64-linux-gnu) && 'gcc-aarch64-linux-gnu linux-libc-dev-arm64-cross libc6-dev-arm64-cross' || '' }}

.github/actions/install-macos-deps/action.yml

@@ -0,0 +1,47 @@
+# This action installs a few dependencies necessary to build RustPython on macOS. By default it installs
+# autoconf, automake and libtool, but can be configured depending on which libraries are needed:
+#
+# ```
+# - uses: ./.github/actions/install-macos-deps
+#   with:
+#     openssl: true
+#     libtool: false
+# ```
+#
+# See the `inputs` section for all options and their defaults. Note that you must checkout the
+# repository before you can use this action.
+#
+# This action will only install dependencies when the current operating system is macOS. It will do
+# nothing on any other OS (Linux, Windows).
+
+name: Install macOS dependencies
+description: Installs the dependencies necessary to build RustPython on macOS.
+inputs:
+  autoconf:
+    description: Install autoconf (autoconf)
+    required: false
+    default: "true"
+  automake:
+    description: Install automake (automake)
+    required: false
+    default: "true"
+  libtool:
+    description: Install libtool (libtool)
+    required: false
+    default: "true"
+  openssl:
+    description: Install openssl (openssl@3)
+    required: false
+    default: "false"
+runs:
+  using: composite
+  steps:
+    - name: Install macOS dependencies
+      shell: bash
+      if: ${{ runner.os == 'macOS' }}
+      run: >
+        brew install
+        ${{ fromJSON(inputs.autoconf) && 'autoconf' || '' }}
+        ${{ fromJSON(inputs.automake) && 'automake' || '' }}
+        ${{ fromJSON(inputs.libtool) && 'libtool' || '' }}
+        ${{ fromJSON(inputs.openssl) && 'openssl@3' || '' }}

.github/aw/actions-lock.json

@@ -0,0 +1,14 @@
+{
+  "entries": {
+    "actions/github-script@v8": {
+      "repo": "actions/github-script",
+      "version": "v8",
+      "sha": "ed597411d8f924073f98dfc5c65a23a2325f34cd"
+    },
+    "github/gh-aw/actions/setup@v0.43.22": {
+      "repo": "github/gh-aw/actions/setup",
+      "version": "v0.43.22",
+      "sha": "fe858c3e14589bf396594a0b106e634d9065823e"
+    }
+  }
+}

.github/dependabot.yml

@@ -1,13 +1,149 @@
-# Keep GitHub Actions up to date with GitHub's Dependabot...
-# https://docs.github.com/en/code-security/dependabot/working-with-dependabot/keeping-your-actions-up-to-date-with-dependabot
-# https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#package-ecosystem
+# cspell:ignore manyhow tinyvec zeroize
 version: 2
 updates:
-  - package-ecosystem: github-actions
+  - package-ecosystem: cargo
+    directory: /
+    schedule:
+      interval: weekly
+    groups:
+      criterion:
+        patterns:
+          - "criterion*"
+      crypto:
+        patterns:
+          - "digest"
+          - "md-5"
+          - "sha-1"
+          - "sha2"
+          - "sha3"
+          - "blake2"
+          - "hmac"
+          - "pbkdf2"
+      futures:
+        patterns:
+          - "futures*"
+      get-size2:
+        patterns:
+          - "get-size*2"
+      iana-time-zone:
+        patterns:
+          - "iana-time-zone*"
+      jiff:
+        patterns:
+          - "jiff*"
+      lexical:
+        patterns:
+          - "lexical*"
+      libffi:
+        patterns:
+          - "libffi*"
+      malachite:
+        patterns:
+          - "malachite*"
+      manyhow:
+        patterns:
+          - "manyhow*"
+      num:
+        patterns:
+          - "num-bigint"
+          - "num-complex"
+          - "num-integer"
+          - "num-iter"
+          - "num-rational"
+          - "num-traits"
+      num_enum:
+        patterns:
+          - "num_enum*"
+      openssl:
+        patterns:
+          - "openssl*"
+      parking_lot:
+        patterns:
+          - "parking_lot*"
+      phf:
+        patterns:
+          - "phf*"
+      plotters:
+        patterns:
+          - "plotters*"
+      portable-atomic:
+        patterns:
+          - "portable-atomic*"
+      pyo3:
+        patterns:
+          - "pyo3*"
+      quote-use:
+        patterns:
+          - "quote-use*"
+      random:
+        patterns:
+          - "getrandom"
+          - "mt19937"
+          - "rand*"
+      rayon:
+        patterns:
+          - "rayon*"
+      regex:
+        patterns:
+          - "regex*"
+      result-like:
+        patterns:
+          - "result-like*"
+      security-framework:
+        patterns:
+          - "security-framework*"
+      serde:
+        patterns:
+          - "serde"
+          - "serde_core"
+          - "serde_derive"
+      system-configuration:
+        patterns:
+          - "system-configuration*"
+      thiserror:
+        patterns:
+          - "thiserror*"
+      time:
+        patterns:
+          - "time*"
+      tinyvec:
+        patterns:
+          - "tinyvec*"
+      tls_codec:
+        patterns:
+          - "tls_codec*"
+      toml:
+        patterns:
+          - "toml*"
+      wasm-bindgen:
+        patterns:
+          - "wasm-bindgen*"
+      wasmtime:
+        patterns:
+          - "cranelift*"
+          - "wasmtime*"
+      webpki-root:
+        patterns:
+          - "webpki-root*"
+      windows:
+        patterns:
+          - "windows*"
+      zerocopy:
+        patterns:
+          - "zerocopy*"
+      zeroize:
+        patterns:
+          - "zeroize*"
+    ignore:
+      # TODO: Remove when we use ruff from crates.io
+      # for some reason dependabot only updates the Cargo.lock file when dealing
+      # with git dependencies. i.e. not updating the version in Cargo.toml
+      - dependency-name: "ruff_*"
+  - package-ecosystem: github-actions
+    directory: /
+    schedule:
+      interval: weekly
+  - package-ecosystem: npm
     directory: /
-    groups:
-      github-actions:
-        patterns:
-          - "*"  # Group all Actions updates into a single larger pull request
     schedule:
       interval: weekly

.github/workflows/ci.yaml

@@ -4,6 +4,7 @@ on:
   pull_request:
     types: [unlabeled, opened, synchronize, reopened]
   merge_group:
+  workflow_dispatch:
 
 name: CI
 
@@ -15,97 +16,17 @@ concurrency:
   cancel-in-progress: true
 
 env:
-  CARGO_ARGS: --no-default-features --features stdlib,zlib,importlib,encodings,sqlite,ssl
-  # Skip additional tests on Windows. They are checked on Linux and MacOS.
-  WINDOWS_SKIPS: >-
-    test_datetime
-    test_glob
-    test_importlib
-    test_io
-    test_os
-    test_pathlib
-    test_posixpath
-    test_venv
-  # configparser: https://github.com/RustPython/RustPython/issues/4995#issuecomment-1582397417
-  # socketserver: seems related to configparser crash.
-  MACOS_SKIPS: >-
-    test_configparser
-    test_socketserver
-  # PLATFORM_INDEPENDENT_TESTS are tests that do not depend on the underlying OS. They are currently
-  # only run on Linux to speed up the CI.
-  PLATFORM_INDEPENDENT_TESTS: >-
-    test_argparse
-    test_array
-    test_asyncgen
-    test_binop
-    test_bisect
-    test_bool
-    test_bytes
-    test_call
-    test_class
-    test_cmath
-    test_collections
-    test_complex
-    test_contains
-    test_copy
-    test_dataclasses
-    test_decimal
-    test_decorators
-    test_defaultdict
-    test_deque
-    test_dict
-    test_dictcomps
-    test_dictviews
-    test_dis
-    test_enumerate
-    test_exception_variations
-    test_exceptions
-    test_float
-    test_format
-    test_fractions
-    test_genericalias
-    test_genericclass
-    test_grammar
-    test_range
-    test_index
-    test_int
-    test_int_literal
-    test_isinstance
-    test_iter
-    test_iterlen
-    test_itertools
-    test_json
-    test_keyword
-    test_keywordonlyarg
-    test_list
-    test_long
-    test_longexp
-    test_math
-    test_operator
-    test_ordered_dict
-    test_pow
-    test_raise
-    test_richcmp
-    test_scope
-    test_set
-    test_slice
-    test_sort
-    test_string
-    test_string_literals
-    test_strtod
-    test_structseq
-    test_subclassinit
-    test_super
-    test_syntax
-    test_tuple
-    test_types
-    test_unary
-    test_unicode
-    test_unpack
-    test_weakref
-    test_yield_from
+  CARGO_ARGS: --no-default-features --features stdlib,importlib,stdio,encodings,sqlite,ssl-rustls,host_env
+  CARGO_ARGS_NO_SSL: --no-default-features --features stdlib,importlib,stdio,encodings,sqlite,host_env
+  # Crates excluded from workspace builds:
+  # - rustpython_wasm: requires wasm target
+  # - rustpython-compiler-source: deprecated
+  # - rustpython-venvlauncher: Windows-only
+  WORKSPACE_EXCLUDES: --exclude rustpython_wasm --exclude rustpython-compiler-source --exclude rustpython-venvlauncher
   # Python version targeted by the CI.
-  PYTHON_VERSION: "3.12.3"
+  PYTHON_VERSION: "3.14.3"
+  X86_64_PC_WINDOWS_MSVC_OPENSSL_LIB_DIR: C:\Program Files\OpenSSL\lib\VC\x64\MD
+  X86_64_PC_WINDOWS_MSVC_OPENSSL_INCLUDE_DIR: C:\Program Files\OpenSSL\include
 
 jobs:
   rust_tests:
@@ -113,274 +34,479 @@ jobs:
     env:
       RUST_BACKTRACE: full
     name: Run rust tests
-    runs-on:  ${{ matrix.os }}
+    runs-on: ${{ matrix.os }}
+    timeout-minutes: 45
     strategy:
       matrix:
-        os: [macos-latest, ubuntu-latest, windows-latest]
+        os: [macos-latest, ubuntu-latest, windows-2025]
       fail-fast: false
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+
       - uses: dtolnay/rust-toolchain@stable
         with:
           components: clippy
-      - uses: Swatinem/rust-cache@v2
 
-      - name: Set up the Windows environment
-        shell: bash
-        run: |
-          cargo install --target-dir=target -v cargo-vcpkg
-          cargo vcpkg -v build
-        if: runner.os == 'Windows'
-      - name: Set up the Mac environment
-        run: brew install autoconf automake libtool
-        if: runner.os == 'macOS'
+      - uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
+        with:
+          save-if: ${{ github.ref == 'refs/heads/main' }}
+
+      - name: Install macOS dependencies
+        uses: ./.github/actions/install-macos-deps
 
       - name: run clippy
-        run: cargo clippy ${{ env.CARGO_ARGS }} --workspace  --exclude rustpython_wasm -- -Dwarnings
+        run: cargo clippy ${{ env.CARGO_ARGS }} --workspace --all-targets ${{ env.WORKSPACE_EXCLUDES }} -- -Dwarnings
 
       - name: run rust tests
-        run: cargo test --workspace --exclude rustpython_wasm --verbose --features threading ${{ env.CARGO_ARGS }}
-        if: runner.os != 'macOS'
-      - name: run rust tests
-        run: cargo test --workspace --exclude rustpython_wasm --exclude rustpython-jit --verbose --features threading  ${{ env.CARGO_ARGS }}
-        if: runner.os == 'macOS'
+        run: cargo test --workspace ${{ env.WORKSPACE_EXCLUDES }} --verbose --features threading ${{ env.CARGO_ARGS }}
 
       - name: check compilation without threading
         run: cargo check ${{ env.CARGO_ARGS }}
 
+      - name: check compilation without host_env (sandbox mode)
+        run: |
+          cargo check -p rustpython-vm --no-default-features --features compiler
+          cargo check -p rustpython-stdlib --no-default-features --features compiler
+          cargo build --no-default-features --features stdlib,importlib,stdio,encodings,freeze-stdlib
+        if: runner.os == 'Linux'
+
+      - name: sandbox smoke test
+        run: |
+          target/debug/rustpython extra_tests/snippets/sandbox_smoke.py
+          target/debug/rustpython extra_tests/snippets/stdlib_re.py
+        if: runner.os == 'Linux'
+
+      - name: Test openssl build
+        run: cargo build --no-default-features --features ssl-openssl
+        if: runner.os == 'Linux'
+
+      # - name: Install tk-dev for tkinter build
+      #   run: sudo apt-get update && sudo apt-get install -y tk-dev
+      #   if: runner.os == 'Linux'
+
+      # - name: Test tkinter build
+      #   run: cargo build --features tkinter
+      #   if: runner.os == 'Linux'
+
       - name: Test example projects
-        run:
+        run: |
           cargo run --manifest-path example_projects/barebone/Cargo.toml
           cargo run --manifest-path example_projects/frozen_stdlib/Cargo.toml
         if: runner.os == 'Linux'
 
-      - name: prepare AppleSilicon build
-        uses: dtolnay/rust-toolchain@stable
-        with:
-          target: aarch64-apple-darwin
-        if: runner.os == 'macOS'
-      - name: Check compilation for Apple Silicon
-        run: cargo check --target aarch64-apple-darwin
-        if: runner.os == 'macOS'
-      - name: prepare iOS build
-        uses: dtolnay/rust-toolchain@stable
-        with:
-          target: aarch64-apple-ios
-        if: runner.os == 'macOS'
-      - name: Check compilation for iOS
-        run: cargo check --target aarch64-apple-ios
-        if: runner.os == 'macOS'
+      - name: run update_lib tests
+        run: cargo run -- -m unittest discover -s scripts/update_lib/tests -v
+        env:
+          PYTHONPATH: scripts
+        if: runner.os == 'Linux'
 
-  exotic_targets:
+  cargo_check:
     if: ${{ !contains(github.event.pull_request.labels.*.name, 'skip:ci') }}
     name: Ensure compilation on various targets
-    runs-on: ubuntu-latest
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        include: 
+        - os: ubuntu-latest
+          targets:
+            - aarch64-linux-android
+            - i686-unknown-linux-gnu
+            - i686-unknown-linux-musl
+            - wasm32-wasip2
+            - x86_64-unknown-freebsd
+          dependencies:
+            gcc-multilib: true
+            musl-tools: true
+        - os: ubuntu-latest
+          targets:
+            - aarch64-unknown-linux-gnu
+          dependencies:
+            gcc-aarch64-linux-gnu: true # conflict with `gcc-multilib`
+        - os: macos-latest
+          targets:
+            - aarch64-apple-ios
+            - x86_64-apple-darwin
+      fail-fast: false
     steps:
-      - uses: actions/checkout@v4
-      - uses: dtolnay/rust-toolchain@stable
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
         with:
-          target: i686-unknown-linux-gnu
+          persist-credentials: false
 
-      - name: Install gcc-multilib and musl-tools
-        run: sudo apt-get update && sudo apt-get install gcc-multilib musl-tools
-      - name: Check compilation for x86 32bit
-        run: cargo check --target i686-unknown-linux-gnu
+      - uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
+        with:
+          prefix-key: v0-rust-${{ join(matrix.targets, '-') }}
+          save-if: ${{ github.ref == 'refs/heads/main' }}
+
+      - name: Install dependencies
+        uses: ./.github/actions/install-linux-deps
+        with: ${{ matrix.dependencies || fromJSON('{}') }}
 
       - uses: dtolnay/rust-toolchain@stable
         with:
-          target: aarch64-linux-android
+          targets: ${{ join(matrix.targets, ',') }}
 
-      - name: Check compilation for android
-        run: cargo check --target aarch64-linux-android
-
-      - uses: dtolnay/rust-toolchain@stable
+      - name: Setup Android NDK
+        if: ${{ contains(matrix.targets, 'aarch64-linux-android') }}
+        id: setup-ndk
+        uses: nttld/setup-ndk@v1
         with:
-          target: aarch64-unknown-linux-gnu
+          ndk-version: r27
+          add-to-path: true
 
-      - name: Install gcc-aarch64-linux-gnu
-        run: sudo apt install gcc-aarch64-linux-gnu
-      - name: Check compilation for aarch64 linux gnu
-        run: cargo check --target aarch64-unknown-linux-gnu
+      # - name: Prepare repository for redox compilation
+      #   run: bash scripts/redox/uncomment-cargo.sh
+      # - name: Check compilation for Redox
+      #   uses: coolreader18/redoxer-action@v1
+      #   with:
+      #     command: check
+      #     args: --ignore-rust-version
 
-      - uses: dtolnay/rust-toolchain@stable
-        with:
-          target: i686-unknown-linux-musl
-
-      - name: Check compilation for musl
-        run: cargo check --target i686-unknown-linux-musl
-
-      - uses: dtolnay/rust-toolchain@stable
-        with:
-          target: x86_64-unknown-freebsd
-
-      - name: Check compilation for freebsd
-        run: cargo check --target x86_64-unknown-freebsd
-
-      - uses: dtolnay/rust-toolchain@stable
-        with:
-          target: x86_64-unknown-freebsd
-
-      - name: Check compilation for freeBSD
-        run: cargo check --target x86_64-unknown-freebsd
-
-      - name: Prepare repository for redox compilation
-        run: bash scripts/redox/uncomment-cargo.sh
-      - name: Check compilation for Redox
-        uses: coolreader18/redoxer-action@v1
-        with:
-          command: check
+      - name: Check compilation
+        run: |
+          for target in ${{ join(matrix.targets, ' ') }}
+          do
+            echo "::group::${target}"
+            cargo check --target $target ${{ env.CARGO_ARGS_NO_SSL }}
+            echo "::endgroup::"
+          done
+        env:
+          CC_aarch64_linux_android: ${{ steps.setup-ndk.outputs.ndk-path }}/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android24-clang
+          AR_aarch64_linux_android: ${{ steps.setup-ndk.outputs.ndk-path }}/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-ar
+          CARGO_TARGET_AARCH64_LINUX_ANDROID_LINKER: ${{ steps.setup-ndk.outputs.ndk-path }}/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android24-clang
 
   snippets_cpython:
     if: ${{ !contains(github.event.pull_request.labels.*.name, 'skip:ci') }}
     env:
       RUST_BACKTRACE: full
+      # PLATFORM_INDEPENDENT_TESTS are tests that do not depend on the underlying OS.
+      # They are currently only run on Linux to speed up the CI.
+      PLATFORM_INDEPENDENT_TESTS: >-
+        test__colorize
+        test_array
+        test_asyncgen
+        test_binop
+        test_bisect
+        test_bool
+        test_bytes
+        test_call
+        test_cmath
+        test_collections
+        test_complex
+        test_contains
+        test_copy
+        test_dataclasses
+        test_decimal
+        test_decorators
+        test_defaultdict
+        test_deque
+        test_dict
+        test_dictcomps
+        test_dictviews
+        test_dis
+        test_enumerate
+        test_exception_variations
+        test_float
+        test_fractions
+        test_genericalias
+        test_genericclass
+        test_grammar
+        test_range
+        test_index
+        test_int
+        test_int_literal
+        test_isinstance
+        test_iter
+        test_iterlen
+        test_itertools
+        test_json
+        test_keyword
+        test_keywordonlyarg
+        test_list
+        test_long
+        test_longexp
+        test_operator
+        test_ordered_dict
+        test_pep646_syntax
+        test_pow
+        test_raise
+        test_richcmp
+        test_scope
+        test_set
+        test_slice
+        test_sort
+        test_string
+        test_string_literals
+        test_strtod
+        test_structseq
+        test_subclassinit
+        test_super
+        test_syntax
+        test_tstring
+        test_tuple
+        test_unary
+        test_unpack
+        test_unpack_ex
+        test_weakref
+        test_yield_from
     name: Run snippets and cpython tests
-    runs-on:  ${{ matrix.os }}
+    runs-on: ${{ matrix.os }}
     strategy:
       matrix:
-        os: [macos-latest, ubuntu-latest, windows-latest]
+        include:
+        - os: macos-latest
+          extra_test_args:
+            - '-u all'
+          env_polluting_tests: []
+          skips: []
+          timeout: 50
+        - os: ubuntu-latest
+          extra_test_args:
+            - '-u all'
+          env_polluting_tests: []
+          skips: []
+          timeout: 60
+        - os: windows-2025
+          extra_test_args: [] # TODO: Enable '-u all'
+          env_polluting_tests: []
+          skips:
+            - test_rlcompleter
+            - test_pathlib # panic by surrogate chars
+            - test_posixpath # OSError: (22, 'The filename, directory name, or volume label syntax is incorrect. (os error 123)')
+            - test_venv # couple of failing tests
+          timeout: 50
       fail-fast: false
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+
       - uses: dtolnay/rust-toolchain@stable
-      - uses: Swatinem/rust-cache@v2
-      - uses: actions/setup-python@v5
+
+      - uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
+        with:
+          save-if: ${{ github.ref == 'refs/heads/main' }}
+
+      - uses: actions/setup-python@v6.2.0
         with:
           python-version: ${{ env.PYTHON_VERSION }}
-      - name: Set up the Windows environment
-        shell: bash
-        run: |
-          cargo install cargo-vcpkg
-          cargo vcpkg build
-        if: runner.os == 'Windows'
-      - name: Set up the Mac environment
-        run: brew install autoconf automake libtool openssl@3
-        if: runner.os == 'macOS'
-      - name: build rustpython
-        run: cargo build --release --verbose --features=threading ${{ env.CARGO_ARGS }}
-        if: runner.os == 'macOS'
-      - name: build rustpython
-        run: cargo build --release --verbose --features=threading ${{ env.CARGO_ARGS }},jit
-        if: runner.os != 'macOS'
-      - uses: actions/setup-python@v5
+
+      - name: Install macOS dependencies
+        uses: ./.github/actions/install-macos-deps
         with:
-          python-version: ${{ env.PYTHON_VERSION }}
+          openssl: true
+
+      - name: build rustpython
+        run: cargo build --release --verbose --features=threading,jit ${{ env.CARGO_ARGS }}
+
       - name: run snippets
         run: python -m pip install -r requirements.txt && pytest -v
         working-directory: ./extra_tests
-      - if: runner.os == 'Linux'
-        name: run cpython platform-independent tests
-        run:
-          target/release/rustpython -m test -j 1 -u all --slowest --fail-env-changed -v ${{ env.PLATFORM_INDEPENDENT_TESTS }}
-      - if: runner.os == 'Linux'
-        name: run cpython platform-dependent tests (Linux)
-        run: target/release/rustpython -m test -j 1 -u all --slowest --fail-env-changed -v -x ${{ env.PLATFORM_INDEPENDENT_TESTS }}
-      - if: runner.os == 'macOS'
-        name: run cpython platform-dependent tests (MacOS)
-        run: target/release/rustpython -m test -j 1 --slowest --fail-env-changed -v -x ${{ env.PLATFORM_INDEPENDENT_TESTS }} ${{ env.MACOS_SKIPS }}
-      - if: runner.os == 'Windows'
-        name: run cpython platform-dependent tests (windows partial - fixme)
-        run:
-          target/release/rustpython -m test -j 1 --slowest --fail-env-changed -v -x ${{ env.PLATFORM_INDEPENDENT_TESTS }} ${{ env.WINDOWS_SKIPS }}
+
+      - name: run cpython platform-independent tests
+        if: runner.os == 'Linux'
+        run: |
+          target/release/rustpython -m test -j 1 -u all --slowest --fail-env-changed --timeout 600 -v ${{ env.PLATFORM_INDEPENDENT_TESTS }}
+        timeout-minutes: 45
+        env:
+          RUSTPYTHON_SKIP_ENV_POLLUTERS: true
+
+      - name: run cpython platform-dependent tests
+        run: |
+          target/release/rustpython -m test -j 1 ${{ join(matrix.extra_test_args, ' ') }} --slowest --fail-env-changed --timeout 600 -v -x ${{ env.PLATFORM_INDEPENDENT_TESTS }} ${{ join(matrix.skips, ' ') }}
+        timeout-minutes: ${{ matrix.timeout }}
+        env:
+          RUSTPYTHON_SKIP_ENV_POLLUTERS: true
+
+      - name: run cpython tests to check if env polluters have stopped polluting
+        shell: bash
+        run: |
+          for thing in ${{ join(matrix.env_polluting_tests, ' ') }}; do
+            for i in $(seq 1 10); do
+              set +e
+              target/release/rustpython -m test -j 1 --slowest --fail-env-changed --timeout 600 -v ${thing}
+              exit_code=$?
+              set -e
+              if [ ${exit_code} -eq 3 ]; then
+                echo "Test ${thing} polluted the environment on attempt ${i}."
+                break
+              fi
+            done
+            if [ ${exit_code} -ne 3 ]; then
+              echo "Test ${thing} is no longer polluting the environment after ${i} attempts!"
+              echo "Please remove ${thing} from matrix.env_polluting_tests in '.github/workflows/ci.yaml'."
+              echo "Please also remove the skip decorators that include the word 'POLLUTERS' in ${thing}."
+              if [ ${exit_code} -ne 0 ]; then
+                echo "Test ${thing} failed with exit code ${exit_code}."
+                echo "Please investigate which test item in ${thing} is failing and either mark it as an expected failure or a skip."
+              fi
+              exit 1
+            fi
+          done
+        timeout-minutes: 15
+
       - if: runner.os != 'Windows'
         name: check that --install-pip succeeds
         run: |
           mkdir site-packages
           target/release/rustpython --install-pip ensurepip --user
           target/release/rustpython -m pip install six
-      - if: runner.os != 'Windows'
-        name: Check that ensurepip succeeds.
+
+      - name: Check that ensurepip succeeds.
         run: |
           target/release/rustpython -m ensurepip
           target/release/rustpython -c "import pip"
+
       - if: runner.os != 'Windows'
         name: Check if pip inside venv is functional
         run: |
           target/release/rustpython -m venv testvenv
           testvenv/bin/rustpython -m pip install wheel
+
       - name: Check whats_left is not broken
-        run: python -I whats_left.py
+        shell: bash
+        run: python -I scripts/whats_left.py ${{ env.CARGO_ARGS }} --features jit
 
   lint:
-    name: Check Rust code with rustfmt and clippy
+    name: Lint Rust & Python code
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v4
-      - uses: dtolnay/rust-toolchain@stable
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
         with:
-            components: rustfmt, clippy
-      - name: run rustfmt
-        run: cargo fmt --check
-      - name: run clippy on wasm
-        run: cargo clippy --manifest-path=wasm/lib/Cargo.toml -- -Dwarnings
-      - uses: actions/setup-python@v5
+          persist-credentials: false
+
+      - uses: actions/setup-python@v6.2.0
         with:
           python-version: ${{ env.PYTHON_VERSION }}
-      - name: install ruff
-        run: python -m pip install ruff==0.0.291 # astral-sh/ruff#7778
-      - name: run python lint
-        run: ruff extra_tests wasm examples  --exclude='./.*',./Lib,./vm/Lib,./benches/ --select=E9,F63,F7,F82 --show-source
+
+      - name: Check for redundant test patches
+        run: python scripts/check_redundant_patches.py
+
+      - uses: dtolnay/rust-toolchain@stable
+        with:
+          components: clippy
+
+      - name: run clippy on wasm
+        run: cargo clippy --manifest-path=crates/wasm/Cargo.toml -- -Dwarnings
+
+      - name: Ensure docs generate no warnings
+        run: cargo doc --locked
+
+      - name: Ensure Lib/_opcode_metadata is updated
+        run: |
+          python scripts/generate_opcode_metadata.py
+          if [ -n "$(git status --porcelain)" ]; then
+            exit 1
+          fi
+
+      - name: Install ruff
+        uses: astral-sh/ruff-action@4919ec5cf1f49eff0871dbcea0da843445b837e6 # v3.6.1
+        with:
+          version: "0.15.5"
+          args: "--version"
+
+      - run: ruff check --diff
+
+      - run: ruff format --check
+
       - name: install prettier
-        run: yarn global add prettier && echo "$(yarn global bin)" >>$GITHUB_PATH
+        run: |
+          yarn global add prettier
+          yarn global bin >> "$GITHUB_PATH"
+
       - name: check wasm code with prettier
         # prettier doesn't handle ignore files very well: https://github.com/prettier/prettier/issues/8506
         run: cd wasm && git ls-files -z | xargs -0 prettier --check -u
+      # Keep cspell check as the last step. This is optional test.
+      - name: install extra dictionaries
+        run: npm install @cspell/dict-en_us @cspell/dict-cpp @cspell/dict-python @cspell/dict-rust @cspell/dict-win32 @cspell/dict-shell
+      - name: spell checker
+        uses: streetsidesoftware/cspell-action@v8
+        with:
+          files: "**/*.rs"
+          incremental_files_only: true
 
   miri:
     if: ${{ !contains(github.event.pull_request.labels.*.name, 'skip:ci') }}
     name: Run tests under miri
     runs-on: ubuntu-latest
+    timeout-minutes: 30
+    env:
+      NIGHTLY_CHANNEL: nightly
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+
       - uses: dtolnay/rust-toolchain@master
         with:
-            toolchain: nightly
-            components: miri
+          toolchain: ${{ env.NIGHTLY_CHANNEL }}
+          components: miri
+
+      - uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
+        with:
+          save-if: ${{ github.ref == 'refs/heads/main' }}
 
-      - uses: Swatinem/rust-cache@v2
       - name: Run tests under miri
-        # miri-ignore-leaks because the type-object circular reference means that there will always be
-        # a memory leak, at least until we have proper cyclic gc
-        run: MIRIFLAGS='-Zmiri-ignore-leaks' cargo +nightly miri test -p rustpython-vm -- miri_test
+        run: cargo +${{ env.NIGHTLY_CHANNEL }} miri test -p rustpython-vm -- miri_test
+        env:
+          # miri-ignore-leaks because the type-object circular reference means that there will always be
+          # a memory leak, at least until we have proper cyclic gc
+          MIRIFLAGS: "-Zmiri-ignore-leaks"
 
   wasm:
     if: ${{ !contains(github.event.pull_request.labels.*.name, 'skip:ci') }}
     name: Check the WASM package and demo
     runs-on: ubuntu-latest
+    timeout-minutes: 30
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+
       - uses: dtolnay/rust-toolchain@stable
 
-      - uses: Swatinem/rust-cache@v2
+      - uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
+        with:
+          save-if: ${{ github.ref == 'refs/heads/main' }}
+
       - name: install wasm-pack
         run: curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh
       - name: install geckodriver
         run: |
-          wget https://github.com/mozilla/geckodriver/releases/download/v0.34.0/geckodriver-v0.34.0-linux64.tar.gz
+          wget https://github.com/mozilla/geckodriver/releases/download/v0.36.0/geckodriver-v0.36.0-linux64.tar.gz
           mkdir geckodriver
-          tar -xzf geckodriver-v0.34.0-linux64.tar.gz -C geckodriver
-      - uses: actions/setup-python@v5
+          tar -xzf geckodriver-v0.36.0-linux64.tar.gz -C geckodriver
+      - uses: actions/setup-python@v6.2.0
         with:
           python-version: ${{ env.PYTHON_VERSION }}
       - run: python -m pip install -r requirements.txt
         working-directory: ./wasm/tests
-      - uses: actions/setup-node@v4
+      - uses: actions/setup-node@v6
+        with:
+          cache: "npm"
+          cache-dependency-path: "wasm/demo/package-lock.json"
       - name: run test
         run: |
-          export PATH=$PATH:`pwd`/../../geckodriver
+          driver_path="$(pwd)/../../geckodriver"
+          export PATH="$PATH:${driver_path}"
           npm install
           npm run test
         env:
           NODE_OPTIONS: "--openssl-legacy-provider"
         working-directory: ./wasm/demo
       - uses: mwilliamson/setup-wabt-action@v3
-        with: { wabt-version: "1.0.30" }
+        with: { wabt-version: "1.0.36" }
       - name: check wasm32-unknown without js
         run: |
-          cargo build --release --manifest-path wasm/wasm-unknown-test/Cargo.toml --target wasm32-unknown-unknown --verbose
-          if wasm-objdump -xj Import target/wasm32-unknown-unknown/release/wasm_unknown_test.wasm; then
-            echo "ERROR: wasm32-unknown module expects imports from the host environment" >2
+          cd example_projects/wasm32_without_js/rustpython-without-js
+          cargo build
+          cd ..
+          if wasm-objdump -xj Import rustpython-without-js/target/wasm32-unknown-unknown/debug/rustpython_without_js.wasm; then
+            echo "ERROR: wasm32-unknown module expects imports from the host environment" >&2
           fi
+          cargo run --release --manifest-path wasm-runtime/Cargo.toml rustpython-without-js/target/wasm32-unknown-unknown/debug/rustpython_without_js.wasm
       - name: build notebook demo
         if: github.ref == 'refs/heads/release'
         run: |
@@ -403,20 +529,31 @@ jobs:
     if: ${{ !contains(github.event.pull_request.labels.*.name, 'skip:ci') }}
     name: Run snippets and cpython tests on wasm-wasi
     runs-on: ubuntu-latest
+    timeout-minutes: 30
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+
       - uses: dtolnay/rust-toolchain@stable
         with:
-          target: wasm32-wasi
+          target: wasm32-wasip1
+
+      - uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
+        with:
+          save-if: ${{ github.ref == 'refs/heads/main' }}
 
-      - uses: Swatinem/rust-cache@v2
       - name: Setup Wasmer
         uses: wasmerio/setup-wasmer@v3
+
       - name: Install clang
-        run: sudo apt-get update && sudo apt-get install clang -y
+        uses: ./.github/actions/install-linux-deps
+        with:
+          clang: true
+
       - name: build rustpython
-        run: cargo build --release --target wasm32-wasi --features freeze-stdlib,stdlib --verbose
+        run: cargo build --release --target wasm32-wasip1 --features freeze-stdlib,stdlib --verbose
       - name: run snippets
-        run: wasmer run --dir `pwd` target/wasm32-wasi/release/rustpython.wasm -- `pwd`/extra_tests/snippets/stdlib_random.py
+        run: wasmer run --dir $(pwd) target/wasm32-wasip1/release/rustpython.wasm -- "$(pwd)/extra_tests/snippets/stdlib_random.py"
       - name: run cpython unittest
-        run: wasmer run --dir `pwd` target/wasm32-wasi/release/rustpython.wasm -- `pwd`/Lib/test/test_int.py
+        run: wasmer run --dir $(pwd) target/wasm32-wasip1/release/rustpython.wasm -- "$(pwd)/Lib/test/test_int.py"

.github/workflows/comment-commands.yml

@@ -0,0 +1,21 @@
+name: Comment Commands
+
+on:
+  issue_comment:
+    types: created
+
+jobs:
+  issue_assign:
+    if: (!github.event.issue.pull_request) && github.event.comment.body == 'take'
+    runs-on: ubuntu-slim
+
+    concurrency:
+      group: ${{ github.actor }}-issue-assign
+
+    permissions:
+      issues: write
+
+    steps:
+      # Using REST API and not `gh issue edit`. https://github.com/cli/cli/issues/6235#issuecomment-1243487651
+      - run: |
+          curl -H "Authorization: token ${{ github.token }}" -d '{"assignees": ["${{ github.event.comment.user.login }}"]}' https://api.github.com/repos/${{ github.repository }}/issues/${{ github.event.issue.number }}/assignees

.github/workflows/cron-ci.yaml

@@ -1,13 +1,19 @@
 on:
   schedule:
-    - cron: '0 0 * * 6'
+    - cron: "0 0 * * 6"
   workflow_dispatch:
+  push:
+    paths:
+      - .github/workflows/cron-ci.yaml
+  pull_request:
+    paths:
+      - .github/workflows/cron-ci.yaml
 
 name: Periodic checks/tasks
 
 env:
-  CARGO_ARGS: --no-default-features --features stdlib,zlib,importlib,encodings,ssl,jit
-  PYTHON_VERSION: "3.12.0"
+  CARGO_ARGS: --no-default-features --features stdlib,importlib,stdio,encodings,ssl-rustls,jit,host_env
+  PYTHON_VERSION: "3.14.3"
 
 jobs:
   # codecov collects code coverage data from the rust tests, python snippets and python test suite.
@@ -15,25 +21,31 @@ jobs:
   codecov:
     name: Collect code coverage data
     runs-on: ubuntu-latest
+    # Disable this scheduled job when running on a fork.
+    if: ${{ github.repository == 'RustPython/RustPython' || github.event_name != 'schedule' }}
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+
       - uses: dtolnay/rust-toolchain@stable
       - uses: taiki-e/install-action@cargo-llvm-cov
-      - uses: actions/setup-python@v5
+      - uses: actions/setup-python@v6.2.0
         with:
           python-version: ${{ env.PYTHON_VERSION }}
       - run: sudo apt-get update && sudo apt-get -y install lcov
       - name: Run cargo-llvm-cov with Rust tests.
-        run: cargo llvm-cov --no-report --workspace --exclude rustpython_wasm --verbose --no-default-features --features stdlib,zlib,importlib,encodings,ssl,jit
+        run: cargo llvm-cov --no-report --workspace --exclude rustpython_wasm --exclude rustpython-compiler-source --exclude rustpython-venvlauncher --verbose --no-default-features --features stdlib,importlib,stdio,encodings,ssl-rustls,jit
       - name: Run cargo-llvm-cov with Python snippets.
         run: python scripts/cargo-llvm-cov.py
         continue-on-error: true
       - name: Run cargo-llvm-cov with Python test suite.
-        run: cargo llvm-cov --no-report run -- -m test -u all --slowest --fail-env-changed 
+        run: cargo llvm-cov --no-report run -- -m test -u all --slowest --fail-env-changed
         continue-on-error: true
       - name: Prepare code coverage data
         run: cargo llvm-cov report --lcov --output-path='codecov.lcov'
       - name: Upload to Codecov
+        if: ${{ github.event_name != 'pull_request' }}
         uses: codecov/codecov-action@v5
         with:
           file: ./codecov.lcov
@@ -41,8 +53,13 @@ jobs:
   testdata:
     name: Collect regression test data
     runs-on: ubuntu-latest
+    # Disable this scheduled job when running on a fork.
+    if: ${{ github.repository == 'RustPython/RustPython' || github.event_name != 'schedule' }}
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: true
+
       - uses: dtolnay/rust-toolchain@stable
       - name: build rustpython
         run: cargo build --release --verbose
@@ -51,6 +68,7 @@ jobs:
         env:
           RUSTPYTHONPATH: ${{ github.workspace }}/Lib
       - name: upload tests data to the website
+        if: ${{ github.event_name != 'pull_request' }}
         env:
           SSHKEY: ${{ secrets.ACTIONS_TESTS_DATA_DEPLOY_KEY }}
           GITHUB_ACTOR: ${{ github.actor }}
@@ -70,21 +88,27 @@ jobs:
   whatsleft:
     name: Collect what is left data
     runs-on: ubuntu-latest
+    # Disable this scheduled job when running on a fork.
+    if: ${{ github.repository == 'RustPython/RustPython' || github.event_name != 'schedule' }}
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: true
+
       - uses: dtolnay/rust-toolchain@stable
-      - uses: actions/setup-python@v5
+      - uses: actions/setup-python@v6.2.0
         with:
           python-version: ${{ env.PYTHON_VERSION }}
       - name: build rustpython
         run: cargo build --release --verbose
       - name: Collect what is left data
         run: |
-          chmod +x ./whats_left.py
-          ./whats_left.py > whats_left.temp
+          chmod +x ./scripts/whats_left.py
+          ./scripts/whats_left.py --features "ssl,sqlite" > whats_left.temp
         env:
           RUSTPYTHONPATH: ${{ github.workspace }}/Lib
       - name: Upload data to the website
+        if: ${{ github.event_name != 'pull_request' }}
         env:
           SSHKEY: ${{ secrets.ACTIONS_TESTS_DATA_DEPLOY_KEY }}
           GITHUB_ACTOR: ${{ github.actor }}
@@ -97,6 +121,26 @@ jobs:
           cd website
           [ -f ./_data/whats_left.temp ] && cp ./_data/whats_left.temp ./_data/whats_left_lastrun.temp
           cp ../whats_left.temp ./_data/whats_left.temp
+          rm ./_data/whats_left/modules.csv
+          echo -e "module" > ./_data/whats_left/modules.csv
+          cat ./_data/whats_left.temp | grep "(entire module)" | cut -d ' ' -f 1 | sort >> ./_data/whats_left/modules.csv
+          awk -f - ./_data/whats_left.temp > ./_data/whats_left/builtin_items.csv <<'EOF'
+          BEGIN {
+              OFS=","
+              print "builtin,name,is_inherited"
+          }
+          /^# builtin items/ { in_section=1; next }
+          /^$/ { if (in_section) exit }
+          in_section {
+              split($1, a, ".")
+              rest = ""
+              idx = index($0, " ")
+              if (idx > 0) {
+                  rest = substr($0, idx+1)
+              }
+              print a[1], $1, rest
+          }
+          EOF
           git add -A
           if git -c user.name="Github Actions" -c user.email="actions@github.com" commit -m "Update what is left results" --author="$GITHUB_ACTOR"; then
             git push
@@ -105,12 +149,17 @@ jobs:
   benchmark:
     name: Collect benchmark data
     runs-on: ubuntu-latest
+    # Disable this scheduled job when running on a fork.
+    if: ${{ github.repository == 'RustPython/RustPython' || github.event_name != 'schedule' }}
     steps:
-      - uses: actions/checkout@v4
-      - uses: dtolnay/rust-toolchain@stable
-      - uses: actions/setup-python@v5
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
         with:
-          python-version: 3.9
+          persist-credentials: true
+
+      - uses: dtolnay/rust-toolchain@stable
+      - uses: actions/setup-python@v6.2.0
+        with:
+          python-version: ${{ env.PYTHON_VERSION }}
       - run: cargo install cargo-criterion
       - name: build benchmarks
         run: cargo build --release --benches
@@ -131,6 +180,7 @@ jobs:
           mv reports/* .
           rmdir reports
       - name: upload benchmark data to the website
+        if: ${{ github.event_name != 'pull_request' }}
         env:
           SSHKEY: ${{ secrets.ACTIONS_TESTS_DATA_DEPLOY_KEY }}
         run: |
@@ -142,7 +192,11 @@ jobs:
           cd website
           rm -rf ./assets/criterion
           cp -r ../target/criterion ./assets/criterion
-          git add ./assets/criterion
+          printf '{\n  "generated_at": "%s",\n  "rustpython_commit": "%s",\n  "rustpython_ref": "%s"\n}\n' \
+            "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
+            "${{ github.sha }}" \
+            "${{ github.ref_name }}" > ./_data/criterion-metadata.json
+          git add ./assets/criterion ./_data/criterion-metadata.json
           if git -c user.name="Github Actions" -c user.email="actions@github.com" commit -m "Update benchmark results"; then
             git push
           fi

.github/workflows/lib-deps-check.yaml

@@ -0,0 +1,123 @@
+name: Lib Dependencies Check
+
+on:
+  pull_request_target:
+    types: [opened, synchronize, reopened]
+    paths:
+      - "Lib/**"
+
+concurrency:
+  group: lib-deps-${{ github.event.pull_request.number }}
+  cancel-in-progress: true
+
+env:
+  PYTHON_VERSION: "3.14.3"
+
+jobs:
+  check_deps:
+    permissions:
+      pull-requests: write
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - name: Checkout base branch
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          # Use base branch for scripts (security: don't run PR code with elevated permissions)
+          ref: ${{ github.event.pull_request.base.ref }}
+          fetch-depth: 0
+          persist-credentials: false
+
+      - name: Fetch PR head
+        run: |
+          git fetch origin ${{ github.event.pull_request.head.sha }}
+
+      - name: Checkout PR Lib files
+        run: |
+          # Checkout only Lib/ directory from PR head for accurate comparison
+          git checkout ${{ github.event.pull_request.head.sha }} -- Lib/
+
+      - name: Checkout CPython
+        run: |
+          git clone --depth 1 --branch "v${{ env.PYTHON_VERSION }}" https://github.com/python/cpython.git cpython
+
+      - name: Get changed Lib files
+        id: changed-files
+        run: |
+          # Get the list of changed files under Lib/
+          changed=$(git diff --name-only ${{ github.event.pull_request.base.sha }} ${{ github.event.pull_request.head.sha }} -- 'Lib/*.py' 'Lib/**/*.py' | head -50)
+          echo "Changed files:"
+          echo "$changed"
+
+          # Extract unique module names
+          modules=""
+          for file in $changed; do
+            if [[ "$file" == Lib/test/* ]]; then
+              # Test files: Lib/test/test_pydoc.py -> test_pydoc, Lib/test/test_pydoc/foo.py -> test_pydoc
+              module=$(echo "$file" | sed -E 's|^Lib/test/||; s|\.py$||; s|/.*||')
+              # Skip non-test files in test/ (e.g., support.py, __init__.py)
+              if [[ ! "$module" == test_* ]]; then
+                continue
+              fi
+            else
+              # Lib files: Lib/foo.py -> foo, Lib/foo/__init__.py -> foo
+              module=$(echo "$file" | sed -E 's|^Lib/||; s|/__init__\.py$||; s|\.py$||; s|/.*||')
+            fi
+            if [[ -n "$module" && ! " $modules " =~ " $module " ]]; then
+              modules="$modules $module"
+            fi
+          done
+
+          modules=$(echo "$modules" | xargs)  # trim whitespace
+          echo "Detected modules: $modules"
+          echo "modules=$modules" >> $GITHUB_OUTPUT
+
+      - name: Setup Python
+        if: steps.changed-files.outputs.modules != ''
+        uses: actions/setup-python@v6.2.0
+        with:
+          python-version: "${{ env.PYTHON_VERSION }}"
+
+      - name: Run deps check
+        if: steps.changed-files.outputs.modules != ''
+        id: deps-check
+        run: |
+          # Run deps for all modules at once
+          python scripts/update_lib deps ${{ steps.changed-files.outputs.modules }} --depth 2 > /tmp/deps_output.txt 2>&1 || true
+
+          # Read output for GitHub Actions
+          echo "deps_output<<EOF" >> $GITHUB_OUTPUT
+          cat /tmp/deps_output.txt >> $GITHUB_OUTPUT
+          echo "EOF" >> $GITHUB_OUTPUT
+
+          # Check if there's any meaningful output
+          if [ -s /tmp/deps_output.txt ]; then
+            echo "has_output=true" >> $GITHUB_OUTPUT
+          else
+            echo "has_output=false" >> $GITHUB_OUTPUT
+          fi
+
+      - name: Post comment
+        if: steps.deps-check.outputs.has_output == 'true'
+        uses: marocchino/sticky-pull-request-comment@v3
+        with:
+          header: lib-deps-check
+          number: ${{ github.event.pull_request.number }}
+          message: |
+            ## 📦 Library Dependencies
+
+            The following Lib/ modules were modified. Here are their dependencies:
+
+            ${{ steps.deps-check.outputs.deps_output }}
+
+            **Legend:**
+            - `[+]` path exists in CPython
+            - `[x]` up-to-date, `[ ]` outdated
+
+      - name: Remove comment if no Lib changes
+        if: steps.changed-files.outputs.modules == ''
+        uses: marocchino/sticky-pull-request-comment@v3
+        with:
+          header: lib-deps-check
+          number: ${{ github.event.pull_request.number }}
+          delete: true

.github/workflows/pr-format.yaml

@@ -0,0 +1,74 @@
+name: Format Check
+
+# This workflow triggers when a PR is opened/updated
+# Posts inline suggestion comments instead of auto-committing
+on:
+  pull_request:
+    types: [opened, synchronize, reopened]
+    branches:
+      - main
+      - release
+
+concurrency:
+  group: format-check-${{ github.event.pull_request.number }}
+  cancel-in-progress: true
+
+env:
+  PYTHON_VERSION: "3.14.3"
+
+jobs:
+  format_check:
+    permissions:
+      contents: read
+      pull-requests: write
+    runs-on: ubuntu-latest
+    timeout-minutes: 60
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+
+      - uses: reviewdog/action-actionlint@0d952c597ef8459f634d7145b0b044a9699e5e43 # v1.71.0
+
+      - name: Setup Rust
+        uses: dtolnay/rust-toolchain@stable
+        with:
+          components: rustfmt
+
+      - name: Run cargo fmt
+        run: cargo fmt --all
+
+      - name: Install ruff
+        uses: astral-sh/ruff-action@4919ec5cf1f49eff0871dbcea0da843445b837e6 # v3.6.1
+        with:
+          version: "0.15.4"
+          args: "--version"
+
+      - name: Run ruff format
+        run: ruff format
+
+      - name: Run ruff check import sorting
+        run: ruff check --select I --fix
+
+      - uses: actions/setup-python@v6.2.0
+        with:
+          python-version: ${{ env.PYTHON_VERSION }}
+
+      - name: Run generate_opcode_metadata.py
+        run: python scripts/generate_opcode_metadata.py
+
+      - name: Check for formatting changes
+        run: |
+          if ! git diff --exit-code; then
+            echo "::error::Formatting changes detected. Please run 'cargo fmt --all', 'ruff format', and 'ruff check --select I --fix' locally."
+            exit 1
+          fi
+
+      - name: Post formatting suggestions
+        if: failure()
+        uses: reviewdog/action-suggester@v1
+        with:
+          tool_name: auto-format
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          level: warning
+          filter_mode: diff_context

.github/workflows/release.yml

@@ -0,0 +1,190 @@
+name: Release
+
+on:
+  schedule:
+    # 9 AM UTC on every Monday
+    - cron: "0 9 * * Mon"
+  workflow_dispatch:
+    inputs:
+      pre-release:
+        type: boolean
+        description: Mark "Pre-Release"
+        required: false
+        default: true
+
+permissions:
+  contents: write
+
+env:
+  CARGO_ARGS: --no-default-features --features stdlib,importlib,encodings,sqlite,ssl
+  X86_64_PC_WINDOWS_MSVC_OPENSSL_LIB_DIR: C:\Program Files\OpenSSL\lib\VC\x64\MD
+  X86_64_PC_WINDOWS_MSVC_OPENSSL_INCLUDE_DIR: C:\Program Files\OpenSSL\include
+
+jobs:
+  build:
+    runs-on: ${{ matrix.platform.runner }}
+    # Disable this scheduled job when running on a fork.
+    if: ${{ github.repository == 'RustPython/RustPython' || github.event_name != 'schedule' }}
+    strategy:
+      matrix:
+        platform:
+          - runner: ubuntu-latest
+            target: x86_64-unknown-linux-gnu
+#          - runner: ubuntu-latest
+#            target: i686-unknown-linux-gnu
+#          - runner: ubuntu-latest
+#            target: aarch64-unknown-linux-gnu
+#          - runner: ubuntu-latest
+#            target: armv7-unknown-linux-gnueabi
+#          - runner: ubuntu-latest
+#            target: s390x-unknown-linux-gnu
+#          - runner: ubuntu-latest
+#            target: powerpc64le-unknown-linux-gnu
+          - runner: macos-latest
+            target: aarch64-apple-darwin
+#          - runner: macos-latest
+#            target: x86_64-apple-darwin
+          - runner: windows-2025
+            target: x86_64-pc-windows-msvc
+#          - runner: windows-2025
+#            target: i686-pc-windows-msvc
+#          - runner: windows-2025
+#            target: aarch64-pc-windows-msvc
+      fail-fast: false
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+
+      - uses: dtolnay/rust-toolchain@stable
+      - uses: cargo-bins/cargo-binstall@main
+
+      - name: Set up Environment
+        shell: bash
+        run: rustup target add ${{ matrix.platform.target }}
+      - name: Set up MacOS Environment
+        run: brew install autoconf automake libtool
+        if: runner.os == 'macOS'
+
+      - name: Build RustPython
+        run: cargo build --release --target=${{ matrix.platform.target }} --verbose --features=threading ${{ env.CARGO_ARGS }}
+        if: runner.os == 'macOS'
+      - name: Build RustPython
+        run: cargo build --release --target=${{ matrix.platform.target }} --verbose --features=threading ${{ env.CARGO_ARGS }},jit
+        if: runner.os != 'macOS'
+
+      - name: Rename Binary
+        run: cp target/${{ matrix.platform.target }}/release/rustpython target/rustpython-release-${{ runner.os }}-${{ matrix.platform.target }}
+        if: runner.os != 'Windows'
+      - name: Rename Binary
+        run: cp target/${{ matrix.platform.target }}/release/rustpython.exe target/rustpython-release-${{ runner.os }}-${{ matrix.platform.target }}.exe
+        if: runner.os == 'Windows'
+
+      - name: Upload Binary Artifacts
+        uses: actions/upload-artifact@v7.0.0
+        with:
+          name: rustpython-release-${{ runner.os }}-${{ matrix.platform.target }}
+          path: target/rustpython-release-${{ runner.os }}-${{ matrix.platform.target }}*
+
+  build-wasm:
+    runs-on: ubuntu-latest
+    # Disable this scheduled job when running on a fork.
+    if: ${{ github.repository == 'RustPython/RustPython' || github.event_name != 'schedule' }}
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+
+      - uses: dtolnay/rust-toolchain@stable
+        with:
+          targets: wasm32-wasip1
+
+      - name: Build RustPython
+        run: cargo build --target wasm32-wasip1 --no-default-features --features freeze-stdlib,stdlib --release
+
+      - name: Rename Binary
+        run: cp target/wasm32-wasip1/release/rustpython.wasm target/rustpython-release-wasm32-wasip1.wasm
+
+      - name: Upload Binary Artifacts
+        uses: actions/upload-artifact@v7.0.0
+        with:
+          name: rustpython-release-wasm32-wasip1
+          path: target/rustpython-release-wasm32-wasip1.wasm
+
+      - name: install wasm-pack
+        run: curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh
+      - uses: actions/setup-node@v6
+      - uses: mwilliamson/setup-wabt-action@v3
+        with: { wabt-version: "1.0.30" }
+      - name: build demo
+        run: |
+          npm install
+          npm run dist
+        env:
+          NODE_OPTIONS: "--openssl-legacy-provider"
+        working-directory: ./wasm/demo
+      - name: build notebook demo
+        run: |
+          npm install
+          npm run dist
+          mv dist ../demo/dist/notebook
+        env:
+          NODE_OPTIONS: "--openssl-legacy-provider"
+        working-directory: ./wasm/notebook
+      - name: Deploy demo to Github Pages
+        uses: peaceiris/actions-gh-pages@v4
+        with:
+          deploy_key: ${{ secrets.ACTIONS_DEMO_DEPLOY_KEY }}
+          publish_dir: ./wasm/demo/dist
+          external_repository: RustPython/demo
+          publish_branch: master
+
+  release:
+    runs-on: ubuntu-latest
+    # Disable this scheduled job when running on a fork.
+    if: ${{ github.repository == 'RustPython/RustPython' || github.event_name != 'schedule' }}
+    needs: [build, build-wasm]
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+
+      - name: Download Binary Artifacts
+        uses: actions/download-artifact@v8.0.1
+        with:
+          path: bin
+          pattern: rustpython-*
+          merge-multiple: true
+
+      - name: Create Lib Archive
+        run: |
+          zip -r bin/rustpython-lib.zip Lib/
+
+      - name: List Binaries
+        run: |
+          ls -lah bin/
+          file bin/*
+      - name: Create Release
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          tag: ${{ github.ref_name }}
+          run: ${{ github.run_number }}
+          PRE_RELEASE_INPUT: ${{ github.event.inputs.pre-release }}
+        run: |
+          if [[ "${PRE_RELEASE_INPUT}" == "false" ]]; then
+            RELEASE_TYPE_NAME=Release
+            PRERELEASE_ARG=
+          else
+            RELEASE_TYPE_NAME=Pre-Release
+            PRERELEASE_ARG=--prerelease
+          fi
+          
+          today=$(date '+%Y-%m-%d')
+          gh release create "$today-$tag-$run" \
+              --repo="$GITHUB_REPOSITORY" \
+              --title="RustPython $RELEASE_TYPE_NAME $today-$tag #$run" \
+              --target="$tag" \
+              --notes "⚠️ **Important**: To run RustPython, you must download both the binary for your platform AND the \`rustpython-lib.zip\` archive. Extract the Lib directory from the archive to the same location as the binary, or set the \`RUSTPYTHONPATH\` environment variable to point to the Lib directory." \
+              --generate-notes \
+              $PRERELEASE_ARG \
+              bin/rustpython-release-*

.github/workflows/update-doc-db.yml

@@ -0,0 +1,124 @@
+name: Update doc DB
+
+permissions:
+  contents: write
+  pull-requests: write
+
+on:
+  workflow_dispatch:
+    inputs:
+      python-version:
+        description: Target python version to generate doc db for
+        type: string
+        default: "3.14.3"
+      base-ref:
+        description: Base branch to create the update branch from
+        type: string
+        default: "main"
+
+defaults:
+  run:
+    shell: bash
+
+jobs:
+  generate:
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        os:
+          - ubuntu-latest
+          - windows-latest
+          - macos-latest
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+          sparse-checkout: |
+            crates/doc
+
+      - uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
+        with:
+          python-version: ${{ inputs.python-version }}
+
+      - name: Generate docs
+        run: python crates/doc/generate.py
+
+      - uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0
+        with:
+          name: doc-db-${{ inputs.python-version }}-${{ matrix.os }}
+          path: "crates/doc/generated/*.json"
+          if-no-files-found: error
+          retention-days: 7
+          overwrite: true
+
+  merge:
+    runs-on: ubuntu-latest
+    needs: generate
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: true
+          ref: ${{ inputs.base-ref }}
+          token: ${{ secrets.AUTO_COMMIT_PAT }}
+
+      - name: Create update branch
+        env:
+          PYTHON_VERSION: ${{ inputs.python-version }}
+        run: git switch -c "update-doc-${PYTHON_VERSION}"
+
+      - name: Download generated doc DBs
+        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
+        with:
+          pattern: "doc-db-${{ inputs.python-version }}-**"
+          path: crates/doc/generated/
+          merge-multiple: true
+
+      - name: Transform JSON
+        env:
+          PYTHON_VERSION: ${{ inputs.python-version }}
+        run: |
+          # Merge all artifacts
+          jq -s "add" --sort-keys crates/doc/generated/*.json > crates/doc/generated/merged.json
+
+          # Format merged json for the phf macro
+          jq -r 'to_entries[] | "    \(.key | @json) => \(.value | @json),"' crates/doc/generated/merged.json > crates/doc/generated/raw_entries.txt
+
+          OUTPUT_FILE='crates/doc/src/data.inc.rs'
+
+          echo -n '' > $OUTPUT_FILE
+
+          echo '// This file was auto-generated by `.github/workflows/update-doc-db.yml`.' >> $OUTPUT_FILE
+          echo "// CPython version: ${PYTHON_VERSION}" >> $OUTPUT_FILE
+          echo '// spell-checker: disable' >> $OUTPUT_FILE
+
+          echo '' >> $OUTPUT_FILE
+
+          echo "pub static DB: phf::Map<&'static str, &'static str> = phf::phf_map! {" >> $OUTPUT_FILE
+          cat crates/doc/generated/raw_entries.txt >> $OUTPUT_FILE
+          echo '};' >> $OUTPUT_FILE
+
+      - uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0
+        with:
+          name: doc-db-${{ inputs.python-version }}
+          path: "crates/doc/src/data.inc.rs"
+          if-no-files-found: error
+          retention-days: 7
+          overwrite: true
+
+      - name: Commit, push and create PR
+        env:
+          GH_TOKEN: ${{ secrets.AUTO_COMMIT_PAT }}
+          PYTHON_VERSION: ${{ inputs.python-version }}
+          BASE_REF: ${{ inputs.base-ref }}
+        run: |
+          git config user.name "github-actions[bot]"
+          git config user.email "github-actions[bot]@users.noreply.github.com"
+          if [ -n "$(git status --porcelain)" ]; then
+            git add crates/doc/src/data.inc.rs
+            git commit -m "Update doc DB for CPython ${PYTHON_VERSION}"
+            git push -u origin HEAD
+            gh pr create \
+              --base "${BASE_REF}" \
+              --title "Update doc DB for CPython ${PYTHON_VERSION}" \
+              --body "Auto-generated by update-doc-db workflow."
+          fi

.github/workflows/update-libs-status.yaml

@@ -0,0 +1,90 @@
+name: Updated libs status
+
+on:
+  push:
+    branches:
+      - main
+    paths:
+      - "Lib/**"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+  issues: write
+
+env:
+  PYTHON_VERSION: "v3.14.3"
+  ISSUE_ID: "6839"
+
+jobs:
+  update-issue:
+    runs-on: ubuntu-latest
+    if: ${{ github.repository == 'RustPython/RustPython' }}
+    steps:
+      - name: Clone RustPython
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          path: rustpython
+          persist-credentials: false
+          sparse-checkout: |-
+              Lib
+              scripts/update_lib
+
+      - name: Clone CPython ${{ env.PYTHON_VERSION }}
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          repository: python/cpython
+          path: cpython
+          ref: ${{ env.PYTHON_VERSION }}
+          persist-credentials: false
+          sparse-checkout: |
+            Lib
+
+      - name: Get current date
+        id: current_date
+        run: |
+          now=$(date -u +"%Y-%m-%d %H:%M:%S")
+          echo "date=$now" >> "$GITHUB_OUTPUT"
+
+      - name: Write body prefix
+        run: |
+          cat > body.txt <<EOF
+
+          <!-- 
+          THIS BODY IS AUTO-GENERATED. DO NOT EDIT MANUALLY!
+          -->
+
+          ## Summary
+
+          Check \`scripts/update_lib\` for tools. As a note, the current latest Python version is \`${{ env.PYTHON_VERSION }}\`.
+
+          Previous versions' issues as reference
+          - 3.13: #5529 
+
+          <!-- 
+          Quick guideline for Copilot:
+          # Clone \`github.com/python/cpython\` \`${{ env.PYTHON_VERSION }}\` tag under RustPython working dir with depth 1 option; never 3.14.0 or 3.14.1 or 3.14.2
+          # Pick a library or test to update. Probably user give one.
+          # Run \`python3 scripts/update_lib quick <name>\`
+          # A commit is automatically created. push the commit.
+          # Do not include keywords that automatically close an issue (e.g., Fix, Close, Resolve) next to an issue link or issue number.
+          -->
+
+          ## Details
+
+          ${{ steps.current_date.outputs.date }} (UTC)
+          \`\`\`shell
+          $ python3 scripts/update_lib todo --done
+          \`\`\`
+          EOF
+        
+      - name: Run todo
+        run: python3 rustpython/scripts/update_lib todo --cpython cpython --lib rustpython/Lib --done >> body.txt
+
+      - name: Update GH issue
+        run: gh issue edit ${{ env.ISSUE_ID }} --body-file ../body.txt
+        env:
+          GH_TOKEN: ${{ github.token }}
+        working-directory: rustpython
+        
+

.github/workflows/upgrade-pylib.md

@@ -0,0 +1,134 @@
+---
+description: |
+  Pick an out-of-sync Python library from the todo list and upgrade it
+  by running `scripts/update_lib quick`, then open a pull request.
+
+on:
+  workflow_dispatch:
+    inputs:
+      name:
+        description: "Module name to upgrade (leave empty to auto-pick)"
+        required: false
+        type: string
+
+timeout-minutes: 45
+
+permissions:
+  contents: read
+  issues: read
+  pull-requests: read
+
+network:
+  allowed:
+    - defaults
+    - rust
+    - python
+
+engine: copilot
+
+runtimes:
+  python:
+    version: "3.14"
+
+tools:
+  bash:
+    - ":*"
+  edit:
+  github:
+    toolsets: [repos, issues, pull_requests]
+    read-only: true
+
+safe-outputs:
+  create-pull-request:
+    title-prefix: "Update "
+    labels: [pylib-sync]
+    draft: false
+    expires: 30
+
+cache:
+  key: cpython-lib-${{ env.PYTHON_VERSION }}
+  path: cpython
+  restore-keys:
+    - cpython-lib-
+
+env:
+  PYTHON_VERSION: "v3.14.3"
+  ISSUE_ID: "6839"
+---
+
+# Upgrade Python Library
+
+You are an automated maintenance agent for RustPython, a Python 3 interpreter written in Rust. Your task is to upgrade one out-of-sync Python standard library module from CPython.
+
+## Step 1: Set up the environment
+
+The CPython source may already be cached. Check if the `cpython` directory exists and has the correct version:
+
+```bash
+if [ -d "cpython/Lib" ]; then
+    echo "CPython cache hit, skipping clone"
+else
+    git clone --depth 1 --branch "$PYTHON_VERSION" https://github.com/python/cpython.git cpython
+fi
+```
+
+## Step 2: Determine module name
+
+Run this script to determine the module name:
+
+```bash
+MODULE_NAME="${{ github.event.inputs.name }}"
+if [ -z "$MODULE_NAME" ]; then
+    echo "No module specified, running todo to find one..."
+    python3 scripts/update_lib todo
+    echo "Pick one module from the list above that is marked [ ], has no unmet deps, and has a small Δ number."
+    echo "Do NOT pick: opcode, datetime, random, hashlib, tokenize, pdb, _pyrepl, concurrent, asyncio, multiprocessing, ctypes, idlelib, tkinter, shutil, tarfile, email, unittest"
+else
+    echo "Module specified by user: $MODULE_NAME"
+fi
+```
+
+If the script printed "Module specified by user: ...", use that exact name. If it printed the todo list, pick one suitable module from it.
+
+## Step 3: Run the upgrade
+
+Run the quick upgrade command. This will copy the library from CPython, migrate test files preserving RustPython markers, auto-mark test failures, and create a git commit:
+
+```bash
+python3 scripts/update_lib quick <module_name>
+```
+
+This takes a while because it builds RustPython (`cargo build --release`) and runs tests to determine which ones pass or fail.
+
+If the command fails, report the error and stop. Do not try to fix Rust code or modify test files manually.
+
+## Step 4: Verify the result
+
+After the script succeeds, check what changed:
+
+```bash
+git log -1 --stat
+git diff HEAD~1 --stat
+```
+
+Make sure the commit was created with the correct message format: `Update <name> from <version>`.
+
+## Step 5: Create the pull request
+
+Create a pull request. Reference issue #${{ env.ISSUE_ID }} in the body but do **NOT** use keywords that auto-close issues (Fix, Close, Resolve).
+
+Use this format for the PR body:
+
+```
+## Summary
+
+Upgrade `<module_name>` from CPython $PYTHON_VERSION.
+
+Part of #$ISSUE_ID
+
+## Changes
+
+- Updated `Lib/<module_name>` from CPython
+- Migrated test files preserving RustPython markers
+- Auto-marked test failures with `@expectedFailure`
+```

.gitignore

@@ -2,11 +2,11 @@
 /*/target
 **/*.rs.bk
 **/*.bytecode
-__pycache__
+__pycache__/
 **/*.pytest_cache
 .*sw*
 .repl_history.txt
-.vscode
+.vscode/
 wasm-pack.log
 .idea/
 .envrc
@@ -21,3 +21,11 @@ flamescope.json
 
 extra_tests/snippets/resources
 extra_tests/not_impl.py
+
+Lib/_sysconfig_vars*.json
+Lib/site-packages/*
+!Lib/site-packages/README.txt
+Lib/test/data/*
+!Lib/test/data/README
+cpython/
+

.vscode/launch.json

@@ -16,18 +16,6 @@
             },
             "cwd": "${workspaceFolder}"
         },
-        {
-            "type": "lldb",
-            "request": "launch",
-            "name": "Debug executable 'rustpython' without SSL",
-            "preLaunchTask": "Build RustPython Debug without SSL",
-            "program": "target/debug/rustpython",
-            "args": [],
-            "env": {
-                "RUST_BACKTRACE": "1"
-            },
-            "cwd": "${workspaceFolder}"
-        },
         {
             "type": "lldb",
             "request": "launch",

.vscode/tasks.json

@@ -1,28 +1,12 @@
 {
     "version": "2.0.0",
     "tasks": [
-        {
-            "label": "Build RustPython Debug without SSL",
-            "type": "shell",
-            "command": "cargo",
-            "args": [
-                "build",
-            ],
-            "problemMatcher": [
-                "$rustc",
-            ],
-            "group": {
-                "kind": "build",
-                "isDefault": true,
-            },
-        },
         {
             "label": "Build RustPython Debug",
             "type": "shell",
             "command": "cargo",
             "args": [
                 "build",
-                "--features=ssl"
             ],
             "problemMatcher": [
                 "$rustc",

AGENTS.md

@@ -0,0 +1,266 @@
+# GitHub Copilot Instructions for RustPython
+
+This document provides guidelines for working with GitHub Copilot when contributing to the RustPython project.
+
+## Project Overview
+
+RustPython is a Python 3 interpreter written in Rust, implementing Python 3.14.0+ compatibility. The project aims to provide:
+
+- A complete Python-3 environment entirely in Rust (not CPython bindings)
+- A clean implementation without compatibility hacks
+- Cross-platform support, including WebAssembly compilation
+- The ability to embed Python scripting in Rust applications
+
+## Repository Structure
+
+- `src/` - Top-level code for the RustPython binary
+- `vm/` - The Python virtual machine implementation
+  - `builtins/` - Python built-in types and functions
+  - `stdlib/` - Essential standard library modules implemented in Rust, required to run the Python core
+- `compiler/` - Python compiler components
+  - `parser/` - Parser for converting Python source to AST
+  - `core/` - Bytecode representation in Rust structures
+  - `codegen/` - AST to bytecode compiler
+- `Lib/` - CPython's standard library in Python (copied from CPython). **IMPORTANT**: Do not edit this directory directly; The only allowed operation is copying files from CPython.
+- `derive/` - Rust macros for RustPython
+- `common/` - Common utilities
+- `extra_tests/` - Integration tests and snippets
+- `stdlib/` - Non-essential Python standard library modules implemented in Rust (useful but not required for core functionality)
+- `wasm/` - WebAssembly support
+- `jit/` - Experimental JIT compiler implementation
+- `pylib/` - Python standard library packaging (do not modify this directory directly - its contents are generated automatically)
+
+## AI Agent Rules
+
+**CRITICAL: Git Operations**
+- NEVER create pull requests directly without explicit user permission
+- NEVER push commits to remote without explicit user permission
+- Always ask the user before performing any git operations that affect the remote repository
+- Commits can be created locally when requested, but pushing and PR creation require explicit approval
+
+## Important Development Notes
+
+### Running Python Code
+
+When testing Python code, always use RustPython instead of the standard `python` command:
+
+```bash
+# Use this instead of python script.py
+cargo run -- script.py
+
+# For interactive REPL
+cargo run
+
+# With specific features
+cargo run --features jit
+
+# Release mode (recommended for better performance)
+cargo run --release -- script.py
+```
+
+### Comparing with CPython
+
+When you need to compare behavior with CPython or run test suites:
+
+```bash
+# Use python command to explicitly run CPython
+python my_test_script.py
+
+# Run RustPython
+cargo run -- my_test_script.py
+```
+
+### Working with the Lib Directory
+
+The `Lib/` directory contains Python standard library files copied from the CPython repository. Important notes:
+
+- These files should be edited very conservatively
+- Modifications should be minimal and only to work around RustPython limitations
+- Tests in `Lib/test` often use one of the following markers:
+  - Add a `# TODO: RUSTPYTHON` comment when modifications are made
+  - `unittest.skip("TODO: RustPython <reason>")`
+  - `unittest.expectedFailure` with `# TODO: RUSTPYTHON <reason>` comment
+
+### Clean Build
+
+When you modify bytecode instructions, a full clean is required:
+
+```bash
+rm -r target/debug/build/rustpython-* && find . | grep -E "\.pyc$" | xargs rm -r
+```
+
+### Testing
+
+```bash
+# Run Rust unit tests
+cargo test --workspace --exclude rustpython_wasm --exclude rustpython-venvlauncher
+
+# Run Python snippets tests (debug mode recommended for faster compilation)
+cargo run -- extra_tests/snippets/builtin_bytes.py
+
+# Run all Python snippets tests with pytest
+cd extra_tests
+pytest -v
+
+# Run the Python test module (release mode recommended for better performance)
+cargo run --release -- -m test ${TEST_MODULE}
+cargo run --release -- -m test test_unicode # to test test_unicode.py
+
+# Run the Python test module with specific function
+cargo run --release -- -m test test_unicode -k test_unicode_escape
+```
+
+**Note**: For `extra_tests/snippets` tests, use debug mode (`cargo run`) as compilation is faster. For `unittest` (`-m test`), use release mode (`cargo run --release`) for better runtime performance.
+
+### Determining What to Implement
+
+Run `./scripts/whats_left.py` to get a list of unimplemented methods, which is helpful when looking for contribution opportunities.
+
+## Coding Guidelines
+
+### Rust Code
+
+- Follow the default rustfmt code style (`cargo fmt` to format)
+- **IMPORTANT**: Always run clippy to lint code (`cargo clippy`) before completing tasks. Fix any warnings or lints that are introduced by your changes
+- Follow Rust best practices for error handling and memory management
+- Use the macro system (`pyclass`, `pymodule`, `pyfunction`, etc.) when implementing Python functionality in Rust
+
+#### Comments
+
+- Do not delete or rewrite existing comments unless they are factually wrong or directly contradict the new code.
+- Do not add decorative section separators (e.g. `// -----------`, `// ===`, `/* *** */`). Use `///` doc-comments or short `//` comments only when they add value.
+
+#### Avoid Duplicate Code in Branches
+
+When branches differ only in a value but share common logic, extract the differing value first, then call the common logic once.
+
+**Bad:**
+```rust
+let result = if condition {
+    let msg = format!("message A: {x}");
+    some_function(msg, shared_arg)
+} else {
+    let msg = format!("message B");
+    some_function(msg, shared_arg)
+};
+```
+
+**Good:**
+```rust
+let msg = if condition {
+    format!("message A: {x}")
+} else {
+    format!("message B")
+};
+let result = some_function(msg, shared_arg);
+```
+
+### Python Code
+
+- **IMPORTANT**: In most cases, Python code should not be edited. Bug fixes should be made through Rust code modifications only
+- Follow PEP 8 style for custom Python code
+- Use ruff for linting Python code
+- Minimize modifications to CPython standard library files
+
+## Integration Between Rust and Python
+
+The project provides several mechanisms for integration:
+
+- `pymodule` macro for creating Python modules in Rust
+- `pyclass` macro for implementing Python classes in Rust
+- `pyfunction` macro for exposing Rust functions to Python
+- `PyObjectRef` and other types for working with Python objects in Rust
+
+## Common Patterns
+
+### Implementing a Python Module in Rust
+
+```rust
+#[pymodule]
+mod mymodule {
+    use rustpython_vm::prelude::*;
+
+    #[pyfunction]
+    fn my_function(value: i32) -> i32 {
+        value * 2
+    }
+
+    #[pyattr]
+    #[pyclass(name = "MyClass")]
+    #[derive(Debug, PyPayload)]
+    struct MyClass {
+        value: usize,
+    }
+
+    #[pyclass]
+    impl MyClass {
+        #[pymethod]
+        fn get_value(&self) -> usize {
+            self.value
+        }
+    }
+}
+```
+
+### Adding a Python Module to the Interpreter
+
+```rust
+vm.add_native_module(
+    "my_module_name".to_owned(),
+    Box::new(my_module::make_module),
+);
+```
+
+## Building for Different Targets
+
+### WebAssembly
+
+```bash
+# Build for WASM
+cargo build --target wasm32-wasip1 --no-default-features --features freeze-stdlib,stdlib --release
+```
+
+### JIT Support
+
+```bash
+# Enable JIT support
+cargo run --features jit
+```
+
+### Linux Build and Debug on macOS
+
+See the "Testing on Linux from macOS" section in [DEVELOPMENT.md](DEVELOPMENT.md#testing-on-linux-from-macos).
+
+### Building venvlauncher (Windows)
+
+See DEVELOPMENT.md "CPython Version Upgrade Checklist" section.
+
+**IMPORTANT**: All 4 venvlauncher binaries use the same source code. Do NOT add multiple `[[bin]]` entries to Cargo.toml. Build once and copy with different names.
+
+## Test Code Modification Rules
+
+**CRITICAL: Test code modification restrictions**
+- NEVER comment out or delete any test code lines except for removing `@unittest.expectedFailure` decorators and upper TODO comments
+- NEVER modify test assertions, test logic, or test data
+- When a test cannot pass due to missing language features, keep it as expectedFailure and document the reason
+- The only acceptable modifications to test files are:
+  1. Removing `@unittest.expectedFailure` decorators and the upper TODO comments when tests actually pass
+  2. Adding `@unittest.expectedFailure` decorators when tests cannot be fixed
+
+**Examples of FORBIDDEN modifications:**
+- Commenting out test lines
+- Changing test assertions
+- Modifying test data or expected results
+- Removing test logic
+
+**Correct approach when tests fail due to unsupported syntax:**
+- Keep the test as `@unittest.expectedFailure`
+- Document that it requires PEP 695 support
+- Focus on tests that can be fixed through Rust code changes only
+
+## Documentation
+
+- Check the [architecture document](/architecture/architecture.md) for a high-level overview
+- Read the [development guide](/DEVELOPMENT.md) for detailed setup instructions
+- Generate documentation with `cargo doc --no-deps --all`
+- Online documentation is available at [docs.rs/rustpython](https://docs.rs/rustpython/)

Cargo.toml

@@ -10,34 +10,40 @@ repository.workspace = true
 license.workspace = true
 
 [features]
-default = ["threading", "stdlib", "zlib", "importlib"]
+default = ["threading", "stdlib", "stdio", "importlib", "ssl-rustls", "host_env"]
+host_env = ["rustpython-vm/host_env", "rustpython-stdlib?/host_env"]
 importlib = ["rustpython-vm/importlib"]
 encodings = ["rustpython-vm/encodings"]
+stdio = ["rustpython-vm/stdio"]
 stdlib = ["rustpython-stdlib", "rustpython-pylib", "encodings"]
-flame-it = ["rustpython-vm/flame-it", "flame", "flamescope"]
+flame-it = ["rustpython-vm/flame-it", "rustpython-stdlib/flame-it", "flame", "flamescope"]
 freeze-stdlib = ["stdlib", "rustpython-vm/freeze-stdlib", "rustpython-pylib?/freeze-stdlib"]
 jit = ["rustpython-vm/jit"]
 threading = ["rustpython-vm/threading", "rustpython-stdlib/threading"]
-zlib = ["stdlib", "rustpython-stdlib/zlib"]
-bz2 = ["stdlib", "rustpython-stdlib/bz2"]
 sqlite = ["rustpython-stdlib/sqlite"]
-ssl = ["rustpython-stdlib/ssl"]
-ssl-vendor = ["ssl", "rustpython-stdlib/ssl-vendor"]
+ssl = []
+ssl-rustls = ["ssl", "rustpython-stdlib/ssl-rustls"]
+ssl-openssl = ["ssl", "rustpython-stdlib/ssl-openssl"]
+ssl-vendor = ["ssl-openssl", "rustpython-stdlib/ssl-vendor"]
+tkinter = ["rustpython-stdlib/tkinter"]
+
+[build-dependencies]
+winresource = "0.1"
 
 [dependencies]
 rustpython-compiler = { workspace = true }
 rustpython-pylib = { workspace = true, optional = true }
 rustpython-stdlib = { workspace = true, optional = true, features = ["compiler"] }
-rustpython-vm = { workspace = true, features = ["compiler"] }
-rustpython-parser = { workspace = true }
+rustpython-vm = { workspace = true, features = ["compiler", "gc"] }
+ruff_python_parser = { workspace = true }
 
 cfg-if = { workspace = true }
 log = { workspace = true }
 flame = { workspace = true, optional = true }
 
-clap = "2.34"
-dirs = { package = "dirs-next", version = "2.0.0" }
-env_logger = { version = "0.9.0", default-features = false, features = ["atty", "termcolor"] }
+lexopt = "0.3"
+dirs = { package = "dirs-next", version = "2.0" }
+env_logger = "0.11"
 flamescope = { version = "0.1.2", optional = true }
 
 [target.'cfg(windows)'.dependencies]
@@ -47,8 +53,9 @@ libc = { workspace = true }
 rustyline = { workspace = true }
 
 [dev-dependencies]
-criterion = { version = "0.3.5", features = ["html_reports"] }
-pyo3 = { version = "0.22", features = ["auto-initialize"] }
+criterion = { workspace = true }
+pyo3 = { version = "0.28.2", features = ["auto-initialize"] }
+rustpython-stdlib = { workspace = true }
 
 [[bench]]
 name = "execution"
@@ -70,6 +77,21 @@ opt-level = 3
 # https://github.com/rust-lang/rust/issues/92869
 # lto = "thin"
 
+# Some crates don't change as much but benefit more from
+# more expensive optimization passes, so we selectively
+# decrease codegen-units in some cases.
+[profile.release.package.rustpython-doc]
+codegen-units = 1
+
+[profile.release.package.rustpython-literal]
+codegen-units = 1
+
+[profile.release.package.rustpython-common]
+codegen-units = 1
+
+[profile.release.package.rustpython-wtf8]
+codegen-units = 1
+
 [profile.bench]
 lto = "thin"
 codegen-units = 1
@@ -79,117 +101,146 @@ opt-level = 3
 lto = "thin"
 
 [patch.crates-io]
+parking_lot_core = { git = "https://github.com/youknowone/parking_lot", branch = "rustpython" }
 # REDOX START, Uncomment when you want to compile/check with redoxer
 # REDOX END
 
-# Used only on Windows to build the vcpkg dependencies
-[package.metadata.vcpkg]
-git = "https://github.com/microsoft/vcpkg"
-# The revision of the vcpkg repository to use
-# https://github.com/microsoft/vcpkg/tags
-rev = "2024.02.14"
+[package.metadata.packager]
+product-name = "RustPython"
+identifier = "com.rustpython.rustpython"
+description = "An open source Python 3 interpreter written in Rust"
+homepage = "https://rustpython.github.io/"
+license_file = "LICENSE"
+authors = ["RustPython Team"]
+publisher = "RustPython Team"
+resources = ["LICENSE", "README.md", "Lib"]
+icons = ["32x32.png"]
+
+[package.metadata.packager.nsis]
+installer_mode = "both"
+template = "installer-config/installer.nsi"
+
+[package.metadata.packager.wix]
+template = "installer-config/installer.wxs"
 
-[package.metadata.vcpkg.target]
-x86_64-pc-windows-msvc = { triplet = "x64-windows-static-md", dev-dependencies = ["openssl" ] }
 
 [workspace]
 resolver = "2"
 members = [
-    "compiler", "compiler/core", "compiler/codegen",
-    ".", "common", "derive", "jit", "vm", "vm/sre_engine", "pylib", "stdlib", "derive-impl",
-    "wasm/lib",
+    ".",
+    "crates/*",
 ]
+exclude = ["pymath"]
 
 [workspace.package]
-version = "0.4.0"
+version = "0.5.0"
 authors = ["RustPython Team"]
-edition = "2021"
-rust-version = "1.80.0"
+edition = "2024"
+rust-version = "1.93.0"
 repository = "https://github.com/RustPython/RustPython"
 license = "MIT"
 
 [workspace.dependencies]
-rustpython-compiler-core = { path = "compiler/core", version = "0.4.0" }
-rustpython-compiler = { path = "compiler", version = "0.4.0" }
-rustpython-codegen = { path = "compiler/codegen", version = "0.4.0" }
-rustpython-common = { path = "common", version = "0.4.0" }
-rustpython-derive = { path = "derive", version = "0.4.0" }
-rustpython-derive-impl = { path = "derive-impl", version = "0.4.0" }
-rustpython-jit = { path = "jit", version = "0.4.0" }
-rustpython-vm = { path = "vm", default-features = false, version = "0.4.0" }
-rustpython-pylib = { path = "pylib", version = "0.4.0" }
-rustpython-stdlib = { path = "stdlib", default-features = false, version = "0.4.0" }
-rustpython-sre_engine = { path = "vm/sre_engine", version = "0.4.0" }
-rustpython-doc = { git = "https://github.com/RustPython/__doc__", tag = "0.3.0", version = "0.3.0" }
+rustpython-compiler-core = { path = "crates/compiler-core", version = "0.5.0" }
+rustpython-compiler = { path = "crates/compiler", version = "0.5.0" }
+rustpython-codegen = { path = "crates/codegen", version = "0.5.0" }
+rustpython-common = { path = "crates/common", version = "0.5.0" }
+rustpython-derive = { path = "crates/derive", version = "0.5.0" }
+rustpython-derive-impl = { path = "crates/derive-impl", version = "0.5.0" }
+rustpython-jit = { path = "crates/jit", version = "0.5.0" }
+rustpython-literal = { path = "crates/literal", version = "0.5.0" }
+rustpython-vm = { path = "crates/vm", default-features = false, version = "0.5.0" }
+rustpython-pylib = { path = "crates/pylib", version = "0.5.0" }
+rustpython-stdlib = { path = "crates/stdlib", default-features = false, version = "0.5.0" }
+rustpython-sre_engine = { path = "crates/sre_engine", version = "0.5.0" }
+rustpython-wtf8 = { path = "crates/wtf8", version = "0.5.0" }
+rustpython-doc = { path = "crates/doc", version = "0.5.0" }
 
-rustpython-literal = { version = "0.4.0" }
-rustpython-parser-core = { version = "0.4.0" }
-rustpython-parser = { version = "0.4.0" }
-rustpython-ast = { version = "0.4.0" }
-rustpython-format= { version = "0.4.0" }
-# rustpython-literal = { git = "https://github.com/RustPython/Parser.git", version = "0.4.0", rev = "00d2f1d1a7522ef9c85c10dfa5f0bb7178dee655" }
-# rustpython-parser-core = { git = "https://github.com/RustPython/Parser.git", version = "0.4.0", rev = "00d2f1d1a7522ef9c85c10dfa5f0bb7178dee655" }
-# rustpython-parser = { git = "https://github.com/RustPython/Parser.git", version = "0.4.0", rev = "00d2f1d1a7522ef9c85c10dfa5f0bb7178dee655" }
-# rustpython-ast = { git = "https://github.com/RustPython/Parser.git", version = "0.4.0", rev = "00d2f1d1a7522ef9c85c10dfa5f0bb7178dee655" }
-# rustpython-format = { git = "https://github.com/RustPython/Parser.git", version = "0.4.0", rev = "00d2f1d1a7522ef9c85c10dfa5f0bb7178dee655" }
-# rustpython-literal = { path = "../RustPython-parser/literal" }
-# rustpython-parser-core = { path = "../RustPython-parser/core" }
-# rustpython-parser = { path = "../RustPython-parser/parser" }
-# rustpython-ast = { path = "../RustPython-parser/ast" }
-# rustpython-format = { path = "../RustPython-parser/format" }
+# Ruff tag 0.15.6 is based on commit e4c7f357777a2fdd34dbe6a98b1b7d3e7488f675
+# at the time of this capture. We use the commit hash to ensure reproducible builds.
+ruff_python_parser = { git = "https://github.com/astral-sh/ruff.git", rev = "e4c7f357777a2fdd34dbe6a98b1b7d3e7488f675" }
+ruff_python_ast = { git = "https://github.com/astral-sh/ruff.git", rev = "e4c7f357777a2fdd34dbe6a98b1b7d3e7488f675" }
+ruff_text_size = { git = "https://github.com/astral-sh/ruff.git", rev = "e4c7f357777a2fdd34dbe6a98b1b7d3e7488f675" }
+ruff_source_file = { git = "https://github.com/astral-sh/ruff.git", rev = "e4c7f357777a2fdd34dbe6a98b1b7d3e7488f675" }
 
-ahash = "0.8.11"
-ascii = "1.0"
-bitflags = "2.4.1"
+phf = { version = "0.13.1", default-features = false, features = ["macros"]}
+ahash = "0.8.12"
+ascii = "1.1"
+bitflags = "2.11.0"
+bitflagset = "0.0.3"
 bstr = "1"
+bytes = "1.11.1"
 cfg-if = "1.0"
-chrono = "0.4.37"
-crossbeam-utils = "0.8.19"
+chrono = { version = "0.4.44", default-features = false, features = ["clock", "oldtime", "std"] }
+constant_time_eq = "0.4"
+criterion = { version = "0.8", features = ["html_reports"] }
+crossbeam-utils = "0.8.21"
 flame = "0.2.2"
-getrandom = "0.2.12"
+getrandom = { version = "0.3", features = ["std"] }
 glob = "0.3"
 hex = "0.4.3"
-indexmap = { version = "2.2.6", features = ["std"] }
-insta = "1.38.0"
-itertools = "0.11.0"
-is-macro = "0.3.0"
-junction = "1.0.0"
-libc = "0.2.153"
-log = "0.4.16"
-nix = { version = "0.29", features = ["fs", "user", "process", "term", "time", "signal", "ioctl", "socket", "sched", "zerocopy", "dir", "hostname", "net", "poll"] }
-malachite-bigint = "0.2.0"
-malachite-q = "0.4.4"
-malachite-base = "0.4.4"
-memchr = "2.7.2"
-num-complex = "0.4.0"
-num-integer = "0.1.44"
+indexmap = { version = "2.13.0", features = ["std"] }
+insta = "1.46"
+itertools = "0.14.0"
+is-macro = "0.3.7"
+junction = "1.4.2"
+libc = "0.2.183"
+libffi = "5"
+log = "0.4.29"
+nix = { version = "0.30", features = ["fs", "user", "process", "term", "time", "signal", "ioctl", "socket", "sched", "zerocopy", "dir", "hostname", "net", "poll"] }
+malachite-bigint = "0.9.1"
+malachite-q = "0.9.1"
+malachite-base = "0.9.1"
+memchr = "2.8.0"
+num-complex = "0.4.6"
+num-integer = "0.1.46"
 num-traits = "0.2"
 num_enum = { version = "0.7", default-features = false }
-once_cell = "1.19.0"
-parking_lot = "0.12.1"
-paste = "1.0.7"
-rand = "0.8.5"
-rustix = { version = "0.38", features = ["event"] }
-rustyline = "14.0.0"
-serde = { version = "1.0.133", default-features = false }
-schannel = "0.1.22"
+optional = "0.5"
+parking_lot = "0.12.3"
+paste = "1.0.15"
+proc-macro2 = "1.0.105"
+pymath = { version = "0.2.0", features = ["mul_add", "malachite-bigint", "complex"] }
+quote = "1.0.45"
+radium = "1.1.1"
+rand = "0.9"
+rand_core = { version = "0.9", features = ["os_rng"] }
+rustix = { version = "1.1", features = ["event"] }
+rustyline = "17.0.1"
+serde = { package = "serde_core", version = "1.0.225", default-features = false, features = ["alloc"] }
+schannel = "0.1.28"
+scoped-tls = "1"
+scopeguard = "1"
 static_assertions = "1.1"
-strum = "0.26"
-strum_macros = "0.26"
-syn = "1.0.109"
-thiserror = "1.0"
-thread_local = "1.1.4"
-unicode_names2 = "1.2.0"
-widestring = "1.1.0"
-windows-sys = "0.52.0"
-wasm-bindgen = "0.2.92"
+strum = "0.27"
+strum_macros = "0.28"
+syn = "2"
+thiserror = "2.0"
+thread_local = "1.1.9"
+unicode-casing = "0.1.1"
+unic-char-property = "0.9.0"
+unic-normal = "0.9.0"
+unic-ucd-age = "0.9.0"
+unic-ucd-bidi = "0.9.0"
+unic-ucd-category = "0.9.0"
+unic-ucd-ident = "0.9.0"
+unicode_names2 = "2.0.0"
+unicode-bidi-mirroring = "0.4"
+widestring = "1.2.0"
+windows-sys = "0.61.2"
+wasm-bindgen = "0.2.106"
 
 # Lints
 
 [workspace.lints.rust]
 unsafe_code = "allow"
+unsafe_op_in_unsafe_fn = "deny"
+elided_lifetimes_in_paths = "warn"
 
 [workspace.lints.clippy]
+alloc_instead_of_core = "warn"
+std_instead_of_alloc = "warn"
+std_instead_of_core = "warn"
 perf = "warn"
 style = "warn"
 complexity = "warn"

DEVELOPMENT.md

@@ -19,13 +19,13 @@ The contents of the Development Guide include:
 
 RustPython requires the following:
 
-- Rust latest stable version (e.g 1.69.0 as of Apr 20 2023)
+- Rust latest stable version (e.g 1.92.0 as of Jan 7 2026)
     - To check Rust version: `rustc --version` 
     - If you have `rustup` on your system, enter to update to the latest
       stable version: `rustup update stable`
     - If you do not have Rust installed, use [rustup](https://rustup.rs/) to
       do so.
-- CPython version 3.12 or higher
+- CPython version 3.14 or higher
     - CPython can be installed by your operating system's package manager,
       from the [Python website](https://www.python.org/downloads/), or
       using a third-party distribution, such as 
@@ -65,7 +65,7 @@ $ pytest -v
 Rust unit tests can be run with `cargo`:
 
 ```shell
-$ cargo test --workspace --exclude rustpython_wasm
+$ cargo test --workspace --exclude rustpython_wasm --exclude rustpython-venvlauncher
 ```
 
 Python unit tests can be run by compiling RustPython and running the test module:
@@ -95,6 +95,41 @@ To run only `test_cmath` (located at `Lib/test/test_cmath`) verbosely:
 $ cargo run --release -- -m test test_cmath -v
 ```
 
+### Testing on Linux from macOS
+
+You can test RustPython on Linux from macOS using Apple's `container` CLI.
+
+**Setup (one-time):**
+
+```shell
+# Install container CLI
+$ brew install container
+
+# Disable Rosetta requirement for arm64-only builds
+$ defaults write com.apple.container.defaults build.rosetta -bool false
+
+# Build the development image
+$ container build --arch arm64 -t rustpython-dev -f .devcontainer/Dockerfile .
+```
+
+**Running tests:**
+
+```shell
+# Start a persistent container in background (8GB memory, 4 CPUs for compilation)
+$ container run -d --name rustpython-test -m 8G -c 4 \
+    --mount type=bind,source=$(pwd),target=/workspace \
+    -w /workspace rustpython-dev sleep infinity
+
+# Run tests inside the container
+$ container exec rustpython-test sh -c "cargo run --release -- -m test test_ensurepip"
+
+# Run any command
+$ container exec rustpython-test sh -c "cargo test --workspace"
+
+# Stop and remove the container when done
+$ container rm -f rustpython-test
+```
+
 ## Profiling
 
 To profile RustPython, build it in `release` mode with the `flame-it` feature.
@@ -118,19 +153,19 @@ exists a raw html viewer which is currently broken, and we welcome a PR to fix i
 Understanding a new codebase takes time. Here's a brief view of the
 repository's structure:
 
-- `compiler/src`: python compilation to bytecode
-  - `core/src`: python bytecode representation in rust structures
-  - `parser/src`: python lexing, parsing and ast
-- `derive/src`: Rust language extensions and macros specific to rustpython
+- `crates/compiler/src`: python compilation to bytecode
+  - `crates/compiler-core/src`: python bytecode representation in rust structures
+- `crates/derive/src` and `crates/derive-impl/src`: Rust language extensions and macros specific to rustpython
 - `Lib`: Carefully selected / copied files from CPython sourcecode. This is
    the python side of the standard library.
   - `test`: CPython test suite
-- `vm/src`: python virtual machine
+- `crates/vm/src`: python virtual machine
   - `builtins`: Builtin functions and types
   - `stdlib`: Standard library parts implemented in rust.
 - `src`: using the other subcrates to bring rustpython to life.
-- `wasm`: Binary crate and resources for WebAssembly build
-- `extra_tests`: extra integration test snippets as a supplement to `Lib/test`
+- `crates/wasm`: Binary crate and resources for WebAssembly build
+- `extra_tests`: extra integration test snippets as a supplement to `Lib/test`.
+  Add new RustPython-only regression tests here; do not place new tests under `Lib/test`.
 
 ## Understanding Internals
 
@@ -140,9 +175,9 @@ implementation is found in the `src` directory (specifically, `src/lib.rs`).
 
 The top-level `rustpython` binary depends on several lower-level crates including:
 
-- `rustpython-parser` (implementation in `compiler/parser/src`)
-- `rustpython-compiler` (implementation in `compiler/src`)
-- `rustpython-vm` (implementation in `vm/src`)
+- `ruff_python_parser` and `ruff_python_ast` (external dependencies from the Ruff project)
+- `rustpython-compiler` (implementation in `crates/compiler/src`)
+- `rustpython-vm` (implementation in `crates/vm/src`)
 
 Together, these crates provide the functions of a programming language and
 enable a line of code to go through a series of steps:
@@ -153,31 +188,26 @@ enable a line of code to go through a series of steps:
 - compile the AST into bytecode
 - execute the bytecode in the virtual machine (VM).
 
-### rustpython-parser
+### Parser and AST
 
-This crate contains the lexer and parser to convert a line of code to
-an Abstract Syntax Tree (AST):
+RustPython uses the Ruff project's parser and AST implementation:
 
-- Lexer: `compiler/parser/src/lexer.rs` converts Python source code into tokens
-- Parser: `compiler/parser/src/parser.rs` takes the tokens generated by the lexer and parses
-  the tokens into an AST (Abstract Syntax Tree) where the nodes of the syntax
-  tree are Rust structs and enums.
-  - The Parser relies on `LALRPOP`, a Rust parser generator framework. The
-    LALRPOP definition of Python's grammar is in `compiler/parser/src/python.lalrpop`.
-  - More information on parsers and a tutorial can be found in the 
-    [LALRPOP book](https://lalrpop.github.io/lalrpop/).
-- AST: `compiler/ast/` implements in Rust the Python types and expressions
-  represented by the AST nodes.
+- Parser: `ruff_python_parser` is used to convert Python source code into tokens
+  and parse them into an Abstract Syntax Tree (AST)
+- AST: `ruff_python_ast` provides the Rust types and expressions represented by
+  the AST nodes
+- These are external dependencies maintained by the Ruff project
+- For more information, visit the [Ruff GitHub repository](https://github.com/astral-sh/ruff)
 
 ### rustpython-compiler
 
 The `rustpython-compiler` crate's purpose is to transform the AST (Abstract Syntax
 Tree) to bytecode. The implementation of the compiler is found in the
-`compiler/src` directory. The compiler implements Python's symbol table,
+`crates/compiler/src` directory. The compiler implements Python's symbol table,
 ast->bytecode compiler, and bytecode optimizer in Rust.
 
-Implementation of bytecode structure in Rust is found in the `compiler/core/src`
-directory. `compiler/core/src/bytecode.rs` contains the representation of
+Implementation of bytecode structure in Rust is found in the `crates/compiler-core/src`
+directory. `crates/compiler-core/src/bytecode.rs` contains the representation of
 instructions and operations in Rust. Further information about Python's
 bytecode instructions can be found in the
 [Python documentation](https://docs.python.org/3/library/dis.html#bytecodes).
@@ -185,14 +215,14 @@ bytecode instructions can be found in the
 ### rustpython-vm
 
 The `rustpython-vm` crate has the important job of running the virtual machine that
-executes Python's instructions. The `vm/src` directory contains code to
+executes Python's instructions. The `crates/vm/src` directory contains code to
 implement the read and evaluation loop that fetches and dispatches
 instructions. This directory also contains the implementation of the
-Python Standard Library modules in Rust (`vm/src/stdlib`). In Python
-everything can be represented as an object. The `vm/src/builtins` directory holds
+Python Standard Library modules in Rust (`crates/vm/src/stdlib`). In Python
+everything can be represented as an object. The `crates/vm/src/builtins` directory holds
 the Rust code used to represent different Python objects and their methods. The
 core implementation of what a Python object is can be found in
-`vm/src/object/core.rs`.
+`crates/vm/src/object/core.rs`.
 
 ### Code generation
 

LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2020 RustPython Team
+Copyright (c) 2025 RustPython Team
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

Lib/_aix_support.py

@@ -0,0 +1,108 @@
+"""Shared AIX support functions."""
+
+import sys
+import sysconfig
+
+
+# Taken from _osx_support _read_output function
+def _read_cmd_output(commandstring, capture_stderr=False):
+    """Output from successful command execution or None"""
+    # Similar to os.popen(commandstring, "r").read(),
+    # but without actually using os.popen because that
+    # function is not usable during python bootstrap.
+    import os
+    import contextlib
+    fp = open("/tmp/_aix_support.%s"%(
+        os.getpid(),), "w+b")
+
+    with contextlib.closing(fp) as fp:
+        if capture_stderr:
+            cmd = "%s >'%s' 2>&1" % (commandstring, fp.name)
+        else:
+            cmd = "%s 2>/dev/null >'%s'" % (commandstring, fp.name)
+        return fp.read() if not os.system(cmd) else None
+
+
+def _aix_tag(vrtl, bd):
+    # type: (List[int], int) -> str
+    # Infer the ABI bitwidth from maxsize (assuming 64 bit as the default)
+    _sz = 32 if sys.maxsize == (2**31-1) else 64
+    _bd = bd if bd != 0 else 9988
+    # vrtl[version, release, technology_level]
+    return "aix-{:1x}{:1d}{:02d}-{:04d}-{}".format(vrtl[0], vrtl[1], vrtl[2], _bd, _sz)
+
+
+# extract version, release and technology level from a VRMF string
+def _aix_vrtl(vrmf):
+    # type: (str) -> List[int]
+    v, r, tl = vrmf.split(".")[:3]
+    return [int(v[-1]), int(r), int(tl)]
+
+
+def _aix_bos_rte():
+    # type: () -> Tuple[str, int]
+    """
+    Return a Tuple[str, int] e.g., ['7.1.4.34', 1806]
+    The fileset bos.rte represents the current AIX run-time level. It's VRMF and
+    builddate reflect the current ABI levels of the runtime environment.
+    If no builddate is found give a value that will satisfy pep425 related queries
+    """
+    # All AIX systems to have lslpp installed in this location
+    # subprocess may not be available during python bootstrap
+    try:
+        import subprocess
+        out = subprocess.check_output(["/usr/bin/lslpp", "-Lqc", "bos.rte"])
+    except ImportError:
+        out = _read_cmd_output("/usr/bin/lslpp -Lqc bos.rte")
+    out = out.decode("utf-8")
+    out = out.strip().split(":")  # type: ignore
+    _bd = int(out[-1]) if out[-1] != '' else 9988
+    return (str(out[2]), _bd)
+
+
+def aix_platform():
+    # type: () -> str
+    """
+    AIX filesets are identified by four decimal values: V.R.M.F.
+    V (version) and R (release) can be retrieved using ``uname``
+    Since 2007, starting with AIX 5.3 TL7, the M value has been
+    included with the fileset bos.rte and represents the Technology
+    Level (TL) of AIX. The F (Fix) value also increases, but is not
+    relevant for comparing releases and binary compatibility.
+    For binary compatibility the so-called builddate is needed.
+    Again, the builddate of an AIX release is associated with bos.rte.
+    AIX ABI compatibility is described  as guaranteed at: https://www.ibm.com/\
+    support/knowledgecenter/en/ssw_aix_72/install/binary_compatability.html
+
+    For pep425 purposes the AIX platform tag becomes:
+    "aix-{:1x}{:1d}{:02d}-{:04d}-{}".format(v, r, tl, builddate, bitsize)
+    e.g., "aix-6107-1415-32" for AIX 6.1 TL7 bd 1415, 32-bit
+    and, "aix-6107-1415-64" for AIX 6.1 TL7 bd 1415, 64-bit
+    """
+    vrmf, bd = _aix_bos_rte()
+    return _aix_tag(_aix_vrtl(vrmf), bd)
+
+
+# extract vrtl from the BUILD_GNU_TYPE as an int
+def _aix_bgt():
+    # type: () -> List[int]
+    gnu_type = sysconfig.get_config_var("BUILD_GNU_TYPE")
+    if not gnu_type:
+        raise ValueError("BUILD_GNU_TYPE is not defined")
+    return _aix_vrtl(vrmf=gnu_type)
+
+
+def aix_buildtag():
+    # type: () -> str
+    """
+    Return the platform_tag of the system Python was built on.
+    """
+    # AIX_BUILDDATE is defined by configure with:
+    # lslpp -Lcq bos.rte | awk -F:  '{ print $NF }'
+    build_date = sysconfig.get_config_var("AIX_BUILDDATE")
+    try:
+        build_date = int(build_date)
+    except (ValueError, TypeError):
+        raise ValueError(f"AIX_BUILDDATE is not defined or invalid: "
+                         f"{build_date!r}")
+    return _aix_tag(_aix_bgt(), build_date)

Lib/_android_support.py

@@ -0,0 +1,185 @@
+import io
+import sys
+from threading import RLock
+from time import sleep, time
+
+# The maximum length of a log message in bytes, including the level marker and
+# tag, is defined as LOGGER_ENTRY_MAX_PAYLOAD at
+# https://cs.android.com/android/platform/superproject/+/android-14.0.0_r1:system/logging/liblog/include/log/log.h;l=71.
+# Messages longer than this will be truncated by logcat. This limit has already
+# been reduced at least once in the history of Android (from 4076 to 4068 between
+# API level 23 and 26), so leave some headroom.
+MAX_BYTES_PER_WRITE = 4000
+
+# UTF-8 uses a maximum of 4 bytes per character, so limiting text writes to this
+# size ensures that we can always avoid exceeding MAX_BYTES_PER_WRITE.
+# However, if the actual number of bytes per character is smaller than that,
+# then we may still join multiple consecutive text writes into binary
+# writes containing a larger number of characters.
+MAX_CHARS_PER_WRITE = MAX_BYTES_PER_WRITE // 4
+
+
+# When embedded in an app on current versions of Android, there's no easy way to
+# monitor the C-level stdout and stderr. The testbed comes with a .c file to
+# redirect them to the system log using a pipe, but that wouldn't be convenient
+# or appropriate for all apps. So we redirect at the Python level instead.
+def init_streams(android_log_write, stdout_prio, stderr_prio):
+    if sys.executable:
+        return  # Not embedded in an app.
+
+    global logcat
+    logcat = Logcat(android_log_write)
+    sys.stdout = TextLogStream(stdout_prio, "python.stdout", sys.stdout)
+    sys.stderr = TextLogStream(stderr_prio, "python.stderr", sys.stderr)
+
+
+class TextLogStream(io.TextIOWrapper):
+    def __init__(self, prio, tag, original=None, **kwargs):
+        # Respect the -u option.
+        if original:
+            kwargs.setdefault("write_through", original.write_through)
+            fileno = original.fileno()
+        else:
+            fileno = None
+
+        # The default is surrogateescape for stdout and backslashreplace for
+        # stderr, but in the context of an Android log, readability is more
+        # important than reversibility.
+        kwargs.setdefault("encoding", "UTF-8")
+        kwargs.setdefault("errors", "backslashreplace")
+
+        super().__init__(BinaryLogStream(prio, tag, fileno), **kwargs)
+        self._lock = RLock()
+        self._pending_bytes = []
+        self._pending_bytes_count = 0
+
+    def __repr__(self):
+        return f"<TextLogStream {self.buffer.tag!r}>"
+
+    def write(self, s):
+        if not isinstance(s, str):
+            raise TypeError(
+                f"write() argument must be str, not {type(s).__name__}")
+
+        # In case `s` is a str subclass that writes itself to stdout or stderr
+        # when we call its methods, convert it to an actual str.
+        s = str.__str__(s)
+
+        # We want to emit one log message per line wherever possible, so split
+        # the string into lines first. Note that "".splitlines() == [], so
+        # nothing will be logged for an empty string.
+        with self._lock:
+            for line in s.splitlines(keepends=True):
+                while line:
+                    chunk = line[:MAX_CHARS_PER_WRITE]
+                    line = line[MAX_CHARS_PER_WRITE:]
+                    self._write_chunk(chunk)
+
+        return len(s)
+
+    # The size and behavior of TextIOWrapper's buffer is not part of its public
+    # API, so we handle buffering ourselves to avoid truncation.
+    def _write_chunk(self, s):
+        b = s.encode(self.encoding, self.errors)
+        if self._pending_bytes_count + len(b) > MAX_BYTES_PER_WRITE:
+            self.flush()
+
+        self._pending_bytes.append(b)
+        self._pending_bytes_count += len(b)
+        if (
+            self.write_through
+            or b.endswith(b"\n")
+            or self._pending_bytes_count > MAX_BYTES_PER_WRITE
+        ):
+            self.flush()
+
+    def flush(self):
+        with self._lock:
+            self.buffer.write(b"".join(self._pending_bytes))
+            self._pending_bytes.clear()
+            self._pending_bytes_count = 0
+
+    # Since this is a line-based logging system, line buffering cannot be turned
+    # off, i.e. a newline always causes a flush.
+    @property
+    def line_buffering(self):
+        return True
+
+
+class BinaryLogStream(io.RawIOBase):
+    def __init__(self, prio, tag, fileno=None):
+        self.prio = prio
+        self.tag = tag
+        self._fileno = fileno
+
+    def __repr__(self):
+        return f"<BinaryLogStream {self.tag!r}>"
+
+    def writable(self):
+        return True
+
+    def write(self, b):
+        if type(b) is not bytes:
+            try:
+                b = bytes(memoryview(b))
+            except TypeError:
+                raise TypeError(
+                    f"write() argument must be bytes-like, not {type(b).__name__}"
+                ) from None
+
+        # Writing an empty string to the stream should have no effect.
+        if b:
+            logcat.write(self.prio, self.tag, b)
+        return len(b)
+
+    # This is needed by the test suite --timeout option, which uses faulthandler.
+    def fileno(self):
+        if self._fileno is None:
+            raise io.UnsupportedOperation("fileno")
+        return self._fileno
+
+
+# When a large volume of data is written to logcat at once, e.g. when a test
+# module fails in --verbose3 mode, there's a risk of overflowing logcat's own
+# buffer and losing messages. We avoid this by imposing a rate limit using the
+# token bucket algorithm, based on a conservative estimate of how fast `adb
+# logcat` can consume data.
+MAX_BYTES_PER_SECOND = 1024 * 1024
+
+# The logcat buffer size of a device can be determined by running `logcat -g`.
+# We set the token bucket size to half of the buffer size of our current minimum
+# API level, because other things on the system will be producing messages as
+# well.
+BUCKET_SIZE = 128 * 1024
+
+# https://cs.android.com/android/platform/superproject/+/android-14.0.0_r1:system/logging/liblog/include/log/log_read.h;l=39
+PER_MESSAGE_OVERHEAD = 28
+
+
+class Logcat:
+    def __init__(self, android_log_write):
+        self.android_log_write = android_log_write
+        self._lock = RLock()
+        self._bucket_level = 0
+        self._prev_write_time = time()
+
+    def write(self, prio, tag, message):
+        # Encode null bytes using "modified UTF-8" to avoid them truncating the
+        # message.
+        message = message.replace(b"\x00", b"\xc0\x80")
+
+        with self._lock:
+            now = time()
+            self._bucket_level += (
+                (now - self._prev_write_time) * MAX_BYTES_PER_SECOND)
+
+            # If the bucket level is still below zero, the clock must have gone
+            # backwards, so reset it to zero and continue.
+            self._bucket_level = max(0, min(self._bucket_level, BUCKET_SIZE))
+            self._prev_write_time = now
+
+            self._bucket_level -= PER_MESSAGE_OVERHEAD + len(tag) + len(message)
+            if self._bucket_level < 0:
+                sleep(-self._bucket_level / MAX_BYTES_PER_SECOND)
+
+            self.android_log_write(prio, tag, message)

Lib/_apple_support.py

@@ -0,0 +1,66 @@
+import io
+import sys
+
+
+def init_streams(log_write, stdout_level, stderr_level):
+    # Redirect stdout and stderr to the Apple system log. This method is
+    # invoked by init_apple_streams() (initconfig.c) if config->use_system_logger
+    # is enabled.
+    sys.stdout = SystemLog(log_write, stdout_level, errors=sys.stderr.errors)
+    sys.stderr = SystemLog(log_write, stderr_level, errors=sys.stderr.errors)
+
+
+class SystemLog(io.TextIOWrapper):
+    def __init__(self, log_write, level, **kwargs):
+        kwargs.setdefault("encoding", "UTF-8")
+        kwargs.setdefault("line_buffering", True)
+        super().__init__(LogStream(log_write, level), **kwargs)
+
+    def __repr__(self):
+        return f"<SystemLog (level {self.buffer.level})>"
+
+    def write(self, s):
+        if not isinstance(s, str):
+            raise TypeError(
+                f"write() argument must be str, not {type(s).__name__}")
+
+        # In case `s` is a str subclass that writes itself to stdout or stderr
+        # when we call its methods, convert it to an actual str.
+        s = str.__str__(s)
+
+        # We want to emit one log message per line, so split
+        # the string before sending it to the superclass.
+        for line in s.splitlines(keepends=True):
+            super().write(line)
+
+        return len(s)
+
+
+class LogStream(io.RawIOBase):
+    def __init__(self, log_write, level):
+        self.log_write = log_write
+        self.level = level
+
+    def __repr__(self):
+        return f"<LogStream (level {self.level!r})>"
+
+    def writable(self):
+        return True
+
+    def write(self, b):
+        if type(b) is not bytes:
+            try:
+                b = bytes(memoryview(b))
+            except TypeError:
+                raise TypeError(
+                    f"write() argument must be bytes-like, not {type(b).__name__}"
+                ) from None
+
+        # Writing an empty string to the stream should have no effect.
+        if b:
+            # Encode null bytes using "modified UTF-8" to avoid truncating the
+            # message. This should not affect the return value, as the caller
+            # may be expecting it to match the length of the input.
+            self.log_write(self.level, b.replace(b"\x00", b"\xc0\x80"))
+
+        return len(b)

Lib/_collections_abc.py

@@ -85,6 +85,10 @@ dict_values = type({}.values())
 dict_items = type({}.items())
 ## misc ##
 mappingproxy = type(type.__dict__)
+def _get_framelocalsproxy():
+    return type(sys._getframe().f_locals)
+framelocalsproxy = _get_framelocalsproxy()
+del _get_framelocalsproxy
 generator = type((lambda: (yield))())
 ## coroutine ##
 async def _coro(): pass
@@ -481,9 +485,10 @@ class _CallableGenericAlias(GenericAlias):
     def __repr__(self):
         if len(self.__args__) == 2 and _is_param_expr(self.__args__[0]):
             return super().__repr__()
+        from annotationlib import type_repr
         return (f'collections.abc.Callable'
-                f'[[{", ".join([_type_repr(a) for a in self.__args__[:-1]])}], '
-                f'{_type_repr(self.__args__[-1])}]')
+                f'[[{", ".join([type_repr(a) for a in self.__args__[:-1]])}], '
+                f'{type_repr(self.__args__[-1])}]')
 
     def __reduce__(self):
         args = self.__args__
@@ -520,23 +525,6 @@ def _is_param_expr(obj):
     names = ('ParamSpec', '_ConcatenateGenericAlias')
     return obj.__module__ == 'typing' and any(obj.__name__ == name for name in names)
 
-def _type_repr(obj):
-    """Return the repr() of an object, special-casing types (internal helper).
-
-    Copied from :mod:`typing` since collections.abc
-    shouldn't depend on that module.
-    (Keep this roughly in sync with the typing version.)
-    """
-    if isinstance(obj, type):
-        if obj.__module__ == 'builtins':
-            return obj.__qualname__
-        return f'{obj.__module__}.{obj.__qualname__}'
-    if obj is Ellipsis:
-        return '...'
-    if isinstance(obj, FunctionType):
-        return obj.__name__
-    return repr(obj)
-
 
 class Callable(metaclass=ABCMeta):
 
@@ -836,6 +824,7 @@ class Mapping(Collection):
     __reversed__ = None
 
 Mapping.register(mappingproxy)
+Mapping.register(framelocalsproxy)
 
 
 class MappingView(Sized):
@@ -973,7 +962,7 @@ class MutableMapping(Mapping):
 
     def update(self, other=(), /, **kwds):
         ''' D.update([E, ]**F) -> None.  Update D from mapping/iterable E and F.
-            If E present and has a .keys() method, does:     for k in E: D[k] = E[k]
+            If E present and has a .keys() method, does:     for k in E.keys(): D[k] = E[k]
             If E present and lacks .keys() method, does:     for (k, v) in E: D[k] = v
             In either case, this is followed by: for k, v in F.items(): D[k] = v
         '''
@@ -1068,6 +1057,7 @@ class Sequence(Reversible, Collection):
 
 Sequence.register(tuple)
 Sequence.register(str)
+Sequence.register(bytes)
 Sequence.register(range)
 Sequence.register(memoryview)
 
@@ -1078,7 +1068,7 @@ class _DeprecateByteStringMeta(ABCMeta):
 
             warnings._deprecated(
                 "collections.abc.ByteString",
-                remove=(3, 14),
+                remove=(3, 17),
             )
         return super().__new__(cls, name, bases, namespace, **kwargs)
 
@@ -1087,14 +1077,18 @@ class _DeprecateByteStringMeta(ABCMeta):
 
         warnings._deprecated(
             "collections.abc.ByteString",
-            remove=(3, 14),
+            remove=(3, 17),
         )
         return super().__instancecheck__(instance)
 
 class ByteString(Sequence, metaclass=_DeprecateByteStringMeta):
-    """This unifies bytes and bytearray.
+    """Deprecated ABC serving as a common supertype of ``bytes`` and ``bytearray``.
 
-    XXX Should add all their methods.
+    This ABC is scheduled for removal in Python 3.17.
+    Use ``isinstance(obj, collections.abc.Buffer)`` to test if ``obj``
+    implements the buffer protocol at runtime. For use in type annotations,
+    either use ``Buffer`` or a union that explicitly specifies the types your
+    code supports (e.g., ``bytes | bytearray | memoryview``).
     """
 
     __slots__ = ()
@@ -1170,4 +1164,4 @@ class MutableSequence(Sequence):
 
 
 MutableSequence.register(list)
-MutableSequence.register(bytearray)  # Multiply inheriting, see ByteString
+MutableSequence.register(bytearray)

Lib/_colorize.py

@@ -0,0 +1,355 @@
+import os
+import sys
+
+from collections.abc import Callable, Iterator, Mapping
+from dataclasses import dataclass, field, Field
+
+COLORIZE = True
+
+
+# types
+if False:
+    from typing import IO, Self, ClassVar
+    _theme: Theme
+
+
+class ANSIColors:
+    RESET = "\x1b[0m"
+
+    BLACK = "\x1b[30m"
+    BLUE = "\x1b[34m"
+    CYAN = "\x1b[36m"
+    GREEN = "\x1b[32m"
+    GREY = "\x1b[90m"
+    MAGENTA = "\x1b[35m"
+    RED = "\x1b[31m"
+    WHITE = "\x1b[37m"  # more like LIGHT GRAY
+    YELLOW = "\x1b[33m"
+
+    BOLD = "\x1b[1m"
+    BOLD_BLACK = "\x1b[1;30m"  # DARK GRAY
+    BOLD_BLUE = "\x1b[1;34m"
+    BOLD_CYAN = "\x1b[1;36m"
+    BOLD_GREEN = "\x1b[1;32m"
+    BOLD_MAGENTA = "\x1b[1;35m"
+    BOLD_RED = "\x1b[1;31m"
+    BOLD_WHITE = "\x1b[1;37m"  # actual WHITE
+    BOLD_YELLOW = "\x1b[1;33m"
+
+    # intense = like bold but without being bold
+    INTENSE_BLACK = "\x1b[90m"
+    INTENSE_BLUE = "\x1b[94m"
+    INTENSE_CYAN = "\x1b[96m"
+    INTENSE_GREEN = "\x1b[92m"
+    INTENSE_MAGENTA = "\x1b[95m"
+    INTENSE_RED = "\x1b[91m"
+    INTENSE_WHITE = "\x1b[97m"
+    INTENSE_YELLOW = "\x1b[93m"
+
+    BACKGROUND_BLACK = "\x1b[40m"
+    BACKGROUND_BLUE = "\x1b[44m"
+    BACKGROUND_CYAN = "\x1b[46m"
+    BACKGROUND_GREEN = "\x1b[42m"
+    BACKGROUND_MAGENTA = "\x1b[45m"
+    BACKGROUND_RED = "\x1b[41m"
+    BACKGROUND_WHITE = "\x1b[47m"
+    BACKGROUND_YELLOW = "\x1b[43m"
+
+    INTENSE_BACKGROUND_BLACK = "\x1b[100m"
+    INTENSE_BACKGROUND_BLUE = "\x1b[104m"
+    INTENSE_BACKGROUND_CYAN = "\x1b[106m"
+    INTENSE_BACKGROUND_GREEN = "\x1b[102m"
+    INTENSE_BACKGROUND_MAGENTA = "\x1b[105m"
+    INTENSE_BACKGROUND_RED = "\x1b[101m"
+    INTENSE_BACKGROUND_WHITE = "\x1b[107m"
+    INTENSE_BACKGROUND_YELLOW = "\x1b[103m"
+
+
+ColorCodes = set()
+NoColors = ANSIColors()
+
+for attr, code in ANSIColors.__dict__.items():
+    if not attr.startswith("__"):
+        ColorCodes.add(code)
+        setattr(NoColors, attr, "")
+
+
+#
+# Experimental theming support (see gh-133346)
+#
+
+# - Create a theme by copying an existing `Theme` with one or more sections
+#   replaced, using `default_theme.copy_with()`;
+# - create a theme section by copying an existing `ThemeSection` with one or
+#   more colors replaced, using for example `default_theme.syntax.copy_with()`;
+# - create a theme from scratch by instantiating a `Theme` data class with
+#   the required sections (which are also dataclass instances).
+#
+# Then call `_colorize.set_theme(your_theme)` to set it.
+#
+# Put your theme configuration in $PYTHONSTARTUP for the interactive shell,
+# or sitecustomize.py in your virtual environment or Python installation for
+# other uses.  Your applications can call `_colorize.set_theme()` too.
+#
+# Note that thanks to the dataclasses providing default values for all fields,
+# creating a new theme or theme section from scratch is possible without
+# specifying all keys.
+#
+# For example, here's a theme that makes punctuation and operators less prominent:
+#
+#   try:
+#       from _colorize import set_theme, default_theme, Syntax, ANSIColors
+#   except ImportError:
+#       pass
+#   else:
+#       theme_with_dim_operators = default_theme.copy_with(
+#           syntax=Syntax(op=ANSIColors.INTENSE_BLACK),
+#       )
+#       set_theme(theme_with_dim_operators)
+#       del set_theme, default_theme, Syntax, ANSIColors, theme_with_dim_operators
+#
+# Guarding the import ensures that your .pythonstartup file will still work in
+# Python 3.13 and older. Deleting the variables ensures they don't remain in your
+# interactive shell's global scope.
+
+class ThemeSection(Mapping[str, str]):
+    """A mixin/base class for theme sections.
+
+    It enables dictionary access to a section, as well as implements convenience
+    methods.
+    """
+
+    # The two types below are just that: types to inform the type checker that the
+    # mixin will work in context of those fields existing
+    __dataclass_fields__: ClassVar[dict[str, Field[str]]]
+    _name_to_value: Callable[[str], str]
+
+    def __post_init__(self) -> None:
+        name_to_value = {}
+        for color_name in self.__dataclass_fields__:
+            name_to_value[color_name] = getattr(self, color_name)
+        super().__setattr__('_name_to_value', name_to_value.__getitem__)
+
+    def copy_with(self, **kwargs: str) -> Self:
+        color_state: dict[str, str] = {}
+        for color_name in self.__dataclass_fields__:
+            color_state[color_name] = getattr(self, color_name)
+        color_state.update(kwargs)
+        return type(self)(**color_state)
+
+    @classmethod
+    def no_colors(cls) -> Self:
+        color_state: dict[str, str] = {}
+        for color_name in cls.__dataclass_fields__:
+            color_state[color_name] = ""
+        return cls(**color_state)
+
+    def __getitem__(self, key: str) -> str:
+        return self._name_to_value(key)
+
+    def __len__(self) -> int:
+        return len(self.__dataclass_fields__)
+
+    def __iter__(self) -> Iterator[str]:
+        return iter(self.__dataclass_fields__)
+
+
+@dataclass(frozen=True, kw_only=True)
+class Argparse(ThemeSection):
+    usage: str = ANSIColors.BOLD_BLUE
+    prog: str = ANSIColors.BOLD_MAGENTA
+    prog_extra: str = ANSIColors.MAGENTA
+    heading: str = ANSIColors.BOLD_BLUE
+    summary_long_option: str = ANSIColors.CYAN
+    summary_short_option: str = ANSIColors.GREEN
+    summary_label: str = ANSIColors.YELLOW
+    summary_action: str = ANSIColors.GREEN
+    long_option: str = ANSIColors.BOLD_CYAN
+    short_option: str = ANSIColors.BOLD_GREEN
+    label: str = ANSIColors.BOLD_YELLOW
+    action: str = ANSIColors.BOLD_GREEN
+    reset: str = ANSIColors.RESET
+
+
+@dataclass(frozen=True)
+class Syntax(ThemeSection):
+    prompt: str = ANSIColors.BOLD_MAGENTA
+    keyword: str = ANSIColors.BOLD_BLUE
+    keyword_constant: str = ANSIColors.BOLD_BLUE
+    builtin: str = ANSIColors.CYAN
+    comment: str = ANSIColors.RED
+    string: str = ANSIColors.GREEN
+    number: str = ANSIColors.YELLOW
+    op: str = ANSIColors.RESET
+    definition: str = ANSIColors.BOLD
+    soft_keyword: str = ANSIColors.BOLD_BLUE
+    reset: str = ANSIColors.RESET
+
+
+@dataclass(frozen=True)
+class Traceback(ThemeSection):
+    type: str = ANSIColors.BOLD_MAGENTA
+    message: str = ANSIColors.MAGENTA
+    filename: str = ANSIColors.MAGENTA
+    line_no: str = ANSIColors.MAGENTA
+    frame: str = ANSIColors.MAGENTA
+    error_highlight: str = ANSIColors.BOLD_RED
+    error_range: str = ANSIColors.RED
+    reset: str = ANSIColors.RESET
+
+
+@dataclass(frozen=True)
+class Unittest(ThemeSection):
+    passed: str = ANSIColors.GREEN
+    warn: str = ANSIColors.YELLOW
+    fail: str = ANSIColors.RED
+    fail_info: str = ANSIColors.BOLD_RED
+    reset: str = ANSIColors.RESET
+
+
+@dataclass(frozen=True)
+class Theme:
+    """A suite of themes for all sections of Python.
+
+    When adding a new one, remember to also modify `copy_with` and `no_colors`
+    below.
+    """
+    argparse: Argparse = field(default_factory=Argparse)
+    syntax: Syntax = field(default_factory=Syntax)
+    traceback: Traceback = field(default_factory=Traceback)
+    unittest: Unittest = field(default_factory=Unittest)
+
+    def copy_with(
+        self,
+        *,
+        argparse: Argparse | None = None,
+        syntax: Syntax | None = None,
+        traceback: Traceback | None = None,
+        unittest: Unittest | None = None,
+    ) -> Self:
+        """Return a new Theme based on this instance with some sections replaced.
+
+        Themes are immutable to protect against accidental modifications that
+        could lead to invalid terminal states.
+        """
+        return type(self)(
+            argparse=argparse or self.argparse,
+            syntax=syntax or self.syntax,
+            traceback=traceback or self.traceback,
+            unittest=unittest or self.unittest,
+        )
+
+    @classmethod
+    def no_colors(cls) -> Self:
+        """Return a new Theme where colors in all sections are empty strings.
+
+        This allows writing user code as if colors are always used. The color
+        fields will be ANSI color code strings when colorization is desired
+        and possible, and empty strings otherwise.
+        """
+        return cls(
+            argparse=Argparse.no_colors(),
+            syntax=Syntax.no_colors(),
+            traceback=Traceback.no_colors(),
+            unittest=Unittest.no_colors(),
+        )
+
+
+def get_colors(
+    colorize: bool = False, *, file: IO[str] | IO[bytes] | None = None
+) -> ANSIColors:
+    if colorize or can_colorize(file=file):
+        return ANSIColors()
+    else:
+        return NoColors
+
+
+def decolor(text: str) -> str:
+    """Remove ANSI color codes from a string."""
+    for code in ColorCodes:
+        text = text.replace(code, "")
+    return text
+
+
+def can_colorize(*, file: IO[str] | IO[bytes] | None = None) -> bool:
+
+    def _safe_getenv(k: str, fallback: str | None = None) -> str | None:
+        """Exception-safe environment retrieval. See gh-128636."""
+        try:
+            return os.environ.get(k, fallback)
+        except Exception:
+            return fallback
+
+    if file is None:
+        file = sys.stdout
+
+    if not sys.flags.ignore_environment:
+        if _safe_getenv("PYTHON_COLORS") == "0":
+            return False
+        if _safe_getenv("PYTHON_COLORS") == "1":
+            return True
+    if _safe_getenv("NO_COLOR"):
+        return False
+    if not COLORIZE:
+        return False
+    if _safe_getenv("FORCE_COLOR"):
+        return True
+    if _safe_getenv("TERM") == "dumb":
+        return False
+
+    if not hasattr(file, "fileno"):
+        return False
+
+    if sys.platform == "win32":
+        try:
+            import nt
+
+            if not nt._supports_virtual_terminal():
+                return False
+        except (ImportError, AttributeError):
+            return False
+
+    try:
+        return os.isatty(file.fileno())
+    except OSError:
+        return hasattr(file, "isatty") and file.isatty()
+
+
+default_theme = Theme()
+theme_no_color = default_theme.no_colors()
+
+
+def get_theme(
+    *,
+    tty_file: IO[str] | IO[bytes] | None = None,
+    force_color: bool = False,
+    force_no_color: bool = False,
+) -> Theme:
+    """Returns the currently set theme, potentially in a zero-color variant.
+
+    In cases where colorizing is not possible (see `can_colorize`), the returned
+    theme contains all empty strings in all color definitions.
+    See `Theme.no_colors()` for more information.
+
+    It is recommended not to cache the result of this function for extended
+    periods of time because the user might influence theme selection by
+    the interactive shell, a debugger, or application-specific code. The
+    environment (including environment variable state and console configuration
+    on Windows) can also change in the course of the application life cycle.
+    """
+    if force_color or (not force_no_color and
+                       can_colorize(file=tty_file)):
+        return _theme
+    return theme_no_color
+
+
+def set_theme(t: Theme) -> None:
+    global _theme
+
+    if not isinstance(t, Theme):
+        raise ValueError(f"Expected Theme object, found {t}")
+
+    _theme = t
+
+
+set_theme(default_theme)

Lib/_compat_pickle.py

@@ -22,7 +22,6 @@ IMPORT_MAPPING = {
     'tkMessageBox': 'tkinter.messagebox',
     'ScrolledText': 'tkinter.scrolledtext',
     'Tkconstants': 'tkinter.constants',
-    'Tix': 'tkinter.tix',
     'ttk': 'tkinter.ttk',
     'Tkinter': 'tkinter',
     'markupbase': '_markupbase',
@@ -257,3 +256,4 @@ PYTHON3_IMPORTERROR_EXCEPTIONS = (
 
 for excname in PYTHON3_IMPORTERROR_EXCEPTIONS:
     REVERSE_NAME_MAPPING[('builtins', excname)] = ('exceptions', 'ImportError')
+del excname

Lib/_dummy_os.py

@@ -5,22 +5,30 @@ A shim of the os module containing only simple path-related utilities
 try:
     from os import *
 except ImportError:
-    import abc
+    import abc, sys
 
     def __getattr__(name):
-        raise OSError("no os specific module found")
+        if name in {"_path_normpath", "__path__"}:
+            raise AttributeError(name)
+        if name.isupper():
+            return 0
+        def dummy(*args, **kwargs):
+            import io
+            return io.UnsupportedOperation(f"{name}: no os specific module found")
+        dummy.__name__ = f"dummy_{name}"
+        return dummy
 
-    def _shim():
-        import _dummy_os, sys
-        sys.modules['os'] = _dummy_os
-        sys.modules['os.path'] = _dummy_os.path
+    sys.modules['os'] = sys.modules['posix'] = sys.modules[__name__]
 
     import posixpath as path
-    import sys
     sys.modules['os.path'] = path
     del sys
 
     sep = path.sep
+    supports_dir_fd = set()
+    supports_effective_ids = set()
+    supports_fd = set()
+    supports_follow_symlinks = set()
 
 
     def fspath(path):

Lib/_dummy_thread.py

@@ -11,15 +11,35 @@ Suggested usage is::
         import _dummy_thread as _thread
 
 """
+
 # Exports only things specified by thread documentation;
 # skipping obsolete synonyms allocate(), start_new(), exit_thread().
-__all__ = ['error', 'start_new_thread', 'exit', 'get_ident', 'allocate_lock',
-           'interrupt_main', 'LockType', 'RLock',
-           '_count']
+__all__ = [
+    "error",
+    "start_new_thread",
+    "exit",
+    "get_ident",
+    "allocate_lock",
+    "interrupt_main",
+    "LockType",
+    "RLock",
+    "_count",
+    "start_joinable_thread",
+    "daemon_threads_allowed",
+    "_shutdown",
+    "_make_thread_handle",
+    "_ThreadHandle",
+    "_get_main_thread_ident",
+    "_is_main_interpreter",
+    "_local",
+]
 
 # A dummy value
 TIMEOUT_MAX = 2**31
 
+# Main thread ident for dummy implementation
+_MAIN_THREAD_IDENT = -1
+
 # NOTE: this module can be imported early in the extension building process,
 # and so top level imports of other modules should be avoided.  Instead, all
 # imports are done when needed on a function-by-function basis.  Since threads
@@ -27,6 +47,7 @@ TIMEOUT_MAX = 2**31
 
 error = RuntimeError
 
+
 def start_new_thread(function, args, kwargs={}):
     """Dummy implementation of _thread.start_new_thread().
 
@@ -52,6 +73,7 @@ def start_new_thread(function, args, kwargs={}):
         pass
     except:
         import traceback
+
         traceback.print_exc()
     _main = True
     global _interrupt
@@ -59,10 +81,58 @@ def start_new_thread(function, args, kwargs={}):
         _interrupt = False
         raise KeyboardInterrupt
 
+
+def start_joinable_thread(function, handle=None, daemon=True):
+    """Dummy implementation of _thread.start_joinable_thread().
+
+    In dummy thread, we just run the function synchronously.
+    """
+    if handle is None:
+        handle = _ThreadHandle()
+    try:
+        function()
+    except SystemExit:
+        pass
+    except:
+        import traceback
+
+        traceback.print_exc()
+    handle._set_done()
+    return handle
+
+
+def daemon_threads_allowed():
+    """Dummy implementation of _thread.daemon_threads_allowed()."""
+    return True
+
+
+def _shutdown():
+    """Dummy implementation of _thread._shutdown()."""
+    pass
+
+
+def _make_thread_handle(ident):
+    """Dummy implementation of _thread._make_thread_handle()."""
+    handle = _ThreadHandle()
+    handle._ident = ident
+    return handle
+
+
+def _get_main_thread_ident():
+    """Dummy implementation of _thread._get_main_thread_ident()."""
+    return _MAIN_THREAD_IDENT
+
+
+def _is_main_interpreter():
+    """Dummy implementation of _thread._is_main_interpreter()."""
+    return True
+
+
 def exit():
     """Dummy implementation of _thread.exit()."""
     raise SystemExit
 
+
 def get_ident():
     """Dummy implementation of _thread.get_ident().
 
@@ -70,26 +140,31 @@ def get_ident():
     available, it is safe to assume that the current process is the
     only thread.  Thus a constant can be safely returned.
     """
-    return -1
+    return _MAIN_THREAD_IDENT
+
 
 def allocate_lock():
     """Dummy implementation of _thread.allocate_lock()."""
     return LockType()
 
+
 def stack_size(size=None):
     """Dummy implementation of _thread.stack_size()."""
     if size is not None:
         raise error("setting thread stack size not supported")
     return 0
 
+
 def _set_sentinel():
     """Dummy implementation of _thread._set_sentinel()."""
     return LockType()
 
+
 def _count():
     """Dummy implementation of _thread._count()."""
     return 0
 
+
 class LockType(object):
     """Class implementing dummy implementation of _thread.LockType.
 
@@ -125,6 +200,7 @@ class LockType(object):
             else:
                 if timeout > 0:
                     import time
+
                     time.sleep(timeout)
                 return False
 
@@ -153,14 +229,41 @@ class LockType(object):
             "locked" if self.locked_status else "unlocked",
             self.__class__.__module__,
             self.__class__.__qualname__,
-            hex(id(self))
+            hex(id(self)),
         )
 
+
+class _ThreadHandle:
+    """Dummy implementation of _thread._ThreadHandle."""
+
+    def __init__(self):
+        self._ident = _MAIN_THREAD_IDENT
+        self._done = False
+
+    @property
+    def ident(self):
+        return self._ident
+
+    def _set_done(self):
+        self._done = True
+
+    def is_done(self):
+        return self._done
+
+    def join(self, timeout=None):
+        # In dummy thread, thread is always done
+        return
+
+    def __repr__(self):
+        return f"<_ThreadHandle ident={self._ident}>"
+
+
 # Used to signal that interrupt_main was called in a "thread"
 _interrupt = False
 # True when not executing in a "thread"
 _main = True
 
+
 def interrupt_main():
     """Set _interrupt flag to True to have start_new_thread raise
     KeyboardInterrupt upon exiting."""
@@ -170,6 +273,7 @@ def interrupt_main():
         global _interrupt
         _interrupt = True
 
+
 class RLock:
     def __init__(self):
         self.locked_count = 0
@@ -190,7 +294,7 @@ class RLock:
         return True
 
     def locked(self):
-        return self.locked_status != 0
+        return self.locked_count != 0
 
     def __repr__(self):
         return "<%s %s.%s object owner=%s count=%s at %s>" % (
@@ -199,5 +303,36 @@ class RLock:
             self.__class__.__qualname__,
             get_ident() if self.locked_count else 0,
             self.locked_count,
-            hex(id(self))
+            hex(id(self)),
         )
+
+
+class _local:
+    """Dummy implementation of _thread._local (thread-local storage)."""
+
+    def __init__(self):
+        object.__setattr__(self, "_local__impl", {})
+
+    def __getattribute__(self, name):
+        if name.startswith("_local__"):
+            return object.__getattribute__(self, name)
+        impl = object.__getattribute__(self, "_local__impl")
+        try:
+            return impl[name]
+        except KeyError:
+            raise AttributeError(name)
+
+    def __setattr__(self, name, value):
+        if name.startswith("_local__"):
+            return object.__setattr__(self, name, value)
+        impl = object.__getattribute__(self, "_local__impl")
+        impl[name] = value
+
+    def __delattr__(self, name):
+        if name.startswith("_local__"):
+            return object.__delattr__(self, name)
+        impl = object.__getattribute__(self, "_local__impl")
+        try:
+            del impl[name]
+        except KeyError:
+            raise AttributeError(name)

Lib/_ios_support.py

@@ -0,0 +1,71 @@
+import sys
+try:
+    from ctypes import cdll, c_void_p, c_char_p, util
+except ImportError:
+    # ctypes is an optional module. If it's not present, we're limited in what
+    # we can tell about the system, but we don't want to prevent the module
+    # from working.
+    print("ctypes isn't available; iOS system calls will not be available", file=sys.stderr)
+    objc = None
+else:
+    # ctypes is available. Load the ObjC library, and wrap the objc_getClass,
+    # sel_registerName methods
+    lib = util.find_library("objc")
+    if lib is None:
+        # Failed to load the objc library
+        raise ImportError("ObjC runtime library couldn't be loaded")
+
+    objc = cdll.LoadLibrary(lib)
+    objc.objc_getClass.restype = c_void_p
+    objc.objc_getClass.argtypes = [c_char_p]
+    objc.sel_registerName.restype = c_void_p
+    objc.sel_registerName.argtypes = [c_char_p]
+
+
+def get_platform_ios():
+    # Determine if this is a simulator using the multiarch value
+    is_simulator = sys.implementation._multiarch.endswith("simulator")
+
+    # We can't use ctypes; abort
+    if not objc:
+        return None
+
+    # Most of the methods return ObjC objects
+    objc.objc_msgSend.restype = c_void_p
+    # All the methods used have no arguments.
+    objc.objc_msgSend.argtypes = [c_void_p, c_void_p]
+
+    # Equivalent of:
+    #   device = [UIDevice currentDevice]
+    UIDevice = objc.objc_getClass(b"UIDevice")
+    SEL_currentDevice = objc.sel_registerName(b"currentDevice")
+    device = objc.objc_msgSend(UIDevice, SEL_currentDevice)
+
+    # Equivalent of:
+    #   device_systemVersion = [device systemVersion]
+    SEL_systemVersion = objc.sel_registerName(b"systemVersion")
+    device_systemVersion = objc.objc_msgSend(device, SEL_systemVersion)
+
+    # Equivalent of:
+    #   device_systemName = [device systemName]
+    SEL_systemName = objc.sel_registerName(b"systemName")
+    device_systemName = objc.objc_msgSend(device, SEL_systemName)
+
+    # Equivalent of:
+    #   device_model = [device model]
+    SEL_model = objc.sel_registerName(b"model")
+    device_model = objc.objc_msgSend(device, SEL_model)
+
+    # UTF8String returns a const char*;
+    SEL_UTF8String = objc.sel_registerName(b"UTF8String")
+    objc.objc_msgSend.restype = c_char_p
+
+    # Equivalent of:
+    #   system = [device_systemName UTF8String]
+    #   release = [device_systemVersion UTF8String]
+    #   model = [device_model UTF8String]
+    system = objc.objc_msgSend(device_systemName, SEL_UTF8String).decode()
+    release = objc.objc_msgSend(device_systemVersion, SEL_UTF8String).decode()
+    model = objc.objc_msgSend(device_model, SEL_UTF8String).decode()
+
+    return system, release, model, is_simulator

Lib/_markupbase.py

@@ -13,7 +13,7 @@ _commentclose = re.compile(r'--\s*>')
 _markedsectionclose = re.compile(r']\s*]\s*>')
 
 # An analysis of the MS-Word extensions is available at
-# http://www.planetpublish.com/xmlarena/xap/Thursday/WordtoXML.pdf
+# http://web.archive.org/web/20060321153828/http://www.planetpublish.com/xmlarena/xap/Thursday/WordtoXML.pdf
 
 _msmarkedsectionclose = re.compile(r']\s*>')
 

Lib/_opcode_metadata.py

@@ -0,0 +1,371 @@
+# This file is generated by scripts/generate_opcode_metadata.py
+# for RustPython bytecode format (CPython 3.14 compatible opcode numbers).
+# Do not edit!
+
+_specializations = {
+    "RESUME": [
+        "RESUME_CHECK",
+    ],
+    "LOAD_CONST": [
+        "LOAD_CONST_MORTAL",
+        "LOAD_CONST_IMMORTAL",
+    ],
+    "TO_BOOL": [
+        "TO_BOOL_ALWAYS_TRUE",
+        "TO_BOOL_BOOL",
+        "TO_BOOL_INT",
+        "TO_BOOL_LIST",
+        "TO_BOOL_NONE",
+        "TO_BOOL_STR",
+    ],
+    "BINARY_OP": [
+        "BINARY_OP_MULTIPLY_INT",
+        "BINARY_OP_ADD_INT",
+        "BINARY_OP_SUBTRACT_INT",
+        "BINARY_OP_MULTIPLY_FLOAT",
+        "BINARY_OP_ADD_FLOAT",
+        "BINARY_OP_SUBTRACT_FLOAT",
+        "BINARY_OP_ADD_UNICODE",
+        "BINARY_OP_SUBSCR_LIST_INT",
+        "BINARY_OP_SUBSCR_LIST_SLICE",
+        "BINARY_OP_SUBSCR_TUPLE_INT",
+        "BINARY_OP_SUBSCR_STR_INT",
+        "BINARY_OP_SUBSCR_DICT",
+        "BINARY_OP_SUBSCR_GETITEM",
+        "BINARY_OP_EXTEND",
+        "BINARY_OP_INPLACE_ADD_UNICODE",
+    ],
+    "STORE_SUBSCR": [
+        "STORE_SUBSCR_DICT",
+        "STORE_SUBSCR_LIST_INT",
+    ],
+    "SEND": [
+        "SEND_GEN",
+    ],
+    "UNPACK_SEQUENCE": [
+        "UNPACK_SEQUENCE_TWO_TUPLE",
+        "UNPACK_SEQUENCE_TUPLE",
+        "UNPACK_SEQUENCE_LIST",
+    ],
+    "STORE_ATTR": [
+        "STORE_ATTR_INSTANCE_VALUE",
+        "STORE_ATTR_SLOT",
+        "STORE_ATTR_WITH_HINT",
+    ],
+    "LOAD_GLOBAL": [
+        "LOAD_GLOBAL_MODULE",
+        "LOAD_GLOBAL_BUILTIN",
+    ],
+    "LOAD_SUPER_ATTR": [
+        "LOAD_SUPER_ATTR_ATTR",
+        "LOAD_SUPER_ATTR_METHOD",
+    ],
+    "LOAD_ATTR": [
+        "LOAD_ATTR_INSTANCE_VALUE",
+        "LOAD_ATTR_MODULE",
+        "LOAD_ATTR_WITH_HINT",
+        "LOAD_ATTR_SLOT",
+        "LOAD_ATTR_CLASS",
+        "LOAD_ATTR_CLASS_WITH_METACLASS_CHECK",
+        "LOAD_ATTR_PROPERTY",
+        "LOAD_ATTR_GETATTRIBUTE_OVERRIDDEN",
+        "LOAD_ATTR_METHOD_WITH_VALUES",
+        "LOAD_ATTR_METHOD_NO_DICT",
+        "LOAD_ATTR_METHOD_LAZY_DICT",
+        "LOAD_ATTR_NONDESCRIPTOR_WITH_VALUES",
+        "LOAD_ATTR_NONDESCRIPTOR_NO_DICT",
+    ],
+    "COMPARE_OP": [
+        "COMPARE_OP_FLOAT",
+        "COMPARE_OP_INT",
+        "COMPARE_OP_STR",
+    ],
+    "CONTAINS_OP": [
+        "CONTAINS_OP_SET",
+        "CONTAINS_OP_DICT",
+    ],
+    "JUMP_BACKWARD": [
+        "JUMP_BACKWARD_NO_JIT",
+        "JUMP_BACKWARD_JIT",
+    ],
+    "FOR_ITER": [
+        "FOR_ITER_LIST",
+        "FOR_ITER_TUPLE",
+        "FOR_ITER_RANGE",
+        "FOR_ITER_GEN",
+    ],
+    "CALL": [
+        "CALL_BOUND_METHOD_EXACT_ARGS",
+        "CALL_PY_EXACT_ARGS",
+        "CALL_TYPE_1",
+        "CALL_STR_1",
+        "CALL_TUPLE_1",
+        "CALL_BUILTIN_CLASS",
+        "CALL_BUILTIN_O",
+        "CALL_BUILTIN_FAST",
+        "CALL_BUILTIN_FAST_WITH_KEYWORDS",
+        "CALL_LEN",
+        "CALL_ISINSTANCE",
+        "CALL_LIST_APPEND",
+        "CALL_METHOD_DESCRIPTOR_O",
+        "CALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS",
+        "CALL_METHOD_DESCRIPTOR_NOARGS",
+        "CALL_METHOD_DESCRIPTOR_FAST",
+        "CALL_ALLOC_AND_ENTER_INIT",
+        "CALL_PY_GENERAL",
+        "CALL_BOUND_METHOD_GENERAL",
+        "CALL_NON_PY_GENERAL",
+    ],
+    "CALL_KW": [
+        "CALL_KW_BOUND_METHOD",
+        "CALL_KW_PY",
+        "CALL_KW_NON_PY",
+    ],
+}
+
+_specialized_opmap = {
+    'BINARY_OP_ADD_FLOAT': 129,
+    'BINARY_OP_ADD_INT': 130,
+    'BINARY_OP_ADD_UNICODE': 131,
+    'BINARY_OP_EXTEND': 132,
+    'BINARY_OP_INPLACE_ADD_UNICODE': 3,
+    'BINARY_OP_MULTIPLY_FLOAT': 133,
+    'BINARY_OP_MULTIPLY_INT': 134,
+    'BINARY_OP_SUBSCR_DICT': 135,
+    'BINARY_OP_SUBSCR_GETITEM': 136,
+    'BINARY_OP_SUBSCR_LIST_INT': 137,
+    'BINARY_OP_SUBSCR_LIST_SLICE': 138,
+    'BINARY_OP_SUBSCR_STR_INT': 139,
+    'BINARY_OP_SUBSCR_TUPLE_INT': 140,
+    'BINARY_OP_SUBTRACT_FLOAT': 141,
+    'BINARY_OP_SUBTRACT_INT': 142,
+    'CALL_ALLOC_AND_ENTER_INIT': 143,
+    'CALL_BOUND_METHOD_EXACT_ARGS': 144,
+    'CALL_BOUND_METHOD_GENERAL': 145,
+    'CALL_BUILTIN_CLASS': 146,
+    'CALL_BUILTIN_FAST': 147,
+    'CALL_BUILTIN_FAST_WITH_KEYWORDS': 148,
+    'CALL_BUILTIN_O': 149,
+    'CALL_ISINSTANCE': 150,
+    'CALL_KW_BOUND_METHOD': 151,
+    'CALL_KW_NON_PY': 152,
+    'CALL_KW_PY': 153,
+    'CALL_LEN': 154,
+    'CALL_LIST_APPEND': 155,
+    'CALL_METHOD_DESCRIPTOR_FAST': 156,
+    'CALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS': 157,
+    'CALL_METHOD_DESCRIPTOR_NOARGS': 158,
+    'CALL_METHOD_DESCRIPTOR_O': 159,
+    'CALL_NON_PY_GENERAL': 160,
+    'CALL_PY_EXACT_ARGS': 161,
+    'CALL_PY_GENERAL': 162,
+    'CALL_STR_1': 163,
+    'CALL_TUPLE_1': 164,
+    'CALL_TYPE_1': 165,
+    'COMPARE_OP_FLOAT': 166,
+    'COMPARE_OP_INT': 167,
+    'COMPARE_OP_STR': 168,
+    'CONTAINS_OP_DICT': 169,
+    'CONTAINS_OP_SET': 170,
+    'FOR_ITER_GEN': 171,
+    'FOR_ITER_LIST': 172,
+    'FOR_ITER_RANGE': 173,
+    'FOR_ITER_TUPLE': 174,
+    'JUMP_BACKWARD_JIT': 175,
+    'JUMP_BACKWARD_NO_JIT': 176,
+    'LOAD_ATTR_CLASS': 177,
+    'LOAD_ATTR_CLASS_WITH_METACLASS_CHECK': 178,
+    'LOAD_ATTR_GETATTRIBUTE_OVERRIDDEN': 179,
+    'LOAD_ATTR_INSTANCE_VALUE': 180,
+    'LOAD_ATTR_METHOD_LAZY_DICT': 181,
+    'LOAD_ATTR_METHOD_NO_DICT': 182,
+    'LOAD_ATTR_METHOD_WITH_VALUES': 183,
+    'LOAD_ATTR_MODULE': 184,
+    'LOAD_ATTR_NONDESCRIPTOR_NO_DICT': 185,
+    'LOAD_ATTR_NONDESCRIPTOR_WITH_VALUES': 186,
+    'LOAD_ATTR_PROPERTY': 187,
+    'LOAD_ATTR_SLOT': 188,
+    'LOAD_ATTR_WITH_HINT': 189,
+    'LOAD_CONST_IMMORTAL': 190,
+    'LOAD_CONST_MORTAL': 191,
+    'LOAD_GLOBAL_BUILTIN': 192,
+    'LOAD_GLOBAL_MODULE': 193,
+    'LOAD_SUPER_ATTR_ATTR': 194,
+    'LOAD_SUPER_ATTR_METHOD': 195,
+    'RESUME_CHECK': 196,
+    'SEND_GEN': 197,
+    'STORE_ATTR_INSTANCE_VALUE': 198,
+    'STORE_ATTR_SLOT': 199,
+    'STORE_ATTR_WITH_HINT': 200,
+    'STORE_SUBSCR_DICT': 201,
+    'STORE_SUBSCR_LIST_INT': 202,
+    'TO_BOOL_ALWAYS_TRUE': 203,
+    'TO_BOOL_BOOL': 204,
+    'TO_BOOL_INT': 205,
+    'TO_BOOL_LIST': 206,
+    'TO_BOOL_NONE': 207,
+    'TO_BOOL_STR': 208,
+    'UNPACK_SEQUENCE_LIST': 209,
+    'UNPACK_SEQUENCE_TUPLE': 210,
+    'UNPACK_SEQUENCE_TWO_TUPLE': 211,
+}
+
+opmap = {
+    'CACHE': 0,
+    'RESERVED': 17,
+    'RESUME': 128,
+    'INSTRUMENTED_LINE': 254,
+    'ENTER_EXECUTOR': 255,
+    'BINARY_SLICE': 1,
+    'BUILD_TEMPLATE': 2,
+    'CALL_FUNCTION_EX': 4,
+    'CHECK_EG_MATCH': 5,
+    'CHECK_EXC_MATCH': 6,
+    'CLEANUP_THROW': 7,
+    'DELETE_SUBSCR': 8,
+    'END_FOR': 9,
+    'END_SEND': 10,
+    'EXIT_INIT_CHECK': 11,
+    'FORMAT_SIMPLE': 12,
+    'FORMAT_WITH_SPEC': 13,
+    'GET_AITER': 14,
+    'GET_ANEXT': 15,
+    'GET_ITER': 16,
+    'GET_LEN': 18,
+    'GET_YIELD_FROM_ITER': 19,
+    'INTERPRETER_EXIT': 20,
+    'LOAD_BUILD_CLASS': 21,
+    'LOAD_LOCALS': 22,
+    'MAKE_FUNCTION': 23,
+    'MATCH_KEYS': 24,
+    'MATCH_MAPPING': 25,
+    'MATCH_SEQUENCE': 26,
+    'NOP': 27,
+    'NOT_TAKEN': 28,
+    'POP_EXCEPT': 29,
+    'POP_ITER': 30,
+    'POP_TOP': 31,
+    'PUSH_EXC_INFO': 32,
+    'PUSH_NULL': 33,
+    'RETURN_GENERATOR': 34,
+    'RETURN_VALUE': 35,
+    'SETUP_ANNOTATIONS': 36,
+    'STORE_SLICE': 37,
+    'STORE_SUBSCR': 38,
+    'TO_BOOL': 39,
+    'UNARY_INVERT': 40,
+    'UNARY_NEGATIVE': 41,
+    'UNARY_NOT': 42,
+    'WITH_EXCEPT_START': 43,
+    'BINARY_OP': 44,
+    'BUILD_INTERPOLATION': 45,
+    'BUILD_LIST': 46,
+    'BUILD_MAP': 47,
+    'BUILD_SET': 48,
+    'BUILD_SLICE': 49,
+    'BUILD_STRING': 50,
+    'BUILD_TUPLE': 51,
+    'CALL': 52,
+    'CALL_INTRINSIC_1': 53,
+    'CALL_INTRINSIC_2': 54,
+    'CALL_KW': 55,
+    'COMPARE_OP': 56,
+    'CONTAINS_OP': 57,
+    'CONVERT_VALUE': 58,
+    'COPY': 59,
+    'COPY_FREE_VARS': 60,
+    'DELETE_ATTR': 61,
+    'DELETE_DEREF': 62,
+    'DELETE_FAST': 63,
+    'DELETE_GLOBAL': 64,
+    'DELETE_NAME': 65,
+    'DICT_MERGE': 66,
+    'DICT_UPDATE': 67,
+    'END_ASYNC_FOR': 68,
+    'EXTENDED_ARG': 69,
+    'FOR_ITER': 70,
+    'GET_AWAITABLE': 71,
+    'IMPORT_FROM': 72,
+    'IMPORT_NAME': 73,
+    'IS_OP': 74,
+    'JUMP_BACKWARD': 75,
+    'JUMP_BACKWARD_NO_INTERRUPT': 76,
+    'JUMP_FORWARD': 77,
+    'LIST_APPEND': 78,
+    'LIST_EXTEND': 79,
+    'LOAD_ATTR': 80,
+    'LOAD_COMMON_CONSTANT': 81,
+    'LOAD_CONST': 82,
+    'LOAD_DEREF': 83,
+    'LOAD_FAST': 84,
+    'LOAD_FAST_AND_CLEAR': 85,
+    'LOAD_FAST_BORROW': 86,
+    'LOAD_FAST_BORROW_LOAD_FAST_BORROW': 87,
+    'LOAD_FAST_CHECK': 88,
+    'LOAD_FAST_LOAD_FAST': 89,
+    'LOAD_FROM_DICT_OR_DEREF': 90,
+    'LOAD_FROM_DICT_OR_GLOBALS': 91,
+    'LOAD_GLOBAL': 92,
+    'LOAD_NAME': 93,
+    'LOAD_SMALL_INT': 94,
+    'LOAD_SPECIAL': 95,
+    'LOAD_SUPER_ATTR': 96,
+    'MAKE_CELL': 97,
+    'MAP_ADD': 98,
+    'MATCH_CLASS': 99,
+    'POP_JUMP_IF_FALSE': 100,
+    'POP_JUMP_IF_NONE': 101,
+    'POP_JUMP_IF_NOT_NONE': 102,
+    'POP_JUMP_IF_TRUE': 103,
+    'RAISE_VARARGS': 104,
+    'RERAISE': 105,
+    'SEND': 106,
+    'SET_ADD': 107,
+    'SET_FUNCTION_ATTRIBUTE': 108,
+    'SET_UPDATE': 109,
+    'STORE_ATTR': 110,
+    'STORE_DEREF': 111,
+    'STORE_FAST': 112,
+    'STORE_FAST_LOAD_FAST': 113,
+    'STORE_FAST_STORE_FAST': 114,
+    'STORE_GLOBAL': 115,
+    'STORE_NAME': 116,
+    'SWAP': 117,
+    'UNPACK_EX': 118,
+    'UNPACK_SEQUENCE': 119,
+    'YIELD_VALUE': 120,
+    'INSTRUMENTED_END_FOR': 234,
+    'INSTRUMENTED_POP_ITER': 235,
+    'INSTRUMENTED_END_SEND': 236,
+    'INSTRUMENTED_FOR_ITER': 237,
+    'INSTRUMENTED_INSTRUCTION': 238,
+    'INSTRUMENTED_JUMP_FORWARD': 239,
+    'INSTRUMENTED_NOT_TAKEN': 240,
+    'INSTRUMENTED_POP_JUMP_IF_TRUE': 241,
+    'INSTRUMENTED_POP_JUMP_IF_FALSE': 242,
+    'INSTRUMENTED_POP_JUMP_IF_NONE': 243,
+    'INSTRUMENTED_POP_JUMP_IF_NOT_NONE': 244,
+    'INSTRUMENTED_RESUME': 245,
+    'INSTRUMENTED_RETURN_VALUE': 246,
+    'INSTRUMENTED_YIELD_VALUE': 247,
+    'INSTRUMENTED_END_ASYNC_FOR': 248,
+    'INSTRUMENTED_LOAD_SUPER_ATTR': 249,
+    'INSTRUMENTED_CALL': 250,
+    'INSTRUMENTED_CALL_KW': 251,
+    'INSTRUMENTED_CALL_FUNCTION_EX': 252,
+    'INSTRUMENTED_JUMP_BACKWARD': 253,
+    'ANNOTATIONS_PLACEHOLDER': 256,
+    'JUMP': 257,
+    'JUMP_IF_FALSE': 258,
+    'JUMP_IF_TRUE': 259,
+    'JUMP_NO_INTERRUPT': 260,
+    'LOAD_CLOSURE': 261,
+    'POP_BLOCK': 262,
+    'SETUP_CLEANUP': 263,
+    'SETUP_FINALLY': 264,
+    'SETUP_WITH': 265,
+    'STORE_FAST_MAYBE_NULL': 266,
+}
+
+HAVE_ARGUMENT = 43
+MIN_INSTRUMENTED_OPCODE = 234

Lib/_osx_support.py

@@ -507,6 +507,11 @@ def get_platform_osx(_config_vars, osname, release, machine):
     # MACOSX_DEPLOYMENT_TARGET.
 
     macver = _config_vars.get('MACOSX_DEPLOYMENT_TARGET', '')
+    if macver and '.' not in macver:
+        # Ensure that the version includes at least a major
+        # and minor version, even if MACOSX_DEPLOYMENT_TARGET
+        # is set to a single-label version like "14".
+        macver += '.0'
     macrelease = _get_system_version() or macver
     macver = macver or macrelease
 

Lib/_py_warnings.py

@@ -0,0 +1,869 @@
+"""Python part of the warnings subsystem."""
+
+import sys
+import _contextvars
+import _thread
+
+
+__all__ = ["warn", "warn_explicit", "showwarning",
+           "formatwarning", "filterwarnings", "simplefilter",
+           "resetwarnings", "catch_warnings", "deprecated"]
+
+
+# Normally '_wm' is sys.modules['warnings'] but for unit tests it can be
+# a different module.  User code is allowed to reassign global attributes
+# of the 'warnings' module, commonly 'filters' or 'showwarning'. So we
+# need to lookup these global attributes dynamically on the '_wm' object,
+# rather than binding them earlier.  The code in this module consistently uses
+# '_wm.<something>' rather than using the globals of this module.  If the
+# '_warnings' C extension is in use, some globals are replaced by functions
+# and variables defined in that extension.
+_wm = None
+
+
+def _set_module(module):
+    global _wm
+    _wm = module
+
+
+# filters contains a sequence of filter 5-tuples
+# The components of the 5-tuple are:
+# - an action: error, ignore, always, all, default, module, or once
+# - a compiled regex that must match the warning message
+# - a class representing the warning category
+# - a compiled regex that must match the module that is being warned
+# - a line number for the line being warning, or 0 to mean any line
+# If either if the compiled regexs are None, match anything.
+filters = []
+
+
+defaultaction = "default"
+onceregistry = {}
+_lock = _thread.RLock()
+_filters_version = 1
+
+
+# If true, catch_warnings() will use a context var to hold the modified
+# filters list.  Otherwise, catch_warnings() will operate on the 'filters'
+# global of the warnings module.
+_use_context = sys.flags.context_aware_warnings
+
+
+class _Context:
+    def __init__(self, filters):
+        self._filters = filters
+        self.log = None  # if set to a list, logging is enabled
+
+    def copy(self):
+        context = _Context(self._filters[:])
+        if self.log is not None:
+            context.log = self.log
+        return context
+
+    def _record_warning(self, msg):
+        self.log.append(msg)
+
+
+class _GlobalContext(_Context):
+    def __init__(self):
+        self.log = None
+
+    @property
+    def _filters(self):
+        # Since there is quite a lot of code that assigns to
+        # warnings.filters, this needs to return the current value of
+        # the module global.
+        try:
+            return _wm.filters
+        except AttributeError:
+            # 'filters' global was deleted.  Do we need to actually handle this case?
+            return []
+
+
+_global_context = _GlobalContext()
+
+
+_warnings_context = _contextvars.ContextVar('warnings_context')
+
+
+def _get_context():
+    if not _use_context:
+        return _global_context
+    try:
+        return _wm._warnings_context.get()
+    except LookupError:
+        return _global_context
+
+
+def _set_context(context):
+    assert _use_context
+    _wm._warnings_context.set(context)
+
+
+def _new_context():
+    assert _use_context
+    old_context = _wm._get_context()
+    new_context = old_context.copy()
+    _wm._set_context(new_context)
+    return old_context, new_context
+
+
+def _get_filters():
+    """Return the current list of filters.  This is a non-public API used by
+    module functions and by the unit tests."""
+    return _wm._get_context()._filters
+
+
+def _filters_mutated_lock_held():
+    _wm._filters_version += 1
+
+
+def showwarning(message, category, filename, lineno, file=None, line=None):
+    """Hook to write a warning to a file; replace if you like."""
+    msg = _wm.WarningMessage(message, category, filename, lineno, file, line)
+    _wm._showwarnmsg_impl(msg)
+
+
+def formatwarning(message, category, filename, lineno, line=None):
+    """Function to format a warning the standard way."""
+    msg = _wm.WarningMessage(message, category, filename, lineno, None, line)
+    return _wm._formatwarnmsg_impl(msg)
+
+
+def _showwarnmsg_impl(msg):
+    context = _wm._get_context()
+    if context.log is not None:
+        context._record_warning(msg)
+        return
+    file = msg.file
+    if file is None:
+        file = sys.stderr
+        if file is None:
+            # sys.stderr is None when run with pythonw.exe:
+            # warnings get lost
+            return
+    text = _wm._formatwarnmsg(msg)
+    try:
+        file.write(text)
+    except OSError:
+        # the file (probably stderr) is invalid - this warning gets lost.
+        pass
+
+
+def _formatwarnmsg_impl(msg):
+    category = msg.category.__name__
+    s =  f"{msg.filename}:{msg.lineno}: {category}: {msg.message}\n"
+
+    if msg.line is None:
+        try:
+            import linecache
+            line = linecache.getline(msg.filename, msg.lineno)
+        except Exception:
+            # When a warning is logged during Python shutdown, linecache
+            # and the import machinery don't work anymore
+            line = None
+            linecache = None
+    else:
+        line = msg.line
+    if line:
+        line = line.strip()
+        s += "  %s\n" % line
+
+    if msg.source is not None:
+        try:
+            import tracemalloc
+        # Logging a warning should not raise a new exception:
+        # catch Exception, not only ImportError and RecursionError.
+        except Exception:
+            # don't suggest to enable tracemalloc if it's not available
+            suggest_tracemalloc = False
+            tb = None
+        else:
+            try:
+                suggest_tracemalloc = not tracemalloc.is_tracing()
+                tb = tracemalloc.get_object_traceback(msg.source)
+            except Exception:
+                # When a warning is logged during Python shutdown, tracemalloc
+                # and the import machinery don't work anymore
+                suggest_tracemalloc = False
+                tb = None
+
+        if tb is not None:
+            s += 'Object allocated at (most recent call last):\n'
+            for frame in tb:
+                s += ('  File "%s", lineno %s\n'
+                      % (frame.filename, frame.lineno))
+
+                try:
+                    if linecache is not None:
+                        line = linecache.getline(frame.filename, frame.lineno)
+                    else:
+                        line = None
+                except Exception:
+                    line = None
+                if line:
+                    line = line.strip()
+                    s += '    %s\n' % line
+        elif suggest_tracemalloc:
+            s += (f'{category}: Enable tracemalloc to get the object '
+                  f'allocation traceback\n')
+    return s
+
+
+# Keep a reference to check if the function was replaced
+_showwarning_orig = showwarning
+
+
+def _showwarnmsg(msg):
+    """Hook to write a warning to a file; replace if you like."""
+    try:
+        sw = _wm.showwarning
+    except AttributeError:
+        pass
+    else:
+        if sw is not _showwarning_orig:
+            # warnings.showwarning() was replaced
+            if not callable(sw):
+                raise TypeError("warnings.showwarning() must be set to a "
+                                "function or method")
+
+            sw(msg.message, msg.category, msg.filename, msg.lineno,
+               msg.file, msg.line)
+            return
+    _wm._showwarnmsg_impl(msg)
+
+
+# Keep a reference to check if the function was replaced
+_formatwarning_orig = formatwarning
+
+
+def _formatwarnmsg(msg):
+    """Function to format a warning the standard way."""
+    try:
+        fw = _wm.formatwarning
+    except AttributeError:
+        pass
+    else:
+        if fw is not _formatwarning_orig:
+            # warnings.formatwarning() was replaced
+            return fw(msg.message, msg.category,
+                      msg.filename, msg.lineno, msg.line)
+    return _wm._formatwarnmsg_impl(msg)
+
+
+def filterwarnings(action, message="", category=Warning, module="", lineno=0,
+                   append=False):
+    """Insert an entry into the list of warnings filters (at the front).
+
+    'action' -- one of "error", "ignore", "always", "all", "default", "module",
+                or "once"
+    'message' -- a regex that the warning message must match
+    'category' -- a class that the warning must be a subclass of
+    'module' -- a regex that the module name must match
+    'lineno' -- an integer line number, 0 matches all warnings
+    'append' -- if true, append to the list of filters
+    """
+    if action not in {"error", "ignore", "always", "all", "default", "module", "once"}:
+        raise ValueError(f"invalid action: {action!r}")
+    if not isinstance(message, str):
+        raise TypeError("message must be a string")
+    if not isinstance(category, type) or not issubclass(category, Warning):
+        raise TypeError("category must be a Warning subclass")
+    if not isinstance(module, str):
+        raise TypeError("module must be a string")
+    if not isinstance(lineno, int):
+        raise TypeError("lineno must be an int")
+    if lineno < 0:
+        raise ValueError("lineno must be an int >= 0")
+
+    if message or module:
+        import re
+
+    if message:
+        message = re.compile(message, re.I)
+    else:
+        message = None
+    if module:
+        module = re.compile(module)
+    else:
+        module = None
+
+    _wm._add_filter(action, message, category, module, lineno, append=append)
+
+
+def simplefilter(action, category=Warning, lineno=0, append=False):
+    """Insert a simple entry into the list of warnings filters (at the front).
+
+    A simple filter matches all modules and messages.
+    'action' -- one of "error", "ignore", "always", "all", "default", "module",
+                or "once"
+    'category' -- a class that the warning must be a subclass of
+    'lineno' -- an integer line number, 0 matches all warnings
+    'append' -- if true, append to the list of filters
+    """
+    if action not in {"error", "ignore", "always", "all", "default", "module", "once"}:
+        raise ValueError(f"invalid action: {action!r}")
+    if not isinstance(lineno, int):
+        raise TypeError("lineno must be an int")
+    if lineno < 0:
+        raise ValueError("lineno must be an int >= 0")
+    _wm._add_filter(action, None, category, None, lineno, append=append)
+
+
+def _filters_mutated():
+    # Even though this function is not part of the public API, it's used by
+    # a fair amount of user code.
+    with _wm._lock:
+        _wm._filters_mutated_lock_held()
+
+
+def _add_filter(*item, append):
+    with _wm._lock:
+        filters = _wm._get_filters()
+        if not append:
+            # Remove possible duplicate filters, so new one will be placed
+            # in correct place. If append=True and duplicate exists, do nothing.
+            try:
+                filters.remove(item)
+            except ValueError:
+                pass
+            filters.insert(0, item)
+        else:
+            if item not in filters:
+                filters.append(item)
+        _wm._filters_mutated_lock_held()
+
+
+def resetwarnings():
+    """Clear the list of warning filters, so that no filters are active."""
+    with _wm._lock:
+        del _wm._get_filters()[:]
+        _wm._filters_mutated_lock_held()
+
+
+class _OptionError(Exception):
+    """Exception used by option processing helpers."""
+    pass
+
+
+# Helper to process -W options passed via sys.warnoptions
+def _processoptions(args):
+    for arg in args:
+        try:
+            _wm._setoption(arg)
+        except _wm._OptionError as msg:
+            print("Invalid -W option ignored:", msg, file=sys.stderr)
+
+
+# Helper for _processoptions()
+def _setoption(arg):
+    parts = arg.split(':')
+    if len(parts) > 5:
+        raise _wm._OptionError("too many fields (max 5): %r" % (arg,))
+    while len(parts) < 5:
+        parts.append('')
+    action, message, category, module, lineno = [s.strip()
+                                                 for s in parts]
+    action = _wm._getaction(action)
+    category = _wm._getcategory(category)
+    if message or module:
+        import re
+    if message:
+        message = re.escape(message)
+    if module:
+        module = re.escape(module) + r'\z'
+    if lineno:
+        try:
+            lineno = int(lineno)
+            if lineno < 0:
+                raise ValueError
+        except (ValueError, OverflowError):
+            raise _wm._OptionError("invalid lineno %r" % (lineno,)) from None
+    else:
+        lineno = 0
+    _wm.filterwarnings(action, message, category, module, lineno)
+
+
+# Helper for _setoption()
+def _getaction(action):
+    if not action:
+        return "default"
+    for a in ('default', 'always', 'all', 'ignore', 'module', 'once', 'error'):
+        if a.startswith(action):
+            return a
+    raise _wm._OptionError("invalid action: %r" % (action,))
+
+
+# Helper for _setoption()
+def _getcategory(category):
+    if not category:
+        return Warning
+    if '.' not in category:
+        import builtins as m
+        klass = category
+    else:
+        module, _, klass = category.rpartition('.')
+        try:
+            m = __import__(module, None, None, [klass])
+        except ImportError:
+            raise _wm._OptionError("invalid module name: %r" % (module,)) from None
+    try:
+        cat = getattr(m, klass)
+    except AttributeError:
+        raise _wm._OptionError("unknown warning category: %r" % (category,)) from None
+    if not issubclass(cat, Warning):
+        raise _wm._OptionError("invalid warning category: %r" % (category,))
+    return cat
+
+
+def _is_internal_filename(filename):
+    return 'importlib' in filename and '_bootstrap' in filename
+
+
+def _is_filename_to_skip(filename, skip_file_prefixes):
+    return any(filename.startswith(prefix) for prefix in skip_file_prefixes)
+
+
+def _is_internal_frame(frame):
+    """Signal whether the frame is an internal CPython implementation detail."""
+    return _is_internal_filename(frame.f_code.co_filename)
+
+
+def _next_external_frame(frame, skip_file_prefixes):
+    """Find the next frame that doesn't involve Python or user internals."""
+    frame = frame.f_back
+    while frame is not None and (
+            _is_internal_filename(filename := frame.f_code.co_filename) or
+            _is_filename_to_skip(filename, skip_file_prefixes)):
+        frame = frame.f_back
+    return frame
+
+
+# Code typically replaced by _warnings
+def warn(message, category=None, stacklevel=1, source=None,
+         *, skip_file_prefixes=()):
+    """Issue a warning, or maybe ignore it or raise an exception."""
+    # Check if message is already a Warning object
+    if isinstance(message, Warning):
+        category = message.__class__
+    # Check category argument
+    if category is None:
+        category = UserWarning
+    if not (isinstance(category, type) and issubclass(category, Warning)):
+        raise TypeError("category must be a Warning subclass, "
+                        "not '{:s}'".format(type(category).__name__))
+    if not isinstance(skip_file_prefixes, tuple):
+        # The C version demands a tuple for implementation performance.
+        raise TypeError('skip_file_prefixes must be a tuple of strs.')
+    if skip_file_prefixes:
+        stacklevel = max(2, stacklevel)
+    # Get context information
+    try:
+        if stacklevel <= 1 or _is_internal_frame(sys._getframe(1)):
+            # If frame is too small to care or if the warning originated in
+            # internal code, then do not try to hide any frames.
+            frame = sys._getframe(stacklevel)
+        else:
+            frame = sys._getframe(1)
+            # Look for one frame less since the above line starts us off.
+            for x in range(stacklevel-1):
+                frame = _next_external_frame(frame, skip_file_prefixes)
+                if frame is None:
+                    raise ValueError
+    except ValueError:
+        globals = sys.__dict__
+        filename = "<sys>"
+        lineno = 0
+    else:
+        globals = frame.f_globals
+        filename = frame.f_code.co_filename
+        lineno = frame.f_lineno
+    if '__name__' in globals:
+        module = globals['__name__']
+    else:
+        module = "<string>"
+    registry = globals.setdefault("__warningregistry__", {})
+    _wm.warn_explicit(
+        message,
+        category,
+        filename,
+        lineno,
+        module,
+        registry,
+        globals,
+        source=source,
+    )
+
+
+def warn_explicit(message, category, filename, lineno,
+                  module=None, registry=None, module_globals=None,
+                  source=None):
+    lineno = int(lineno)
+    if module is None:
+        module = filename or "<unknown>"
+        if module[-3:].lower() == ".py":
+            module = module[:-3] # XXX What about leading pathname?
+    if isinstance(message, Warning):
+        text = str(message)
+        category = message.__class__
+    else:
+        text = message
+        message = category(message)
+    key = (text, category, lineno)
+    with _wm._lock:
+        if registry is None:
+            registry = {}
+        if registry.get('version', 0) != _wm._filters_version:
+            registry.clear()
+            registry['version'] = _wm._filters_version
+        # Quick test for common case
+        if registry.get(key):
+            return
+        # Search the filters
+        for item in _wm._get_filters():
+            action, msg, cat, mod, ln = item
+            if ((msg is None or msg.match(text)) and
+                issubclass(category, cat) and
+                (mod is None or mod.match(module)) and
+                (ln == 0 or lineno == ln)):
+                break
+        else:
+            action = _wm.defaultaction
+        # Early exit actions
+        if action == "ignore":
+            return
+
+        if action == "error":
+            raise message
+        # Other actions
+        if action == "once":
+            registry[key] = 1
+            oncekey = (text, category)
+            if _wm.onceregistry.get(oncekey):
+                return
+            _wm.onceregistry[oncekey] = 1
+        elif action in {"always", "all"}:
+            pass
+        elif action == "module":
+            registry[key] = 1
+            altkey = (text, category, 0)
+            if registry.get(altkey):
+                return
+            registry[altkey] = 1
+        elif action == "default":
+            registry[key] = 1
+        else:
+            # Unrecognized actions are errors
+            raise RuntimeError(
+                  "Unrecognized action (%r) in warnings.filters:\n %s" %
+                  (action, item))
+
+    # Prime the linecache for formatting, in case the
+    # "file" is actually in a zipfile or something.
+    import linecache
+    linecache.getlines(filename, module_globals)
+
+    # Print message and context
+    msg = _wm.WarningMessage(message, category, filename, lineno, source=source)
+    _wm._showwarnmsg(msg)
+
+
+class WarningMessage(object):
+
+    _WARNING_DETAILS = ("message", "category", "filename", "lineno", "file",
+                        "line", "source")
+
+    def __init__(self, message, category, filename, lineno, file=None,
+                 line=None, source=None):
+        self.message = message
+        self.category = category
+        self.filename = filename
+        self.lineno = lineno
+        self.file = file
+        self.line = line
+        self.source = source
+        self._category_name = category.__name__ if category else None
+
+    def __str__(self):
+        return ("{message : %r, category : %r, filename : %r, lineno : %s, "
+                    "line : %r}" % (self.message, self._category_name,
+                                    self.filename, self.lineno, self.line))
+
+    def __repr__(self):
+        return f'<{type(self).__qualname__} {self}>'
+
+
+class catch_warnings(object):
+
+    """A context manager that copies and restores the warnings filter upon
+    exiting the context.
+
+    The 'record' argument specifies whether warnings should be captured by a
+    custom implementation of warnings.showwarning() and be appended to a list
+    returned by the context manager. Otherwise None is returned by the context
+    manager. The objects appended to the list are arguments whose attributes
+    mirror the arguments to showwarning().
+
+    The 'module' argument is to specify an alternative module to the module
+    named 'warnings' and imported under that name. This argument is only useful
+    when testing the warnings module itself.
+
+    If the 'action' argument is not None, the remaining arguments are passed
+    to warnings.simplefilter() as if it were called immediately on entering the
+    context.
+    """
+
+    def __init__(self, *, record=False, module=None,
+                 action=None, category=Warning, lineno=0, append=False):
+        """Specify whether to record warnings and if an alternative module
+        should be used other than sys.modules['warnings'].
+
+        """
+        self._record = record
+        self._module = sys.modules['warnings'] if module is None else module
+        self._entered = False
+        if action is None:
+            self._filter = None
+        else:
+            self._filter = (action, category, lineno, append)
+
+    def __repr__(self):
+        args = []
+        if self._record:
+            args.append("record=True")
+        if self._module is not sys.modules['warnings']:
+            args.append("module=%r" % self._module)
+        name = type(self).__name__
+        return "%s(%s)" % (name, ", ".join(args))
+
+    def __enter__(self):
+        if self._entered:
+            raise RuntimeError("Cannot enter %r twice" % self)
+        self._entered = True
+        with _wm._lock:
+            if _use_context:
+                self._saved_context, context = self._module._new_context()
+            else:
+                context = None
+                self._filters = self._module.filters
+                self._module.filters = self._filters[:]
+                self._showwarning = self._module.showwarning
+                self._showwarnmsg_impl = self._module._showwarnmsg_impl
+            self._module._filters_mutated_lock_held()
+            if self._record:
+                if _use_context:
+                    context.log = log = []
+                else:
+                    log = []
+                    self._module._showwarnmsg_impl = log.append
+                    # Reset showwarning() to the default implementation to make sure
+                    # that _showwarnmsg() calls _showwarnmsg_impl()
+                    self._module.showwarning = self._module._showwarning_orig
+            else:
+                log = None
+        if self._filter is not None:
+            self._module.simplefilter(*self._filter)
+        return log
+
+    def __exit__(self, *exc_info):
+        if not self._entered:
+            raise RuntimeError("Cannot exit %r without entering first" % self)
+        with _wm._lock:
+            if _use_context:
+                self._module._warnings_context.set(self._saved_context)
+            else:
+                self._module.filters = self._filters
+                self._module.showwarning = self._showwarning
+                self._module._showwarnmsg_impl = self._showwarnmsg_impl
+            self._module._filters_mutated_lock_held()
+
+
+class deprecated:
+    """Indicate that a class, function or overload is deprecated.
+
+    When this decorator is applied to an object, the type checker
+    will generate a diagnostic on usage of the deprecated object.
+
+    Usage:
+
+        @deprecated("Use B instead")
+        class A:
+            pass
+
+        @deprecated("Use g instead")
+        def f():
+            pass
+
+        @overload
+        @deprecated("int support is deprecated")
+        def g(x: int) -> int: ...
+        @overload
+        def g(x: str) -> int: ...
+
+    The warning specified by *category* will be emitted at runtime
+    on use of deprecated objects. For functions, that happens on calls;
+    for classes, on instantiation and on creation of subclasses.
+    If the *category* is ``None``, no warning is emitted at runtime.
+    The *stacklevel* determines where the
+    warning is emitted. If it is ``1`` (the default), the warning
+    is emitted at the direct caller of the deprecated object; if it
+    is higher, it is emitted further up the stack.
+    Static type checker behavior is not affected by the *category*
+    and *stacklevel* arguments.
+
+    The deprecation message passed to the decorator is saved in the
+    ``__deprecated__`` attribute on the decorated object.
+    If applied to an overload, the decorator
+    must be after the ``@overload`` decorator for the attribute to
+    exist on the overload as returned by ``get_overloads()``.
+
+    See PEP 702 for details.
+
+    """
+    def __init__(
+        self,
+        message: str,
+        /,
+        *,
+        category: type[Warning] | None = DeprecationWarning,
+        stacklevel: int = 1,
+    ) -> None:
+        if not isinstance(message, str):
+            raise TypeError(
+                f"Expected an object of type str for 'message', not {type(message).__name__!r}"
+            )
+        self.message = message
+        self.category = category
+        self.stacklevel = stacklevel
+
+    def __call__(self, arg, /):
+        # Make sure the inner functions created below don't
+        # retain a reference to self.
+        msg = self.message
+        category = self.category
+        stacklevel = self.stacklevel
+        if category is None:
+            arg.__deprecated__ = msg
+            return arg
+        elif isinstance(arg, type):
+            import functools
+            from types import MethodType
+
+            original_new = arg.__new__
+
+            @functools.wraps(original_new)
+            def __new__(cls, /, *args, **kwargs):
+                if cls is arg:
+                    _wm.warn(msg, category=category, stacklevel=stacklevel + 1)
+                if original_new is not object.__new__:
+                    return original_new(cls, *args, **kwargs)
+                # Mirrors a similar check in object.__new__.
+                elif cls.__init__ is object.__init__ and (args or kwargs):
+                    raise TypeError(f"{cls.__name__}() takes no arguments")
+                else:
+                    return original_new(cls)
+
+            arg.__new__ = staticmethod(__new__)
+
+            if "__init_subclass__" in arg.__dict__:
+                # __init_subclass__ is directly present on the decorated class.
+                # Synthesize a wrapper that calls this method directly.
+                original_init_subclass = arg.__init_subclass__
+                # We need slightly different behavior if __init_subclass__
+                # is a bound method (likely if it was implemented in Python).
+                # Otherwise, it likely means it's a builtin such as
+                # object's implementation of __init_subclass__.
+                if isinstance(original_init_subclass, MethodType):
+                    original_init_subclass = original_init_subclass.__func__
+
+                @functools.wraps(original_init_subclass)
+                def __init_subclass__(*args, **kwargs):
+                    _wm.warn(msg, category=category, stacklevel=stacklevel + 1)
+                    return original_init_subclass(*args, **kwargs)
+            else:
+                def __init_subclass__(cls, *args, **kwargs):
+                    _wm.warn(msg, category=category, stacklevel=stacklevel + 1)
+                    return super(arg, cls).__init_subclass__(*args, **kwargs)
+
+            arg.__init_subclass__ = classmethod(__init_subclass__)
+
+            arg.__deprecated__ = __new__.__deprecated__ = msg
+            __init_subclass__.__deprecated__ = msg
+            return arg
+        elif callable(arg):
+            import functools
+            import inspect
+
+            @functools.wraps(arg)
+            def wrapper(*args, **kwargs):
+                _wm.warn(msg, category=category, stacklevel=stacklevel + 1)
+                return arg(*args, **kwargs)
+
+            if inspect.iscoroutinefunction(arg):
+                wrapper = inspect.markcoroutinefunction(wrapper)
+
+            arg.__deprecated__ = wrapper.__deprecated__ = msg
+            return wrapper
+        else:
+            raise TypeError(
+                "@deprecated decorator with non-None category must be applied to "
+                f"a class or callable, not {arg!r}"
+            )
+
+
+_DEPRECATED_MSG = "{name!r} is deprecated and slated for removal in Python {remove}"
+
+
+def _deprecated(name, message=_DEPRECATED_MSG, *, remove, _version=sys.version_info):
+    """Warn that *name* is deprecated or should be removed.
+
+    RuntimeError is raised if *remove* specifies a major/minor tuple older than
+    the current Python version or the same version but past the alpha.
+
+    The *message* argument is formatted with *name* and *remove* as a Python
+    version tuple (e.g. (3, 11)).
+
+    """
+    remove_formatted = f"{remove[0]}.{remove[1]}"
+    if (_version[:2] > remove) or (_version[:2] == remove and _version[3] != "alpha"):
+        msg = f"{name!r} was slated for removal after Python {remove_formatted} alpha"
+        raise RuntimeError(msg)
+    else:
+        msg = message.format(name=name, remove=remove_formatted)
+        _wm.warn(msg, DeprecationWarning, stacklevel=3)
+
+
+# Private utility function called by _PyErr_WarnUnawaitedCoroutine
+def _warn_unawaited_coroutine(coro):
+    msg_lines = [
+        f"coroutine '{coro.__qualname__}' was never awaited\n"
+    ]
+    if coro.cr_origin is not None:
+        import linecache, traceback
+        def extract():
+            for filename, lineno, funcname in reversed(coro.cr_origin):
+                line = linecache.getline(filename, lineno)
+                yield (filename, lineno, funcname, line)
+        msg_lines.append("Coroutine created at (most recent call last)\n")
+        msg_lines += traceback.format_list(list(extract()))
+    msg = "".join(msg_lines).rstrip("\n")
+    # Passing source= here means that if the user happens to have tracemalloc
+    # enabled and tracking where the coroutine was created, the warning will
+    # contain that traceback. This does mean that if they have *both*
+    # coroutine origin tracking *and* tracemalloc enabled, they'll get two
+    # partially-redundant tracebacks. If we wanted to be clever we could
+    # probably detect this case and avoid it, but for now we don't bother.
+    _wm.warn(
+        msg, category=RuntimeWarning, stacklevel=2, source=coro
+    )
+
+
+def _setup_defaults():
+    # Several warning categories are ignored by default in regular builds
+    if hasattr(sys, 'gettotalrefcount'):
+        return
+    _wm.filterwarnings("default", category=DeprecationWarning, module="__main__", append=1)
+    _wm.simplefilter("ignore", category=DeprecationWarning, append=1)
+    _wm.simplefilter("ignore", category=PendingDeprecationWarning, append=1)
+    _wm.simplefilter("ignore", category=ImportWarning, append=1)
+    _wm.simplefilter("ignore", category=ResourceWarning, append=1)

Lib/_pydatetime.py

@@ -1,12 +1,10 @@
-"""Concrete date/time and related types.
-
-See http://www.iana.org/time-zones/repository/tz-link.html for
-time zone and DST data sources.
-"""
+"""Pure Python implementation of the datetime module."""
 
 __all__ = ("date", "datetime", "time", "timedelta", "timezone", "tzinfo",
            "MINYEAR", "MAXYEAR", "UTC")
 
+__name__ = "datetime"
+
 
 import time as _time
 import math as _math
@@ -18,10 +16,10 @@ def _cmp(x, y):
 
 def _get_class_module(self):
     module_name = self.__class__.__module__
-    if module_name == '_pydatetime':
-        return 'datetime'
+    if module_name == 'datetime':
+        return 'datetime.'
     else:
-        return module_name
+        return ''
 
 MINYEAR = 1
 MAXYEAR = 9999
@@ -64,14 +62,14 @@ def _days_in_month(year, month):
 
 def _days_before_month(year, month):
     "year, month -> number of days in year preceding first day of month."
-    assert 1 <= month <= 12, 'month must be in 1..12'
+    assert 1 <= month <= 12, f"month must be in 1..12, not {month}"
     return _DAYS_BEFORE_MONTH[month] + (month > 2 and _is_leap(year))
 
 def _ymd2ord(year, month, day):
     "year, month, day -> ordinal, considering 01-Jan-0001 as day 1."
-    assert 1 <= month <= 12, 'month must be in 1..12'
+    assert 1 <= month <= 12, f"month must be in 1..12, not {month}"
     dim = _days_in_month(year, month)
-    assert 1 <= day <= dim, ('day must be in 1..%d' % dim)
+    assert 1 <= day <= dim, f"day must be in 1..{dim}, not {day}"
     return (_days_before_year(year) +
             _days_before_month(year, month) +
             day)
@@ -204,6 +202,17 @@ def _format_offset(off, sep=':'):
                 s += '.%06d' % ss.microseconds
     return s
 
+_normalize_century = None
+def _need_normalize_century():
+    global _normalize_century
+    if _normalize_century is None:
+        try:
+            _normalize_century = (
+                _time.strftime("%Y", (99, 1, 1, 0, 0, 0, 0, 1, 0)) != "0099")
+        except ValueError:
+            _normalize_century = True
+    return _normalize_century
+
 # Correctly substitute for %z and %Z escapes in strftime formats.
 def _wrap_strftime(object, format, timetuple):
     # Don't call utcoffset() or tzname() unless actually needed.
@@ -261,6 +270,20 @@ def _wrap_strftime(object, format, timetuple):
                                 # strftime is going to have at this: escape %
                                 Zreplace = s.replace('%', '%%')
                     newformat.append(Zreplace)
+                # Note that datetime(1000, 1, 1).strftime('%G') == '1000' so
+                # year 1000 for %G can go on the fast path.
+                elif ((ch in 'YG' or ch in 'FC') and
+                        object.year < 1000 and _need_normalize_century()):
+                    if ch == 'G':
+                        year = int(_time.strftime("%G", timetuple))
+                    else:
+                        year = object.year
+                    if ch == 'C':
+                        push('{:02}'.format(year // 100))
+                    else:
+                        push('{:04}'.format(year))
+                        if ch == 'F':
+                            push('-{:02}-{:02}'.format(*timetuple[1:3]))
                 else:
                     push('%')
                     push(ch)
@@ -399,9 +422,11 @@ def _parse_hh_mm_ss_ff(tstr):
 
     if pos < len_str:
         if tstr[pos] not in '.,':
-            raise ValueError("Invalid microsecond component")
+            raise ValueError("Invalid microsecond separator")
         else:
             pos += 1
+            if not all(map(_is_ascii_digit, tstr[pos:])):
+                raise ValueError("Non-digit values in fraction")
 
             len_remainder = len_str - pos
 
@@ -413,9 +438,6 @@ def _parse_hh_mm_ss_ff(tstr):
             time_comps[3] = int(tstr[pos:(pos+to_parse)])
             if to_parse < 6:
                 time_comps[3] *= _FRACTION_CORRECTION[to_parse-1]
-            if (len_remainder > to_parse
-                    and not all(map(_is_ascii_digit, tstr[(pos+to_parse):]))):
-                raise ValueError("Non-digit values in unparsed fraction")
 
     return time_comps
 
@@ -431,6 +453,17 @@ def _parse_isoformat_time(tstr):
 
     time_comps = _parse_hh_mm_ss_ff(timestr)
 
+    hour, minute, second, microsecond = time_comps
+    became_next_day = False
+    error_from_components = False
+    if (hour == 24):
+        if all(time_comp == 0 for time_comp in time_comps[1:]):
+            hour = 0
+            time_comps[0] = hour
+            became_next_day = True
+        else:
+            error_from_components = True
+
     tzi = None
     if tz_pos == len_str and tstr[-1] == 'Z':
         tzi = timezone.utc
@@ -446,7 +479,7 @@ def _parse_isoformat_time(tstr):
         # HH:MM:SS            len: 8
         # HH:MM:SS.f+         len: 10+
 
-        if len(tzstr) in (0, 1, 3):
+        if len(tzstr) in (0, 1, 3) or tstr[tz_pos-1] == 'Z':
             raise ValueError("Malformed time zone string")
 
         tz_comps = _parse_hh_mm_ss_ff(tzstr)
@@ -463,13 +496,13 @@ def _parse_isoformat_time(tstr):
 
     time_comps.append(tzi)
 
-    return time_comps
+    return time_comps, became_next_day, error_from_components
 
 # tuple[int, int, int] -> tuple[int, int, int] version of date.fromisocalendar
 def _isoweek_to_gregorian(year, week, day):
     # Year is bounded this way because 9999-12-31 is (9999, 52, 5)
     if not MINYEAR <= year <= MAXYEAR:
-        raise ValueError(f"Year is out of range: {year}")
+        raise ValueError(f"year must be in {MINYEAR}..{MAXYEAR}, not {year}")
 
     if not 0 < week < 53:
         out_of_range = True
@@ -502,7 +535,7 @@ def _isoweek_to_gregorian(year, week, day):
 def _check_tzname(name):
     if name is not None and not isinstance(name, str):
         raise TypeError("tzinfo.tzname() must return None or string, "
-                        "not '%s'" % type(name))
+                        f"not {type(name).__name__!r}")
 
 # name is the offset-producing method, "utcoffset" or "dst".
 # offset is what it returned.
@@ -515,24 +548,24 @@ def _check_utc_offset(name, offset):
     if offset is None:
         return
     if not isinstance(offset, timedelta):
-        raise TypeError("tzinfo.%s() must return None "
-                        "or timedelta, not '%s'" % (name, type(offset)))
+        raise TypeError(f"tzinfo.{name}() must return None "
+                        f"or timedelta, not {type(offset).__name__!r}")
     if not -timedelta(1) < offset < timedelta(1):
-        raise ValueError("%s()=%s, must be strictly between "
-                         "-timedelta(hours=24) and timedelta(hours=24)" %
-                         (name, offset))
+        raise ValueError("offset must be a timedelta "
+                         "strictly between -timedelta(hours=24) and "
+                         f"timedelta(hours=24), not {offset!r}")
 
 def _check_date_fields(year, month, day):
     year = _index(year)
     month = _index(month)
     day = _index(day)
     if not MINYEAR <= year <= MAXYEAR:
-        raise ValueError('year must be in %d..%d' % (MINYEAR, MAXYEAR), year)
+        raise ValueError(f"year must be in {MINYEAR}..{MAXYEAR}, not {year}")
     if not 1 <= month <= 12:
-        raise ValueError('month must be in 1..12', month)
+        raise ValueError(f"month must be in 1..12, not {month}")
     dim = _days_in_month(year, month)
     if not 1 <= day <= dim:
-        raise ValueError('day must be in 1..%d' % dim, day)
+        raise ValueError(f"day {day} must be in range 1..{dim} for month {month} in year {year}")
     return year, month, day
 
 def _check_time_fields(hour, minute, second, microsecond, fold):
@@ -541,24 +574,23 @@ def _check_time_fields(hour, minute, second, microsecond, fold):
     second = _index(second)
     microsecond = _index(microsecond)
     if not 0 <= hour <= 23:
-        raise ValueError('hour must be in 0..23', hour)
+        raise ValueError(f"hour must be in 0..23, not {hour}")
     if not 0 <= minute <= 59:
-        raise ValueError('minute must be in 0..59', minute)
+        raise ValueError(f"minute must be in 0..59, not {minute}")
     if not 0 <= second <= 59:
-        raise ValueError('second must be in 0..59', second)
+        raise ValueError(f"second must be in 0..59, not {second}")
     if not 0 <= microsecond <= 999999:
-        raise ValueError('microsecond must be in 0..999999', microsecond)
+        raise ValueError(f"microsecond must be in 0..999999, not {microsecond}")
     if fold not in (0, 1):
-        raise ValueError('fold must be either 0 or 1', fold)
+        raise ValueError(f"fold must be either 0 or 1, not {fold}")
     return hour, minute, second, microsecond, fold
 
 def _check_tzinfo_arg(tz):
     if tz is not None and not isinstance(tz, tzinfo):
-        raise TypeError("tzinfo argument must be None or of a tzinfo subclass")
-
-def _cmperror(x, y):
-    raise TypeError("can't compare '%s' to '%s'" % (
-                    type(x).__name__, type(y).__name__))
+        raise TypeError(
+            "tzinfo argument must be None or of a tzinfo subclass, "
+            f"not {type(tz).__name__!r}"
+        )
 
 def _divide_and_round(a, b):
     """divide a by b and round result to the nearest integer
@@ -612,7 +644,19 @@ class timedelta:
         # guide the C implementation; it's way more convoluted than speed-
         # ignoring auto-overflow-to-long idiomatic Python could be.
 
-        # XXX Check that all inputs are ints or floats.
+        for name, value in (
+            ("days", days),
+            ("seconds", seconds),
+            ("microseconds", microseconds),
+            ("milliseconds", milliseconds),
+            ("minutes", minutes),
+            ("hours", hours),
+            ("weeks", weeks)
+        ):
+            if not isinstance(value, (int, float)):
+                raise TypeError(
+                    f"unsupported type for timedelta {name} component: {type(value).__name__}"
+                )
 
         # Final values, all integer.
         # s and us fit in 32-bit signed ints; d isn't bounded.
@@ -713,9 +757,9 @@ class timedelta:
             args.append("microseconds=%d" % self._microseconds)
         if not args:
             args.append('0')
-        return "%s.%s(%s)" % (_get_class_module(self),
-                              self.__class__.__qualname__,
-                              ', '.join(args))
+        return "%s%s(%s)" % (_get_class_module(self),
+                             self.__class__.__qualname__,
+                             ', '.join(args))
 
     def __str__(self):
         mm, ss = divmod(self._seconds, 60)
@@ -912,6 +956,7 @@ class date:
     fromtimestamp()
     today()
     fromordinal()
+    strptime()
 
     Operators:
 
@@ -970,6 +1015,8 @@ class date:
     @classmethod
     def fromtimestamp(cls, t):
         "Construct a date from a POSIX timestamp (like time.time())."
+        if t is None:
+            raise TypeError("'NoneType' object cannot be interpreted as an integer")
         y, m, d, hh, mm, ss, weekday, jday, dst = _time.localtime(t)
         return cls(y, m, d)
 
@@ -992,8 +1039,12 @@ class date:
     @classmethod
     def fromisoformat(cls, date_string):
         """Construct a date from a string in ISO 8601 format."""
+
         if not isinstance(date_string, str):
-            raise TypeError('fromisoformat: argument must be str')
+            raise TypeError('Argument must be a str')
+
+        if not date_string.isascii():
+            raise ValueError('Argument must be an ASCII str')
 
         if len(date_string) not in (7, 8, 10):
             raise ValueError(f'Invalid isoformat string: {date_string!r}')
@@ -1010,6 +1061,12 @@ class date:
         This is the inverse of the date.isocalendar() function"""
         return cls(*_isoweek_to_gregorian(year, week, day))
 
+    @classmethod
+    def strptime(cls, date_string, format):
+        """Parse a date string according to the given format (like time.strptime())."""
+        import _strptime
+        return _strptime._strptime_datetime_date(cls, date_string, format)
+
     # Conversions to string
 
     def __repr__(self):
@@ -1019,11 +1076,11 @@ class date:
         >>> repr(d)
         'datetime.date(2010, 1, 1)'
         """
-        return "%s.%s(%d, %d, %d)" % (_get_class_module(self),
-                                      self.__class__.__qualname__,
-                                      self._year,
-                                      self._month,
-                                      self._day)
+        return "%s%s(%d, %d, %d)" % (_get_class_module(self),
+                                     self.__class__.__qualname__,
+                                     self._year,
+                                     self._month,
+                                     self._day)
     # XXX These shouldn't depend on time.localtime(), because that
     # clips the usable dates to [1970 .. 2038).  At least ctime() is
     # easily done without using strftime() -- that's better too because
@@ -1059,8 +1116,8 @@ class date:
         This is 'YYYY-MM-DD'.
 
         References:
-        - http://www.w3.org/TR/NOTE-datetime
-        - http://www.cl.cam.ac.uk/~mgk25/iso-time.html
+        - https://www.w3.org/TR/NOTE-datetime
+        - https://www.cl.cam.ac.uk/~mgk25/iso-time.html
         """
         return "%04d-%02d-%02d" % (self._year, self._month, self._day)
 
@@ -1108,35 +1165,38 @@ class date:
             day = self._day
         return type(self)(year, month, day)
 
+    __replace__ = replace
+
     # Comparisons of date objects with other.
 
     def __eq__(self, other):
-        if isinstance(other, date):
+        if isinstance(other, date) and not isinstance(other, datetime):
             return self._cmp(other) == 0
         return NotImplemented
 
     def __le__(self, other):
-        if isinstance(other, date):
+        if isinstance(other, date) and not isinstance(other, datetime):
             return self._cmp(other) <= 0
         return NotImplemented
 
     def __lt__(self, other):
-        if isinstance(other, date):
+        if isinstance(other, date) and not isinstance(other, datetime):
             return self._cmp(other) < 0
         return NotImplemented
 
     def __ge__(self, other):
-        if isinstance(other, date):
+        if isinstance(other, date) and not isinstance(other, datetime):
             return self._cmp(other) >= 0
         return NotImplemented
 
     def __gt__(self, other):
-        if isinstance(other, date):
+        if isinstance(other, date) and not isinstance(other, datetime):
             return self._cmp(other) > 0
         return NotImplemented
 
     def _cmp(self, other):
         assert isinstance(other, date)
+        assert not isinstance(other, datetime)
         y, m, d = self._year, self._month, self._day
         y2, m2, d2 = other._year, other._month, other._day
         return _cmp((y, m, d), (y2, m2, d2))
@@ -1191,7 +1251,7 @@ class date:
         The first week is 1; Monday is 1 ... Sunday is 7.
 
         ISO calendar algorithm taken from
-        http://www.phys.uu.nl/~vgent/calendar/isocalendar.htm
+        https://www.phys.uu.nl/~vgent/calendar/isocalendar.htm
         (used with permission)
         """
         year = self._year
@@ -1327,6 +1387,7 @@ class time:
     Constructors:
 
     __new__()
+    strptime()
 
     Operators:
 
@@ -1385,6 +1446,12 @@ class time:
         self._fold = fold
         return self
 
+    @classmethod
+    def strptime(cls, date_string, format):
+        """string, format -> new time parsed from a string (like time.strptime())."""
+        import _strptime
+        return _strptime._strptime_datetime_time(cls, date_string, format)
+
     # Read-only field accessors
     @property
     def hour(self):
@@ -1513,7 +1580,7 @@ class time:
             s = ", %d" % self._second
         else:
             s = ""
-        s= "%s.%s(%d, %d%s)" % (_get_class_module(self),
+        s = "%s%s(%d, %d%s)" % (_get_class_module(self),
                                 self.__class__.__qualname__,
                                 self._hour, self._minute, s)
         if self._tzinfo is not None:
@@ -1555,7 +1622,7 @@ class time:
         time_string = time_string.removeprefix('T')
 
         try:
-            return cls(*_parse_isoformat_time(time_string))
+            return cls(*_parse_isoformat_time(time_string)[0])
         except Exception:
             raise ValueError(f'Invalid isoformat string: {time_string!r}')
 
@@ -1633,6 +1700,8 @@ class time:
             fold = self._fold
         return type(self)(hour, minute, second, microsecond, tzinfo, fold=fold)
 
+    __replace__ = replace
+
     # Pickle support.
 
     def _getstate(self, protocol=3):
@@ -1680,7 +1749,7 @@ class datetime(date):
     The year, month and day arguments are required. tzinfo may be None, or an
     instance of a tzinfo subclass. The remaining arguments may be ints.
     """
-    __slots__ = date.__slots__ + time.__slots__
+    __slots__ = time.__slots__
 
     def __new__(cls, year, month=None, day=None, hour=0, minute=0, second=0,
                 microsecond=0, tzinfo=None, *, fold=0):
@@ -1867,10 +1936,27 @@ class datetime(date):
 
         if tstr:
             try:
-                time_components = _parse_isoformat_time(tstr)
+                time_components, became_next_day, error_from_components = _parse_isoformat_time(tstr)
             except ValueError:
                 raise ValueError(
                     f'Invalid isoformat string: {date_string!r}') from None
+            else:
+                if error_from_components:
+                    raise ValueError("minute, second, and microsecond must be 0 when hour is 24")
+
+                if became_next_day:
+                    year, month, day = date_components
+                    # Only wrap day/month when it was previously valid
+                    if month <= 12 and day <= (days_in_month := _days_in_month(year, month)):
+                        # Calculate midnight of the next day
+                        day += 1
+                        if day > days_in_month:
+                            day = 1
+                            month += 1
+                            if month > 12:
+                                month = 1
+                                year += 1
+                        date_components = [year, month, day]
         else:
             time_components = [0, 0, 0, 0, None]
 
@@ -1979,6 +2065,8 @@ class datetime(date):
         return type(self)(year, month, day, hour, minute, second,
                           microsecond, tzinfo, fold=fold)
 
+    __replace__ = replace
+
     def _local_timezone(self):
         if self.tzinfo is None:
             ts = self._mktime()
@@ -2040,7 +2128,7 @@ class datetime(date):
         By default, the fractional part is omitted if self.microsecond == 0.
 
         If self.tzinfo is not None, the UTC offset is also attached, giving
-        giving a full format of 'YYYY-MM-DD HH:MM:SS.mmmmmm+HH:MM'.
+        a full format of 'YYYY-MM-DD HH:MM:SS.mmmmmm+HH:MM'.
 
         Optional argument sep specifies the separator between date and
         time, default 'T'.
@@ -2068,9 +2156,9 @@ class datetime(date):
             del L[-1]
         if L[-1] == 0:
             del L[-1]
-        s = "%s.%s(%s)" % (_get_class_module(self),
-                           self.__class__.__qualname__,
-                           ", ".join(map(str, L)))
+        s = "%s%s(%s)" % (_get_class_module(self),
+                          self.__class__.__qualname__,
+                          ", ".join(map(str, L)))
         if self._tzinfo is not None:
             assert s[-1:] == ")"
             s = s[:-1] + ", tzinfo=%r" % self._tzinfo + ")"
@@ -2087,7 +2175,7 @@ class datetime(date):
     def strptime(cls, date_string, format):
         'string, format -> new datetime parsed from a string (like time.strptime()).'
         import _strptime
-        return _strptime._strptime_datetime(cls, date_string, format)
+        return _strptime._strptime_datetime_datetime(cls, date_string, format)
 
     def utcoffset(self):
         """Return the timezone offset as timedelta positive east of UTC (negative west of
@@ -2131,42 +2219,32 @@ class datetime(date):
     def __eq__(self, other):
         if isinstance(other, datetime):
             return self._cmp(other, allow_mixed=True) == 0
-        elif not isinstance(other, date):
-            return NotImplemented
         else:
-            return False
+            return NotImplemented
 
     def __le__(self, other):
         if isinstance(other, datetime):
             return self._cmp(other) <= 0
-        elif not isinstance(other, date):
-            return NotImplemented
         else:
-            _cmperror(self, other)
+            return NotImplemented
 
     def __lt__(self, other):
         if isinstance(other, datetime):
             return self._cmp(other) < 0
-        elif not isinstance(other, date):
-            return NotImplemented
         else:
-            _cmperror(self, other)
+            return NotImplemented
 
     def __ge__(self, other):
         if isinstance(other, datetime):
             return self._cmp(other) >= 0
-        elif not isinstance(other, date):
-            return NotImplemented
         else:
-            _cmperror(self, other)
+            return NotImplemented
 
     def __gt__(self, other):
         if isinstance(other, datetime):
             return self._cmp(other) > 0
-        elif not isinstance(other, date):
-            return NotImplemented
         else:
-            _cmperror(self, other)
+            return NotImplemented
 
     def _cmp(self, other, allow_mixed=False):
         assert isinstance(other, datetime)
@@ -2311,7 +2389,6 @@ datetime.resolution = timedelta(microseconds=1)
 
 def _isoweek1monday(year):
     # Helper to calculate the day number of the Monday starting week 1
-    # XXX This could be done more efficiently
     THURSDAY = 3
     firstday = _ymd2ord(year, 1, 1)
     firstweekday = (firstday + 6) % 7  # See weekday() above
@@ -2338,9 +2415,12 @@ class timezone(tzinfo):
         if not cls._minoffset <= offset <= cls._maxoffset:
             raise ValueError("offset must be a timedelta "
                              "strictly between -timedelta(hours=24) and "
-                             "timedelta(hours=24).")
+                             f"timedelta(hours=24), not {offset!r}")
         return cls._create(offset, name)
 
+    def __init_subclass__(cls):
+        raise TypeError("type 'datetime.timezone' is not an acceptable base type")
+
     @classmethod
     def _create(cls, offset, name=None):
         self = tzinfo.__new__(cls)
@@ -2375,12 +2455,12 @@ class timezone(tzinfo):
         if self is self.utc:
             return 'datetime.timezone.utc'
         if self._name is None:
-            return "%s.%s(%r)" % (_get_class_module(self),
-                                  self.__class__.__qualname__,
-                                  self._offset)
-        return "%s.%s(%r, %r)" % (_get_class_module(self),
-                                  self.__class__.__qualname__,
-                                  self._offset, self._name)
+            return "%s%s(%r)" % (_get_class_module(self),
+                                 self.__class__.__qualname__,
+                                 self._offset)
+        return "%s%s(%r, %r)" % (_get_class_module(self),
+                                 self.__class__.__qualname__,
+                                 self._offset, self._name)
 
     def __str__(self):
         return self.tzname(None)

Lib/_pydecimal.py

@@ -13,104 +13,7 @@
 # bug) and will be backported.  At this point the spec is stabilizing
 # and the updates are becoming fewer, smaller, and less significant.
 
-"""
-This is an implementation of decimal floating point arithmetic based on
-the General Decimal Arithmetic Specification:
-
-    http://speleotrove.com/decimal/decarith.html
-
-and IEEE standard 854-1987:
-
-    http://en.wikipedia.org/wiki/IEEE_854-1987
-
-Decimal floating point has finite precision with arbitrarily large bounds.
-
-The purpose of this module is to support arithmetic using familiar
-"schoolhouse" rules and to avoid some of the tricky representation
-issues associated with binary floating point.  The package is especially
-useful for financial applications or for contexts where users have
-expectations that are at odds with binary floating point (for instance,
-in binary floating point, 1.00 % 0.1 gives 0.09999999999999995 instead
-of 0.0; Decimal('1.00') % Decimal('0.1') returns the expected
-Decimal('0.00')).
-
-Here are some examples of using the decimal module:
-
->>> from decimal import *
->>> setcontext(ExtendedContext)
->>> Decimal(0)
-Decimal('0')
->>> Decimal('1')
-Decimal('1')
->>> Decimal('-.0123')
-Decimal('-0.0123')
->>> Decimal(123456)
-Decimal('123456')
->>> Decimal('123.45e12345678')
-Decimal('1.2345E+12345680')
->>> Decimal('1.33') + Decimal('1.27')
-Decimal('2.60')
->>> Decimal('12.34') + Decimal('3.87') - Decimal('18.41')
-Decimal('-2.20')
->>> dig = Decimal(1)
->>> print(dig / Decimal(3))
-0.333333333
->>> getcontext().prec = 18
->>> print(dig / Decimal(3))
-0.333333333333333333
->>> print(dig.sqrt())
-1
->>> print(Decimal(3).sqrt())
-1.73205080756887729
->>> print(Decimal(3) ** 123)
-4.85192780976896427E+58
->>> inf = Decimal(1) / Decimal(0)
->>> print(inf)
-Infinity
->>> neginf = Decimal(-1) / Decimal(0)
->>> print(neginf)
--Infinity
->>> print(neginf + inf)
-NaN
->>> print(neginf * inf)
--Infinity
->>> print(dig / 0)
-Infinity
->>> getcontext().traps[DivisionByZero] = 1
->>> print(dig / 0)
-Traceback (most recent call last):
-  ...
-  ...
-  ...
-decimal.DivisionByZero: x / 0
->>> c = Context()
->>> c.traps[InvalidOperation] = 0
->>> print(c.flags[InvalidOperation])
-0
->>> c.divide(Decimal(0), Decimal(0))
-Decimal('NaN')
->>> c.traps[InvalidOperation] = 1
->>> print(c.flags[InvalidOperation])
-1
->>> c.flags[InvalidOperation] = 0
->>> print(c.flags[InvalidOperation])
-0
->>> print(c.divide(Decimal(0), Decimal(0)))
-Traceback (most recent call last):
-  ...
-  ...
-  ...
-decimal.InvalidOperation: 0 / 0
->>> print(c.flags[InvalidOperation])
-1
->>> c.flags[InvalidOperation] = 0
->>> c.traps[InvalidOperation] = 0
->>> print(c.divide(Decimal(0), Decimal(0)))
-NaN
->>> print(c.flags[InvalidOperation])
-1
->>>
-"""
+"""Python decimal arithmetic module"""
 
 __all__ = [
     # Two major classes
@@ -135,13 +38,16 @@ __all__ = [
     'ROUND_FLOOR', 'ROUND_UP', 'ROUND_HALF_DOWN', 'ROUND_05UP',
 
     # Functions for manipulating contexts
-    'setcontext', 'getcontext', 'localcontext',
+    'setcontext', 'getcontext', 'localcontext', 'IEEEContext',
 
     # Limits for the C version for compatibility
-    'MAX_PREC',  'MAX_EMAX', 'MIN_EMIN', 'MIN_ETINY',
+    'MAX_PREC',  'MAX_EMAX', 'MIN_EMIN', 'MIN_ETINY', 'IEEE_CONTEXT_MAX_BITS',
 
-    # C version: compile time choice that enables the thread local context
-    'HAVE_THREADS'
+    # C version: compile time choice that enables the thread local context (deprecated, now always true)
+    'HAVE_THREADS',
+
+    # C version: compile time choice that enables the coroutine local context
+    'HAVE_CONTEXTVAR'
 ]
 
 __xname__ = __name__    # sys.modules lookup (--without-threads)
@@ -156,7 +62,7 @@ import sys
 
 try:
     from collections import namedtuple as _namedtuple
-    DecimalTuple = _namedtuple('DecimalTuple', 'sign digits exponent')
+    DecimalTuple = _namedtuple('DecimalTuple', 'sign digits exponent', module='decimal')
 except ImportError:
     DecimalTuple = lambda *args: args
 
@@ -172,14 +78,17 @@ ROUND_05UP = 'ROUND_05UP'
 
 # Compatibility with the C version
 HAVE_THREADS = True
+HAVE_CONTEXTVAR = True
 if sys.maxsize == 2**63-1:
     MAX_PREC = 999999999999999999
     MAX_EMAX = 999999999999999999
     MIN_EMIN = -999999999999999999
+    IEEE_CONTEXT_MAX_BITS = 512
 else:
     MAX_PREC = 425000000
     MAX_EMAX = 425000000
     MIN_EMIN = -425000000
+    IEEE_CONTEXT_MAX_BITS = 256
 
 MIN_ETINY = MIN_EMIN - (MAX_PREC-1)
 
@@ -190,7 +99,7 @@ class DecimalException(ArithmeticError):
 
     Used exceptions derive from this.
     If an exception derives from another exception besides this (such as
-    Underflow (Inexact, Rounded, Subnormal) that indicates that it is only
+    Underflow (Inexact, Rounded, Subnormal)) that indicates that it is only
     called if the others are present.  This isn't actually used for
     anything, though.
 
@@ -238,7 +147,7 @@ class InvalidOperation(DecimalException):
     x ** (+-)INF
     An operand is invalid
 
-    The result of the operation after these is a quiet positive NaN,
+    The result of the operation after this is a quiet positive NaN,
     except when the cause is a signaling NaN, in which case the result is
     also a quiet NaN, but with the original sign, and an optional
     diagnostic information.
@@ -431,82 +340,40 @@ _rounding_modes = (ROUND_DOWN, ROUND_HALF_UP, ROUND_HALF_EVEN, ROUND_CEILING,
 ##### Context Functions ##################################################
 
 # The getcontext() and setcontext() function manage access to a thread-local
-# current context.  Py2.4 offers direct support for thread locals.  If that
-# is not available, use threading.current_thread() which is slower but will
-# work for older Pythons.  If threads are not part of the build, create a
-# mock threading object with threading.local() returning the module namespace.
+# current context.
 
-try:
-    import threading
-except ImportError:
-    # Python was compiled without threads; create a mock object instead
-    class MockThreading(object):
-        def local(self, sys=sys):
-            return sys.modules[__xname__]
-    threading = MockThreading()
-    del MockThreading
+import contextvars
 
-try:
-    threading.local
+_current_context_var = contextvars.ContextVar('decimal_context')
 
-except AttributeError:
+_context_attributes = frozenset(
+    ['prec', 'Emin', 'Emax', 'capitals', 'clamp', 'rounding', 'flags', 'traps']
+)
 
-    # To fix reloading, force it to create a new context
-    # Old contexts have different exceptions in their dicts, making problems.
-    if hasattr(threading.current_thread(), '__decimal_context__'):
-        del threading.current_thread().__decimal_context__
+def getcontext():
+    """Returns this thread's context.
 
-    def setcontext(context):
-        """Set this thread's context to context."""
-        if context in (DefaultContext, BasicContext, ExtendedContext):
-            context = context.copy()
-            context.clear_flags()
-        threading.current_thread().__decimal_context__ = context
+    If this thread does not yet have a context, returns
+    a new context and sets this thread's context.
+    New contexts are copies of DefaultContext.
+    """
+    try:
+        return _current_context_var.get()
+    except LookupError:
+        context = Context()
+        _current_context_var.set(context)
+        return context
 
-    def getcontext():
-        """Returns this thread's context.
+def setcontext(context):
+    """Set this thread's context to context."""
+    if context in (DefaultContext, BasicContext, ExtendedContext):
+        context = context.copy()
+        context.clear_flags()
+    _current_context_var.set(context)
 
-        If this thread does not yet have a context, returns
-        a new context and sets this thread's context.
-        New contexts are copies of DefaultContext.
-        """
-        try:
-            return threading.current_thread().__decimal_context__
-        except AttributeError:
-            context = Context()
-            threading.current_thread().__decimal_context__ = context
-            return context
+del contextvars        # Don't contaminate the namespace
 
-else:
-
-    local = threading.local()
-    if hasattr(local, '__decimal_context__'):
-        del local.__decimal_context__
-
-    def getcontext(_local=local):
-        """Returns this thread's context.
-
-        If this thread does not yet have a context, returns
-        a new context and sets this thread's context.
-        New contexts are copies of DefaultContext.
-        """
-        try:
-            return _local.__decimal_context__
-        except AttributeError:
-            context = Context()
-            _local.__decimal_context__ = context
-            return context
-
-    def setcontext(context, _local=local):
-        """Set this thread's context to context."""
-        if context in (DefaultContext, BasicContext, ExtendedContext):
-            context = context.copy()
-            context.clear_flags()
-        _local.__decimal_context__ = context
-
-    del threading, local        # Don't contaminate the namespace
-
-def localcontext(ctx=None):
+def localcontext(ctx=None, **kwargs):
     """Return a context manager for a copy of the supplied context
 
     Uses a copy of the current context if no context is specified
@@ -542,8 +409,35 @@ def localcontext(ctx=None):
     >>> print(getcontext().prec)
     28
     """
-    if ctx is None: ctx = getcontext()
-    return _ContextManager(ctx)
+    if ctx is None:
+        ctx = getcontext()
+    ctx_manager = _ContextManager(ctx)
+    for key, value in kwargs.items():
+        if key not in _context_attributes:
+            raise TypeError(f"'{key}' is an invalid keyword argument for this function")
+        setattr(ctx_manager.new_context, key, value)
+    return ctx_manager
+
+
+def IEEEContext(bits, /):
+    """
+    Return a context object initialized to the proper values for one of the
+    IEEE interchange formats.  The argument must be a multiple of 32 and less
+    than IEEE_CONTEXT_MAX_BITS.
+    """
+    if bits <= 0 or bits > IEEE_CONTEXT_MAX_BITS or bits % 32:
+        raise ValueError("argument must be a multiple of 32, "
+                         f"with a maximum of {IEEE_CONTEXT_MAX_BITS}")
+
+    ctx = Context()
+    ctx.prec = 9 * (bits//32) - 2
+    ctx.Emax = 3 * (1 << (bits//16 + 3))
+    ctx.Emin = 1 - ctx.Emax
+    ctx.rounding = ROUND_HALF_EVEN
+    ctx.clamp = 1
+    ctx.traps = dict.fromkeys(_signals, False)
+
+    return ctx
 
 
 ##### Decimal class #######################################################
@@ -553,7 +447,7 @@ def localcontext(ctx=None):
 # numbers.py for more detail.
 
 class Decimal(object):
-    """Floating point class for decimal arithmetic."""
+    """Floating-point class for decimal arithmetic."""
 
     __slots__ = ('_exp','_int','_sign', '_is_special')
     # Generally, the value of the Decimal instance is given by
@@ -711,6 +605,21 @@ class Decimal(object):
 
         raise TypeError("Cannot convert %r to Decimal" % value)
 
+    @classmethod
+    def from_number(cls, number):
+        """Converts a real number to a decimal number, exactly.
+
+        >>> Decimal.from_number(314)              # int
+        Decimal('314')
+        >>> Decimal.from_number(0.1)              # float
+        Decimal('0.1000000000000000055511151231257827021181583404541015625')
+        >>> Decimal.from_number(Decimal('3.14'))  # another decimal instance
+        Decimal('3.14')
+        """
+        if isinstance(number, (int, Decimal, float)):
+            return cls(number)
+        raise TypeError("Cannot convert %r to Decimal" % number)
+
     @classmethod
     def from_float(cls, f):
         """Converts a float to a decimal number, exactly.
@@ -993,7 +902,7 @@ class Decimal(object):
             if self.is_snan():
                 raise TypeError('Cannot hash a signaling NaN value.')
             elif self.is_nan():
-                return _PyHASH_NAN
+                return object.__hash__(self)
             else:
                 if self._sign:
                     return -_PyHASH_INF
@@ -1674,13 +1583,13 @@ class Decimal(object):
 
     __trunc__ = __int__
 
+    @property
     def real(self):
         return self
-    real = property(real)
 
+    @property
     def imag(self):
         return Decimal(0)
-    imag = property(imag)
 
     def conjugate(self):
         return self
@@ -2260,10 +2169,16 @@ class Decimal(object):
             else:
                 return None
 
-            if xc >= 10**p:
+            # An exact power of 10 is representable, but can convert to a
+            # string of any length. But an exact power of 10 shouldn't be
+            # possible at this point.
+            assert xc > 1, self
+            assert xc % 10 != 0, self
+            strxc = str(xc)
+            if len(strxc) > p:
                 return None
             xe = -e-xe
-            return _dec_from_triple(0, str(xc), xe)
+            return _dec_from_triple(0, strxc, xe)
 
         # now y is positive; find m and n such that y = m/n
         if ye >= 0:
@@ -2272,7 +2187,7 @@ class Decimal(object):
             if xe != 0 and len(str(abs(yc*xe))) <= -ye:
                 return None
             xc_bits = _nbits(xc)
-            if xc != 1 and len(str(abs(yc)*xc_bits)) <= -ye:
+            if len(str(abs(yc)*xc_bits)) <= -ye:
                 return None
             m, n = yc, 10**(-ye)
             while m % 2 == n % 2 == 0:
@@ -2285,7 +2200,7 @@ class Decimal(object):
         # compute nth root of xc*10**xe
         if n > 1:
             # if 1 < xc < 2**n then xc isn't an nth power
-            if xc != 1 and xc_bits <= n:
+            if xc_bits <= n:
                 return None
 
             xe, rem = divmod(xe, n)
@@ -2313,13 +2228,18 @@ class Decimal(object):
             return None
         xc = xc**m
         xe *= m
-        if xc > 10**p:
+        # An exact power of 10 is representable, but can convert to a string
+        # of any length. But an exact power of 10 shouldn't be possible at
+        # this point.
+        assert xc > 1, self
+        assert xc % 10 != 0, self
+        str_xc = str(xc)
+        if len(str_xc) > p:
             return None
 
         # by this point the result *is* exactly representable
         # adjust the exponent to get as close as possible to the ideal
         # exponent, if necessary
-        str_xc = str(xc)
         if other._isinteger() and other._sign == 0:
             ideal_exponent = self._exp*int(other)
             zeros = min(xe-ideal_exponent, p-len(str_xc))
@@ -2543,12 +2463,12 @@ class Decimal(object):
 
         return ans
 
-    def __rpow__(self, other, context=None):
+    def __rpow__(self, other, modulo=None, context=None):
         """Swaps self/other and returns __pow__."""
         other = _convert_other(other)
         if other is NotImplemented:
             return other
-        return other.__pow__(self, context=context)
+        return other.__pow__(self, modulo, context=context)
 
     def normalize(self, context=None):
         """Normalize- strip trailing 0s, change anything equal to 0 to 0e0"""
@@ -3420,7 +3340,10 @@ class Decimal(object):
         return opa, opb
 
     def logical_and(self, other, context=None):
-        """Applies an 'and' operation between self and other's digits."""
+        """Applies an 'and' operation between self and other's digits.
+
+        Both self and other must be logical numbers.
+        """
         if context is None:
             context = getcontext()
 
@@ -3437,14 +3360,20 @@ class Decimal(object):
         return _dec_from_triple(0, result.lstrip('0') or '0', 0)
 
     def logical_invert(self, context=None):
-        """Invert all its digits."""
+        """Invert all its digits.
+
+        The self must be logical number.
+        """
         if context is None:
             context = getcontext()
         return self.logical_xor(_dec_from_triple(0,'1'*context.prec,0),
                                 context)
 
     def logical_or(self, other, context=None):
-        """Applies an 'or' operation between self and other's digits."""
+        """Applies an 'or' operation between self and other's digits.
+
+        Both self and other must be logical numbers.
+        """
         if context is None:
             context = getcontext()
 
@@ -3461,7 +3390,10 @@ class Decimal(object):
         return _dec_from_triple(0, result.lstrip('0') or '0', 0)
 
     def logical_xor(self, other, context=None):
-        """Applies an 'xor' operation between self and other's digits."""
+        """Applies an 'xor' operation between self and other's digits.
+
+        Both self and other must be logical numbers.
+        """
         if context is None:
             context = getcontext()
 
@@ -3837,6 +3769,10 @@ class Decimal(object):
         # represented in fixed point; rescale them to 0e0.
         if not self and self._exp > 0 and spec['type'] in 'fF%':
             self = self._rescale(0, rounding)
+        if not self and spec['no_neg_0'] and self._sign:
+            adjusted_sign = 0
+        else:
+            adjusted_sign = self._sign
 
         # figure out placement of the decimal point
         leftdigits = self._exp + len(self._int)
@@ -3867,7 +3803,7 @@ class Decimal(object):
 
         # done with the decimal-specific stuff;  hand over the rest
         # of the formatting to the _format_number function
-        return _format_number(self._sign, intpart, fracpart, exp, spec)
+        return _format_number(adjusted_sign, intpart, fracpart, exp, spec)
 
 def _dec_from_triple(sign, coefficient, exponent, special=False):
     """Create a decimal instance directly, without any validation,
@@ -5677,8 +5613,6 @@ class _WorkRep(object):
     def __repr__(self):
         return "(%r, %r, %r)" % (self.sign, self.int, self.exp)
 
-    __str__ = __repr__
-
 
 
 def _normalize(op1, op2, prec = 0):
@@ -6174,7 +6108,7 @@ _parser = re.compile(r"""        # A numeric string consists of:
         (?P<diag>\d*)            # with (possibly empty) diagnostic info.
     )
 #    \s*
-    \Z
+    \z
 """, re.VERBOSE | re.IGNORECASE).match
 
 _all_zeros = re.compile('0*$').match
@@ -6187,7 +6121,7 @@ _exact_half = re.compile('50*$').match
 #
 # A format specifier for Decimal looks like:
 #
-#   [[fill]align][sign][#][0][minimumwidth][,][.precision][type]
+#   [[fill]align][sign][z][#][0][minimumwidth][,][.precision][type]
 
 _parse_format_specifier_regex = re.compile(r"""\A
 (?:
@@ -6195,13 +6129,18 @@ _parse_format_specifier_regex = re.compile(r"""\A
    (?P<align>[<>=^])
 )?
 (?P<sign>[-+ ])?
+(?P<no_neg_0>z)?
 (?P<alt>\#)?
 (?P<zeropad>0)?
-(?P<minimumwidth>(?!0)\d+)?
-(?P<thousands_sep>,)?
-(?:\.(?P<precision>0|(?!0)\d+))?
+(?P<minimumwidth>\d+)?
+(?P<thousands_sep>[,_])?
+(?:\.
+    (?=[\d,_])  # lookahead for digit or separator
+    (?P<precision>\d+)?
+    (?P<frac_separators>[,_])?
+)?
 (?P<type>[eEfFgGn%])?
-\Z
+\z
 """, re.VERBOSE|re.DOTALL)
 
 del re
@@ -6292,6 +6231,9 @@ def _parse_format_specifier(format_spec, _localeconv=None):
         format_dict['grouping'] = [3, 0]
         format_dict['decimal_point'] = '.'
 
+    if format_dict['frac_separators'] is None:
+        format_dict['frac_separators'] = ''
+
     return format_dict
 
 def _format_align(sign, body, spec):
@@ -6411,6 +6353,11 @@ def _format_number(is_negative, intpart, fracpart, exp, spec):
 
     sign = _format_sign(is_negative, spec)
 
+    frac_sep = spec['frac_separators']
+    if fracpart and frac_sep:
+        fracpart = frac_sep.join(fracpart[pos:pos + 3]
+                                 for pos in range(0, len(fracpart), 3))
+
     if fracpart or spec['alt']:
         fracpart = spec['decimal_point'] + fracpart
 

Lib/_pyio.py

@@ -16,15 +16,16 @@ else:
     _setmode = None
 
 import io
-from io import (__all__, SEEK_SET, SEEK_CUR, SEEK_END)
+from io import (__all__, SEEK_SET, SEEK_CUR, SEEK_END, Reader, Writer)  # noqa: F401
 
 valid_seek_flags = {0, 1, 2}  # Hardwired values
 if hasattr(os, 'SEEK_HOLE') :
     valid_seek_flags.add(os.SEEK_HOLE)
     valid_seek_flags.add(os.SEEK_DATA)
 
-# open() uses st_blksize whenever we can
-DEFAULT_BUFFER_SIZE = 8 * 1024  # bytes
+# open() uses max(min(blocksize, 8 MiB), DEFAULT_BUFFER_SIZE)
+# when the device block size is available.
+DEFAULT_BUFFER_SIZE = 128 * 1024  # bytes
 
 # NOTE: Base classes defined here are registered with the "official" ABCs
 # defined in io.py. We don't use real inheritance though, because we don't want
@@ -33,11 +34,8 @@ DEFAULT_BUFFER_SIZE = 8 * 1024  # bytes
 # Rebind for compatibility
 BlockingIOError = BlockingIOError
 
-# Does io.IOBase finalizer log the exception if the close() method fails?
-# The exception is ignored silently by default in release build.
-_IOBASE_EMITS_UNRAISABLE = (hasattr(sys, "gettotalrefcount") or sys.flags.dev_mode)
 # Does open() check its 'errors' argument?
-_CHECK_ERRORS = _IOBASE_EMITS_UNRAISABLE
+_CHECK_ERRORS = (hasattr(sys, "gettotalrefcount") or sys.flags.dev_mode)
 
 
 def text_encoding(encoding, stacklevel=2):
@@ -126,10 +124,10 @@ def open(file, mode="r", buffering=-1, encoding=None, errors=None,
     the size of a fixed-size chunk buffer.  When no buffering argument is
     given, the default buffering policy works as follows:
 
-    * Binary files are buffered in fixed-size chunks; the size of the buffer
-      is chosen using a heuristic trying to determine the underlying device's
-      "block size" and falling back on `io.DEFAULT_BUFFER_SIZE`.
-      On many systems, the buffer will typically be 4096 or 8192 bytes long.
+   * Binary files are buffered in fixed-size chunks; the size of the buffer
+     is max(min(blocksize, 8 MiB), DEFAULT_BUFFER_SIZE)
+     when the device block size is available.
+     On most systems, the buffer will typically be 128 kilobytes long.
 
     * "Interactive" text files (files for which isatty() returns True)
       use line buffering.  Other text files use the policy described above
@@ -241,18 +239,11 @@ def open(file, mode="r", buffering=-1, encoding=None, errors=None,
     result = raw
     try:
         line_buffering = False
-        if buffering == 1 or buffering < 0 and raw.isatty():
+        if buffering == 1 or buffering < 0 and raw._isatty_open_only():
             buffering = -1
             line_buffering = True
         if buffering < 0:
-            buffering = DEFAULT_BUFFER_SIZE
-            try:
-                bs = os.fstat(raw.fileno()).st_blksize
-            except (OSError, AttributeError):
-                pass
-            else:
-                if bs > 1:
-                    buffering = bs
+            buffering = max(min(raw._blksize, 8192 * 1024), DEFAULT_BUFFER_SIZE)
         if buffering < 0:
             raise ValueError("invalid buffering size")
         if buffering == 0:
@@ -416,18 +407,12 @@ class IOBase(metaclass=abc.ABCMeta):
         if closed:
             return
 
-        if _IOBASE_EMITS_UNRAISABLE:
-            self.close()
-        else:
-            # The try/except block is in case this is called at program
-            # exit time, when it's possible that globals have already been
-            # deleted, and then the close() call might fail.  Since
-            # there's nothing we can do about such failures and they annoy
-            # the end users, we suppress the traceback.
-            try:
-                self.close()
-            except:
-                pass
+        if dealloc_warn := getattr(self, "_dealloc_warn", None):
+            dealloc_warn(self)
+
+        # If close() fails, the caller logs the exception with
+        # sys.unraisablehook. close() must be called at the end at __del__().
+        self.close()
 
     ### Inquiries ###
 
@@ -632,16 +617,15 @@ class RawIOBase(IOBase):
         n = self.readinto(b)
         if n is None:
             return None
+        if n < 0 or n > len(b):
+            raise ValueError(f"readinto returned {n} outside buffer size {len(b)}")
         del b[n:]
         return bytes(b)
 
     def readall(self):
         """Read until EOF, using multiple read() call."""
         res = bytearray()
-        while True:
-            data = self.read(DEFAULT_BUFFER_SIZE)
-            if not data:
-                break
+        while data := self.read(DEFAULT_BUFFER_SIZE):
             res += data
         if res:
             return bytes(res)
@@ -666,8 +650,6 @@ class RawIOBase(IOBase):
         self._unsupported("write")
 
 io.RawIOBase.register(RawIOBase)
-from _io import FileIO
-RawIOBase.register(FileIO)
 
 
 class BufferedIOBase(IOBase):
@@ -874,6 +856,10 @@ class _BufferedIOMixin(BufferedIOBase):
         else:
             return "<{}.{} name={!r}>".format(modname, clsname, name)
 
+    def _dealloc_warn(self, source):
+        if dealloc_warn := getattr(self.raw, "_dealloc_warn", None):
+            dealloc_warn(source)
+
     ### Lower-level APIs ###
 
     def fileno(self):
@@ -949,22 +935,22 @@ class BytesIO(BufferedIOBase):
         return self.read(size)
 
     def write(self, b):
-        if self.closed:
-            raise ValueError("write to closed file")
         if isinstance(b, str):
             raise TypeError("can't write str to binary stream")
         with memoryview(b) as view:
+            if self.closed:
+                raise ValueError("write to closed file")
+
             n = view.nbytes  # Size of any bytes-like object
-        if n == 0:
-            return 0
-        pos = self._pos
-        if pos > len(self._buffer):
-            # Inserts null bytes between the current end of the file
-            # and the new write position.
-            padding = b'\x00' * (pos - len(self._buffer))
-            self._buffer += padding
-        self._buffer[pos:pos + n] = b
-        self._pos += n
+            if n == 0:
+                return 0
+
+            pos = self._pos
+            if pos > len(self._buffer):
+                # Pad buffer to pos with null bytes.
+                self._buffer.resize(pos)
+            self._buffer[pos:pos + n] = view
+            self._pos += n
         return n
 
     def seek(self, pos, whence=0):
@@ -1478,6 +1464,17 @@ class BufferedRandom(BufferedWriter, BufferedReader):
         return BufferedWriter.write(self, b)
 
 
+def _new_buffersize(bytes_read):
+    # Parallels _io/fileio.c new_buffersize
+    if bytes_read > 65536:
+        addend = bytes_read >> 3
+    else:
+        addend = 256 + bytes_read
+    if addend < DEFAULT_BUFFER_SIZE:
+        addend = DEFAULT_BUFFER_SIZE
+    return bytes_read + addend
+
+
 class FileIO(RawIOBase):
     _fd = -1
     _created = False
@@ -1502,6 +1499,7 @@ class FileIO(RawIOBase):
         """
         if self._fd >= 0:
             # Have to close the existing file first.
+            self._stat_atopen = None
             try:
                 if self._closefd:
                     os.close(self._fd)
@@ -1511,6 +1509,11 @@ class FileIO(RawIOBase):
         if isinstance(file, float):
             raise TypeError('integer argument expected, got float')
         if isinstance(file, int):
+            if isinstance(file, bool):
+                import warnings
+                warnings.warn("bool is used as a file descriptor",
+                              RuntimeWarning, stacklevel=2)
+                file = int(file)
             fd = file
             if fd < 0:
                 raise ValueError('negative file descriptor')
@@ -1569,24 +1572,22 @@ class FileIO(RawIOBase):
                     if not isinstance(fd, int):
                         raise TypeError('expected integer from opener')
                     if fd < 0:
-                        raise OSError('Negative file descriptor')
+                        # bpo-27066: Raise a ValueError for bad value.
+                        raise ValueError(f'opener returned {fd}')
                 owned_fd = fd
                 if not noinherit_flag:
                     os.set_inheritable(fd, False)
 
             self._closefd = closefd
-            fdfstat = os.fstat(fd)
+            self._stat_atopen = os.fstat(fd)
             try:
-                if stat.S_ISDIR(fdfstat.st_mode):
+                if stat.S_ISDIR(self._stat_atopen.st_mode):
                     raise IsADirectoryError(errno.EISDIR,
                                             os.strerror(errno.EISDIR), file)
             except AttributeError:
                 # Ignore the AttributeError if stat.S_ISDIR or errno.EISDIR
                 # don't exist.
                 pass
-            self._blksize = getattr(fdfstat, 'st_blksize', 0)
-            if self._blksize <= 1:
-                self._blksize = DEFAULT_BUFFER_SIZE
 
             if _setmode:
                 # don't translate newlines (\r\n <=> \n)
@@ -1603,17 +1604,17 @@ class FileIO(RawIOBase):
                     if e.errno != errno.ESPIPE:
                         raise
         except:
+            self._stat_atopen = None
             if owned_fd is not None:
                 os.close(owned_fd)
             raise
         self._fd = fd
 
-    def __del__(self):
+    def _dealloc_warn(self, source):
         if self._fd >= 0 and self._closefd and not self.closed:
             import warnings
-            warnings.warn('unclosed file %r' % (self,), ResourceWarning,
+            warnings.warn(f'unclosed file {source!r}', ResourceWarning,
                           stacklevel=2, source=self)
-            self.close()
 
     def __getstate__(self):
         raise TypeError(f"cannot pickle {self.__class__.__name__!r} object")
@@ -1632,6 +1633,17 @@ class FileIO(RawIOBase):
             return ('<%s name=%r mode=%r closefd=%r>' %
                     (class_name, name, self.mode, self._closefd))
 
+    @property
+    def _blksize(self):
+        if self._stat_atopen is None:
+            return DEFAULT_BUFFER_SIZE
+
+        blksize = getattr(self._stat_atopen, "st_blksize", 0)
+        # WASI sets blsize to 0
+        if not blksize:
+            return DEFAULT_BUFFER_SIZE
+        return blksize
+
     def _checkReadable(self):
         if not self._readable:
             raise UnsupportedOperation('File not open for reading')
@@ -1643,7 +1655,13 @@ class FileIO(RawIOBase):
     def read(self, size=None):
         """Read at most size bytes, returned as bytes.
 
-        Only makes one system call, so less data may be returned than requested
+        If size is less than 0, read all bytes in the file making
+        multiple read calls. See ``FileIO.readall``.
+
+        Attempts to make only one system call, retrying only per
+        PEP 475 (EINTR). This means less data may be returned than
+        requested.
+
         In non-blocking mode, returns None if no data is available.
         Return an empty bytes object at EOF.
         """
@@ -1659,45 +1677,57 @@ class FileIO(RawIOBase):
     def readall(self):
         """Read all data from the file, returned as bytes.
 
-        In non-blocking mode, returns as much as is immediately available,
-        or None if no data is available.  Return an empty bytes object at EOF.
+        Reads until either there is an error or read() returns size 0
+        (indicates EOF). If the file is already at EOF, returns an
+        empty bytes object.
+
+        In non-blocking mode, returns as much data as could be read
+        before EAGAIN. If no data is available (EAGAIN is returned
+        before bytes are read) returns None.
         """
         self._checkClosed()
         self._checkReadable()
-        bufsize = DEFAULT_BUFFER_SIZE
+        if self._stat_atopen is None or self._stat_atopen.st_size <= 0:
+            bufsize = DEFAULT_BUFFER_SIZE
+        else:
+            # In order to detect end of file, need a read() of at least 1
+            # byte which returns size 0. Oversize the buffer by 1 byte so the
+            # I/O can be completed with two read() calls (one for all data, one
+            # for EOF) without needing to resize the buffer.
+            bufsize = self._stat_atopen.st_size + 1
+
+            if self._stat_atopen.st_size > 65536:
+                try:
+                    pos = os.lseek(self._fd, 0, SEEK_CUR)
+                    if self._stat_atopen.st_size >= pos:
+                        bufsize = self._stat_atopen.st_size - pos + 1
+                except OSError:
+                    pass
+
+        result = bytearray(bufsize)
+        bytes_read = 0
         try:
-            pos = os.lseek(self._fd, 0, SEEK_CUR)
-            end = os.fstat(self._fd).st_size
-            if end >= pos:
-                bufsize = end - pos + 1
-        except OSError:
-            pass
-
-        result = bytearray()
-        while True:
-            if len(result) >= bufsize:
-                bufsize = len(result)
-                bufsize += max(bufsize, DEFAULT_BUFFER_SIZE)
-            n = bufsize - len(result)
-            try:
-                chunk = os.read(self._fd, n)
-            except BlockingIOError:
-                if result:
-                    break
+            while n := os.readinto(self._fd, memoryview(result)[bytes_read:]):
+                bytes_read += n
+                if bytes_read >= len(result):
+                    result.resize(_new_buffersize(bytes_read))
+        except BlockingIOError:
+            if not bytes_read:
                 return None
-            if not chunk: # reached the end of the file
-                break
-            result += chunk
 
+        assert len(result) - bytes_read >= 1, \
+            "os.readinto buffer size 0 will result in erroneous EOF / returns 0"
+        result.resize(bytes_read)
         return bytes(result)
 
-    def readinto(self, b):
+    def readinto(self, buffer):
         """Same as RawIOBase.readinto()."""
-        m = memoryview(b).cast('B')
-        data = self.read(len(m))
-        n = len(data)
-        m[:n] = data
-        return n
+        self._checkClosed()
+        self._checkReadable()
+        try:
+            return os.readinto(self._fd, buffer)
+        except BlockingIOError:
+            return None
 
     def write(self, b):
         """Write bytes b to file, return number written.
@@ -1747,6 +1777,7 @@ class FileIO(RawIOBase):
         if size is None:
             size = self.tell()
         os.ftruncate(self._fd, size)
+        self._stat_atopen = None
         return size
 
     def close(self):
@@ -1756,8 +1787,9 @@ class FileIO(RawIOBase):
         called more than once without error.
         """
         if not self.closed:
+            self._stat_atopen = None
             try:
-                if self._closefd:
+                if self._closefd and self._fd >= 0:
                     os.close(self._fd)
             finally:
                 super().close()
@@ -1794,6 +1826,21 @@ class FileIO(RawIOBase):
         self._checkClosed()
         return os.isatty(self._fd)
 
+    def _isatty_open_only(self):
+        """Checks whether the file is a TTY using an open-only optimization.
+
+        TTYs are always character devices. If the interpreter knows a file is
+        not a character device when it would call ``isatty``, can skip that
+        call. Inside ``open()``  there is a fresh stat result that contains that
+        information. Use the stat result to skip a system call. Outside of that
+        context TOCTOU issues (the fd could be arbitrarily modified by
+        surrounding code).
+        """
+        if (self._stat_atopen is not None
+            and not stat.S_ISCHR(self._stat_atopen.st_mode)):
+            return False
+        return os.isatty(self._fd)
+
     @property
     def closefd(self):
         """True if the file descriptor will be closed by close()."""
@@ -2018,8 +2065,7 @@ class TextIOWrapper(TextIOBase):
             raise ValueError("invalid encoding: %r" % encoding)
 
         if not codecs.lookup(encoding)._is_text_encoding:
-            msg = ("%r is not a text encoding; "
-                   "use codecs.open() to handle arbitrary codecs")
+            msg = "%r is not a text encoding"
             raise LookupError(msg % encoding)
 
         if errors is None:
@@ -2527,9 +2573,12 @@ class TextIOWrapper(TextIOBase):
                 size = size_index()
         decoder = self._decoder or self._get_decoder()
         if size < 0:
+            chunk = self.buffer.read()
+            if chunk is None:
+                raise BlockingIOError("Read returned None.")
             # Read everything.
             result = (self._get_decoded_chars() +
-                      decoder.decode(self.buffer.read(), final=True))
+                      decoder.decode(chunk, final=True))
             if self._snapshot is not None:
                 self._set_decoded_chars('')
                 self._snapshot = None
@@ -2649,6 +2698,10 @@ class TextIOWrapper(TextIOBase):
     def newlines(self):
         return self._decoder.newlines if self._decoder else None
 
+    def _dealloc_warn(self, source):
+        if dealloc_warn := getattr(self.buffer, "_dealloc_warn", None):
+            dealloc_warn(source)
+
 
 class StringIO(TextIOWrapper):
     """Text I/O implementation using an in-memory buffer.

Lib/_pylong.py

@@ -0,0 +1,729 @@
+"""Python implementations of some algorithms for use by longobject.c.
+The goal is to provide asymptotically faster algorithms that can be
+used for operations on integers with many digits.  In those cases, the
+performance overhead of the Python implementation is not significant
+since the asymptotic behavior is what dominates runtime. Functions
+provided by this module should be considered private and not part of any
+public API.
+
+Note: for ease of maintainability, please prefer clear code and avoid
+"micro-optimizations".  This module will only be imported and used for
+integers with a huge number of digits.  Saving a few microseconds with
+tricky or non-obvious code is not worth it.  For people looking for
+maximum performance, they should use something like gmpy2."""
+
+import re
+import decimal
+try:
+    import _decimal
+except ImportError:
+    _decimal = None
+
+# A number of functions have this form, where `w` is a desired number of
+# digits in base `base`:
+#
+#    def inner(...w...):
+#        if w <= LIMIT:
+#            return something
+#        lo = w >> 1
+#        hi = w - lo
+#        something involving base**lo, inner(...lo...), j, and inner(...hi...)
+#    figure out largest w needed
+#    result = inner(w)
+#
+# They all had some on-the-fly scheme to cache `base**lo` results for reuse.
+# Power is costly.
+#
+# This routine aims to compute all amd only the needed powers in advance, as
+# efficiently as reasonably possible. This isn't trivial, and all the
+# on-the-fly methods did needless work in many cases. The driving code above
+# changes to:
+#
+#    figure out largest w needed
+#    mycache = compute_powers(w, base, LIMIT)
+#    result = inner(w)
+#
+# and `mycache[lo]` replaces `base**lo` in the inner function.
+#
+# If an algorithm wants the powers of ceiling(w/2) instead of the floor,
+# pass keyword argument `need_hi=True`.
+#
+# While this does give minor speedups (a few percent at best), the
+# primary intent is to simplify the functions using this, by eliminating
+# the need for them to craft their own ad-hoc caching schemes.
+#
+# See code near end of file for a block of code that can be enabled to
+# run millions of tests.
+def compute_powers(w, base, more_than, *, need_hi=False, show=False):
+    seen = set()
+    need = set()
+    ws = {w}
+    while ws:
+        w = ws.pop() # any element is fine to use next
+        if w in seen or w <= more_than:
+            continue
+        seen.add(w)
+        lo = w >> 1
+        hi = w - lo
+        # only _need_ one here; the other may, or may not, be needed
+        which = hi if need_hi else lo
+        need.add(which)
+        ws.add(which)
+        if lo != hi:
+            ws.add(w - which)
+
+    # `need` is the set of exponents needed. To compute them all
+    # efficiently, possibly add other exponents to `extra`. The goal is
+    # to ensure that each exponent can be gotten from a smaller one via
+    # multiplying by the base, squaring it, or squaring and then
+    # multiplying by the base.
+    #
+    # If need_hi is False, this is already the case (w can always be
+    # gotten from w >> 1 via one of the squaring strategies). But we do
+    # the work anyway, just in case ;-)
+    #
+    # Note that speed is irrelevant. These loops are working on little
+    # ints (exponents) and go around O(log w) times. The total cost is
+    # insignificant compared to just one of the bigint multiplies.
+    cands = need.copy()
+    extra = set()
+    while cands:
+        w = max(cands)
+        cands.remove(w)
+        lo = w >> 1
+        if lo > more_than and w-1 not in cands and lo not in cands:
+            extra.add(lo)
+            cands.add(lo)
+    assert need_hi or not extra
+
+    d = {}
+    for n in sorted(need | extra):
+        lo = n >> 1
+        hi = n - lo
+        if n-1 in d:
+            if show:
+                print("* base", end="")
+            result = d[n-1] * base # cheap!
+        elif lo in d:
+            # Multiplying a bigint by itself is about twice as fast
+            # in CPython provided it's the same object.
+            if show:
+                print("square", end="")
+            result = d[lo] * d[lo] # same object
+            if hi != lo:
+                if show:
+                    print(" * base", end="")
+                assert 2 * lo + 1 == n
+                result *= base
+        else: # rare
+            if show:
+                print("pow", end='')
+            result = base ** n
+        if show:
+            print(" at", n, "needed" if n in need else "extra")
+        d[n] = result
+
+    assert need <= d.keys()
+    if excess := d.keys() - need:
+        assert need_hi
+        for n in excess:
+            del d[n]
+    return d
+
+_unbounded_dec_context = decimal.getcontext().copy()
+_unbounded_dec_context.prec = decimal.MAX_PREC
+_unbounded_dec_context.Emax = decimal.MAX_EMAX
+_unbounded_dec_context.Emin = decimal.MIN_EMIN
+_unbounded_dec_context.traps[decimal.Inexact] = 1 # sanity check
+
+def int_to_decimal(n):
+    """Asymptotically fast conversion of an 'int' to Decimal."""
+
+    # Function due to Tim Peters.  See GH issue #90716 for details.
+    # https://github.com/python/cpython/issues/90716
+    #
+    # The implementation in longobject.c of base conversion algorithms
+    # between power-of-2 and non-power-of-2 bases are quadratic time.
+    # This function implements a divide-and-conquer algorithm that is
+    # faster for large numbers.  Builds an equal decimal.Decimal in a
+    # "clever" recursive way.  If we want a string representation, we
+    # apply str to _that_.
+
+    from decimal import Decimal as D
+    BITLIM = 200
+
+    # Don't bother caching the "lo" mask in this; the time to compute it is
+    # tiny compared to the multiply.
+    def inner(n, w):
+        if w <= BITLIM:
+            return D(n)
+        w2 = w >> 1
+        hi = n >> w2
+        lo = n & ((1 << w2) - 1)
+        return inner(lo, w2) + inner(hi, w - w2) * w2pow[w2]
+
+    with decimal.localcontext(_unbounded_dec_context):
+        nbits = n.bit_length()
+        w2pow = compute_powers(nbits, D(2), BITLIM)
+        if n < 0:
+            negate = True
+            n = -n
+        else:
+            negate = False
+        result = inner(n, nbits)
+        if negate:
+            result = -result
+    return result
+
+def int_to_decimal_string(n):
+    """Asymptotically fast conversion of an 'int' to a decimal string."""
+    w = n.bit_length()
+    if w > 450_000 and _decimal is not None:
+        # It is only usable with the C decimal implementation.
+        # _pydecimal.py calls str() on very large integers, which in its
+        # turn calls int_to_decimal_string(), causing very deep recursion.
+        return str(int_to_decimal(n))
+
+    # Fallback algorithm for the case when the C decimal module isn't
+    # available.  This algorithm is asymptotically worse than the algorithm
+    # using the decimal module, but better than the quadratic time
+    # implementation in longobject.c.
+
+    DIGLIM = 1000
+    def inner(n, w):
+        if w <= DIGLIM:
+            return str(n)
+        w2 = w >> 1
+        hi, lo = divmod(n, pow10[w2])
+        return inner(hi, w - w2) + inner(lo, w2).zfill(w2)
+
+    # The estimation of the number of decimal digits.
+    # There is no harm in small error.  If we guess too large, there may
+    # be leading 0's that need to be stripped.  If we guess too small, we
+    # may need to call str() recursively for the remaining highest digits,
+    # which can still potentially be a large integer. This is manifested
+    # only if the number has way more than 10**15 digits, that exceeds
+    # the 52-bit physical address limit in both Intel64 and AMD64.
+    w = int(w * 0.3010299956639812 + 1)  # log10(2)
+    pow10 = compute_powers(w, 5, DIGLIM)
+    for k, v in pow10.items():
+        pow10[k] = v << k # 5**k << k == 5**k * 2**k == 10**k
+    if n < 0:
+        n = -n
+        sign = '-'
+    else:
+        sign = ''
+    s = inner(n, w)
+    if s[0] == '0' and n:
+        # If our guess of w is too large, there may be leading 0's that
+        # need to be stripped.
+        s = s.lstrip('0')
+    return sign + s
+
+def _str_to_int_inner(s):
+    """Asymptotically fast conversion of a 'str' to an 'int'."""
+
+    # Function due to Bjorn Martinsson.  See GH issue #90716 for details.
+    # https://github.com/python/cpython/issues/90716
+    #
+    # The implementation in longobject.c of base conversion algorithms
+    # between power-of-2 and non-power-of-2 bases are quadratic time.
+    # This function implements a divide-and-conquer algorithm making use
+    # of Python's built in big int multiplication. Since Python uses the
+    # Karatsuba algorithm for multiplication, the time complexity
+    # of this function is O(len(s)**1.58).
+
+    DIGLIM = 2048
+
+    def inner(a, b):
+        if b - a <= DIGLIM:
+            return int(s[a:b])
+        mid = (a + b + 1) >> 1
+        return (inner(mid, b)
+                + ((inner(a, mid) * w5pow[b - mid])
+                    << (b - mid)))
+
+    w5pow = compute_powers(len(s), 5, DIGLIM)
+    return inner(0, len(s))
+
+
+# Asymptotically faster version, using the C decimal module. See
+# comments at the end of the file. This uses decimal arithmetic to
+# convert from base 10 to base 256. The latter is just a string of
+# bytes, which CPython can convert very efficiently to a Python int.
+
+# log of 10 to base 256 with best-possible 53-bit precision. Obtained
+# via:
+#    from mpmath import mp
+#    mp.prec = 1000
+#    print(float(mp.log(10, 256)).hex())
+_LOG_10_BASE_256 = float.fromhex('0x1.a934f0979a371p-2') # about 0.415
+
+# _spread is for internal testing. It maps a key to the number of times
+# that condition obtained in _dec_str_to_int_inner:
+#     key 0 - quotient guess was right
+#     key 1 - quotient had to be boosted by 1, one time
+#     key 999 - one adjustment wasn't enough, so fell back to divmod
+from collections import defaultdict
+_spread = defaultdict(int)
+del defaultdict
+
+def _dec_str_to_int_inner(s, *, GUARD=8):
+    # Yes, BYTELIM is "large". Large enough that CPython will usually
+    # use the Karatsuba _str_to_int_inner to convert the string. This
+    # allowed reducing the cutoff for calling _this_ function from 3.5M
+    # to 2M digits. We could almost certainly do even better by
+    # fine-tuning this and/or using a larger output base than 256.
+    BYTELIM = 100_000
+    D = decimal.Decimal
+    result = bytearray()
+    # See notes at end of file for discussion of GUARD.
+    assert GUARD > 0 # if 0, `decimal` can blow up - .prec 0 not allowed
+
+    def inner(n, w):
+        #assert n < D256 ** w # required, but too expensive to check
+        if w <= BYTELIM:
+            # XXX Stefan Pochmann discovered that, for 1024-bit ints,
+            # `int(Decimal)` took 2.5x longer than `int(str(Decimal))`.
+            # Worse, `int(Decimal) is still quadratic-time for much
+            # larger ints. So unless/until all that is repaired, the
+            # seemingly redundant `str(Decimal)` is crucial to speed.
+            result.extend(int(str(n)).to_bytes(w)) # big-endian default
+            return
+        w1 = w >> 1
+        w2 = w - w1
+        if 0:
+            # This is maximally clear, but "too slow". `decimal`
+            # division is asymptotically fast, but we have no way to
+            # tell it to reuse the high-precision reciprocal it computes
+            # for pow256[w2], so it has to recompute it over & over &
+            # over again :-(
+            hi, lo = divmod(n, pow256[w2][0])
+        else:
+            p256, recip = pow256[w2]
+            # The integer part will have a number of digits about equal
+            # to the difference between the log10s of `n` and `pow256`
+            # (which, since these are integers, is roughly approximated
+            # by `.adjusted()`). That's the working precision we need,
+            ctx.prec = max(n.adjusted() - p256.adjusted(), 0) + GUARD
+            hi = +n * +recip # unary `+` chops back to ctx.prec digits
+            ctx.prec = decimal.MAX_PREC
+            hi = hi.to_integral_value() # lose the fractional digits
+            lo = n - hi * p256
+            # Because we've been uniformly rounding down, `hi` is a
+            # lower bound on the correct quotient.
+            assert lo >= 0
+            # Adjust quotient up if needed. It usually isn't. In random
+            # testing on inputs through 5 billion digit strings, the
+            # test triggered once in about 200 thousand tries.
+            count = 0
+            if lo >= p256:
+                count = 1
+                lo -= p256
+                hi += 1
+                if lo >= p256:
+                    # Complete correction via an exact computation. I
+                    # believe it's not possible to get here provided
+                    # GUARD >= 3. It's tested by reducing GUARD below
+                    # that.
+                    count = 999
+                    hi2, lo = divmod(lo, p256)
+                    hi += hi2
+            _spread[count] += 1
+            # The assert should always succeed, but way too slow to keep
+            # enabled.
+            #assert hi, lo == divmod(n, pow256[w2][0])
+        inner(hi, w1)
+        del hi # at top levels, can free a lot of RAM "early"
+        inner(lo, w2)
+
+    # How many base 256 digits are needed?. Mathematically, exactly
+    # floor(log256(int(s))) + 1. There is no cheap way to compute this.
+    # But we can get an upper bound, and that's necessary for our error
+    # analysis to make sense. int(s) < 10**len(s), so the log needed is
+    # < log256(10**len(s)) = len(s) * log256(10). However, using
+    # finite-precision floating point for this, it's possible that the
+    # computed value is a little less than the true value. If the true
+    # value is at - or a little higher than - an integer, we can get an
+    # off-by-1 error too low. So we add 2 instead of 1 if chopping lost
+    # a fraction > 0.9.
+
+    # The "WASI" test platform can complain about `len(s)` if it's too
+    # large to fit in its idea of "an index-sized integer".
+    lenS = s.__len__()
+    log_ub = lenS * _LOG_10_BASE_256
+    log_ub_as_int = int(log_ub)
+    w = log_ub_as_int + 1 + (log_ub - log_ub_as_int > 0.9)
+    # And what if we've plain exhausted the limits of HW floats? We
+    # could compute the log to any desired precision using `decimal`,
+    # but it's not plausible that anyone will pass a string requiring
+    # trillions of bytes (unless they're just trying to "break things").
+    if w.bit_length() >= 46:
+        # "Only" had < 53 - 46 = 7 bits to spare in IEEE-754 double.
+        raise ValueError(f"cannot convert string of len {lenS} to int")
+    with decimal.localcontext(_unbounded_dec_context) as ctx:
+        D256 = D(256)
+        pow256 = compute_powers(w, D256, BYTELIM, need_hi=True)
+        rpow256 = compute_powers(w, 1 / D256, BYTELIM, need_hi=True)
+        # We're going to do inexact, chopped arithmetic, multiplying by
+        # an approximation to the reciprocal of 256**i. We chop to get a
+        # lower bound on the true integer quotient. Our approximation is
+        # a lower bound, the multiplication is chopped too, and
+        # to_integral_value() is also chopped.
+        ctx.traps[decimal.Inexact] = 0
+        ctx.rounding = decimal.ROUND_DOWN
+        for k, v in pow256.items():
+            # No need to save much more precision in the reciprocal than
+            # the power of 256 has, plus some guard digits to absorb
+            # most relevant rounding errors. This is highly significant:
+            # 1/2**i has the same number of significant decimal digits
+            # as 5**i, generally over twice the number in 2**i,
+            ctx.prec = v.adjusted() + GUARD + 1
+            # The unary "+" chops the reciprocal back to that precision.
+            pow256[k] = v, +rpow256[k]
+        del rpow256 # exact reciprocals no longer needed
+        ctx.prec = decimal.MAX_PREC
+        inner(D(s), w)
+    return int.from_bytes(result)
+
+def int_from_string(s):
+    """Asymptotically fast version of PyLong_FromString(), conversion
+    of a string of decimal digits into an 'int'."""
+    # PyLong_FromString() has already removed leading +/-, checked for invalid
+    # use of underscore characters, checked that string consists of only digits
+    # and underscores, and stripped leading whitespace.  The input can still
+    # contain underscores and have trailing whitespace.
+    s = s.rstrip().replace('_', '')
+    func = _str_to_int_inner
+    if len(s) >= 2_000_000 and _decimal is not None:
+        func = _dec_str_to_int_inner
+    return func(s)
+
+def str_to_int(s):
+    """Asymptotically fast version of decimal string to 'int' conversion."""
+    # FIXME: this doesn't support the full syntax that int() supports.
+    m = re.match(r'\s*([+-]?)([0-9_]+)\s*', s)
+    if not m:
+        raise ValueError('invalid literal for int() with base 10')
+    v = int_from_string(m.group(2))
+    if m.group(1) == '-':
+        v = -v
+    return v
+
+
+# Fast integer division, based on code from Mark Dickinson, fast_div.py
+# GH-47701. Additional refinements and optimizations by Bjorn Martinsson.  The
+# algorithm is due to Burnikel and Ziegler, in their paper "Fast Recursive
+# Division".
+
+_DIV_LIMIT = 4000
+
+
+def _div2n1n(a, b, n):
+    """Divide a 2n-bit nonnegative integer a by an n-bit positive integer
+    b, using a recursive divide-and-conquer algorithm.
+
+    Inputs:
+      n is a positive integer
+      b is a positive integer with exactly n bits
+      a is a nonnegative integer such that a < 2**n * b
+
+    Output:
+      (q, r) such that a = b*q+r and 0 <= r < b.
+
+    """
+    if a.bit_length() - n <= _DIV_LIMIT:
+        return divmod(a, b)
+    pad = n & 1
+    if pad:
+        a <<= 1
+        b <<= 1
+        n += 1
+    half_n = n >> 1
+    mask = (1 << half_n) - 1
+    b1, b2 = b >> half_n, b & mask
+    q1, r = _div3n2n(a >> n, (a >> half_n) & mask, b, b1, b2, half_n)
+    q2, r = _div3n2n(r, a & mask, b, b1, b2, half_n)
+    if pad:
+        r >>= 1
+    return q1 << half_n | q2, r
+
+
+def _div3n2n(a12, a3, b, b1, b2, n):
+    """Helper function for _div2n1n; not intended to be called directly."""
+    if a12 >> n == b1:
+        q, r = (1 << n) - 1, a12 - (b1 << n) + b1
+    else:
+        q, r = _div2n1n(a12, b1, n)
+    r = (r << n | a3) - q * b2
+    while r < 0:
+        q -= 1
+        r += b
+    return q, r
+
+
+def _int2digits(a, n):
+    """Decompose non-negative int a into base 2**n
+
+    Input:
+      a is a non-negative integer
+
+    Output:
+      List of the digits of a in base 2**n in little-endian order,
+      meaning the most significant digit is last. The most
+      significant digit is guaranteed to be non-zero.
+      If a is 0 then the output is an empty list.
+
+    """
+    a_digits = [0] * ((a.bit_length() + n - 1) // n)
+
+    def inner(x, L, R):
+        if L + 1 == R:
+            a_digits[L] = x
+            return
+        mid = (L + R) >> 1
+        shift = (mid - L) * n
+        upper = x >> shift
+        lower = x ^ (upper << shift)
+        inner(lower, L, mid)
+        inner(upper, mid, R)
+
+    if a:
+        inner(a, 0, len(a_digits))
+    return a_digits
+
+
+def _digits2int(digits, n):
+    """Combine base-2**n digits into an int. This function is the
+    inverse of `_int2digits`. For more details, see _int2digits.
+    """
+
+    def inner(L, R):
+        if L + 1 == R:
+            return digits[L]
+        mid = (L + R) >> 1
+        shift = (mid - L) * n
+        return (inner(mid, R) << shift) + inner(L, mid)
+
+    return inner(0, len(digits)) if digits else 0
+
+
+def _divmod_pos(a, b):
+    """Divide a non-negative integer a by a positive integer b, giving
+    quotient and remainder."""
+    # Use grade-school algorithm in base 2**n, n = nbits(b)
+    n = b.bit_length()
+    a_digits = _int2digits(a, n)
+
+    r = 0
+    q_digits = []
+    for a_digit in reversed(a_digits):
+        q_digit, r = _div2n1n((r << n) + a_digit, b, n)
+        q_digits.append(q_digit)
+    q_digits.reverse()
+    q = _digits2int(q_digits, n)
+    return q, r
+
+
+def int_divmod(a, b):
+    """Asymptotically fast replacement for divmod, for 'int'.
+    Its time complexity is O(n**1.58), where n = #bits(a) + #bits(b).
+    """
+    if b == 0:
+        raise ZeroDivisionError('division by zero')
+    elif b < 0:
+        q, r = int_divmod(-a, -b)
+        return q, -r
+    elif a < 0:
+        q, r = int_divmod(~a, b)
+        return ~q, b + ~r
+    else:
+        return _divmod_pos(a, b)
+
+
+# Notes on _dec_str_to_int_inner:
+#
+# Stefan Pochmann worked up a str->int function that used the decimal
+# module to, in effect, convert from base 10 to base 256. This is
+# "unnatural", in that it requires multiplying and dividing by large
+# powers of 2, which `decimal` isn't naturally suited to. But
+# `decimal`'s `*` and `/` are asymptotically superior to CPython's, so
+# at _some_ point it could be expected to win.
+#
+# Alas, the crossover point was too high to be of much real interest. I
+# (Tim) then worked on ways to replace its division with multiplication
+# by a cached reciprocal approximation instead, fixing up errors
+# afterwards. This reduced the crossover point significantly,
+#
+# I revisited the code, and found ways to improve and simplify it. The
+# crossover point is at about 3.4 million digits now.
+#
+# About .adjusted()
+# -----------------
+# Restrict to Decimal values x > 0. We don't use negative numbers in the
+# code, and I don't want to have to keep typing, e.g., "absolute value".
+#
+# For convenience, I'll use `x.a` to mean `x.adjusted()`. x.a doesn't
+# look at the digits of x, but instead returns an integer giving x's
+# order of magnitude. These are equivalent:
+#
+# - x.a is the power-of-10 exponent of x's most significant digit.
+# - x.a = the infinitely precise floor(log10(x))
+# - x can be written in this form, where f is a real with 1 <= f < 10:
+#    x = f * 10**x.a
+#
+# Observation; if x is an integer, len(str(x)) = x.a + 1.
+#
+# Lemma 1: (x * y).a = x.a + y.a, or one larger
+#
+# Proof: Write x = f * 10**x.a and y = g * 10**y.a, where f and g are in
+# [1, 10). Then x*y = f*g * 10**(x.a + y.a), where 1 <= f*g < 100. If
+# f*g < 10, (x*y).a is x.a+y.a. Else divide f*g by 10 to bring it back
+# into [1, 10], and add 1 to the exponent to compensate. Then (x*y).a is
+# x.a+y.a+1.
+#
+# Lemma 2: ceiling(log10(x/y)) <= x.a - y.a + 1
+#
+# Proof: Express x and y as in Lemma 1. Then x/y = f/g * 10**(x.a -
+# y.a), where 1/10 < f/g < 10. If 1 <= f/g, (x/y).a is x.a-y.a. Else
+# multiply f/g by 10 to bring it back into [1, 10], and subtract 1 from
+# the exponent to compensate. Then (x/y).a is x.a-y.a-1. So the largest
+# (x/y).a can be is x.a-y.a. Since that's the floor of log10(x/y). the
+# ceiling is at most 1 larger (with equality iff f/g = 1 exactly).
+#
+# GUARD digits
+# ------------
+# We only want the integer part of divisions, so don't need to build
+# the full multiplication tree. But using _just_ the number of
+# digits expected in the integer part ignores too much. What's left
+# out can have a very significant effect on the quotient. So we use
+# GUARD additional digits.
+#
+# The default 8 is more than enough so no more than 1 correction step
+# was ever needed for all inputs tried through 2.5 billion digits. In
+# fact, I believe 3 guard digits are always enough - but the proof is
+# very involved, so better safe than sorry.
+#
+# Short course:
+#
+# If prec is the decimal precision in effect, and we're rounding down,
+# the result of an operation is exactly equal to the infinitely precise
+# result times 1-e for some real e with 0 <= e < 10**(1-prec). In
+#
+#     ctx.prec = max(n.adjusted() - p256.adjusted(), 0) + GUARD
+#     hi = +n * +recip # unary `+` chops to ctx.prec digits
+#
+# we have 3 visible chopped operations, but there's also a 4th:
+# precomputing a truncated `recip` as part of setup.
+#
+# So the computed product is exactly equal to the true product times
+# (1-e1)*(1-e2)*(1-e3)*(1-e4); since the e's are all very small, an
+# excellent approximation to the second factor is 1-(e1+e2+e3+e4) (the
+# 2nd and higher order terms in the expanded product are too tiny to
+# matter). If they're all as large as possible, that's
+#
+# 1 - 4*10**(1-prec). This, BTW, is all bog-standard FP error analysis.
+#
+# That implies the computed product is within 1 of the true product
+# provided prec >= log10(true_product) + 1.602.
+#
+# Here are telegraphic details, rephrasing the initial condition in
+# equivalent ways, step by step:
+#
+# prod - prod * (1 - 4*10**(1-prec)) <= 1
+# prod - prod + prod * 4*10**(1-prec)) <= 1
+# prod * 4*10**(1-prec)) <= 1
+# 10**(log10(prod)) * 4*10**(1-prec)) <= 1
+# 4*10**(1-prec+log10(prod))) <= 1
+# 10**(1-prec+log10(prod))) <= 1/4
+# 1-prec+log10(prod) <= log10(1/4) = -0.602
+# -prec <= -1.602 - log10(prod)
+# prec >= log10(prod) + 1.602
+#
+# The true product is the same as the true ratio n/p256. By Lemma 2
+# above, n.a - p256.a + 1 is an upper bound on the ceiling of
+# log10(prod). Then 2 is the ceiling of 1.602. so n.a - p256.a + 3 is an
+# upper bound on the right hand side of the inequality. Any prec >= that
+# will work.
+#
+# But since this is just a sketch of a proof ;-), the code uses the
+# empirically tested 8 instead of 3. 5 digits more or less makes no
+# practical difference to speed - these ints are huge. And while
+# increasing GUARD above 3 may not be necessary, every increase cuts the
+# percentage of cases that need a correction at all.
+#
+# On Computing Reciprocals
+# ------------------------
+# In general, the exact reciprocals we compute have over twice as many
+# significant digits as needed. 1/256**i has the same number of
+# significant decimal digits as 5**i. It's a significant waste of RAM
+# to store all those unneeded digits.
+#
+# So we cut exact reciprocals back to the least precision that can
+# be needed so that the error analysis above is valid,
+#
+# [Note: turns out it's very significantly faster to do it this way than
+# to compute  1 / 256**i  directly to the desired precision, because the
+# power method doesn't require division. It's also faster than computing
+# (1/256)**i directly to the desired precision - no material division
+# there, but `compute_powers()` is much smarter about _how_ to compute
+# all the powers needed than repeated applications of `**` - that
+# function invokes `**` for at most the few smallest powers needed.]
+#
+# The hard part is that chopping back to a shorter width occurs
+# _outside_ of `inner`. We can't know then what `prec` `inner()` will
+# need. We have to pick, for each value of `w2`, the largest possible
+# value `prec` can become when `inner()` is working on `w2`.
+#
+# This is the `prec` inner() uses:
+#     max(n.a - p256.a, 0) + GUARD
+# and what setup uses (renaming its `v` to `p256` - same thing):
+#     p256.a + GUARD + 1
+#
+# We need that the second is always at least as large as the first,
+# which is the same as requiring
+#
+#     n.a - 2 * p256.a <= 1
+#
+# What's the largest n can be? n < 255**w = 256**(w2 + (w - w2)). The
+# worst case in this context is when w ix even. and then w = 2*w2, so
+# n < 256**(2*w2) = (256**w2)**2 = p256**2. By Lemma 1, then, n.a
+# is at most p256.a + p256.a + 1.
+#
+# So the most n.a - 2 * p256.a can be is
+# p256.a + p256.a + 1 - 2 * p256.a = 1. QED
+#
+# Note: an earlier version of the code split on floor(e/2) instead of on
+# the ceiling. The worst case then is odd `w`, and a more involved proof
+# was needed to show that adding 4 (instead of 1) may be necessary.
+# Basically because, in that case, n may be up to 256 times larger than
+# p256**2. Curiously enough, by splitting on the ceiling instead,
+# nothing in any proof here actually depends on the output base (256).
+
+# Enable for brute-force testing of compute_powers(). This takes about a
+# minute, because it tries millions of cases.
+if 0:
+    def consumer(w, limit, need_hi):
+        seen = set()
+        need = set()
+        def inner(w):
+            if w <= limit:
+                return
+            if w in seen:
+                return
+            seen.add(w)
+            lo = w >> 1
+            hi = w - lo
+            need.add(hi if need_hi else lo)
+            inner(lo)
+            inner(hi)
+        inner(w)
+        exp = compute_powers(w, 1, limit, need_hi=need_hi)
+        assert exp.keys() == need
+
+    from itertools import chain
+    for need_hi in (False, True):
+        for limit in (0, 1, 10, 100, 1_000, 10_000, 100_000):
+            for w in chain(range(1, 100_000),
+                           (10**i for i in range(5, 30))):
+                consumer(w, limit, need_hi)

Lib/_pyrepl/__init__.py

@@ -0,0 +1,19 @@
+#   Copyright 2000-2008 Michael Hudson-Doyle <micahel@gmail.com>
+#                       Armin Rigo
+#
+#                        All Rights Reserved
+#
+#
+# Permission to use, copy, modify, and distribute this software and
+# its documentation for any purpose is hereby granted without fee,
+# provided that the above copyright notice appear in all copies and
+# that both that copyright notice and this permission notice appear in
+# supporting documentation.
+#
+# THE AUTHOR MICHAEL HUDSON DISCLAIMS ALL WARRANTIES WITH REGARD TO
+# THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+# AND FITNESS, IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL,
+# INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER
+# RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
+# CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
+# CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

Lib/_pyrepl/__main__.py

@@ -0,0 +1,6 @@
+# Important: don't add things to this module, as they will end up in the REPL's
+# default globals.  Use _pyrepl.main instead.
+
+if __name__ == "__main__":
+    from .main import interactive_console as __pyrepl_interactive_console
+    __pyrepl_interactive_console()

Lib/_pyrepl/_minimal_curses.py

@@ -0,0 +1,68 @@
+"""Minimal '_curses' module, the low-level interface for curses module
+which is not meant to be used directly.
+
+Based on ctypes.  It's too incomplete to be really called '_curses', so
+to use it, you have to import it and stick it in sys.modules['_curses']
+manually.
+
+Note that there is also a built-in module _minimal_curses which will
+hide this one if compiled in.
+"""
+
+import ctypes
+import ctypes.util
+
+
+class error(Exception):
+    pass
+
+
+def _find_clib() -> str:
+    trylibs = ["ncursesw", "ncurses", "curses"]
+
+    for lib in trylibs:
+        path = ctypes.util.find_library(lib)
+        if path:
+            return path
+    raise ModuleNotFoundError("curses library not found", name="_pyrepl._minimal_curses")
+
+
+_clibpath = _find_clib()
+clib = ctypes.cdll.LoadLibrary(_clibpath)
+
+clib.setupterm.argtypes = [ctypes.c_char_p, ctypes.c_int, ctypes.POINTER(ctypes.c_int)]
+clib.setupterm.restype = ctypes.c_int
+
+clib.tigetstr.argtypes = [ctypes.c_char_p]
+clib.tigetstr.restype = ctypes.c_ssize_t
+
+clib.tparm.argtypes = [ctypes.c_char_p] + 9 * [ctypes.c_int]  # type: ignore[operator]
+clib.tparm.restype = ctypes.c_char_p
+
+OK = 0
+ERR = -1
+
+# ____________________________________________________________
+
+
+def setupterm(termstr, fd):
+    err = ctypes.c_int(0)
+    result = clib.setupterm(termstr, fd, ctypes.byref(err))
+    if result == ERR:
+        raise error("setupterm() failed (err=%d)" % err.value)
+
+
+def tigetstr(cap):
+    if not isinstance(cap, bytes):
+        cap = cap.encode("ascii")
+    result = clib.tigetstr(cap)
+    if result == ERR:
+        return None
+    return ctypes.cast(result, ctypes.c_char_p).value
+
+
+def tparm(str, i1=0, i2=0, i3=0, i4=0, i5=0, i6=0, i7=0, i8=0, i9=0):
+    result = clib.tparm(str, i1, i2, i3, i4, i5, i6, i7, i8, i9)
+    if result is None:
+        raise error("tparm() returned NULL")
+    return result

Lib/_pyrepl/_threading_handler.py

@@ -0,0 +1,74 @@
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+import traceback
+
+
+TYPE_CHECKING = False
+if TYPE_CHECKING:
+    from threading import Thread
+    from types import TracebackType
+    from typing import Protocol
+
+    class ExceptHookArgs(Protocol):
+        @property
+        def exc_type(self) -> type[BaseException]: ...
+        @property
+        def exc_value(self) -> BaseException | None: ...
+        @property
+        def exc_traceback(self) -> TracebackType | None: ...
+        @property
+        def thread(self) -> Thread | None: ...
+
+    class ShowExceptions(Protocol):
+        def __call__(self) -> int: ...
+        def add(self, s: str) -> None: ...
+
+    from .reader import Reader
+
+
+def install_threading_hook(reader: Reader) -> None:
+    import threading
+
+    @dataclass
+    class ExceptHookHandler:
+        lock: threading.Lock = field(default_factory=threading.Lock)
+        messages: list[str] = field(default_factory=list)
+
+        def show(self) -> int:
+            count = 0
+            with self.lock:
+                if not self.messages:
+                    return 0
+                reader.restore()
+                for tb in self.messages:
+                    count += 1
+                    if tb:
+                        print(tb)
+                self.messages.clear()
+                reader.scheduled_commands.append("ctrl-c")
+                reader.prepare()
+            return count
+
+        def add(self, s: str) -> None:
+            with self.lock:
+                self.messages.append(s)
+
+        def exception(self, args: ExceptHookArgs) -> None:
+            lines = traceback.format_exception(
+                args.exc_type,
+                args.exc_value,
+                args.exc_traceback,
+                colorize=reader.can_colorize,
+            )  # type: ignore[call-overload]
+            pre = f"\nException in {args.thread.name}:\n" if args.thread else "\n"
+            tb = pre + "".join(lines)
+            self.add(tb)
+
+        def __call__(self) -> int:
+            return self.show()
+
+
+    handler = ExceptHookHandler()
+    reader.threading_hook = handler
+    threading.excepthook = handler.exception

Lib/_pyrepl/commands.py

@@ -0,0 +1,489 @@
+#   Copyright 2000-2010 Michael Hudson-Doyle <micahel@gmail.com>
+#                       Antonio Cuni
+#                       Armin Rigo
+#
+#                        All Rights Reserved
+#
+#
+# Permission to use, copy, modify, and distribute this software and
+# its documentation for any purpose is hereby granted without fee,
+# provided that the above copyright notice appear in all copies and
+# that both that copyright notice and this permission notice appear in
+# supporting documentation.
+#
+# THE AUTHOR MICHAEL HUDSON DISCLAIMS ALL WARRANTIES WITH REGARD TO
+# THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+# AND FITNESS, IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL,
+# INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER
+# RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
+# CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
+# CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+
+from __future__ import annotations
+import os
+
+# Categories of actions:
+#  killing
+#  yanking
+#  motion
+#  editing
+#  history
+#  finishing
+# [completion]
+
+
+# types
+if False:
+    from .historical_reader import HistoricalReader
+
+
+class Command:
+    finish: bool = False
+    kills_digit_arg: bool = True
+
+    def __init__(
+        self, reader: HistoricalReader, event_name: str, event: list[str]
+    ) -> None:
+        # Reader should really be "any reader" but there's too much usage of
+        # HistoricalReader methods and fields in the code below for us to
+        # refactor at the moment.
+
+        self.reader = reader
+        self.event = event
+        self.event_name = event_name
+
+    def do(self) -> None:
+        pass
+
+
+class KillCommand(Command):
+    def kill_range(self, start: int, end: int) -> None:
+        if start == end:
+            return
+        r = self.reader
+        b = r.buffer
+        text = b[start:end]
+        del b[start:end]
+        if is_kill(r.last_command):
+            if start < r.pos:
+                r.kill_ring[-1] = text + r.kill_ring[-1]
+            else:
+                r.kill_ring[-1] = r.kill_ring[-1] + text
+        else:
+            r.kill_ring.append(text)
+        r.pos = start
+        r.dirty = True
+
+
+class YankCommand(Command):
+    pass
+
+
+class MotionCommand(Command):
+    pass
+
+
+class EditCommand(Command):
+    pass
+
+
+class FinishCommand(Command):
+    finish = True
+    pass
+
+
+def is_kill(command: type[Command] | None) -> bool:
+    return command is not None and issubclass(command, KillCommand)
+
+
+def is_yank(command: type[Command] | None) -> bool:
+    return command is not None and issubclass(command, YankCommand)
+
+
+# etc
+
+
+class digit_arg(Command):
+    kills_digit_arg = False
+
+    def do(self) -> None:
+        r = self.reader
+        c = self.event[-1]
+        if c == "-":
+            if r.arg is not None:
+                r.arg = -r.arg
+            else:
+                r.arg = -1
+        else:
+            d = int(c)
+            if r.arg is None:
+                r.arg = d
+            else:
+                if r.arg < 0:
+                    r.arg = 10 * r.arg - d
+                else:
+                    r.arg = 10 * r.arg + d
+        r.dirty = True
+
+
+class clear_screen(Command):
+    def do(self) -> None:
+        r = self.reader
+        r.console.clear()
+        r.dirty = True
+
+
+class refresh(Command):
+    def do(self) -> None:
+        self.reader.dirty = True
+
+
+class repaint(Command):
+    def do(self) -> None:
+        self.reader.dirty = True
+        self.reader.console.repaint()
+
+
+class kill_line(KillCommand):
+    def do(self) -> None:
+        r = self.reader
+        b = r.buffer
+        eol = r.eol()
+        for c in b[r.pos : eol]:
+            if not c.isspace():
+                self.kill_range(r.pos, eol)
+                return
+        else:
+            self.kill_range(r.pos, eol + 1)
+
+
+class unix_line_discard(KillCommand):
+    def do(self) -> None:
+        r = self.reader
+        self.kill_range(r.bol(), r.pos)
+
+
+class unix_word_rubout(KillCommand):
+    def do(self) -> None:
+        r = self.reader
+        for i in range(r.get_arg()):
+            self.kill_range(r.bow(), r.pos)
+
+
+class kill_word(KillCommand):
+    def do(self) -> None:
+        r = self.reader
+        for i in range(r.get_arg()):
+            self.kill_range(r.pos, r.eow())
+
+
+class backward_kill_word(KillCommand):
+    def do(self) -> None:
+        r = self.reader
+        for i in range(r.get_arg()):
+            self.kill_range(r.bow(), r.pos)
+
+
+class yank(YankCommand):
+    def do(self) -> None:
+        r = self.reader
+        if not r.kill_ring:
+            r.error("nothing to yank")
+            return
+        r.insert(r.kill_ring[-1])
+
+
+class yank_pop(YankCommand):
+    def do(self) -> None:
+        r = self.reader
+        b = r.buffer
+        if not r.kill_ring:
+            r.error("nothing to yank")
+            return
+        if not is_yank(r.last_command):
+            r.error("previous command was not a yank")
+            return
+        repl = len(r.kill_ring[-1])
+        r.kill_ring.insert(0, r.kill_ring.pop())
+        t = r.kill_ring[-1]
+        b[r.pos - repl : r.pos] = t
+        r.pos = r.pos - repl + len(t)
+        r.dirty = True
+
+
+class interrupt(FinishCommand):
+    def do(self) -> None:
+        import signal
+
+        self.reader.console.finish()
+        self.reader.finish()
+        os.kill(os.getpid(), signal.SIGINT)
+
+
+class ctrl_c(Command):
+    def do(self) -> None:
+        self.reader.console.finish()
+        self.reader.finish()
+        raise KeyboardInterrupt
+
+
+class suspend(Command):
+    def do(self) -> None:
+        import signal
+
+        r = self.reader
+        p = r.pos
+        r.console.finish()
+        os.kill(os.getpid(), signal.SIGSTOP)
+        ## this should probably be done
+        ## in a handler for SIGCONT?
+        r.console.prepare()
+        r.pos = p
+        # r.posxy = 0, 0  # XXX this is invalid
+        r.dirty = True
+        r.console.screen = []
+
+
+class up(MotionCommand):
+    def do(self) -> None:
+        r = self.reader
+        for _ in range(r.get_arg()):
+            x, y = r.pos2xy()
+            new_y = y - 1
+
+            if r.bol() == 0:
+                if r.historyi > 0:
+                    r.select_item(r.historyi - 1)
+                    return
+                r.pos = 0
+                r.error("start of buffer")
+                return
+
+            if (
+                x
+                > (
+                    new_x := r.max_column(new_y)
+                )  # we're past the end of the previous line
+                or x == r.max_column(y)
+                and any(
+                    not i.isspace() for i in r.buffer[r.bol() :]
+                )  # move between eols
+            ):
+                x = new_x
+
+            r.setpos_from_xy(x, new_y)
+
+
+class down(MotionCommand):
+    def do(self) -> None:
+        r = self.reader
+        b = r.buffer
+        for _ in range(r.get_arg()):
+            x, y = r.pos2xy()
+            new_y = y + 1
+
+            if r.eol() == len(b):
+                if r.historyi < len(r.history):
+                    r.select_item(r.historyi + 1)
+                    r.pos = r.eol(0)
+                    return
+                r.pos = len(b)
+                r.error("end of buffer")
+                return
+
+            if (
+                x
+                > (
+                    new_x := r.max_column(new_y)
+                )  # we're past the end of the previous line
+                or x == r.max_column(y)
+                and any(
+                    not i.isspace() for i in r.buffer[r.bol() :]
+                )  # move between eols
+            ):
+                x = new_x
+
+            r.setpos_from_xy(x, new_y)
+
+
+class left(MotionCommand):
+    def do(self) -> None:
+        r = self.reader
+        for _ in range(r.get_arg()):
+            p = r.pos - 1
+            if p >= 0:
+                r.pos = p
+            else:
+                self.reader.error("start of buffer")
+
+
+class right(MotionCommand):
+    def do(self) -> None:
+        r = self.reader
+        b = r.buffer
+        for _ in range(r.get_arg()):
+            p = r.pos + 1
+            if p <= len(b):
+                r.pos = p
+            else:
+                self.reader.error("end of buffer")
+
+
+class beginning_of_line(MotionCommand):
+    def do(self) -> None:
+        self.reader.pos = self.reader.bol()
+
+
+class end_of_line(MotionCommand):
+    def do(self) -> None:
+        self.reader.pos = self.reader.eol()
+
+
+class home(MotionCommand):
+    def do(self) -> None:
+        self.reader.pos = 0
+
+
+class end(MotionCommand):
+    def do(self) -> None:
+        self.reader.pos = len(self.reader.buffer)
+
+
+class forward_word(MotionCommand):
+    def do(self) -> None:
+        r = self.reader
+        for i in range(r.get_arg()):
+            r.pos = r.eow()
+
+
+class backward_word(MotionCommand):
+    def do(self) -> None:
+        r = self.reader
+        for i in range(r.get_arg()):
+            r.pos = r.bow()
+
+
+class self_insert(EditCommand):
+    def do(self) -> None:
+        r = self.reader
+        text = self.event * r.get_arg()
+        r.insert(text)
+
+
+class insert_nl(EditCommand):
+    def do(self) -> None:
+        r = self.reader
+        r.insert("\n" * r.get_arg())
+
+
+class transpose_characters(EditCommand):
+    def do(self) -> None:
+        r = self.reader
+        b = r.buffer
+        s = r.pos - 1
+        if s < 0:
+            r.error("cannot transpose at start of buffer")
+        else:
+            if s == len(b):
+                s -= 1
+            t = min(s + r.get_arg(), len(b) - 1)
+            c = b[s]
+            del b[s]
+            b.insert(t, c)
+            r.pos = t
+            r.dirty = True
+
+
+class backspace(EditCommand):
+    def do(self) -> None:
+        r = self.reader
+        b = r.buffer
+        for i in range(r.get_arg()):
+            if r.pos > 0:
+                r.pos -= 1
+                del b[r.pos]
+                r.dirty = True
+            else:
+                self.reader.error("can't backspace at start")
+
+
+class delete(EditCommand):
+    def do(self) -> None:
+        r = self.reader
+        b = r.buffer
+        if (
+            r.pos == 0
+            and len(b) == 0  # this is something of a hack
+            and self.event[-1] == "\004"
+        ):
+            r.update_screen()
+            r.console.finish()
+            raise EOFError
+        for i in range(r.get_arg()):
+            if r.pos != len(b):
+                del b[r.pos]
+                r.dirty = True
+            else:
+                self.reader.error("end of buffer")
+
+
+class accept(FinishCommand):
+    def do(self) -> None:
+        pass
+
+
+class help(Command):
+    def do(self) -> None:
+        import _sitebuiltins
+
+        with self.reader.suspend():
+            self.reader.msg = _sitebuiltins._Helper()()  # type: ignore[assignment, call-arg]
+
+
+class invalid_key(Command):
+    def do(self) -> None:
+        pending = self.reader.console.getpending()
+        s = "".join(self.event) + pending.data
+        self.reader.error("`%r' not bound" % s)
+
+
+class invalid_command(Command):
+    def do(self) -> None:
+        s = self.event_name
+        self.reader.error("command `%s' not known" % s)
+
+
+class show_history(Command):
+    def do(self) -> None:
+        from .pager import get_pager
+        from site import gethistoryfile  # type: ignore[attr-defined]
+
+        history = os.linesep.join(self.reader.history[:])
+        self.reader.console.restore()
+        pager = get_pager()
+        pager(history, gethistoryfile())
+        self.reader.console.prepare()
+
+        # We need to copy over the state so that it's consistent between
+        # console and reader, and console does not overwrite/append stuff
+        self.reader.console.screen = self.reader.screen.copy()
+        self.reader.console.posxy = self.reader.cxy
+
+
+class paste_mode(Command):
+
+    def do(self) -> None:
+        self.reader.paste_mode = not self.reader.paste_mode
+        self.reader.dirty = True
+
+
+class enable_bracketed_paste(Command):
+    def do(self) -> None:
+        self.reader.paste_mode = True
+        self.reader.in_bracketed_paste = True
+
+class disable_bracketed_paste(Command):
+    def do(self) -> None:
+        self.reader.paste_mode = False
+        self.reader.in_bracketed_paste = False
+        self.reader.dirty = True

Lib/_pyrepl/completing_reader.py

@@ -0,0 +1,295 @@
+#   Copyright 2000-2010 Michael Hudson-Doyle <micahel@gmail.com>
+#                       Antonio Cuni
+#
+#                        All Rights Reserved
+#
+#
+# Permission to use, copy, modify, and distribute this software and
+# its documentation for any purpose is hereby granted without fee,
+# provided that the above copyright notice appear in all copies and
+# that both that copyright notice and this permission notice appear in
+# supporting documentation.
+#
+# THE AUTHOR MICHAEL HUDSON DISCLAIMS ALL WARRANTIES WITH REGARD TO
+# THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+# AND FITNESS, IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL,
+# INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER
+# RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
+# CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
+# CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+
+import re
+from . import commands, console, reader
+from .reader import Reader
+
+
+# types
+Command = commands.Command
+if False:
+    from .types import KeySpec, CommandName
+
+
+def prefix(wordlist: list[str], j: int = 0) -> str:
+    d = {}
+    i = j
+    try:
+        while 1:
+            for word in wordlist:
+                d[word[i]] = 1
+            if len(d) > 1:
+                return wordlist[0][j:i]
+            i += 1
+            d = {}
+    except IndexError:
+        return wordlist[0][j:i]
+    return ""
+
+
+STRIPCOLOR_REGEX = re.compile(r"\x1B\[([0-9]{1,3}(;[0-9]{1,2})?)?[m|K]")
+
+def stripcolor(s: str) -> str:
+    return STRIPCOLOR_REGEX.sub('', s)
+
+
+def real_len(s: str) -> int:
+    return len(stripcolor(s))
+
+
+def left_align(s: str, maxlen: int) -> str:
+    stripped = stripcolor(s)
+    if len(stripped) > maxlen:
+        # too bad, we remove the color
+        return stripped[:maxlen]
+    padding = maxlen - len(stripped)
+    return s + ' '*padding
+
+
+def build_menu(
+        cons: console.Console,
+        wordlist: list[str],
+        start: int,
+        use_brackets: bool,
+        sort_in_column: bool,
+) -> tuple[list[str], int]:
+    if use_brackets:
+        item = "[ %s ]"
+        padding = 4
+    else:
+        item = "%s  "
+        padding = 2
+    maxlen = min(max(map(real_len, wordlist)), cons.width - padding)
+    cols = int(cons.width / (maxlen + padding))
+    rows = int((len(wordlist) - 1)/cols + 1)
+
+    if sort_in_column:
+        # sort_in_column=False (default)     sort_in_column=True
+        #          A B C                       A D G
+        #          D E F                       B E
+        #          G                           C F
+        #
+        # "fill" the table with empty words, so we always have the same amout
+        # of rows for each column
+        missing = cols*rows - len(wordlist)
+        wordlist = wordlist + ['']*missing
+        indexes = [(i % cols) * rows + i // cols for i in range(len(wordlist))]
+        wordlist = [wordlist[i] for i in indexes]
+    menu = []
+    i = start
+    for r in range(rows):
+        row = []
+        for col in range(cols):
+            row.append(item % left_align(wordlist[i], maxlen))
+            i += 1
+            if i >= len(wordlist):
+                break
+        menu.append(''.join(row))
+        if i >= len(wordlist):
+            i = 0
+            break
+        if r + 5 > cons.height:
+            menu.append("   %d more... " % (len(wordlist) - i))
+            break
+    return menu, i
+
+# this gets somewhat user interface-y, and as a result the logic gets
+# very convoluted.
+#
+#  To summarise the summary of the summary:- people are a problem.
+#                  -- The Hitch-Hikers Guide to the Galaxy, Episode 12
+
+#### Desired behaviour of the completions commands.
+# the considerations are:
+# (1) how many completions are possible
+# (2) whether the last command was a completion
+# (3) if we can assume that the completer is going to return the same set of
+#     completions: this is controlled by the ``assume_immutable_completions``
+#     variable on the reader, which is True by default to match the historical
+#     behaviour of pyrepl, but e.g. False in the ReadlineAlikeReader to match
+#     more closely readline's semantics (this is needed e.g. by
+#     fancycompleter)
+#
+# if there's no possible completion, beep at the user and point this out.
+# this is easy.
+#
+# if there's only one possible completion, stick it in.  if the last thing
+# user did was a completion, point out that he isn't getting anywhere, but
+# only if the ``assume_immutable_completions`` is True.
+#
+# now it gets complicated.
+#
+# for the first press of a completion key:
+#  if there's a common prefix, stick it in.
+
+#  irrespective of whether anything got stuck in, if the word is now
+#  complete, show the "complete but not unique" message
+
+#  if there's no common prefix and if the word is not now complete,
+#  beep.
+
+#        common prefix ->    yes          no
+#        word complete \/
+#            yes           "cbnu"      "cbnu"
+#            no              -          beep
+
+# for the second bang on the completion key
+#  there will necessarily be no common prefix
+#  show a menu of the choices.
+
+# for subsequent bangs, rotate the menu around (if there are sufficient
+# choices).
+
+
+class complete(commands.Command):
+    def do(self) -> None:
+        r: CompletingReader
+        r = self.reader  # type: ignore[assignment]
+        last_is_completer = r.last_command_is(self.__class__)
+        immutable_completions = r.assume_immutable_completions
+        completions_unchangable = last_is_completer and immutable_completions
+        stem = r.get_stem()
+        if not completions_unchangable:
+            r.cmpltn_menu_choices = r.get_completions(stem)
+
+        completions = r.cmpltn_menu_choices
+        if not completions:
+            r.error("no matches")
+        elif len(completions) == 1:
+            if completions_unchangable and len(completions[0]) == len(stem):
+                r.msg = "[ sole completion ]"
+                r.dirty = True
+            r.insert(completions[0][len(stem):])
+        else:
+            p = prefix(completions, len(stem))
+            if p:
+                r.insert(p)
+            if last_is_completer:
+                r.cmpltn_menu_visible = True
+                r.cmpltn_message_visible = False
+                r.cmpltn_menu, r.cmpltn_menu_end = build_menu(
+                    r.console, completions, r.cmpltn_menu_end,
+                    r.use_brackets, r.sort_in_column)
+                r.dirty = True
+            elif not r.cmpltn_menu_visible:
+                r.cmpltn_message_visible = True
+                if stem + p in completions:
+                    r.msg = "[ complete but not unique ]"
+                    r.dirty = True
+                else:
+                    r.msg = "[ not unique ]"
+                    r.dirty = True
+
+
+class self_insert(commands.self_insert):
+    def do(self) -> None:
+        r: CompletingReader
+        r = self.reader  # type: ignore[assignment]
+
+        commands.self_insert.do(self)
+        if r.cmpltn_menu_visible:
+            stem = r.get_stem()
+            if len(stem) < 1:
+                r.cmpltn_reset()
+            else:
+                completions = [w for w in r.cmpltn_menu_choices
+                               if w.startswith(stem)]
+                if completions:
+                    r.cmpltn_menu, r.cmpltn_menu_end = build_menu(
+                        r.console, completions, 0,
+                        r.use_brackets, r.sort_in_column)
+                else:
+                    r.cmpltn_reset()
+
+
+@dataclass
+class CompletingReader(Reader):
+    """Adds completion support"""
+
+    ### Class variables
+    # see the comment for the complete command
+    assume_immutable_completions = True
+    use_brackets = True  # display completions inside []
+    sort_in_column = False
+
+    ### Instance variables
+    cmpltn_menu: list[str] = field(init=False)
+    cmpltn_menu_visible: bool = field(init=False)
+    cmpltn_message_visible: bool = field(init=False)
+    cmpltn_menu_end: int = field(init=False)
+    cmpltn_menu_choices: list[str] = field(init=False)
+
+    def __post_init__(self) -> None:
+        super().__post_init__()
+        self.cmpltn_reset()
+        for c in (complete, self_insert):
+            self.commands[c.__name__] = c
+            self.commands[c.__name__.replace('_', '-')] = c
+
+    def collect_keymap(self) -> tuple[tuple[KeySpec, CommandName], ...]:
+        return super().collect_keymap() + (
+            (r'\t', 'complete'),)
+
+    def after_command(self, cmd: Command) -> None:
+        super().after_command(cmd)
+        if not isinstance(cmd, (complete, self_insert)):
+            self.cmpltn_reset()
+
+    def calc_screen(self) -> list[str]:
+        screen = super().calc_screen()
+        if self.cmpltn_menu_visible:
+            # We display the completions menu below the current prompt
+            ly = self.lxy[1] + 1
+            screen[ly:ly] = self.cmpltn_menu
+            # If we're not in the middle of multiline edit, don't append to screeninfo
+            # since that screws up the position calculation in pos2xy function.
+            # This is a hack to prevent the cursor jumping
+            # into the completions menu when pressing left or down arrow.
+            if self.pos != len(self.buffer):
+                self.screeninfo[ly:ly] = [(0, [])]*len(self.cmpltn_menu)
+        return screen
+
+    def finish(self) -> None:
+        super().finish()
+        self.cmpltn_reset()
+
+    def cmpltn_reset(self) -> None:
+        self.cmpltn_menu = []
+        self.cmpltn_menu_visible = False
+        self.cmpltn_message_visible = False
+        self.cmpltn_menu_end = 0
+        self.cmpltn_menu_choices = []
+
+    def get_stem(self) -> str:
+        st = self.syntax_table
+        SW = reader.SYNTAX_WORD
+        b = self.buffer
+        p = self.pos - 1
+        while p >= 0 and st.get(b[p], SW) == SW:
+            p -= 1
+        return ''.join(b[p+1:self.pos])
+
+    def get_completions(self, stem: str) -> list[str]:
+        return []

Lib/_pyrepl/console.py

@@ -0,0 +1,213 @@
+#   Copyright 2000-2004 Michael Hudson-Doyle <micahel@gmail.com>
+#
+#                        All Rights Reserved
+#
+#
+# Permission to use, copy, modify, and distribute this software and
+# its documentation for any purpose is hereby granted without fee,
+# provided that the above copyright notice appear in all copies and
+# that both that copyright notice and this permission notice appear in
+# supporting documentation.
+#
+# THE AUTHOR MICHAEL HUDSON DISCLAIMS ALL WARRANTIES WITH REGARD TO
+# THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+# AND FITNESS, IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL,
+# INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER
+# RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
+# CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
+# CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+
+from __future__ import annotations
+
+import _colorize  # type: ignore[import-not-found]
+
+from abc import ABC, abstractmethod
+import ast
+import code
+from dataclasses import dataclass, field
+import os.path
+import sys
+
+
+TYPE_CHECKING = False
+
+if TYPE_CHECKING:
+    from typing import IO
+    from typing import Callable
+
+
+@dataclass
+class Event:
+    evt: str
+    data: str
+    raw: bytes = b""
+
+
+@dataclass
+class Console(ABC):
+    posxy: tuple[int, int]
+    screen: list[str] = field(default_factory=list)
+    height: int = 25
+    width: int = 80
+
+    def __init__(
+        self,
+        f_in: IO[bytes] | int = 0,
+        f_out: IO[bytes] | int = 1,
+        term: str = "",
+        encoding: str = "",
+    ):
+        self.encoding = encoding or sys.getdefaultencoding()
+
+        if isinstance(f_in, int):
+            self.input_fd = f_in
+        else:
+            self.input_fd = f_in.fileno()
+
+        if isinstance(f_out, int):
+            self.output_fd = f_out
+        else:
+            self.output_fd = f_out.fileno()
+
+    @abstractmethod
+    def refresh(self, screen: list[str], xy: tuple[int, int]) -> None: ...
+
+    @abstractmethod
+    def prepare(self) -> None: ...
+
+    @abstractmethod
+    def restore(self) -> None: ...
+
+    @abstractmethod
+    def move_cursor(self, x: int, y: int) -> None: ...
+
+    @abstractmethod
+    def set_cursor_vis(self, visible: bool) -> None: ...
+
+    @abstractmethod
+    def getheightwidth(self) -> tuple[int, int]:
+        """Return (height, width) where height and width are the height
+        and width of the terminal window in characters."""
+        ...
+
+    @abstractmethod
+    def get_event(self, block: bool = True) -> Event | None:
+        """Return an Event instance.  Returns None if |block| is false
+        and there is no event pending, otherwise waits for the
+        completion of an event."""
+        ...
+
+    @abstractmethod
+    def push_char(self, char: int | bytes) -> None:
+        """
+        Push a character to the console event queue.
+        """
+        ...
+
+    @abstractmethod
+    def beep(self) -> None: ...
+
+    @abstractmethod
+    def clear(self) -> None:
+        """Wipe the screen"""
+        ...
+
+    @abstractmethod
+    def finish(self) -> None:
+        """Move the cursor to the end of the display and otherwise get
+        ready for end.  XXX could be merged with restore?  Hmm."""
+        ...
+
+    @abstractmethod
+    def flushoutput(self) -> None:
+        """Flush all output to the screen (assuming there's some
+        buffering going on somewhere)."""
+        ...
+
+    @abstractmethod
+    def forgetinput(self) -> None:
+        """Forget all pending, but not yet processed input."""
+        ...
+
+    @abstractmethod
+    def getpending(self) -> Event:
+        """Return the characters that have been typed but not yet
+        processed."""
+        ...
+
+    @abstractmethod
+    def wait(self, timeout: float | None) -> bool:
+        """Wait for an event. The return value is True if an event is
+        available, False if the timeout has been reached. If timeout is
+        None, wait forever. The timeout is in milliseconds."""
+        ...
+
+    @property
+    def input_hook(self) -> Callable[[], int] | None:
+        """Returns the current input hook."""
+        ...
+
+    @abstractmethod
+    def repaint(self) -> None: ...
+
+
+class InteractiveColoredConsole(code.InteractiveConsole):
+    def __init__(
+        self,
+        locals: dict[str, object] | None = None,
+        filename: str = "<console>",
+        *,
+        local_exit: bool = False,
+    ) -> None:
+        super().__init__(locals=locals, filename=filename, local_exit=local_exit)  # type: ignore[call-arg]
+        self.can_colorize = _colorize.can_colorize()
+
+    def showsyntaxerror(self, filename=None, **kwargs):
+        super().showsyntaxerror(filename=filename, **kwargs)
+
+    def _excepthook(self, typ, value, tb):
+        import traceback
+        lines = traceback.format_exception(
+                typ, value, tb,
+                colorize=self.can_colorize,
+                limit=traceback.BUILTIN_EXCEPTION_LIMIT)
+        self.write(''.join(lines))
+
+    def runsource(self, source, filename="<input>", symbol="single"):
+        try:
+            tree = self.compile.compiler(
+                source,
+                filename,
+                "exec",
+                ast.PyCF_ONLY_AST,
+                incomplete_input=False,
+            )
+        except (SyntaxError, OverflowError, ValueError):
+            self.showsyntaxerror(filename, source=source)
+            return False
+        if tree.body:
+            *_, last_stmt = tree.body
+        for stmt in tree.body:
+            wrapper = ast.Interactive if stmt is last_stmt else ast.Module
+            the_symbol = symbol if stmt is last_stmt else "exec"
+            item = wrapper([stmt])
+            try:
+                code = self.compile.compiler(item, filename, the_symbol)
+            except SyntaxError as e:
+                if e.args[0] == "'await' outside function":
+                    python = os.path.basename(sys.executable)
+                    e.add_note(
+                        f"Try the asyncio REPL ({python} -m asyncio) to use"
+                        f" top-level 'await' and run background asyncio tasks."
+                    )
+                self.showsyntaxerror(filename, source=source)
+                return False
+            except (OverflowError, ValueError):
+                self.showsyntaxerror(filename, source=source)
+                return False
+
+            if code is None:
+                return True
+
+            self.runcode(code)
+        return False

Lib/_pyrepl/curses.py

@@ -0,0 +1,33 @@
+#   Copyright 2000-2010 Michael Hudson-Doyle <micahel@gmail.com>
+#                       Armin Rigo
+#
+#                        All Rights Reserved
+#
+#
+# Permission to use, copy, modify, and distribute this software and
+# its documentation for any purpose is hereby granted without fee,
+# provided that the above copyright notice appear in all copies and
+# that both that copyright notice and this permission notice appear in
+# supporting documentation.
+#
+# THE AUTHOR MICHAEL HUDSON DISCLAIMS ALL WARRANTIES WITH REGARD TO
+# THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+# AND FITNESS, IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL,
+# INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER
+# RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
+# CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
+# CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+
+
+try:
+    import _curses
+except ImportError:
+    try:
+        import curses as _curses  # type: ignore[no-redef]
+    except ImportError:
+        from . import _minimal_curses as _curses  # type: ignore[no-redef]
+
+setupterm = _curses.setupterm
+tigetstr = _curses.tigetstr
+tparm = _curses.tparm
+error = _curses.error

Lib/_pyrepl/fancy_termios.py

@@ -0,0 +1,76 @@
+#   Copyright 2000-2004 Michael Hudson-Doyle <micahel@gmail.com>
+#
+#                        All Rights Reserved
+#
+#
+# Permission to use, copy, modify, and distribute this software and
+# its documentation for any purpose is hereby granted without fee,
+# provided that the above copyright notice appear in all copies and
+# that both that copyright notice and this permission notice appear in
+# supporting documentation.
+#
+# THE AUTHOR MICHAEL HUDSON DISCLAIMS ALL WARRANTIES WITH REGARD TO
+# THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+# AND FITNESS, IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL,
+# INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER
+# RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
+# CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
+# CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+
+import termios
+
+
+class TermState:
+    def __init__(self, tuples):
+        (
+            self.iflag,
+            self.oflag,
+            self.cflag,
+            self.lflag,
+            self.ispeed,
+            self.ospeed,
+            self.cc,
+        ) = tuples
+
+    def as_list(self):
+        return [
+            self.iflag,
+            self.oflag,
+            self.cflag,
+            self.lflag,
+            self.ispeed,
+            self.ospeed,
+            # Always return a copy of the control characters list to ensure
+            # there are not any additional references to self.cc
+            self.cc[:],
+        ]
+
+    def copy(self):
+        return self.__class__(self.as_list())
+
+
+def tcgetattr(fd):
+    return TermState(termios.tcgetattr(fd))
+
+
+def tcsetattr(fd, when, attrs):
+    termios.tcsetattr(fd, when, attrs.as_list())
+
+
+class Term(TermState):
+    TS__init__ = TermState.__init__
+
+    def __init__(self, fd=0):
+        self.TS__init__(termios.tcgetattr(fd))
+        self.fd = fd
+        self.stack = []
+
+    def save(self):
+        self.stack.append(self.as_list())
+
+    def set(self, when=termios.TCSANOW):
+        termios.tcsetattr(self.fd, when, self.as_list())
+
+    def restore(self):
+        self.TS__init__(self.stack.pop())
+        self.set()

Lib/_pyrepl/historical_reader.py

@@ -0,0 +1,419 @@
+#   Copyright 2000-2004 Michael Hudson-Doyle <micahel@gmail.com>
+#
+#                        All Rights Reserved
+#
+#
+# Permission to use, copy, modify, and distribute this software and
+# its documentation for any purpose is hereby granted without fee,
+# provided that the above copyright notice appear in all copies and
+# that both that copyright notice and this permission notice appear in
+# supporting documentation.
+#
+# THE AUTHOR MICHAEL HUDSON DISCLAIMS ALL WARRANTIES WITH REGARD TO
+# THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+# AND FITNESS, IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL,
+# INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER
+# RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
+# CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
+# CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+
+from __future__ import annotations
+
+from contextlib import contextmanager
+from dataclasses import dataclass, field
+
+from . import commands, input
+from .reader import Reader
+
+
+if False:
+    from .types import SimpleContextManager, KeySpec, CommandName
+
+
+isearch_keymap: tuple[tuple[KeySpec, CommandName], ...] = tuple(
+    [("\\%03o" % c, "isearch-end") for c in range(256) if chr(c) != "\\"]
+    + [(c, "isearch-add-character") for c in map(chr, range(32, 127)) if c != "\\"]
+    + [
+        ("\\%03o" % c, "isearch-add-character")
+        for c in range(256)
+        if chr(c).isalpha() and chr(c) != "\\"
+    ]
+    + [
+        ("\\\\", "self-insert"),
+        (r"\C-r", "isearch-backwards"),
+        (r"\C-s", "isearch-forwards"),
+        (r"\C-c", "isearch-cancel"),
+        (r"\C-g", "isearch-cancel"),
+        (r"\<backspace>", "isearch-backspace"),
+    ]
+)
+
+ISEARCH_DIRECTION_NONE = ""
+ISEARCH_DIRECTION_BACKWARDS = "r"
+ISEARCH_DIRECTION_FORWARDS = "f"
+
+
+class next_history(commands.Command):
+    def do(self) -> None:
+        r = self.reader
+        if r.historyi == len(r.history):
+            r.error("end of history list")
+            return
+        r.select_item(r.historyi + 1)
+
+
+class previous_history(commands.Command):
+    def do(self) -> None:
+        r = self.reader
+        if r.historyi == 0:
+            r.error("start of history list")
+            return
+        r.select_item(r.historyi - 1)
+
+
+class history_search_backward(commands.Command):
+    def do(self) -> None:
+        r = self.reader
+        r.search_next(forwards=False)
+
+
+class history_search_forward(commands.Command):
+    def do(self) -> None:
+        r = self.reader
+        r.search_next(forwards=True)
+
+
+class restore_history(commands.Command):
+    def do(self) -> None:
+        r = self.reader
+        if r.historyi != len(r.history):
+            if r.get_unicode() != r.history[r.historyi]:
+                r.buffer = list(r.history[r.historyi])
+                r.pos = len(r.buffer)
+                r.dirty = True
+
+
+class first_history(commands.Command):
+    def do(self) -> None:
+        self.reader.select_item(0)
+
+
+class last_history(commands.Command):
+    def do(self) -> None:
+        self.reader.select_item(len(self.reader.history))
+
+
+class operate_and_get_next(commands.FinishCommand):
+    def do(self) -> None:
+        self.reader.next_history = self.reader.historyi + 1
+
+
+class yank_arg(commands.Command):
+    def do(self) -> None:
+        r = self.reader
+        if r.last_command is self.__class__:
+            r.yank_arg_i += 1
+        else:
+            r.yank_arg_i = 0
+        if r.historyi < r.yank_arg_i:
+            r.error("beginning of history list")
+            return
+        a = r.get_arg(-1)
+        # XXX how to split?
+        words = r.get_item(r.historyi - r.yank_arg_i - 1).split()
+        if a < -len(words) or a >= len(words):
+            r.error("no such arg")
+            return
+        w = words[a]
+        b = r.buffer
+        if r.yank_arg_i > 0:
+            o = len(r.yank_arg_yanked)
+        else:
+            o = 0
+        b[r.pos - o : r.pos] = list(w)
+        r.yank_arg_yanked = w
+        r.pos += len(w) - o
+        r.dirty = True
+
+
+class forward_history_isearch(commands.Command):
+    def do(self) -> None:
+        r = self.reader
+        r.isearch_direction = ISEARCH_DIRECTION_FORWARDS
+        r.isearch_start = r.historyi, r.pos
+        r.isearch_term = ""
+        r.dirty = True
+        r.push_input_trans(r.isearch_trans)
+
+
+class reverse_history_isearch(commands.Command):
+    def do(self) -> None:
+        r = self.reader
+        r.isearch_direction = ISEARCH_DIRECTION_BACKWARDS
+        r.dirty = True
+        r.isearch_term = ""
+        r.push_input_trans(r.isearch_trans)
+        r.isearch_start = r.historyi, r.pos
+
+
+class isearch_cancel(commands.Command):
+    def do(self) -> None:
+        r = self.reader
+        r.isearch_direction = ISEARCH_DIRECTION_NONE
+        r.pop_input_trans()
+        r.select_item(r.isearch_start[0])
+        r.pos = r.isearch_start[1]
+        r.dirty = True
+
+
+class isearch_add_character(commands.Command):
+    def do(self) -> None:
+        r = self.reader
+        b = r.buffer
+        r.isearch_term += self.event[-1]
+        r.dirty = True
+        p = r.pos + len(r.isearch_term) - 1
+        if b[p : p + 1] != [r.isearch_term[-1]]:
+            r.isearch_next()
+
+
+class isearch_backspace(commands.Command):
+    def do(self) -> None:
+        r = self.reader
+        if len(r.isearch_term) > 0:
+            r.isearch_term = r.isearch_term[:-1]
+            r.dirty = True
+        else:
+            r.error("nothing to rubout")
+
+
+class isearch_forwards(commands.Command):
+    def do(self) -> None:
+        r = self.reader
+        r.isearch_direction = ISEARCH_DIRECTION_FORWARDS
+        r.isearch_next()
+
+
+class isearch_backwards(commands.Command):
+    def do(self) -> None:
+        r = self.reader
+        r.isearch_direction = ISEARCH_DIRECTION_BACKWARDS
+        r.isearch_next()
+
+
+class isearch_end(commands.Command):
+    def do(self) -> None:
+        r = self.reader
+        r.isearch_direction = ISEARCH_DIRECTION_NONE
+        r.console.forgetinput()
+        r.pop_input_trans()
+        r.dirty = True
+
+
+@dataclass
+class HistoricalReader(Reader):
+    """Adds history support (with incremental history searching) to the
+    Reader class.
+    """
+
+    history: list[str] = field(default_factory=list)
+    historyi: int = 0
+    next_history: int | None = None
+    transient_history: dict[int, str] = field(default_factory=dict)
+    isearch_term: str = ""
+    isearch_direction: str = ISEARCH_DIRECTION_NONE
+    isearch_start: tuple[int, int] = field(init=False)
+    isearch_trans: input.KeymapTranslator = field(init=False)
+    yank_arg_i: int = 0
+    yank_arg_yanked: str = ""
+
+    def __post_init__(self) -> None:
+        super().__post_init__()
+        for c in [
+            next_history,
+            previous_history,
+            restore_history,
+            first_history,
+            last_history,
+            yank_arg,
+            forward_history_isearch,
+            reverse_history_isearch,
+            isearch_end,
+            isearch_add_character,
+            isearch_cancel,
+            isearch_add_character,
+            isearch_backspace,
+            isearch_forwards,
+            isearch_backwards,
+            operate_and_get_next,
+            history_search_backward,
+            history_search_forward,
+        ]:
+            self.commands[c.__name__] = c
+            self.commands[c.__name__.replace("_", "-")] = c
+        self.isearch_start = self.historyi, self.pos
+        self.isearch_trans = input.KeymapTranslator(
+            isearch_keymap, invalid_cls=isearch_end, character_cls=isearch_add_character
+        )
+
+    def collect_keymap(self) -> tuple[tuple[KeySpec, CommandName], ...]:
+        return super().collect_keymap() + (
+            (r"\C-n", "next-history"),
+            (r"\C-p", "previous-history"),
+            (r"\C-o", "operate-and-get-next"),
+            (r"\C-r", "reverse-history-isearch"),
+            (r"\C-s", "forward-history-isearch"),
+            (r"\M-r", "restore-history"),
+            (r"\M-.", "yank-arg"),
+            (r"\<page down>", "history-search-forward"),
+            (r"\x1b[6~", "history-search-forward"),
+            (r"\<page up>", "history-search-backward"),
+            (r"\x1b[5~", "history-search-backward"),
+        )
+
+    def select_item(self, i: int) -> None:
+        self.transient_history[self.historyi] = self.get_unicode()
+        buf = self.transient_history.get(i)
+        if buf is None:
+            buf = self.history[i].rstrip()
+        self.buffer = list(buf)
+        self.historyi = i
+        self.pos = len(self.buffer)
+        self.dirty = True
+        self.last_refresh_cache.invalidated = True
+
+    def get_item(self, i: int) -> str:
+        if i != len(self.history):
+            return self.transient_history.get(i, self.history[i])
+        else:
+            return self.transient_history.get(i, self.get_unicode())
+
+    @contextmanager
+    def suspend(self) -> SimpleContextManager:
+        with super().suspend(), self.suspend_history():
+            yield
+
+    @contextmanager
+    def suspend_history(self) -> SimpleContextManager:
+        try:
+            old_history = self.history[:]
+            del self.history[:]
+            yield
+        finally:
+            self.history[:] = old_history
+
+    def prepare(self) -> None:
+        super().prepare()
+        try:
+            self.transient_history = {}
+            if self.next_history is not None and self.next_history < len(self.history):
+                self.historyi = self.next_history
+                self.buffer[:] = list(self.history[self.next_history])
+                self.pos = len(self.buffer)
+                self.transient_history[len(self.history)] = ""
+            else:
+                self.historyi = len(self.history)
+            self.next_history = None
+        except:
+            self.restore()
+            raise
+
+    def get_prompt(self, lineno: int, cursor_on_line: bool) -> str:
+        if cursor_on_line and self.isearch_direction != ISEARCH_DIRECTION_NONE:
+            d = "rf"[self.isearch_direction == ISEARCH_DIRECTION_FORWARDS]
+            return "(%s-search `%s') " % (d, self.isearch_term)
+        else:
+            return super().get_prompt(lineno, cursor_on_line)
+
+    def search_next(self, *, forwards: bool) -> None:
+        """Search history for the current line contents up to the cursor.
+
+        Selects the first item found. If nothing is under the cursor, any next
+        item in history is selected.
+        """
+        pos = self.pos
+        s = self.get_unicode()
+        history_index = self.historyi
+
+        # In multiline contexts, we're only interested in the current line.
+        nl_index = s.rfind('\n', 0, pos)
+        prefix = s[nl_index + 1:pos]
+        pos = len(prefix)
+
+        match_prefix = len(prefix)
+        len_item = 0
+        if history_index < len(self.history):
+            len_item = len(self.get_item(history_index))
+        if len_item and pos == len_item:
+            match_prefix = False
+        elif not pos:
+            match_prefix = False
+
+        while 1:
+            if forwards:
+                out_of_bounds = history_index >= len(self.history) - 1
+            else:
+                out_of_bounds = history_index == 0
+            if out_of_bounds:
+                if forwards and not match_prefix:
+                    self.pos = 0
+                    self.buffer = []
+                    self.dirty = True
+                else:
+                    self.error("not found")
+                return
+
+            history_index += 1 if forwards else -1
+            s = self.get_item(history_index)
+
+            if not match_prefix:
+                self.select_item(history_index)
+                return
+
+            len_acc = 0
+            for i, line in enumerate(s.splitlines(keepends=True)):
+                if line.startswith(prefix):
+                    self.select_item(history_index)
+                    self.pos = pos + len_acc
+                    return
+                len_acc += len(line)
+
+    def isearch_next(self) -> None:
+        st = self.isearch_term
+        p = self.pos
+        i = self.historyi
+        s = self.get_unicode()
+        forwards = self.isearch_direction == ISEARCH_DIRECTION_FORWARDS
+        while 1:
+            if forwards:
+                p = s.find(st, p + 1)
+            else:
+                p = s.rfind(st, 0, p + len(st) - 1)
+            if p != -1:
+                self.select_item(i)
+                self.pos = p
+                return
+            elif (forwards and i >= len(self.history) - 1) or (not forwards and i == 0):
+                self.error("not found")
+                return
+            else:
+                if forwards:
+                    i += 1
+                    s = self.get_item(i)
+                    p = -1
+                else:
+                    i -= 1
+                    s = self.get_item(i)
+                    p = len(s)
+
+    def finish(self) -> None:
+        super().finish()
+        ret = self.get_unicode()
+        for i, t in self.transient_history.items():
+            if i < len(self.history) and i != self.historyi:
+                self.history[i] = t
+        if ret and should_auto_add_history:
+            self.history.append(ret)
+
+
+should_auto_add_history = True

Lib/_pyrepl/input.py

@@ -0,0 +1,114 @@
+#   Copyright 2000-2004 Michael Hudson-Doyle <micahel@gmail.com>
+#
+#                        All Rights Reserved
+#
+#
+# Permission to use, copy, modify, and distribute this software and
+# its documentation for any purpose is hereby granted without fee,
+# provided that the above copyright notice appear in all copies and
+# that both that copyright notice and this permission notice appear in
+# supporting documentation.
+#
+# THE AUTHOR MICHAEL HUDSON DISCLAIMS ALL WARRANTIES WITH REGARD TO
+# THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+# AND FITNESS, IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL,
+# INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER
+# RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
+# CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
+# CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+
+# (naming modules after builtin functions is not such a hot idea...)
+
+# an KeyTrans instance translates Event objects into Command objects
+
+# hmm, at what level do we want [C-i] and [tab] to be equivalent?
+# [meta-a] and [esc a]?  obviously, these are going to be equivalent
+# for the UnixConsole, but should they be for PygameConsole?
+
+# it would in any situation seem to be a bad idea to bind, say, [tab]
+# and [C-i] to *different* things... but should binding one bind the
+# other?
+
+# executive, temporary decision: [tab] and [C-i] are distinct, but
+# [meta-key] is identified with [esc key].  We demand that any console
+# class does quite a lot towards emulating a unix terminal.
+
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+import unicodedata
+from collections import deque
+
+
+# types
+if False:
+    from .types import EventTuple
+
+
+class InputTranslator(ABC):
+    @abstractmethod
+    def push(self, evt: EventTuple) -> None:
+        pass
+
+    @abstractmethod
+    def get(self) -> EventTuple | None:
+        return None
+
+    @abstractmethod
+    def empty(self) -> bool:
+        return True
+
+
+class KeymapTranslator(InputTranslator):
+    def __init__(self, keymap, verbose=False, invalid_cls=None, character_cls=None):
+        self.verbose = verbose
+        from .keymap import compile_keymap, parse_keys
+
+        self.keymap = keymap
+        self.invalid_cls = invalid_cls
+        self.character_cls = character_cls
+        d = {}
+        for keyspec, command in keymap:
+            keyseq = tuple(parse_keys(keyspec))
+            d[keyseq] = command
+        if self.verbose:
+            print(d)
+        self.k = self.ck = compile_keymap(d, ())
+        self.results = deque()
+        self.stack = []
+
+    def push(self, evt):
+        if self.verbose:
+            print("pushed", evt.data, end="")
+        key = evt.data
+        d = self.k.get(key)
+        if isinstance(d, dict):
+            if self.verbose:
+                print("transition")
+            self.stack.append(key)
+            self.k = d
+        else:
+            if d is None:
+                if self.verbose:
+                    print("invalid")
+                if self.stack or len(key) > 1 or unicodedata.category(key) == "C":
+                    self.results.append((self.invalid_cls, self.stack + [key]))
+                else:
+                    # small optimization:
+                    self.k[key] = self.character_cls
+                    self.results.append((self.character_cls, [key]))
+            else:
+                if self.verbose:
+                    print("matched", d)
+                self.results.append((d, self.stack + [key]))
+            self.stack = []
+            self.k = self.ck
+
+    def get(self):
+        if self.results:
+            return self.results.popleft()
+        else:
+            return None
+
+    def empty(self) -> bool:
+        return not self.results

Lib/_pyrepl/keymap.py

@@ -0,0 +1,213 @@
+#   Copyright 2000-2008 Michael Hudson-Doyle <micahel@gmail.com>
+#                       Armin Rigo
+#
+#                        All Rights Reserved
+#
+#
+# Permission to use, copy, modify, and distribute this software and
+# its documentation for any purpose is hereby granted without fee,
+# provided that the above copyright notice appear in all copies and
+# that both that copyright notice and this permission notice appear in
+# supporting documentation.
+#
+# THE AUTHOR MICHAEL HUDSON DISCLAIMS ALL WARRANTIES WITH REGARD TO
+# THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+# AND FITNESS, IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL,
+# INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER
+# RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
+# CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
+# CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+
+"""
+Keymap contains functions for parsing keyspecs and turning keyspecs into
+appropriate sequences.
+
+A keyspec is a string representing a sequence of key presses that can
+be bound to a command. All characters other than the backslash represent
+themselves. In the traditional manner, a backslash introduces an escape
+sequence.
+
+pyrepl uses its own keyspec format that is meant to be a strict superset of
+readline's KEYSEQ format. This means that if a spec is found that readline
+accepts that this doesn't, it should be logged as a bug. Note that this means
+we're using the `\\C-o' style of readline's keyspec, not the `Control-o' sort.
+
+The extension to readline is that the sequence \\<KEY> denotes the
+sequence of characters produced by hitting KEY.
+
+Examples:
+`a'      - what you get when you hit the `a' key
+`\\EOA'  - Escape - O - A (up, on my terminal)
+`\\<UP>' - the up arrow key
+`\\<up>' - ditto (keynames are case-insensitive)
+`\\C-o', `\\c-o'  - control-o
+`\\M-.'  - meta-period
+`\\E.'   - ditto (that's how meta works for pyrepl)
+`\\<tab>', `\\<TAB>', `\\t', `\\011', '\\x09', '\\X09', '\\C-i', '\\C-I'
+   - all of these are the tab character.
+"""
+
+_escapes = {
+    "\\": "\\",
+    "'": "'",
+    '"': '"',
+    "a": "\a",
+    "b": "\b",
+    "e": "\033",
+    "f": "\f",
+    "n": "\n",
+    "r": "\r",
+    "t": "\t",
+    "v": "\v",
+}
+
+_keynames = {
+    "backspace": "backspace",
+    "delete": "delete",
+    "down": "down",
+    "end": "end",
+    "enter": "\r",
+    "escape": "\033",
+    "f1": "f1",
+    "f2": "f2",
+    "f3": "f3",
+    "f4": "f4",
+    "f5": "f5",
+    "f6": "f6",
+    "f7": "f7",
+    "f8": "f8",
+    "f9": "f9",
+    "f10": "f10",
+    "f11": "f11",
+    "f12": "f12",
+    "f13": "f13",
+    "f14": "f14",
+    "f15": "f15",
+    "f16": "f16",
+    "f17": "f17",
+    "f18": "f18",
+    "f19": "f19",
+    "f20": "f20",
+    "home": "home",
+    "insert": "insert",
+    "left": "left",
+    "page down": "page down",
+    "page up": "page up",
+    "return": "\r",
+    "right": "right",
+    "space": " ",
+    "tab": "\t",
+    "up": "up",
+}
+
+
+class KeySpecError(Exception):
+    pass
+
+
+def parse_keys(keys: str) -> list[str]:
+    """Parse keys in keyspec format to a sequence of keys."""
+    s = 0
+    r: list[str] = []
+    while s < len(keys):
+        k, s = _parse_single_key_sequence(keys, s)
+        r.extend(k)
+    return r
+
+
+def _parse_single_key_sequence(key: str, s: int) -> tuple[list[str], int]:
+    ctrl = 0
+    meta = 0
+    ret = ""
+    while not ret and s < len(key):
+        if key[s] == "\\":
+            c = key[s + 1].lower()
+            if c in _escapes:
+                ret = _escapes[c]
+                s += 2
+            elif c == "c":
+                if key[s + 2] != "-":
+                    raise KeySpecError(
+                        "\\C must be followed by `-' (char %d of %s)"
+                        % (s + 2, repr(key))
+                    )
+                if ctrl:
+                    raise KeySpecError(
+                        "doubled \\C- (char %d of %s)" % (s + 1, repr(key))
+                    )
+                ctrl = 1
+                s += 3
+            elif c == "m":
+                if key[s + 2] != "-":
+                    raise KeySpecError(
+                        "\\M must be followed by `-' (char %d of %s)"
+                        % (s + 2, repr(key))
+                    )
+                if meta:
+                    raise KeySpecError(
+                        "doubled \\M- (char %d of %s)" % (s + 1, repr(key))
+                    )
+                meta = 1
+                s += 3
+            elif c.isdigit():
+                n = key[s + 1 : s + 4]
+                ret = chr(int(n, 8))
+                s += 4
+            elif c == "x":
+                n = key[s + 2 : s + 4]
+                ret = chr(int(n, 16))
+                s += 4
+            elif c == "<":
+                t = key.find(">", s)
+                if t == -1:
+                    raise KeySpecError(
+                        "unterminated \\< starting at char %d of %s"
+                        % (s + 1, repr(key))
+                    )
+                ret = key[s + 2 : t].lower()
+                if ret not in _keynames:
+                    raise KeySpecError(
+                        "unrecognised keyname `%s' at char %d of %s"
+                        % (ret, s + 2, repr(key))
+                    )
+                ret = _keynames[ret]
+                s = t + 1
+            else:
+                raise KeySpecError(
+                    "unknown backslash escape %s at char %d of %s"
+                    % (repr(c), s + 2, repr(key))
+                )
+        else:
+            ret = key[s]
+            s += 1
+    if ctrl:
+        if len(ret) == 1:
+            ret = chr(ord(ret) & 0x1F)  # curses.ascii.ctrl()
+        elif ret in {"left", "right"}:
+            ret = f"ctrl {ret}"
+        else:
+            raise KeySpecError("\\C- followed by invalid key")
+
+    result = [ret], s
+    if meta:
+        result[0].insert(0, "\033")
+    return result
+
+
+def compile_keymap(keymap, empty=b""):
+    r = {}
+    for key, value in keymap.items():
+        if isinstance(key, bytes):
+            first = key[:1]
+        else:
+            first = key[0]
+        r.setdefault(first, {})[key[1:]] = value
+    for key, value in r.items():
+        if empty in value:
+            if len(value) != 1:
+                raise KeySpecError("key definitions for %s clash" % (value.values(),))
+            else:
+                r[key] = value[empty]
+        else:
+            r[key] = compile_keymap(value, empty)
+    return r

Lib/_pyrepl/main.py

@@ -0,0 +1,59 @@
+import errno
+import os
+import sys
+
+
+CAN_USE_PYREPL: bool
+FAIL_REASON: str
+try:
+    if sys.platform == "win32" and sys.getwindowsversion().build < 10586:
+        raise RuntimeError("Windows 10 TH2 or later required")
+    if not os.isatty(sys.stdin.fileno()):
+        raise OSError(errno.ENOTTY, "tty required", "stdin")
+    from .simple_interact import check
+    if err := check():
+        raise RuntimeError(err)
+except Exception as e:
+    CAN_USE_PYREPL = False
+    FAIL_REASON = f"warning: can't use pyrepl: {e}"
+else:
+    CAN_USE_PYREPL = True
+    FAIL_REASON = ""
+
+
+def interactive_console(mainmodule=None, quiet=False, pythonstartup=False):
+    if not CAN_USE_PYREPL:
+        if not os.getenv('PYTHON_BASIC_REPL') and FAIL_REASON:
+            from .trace import trace
+            trace(FAIL_REASON)
+            print(FAIL_REASON, file=sys.stderr)
+        return sys._baserepl()
+
+    if mainmodule:
+        namespace = mainmodule.__dict__
+    else:
+        import __main__
+        namespace = __main__.__dict__
+        namespace.pop("__pyrepl_interactive_console", None)
+
+    # sys._baserepl() above does this internally, we do it here
+    startup_path = os.getenv("PYTHONSTARTUP")
+    if pythonstartup and startup_path:
+        sys.audit("cpython.run_startup", startup_path)
+
+        import tokenize
+        with tokenize.open(startup_path) as f:
+            startup_code = compile(f.read(), startup_path, "exec")
+            exec(startup_code, namespace)
+
+    # set sys.{ps1,ps2} just before invoking the interactive interpreter. This
+    # mimics what CPython does in pythonrun.c
+    if not hasattr(sys, "ps1"):
+        sys.ps1 = ">>> "
+    if not hasattr(sys, "ps2"):
+        sys.ps2 = "... "
+
+    from .console import InteractiveColoredConsole
+    from .simple_interact import run_multiline_interactive_console
+    console = InteractiveColoredConsole(namespace, filename="<stdin>")
+    run_multiline_interactive_console(console)

Lib/_pyrepl/mypy.ini

@@ -0,0 +1,24 @@
+# Config file for running mypy on _pyrepl.
+# Run mypy by invoking `mypy --config-file Lib/_pyrepl/mypy.ini`
+# on the command-line from the repo root
+
+[mypy]
+files = Lib/_pyrepl
+explicit_package_bases = True
+python_version = 3.12
+platform = linux
+pretty = True
+
+# Enable most stricter settings
+enable_error_code = ignore-without-code,redundant-expr
+strict = True
+
+# Various stricter settings that we can't yet enable
+# Try to enable these in the following order:
+disallow_untyped_calls = False
+disallow_untyped_defs = False
+check_untyped_defs = False
+
+# Various internal modules that typeshed deliberately doesn't have stubs for:
+[mypy-_abc.*,_opcode.*,_overlapped.*,_testcapi.*,_testinternalcapi.*,test.*]
+ignore_missing_imports = True

Lib/_pyrepl/pager.py

@@ -0,0 +1,175 @@
+from __future__ import annotations
+
+import io
+import os
+import re
+import sys
+
+
+# types
+if False:
+    from typing import Protocol
+    class Pager(Protocol):
+        def __call__(self, text: str, title: str = "") -> None:
+            ...
+
+
+def get_pager() -> Pager:
+    """Decide what method to use for paging through text."""
+    if not hasattr(sys.stdin, "isatty"):
+        return plain_pager
+    if not hasattr(sys.stdout, "isatty"):
+        return plain_pager
+    if not sys.stdin.isatty() or not sys.stdout.isatty():
+        return plain_pager
+    if sys.platform == "emscripten":
+        return plain_pager
+    use_pager = os.environ.get('MANPAGER') or os.environ.get('PAGER')
+    if use_pager:
+        if sys.platform == 'win32': # pipes completely broken in Windows
+            return lambda text, title='': tempfile_pager(plain(text), use_pager)
+        elif os.environ.get('TERM') in ('dumb', 'emacs'):
+            return lambda text, title='': pipe_pager(plain(text), use_pager, title)
+        else:
+            return lambda text, title='': pipe_pager(text, use_pager, title)
+    if os.environ.get('TERM') in ('dumb', 'emacs'):
+        return plain_pager
+    if sys.platform == 'win32':
+        return lambda text, title='': tempfile_pager(plain(text), 'more <')
+    if hasattr(os, 'system') and os.system('(pager) 2>/dev/null') == 0:
+        return lambda text, title='': pipe_pager(text, 'pager', title)
+    if hasattr(os, 'system') and os.system('(less) 2>/dev/null') == 0:
+        return lambda text, title='': pipe_pager(text, 'less', title)
+
+    import tempfile
+    (fd, filename) = tempfile.mkstemp()
+    os.close(fd)
+    try:
+        if hasattr(os, 'system') and os.system('more "%s"' % filename) == 0:
+            return lambda text, title='': pipe_pager(text, 'more', title)
+        else:
+            return tty_pager
+    finally:
+        os.unlink(filename)
+
+
+def escape_stdout(text: str) -> str:
+    # Escape non-encodable characters to avoid encoding errors later
+    encoding = getattr(sys.stdout, 'encoding', None) or 'utf-8'
+    return text.encode(encoding, 'backslashreplace').decode(encoding)
+
+
+def escape_less(s: str) -> str:
+    return re.sub(r'([?:.%\\])', r'\\\1', s)
+
+
+def plain(text: str) -> str:
+    """Remove boldface formatting from text."""
+    return re.sub('.\b', '', text)
+
+
+def tty_pager(text: str, title: str = '') -> None:
+    """Page through text on a text terminal."""
+    lines = plain(escape_stdout(text)).split('\n')
+    has_tty = False
+    try:
+        import tty
+        import termios
+        fd = sys.stdin.fileno()
+        old = termios.tcgetattr(fd)
+        tty.setcbreak(fd)
+        has_tty = True
+
+        def getchar() -> str:
+            return sys.stdin.read(1)
+
+    except (ImportError, AttributeError, io.UnsupportedOperation):
+        def getchar() -> str:
+            return sys.stdin.readline()[:-1][:1]
+
+    try:
+        try:
+            h = int(os.environ.get('LINES', 0))
+        except ValueError:
+            h = 0
+        if h <= 1:
+            h = 25
+        r = inc = h - 1
+        sys.stdout.write('\n'.join(lines[:inc]) + '\n')
+        while lines[r:]:
+            sys.stdout.write('-- more --')
+            sys.stdout.flush()
+            c = getchar()
+
+            if c in ('q', 'Q'):
+                sys.stdout.write('\r          \r')
+                break
+            elif c in ('\r', '\n'):
+                sys.stdout.write('\r          \r' + lines[r] + '\n')
+                r = r + 1
+                continue
+            if c in ('b', 'B', '\x1b'):
+                r = r - inc - inc
+                if r < 0: r = 0
+            sys.stdout.write('\n' + '\n'.join(lines[r:r+inc]) + '\n')
+            r = r + inc
+
+    finally:
+        if has_tty:
+            termios.tcsetattr(fd, termios.TCSAFLUSH, old)
+
+
+def plain_pager(text: str, title: str = '') -> None:
+    """Simply print unformatted text.  This is the ultimate fallback."""
+    sys.stdout.write(plain(escape_stdout(text)))
+
+
+def pipe_pager(text: str, cmd: str, title: str = '') -> None:
+    """Page through text by feeding it to another program."""
+    import subprocess
+    env = os.environ.copy()
+    if title:
+        title += ' '
+    esc_title = escape_less(title)
+    prompt_string = (
+        f' {esc_title}' +
+        '?ltline %lt?L/%L.'
+        ':byte %bB?s/%s.'
+        '.'
+        '?e (END):?pB %pB\\%..'
+        ' (press h for help or q to quit)')
+    env['LESS'] = '-RmPm{0}$PM{0}$'.format(prompt_string)
+    proc = subprocess.Popen(cmd, shell=True, stdin=subprocess.PIPE,
+                            errors='backslashreplace', env=env)
+    assert proc.stdin is not None
+    try:
+        with proc.stdin as pipe:
+            try:
+                pipe.write(text)
+            except KeyboardInterrupt:
+                # We've hereby abandoned whatever text hasn't been written,
+                # but the pager is still in control of the terminal.
+                pass
+    except OSError:
+        pass # Ignore broken pipes caused by quitting the pager program.
+    while True:
+        try:
+            proc.wait()
+            break
+        except KeyboardInterrupt:
+            # Ignore ctl-c like the pager itself does.  Otherwise the pager is
+            # left running and the terminal is in raw mode and unusable.
+            pass
+
+
+def tempfile_pager(text: str, cmd: str, title: str = '') -> None:
+    """Page through text by invoking a program on a temporary file."""
+    import tempfile
+    with tempfile.TemporaryDirectory() as tempdir:
+        filename = os.path.join(tempdir, 'pydoc.out')
+        with open(filename, 'w', errors='backslashreplace',
+                  encoding=os.device_encoding(0) if
+                  sys.platform == 'win32' else None
+                  ) as file:
+            file.write(text)
+        os.system(cmd + ' "' + filename + '"')

Lib/_pyrepl/reader.py

@@ -0,0 +1,816 @@
+#   Copyright 2000-2010 Michael Hudson-Doyle <micahel@gmail.com>
+#                       Antonio Cuni
+#                       Armin Rigo
+#
+#                        All Rights Reserved
+#
+#
+# Permission to use, copy, modify, and distribute this software and
+# its documentation for any purpose is hereby granted without fee,
+# provided that the above copyright notice appear in all copies and
+# that both that copyright notice and this permission notice appear in
+# supporting documentation.
+#
+# THE AUTHOR MICHAEL HUDSON DISCLAIMS ALL WARRANTIES WITH REGARD TO
+# THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+# AND FITNESS, IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL,
+# INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER
+# RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
+# CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
+# CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+
+from __future__ import annotations
+
+import sys
+
+from contextlib import contextmanager
+from dataclasses import dataclass, field, fields
+import unicodedata
+from _colorize import can_colorize, ANSIColors  # type: ignore[import-not-found]
+
+
+from . import commands, console, input
+from .utils import ANSI_ESCAPE_SEQUENCE, wlen, str_width
+from .trace import trace
+
+
+# types
+Command = commands.Command
+from .types import Callback, SimpleContextManager, KeySpec, CommandName
+
+
+def disp_str(buffer: str) -> tuple[str, list[int]]:
+    """disp_str(buffer:string) -> (string, [int])
+
+    Return the string that should be the printed representation of
+    |buffer| and a list detailing where the characters of |buffer|
+    get used up.  E.g.:
+
+    >>> disp_str(chr(3))
+    ('^C', [1, 0])
+
+    """
+    b: list[int] = []
+    s: list[str] = []
+    for c in buffer:
+        if c == '\x1a':
+            s.append(c)
+            b.append(2)
+        elif ord(c) < 128:
+            s.append(c)
+            b.append(1)
+        elif unicodedata.category(c).startswith("C"):
+            c = r"\u%04x" % ord(c)
+            s.append(c)
+            b.extend([0] * (len(c) - 1))
+        else:
+            s.append(c)
+            b.append(str_width(c))
+    return "".join(s), b
+
+
+# syntax classes:
+
+SYNTAX_WHITESPACE, SYNTAX_WORD, SYNTAX_SYMBOL = range(3)
+
+
+def make_default_syntax_table() -> dict[str, int]:
+    # XXX perhaps should use some unicodedata here?
+    st: dict[str, int] = {}
+    for c in map(chr, range(256)):
+        st[c] = SYNTAX_SYMBOL
+    for c in [a for a in map(chr, range(256)) if a.isalnum()]:
+        st[c] = SYNTAX_WORD
+    st["\n"] = st[" "] = SYNTAX_WHITESPACE
+    return st
+
+
+def make_default_commands() -> dict[CommandName, type[Command]]:
+    result: dict[CommandName, type[Command]] = {}
+    for v in vars(commands).values():
+        if isinstance(v, type) and issubclass(v, Command) and v.__name__[0].islower():
+            result[v.__name__] = v
+            result[v.__name__.replace("_", "-")] = v
+    return result
+
+
+default_keymap: tuple[tuple[KeySpec, CommandName], ...] = tuple(
+    [
+        (r"\C-a", "beginning-of-line"),
+        (r"\C-b", "left"),
+        (r"\C-c", "interrupt"),
+        (r"\C-d", "delete"),
+        (r"\C-e", "end-of-line"),
+        (r"\C-f", "right"),
+        (r"\C-g", "cancel"),
+        (r"\C-h", "backspace"),
+        (r"\C-j", "accept"),
+        (r"\<return>", "accept"),
+        (r"\C-k", "kill-line"),
+        (r"\C-l", "clear-screen"),
+        (r"\C-m", "accept"),
+        (r"\C-t", "transpose-characters"),
+        (r"\C-u", "unix-line-discard"),
+        (r"\C-w", "unix-word-rubout"),
+        (r"\C-x\C-u", "upcase-region"),
+        (r"\C-y", "yank"),
+        *(() if sys.platform == "win32" else ((r"\C-z", "suspend"), )),
+        (r"\M-b", "backward-word"),
+        (r"\M-c", "capitalize-word"),
+        (r"\M-d", "kill-word"),
+        (r"\M-f", "forward-word"),
+        (r"\M-l", "downcase-word"),
+        (r"\M-t", "transpose-words"),
+        (r"\M-u", "upcase-word"),
+        (r"\M-y", "yank-pop"),
+        (r"\M--", "digit-arg"),
+        (r"\M-0", "digit-arg"),
+        (r"\M-1", "digit-arg"),
+        (r"\M-2", "digit-arg"),
+        (r"\M-3", "digit-arg"),
+        (r"\M-4", "digit-arg"),
+        (r"\M-5", "digit-arg"),
+        (r"\M-6", "digit-arg"),
+        (r"\M-7", "digit-arg"),
+        (r"\M-8", "digit-arg"),
+        (r"\M-9", "digit-arg"),
+        (r"\M-\n", "accept"),
+        ("\\\\", "self-insert"),
+        (r"\x1b[200~", "enable_bracketed_paste"),
+        (r"\x1b[201~", "disable_bracketed_paste"),
+        (r"\x03", "ctrl-c"),
+    ]
+    + [(c, "self-insert") for c in map(chr, range(32, 127)) if c != "\\"]
+    + [(c, "self-insert") for c in map(chr, range(128, 256)) if c.isalpha()]
+    + [
+        (r"\<up>", "up"),
+        (r"\<down>", "down"),
+        (r"\<left>", "left"),
+        (r"\C-\<left>", "backward-word"),
+        (r"\<right>", "right"),
+        (r"\C-\<right>", "forward-word"),
+        (r"\<delete>", "delete"),
+        (r"\x1b[3~", "delete"),
+        (r"\<backspace>", "backspace"),
+        (r"\M-\<backspace>", "backward-kill-word"),
+        (r"\<end>", "end-of-line"),  # was 'end'
+        (r"\<home>", "beginning-of-line"),  # was 'home'
+        (r"\<f1>", "help"),
+        (r"\<f2>", "show-history"),
+        (r"\<f3>", "paste-mode"),
+        (r"\EOF", "end"),  # the entries in the terminfo database for xterms
+        (r"\EOH", "home"),  # seem to be wrong.  this is a less than ideal
+        # workaround
+    ]
+)
+
+
+@dataclass(slots=True)
+class Reader:
+    """The Reader class implements the bare bones of a command reader,
+    handling such details as editing and cursor motion.  What it does
+    not support are such things as completion or history support -
+    these are implemented elsewhere.
+
+    Instance variables of note include:
+
+      * buffer:
+        A *list* (*not* a string at the moment :-) containing all the
+        characters that have been entered.
+      * console:
+        Hopefully encapsulates the OS dependent stuff.
+      * pos:
+        A 0-based index into `buffer' for where the insertion point
+        is.
+      * screeninfo:
+        Ahem.  This list contains some info needed to move the
+        insertion point around reasonably efficiently.
+      * cxy, lxy:
+        the position of the insertion point in screen ...
+      * syntax_table:
+        Dictionary mapping characters to `syntax class'; read the
+        emacs docs to see what this means :-)
+      * commands:
+        Dictionary mapping command names to command classes.
+      * arg:
+        The emacs-style prefix argument.  It will be None if no such
+        argument has been provided.
+      * dirty:
+        True if we need to refresh the display.
+      * kill_ring:
+        The emacs-style kill-ring; manipulated with yank & yank-pop
+      * ps1, ps2, ps3, ps4:
+        prompts.  ps1 is the prompt for a one-line input; for a
+        multiline input it looks like:
+            ps2> first line of input goes here
+            ps3> second and further
+            ps3> lines get ps3
+            ...
+            ps4> and the last one gets ps4
+        As with the usual top-level, you can set these to instances if
+        you like; str() will be called on them (once) at the beginning
+        of each command.  Don't put really long or newline containing
+        strings here, please!
+        This is just the default policy; you can change it freely by
+        overriding get_prompt() (and indeed some standard subclasses
+        do).
+      * finished:
+        handle1 will set this to a true value if a command signals
+        that we're done.
+    """
+
+    console: console.Console
+
+    ## state
+    buffer: list[str] = field(default_factory=list)
+    pos: int = 0
+    ps1: str = "->> "
+    ps2: str = "/>> "
+    ps3: str = "|.. "
+    ps4: str = R"\__ "
+    kill_ring: list[list[str]] = field(default_factory=list)
+    msg: str = ""
+    arg: int | None = None
+    dirty: bool = False
+    finished: bool = False
+    paste_mode: bool = False
+    in_bracketed_paste: bool = False
+    commands: dict[str, type[Command]] = field(default_factory=make_default_commands)
+    last_command: type[Command] | None = None
+    syntax_table: dict[str, int] = field(default_factory=make_default_syntax_table)
+    keymap: tuple[tuple[str, str], ...] = ()
+    input_trans: input.KeymapTranslator = field(init=False)
+    input_trans_stack: list[input.KeymapTranslator] = field(default_factory=list)
+    screen: list[str] = field(default_factory=list)
+    screeninfo: list[tuple[int, list[int]]] = field(init=False)
+    cxy: tuple[int, int] = field(init=False)
+    lxy: tuple[int, int] = field(init=False)
+    scheduled_commands: list[str] = field(default_factory=list)
+    can_colorize: bool = False
+    threading_hook: Callback | None = None
+
+    ## cached metadata to speed up screen refreshes
+    @dataclass
+    class RefreshCache:
+        in_bracketed_paste: bool = False
+        screen: list[str] = field(default_factory=list)
+        screeninfo: list[tuple[int, list[int]]] = field(init=False)
+        line_end_offsets: list[int] = field(default_factory=list)
+        pos: int = field(init=False)
+        cxy: tuple[int, int] = field(init=False)
+        dimensions: tuple[int, int] = field(init=False)
+        invalidated: bool = False
+
+        def update_cache(self,
+                         reader: Reader,
+                         screen: list[str],
+                         screeninfo: list[tuple[int, list[int]]],
+            ) -> None:
+            self.in_bracketed_paste = reader.in_bracketed_paste
+            self.screen = screen.copy()
+            self.screeninfo = screeninfo.copy()
+            self.pos = reader.pos
+            self.cxy = reader.cxy
+            self.dimensions = reader.console.width, reader.console.height
+            self.invalidated = False
+
+        def valid(self, reader: Reader) -> bool:
+            if self.invalidated:
+                return False
+            dimensions = reader.console.width, reader.console.height
+            dimensions_changed = dimensions != self.dimensions
+            paste_changed = reader.in_bracketed_paste != self.in_bracketed_paste
+            return not (dimensions_changed or paste_changed)
+
+        def get_cached_location(self, reader: Reader) -> tuple[int, int]:
+            if self.invalidated:
+                raise ValueError("Cache is invalidated")
+            offset = 0
+            earliest_common_pos = min(reader.pos, self.pos)
+            num_common_lines = len(self.line_end_offsets)
+            while num_common_lines > 0:
+                offset = self.line_end_offsets[num_common_lines - 1]
+                if earliest_common_pos > offset:
+                    break
+                num_common_lines -= 1
+            else:
+                offset = 0
+            return offset, num_common_lines
+
+    last_refresh_cache: RefreshCache = field(default_factory=RefreshCache)
+
+    def __post_init__(self) -> None:
+        # Enable the use of `insert` without a `prepare` call - necessary to
+        # facilitate the tab completion hack implemented for
+        # <https://bugs.python.org/issue25660>.
+        self.keymap = self.collect_keymap()
+        self.input_trans = input.KeymapTranslator(
+            self.keymap, invalid_cls="invalid-key", character_cls="self-insert"
+        )
+        self.screeninfo = [(0, [])]
+        self.cxy = self.pos2xy()
+        self.lxy = (self.pos, 0)
+        self.can_colorize = can_colorize()
+
+        self.last_refresh_cache.screeninfo = self.screeninfo
+        self.last_refresh_cache.pos = self.pos
+        self.last_refresh_cache.cxy = self.cxy
+        self.last_refresh_cache.dimensions = (0, 0)
+
+    def collect_keymap(self) -> tuple[tuple[KeySpec, CommandName], ...]:
+        return default_keymap
+
+    def calc_screen(self) -> list[str]:
+        """Translate changes in self.buffer into changes in self.console.screen."""
+        # Since the last call to calc_screen:
+        # screen and screeninfo may differ due to a completion menu being shown
+        # pos and cxy may differ due to edits, cursor movements, or completion menus
+
+        # Lines that are above both the old and new cursor position can't have changed,
+        # unless the terminal has been resized (which might cause reflowing) or we've
+        # entered or left paste mode (which changes prompts, causing reflowing).
+        num_common_lines = 0
+        offset = 0
+        if self.last_refresh_cache.valid(self):
+            offset, num_common_lines = self.last_refresh_cache.get_cached_location(self)
+
+        screen = self.last_refresh_cache.screen
+        del screen[num_common_lines:]
+
+        screeninfo = self.last_refresh_cache.screeninfo
+        del screeninfo[num_common_lines:]
+
+        last_refresh_line_end_offsets = self.last_refresh_cache.line_end_offsets
+        del last_refresh_line_end_offsets[num_common_lines:]
+
+        pos = self.pos
+        pos -= offset
+
+        prompt_from_cache = (offset and self.buffer[offset - 1] != "\n")
+
+        lines = "".join(self.buffer[offset:]).split("\n")
+
+        cursor_found = False
+        lines_beyond_cursor = 0
+        for ln, line in enumerate(lines, num_common_lines):
+            ll = len(line)
+            if 0 <= pos <= ll:
+                self.lxy = pos, ln
+                cursor_found = True
+            elif cursor_found:
+                lines_beyond_cursor += 1
+                if lines_beyond_cursor > self.console.height:
+                    # No need to keep formatting lines.
+                    # The console can't show them.
+                    break
+            if prompt_from_cache:
+                # Only the first line's prompt can come from the cache
+                prompt_from_cache = False
+                prompt = ""
+            else:
+                prompt = self.get_prompt(ln, ll >= pos >= 0)
+            while "\n" in prompt:
+                pre_prompt, _, prompt = prompt.partition("\n")
+                last_refresh_line_end_offsets.append(offset)
+                screen.append(pre_prompt)
+                screeninfo.append((0, []))
+            pos -= ll + 1
+            prompt, lp = self.process_prompt(prompt)
+            l, l2 = disp_str(line)
+            wrapcount = (wlen(l) + lp) // self.console.width
+            if wrapcount == 0:
+                offset += ll + 1  # Takes all of the line plus the newline
+                last_refresh_line_end_offsets.append(offset)
+                screen.append(prompt + l)
+                screeninfo.append((lp, l2))
+            else:
+                i = 0
+                while l:
+                    prelen = lp if i == 0 else 0
+                    index_to_wrap_before = 0
+                    column = 0
+                    for character_width in l2:
+                        if column + character_width >= self.console.width - prelen:
+                            break
+                        index_to_wrap_before += 1
+                        column += character_width
+                    pre = prompt if i == 0 else ""
+                    if len(l) > index_to_wrap_before:
+                        offset += index_to_wrap_before
+                        post = "\\"
+                        after = [1]
+                    else:
+                        offset += index_to_wrap_before + 1  # Takes the newline
+                        post = ""
+                        after = []
+                    last_refresh_line_end_offsets.append(offset)
+                    screen.append(pre + l[:index_to_wrap_before] + post)
+                    screeninfo.append((prelen, l2[:index_to_wrap_before] + after))
+                    l = l[index_to_wrap_before:]
+                    l2 = l2[index_to_wrap_before:]
+                    i += 1
+        self.screeninfo = screeninfo
+        self.cxy = self.pos2xy()
+        if self.msg:
+            for mline in self.msg.split("\n"):
+                screen.append(mline)
+                screeninfo.append((0, []))
+
+        self.last_refresh_cache.update_cache(self, screen, screeninfo)
+        return screen
+
+    @staticmethod
+    def process_prompt(prompt: str) -> tuple[str, int]:
+        """Process the prompt.
+
+        This means calculate the length of the prompt. The character \x01
+        and \x02 are used to bracket ANSI control sequences and need to be
+        excluded from the length calculation.  So also a copy of the prompt
+        is returned with these control characters removed."""
+
+        # The logic below also ignores the length of common escape
+        # sequences if they were not explicitly within \x01...\x02.
+        # They are CSI (or ANSI) sequences  ( ESC [ ... LETTER )
+
+        # wlen from utils already excludes ANSI_ESCAPE_SEQUENCE chars,
+        # which breaks the logic below so we redefine it here.
+        def wlen(s: str) -> int:
+            return sum(str_width(i) for i in s)
+
+        out_prompt = ""
+        l = wlen(prompt)
+        pos = 0
+        while True:
+            s = prompt.find("\x01", pos)
+            if s == -1:
+                break
+            e = prompt.find("\x02", s)
+            if e == -1:
+                break
+            # Found start and end brackets, subtract from string length
+            l = l - (e - s + 1)
+            keep = prompt[pos:s]
+            l -= sum(map(wlen, ANSI_ESCAPE_SEQUENCE.findall(keep)))
+            out_prompt += keep + prompt[s + 1 : e]
+            pos = e + 1
+        keep = prompt[pos:]
+        l -= sum(map(wlen, ANSI_ESCAPE_SEQUENCE.findall(keep)))
+        out_prompt += keep
+        return out_prompt, l
+
+    def bow(self, p: int | None = None) -> int:
+        """Return the 0-based index of the word break preceding p most
+        immediately.
+
+        p defaults to self.pos; word boundaries are determined using
+        self.syntax_table."""
+        if p is None:
+            p = self.pos
+        st = self.syntax_table
+        b = self.buffer
+        p -= 1
+        while p >= 0 and st.get(b[p], SYNTAX_WORD) != SYNTAX_WORD:
+            p -= 1
+        while p >= 0 and st.get(b[p], SYNTAX_WORD) == SYNTAX_WORD:
+            p -= 1
+        return p + 1
+
+    def eow(self, p: int | None = None) -> int:
+        """Return the 0-based index of the word break following p most
+        immediately.
+
+        p defaults to self.pos; word boundaries are determined using
+        self.syntax_table."""
+        if p is None:
+            p = self.pos
+        st = self.syntax_table
+        b = self.buffer
+        while p < len(b) and st.get(b[p], SYNTAX_WORD) != SYNTAX_WORD:
+            p += 1
+        while p < len(b) and st.get(b[p], SYNTAX_WORD) == SYNTAX_WORD:
+            p += 1
+        return p
+
+    def bol(self, p: int | None = None) -> int:
+        """Return the 0-based index of the line break preceding p most
+        immediately.
+
+        p defaults to self.pos."""
+        if p is None:
+            p = self.pos
+        b = self.buffer
+        p -= 1
+        while p >= 0 and b[p] != "\n":
+            p -= 1
+        return p + 1
+
+    def eol(self, p: int | None = None) -> int:
+        """Return the 0-based index of the line break following p most
+        immediately.
+
+        p defaults to self.pos."""
+        if p is None:
+            p = self.pos
+        b = self.buffer
+        while p < len(b) and b[p] != "\n":
+            p += 1
+        return p
+
+    def max_column(self, y: int) -> int:
+        """Return the last x-offset for line y"""
+        return self.screeninfo[y][0] + sum(self.screeninfo[y][1])
+
+    def max_row(self) -> int:
+        return len(self.screeninfo) - 1
+
+    def get_arg(self, default: int = 1) -> int:
+        """Return any prefix argument that the user has supplied,
+        returning `default' if there is None.  Defaults to 1.
+        """
+        if self.arg is None:
+            return default
+        return self.arg
+
+    def get_prompt(self, lineno: int, cursor_on_line: bool) -> str:
+        """Return what should be in the left-hand margin for line
+        `lineno'."""
+        if self.arg is not None and cursor_on_line:
+            prompt = f"(arg: {self.arg}) "
+        elif self.paste_mode and not self.in_bracketed_paste:
+            prompt = "(paste) "
+        elif "\n" in self.buffer:
+            if lineno == 0:
+                prompt = self.ps2
+            elif self.ps4 and lineno == self.buffer.count("\n"):
+                prompt = self.ps4
+            else:
+                prompt = self.ps3
+        else:
+            prompt = self.ps1
+
+        if self.can_colorize:
+            prompt = f"{ANSIColors.BOLD_MAGENTA}{prompt}{ANSIColors.RESET}"
+        return prompt
+
+    def push_input_trans(self, itrans: input.KeymapTranslator) -> None:
+        self.input_trans_stack.append(self.input_trans)
+        self.input_trans = itrans
+
+    def pop_input_trans(self) -> None:
+        self.input_trans = self.input_trans_stack.pop()
+
+    def setpos_from_xy(self, x: int, y: int) -> None:
+        """Set pos according to coordinates x, y"""
+        pos = 0
+        i = 0
+        while i < y:
+            prompt_len, character_widths = self.screeninfo[i]
+            offset = len(character_widths) - character_widths.count(0)
+            in_wrapped_line = prompt_len + sum(character_widths) >= self.console.width
+            if in_wrapped_line:
+                pos += offset - 1  # -1 cause backslash is not in buffer
+            else:
+                pos += offset + 1  # +1 cause newline is in buffer
+            i += 1
+
+        j = 0
+        cur_x = self.screeninfo[i][0]
+        while cur_x < x:
+            if self.screeninfo[i][1][j] == 0:
+                continue
+            cur_x += self.screeninfo[i][1][j]
+            j += 1
+            pos += 1
+
+        self.pos = pos
+
+    def pos2xy(self) -> tuple[int, int]:
+        """Return the x, y coordinates of position 'pos'."""
+        # this *is* incomprehensible, yes.
+        p, y = 0, 0
+        l2: list[int] = []
+        pos = self.pos
+        assert 0 <= pos <= len(self.buffer)
+        if pos == len(self.buffer) and len(self.screeninfo) > 0:
+            y = len(self.screeninfo) - 1
+            p, l2 = self.screeninfo[y]
+            return p + sum(l2) + l2.count(0), y
+
+        for p, l2 in self.screeninfo:
+            l = len(l2) - l2.count(0)
+            in_wrapped_line = p + sum(l2) >= self.console.width
+            offset = l - 1 if in_wrapped_line else l  # need to remove backslash
+            if offset >= pos:
+                break
+
+            if p + sum(l2) >= self.console.width:
+                pos -= l - 1  # -1 cause backslash is not in buffer
+            else:
+                pos -= l + 1  # +1 cause newline is in buffer
+            y += 1
+        return p + sum(l2[:pos]), y
+
+    def insert(self, text: str | list[str]) -> None:
+        """Insert 'text' at the insertion point."""
+        self.buffer[self.pos : self.pos] = list(text)
+        self.pos += len(text)
+        self.dirty = True
+
+    def update_cursor(self) -> None:
+        """Move the cursor to reflect changes in self.pos"""
+        self.cxy = self.pos2xy()
+        self.console.move_cursor(*self.cxy)
+
+    def after_command(self, cmd: Command) -> None:
+        """This function is called to allow post command cleanup."""
+        if getattr(cmd, "kills_digit_arg", True):
+            if self.arg is not None:
+                self.dirty = True
+            self.arg = None
+
+    def prepare(self) -> None:
+        """Get ready to run.  Call restore when finished.  You must not
+        write to the console in between the calls to prepare and
+        restore."""
+        try:
+            self.console.prepare()
+            self.arg = None
+            self.finished = False
+            del self.buffer[:]
+            self.pos = 0
+            self.dirty = True
+            self.last_command = None
+            self.calc_screen()
+        except BaseException:
+            self.restore()
+            raise
+
+        while self.scheduled_commands:
+            cmd = self.scheduled_commands.pop()
+            self.do_cmd((cmd, []))
+
+    def last_command_is(self, cls: type) -> bool:
+        if not self.last_command:
+            return False
+        return issubclass(cls, self.last_command)
+
+    def restore(self) -> None:
+        """Clean up after a run."""
+        self.console.restore()
+
+    @contextmanager
+    def suspend(self) -> SimpleContextManager:
+        """A context manager to delegate to another reader."""
+        prev_state = {f.name: getattr(self, f.name) for f in fields(self)}
+        try:
+            self.restore()
+            yield
+        finally:
+            for arg in ("msg", "ps1", "ps2", "ps3", "ps4", "paste_mode"):
+                setattr(self, arg, prev_state[arg])
+            self.prepare()
+
+    def finish(self) -> None:
+        """Called when a command signals that we're finished."""
+        pass
+
+    def error(self, msg: str = "none") -> None:
+        self.msg = "! " + msg + " "
+        self.dirty = True
+        self.console.beep()
+
+    def update_screen(self) -> None:
+        if self.dirty:
+            self.refresh()
+
+    def refresh(self) -> None:
+        """Recalculate and refresh the screen."""
+        if self.in_bracketed_paste and self.buffer and not self.buffer[-1] == "\n":
+            return
+
+        # this call sets up self.cxy, so call it first.
+        self.screen = self.calc_screen()
+        self.console.refresh(self.screen, self.cxy)
+        self.dirty = False
+
+    def do_cmd(self, cmd: tuple[str, list[str]]) -> None:
+        """`cmd` is a tuple of "event_name" and "event", which in the current
+        implementation is always just the "buffer" which happens to be a list
+        of single-character strings."""
+
+        trace("received command {cmd}", cmd=cmd)
+        if isinstance(cmd[0], str):
+            command_type = self.commands.get(cmd[0], commands.invalid_command)
+        elif isinstance(cmd[0], type):
+            command_type = cmd[0]
+        else:
+            return  # nothing to do
+
+        command = command_type(self, *cmd)  # type: ignore[arg-type]
+        command.do()
+
+        self.after_command(command)
+
+        if self.dirty:
+            self.refresh()
+        else:
+            self.update_cursor()
+
+        if not isinstance(cmd, commands.digit_arg):
+            self.last_command = command_type
+
+        self.finished = bool(command.finish)
+        if self.finished:
+            self.console.finish()
+            self.finish()
+
+    def run_hooks(self) -> None:
+        threading_hook = self.threading_hook
+        if threading_hook is None and 'threading' in sys.modules:
+            from ._threading_handler import install_threading_hook
+            install_threading_hook(self)
+        if threading_hook is not None:
+            try:
+                threading_hook()
+            except Exception:
+                pass
+
+        input_hook = self.console.input_hook
+        if input_hook:
+            try:
+                input_hook()
+            except Exception:
+                pass
+
+    def handle1(self, block: bool = True) -> bool:
+        """Handle a single event.  Wait as long as it takes if block
+        is true (the default), otherwise return False if no event is
+        pending."""
+
+        if self.msg:
+            self.msg = ""
+            self.dirty = True
+
+        while True:
+            # We use the same timeout as in readline.c: 100ms
+            self.run_hooks()
+            self.console.wait(100)
+            event = self.console.get_event(block=False)
+            if not event:
+                if block:
+                    continue
+                return False
+
+            translate = True
+
+            if event.evt == "key":
+                self.input_trans.push(event)
+            elif event.evt == "scroll":
+                self.refresh()
+            elif event.evt == "resize":
+                self.refresh()
+            else:
+                translate = False
+
+            if translate:
+                cmd = self.input_trans.get()
+            else:
+                cmd = [event.evt, event.data]
+
+            if cmd is None:
+                if block:
+                    continue
+                return False
+
+            self.do_cmd(cmd)
+            return True
+
+    def push_char(self, char: int | bytes) -> None:
+        self.console.push_char(char)
+        self.handle1(block=False)
+
+    def readline(self, startup_hook: Callback | None = None) -> str:
+        """Read a line.  The implementation of this method also shows
+        how to drive Reader if you want more control over the event
+        loop."""
+        self.prepare()
+        try:
+            if startup_hook is not None:
+                startup_hook()
+            self.refresh()
+            while not self.finished:
+                self.handle1()
+            return self.get_unicode()
+
+        finally:
+            self.restore()
+
+    def bind(self, spec: KeySpec, command: CommandName) -> None:
+        self.keymap = self.keymap + ((spec, command),)
+        self.input_trans = input.KeymapTranslator(
+            self.keymap, invalid_cls="invalid-key", character_cls="self-insert"
+        )
+
+    def get_unicode(self) -> str:
+        """Return the current buffer as a unicode string."""
+        return "".join(self.buffer)

Lib/_pyrepl/readline.py

@@ -0,0 +1,598 @@
+#   Copyright 2000-2010 Michael Hudson-Doyle <micahel@gmail.com>
+#                       Alex Gaynor
+#                       Antonio Cuni
+#                       Armin Rigo
+#                       Holger Krekel
+#
+#                        All Rights Reserved
+#
+#
+# Permission to use, copy, modify, and distribute this software and
+# its documentation for any purpose is hereby granted without fee,
+# provided that the above copyright notice appear in all copies and
+# that both that copyright notice and this permission notice appear in
+# supporting documentation.
+#
+# THE AUTHOR MICHAEL HUDSON DISCLAIMS ALL WARRANTIES WITH REGARD TO
+# THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+# AND FITNESS, IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL,
+# INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER
+# RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
+# CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
+# CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+
+"""A compatibility wrapper reimplementing the 'readline' standard module
+on top of pyrepl.  Not all functionalities are supported.  Contains
+extensions for multiline input.
+"""
+
+from __future__ import annotations
+
+import warnings
+from dataclasses import dataclass, field
+
+import os
+from site import gethistoryfile   # type: ignore[attr-defined]
+import sys
+from rlcompleter import Completer as RLCompleter
+
+from . import commands, historical_reader
+from .completing_reader import CompletingReader
+from .console import Console as ConsoleType
+
+Console: type[ConsoleType]
+_error: tuple[type[Exception], ...] | type[Exception]
+try:
+    from .unix_console import UnixConsole as Console, _error
+except ImportError:
+    from .windows_console import WindowsConsole as Console, _error
+
+ENCODING = sys.getdefaultencoding() or "latin1"
+
+
+# types
+Command = commands.Command
+from collections.abc import Callable, Collection
+from .types import Callback, Completer, KeySpec, CommandName
+
+TYPE_CHECKING = False
+
+if TYPE_CHECKING:
+    from typing import Any, Mapping
+
+
+MoreLinesCallable = Callable[[str], bool]
+
+
+__all__ = [
+    "add_history",
+    "clear_history",
+    "get_begidx",
+    "get_completer",
+    "get_completer_delims",
+    "get_current_history_length",
+    "get_endidx",
+    "get_history_item",
+    "get_history_length",
+    "get_line_buffer",
+    "insert_text",
+    "parse_and_bind",
+    "read_history_file",
+    # "read_init_file",
+    # "redisplay",
+    "remove_history_item",
+    "replace_history_item",
+    "set_auto_history",
+    "set_completer",
+    "set_completer_delims",
+    "set_history_length",
+    # "set_pre_input_hook",
+    "set_startup_hook",
+    "write_history_file",
+    # ---- multiline extensions ----
+    "multiline_input",
+]
+
+# ____________________________________________________________
+
+@dataclass
+class ReadlineConfig:
+    readline_completer: Completer | None = None
+    completer_delims: frozenset[str] = frozenset(" \t\n`~!@#$%^&*()-=+[{]}\\|;:'\",<>/?")
+
+
+@dataclass(kw_only=True)
+class ReadlineAlikeReader(historical_reader.HistoricalReader, CompletingReader):
+    # Class fields
+    assume_immutable_completions = False
+    use_brackets = False
+    sort_in_column = True
+
+    # Instance fields
+    config: ReadlineConfig
+    more_lines: MoreLinesCallable | None = None
+    last_used_indentation: str | None = None
+
+    def __post_init__(self) -> None:
+        super().__post_init__()
+        self.commands["maybe_accept"] = maybe_accept
+        self.commands["maybe-accept"] = maybe_accept
+        self.commands["backspace_dedent"] = backspace_dedent
+        self.commands["backspace-dedent"] = backspace_dedent
+
+    def error(self, msg: str = "none") -> None:
+        pass  # don't show error messages by default
+
+    def get_stem(self) -> str:
+        b = self.buffer
+        p = self.pos - 1
+        completer_delims = self.config.completer_delims
+        while p >= 0 and b[p] not in completer_delims:
+            p -= 1
+        return "".join(b[p + 1 : self.pos])
+
+    def get_completions(self, stem: str) -> list[str]:
+        if len(stem) == 0 and self.more_lines is not None:
+            b = self.buffer
+            p = self.pos
+            while p > 0 and b[p - 1] != "\n":
+                p -= 1
+            num_spaces = 4 - ((self.pos - p) % 4)
+            return [" " * num_spaces]
+        result = []
+        function = self.config.readline_completer
+        if function is not None:
+            try:
+                stem = str(stem)  # rlcompleter.py seems to not like unicode
+            except UnicodeEncodeError:
+                pass  # but feed unicode anyway if we have no choice
+            state = 0
+            while True:
+                try:
+                    next = function(stem, state)
+                except Exception:
+                    break
+                if not isinstance(next, str):
+                    break
+                result.append(next)
+                state += 1
+            # emulate the behavior of the standard readline that sorts
+            # the completions before displaying them.
+            result.sort()
+        return result
+
+    def get_trimmed_history(self, maxlength: int) -> list[str]:
+        if maxlength >= 0:
+            cut = len(self.history) - maxlength
+            if cut < 0:
+                cut = 0
+        else:
+            cut = 0
+        return self.history[cut:]
+
+    def update_last_used_indentation(self) -> None:
+        indentation = _get_first_indentation(self.buffer)
+        if indentation is not None:
+            self.last_used_indentation = indentation
+
+    # --- simplified support for reading multiline Python statements ---
+
+    def collect_keymap(self) -> tuple[tuple[KeySpec, CommandName], ...]:
+        return super().collect_keymap() + (
+            (r"\n", "maybe-accept"),
+            (r"\<backspace>", "backspace-dedent"),
+        )
+
+    def after_command(self, cmd: Command) -> None:
+        super().after_command(cmd)
+        if self.more_lines is None:
+            # Force single-line input if we are in raw_input() mode.
+            # Although there is no direct way to add a \n in this mode,
+            # multiline buffers can still show up using various
+            # commands, e.g. navigating the history.
+            try:
+                index = self.buffer.index("\n")
+            except ValueError:
+                pass
+            else:
+                self.buffer = self.buffer[:index]
+                if self.pos > len(self.buffer):
+                    self.pos = len(self.buffer)
+
+
+def set_auto_history(_should_auto_add_history: bool) -> None:
+    """Enable or disable automatic history"""
+    historical_reader.should_auto_add_history = bool(_should_auto_add_history)
+
+
+def _get_this_line_indent(buffer: list[str], pos: int) -> int:
+    indent = 0
+    while pos > 0 and buffer[pos - 1] in " \t":
+        indent += 1
+        pos -= 1
+    if pos > 0 and buffer[pos - 1] == "\n":
+        return indent
+    return 0
+
+
+def _get_previous_line_indent(buffer: list[str], pos: int) -> tuple[int, int | None]:
+    prevlinestart = pos
+    while prevlinestart > 0 and buffer[prevlinestart - 1] != "\n":
+        prevlinestart -= 1
+    prevlinetext = prevlinestart
+    while prevlinetext < pos and buffer[prevlinetext] in " \t":
+        prevlinetext += 1
+    if prevlinetext == pos:
+        indent = None
+    else:
+        indent = prevlinetext - prevlinestart
+    return prevlinestart, indent
+
+
+def _get_first_indentation(buffer: list[str]) -> str | None:
+    indented_line_start = None
+    for i in range(len(buffer)):
+        if (i < len(buffer) - 1
+            and buffer[i] == "\n"
+            and buffer[i + 1] in " \t"
+        ):
+            indented_line_start = i + 1
+        elif indented_line_start is not None and buffer[i] not in " \t\n":
+            return ''.join(buffer[indented_line_start : i])
+    return None
+
+
+def _should_auto_indent(buffer: list[str], pos: int) -> bool:
+    # check if last character before "pos" is a colon, ignoring
+    # whitespaces and comments.
+    last_char = None
+    while pos > 0:
+        pos -= 1
+        if last_char is None:
+            if buffer[pos] not in " \t\n#":  # ignore whitespaces and comments
+                last_char = buffer[pos]
+        else:
+            # even if we found a non-whitespace character before
+            # original pos, we keep going back until newline is reached
+            # to make sure we ignore comments
+            if buffer[pos] == "\n":
+                break
+            if buffer[pos] == "#":
+                last_char = None
+    return last_char == ":"
+
+
+class maybe_accept(commands.Command):
+    def do(self) -> None:
+        r: ReadlineAlikeReader
+        r = self.reader  # type: ignore[assignment]
+        r.dirty = True  # this is needed to hide the completion menu, if visible
+
+        if self.reader.in_bracketed_paste:
+            r.insert("\n")
+            return
+
+        # if there are already several lines and the cursor
+        # is not on the last one, always insert a new \n.
+        text = r.get_unicode()
+
+        if "\n" in r.buffer[r.pos :] or (
+            r.more_lines is not None and r.more_lines(text)
+        ):
+            def _newline_before_pos():
+                before_idx = r.pos - 1
+                while before_idx > 0 and text[before_idx].isspace():
+                    before_idx -= 1
+                return text[before_idx : r.pos].count("\n") > 0
+
+            # if there's already a new line before the cursor then
+            # even if the cursor is followed by whitespace, we assume
+            # the user is trying to terminate the block
+            if _newline_before_pos() and text[r.pos:].isspace():
+                self.finish = True
+                return
+
+            # auto-indent the next line like the previous line
+            prevlinestart, indent = _get_previous_line_indent(r.buffer, r.pos)
+            r.insert("\n")
+            if not self.reader.paste_mode:
+                if indent:
+                    for i in range(prevlinestart, prevlinestart + indent):
+                        r.insert(r.buffer[i])
+                r.update_last_used_indentation()
+                if _should_auto_indent(r.buffer, r.pos):
+                    if r.last_used_indentation is not None:
+                        indentation = r.last_used_indentation
+                    else:
+                        # default
+                        indentation = " " * 4
+                    r.insert(indentation)
+        elif not self.reader.paste_mode:
+            self.finish = True
+        else:
+            r.insert("\n")
+
+
+class backspace_dedent(commands.Command):
+    def do(self) -> None:
+        r = self.reader
+        b = r.buffer
+        if r.pos > 0:
+            repeat = 1
+            if b[r.pos - 1] != "\n":
+                indent = _get_this_line_indent(b, r.pos)
+                if indent > 0:
+                    ls = r.pos - indent
+                    while ls > 0:
+                        ls, pi = _get_previous_line_indent(b, ls - 1)
+                        if pi is not None and pi < indent:
+                            repeat = indent - pi
+                            break
+            r.pos -= repeat
+            del b[r.pos : r.pos + repeat]
+            r.dirty = True
+        else:
+            self.reader.error("can't backspace at start")
+
+
+# ____________________________________________________________
+
+
+@dataclass(slots=True)
+class _ReadlineWrapper:
+    f_in: int = -1
+    f_out: int = -1
+    reader: ReadlineAlikeReader | None = field(default=None, repr=False)
+    saved_history_length: int = -1
+    startup_hook: Callback | None = None
+    config: ReadlineConfig = field(default_factory=ReadlineConfig, repr=False)
+
+    def __post_init__(self) -> None:
+        if self.f_in == -1:
+            self.f_in = os.dup(0)
+        if self.f_out == -1:
+            self.f_out = os.dup(1)
+
+    def get_reader(self) -> ReadlineAlikeReader:
+        if self.reader is None:
+            console = Console(self.f_in, self.f_out, encoding=ENCODING)
+            self.reader = ReadlineAlikeReader(console=console, config=self.config)
+        return self.reader
+
+    def input(self, prompt: object = "") -> str:
+        try:
+            reader = self.get_reader()
+        except _error:
+            assert raw_input is not None
+            return raw_input(prompt)
+        prompt_str = str(prompt)
+        reader.ps1 = prompt_str
+        sys.audit("builtins.input", prompt_str)
+        result = reader.readline(startup_hook=self.startup_hook)
+        sys.audit("builtins.input/result", result)
+        return result
+
+    def multiline_input(self, more_lines: MoreLinesCallable, ps1: str, ps2: str) -> str:
+        """Read an input on possibly multiple lines, asking for more
+        lines as long as 'more_lines(unicodetext)' returns an object whose
+        boolean value is true.
+        """
+        reader = self.get_reader()
+        saved = reader.more_lines
+        try:
+            reader.more_lines = more_lines
+            reader.ps1 = ps1
+            reader.ps2 = ps1
+            reader.ps3 = ps2
+            reader.ps4 = ""
+            with warnings.catch_warnings(action="ignore"):
+                return reader.readline()
+        finally:
+            reader.more_lines = saved
+            reader.paste_mode = False
+
+    def parse_and_bind(self, string: str) -> None:
+        pass  # XXX we don't support parsing GNU-readline-style init files
+
+    def set_completer(self, function: Completer | None = None) -> None:
+        self.config.readline_completer = function
+
+    def get_completer(self) -> Completer | None:
+        return self.config.readline_completer
+
+    def set_completer_delims(self, delimiters: Collection[str]) -> None:
+        self.config.completer_delims = frozenset(delimiters)
+
+    def get_completer_delims(self) -> str:
+        return "".join(sorted(self.config.completer_delims))
+
+    def _histline(self, line: str) -> str:
+        line = line.rstrip("\n")
+        return line
+
+    def get_history_length(self) -> int:
+        return self.saved_history_length
+
+    def set_history_length(self, length: int) -> None:
+        self.saved_history_length = length
+
+    def get_current_history_length(self) -> int:
+        return len(self.get_reader().history)
+
+    def read_history_file(self, filename: str = gethistoryfile()) -> None:
+        # multiline extension (really a hack) for the end of lines that
+        # are actually continuations inside a single multiline_input()
+        # history item: we use \r\n instead of just \n.  If the history
+        # file is passed to GNU readline, the extra \r are just ignored.
+        history = self.get_reader().history
+
+        with open(os.path.expanduser(filename), 'rb') as f:
+            is_editline = f.readline().startswith(b"_HiStOrY_V2_")
+            if is_editline:
+                encoding = "unicode-escape"
+            else:
+                f.seek(0)
+                encoding = "utf-8"
+
+            lines = [line.decode(encoding, errors='replace') for line in f.read().split(b'\n')]
+            buffer = []
+            for line in lines:
+                if line.endswith("\r"):
+                    buffer.append(line+'\n')
+                else:
+                    line = self._histline(line)
+                    if buffer:
+                        line = self._histline("".join(buffer).replace("\r", "") + line)
+                        del buffer[:]
+                    if line:
+                        history.append(line)
+
+    def write_history_file(self, filename: str = gethistoryfile()) -> None:
+        maxlength = self.saved_history_length
+        history = self.get_reader().get_trimmed_history(maxlength)
+        f = open(os.path.expanduser(filename), "w",
+                 encoding="utf-8", newline="\n")
+        with f:
+            for entry in history:
+                entry = entry.replace("\n", "\r\n")  # multiline history support
+                f.write(entry + "\n")
+
+    def clear_history(self) -> None:
+        del self.get_reader().history[:]
+
+    def get_history_item(self, index: int) -> str | None:
+        history = self.get_reader().history
+        if 1 <= index <= len(history):
+            return history[index - 1]
+        else:
+            return None  # like readline.c
+
+    def remove_history_item(self, index: int) -> None:
+        history = self.get_reader().history
+        if 0 <= index < len(history):
+            del history[index]
+        else:
+            raise ValueError("No history item at position %d" % index)
+            # like readline.c
+
+    def replace_history_item(self, index: int, line: str) -> None:
+        history = self.get_reader().history
+        if 0 <= index < len(history):
+            history[index] = self._histline(line)
+        else:
+            raise ValueError("No history item at position %d" % index)
+            # like readline.c
+
+    def add_history(self, line: str) -> None:
+        self.get_reader().history.append(self._histline(line))
+
+    def set_startup_hook(self, function: Callback | None = None) -> None:
+        self.startup_hook = function
+
+    def get_line_buffer(self) -> str:
+        return self.get_reader().get_unicode()
+
+    def _get_idxs(self) -> tuple[int, int]:
+        start = cursor = self.get_reader().pos
+        buf = self.get_line_buffer()
+        for i in range(cursor - 1, -1, -1):
+            if buf[i] in self.get_completer_delims():
+                break
+            start = i
+        return start, cursor
+
+    def get_begidx(self) -> int:
+        return self._get_idxs()[0]
+
+    def get_endidx(self) -> int:
+        return self._get_idxs()[1]
+
+    def insert_text(self, text: str) -> None:
+        self.get_reader().insert(text)
+
+
+_wrapper = _ReadlineWrapper()
+
+# ____________________________________________________________
+# Public API
+
+parse_and_bind = _wrapper.parse_and_bind
+set_completer = _wrapper.set_completer
+get_completer = _wrapper.get_completer
+set_completer_delims = _wrapper.set_completer_delims
+get_completer_delims = _wrapper.get_completer_delims
+get_history_length = _wrapper.get_history_length
+set_history_length = _wrapper.set_history_length
+get_current_history_length = _wrapper.get_current_history_length
+read_history_file = _wrapper.read_history_file
+write_history_file = _wrapper.write_history_file
+clear_history = _wrapper.clear_history
+get_history_item = _wrapper.get_history_item
+remove_history_item = _wrapper.remove_history_item
+replace_history_item = _wrapper.replace_history_item
+add_history = _wrapper.add_history
+set_startup_hook = _wrapper.set_startup_hook
+get_line_buffer = _wrapper.get_line_buffer
+get_begidx = _wrapper.get_begidx
+get_endidx = _wrapper.get_endidx
+insert_text = _wrapper.insert_text
+
+# Extension
+multiline_input = _wrapper.multiline_input
+
+# Internal hook
+_get_reader = _wrapper.get_reader
+
+# ____________________________________________________________
+# Stubs
+
+
+def _make_stub(_name: str, _ret: object) -> None:
+    def stub(*args: object, **kwds: object) -> None:
+        import warnings
+
+        warnings.warn("readline.%s() not implemented" % _name, stacklevel=2)
+
+    stub.__name__ = _name
+    globals()[_name] = stub
+
+
+for _name, _ret in [
+    ("read_init_file", None),
+    ("redisplay", None),
+    ("set_pre_input_hook", None),
+]:
+    assert _name not in globals(), _name
+    _make_stub(_name, _ret)
+
+# ____________________________________________________________
+
+
+def _setup(namespace: Mapping[str, Any]) -> None:
+    global raw_input
+    if raw_input is not None:
+        return  # don't run _setup twice
+
+    try:
+        f_in = sys.stdin.fileno()
+        f_out = sys.stdout.fileno()
+    except (AttributeError, ValueError):
+        return
+    if not os.isatty(f_in) or not os.isatty(f_out):
+        return
+
+    _wrapper.f_in = f_in
+    _wrapper.f_out = f_out
+
+    # set up namespace in rlcompleter, which requires it to be a bona fide dict
+    if not isinstance(namespace, dict):
+        namespace = dict(namespace)
+    _wrapper.config.readline_completer = RLCompleter(namespace).complete
+
+    # this is not really what readline.c does.  Better than nothing I guess
+    import builtins
+    raw_input = builtins.input
+    builtins.input = _wrapper.input
+
+
+raw_input: Callable[[object], str] | None = None

Lib/_pyrepl/simple_interact.py

@@ -0,0 +1,167 @@
+#   Copyright 2000-2010 Michael Hudson-Doyle <micahel@gmail.com>
+#                       Armin Rigo
+#
+#                        All Rights Reserved
+#
+#
+# Permission to use, copy, modify, and distribute this software and
+# its documentation for any purpose is hereby granted without fee,
+# provided that the above copyright notice appear in all copies and
+# that both that copyright notice and this permission notice appear in
+# supporting documentation.
+#
+# THE AUTHOR MICHAEL HUDSON DISCLAIMS ALL WARRANTIES WITH REGARD TO
+# THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+# AND FITNESS, IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL,
+# INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER
+# RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
+# CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
+# CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+
+"""This is an alternative to python_reader which tries to emulate
+the CPython prompt as closely as possible, with the exception of
+allowing multiline input and multiline history entries.
+"""
+
+from __future__ import annotations
+
+import _sitebuiltins
+import linecache
+import functools
+import os
+import sys
+import code
+
+from .readline import _get_reader, multiline_input
+
+TYPE_CHECKING = False
+
+if TYPE_CHECKING:
+    from typing import Any
+
+
+_error: tuple[type[Exception], ...] | type[Exception]
+try:
+    from .unix_console import _error
+except ModuleNotFoundError:
+    from .windows_console import _error
+
+def check() -> str:
+    """Returns the error message if there is a problem initializing the state."""
+    try:
+        _get_reader()
+    except _error as e:
+        if term := os.environ.get("TERM", ""):
+            term = f"; TERM={term}"
+        return str(str(e) or repr(e) or "unknown error") + term
+    return ""
+
+
+def _strip_final_indent(text: str) -> str:
+    # kill spaces and tabs at the end, but only if they follow '\n'.
+    # meant to remove the auto-indentation only (although it would of
+    # course also remove explicitly-added indentation).
+    short = text.rstrip(" \t")
+    n = len(short)
+    if n > 0 and text[n - 1] == "\n":
+        return short
+    return text
+
+
+def _clear_screen():
+    reader = _get_reader()
+    reader.scheduled_commands.append("clear_screen")
+
+
+REPL_COMMANDS = {
+    "exit": _sitebuiltins.Quitter('exit', ''),
+    "quit": _sitebuiltins.Quitter('quit' ,''),
+    "copyright": _sitebuiltins._Printer('copyright', sys.copyright),
+    "help": _sitebuiltins._Helper(),
+    "clear": _clear_screen,
+    "\x1a": _sitebuiltins.Quitter('\x1a', ''),
+}
+
+
+def _more_lines(console: code.InteractiveConsole, unicodetext: str) -> bool:
+    # ooh, look at the hack:
+    src = _strip_final_indent(unicodetext)
+    try:
+        code = console.compile(src, "<stdin>", "single")
+    except (OverflowError, SyntaxError, ValueError):
+        lines = src.splitlines(keepends=True)
+        if len(lines) == 1:
+            return False
+
+        last_line = lines[-1]
+        was_indented = last_line.startswith((" ", "\t"))
+        not_empty = last_line.strip() != ""
+        incomplete = not last_line.endswith("\n")
+        return (was_indented or not_empty) and incomplete
+    else:
+        return code is None
+
+
+def run_multiline_interactive_console(
+    console: code.InteractiveConsole,
+    *,
+    future_flags: int = 0,
+) -> None:
+    from .readline import _setup
+    _setup(console.locals)
+    if future_flags:
+        console.compile.compiler.flags |= future_flags
+
+    more_lines = functools.partial(_more_lines, console)
+    input_n = 0
+
+    def maybe_run_command(statement: str) -> bool:
+        statement = statement.strip()
+        if statement in console.locals or statement not in REPL_COMMANDS:
+            return False
+
+        reader = _get_reader()
+        reader.history.pop()  # skip internal commands in history
+        command = REPL_COMMANDS[statement]
+        if callable(command):
+            # Make sure that history does not change because of commands
+            with reader.suspend_history():
+                command()
+            return True
+        return False
+
+    while 1:
+        try:
+            try:
+                sys.stdout.flush()
+            except Exception:
+                pass
+
+            ps1 = getattr(sys, "ps1", ">>> ")
+            ps2 = getattr(sys, "ps2", "... ")
+            try:
+                statement = multiline_input(more_lines, ps1, ps2)
+            except EOFError:
+                break
+
+            if maybe_run_command(statement):
+                continue
+
+            input_name = f"<python-input-{input_n}>"
+            linecache._register_code(input_name, statement, "<stdin>")  # type: ignore[attr-defined]
+            more = console.push(_strip_final_indent(statement), filename=input_name, _symbol="single")  # type: ignore[call-arg]
+            assert not more
+            input_n += 1
+        except KeyboardInterrupt:
+            r = _get_reader()
+            if r.input_trans is r.isearch_trans:
+                r.do_cmd(("isearch-end", [""]))
+            r.pos = len(r.get_unicode())
+            r.dirty = True
+            r.refresh()
+            r.in_bracketed_paste = False
+            console.write("\nKeyboardInterrupt\n")
+            console.resetbuffer()
+        except MemoryError:
+            console.write("\nMemoryError\n")
+            console.resetbuffer()

Lib/_pyrepl/trace.py

@@ -0,0 +1,21 @@
+from __future__ import annotations
+
+import os
+
+# types
+if False:
+    from typing import IO
+
+
+trace_file: IO[str] | None = None
+if trace_filename := os.environ.get("PYREPL_TRACE"):
+    trace_file = open(trace_filename, "a")
+
+
+def trace(line: str, *k: object, **kw: object) -> None:
+    if trace_file is None:
+        return
+    if k or kw:
+        line = line.format(*k, **kw)
+    trace_file.write(line + "\n")
+    trace_file.flush()

Lib/_pyrepl/types.py

@@ -0,0 +1,8 @@
+from collections.abc import Callable, Iterator
+
+Callback = Callable[[], object]
+SimpleContextManager = Iterator[None]
+KeySpec = str  # like r"\C-c"
+CommandName = str  # like "interrupt"
+EventTuple = tuple[CommandName, str]
+Completer = Callable[[str, int], str | None]

Lib/_pyrepl/unix_console.py

@@ -0,0 +1,810 @@
+#   Copyright 2000-2010 Michael Hudson-Doyle <micahel@gmail.com>
+#                       Antonio Cuni
+#                       Armin Rigo
+#
+#                        All Rights Reserved
+#
+#
+# Permission to use, copy, modify, and distribute this software and
+# its documentation for any purpose is hereby granted without fee,
+# provided that the above copyright notice appear in all copies and
+# that both that copyright notice and this permission notice appear in
+# supporting documentation.
+#
+# THE AUTHOR MICHAEL HUDSON DISCLAIMS ALL WARRANTIES WITH REGARD TO
+# THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+# AND FITNESS, IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL,
+# INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER
+# RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
+# CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
+# CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+
+from __future__ import annotations
+
+import errno
+import os
+import re
+import select
+import signal
+import struct
+import termios
+import time
+import platform
+from fcntl import ioctl
+
+from . import curses
+from .console import Console, Event
+from .fancy_termios import tcgetattr, tcsetattr
+from .trace import trace
+from .unix_eventqueue import EventQueue
+from .utils import wlen
+
+
+TYPE_CHECKING = False
+
+# types
+if TYPE_CHECKING:
+    from typing import IO, Literal, overload
+else:
+    overload = lambda func: None
+
+
+class InvalidTerminal(RuntimeError):
+    pass
+
+
+_error = (termios.error, curses.error, InvalidTerminal)
+
+SIGWINCH_EVENT = "repaint"
+
+FIONREAD = getattr(termios, "FIONREAD", None)
+TIOCGWINSZ = getattr(termios, "TIOCGWINSZ", None)
+
+# ------------ start of baudrate definitions ------------
+
+# Add (possibly) missing baudrates (check termios man page) to termios
+
+
+def add_baudrate_if_supported(dictionary: dict[int, int], rate: int) -> None:
+    baudrate_name = "B%d" % rate
+    if hasattr(termios, baudrate_name):
+        dictionary[getattr(termios, baudrate_name)] = rate
+
+
+# Check the termios man page (Line speed) to know where these
+# values come from.
+potential_baudrates = [
+    0,
+    110,
+    115200,
+    1200,
+    134,
+    150,
+    1800,
+    19200,
+    200,
+    230400,
+    2400,
+    300,
+    38400,
+    460800,
+    4800,
+    50,
+    57600,
+    600,
+    75,
+    9600,
+]
+
+ratedict: dict[int, int] = {}
+for rate in potential_baudrates:
+    add_baudrate_if_supported(ratedict, rate)
+
+# Clean up variables to avoid unintended usage
+del rate, add_baudrate_if_supported
+
+# ------------ end of baudrate definitions ------------
+
+delayprog = re.compile(b"\\$<([0-9]+)((?:/|\\*){0,2})>")
+
+try:
+    poll: type[select.poll] = select.poll
+except AttributeError:
+    # this is exactly the minumum necessary to support what we
+    # do with poll objects
+    class MinimalPoll:
+        def __init__(self):
+            pass
+
+        def register(self, fd, flag):
+            self.fd = fd
+        # note: The 'timeout' argument is received as *milliseconds*
+        def poll(self, timeout: float | None = None) -> list[int]:
+            if timeout is None:
+                r, w, e = select.select([self.fd], [], [])
+            else:
+                r, w, e = select.select([self.fd], [], [], timeout/1000)
+            return r
+
+    poll = MinimalPoll  # type: ignore[assignment]
+
+
+class UnixConsole(Console):
+    def __init__(
+        self,
+        f_in: IO[bytes] | int = 0,
+        f_out: IO[bytes] | int = 1,
+        term: str = "",
+        encoding: str = "",
+    ):
+        """
+        Initialize the UnixConsole.
+
+        Parameters:
+        - f_in (int or file-like object): Input file descriptor or object.
+        - f_out (int or file-like object): Output file descriptor or object.
+        - term (str): Terminal name.
+        - encoding (str): Encoding to use for I/O operations.
+        """
+        super().__init__(f_in, f_out, term, encoding)
+
+        self.pollob = poll()
+        self.pollob.register(self.input_fd, select.POLLIN)
+        self.input_buffer = b""
+        self.input_buffer_pos = 0
+        curses.setupterm(term or None, self.output_fd)
+        self.term = term
+
+        @overload
+        def _my_getstr(cap: str, optional: Literal[False] = False) -> bytes: ...
+
+        @overload
+        def _my_getstr(cap: str, optional: bool) -> bytes | None: ...
+
+        def _my_getstr(cap: str, optional: bool = False) -> bytes | None:
+            r = curses.tigetstr(cap)
+            if not optional and r is None:
+                raise InvalidTerminal(
+                    f"terminal doesn't have the required {cap} capability"
+                )
+            return r
+
+        self._bel = _my_getstr("bel")
+        self._civis = _my_getstr("civis", optional=True)
+        self._clear = _my_getstr("clear")
+        self._cnorm = _my_getstr("cnorm", optional=True)
+        self._cub = _my_getstr("cub", optional=True)
+        self._cub1 = _my_getstr("cub1", optional=True)
+        self._cud = _my_getstr("cud", optional=True)
+        self._cud1 = _my_getstr("cud1", optional=True)
+        self._cuf = _my_getstr("cuf", optional=True)
+        self._cuf1 = _my_getstr("cuf1", optional=True)
+        self._cup = _my_getstr("cup")
+        self._cuu = _my_getstr("cuu", optional=True)
+        self._cuu1 = _my_getstr("cuu1", optional=True)
+        self._dch1 = _my_getstr("dch1", optional=True)
+        self._dch = _my_getstr("dch", optional=True)
+        self._el = _my_getstr("el")
+        self._hpa = _my_getstr("hpa", optional=True)
+        self._ich = _my_getstr("ich", optional=True)
+        self._ich1 = _my_getstr("ich1", optional=True)
+        self._ind = _my_getstr("ind", optional=True)
+        self._pad = _my_getstr("pad", optional=True)
+        self._ri = _my_getstr("ri", optional=True)
+        self._rmkx = _my_getstr("rmkx", optional=True)
+        self._smkx = _my_getstr("smkx", optional=True)
+
+        self.__setup_movement()
+
+        self.event_queue = EventQueue(self.input_fd, self.encoding)
+        self.cursor_visible = 1
+
+    def more_in_buffer(self) -> bool:
+        return bool(
+            self.input_buffer
+            and self.input_buffer_pos < len(self.input_buffer)
+        )
+
+    def __read(self, n: int) -> bytes:
+        if not self.more_in_buffer():
+            self.input_buffer = os.read(self.input_fd, 10000)
+
+        ret = self.input_buffer[self.input_buffer_pos : self.input_buffer_pos + n]
+        self.input_buffer_pos += len(ret)
+        if self.input_buffer_pos >= len(self.input_buffer):
+            self.input_buffer = b""
+            self.input_buffer_pos = 0
+        return ret
+
+
+    def change_encoding(self, encoding: str) -> None:
+        """
+        Change the encoding used for I/O operations.
+
+        Parameters:
+        - encoding (str): New encoding to use.
+        """
+        self.encoding = encoding
+
+    def refresh(self, screen, c_xy):
+        """
+        Refresh the console screen.
+
+        Parameters:
+        - screen (list): List of strings representing the screen contents.
+        - c_xy (tuple): Cursor position (x, y) on the screen.
+        """
+        cx, cy = c_xy
+        if not self.__gone_tall:
+            while len(self.screen) < min(len(screen), self.height):
+                self.__hide_cursor()
+                self.__move(0, len(self.screen) - 1)
+                self.__write("\n")
+                self.posxy = 0, len(self.screen)
+                self.screen.append("")
+        else:
+            while len(self.screen) < len(screen):
+                self.screen.append("")
+
+        if len(screen) > self.height:
+            self.__gone_tall = 1
+            self.__move = self.__move_tall
+
+        px, py = self.posxy
+        old_offset = offset = self.__offset
+        height = self.height
+
+        # we make sure the cursor is on the screen, and that we're
+        # using all of the screen if we can
+        if cy < offset:
+            offset = cy
+        elif cy >= offset + height:
+            offset = cy - height + 1
+        elif offset > 0 and len(screen) < offset + height:
+            offset = max(len(screen) - height, 0)
+            screen.append("")
+
+        oldscr = self.screen[old_offset : old_offset + height]
+        newscr = screen[offset : offset + height]
+
+        # use hardware scrolling if we have it.
+        if old_offset > offset and self._ri:
+            self.__hide_cursor()
+            self.__write_code(self._cup, 0, 0)
+            self.posxy = 0, old_offset
+            for i in range(old_offset - offset):
+                self.__write_code(self._ri)
+                oldscr.pop(-1)
+                oldscr.insert(0, "")
+        elif old_offset < offset and self._ind:
+            self.__hide_cursor()
+            self.__write_code(self._cup, self.height - 1, 0)
+            self.posxy = 0, old_offset + self.height - 1
+            for i in range(offset - old_offset):
+                self.__write_code(self._ind)
+                oldscr.pop(0)
+                oldscr.append("")
+
+        self.__offset = offset
+
+        for (
+            y,
+            oldline,
+            newline,
+        ) in zip(range(offset, offset + height), oldscr, newscr):
+            if oldline != newline:
+                self.__write_changed_line(y, oldline, newline, px)
+
+        y = len(newscr)
+        while y < len(oldscr):
+            self.__hide_cursor()
+            self.__move(0, y)
+            self.posxy = 0, y
+            self.__write_code(self._el)
+            y += 1
+
+        self.__show_cursor()
+
+        self.screen = screen.copy()
+        self.move_cursor(cx, cy)
+        self.flushoutput()
+
+    def move_cursor(self, x, y):
+        """
+        Move the cursor to the specified position on the screen.
+
+        Parameters:
+        - x (int): X coordinate.
+        - y (int): Y coordinate.
+        """
+        if y < self.__offset or y >= self.__offset + self.height:
+            self.event_queue.insert(Event("scroll", None))
+        else:
+            self.__move(x, y)
+            self.posxy = x, y
+            self.flushoutput()
+
+    def prepare(self):
+        """
+        Prepare the console for input/output operations.
+        """
+        self.__svtermstate = tcgetattr(self.input_fd)
+        raw = self.__svtermstate.copy()
+        raw.iflag &= ~(termios.INPCK | termios.ISTRIP | termios.IXON)
+        raw.oflag &= ~(termios.OPOST)
+        raw.cflag &= ~(termios.CSIZE | termios.PARENB)
+        raw.cflag |= termios.CS8
+        raw.iflag |= termios.BRKINT
+        raw.lflag &= ~(termios.ICANON | termios.ECHO | termios.IEXTEN)
+        raw.lflag |= termios.ISIG
+        raw.cc[termios.VMIN] = 1
+        raw.cc[termios.VTIME] = 0
+        tcsetattr(self.input_fd, termios.TCSADRAIN, raw)
+
+        # In macOS terminal we need to deactivate line wrap via ANSI escape code
+        if platform.system() == "Darwin" and os.getenv("TERM_PROGRAM") == "Apple_Terminal":
+            os.write(self.output_fd, b"\033[?7l")
+
+        self.screen = []
+        self.height, self.width = self.getheightwidth()
+
+        self.__buffer = []
+
+        self.posxy = 0, 0
+        self.__gone_tall = 0
+        self.__move = self.__move_short
+        self.__offset = 0
+
+        self.__maybe_write_code(self._smkx)
+
+        try:
+            self.old_sigwinch = signal.signal(signal.SIGWINCH, self.__sigwinch)
+        except ValueError:
+            pass
+
+        self.__enable_bracketed_paste()
+
+    def restore(self):
+        """
+        Restore the console to the default state
+        """
+        self.__disable_bracketed_paste()
+        self.__maybe_write_code(self._rmkx)
+        self.flushoutput()
+        tcsetattr(self.input_fd, termios.TCSADRAIN, self.__svtermstate)
+
+        if platform.system() == "Darwin" and os.getenv("TERM_PROGRAM") == "Apple_Terminal":
+            os.write(self.output_fd, b"\033[?7h")
+
+        if hasattr(self, "old_sigwinch"):
+            signal.signal(signal.SIGWINCH, self.old_sigwinch)
+            del self.old_sigwinch
+
+    def push_char(self, char: int | bytes) -> None:
+        """
+        Push a character to the console event queue.
+        """
+        trace("push char {char!r}", char=char)
+        self.event_queue.push(char)
+
+    def get_event(self, block: bool = True) -> Event | None:
+        """
+        Get an event from the console event queue.
+
+        Parameters:
+        - block (bool): Whether to block until an event is available.
+
+        Returns:
+        - Event: Event object from the event queue.
+        """
+        if not block and not self.wait(timeout=0):
+            return None
+
+        while self.event_queue.empty():
+            while True:
+                try:
+                    self.push_char(self.__read(1))
+                except OSError as err:
+                    if err.errno == errno.EINTR:
+                        if not self.event_queue.empty():
+                            return self.event_queue.get()
+                        else:
+                            continue
+                    else:
+                        raise
+                else:
+                    break
+        return self.event_queue.get()
+
+    def wait(self, timeout: float | None = None) -> bool:
+        """
+        Wait for events on the console.
+        """
+        return (
+            not self.event_queue.empty()
+            or self.more_in_buffer()
+            or bool(self.pollob.poll(timeout))
+        )
+
+    def set_cursor_vis(self, visible):
+        """
+        Set the visibility of the cursor.
+
+        Parameters:
+        - visible (bool): Visibility flag.
+        """
+        if visible:
+            self.__show_cursor()
+        else:
+            self.__hide_cursor()
+
+    if TIOCGWINSZ:
+
+        def getheightwidth(self):
+            """
+            Get the height and width of the console.
+
+            Returns:
+            - tuple: Height and width of the console.
+            """
+            try:
+                return int(os.environ["LINES"]), int(os.environ["COLUMNS"])
+            except (KeyError, TypeError, ValueError):
+                try:
+                    size = ioctl(self.input_fd, TIOCGWINSZ, b"\000" * 8)
+                except OSError:
+                    return 25, 80
+                height, width = struct.unpack("hhhh", size)[0:2]
+                if not height:
+                    return 25, 80
+                return height, width
+
+    else:
+
+        def getheightwidth(self):
+            """
+            Get the height and width of the console.
+
+            Returns:
+            - tuple: Height and width of the console.
+            """
+            try:
+                return int(os.environ["LINES"]), int(os.environ["COLUMNS"])
+            except (KeyError, TypeError, ValueError):
+                return 25, 80
+
+    def forgetinput(self):
+        """
+        Discard any pending input on the console.
+        """
+        termios.tcflush(self.input_fd, termios.TCIFLUSH)
+
+    def flushoutput(self):
+        """
+        Flush the output buffer.
+        """
+        for text, iscode in self.__buffer:
+            if iscode:
+                self.__tputs(text)
+            else:
+                os.write(self.output_fd, text.encode(self.encoding, "replace"))
+        del self.__buffer[:]
+
+    def finish(self):
+        """
+        Finish console operations and flush the output buffer.
+        """
+        y = len(self.screen) - 1
+        while y >= 0 and not self.screen[y]:
+            y -= 1
+        self.__move(0, min(y, self.height + self.__offset - 1))
+        self.__write("\n\r")
+        self.flushoutput()
+
+    def beep(self):
+        """
+        Emit a beep sound.
+        """
+        self.__maybe_write_code(self._bel)
+        self.flushoutput()
+
+    if FIONREAD:
+
+        def getpending(self):
+            """
+            Get pending events from the console event queue.
+
+            Returns:
+            - Event: Pending event from the event queue.
+            """
+            e = Event("key", "", b"")
+
+            while not self.event_queue.empty():
+                e2 = self.event_queue.get()
+                e.data += e2.data
+                e.raw += e.raw
+
+            amount = struct.unpack("i", ioctl(self.input_fd, FIONREAD, b"\0\0\0\0"))[0]
+            raw = self.__read(amount)
+            data = str(raw, self.encoding, "replace")
+            e.data += data
+            e.raw += raw
+            return e
+
+    else:
+
+        def getpending(self):
+            """
+            Get pending events from the console event queue.
+
+            Returns:
+            - Event: Pending event from the event queue.
+            """
+            e = Event("key", "", b"")
+
+            while not self.event_queue.empty():
+                e2 = self.event_queue.get()
+                e.data += e2.data
+                e.raw += e.raw
+
+            amount = 10000
+            raw = self.__read(amount)
+            data = str(raw, self.encoding, "replace")
+            e.data += data
+            e.raw += raw
+            return e
+
+    def clear(self):
+        """
+        Clear the console screen.
+        """
+        self.__write_code(self._clear)
+        self.__gone_tall = 1
+        self.__move = self.__move_tall
+        self.posxy = 0, 0
+        self.screen = []
+
+    @property
+    def input_hook(self):
+        try:
+            import posix
+        except ImportError:
+            return None
+        if posix._is_inputhook_installed():
+            return posix._inputhook
+
+    def __enable_bracketed_paste(self) -> None:
+        os.write(self.output_fd, b"\x1b[?2004h")
+
+    def __disable_bracketed_paste(self) -> None:
+        os.write(self.output_fd, b"\x1b[?2004l")
+
+    def __setup_movement(self):
+        """
+        Set up the movement functions based on the terminal capabilities.
+        """
+        if 0 and self._hpa:  # hpa don't work in windows telnet :-(
+            self.__move_x = self.__move_x_hpa
+        elif self._cub and self._cuf:
+            self.__move_x = self.__move_x_cub_cuf
+        elif self._cub1 and self._cuf1:
+            self.__move_x = self.__move_x_cub1_cuf1
+        else:
+            raise RuntimeError("insufficient terminal (horizontal)")
+
+        if self._cuu and self._cud:
+            self.__move_y = self.__move_y_cuu_cud
+        elif self._cuu1 and self._cud1:
+            self.__move_y = self.__move_y_cuu1_cud1
+        else:
+            raise RuntimeError("insufficient terminal (vertical)")
+
+        if self._dch1:
+            self.dch1 = self._dch1
+        elif self._dch:
+            self.dch1 = curses.tparm(self._dch, 1)
+        else:
+            self.dch1 = None
+
+        if self._ich1:
+            self.ich1 = self._ich1
+        elif self._ich:
+            self.ich1 = curses.tparm(self._ich, 1)
+        else:
+            self.ich1 = None
+
+        self.__move = self.__move_short
+
+    def __write_changed_line(self, y, oldline, newline, px_coord):
+        # this is frustrating; there's no reason to test (say)
+        # self.dch1 inside the loop -- but alternative ways of
+        # structuring this function are equally painful (I'm trying to
+        # avoid writing code generators these days...)
+        minlen = min(wlen(oldline), wlen(newline))
+        x_pos = 0
+        x_coord = 0
+
+        px_pos = 0
+        j = 0
+        for c in oldline:
+            if j >= px_coord:
+                break
+            j += wlen(c)
+            px_pos += 1
+
+        # reuse the oldline as much as possible, but stop as soon as we
+        # encounter an ESCAPE, because it might be the start of an escape
+        # sequene
+        while (
+            x_coord < minlen
+            and oldline[x_pos] == newline[x_pos]
+            and newline[x_pos] != "\x1b"
+        ):
+            x_coord += wlen(newline[x_pos])
+            x_pos += 1
+
+        # if we need to insert a single character right after the first detected change
+        if oldline[x_pos:] == newline[x_pos + 1 :] and self.ich1:
+            if (
+                y == self.posxy[1]
+                and x_coord > self.posxy[0]
+                and oldline[px_pos:x_pos] == newline[px_pos + 1 : x_pos + 1]
+            ):
+                x_pos = px_pos
+                x_coord = px_coord
+            character_width = wlen(newline[x_pos])
+            self.__move(x_coord, y)
+            self.__write_code(self.ich1)
+            self.__write(newline[x_pos])
+            self.posxy = x_coord + character_width, y
+
+        # if it's a single character change in the middle of the line
+        elif (
+            x_coord < minlen
+            and oldline[x_pos + 1 :] == newline[x_pos + 1 :]
+            and wlen(oldline[x_pos]) == wlen(newline[x_pos])
+        ):
+            character_width = wlen(newline[x_pos])
+            self.__move(x_coord, y)
+            self.__write(newline[x_pos])
+            self.posxy = x_coord + character_width, y
+
+        # if this is the last character to fit in the line and we edit in the middle of the line
+        elif (
+            self.dch1
+            and self.ich1
+            and wlen(newline) == self.width
+            and x_coord < wlen(newline) - 2
+            and newline[x_pos + 1 : -1] == oldline[x_pos:-2]
+        ):
+            self.__hide_cursor()
+            self.__move(self.width - 2, y)
+            self.posxy = self.width - 2, y
+            self.__write_code(self.dch1)
+
+            character_width = wlen(newline[x_pos])
+            self.__move(x_coord, y)
+            self.__write_code(self.ich1)
+            self.__write(newline[x_pos])
+            self.posxy = character_width + 1, y
+
+        else:
+            self.__hide_cursor()
+            self.__move(x_coord, y)
+            if wlen(oldline) > wlen(newline):
+                self.__write_code(self._el)
+            self.__write(newline[x_pos:])
+            self.posxy = wlen(newline), y
+
+        if "\x1b" in newline:
+            # ANSI escape characters are present, so we can't assume
+            # anything about the position of the cursor.  Moving the cursor
+            # to the left margin should work to get to a known position.
+            self.move_cursor(0, y)
+
+    def __write(self, text):
+        self.__buffer.append((text, 0))
+
+    def __write_code(self, fmt, *args):
+        self.__buffer.append((curses.tparm(fmt, *args), 1))
+
+    def __maybe_write_code(self, fmt, *args):
+        if fmt:
+            self.__write_code(fmt, *args)
+
+    def __move_y_cuu1_cud1(self, y):
+        assert self._cud1 is not None
+        assert self._cuu1 is not None
+        dy = y - self.posxy[1]
+        if dy > 0:
+            self.__write_code(dy * self._cud1)
+        elif dy < 0:
+            self.__write_code((-dy) * self._cuu1)
+
+    def __move_y_cuu_cud(self, y):
+        dy = y - self.posxy[1]
+        if dy > 0:
+            self.__write_code(self._cud, dy)
+        elif dy < 0:
+            self.__write_code(self._cuu, -dy)
+
+    def __move_x_hpa(self, x: int) -> None:
+        if x != self.posxy[0]:
+            self.__write_code(self._hpa, x)
+
+    def __move_x_cub1_cuf1(self, x: int) -> None:
+        assert self._cuf1 is not None
+        assert self._cub1 is not None
+        dx = x - self.posxy[0]
+        if dx > 0:
+            self.__write_code(self._cuf1 * dx)
+        elif dx < 0:
+            self.__write_code(self._cub1 * (-dx))
+
+    def __move_x_cub_cuf(self, x: int) -> None:
+        dx = x - self.posxy[0]
+        if dx > 0:
+            self.__write_code(self._cuf, dx)
+        elif dx < 0:
+            self.__write_code(self._cub, -dx)
+
+    def __move_short(self, x, y):
+        self.__move_x(x)
+        self.__move_y(y)
+
+    def __move_tall(self, x, y):
+        assert 0 <= y - self.__offset < self.height, y - self.__offset
+        self.__write_code(self._cup, y - self.__offset, x)
+
+    def __sigwinch(self, signum, frame):
+        self.height, self.width = self.getheightwidth()
+        self.event_queue.insert(Event("resize", None))
+
+    def __hide_cursor(self):
+        if self.cursor_visible:
+            self.__maybe_write_code(self._civis)
+            self.cursor_visible = 0
+
+    def __show_cursor(self):
+        if not self.cursor_visible:
+            self.__maybe_write_code(self._cnorm)
+            self.cursor_visible = 1
+
+    def repaint(self):
+        if not self.__gone_tall:
+            self.posxy = 0, self.posxy[1]
+            self.__write("\r")
+            ns = len(self.screen) * ["\000" * self.width]
+            self.screen = ns
+        else:
+            self.posxy = 0, self.__offset
+            self.__move(0, self.__offset)
+            ns = self.height * ["\000" * self.width]
+            self.screen = ns
+
+    def __tputs(self, fmt, prog=delayprog):
+        """A Python implementation of the curses tputs function; the
+        curses one can't really be wrapped in a sane manner.
+
+        I have the strong suspicion that this is complexity that
+        will never do anyone any good."""
+        # using .get() means that things will blow up
+        # only if the bps is actually needed (which I'm
+        # betting is pretty unlkely)
+        bps = ratedict.get(self.__svtermstate.ospeed)
+        while 1:
+            m = prog.search(fmt)
+            if not m:
+                os.write(self.output_fd, fmt)
+                break
+            x, y = m.span()
+            os.write(self.output_fd, fmt[:x])
+            fmt = fmt[y:]
+            delay = int(m.group(1))
+            if b"*" in m.group(2):
+                delay *= self.height
+            if self._pad and bps is not None:
+                nchars = (bps * delay) / 1000
+                os.write(self.output_fd, self._pad * nchars)
+            else:
+                time.sleep(float(delay) / 1000.0)

Lib/_pyrepl/unix_eventqueue.py

@@ -0,0 +1,152 @@
+#   Copyright 2000-2008 Michael Hudson-Doyle <micahel@gmail.com>
+#                       Armin Rigo
+#
+#                        All Rights Reserved
+#
+#
+# Permission to use, copy, modify, and distribute this software and
+# its documentation for any purpose is hereby granted without fee,
+# provided that the above copyright notice appear in all copies and
+# that both that copyright notice and this permission notice appear in
+# supporting documentation.
+#
+# THE AUTHOR MICHAEL HUDSON DISCLAIMS ALL WARRANTIES WITH REGARD TO
+# THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+# AND FITNESS, IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL,
+# INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER
+# RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
+# CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
+# CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+
+from collections import deque
+
+from . import keymap
+from .console import Event
+from . import curses
+from .trace import trace
+from termios import tcgetattr, VERASE
+import os
+
+
+# Mapping of human-readable key names to their terminal-specific codes
+TERMINAL_KEYNAMES = {
+    "delete": "kdch1",
+    "down": "kcud1",
+    "end": "kend",
+    "enter": "kent",
+    "home": "khome",
+    "insert": "kich1",
+    "left": "kcub1",
+    "page down": "knp",
+    "page up": "kpp",
+    "right": "kcuf1",
+    "up": "kcuu1",
+}
+
+
+# Function keys F1-F20 mapping
+TERMINAL_KEYNAMES.update(("f%d" % i, "kf%d" % i) for i in range(1, 21))
+
+# Known CTRL-arrow keycodes
+CTRL_ARROW_KEYCODES= {
+    # for xterm, gnome-terminal, xfce terminal, etc.
+    b'\033[1;5D': 'ctrl left',
+    b'\033[1;5C': 'ctrl right',
+    # for rxvt
+    b'\033Od': 'ctrl left',
+    b'\033Oc': 'ctrl right',
+}
+
+def get_terminal_keycodes() -> dict[bytes, str]:
+    """
+    Generates a dictionary mapping terminal keycodes to human-readable names.
+    """
+    keycodes = {}
+    for key, terminal_code in TERMINAL_KEYNAMES.items():
+        keycode = curses.tigetstr(terminal_code)
+        trace('key {key} tiname {terminal_code} keycode {keycode!r}', **locals())
+        if keycode:
+            keycodes[keycode] = key
+    keycodes.update(CTRL_ARROW_KEYCODES)
+    return keycodes
+
+class EventQueue:
+    def __init__(self, fd: int, encoding: str) -> None:
+        self.keycodes = get_terminal_keycodes()
+        if os.isatty(fd):
+            backspace = tcgetattr(fd)[6][VERASE]
+            self.keycodes[backspace] = "backspace"
+        self.compiled_keymap = keymap.compile_keymap(self.keycodes)
+        self.keymap = self.compiled_keymap
+        trace("keymap {k!r}", k=self.keymap)
+        self.encoding = encoding
+        self.events: deque[Event] = deque()
+        self.buf = bytearray()
+
+    def get(self) -> Event | None:
+        """
+        Retrieves the next event from the queue.
+        """
+        if self.events:
+            return self.events.popleft()
+        else:
+            return None
+
+    def empty(self) -> bool:
+        """
+        Checks if the queue is empty.
+        """
+        return not self.events
+
+    def flush_buf(self) -> bytearray:
+        """
+        Flushes the buffer and returns its contents.
+        """
+        old = self.buf
+        self.buf = bytearray()
+        return old
+
+    def insert(self, event: Event) -> None:
+        """
+        Inserts an event into the queue.
+        """
+        trace('added event {event}', event=event)
+        self.events.append(event)
+
+    def push(self, char: int | bytes) -> None:
+        """
+        Processes a character by updating the buffer and handling special key mappings.
+        """
+        ord_char = char if isinstance(char, int) else ord(char)
+        char = bytes(bytearray((ord_char,)))
+        self.buf.append(ord_char)
+        if char in self.keymap:
+            if self.keymap is self.compiled_keymap:
+                #sanity check, buffer is empty when a special key comes
+                assert len(self.buf) == 1
+            k = self.keymap[char]
+            trace('found map {k!r}', k=k)
+            if isinstance(k, dict):
+                self.keymap = k
+            else:
+                self.insert(Event('key', k, self.flush_buf()))
+                self.keymap = self.compiled_keymap
+
+        elif self.buf and self.buf[0] == 27:  # escape
+            # escape sequence not recognized by our keymap: propagate it
+            # outside so that i can be recognized as an M-... key (see also
+            # the docstring in keymap.py
+            trace('unrecognized escape sequence, propagating...')
+            self.keymap = self.compiled_keymap
+            self.insert(Event('key', '\033', bytearray(b'\033')))
+            for _c in self.flush_buf()[1:]:
+                self.push(_c)
+
+        else:
+            try:
+                decoded = bytes(self.buf).decode(self.encoding)
+            except UnicodeError:
+                return
+            else:
+                self.insert(Event('key', decoded, self.flush_buf()))
+            self.keymap = self.compiled_keymap

Lib/_pyrepl/utils.py

@@ -0,0 +1,25 @@
+import re
+import unicodedata
+import functools
+
+ANSI_ESCAPE_SEQUENCE = re.compile(r"\x1b\[[ -@]*[A-~]")
+
+
+@functools.cache
+def str_width(c: str) -> int:
+    if ord(c) < 128:
+        return 1
+    w = unicodedata.east_asian_width(c)
+    if w in ('N', 'Na', 'H', 'A'):
+        return 1
+    return 2
+
+
+def wlen(s: str) -> int:
+    if len(s) == 1 and s != '\x1a':
+        return str_width(s)
+    length = sum(str_width(i) for i in s)
+    # remove lengths of any escape sequences
+    sequence = ANSI_ESCAPE_SEQUENCE.findall(s)
+    ctrl_z_cnt = s.count('\x1a')
+    return length - sum(len(i) for i in sequence) + ctrl_z_cnt

Lib/_pyrepl/windows_console.py

@@ -0,0 +1,618 @@
+#   Copyright 2000-2004 Michael Hudson-Doyle <micahel@gmail.com>
+#
+#                        All Rights Reserved
+#
+#
+# Permission to use, copy, modify, and distribute this software and
+# its documentation for any purpose is hereby granted without fee,
+# provided that the above copyright notice appear in all copies and
+# that both that copyright notice and this permission notice appear in
+# supporting documentation.
+#
+# THE AUTHOR MICHAEL HUDSON DISCLAIMS ALL WARRANTIES WITH REGARD TO
+# THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+# AND FITNESS, IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL,
+# INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER
+# RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
+# CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
+# CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+
+from __future__ import annotations
+
+import io
+import os
+import sys
+import time
+import msvcrt
+
+from collections import deque
+import ctypes
+from ctypes.wintypes import (
+    _COORD,
+    WORD,
+    SMALL_RECT,
+    BOOL,
+    HANDLE,
+    CHAR,
+    DWORD,
+    WCHAR,
+    SHORT,
+)
+from ctypes import Structure, POINTER, Union
+from .console import Event, Console
+from .trace import trace
+from .utils import wlen
+
+try:
+    from ctypes import GetLastError, WinDLL, windll, WinError  # type: ignore[attr-defined]
+except:
+    # Keep MyPy happy off Windows
+    from ctypes import CDLL as WinDLL, cdll as windll
+
+    def GetLastError() -> int:
+        return 42
+
+    class WinError(OSError):  # type: ignore[no-redef]
+        def __init__(self, err: int | None, descr: str | None = None) -> None:
+            self.err = err
+            self.descr = descr
+
+
+TYPE_CHECKING = False
+
+if TYPE_CHECKING:
+    from typing import IO
+
+# Virtual-Key Codes: https://learn.microsoft.com/en-us/windows/win32/inputdev/virtual-key-codes
+VK_MAP: dict[int, str] = {
+    0x23: "end",  # VK_END
+    0x24: "home",  # VK_HOME
+    0x25: "left",  # VK_LEFT
+    0x26: "up",  # VK_UP
+    0x27: "right",  # VK_RIGHT
+    0x28: "down",  # VK_DOWN
+    0x2E: "delete",  # VK_DELETE
+    0x70: "f1",  # VK_F1
+    0x71: "f2",  # VK_F2
+    0x72: "f3",  # VK_F3
+    0x73: "f4",  # VK_F4
+    0x74: "f5",  # VK_F5
+    0x75: "f6",  # VK_F6
+    0x76: "f7",  # VK_F7
+    0x77: "f8",  # VK_F8
+    0x78: "f9",  # VK_F9
+    0x79: "f10",  # VK_F10
+    0x7A: "f11",  # VK_F11
+    0x7B: "f12",  # VK_F12
+    0x7C: "f13",  # VK_F13
+    0x7D: "f14",  # VK_F14
+    0x7E: "f15",  # VK_F15
+    0x7F: "f16",  # VK_F16
+    0x80: "f17",  # VK_F17
+    0x81: "f18",  # VK_F18
+    0x82: "f19",  # VK_F19
+    0x83: "f20",  # VK_F20
+}
+
+# Console escape codes: https://learn.microsoft.com/en-us/windows/console/console-virtual-terminal-sequences
+ERASE_IN_LINE = "\x1b[K"
+MOVE_LEFT = "\x1b[{}D"
+MOVE_RIGHT = "\x1b[{}C"
+MOVE_UP = "\x1b[{}A"
+MOVE_DOWN = "\x1b[{}B"
+CLEAR = "\x1b[H\x1b[J"
+
+
+class _error(Exception):
+    pass
+
+
+class WindowsConsole(Console):
+    def __init__(
+        self,
+        f_in: IO[bytes] | int = 0,
+        f_out: IO[bytes] | int = 1,
+        term: str = "",
+        encoding: str = "",
+    ):
+        super().__init__(f_in, f_out, term, encoding)
+
+        SetConsoleMode(
+            OutHandle,
+            ENABLE_WRAP_AT_EOL_OUTPUT
+            | ENABLE_PROCESSED_OUTPUT
+            | ENABLE_VIRTUAL_TERMINAL_PROCESSING,
+        )
+        self.screen: list[str] = []
+        self.width = 80
+        self.height = 25
+        self.__offset = 0
+        self.event_queue: deque[Event] = deque()
+        try:
+            self.out = io._WindowsConsoleIO(self.output_fd, "w")  # type: ignore[attr-defined]
+        except ValueError:
+            # Console I/O is redirected, fallback...
+            self.out = None
+
+    def refresh(self, screen: list[str], c_xy: tuple[int, int]) -> None:
+        """
+        Refresh the console screen.
+
+        Parameters:
+        - screen (list): List of strings representing the screen contents.
+        - c_xy (tuple): Cursor position (x, y) on the screen.
+        """
+        cx, cy = c_xy
+
+        while len(self.screen) < min(len(screen), self.height):
+            self._hide_cursor()
+            self._move_relative(0, len(self.screen) - 1)
+            self.__write("\n")
+            self.posxy = 0, len(self.screen)
+            self.screen.append("")
+
+        px, py = self.posxy
+        old_offset = offset = self.__offset
+        height = self.height
+
+        # we make sure the cursor is on the screen, and that we're
+        # using all of the screen if we can
+        if cy < offset:
+            offset = cy
+        elif cy >= offset + height:
+            offset = cy - height + 1
+            scroll_lines = offset - old_offset
+
+            # Scrolling the buffer as the current input is greater than the visible
+            # portion of the window.  We need to scroll the visible portion and the
+            # entire history
+            self._scroll(scroll_lines, self._getscrollbacksize())
+            self.posxy = self.posxy[0], self.posxy[1] + scroll_lines
+            self.__offset += scroll_lines
+
+            for i in range(scroll_lines):
+                self.screen.append("")
+        elif offset > 0 and len(screen) < offset + height:
+            offset = max(len(screen) - height, 0)
+            screen.append("")
+
+        oldscr = self.screen[old_offset : old_offset + height]
+        newscr = screen[offset : offset + height]
+
+        self.__offset = offset
+
+        self._hide_cursor()
+        for (
+            y,
+            oldline,
+            newline,
+        ) in zip(range(offset, offset + height), oldscr, newscr):
+            if oldline != newline:
+                self.__write_changed_line(y, oldline, newline, px)
+
+        y = len(newscr)
+        while y < len(oldscr):
+            self._move_relative(0, y)
+            self.posxy = 0, y
+            self._erase_to_end()
+            y += 1
+
+        self._show_cursor()
+
+        self.screen = screen
+        self.move_cursor(cx, cy)
+
+    @property
+    def input_hook(self):
+        try:
+            import nt
+        except ImportError:
+            return None
+        if nt._is_inputhook_installed():
+            return nt._inputhook
+
+    def __write_changed_line(
+        self, y: int, oldline: str, newline: str, px_coord: int
+    ) -> None:
+        # this is frustrating; there's no reason to test (say)
+        # self.dch1 inside the loop -- but alternative ways of
+        # structuring this function are equally painful (I'm trying to
+        # avoid writing code generators these days...)
+        minlen = min(wlen(oldline), wlen(newline))
+        x_pos = 0
+        x_coord = 0
+
+        px_pos = 0
+        j = 0
+        for c in oldline:
+            if j >= px_coord:
+                break
+            j += wlen(c)
+            px_pos += 1
+
+        # reuse the oldline as much as possible, but stop as soon as we
+        # encounter an ESCAPE, because it might be the start of an escape
+        # sequene
+        while (
+            x_coord < minlen
+            and oldline[x_pos] == newline[x_pos]
+            and newline[x_pos] != "\x1b"
+        ):
+            x_coord += wlen(newline[x_pos])
+            x_pos += 1
+
+        self._hide_cursor()
+        self._move_relative(x_coord, y)
+        if wlen(oldline) > wlen(newline):
+            self._erase_to_end()
+
+        self.__write(newline[x_pos:])
+        if wlen(newline) == self.width:
+            # If we wrapped we want to start at the next line
+            self._move_relative(0, y + 1)
+            self.posxy = 0, y + 1
+        else:
+            self.posxy = wlen(newline), y
+
+            if "\x1b" in newline or y != self.posxy[1] or '\x1a' in newline:
+                # ANSI escape characters are present, so we can't assume
+                # anything about the position of the cursor.  Moving the cursor
+                # to the left margin should work to get to a known position.
+                self.move_cursor(0, y)
+
+    def _scroll(
+        self, top: int, bottom: int, left: int | None = None, right: int | None = None
+    ) -> None:
+        scroll_rect = SMALL_RECT()
+        scroll_rect.Top = SHORT(top)
+        scroll_rect.Bottom = SHORT(bottom)
+        scroll_rect.Left = SHORT(0 if left is None else left)
+        scroll_rect.Right = SHORT(
+            self.getheightwidth()[1] - 1 if right is None else right
+        )
+        destination_origin = _COORD()
+        fill_info = CHAR_INFO()
+        fill_info.UnicodeChar = " "
+
+        if not ScrollConsoleScreenBuffer(
+            OutHandle, scroll_rect, None, destination_origin, fill_info
+        ):
+            raise WinError(GetLastError())
+
+    def _hide_cursor(self):
+        self.__write("\x1b[?25l")
+
+    def _show_cursor(self):
+        self.__write("\x1b[?25h")
+
+    def _enable_blinking(self):
+        self.__write("\x1b[?12h")
+
+    def _disable_blinking(self):
+        self.__write("\x1b[?12l")
+
+    def __write(self, text: str) -> None:
+        if "\x1a" in text:
+            text = ''.join(["^Z" if x == '\x1a' else x for x in text])
+
+        if self.out is not None:
+            self.out.write(text.encode(self.encoding, "replace"))
+            self.out.flush()
+        else:
+            os.write(self.output_fd, text.encode(self.encoding, "replace"))
+
+    @property
+    def screen_xy(self) -> tuple[int, int]:
+        info = CONSOLE_SCREEN_BUFFER_INFO()
+        if not GetConsoleScreenBufferInfo(OutHandle, info):
+            raise WinError(GetLastError())
+        return info.dwCursorPosition.X, info.dwCursorPosition.Y
+
+    def _erase_to_end(self) -> None:
+        self.__write(ERASE_IN_LINE)
+
+    def prepare(self) -> None:
+        trace("prepare")
+        self.screen = []
+        self.height, self.width = self.getheightwidth()
+
+        self.posxy = 0, 0
+        self.__gone_tall = 0
+        self.__offset = 0
+
+    def restore(self) -> None:
+        pass
+
+    def _move_relative(self, x: int, y: int) -> None:
+        """Moves relative to the current posxy"""
+        dx = x - self.posxy[0]
+        dy = y - self.posxy[1]
+        if dx < 0:
+            self.__write(MOVE_LEFT.format(-dx))
+        elif dx > 0:
+            self.__write(MOVE_RIGHT.format(dx))
+
+        if dy < 0:
+            self.__write(MOVE_UP.format(-dy))
+        elif dy > 0:
+            self.__write(MOVE_DOWN.format(dy))
+
+    def move_cursor(self, x: int, y: int) -> None:
+        if x < 0 or y < 0:
+            raise ValueError(f"Bad cursor position {x}, {y}")
+
+        if y < self.__offset or y >= self.__offset + self.height:
+            self.event_queue.insert(0, Event("scroll", ""))
+        else:
+            self._move_relative(x, y)
+            self.posxy = x, y
+
+    def set_cursor_vis(self, visible: bool) -> None:
+        if visible:
+            self._show_cursor()
+        else:
+            self._hide_cursor()
+
+    def getheightwidth(self) -> tuple[int, int]:
+        """Return (height, width) where height and width are the height
+        and width of the terminal window in characters."""
+        info = CONSOLE_SCREEN_BUFFER_INFO()
+        if not GetConsoleScreenBufferInfo(OutHandle, info):
+            raise WinError(GetLastError())
+        return (
+            info.srWindow.Bottom - info.srWindow.Top + 1,
+            info.srWindow.Right - info.srWindow.Left + 1,
+        )
+
+    def _getscrollbacksize(self) -> int:
+        info = CONSOLE_SCREEN_BUFFER_INFO()
+        if not GetConsoleScreenBufferInfo(OutHandle, info):
+            raise WinError(GetLastError())
+
+        return info.srWindow.Bottom  # type: ignore[no-any-return]
+
+    def _read_input(self, block: bool = True) -> INPUT_RECORD | None:
+        if not block:
+            events = DWORD()
+            if not GetNumberOfConsoleInputEvents(InHandle, events):
+                raise WinError(GetLastError())
+            if not events.value:
+                return None
+
+        rec = INPUT_RECORD()
+        read = DWORD()
+        if not ReadConsoleInput(InHandle, rec, 1, read):
+            raise WinError(GetLastError())
+
+        return rec
+
+    def get_event(self, block: bool = True) -> Event | None:
+        """Return an Event instance.  Returns None if |block| is false
+        and there is no event pending, otherwise waits for the
+        completion of an event."""
+        if self.event_queue:
+            return self.event_queue.pop()
+
+        while True:
+            rec = self._read_input(block)
+            if rec is None:
+                return None
+
+            if rec.EventType == WINDOW_BUFFER_SIZE_EVENT:
+                return Event("resize", "")
+
+            if rec.EventType != KEY_EVENT or not rec.Event.KeyEvent.bKeyDown:
+                # Only process keys and keydown events
+                if block:
+                    continue
+                return None
+
+            key = rec.Event.KeyEvent.uChar.UnicodeChar
+
+            if rec.Event.KeyEvent.uChar.UnicodeChar == "\r":
+                # Make enter make unix-like
+                return Event(evt="key", data="\n", raw=b"\n")
+            elif rec.Event.KeyEvent.wVirtualKeyCode == 8:
+                # Turn backspace directly into the command
+                return Event(
+                    evt="key",
+                    data="backspace",
+                    raw=rec.Event.KeyEvent.uChar.UnicodeChar,
+                )
+            elif rec.Event.KeyEvent.uChar.UnicodeChar == "\x00":
+                # Handle special keys like arrow keys and translate them into the appropriate command
+                code = VK_MAP.get(rec.Event.KeyEvent.wVirtualKeyCode)
+                if code:
+                    return Event(
+                        evt="key", data=code, raw=rec.Event.KeyEvent.uChar.UnicodeChar
+                    )
+                if block:
+                    continue
+
+                return None
+
+            return Event(evt="key", data=key, raw=rec.Event.KeyEvent.uChar.UnicodeChar)
+
+    def push_char(self, char: int | bytes) -> None:
+        """
+        Push a character to the console event queue.
+        """
+        raise NotImplementedError("push_char not supported on Windows")
+
+    def beep(self) -> None:
+        self.__write("\x07")
+
+    def clear(self) -> None:
+        """Wipe the screen"""
+        self.__write(CLEAR)
+        self.posxy = 0, 0
+        self.screen = [""]
+
+    def finish(self) -> None:
+        """Move the cursor to the end of the display and otherwise get
+        ready for end.  XXX could be merged with restore?  Hmm."""
+        y = len(self.screen) - 1
+        while y >= 0 and not self.screen[y]:
+            y -= 1
+        self._move_relative(0, min(y, self.height + self.__offset - 1))
+        self.__write("\r\n")
+
+    def flushoutput(self) -> None:
+        """Flush all output to the screen (assuming there's some
+        buffering going on somewhere).
+
+        All output on Windows is unbuffered so this is a nop"""
+        pass
+
+    def forgetinput(self) -> None:
+        """Forget all pending, but not yet processed input."""
+        if not FlushConsoleInputBuffer(InHandle):
+            raise WinError(GetLastError())
+
+    def getpending(self) -> Event:
+        """Return the characters that have been typed but not yet
+        processed."""
+        return Event("key", "", b"")
+
+    def wait(self, timeout: float | None) -> bool:
+        """Wait for an event."""
+        # Poor man's Windows select loop
+        start_time = time.time()
+        while True:
+            if msvcrt.kbhit(): # type: ignore[attr-defined]
+                return True
+            if timeout and time.time() - start_time > timeout / 1000:
+                return False
+            time.sleep(0.01)
+
+    def repaint(self) -> None:
+        raise NotImplementedError("No repaint support")
+
+
+# Windows interop
+class CONSOLE_SCREEN_BUFFER_INFO(Structure):
+    _fields_ = [
+        ("dwSize", _COORD),
+        ("dwCursorPosition", _COORD),
+        ("wAttributes", WORD),
+        ("srWindow", SMALL_RECT),
+        ("dwMaximumWindowSize", _COORD),
+    ]
+
+
+class CONSOLE_CURSOR_INFO(Structure):
+    _fields_ = [
+        ("dwSize", DWORD),
+        ("bVisible", BOOL),
+    ]
+
+
+class CHAR_INFO(Structure):
+    _fields_ = [
+        ("UnicodeChar", WCHAR),
+        ("Attributes", WORD),
+    ]
+
+
+class Char(Union):
+    _fields_ = [
+        ("UnicodeChar", WCHAR),
+        ("Char", CHAR),
+    ]
+
+
+class KeyEvent(ctypes.Structure):
+    _fields_ = [
+        ("bKeyDown", BOOL),
+        ("wRepeatCount", WORD),
+        ("wVirtualKeyCode", WORD),
+        ("wVirtualScanCode", WORD),
+        ("uChar", Char),
+        ("dwControlKeyState", DWORD),
+    ]
+
+
+class WindowsBufferSizeEvent(ctypes.Structure):
+    _fields_ = [("dwSize", _COORD)]
+
+
+class ConsoleEvent(ctypes.Union):
+    _fields_ = [
+        ("KeyEvent", KeyEvent),
+        ("WindowsBufferSizeEvent", WindowsBufferSizeEvent),
+    ]
+
+
+class INPUT_RECORD(Structure):
+    _fields_ = [("EventType", WORD), ("Event", ConsoleEvent)]
+
+
+KEY_EVENT = 0x01
+FOCUS_EVENT = 0x10
+MENU_EVENT = 0x08
+MOUSE_EVENT = 0x02
+WINDOW_BUFFER_SIZE_EVENT = 0x04
+
+ENABLE_PROCESSED_OUTPUT = 0x01
+ENABLE_WRAP_AT_EOL_OUTPUT = 0x02
+ENABLE_VIRTUAL_TERMINAL_PROCESSING = 0x04
+
+STD_INPUT_HANDLE = -10
+STD_OUTPUT_HANDLE = -11
+
+if sys.platform == "win32":
+    _KERNEL32 = WinDLL("kernel32", use_last_error=True)
+
+    GetStdHandle = windll.kernel32.GetStdHandle
+    GetStdHandle.argtypes = [DWORD]
+    GetStdHandle.restype = HANDLE
+
+    GetConsoleScreenBufferInfo = _KERNEL32.GetConsoleScreenBufferInfo
+    GetConsoleScreenBufferInfo.argtypes = [
+        HANDLE,
+        ctypes.POINTER(CONSOLE_SCREEN_BUFFER_INFO),
+    ]
+    GetConsoleScreenBufferInfo.restype = BOOL
+
+    ScrollConsoleScreenBuffer = _KERNEL32.ScrollConsoleScreenBufferW
+    ScrollConsoleScreenBuffer.argtypes = [
+        HANDLE,
+        POINTER(SMALL_RECT),
+        POINTER(SMALL_RECT),
+        _COORD,
+        POINTER(CHAR_INFO),
+    ]
+    ScrollConsoleScreenBuffer.restype = BOOL
+
+    SetConsoleMode = _KERNEL32.SetConsoleMode
+    SetConsoleMode.argtypes = [HANDLE, DWORD]
+    SetConsoleMode.restype = BOOL
+
+    ReadConsoleInput = _KERNEL32.ReadConsoleInputW
+    ReadConsoleInput.argtypes = [HANDLE, POINTER(INPUT_RECORD), DWORD, POINTER(DWORD)]
+    ReadConsoleInput.restype = BOOL
+
+    GetNumberOfConsoleInputEvents = _KERNEL32.GetNumberOfConsoleInputEvents
+    GetNumberOfConsoleInputEvents.argtypes = [HANDLE, POINTER(DWORD)]
+    GetNumberOfConsoleInputEvents.restype = BOOL
+
+    FlushConsoleInputBuffer = _KERNEL32.FlushConsoleInputBuffer
+    FlushConsoleInputBuffer.argtypes = [HANDLE]
+    FlushConsoleInputBuffer.restype = BOOL
+
+    OutHandle = GetStdHandle(STD_OUTPUT_HANDLE)
+    InHandle = GetStdHandle(STD_INPUT_HANDLE)
+else:
+
+    def _win_only(*args, **kwargs):
+        raise NotImplementedError("Windows only")
+
+    GetStdHandle = _win_only
+    GetConsoleScreenBufferInfo = _win_only
+    ScrollConsoleScreenBuffer = _win_only
+    SetConsoleMode = _win_only
+    ReadConsoleInput = _win_only
+    GetNumberOfConsoleInputEvents = _win_only
+    FlushConsoleInputBuffer = _win_only
+    OutHandle = 0
+    InHandle = 0

Lib/_strptime.py

@@ -10,10 +10,13 @@ FUNCTIONS:
     strptime -- Calculates the time struct represented by the passed-in string
 
 """
+import os
 import time
 import locale
 import calendar
+import re
 from re import compile as re_compile
+from re import sub as re_sub
 from re import IGNORECASE
 from re import escape as re_escape
 from datetime import (date as datetime_date,
@@ -27,6 +30,41 @@ def _getlang():
     # Figure out what the current language is set to.
     return locale.getlocale(locale.LC_TIME)
 
+def _findall(haystack, needle):
+    # Find all positions of needle in haystack.
+    if not needle:
+        return
+    i = 0
+    while True:
+        i = haystack.find(needle, i)
+        if i < 0:
+            break
+        yield i
+        i += len(needle)
+
+def _fixmonths(months):
+    yield from months
+    # The lower case of 'İ' ('\u0130') is 'i\u0307'.
+    # The re module only supports 1-to-1 character matching in
+    # case-insensitive mode.
+    for s in months:
+        if 'i\u0307' in s:
+            yield s.replace('i\u0307', '\u0130')
+
+lzh_TW_alt_digits = (
+    # 〇:一:二:三:四:五:六:七:八:九
+    '\u3007', '\u4e00', '\u4e8c', '\u4e09', '\u56db',
+    '\u4e94', '\u516d', '\u4e03', '\u516b', '\u4e5d',
+    # 十:十一:十二:十三:十四:十五:十六:十七:十八:十九
+    '\u5341', '\u5341\u4e00', '\u5341\u4e8c', '\u5341\u4e09', '\u5341\u56db',
+    '\u5341\u4e94', '\u5341\u516d', '\u5341\u4e03', '\u5341\u516b', '\u5341\u4e5d',
+    # 廿:廿一:廿二:廿三:廿四:廿五:廿六:廿七:廿八:廿九
+    '\u5eff', '\u5eff\u4e00', '\u5eff\u4e8c', '\u5eff\u4e09', '\u5eff\u56db',
+    '\u5eff\u4e94', '\u5eff\u516d', '\u5eff\u4e03', '\u5eff\u516b', '\u5eff\u4e5d',
+    # 卅:卅一
+    '\u5345', '\u5345\u4e00')
+
+
 class LocaleTime(object):
     """Stores and handles locale-specific information related to time.
 
@@ -70,6 +108,7 @@ class LocaleTime(object):
         self.__calc_weekday()
         self.__calc_month()
         self.__calc_am_pm()
+        self.__calc_alt_digits()
         self.__calc_timezone()
         self.__calc_date_time()
         if _getlang() != self.lang:
@@ -101,53 +140,184 @@ class LocaleTime(object):
         am_pm = []
         for hour in (1, 22):
             time_tuple = time.struct_time((1999,3,17,hour,44,55,2,76,0))
-            am_pm.append(time.strftime("%p", time_tuple).lower())
+            # br_FR has AM/PM info (' ',' ').
+            am_pm.append(time.strftime("%p", time_tuple).lower().strip())
         self.am_pm = am_pm
 
+    def __calc_alt_digits(self):
+        # Set self.LC_alt_digits by using time.strftime().
+
+        # The magic data should contain all decimal digits.
+        time_tuple = time.struct_time((1998, 1, 27, 10, 43, 56, 1, 27, 0))
+        s = time.strftime("%x%X", time_tuple)
+        if s.isascii():
+            # Fast path -- all digits are ASCII.
+            self.LC_alt_digits = ()
+            return
+
+        digits = ''.join(sorted(set(re.findall(r'\d', s))))
+        if len(digits) == 10 and ord(digits[-1]) == ord(digits[0]) + 9:
+            # All 10 decimal digits from the same set.
+            if digits.isascii():
+                # All digits are ASCII.
+                self.LC_alt_digits = ()
+                return
+
+            self.LC_alt_digits = [a + b for a in digits for b in digits]
+            # Test whether the numbers contain leading zero.
+            time_tuple2 = time.struct_time((2000, 1, 1, 1, 1, 1, 5, 1, 0))
+            if self.LC_alt_digits[1] not in time.strftime("%x %X", time_tuple2):
+                self.LC_alt_digits[:10] = digits
+            return
+
+        # Either non-Gregorian calendar or non-decimal numbers.
+        if {'\u4e00', '\u4e03', '\u4e5d', '\u5341', '\u5eff'}.issubset(s):
+            # lzh_TW
+            self.LC_alt_digits = lzh_TW_alt_digits
+            return
+
+        self.LC_alt_digits = None
+
     def __calc_date_time(self):
-        # Set self.date_time, self.date, & self.time by using
-        # time.strftime().
+        # Set self.LC_date_time, self.LC_date, self.LC_time and
+        # self.LC_time_ampm by using time.strftime().
 
         # Use (1999,3,17,22,44,55,2,76,0) for magic date because the amount of
         # overloaded numbers is minimized.  The order in which searches for
         # values within the format string is very important; it eliminates
         # possible ambiguity for what something represents.
         time_tuple = time.struct_time((1999,3,17,22,44,55,2,76,0))
-        date_time = [None, None, None]
-        date_time[0] = time.strftime("%c", time_tuple).lower()
-        date_time[1] = time.strftime("%x", time_tuple).lower()
-        date_time[2] = time.strftime("%X", time_tuple).lower()
-        replacement_pairs = [('%', '%%'), (self.f_weekday[2], '%A'),
-                    (self.f_month[3], '%B'), (self.a_weekday[2], '%a'),
-                    (self.a_month[3], '%b'), (self.am_pm[1], '%p'),
-                    ('1999', '%Y'), ('99', '%y'), ('22', '%H'),
-                    ('44', '%M'), ('55', '%S'), ('76', '%j'),
-                    ('17', '%d'), ('03', '%m'), ('3', '%m'),
-                    # '3' needed for when no leading zero.
-                    ('2', '%w'), ('10', '%I')]
-        replacement_pairs.extend([(tz, "%Z") for tz_values in self.timezone
-                                                for tz in tz_values])
-        for offset,directive in ((0,'%c'), (1,'%x'), (2,'%X')):
-            current_format = date_time[offset]
-            for old, new in replacement_pairs:
+        time_tuple2 = time.struct_time((1999,1,3,1,1,1,6,3,0))
+        replacement_pairs = []
+
+        # Non-ASCII digits
+        if self.LC_alt_digits or self.LC_alt_digits is None:
+            for n, d in [(19, '%OC'), (99, '%Oy'), (22, '%OH'),
+                         (44, '%OM'), (55, '%OS'), (17, '%Od'),
+                         (3, '%Om'), (2, '%Ow'), (10, '%OI')]:
+                if self.LC_alt_digits is None:
+                    s = chr(0x660 + n // 10) + chr(0x660 + n % 10)
+                    replacement_pairs.append((s, d))
+                    if n < 10:
+                        replacement_pairs.append((s[1], d))
+                elif len(self.LC_alt_digits) > n:
+                    replacement_pairs.append((self.LC_alt_digits[n], d))
+                else:
+                    replacement_pairs.append((time.strftime(d, time_tuple), d))
+        replacement_pairs += [
+            ('1999', '%Y'), ('99', '%y'), ('22', '%H'),
+            ('44', '%M'), ('55', '%S'), ('76', '%j'),
+            ('17', '%d'), ('03', '%m'), ('3', '%m'),
+            # '3' needed for when no leading zero.
+            ('2', '%w'), ('10', '%I'),
+        ]
+
+        date_time = []
+        for directive in ('%c', '%x', '%X', '%r'):
+            current_format = time.strftime(directive, time_tuple).lower()
+            current_format = current_format.replace('%', '%%')
+            # The month and the day of the week formats are treated specially
+            # because of a possible ambiguity in some locales where the full
+            # and abbreviated names are equal or names of different types
+            # are equal. See doc of __find_month_format for more details.
+            lst, fmt = self.__find_weekday_format(directive)
+            if lst:
+                current_format = current_format.replace(lst[2], fmt, 1)
+            lst, fmt = self.__find_month_format(directive)
+            if lst:
+                current_format = current_format.replace(lst[3], fmt, 1)
+            if self.am_pm[1]:
                 # Must deal with possible lack of locale info
                 # manifesting itself as the empty string (e.g., Swedish's
                 # lack of AM/PM info) or a platform returning a tuple of empty
                 # strings (e.g., MacOS 9 having timezone as ('','')).
-                if old:
-                    current_format = current_format.replace(old, new)
+                current_format = current_format.replace(self.am_pm[1], '%p')
+            for tz_values in self.timezone:
+                for tz in tz_values:
+                    if tz:
+                        current_format = current_format.replace(tz, "%Z")
+            # Transform all non-ASCII digits to digits in range U+0660 to U+0669.
+            if not current_format.isascii() and self.LC_alt_digits is None:
+                current_format = re_sub(r'\d(?<![0-9])',
+                                        lambda m: chr(0x0660 + int(m[0])),
+                                        current_format)
+            for old, new in replacement_pairs:
+                current_format = current_format.replace(old, new)
             # If %W is used, then Sunday, 2005-01-03 will fall on week 0 since
             # 2005-01-03 occurs before the first Monday of the year.  Otherwise
             # %U is used.
-            time_tuple = time.struct_time((1999,1,3,1,1,1,6,3,0))
-            if '00' in time.strftime(directive, time_tuple):
+            if '00' in time.strftime(directive, time_tuple2):
                 U_W = '%W'
             else:
                 U_W = '%U'
-            date_time[offset] = current_format.replace('11', U_W)
+            current_format = current_format.replace('11', U_W)
+            date_time.append(current_format)
         self.LC_date_time = date_time[0]
         self.LC_date = date_time[1]
         self.LC_time = date_time[2]
+        self.LC_time_ampm = date_time[3]
+
+    def __find_month_format(self, directive):
+        """Find the month format appropriate for the current locale.
+
+        In some locales (for example French and Hebrew), the default month
+        used in __calc_date_time has the same name in full and abbreviated
+        form.  Also, the month name can by accident match other part of the
+        representation: the day of the week name (for example in Morisyen)
+        or the month number (for example in Japanese).  Thus, cycle months
+        of the year and find all positions that match the month name for
+        each month,  If no common positions are found, the representation
+        does not use the month name.
+        """
+        full_indices = abbr_indices = None
+        for m in range(1, 13):
+            time_tuple = time.struct_time((1999, m, 17, 22, 44, 55, 2, 76, 0))
+            datetime = time.strftime(directive, time_tuple).lower()
+            indices = set(_findall(datetime, self.f_month[m]))
+            if full_indices is None:
+                full_indices = indices
+            else:
+                full_indices &= indices
+            indices = set(_findall(datetime, self.a_month[m]))
+            if abbr_indices is None:
+                abbr_indices = set(indices)
+            else:
+                abbr_indices &= indices
+            if not full_indices and not abbr_indices:
+                return None, None
+        if full_indices:
+            return self.f_month, '%B'
+        if abbr_indices:
+            return self.a_month, '%b'
+        return None, None
+
+    def __find_weekday_format(self, directive):
+        """Find the day of the week format appropriate for the current locale.
+
+        Similar to __find_month_format().
+        """
+        full_indices = abbr_indices = None
+        for wd in range(7):
+            time_tuple = time.struct_time((1999, 3, 17, 22, 44, 55, wd, 76, 0))
+            datetime = time.strftime(directive, time_tuple).lower()
+            indices = set(_findall(datetime, self.f_weekday[wd]))
+            if full_indices is None:
+                full_indices = indices
+            else:
+                full_indices &= indices
+            if self.f_weekday[wd] != self.a_weekday[wd]:
+                indices = set(_findall(datetime, self.a_weekday[wd]))
+            if abbr_indices is None:
+                abbr_indices = set(indices)
+            else:
+                abbr_indices &= indices
+            if not full_indices and not abbr_indices:
+                return None, None
+        if full_indices:
+            return self.f_weekday, '%A'
+        if abbr_indices:
+            return self.a_weekday, '%a'
+        return None, None
 
     def __calc_timezone(self):
         # Set self.timezone by using time.tzname.
@@ -181,12 +351,14 @@ class TimeRE(dict):
         else:
             self.locale_time = LocaleTime()
         base = super()
-        base.__init__({
+        mapping = {
             # The " [1-9]" part of the regex is to make %c from ANSI C work
             'd': r"(?P<d>3[0-1]|[1-2]\d|0[1-9]|[1-9]| [1-9])",
             'f': r"(?P<f>[0-9]{1,6})",
-            'H': r"(?P<H>2[0-3]|[0-1]\d|\d)",
-            'I': r"(?P<I>1[0-2]|0[1-9]|[1-9])",
+            'H': r"(?P<H>2[0-3]|[0-1]\d|\d| \d)",
+            'k': r"(?P<H>2[0-3]|[0-1]\d|\d| \d)",
+            'I': r"(?P<I>1[0-2]|0[1-9]|[1-9]| [1-9])",
+            'l': r"(?P<I>1[0-2]|0[1-9]|[1-9]| [1-9])",
             'G': r"(?P<G>\d\d\d\d)",
             'j': r"(?P<j>36[0-6]|3[0-5]\d|[1-2]\d\d|0[1-9]\d|00[1-9]|[1-9]\d|0[1-9]|[1-9])",
             'm': r"(?P<m>1[0-2]|0[1-9]|[1-9])",
@@ -198,25 +370,60 @@ class TimeRE(dict):
             'V': r"(?P<V>5[0-3]|0[1-9]|[1-4]\d|\d)",
             # W is set below by using 'U'
             'y': r"(?P<y>\d\d)",
-            #XXX: Does 'Y' need to worry about having less or more than
-            #     4 digits?
             'Y': r"(?P<Y>\d\d\d\d)",
             'z': r"(?P<z>[+-]\d\d:?[0-5]\d(:?[0-5]\d(\.\d{1,6})?)?|(?-i:Z))",
             'A': self.__seqToRE(self.locale_time.f_weekday, 'A'),
             'a': self.__seqToRE(self.locale_time.a_weekday, 'a'),
-            'B': self.__seqToRE(self.locale_time.f_month[1:], 'B'),
-            'b': self.__seqToRE(self.locale_time.a_month[1:], 'b'),
+            'B': self.__seqToRE(_fixmonths(self.locale_time.f_month[1:]), 'B'),
+            'b': self.__seqToRE(_fixmonths(self.locale_time.a_month[1:]), 'b'),
             'p': self.__seqToRE(self.locale_time.am_pm, 'p'),
             'Z': self.__seqToRE((tz for tz_names in self.locale_time.timezone
                                         for tz in tz_names),
                                 'Z'),
-            '%': '%'})
-        base.__setitem__('W', base.__getitem__('U').replace('U', 'W'))
-        base.__setitem__('c', self.pattern(self.locale_time.LC_date_time))
-        base.__setitem__('x', self.pattern(self.locale_time.LC_date))
-        base.__setitem__('X', self.pattern(self.locale_time.LC_time))
+            '%': '%'}
+        if self.locale_time.LC_alt_digits is None:
+            for d in 'dmyCHIMS':
+                mapping['O' + d] = r'(?P<%s>\d\d|\d| \d)' % d
+            mapping['Ow'] = r'(?P<w>\d)'
+        else:
+            mapping.update({
+                'Od': self.__seqToRE(self.locale_time.LC_alt_digits[1:32], 'd',
+                                     '3[0-1]|[1-2][0-9]|0[1-9]|[1-9]'),
+                'Om': self.__seqToRE(self.locale_time.LC_alt_digits[1:13], 'm',
+                                     '1[0-2]|0[1-9]|[1-9]'),
+                'Ow': self.__seqToRE(self.locale_time.LC_alt_digits[:7], 'w',
+                                     '[0-6]'),
+                'Oy': self.__seqToRE(self.locale_time.LC_alt_digits, 'y',
+                                     '[0-9][0-9]'),
+                'OC': self.__seqToRE(self.locale_time.LC_alt_digits, 'C',
+                                     '[0-9][0-9]'),
+                'OH': self.__seqToRE(self.locale_time.LC_alt_digits[:24], 'H',
+                                     '2[0-3]|[0-1][0-9]|[0-9]'),
+                'OI': self.__seqToRE(self.locale_time.LC_alt_digits[1:13], 'I',
+                                     '1[0-2]|0[1-9]|[1-9]'),
+                'OM': self.__seqToRE(self.locale_time.LC_alt_digits[:60], 'M',
+                                     '[0-5][0-9]|[0-9]'),
+                'OS': self.__seqToRE(self.locale_time.LC_alt_digits[:62], 'S',
+                                     '6[0-1]|[0-5][0-9]|[0-9]'),
+            })
+        mapping.update({
+            'e': mapping['d'],
+            'Oe': mapping['Od'],
+            'P': mapping['p'],
+            'Op': mapping['p'],
+            'W': mapping['U'].replace('U', 'W'),
+        })
+        mapping['W'] = mapping['U'].replace('U', 'W')
 
-    def __seqToRE(self, to_convert, directive):
+        base.__init__(mapping)
+        base.__setitem__('T', self.pattern('%H:%M:%S'))
+        base.__setitem__('R', self.pattern('%H:%M'))
+        base.__setitem__('r', self.pattern(self.locale_time.LC_time_ampm))
+        base.__setitem__('X', self.pattern(self.locale_time.LC_time))
+        base.__setitem__('x', self.pattern(self.locale_time.LC_date))
+        base.__setitem__('c', self.pattern(self.locale_time.LC_date_time))
+
+    def __seqToRE(self, to_convert, directive, altregex=None):
         """Convert a list to a regex string for matching a directive.
 
         Want possible matching values to be from longest to shortest.  This
@@ -232,8 +439,9 @@ class TimeRE(dict):
         else:
             return ''
         regex = '|'.join(re_escape(stuff) for stuff in to_convert)
-        regex = '(?P<%s>%s' % (directive, regex)
-        return '%s)' % regex
+        if altregex is not None:
+            regex += '|' + altregex
+        return '(?P<%s>%s)' % (directive, regex)
 
     def pattern(self, format):
         """Return regex pattern for the format string.
@@ -242,21 +450,36 @@ class TimeRE(dict):
         regex syntax are escaped.
 
         """
-        processed_format = ''
         # The sub() call escapes all characters that might be misconstrued
         # as regex syntax.  Cannot use re.escape since we have to deal with
         # format directives (%m, etc.).
-        regex_chars = re_compile(r"([\\.^$*+?\(\){}\[\]|])")
-        format = regex_chars.sub(r"\\\1", format)
-        whitespace_replacement = re_compile(r'\s+')
-        format = whitespace_replacement.sub(r'\\s+', format)
-        while '%' in format:
-            directive_index = format.index('%')+1
-            processed_format = "%s%s%s" % (processed_format,
-                                           format[:directive_index-1],
-                                           self[format[directive_index]])
-            format = format[directive_index+1:]
-        return "%s%s" % (processed_format, format)
+        format = re_sub(r"([\\.^$*+?\(\){}\[\]|])", r"\\\1", format)
+        format = re_sub(r'\s+', r'\\s+', format)
+        format = re_sub(r"'", "['\u02bc]", format)  # needed for br_FR
+        year_in_format = False
+        day_of_month_in_format = False
+        def repl(m):
+            format_char = m[1]
+            match format_char:
+                case 'Y' | 'y' | 'G':
+                    nonlocal year_in_format
+                    year_in_format = True
+                case 'd':
+                    nonlocal day_of_month_in_format
+                    day_of_month_in_format = True
+            return self[format_char]
+        format = re_sub(r'%[-_0^#]*[0-9]*([OE]?\\?.?)', repl, format)
+        if day_of_month_in_format and not year_in_format:
+            import warnings
+            warnings.warn("""\
+Parsing dates involving a day of month without a year specified is ambiguous
+and fails to parse leap day. The default behavior will change in Python 3.15
+to either always raise an exception or to use a different default year (TBD).
+To avoid trouble, add a specific year to the input & format.
+See https://github.com/python/cpython/issues/70647.""",
+                          DeprecationWarning,
+                          skip_file_prefixes=(os.path.dirname(__file__),))
+        return format
 
     def compile(self, format):
         """Return a compiled re object for the format string."""
@@ -319,14 +542,13 @@ def _strptime(data_string, format="%a %b %d %H:%M:%S %Y"):
             # \\, in which case it was a stray % but with a space after it
             except KeyError as err:
                 bad_directive = err.args[0]
-                if bad_directive == "\\":
-                    bad_directive = "%"
                 del err
+                bad_directive = bad_directive.replace('\\s', '')
+                if not bad_directive:
+                    raise ValueError("stray %% in format '%s'" % format) from None
+                bad_directive = bad_directive.replace('\\', '', 1)
                 raise ValueError("'%s' is a bad directive in format '%s'" %
                                     (bad_directive, format)) from None
-            # IndexError only occurs when the format string is "%"
-            except IndexError:
-                raise ValueError("stray %% in format '%s'" % format) from None
             _regex_cache[format] = format_regex
     found = format_regex.match(data_string)
     if not found:
@@ -348,6 +570,15 @@ def _strptime(data_string, format="%a %b %d %H:%M:%S %Y"):
     # values
     weekday = julian = None
     found_dict = found.groupdict()
+    if locale_time.LC_alt_digits:
+        def parse_int(s):
+            try:
+                return locale_time.LC_alt_digits.index(s)
+            except ValueError:
+                return int(s)
+    else:
+        parse_int = int
+
     for group_key in found_dict.keys():
         # Directives not explicitly handled below:
         #   c, x, X
@@ -355,30 +586,34 @@ def _strptime(data_string, format="%a %b %d %H:%M:%S %Y"):
         #   U, W
         #      worthless without day of the week
         if group_key == 'y':
-            year = int(found_dict['y'])
-            # Open Group specification for strptime() states that a %y
-            #value in the range of [00, 68] is in the century 2000, while
-            #[69,99] is in the century 1900
-            if year <= 68:
-                year += 2000
+            year = parse_int(found_dict['y'])
+            if 'C' in found_dict:
+                century = parse_int(found_dict['C'])
+                year += century * 100
             else:
-                year += 1900
+                # Open Group specification for strptime() states that a %y
+                #value in the range of [00, 68] is in the century 2000, while
+                #[69,99] is in the century 1900
+                if year <= 68:
+                    year += 2000
+                else:
+                    year += 1900
         elif group_key == 'Y':
             year = int(found_dict['Y'])
         elif group_key == 'G':
             iso_year = int(found_dict['G'])
         elif group_key == 'm':
-            month = int(found_dict['m'])
+            month = parse_int(found_dict['m'])
         elif group_key == 'B':
             month = locale_time.f_month.index(found_dict['B'].lower())
         elif group_key == 'b':
             month = locale_time.a_month.index(found_dict['b'].lower())
         elif group_key == 'd':
-            day = int(found_dict['d'])
+            day = parse_int(found_dict['d'])
         elif group_key == 'H':
-            hour = int(found_dict['H'])
+            hour = parse_int(found_dict['H'])
         elif group_key == 'I':
-            hour = int(found_dict['I'])
+            hour = parse_int(found_dict['I'])
             ampm = found_dict.get('p', '').lower()
             # If there was no AM/PM indicator, we'll treat this like AM
             if ampm in ('', locale_time.am_pm[0]):
@@ -394,9 +629,9 @@ def _strptime(data_string, format="%a %b %d %H:%M:%S %Y"):
                 if hour != 12:
                     hour += 12
         elif group_key == 'M':
-            minute = int(found_dict['M'])
+            minute = parse_int(found_dict['M'])
         elif group_key == 'S':
-            second = int(found_dict['S'])
+            second = parse_int(found_dict['S'])
         elif group_key == 'f':
             s = found_dict['f']
             # Pad to always return microseconds.
@@ -548,18 +783,40 @@ def _strptime_time(data_string, format="%a %b %d %H:%M:%S %Y"):
     tt = _strptime(data_string, format)[0]
     return time.struct_time(tt[:time._STRUCT_TM_ITEMS])
 
-def _strptime_datetime(cls, data_string, format="%a %b %d %H:%M:%S %Y"):
-    """Return a class cls instance based on the input string and the
+def _strptime_datetime_date(cls, data_string, format="%a %b %d %Y"):
+    """Return a date instance based on the input string and the
+    format string."""
+    tt, _, _ = _strptime(data_string, format)
+    args = tt[:3]
+    return cls(*args)
+
+def _parse_tz(tzname, gmtoff, gmtoff_fraction):
+    tzdelta = datetime_timedelta(seconds=gmtoff, microseconds=gmtoff_fraction)
+    if tzname:
+        return datetime_timezone(tzdelta, tzname)
+    else:
+        return datetime_timezone(tzdelta)
+
+def _strptime_datetime_time(cls, data_string, format="%H:%M:%S"):
+    """Return a time instance based on the input string and the
+    format string."""
+    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
+    tzname, gmtoff = tt[-2:]
+    args = tt[3:6] + (fraction,)
+    if gmtoff is None:
+        return cls(*args)
+    else:
+        tz = _parse_tz(tzname, gmtoff, gmtoff_fraction)
+        return cls(*args, tz)
+
+def _strptime_datetime_datetime(cls, data_string, format="%a %b %d %H:%M:%S %Y"):
+    """Return a datetime instance based on the input string and the
     format string."""
     tt, fraction, gmtoff_fraction = _strptime(data_string, format)
     tzname, gmtoff = tt[-2:]
     args = tt[:6] + (fraction,)
-    if gmtoff is not None:
-        tzdelta = datetime_timedelta(seconds=gmtoff, microseconds=gmtoff_fraction)
-        if tzname:
-            tz = datetime_timezone(tzdelta, tzname)
-        else:
-            tz = datetime_timezone(tzdelta)
-        args += (tz,)
-
-    return cls(*args)
+    if gmtoff is None:
+        return cls(*args)
+    else:
+        tz = _parse_tz(tzname, gmtoff, gmtoff_fraction)
+        return cls(*args, tz)

Lib/_threading_local.py

@@ -4,133 +4,6 @@
  class.  Depending on the version of Python you're using, there may be a
  faster one available.  You should always import the `local` class from
  `threading`.)
-
-Thread-local objects support the management of thread-local data.
-If you have data that you want to be local to a thread, simply create
-a thread-local object and use its attributes:
-
-  >>> mydata = local()
-  >>> mydata.number = 42
-  >>> mydata.number
-  42
-
-You can also access the local-object's dictionary:
-
-  >>> mydata.__dict__
-  {'number': 42}
-  >>> mydata.__dict__.setdefault('widgets', [])
-  []
-  >>> mydata.widgets
-  []
-
-What's important about thread-local objects is that their data are
-local to a thread. If we access the data in a different thread:
-
-  >>> log = []
-  >>> def f():
-  ...     items = sorted(mydata.__dict__.items())
-  ...     log.append(items)
-  ...     mydata.number = 11
-  ...     log.append(mydata.number)
-
-  >>> import threading
-  >>> thread = threading.Thread(target=f)
-  >>> thread.start()
-  >>> thread.join()
-  >>> log
-  [[], 11]
-
-we get different data.  Furthermore, changes made in the other thread
-don't affect data seen in this thread:
-
-  >>> mydata.number
-  42
-
-Of course, values you get from a local object, including a __dict__
-attribute, are for whatever thread was current at the time the
-attribute was read.  For that reason, you generally don't want to save
-these values across threads, as they apply only to the thread they
-came from.
-
-You can create custom local objects by subclassing the local class:
-
-  >>> class MyLocal(local):
-  ...     number = 2
-  ...     initialized = False
-  ...     def __init__(self, **kw):
-  ...         if self.initialized:
-  ...             raise SystemError('__init__ called too many times')
-  ...         self.initialized = True
-  ...         self.__dict__.update(kw)
-  ...     def squared(self):
-  ...         return self.number ** 2
-
-This can be useful to support default values, methods and
-initialization.  Note that if you define an __init__ method, it will be
-called each time the local object is used in a separate thread.  This
-is necessary to initialize each thread's dictionary.
-
-Now if we create a local object:
-
-  >>> mydata = MyLocal(color='red')
-
-Now we have a default number:
-
-  >>> mydata.number
-  2
-
-an initial color:
-
-  >>> mydata.color
-  'red'
-  >>> del mydata.color
-
-And a method that operates on the data:
-
-  >>> mydata.squared()
-  4
-
-As before, we can access the data in a separate thread:
-
-  >>> log = []
-  >>> thread = threading.Thread(target=f)
-  >>> thread.start()
-  >>> thread.join()
-  >>> log
-  [[('color', 'red'), ('initialized', True)], 11]
-
-without affecting this thread's data:
-
-  >>> mydata.number
-  2
-  >>> mydata.color
-  Traceback (most recent call last):
-  ...
-  AttributeError: 'MyLocal' object has no attribute 'color'
-
-Note that subclasses can define slots, but they are not thread
-local. They are shared across threads:
-
-  >>> class MyLocal(local):
-  ...     __slots__ = 'number'
-
-  >>> mydata = MyLocal()
-  >>> mydata.number = 42
-  >>> mydata.color = 'red'
-
-So, the separate thread:
-
-  >>> thread = threading.Thread(target=f)
-  >>> thread.start()
-  >>> thread.join()
-
-affects what we see:
-
-  >>> # TODO: RUSTPYTHON, __slots__
-  >>> mydata.number #doctest: +SKIP
-  11
-
->>> del mydata
 """
 
 from weakref import ref
@@ -194,7 +67,6 @@ class _localimpl:
 
 @contextmanager
 def _patch(self):
-    old = object.__getattribute__(self, '__dict__')
     impl = object.__getattribute__(self, '_local__impl')
     try:
         dct = impl.get_dict()
@@ -205,13 +77,12 @@ def _patch(self):
     with impl.locallock:
         object.__setattr__(self, '__dict__', dct)
         yield
-        object.__setattr__(self, '__dict__', old)
 
 
 class local:
     __slots__ = '_local__impl', '__dict__'
 
-    def __new__(cls, *args, **kw):
+    def __new__(cls, /, *args, **kw):
         if (args or kw) and (cls.__init__ is object.__init__):
             raise TypeError("Initialization arguments are not supported")
         self = object.__new__(cls)

Lib/_weakrefset.py

@@ -8,69 +8,29 @@ from types import GenericAlias
 __all__ = ['WeakSet']
 
 
-class _IterationGuard:
-    # This context manager registers itself in the current iterators of the
-    # weak container, such as to delay all removals until the context manager
-    # exits.
-    # This technique should be relatively thread-safe (since sets are).
-
-    def __init__(self, weakcontainer):
-        # Don't create cycles
-        self.weakcontainer = ref(weakcontainer)
-
-    def __enter__(self):
-        w = self.weakcontainer()
-        if w is not None:
-            w._iterating.add(self)
-        return self
-
-    def __exit__(self, e, t, b):
-        w = self.weakcontainer()
-        if w is not None:
-            s = w._iterating
-            s.remove(self)
-            if not s:
-                w._commit_removals()
-
-
 class WeakSet:
     def __init__(self, data=None):
         self.data = set()
+
         def _remove(item, selfref=ref(self)):
             self = selfref()
             if self is not None:
-                if self._iterating:
-                    self._pending_removals.append(item)
-                else:
-                    self.data.discard(item)
+                self.data.discard(item)
+
         self._remove = _remove
-        # A list of keys to be removed
-        self._pending_removals = []
-        self._iterating = set()
         if data is not None:
             self.update(data)
 
-    def _commit_removals(self):
-        pop = self._pending_removals.pop
-        discard = self.data.discard
-        while True:
-            try:
-                item = pop()
-            except IndexError:
-                return
-            discard(item)
-
     def __iter__(self):
-        with _IterationGuard(self):
-            for itemref in self.data:
-                item = itemref()
-                if item is not None:
-                    # Caveat: the iterator will keep a strong reference to
-                    # `item` until it is resumed or closed.
-                    yield item
+        for itemref in self.data.copy():
+            item = itemref()
+            if item is not None:
+                # Caveat: the iterator will keep a strong reference to
+                # `item` until it is resumed or closed.
+                yield item
 
     def __len__(self):
-        return len(self.data) - len(self._pending_removals)
+        return len(self.data)
 
     def __contains__(self, item):
         try:
@@ -80,25 +40,18 @@ class WeakSet:
         return wr in self.data
 
     def __reduce__(self):
-        return (self.__class__, (list(self),),
-                getattr(self, '__dict__', None))
+        return self.__class__, (list(self),), self.__getstate__()
 
     def add(self, item):
-        if self._pending_removals:
-            self._commit_removals()
         self.data.add(ref(item, self._remove))
 
     def clear(self):
-        if self._pending_removals:
-            self._commit_removals()
         self.data.clear()
 
     def copy(self):
         return self.__class__(self)
 
     def pop(self):
-        if self._pending_removals:
-            self._commit_removals()
         while True:
             try:
                 itemref = self.data.pop()
@@ -109,18 +62,12 @@ class WeakSet:
                 return item
 
     def remove(self, item):
-        if self._pending_removals:
-            self._commit_removals()
         self.data.remove(ref(item))
 
     def discard(self, item):
-        if self._pending_removals:
-            self._commit_removals()
         self.data.discard(ref(item))
 
     def update(self, other):
-        if self._pending_removals:
-            self._commit_removals()
         for element in other:
             self.add(element)
 
@@ -137,8 +84,6 @@ class WeakSet:
     def difference_update(self, other):
         self.__isub__(other)
     def __isub__(self, other):
-        if self._pending_removals:
-            self._commit_removals()
         if self is other:
             self.data.clear()
         else:
@@ -152,8 +97,6 @@ class WeakSet:
     def intersection_update(self, other):
         self.__iand__(other)
     def __iand__(self, other):
-        if self._pending_removals:
-            self._commit_removals()
         self.data.intersection_update(ref(item) for item in other)
         return self
 
@@ -185,8 +128,6 @@ class WeakSet:
     def symmetric_difference_update(self, other):
         self.__ixor__(other)
     def __ixor__(self, other):
-        if self._pending_removals:
-            self._commit_removals()
         if self is other:
             self.data.clear()
         else:

Lib/abc.py

@@ -85,10 +85,6 @@ try:
     from _abc import (get_cache_token, _abc_init, _abc_register,
                       _abc_instancecheck, _abc_subclasscheck, _get_dump,
                       _reset_registry, _reset_caches)
-# TODO: RUSTPYTHON missing _abc module implementation.
-except ModuleNotFoundError:
-    from _py_abc import ABCMeta, get_cache_token
-    ABCMeta.__module__ = 'abc'
 except ImportError:
     from _py_abc import ABCMeta, get_cache_token
     ABCMeta.__module__ = 'abc'

Lib/aifc.py

@@ -1,984 +0,0 @@
-"""Stuff to parse AIFF-C and AIFF files.
-
-Unless explicitly stated otherwise, the description below is true
-both for AIFF-C files and AIFF files.
-
-An AIFF-C file has the following structure.
-
-  +-----------------+
-  | FORM            |
-  +-----------------+
-  | <size>          |
-  +----+------------+
-  |    | AIFC       |
-  |    +------------+
-  |    | <chunks>   |
-  |    |    .       |
-  |    |    .       |
-  |    |    .       |
-  +----+------------+
-
-An AIFF file has the string "AIFF" instead of "AIFC".
-
-A chunk consists of an identifier (4 bytes) followed by a size (4 bytes,
-big endian order), followed by the data.  The size field does not include
-the size of the 8 byte header.
-
-The following chunk types are recognized.
-
-  FVER
-      <version number of AIFF-C defining document> (AIFF-C only).
-  MARK
-      <# of markers> (2 bytes)
-      list of markers:
-          <marker ID> (2 bytes, must be > 0)
-          <position> (4 bytes)
-          <marker name> ("pstring")
-  COMM
-      <# of channels> (2 bytes)
-      <# of sound frames> (4 bytes)
-      <size of the samples> (2 bytes)
-      <sampling frequency> (10 bytes, IEEE 80-bit extended
-          floating point)
-      in AIFF-C files only:
-      <compression type> (4 bytes)
-      <human-readable version of compression type> ("pstring")
-  SSND
-      <offset> (4 bytes, not used by this program)
-      <blocksize> (4 bytes, not used by this program)
-      <sound data>
-
-A pstring consists of 1 byte length, a string of characters, and 0 or 1
-byte pad to make the total length even.
-
-Usage.
-
-Reading AIFF files:
-  f = aifc.open(file, 'r')
-where file is either the name of a file or an open file pointer.
-The open file pointer must have methods read(), seek(), and close().
-In some types of audio files, if the setpos() method is not used,
-the seek() method is not necessary.
-
-This returns an instance of a class with the following public methods:
-  getnchannels()  -- returns number of audio channels (1 for
-             mono, 2 for stereo)
-  getsampwidth()  -- returns sample width in bytes
-  getframerate()  -- returns sampling frequency
-  getnframes()    -- returns number of audio frames
-  getcomptype()   -- returns compression type ('NONE' for AIFF files)
-  getcompname()   -- returns human-readable version of
-             compression type ('not compressed' for AIFF files)
-  getparams() -- returns a namedtuple consisting of all of the
-             above in the above order
-  getmarkers()    -- get the list of marks in the audio file or None
-             if there are no marks
-  getmark(id) -- get mark with the specified id (raises an error
-             if the mark does not exist)
-  readframes(n)   -- returns at most n frames of audio
-  rewind()    -- rewind to the beginning of the audio stream
-  setpos(pos) -- seek to the specified position
-  tell()      -- return the current position
-  close()     -- close the instance (make it unusable)
-The position returned by tell(), the position given to setpos() and
-the position of marks are all compatible and have nothing to do with
-the actual position in the file.
-The close() method is called automatically when the class instance
-is destroyed.
-
-Writing AIFF files:
-  f = aifc.open(file, 'w')
-where file is either the name of a file or an open file pointer.
-The open file pointer must have methods write(), tell(), seek(), and
-close().
-
-This returns an instance of a class with the following public methods:
-  aiff()      -- create an AIFF file (AIFF-C default)
-  aifc()      -- create an AIFF-C file
-  setnchannels(n) -- set the number of channels
-  setsampwidth(n) -- set the sample width
-  setframerate(n) -- set the frame rate
-  setnframes(n)   -- set the number of frames
-  setcomptype(type, name)
-          -- set the compression type and the
-             human-readable compression type
-  setparams(tuple)
-          -- set all parameters at once
-  setmark(id, pos, name)
-          -- add specified mark to the list of marks
-  tell()      -- return current position in output file (useful
-             in combination with setmark())
-  writeframesraw(data)
-          -- write audio frames without pathing up the
-             file header
-  writeframes(data)
-          -- write audio frames and patch up the file header
-  close()     -- patch up the file header and close the
-             output file
-You should set the parameters before the first writeframesraw or
-writeframes.  The total number of frames does not need to be set,
-but when it is set to the correct value, the header does not have to
-be patched up.
-It is best to first set all parameters, perhaps possibly the
-compression type, and then write audio frames using writeframesraw.
-When all frames have been written, either call writeframes(b'') or
-close() to patch up the sizes in the header.
-Marks can be added anytime.  If there are any marks, you must call
-close() after all frames have been written.
-The close() method is called automatically when the class instance
-is destroyed.
-
-When a file is opened with the extension '.aiff', an AIFF file is
-written, otherwise an AIFF-C file is written.  This default can be
-changed by calling aiff() or aifc() before the first writeframes or
-writeframesraw.
-"""
-
-import struct
-import builtins
-import warnings
-
-__all__ = ["Error", "open"]
-
-
-warnings._deprecated(__name__, remove=(3, 13))
-
-
-class Error(Exception):
-    pass
-
-_AIFC_version = 0xA2805140     # Version 1 of AIFF-C
-
-def _read_long(file):
-    try:
-        return struct.unpack('>l', file.read(4))[0]
-    except struct.error:
-        raise EOFError from None
-
-def _read_ulong(file):
-    try:
-        return struct.unpack('>L', file.read(4))[0]
-    except struct.error:
-        raise EOFError from None
-
-def _read_short(file):
-    try:
-        return struct.unpack('>h', file.read(2))[0]
-    except struct.error:
-        raise EOFError from None
-
-def _read_ushort(file):
-    try:
-        return struct.unpack('>H', file.read(2))[0]
-    except struct.error:
-        raise EOFError from None
-
-def _read_string(file):
-    length = ord(file.read(1))
-    if length == 0:
-        data = b''
-    else:
-        data = file.read(length)
-    if length & 1 == 0:
-        dummy = file.read(1)
-    return data
-
-_HUGE_VAL = 1.79769313486231e+308 # See <limits.h>
-
-def _read_float(f): # 10 bytes
-    expon = _read_short(f) # 2 bytes
-    sign = 1
-    if expon < 0:
-        sign = -1
-        expon = expon + 0x8000
-    himant = _read_ulong(f) # 4 bytes
-    lomant = _read_ulong(f) # 4 bytes
-    if expon == himant == lomant == 0:
-        f = 0.0
-    elif expon == 0x7FFF:
-        f = _HUGE_VAL
-    else:
-        expon = expon - 16383
-        f = (himant * 0x100000000 + lomant) * pow(2.0, expon - 63)
-    return sign * f
-
-def _write_short(f, x):
-    f.write(struct.pack('>h', x))
-
-def _write_ushort(f, x):
-    f.write(struct.pack('>H', x))
-
-def _write_long(f, x):
-    f.write(struct.pack('>l', x))
-
-def _write_ulong(f, x):
-    f.write(struct.pack('>L', x))
-
-def _write_string(f, s):
-    if len(s) > 255:
-        raise ValueError("string exceeds maximum pstring length")
-    f.write(struct.pack('B', len(s)))
-    f.write(s)
-    if len(s) & 1 == 0:
-        f.write(b'\x00')
-
-def _write_float(f, x):
-    import math
-    if x < 0:
-        sign = 0x8000
-        x = x * -1
-    else:
-        sign = 0
-    if x == 0:
-        expon = 0
-        himant = 0
-        lomant = 0
-    else:
-        fmant, expon = math.frexp(x)
-        if expon > 16384 or fmant >= 1 or fmant != fmant: # Infinity or NaN
-            expon = sign|0x7FFF
-            himant = 0
-            lomant = 0
-        else:                   # Finite
-            expon = expon + 16382
-            if expon < 0:           # denormalized
-                fmant = math.ldexp(fmant, expon)
-                expon = 0
-            expon = expon | sign
-            fmant = math.ldexp(fmant, 32)
-            fsmant = math.floor(fmant)
-            himant = int(fsmant)
-            fmant = math.ldexp(fmant - fsmant, 32)
-            fsmant = math.floor(fmant)
-            lomant = int(fsmant)
-    _write_ushort(f, expon)
-    _write_ulong(f, himant)
-    _write_ulong(f, lomant)
-
-with warnings.catch_warnings():
-    warnings.simplefilter("ignore", DeprecationWarning)
-    from chunk import Chunk
-from collections import namedtuple
-
-_aifc_params = namedtuple('_aifc_params',
-                          'nchannels sampwidth framerate nframes comptype compname')
-
-_aifc_params.nchannels.__doc__ = 'Number of audio channels (1 for mono, 2 for stereo)'
-_aifc_params.sampwidth.__doc__ = 'Sample width in bytes'
-_aifc_params.framerate.__doc__ = 'Sampling frequency'
-_aifc_params.nframes.__doc__ = 'Number of audio frames'
-_aifc_params.comptype.__doc__ = 'Compression type ("NONE" for AIFF files)'
-_aifc_params.compname.__doc__ = ("""\
-A human-readable version of the compression type
-('not compressed' for AIFF files)""")
-
-
-class Aifc_read:
-    # Variables used in this class:
-    #
-    # These variables are available to the user though appropriate
-    # methods of this class:
-    # _file -- the open file with methods read(), close(), and seek()
-    #       set through the __init__() method
-    # _nchannels -- the number of audio channels
-    #       available through the getnchannels() method
-    # _nframes -- the number of audio frames
-    #       available through the getnframes() method
-    # _sampwidth -- the number of bytes per audio sample
-    #       available through the getsampwidth() method
-    # _framerate -- the sampling frequency
-    #       available through the getframerate() method
-    # _comptype -- the AIFF-C compression type ('NONE' if AIFF)
-    #       available through the getcomptype() method
-    # _compname -- the human-readable AIFF-C compression type
-    #       available through the getcomptype() method
-    # _markers -- the marks in the audio file
-    #       available through the getmarkers() and getmark()
-    #       methods
-    # _soundpos -- the position in the audio stream
-    #       available through the tell() method, set through the
-    #       setpos() method
-    #
-    # These variables are used internally only:
-    # _version -- the AIFF-C version number
-    # _decomp -- the decompressor from builtin module cl
-    # _comm_chunk_read -- 1 iff the COMM chunk has been read
-    # _aifc -- 1 iff reading an AIFF-C file
-    # _ssnd_seek_needed -- 1 iff positioned correctly in audio
-    #       file for readframes()
-    # _ssnd_chunk -- instantiation of a chunk class for the SSND chunk
-    # _framesize -- size of one frame in the file
-
-    _file = None  # Set here since __del__ checks it
-
-    def initfp(self, file):
-        self._version = 0
-        self._convert = None
-        self._markers = []
-        self._soundpos = 0
-        self._file = file
-        chunk = Chunk(file)
-        if chunk.getname() != b'FORM':
-            raise Error('file does not start with FORM id')
-        formdata = chunk.read(4)
-        if formdata == b'AIFF':
-            self._aifc = 0
-        elif formdata == b'AIFC':
-            self._aifc = 1
-        else:
-            raise Error('not an AIFF or AIFF-C file')
-        self._comm_chunk_read = 0
-        self._ssnd_chunk = None
-        while 1:
-            self._ssnd_seek_needed = 1
-            try:
-                chunk = Chunk(self._file)
-            except EOFError:
-                break
-            chunkname = chunk.getname()
-            if chunkname == b'COMM':
-                self._read_comm_chunk(chunk)
-                self._comm_chunk_read = 1
-            elif chunkname == b'SSND':
-                self._ssnd_chunk = chunk
-                dummy = chunk.read(8)
-                self._ssnd_seek_needed = 0
-            elif chunkname == b'FVER':
-                self._version = _read_ulong(chunk)
-            elif chunkname == b'MARK':
-                self._readmark(chunk)
-            chunk.skip()
-        if not self._comm_chunk_read or not self._ssnd_chunk:
-            raise Error('COMM chunk and/or SSND chunk missing')
-
-    def __init__(self, f):
-        if isinstance(f, str):
-            file_object = builtins.open(f, 'rb')
-            try:
-                self.initfp(file_object)
-            except:
-                file_object.close()
-                raise
-        else:
-            # assume it is an open file object already
-            self.initfp(f)
-
-    def __enter__(self):
-        return self
-
-    def __exit__(self, *args):
-        self.close()
-
-    #
-    # User visible methods.
-    #
-    def getfp(self):
-        return self._file
-
-    def rewind(self):
-        self._ssnd_seek_needed = 1
-        self._soundpos = 0
-
-    def close(self):
-        file = self._file
-        if file is not None:
-            self._file = None
-            file.close()
-
-    def tell(self):
-        return self._soundpos
-
-    def getnchannels(self):
-        return self._nchannels
-
-    def getnframes(self):
-        return self._nframes
-
-    def getsampwidth(self):
-        return self._sampwidth
-
-    def getframerate(self):
-        return self._framerate
-
-    def getcomptype(self):
-        return self._comptype
-
-    def getcompname(self):
-        return self._compname
-
-##  def getversion(self):
-##      return self._version
-
-    def getparams(self):
-        return _aifc_params(self.getnchannels(), self.getsampwidth(),
-                            self.getframerate(), self.getnframes(),
-                            self.getcomptype(), self.getcompname())
-
-    def getmarkers(self):
-        if len(self._markers) == 0:
-            return None
-        return self._markers
-
-    def getmark(self, id):
-        for marker in self._markers:
-            if id == marker[0]:
-                return marker
-        raise Error('marker {0!r} does not exist'.format(id))
-
-    def setpos(self, pos):
-        if pos < 0 or pos > self._nframes:
-            raise Error('position not in range')
-        self._soundpos = pos
-        self._ssnd_seek_needed = 1
-
-    def readframes(self, nframes):
-        if self._ssnd_seek_needed:
-            self._ssnd_chunk.seek(0)
-            dummy = self._ssnd_chunk.read(8)
-            pos = self._soundpos * self._framesize
-            if pos:
-                self._ssnd_chunk.seek(pos + 8)
-            self._ssnd_seek_needed = 0
-        if nframes == 0:
-            return b''
-        data = self._ssnd_chunk.read(nframes * self._framesize)
-        if self._convert and data:
-            data = self._convert(data)
-        self._soundpos = self._soundpos + len(data) // (self._nchannels
-                                                        * self._sampwidth)
-        return data
-
-    #
-    # Internal methods.
-    #
-
-    def _alaw2lin(self, data):
-        with warnings.catch_warnings():
-            warnings.simplefilter('ignore', category=DeprecationWarning)
-            import audioop
-        return audioop.alaw2lin(data, 2)
-
-    def _ulaw2lin(self, data):
-        with warnings.catch_warnings():
-            warnings.simplefilter('ignore', category=DeprecationWarning)
-            import audioop
-        return audioop.ulaw2lin(data, 2)
-
-    def _adpcm2lin(self, data):
-        with warnings.catch_warnings():
-            warnings.simplefilter('ignore', category=DeprecationWarning)
-            import audioop
-        if not hasattr(self, '_adpcmstate'):
-            # first time
-            self._adpcmstate = None
-        data, self._adpcmstate = audioop.adpcm2lin(data, 2, self._adpcmstate)
-        return data
-
-    def _sowt2lin(self, data):
-        with warnings.catch_warnings():
-            warnings.simplefilter('ignore', category=DeprecationWarning)
-            import audioop
-        return audioop.byteswap(data, 2)
-
-    def _read_comm_chunk(self, chunk):
-        self._nchannels = _read_short(chunk)
-        self._nframes = _read_long(chunk)
-        self._sampwidth = (_read_short(chunk) + 7) // 8
-        self._framerate = int(_read_float(chunk))
-        if self._sampwidth <= 0:
-            raise Error('bad sample width')
-        if self._nchannels <= 0:
-            raise Error('bad # of channels')
-        self._framesize = self._nchannels * self._sampwidth
-        if self._aifc:
-            #DEBUG: SGI's soundeditor produces a bad size :-(
-            kludge = 0
-            if chunk.chunksize == 18:
-                kludge = 1
-                warnings.warn('Warning: bad COMM chunk size')
-                chunk.chunksize = 23
-            #DEBUG end
-            self._comptype = chunk.read(4)
-            #DEBUG start
-            if kludge:
-                length = ord(chunk.file.read(1))
-                if length & 1 == 0:
-                    length = length + 1
-                chunk.chunksize = chunk.chunksize + length
-                chunk.file.seek(-1, 1)
-            #DEBUG end
-            self._compname = _read_string(chunk)
-            if self._comptype != b'NONE':
-                if self._comptype == b'G722':
-                    self._convert = self._adpcm2lin
-                elif self._comptype in (b'ulaw', b'ULAW'):
-                    self._convert = self._ulaw2lin
-                elif self._comptype in (b'alaw', b'ALAW'):
-                    self._convert = self._alaw2lin
-                elif self._comptype in (b'sowt', b'SOWT'):
-                    self._convert = self._sowt2lin
-                else:
-                    raise Error('unsupported compression type')
-                self._sampwidth = 2
-        else:
-            self._comptype = b'NONE'
-            self._compname = b'not compressed'
-
-    def _readmark(self, chunk):
-        nmarkers = _read_short(chunk)
-        # Some files appear to contain invalid counts.
-        # Cope with this by testing for EOF.
-        try:
-            for i in range(nmarkers):
-                id = _read_short(chunk)
-                pos = _read_long(chunk)
-                name = _read_string(chunk)
-                if pos or name:
-                    # some files appear to have
-                    # dummy markers consisting of
-                    # a position 0 and name ''
-                    self._markers.append((id, pos, name))
-        except EOFError:
-            w = ('Warning: MARK chunk contains only %s marker%s instead of %s' %
-                 (len(self._markers), '' if len(self._markers) == 1 else 's',
-                  nmarkers))
-            warnings.warn(w)
-
-class Aifc_write:
-    # Variables used in this class:
-    #
-    # These variables are user settable through appropriate methods
-    # of this class:
-    # _file -- the open file with methods write(), close(), tell(), seek()
-    #       set through the __init__() method
-    # _comptype -- the AIFF-C compression type ('NONE' in AIFF)
-    #       set through the setcomptype() or setparams() method
-    # _compname -- the human-readable AIFF-C compression type
-    #       set through the setcomptype() or setparams() method
-    # _nchannels -- the number of audio channels
-    #       set through the setnchannels() or setparams() method
-    # _sampwidth -- the number of bytes per audio sample
-    #       set through the setsampwidth() or setparams() method
-    # _framerate -- the sampling frequency
-    #       set through the setframerate() or setparams() method
-    # _nframes -- the number of audio frames written to the header
-    #       set through the setnframes() or setparams() method
-    # _aifc -- whether we're writing an AIFF-C file or an AIFF file
-    #       set through the aifc() method, reset through the
-    #       aiff() method
-    #
-    # These variables are used internally only:
-    # _version -- the AIFF-C version number
-    # _comp -- the compressor from builtin module cl
-    # _nframeswritten -- the number of audio frames actually written
-    # _datalength -- the size of the audio samples written to the header
-    # _datawritten -- the size of the audio samples actually written
-
-    _file = None  # Set here since __del__ checks it
-
-    def __init__(self, f):
-        if isinstance(f, str):
-            file_object = builtins.open(f, 'wb')
-            try:
-                self.initfp(file_object)
-            except:
-                file_object.close()
-                raise
-
-            # treat .aiff file extensions as non-compressed audio
-            if f.endswith('.aiff'):
-                self._aifc = 0
-        else:
-            # assume it is an open file object already
-            self.initfp(f)
-
-    def initfp(self, file):
-        self._file = file
-        self._version = _AIFC_version
-        self._comptype = b'NONE'
-        self._compname = b'not compressed'
-        self._convert = None
-        self._nchannels = 0
-        self._sampwidth = 0
-        self._framerate = 0
-        self._nframes = 0
-        self._nframeswritten = 0
-        self._datawritten = 0
-        self._datalength = 0
-        self._markers = []
-        self._marklength = 0
-        self._aifc = 1      # AIFF-C is default
-
-    def __del__(self):
-        self.close()
-
-    def __enter__(self):
-        return self
-
-    def __exit__(self, *args):
-        self.close()
-
-    #
-    # User visible methods.
-    #
-    def aiff(self):
-        if self._nframeswritten:
-            raise Error('cannot change parameters after starting to write')
-        self._aifc = 0
-
-    def aifc(self):
-        if self._nframeswritten:
-            raise Error('cannot change parameters after starting to write')
-        self._aifc = 1
-
-    def setnchannels(self, nchannels):
-        if self._nframeswritten:
-            raise Error('cannot change parameters after starting to write')
-        if nchannels < 1:
-            raise Error('bad # of channels')
-        self._nchannels = nchannels
-
-    def getnchannels(self):
-        if not self._nchannels:
-            raise Error('number of channels not set')
-        return self._nchannels
-
-    def setsampwidth(self, sampwidth):
-        if self._nframeswritten:
-            raise Error('cannot change parameters after starting to write')
-        if sampwidth < 1 or sampwidth > 4:
-            raise Error('bad sample width')
-        self._sampwidth = sampwidth
-
-    def getsampwidth(self):
-        if not self._sampwidth:
-            raise Error('sample width not set')
-        return self._sampwidth
-
-    def setframerate(self, framerate):
-        if self._nframeswritten:
-            raise Error('cannot change parameters after starting to write')
-        if framerate <= 0:
-            raise Error('bad frame rate')
-        self._framerate = framerate
-
-    def getframerate(self):
-        if not self._framerate:
-            raise Error('frame rate not set')
-        return self._framerate
-
-    def setnframes(self, nframes):
-        if self._nframeswritten:
-            raise Error('cannot change parameters after starting to write')
-        self._nframes = nframes
-
-    def getnframes(self):
-        return self._nframeswritten
-
-    def setcomptype(self, comptype, compname):
-        if self._nframeswritten:
-            raise Error('cannot change parameters after starting to write')
-        if comptype not in (b'NONE', b'ulaw', b'ULAW',
-                            b'alaw', b'ALAW', b'G722', b'sowt', b'SOWT'):
-            raise Error('unsupported compression type')
-        self._comptype = comptype
-        self._compname = compname
-
-    def getcomptype(self):
-        return self._comptype
-
-    def getcompname(self):
-        return self._compname
-
-##  def setversion(self, version):
-##      if self._nframeswritten:
-##          raise Error, 'cannot change parameters after starting to write'
-##      self._version = version
-
-    def setparams(self, params):
-        nchannels, sampwidth, framerate, nframes, comptype, compname = params
-        if self._nframeswritten:
-            raise Error('cannot change parameters after starting to write')
-        if comptype not in (b'NONE', b'ulaw', b'ULAW',
-                            b'alaw', b'ALAW', b'G722', b'sowt', b'SOWT'):
-            raise Error('unsupported compression type')
-        self.setnchannels(nchannels)
-        self.setsampwidth(sampwidth)
-        self.setframerate(framerate)
-        self.setnframes(nframes)
-        self.setcomptype(comptype, compname)
-
-    def getparams(self):
-        if not self._nchannels or not self._sampwidth or not self._framerate:
-            raise Error('not all parameters set')
-        return _aifc_params(self._nchannels, self._sampwidth, self._framerate,
-                            self._nframes, self._comptype, self._compname)
-
-    def setmark(self, id, pos, name):
-        if id <= 0:
-            raise Error('marker ID must be > 0')
-        if pos < 0:
-            raise Error('marker position must be >= 0')
-        if not isinstance(name, bytes):
-            raise Error('marker name must be bytes')
-        for i in range(len(self._markers)):
-            if id == self._markers[i][0]:
-                self._markers[i] = id, pos, name
-                return
-        self._markers.append((id, pos, name))
-
-    def getmark(self, id):
-        for marker in self._markers:
-            if id == marker[0]:
-                return marker
-        raise Error('marker {0!r} does not exist'.format(id))
-
-    def getmarkers(self):
-        if len(self._markers) == 0:
-            return None
-        return self._markers
-
-    def tell(self):
-        return self._nframeswritten
-
-    def writeframesraw(self, data):
-        if not isinstance(data, (bytes, bytearray)):
-            data = memoryview(data).cast('B')
-        self._ensure_header_written(len(data))
-        nframes = len(data) // (self._sampwidth * self._nchannels)
-        if self._convert:
-            data = self._convert(data)
-        self._file.write(data)
-        self._nframeswritten = self._nframeswritten + nframes
-        self._datawritten = self._datawritten + len(data)
-
-    def writeframes(self, data):
-        self.writeframesraw(data)
-        if self._nframeswritten != self._nframes or \
-              self._datalength != self._datawritten:
-            self._patchheader()
-
-    def close(self):
-        if self._file is None:
-            return
-        try:
-            self._ensure_header_written(0)
-            if self._datawritten & 1:
-                # quick pad to even size
-                self._file.write(b'\x00')
-                self._datawritten = self._datawritten + 1
-            self._writemarkers()
-            if self._nframeswritten != self._nframes or \
-                  self._datalength != self._datawritten or \
-                  self._marklength:
-                self._patchheader()
-        finally:
-            # Prevent ref cycles
-            self._convert = None
-            f = self._file
-            self._file = None
-            f.close()
-
-    #
-    # Internal methods.
-    #
-
-    def _lin2alaw(self, data):
-        with warnings.catch_warnings():
-            warnings.simplefilter('ignore', category=DeprecationWarning)
-            import audioop
-        return audioop.lin2alaw(data, 2)
-
-    def _lin2ulaw(self, data):
-        with warnings.catch_warnings():
-            warnings.simplefilter('ignore', category=DeprecationWarning)
-            import audioop
-        return audioop.lin2ulaw(data, 2)
-
-    def _lin2adpcm(self, data):
-        with warnings.catch_warnings():
-            warnings.simplefilter('ignore', category=DeprecationWarning)
-            import audioop
-        if not hasattr(self, '_adpcmstate'):
-            self._adpcmstate = None
-        data, self._adpcmstate = audioop.lin2adpcm(data, 2, self._adpcmstate)
-        return data
-
-    def _lin2sowt(self, data):
-        with warnings.catch_warnings():
-            warnings.simplefilter('ignore', category=DeprecationWarning)
-            import audioop
-        return audioop.byteswap(data, 2)
-
-    def _ensure_header_written(self, datasize):
-        if not self._nframeswritten:
-            if self._comptype in (b'ULAW', b'ulaw',
-                b'ALAW', b'alaw', b'G722',
-                b'sowt', b'SOWT'):
-                if not self._sampwidth:
-                    self._sampwidth = 2
-                if self._sampwidth != 2:
-                    raise Error('sample width must be 2 when compressing '
-                                'with ulaw/ULAW, alaw/ALAW, sowt/SOWT '
-                                'or G7.22 (ADPCM)')
-            if not self._nchannels:
-                raise Error('# channels not specified')
-            if not self._sampwidth:
-                raise Error('sample width not specified')
-            if not self._framerate:
-                raise Error('sampling rate not specified')
-            self._write_header(datasize)
-
-    def _init_compression(self):
-        if self._comptype == b'G722':
-            self._convert = self._lin2adpcm
-        elif self._comptype in (b'ulaw', b'ULAW'):
-            self._convert = self._lin2ulaw
-        elif self._comptype in (b'alaw', b'ALAW'):
-            self._convert = self._lin2alaw
-        elif self._comptype in (b'sowt', b'SOWT'):
-            self._convert = self._lin2sowt
-
-    def _write_header(self, initlength):
-        if self._aifc and self._comptype != b'NONE':
-            self._init_compression()
-        self._file.write(b'FORM')
-        if not self._nframes:
-            self._nframes = initlength // (self._nchannels * self._sampwidth)
-        self._datalength = self._nframes * self._nchannels * self._sampwidth
-        if self._datalength & 1:
-            self._datalength = self._datalength + 1
-        if self._aifc:
-            if self._comptype in (b'ulaw', b'ULAW', b'alaw', b'ALAW'):
-                self._datalength = self._datalength // 2
-                if self._datalength & 1:
-                    self._datalength = self._datalength + 1
-            elif self._comptype == b'G722':
-                self._datalength = (self._datalength + 3) // 4
-                if self._datalength & 1:
-                    self._datalength = self._datalength + 1
-        try:
-            self._form_length_pos = self._file.tell()
-        except (AttributeError, OSError):
-            self._form_length_pos = None
-        commlength = self._write_form_length(self._datalength)
-        if self._aifc:
-            self._file.write(b'AIFC')
-            self._file.write(b'FVER')
-            _write_ulong(self._file, 4)
-            _write_ulong(self._file, self._version)
-        else:
-            self._file.write(b'AIFF')
-        self._file.write(b'COMM')
-        _write_ulong(self._file, commlength)
-        _write_short(self._file, self._nchannels)
-        if self._form_length_pos is not None:
-            self._nframes_pos = self._file.tell()
-        _write_ulong(self._file, self._nframes)
-        if self._comptype in (b'ULAW', b'ulaw', b'ALAW', b'alaw', b'G722'):
-            _write_short(self._file, 8)
-        else:
-            _write_short(self._file, self._sampwidth * 8)
-        _write_float(self._file, self._framerate)
-        if self._aifc:
-            self._file.write(self._comptype)
-            _write_string(self._file, self._compname)
-        self._file.write(b'SSND')
-        if self._form_length_pos is not None:
-            self._ssnd_length_pos = self._file.tell()
-        _write_ulong(self._file, self._datalength + 8)
-        _write_ulong(self._file, 0)
-        _write_ulong(self._file, 0)
-
-    def _write_form_length(self, datalength):
-        if self._aifc:
-            commlength = 18 + 5 + len(self._compname)
-            if commlength & 1:
-                commlength = commlength + 1
-            verslength = 12
-        else:
-            commlength = 18
-            verslength = 0
-        _write_ulong(self._file, 4 + verslength + self._marklength + \
-                     8 + commlength + 16 + datalength)
-        return commlength
-
-    def _patchheader(self):
-        curpos = self._file.tell()
-        if self._datawritten & 1:
-            datalength = self._datawritten + 1
-            self._file.write(b'\x00')
-        else:
-            datalength = self._datawritten
-        if datalength == self._datalength and \
-              self._nframes == self._nframeswritten and \
-              self._marklength == 0:
-            self._file.seek(curpos, 0)
-            return
-        self._file.seek(self._form_length_pos, 0)
-        dummy = self._write_form_length(datalength)
-        self._file.seek(self._nframes_pos, 0)
-        _write_ulong(self._file, self._nframeswritten)
-        self._file.seek(self._ssnd_length_pos, 0)
-        _write_ulong(self._file, datalength + 8)
-        self._file.seek(curpos, 0)
-        self._nframes = self._nframeswritten
-        self._datalength = datalength
-
-    def _writemarkers(self):
-        if len(self._markers) == 0:
-            return
-        self._file.write(b'MARK')
-        length = 2
-        for marker in self._markers:
-            id, pos, name = marker
-            length = length + len(name) + 1 + 6
-            if len(name) & 1 == 0:
-                length = length + 1
-        _write_ulong(self._file, length)
-        self._marklength = length + 8
-        _write_short(self._file, len(self._markers))
-        for marker in self._markers:
-            id, pos, name = marker
-            _write_short(self._file, id)
-            _write_ulong(self._file, pos)
-            _write_string(self._file, name)
-
-def open(f, mode=None):
-    if mode is None:
-        if hasattr(f, 'mode'):
-            mode = f.mode
-        else:
-            mode = 'rb'
-    if mode in ('r', 'rb'):
-        return Aifc_read(f)
-    elif mode in ('w', 'wb'):
-        return Aifc_write(f)
-    else:
-        raise Error("mode must be 'r', 'rb', 'w', or 'wb'")
-
-
-if __name__ == '__main__':
-    import sys
-    if not sys.argv[1:]:
-        sys.argv.append('/usr/demos/data/audio/bach.aiff')
-    fn = sys.argv[1]
-    with open(fn, 'r') as f:
-        print("Reading", fn)
-        print("nchannels =", f.getnchannels())
-        print("nframes   =", f.getnframes())
-        print("sampwidth =", f.getsampwidth())
-        print("framerate =", f.getframerate())
-        print("comptype  =", f.getcomptype())
-        print("compname  =", f.getcompname())
-        if sys.argv[2:]:
-            gn = sys.argv[2]
-            print("Writing", gn)
-            with open(gn, 'w') as g:
-                g.setparams(f.getparams())
-                while 1:
-                    data = f.readframes(1024)
-                    if not data:
-                        break
-                    g.writeframes(data)
-            print("Done.")

Lib/asynchat.py

@@ -1,307 +0,0 @@
-# -*- Mode: Python; tab-width: 4 -*-
-#       Id: asynchat.py,v 2.26 2000/09/07 22:29:26 rushing Exp
-#       Author: Sam Rushing <rushing@nightmare.com>
-
-# ======================================================================
-# Copyright 1996 by Sam Rushing
-#
-#                         All Rights Reserved
-#
-# Permission to use, copy, modify, and distribute this software and
-# its documentation for any purpose and without fee is hereby
-# granted, provided that the above copyright notice appear in all
-# copies and that both that copyright notice and this permission
-# notice appear in supporting documentation, and that the name of Sam
-# Rushing not be used in advertising or publicity pertaining to
-# distribution of the software without specific, written prior
-# permission.
-#
-# SAM RUSHING DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE,
-# INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS, IN
-# NO EVENT SHALL SAM RUSHING BE LIABLE FOR ANY SPECIAL, INDIRECT OR
-# CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS
-# OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT,
-# NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
-# CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
-# ======================================================================
-
-r"""A class supporting chat-style (command/response) protocols.
-
-This class adds support for 'chat' style protocols - where one side
-sends a 'command', and the other sends a response (examples would be
-the common internet protocols - smtp, nntp, ftp, etc..).
-
-The handle_read() method looks at the input stream for the current
-'terminator' (usually '\r\n' for single-line responses, '\r\n.\r\n'
-for multi-line output), calling self.found_terminator() on its
-receipt.
-
-for example:
-Say you build an async nntp client using this class.  At the start
-of the connection, you'll have self.terminator set to '\r\n', in
-order to process the single-line greeting.  Just before issuing a
-'LIST' command you'll set it to '\r\n.\r\n'.  The output of the LIST
-command will be accumulated (using your own 'collect_incoming_data'
-method) up to the terminator, and then control will be returned to
-you - by calling your self.found_terminator() method.
-"""
-import asyncore
-from collections import deque
-
-
-class async_chat(asyncore.dispatcher):
-    """This is an abstract class.  You must derive from this class, and add
-    the two methods collect_incoming_data() and found_terminator()"""
-
-    # these are overridable defaults
-
-    ac_in_buffer_size = 65536
-    ac_out_buffer_size = 65536
-
-    # we don't want to enable the use of encoding by default, because that is a
-    # sign of an application bug that we don't want to pass silently
-
-    use_encoding = 0
-    encoding = 'latin-1'
-
-    def __init__(self, sock=None, map=None):
-        # for string terminator matching
-        self.ac_in_buffer = b''
-
-        # we use a list here rather than io.BytesIO for a few reasons...
-        # del lst[:] is faster than bio.truncate(0)
-        # lst = [] is faster than bio.truncate(0)
-        self.incoming = []
-
-        # we toss the use of the "simple producer" and replace it with
-        # a pure deque, which the original fifo was a wrapping of
-        self.producer_fifo = deque()
-        asyncore.dispatcher.__init__(self, sock, map)
-
-    def collect_incoming_data(self, data):
-        raise NotImplementedError("must be implemented in subclass")
-
-    def _collect_incoming_data(self, data):
-        self.incoming.append(data)
-
-    def _get_data(self):
-        d = b''.join(self.incoming)
-        del self.incoming[:]
-        return d
-
-    def found_terminator(self):
-        raise NotImplementedError("must be implemented in subclass")
-
-    def set_terminator(self, term):
-        """Set the input delimiter.
-
-        Can be a fixed string of any length, an integer, or None.
-        """
-        if isinstance(term, str) and self.use_encoding:
-            term = bytes(term, self.encoding)
-        elif isinstance(term, int) and term < 0:
-            raise ValueError('the number of received bytes must be positive')
-        self.terminator = term
-
-    def get_terminator(self):
-        return self.terminator
-
-    # grab some more data from the socket,
-    # throw it to the collector method,
-    # check for the terminator,
-    # if found, transition to the next state.
-
-    def handle_read(self):
-
-        try:
-            data = self.recv(self.ac_in_buffer_size)
-        except BlockingIOError:
-            return
-        except OSError as why:
-            self.handle_error()
-            return
-
-        if isinstance(data, str) and self.use_encoding:
-            data = bytes(str, self.encoding)
-        self.ac_in_buffer = self.ac_in_buffer + data
-
-        # Continue to search for self.terminator in self.ac_in_buffer,
-        # while calling self.collect_incoming_data.  The while loop
-        # is necessary because we might read several data+terminator
-        # combos with a single recv(4096).
-
-        while self.ac_in_buffer:
-            lb = len(self.ac_in_buffer)
-            terminator = self.get_terminator()
-            if not terminator:
-                # no terminator, collect it all
-                self.collect_incoming_data(self.ac_in_buffer)
-                self.ac_in_buffer = b''
-            elif isinstance(terminator, int):
-                # numeric terminator
-                n = terminator
-                if lb < n:
-                    self.collect_incoming_data(self.ac_in_buffer)
-                    self.ac_in_buffer = b''
-                    self.terminator = self.terminator - lb
-                else:
-                    self.collect_incoming_data(self.ac_in_buffer[:n])
-                    self.ac_in_buffer = self.ac_in_buffer[n:]
-                    self.terminator = 0
-                    self.found_terminator()
-            else:
-                # 3 cases:
-                # 1) end of buffer matches terminator exactly:
-                #    collect data, transition
-                # 2) end of buffer matches some prefix:
-                #    collect data to the prefix
-                # 3) end of buffer does not match any prefix:
-                #    collect data
-                terminator_len = len(terminator)
-                index = self.ac_in_buffer.find(terminator)
-                if index != -1:
-                    # we found the terminator
-                    if index > 0:
-                        # don't bother reporting the empty string
-                        # (source of subtle bugs)
-                        self.collect_incoming_data(self.ac_in_buffer[:index])
-                    self.ac_in_buffer = self.ac_in_buffer[index+terminator_len:]
-                    # This does the Right Thing if the terminator
-                    # is changed here.
-                    self.found_terminator()
-                else:
-                    # check for a prefix of the terminator
-                    index = find_prefix_at_end(self.ac_in_buffer, terminator)
-                    if index:
-                        if index != lb:
-                            # we found a prefix, collect up to the prefix
-                            self.collect_incoming_data(self.ac_in_buffer[:-index])
-                            self.ac_in_buffer = self.ac_in_buffer[-index:]
-                        break
-                    else:
-                        # no prefix, collect it all
-                        self.collect_incoming_data(self.ac_in_buffer)
-                        self.ac_in_buffer = b''
-
-    def handle_write(self):
-        self.initiate_send()
-
-    def handle_close(self):
-        self.close()
-
-    def push(self, data):
-        if not isinstance(data, (bytes, bytearray, memoryview)):
-            raise TypeError('data argument must be byte-ish (%r)',
-                            type(data))
-        sabs = self.ac_out_buffer_size
-        if len(data) > sabs:
-            for i in range(0, len(data), sabs):
-                self.producer_fifo.append(data[i:i+sabs])
-        else:
-            self.producer_fifo.append(data)
-        self.initiate_send()
-
-    def push_with_producer(self, producer):
-        self.producer_fifo.append(producer)
-        self.initiate_send()
-
-    def readable(self):
-        "predicate for inclusion in the readable for select()"
-        # cannot use the old predicate, it violates the claim of the
-        # set_terminator method.
-
-        # return (len(self.ac_in_buffer) <= self.ac_in_buffer_size)
-        return 1
-
-    def writable(self):
-        "predicate for inclusion in the writable for select()"
-        return self.producer_fifo or (not self.connected)
-
-    def close_when_done(self):
-        "automatically close this channel once the outgoing queue is empty"
-        self.producer_fifo.append(None)
-
-    def initiate_send(self):
-        while self.producer_fifo and self.connected:
-            first = self.producer_fifo[0]
-            # handle empty string/buffer or None entry
-            if not first:
-                del self.producer_fifo[0]
-                if first is None:
-                    self.handle_close()
-                    return
-
-            # handle classic producer behavior
-            obs = self.ac_out_buffer_size
-            try:
-                data = first[:obs]
-            except TypeError:
-                data = first.more()
-                if data:
-                    self.producer_fifo.appendleft(data)
-                else:
-                    del self.producer_fifo[0]
-                continue
-
-            if isinstance(data, str) and self.use_encoding:
-                data = bytes(data, self.encoding)
-
-            # send the data
-            try:
-                num_sent = self.send(data)
-            except OSError:
-                self.handle_error()
-                return
-
-            if num_sent:
-                if num_sent < len(data) or obs < len(first):
-                    self.producer_fifo[0] = first[num_sent:]
-                else:
-                    del self.producer_fifo[0]
-            # we tried to send some actual data
-            return
-
-    def discard_buffers(self):
-        # Emergencies only!
-        self.ac_in_buffer = b''
-        del self.incoming[:]
-        self.producer_fifo.clear()
-
-
-class simple_producer:
-
-    def __init__(self, data, buffer_size=512):
-        self.data = data
-        self.buffer_size = buffer_size
-
-    def more(self):
-        if len(self.data) > self.buffer_size:
-            result = self.data[:self.buffer_size]
-            self.data = self.data[self.buffer_size:]
-            return result
-        else:
-            result = self.data
-            self.data = b''
-            return result
-
-
-# Given 'haystack', see if any prefix of 'needle' is at its end.  This
-# assumes an exact match has already been checked.  Return the number of
-# characters matched.
-# for example:
-# f_p_a_e("qwerty\r", "\r\n") => 1
-# f_p_a_e("qwertydkjf", "\r\n") => 0
-# f_p_a_e("qwerty\r\n", "\r\n") => <undefined>
-
-# this could maybe be made faster with a computed regex?
-# [answer: no; circa Python-2.0, Jan 2001]
-# new python:   28961/s
-# old python:   18307/s
-# re:        12820/s
-# regex:     14035/s
-
-def find_prefix_at_end(haystack, needle):
-    l = len(needle) - 1
-    while l and not haystack.endswith(needle[:l]):
-        l -= 1
-    return l

Lib/asyncio/__init__.py

@@ -10,6 +10,7 @@ from .coroutines import *
 from .events import *
 from .exceptions import *
 from .futures import *
+from .graph import *
 from .locks import *
 from .protocols import *
 from .runners import *
@@ -27,6 +28,7 @@ __all__ = (base_events.__all__ +
            events.__all__ +
            exceptions.__all__ +
            futures.__all__ +
+           graph.__all__ +
            locks.__all__ +
            protocols.__all__ +
            runners.__all__ +
@@ -45,3 +47,28 @@ if sys.platform == 'win32':  # pragma: no cover
 else:
     from .unix_events import *  # pragma: no cover
     __all__ += unix_events.__all__
+
+def __getattr__(name: str):
+    import warnings
+
+    match name:
+        case "AbstractEventLoopPolicy":
+            warnings._deprecated(f"asyncio.{name}", remove=(3, 16))
+            return events._AbstractEventLoopPolicy
+        case "DefaultEventLoopPolicy":
+            warnings._deprecated(f"asyncio.{name}", remove=(3, 16))
+            if sys.platform == 'win32':
+                return windows_events._DefaultEventLoopPolicy
+            return unix_events._DefaultEventLoopPolicy
+        case "WindowsSelectorEventLoopPolicy":
+            if sys.platform == 'win32':
+                warnings._deprecated(f"asyncio.{name}", remove=(3, 16))
+                return windows_events._WindowsSelectorEventLoopPolicy
+            # Else fall through to the AttributeError below.
+        case "WindowsProactorEventLoopPolicy":
+            if sys.platform == 'win32':
+                warnings._deprecated(f"asyncio.{name}", remove=(3, 16))
+                return windows_events._WindowsProactorEventLoopPolicy
+            # Else fall through to the AttributeError below.
+
+    raise AttributeError(f"module {__name__!r} has no attribute {name!r}")

Lib/asyncio/__main__.py

@@ -1,41 +1,53 @@
+import argparse
 import ast
 import asyncio
-import code
+import asyncio.tools
 import concurrent.futures
+import contextvars
 import inspect
+import os
+import site
 import sys
 import threading
 import types
 import warnings
 
+from _colorize import get_theme
+from _pyrepl.console import InteractiveColoredConsole
+
 from . import futures
 
 
-class AsyncIOInteractiveConsole(code.InteractiveConsole):
+class AsyncIOInteractiveConsole(InteractiveColoredConsole):
 
     def __init__(self, locals, loop):
-        super().__init__(locals)
+        super().__init__(locals, filename="<stdin>")
         self.compile.compiler.flags |= ast.PyCF_ALLOW_TOP_LEVEL_AWAIT
 
         self.loop = loop
+        self.context = contextvars.copy_context()
 
     def runcode(self, code):
+        global return_code
         future = concurrent.futures.Future()
 
         def callback():
+            global return_code
             global repl_future
-            global repl_future_interrupted
+            global keyboard_interrupted
 
             repl_future = None
-            repl_future_interrupted = False
+            keyboard_interrupted = False
 
             func = types.FunctionType(code, self.locals)
             try:
                 coro = func()
-            except SystemExit:
-                raise
+            except SystemExit as se:
+                return_code = se.code
+                self.loop.stop()
+                return
             except KeyboardInterrupt as ex:
-                repl_future_interrupted = True
+                keyboard_interrupted = True
                 future.set_exception(ex)
                 return
             except BaseException as ex:
@@ -47,39 +59,72 @@ class AsyncIOInteractiveConsole(code.InteractiveConsole):
                 return
 
             try:
-                repl_future = self.loop.create_task(coro)
+                repl_future = self.loop.create_task(coro, context=self.context)
                 futures._chain_future(repl_future, future)
             except BaseException as exc:
                 future.set_exception(exc)
 
-        loop.call_soon_threadsafe(callback)
+        self.loop.call_soon_threadsafe(callback, context=self.context)
 
         try:
             return future.result()
-        except SystemExit:
-            raise
+        except SystemExit as se:
+            return_code = se.code
+            self.loop.stop()
+            return
         except BaseException:
-            if repl_future_interrupted:
-                self.write("\nKeyboardInterrupt\n")
+            if keyboard_interrupted:
+                if not CAN_USE_PYREPL:
+                    self.write("\nKeyboardInterrupt\n")
             else:
                 self.showtraceback()
-
+            return self.STATEMENT_FAILED
 
 class REPLThread(threading.Thread):
 
     def run(self):
+        global return_code
+
         try:
             banner = (
                 f'asyncio REPL {sys.version} on {sys.platform}\n'
                 f'Use "await" directly instead of "asyncio.run()".\n'
                 f'Type "help", "copyright", "credits" or "license" '
                 f'for more information.\n'
-                f'{getattr(sys, "ps1", ">>> ")}import asyncio'
             )
 
-            console.interact(
-                banner=banner,
-                exitmsg='exiting asyncio REPL...')
+            console.write(banner)
+
+            if startup_path := os.getenv("PYTHONSTARTUP"):
+                sys.audit("cpython.run_startup", startup_path)
+
+                import tokenize
+                with tokenize.open(startup_path) as f:
+                    startup_code = compile(f.read(), startup_path, "exec")
+                    exec(startup_code, console.locals)
+
+            ps1 = getattr(sys, "ps1", ">>> ")
+            if CAN_USE_PYREPL:
+                theme = get_theme().syntax
+                ps1 = f"{theme.prompt}{ps1}{theme.reset}"
+            console.write(f"{ps1}import asyncio\n")
+
+            if CAN_USE_PYREPL:
+                from _pyrepl.simple_interact import (
+                    run_multiline_interactive_console,
+                )
+                try:
+                    run_multiline_interactive_console(console)
+                except SystemExit:
+                    # expected via the `exit` and `quit` commands
+                    pass
+                except BaseException:
+                    # unexpected issue
+                    console.showtraceback()
+                    console.write("Internal error, ")
+                    return_code = 1
+            else:
+                console.interact(banner="", exitmsg="")
         finally:
             warnings.filterwarnings(
                 'ignore',
@@ -88,8 +133,56 @@ class REPLThread(threading.Thread):
 
             loop.call_soon_threadsafe(loop.stop)
 
+    def interrupt(self) -> None:
+        if not CAN_USE_PYREPL:
+            return
+
+        from _pyrepl.simple_interact import _get_reader
+        r = _get_reader()
+        if r.threading_hook is not None:
+            r.threading_hook.add("")  # type: ignore
+
 
 if __name__ == '__main__':
+    parser = argparse.ArgumentParser(
+        prog="python3 -m asyncio",
+        description="Interactive asyncio shell and CLI tools",
+        color=True,
+    )
+    subparsers = parser.add_subparsers(help="sub-commands", dest="command")
+    ps = subparsers.add_parser(
+        "ps", help="Display a table of all pending tasks in a process"
+    )
+    ps.add_argument("pid", type=int, help="Process ID to inspect")
+    pstree = subparsers.add_parser(
+        "pstree", help="Display a tree of all pending tasks in a process"
+    )
+    pstree.add_argument("pid", type=int, help="Process ID to inspect")
+    args = parser.parse_args()
+    match args.command:
+        case "ps":
+            asyncio.tools.display_awaited_by_tasks_table(args.pid)
+            sys.exit(0)
+        case "pstree":
+            asyncio.tools.display_awaited_by_tasks_tree(args.pid)
+            sys.exit(0)
+        case None:
+            pass  # continue to the interactive shell
+        case _:
+            # shouldn't happen as an invalid command-line wouldn't parse
+            # but let's keep it for the next person adding a command
+            print(f"error: unhandled command {args.command}", file=sys.stderr)
+            parser.print_usage(file=sys.stderr)
+            sys.exit(1)
+
+    sys.audit("cpython.run_stdin")
+
+    if os.getenv('PYTHON_BASIC_REPL'):
+        CAN_USE_PYREPL = False
+    else:
+        from _pyrepl.main import CAN_USE_PYREPL
+
+    return_code = 0
     loop = asyncio.new_event_loop()
     asyncio.set_event_loop(loop)
 
@@ -102,14 +195,31 @@ if __name__ == '__main__':
     console = AsyncIOInteractiveConsole(repl_locals, loop)
 
     repl_future = None
-    repl_future_interrupted = False
+    keyboard_interrupted = False
 
     try:
         import readline  # NoQA
     except ImportError:
-        pass
+        readline = None
 
-    repl_thread = REPLThread()
+    interactive_hook = getattr(sys, "__interactivehook__", None)
+
+    if interactive_hook is not None:
+        sys.audit("cpython.run_interactivehook", interactive_hook)
+        interactive_hook()
+
+    if interactive_hook is site.register_readline:
+        # Fix the completer function to use the interactive console locals
+        try:
+            import rlcompleter
+        except:
+            pass
+        else:
+            if readline is not None:
+                completer = rlcompleter.Completer(console.locals)
+                readline.set_completer(completer.complete)
+
+    repl_thread = REPLThread(name="Interactive thread")
     repl_thread.daemon = True
     repl_thread.start()
 
@@ -117,9 +227,13 @@ if __name__ == '__main__':
         try:
             loop.run_forever()
         except KeyboardInterrupt:
+            keyboard_interrupted = True
             if repl_future and not repl_future.done():
                 repl_future.cancel()
-                repl_future_interrupted = True
+            repl_thread.interrupt()
             continue
         else:
             break
+
+    console.write('exiting asyncio REPL...\n')
+    sys.exit(return_code)

Lib/asyncio/base_events.py

@@ -17,7 +17,6 @@ import collections
 import collections.abc
 import concurrent.futures
 import errno
-import functools
 import heapq
 import itertools
 import os
@@ -279,7 +278,9 @@ class Server(events.AbstractServer):
                  ssl_handshake_timeout, ssl_shutdown_timeout=None):
         self._loop = loop
         self._sockets = sockets
-        self._active_count = 0
+        # Weak references so we don't break Transport's ability to
+        # detect abandoned transports
+        self._clients = weakref.WeakSet()
         self._waiters = []
         self._protocol_factory = protocol_factory
         self._backlog = backlog
@@ -292,14 +293,13 @@ class Server(events.AbstractServer):
     def __repr__(self):
         return f'<{self.__class__.__name__} sockets={self.sockets!r}>'
 
-    def _attach(self):
+    def _attach(self, transport):
         assert self._sockets is not None
-        self._active_count += 1
+        self._clients.add(transport)
 
-    def _detach(self):
-        assert self._active_count > 0
-        self._active_count -= 1
-        if self._active_count == 0 and self._sockets is None:
+    def _detach(self, transport):
+        self._clients.discard(transport)
+        if len(self._clients) == 0 and self._sockets is None:
             self._wakeup()
 
     def _wakeup(self):
@@ -348,9 +348,17 @@ class Server(events.AbstractServer):
             self._serving_forever_fut.cancel()
             self._serving_forever_fut = None
 
-        if self._active_count == 0:
+        if len(self._clients) == 0:
             self._wakeup()
 
+    def close_clients(self):
+        for transport in self._clients.copy():
+            transport.close()
+
+    def abort_clients(self):
+        for transport in self._clients.copy():
+            transport.abort()
+
     async def start_serving(self):
         self._start_serving()
         # Skip one loop iteration so that all 'loop.add_reader'
@@ -422,6 +430,8 @@ class BaseEventLoop(events.AbstractEventLoop):
         self._clock_resolution = time.get_clock_info('monotonic').resolution
         self._exception_handler = None
         self.set_debug(coroutines._is_debug_mode())
+        # The preserved state of async generator hooks.
+        self._old_agen_hooks = None
         # In debug mode, if the execution of a callback or a step of a task
         # exceed this duration in seconds, the slow callback/task is logged.
         self.slow_callback_duration = 0.1
@@ -448,26 +458,24 @@ class BaseEventLoop(events.AbstractEventLoop):
         """Create a Future object attached to the loop."""
         return futures.Future(loop=self)
 
-    def create_task(self, coro, *, name=None, context=None):
-        """Schedule a coroutine object.
+    def create_task(self, coro, **kwargs):
+        """Schedule or begin executing a coroutine object.
 
         Return a task object.
         """
         self._check_closed()
-        if self._task_factory is None:
-            task = tasks.Task(coro, loop=self, name=name, context=context)
-            if task._source_traceback:
-                del task._source_traceback[-1]
-        else:
-            if context is None:
-                # Use legacy API if context is not needed
-                task = self._task_factory(self, coro)
-            else:
-                task = self._task_factory(self, coro, context=context)
+        if self._task_factory is not None:
+            return self._task_factory(self, coro, **kwargs)
 
-            tasks._set_task_name(task, name)
-
-        return task
+        task = tasks.Task(coro, loop=self, **kwargs)
+        if task._source_traceback:
+            del task._source_traceback[-1]
+        try:
+            return task
+        finally:
+            # gh-128552: prevent a refcycle of
+            # task.exception().__traceback__->BaseEventLoop.create_task->task
+            del task
 
     def set_task_factory(self, factory):
         """Set a task factory that will be used by loop.create_task().
@@ -475,9 +483,10 @@ class BaseEventLoop(events.AbstractEventLoop):
         If factory is None the default task factory will be set.
 
         If factory is a callable, it should have a signature matching
-        '(loop, coro)', where 'loop' will be a reference to the active
-        event loop, 'coro' will be a coroutine object.  The callable
-        must return a Future.
+        '(loop, coro, **kwargs)', where 'loop' will be a reference to the active
+        event loop, 'coro' will be a coroutine object, and **kwargs will be
+        arbitrary keyword arguments that should be passed on to Task.
+        The callable must return a Task.
         """
         if factory is not None and not callable(factory):
             raise TypeError('task factory must be a callable or None')
@@ -624,29 +633,52 @@ class BaseEventLoop(events.AbstractEventLoop):
             raise RuntimeError(
                 'Cannot run the event loop while another loop is running')
 
-    def run_forever(self):
-        """Run until stop() is called."""
+    def _run_forever_setup(self):
+        """Prepare the run loop to process events.
+
+        This method exists so that custom event loop subclasses (e.g., event loops
+        that integrate a GUI event loop with Python's event loop) have access to all the
+        loop setup logic.
+        """
         self._check_closed()
         self._check_running()
         self._set_coroutine_origin_tracking(self._debug)
 
-        old_agen_hooks = sys.get_asyncgen_hooks()
-        try:
-            self._thread_id = threading.get_ident()
-            sys.set_asyncgen_hooks(firstiter=self._asyncgen_firstiter_hook,
-                                   finalizer=self._asyncgen_finalizer_hook)
+        self._old_agen_hooks = sys.get_asyncgen_hooks()
+        self._thread_id = threading.get_ident()
+        sys.set_asyncgen_hooks(
+            firstiter=self._asyncgen_firstiter_hook,
+            finalizer=self._asyncgen_finalizer_hook
+        )
 
-            events._set_running_loop(self)
+        events._set_running_loop(self)
+
+    def _run_forever_cleanup(self):
+        """Clean up after an event loop finishes the looping over events.
+
+        This method exists so that custom event loop subclasses (e.g., event loops
+        that integrate a GUI event loop with Python's event loop) have access to all the
+        loop cleanup logic.
+        """
+        self._stopping = False
+        self._thread_id = None
+        events._set_running_loop(None)
+        self._set_coroutine_origin_tracking(False)
+        # Restore any pre-existing async generator hooks.
+        if self._old_agen_hooks is not None:
+            sys.set_asyncgen_hooks(*self._old_agen_hooks)
+            self._old_agen_hooks = None
+
+    def run_forever(self):
+        """Run until stop() is called."""
+        self._run_forever_setup()
+        try:
             while True:
                 self._run_once()
                 if self._stopping:
                     break
         finally:
-            self._stopping = False
-            self._thread_id = None
-            events._set_running_loop(None)
-            self._set_coroutine_origin_tracking(False)
-            sys.set_asyncgen_hooks(*old_agen_hooks)
+            self._run_forever_cleanup()
 
     def run_until_complete(self, future):
         """Run until the Future is done.
@@ -803,7 +835,7 @@ class BaseEventLoop(events.AbstractEventLoop):
 
     def _check_callback(self, callback, method):
         if (coroutines.iscoroutine(callback) or
-                coroutines.iscoroutinefunction(callback)):
+                coroutines._iscoroutinefunction(callback)):
             raise TypeError(
                 f"coroutines cannot be used with {method}()")
         if not callable(callback):
@@ -840,7 +872,10 @@ class BaseEventLoop(events.AbstractEventLoop):
         self._check_closed()
         if self._debug:
             self._check_callback(callback, 'call_soon_threadsafe')
-        handle = self._call_soon(callback, args, context)
+        handle = events._ThreadSafeHandle(callback, args, self, context)
+        self._ready.append(handle)
+        if handle._source_traceback:
+            del handle._source_traceback[-1]
         if handle._source_traceback:
             del handle._source_traceback[-1]
         self._write_to_self()
@@ -981,39 +1016,43 @@ class BaseEventLoop(events.AbstractEventLoop):
         family, type_, proto, _, address = addr_info
         sock = None
         try:
-            sock = socket.socket(family=family, type=type_, proto=proto)
-            sock.setblocking(False)
-            if local_addr_infos is not None:
-                for lfamily, _, _, _, laddr in local_addr_infos:
-                    # skip local addresses of different family
-                    if lfamily != family:
-                        continue
-                    try:
-                        sock.bind(laddr)
-                        break
-                    except OSError as exc:
-                        msg = (
-                            f'error while attempting to bind on '
-                            f'address {laddr!r}: '
-                            f'{exc.strerror.lower()}'
-                        )
-                        exc = OSError(exc.errno, msg)
-                        my_exceptions.append(exc)
-                else:  # all bind attempts failed
-                    if my_exceptions:
-                        raise my_exceptions.pop()
-                    else:
-                        raise OSError(f"no matching local address with {family=} found")
-            await self.sock_connect(sock, address)
-            return sock
-        except OSError as exc:
-            my_exceptions.append(exc)
-            if sock is not None:
-                sock.close()
-            raise
+            try:
+                sock = socket.socket(family=family, type=type_, proto=proto)
+                sock.setblocking(False)
+                if local_addr_infos is not None:
+                    for lfamily, _, _, _, laddr in local_addr_infos:
+                        # skip local addresses of different family
+                        if lfamily != family:
+                            continue
+                        try:
+                            sock.bind(laddr)
+                            break
+                        except OSError as exc:
+                            msg = (
+                                f'error while attempting to bind on '
+                                f'address {laddr!r}: {str(exc).lower()}'
+                            )
+                            exc = OSError(exc.errno, msg)
+                            my_exceptions.append(exc)
+                    else:  # all bind attempts failed
+                        if my_exceptions:
+                            raise my_exceptions.pop()
+                        else:
+                            raise OSError(f"no matching local address with {family=} found")
+                await self.sock_connect(sock, address)
+                return sock
+            except OSError as exc:
+                my_exceptions.append(exc)
+                raise
         except:
             if sock is not None:
-                sock.close()
+                try:
+                    sock.close()
+                except OSError:
+                    # An error when closing a newly created socket is
+                    # not important, but it can overwrite more important
+                    # non-OSError error. So ignore it.
+                    pass
             raise
         finally:
             exceptions = my_exceptions = None
@@ -1107,11 +1146,18 @@ class BaseEventLoop(events.AbstractEventLoop):
                     except OSError:
                         continue
             else:  # using happy eyeballs
-                sock, _, _ = await staggered.staggered_race(
-                    (functools.partial(self._connect_sock,
-                                       exceptions, addrinfo, laddr_infos)
-                     for addrinfo in infos),
-                    happy_eyeballs_delay, loop=self)
+                sock = (await staggered.staggered_race(
+                    (
+                        # can't use functools.partial as it keeps a reference
+                        # to exceptions
+                        lambda addrinfo=addrinfo: self._connect_sock(
+                            exceptions, addrinfo, laddr_infos
+                        )
+                        for addrinfo in infos
+                    ),
+                    happy_eyeballs_delay,
+                    loop=self,
+                ))[0]  # can't use sock, _, _ as it keeks a reference to exceptions
 
             if sock is None:
                 exceptions = [exc for sub in exceptions for exc in sub]
@@ -1120,7 +1166,7 @@ class BaseEventLoop(events.AbstractEventLoop):
                         raise ExceptionGroup("create_connection failed", exceptions)
                     if len(exceptions) == 1:
                         raise exceptions[0]
-                    else:
+                    elif exceptions:
                         # If they all have the same str(), raise one.
                         model = str(exceptions[0])
                         if all(str(exc) == model for exc in exceptions):
@@ -1129,6 +1175,9 @@ class BaseEventLoop(events.AbstractEventLoop):
                         # the various error messages.
                         raise OSError('Multiple exceptions: {}'.format(
                             ', '.join(str(exc) for exc in exceptions)))
+                    else:
+                        # No exceptions were collected, raise a timeout error
+                        raise TimeoutError('create_connection failed')
                 finally:
                     exceptions = None
 
@@ -1254,8 +1303,8 @@ class BaseEventLoop(events.AbstractEventLoop):
                 read = await self.run_in_executor(None, file.readinto, view)
                 if not read:
                     return total_sent  # EOF
-                await proto.drain()
                 transp.write(view[:read])
+                await proto.drain()
                 total_sent += read
         finally:
             if total_sent > 0 and hasattr(file, 'seek'):
@@ -1474,6 +1523,7 @@ class BaseEventLoop(events.AbstractEventLoop):
             ssl=None,
             reuse_address=None,
             reuse_port=None,
+            keep_alive=None,
             ssl_handshake_timeout=None,
             ssl_shutdown_timeout=None,
             start_serving=True):
@@ -1545,8 +1595,13 @@ class BaseEventLoop(events.AbstractEventLoop):
                     if reuse_address:
                         sock.setsockopt(
                             socket.SOL_SOCKET, socket.SO_REUSEADDR, True)
-                    if reuse_port:
+                    # Since Linux 6.12.9, SO_REUSEPORT is not allowed
+                    # on other address families than AF_INET/AF_INET6.
+                    if reuse_port and af in (socket.AF_INET, socket.AF_INET6):
                         _set_reuseport(sock)
+                    if keep_alive:
+                        sock.setsockopt(
+                            socket.SOL_SOCKET, socket.SO_KEEPALIVE, True)
                     # Disable IPv4/IPv6 dual stack support (enabled by
                     # default on Linux) which makes a single socket
                     # listen on both address families.
@@ -1561,7 +1616,7 @@ class BaseEventLoop(events.AbstractEventLoop):
                     except OSError as err:
                         msg = ('error while attempting '
                                'to bind on address %r: %s'
-                               % (sa, err.strerror.lower()))
+                               % (sa, str(err).lower()))
                         if err.errno == errno.EADDRNOTAVAIL:
                             # Assume the family is not enabled (bpo-30945)
                             sockets.pop()
@@ -1619,8 +1674,7 @@ class BaseEventLoop(events.AbstractEventLoop):
             raise ValueError(
                 'ssl_shutdown_timeout is only meaningful with ssl')
 
-        if sock is not None:
-            _check_ssl_socket(sock)
+        _check_ssl_socket(sock)
 
         transport, protocol = await self._create_connection_transport(
             sock, protocol_factory, ssl, '', server_side=True,
@@ -1833,6 +1887,8 @@ class BaseEventLoop(events.AbstractEventLoop):
         - 'protocol' (optional): Protocol instance;
         - 'transport' (optional): Transport instance;
         - 'socket' (optional): Socket instance;
+        - 'source_traceback' (optional): Traceback of the source;
+        - 'handle_traceback' (optional): Traceback of the handle;
         - 'asyncgen' (optional): Asynchronous generator that caused
                                  the exception.
 
@@ -1943,8 +1999,11 @@ class BaseEventLoop(events.AbstractEventLoop):
             timeout = 0
         elif self._scheduled:
             # Compute the desired timeout.
-            when = self._scheduled[0]._when
-            timeout = min(max(0, when - self.time()), MAXIMUM_SELECT_TIMEOUT)
+            timeout = self._scheduled[0]._when - self.time()
+            if timeout > MAXIMUM_SELECT_TIMEOUT:
+                timeout = MAXIMUM_SELECT_TIMEOUT
+            elif timeout < 0:
+                timeout = 0
 
         event_list = self._selector.select(timeout)
         self._process_events(event_list)

Lib/asyncio/base_subprocess.py

@@ -1,6 +1,9 @@
 import collections
 import subprocess
 import warnings
+import os
+import signal
+import sys
 
 from . import protocols
 from . import transports
@@ -23,6 +26,7 @@ class BaseSubprocessTransport(transports.SubprocessTransport):
         self._pending_calls = collections.deque()
         self._pipes = {}
         self._finished = False
+        self._pipes_connected = False
 
         if stdin == subprocess.PIPE:
             self._pipes[0] = None
@@ -101,7 +105,12 @@ class BaseSubprocessTransport(transports.SubprocessTransport):
         for proto in self._pipes.values():
             if proto is None:
                 continue
-            proto.pipe.close()
+            # See gh-114177
+            # skip closing the pipe if loop is already closed
+            # this can happen e.g. when loop is closed immediately after
+            # process is killed
+            if self._loop and not self._loop.is_closed():
+                proto.pipe.close()
 
         if (self._proc is not None and
                 # has the child process finished?
@@ -115,7 +124,8 @@ class BaseSubprocessTransport(transports.SubprocessTransport):
 
             try:
                 self._proc.kill()
-            except ProcessLookupError:
+            except (ProcessLookupError, PermissionError):
+                # the process may have already exited or may be running setuid
                 pass
 
             # Don't clear the _proc reference yet: _post_init() may still run
@@ -141,17 +151,31 @@ class BaseSubprocessTransport(transports.SubprocessTransport):
         if self._proc is None:
             raise ProcessLookupError()
 
-    def send_signal(self, signal):
-        self._check_proc()
-        self._proc.send_signal(signal)
+    if sys.platform == 'win32':
+        def send_signal(self, signal):
+            self._check_proc()
+            self._proc.send_signal(signal)
 
-    def terminate(self):
-        self._check_proc()
-        self._proc.terminate()
+        def terminate(self):
+            self._check_proc()
+            self._proc.terminate()
 
-    def kill(self):
-        self._check_proc()
-        self._proc.kill()
+        def kill(self):
+            self._check_proc()
+            self._proc.kill()
+    else:
+        def send_signal(self, signal):
+            self._check_proc()
+            try:
+                os.kill(self._proc.pid, signal)
+            except ProcessLookupError:
+                pass
+
+        def terminate(self):
+            self.send_signal(signal.SIGTERM)
+
+        def kill(self):
+            self.send_signal(signal.SIGKILL)
 
     async def _connect_pipes(self, waiter):
         try:
@@ -190,6 +214,7 @@ class BaseSubprocessTransport(transports.SubprocessTransport):
         else:
             if waiter is not None and not waiter.cancelled():
                 waiter.set_result(None)
+            self._pipes_connected = True
 
     def _call(self, cb, *data):
         if self._pending_calls is not None:
@@ -233,6 +258,15 @@ class BaseSubprocessTransport(transports.SubprocessTransport):
         assert not self._finished
         if self._returncode is None:
             return
+        if not self._pipes_connected:
+            # self._pipes_connected can be False if not all pipes were connected
+            # because either the process failed to start or the self._connect_pipes task
+            # got cancelled. In this broken state we consider all pipes disconnected and
+            # to avoid hanging forever in self._wait as otherwise _exit_waiters
+            # would never be woken up, we wake them up here.
+            for waiter in self._exit_waiters:
+                if not waiter.cancelled():
+                    waiter.set_result(self._returncode)
         if all(p is not None and p.disconnected
                for p in self._pipes.values()):
             self._finished = True

Lib/asyncio/coroutines.py

@@ -18,7 +18,16 @@ _is_coroutine = object()
 
 
 def iscoroutinefunction(func):
+    import warnings
     """Return True if func is a decorated coroutine function."""
+    warnings._deprecated("asyncio.iscoroutinefunction",
+                         f"{warnings._DEPRECATED_MSG}; "
+                         "use inspect.iscoroutinefunction() instead",
+                         remove=(3,16))
+    return _iscoroutinefunction(func)
+
+
+def _iscoroutinefunction(func):
     return (inspect.iscoroutinefunction(func) or
             getattr(func, '_is_coroutine', None) is _is_coroutine)
 

Lib/asyncio/events.py

@@ -5,14 +5,18 @@
 # SPDX-FileCopyrightText: Copyright (c) 2015-2021 MagicStack Inc.  http://magic.io
 
 __all__ = (
-    'AbstractEventLoopPolicy',
-    'AbstractEventLoop', 'AbstractServer',
-    'Handle', 'TimerHandle',
-    'get_event_loop_policy', 'set_event_loop_policy',
-    'get_event_loop', 'set_event_loop', 'new_event_loop',
-    'get_child_watcher', 'set_child_watcher',
-    '_set_running_loop', 'get_running_loop',
-    '_get_running_loop',
+    "AbstractEventLoop",
+    "AbstractServer",
+    "Handle",
+    "TimerHandle",
+    "get_event_loop_policy",
+    "set_event_loop_policy",
+    "get_event_loop",
+    "set_event_loop",
+    "new_event_loop",
+    "_set_running_loop",
+    "get_running_loop",
+    "_get_running_loop",
 )
 
 import contextvars
@@ -22,6 +26,7 @@ import socket
 import subprocess
 import sys
 import threading
+import warnings
 
 from . import format_helpers
 
@@ -54,7 +59,8 @@ class Handle:
             info.append('cancelled')
         if self._callback is not None:
             info.append(format_helpers._format_callback_source(
-                self._callback, self._args))
+                self._callback, self._args,
+                debug=self._loop.get_debug()))
         if self._source_traceback:
             frame = self._source_traceback[-1]
             info.append(f'created at {frame[0]}:{frame[1]}')
@@ -90,7 +96,8 @@ class Handle:
             raise
         except BaseException as exc:
             cb = format_helpers._format_callback_source(
-                self._callback, self._args)
+                self._callback, self._args,
+                debug=self._loop.get_debug())
             msg = f'Exception in callback {cb}'
             context = {
                 'message': msg,
@@ -102,6 +109,34 @@ class Handle:
             self._loop.call_exception_handler(context)
         self = None  # Needed to break cycles when an exception occurs.
 
+# _ThreadSafeHandle is used for callbacks scheduled with call_soon_threadsafe
+# and is thread safe unlike Handle which is not thread safe.
+class _ThreadSafeHandle(Handle):
+
+    __slots__ = ('_lock',)
+
+    def __init__(self, callback, args, loop, context=None):
+        super().__init__(callback, args, loop, context)
+        self._lock = threading.RLock()
+
+    def cancel(self):
+        with self._lock:
+            return super().cancel()
+
+    def cancelled(self):
+        with self._lock:
+            return super().cancelled()
+
+    def _run(self):
+        # The event loop checks for cancellation without holding the lock
+        # It is possible that the handle is cancelled after the check
+        # but before the callback is called so check it again after acquiring
+        # the lock and return without calling the callback if it is cancelled.
+        with self._lock:
+            if self._cancelled:
+                return
+            return super()._run()
+
 
 class TimerHandle(Handle):
     """Object returned by timed callback registration methods."""
@@ -173,6 +208,14 @@ class AbstractServer:
         """Stop serving.  This leaves existing connections open."""
         raise NotImplementedError
 
+    def close_clients(self):
+        """Close all active connections."""
+        raise NotImplementedError
+
+    def abort_clients(self):
+        """Close all active connections immediately."""
+        raise NotImplementedError
+
     def get_loop(self):
         """Get the event loop the Server object is attached to."""
         raise NotImplementedError
@@ -282,7 +325,7 @@ class AbstractEventLoop:
 
     # Method scheduling a coroutine object: create a task.
 
-    def create_task(self, coro, *, name=None, context=None):
+    def create_task(self, coro, **kwargs):
         raise NotImplementedError
 
     # Methods for interacting with threads.
@@ -320,6 +363,7 @@ class AbstractEventLoop:
             *, family=socket.AF_UNSPEC,
             flags=socket.AI_PASSIVE, sock=None, backlog=100,
             ssl=None, reuse_address=None, reuse_port=None,
+            keep_alive=None,
             ssl_handshake_timeout=None,
             ssl_shutdown_timeout=None,
             start_serving=True):
@@ -358,6 +402,9 @@ class AbstractEventLoop:
         they all set this flag when being created. This option is not
         supported on Windows.
 
+        keep_alive set to True keeps connections active by enabling the
+        periodic transmission of messages.
+
         ssl_handshake_timeout is the time in seconds that an SSL server
         will wait for completion of the SSL handshake before aborting the
         connection. Default is 60s.
@@ -615,7 +662,7 @@ class AbstractEventLoop:
         raise NotImplementedError
 
 
-class AbstractEventLoopPolicy:
+class _AbstractEventLoopPolicy:
     """Abstract policy for accessing the event loop."""
 
     def get_event_loop(self):
@@ -638,18 +685,7 @@ class AbstractEventLoopPolicy:
         the current context, set_event_loop must be called explicitly."""
         raise NotImplementedError
 
-    # Child processes handling (Unix only).
-
-    def get_child_watcher(self):
-        "Get the watcher for child processes."
-        raise NotImplementedError
-
-    def set_child_watcher(self, watcher):
-        """Set the watcher for child processes."""
-        raise NotImplementedError
-
-
-class BaseDefaultEventLoopPolicy(AbstractEventLoopPolicy):
+class _BaseDefaultEventLoopPolicy(_AbstractEventLoopPolicy):
     """Default policy implementation for accessing the event loop.
 
     In this policy, each thread has its own event loop.  However, we
@@ -666,7 +702,6 @@ class BaseDefaultEventLoopPolicy(AbstractEventLoopPolicy):
 
     class _Local(threading.local):
         _loop = None
-        _set_called = False
 
     def __init__(self):
         self._local = self._Local()
@@ -676,28 +711,6 @@ class BaseDefaultEventLoopPolicy(AbstractEventLoopPolicy):
 
         Returns an instance of EventLoop or raises an exception.
         """
-        if (self._local._loop is None and
-                not self._local._set_called and
-                threading.current_thread() is threading.main_thread()):
-            stacklevel = 2
-            try:
-                f = sys._getframe(1)
-            except AttributeError:
-                pass
-            else:
-                # Move up the call stack so that the warning is attached
-                # to the line outside asyncio itself.
-                while f:
-                    module = f.f_globals.get('__name__')
-                    if not (module == 'asyncio' or module.startswith('asyncio.')):
-                        break
-                    f = f.f_back
-                    stacklevel += 1
-            import warnings
-            warnings.warn('There is no current event loop',
-                          DeprecationWarning, stacklevel=stacklevel)
-            self.set_event_loop(self.new_event_loop())
-
         if self._local._loop is None:
             raise RuntimeError('There is no current event loop in thread %r.'
                                % threading.current_thread().name)
@@ -706,7 +719,6 @@ class BaseDefaultEventLoopPolicy(AbstractEventLoopPolicy):
 
     def set_event_loop(self, loop):
         """Set the event loop."""
-        self._local._set_called = True
         if loop is not None and not isinstance(loop, AbstractEventLoop):
             raise TypeError(f"loop must be an instance of AbstractEventLoop or None, not '{type(loop).__name__}'")
         self._local._loop = loop
@@ -776,26 +788,35 @@ def _init_event_loop_policy():
     global _event_loop_policy
     with _lock:
         if _event_loop_policy is None:  # pragma: no branch
-            from . import DefaultEventLoopPolicy
-            _event_loop_policy = DefaultEventLoopPolicy()
+            if sys.platform == 'win32':
+                from .windows_events import _DefaultEventLoopPolicy
+            else:
+                from .unix_events import _DefaultEventLoopPolicy
+            _event_loop_policy = _DefaultEventLoopPolicy()
 
 
-def get_event_loop_policy():
+def _get_event_loop_policy():
     """Get the current event loop policy."""
     if _event_loop_policy is None:
         _init_event_loop_policy()
     return _event_loop_policy
 
+def get_event_loop_policy():
+    warnings._deprecated('asyncio.get_event_loop_policy', remove=(3, 16))
+    return _get_event_loop_policy()
 
-def set_event_loop_policy(policy):
+def _set_event_loop_policy(policy):
     """Set the current event loop policy.
 
     If policy is None, the default policy is restored."""
     global _event_loop_policy
-    if policy is not None and not isinstance(policy, AbstractEventLoopPolicy):
+    if policy is not None and not isinstance(policy, _AbstractEventLoopPolicy):
         raise TypeError(f"policy must be an instance of AbstractEventLoopPolicy or None, not '{type(policy).__name__}'")
     _event_loop_policy = policy
 
+def set_event_loop_policy(policy):
+    warnings._deprecated('asyncio.set_event_loop_policy', remove=(3,16))
+    _set_event_loop_policy(policy)
 
 def get_event_loop():
     """Return an asyncio event loop.
@@ -810,28 +831,17 @@ def get_event_loop():
     current_loop = _get_running_loop()
     if current_loop is not None:
         return current_loop
-    return get_event_loop_policy().get_event_loop()
+    return _get_event_loop_policy().get_event_loop()
 
 
 def set_event_loop(loop):
     """Equivalent to calling get_event_loop_policy().set_event_loop(loop)."""
-    get_event_loop_policy().set_event_loop(loop)
+    _get_event_loop_policy().set_event_loop(loop)
 
 
 def new_event_loop():
     """Equivalent to calling get_event_loop_policy().new_event_loop()."""
-    return get_event_loop_policy().new_event_loop()
-
-
-def get_child_watcher():
-    """Equivalent to calling get_event_loop_policy().get_child_watcher()."""
-    return get_event_loop_policy().get_child_watcher()
-
-
-def set_child_watcher(watcher):
-    """Equivalent to calling
-    get_event_loop_policy().set_child_watcher(watcher)."""
-    return get_event_loop_policy().set_child_watcher(watcher)
+    return _get_event_loop_policy().new_event_loop()
 
 
 # Alias pure-Python implementations for testing purposes.
@@ -861,7 +871,7 @@ if hasattr(os, 'fork'):
     def on_fork():
         # Reset the loop and wakeupfd in the forked child process.
         if _event_loop_policy is not None:
-            _event_loop_policy._local = BaseDefaultEventLoopPolicy._Local()
+            _event_loop_policy._local = _BaseDefaultEventLoopPolicy._Local()
         _set_running_loop(None)
         signal.set_wakeup_fd(-1)
 

Lib/asyncio/format_helpers.py

@@ -19,19 +19,26 @@ def _get_function_source(func):
     return None
 
 
-def _format_callback_source(func, args):
-    func_repr = _format_callback(func, args, None)
+def _format_callback_source(func, args, *, debug=False):
+    func_repr = _format_callback(func, args, None, debug=debug)
     source = _get_function_source(func)
     if source:
         func_repr += f' at {source[0]}:{source[1]}'
     return func_repr
 
 
-def _format_args_and_kwargs(args, kwargs):
+def _format_args_and_kwargs(args, kwargs, *, debug=False):
     """Format function arguments and keyword arguments.
 
     Special case for a single parameter: ('hello',) is formatted as ('hello').
+
+    Note that this function only returns argument details when
+    debug=True is specified, as arguments may contain sensitive
+    information.
     """
+    if not debug:
+        return '()'
+
     # use reprlib to limit the length of the output
     items = []
     if args:
@@ -41,10 +48,11 @@ def _format_args_and_kwargs(args, kwargs):
     return '({})'.format(', '.join(items))
 
 
-def _format_callback(func, args, kwargs, suffix=''):
+def _format_callback(func, args, kwargs, *, debug=False, suffix=''):
     if isinstance(func, functools.partial):
-        suffix = _format_args_and_kwargs(args, kwargs) + suffix
-        return _format_callback(func.func, func.args, func.keywords, suffix)
+        suffix = _format_args_and_kwargs(args, kwargs, debug=debug) + suffix
+        return _format_callback(func.func, func.args, func.keywords,
+                                debug=debug, suffix=suffix)
 
     if hasattr(func, '__qualname__') and func.__qualname__:
         func_repr = func.__qualname__
@@ -53,7 +61,7 @@ def _format_callback(func, args, kwargs, suffix=''):
     else:
         func_repr = repr(func)
 
-    func_repr += _format_args_and_kwargs(args, kwargs)
+    func_repr += _format_args_and_kwargs(args, kwargs, debug=debug)
     if suffix:
         func_repr += suffix
     return func_repr

Lib/asyncio/futures.py

@@ -2,6 +2,7 @@
 
 __all__ = (
     'Future', 'wrap_future', 'isfuture',
+    'future_add_to_awaited_by', 'future_discard_from_awaited_by',
 )
 
 import concurrent.futures
@@ -43,7 +44,6 @@ class Future:
     - This class is not compatible with the wait() and as_completed()
       methods in the concurrent.futures package.
 
-    (In Python 3.4 or later we may be able to unify the implementations.)
     """
 
     # Class variables serving as defaults for instance variables.
@@ -61,12 +61,15 @@ class Future:
     #   the Future protocol (i.e. is intended to be duck-type compatible).
     #   The value must also be not-None, to enable a subclass to declare
     #   that it is not compatible by setting this to None.
-    # - It is set by __iter__() below so that Task._step() can tell
+    # - It is set by __iter__() below so that Task.__step() can tell
     #   the difference between
-    #   `await Future()` or`yield from Future()` (correct) vs.
+    #   `await Future()` or `yield from Future()` (correct) vs.
     #   `yield Future()` (incorrect).
     _asyncio_future_blocking = False
 
+    # Used by the capture_call_stack() API.
+    __asyncio_awaited_by = None
+
     __log_traceback = False
 
     def __init__(self, *, loop=None):
@@ -116,6 +119,12 @@ class Future:
             raise ValueError('_log_traceback can only be set to False')
         self.__log_traceback = False
 
+    @property
+    def _asyncio_awaited_by(self):
+        if self.__asyncio_awaited_by is None:
+            return None
+        return frozenset(self.__asyncio_awaited_by)
+
     def get_loop(self):
         """Return the event loop the Future is bound to."""
         loop = self._loop
@@ -138,9 +147,6 @@ class Future:
             exc = exceptions.CancelledError()
         else:
             exc = exceptions.CancelledError(self._cancel_message)
-        exc.__context__ = self._cancelled_exc
-        # Remove the reference since we don't need this anymore.
-        self._cancelled_exc = None
         return exc
 
     def cancel(self, msg=None):
@@ -194,8 +200,7 @@ class Future:
         the future is done and has an exception set, this exception is raised.
         """
         if self._state == _CANCELLED:
-            exc = self._make_cancelled_error()
-            raise exc
+            raise self._make_cancelled_error()
         if self._state != _FINISHED:
             raise exceptions.InvalidStateError('Result is not ready.')
         self.__log_traceback = False
@@ -212,8 +217,7 @@ class Future:
         InvalidStateError.
         """
         if self._state == _CANCELLED:
-            exc = self._make_cancelled_error()
-            raise exc
+            raise self._make_cancelled_error()
         if self._state != _FINISHED:
             raise exceptions.InvalidStateError('Exception is not set.')
         self.__log_traceback = False
@@ -272,9 +276,13 @@ class Future:
             raise exceptions.InvalidStateError(f'{self._state}: {self!r}')
         if isinstance(exception, type):
             exception = exception()
-        if type(exception) is StopIteration:
-            raise TypeError("StopIteration interacts badly with generators "
-                            "and cannot be raised into a Future")
+        if isinstance(exception, StopIteration):
+            new_exc = RuntimeError("StopIteration interacts badly with "
+                                   "generators and cannot be raised into a "
+                                   "Future")
+            new_exc.__cause__ = exception
+            new_exc.__context__ = exception
+            exception = new_exc
         self._exception = exception
         self._exception_tb = exception.__traceback__
         self._state = _FINISHED
@@ -318,11 +326,9 @@ def _set_result_unless_cancelled(fut, result):
 def _convert_future_exc(exc):
     exc_class = type(exc)
     if exc_class is concurrent.futures.CancelledError:
-        return exceptions.CancelledError(*exc.args)
-    elif exc_class is concurrent.futures.TimeoutError:
-        return exceptions.TimeoutError(*exc.args)
+        return exceptions.CancelledError(*exc.args).with_traceback(exc.__traceback__)
     elif exc_class is concurrent.futures.InvalidStateError:
-        return exceptions.InvalidStateError(*exc.args)
+        return exceptions.InvalidStateError(*exc.args).with_traceback(exc.__traceback__)
     else:
         return exc
 
@@ -419,6 +425,49 @@ def wrap_future(future, *, loop=None):
     return new_future
 
 
+def future_add_to_awaited_by(fut, waiter, /):
+    """Record that `fut` is awaited on by `waiter`."""
+    # For the sake of keeping the implementation minimal and assuming
+    # that most of asyncio users use the built-in Futures and Tasks
+    # (or their subclasses), we only support native Future objects
+    # and their subclasses.
+    #
+    # Longer version: tracking requires storing the caller-callee
+    # dependency somewhere. One obvious choice is to store that
+    # information right in the future itself in a dedicated attribute.
+    # This means that we'd have to require all duck-type compatible
+    # futures to implement a specific attribute used by asyncio for
+    # the book keeping. Another solution would be to store that in
+    # a global dictionary. The downside here is that that would create
+    # strong references and any scenario where the "add" call isn't
+    # followed by a "discard" call would lead to a memory leak.
+    # Using WeakDict would resolve that issue, but would complicate
+    # the C code (_asynciomodule.c). The bottom line here is that
+    # it's not clear that all this work would be worth the effort.
+    #
+    # Note that there's an accelerated version of this function
+    # shadowing this implementation later in this file.
+    if isinstance(fut, _PyFuture) and isinstance(waiter, _PyFuture):
+        if fut._Future__asyncio_awaited_by is None:
+            fut._Future__asyncio_awaited_by = set()
+        fut._Future__asyncio_awaited_by.add(waiter)
+
+
+def future_discard_from_awaited_by(fut, waiter, /):
+    """Record that `fut` is no longer awaited on by `waiter`."""
+    # See the comment in "future_add_to_awaited_by()" body for
+    # details on implementation.
+    #
+    # Note that there's an accelerated version of this function
+    # shadowing this implementation later in this file.
+    if isinstance(fut, _PyFuture) and isinstance(waiter, _PyFuture):
+        if fut._Future__asyncio_awaited_by is not None:
+            fut._Future__asyncio_awaited_by.discard(waiter)
+
+
+_py_future_add_to_awaited_by = future_add_to_awaited_by
+_py_future_discard_from_awaited_by = future_discard_from_awaited_by
+
 try:
     import _asyncio
 except ImportError:
@@ -426,3 +475,7 @@ except ImportError:
 else:
     # _CFuture is needed for tests.
     Future = _CFuture = _asyncio.Future
+    future_add_to_awaited_by = _asyncio.future_add_to_awaited_by
+    future_discard_from_awaited_by = _asyncio.future_discard_from_awaited_by
+    _c_future_add_to_awaited_by = future_add_to_awaited_by
+    _c_future_discard_from_awaited_by = future_discard_from_awaited_by

Lib/asyncio/graph.py

@@ -0,0 +1,276 @@
+"""Introspection utils for tasks call graphs."""
+
+import dataclasses
+import io
+import sys
+import types
+
+from . import events
+from . import futures
+from . import tasks
+
+__all__ = (
+    'capture_call_graph',
+    'format_call_graph',
+    'print_call_graph',
+    'FrameCallGraphEntry',
+    'FutureCallGraph',
+)
+
+# Sadly, we can't re-use the traceback module's datastructures as those
+# are tailored for error reporting, whereas we need to represent an
+# async call graph.
+#
+# Going with pretty verbose names as we'd like to export them to the
+# top level asyncio namespace, and want to avoid future name clashes.
+
+
+@dataclasses.dataclass(frozen=True, slots=True)
+class FrameCallGraphEntry:
+    frame: types.FrameType
+
+
+@dataclasses.dataclass(frozen=True, slots=True)
+class FutureCallGraph:
+    future: futures.Future
+    call_stack: tuple["FrameCallGraphEntry", ...]
+    awaited_by: tuple["FutureCallGraph", ...]
+
+
+def _build_graph_for_future(
+    future: futures.Future,
+    *,
+    limit: int | None = None,
+) -> FutureCallGraph:
+    if not isinstance(future, futures.Future):
+        raise TypeError(
+            f"{future!r} object does not appear to be compatible "
+            f"with asyncio.Future"
+        )
+
+    coro = None
+    if get_coro := getattr(future, 'get_coro', None):
+        coro = get_coro() if limit != 0 else None
+
+    st: list[FrameCallGraphEntry] = []
+    awaited_by: list[FutureCallGraph] = []
+
+    while coro is not None:
+        if hasattr(coro, 'cr_await'):
+            # A native coroutine or duck-type compatible iterator
+            st.append(FrameCallGraphEntry(coro.cr_frame))
+            coro = coro.cr_await
+        elif hasattr(coro, 'ag_await'):
+            # A native async generator or duck-type compatible iterator
+            st.append(FrameCallGraphEntry(coro.cr_frame))
+            coro = coro.ag_await
+        else:
+            break
+
+    if future._asyncio_awaited_by:
+        for parent in future._asyncio_awaited_by:
+            awaited_by.append(_build_graph_for_future(parent, limit=limit))
+
+    if limit is not None:
+        if limit > 0:
+            st = st[:limit]
+        elif limit < 0:
+            st = st[limit:]
+    st.reverse()
+    return FutureCallGraph(future, tuple(st), tuple(awaited_by))
+
+
+def capture_call_graph(
+    future: futures.Future | None = None,
+    /,
+    *,
+    depth: int = 1,
+    limit: int | None = None,
+) -> FutureCallGraph | None:
+    """Capture the async call graph for the current task or the provided Future.
+
+    The graph is represented with three data structures:
+
+    * FutureCallGraph(future, call_stack, awaited_by)
+
+      Where 'future' is an instance of asyncio.Future or asyncio.Task.
+
+      'call_stack' is a tuple of FrameGraphEntry objects.
+
+      'awaited_by' is a tuple of FutureCallGraph objects.
+
+    * FrameCallGraphEntry(frame)
+
+      Where 'frame' is a frame object of a regular Python function
+      in the call stack.
+
+    Receives an optional 'future' argument. If not passed,
+    the current task will be used. If there's no current task, the function
+    returns None.
+
+    If "capture_call_graph()" is introspecting *the current task*, the
+    optional keyword-only 'depth' argument can be used to skip the specified
+    number of frames from top of the stack.
+
+    If the optional keyword-only 'limit' argument is provided, each call stack
+    in the resulting graph is truncated to include at most ``abs(limit)``
+    entries. If 'limit' is positive, the entries left are the closest to
+    the invocation point. If 'limit' is negative, the topmost entries are
+    left. If 'limit' is omitted or None, all entries are present.
+    If 'limit' is 0, the call stack is not captured at all, only
+    "awaited by" information is present.
+    """
+
+    loop = events._get_running_loop()
+
+    if future is not None:
+        # Check if we're in a context of a running event loop;
+        # if yes - check if the passed future is the currently
+        # running task or not.
+        if loop is None or future is not tasks.current_task(loop=loop):
+            return _build_graph_for_future(future, limit=limit)
+        # else: future is the current task, move on.
+    else:
+        if loop is None:
+            raise RuntimeError(
+                'capture_call_graph() is called outside of a running '
+                'event loop and no *future* to introspect was provided')
+        future = tasks.current_task(loop=loop)
+
+    if future is None:
+        # This isn't a generic call stack introspection utility. If we
+        # can't determine the current task and none was provided, we
+        # just return.
+        return None
+
+    if not isinstance(future, futures.Future):
+        raise TypeError(
+            f"{future!r} object does not appear to be compatible "
+            f"with asyncio.Future"
+        )
+
+    call_stack: list[FrameCallGraphEntry] = []
+
+    f = sys._getframe(depth) if limit != 0 else None
+    try:
+        while f is not None:
+            is_async = f.f_generator is not None
+            call_stack.append(FrameCallGraphEntry(f))
+
+            if is_async:
+                if f.f_back is not None and f.f_back.f_generator is None:
+                    # We've reached the bottom of the coroutine stack, which
+                    # must be the Task that runs it.
+                    break
+
+            f = f.f_back
+    finally:
+        del f
+
+    awaited_by = []
+    if future._asyncio_awaited_by:
+        for parent in future._asyncio_awaited_by:
+            awaited_by.append(_build_graph_for_future(parent, limit=limit))
+
+    if limit is not None:
+        limit *= -1
+        if limit > 0:
+            call_stack = call_stack[:limit]
+        elif limit < 0:
+            call_stack = call_stack[limit:]
+
+    return FutureCallGraph(future, tuple(call_stack), tuple(awaited_by))
+
+
+def format_call_graph(
+    future: futures.Future | None = None,
+    /,
+    *,
+    depth: int = 1,
+    limit: int | None = None,
+) -> str:
+    """Return the async call graph as a string for `future`.
+
+    If `future` is not provided, format the call graph for the current task.
+    """
+
+    def render_level(st: FutureCallGraph, buf: list[str], level: int) -> None:
+        def add_line(line: str) -> None:
+            buf.append(level * '    ' + line)
+
+        if isinstance(st.future, tasks.Task):
+            add_line(
+                f'* Task(name={st.future.get_name()!r}, id={id(st.future):#x})'
+            )
+        else:
+            add_line(
+                f'* Future(id={id(st.future):#x})'
+            )
+
+        if st.call_stack:
+            add_line(
+                f'  + Call stack:'
+            )
+            for ste in st.call_stack:
+                f = ste.frame
+
+                if f.f_generator is None:
+                    f = ste.frame
+                    add_line(
+                        f'  |   File {f.f_code.co_filename!r},'
+                        f' line {f.f_lineno}, in'
+                        f' {f.f_code.co_qualname}()'
+                    )
+                else:
+                    c = f.f_generator
+
+                    try:
+                        f = c.cr_frame
+                        code = c.cr_code
+                        tag = 'async'
+                    except AttributeError:
+                        try:
+                            f = c.ag_frame
+                            code = c.ag_code
+                            tag = 'async generator'
+                        except AttributeError:
+                            f = c.gi_frame
+                            code = c.gi_code
+                            tag = 'generator'
+
+                    add_line(
+                        f'  |   File {f.f_code.co_filename!r},'
+                        f' line {f.f_lineno}, in'
+                        f' {tag} {code.co_qualname}()'
+                    )
+
+        if st.awaited_by:
+            add_line(
+                f'  + Awaited by:'
+            )
+            for fut in st.awaited_by:
+                render_level(fut, buf, level + 1)
+
+    graph = capture_call_graph(future, depth=depth + 1, limit=limit)
+    if graph is None:
+        return ""
+
+    buf: list[str] = []
+    try:
+        render_level(graph, buf, 0)
+    finally:
+        # 'graph' has references to frames so we should
+        # make sure it's GC'ed as soon as we don't need it.
+        del graph
+    return '\n'.join(buf)
+
+def print_call_graph(
+    future: futures.Future | None = None,
+    /,
+    *,
+    file: io.Writer[str] | None = None,
+    depth: int = 1,
+    limit: int | None = None,
+) -> None:
+    """Print the async call graph for the current task or the provided Future."""
+    print(format_call_graph(future, depth=depth, limit=limit), file=file)