Drop old PyObjectRef outside type lock to prevent deadlock

Dropping values inside with_type_lock can trigger weakref callbacks,
which may access attributes (LOAD_ATTR specialization) and re-acquire
the non-reentrant type mutex, causing deadlock.

Return old values from lock closures so they drop after lock release.

.cargo/config.toml

@@ -3,3 +3,6 @@ rustflags = "-C link-arg=/STACK:8000000"
 
 [target.'cfg(all(target_os = "windows", not(target_env = "msvc")))']
 rustflags = "-C link-args=-Wl,--stack,8000000"
+
+[target.wasm32-unknown-unknown]
+rustflags = ["--cfg=getrandom_backend=\"wasm_js\""]

.claude/commands/apple-container.md

@@ -0,0 +1,46 @@
+---
+allowed-tools: Bash(container *), Bash(cargo *), Read, Grep, Glob
+---
+
+# Run Tests in Linux Container (Apple `container` CLI)
+
+Run RustPython tests inside a Linux container using Apple's `container` CLI.
+**NEVER use Docker, Podman, or any other container runtime.** Only use the `container` command.
+
+## Arguments
+- `$ARGUMENTS`: Test command to run (e.g., `test_io`, `test_codecs -v`, `test_io -v -m "test_errors"`)
+
+## Prerequisites
+
+The `container` CLI is installed via `brew install container`.
+The dev image `rustpython-dev` is already built.
+
+## Steps
+
+1. **Check if the container is already running**
+   ```shell
+   container list 2>/dev/null | grep rustpython-test
+   ```
+
+2. **Start the container if not running**
+   ```shell
+   container run -d --name rustpython-test -m 8G -c 4 \
+       --mount type=bind,source=/Users/al03219714/Projects/RustPython3,target=/workspace \
+       -w /workspace rustpython-dev sleep infinity
+   ```
+
+3. **Run the test inside the container**
+   ```shell
+   container exec rustpython-test sh -c "cargo run --release -- -m test $ARGUMENTS"
+   ```
+
+4. **Report results**
+   - Show test summary (pass/fail counts, expected failures, unexpected successes)
+   - Highlight any new failures compared to macOS results if available
+   - Do NOT stop or remove the container after testing (keep it for reuse)
+
+## Notes
+- The workspace is bind-mounted, so local code changes are immediately available
+- Use `container exec rustpython-test sh -c "..."` for any command inside the container
+- To rebuild after code changes, run: `container exec rustpython-test sh -c "cargo build --release"`
+- To stop the container when done: `container rm -f rustpython-test`

.claude/commands/investigate-test-failure.md

@@ -0,0 +1,49 @@
+---
+allowed-tools: Bash(python3:*), Bash(cargo run:*), Read, Grep, Glob, Bash(git add:*), Bash(git commit:*), Bash(cargo fmt:*), Bash(git diff:*), Task
+---
+
+# Investigate Test Failure
+
+Investigate why a specific test is failing and determine if it can be fixed or needs an issue.
+
+## Arguments
+- `$ARGUMENTS`: Failed test identifier (e.g., `test_inspect.TestGetSourceBase.test_getsource_reload`)
+
+## Steps
+
+1. **Analyze failure cause**
+   - Read the test code
+   - Analyze failure message/traceback
+   - Check related RustPython code
+
+2. **Verify behavior in CPython**
+   - Run the test with `python3 -m unittest` to confirm expected behavior
+   - Document the expected output
+
+3. **Determine fix feasibility**
+   - **Simple fix** (import issues, small logic bugs): Fix code → Run `cargo fmt --all` → Pre-commit review → Commit
+   - **Complex fix** (major unimplemented features): Collect issue info and report to user
+
+   **Pre-commit review process**:
+   - Run `git diff` to see the changes
+   - Use Task tool with `general-purpose` subagent to review:
+     - Compare implementation against cpython/ source code
+     - Verify the fix aligns with CPython behavior
+     - Check for any missed edge cases
+   - Proceed to commit only after review passes
+
+4. **For complex issues - Collect issue information**
+   Following `.github/ISSUE_TEMPLATE/report-incompatibility.md` format:
+
+   - **Feature**: Description of missing/broken Python feature
+   - **Minimal reproduction code**: Smallest code that reproduces the issue
+   - **CPython behavior**: Result when running with python3
+   - **RustPython behavior**: Result when running with cargo run
+   - **Python Documentation link**: Link to relevant CPython docs
+
+   Report collected information to the user. Issue creation is done only upon user request.
+
+   Example issue creation command:
+   ```
+   gh issue create --template report-incompatibility.md --title "..." --body "..."
+   ```

.claude/commands/upgrade-pylib-next.md

@@ -0,0 +1,33 @@
+---
+allowed-tools: Skill(upgrade-pylib), Bash(gh pr list:*)
+---
+
+# Upgrade Next Python Library
+
+Find the next Python library module ready for upgrade and run `/upgrade-pylib` for it.
+
+## Current TODO Status
+
+!`cargo run --release -- scripts/update_lib todo 2>/dev/null`
+
+## Open Upgrade PRs
+
+!`gh pr list --search "Update in:title" --json number,title --template '{{range .}}#{{.number}} {{.title}}{{"\n"}}{{end}}'`
+
+## Instructions
+
+From the TODO list above, find modules matching these patterns (in priority order):
+
+1. `[ ] [no deps]` - Modules with no dependencies (can be upgraded immediately)
+2. `[ ] [0/n]` - Modules where all dependencies are already upgraded (e.g., `[0/3]`, `[0/5]`)
+
+These patterns indicate modules that are ready to upgrade without blocking dependencies.
+
+**Important**: Skip any modules that already have an open PR in the "Open Upgrade PRs" list above.
+
+**After identifying a suitable module**, run:
+```
+/upgrade-pylib <module_name>
+```
+
+If no modules match these criteria, inform the user that all eligible modules have dependencies that need to be upgraded first.

.claude/commands/upgrade-pylib.md

@@ -0,0 +1,157 @@
+---
+allowed-tools: Bash(git add:*), Bash(git commit:*), Bash(python3 scripts/update_lib quick:*), Bash(python3 scripts/update_lib auto-mark:*)
+---
+
+# Upgrade Python Library from CPython
+
+Upgrade a Python standard library module from CPython to RustPython.
+
+## Arguments
+- `$ARGUMENTS`: Library name to upgrade (e.g., `inspect`, `asyncio`, `json`)
+
+## Important: Report Tool Issues First
+
+If during the upgrade process you encounter any of the following issues with `scripts/update_lib`:
+- A feature that should be automated but isn't supported
+- A bug or unexpected behavior in the tool
+- Missing functionality that would make the upgrade easier
+
+**STOP the upgrade and report the issue first.** Describe:
+1. What you were trying to do
+    - Library name
+    - The full command executed (e.g. python scripts/update_lib quick cpython/Lib/$ARGUMENTS.py)
+2. What went wrong or what's missing
+3. Expected vs actual behavior
+
+This helps improve the tooling for future upgrades.
+
+## Steps
+
+1. **Run quick upgrade with update_lib**
+   - Run: `python3 scripts/update_lib quick $ARGUMENTS` (module name)
+   - Or: `python3 scripts/update_lib quick cpython/Lib/$ARGUMENTS.py` (library file path)
+   - Or: `python3 scripts/update_lib quick cpython/Lib/$ARGUMENTS/` (library directory path)
+   - This will:
+     - Copy library files (delete existing `Lib/$ARGUMENTS.py` or `Lib/$ARGUMENTS/`, then copy from `cpython/Lib/`)
+     - Patch test files preserving existing RustPython markers
+     - Run tests and auto-mark new test failures (not regressions)
+     - Remove `@unittest.expectedFailure` from tests that now pass
+     - Create a git commit with the changes
+   - **Handle warnings**: If you see warnings like `WARNING: TestCFoo does not exist in remote file`, it means the class structure changed and markers couldn't be transferred automatically. These need to be manually restored in step 2 or added in step 3.
+
+2. **Review git diff and restore RUSTPYTHON-specific changes**
+   - Run `git diff Lib/test/test_$ARGUMENTS` to review all changes
+   - **Only restore changes that have explicit `RUSTPYTHON` comments**. Look for:
+     - `# XXX: RUSTPYTHON` or `# XXX RUSTPYTHON` - Comments marking RustPython-specific code modifications
+     - `# TODO: RUSTPYTHON` - Comments marking tests that need work
+     - Code changes with inline `# ... RUSTPYTHON` comments
+   - **Do NOT restore other diff changes** - these are likely upstream CPython changes, not RustPython-specific modifications
+   - When restoring, preserve the original context and formatting
+
+3. **Investigate test failures with subagent**
+   - First, get dependent tests using the deps command:
+     ```
+     cargo run --release -- scripts/update_lib deps $ARGUMENTS
+     ```
+   - Look for the line `- [ ] $ARGUMENTS: test_xxx test_yyy ...` to get the direct dependent tests
+   - Run those tests to collect failures:
+     ```
+     cargo run --release -- -m test test_xxx test_yyy ... 2>&1 | grep -E "^(FAIL|ERROR):"
+     ```
+   - For example, if deps output shows `- [ ] linecache: test_bdb test_inspect test_linecache test_traceback test_zipimport`, run:
+     ```
+     cargo run --release -- -m test test_bdb test_inspect test_linecache test_traceback test_zipimport 2>&1 | grep -E "^(FAIL|ERROR):"
+     ```
+   - For each failure, use the Task tool with `general-purpose` subagent to investigate:
+     - Subagent should follow the `/investigate-test-failure` skill workflow
+     - Pass the failed test identifier as the argument (e.g., `test_inspect.TestGetSourceBase.test_getsource_reload`)
+   - If subagent can fix the issue easily: fix and commit
+   - If complex issue: subagent collects issue info and reports back (issue creation on user request only)
+   - Using subagent prevents context pollution in the main conversation
+
+4. **Mark remaining test failures with auto-mark**
+   - Run: `python3 scripts/update_lib auto-mark Lib/test/test_$ARGUMENTS.py --mark-failure`
+   - Or for directory: `python3 scripts/update_lib auto-mark Lib/test/test_$ARGUMENTS/ --mark-failure`
+   - This will:
+     - Run tests and mark ALL failing tests with `@unittest.expectedFailure`
+     - Remove `@unittest.expectedFailure` from tests that now pass
+   - **Note**: The `--mark-failure` flag marks all failures including regressions. Review the changes before committing.
+
+5. **Handle panics manually**
+   - If any tests cause panics/crashes (not just assertion failures), they need `@unittest.skip` instead:
+     ```python
+     @unittest.skip("TODO: RUSTPYTHON; panics with 'index out of bounds'")
+     def test_crashes(self):
+         ...
+     ```
+   - auto-mark cannot detect panics automatically - check the test output for crash messages
+
+6. **Handle class-specific failures**
+   - If a test fails only in the C implementation (TestCFoo) but passes in the Python implementation (TestPyFoo), or vice versa, move the marker to the specific subclass:
+     ```python
+     # Base class - no marker here
+     class TestFoo:
+         def test_something(self):
+             ...
+
+     class TestPyFoo(TestFoo, PyTest): pass
+
+     class TestCFoo(TestFoo, CTest):
+         # TODO: RUSTPYTHON
+         @unittest.expectedFailure
+         def test_something(self):
+             return super().test_something()
+     ```
+
+7. **Commit the test fixes**
+   - Run: `git add -u && git commit -m "Mark failing tests"`
+   - This creates a separate commit for the test markers added in steps 2-6
+
+## Example Usage
+```
+# Using module names (recommended)
+/upgrade-pylib inspect
+/upgrade-pylib json
+/upgrade-pylib asyncio
+
+# Using library paths (alternative)
+/upgrade-pylib cpython/Lib/inspect.py
+/upgrade-pylib cpython/Lib/json/
+```
+
+## Example: Restoring RUSTPYTHON changes
+
+When git diff shows removed RUSTPYTHON-specific code like:
+```diff
+-# XXX RUSTPYTHON: we don't import _json as fresh since...
+-cjson = import_helper.import_fresh_module('json') #, fresh=['_json'])
++cjson = import_helper.import_fresh_module('json', fresh=['_json'])
+```
+
+You should restore the RustPython version:
+```python
+# XXX RUSTPYTHON: we don't import _json as fresh since...
+cjson = import_helper.import_fresh_module('json') #, fresh=['_json'])
+```
+
+## Notes
+- The cpython/ directory should contain the CPython source that we're syncing from
+- `scripts/update_lib` package handles patching and auto-marking:
+  - `quick` - Combined patch + auto-mark (recommended)
+  - `migrate` - Only migrate (patch), no test running
+  - `auto-mark` - Only run tests and mark failures
+  - `copy-lib` - Copy library files (not tests)
+- The patching:
+  - Transfers `@unittest.expectedFailure` and `@unittest.skip` decorators with `TODO: RUSTPYTHON` markers
+  - Adds `import unittest # XXX: RUSTPYTHON` if needed for the decorators
+  - **Limitation**: If a class was restructured (e.g., method overrides removed), update_lib will warn and skip those markers
+- The smart auto-mark:
+  - Marks NEW test failures automatically (tests that didn't exist before)
+  - Does NOT mark regressions (existing tests that now fail) - these are warnings
+  - Removes `@unittest.expectedFailure` from tests that now pass
+- The script does NOT preserve all RustPython-specific changes - you must review `git diff` and restore them
+- Common RustPython markers to look for:
+  - `# XXX: RUSTPYTHON` or `# XXX RUSTPYTHON` - Inline comments for code modifications
+  - `# TODO: RUSTPYTHON` - Test skip/failure markers
+  - Any code with `RUSTPYTHON` in comments that was removed in the diff
+- **Important**: Not all changes in the git diff need to be restored. Only restore changes that have explicit `RUSTPYTHON` comments. Other changes are upstream CPython updates.

.claude/scripts/setup-env.sh

@@ -0,0 +1,50 @@
+#!/bin/bash
+# Claude Code web session startup script
+# Sets up the development environment for RustPython
+
+set -e
+
+cd /home/user/RustPython
+
+echo "=== RustPython dev environment setup ==="
+
+# 1. Ensure python3 points to 3.13+ (needed for scripts/update_lib)
+#    /usr/local/bin takes precedence over /usr/bin in PATH,
+#    so we update the symlink there directly.
+CURRENT_PY=$(python3 --version 2>&1 | grep -oP '\d+\.\d+')
+if [ "$(printf '%s\n' "3.13" "$CURRENT_PY" | sort -V | head -1)" != "3.13" ]; then
+    echo "Upgrading python3 default to 3.13..."
+    # Find best available Python >= 3.13
+    TARGET=""
+    for ver in python3.14 python3.13; do
+        if command -v "$ver" &>/dev/null; then
+            TARGET=$(command -v "$ver")
+            break
+        fi
+    done
+    if [ -n "$TARGET" ]; then
+        # Override /usr/local/bin/python3 if it exists and is outdated
+        if [ -e /usr/local/bin/python3 ]; then
+            sudo ln -sf "$TARGET" /usr/local/bin/python3
+        fi
+        # Also set /usr/bin via update-alternatives
+        sudo update-alternatives --install /usr/bin/python3 python3 "$TARGET" 3 2>/dev/null || true
+        sudo update-alternatives --set python3 "$TARGET" 2>/dev/null || true
+        echo "python3 now: $(python3 --version)"
+    else
+        echo "WARNING: No Python 3.13+ found. scripts/update_lib may not work."
+    fi
+else
+    echo "python3 already >= 3.13: $(python3 --version)"
+fi
+
+# 2. Clone CPython source if not present (needed for scripts/update_lib)
+if [ ! -d "cpython" ]; then
+    echo "Cloning CPython v3.14.3 (shallow)..."
+    git clone --depth 1 --branch v3.14.3 https://github.com/python/cpython.git cpython
+    echo "CPython source ready."
+else
+    echo "CPython source already present."
+fi
+
+echo "=== Setup complete ==="

.claude/settings.json

@@ -0,0 +1,15 @@
+{
+  "hooks": {
+    "SessionStart": [
+      {
+        "matcher": "",
+        "hooks": [
+          {
+            "type": "command",
+            "command": "bash .claude/scripts/setup-env.sh"
+          }
+        ]
+      }
+    ]
+  }
+}

.cspell.dict/cpython.txt

@@ -1,61 +1,233 @@
+ADDOP
+aftersign
+argdefs
 argtypes
 asdl
 asname
+attro
 augassign
+badcert
 badsyntax
+baseinfo
 basetype
+binop
+bltin
 boolop
+BUFMAX
+BUILDSTDLIB
 bxor
+byteswap
 cached_tsver
+cadata
+cafile
+calldepth
+callinfo
+callproc
+capath
+carg
 cellarg
 cellvar
 cellvars
+ceval
+cfield
+CLASSDEREF
+classdict
 cmpop
+codedepth
+CODEUNIT
+CONIN
+CONOUT
+constevaluator
+consti
+CONVFUNC
+convparam
+copyslot
+cpucount
+datastack
+defaultdict
 denom
+deopt
+dictbytype
+DICTFLAG
 dictoffset
+distpoint
+dynload
 elts
+eofs
+evalloop
 excepthandler
+exceptiontable
+fastlocal
+fastlocals
+fblock
+fblocks
+fdescr
+ffi_argtypes
+fielddesc
+fieldlist
 fileutils
 finalbody
+finalizers
+firsttraceable
+flowgraph
 formatfloat
+freelist
 freevar
 freevars
 fromlist
+getdict
+getfunc
+getiter
+getsets
+getslice
+globalgetvar
+HASARRAY
+HASBITFIELD
+HASPOINTER
+HASSTRUCT
+HASUNION
 heaptype
+hexdigit
 HIGHRES
+IFUNC
 IMMUTABLETYPE
+INCREF
+inlinedepth
+inplace
+ismine
+ISPOINTER
+iteminfo
+Itertool
+keeped
+kwnames
 kwonlyarg
 kwonlyargs
 lasti
+libffi
 linearise
+lineiterator
+linetable
+loadfast
+localsplus
+Lshift
+lsprof
+MAXBLOCKS
 maxdepth
+metavars
+miscompiles
 mult
+multibytecodec
+nameobj
+nameop
+ncells
+nconsts
+newargs
+newfree
+NEWLOCALS
+newsemlockobject
+nfrees
 nkwargs
+nkwelts
+nlocalsplus
+Nondescriptor
+noninteger
+nops
 noraise
+nseen
+NSIGNALS
 numer
+opname
+opnames
 orelse
+outparam
+outparm
+paramfunc
+parg
 pathconfig
 patma
+peepholer
+phcount
+platstdlib
 posonlyarg
 posonlyargs
 prec
 preinitialized
+pybuilddir
+pycore
+pyinner
+pydecimal
+Pyfunc
+pylifecycle
+pymain
+pyrepl
+PYTHONTRACEMALLOC
+PYTHONUTF8
+pythonw
 PYTHREAD_NAME
+releasebuffer
+repr
+resinfo
+Rshift
 SA_ONSTACK
+saveall
+scls
+setdict
+setfunc
+setprofileallthreads
+SETREF
+setresult
+setslice
+settraceallthreads
+SLOTDEFINED
+SMALLBUF
+SOABI
+SSLEOF
 stackdepth
+stackref
+staticbase
+stginfo
+storefast
 stringlib
 structseq
+subkwargs
 subparams
+subscr
+sval
+swappedbytes
+sysdict
+templatelib
+testconsole
+threadstate
+ticketer
+tmptype
 tok_oldval
+tstate
 tvars
+typeobject
+typeparam
+Typeparam
+typeparams
+typeslots
 unaryop
+uncollectable
+Unhandle
 unparse
 unparser
+untracking
 VARKEYWORDS
 varkwarg
+venvlauncher
+venvlaunchert
+venvw
+venvwlauncher
+venvwlaunchert
 wbits
 weakreflist
+weakrefobject
+webpki
+winconsoleio
 withitem
 withs
+worklist
 xstat
-XXPRIME
+XXPRIME

.cspell.dict/python-more.txt

@@ -1,9 +1,13 @@
 abiflags
 abstractmethods
+addcompare
 aenter
 aexit
 aiter
+altzone
 anext
+anextawaitable
+annotationlib
 appendleft
 argcount
 arrayiterator
@@ -22,6 +26,7 @@ breakpointhook
 cformat
 chunksize
 classcell
+classmethods
 closefd
 closesocket
 codepoint
@@ -30,6 +35,8 @@ codesize
 contextvar
 cpython
 cratio
+ctype
+ctypes
 dealloc
 debugbuild
 decompressor
@@ -65,12 +72,15 @@ fnctl
 frombytes
 fromhex
 fromunicode
+frozensets
 fset
 fspath
 fstring
 fstrings
 ftruncate
 genexpr
+genexpressions
+getargs
 getattro
 getcodesize
 getdefaultencoding
@@ -80,14 +90,17 @@ getformat
 getframe
 getframemodulename
 getnewargs
+getopt
 getpip
 getrandom
 getrecursionlimit
 getrefcount
 getsizeof
 getswitchinterval
+getweakref
 getweakrefcount
 getweakrefs
+getweakrefs
 getwindowsversion
 gmtoff
 groupdict
@@ -100,8 +113,12 @@ idxs
 impls
 indexgroup
 infj
+inittab
+Inittab
 instancecheck
 instanceof
+interpchannels
+interpqueues
 irepeat
 isabstractmethod
 isbytes
@@ -126,6 +143,7 @@ listcomp
 longrange
 lvalue
 mappingproxy
+markupbase
 maskpri
 maxdigits
 MAXGROUPS
@@ -141,17 +159,20 @@ mformat
 mro
 mros
 multiarch
+mymodule
 namereplace
 nanj
 nbytes
 ncallbacks
 ndigits
 ndim
+needsfree
 nldecoder
 nlocals
 NOARGS
 nonbytes
 Nonprintable
+onceregistry
 origname
 ospath
 pendingcr
@@ -166,14 +187,21 @@ profilefunc
 pycache
 pycodecs
 pycs
+pydatetime
 pyexpat
+pyio
+pymain
+PYTHONAPI
 PYTHONBREAKPOINT
 PYTHONDEBUG
 PYTHONDONTWRITEBYTECODE
+PYTHONFAULTHANDLER
 PYTHONHASHSEED
 PYTHONHOME
 PYTHONINSPECT
 PYTHONINTMAXSTRDIGITS
+PYTHONIOENCODING
+PYTHONNODEBUGRANGES
 PYTHONNOUSERSITE
 PYTHONOPTIMIZE
 PYTHONPATH
@@ -195,6 +223,7 @@ readbuffer
 reconstructor
 refcnt
 releaselevel
+reraised
 reverseitemiterator
 reverseiterator
 reversekeyiterator
@@ -211,10 +240,13 @@ scproxy
 seennl
 setattro
 setcomp
+setprofileallthreads
 setrecursionlimit
 setswitchinterval
+settraceallthreads
 showwarnmsg
 signum
+sitebuiltins
 slotnames
 STACKLESS
 stacklevel
@@ -223,14 +255,17 @@ startpos
 subclassable
 subclasscheck
 subclasshook
+subclassing
 suboffset
 suboffsets
 SUBPATTERN
+subpatterns
 sumprod
 surrogateescape
 surrogatepass
 sysconf
 sysconfigdata
+sysdict
 sysvars
 teedata
 thisclass
@@ -257,6 +292,7 @@ warnopts
 weaklist
 weakproxy
 weakrefs
+weakrefset
 winver
 withdata
 xmlcharrefreplace

.cspell.dict/rust-more.txt

@@ -5,7 +5,9 @@ biguint
 bindgen
 bitand
 bitflags
+bitflagset
 bitor
+bitvec
 bitxor
 bstr
 byteorder
@@ -28,6 +30,7 @@ hexf
 hexversion
 idents
 illumos
+ilog
 indexmap
 insta
 keccak
@@ -50,17 +53,20 @@ nanos
 nonoverlapping
 objclass
 peekable
+pemfile
 powc
 powf
 powi
 prepended
 punct
 replacen
+retag
 rmatch
 rposition
 rsplitn
 rustc
 rustfmt
+rustls
 rustyline
 seedable
 seekfrom
@@ -80,7 +86,9 @@ unsync
 wasip1
 wasip2
 wasmbind
+wasmer
 wasmtime
 widestring
 winapi
+winresource
 winsock

.cspell.dict/rustpython.txt

@@ -0,0 +1,32 @@
+cfgs
+miri
+py
+pyarg
+pyargs
+pyast
+pyattr
+pyclass
+pyclassmethod
+pyexception
+pyfunction
+pygetset
+pyimpl
+pylib
+pymath
+pymethod
+pymodule
+pyname
+pyobj
+pyobject
+pypayload
+pyref
+pyslot
+pystaticmethod
+pystone
+pystr
+pystruct
+pystructseq
+pytype
+rustix
+struc
+zelf

.cspell.json

@@ -9,6 +9,7 @@
     "@cspell/dict-win32/cspell-ext.json",
     "@cspell/dict-shell/cspell-ext.json",
   ],
+  "allowCompoundWords": true,
   // language - current active spelling language
   "language": "en",
   // dictionaries - list of the names of the dictionaries to use
@@ -16,6 +17,7 @@
     "cpython",  // Sometimes keeping same terms with cpython is easy
     "python-more",  // Python API terms not listed in python
     "rust-more",  // Rust API terms not listed in rust
+    "rustpython",  // RustPython derive macros and internal terms
     "en_US",
     "softwareTerms",
     "c",
@@ -38,6 +40,10 @@
     {
       "name": "rust-more",
       "path": "./.cspell.dict/rust-more.txt"
+    },
+    {
+      "name": "rustpython",
+      "path": "./.cspell.dict/rustpython.txt"
     }
   ],
   "ignorePaths": [
@@ -46,97 +52,44 @@
     "Lib/**"
   ],
   // words - list of words to be always considered correct
+  // (compound words like pyarg, baseclass, microbenchmark are handled by allowCompoundWords)
   "words": [
-    "RUSTPYTHONPATH",
-    // RustPython terms
     "aiterable",
     "alnum",
-    "baseclass",
-    "boxvec",
-    "Bytecode",
-    "cfgs",
-    "codegen",
     "coro",
     "dedentations",
     "dedents",
     "deduped",
+    "deoptimize",
     "downcastable",
     "downcasted",
     "dumpable",
     "emscripten",
     "excs",
-    "finalizer",
-    "GetSet",
-    "groupref",
-    "internable",
+    "interps",
     "jitted",
     "jitting",
+    "kwonly",
     "lossily",
-    "makeunicodedata",
-    "miri",
-    "notrace",
-    "openat",
-    "pyarg",
-    "pyarg",
-    "pyargs",
-    "pyast",
-    "PyAttr",
+    "mcache",
+    "oparg",
     "pyc",
-    "PyClass",
-    "PyClassMethod",
-    "PyException",
-    "PyFunction",
-    "pygetset",
-    "pyimpl",
-    "pylib",
-    "pymath",
-    "pymember",
-    "PyMethod",
-    "PyModule",
-    "pyname",
-    "pyobj",
-    "PyObject",
-    "pypayload",
-    "PyProperty",
-    "pyref",
-    "PyResult",
-    "pyslot",
-    "PyStaticMethod",
-    "pystone",
-    "pystr",
-    "pystruct",
-    "pystructseq",
-    "pytrace",
-    "reducelib",
-    "richcompare",
-    "RustPython",
     "significand",
-    "struc",
-    "summands", // plural of summand
-    "sysmodule",
-    "tracebacks",
-    "typealiases",
-    "unconstructible",
-    "unhashable",
-    "uninit",
+    "summands",
     "unraisable",
-    "unresizable",
     "wasi",
-    "zelf",
+    "weaked",
     // unix
+    "posixshmem",
+    "shm",
     "CLOEXEC",
-    "codeset",
     "endgrent",
     "gethrvtime",
     "getrusage",
-    "nanosleep",
     "sigaction",
     "WRLCK",
     // win32
-    "birthtime",
-    "IFEXEC",
-    // "stat"
-    "FIRMLINK"
+    "IFEXEC"
   ],
   // flagWords - list of words to be always considered incorrect
   "flagWords": [

.devcontainer/Dockerfile

@@ -1,4 +1,4 @@
-FROM mcr.microsoft.com/vscode/devcontainers/rust:1-bullseye
+FROM rust:bullseye
 
 # Install clang
 RUN apt-get update \

.gitattributes

@@ -4,4 +4,67 @@ Cargo.lock  linguist-generated
 vm/src/stdlib/ast/gen.rs    linguist-generated -merge
 Lib/*.py    text working-tree-encoding=UTF-8 eol=LF
 **/*.rs     text working-tree-encoding=UTF-8 eol=LF
-*.pck       binary
+crates/rustpython_doc_db/src/*.inc.rs   linguist-generated=true
+
+# Binary data types
+*.aif binary
+*.aifc binary
+*.aiff binary
+*.au binary
+*.bmp binary
+*.exe binary
+*.icns binary
+*.gif binary
+*.ico binary
+*.jpg binary
+*.pck binary
+*.pdf binary
+*.png binary
+*.psd binary
+*.tar binary
+*.wav binary
+*.whl binary
+*.zip binary
+
+# Text files that should not be subject to eol conversion
+[attr]noeol -text
+
+Lib/test/cjkencodings/*                    noeol
+Lib/test/tokenizedata/coding20731.py       noeol
+Lib/test/decimaltestdata/*.decTest         noeol
+Lib/test/test_email/data/*.txt             noeol
+Lib/test/xmltestdata/*                     noeol
+
+# Shell scripts should have LF even on Windows because of Cygwin
+Lib/venv/scripts/common/activate text eol=lf
+Lib/venv/scripts/posix/* text eol=lf
+
+# CRLF files
+[attr]dos text eol=crlf
+
+# Language aware diff headers
+# https://tekin.co.uk/2020/10/better-git-diff-output-for-ruby-python-elixir-and-more
+# https://gist.github.com/tekin/12500956bd56784728e490d8cef9cb81
+*.css   diff=css
+*.html  diff=html
+*.py    diff=python
+*.md    diff=markdown
+
+# Generated files
+# https://github.com/github/linguist/blob/master/docs/overrides.md
+#
+# To always hide generated files in local diffs, mark them as binary:
+# $ git config diff.generated.binary true
+#
+[attr]generated linguist-generated=true diff=generated
+
+Lib/_opcode_metadata.py                             generated
+Lib/keyword.py                                      generated
+Lib/idlelib/help.html                               generated
+Lib/test/certdata/*.pem                             generated
+Lib/test/certdata/*.0                               generated
+Lib/test/levenshtein_examples.json                  generated
+Lib/test/test_stable_abi_ctypes.py                  generated
+Lib/token.py                                        generated
+
+.github/workflows/*.lock.yml linguist-generated=true merge=ours

.github/actions/install-linux-deps/action.yml

@@ -0,0 +1,49 @@
+# This action installs a few dependencies necessary to build RustPython on Linux.
+# It can be configured depending on which libraries are needed:
+#
+# ```
+# - uses: ./.github/actions/install-linux-deps
+#   with:
+#     gcc-multilib: true
+#     musl-tools: false
+# ```
+#
+# See the `inputs` section for all options and their defaults. Note that you must checkout the
+# repository before you can use this action.
+#
+# This action will only install dependencies when the current operating system is Linux. It will do
+# nothing on any other OS (macOS, Windows).
+
+name: Install Linux dependencies
+description: Installs the dependencies necessary to build RustPython on Linux.
+inputs:
+  gcc-multilib:
+    description: Install gcc-multilib (gcc-multilib)
+    required: false
+    default: "false"
+  musl-tools:
+    description: Install musl-tools (musl-tools)
+    required: false
+    default: "false"
+  gcc-aarch64-linux-gnu:
+    description: Install gcc-aarch64-linux-gnu (gcc-aarch64-linux-gnu)
+    required: false
+    default: "false"
+  clang:
+    description: Install clang (clang)
+    required: false
+    default: "false"
+runs:
+  using: composite
+  steps:
+    - name: Install Linux dependencies
+      shell: bash
+      if: ${{ runner.os == 'Linux' }}
+      run: >
+        sudo apt-get update
+
+        sudo apt-get install --no-install-recommends
+        ${{ fromJSON(inputs.gcc-multilib) && 'gcc-multilib' || '' }}
+        ${{ fromJSON(inputs.musl-tools) && 'musl-tools' || '' }}
+        ${{ fromJSON(inputs.clang) && 'clang' || '' }}
+        ${{ fromJSON(inputs.gcc-aarch64-linux-gnu) && 'gcc-aarch64-linux-gnu linux-libc-dev-arm64-cross libc6-dev-arm64-cross' || '' }}

.github/actions/install-macos-deps/action.yml

@@ -0,0 +1,47 @@
+# This action installs a few dependencies necessary to build RustPython on macOS. By default it installs
+# autoconf, automake and libtool, but can be configured depending on which libraries are needed:
+#
+# ```
+# - uses: ./.github/actions/install-macos-deps
+#   with:
+#     openssl: true
+#     libtool: false
+# ```
+#
+# See the `inputs` section for all options and their defaults. Note that you must checkout the
+# repository before you can use this action.
+#
+# This action will only install dependencies when the current operating system is macOS. It will do
+# nothing on any other OS (Linux, Windows).
+
+name: Install macOS dependencies
+description: Installs the dependencies necessary to build RustPython on macOS.
+inputs:
+  autoconf:
+    description: Install autoconf (autoconf)
+    required: false
+    default: "true"
+  automake:
+    description: Install automake (automake)
+    required: false
+    default: "true"
+  libtool:
+    description: Install libtool (libtool)
+    required: false
+    default: "true"
+  openssl:
+    description: Install openssl (openssl@3)
+    required: false
+    default: "false"
+runs:
+  using: composite
+  steps:
+    - name: Install macOS dependencies
+      shell: bash
+      if: ${{ runner.os == 'macOS' }}
+      run: >
+        brew install
+        ${{ fromJSON(inputs.autoconf) && 'autoconf' || '' }}
+        ${{ fromJSON(inputs.automake) && 'automake' || '' }}
+        ${{ fromJSON(inputs.libtool) && 'libtool' || '' }}
+        ${{ fromJSON(inputs.openssl) && 'openssl@3' || '' }}

.github/aw/actions-lock.json

@@ -0,0 +1,14 @@
+{
+  "entries": {
+    "actions/github-script@v8": {
+      "repo": "actions/github-script",
+      "version": "v8",
+      "sha": "ed597411d8f924073f98dfc5c65a23a2325f34cd"
+    },
+    "github/gh-aw/actions/setup@v0.43.22": {
+      "repo": "github/gh-aw/actions/setup",
+      "version": "v0.43.22",
+      "sha": "fe858c3e14589bf396594a0b106e634d9065823e"
+    }
+  }
+}

.github/dependabot.yml

@@ -1,13 +1,149 @@
-# Keep GitHub Actions up to date with GitHub's Dependabot...
-# https://docs.github.com/en/code-security/dependabot/working-with-dependabot/keeping-your-actions-up-to-date-with-dependabot
-# https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#package-ecosystem
+# cspell:ignore manyhow tinyvec zeroize
 version: 2
 updates:
-  - package-ecosystem: github-actions
+  - package-ecosystem: cargo
+    directory: /
+    schedule:
+      interval: weekly
+    groups:
+      criterion:
+        patterns:
+          - "criterion*"
+      crypto:
+        patterns:
+          - "digest"
+          - "md-5"
+          - "sha-1"
+          - "sha2"
+          - "sha3"
+          - "blake2"
+          - "hmac"
+          - "pbkdf2"
+      futures:
+        patterns:
+          - "futures*"
+      get-size2:
+        patterns:
+          - "get-size*2"
+      iana-time-zone:
+        patterns:
+          - "iana-time-zone*"
+      jiff:
+        patterns:
+          - "jiff*"
+      lexical:
+        patterns:
+          - "lexical*"
+      libffi:
+        patterns:
+          - "libffi*"
+      malachite:
+        patterns:
+          - "malachite*"
+      manyhow:
+        patterns:
+          - "manyhow*"
+      num:
+        patterns:
+          - "num-bigint"
+          - "num-complex"
+          - "num-integer"
+          - "num-iter"
+          - "num-rational"
+          - "num-traits"
+      num_enum:
+        patterns:
+          - "num_enum*"
+      openssl:
+        patterns:
+          - "openssl*"
+      parking_lot:
+        patterns:
+          - "parking_lot*"
+      phf:
+        patterns:
+          - "phf*"
+      plotters:
+        patterns:
+          - "plotters*"
+      portable-atomic:
+        patterns:
+          - "portable-atomic*"
+      pyo3:
+        patterns:
+          - "pyo3*"
+      quote-use:
+        patterns:
+          - "quote-use*"
+      random:
+        patterns:
+          - "getrandom"
+          - "mt19937"
+          - "rand*"
+      rayon:
+        patterns:
+          - "rayon*"
+      regex:
+        patterns:
+          - "regex*"
+      result-like:
+        patterns:
+          - "result-like*"
+      security-framework:
+        patterns:
+          - "security-framework*"
+      serde:
+        patterns:
+          - "serde"
+          - "serde_core"
+          - "serde_derive"
+      system-configuration:
+        patterns:
+          - "system-configuration*"
+      thiserror:
+        patterns:
+          - "thiserror*"
+      time:
+        patterns:
+          - "time*"
+      tinyvec:
+        patterns:
+          - "tinyvec*"
+      tls_codec:
+        patterns:
+          - "tls_codec*"
+      toml:
+        patterns:
+          - "toml*"
+      wasm-bindgen:
+        patterns:
+          - "wasm-bindgen*"
+      wasmtime:
+        patterns:
+          - "cranelift*"
+          - "wasmtime*"
+      webpki-root:
+        patterns:
+          - "webpki-root*"
+      windows:
+        patterns:
+          - "windows*"
+      zerocopy:
+        patterns:
+          - "zerocopy*"
+      zeroize:
+        patterns:
+          - "zeroize*"
+    ignore:
+      # TODO: Remove when we use ruff from crates.io
+      # for some reason dependabot only updates the Cargo.lock file when dealing
+      # with git dependencies. i.e. not updating the version in Cargo.toml
+      - dependency-name: "ruff_*"
+  - package-ecosystem: github-actions
+    directory: /
+    schedule:
+      interval: weekly
+  - package-ecosystem: npm
     directory: /
-    groups:
-      github-actions:
-        patterns:
-          - "*"  # Group all Actions updates into a single larger pull request
     schedule:
       interval: weekly

.github/workflows/ci.yaml

@@ -16,95 +16,17 @@ concurrency:
   cancel-in-progress: true
 
 env:
-  CARGO_ARGS: --no-default-features --features stdlib,importlib,stdio,encodings,sqlite,ssl
-  # Skip additional tests on Windows. They are checked on Linux and MacOS.
-  # test_glob: many failing tests
-  # test_io: many failing tests
-  # test_os: many failing tests
-  # test_pathlib: support.rmtree() failing
-  # test_posixpath: OSError: (22, 'The filename, directory name, or volume label syntax is incorrect. (os error 123)')
-  # test_venv: couple of failing tests
-  WINDOWS_SKIPS: >-
-    test_glob
-    test_io
-    test_os
-    test_rlcompleter
-    test_pathlib
-    test_posixpath
-    test_venv
-  # PLATFORM_INDEPENDENT_TESTS are tests that do not depend on the underlying OS. They are currently
-  # only run on Linux to speed up the CI.
-  PLATFORM_INDEPENDENT_TESTS: >-
-    test__colorize
-    test_array
-    test_asyncgen
-    test_binop
-    test_bisect
-    test_bool
-    test_bytes
-    test_call
-    test_class
-    test_cmath
-    test_collections
-    test_complex
-    test_contains
-    test_copy
-    test_dataclasses
-    test_decimal
-    test_decorators
-    test_defaultdict
-    test_deque
-    test_dict
-    test_dictcomps
-    test_dictviews
-    test_dis
-    test_enumerate
-    test_exception_variations
-    test_float
-    test_format
-    test_fractions
-    test_genericalias
-    test_genericclass
-    test_grammar
-    test_range
-    test_index
-    test_int
-    test_int_literal
-    test_isinstance
-    test_iter
-    test_iterlen
-    test_itertools
-    test_json
-    test_keyword
-    test_keywordonlyarg
-    test_list
-    test_long
-    test_longexp
-    test_math
-    test_operator
-    test_ordered_dict
-    test_pow
-    test_raise
-    test_richcmp
-    test_scope
-    test_set
-    test_slice
-    test_sort
-    test_string
-    test_string_literals
-    test_strtod
-    test_structseq
-    test_subclassinit
-    test_super
-    test_syntax
-    test_tuple
-    test_types
-    test_unary
-    test_unpack
-    test_weakref
-    test_yield_from
+  CARGO_ARGS: --no-default-features --features stdlib,importlib,stdio,encodings,sqlite,ssl-rustls,host_env
+  CARGO_ARGS_NO_SSL: --no-default-features --features stdlib,importlib,stdio,encodings,sqlite,host_env
+  # Crates excluded from workspace builds:
+  # - rustpython_wasm: requires wasm target
+  # - rustpython-compiler-source: deprecated
+  # - rustpython-venvlauncher: Windows-only
+  WORKSPACE_EXCLUDES: --exclude rustpython_wasm --exclude rustpython-compiler-source --exclude rustpython-venvlauncher
   # Python version targeted by the CI.
-  PYTHON_VERSION: "3.13.1"
+  PYTHON_VERSION: "3.14.3"
+  X86_64_PC_WINDOWS_MSVC_OPENSSL_LIB_DIR: C:\Program Files\OpenSSL\lib\VC\x64\MD
+  X86_64_PC_WINDOWS_MSVC_OPENSSL_INCLUDE_DIR: C:\Program Files\OpenSSL\include
 
 jobs:
   rust_tests:
@@ -112,118 +34,131 @@ jobs:
     env:
       RUST_BACKTRACE: full
     name: Run rust tests
-    runs-on:  ${{ matrix.os }}
-    timeout-minutes: ${{ contains(matrix.os, 'windows') && 45 || 30 }}
+    runs-on: ${{ matrix.os }}
+    timeout-minutes: 45
     strategy:
       matrix:
-        os: [macos-latest, ubuntu-latest, windows-latest]
+        os: [macos-latest, ubuntu-latest, windows-2025]
       fail-fast: false
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+
       - uses: dtolnay/rust-toolchain@stable
         with:
           components: clippy
-      - uses: Swatinem/rust-cache@v2
 
-      - name: Set up the Windows environment
-        shell: bash
-        run: |
-          git config --system core.longpaths true
-          cargo install --target-dir=target -v cargo-vcpkg
-          cargo vcpkg -v build
-        if: runner.os == 'Windows'
-      - name: Set up the Mac environment
-        run: brew install autoconf automake libtool
-        if: runner.os == 'macOS'
+      - uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
+        with:
+          save-if: ${{ github.ref == 'refs/heads/main' }}
+
+      - name: Install macOS dependencies
+        uses: ./.github/actions/install-macos-deps
 
       - name: run clippy
-        run: cargo clippy ${{ env.CARGO_ARGS }} --workspace --all-targets --exclude rustpython_wasm -- -Dwarnings
+        run: cargo clippy ${{ env.CARGO_ARGS }} --workspace --all-targets ${{ env.WORKSPACE_EXCLUDES }} -- -Dwarnings
 
       - name: run rust tests
-        run: cargo test --workspace --exclude rustpython_wasm --verbose --features threading ${{ env.CARGO_ARGS }}
-        if: runner.os != 'macOS'
-      - name: run rust tests
-        run: cargo test --workspace --exclude rustpython_wasm --exclude rustpython-jit --verbose --features threading  ${{ env.CARGO_ARGS }}
-        if: runner.os == 'macOS'
+        run: cargo test --workspace ${{ env.WORKSPACE_EXCLUDES }} --verbose --features threading ${{ env.CARGO_ARGS }}
 
       - name: check compilation without threading
         run: cargo check ${{ env.CARGO_ARGS }}
 
+      - name: check compilation without host_env (sandbox mode)
+        run: |
+          cargo check -p rustpython-vm --no-default-features --features compiler
+          cargo check -p rustpython-stdlib --no-default-features --features compiler
+          cargo build --no-default-features --features stdlib,importlib,stdio,encodings,freeze-stdlib
+        if: runner.os == 'Linux'
+
+      - name: sandbox smoke test
+        run: |
+          target/debug/rustpython extra_tests/snippets/sandbox_smoke.py
+          target/debug/rustpython extra_tests/snippets/stdlib_re.py
+        if: runner.os == 'Linux'
+
+      - name: Test openssl build
+        run: cargo build --no-default-features --features ssl-openssl
+        if: runner.os == 'Linux'
+
+      # - name: Install tk-dev for tkinter build
+      #   run: sudo apt-get update && sudo apt-get install -y tk-dev
+      #   if: runner.os == 'Linux'
+
+      # - name: Test tkinter build
+      #   run: cargo build --features tkinter
+      #   if: runner.os == 'Linux'
+
       - name: Test example projects
-        run:
+        run: |
           cargo run --manifest-path example_projects/barebone/Cargo.toml
           cargo run --manifest-path example_projects/frozen_stdlib/Cargo.toml
         if: runner.os == 'Linux'
 
-      - name: prepare AppleSilicon build
-        uses: dtolnay/rust-toolchain@stable
-        with:
-          target: aarch64-apple-darwin
-        if: runner.os == 'macOS'
-      - name: Check compilation for Apple Silicon
-        run: cargo check --target aarch64-apple-darwin
-        if: runner.os == 'macOS'
-      - name: prepare iOS build
-        uses: dtolnay/rust-toolchain@stable
-        with:
-          target: aarch64-apple-ios
-        if: runner.os == 'macOS'
-      - name: Check compilation for iOS
-        run: cargo check --target aarch64-apple-ios
-        if: runner.os == 'macOS'
+      - name: run update_lib tests
+        run: cargo run -- -m unittest discover -s scripts/update_lib/tests -v
+        env:
+          PYTHONPATH: scripts
+        if: runner.os == 'Linux'
 
-  exotic_targets:
+  cargo_check:
     if: ${{ !contains(github.event.pull_request.labels.*.name, 'skip:ci') }}
     name: Ensure compilation on various targets
-    runs-on: ubuntu-latest
-    timeout-minutes: 30
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        include: 
+        - os: ubuntu-latest
+          targets:
+            - aarch64-linux-android
+            - i686-unknown-linux-gnu
+            - i686-unknown-linux-musl
+            - wasm32-wasip2
+            - x86_64-unknown-freebsd
+          dependencies:
+            gcc-multilib: true
+            musl-tools: true
+        - os: ubuntu-latest
+          targets:
+            - aarch64-unknown-linux-gnu
+          dependencies:
+            gcc-aarch64-linux-gnu: true # conflict with `gcc-multilib`
+        - os: macos-latest
+          targets:
+            - aarch64-apple-ios
+            - x86_64-apple-darwin
+      fail-fast: false
     steps:
-      - uses: actions/checkout@v4
-      - uses: dtolnay/rust-toolchain@stable
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
         with:
-          target: i686-unknown-linux-gnu
+          persist-credentials: false
 
-      - name: Install gcc-multilib and musl-tools
-        run: sudo apt-get update && sudo apt-get install gcc-multilib musl-tools
-      - name: Check compilation for x86 32bit
-        run: cargo check --target i686-unknown-linux-gnu
+      - uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
+        with:
+          prefix-key: v0-rust-${{ join(matrix.targets, '-') }}
+          save-if: ${{ github.ref == 'refs/heads/main' }}
+
+      - name: Install dependencies
+        uses: ./.github/actions/install-linux-deps
+        # zizmor has an issue with dynamic `with`
+        # with: ${{ matrix.dependencies || fromJSON('{}') }}
+        with:
+          gcc-multilib: ${{ matrix.dependencies.gcc-multilib || false }}
+          musl-tools: ${{ matrix.dependencies.musl-tools || false }}
+          gcc-aarch64-linux-gnu: ${{ matrix.dependencies.gcc-aarch64-linux-gnu || false }}
 
       - uses: dtolnay/rust-toolchain@stable
         with:
-          target: aarch64-linux-android
+          targets: ${{ join(matrix.targets, ',') }}
 
-      - name: Check compilation for android
-        run: cargo check --target aarch64-linux-android
-
-      - uses: dtolnay/rust-toolchain@stable
+      - name: Setup Android NDK
+        if: ${{ contains(matrix.targets, 'aarch64-linux-android') }}
+        id: setup-ndk
+        uses: nttld/setup-ndk@v1
         with:
-          target: aarch64-unknown-linux-gnu
-
-      - name: Install gcc-aarch64-linux-gnu
-        run: sudo apt install gcc-aarch64-linux-gnu
-      - name: Check compilation for aarch64 linux gnu
-        run: cargo check --target aarch64-unknown-linux-gnu
-
-      - uses: dtolnay/rust-toolchain@stable
-        with:
-          target: i686-unknown-linux-musl
-
-      - name: Check compilation for musl
-        run: cargo check --target i686-unknown-linux-musl
-
-      - uses: dtolnay/rust-toolchain@stable
-        with:
-          target: x86_64-unknown-freebsd
-
-      - name: Check compilation for freebsd
-        run: cargo check --target x86_64-unknown-freebsd
-
-      - uses: dtolnay/rust-toolchain@stable
-        with:
-          target: x86_64-unknown-freebsd
-
-      - name: Check compilation for freeBSD
-        run: cargo check --target x86_64-unknown-freebsd
+          ndk-version: r27
+          add-to-path: true
 
       # - name: Prepare repository for redox compilation
       #   run: bash scripts/redox/uncomment-cargo.sh
@@ -233,104 +168,201 @@ jobs:
       #     command: check
       #     args: --ignore-rust-version
 
+      - name: Check compilation
+        run: |
+          for target in ${{ join(matrix.targets, ' ') }}
+          do
+            echo "::group::${target}"
+            cargo check --target $target ${{ env.CARGO_ARGS_NO_SSL }}
+            echo "::endgroup::"
+          done
+        env:
+          CC_aarch64_linux_android: ${{ steps.setup-ndk.outputs.ndk-path }}/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android24-clang
+          AR_aarch64_linux_android: ${{ steps.setup-ndk.outputs.ndk-path }}/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-ar
+          CARGO_TARGET_AARCH64_LINUX_ANDROID_LINKER: ${{ steps.setup-ndk.outputs.ndk-path }}/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android24-clang
+
   snippets_cpython:
     if: ${{ !contains(github.event.pull_request.labels.*.name, 'skip:ci') }}
     env:
       RUST_BACKTRACE: full
+      # Tests that can be flaky when running with multiple processes `-j 2`. We will use `-j 1` for these.
+      FLAKY_MP_TESTS: >-
+        test_class
+        test_eintr
+        test_multiprocessing_fork
+        test_multiprocessing_forkserver
+        test_multiprocessing_spawn
     name: Run snippets and cpython tests
-    runs-on:  ${{ matrix.os }}
-    timeout-minutes: ${{ contains(matrix.os, 'windows') && 45 || 30 }}
+    runs-on: ${{ matrix.os }}
     strategy:
       matrix:
-        os: [macos-latest, ubuntu-latest, windows-latest]
+        include:
+        - os: macos-latest
+          extra_test_args:
+            - '-u all'
+          env_polluting_tests: []
+          skips: []
+          timeout: 50
+        - os: ubuntu-latest
+          extra_test_args:
+            - '-u all'
+          env_polluting_tests: []
+          skips: []
+          timeout: 60
+        - os: windows-2025
+          extra_test_args: [] # TODO: Enable '-u all'
+          env_polluting_tests: []
+          skips:
+            - test_rlcompleter
+            - test_pathlib # panic by surrogate chars
+            - test_posixpath # OSError: (22, 'The filename, directory name, or volume label syntax is incorrect. (os error 123)')
+            - test_venv # couple of failing tests
+          timeout: 50
       fail-fast: false
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+
       - uses: dtolnay/rust-toolchain@stable
-      - uses: Swatinem/rust-cache@v2
-      - uses: actions/setup-python@v5
+
+      - uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
+        with:
+          save-if: ${{ github.ref == 'refs/heads/main' }}
+
+      - uses: actions/setup-python@v6.2.0
         with:
           python-version: ${{ env.PYTHON_VERSION }}
-      - name: Set up the Windows environment
-        shell: bash
-        run: |
-          git config --system core.longpaths true
-          cargo install cargo-vcpkg
-          cargo vcpkg build
-        if: runner.os == 'Windows'
-      - name: Set up the Mac environment
-        run: brew install autoconf automake libtool openssl@3
-        if: runner.os == 'macOS'
-      - name: build rustpython
-        run: cargo build --release --verbose --features=threading ${{ env.CARGO_ARGS }}
-        if: runner.os == 'macOS'
-      - name: build rustpython
-        run: cargo build --release --verbose --features=threading ${{ env.CARGO_ARGS }},jit
-        if: runner.os != 'macOS'
-      - uses: actions/setup-python@v5
+
+      - name: Install macOS dependencies
+        uses: ./.github/actions/install-macos-deps
         with:
-          python-version: ${{ env.PYTHON_VERSION }}
+          openssl: true
+
+      - name: build rustpython
+        run: cargo build --release --verbose --features=threading,jit ${{ env.CARGO_ARGS }}
+
       - name: run snippets
         run: python -m pip install -r requirements.txt && pytest -v
         working-directory: ./extra_tests
-      - if: runner.os == 'Linux'
-        name: run cpython platform-independent tests
-        run:
-          target/release/rustpython -m test -j 1 -u all --slowest --fail-env-changed -v ${{ env.PLATFORM_INDEPENDENT_TESTS }}
-      - if: runner.os == 'Linux'
-        name: run cpython platform-dependent tests (Linux)
-        run: target/release/rustpython -m test -j 1 -u all --slowest --fail-env-changed -v -x ${{ env.PLATFORM_INDEPENDENT_TESTS }}
-      - if: runner.os == 'macOS'
-        name: run cpython platform-dependent tests (MacOS)
-        run: target/release/rustpython -m test -j 1 --slowest --fail-env-changed -v -x ${{ env.PLATFORM_INDEPENDENT_TESTS }}
-      - if: runner.os == 'Windows'
-        name: run cpython platform-dependent tests (windows partial - fixme)
-        run:
-          target/release/rustpython -m test -j 1 --slowest --fail-env-changed -v -x ${{ env.PLATFORM_INDEPENDENT_TESTS }} ${{ env.WINDOWS_SKIPS }}
+
+      - name: Detect available cores
+        id: cores
+        shell: bash
+        run: |
+          cores=$(python -c 'print(__import__("os").process_cpu_count())')
+          echo "cores=${cores}" >> $GITHUB_OUTPUT
+
+      - name: Run CPython tests
+        run: |
+          target/release/rustpython -m test -j ${{ steps.cores.outputs.cores }} ${{ join(matrix.extra_test_args, ' ') }} --slowest --fail-env-changed --timeout 600 -v -x ${{ env.FLAKY_MP_TESTS }} ${{ join(matrix.skips, ' ') }}
+        timeout-minutes: ${{ matrix.timeout }}
+        env:
+          RUSTPYTHON_SKIP_ENV_POLLUTERS: true
+
+      - name: Run flaky MP CPython tests
+        run: |
+          target/release/rustpython -m test -j 1 ${{ join(matrix.extra_test_args, ' ') }} --slowest --fail-env-changed --timeout 600 -v ${{ env.FLAKY_MP_TESTS }}
+        timeout-minutes: ${{ matrix.timeout }}
+        env:
+          RUSTPYTHON_SKIP_ENV_POLLUTERS: true
+
+      - name: run cpython tests to check if env polluters have stopped polluting
+        shell: bash
+        run: |
+          for thing in ${{ join(matrix.env_polluting_tests, ' ') }}; do
+            for i in $(seq 1 10); do
+              set +e
+              target/release/rustpython -m test -j 1 --slowest --fail-env-changed --timeout 600 -v ${thing}
+              exit_code=$?
+              set -e
+              if [ ${exit_code} -eq 3 ]; then
+                echo "Test ${thing} polluted the environment on attempt ${i}."
+                break
+              fi
+            done
+            if [ ${exit_code} -ne 3 ]; then
+              echo "Test ${thing} is no longer polluting the environment after ${i} attempts!"
+              echo "Please remove ${thing} from matrix.env_polluting_tests in '.github/workflows/ci.yaml'."
+              echo "Please also remove the skip decorators that include the word 'POLLUTERS' in ${thing}."
+              if [ ${exit_code} -ne 0 ]; then
+                echo "Test ${thing} failed with exit code ${exit_code}."
+                echo "Please investigate which test item in ${thing} is failing and either mark it as an expected failure or a skip."
+              fi
+              exit 1
+            fi
+          done
+        timeout-minutes: 15
+
       - if: runner.os != 'Windows'
         name: check that --install-pip succeeds
         run: |
           mkdir site-packages
           target/release/rustpython --install-pip ensurepip --user
           target/release/rustpython -m pip install six
-      - if: runner.os != 'Windows'
-        name: Check that ensurepip succeeds.
+
+      - name: Check that ensurepip succeeds.
         run: |
           target/release/rustpython -m ensurepip
           target/release/rustpython -c "import pip"
+
       - if: runner.os != 'Windows'
         name: Check if pip inside venv is functional
         run: |
           target/release/rustpython -m venv testvenv
           testvenv/bin/rustpython -m pip install wheel
+
       - name: Check whats_left is not broken
-        run: python -I whats_left.py
+        shell: bash
+        run: python -I scripts/whats_left.py ${{ env.CARGO_ARGS }} --features jit
 
   lint:
-    name: Check Rust code with rustfmt and clippy
+    name: Lint Rust & Python code
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v4
-      - uses: dtolnay/rust-toolchain@stable
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
         with:
-            components: rustfmt, clippy
-      - name: run rustfmt
-        run: cargo fmt --check
-      - name: run clippy on wasm
-        run: cargo clippy --manifest-path=wasm/lib/Cargo.toml -- -Dwarnings
-      - uses: actions/setup-python@v5
+          persist-credentials: false
+
+      - uses: actions/setup-python@v6.2.0
         with:
           python-version: ${{ env.PYTHON_VERSION }}
-      - name: install ruff
-        run: python -m pip install ruff==0.11.8
+
+      - name: Check for redundant test patches
+        run: python scripts/check_redundant_patches.py
+
+      - uses: dtolnay/rust-toolchain@stable
+        with:
+          components: clippy
+
+      - name: run clippy on wasm
+        run: cargo clippy --manifest-path=crates/wasm/Cargo.toml -- -Dwarnings
+
       - name: Ensure docs generate no warnings
-        run: cargo doc
-      - name: run ruff check
-        run: ruff check --diff
-      - name: run ruff format
-        run: ruff format --check
+        run: cargo doc --locked
+
+      - name: Ensure Lib/_opcode_metadata is updated
+        run: |
+          python scripts/generate_opcode_metadata.py
+          if [ -n "$(git status --porcelain)" ]; then
+            exit 1
+          fi
+
+      - name: Install ruff
+        uses: astral-sh/ruff-action@4919ec5cf1f49eff0871dbcea0da843445b837e6 # v3.6.1
+        with:
+          version: "0.15.5"
+          args: "--version"
+
+      - run: ruff check --diff
+
+      - run: ruff format --check
+
       - name: install prettier
-        run: yarn global add prettier && echo "$(yarn global bin)" >>$GITHUB_PATH
+        run: |
+          yarn global add prettier
+          yarn global bin >> "$GITHUB_PATH"
+
       - name: check wasm code with prettier
         # prettier doesn't handle ignore files very well: https://github.com/prettier/prettier/issues/8506
         run: cd wasm && git ls-files -z | xargs -0 prettier --check -u
@@ -338,9 +370,9 @@ jobs:
       - name: install extra dictionaries
         run: npm install @cspell/dict-en_us @cspell/dict-cpp @cspell/dict-python @cspell/dict-rust @cspell/dict-win32 @cspell/dict-shell
       - name: spell checker
-        uses: streetsidesoftware/cspell-action@v7
+        uses: streetsidesoftware/cspell-action@v8
         with:
-          files: '**/*.rs'
+          files: "**/*.rs"
           incremental_files_only: true
 
   miri:
@@ -349,19 +381,27 @@ jobs:
     runs-on: ubuntu-latest
     timeout-minutes: 30
     env:
-      RUSTUP_TOOLCHAIN: nightly-2025-07-18 # TODO: Set to "nightly" once https://github.com/rust-lang/rust/issues/144168 is fixed
+      NIGHTLY_CHANNEL: nightly
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+
       - uses: dtolnay/rust-toolchain@master
         with:
-            toolchain: ${{ env.RUSTUP_TOOLCHAIN }}
-            components: miri
+          toolchain: ${{ env.NIGHTLY_CHANNEL }}
+          components: miri
+
+      - uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
+        with:
+          save-if: ${{ github.ref == 'refs/heads/main' }}
 
-      - uses: Swatinem/rust-cache@v2
       - name: Run tests under miri
-        # miri-ignore-leaks because the type-object circular reference means that there will always be
-        # a memory leak, at least until we have proper cyclic gc
-        run: MIRIFLAGS='-Zmiri-ignore-leaks' cargo +${{ env.RUSTUP_TOOLCHAIN }} miri test -p rustpython-vm -- miri_test
+        run: cargo +${{ env.NIGHTLY_CHANNEL }} miri test -p rustpython-vm -- miri_test
+        env:
+          # miri-ignore-leaks because the type-object circular reference means that there will always be
+          # a memory leak, at least until we have proper cyclic gc
+          MIRIFLAGS: "-Zmiri-ignore-leaks"
 
   wasm:
     if: ${{ !contains(github.event.pull_request.labels.*.name, 'skip:ci') }}
@@ -369,10 +409,16 @@ jobs:
     runs-on: ubuntu-latest
     timeout-minutes: 30
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+
       - uses: dtolnay/rust-toolchain@stable
 
-      - uses: Swatinem/rust-cache@v2
+      - uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
+        with:
+          save-if: ${{ github.ref == 'refs/heads/main' }}
+
       - name: install wasm-pack
         run: curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh
       - name: install geckodriver
@@ -380,18 +426,19 @@ jobs:
           wget https://github.com/mozilla/geckodriver/releases/download/v0.36.0/geckodriver-v0.36.0-linux64.tar.gz
           mkdir geckodriver
           tar -xzf geckodriver-v0.36.0-linux64.tar.gz -C geckodriver
-      - uses: actions/setup-python@v5
+      - uses: actions/setup-python@v6.2.0
         with:
           python-version: ${{ env.PYTHON_VERSION }}
       - run: python -m pip install -r requirements.txt
         working-directory: ./wasm/tests
-      - uses: actions/setup-node@v4
+      - uses: actions/setup-node@v6
         with:
           cache: "npm"
           cache-dependency-path: "wasm/demo/package-lock.json"
       - name: run test
         run: |
-          export PATH=$PATH:`pwd`/../../geckodriver
+          driver_path="$(pwd)/../../geckodriver"
+          export PATH="$PATH:${driver_path}"
           npm install
           npm run test
         env:
@@ -401,11 +448,13 @@ jobs:
         with: { wabt-version: "1.0.36" }
       - name: check wasm32-unknown without js
         run: |
-          cd wasm/wasm-unknown-test
-          cargo build --release --verbose
-          if wasm-objdump -xj Import target/wasm32-unknown-unknown/release/wasm_unknown_test.wasm; then
-            echo "ERROR: wasm32-unknown module expects imports from the host environment" >2
+          cd example_projects/wasm32_without_js/rustpython-without-js
+          cargo build
+          cd ..
+          if wasm-objdump -xj Import rustpython-without-js/target/wasm32-unknown-unknown/debug/rustpython_without_js.wasm; then
+            echo "ERROR: wasm32-unknown module expects imports from the host environment" >&2
           fi
+          cargo run --release --manifest-path wasm-runtime/Cargo.toml rustpython-without-js/target/wasm32-unknown-unknown/debug/rustpython_without_js.wasm
       - name: build notebook demo
         if: github.ref == 'refs/heads/release'
         run: |
@@ -430,19 +479,55 @@ jobs:
     runs-on: ubuntu-latest
     timeout-minutes: 30
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+
       - uses: dtolnay/rust-toolchain@stable
         with:
           target: wasm32-wasip1
 
-      - uses: Swatinem/rust-cache@v2
+      - uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
+        with:
+          save-if: ${{ github.ref == 'refs/heads/main' }}
+
       - name: Setup Wasmer
         uses: wasmerio/setup-wasmer@v3
+
       - name: Install clang
-        run: sudo apt-get update && sudo apt-get install clang -y
+        uses: ./.github/actions/install-linux-deps
+        with:
+          clang: true
+
       - name: build rustpython
         run: cargo build --release --target wasm32-wasip1 --features freeze-stdlib,stdlib --verbose
       - name: run snippets
-        run: wasmer run --dir `pwd` target/wasm32-wasip1/release/rustpython.wasm -- `pwd`/extra_tests/snippets/stdlib_random.py
+        run: wasmer run --dir $(pwd) target/wasm32-wasip1/release/rustpython.wasm -- "$(pwd)/extra_tests/snippets/stdlib_random.py"
       - name: run cpython unittest
-        run: wasmer run --dir `pwd` target/wasm32-wasip1/release/rustpython.wasm -- `pwd`/Lib/test/test_int.py
+        run: wasmer run --dir $(pwd) target/wasm32-wasip1/release/rustpython.wasm -- "$(pwd)/Lib/test/test_int.py"
+
+  cargo-shear:
+    name: cargo shear
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+
+      - uses: cargo-bins/cargo-binstall@1800853f2578f8c34492ec76154caef8e163fbca # v1.17.7
+
+      - run: cargo binstall --no-confirm cargo-shear
+
+      - run: cargo shear
+
+  security-lint:
+    runs-on: ubuntu-latest
+    permissions:
+      security-events: write
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+
+      - name: Run zizmor
+        uses: zizmorcore/zizmor-action@71321a20a9ded102f6e9ce5718a2fcec2c4f70d8 # v0.5.2

.github/workflows/comment-commands.yml

@@ -7,7 +7,7 @@ on:
 jobs:
   issue_assign:
     if: (!github.event.issue.pull_request) && github.event.comment.body == 'take'
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-slim
 
     concurrency:
       group: ${{ github.actor }}-issue-assign

.github/workflows/cron-ci.yaml

@@ -1,16 +1,19 @@
 on:
   schedule:
-    - cron: '0 0 * * 6'
+    - cron: "0 0 * * 6"
   workflow_dispatch:
   push:
     paths:
       - .github/workflows/cron-ci.yaml
+  pull_request:
+    paths:
+      - .github/workflows/cron-ci.yaml
 
 name: Periodic checks/tasks
 
 env:
-  CARGO_ARGS: --no-default-features --features stdlib,importlib,encodings,ssl,jit
-  PYTHON_VERSION: "3.13.1"
+  CARGO_ARGS: --no-default-features --features stdlib,importlib,stdio,encodings,ssl-rustls,jit,host_env
+  PYTHON_VERSION: "3.14.3"
 
 jobs:
   # codecov collects code coverage data from the rust tests, python snippets and python test suite.
@@ -18,34 +21,45 @@ jobs:
   codecov:
     name: Collect code coverage data
     runs-on: ubuntu-latest
+    # Disable this scheduled job when running on a fork.
+    if: ${{ github.repository == 'RustPython/RustPython' || github.event_name != 'schedule' }}
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+
       - uses: dtolnay/rust-toolchain@stable
       - uses: taiki-e/install-action@cargo-llvm-cov
-      - uses: actions/setup-python@v5
+      - uses: actions/setup-python@v6.2.0
         with:
           python-version: ${{ env.PYTHON_VERSION }}
       - run: sudo apt-get update && sudo apt-get -y install lcov
       - name: Run cargo-llvm-cov with Rust tests.
-        run: cargo llvm-cov --no-report --workspace --exclude rustpython_wasm --verbose --no-default-features --features stdlib,importlib,encodings,ssl,jit
+        run: cargo llvm-cov --no-report --workspace --exclude rustpython_wasm --exclude rustpython-compiler-source --exclude rustpython-venvlauncher --verbose --no-default-features --features stdlib,importlib,stdio,encodings,ssl-rustls,jit,host_env
       - name: Run cargo-llvm-cov with Python snippets.
         run: python scripts/cargo-llvm-cov.py
         continue-on-error: true
       - name: Run cargo-llvm-cov with Python test suite.
-        run: cargo llvm-cov --no-report run -- -m test -u all --slowest --fail-env-changed 
+        run: cargo llvm-cov --no-report run -- -m test -u all --slowest --fail-env-changed
         continue-on-error: true
       - name: Prepare code coverage data
         run: cargo llvm-cov report --lcov --output-path='codecov.lcov'
       - name: Upload to Codecov
+        if: ${{ github.event_name != 'pull_request' }}
         uses: codecov/codecov-action@v5
         with:
-          file: ./codecov.lcov
+          files: ./codecov.lcov
 
   testdata:
     name: Collect regression test data
     runs-on: ubuntu-latest
+    # Disable this scheduled job when running on a fork.
+    if: ${{ github.repository == 'RustPython/RustPython' || github.event_name != 'schedule' }}
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: true
+
       - uses: dtolnay/rust-toolchain@stable
       - name: build rustpython
         run: cargo build --release --verbose
@@ -54,6 +68,7 @@ jobs:
         env:
           RUSTPYTHONPATH: ${{ github.workspace }}/Lib
       - name: upload tests data to the website
+        if: ${{ github.event_name != 'pull_request' }}
         env:
           SSHKEY: ${{ secrets.ACTIONS_TESTS_DATA_DEPLOY_KEY }}
           GITHUB_ACTOR: ${{ github.actor }}
@@ -73,21 +88,27 @@ jobs:
   whatsleft:
     name: Collect what is left data
     runs-on: ubuntu-latest
+    # Disable this scheduled job when running on a fork.
+    if: ${{ github.repository == 'RustPython/RustPython' || github.event_name != 'schedule' }}
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: true
+
       - uses: dtolnay/rust-toolchain@stable
-      - uses: actions/setup-python@v5
+      - uses: actions/setup-python@v6.2.0
         with:
           python-version: ${{ env.PYTHON_VERSION }}
       - name: build rustpython
         run: cargo build --release --verbose
       - name: Collect what is left data
         run: |
-          chmod +x ./whats_left.py
-          ./whats_left.py --features "ssl,sqlite" > whats_left.temp
+          chmod +x ./scripts/whats_left.py
+          ./scripts/whats_left.py --features "ssl,sqlite" > whats_left.temp
         env:
           RUSTPYTHONPATH: ${{ github.workspace }}/Lib
       - name: Upload data to the website
+        if: ${{ github.event_name != 'pull_request' }}
         env:
           SSHKEY: ${{ secrets.ACTIONS_TESTS_DATA_DEPLOY_KEY }}
           GITHUB_ACTOR: ${{ github.actor }}
@@ -103,6 +124,23 @@ jobs:
           rm ./_data/whats_left/modules.csv
           echo -e "module" > ./_data/whats_left/modules.csv
           cat ./_data/whats_left.temp | grep "(entire module)" | cut -d ' ' -f 1 | sort >> ./_data/whats_left/modules.csv
+          awk -f - ./_data/whats_left.temp > ./_data/whats_left/builtin_items.csv <<'EOF'
+          BEGIN {
+              OFS=","
+              print "builtin,name,is_inherited"
+          }
+          /^# builtin items/ { in_section=1; next }
+          /^$/ { if (in_section) exit }
+          in_section {
+              split($1, a, ".")
+              rest = ""
+              idx = index($0, " ")
+              if (idx > 0) {
+                  rest = substr($0, idx+1)
+              }
+              print a[1], $1, rest
+          }
+          EOF
           git add -A
           if git -c user.name="Github Actions" -c user.email="actions@github.com" commit -m "Update what is left results" --author="$GITHUB_ACTOR"; then
             git push
@@ -111,12 +149,17 @@ jobs:
   benchmark:
     name: Collect benchmark data
     runs-on: ubuntu-latest
+    # Disable this scheduled job when running on a fork.
+    if: ${{ github.repository == 'RustPython/RustPython' || github.event_name != 'schedule' }}
     steps:
-      - uses: actions/checkout@v4
-      - uses: dtolnay/rust-toolchain@stable
-      - uses: actions/setup-python@v5
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
         with:
-          python-version: 3.9
+          persist-credentials: true
+
+      - uses: dtolnay/rust-toolchain@stable
+      - uses: actions/setup-python@v6.2.0
+        with:
+          python-version: ${{ env.PYTHON_VERSION }}
       - run: cargo install cargo-criterion
       - name: build benchmarks
         run: cargo build --release --benches
@@ -127,16 +170,17 @@ jobs:
       - name: restructure generated files
         run: |
           cd ./target/criterion/reports
-          find -type d -name cpython | xargs rm -rf
-          find -type d -name rustpython | xargs rm -rf
-          find -mindepth 2 -maxdepth 2 -name violin.svg | xargs rm -rf
-          find -type f -not -name violin.svg | xargs rm -rf
-          for file in $(find -type f -name violin.svg); do mv $file $(echo $file | sed -E "s_\./([^/]+)/([^/]+)/violin\.svg_./\1/\2.svg_"); done
-          find -mindepth 2 -maxdepth 2 -type d | xargs rm -rf
+          find . -type d -name cpython -print0 | xargs -0 rm -rf
+          find . -type d -name rustpython -print0 | xargs -0 rm -rf
+          find . -mindepth 2 -maxdepth 2 -name violin.svg -print0 | xargs -0 rm -rf
+          find . -type f -not -name violin.svg -print0 | xargs -0 rm -rf
+          find . -type f -name violin.svg -exec sh -c 'for file; do mv "$file" "$(echo "$file" | sed -E "s_\./([^/]+)/([^/]+)/violin\.svg_./\1/\2.svg_")"; done' _ {} +
+          find . -mindepth 2 -maxdepth 2 -type d -print0 | xargs -0 rm -rf
           cd ..
           mv reports/* .
           rmdir reports
       - name: upload benchmark data to the website
+        if: ${{ github.event_name != 'pull_request' }}
         env:
           SSHKEY: ${{ secrets.ACTIONS_TESTS_DATA_DEPLOY_KEY }}
         run: |
@@ -148,7 +192,11 @@ jobs:
           cd website
           rm -rf ./assets/criterion
           cp -r ../target/criterion ./assets/criterion
-          git add ./assets/criterion
+          printf '{\n  "generated_at": "%s",\n  "rustpython_commit": "%s",\n  "rustpython_ref": "%s"\n}\n' \
+            "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
+            "${{ github.sha }}" \
+            "${{ github.ref_name }}" > ./_data/criterion-metadata.json
+          git add ./assets/criterion ./_data/criterion-metadata.json
           if git -c user.name="Github Actions" -c user.email="actions@github.com" commit -m "Update benchmark results"; then
             git push
           fi

.github/workflows/lib-deps-check.yaml

@@ -0,0 +1,123 @@
+name: Lib Dependencies Check
+
+on:
+  pull_request_target:
+    types: [opened, synchronize, reopened]
+    paths:
+      - "Lib/**"
+
+concurrency:
+  group: lib-deps-${{ github.event.pull_request.number }}
+  cancel-in-progress: true
+
+env:
+  PYTHON_VERSION: "3.14.3"
+
+jobs:
+  check_deps:
+    permissions:
+      pull-requests: write
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - name: Checkout base branch
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          # Use base branch for scripts (security: don't run PR code with elevated permissions)
+          ref: ${{ github.event.pull_request.base.ref }}
+          fetch-depth: 0
+          persist-credentials: false
+
+      - name: Fetch PR head
+        run: |
+          git fetch origin ${{ github.event.pull_request.head.sha }}
+
+      - name: Checkout PR Lib files
+        run: |
+          # Checkout only Lib/ directory from PR head for accurate comparison
+          git checkout ${{ github.event.pull_request.head.sha }} -- Lib/
+
+      - name: Checkout CPython
+        run: |
+          git clone --depth 1 --branch "v${{ env.PYTHON_VERSION }}" https://github.com/python/cpython.git cpython
+
+      - name: Get changed Lib files
+        id: changed-files
+        run: |
+          # Get the list of changed files under Lib/
+          changed=$(git diff --name-only ${{ github.event.pull_request.base.sha }} ${{ github.event.pull_request.head.sha }} -- 'Lib/*.py' 'Lib/**/*.py' | head -50)
+          echo "Changed files:"
+          echo "$changed"
+
+          # Extract unique module names
+          modules=""
+          for file in $changed; do
+            if [[ "$file" == Lib/test/* ]]; then
+              # Test files: Lib/test/test_pydoc.py -> test_pydoc, Lib/test/test_pydoc/foo.py -> test_pydoc
+              module=$(echo "$file" | sed -E 's|^Lib/test/||; s|\.py$||; s|/.*||')
+              # Skip non-test files in test/ (e.g., support.py, __init__.py)
+              if [[ ! "$module" == test_* ]]; then
+                continue
+              fi
+            else
+              # Lib files: Lib/foo.py -> foo, Lib/foo/__init__.py -> foo
+              module=$(echo "$file" | sed -E 's|^Lib/||; s|/__init__\.py$||; s|\.py$||; s|/.*||')
+            fi
+            if [[ -n "$module" && ! " $modules " =~ " $module " ]]; then
+              modules="$modules $module"
+            fi
+          done
+
+          modules=$(echo "$modules" | xargs)  # trim whitespace
+          echo "Detected modules: $modules"
+          echo "modules=$modules" >> $GITHUB_OUTPUT
+
+      - name: Setup Python
+        if: steps.changed-files.outputs.modules != ''
+        uses: actions/setup-python@v6.2.0
+        with:
+          python-version: "${{ env.PYTHON_VERSION }}"
+
+      - name: Run deps check
+        if: steps.changed-files.outputs.modules != ''
+        id: deps-check
+        run: |
+          # Run deps for all modules at once
+          python scripts/update_lib deps ${{ steps.changed-files.outputs.modules }} --depth 2 > /tmp/deps_output.txt 2>&1 || true
+
+          # Read output for GitHub Actions
+          echo "deps_output<<EOF" >> $GITHUB_OUTPUT
+          cat /tmp/deps_output.txt >> $GITHUB_OUTPUT
+          echo "EOF" >> $GITHUB_OUTPUT
+
+          # Check if there's any meaningful output
+          if [ -s /tmp/deps_output.txt ]; then
+            echo "has_output=true" >> $GITHUB_OUTPUT
+          else
+            echo "has_output=false" >> $GITHUB_OUTPUT
+          fi
+
+      - name: Post comment
+        if: steps.deps-check.outputs.has_output == 'true'
+        uses: marocchino/sticky-pull-request-comment@v3
+        with:
+          header: lib-deps-check
+          number: ${{ github.event.pull_request.number }}
+          message: |
+            ## 📦 Library Dependencies
+
+            The following Lib/ modules were modified. Here are their dependencies:
+
+            ${{ steps.deps-check.outputs.deps_output }}
+
+            **Legend:**
+            - `[+]` path exists in CPython
+            - `[x]` up-to-date, `[ ]` outdated
+
+      - name: Remove comment if no Lib changes
+        if: steps.changed-files.outputs.modules == ''
+        uses: marocchino/sticky-pull-request-comment@v3
+        with:
+          header: lib-deps-check
+          number: ${{ github.event.pull_request.number }}
+          delete: true

.github/workflows/pr-format.yaml

@@ -0,0 +1,74 @@
+name: Format Check
+
+# This workflow triggers when a PR is opened/updated
+# Posts inline suggestion comments instead of auto-committing
+on:
+  pull_request:
+    types: [opened, synchronize, reopened]
+    branches:
+      - main
+      - release
+
+concurrency:
+  group: format-check-${{ github.event.pull_request.number }}
+  cancel-in-progress: true
+
+env:
+  PYTHON_VERSION: "3.14.3"
+
+jobs:
+  format_check:
+    permissions:
+      contents: read
+      pull-requests: write
+    runs-on: ubuntu-latest
+    timeout-minutes: 60
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+
+      - uses: reviewdog/action-actionlint@0d952c597ef8459f634d7145b0b044a9699e5e43 # v1.71.0
+
+      - name: Setup Rust
+        uses: dtolnay/rust-toolchain@stable
+        with:
+          components: rustfmt
+
+      - name: Run cargo fmt
+        run: cargo fmt --all
+
+      - name: Install ruff
+        uses: astral-sh/ruff-action@4919ec5cf1f49eff0871dbcea0da843445b837e6 # v3.6.1
+        with:
+          version: "0.15.4"
+          args: "--version"
+
+      - name: Run ruff format
+        run: ruff format
+
+      - name: Run ruff check import sorting
+        run: ruff check --select I --fix
+
+      - uses: actions/setup-python@v6.2.0
+        with:
+          python-version: ${{ env.PYTHON_VERSION }}
+
+      - name: Run generate_opcode_metadata.py
+        run: python scripts/generate_opcode_metadata.py
+
+      - name: Check for formatting changes
+        run: |
+          if ! git diff --exit-code; then
+            echo "::error::Formatting changes detected. Please run 'cargo fmt --all', 'ruff format', and 'ruff check --select I --fix' locally."
+            exit 1
+          fi
+
+      - name: Post formatting suggestions
+        if: failure()
+        uses: reviewdog/action-suggester@v1
+        with:
+          tool_name: auto-format
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          level: warning
+          filter_mode: diff_context

.github/workflows/release.yml

@@ -17,10 +17,14 @@ permissions:
 
 env:
   CARGO_ARGS: --no-default-features --features stdlib,importlib,encodings,sqlite,ssl
+  X86_64_PC_WINDOWS_MSVC_OPENSSL_LIB_DIR: C:\Program Files\OpenSSL\lib\VC\x64\MD
+  X86_64_PC_WINDOWS_MSVC_OPENSSL_INCLUDE_DIR: C:\Program Files\OpenSSL\include
 
 jobs:
   build:
     runs-on: ${{ matrix.platform.runner }}
+    # Disable this scheduled job when running on a fork.
+    if: ${{ github.repository == 'RustPython/RustPython' || github.event_name != 'schedule' }}
     strategy:
       matrix:
         platform:
@@ -40,28 +44,24 @@ jobs:
             target: aarch64-apple-darwin
 #          - runner: macos-latest
 #            target: x86_64-apple-darwin
-          - runner: windows-latest
+          - runner: windows-2025
             target: x86_64-pc-windows-msvc
-#          - runner: windows-latest
+#          - runner: windows-2025
 #            target: i686-pc-windows-msvc
-#          - runner: windows-latest
+#          - runner: windows-2025
 #            target: aarch64-pc-windows-msvc
       fail-fast: false
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+
       - uses: dtolnay/rust-toolchain@stable
       - uses: cargo-bins/cargo-binstall@main
 
       - name: Set up Environment
         shell: bash
         run: rustup target add ${{ matrix.platform.target }}
-      - name: Set up Windows Environment
-        shell: bash
-        run: |
-          git config --global core.longpaths true
-          cargo install --target-dir=target -v cargo-vcpkg
-          cargo vcpkg -v build
-        if: runner.os == 'Windows'
       - name: Set up MacOS Environment
         run: brew install autoconf automake libtool
         if: runner.os == 'macOS'
@@ -81,15 +81,20 @@ jobs:
         if: runner.os == 'Windows'
 
       - name: Upload Binary Artifacts
-        uses: actions/upload-artifact@v4
+        uses: actions/upload-artifact@v7.0.0
         with:
           name: rustpython-release-${{ runner.os }}-${{ matrix.platform.target }}
           path: target/rustpython-release-${{ runner.os }}-${{ matrix.platform.target }}*
 
   build-wasm:
     runs-on: ubuntu-latest
+    # Disable this scheduled job when running on a fork.
+    if: ${{ github.repository == 'RustPython/RustPython' || github.event_name != 'schedule' }}
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+
       - uses: dtolnay/rust-toolchain@stable
         with:
           targets: wasm32-wasip1
@@ -101,14 +106,14 @@ jobs:
         run: cp target/wasm32-wasip1/release/rustpython.wasm target/rustpython-release-wasm32-wasip1.wasm
 
       - name: Upload Binary Artifacts
-        uses: actions/upload-artifact@v4
+        uses: actions/upload-artifact@v7.0.0
         with:
           name: rustpython-release-wasm32-wasip1
           path: target/rustpython-release-wasm32-wasip1.wasm
 
       - name: install wasm-pack
         run: curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh
-      - uses: actions/setup-node@v4
+      - uses: actions/setup-node@v6
       - uses: mwilliamson/setup-wabt-action@v3
         with: { wabt-version: "1.0.30" }
       - name: build demo
@@ -136,15 +141,25 @@ jobs:
 
   release:
     runs-on: ubuntu-latest
+    # Disable this scheduled job when running on a fork.
+    if: ${{ github.repository == 'RustPython/RustPython' || github.event_name != 'schedule' }}
     needs: [build, build-wasm]
     steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+
       - name: Download Binary Artifacts
-        uses: actions/download-artifact@v4
+        uses: actions/download-artifact@v8.0.1
         with:
           path: bin
           pattern: rustpython-*
           merge-multiple: true
 
+      - name: Create Lib Archive
+        run: |
+          zip -r bin/rustpython-lib.zip Lib/
+
       - name: List Binaries
         run: |
           ls -lah bin/
@@ -154,8 +169,9 @@ jobs:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           tag: ${{ github.ref_name }}
           run: ${{ github.run_number }}
+          PRE_RELEASE_INPUT: ${{ github.event.inputs.pre-release }}
         run: |
-          if [[ "${{ github.event.inputs.pre-release }}" == "false" ]]; then
+          if [[ "${PRE_RELEASE_INPUT}" == "false" ]]; then
             RELEASE_TYPE_NAME=Release
             PRERELEASE_ARG=
           else
@@ -168,6 +184,7 @@ jobs:
               --repo="$GITHUB_REPOSITORY" \
               --title="RustPython $RELEASE_TYPE_NAME $today-$tag #$run" \
               --target="$tag" \
+              --notes "⚠️ **Important**: To run RustPython, you must download both the binary for your platform AND the \`rustpython-lib.zip\` archive. Extract the Lib directory from the archive to the same location as the binary, or set the \`RUSTPYTHONPATH\` environment variable to point to the Lib directory." \
               --generate-notes \
               $PRERELEASE_ARG \
               bin/rustpython-release-*

.github/workflows/update-doc-db.yml

@@ -0,0 +1,124 @@
+name: Update doc DB
+
+permissions:
+  contents: write
+  pull-requests: write
+
+on:
+  workflow_dispatch:
+    inputs:
+      python-version:
+        description: Target python version to generate doc db for
+        type: string
+        default: "3.14.3"
+      base-ref:
+        description: Base branch to create the update branch from
+        type: string
+        default: "main"
+
+defaults:
+  run:
+    shell: bash
+
+jobs:
+  generate:
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        os:
+          - ubuntu-latest
+          - windows-latest
+          - macos-latest
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+          sparse-checkout: |
+            crates/doc
+
+      - uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
+        with:
+          python-version: ${{ inputs.python-version }}
+
+      - name: Generate docs
+        run: python crates/doc/generate.py
+
+      - uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0
+        with:
+          name: doc-db-${{ inputs.python-version }}-${{ matrix.os }}
+          path: "crates/doc/generated/*.json"
+          if-no-files-found: error
+          retention-days: 7
+          overwrite: true
+
+  merge:
+    runs-on: ubuntu-latest
+    needs: generate
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: true
+          ref: ${{ inputs.base-ref }}
+          token: ${{ secrets.AUTO_COMMIT_PAT }}
+
+      - name: Create update branch
+        env:
+          PYTHON_VERSION: ${{ inputs.python-version }}
+        run: git switch -c "update-doc-${PYTHON_VERSION}"
+
+      - name: Download generated doc DBs
+        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
+        with:
+          pattern: "doc-db-${{ inputs.python-version }}-**"
+          path: crates/doc/generated/
+          merge-multiple: true
+
+      - name: Transform JSON
+        env:
+          PYTHON_VERSION: ${{ inputs.python-version }}
+        run: |
+          # Merge all artifacts
+          jq -s "add" --sort-keys crates/doc/generated/*.json > crates/doc/generated/merged.json
+
+          # Format merged json for the phf macro
+          jq -r 'to_entries[] | "    \(.key | @json) => \(.value | @json),"' crates/doc/generated/merged.json > crates/doc/generated/raw_entries.txt
+
+          OUTPUT_FILE='crates/doc/src/data.inc.rs'
+
+          echo -n '' > $OUTPUT_FILE
+
+          echo '// This file was auto-generated by `.github/workflows/update-doc-db.yml`.' >> $OUTPUT_FILE
+          echo "// CPython version: ${PYTHON_VERSION}" >> $OUTPUT_FILE
+          echo '// spell-checker: disable' >> $OUTPUT_FILE
+
+          echo '' >> $OUTPUT_FILE
+
+          echo "pub static DB: phf::Map<&'static str, &'static str> = phf::phf_map! {" >> $OUTPUT_FILE
+          cat crates/doc/generated/raw_entries.txt >> $OUTPUT_FILE
+          echo '};' >> $OUTPUT_FILE
+
+      - uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0
+        with:
+          name: doc-db-${{ inputs.python-version }}
+          path: "crates/doc/src/data.inc.rs"
+          if-no-files-found: error
+          retention-days: 7
+          overwrite: true
+
+      - name: Commit, push and create PR
+        env:
+          GH_TOKEN: ${{ secrets.AUTO_COMMIT_PAT }}
+          PYTHON_VERSION: ${{ inputs.python-version }}
+          BASE_REF: ${{ inputs.base-ref }}
+        run: |
+          git config user.name "github-actions[bot]"
+          git config user.email "github-actions[bot]@users.noreply.github.com"
+          if [ -n "$(git status --porcelain)" ]; then
+            git add crates/doc/src/data.inc.rs
+            git commit -m "Update doc DB for CPython ${PYTHON_VERSION}"
+            git push -u origin HEAD
+            gh pr create \
+              --base "${BASE_REF}" \
+              --title "Update doc DB for CPython ${PYTHON_VERSION}" \
+              --body "Auto-generated by update-doc-db workflow."
+          fi

.github/workflows/update-libs-status.yaml

@@ -0,0 +1,90 @@
+name: Updated libs status
+
+on:
+  push:
+    branches:
+      - main
+    paths:
+      - "Lib/**"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+  issues: write
+
+env:
+  PYTHON_VERSION: "v3.14.3"
+  ISSUE_ID: "6839"
+
+jobs:
+  update-issue:
+    runs-on: ubuntu-latest
+    if: ${{ github.repository == 'RustPython/RustPython' }}
+    steps:
+      - name: Clone RustPython
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          path: rustpython
+          persist-credentials: false
+          sparse-checkout: |-
+              Lib
+              scripts/update_lib
+
+      - name: Clone CPython ${{ env.PYTHON_VERSION }}
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          repository: python/cpython
+          path: cpython
+          ref: ${{ env.PYTHON_VERSION }}
+          persist-credentials: false
+          sparse-checkout: |
+            Lib
+
+      - name: Get current date
+        id: current_date
+        run: |
+          now=$(date -u +"%Y-%m-%d %H:%M:%S")
+          echo "date=$now" >> "$GITHUB_OUTPUT"
+
+      - name: Write body prefix
+        run: |
+          cat > body.txt <<EOF
+
+          <!-- 
+          THIS BODY IS AUTO-GENERATED. DO NOT EDIT MANUALLY!
+          -->
+
+          ## Summary
+
+          Check \`scripts/update_lib\` for tools. As a note, the current latest Python version is \`${{ env.PYTHON_VERSION }}\`.
+
+          Previous versions' issues as reference
+          - 3.13: #5529 
+
+          <!-- 
+          Quick guideline for Copilot:
+          # Clone \`github.com/python/cpython\` \`${{ env.PYTHON_VERSION }}\` tag under RustPython working dir with depth 1 option; never 3.14.0 or 3.14.1 or 3.14.2
+          # Pick a library or test to update. Probably user give one.
+          # Run \`python3 scripts/update_lib quick <name>\`
+          # A commit is automatically created. push the commit.
+          # Do not include keywords that automatically close an issue (e.g., Fix, Close, Resolve) next to an issue link or issue number.
+          -->
+
+          ## Details
+
+          ${{ steps.current_date.outputs.date }} (UTC)
+          \`\`\`shell
+          $ python3 scripts/update_lib todo --done
+          \`\`\`
+          EOF
+        
+      - name: Run todo
+        run: python3 rustpython/scripts/update_lib todo --cpython cpython --lib rustpython/Lib --done >> body.txt
+
+      - name: Update GH issue
+        run: gh issue edit ${{ env.ISSUE_ID }} --body-file ../body.txt
+        env:
+          GH_TOKEN: ${{ github.token }}
+        working-directory: rustpython
+        
+

.github/workflows/upgrade-pylib.md

@@ -0,0 +1,134 @@
+---
+description: |
+  Pick an out-of-sync Python library from the todo list and upgrade it
+  by running `scripts/update_lib quick`, then open a pull request.
+
+on:
+  workflow_dispatch:
+    inputs:
+      name:
+        description: "Module name to upgrade (leave empty to auto-pick)"
+        required: false
+        type: string
+
+timeout-minutes: 45
+
+permissions:
+  contents: read
+  issues: read
+  pull-requests: read
+
+network:
+  allowed:
+    - defaults
+    - rust
+    - python
+
+engine: copilot
+
+runtimes:
+  python:
+    version: "3.14"
+
+tools:
+  bash:
+    - ":*"
+  edit:
+  github:
+    toolsets: [repos, issues, pull_requests]
+    read-only: true
+
+safe-outputs:
+  create-pull-request:
+    title-prefix: "Update "
+    labels: [pylib-sync]
+    draft: false
+    expires: 30
+
+cache:
+  key: cpython-lib-${{ env.PYTHON_VERSION }}
+  path: cpython
+  restore-keys:
+    - cpython-lib-
+
+env:
+  PYTHON_VERSION: "v3.14.3"
+  ISSUE_ID: "6839"
+---
+
+# Upgrade Python Library
+
+You are an automated maintenance agent for RustPython, a Python 3 interpreter written in Rust. Your task is to upgrade one out-of-sync Python standard library module from CPython.
+
+## Step 1: Set up the environment
+
+The CPython source may already be cached. Check if the `cpython` directory exists and has the correct version:
+
+```bash
+if [ -d "cpython/Lib" ]; then
+    echo "CPython cache hit, skipping clone"
+else
+    git clone --depth 1 --branch "$PYTHON_VERSION" https://github.com/python/cpython.git cpython
+fi
+```
+
+## Step 2: Determine module name
+
+Run this script to determine the module name:
+
+```bash
+MODULE_NAME="${{ github.event.inputs.name }}"
+if [ -z "$MODULE_NAME" ]; then
+    echo "No module specified, running todo to find one..."
+    python3 scripts/update_lib todo
+    echo "Pick one module from the list above that is marked [ ], has no unmet deps, and has a small Δ number."
+    echo "Do NOT pick: opcode, datetime, random, hashlib, tokenize, pdb, _pyrepl, concurrent, asyncio, multiprocessing, ctypes, idlelib, tkinter, shutil, tarfile, email, unittest"
+else
+    echo "Module specified by user: $MODULE_NAME"
+fi
+```
+
+If the script printed "Module specified by user: ...", use that exact name. If it printed the todo list, pick one suitable module from it.
+
+## Step 3: Run the upgrade
+
+Run the quick upgrade command. This will copy the library from CPython, migrate test files preserving RustPython markers, auto-mark test failures, and create a git commit:
+
+```bash
+python3 scripts/update_lib quick <module_name>
+```
+
+This takes a while because it builds RustPython (`cargo build --release`) and runs tests to determine which ones pass or fail.
+
+If the command fails, report the error and stop. Do not try to fix Rust code or modify test files manually.
+
+## Step 4: Verify the result
+
+After the script succeeds, check what changed:
+
+```bash
+git log -1 --stat
+git diff HEAD~1 --stat
+```
+
+Make sure the commit was created with the correct message format: `Update <name> from <version>`.
+
+## Step 5: Create the pull request
+
+Create a pull request. Reference issue #${{ env.ISSUE_ID }} in the body but do **NOT** use keywords that auto-close issues (Fix, Close, Resolve).
+
+Use this format for the PR body:
+
+```
+## Summary
+
+Upgrade `<module_name>` from CPython $PYTHON_VERSION.
+
+Part of #$ISSUE_ID
+
+## Changes
+
+- Updated `Lib/<module_name>` from CPython
+- Migrated test files preserving RustPython markers
+- Auto-marked test failures with `@expectedFailure`
+```

.gitignore

@@ -21,3 +21,11 @@ flamescope.json
 
 extra_tests/snippets/resources
 extra_tests/not_impl.py
+
+Lib/_sysconfig_vars*.json
+Lib/site-packages/*
+!Lib/site-packages/README.txt
+Lib/test/data/*
+!Lib/test/data/README
+cpython/
+

.vscode/launch.json

@@ -16,18 +16,6 @@
             },
             "cwd": "${workspaceFolder}"
         },
-        {
-            "type": "lldb",
-            "request": "launch",
-            "name": "Debug executable 'rustpython' without SSL",
-            "preLaunchTask": "Build RustPython Debug without SSL",
-            "program": "target/debug/rustpython",
-            "args": [],
-            "env": {
-                "RUST_BACKTRACE": "1"
-            },
-            "cwd": "${workspaceFolder}"
-        },
         {
             "type": "lldb",
             "request": "launch",

.vscode/tasks.json

@@ -1,28 +1,12 @@
 {
     "version": "2.0.0",
     "tasks": [
-        {
-            "label": "Build RustPython Debug without SSL",
-            "type": "shell",
-            "command": "cargo",
-            "args": [
-                "build",
-            ],
-            "problemMatcher": [
-                "$rustc",
-            ],
-            "group": {
-                "kind": "build",
-                "isDefault": true,
-            },
-        },
         {
             "label": "Build RustPython Debug",
             "type": "shell",
             "command": "cargo",
             "args": [
                 "build",
-                "--features=ssl"
             ],
             "problemMatcher": [
                 "$rustc",

AGENTS.md

@@ -4,7 +4,7 @@ This document provides guidelines for working with GitHub Copilot when contribut
 
 ## Project Overview
 
-RustPython is a Python 3 interpreter written in Rust, implementing Python 3.13.0+ compatibility. The project aims to provide:
+RustPython is a Python 3 interpreter written in Rust, implementing Python 3.14.0+ compatibility. The project aims to provide:
 
 - A complete Python-3 environment entirely in Rust (not CPython bindings)
 - A clean implementation without compatibility hacks
@@ -30,6 +30,14 @@ RustPython is a Python 3 interpreter written in Rust, implementing Python 3.13.0
 - `jit/` - Experimental JIT compiler implementation
 - `pylib/` - Python standard library packaging (do not modify this directory directly - its contents are generated automatically)
 
+## AI Agent Rules
+
+**CRITICAL: Git Operations**
+- NEVER create pull requests directly without explicit user permission
+- NEVER push commits to remote without explicit user permission
+- Always ask the user before performing any git operations that affect the remote repository
+- Commits can be created locally when requested, but pushing and PR creation require explicit approval
+
 ## Important Development Notes
 
 ### Running Python Code
@@ -44,7 +52,7 @@ cargo run -- script.py
 cargo run
 
 # With specific features
-cargo run --features ssl
+cargo run --features jit
 
 # Release mode (recommended for better performance)
 cargo run --release -- script.py
@@ -73,17 +81,28 @@ The `Lib/` directory contains Python standard library files copied from the CPyt
   - `unittest.skip("TODO: RustPython <reason>")`
   - `unittest.expectedFailure` with `# TODO: RUSTPYTHON <reason>` comment
 
+### Clean Build
+
+When you modify bytecode instructions, a full clean is required:
+
+```bash
+rm -r target/debug/build/rustpython-* && find . | grep -E "\.pyc$" | xargs rm -r
+```
+
 ### Testing
 
 ```bash
 # Run Rust unit tests
-cargo test --workspace --exclude rustpython_wasm
+cargo test --workspace --exclude rustpython_wasm --exclude rustpython-venvlauncher
 
-# Run Python snippets tests
+# Run Python snippets tests (debug mode recommended for faster compilation)
+cargo run -- extra_tests/snippets/builtin_bytes.py
+
+# Run all Python snippets tests with pytest
 cd extra_tests
 pytest -v
 
-# Run the Python test module
+# Run the Python test module (release mode recommended for better performance)
 cargo run --release -- -m test ${TEST_MODULE}
 cargo run --release -- -m test test_unicode # to test test_unicode.py
 
@@ -91,9 +110,11 @@ cargo run --release -- -m test test_unicode # to test test_unicode.py
 cargo run --release -- -m test test_unicode -k test_unicode_escape
 ```
 
+**Note**: For `extra_tests/snippets` tests, use debug mode (`cargo run`) as compilation is faster. For `unittest` (`-m test`), use release mode (`cargo run --release`) for better runtime performance.
+
 ### Determining What to Implement
 
-Run `./whats_left.py` to get a list of unimplemented methods, which is helpful when looking for contribution opportunities.
+Run `./scripts/whats_left.py` to get a list of unimplemented methods, which is helpful when looking for contribution opportunities.
 
 ## Coding Guidelines
 
@@ -104,6 +125,36 @@ Run `./whats_left.py` to get a list of unimplemented methods, which is helpful w
 - Follow Rust best practices for error handling and memory management
 - Use the macro system (`pyclass`, `pymodule`, `pyfunction`, etc.) when implementing Python functionality in Rust
 
+#### Comments
+
+- Do not delete or rewrite existing comments unless they are factually wrong or directly contradict the new code.
+- Do not add decorative section separators (e.g. `// -----------`, `// ===`, `/* *** */`). Use `///` doc-comments or short `//` comments only when they add value.
+
+#### Avoid Duplicate Code in Branches
+
+When branches differ only in a value but share common logic, extract the differing value first, then call the common logic once.
+
+**Bad:**
+```rust
+let result = if condition {
+    let msg = format!("message A: {x}");
+    some_function(msg, shared_arg)
+} else {
+    let msg = format!("message B");
+    some_function(msg, shared_arg)
+};
+```
+
+**Good:**
+```rust
+let msg = if condition {
+    format!("message A: {x}")
+} else {
+    format!("message B")
+};
+let result = some_function(msg, shared_arg);
+```
+
 ### Python Code
 
 - **IMPORTANT**: In most cases, Python code should not be edited. Bug fixes should be made through Rust code modifications only
@@ -176,12 +227,15 @@ cargo build --target wasm32-wasip1 --no-default-features --features freeze-stdli
 cargo run --features jit
 ```
 
-### SSL Support
+### Linux Build and Debug on macOS
 
-```bash
-# Enable SSL support
-cargo run --features ssl
-```
+See the "Testing on Linux from macOS" section in [DEVELOPMENT.md](DEVELOPMENT.md#testing-on-linux-from-macos).
+
+### Building venvlauncher (Windows)
+
+See DEVELOPMENT.md "CPython Version Upgrade Checklist" section.
+
+**IMPORTANT**: All 4 venvlauncher binaries use the same source code. Do NOT add multiple `[[bin]]` entries to Cargo.toml. Build once and copy with different names.
 
 ## Test Code Modification Rules
 
@@ -206,7 +260,7 @@ cargo run --features ssl
 
 ## Documentation
 
-- Check the [architecture document](architecture/architecture.md) for a high-level overview
-- Read the [development guide](DEVELOPMENT.md) for detailed setup instructions
+- Check the [architecture document](/architecture/architecture.md) for a high-level overview
+- Read the [development guide](/DEVELOPMENT.md) for detailed setup instructions
 - Generate documentation with `cargo doc --no-deps --all`
-- Online documentation is available at [docs.rs/rustpython](https://docs.rs/rustpython/)
+- Online documentation is available at [docs.rs/rustpython](https://docs.rs/rustpython/)

Cargo.toml

@@ -10,18 +10,21 @@ repository.workspace = true
 license.workspace = true
 
 [features]
-default = ["threading", "stdlib", "stdio", "importlib"]
+default = ["threading", "stdlib", "stdio", "importlib", "ssl-rustls", "host_env"]
+host_env = ["rustpython-vm/host_env", "rustpython-stdlib?/host_env"]
 importlib = ["rustpython-vm/importlib"]
 encodings = ["rustpython-vm/encodings"]
 stdio = ["rustpython-vm/stdio"]
 stdlib = ["rustpython-stdlib", "rustpython-pylib", "encodings"]
-flame-it = ["rustpython-vm/flame-it", "flame", "flamescope"]
+flame-it = ["rustpython-vm/flame-it", "rustpython-stdlib/flame-it", "flame", "flamescope"]
 freeze-stdlib = ["stdlib", "rustpython-vm/freeze-stdlib", "rustpython-pylib?/freeze-stdlib"]
 jit = ["rustpython-vm/jit"]
 threading = ["rustpython-vm/threading", "rustpython-stdlib/threading"]
 sqlite = ["rustpython-stdlib/sqlite"]
-ssl = ["rustpython-stdlib/ssl"]
-ssl-vendor = ["ssl", "rustpython-stdlib/ssl-vendor"]
+ssl = []
+ssl-rustls = ["ssl", "rustpython-stdlib/ssl-rustls"]
+ssl-openssl = ["ssl", "rustpython-stdlib/ssl-openssl"]
+ssl-vendor = ["ssl-openssl", "rustpython-stdlib/ssl-vendor"]
 tkinter = ["rustpython-stdlib/tkinter"]
 
 [build-dependencies]
@@ -31,8 +34,7 @@ winresource = "0.1"
 rustpython-compiler = { workspace = true }
 rustpython-pylib = { workspace = true, optional = true }
 rustpython-stdlib = { workspace = true, optional = true, features = ["compiler"] }
-rustpython-vm = { workspace = true, features = ["compiler"] }
-ruff_python_parser = { workspace = true }
+rustpython-vm = { workspace = true, features = ["compiler", "gc"] }
 
 cfg-if = { workspace = true }
 log = { workspace = true }
@@ -51,7 +53,9 @@ rustyline = { workspace = true }
 
 [dev-dependencies]
 criterion = { workspace = true }
-pyo3 = { version = "0.24", features = ["auto-initialize"] }
+pyo3 = { version = "0.28.2", features = ["auto-initialize"] }
+rustpython-stdlib = { workspace = true }
+ruff_python_parser = { workspace = true }
 
 [[bench]]
 name = "execution"
@@ -73,6 +77,21 @@ opt-level = 3
 # https://github.com/rust-lang/rust/issues/92869
 # lto = "thin"
 
+# Some crates don't change as much but benefit more from
+# more expensive optimization passes, so we selectively
+# decrease codegen-units in some cases.
+[profile.release.package.rustpython-doc]
+codegen-units = 1
+
+[profile.release.package.rustpython-literal]
+codegen-units = 1
+
+[profile.release.package.rustpython-common]
+codegen-units = 1
+
+[profile.release.package.rustpython-wtf8]
+codegen-units = 1
+
 [profile.bench]
 lto = "thin"
 codegen-units = 1
@@ -82,19 +101,10 @@ opt-level = 3
 lto = "thin"
 
 [patch.crates-io]
+parking_lot_core = { git = "https://github.com/youknowone/parking_lot", branch = "rustpython" }
 # REDOX START, Uncomment when you want to compile/check with redoxer
 # REDOX END
 
-# Used only on Windows to build the vcpkg dependencies
-[package.metadata.vcpkg]
-git = "https://github.com/microsoft/vcpkg"
-# The revision of the vcpkg repository to use
-# https://github.com/microsoft/vcpkg/tags
-rev = "2024.02.14"
-
-[package.metadata.vcpkg.target]
-x86_64-pc-windows-msvc = { triplet = "x64-windows-static-md", dev-dependencies = ["openssl" ] }
-
 [package.metadata.packager]
 product-name = "RustPython"
 identifier = "com.rustpython.rustpython"
@@ -117,103 +127,106 @@ template = "installer-config/installer.wxs"
 [workspace]
 resolver = "2"
 members = [
-    "compiler", "compiler/core", "compiler/codegen", "compiler/literal", "compiler/source",
-    ".", "common", "derive", "jit", "vm", "vm/sre_engine", "pylib", "stdlib", "derive-impl", "wtf8",
-    "wasm/lib",
+    ".",
+    "crates/*",
 ]
+exclude = ["pymath"]
 
 [workspace.package]
-version = "0.4.0"
+version = "0.5.0"
 authors = ["RustPython Team"]
 edition = "2024"
-rust-version = "1.85.0"
+rust-version = "1.93.0"
 repository = "https://github.com/RustPython/RustPython"
 license = "MIT"
 
 [workspace.dependencies]
-rustpython-compiler-source = { path = "compiler/source" }
-rustpython-compiler-core = { path = "compiler/core", version = "0.4.0" }
-rustpython-compiler = { path = "compiler", version = "0.4.0" }
-rustpython-codegen = { path = "compiler/codegen", version = "0.4.0" }
-rustpython-common = { path = "common", version = "0.4.0" }
-rustpython-derive = { path = "derive", version = "0.4.0" }
-rustpython-derive-impl = { path = "derive-impl", version = "0.4.0" }
-rustpython-jit = { path = "jit", version = "0.4.0" }
-rustpython-literal = { path = "compiler/literal", version = "0.4.0" }
-rustpython-vm = { path = "vm", default-features = false, version = "0.4.0" }
-rustpython-pylib = { path = "pylib", version = "0.4.0" }
-rustpython-stdlib = { path = "stdlib", default-features = false, version = "0.4.0" }
-rustpython-sre_engine = { path = "vm/sre_engine", version = "0.4.0" }
-rustpython-wtf8 = { path = "wtf8", version = "0.4.0" }
-rustpython-doc = { git = "https://github.com/RustPython/__doc__", tag = "0.3.0", version = "0.3.0" }
+rustpython-compiler-core = { path = "crates/compiler-core", version = "0.5.0" }
+rustpython-compiler = { path = "crates/compiler", version = "0.5.0" }
+rustpython-codegen = { path = "crates/codegen", version = "0.5.0" }
+rustpython-common = { path = "crates/common", version = "0.5.0" }
+rustpython-derive = { path = "crates/derive", version = "0.5.0" }
+rustpython-derive-impl = { path = "crates/derive-impl", version = "0.5.0" }
+rustpython-jit = { path = "crates/jit", version = "0.5.0" }
+rustpython-literal = { path = "crates/literal", version = "0.5.0" }
+rustpython-vm = { path = "crates/vm", default-features = false, version = "0.5.0" }
+rustpython-pylib = { path = "crates/pylib", version = "0.5.0" }
+rustpython-stdlib = { path = "crates/stdlib", default-features = false, version = "0.5.0" }
+rustpython-sre_engine = { path = "crates/sre_engine", version = "0.5.0" }
+rustpython-wtf8 = { path = "crates/wtf8", version = "0.5.0" }
+rustpython-doc = { path = "crates/doc", version = "0.5.0" }
 
-ruff_python_parser = { git = "https://github.com/astral-sh/ruff.git", tag = "0.11.0" }
-ruff_python_ast = { git = "https://github.com/astral-sh/ruff.git", tag = "0.11.0" }
-ruff_text_size = { git = "https://github.com/astral-sh/ruff.git", tag = "0.11.0" }
-ruff_source_file = { git = "https://github.com/astral-sh/ruff.git", tag = "0.11.0" }
+# Ruff tag 0.15.6 is based on commit e4c7f357777a2fdd34dbe6a98b1b7d3e7488f675
+# at the time of this capture. We use the commit hash to ensure reproducible builds.
+ruff_python_parser = { git = "https://github.com/astral-sh/ruff.git", rev = "e4c7f357777a2fdd34dbe6a98b1b7d3e7488f675" }
+ruff_python_ast = { git = "https://github.com/astral-sh/ruff.git", rev = "e4c7f357777a2fdd34dbe6a98b1b7d3e7488f675" }
+ruff_text_size = { git = "https://github.com/astral-sh/ruff.git", rev = "e4c7f357777a2fdd34dbe6a98b1b7d3e7488f675" }
+ruff_source_file = { git = "https://github.com/astral-sh/ruff.git", rev = "e4c7f357777a2fdd34dbe6a98b1b7d3e7488f675" }
 
-ahash = "0.8.11"
+phf = { version = "0.13.1", default-features = false, features = ["macros"]}
+ahash = "0.8.12"
 ascii = "1.1"
-bitflags = "2.4.2"
+bitflags = "2.11.0"
+bitflagset = "0.0.3"
 bstr = "1"
 cfg-if = "1.0"
-chrono = "0.4.39"
+chrono = { version = "0.4.44", default-features = false, features = ["clock", "oldtime", "std"] }
 constant_time_eq = "0.4"
-criterion = { version = "0.5", features = ["html_reports"] }
+criterion = { version = "0.8", features = ["html_reports"] }
 crossbeam-utils = "0.8.21"
 flame = "0.2.2"
 getrandom = { version = "0.3", features = ["std"] }
 glob = "0.3"
 hex = "0.4.3"
-indexmap = { version = "2.2.6", features = ["std"] }
-insta = "1.42"
+indexmap = { version = "2.13.0", features = ["std"] }
+insta = "1.46"
 itertools = "0.14.0"
 is-macro = "0.3.7"
-junction = "1.2.0"
-libc = "0.2.169"
-libffi = "4.1"
-log = "0.4.27"
-nix = { version = "0.29", features = ["fs", "user", "process", "term", "time", "signal", "ioctl", "socket", "sched", "zerocopy", "dir", "hostname", "net", "poll"] }
-malachite-bigint = "0.6"
-malachite-q = "0.6"
-malachite-base = "0.6"
-memchr = "2.7.4"
+junction = "1.4.2"
+libc = "0.2.183"
+libffi = "5"
+log = "0.4.29"
+nix = { version = "0.30", features = ["fs", "user", "process", "term", "time", "signal", "ioctl", "socket", "sched", "zerocopy", "dir", "hostname", "net", "poll"] }
+malachite-bigint = "0.9.1"
+malachite-q = "0.9.1"
+malachite-base = "0.9.1"
+memchr = "2.8.0"
 num-complex = "0.4.6"
 num-integer = "0.1.46"
 num-traits = "0.2"
 num_enum = { version = "0.7", default-features = false }
 optional = "0.5"
-once_cell = "1.20.3"
 parking_lot = "0.12.3"
 paste = "1.0.15"
-proc-macro2 = "1.0.93"
-pymath = "0.0.2"
-quote = "1.0.38"
+proc-macro2 = "1.0.105"
+pymath = { version = "0.2.0", features = ["mul_add", "malachite-bigint", "complex"] }
+quote = "1.0.45"
 radium = "1.1.1"
 rand = "0.9"
 rand_core = { version = "0.9", features = ["os_rng"] }
-rustix = { version = "1.0", features = ["event"] }
-rustyline = "15.0.0"
-serde = { version = "1.0.133", default-features = false }
-schannel = "0.1.27"
+rustix = { version = "1.1", features = ["event"] }
+rustyline = "17.0.1"
+serde = { package = "serde_core", version = "1.0.225", default-features = false, features = ["alloc"] }
+schannel = "0.1.28"
+scoped-tls = "1"
+scopeguard = "1"
 static_assertions = "1.1"
 strum = "0.27"
-strum_macros = "0.27"
+strum_macros = "0.28"
 syn = "2"
 thiserror = "2.0"
-thread_local = "1.1.8"
-unicode-casing = "0.1.0"
+unicode-casing = "0.1.1"
 unic-char-property = "0.9.0"
 unic-normal = "0.9.0"
 unic-ucd-age = "0.9.0"
 unic-ucd-bidi = "0.9.0"
 unic-ucd-category = "0.9.0"
 unic-ucd-ident = "0.9.0"
-unicode_names2 = "1.3.0"
-unicode-bidi-mirroring = "0.2"
-widestring = "1.1.0"
-windows-sys = "0.59.0"
-wasm-bindgen = "0.2.100"
+unicode_names2 = "2.0.0"
+unicode-bidi-mirroring = "0.4"
+widestring = "1.2.0"
+windows-sys = "0.61.2"
+wasm-bindgen = "0.2.106"
 
 # Lints
 
@@ -223,6 +236,9 @@ unsafe_op_in_unsafe_fn = "deny"
 elided_lifetimes_in_paths = "warn"
 
 [workspace.lints.clippy]
+alloc_instead_of_core = "warn"
+std_instead_of_alloc = "warn"
+std_instead_of_core = "warn"
 perf = "warn"
 style = "warn"
 complexity = "warn"

DEVELOPMENT.md

@@ -19,13 +19,13 @@ The contents of the Development Guide include:
 
 RustPython requires the following:
 
-- Rust latest stable version (e.g 1.69.0 as of Apr 20 2023)
+- Rust latest stable version (e.g 1.92.0 as of Jan 7 2026)
     - To check Rust version: `rustc --version` 
     - If you have `rustup` on your system, enter to update to the latest
       stable version: `rustup update stable`
     - If you do not have Rust installed, use [rustup](https://rustup.rs/) to
       do so.
-- CPython version 3.13 or higher
+- CPython version 3.14 or higher
     - CPython can be installed by your operating system's package manager,
       from the [Python website](https://www.python.org/downloads/), or
       using a third-party distribution, such as 
@@ -65,7 +65,7 @@ $ pytest -v
 Rust unit tests can be run with `cargo`:
 
 ```shell
-$ cargo test --workspace --exclude rustpython_wasm
+$ cargo test --workspace --exclude rustpython_wasm --exclude rustpython-venvlauncher
 ```
 
 Python unit tests can be run by compiling RustPython and running the test module:
@@ -95,6 +95,41 @@ To run only `test_cmath` (located at `Lib/test/test_cmath`) verbosely:
 $ cargo run --release -- -m test test_cmath -v
 ```
 
+### Testing on Linux from macOS
+
+You can test RustPython on Linux from macOS using Apple's `container` CLI.
+
+**Setup (one-time):**
+
+```shell
+# Install container CLI
+$ brew install container
+
+# Disable Rosetta requirement for arm64-only builds
+$ defaults write com.apple.container.defaults build.rosetta -bool false
+
+# Build the development image
+$ container build --arch arm64 -t rustpython-dev -f .devcontainer/Dockerfile .
+```
+
+**Running tests:**
+
+```shell
+# Start a persistent container in background (8GB memory, 4 CPUs for compilation)
+$ container run -d --name rustpython-test -m 8G -c 4 \
+    --mount type=bind,source=$(pwd),target=/workspace \
+    -w /workspace rustpython-dev sleep infinity
+
+# Run tests inside the container
+$ container exec rustpython-test sh -c "cargo run --release -- -m test test_ensurepip"
+
+# Run any command
+$ container exec rustpython-test sh -c "cargo test --workspace"
+
+# Stop and remove the container when done
+$ container rm -f rustpython-test
+```
+
 ## Profiling
 
 To profile RustPython, build it in `release` mode with the `flame-it` feature.
@@ -118,19 +153,19 @@ exists a raw html viewer which is currently broken, and we welcome a PR to fix i
 Understanding a new codebase takes time. Here's a brief view of the
 repository's structure:
 
-- `compiler/src`: python compilation to bytecode
-  - `core/src`: python bytecode representation in rust structures
-  - `parser/src`: python lexing, parsing and ast
-- `derive/src`: Rust language extensions and macros specific to rustpython
+- `crates/compiler/src`: python compilation to bytecode
+  - `crates/compiler-core/src`: python bytecode representation in rust structures
+- `crates/derive/src` and `crates/derive-impl/src`: Rust language extensions and macros specific to rustpython
 - `Lib`: Carefully selected / copied files from CPython sourcecode. This is
    the python side of the standard library.
   - `test`: CPython test suite
-- `vm/src`: python virtual machine
+- `crates/vm/src`: python virtual machine
   - `builtins`: Builtin functions and types
   - `stdlib`: Standard library parts implemented in rust.
 - `src`: using the other subcrates to bring rustpython to life.
-- `wasm`: Binary crate and resources for WebAssembly build
-- `extra_tests`: extra integration test snippets as a supplement to `Lib/test`
+- `crates/wasm`: Binary crate and resources for WebAssembly build
+- `extra_tests`: extra integration test snippets as a supplement to `Lib/test`.
+  Add new RustPython-only regression tests here; do not place new tests under `Lib/test`.
 
 ## Understanding Internals
 
@@ -140,9 +175,9 @@ implementation is found in the `src` directory (specifically, `src/lib.rs`).
 
 The top-level `rustpython` binary depends on several lower-level crates including:
 
-- `rustpython-parser` (implementation in `compiler/parser/src`)
-- `rustpython-compiler` (implementation in `compiler/src`)
-- `rustpython-vm` (implementation in `vm/src`)
+- `ruff_python_parser` and `ruff_python_ast` (external dependencies from the Ruff project)
+- `rustpython-compiler` (implementation in `crates/compiler/src`)
+- `rustpython-vm` (implementation in `crates/vm/src`)
 
 Together, these crates provide the functions of a programming language and
 enable a line of code to go through a series of steps:
@@ -153,31 +188,26 @@ enable a line of code to go through a series of steps:
 - compile the AST into bytecode
 - execute the bytecode in the virtual machine (VM).
 
-### rustpython-parser
+### Parser and AST
 
-This crate contains the lexer and parser to convert a line of code to
-an Abstract Syntax Tree (AST):
+RustPython uses the Ruff project's parser and AST implementation:
 
-- Lexer: `compiler/parser/src/lexer.rs` converts Python source code into tokens
-- Parser: `compiler/parser/src/parser.rs` takes the tokens generated by the lexer and parses
-  the tokens into an AST (Abstract Syntax Tree) where the nodes of the syntax
-  tree are Rust structs and enums.
-  - The Parser relies on `LALRPOP`, a Rust parser generator framework. The
-    LALRPOP definition of Python's grammar is in `compiler/parser/src/python.lalrpop`.
-  - More information on parsers and a tutorial can be found in the 
-    [LALRPOP book](https://lalrpop.github.io/lalrpop/).
-- AST: `compiler/ast/` implements in Rust the Python types and expressions
-  represented by the AST nodes.
+- Parser: `ruff_python_parser` is used to convert Python source code into tokens
+  and parse them into an Abstract Syntax Tree (AST)
+- AST: `ruff_python_ast` provides the Rust types and expressions represented by
+  the AST nodes
+- These are external dependencies maintained by the Ruff project
+- For more information, visit the [Ruff GitHub repository](https://github.com/astral-sh/ruff)
 
 ### rustpython-compiler
 
 The `rustpython-compiler` crate's purpose is to transform the AST (Abstract Syntax
 Tree) to bytecode. The implementation of the compiler is found in the
-`compiler/src` directory. The compiler implements Python's symbol table,
+`crates/compiler/src` directory. The compiler implements Python's symbol table,
 ast->bytecode compiler, and bytecode optimizer in Rust.
 
-Implementation of bytecode structure in Rust is found in the `compiler/core/src`
-directory. `compiler/core/src/bytecode.rs` contains the representation of
+Implementation of bytecode structure in Rust is found in the `crates/compiler-core/src`
+directory. `crates/compiler-core/src/bytecode.rs` contains the representation of
 instructions and operations in Rust. Further information about Python's
 bytecode instructions can be found in the
 [Python documentation](https://docs.python.org/3/library/dis.html#bytecodes).
@@ -185,14 +215,14 @@ bytecode instructions can be found in the
 ### rustpython-vm
 
 The `rustpython-vm` crate has the important job of running the virtual machine that
-executes Python's instructions. The `vm/src` directory contains code to
+executes Python's instructions. The `crates/vm/src` directory contains code to
 implement the read and evaluation loop that fetches and dispatches
 instructions. This directory also contains the implementation of the
-Python Standard Library modules in Rust (`vm/src/stdlib`). In Python
-everything can be represented as an object. The `vm/src/builtins` directory holds
+Python Standard Library modules in Rust (`crates/vm/src/stdlib`). In Python
+everything can be represented as an object. The `crates/vm/src/builtins` directory holds
 the Rust code used to represent different Python objects and their methods. The
 core implementation of what a Python object is can be found in
-`vm/src/object/core.rs`.
+`crates/vm/src/object/core.rs`.
 
 ### Code generation
 

Lib/_android_support.py

@@ -29,15 +29,19 @@ def init_streams(android_log_write, stdout_prio, stderr_prio):
 
     global logcat
     logcat = Logcat(android_log_write)
-
-    sys.stdout = TextLogStream(
-        stdout_prio, "python.stdout", sys.stdout.fileno())
-    sys.stderr = TextLogStream(
-        stderr_prio, "python.stderr", sys.stderr.fileno())
+    sys.stdout = TextLogStream(stdout_prio, "python.stdout", sys.stdout)
+    sys.stderr = TextLogStream(stderr_prio, "python.stderr", sys.stderr)
 
 
 class TextLogStream(io.TextIOWrapper):
-    def __init__(self, prio, tag, fileno=None, **kwargs):
+    def __init__(self, prio, tag, original=None, **kwargs):
+        # Respect the -u option.
+        if original:
+            kwargs.setdefault("write_through", original.write_through)
+            fileno = original.fileno()
+        else:
+            fileno = None
+
         # The default is surrogateescape for stdout and backslashreplace for
         # stderr, but in the context of an Android log, readability is more
         # important than reversibility.

Lib/_collections_abc.py

@@ -485,9 +485,10 @@ class _CallableGenericAlias(GenericAlias):
     def __repr__(self):
         if len(self.__args__) == 2 and _is_param_expr(self.__args__[0]):
             return super().__repr__()
+        from annotationlib import type_repr
         return (f'collections.abc.Callable'
-                f'[[{", ".join([_type_repr(a) for a in self.__args__[:-1]])}], '
-                f'{_type_repr(self.__args__[-1])}]')
+                f'[[{", ".join([type_repr(a) for a in self.__args__[:-1]])}], '
+                f'{type_repr(self.__args__[-1])}]')
 
     def __reduce__(self):
         args = self.__args__
@@ -512,10 +513,6 @@ class _CallableGenericAlias(GenericAlias):
             new_args = (t_args, t_result)
         return _CallableGenericAlias(Callable, tuple(new_args))
 
-    # TODO: RUSTPYTHON patch for common call
-    def __or__(self, other):
-        super().__or__(other)
-
 def _is_param_expr(obj):
     """Checks if obj matches either a list of types, ``...``, ``ParamSpec`` or
     ``_ConcatenateGenericAlias`` from typing.py
@@ -528,23 +525,6 @@ def _is_param_expr(obj):
     names = ('ParamSpec', '_ConcatenateGenericAlias')
     return obj.__module__ == 'typing' and any(obj.__name__ == name for name in names)
 
-def _type_repr(obj):
-    """Return the repr() of an object, special-casing types (internal helper).
-
-    Copied from :mod:`typing` since collections.abc
-    shouldn't depend on that module.
-    (Keep this roughly in sync with the typing version.)
-    """
-    if isinstance(obj, type):
-        if obj.__module__ == 'builtins':
-            return obj.__qualname__
-        return f'{obj.__module__}.{obj.__qualname__}'
-    if obj is Ellipsis:
-        return '...'
-    if isinstance(obj, FunctionType):
-        return obj.__name__
-    return repr(obj)
-
 
 class Callable(metaclass=ABCMeta):
 
@@ -1077,6 +1057,7 @@ class Sequence(Reversible, Collection):
 
 Sequence.register(tuple)
 Sequence.register(str)
+Sequence.register(bytes)
 Sequence.register(range)
 Sequence.register(memoryview)
 
@@ -1087,7 +1068,7 @@ class _DeprecateByteStringMeta(ABCMeta):
 
             warnings._deprecated(
                 "collections.abc.ByteString",
-                remove=(3, 14),
+                remove=(3, 17),
             )
         return super().__new__(cls, name, bases, namespace, **kwargs)
 
@@ -1096,14 +1077,18 @@ class _DeprecateByteStringMeta(ABCMeta):
 
         warnings._deprecated(
             "collections.abc.ByteString",
-            remove=(3, 14),
+            remove=(3, 17),
         )
         return super().__instancecheck__(instance)
 
 class ByteString(Sequence, metaclass=_DeprecateByteStringMeta):
-    """This unifies bytes and bytearray.
+    """Deprecated ABC serving as a common supertype of ``bytes`` and ``bytearray``.
 
-    XXX Should add all their methods.
+    This ABC is scheduled for removal in Python 3.17.
+    Use ``isinstance(obj, collections.abc.Buffer)`` to test if ``obj``
+    implements the buffer protocol at runtime. For use in type annotations,
+    either use ``Buffer`` or a union that explicitly specifies the types your
+    code supports (e.g., ``bytes | bytearray | memoryview``).
     """
 
     __slots__ = ()
@@ -1179,4 +1164,4 @@ class MutableSequence(Sequence):
 
 
 MutableSequence.register(list)
-MutableSequence.register(bytearray)  # Multiply inheriting, see ByteString
+MutableSequence.register(bytearray)

Lib/_colorize.py

@@ -1,13 +1,16 @@
-from __future__ import annotations
-import io
 import os
 import sys
 
+from collections.abc import Callable, Iterator, Mapping
+from dataclasses import dataclass, field, Field
+
 COLORIZE = True
 
+
 # types
 if False:
-    from typing import IO
+    from typing import IO, Self, ClassVar
+    _theme: Theme
 
 
 class ANSIColors:
@@ -17,11 +20,13 @@ class ANSIColors:
     BLUE = "\x1b[34m"
     CYAN = "\x1b[36m"
     GREEN = "\x1b[32m"
+    GREY = "\x1b[90m"
     MAGENTA = "\x1b[35m"
     RED = "\x1b[31m"
     WHITE = "\x1b[37m"  # more like LIGHT GRAY
     YELLOW = "\x1b[33m"
 
+    BOLD = "\x1b[1m"
     BOLD_BLACK = "\x1b[1;30m"  # DARK GRAY
     BOLD_BLUE = "\x1b[1;34m"
     BOLD_CYAN = "\x1b[1;36m"
@@ -60,13 +65,196 @@ class ANSIColors:
     INTENSE_BACKGROUND_YELLOW = "\x1b[103m"
 
 
+ColorCodes = set()
 NoColors = ANSIColors()
 
-for attr in dir(NoColors):
+for attr, code in ANSIColors.__dict__.items():
     if not attr.startswith("__"):
+        ColorCodes.add(code)
         setattr(NoColors, attr, "")
 
 
+#
+# Experimental theming support (see gh-133346)
+#
+
+# - Create a theme by copying an existing `Theme` with one or more sections
+#   replaced, using `default_theme.copy_with()`;
+# - create a theme section by copying an existing `ThemeSection` with one or
+#   more colors replaced, using for example `default_theme.syntax.copy_with()`;
+# - create a theme from scratch by instantiating a `Theme` data class with
+#   the required sections (which are also dataclass instances).
+#
+# Then call `_colorize.set_theme(your_theme)` to set it.
+#
+# Put your theme configuration in $PYTHONSTARTUP for the interactive shell,
+# or sitecustomize.py in your virtual environment or Python installation for
+# other uses.  Your applications can call `_colorize.set_theme()` too.
+#
+# Note that thanks to the dataclasses providing default values for all fields,
+# creating a new theme or theme section from scratch is possible without
+# specifying all keys.
+#
+# For example, here's a theme that makes punctuation and operators less prominent:
+#
+#   try:
+#       from _colorize import set_theme, default_theme, Syntax, ANSIColors
+#   except ImportError:
+#       pass
+#   else:
+#       theme_with_dim_operators = default_theme.copy_with(
+#           syntax=Syntax(op=ANSIColors.INTENSE_BLACK),
+#       )
+#       set_theme(theme_with_dim_operators)
+#       del set_theme, default_theme, Syntax, ANSIColors, theme_with_dim_operators
+#
+# Guarding the import ensures that your .pythonstartup file will still work in
+# Python 3.13 and older. Deleting the variables ensures they don't remain in your
+# interactive shell's global scope.
+
+class ThemeSection(Mapping[str, str]):
+    """A mixin/base class for theme sections.
+
+    It enables dictionary access to a section, as well as implements convenience
+    methods.
+    """
+
+    # The two types below are just that: types to inform the type checker that the
+    # mixin will work in context of those fields existing
+    __dataclass_fields__: ClassVar[dict[str, Field[str]]]
+    _name_to_value: Callable[[str], str]
+
+    def __post_init__(self) -> None:
+        name_to_value = {}
+        for color_name in self.__dataclass_fields__:
+            name_to_value[color_name] = getattr(self, color_name)
+        super().__setattr__('_name_to_value', name_to_value.__getitem__)
+
+    def copy_with(self, **kwargs: str) -> Self:
+        color_state: dict[str, str] = {}
+        for color_name in self.__dataclass_fields__:
+            color_state[color_name] = getattr(self, color_name)
+        color_state.update(kwargs)
+        return type(self)(**color_state)
+
+    @classmethod
+    def no_colors(cls) -> Self:
+        color_state: dict[str, str] = {}
+        for color_name in cls.__dataclass_fields__:
+            color_state[color_name] = ""
+        return cls(**color_state)
+
+    def __getitem__(self, key: str) -> str:
+        return self._name_to_value(key)
+
+    def __len__(self) -> int:
+        return len(self.__dataclass_fields__)
+
+    def __iter__(self) -> Iterator[str]:
+        return iter(self.__dataclass_fields__)
+
+
+@dataclass(frozen=True, kw_only=True)
+class Argparse(ThemeSection):
+    usage: str = ANSIColors.BOLD_BLUE
+    prog: str = ANSIColors.BOLD_MAGENTA
+    prog_extra: str = ANSIColors.MAGENTA
+    heading: str = ANSIColors.BOLD_BLUE
+    summary_long_option: str = ANSIColors.CYAN
+    summary_short_option: str = ANSIColors.GREEN
+    summary_label: str = ANSIColors.YELLOW
+    summary_action: str = ANSIColors.GREEN
+    long_option: str = ANSIColors.BOLD_CYAN
+    short_option: str = ANSIColors.BOLD_GREEN
+    label: str = ANSIColors.BOLD_YELLOW
+    action: str = ANSIColors.BOLD_GREEN
+    reset: str = ANSIColors.RESET
+
+
+@dataclass(frozen=True)
+class Syntax(ThemeSection):
+    prompt: str = ANSIColors.BOLD_MAGENTA
+    keyword: str = ANSIColors.BOLD_BLUE
+    keyword_constant: str = ANSIColors.BOLD_BLUE
+    builtin: str = ANSIColors.CYAN
+    comment: str = ANSIColors.RED
+    string: str = ANSIColors.GREEN
+    number: str = ANSIColors.YELLOW
+    op: str = ANSIColors.RESET
+    definition: str = ANSIColors.BOLD
+    soft_keyword: str = ANSIColors.BOLD_BLUE
+    reset: str = ANSIColors.RESET
+
+
+@dataclass(frozen=True)
+class Traceback(ThemeSection):
+    type: str = ANSIColors.BOLD_MAGENTA
+    message: str = ANSIColors.MAGENTA
+    filename: str = ANSIColors.MAGENTA
+    line_no: str = ANSIColors.MAGENTA
+    frame: str = ANSIColors.MAGENTA
+    error_highlight: str = ANSIColors.BOLD_RED
+    error_range: str = ANSIColors.RED
+    reset: str = ANSIColors.RESET
+
+
+@dataclass(frozen=True)
+class Unittest(ThemeSection):
+    passed: str = ANSIColors.GREEN
+    warn: str = ANSIColors.YELLOW
+    fail: str = ANSIColors.RED
+    fail_info: str = ANSIColors.BOLD_RED
+    reset: str = ANSIColors.RESET
+
+
+@dataclass(frozen=True)
+class Theme:
+    """A suite of themes for all sections of Python.
+
+    When adding a new one, remember to also modify `copy_with` and `no_colors`
+    below.
+    """
+    argparse: Argparse = field(default_factory=Argparse)
+    syntax: Syntax = field(default_factory=Syntax)
+    traceback: Traceback = field(default_factory=Traceback)
+    unittest: Unittest = field(default_factory=Unittest)
+
+    def copy_with(
+        self,
+        *,
+        argparse: Argparse | None = None,
+        syntax: Syntax | None = None,
+        traceback: Traceback | None = None,
+        unittest: Unittest | None = None,
+    ) -> Self:
+        """Return a new Theme based on this instance with some sections replaced.
+
+        Themes are immutable to protect against accidental modifications that
+        could lead to invalid terminal states.
+        """
+        return type(self)(
+            argparse=argparse or self.argparse,
+            syntax=syntax or self.syntax,
+            traceback=traceback or self.traceback,
+            unittest=unittest or self.unittest,
+        )
+
+    @classmethod
+    def no_colors(cls) -> Self:
+        """Return a new Theme where colors in all sections are empty strings.
+
+        This allows writing user code as if colors are always used. The color
+        fields will be ANSI color code strings when colorization is desired
+        and possible, and empty strings otherwise.
+        """
+        return cls(
+            argparse=Argparse.no_colors(),
+            syntax=Syntax.no_colors(),
+            traceback=Traceback.no_colors(),
+            unittest=Unittest.no_colors(),
+        )
+
+
 def get_colors(
     colorize: bool = False, *, file: IO[str] | IO[bytes] | None = None
 ) -> ANSIColors:
@@ -76,22 +264,37 @@ def get_colors(
         return NoColors
 
 
+def decolor(text: str) -> str:
+    """Remove ANSI color codes from a string."""
+    for code in ColorCodes:
+        text = text.replace(code, "")
+    return text
+
+
 def can_colorize(*, file: IO[str] | IO[bytes] | None = None) -> bool:
+
+    def _safe_getenv(k: str, fallback: str | None = None) -> str | None:
+        """Exception-safe environment retrieval. See gh-128636."""
+        try:
+            return os.environ.get(k, fallback)
+        except Exception:
+            return fallback
+
     if file is None:
         file = sys.stdout
 
     if not sys.flags.ignore_environment:
-        if os.environ.get("PYTHON_COLORS") == "0":
+        if _safe_getenv("PYTHON_COLORS") == "0":
             return False
-        if os.environ.get("PYTHON_COLORS") == "1":
+        if _safe_getenv("PYTHON_COLORS") == "1":
             return True
-    if os.environ.get("NO_COLOR"):
+    if _safe_getenv("NO_COLOR"):
         return False
     if not COLORIZE:
         return False
-    if os.environ.get("FORCE_COLOR"):
+    if _safe_getenv("FORCE_COLOR"):
         return True
-    if os.environ.get("TERM") == "dumb":
+    if _safe_getenv("TERM") == "dumb":
         return False
 
     if not hasattr(file, "fileno"):
@@ -108,5 +311,45 @@ def can_colorize(*, file: IO[str] | IO[bytes] | None = None) -> bool:
 
     try:
         return os.isatty(file.fileno())
-    except io.UnsupportedOperation:
+    except OSError:
         return hasattr(file, "isatty") and file.isatty()
+
+
+default_theme = Theme()
+theme_no_color = default_theme.no_colors()
+
+
+def get_theme(
+    *,
+    tty_file: IO[str] | IO[bytes] | None = None,
+    force_color: bool = False,
+    force_no_color: bool = False,
+) -> Theme:
+    """Returns the currently set theme, potentially in a zero-color variant.
+
+    In cases where colorizing is not possible (see `can_colorize`), the returned
+    theme contains all empty strings in all color definitions.
+    See `Theme.no_colors()` for more information.
+
+    It is recommended not to cache the result of this function for extended
+    periods of time because the user might influence theme selection by
+    the interactive shell, a debugger, or application-specific code. The
+    environment (including environment variable state and console configuration
+    on Windows) can also change in the course of the application life cycle.
+    """
+    if force_color or (not force_no_color and
+                       can_colorize(file=tty_file)):
+        return _theme
+    return theme_no_color
+
+
+def set_theme(t: Theme) -> None:
+    global _theme
+
+    if not isinstance(t, Theme):
+        raise ValueError(f"Expected Theme object, found {t}")
+
+    _theme = t
+
+
+set_theme(default_theme)

Lib/_compat_pickle.py

@@ -22,7 +22,6 @@ IMPORT_MAPPING = {
     'tkMessageBox': 'tkinter.messagebox',
     'ScrolledText': 'tkinter.scrolledtext',
     'Tkconstants': 'tkinter.constants',
-    'Tix': 'tkinter.tix',
     'ttk': 'tkinter.ttk',
     'Tkinter': 'tkinter',
     'markupbase': '_markupbase',
@@ -257,3 +256,4 @@ PYTHON3_IMPORTERROR_EXCEPTIONS = (
 
 for excname in PYTHON3_IMPORTERROR_EXCEPTIONS:
     REVERSE_NAME_MAPPING[('builtins', excname)] = ('exceptions', 'ImportError')
+del excname

Lib/_dummy_thread.py

@@ -11,15 +11,35 @@ Suggested usage is::
         import _dummy_thread as _thread
 
 """
+
 # Exports only things specified by thread documentation;
 # skipping obsolete synonyms allocate(), start_new(), exit_thread().
-__all__ = ['error', 'start_new_thread', 'exit', 'get_ident', 'allocate_lock',
-           'interrupt_main', 'LockType', 'RLock',
-           '_count']
+__all__ = [
+    "error",
+    "start_new_thread",
+    "exit",
+    "get_ident",
+    "allocate_lock",
+    "interrupt_main",
+    "LockType",
+    "RLock",
+    "_count",
+    "start_joinable_thread",
+    "daemon_threads_allowed",
+    "_shutdown",
+    "_make_thread_handle",
+    "_ThreadHandle",
+    "_get_main_thread_ident",
+    "_is_main_interpreter",
+    "_local",
+]
 
 # A dummy value
 TIMEOUT_MAX = 2**31
 
+# Main thread ident for dummy implementation
+_MAIN_THREAD_IDENT = -1
+
 # NOTE: this module can be imported early in the extension building process,
 # and so top level imports of other modules should be avoided.  Instead, all
 # imports are done when needed on a function-by-function basis.  Since threads
@@ -27,6 +47,7 @@ TIMEOUT_MAX = 2**31
 
 error = RuntimeError
 
+
 def start_new_thread(function, args, kwargs={}):
     """Dummy implementation of _thread.start_new_thread().
 
@@ -52,6 +73,7 @@ def start_new_thread(function, args, kwargs={}):
         pass
     except:
         import traceback
+
         traceback.print_exc()
     _main = True
     global _interrupt
@@ -59,10 +81,58 @@ def start_new_thread(function, args, kwargs={}):
         _interrupt = False
         raise KeyboardInterrupt
 
+
+def start_joinable_thread(function, handle=None, daemon=True):
+    """Dummy implementation of _thread.start_joinable_thread().
+
+    In dummy thread, we just run the function synchronously.
+    """
+    if handle is None:
+        handle = _ThreadHandle()
+    try:
+        function()
+    except SystemExit:
+        pass
+    except:
+        import traceback
+
+        traceback.print_exc()
+    handle._set_done()
+    return handle
+
+
+def daemon_threads_allowed():
+    """Dummy implementation of _thread.daemon_threads_allowed()."""
+    return True
+
+
+def _shutdown():
+    """Dummy implementation of _thread._shutdown()."""
+    pass
+
+
+def _make_thread_handle(ident):
+    """Dummy implementation of _thread._make_thread_handle()."""
+    handle = _ThreadHandle()
+    handle._ident = ident
+    return handle
+
+
+def _get_main_thread_ident():
+    """Dummy implementation of _thread._get_main_thread_ident()."""
+    return _MAIN_THREAD_IDENT
+
+
+def _is_main_interpreter():
+    """Dummy implementation of _thread._is_main_interpreter()."""
+    return True
+
+
 def exit():
     """Dummy implementation of _thread.exit()."""
     raise SystemExit
 
+
 def get_ident():
     """Dummy implementation of _thread.get_ident().
 
@@ -70,26 +140,31 @@ def get_ident():
     available, it is safe to assume that the current process is the
     only thread.  Thus a constant can be safely returned.
     """
-    return -1
+    return _MAIN_THREAD_IDENT
+
 
 def allocate_lock():
     """Dummy implementation of _thread.allocate_lock()."""
     return LockType()
 
+
 def stack_size(size=None):
     """Dummy implementation of _thread.stack_size()."""
     if size is not None:
         raise error("setting thread stack size not supported")
     return 0
 
+
 def _set_sentinel():
     """Dummy implementation of _thread._set_sentinel()."""
     return LockType()
 
+
 def _count():
     """Dummy implementation of _thread._count()."""
     return 0
 
+
 class LockType(object):
     """Class implementing dummy implementation of _thread.LockType.
 
@@ -125,6 +200,7 @@ class LockType(object):
             else:
                 if timeout > 0:
                     import time
+
                     time.sleep(timeout)
                 return False
 
@@ -153,14 +229,41 @@ class LockType(object):
             "locked" if self.locked_status else "unlocked",
             self.__class__.__module__,
             self.__class__.__qualname__,
-            hex(id(self))
+            hex(id(self)),
         )
 
+
+class _ThreadHandle:
+    """Dummy implementation of _thread._ThreadHandle."""
+
+    def __init__(self):
+        self._ident = _MAIN_THREAD_IDENT
+        self._done = False
+
+    @property
+    def ident(self):
+        return self._ident
+
+    def _set_done(self):
+        self._done = True
+
+    def is_done(self):
+        return self._done
+
+    def join(self, timeout=None):
+        # In dummy thread, thread is always done
+        return
+
+    def __repr__(self):
+        return f"<_ThreadHandle ident={self._ident}>"
+
+
 # Used to signal that interrupt_main was called in a "thread"
 _interrupt = False
 # True when not executing in a "thread"
 _main = True
 
+
 def interrupt_main():
     """Set _interrupt flag to True to have start_new_thread raise
     KeyboardInterrupt upon exiting."""
@@ -170,6 +273,7 @@ def interrupt_main():
         global _interrupt
         _interrupt = True
 
+
 class RLock:
     def __init__(self):
         self.locked_count = 0
@@ -190,7 +294,7 @@ class RLock:
         return True
 
     def locked(self):
-        return self.locked_status != 0
+        return self.locked_count != 0
 
     def __repr__(self):
         return "<%s %s.%s object owner=%s count=%s at %s>" % (
@@ -199,5 +303,36 @@ class RLock:
             self.__class__.__qualname__,
             get_ident() if self.locked_count else 0,
             self.locked_count,
-            hex(id(self))
+            hex(id(self)),
         )
+
+
+class _local:
+    """Dummy implementation of _thread._local (thread-local storage)."""
+
+    def __init__(self):
+        object.__setattr__(self, "_local__impl", {})
+
+    def __getattribute__(self, name):
+        if name.startswith("_local__"):
+            return object.__getattribute__(self, name)
+        impl = object.__getattribute__(self, "_local__impl")
+        try:
+            return impl[name]
+        except KeyError:
+            raise AttributeError(name)
+
+    def __setattr__(self, name, value):
+        if name.startswith("_local__"):
+            return object.__setattr__(self, name, value)
+        impl = object.__getattribute__(self, "_local__impl")
+        impl[name] = value
+
+    def __delattr__(self, name):
+        if name.startswith("_local__"):
+            return object.__delattr__(self, name)
+        impl = object.__getattribute__(self, "_local__impl")
+        try:
+            del impl[name]
+        except KeyError:
+            raise AttributeError(name)

Lib/_markupbase.py

@@ -13,7 +13,7 @@ _commentclose = re.compile(r'--\s*>')
 _markedsectionclose = re.compile(r']\s*]\s*>')
 
 # An analysis of the MS-Word extensions is available at
-# http://www.planetpublish.com/xmlarena/xap/Thursday/WordtoXML.pdf
+# http://web.archive.org/web/20060321153828/http://www.planetpublish.com/xmlarena/xap/Thursday/WordtoXML.pdf
 
 _msmarkedsectionclose = re.compile(r']\s*>')
 

Lib/_opcode_metadata.py

@@ -0,0 +1,371 @@
+# This file is generated by scripts/generate_opcode_metadata.py
+# for RustPython bytecode format (CPython 3.14 compatible opcode numbers).
+# Do not edit!
+
+_specializations = {
+    "RESUME": [
+        "RESUME_CHECK",
+    ],
+    "LOAD_CONST": [
+        "LOAD_CONST_MORTAL",
+        "LOAD_CONST_IMMORTAL",
+    ],
+    "TO_BOOL": [
+        "TO_BOOL_ALWAYS_TRUE",
+        "TO_BOOL_BOOL",
+        "TO_BOOL_INT",
+        "TO_BOOL_LIST",
+        "TO_BOOL_NONE",
+        "TO_BOOL_STR",
+    ],
+    "BINARY_OP": [
+        "BINARY_OP_MULTIPLY_INT",
+        "BINARY_OP_ADD_INT",
+        "BINARY_OP_SUBTRACT_INT",
+        "BINARY_OP_MULTIPLY_FLOAT",
+        "BINARY_OP_ADD_FLOAT",
+        "BINARY_OP_SUBTRACT_FLOAT",
+        "BINARY_OP_ADD_UNICODE",
+        "BINARY_OP_SUBSCR_LIST_INT",
+        "BINARY_OP_SUBSCR_LIST_SLICE",
+        "BINARY_OP_SUBSCR_TUPLE_INT",
+        "BINARY_OP_SUBSCR_STR_INT",
+        "BINARY_OP_SUBSCR_DICT",
+        "BINARY_OP_SUBSCR_GETITEM",
+        "BINARY_OP_EXTEND",
+        "BINARY_OP_INPLACE_ADD_UNICODE",
+    ],
+    "STORE_SUBSCR": [
+        "STORE_SUBSCR_DICT",
+        "STORE_SUBSCR_LIST_INT",
+    ],
+    "SEND": [
+        "SEND_GEN",
+    ],
+    "UNPACK_SEQUENCE": [
+        "UNPACK_SEQUENCE_TWO_TUPLE",
+        "UNPACK_SEQUENCE_TUPLE",
+        "UNPACK_SEQUENCE_LIST",
+    ],
+    "STORE_ATTR": [
+        "STORE_ATTR_INSTANCE_VALUE",
+        "STORE_ATTR_SLOT",
+        "STORE_ATTR_WITH_HINT",
+    ],
+    "LOAD_GLOBAL": [
+        "LOAD_GLOBAL_MODULE",
+        "LOAD_GLOBAL_BUILTIN",
+    ],
+    "LOAD_SUPER_ATTR": [
+        "LOAD_SUPER_ATTR_ATTR",
+        "LOAD_SUPER_ATTR_METHOD",
+    ],
+    "LOAD_ATTR": [
+        "LOAD_ATTR_INSTANCE_VALUE",
+        "LOAD_ATTR_MODULE",
+        "LOAD_ATTR_WITH_HINT",
+        "LOAD_ATTR_SLOT",
+        "LOAD_ATTR_CLASS",
+        "LOAD_ATTR_CLASS_WITH_METACLASS_CHECK",
+        "LOAD_ATTR_PROPERTY",
+        "LOAD_ATTR_GETATTRIBUTE_OVERRIDDEN",
+        "LOAD_ATTR_METHOD_WITH_VALUES",
+        "LOAD_ATTR_METHOD_NO_DICT",
+        "LOAD_ATTR_METHOD_LAZY_DICT",
+        "LOAD_ATTR_NONDESCRIPTOR_WITH_VALUES",
+        "LOAD_ATTR_NONDESCRIPTOR_NO_DICT",
+    ],
+    "COMPARE_OP": [
+        "COMPARE_OP_FLOAT",
+        "COMPARE_OP_INT",
+        "COMPARE_OP_STR",
+    ],
+    "CONTAINS_OP": [
+        "CONTAINS_OP_SET",
+        "CONTAINS_OP_DICT",
+    ],
+    "JUMP_BACKWARD": [
+        "JUMP_BACKWARD_NO_JIT",
+        "JUMP_BACKWARD_JIT",
+    ],
+    "FOR_ITER": [
+        "FOR_ITER_LIST",
+        "FOR_ITER_TUPLE",
+        "FOR_ITER_RANGE",
+        "FOR_ITER_GEN",
+    ],
+    "CALL": [
+        "CALL_BOUND_METHOD_EXACT_ARGS",
+        "CALL_PY_EXACT_ARGS",
+        "CALL_TYPE_1",
+        "CALL_STR_1",
+        "CALL_TUPLE_1",
+        "CALL_BUILTIN_CLASS",
+        "CALL_BUILTIN_O",
+        "CALL_BUILTIN_FAST",
+        "CALL_BUILTIN_FAST_WITH_KEYWORDS",
+        "CALL_LEN",
+        "CALL_ISINSTANCE",
+        "CALL_LIST_APPEND",
+        "CALL_METHOD_DESCRIPTOR_O",
+        "CALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS",
+        "CALL_METHOD_DESCRIPTOR_NOARGS",
+        "CALL_METHOD_DESCRIPTOR_FAST",
+        "CALL_ALLOC_AND_ENTER_INIT",
+        "CALL_PY_GENERAL",
+        "CALL_BOUND_METHOD_GENERAL",
+        "CALL_NON_PY_GENERAL",
+    ],
+    "CALL_KW": [
+        "CALL_KW_BOUND_METHOD",
+        "CALL_KW_PY",
+        "CALL_KW_NON_PY",
+    ],
+}
+
+_specialized_opmap = {
+    'BINARY_OP_ADD_FLOAT': 129,
+    'BINARY_OP_ADD_INT': 130,
+    'BINARY_OP_ADD_UNICODE': 131,
+    'BINARY_OP_EXTEND': 132,
+    'BINARY_OP_INPLACE_ADD_UNICODE': 3,
+    'BINARY_OP_MULTIPLY_FLOAT': 133,
+    'BINARY_OP_MULTIPLY_INT': 134,
+    'BINARY_OP_SUBSCR_DICT': 135,
+    'BINARY_OP_SUBSCR_GETITEM': 136,
+    'BINARY_OP_SUBSCR_LIST_INT': 137,
+    'BINARY_OP_SUBSCR_LIST_SLICE': 138,
+    'BINARY_OP_SUBSCR_STR_INT': 139,
+    'BINARY_OP_SUBSCR_TUPLE_INT': 140,
+    'BINARY_OP_SUBTRACT_FLOAT': 141,
+    'BINARY_OP_SUBTRACT_INT': 142,
+    'CALL_ALLOC_AND_ENTER_INIT': 143,
+    'CALL_BOUND_METHOD_EXACT_ARGS': 144,
+    'CALL_BOUND_METHOD_GENERAL': 145,
+    'CALL_BUILTIN_CLASS': 146,
+    'CALL_BUILTIN_FAST': 147,
+    'CALL_BUILTIN_FAST_WITH_KEYWORDS': 148,
+    'CALL_BUILTIN_O': 149,
+    'CALL_ISINSTANCE': 150,
+    'CALL_KW_BOUND_METHOD': 151,
+    'CALL_KW_NON_PY': 152,
+    'CALL_KW_PY': 153,
+    'CALL_LEN': 154,
+    'CALL_LIST_APPEND': 155,
+    'CALL_METHOD_DESCRIPTOR_FAST': 156,
+    'CALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS': 157,
+    'CALL_METHOD_DESCRIPTOR_NOARGS': 158,
+    'CALL_METHOD_DESCRIPTOR_O': 159,
+    'CALL_NON_PY_GENERAL': 160,
+    'CALL_PY_EXACT_ARGS': 161,
+    'CALL_PY_GENERAL': 162,
+    'CALL_STR_1': 163,
+    'CALL_TUPLE_1': 164,
+    'CALL_TYPE_1': 165,
+    'COMPARE_OP_FLOAT': 166,
+    'COMPARE_OP_INT': 167,
+    'COMPARE_OP_STR': 168,
+    'CONTAINS_OP_DICT': 169,
+    'CONTAINS_OP_SET': 170,
+    'FOR_ITER_GEN': 171,
+    'FOR_ITER_LIST': 172,
+    'FOR_ITER_RANGE': 173,
+    'FOR_ITER_TUPLE': 174,
+    'JUMP_BACKWARD_JIT': 175,
+    'JUMP_BACKWARD_NO_JIT': 176,
+    'LOAD_ATTR_CLASS': 177,
+    'LOAD_ATTR_CLASS_WITH_METACLASS_CHECK': 178,
+    'LOAD_ATTR_GETATTRIBUTE_OVERRIDDEN': 179,
+    'LOAD_ATTR_INSTANCE_VALUE': 180,
+    'LOAD_ATTR_METHOD_LAZY_DICT': 181,
+    'LOAD_ATTR_METHOD_NO_DICT': 182,
+    'LOAD_ATTR_METHOD_WITH_VALUES': 183,
+    'LOAD_ATTR_MODULE': 184,
+    'LOAD_ATTR_NONDESCRIPTOR_NO_DICT': 185,
+    'LOAD_ATTR_NONDESCRIPTOR_WITH_VALUES': 186,
+    'LOAD_ATTR_PROPERTY': 187,
+    'LOAD_ATTR_SLOT': 188,
+    'LOAD_ATTR_WITH_HINT': 189,
+    'LOAD_CONST_IMMORTAL': 190,
+    'LOAD_CONST_MORTAL': 191,
+    'LOAD_GLOBAL_BUILTIN': 192,
+    'LOAD_GLOBAL_MODULE': 193,
+    'LOAD_SUPER_ATTR_ATTR': 194,
+    'LOAD_SUPER_ATTR_METHOD': 195,
+    'RESUME_CHECK': 196,
+    'SEND_GEN': 197,
+    'STORE_ATTR_INSTANCE_VALUE': 198,
+    'STORE_ATTR_SLOT': 199,
+    'STORE_ATTR_WITH_HINT': 200,
+    'STORE_SUBSCR_DICT': 201,
+    'STORE_SUBSCR_LIST_INT': 202,
+    'TO_BOOL_ALWAYS_TRUE': 203,
+    'TO_BOOL_BOOL': 204,
+    'TO_BOOL_INT': 205,
+    'TO_BOOL_LIST': 206,
+    'TO_BOOL_NONE': 207,
+    'TO_BOOL_STR': 208,
+    'UNPACK_SEQUENCE_LIST': 209,
+    'UNPACK_SEQUENCE_TUPLE': 210,
+    'UNPACK_SEQUENCE_TWO_TUPLE': 211,
+}
+
+opmap = {
+    'CACHE': 0,
+    'RESERVED': 17,
+    'RESUME': 128,
+    'INSTRUMENTED_LINE': 254,
+    'ENTER_EXECUTOR': 255,
+    'BINARY_SLICE': 1,
+    'BUILD_TEMPLATE': 2,
+    'CALL_FUNCTION_EX': 4,
+    'CHECK_EG_MATCH': 5,
+    'CHECK_EXC_MATCH': 6,
+    'CLEANUP_THROW': 7,
+    'DELETE_SUBSCR': 8,
+    'END_FOR': 9,
+    'END_SEND': 10,
+    'EXIT_INIT_CHECK': 11,
+    'FORMAT_SIMPLE': 12,
+    'FORMAT_WITH_SPEC': 13,
+    'GET_AITER': 14,
+    'GET_ANEXT': 15,
+    'GET_ITER': 16,
+    'GET_LEN': 18,
+    'GET_YIELD_FROM_ITER': 19,
+    'INTERPRETER_EXIT': 20,
+    'LOAD_BUILD_CLASS': 21,
+    'LOAD_LOCALS': 22,
+    'MAKE_FUNCTION': 23,
+    'MATCH_KEYS': 24,
+    'MATCH_MAPPING': 25,
+    'MATCH_SEQUENCE': 26,
+    'NOP': 27,
+    'NOT_TAKEN': 28,
+    'POP_EXCEPT': 29,
+    'POP_ITER': 30,
+    'POP_TOP': 31,
+    'PUSH_EXC_INFO': 32,
+    'PUSH_NULL': 33,
+    'RETURN_GENERATOR': 34,
+    'RETURN_VALUE': 35,
+    'SETUP_ANNOTATIONS': 36,
+    'STORE_SLICE': 37,
+    'STORE_SUBSCR': 38,
+    'TO_BOOL': 39,
+    'UNARY_INVERT': 40,
+    'UNARY_NEGATIVE': 41,
+    'UNARY_NOT': 42,
+    'WITH_EXCEPT_START': 43,
+    'BINARY_OP': 44,
+    'BUILD_INTERPOLATION': 45,
+    'BUILD_LIST': 46,
+    'BUILD_MAP': 47,
+    'BUILD_SET': 48,
+    'BUILD_SLICE': 49,
+    'BUILD_STRING': 50,
+    'BUILD_TUPLE': 51,
+    'CALL': 52,
+    'CALL_INTRINSIC_1': 53,
+    'CALL_INTRINSIC_2': 54,
+    'CALL_KW': 55,
+    'COMPARE_OP': 56,
+    'CONTAINS_OP': 57,
+    'CONVERT_VALUE': 58,
+    'COPY': 59,
+    'COPY_FREE_VARS': 60,
+    'DELETE_ATTR': 61,
+    'DELETE_DEREF': 62,
+    'DELETE_FAST': 63,
+    'DELETE_GLOBAL': 64,
+    'DELETE_NAME': 65,
+    'DICT_MERGE': 66,
+    'DICT_UPDATE': 67,
+    'END_ASYNC_FOR': 68,
+    'EXTENDED_ARG': 69,
+    'FOR_ITER': 70,
+    'GET_AWAITABLE': 71,
+    'IMPORT_FROM': 72,
+    'IMPORT_NAME': 73,
+    'IS_OP': 74,
+    'JUMP_BACKWARD': 75,
+    'JUMP_BACKWARD_NO_INTERRUPT': 76,
+    'JUMP_FORWARD': 77,
+    'LIST_APPEND': 78,
+    'LIST_EXTEND': 79,
+    'LOAD_ATTR': 80,
+    'LOAD_COMMON_CONSTANT': 81,
+    'LOAD_CONST': 82,
+    'LOAD_DEREF': 83,
+    'LOAD_FAST': 84,
+    'LOAD_FAST_AND_CLEAR': 85,
+    'LOAD_FAST_BORROW': 86,
+    'LOAD_FAST_BORROW_LOAD_FAST_BORROW': 87,
+    'LOAD_FAST_CHECK': 88,
+    'LOAD_FAST_LOAD_FAST': 89,
+    'LOAD_FROM_DICT_OR_DEREF': 90,
+    'LOAD_FROM_DICT_OR_GLOBALS': 91,
+    'LOAD_GLOBAL': 92,
+    'LOAD_NAME': 93,
+    'LOAD_SMALL_INT': 94,
+    'LOAD_SPECIAL': 95,
+    'LOAD_SUPER_ATTR': 96,
+    'MAKE_CELL': 97,
+    'MAP_ADD': 98,
+    'MATCH_CLASS': 99,
+    'POP_JUMP_IF_FALSE': 100,
+    'POP_JUMP_IF_NONE': 101,
+    'POP_JUMP_IF_NOT_NONE': 102,
+    'POP_JUMP_IF_TRUE': 103,
+    'RAISE_VARARGS': 104,
+    'RERAISE': 105,
+    'SEND': 106,
+    'SET_ADD': 107,
+    'SET_FUNCTION_ATTRIBUTE': 108,
+    'SET_UPDATE': 109,
+    'STORE_ATTR': 110,
+    'STORE_DEREF': 111,
+    'STORE_FAST': 112,
+    'STORE_FAST_LOAD_FAST': 113,
+    'STORE_FAST_STORE_FAST': 114,
+    'STORE_GLOBAL': 115,
+    'STORE_NAME': 116,
+    'SWAP': 117,
+    'UNPACK_EX': 118,
+    'UNPACK_SEQUENCE': 119,
+    'YIELD_VALUE': 120,
+    'INSTRUMENTED_END_FOR': 234,
+    'INSTRUMENTED_POP_ITER': 235,
+    'INSTRUMENTED_END_SEND': 236,
+    'INSTRUMENTED_FOR_ITER': 237,
+    'INSTRUMENTED_INSTRUCTION': 238,
+    'INSTRUMENTED_JUMP_FORWARD': 239,
+    'INSTRUMENTED_NOT_TAKEN': 240,
+    'INSTRUMENTED_POP_JUMP_IF_TRUE': 241,
+    'INSTRUMENTED_POP_JUMP_IF_FALSE': 242,
+    'INSTRUMENTED_POP_JUMP_IF_NONE': 243,
+    'INSTRUMENTED_POP_JUMP_IF_NOT_NONE': 244,
+    'INSTRUMENTED_RESUME': 245,
+    'INSTRUMENTED_RETURN_VALUE': 246,
+    'INSTRUMENTED_YIELD_VALUE': 247,
+    'INSTRUMENTED_END_ASYNC_FOR': 248,
+    'INSTRUMENTED_LOAD_SUPER_ATTR': 249,
+    'INSTRUMENTED_CALL': 250,
+    'INSTRUMENTED_CALL_KW': 251,
+    'INSTRUMENTED_CALL_FUNCTION_EX': 252,
+    'INSTRUMENTED_JUMP_BACKWARD': 253,
+    'ANNOTATIONS_PLACEHOLDER': 256,
+    'JUMP': 257,
+    'JUMP_IF_FALSE': 258,
+    'JUMP_IF_TRUE': 259,
+    'JUMP_NO_INTERRUPT': 260,
+    'LOAD_CLOSURE': 261,
+    'POP_BLOCK': 262,
+    'SETUP_CLEANUP': 263,
+    'SETUP_FINALLY': 264,
+    'SETUP_WITH': 265,
+    'STORE_FAST_MAYBE_NULL': 266,
+}
+
+HAVE_ARGUMENT = 43
+MIN_INSTRUMENTED_OPCODE = 234

Lib/_py_warnings.py

@@ -0,0 +1,869 @@
+"""Python part of the warnings subsystem."""
+
+import sys
+import _contextvars
+import _thread
+
+
+__all__ = ["warn", "warn_explicit", "showwarning",
+           "formatwarning", "filterwarnings", "simplefilter",
+           "resetwarnings", "catch_warnings", "deprecated"]
+
+
+# Normally '_wm' is sys.modules['warnings'] but for unit tests it can be
+# a different module.  User code is allowed to reassign global attributes
+# of the 'warnings' module, commonly 'filters' or 'showwarning'. So we
+# need to lookup these global attributes dynamically on the '_wm' object,
+# rather than binding them earlier.  The code in this module consistently uses
+# '_wm.<something>' rather than using the globals of this module.  If the
+# '_warnings' C extension is in use, some globals are replaced by functions
+# and variables defined in that extension.
+_wm = None
+
+
+def _set_module(module):
+    global _wm
+    _wm = module
+
+
+# filters contains a sequence of filter 5-tuples
+# The components of the 5-tuple are:
+# - an action: error, ignore, always, all, default, module, or once
+# - a compiled regex that must match the warning message
+# - a class representing the warning category
+# - a compiled regex that must match the module that is being warned
+# - a line number for the line being warning, or 0 to mean any line
+# If either if the compiled regexs are None, match anything.
+filters = []
+
+
+defaultaction = "default"
+onceregistry = {}
+_lock = _thread.RLock()
+_filters_version = 1
+
+
+# If true, catch_warnings() will use a context var to hold the modified
+# filters list.  Otherwise, catch_warnings() will operate on the 'filters'
+# global of the warnings module.
+_use_context = sys.flags.context_aware_warnings
+
+
+class _Context:
+    def __init__(self, filters):
+        self._filters = filters
+        self.log = None  # if set to a list, logging is enabled
+
+    def copy(self):
+        context = _Context(self._filters[:])
+        if self.log is not None:
+            context.log = self.log
+        return context
+
+    def _record_warning(self, msg):
+        self.log.append(msg)
+
+
+class _GlobalContext(_Context):
+    def __init__(self):
+        self.log = None
+
+    @property
+    def _filters(self):
+        # Since there is quite a lot of code that assigns to
+        # warnings.filters, this needs to return the current value of
+        # the module global.
+        try:
+            return _wm.filters
+        except AttributeError:
+            # 'filters' global was deleted.  Do we need to actually handle this case?
+            return []
+
+
+_global_context = _GlobalContext()
+
+
+_warnings_context = _contextvars.ContextVar('warnings_context')
+
+
+def _get_context():
+    if not _use_context:
+        return _global_context
+    try:
+        return _wm._warnings_context.get()
+    except LookupError:
+        return _global_context
+
+
+def _set_context(context):
+    assert _use_context
+    _wm._warnings_context.set(context)
+
+
+def _new_context():
+    assert _use_context
+    old_context = _wm._get_context()
+    new_context = old_context.copy()
+    _wm._set_context(new_context)
+    return old_context, new_context
+
+
+def _get_filters():
+    """Return the current list of filters.  This is a non-public API used by
+    module functions and by the unit tests."""
+    return _wm._get_context()._filters
+
+
+def _filters_mutated_lock_held():
+    _wm._filters_version += 1
+
+
+def showwarning(message, category, filename, lineno, file=None, line=None):
+    """Hook to write a warning to a file; replace if you like."""
+    msg = _wm.WarningMessage(message, category, filename, lineno, file, line)
+    _wm._showwarnmsg_impl(msg)
+
+
+def formatwarning(message, category, filename, lineno, line=None):
+    """Function to format a warning the standard way."""
+    msg = _wm.WarningMessage(message, category, filename, lineno, None, line)
+    return _wm._formatwarnmsg_impl(msg)
+
+
+def _showwarnmsg_impl(msg):
+    context = _wm._get_context()
+    if context.log is not None:
+        context._record_warning(msg)
+        return
+    file = msg.file
+    if file is None:
+        file = sys.stderr
+        if file is None:
+            # sys.stderr is None when run with pythonw.exe:
+            # warnings get lost
+            return
+    text = _wm._formatwarnmsg(msg)
+    try:
+        file.write(text)
+    except OSError:
+        # the file (probably stderr) is invalid - this warning gets lost.
+        pass
+
+
+def _formatwarnmsg_impl(msg):
+    category = msg.category.__name__
+    s =  f"{msg.filename}:{msg.lineno}: {category}: {msg.message}\n"
+
+    if msg.line is None:
+        try:
+            import linecache
+            line = linecache.getline(msg.filename, msg.lineno)
+        except Exception:
+            # When a warning is logged during Python shutdown, linecache
+            # and the import machinery don't work anymore
+            line = None
+            linecache = None
+    else:
+        line = msg.line
+    if line:
+        line = line.strip()
+        s += "  %s\n" % line
+
+    if msg.source is not None:
+        try:
+            import tracemalloc
+        # Logging a warning should not raise a new exception:
+        # catch Exception, not only ImportError and RecursionError.
+        except Exception:
+            # don't suggest to enable tracemalloc if it's not available
+            suggest_tracemalloc = False
+            tb = None
+        else:
+            try:
+                suggest_tracemalloc = not tracemalloc.is_tracing()
+                tb = tracemalloc.get_object_traceback(msg.source)
+            except Exception:
+                # When a warning is logged during Python shutdown, tracemalloc
+                # and the import machinery don't work anymore
+                suggest_tracemalloc = False
+                tb = None
+
+        if tb is not None:
+            s += 'Object allocated at (most recent call last):\n'
+            for frame in tb:
+                s += ('  File "%s", lineno %s\n'
+                      % (frame.filename, frame.lineno))
+
+                try:
+                    if linecache is not None:
+                        line = linecache.getline(frame.filename, frame.lineno)
+                    else:
+                        line = None
+                except Exception:
+                    line = None
+                if line:
+                    line = line.strip()
+                    s += '    %s\n' % line
+        elif suggest_tracemalloc:
+            s += (f'{category}: Enable tracemalloc to get the object '
+                  f'allocation traceback\n')
+    return s
+
+
+# Keep a reference to check if the function was replaced
+_showwarning_orig = showwarning
+
+
+def _showwarnmsg(msg):
+    """Hook to write a warning to a file; replace if you like."""
+    try:
+        sw = _wm.showwarning
+    except AttributeError:
+        pass
+    else:
+        if sw is not _showwarning_orig:
+            # warnings.showwarning() was replaced
+            if not callable(sw):
+                raise TypeError("warnings.showwarning() must be set to a "
+                                "function or method")
+
+            sw(msg.message, msg.category, msg.filename, msg.lineno,
+               msg.file, msg.line)
+            return
+    _wm._showwarnmsg_impl(msg)
+
+
+# Keep a reference to check if the function was replaced
+_formatwarning_orig = formatwarning
+
+
+def _formatwarnmsg(msg):
+    """Function to format a warning the standard way."""
+    try:
+        fw = _wm.formatwarning
+    except AttributeError:
+        pass
+    else:
+        if fw is not _formatwarning_orig:
+            # warnings.formatwarning() was replaced
+            return fw(msg.message, msg.category,
+                      msg.filename, msg.lineno, msg.line)
+    return _wm._formatwarnmsg_impl(msg)
+
+
+def filterwarnings(action, message="", category=Warning, module="", lineno=0,
+                   append=False):
+    """Insert an entry into the list of warnings filters (at the front).
+
+    'action' -- one of "error", "ignore", "always", "all", "default", "module",
+                or "once"
+    'message' -- a regex that the warning message must match
+    'category' -- a class that the warning must be a subclass of
+    'module' -- a regex that the module name must match
+    'lineno' -- an integer line number, 0 matches all warnings
+    'append' -- if true, append to the list of filters
+    """
+    if action not in {"error", "ignore", "always", "all", "default", "module", "once"}:
+        raise ValueError(f"invalid action: {action!r}")
+    if not isinstance(message, str):
+        raise TypeError("message must be a string")
+    if not isinstance(category, type) or not issubclass(category, Warning):
+        raise TypeError("category must be a Warning subclass")
+    if not isinstance(module, str):
+        raise TypeError("module must be a string")
+    if not isinstance(lineno, int):
+        raise TypeError("lineno must be an int")
+    if lineno < 0:
+        raise ValueError("lineno must be an int >= 0")
+
+    if message or module:
+        import re
+
+    if message:
+        message = re.compile(message, re.I)
+    else:
+        message = None
+    if module:
+        module = re.compile(module)
+    else:
+        module = None
+
+    _wm._add_filter(action, message, category, module, lineno, append=append)
+
+
+def simplefilter(action, category=Warning, lineno=0, append=False):
+    """Insert a simple entry into the list of warnings filters (at the front).
+
+    A simple filter matches all modules and messages.
+    'action' -- one of "error", "ignore", "always", "all", "default", "module",
+                or "once"
+    'category' -- a class that the warning must be a subclass of
+    'lineno' -- an integer line number, 0 matches all warnings
+    'append' -- if true, append to the list of filters
+    """
+    if action not in {"error", "ignore", "always", "all", "default", "module", "once"}:
+        raise ValueError(f"invalid action: {action!r}")
+    if not isinstance(lineno, int):
+        raise TypeError("lineno must be an int")
+    if lineno < 0:
+        raise ValueError("lineno must be an int >= 0")
+    _wm._add_filter(action, None, category, None, lineno, append=append)
+
+
+def _filters_mutated():
+    # Even though this function is not part of the public API, it's used by
+    # a fair amount of user code.
+    with _wm._lock:
+        _wm._filters_mutated_lock_held()
+
+
+def _add_filter(*item, append):
+    with _wm._lock:
+        filters = _wm._get_filters()
+        if not append:
+            # Remove possible duplicate filters, so new one will be placed
+            # in correct place. If append=True and duplicate exists, do nothing.
+            try:
+                filters.remove(item)
+            except ValueError:
+                pass
+            filters.insert(0, item)
+        else:
+            if item not in filters:
+                filters.append(item)
+        _wm._filters_mutated_lock_held()
+
+
+def resetwarnings():
+    """Clear the list of warning filters, so that no filters are active."""
+    with _wm._lock:
+        del _wm._get_filters()[:]
+        _wm._filters_mutated_lock_held()
+
+
+class _OptionError(Exception):
+    """Exception used by option processing helpers."""
+    pass
+
+
+# Helper to process -W options passed via sys.warnoptions
+def _processoptions(args):
+    for arg in args:
+        try:
+            _wm._setoption(arg)
+        except _wm._OptionError as msg:
+            print("Invalid -W option ignored:", msg, file=sys.stderr)
+
+
+# Helper for _processoptions()
+def _setoption(arg):
+    parts = arg.split(':')
+    if len(parts) > 5:
+        raise _wm._OptionError("too many fields (max 5): %r" % (arg,))
+    while len(parts) < 5:
+        parts.append('')
+    action, message, category, module, lineno = [s.strip()
+                                                 for s in parts]
+    action = _wm._getaction(action)
+    category = _wm._getcategory(category)
+    if message or module:
+        import re
+    if message:
+        message = re.escape(message)
+    if module:
+        module = re.escape(module) + r'\z'
+    if lineno:
+        try:
+            lineno = int(lineno)
+            if lineno < 0:
+                raise ValueError
+        except (ValueError, OverflowError):
+            raise _wm._OptionError("invalid lineno %r" % (lineno,)) from None
+    else:
+        lineno = 0
+    _wm.filterwarnings(action, message, category, module, lineno)
+
+
+# Helper for _setoption()
+def _getaction(action):
+    if not action:
+        return "default"
+    for a in ('default', 'always', 'all', 'ignore', 'module', 'once', 'error'):
+        if a.startswith(action):
+            return a
+    raise _wm._OptionError("invalid action: %r" % (action,))
+
+
+# Helper for _setoption()
+def _getcategory(category):
+    if not category:
+        return Warning
+    if '.' not in category:
+        import builtins as m
+        klass = category
+    else:
+        module, _, klass = category.rpartition('.')
+        try:
+            m = __import__(module, None, None, [klass])
+        except ImportError:
+            raise _wm._OptionError("invalid module name: %r" % (module,)) from None
+    try:
+        cat = getattr(m, klass)
+    except AttributeError:
+        raise _wm._OptionError("unknown warning category: %r" % (category,)) from None
+    if not issubclass(cat, Warning):
+        raise _wm._OptionError("invalid warning category: %r" % (category,))
+    return cat
+
+
+def _is_internal_filename(filename):
+    return 'importlib' in filename and '_bootstrap' in filename
+
+
+def _is_filename_to_skip(filename, skip_file_prefixes):
+    return any(filename.startswith(prefix) for prefix in skip_file_prefixes)
+
+
+def _is_internal_frame(frame):
+    """Signal whether the frame is an internal CPython implementation detail."""
+    return _is_internal_filename(frame.f_code.co_filename)
+
+
+def _next_external_frame(frame, skip_file_prefixes):
+    """Find the next frame that doesn't involve Python or user internals."""
+    frame = frame.f_back
+    while frame is not None and (
+            _is_internal_filename(filename := frame.f_code.co_filename) or
+            _is_filename_to_skip(filename, skip_file_prefixes)):
+        frame = frame.f_back
+    return frame
+
+
+# Code typically replaced by _warnings
+def warn(message, category=None, stacklevel=1, source=None,
+         *, skip_file_prefixes=()):
+    """Issue a warning, or maybe ignore it or raise an exception."""
+    # Check if message is already a Warning object
+    if isinstance(message, Warning):
+        category = message.__class__
+    # Check category argument
+    if category is None:
+        category = UserWarning
+    if not (isinstance(category, type) and issubclass(category, Warning)):
+        raise TypeError("category must be a Warning subclass, "
+                        "not '{:s}'".format(type(category).__name__))
+    if not isinstance(skip_file_prefixes, tuple):
+        # The C version demands a tuple for implementation performance.
+        raise TypeError('skip_file_prefixes must be a tuple of strs.')
+    if skip_file_prefixes:
+        stacklevel = max(2, stacklevel)
+    # Get context information
+    try:
+        if stacklevel <= 1 or _is_internal_frame(sys._getframe(1)):
+            # If frame is too small to care or if the warning originated in
+            # internal code, then do not try to hide any frames.
+            frame = sys._getframe(stacklevel)
+        else:
+            frame = sys._getframe(1)
+            # Look for one frame less since the above line starts us off.
+            for x in range(stacklevel-1):
+                frame = _next_external_frame(frame, skip_file_prefixes)
+                if frame is None:
+                    raise ValueError
+    except ValueError:
+        globals = sys.__dict__
+        filename = "<sys>"
+        lineno = 0
+    else:
+        globals = frame.f_globals
+        filename = frame.f_code.co_filename
+        lineno = frame.f_lineno
+    if '__name__' in globals:
+        module = globals['__name__']
+    else:
+        module = "<string>"
+    registry = globals.setdefault("__warningregistry__", {})
+    _wm.warn_explicit(
+        message,
+        category,
+        filename,
+        lineno,
+        module,
+        registry,
+        globals,
+        source=source,
+    )
+
+
+def warn_explicit(message, category, filename, lineno,
+                  module=None, registry=None, module_globals=None,
+                  source=None):
+    lineno = int(lineno)
+    if module is None:
+        module = filename or "<unknown>"
+        if module[-3:].lower() == ".py":
+            module = module[:-3] # XXX What about leading pathname?
+    if isinstance(message, Warning):
+        text = str(message)
+        category = message.__class__
+    else:
+        text = message
+        message = category(message)
+    key = (text, category, lineno)
+    with _wm._lock:
+        if registry is None:
+            registry = {}
+        if registry.get('version', 0) != _wm._filters_version:
+            registry.clear()
+            registry['version'] = _wm._filters_version
+        # Quick test for common case
+        if registry.get(key):
+            return
+        # Search the filters
+        for item in _wm._get_filters():
+            action, msg, cat, mod, ln = item
+            if ((msg is None or msg.match(text)) and
+                issubclass(category, cat) and
+                (mod is None or mod.match(module)) and
+                (ln == 0 or lineno == ln)):
+                break
+        else:
+            action = _wm.defaultaction
+        # Early exit actions
+        if action == "ignore":
+            return
+
+        if action == "error":
+            raise message
+        # Other actions
+        if action == "once":
+            registry[key] = 1
+            oncekey = (text, category)
+            if _wm.onceregistry.get(oncekey):
+                return
+            _wm.onceregistry[oncekey] = 1
+        elif action in {"always", "all"}:
+            pass
+        elif action == "module":
+            registry[key] = 1
+            altkey = (text, category, 0)
+            if registry.get(altkey):
+                return
+            registry[altkey] = 1
+        elif action == "default":
+            registry[key] = 1
+        else:
+            # Unrecognized actions are errors
+            raise RuntimeError(
+                  "Unrecognized action (%r) in warnings.filters:\n %s" %
+                  (action, item))
+
+    # Prime the linecache for formatting, in case the
+    # "file" is actually in a zipfile or something.
+    import linecache
+    linecache.getlines(filename, module_globals)
+
+    # Print message and context
+    msg = _wm.WarningMessage(message, category, filename, lineno, source=source)
+    _wm._showwarnmsg(msg)
+
+
+class WarningMessage(object):
+
+    _WARNING_DETAILS = ("message", "category", "filename", "lineno", "file",
+                        "line", "source")
+
+    def __init__(self, message, category, filename, lineno, file=None,
+                 line=None, source=None):
+        self.message = message
+        self.category = category
+        self.filename = filename
+        self.lineno = lineno
+        self.file = file
+        self.line = line
+        self.source = source
+        self._category_name = category.__name__ if category else None
+
+    def __str__(self):
+        return ("{message : %r, category : %r, filename : %r, lineno : %s, "
+                    "line : %r}" % (self.message, self._category_name,
+                                    self.filename, self.lineno, self.line))
+
+    def __repr__(self):
+        return f'<{type(self).__qualname__} {self}>'
+
+
+class catch_warnings(object):
+
+    """A context manager that copies and restores the warnings filter upon
+    exiting the context.
+
+    The 'record' argument specifies whether warnings should be captured by a
+    custom implementation of warnings.showwarning() and be appended to a list
+    returned by the context manager. Otherwise None is returned by the context
+    manager. The objects appended to the list are arguments whose attributes
+    mirror the arguments to showwarning().
+
+    The 'module' argument is to specify an alternative module to the module
+    named 'warnings' and imported under that name. This argument is only useful
+    when testing the warnings module itself.
+
+    If the 'action' argument is not None, the remaining arguments are passed
+    to warnings.simplefilter() as if it were called immediately on entering the
+    context.
+    """
+
+    def __init__(self, *, record=False, module=None,
+                 action=None, category=Warning, lineno=0, append=False):
+        """Specify whether to record warnings and if an alternative module
+        should be used other than sys.modules['warnings'].
+
+        """
+        self._record = record
+        self._module = sys.modules['warnings'] if module is None else module
+        self._entered = False
+        if action is None:
+            self._filter = None
+        else:
+            self._filter = (action, category, lineno, append)
+
+    def __repr__(self):
+        args = []
+        if self._record:
+            args.append("record=True")
+        if self._module is not sys.modules['warnings']:
+            args.append("module=%r" % self._module)
+        name = type(self).__name__
+        return "%s(%s)" % (name, ", ".join(args))
+
+    def __enter__(self):
+        if self._entered:
+            raise RuntimeError("Cannot enter %r twice" % self)
+        self._entered = True
+        with _wm._lock:
+            if _use_context:
+                self._saved_context, context = self._module._new_context()
+            else:
+                context = None
+                self._filters = self._module.filters
+                self._module.filters = self._filters[:]
+                self._showwarning = self._module.showwarning
+                self._showwarnmsg_impl = self._module._showwarnmsg_impl
+            self._module._filters_mutated_lock_held()
+            if self._record:
+                if _use_context:
+                    context.log = log = []
+                else:
+                    log = []
+                    self._module._showwarnmsg_impl = log.append
+                    # Reset showwarning() to the default implementation to make sure
+                    # that _showwarnmsg() calls _showwarnmsg_impl()
+                    self._module.showwarning = self._module._showwarning_orig
+            else:
+                log = None
+        if self._filter is not None:
+            self._module.simplefilter(*self._filter)
+        return log
+
+    def __exit__(self, *exc_info):
+        if not self._entered:
+            raise RuntimeError("Cannot exit %r without entering first" % self)
+        with _wm._lock:
+            if _use_context:
+                self._module._warnings_context.set(self._saved_context)
+            else:
+                self._module.filters = self._filters
+                self._module.showwarning = self._showwarning
+                self._module._showwarnmsg_impl = self._showwarnmsg_impl
+            self._module._filters_mutated_lock_held()
+
+
+class deprecated:
+    """Indicate that a class, function or overload is deprecated.
+
+    When this decorator is applied to an object, the type checker
+    will generate a diagnostic on usage of the deprecated object.
+
+    Usage:
+
+        @deprecated("Use B instead")
+        class A:
+            pass
+
+        @deprecated("Use g instead")
+        def f():
+            pass
+
+        @overload
+        @deprecated("int support is deprecated")
+        def g(x: int) -> int: ...
+        @overload
+        def g(x: str) -> int: ...
+
+    The warning specified by *category* will be emitted at runtime
+    on use of deprecated objects. For functions, that happens on calls;
+    for classes, on instantiation and on creation of subclasses.
+    If the *category* is ``None``, no warning is emitted at runtime.
+    The *stacklevel* determines where the
+    warning is emitted. If it is ``1`` (the default), the warning
+    is emitted at the direct caller of the deprecated object; if it
+    is higher, it is emitted further up the stack.
+    Static type checker behavior is not affected by the *category*
+    and *stacklevel* arguments.
+
+    The deprecation message passed to the decorator is saved in the
+    ``__deprecated__`` attribute on the decorated object.
+    If applied to an overload, the decorator
+    must be after the ``@overload`` decorator for the attribute to
+    exist on the overload as returned by ``get_overloads()``.
+
+    See PEP 702 for details.
+
+    """
+    def __init__(
+        self,
+        message: str,
+        /,
+        *,
+        category: type[Warning] | None = DeprecationWarning,
+        stacklevel: int = 1,
+    ) -> None:
+        if not isinstance(message, str):
+            raise TypeError(
+                f"Expected an object of type str for 'message', not {type(message).__name__!r}"
+            )
+        self.message = message
+        self.category = category
+        self.stacklevel = stacklevel
+
+    def __call__(self, arg, /):
+        # Make sure the inner functions created below don't
+        # retain a reference to self.
+        msg = self.message
+        category = self.category
+        stacklevel = self.stacklevel
+        if category is None:
+            arg.__deprecated__ = msg
+            return arg
+        elif isinstance(arg, type):
+            import functools
+            from types import MethodType
+
+            original_new = arg.__new__
+
+            @functools.wraps(original_new)
+            def __new__(cls, /, *args, **kwargs):
+                if cls is arg:
+                    _wm.warn(msg, category=category, stacklevel=stacklevel + 1)
+                if original_new is not object.__new__:
+                    return original_new(cls, *args, **kwargs)
+                # Mirrors a similar check in object.__new__.
+                elif cls.__init__ is object.__init__ and (args or kwargs):
+                    raise TypeError(f"{cls.__name__}() takes no arguments")
+                else:
+                    return original_new(cls)
+
+            arg.__new__ = staticmethod(__new__)
+
+            if "__init_subclass__" in arg.__dict__:
+                # __init_subclass__ is directly present on the decorated class.
+                # Synthesize a wrapper that calls this method directly.
+                original_init_subclass = arg.__init_subclass__
+                # We need slightly different behavior if __init_subclass__
+                # is a bound method (likely if it was implemented in Python).
+                # Otherwise, it likely means it's a builtin such as
+                # object's implementation of __init_subclass__.
+                if isinstance(original_init_subclass, MethodType):
+                    original_init_subclass = original_init_subclass.__func__
+
+                @functools.wraps(original_init_subclass)
+                def __init_subclass__(*args, **kwargs):
+                    _wm.warn(msg, category=category, stacklevel=stacklevel + 1)
+                    return original_init_subclass(*args, **kwargs)
+            else:
+                def __init_subclass__(cls, *args, **kwargs):
+                    _wm.warn(msg, category=category, stacklevel=stacklevel + 1)
+                    return super(arg, cls).__init_subclass__(*args, **kwargs)
+
+            arg.__init_subclass__ = classmethod(__init_subclass__)
+
+            arg.__deprecated__ = __new__.__deprecated__ = msg
+            __init_subclass__.__deprecated__ = msg
+            return arg
+        elif callable(arg):
+            import functools
+            import inspect
+
+            @functools.wraps(arg)
+            def wrapper(*args, **kwargs):
+                _wm.warn(msg, category=category, stacklevel=stacklevel + 1)
+                return arg(*args, **kwargs)
+
+            if inspect.iscoroutinefunction(arg):
+                wrapper = inspect.markcoroutinefunction(wrapper)
+
+            arg.__deprecated__ = wrapper.__deprecated__ = msg
+            return wrapper
+        else:
+            raise TypeError(
+                "@deprecated decorator with non-None category must be applied to "
+                f"a class or callable, not {arg!r}"
+            )
+
+
+_DEPRECATED_MSG = "{name!r} is deprecated and slated for removal in Python {remove}"
+
+
+def _deprecated(name, message=_DEPRECATED_MSG, *, remove, _version=sys.version_info):
+    """Warn that *name* is deprecated or should be removed.
+
+    RuntimeError is raised if *remove* specifies a major/minor tuple older than
+    the current Python version or the same version but past the alpha.
+
+    The *message* argument is formatted with *name* and *remove* as a Python
+    version tuple (e.g. (3, 11)).
+
+    """
+    remove_formatted = f"{remove[0]}.{remove[1]}"
+    if (_version[:2] > remove) or (_version[:2] == remove and _version[3] != "alpha"):
+        msg = f"{name!r} was slated for removal after Python {remove_formatted} alpha"
+        raise RuntimeError(msg)
+    else:
+        msg = message.format(name=name, remove=remove_formatted)
+        _wm.warn(msg, DeprecationWarning, stacklevel=3)
+
+
+# Private utility function called by _PyErr_WarnUnawaitedCoroutine
+def _warn_unawaited_coroutine(coro):
+    msg_lines = [
+        f"coroutine '{coro.__qualname__}' was never awaited\n"
+    ]
+    if coro.cr_origin is not None:
+        import linecache, traceback
+        def extract():
+            for filename, lineno, funcname in reversed(coro.cr_origin):
+                line = linecache.getline(filename, lineno)
+                yield (filename, lineno, funcname, line)
+        msg_lines.append("Coroutine created at (most recent call last)\n")
+        msg_lines += traceback.format_list(list(extract()))
+    msg = "".join(msg_lines).rstrip("\n")
+    # Passing source= here means that if the user happens to have tracemalloc
+    # enabled and tracking where the coroutine was created, the warning will
+    # contain that traceback. This does mean that if they have *both*
+    # coroutine origin tracking *and* tracemalloc enabled, they'll get two
+    # partially-redundant tracebacks. If we wanted to be clever we could
+    # probably detect this case and avoid it, but for now we don't bother.
+    _wm.warn(
+        msg, category=RuntimeWarning, stacklevel=2, source=coro
+    )
+
+
+def _setup_defaults():
+    # Several warning categories are ignored by default in regular builds
+    if hasattr(sys, 'gettotalrefcount'):
+        return
+    _wm.filterwarnings("default", category=DeprecationWarning, module="__main__", append=1)
+    _wm.simplefilter("ignore", category=DeprecationWarning, append=1)
+    _wm.simplefilter("ignore", category=PendingDeprecationWarning, append=1)
+    _wm.simplefilter("ignore", category=ImportWarning, append=1)
+    _wm.simplefilter("ignore", category=ResourceWarning, append=1)

Lib/_pydatetime.py

@@ -1,12 +1,10 @@
-"""Concrete date/time and related types.
-
-See http://www.iana.org/time-zones/repository/tz-link.html for
-time zone and DST data sources.
-"""
+"""Pure Python implementation of the datetime module."""
 
 __all__ = ("date", "datetime", "time", "timedelta", "timezone", "tzinfo",
            "MINYEAR", "MAXYEAR", "UTC")
 
+__name__ = "datetime"
+
 
 import time as _time
 import math as _math
@@ -18,10 +16,10 @@ def _cmp(x, y):
 
 def _get_class_module(self):
     module_name = self.__class__.__module__
-    if module_name == '_pydatetime':
-        return 'datetime'
+    if module_name == 'datetime':
+        return 'datetime.'
     else:
-        return module_name
+        return ''
 
 MINYEAR = 1
 MAXYEAR = 9999
@@ -64,14 +62,14 @@ def _days_in_month(year, month):
 
 def _days_before_month(year, month):
     "year, month -> number of days in year preceding first day of month."
-    assert 1 <= month <= 12, 'month must be in 1..12'
+    assert 1 <= month <= 12, f"month must be in 1..12, not {month}"
     return _DAYS_BEFORE_MONTH[month] + (month > 2 and _is_leap(year))
 
 def _ymd2ord(year, month, day):
     "year, month, day -> ordinal, considering 01-Jan-0001 as day 1."
-    assert 1 <= month <= 12, 'month must be in 1..12'
+    assert 1 <= month <= 12, f"month must be in 1..12, not {month}"
     dim = _days_in_month(year, month)
-    assert 1 <= day <= dim, ('day must be in 1..%d' % dim)
+    assert 1 <= day <= dim, f"day must be in 1..{dim}, not {day}"
     return (_days_before_year(year) +
             _days_before_month(year, month) +
             day)
@@ -204,6 +202,17 @@ def _format_offset(off, sep=':'):
                 s += '.%06d' % ss.microseconds
     return s
 
+_normalize_century = None
+def _need_normalize_century():
+    global _normalize_century
+    if _normalize_century is None:
+        try:
+            _normalize_century = (
+                _time.strftime("%Y", (99, 1, 1, 0, 0, 0, 0, 1, 0)) != "0099")
+        except ValueError:
+            _normalize_century = True
+    return _normalize_century
+
 # Correctly substitute for %z and %Z escapes in strftime formats.
 def _wrap_strftime(object, format, timetuple):
     # Don't call utcoffset() or tzname() unless actually needed.
@@ -261,6 +270,20 @@ def _wrap_strftime(object, format, timetuple):
                                 # strftime is going to have at this: escape %
                                 Zreplace = s.replace('%', '%%')
                     newformat.append(Zreplace)
+                # Note that datetime(1000, 1, 1).strftime('%G') == '1000' so
+                # year 1000 for %G can go on the fast path.
+                elif ((ch in 'YG' or ch in 'FC') and
+                        object.year < 1000 and _need_normalize_century()):
+                    if ch == 'G':
+                        year = int(_time.strftime("%G", timetuple))
+                    else:
+                        year = object.year
+                    if ch == 'C':
+                        push('{:02}'.format(year // 100))
+                    else:
+                        push('{:04}'.format(year))
+                        if ch == 'F':
+                            push('-{:02}-{:02}'.format(*timetuple[1:3]))
                 else:
                     push('%')
                     push(ch)
@@ -399,9 +422,11 @@ def _parse_hh_mm_ss_ff(tstr):
 
     if pos < len_str:
         if tstr[pos] not in '.,':
-            raise ValueError("Invalid microsecond component")
+            raise ValueError("Invalid microsecond separator")
         else:
             pos += 1
+            if not all(map(_is_ascii_digit, tstr[pos:])):
+                raise ValueError("Non-digit values in fraction")
 
             len_remainder = len_str - pos
 
@@ -413,9 +438,6 @@ def _parse_hh_mm_ss_ff(tstr):
             time_comps[3] = int(tstr[pos:(pos+to_parse)])
             if to_parse < 6:
                 time_comps[3] *= _FRACTION_CORRECTION[to_parse-1]
-            if (len_remainder > to_parse
-                    and not all(map(_is_ascii_digit, tstr[(pos+to_parse):]))):
-                raise ValueError("Non-digit values in unparsed fraction")
 
     return time_comps
 
@@ -431,6 +453,17 @@ def _parse_isoformat_time(tstr):
 
     time_comps = _parse_hh_mm_ss_ff(timestr)
 
+    hour, minute, second, microsecond = time_comps
+    became_next_day = False
+    error_from_components = False
+    if (hour == 24):
+        if all(time_comp == 0 for time_comp in time_comps[1:]):
+            hour = 0
+            time_comps[0] = hour
+            became_next_day = True
+        else:
+            error_from_components = True
+
     tzi = None
     if tz_pos == len_str and tstr[-1] == 'Z':
         tzi = timezone.utc
@@ -446,7 +479,7 @@ def _parse_isoformat_time(tstr):
         # HH:MM:SS            len: 8
         # HH:MM:SS.f+         len: 10+
 
-        if len(tzstr) in (0, 1, 3):
+        if len(tzstr) in (0, 1, 3) or tstr[tz_pos-1] == 'Z':
             raise ValueError("Malformed time zone string")
 
         tz_comps = _parse_hh_mm_ss_ff(tzstr)
@@ -463,13 +496,13 @@ def _parse_isoformat_time(tstr):
 
     time_comps.append(tzi)
 
-    return time_comps
+    return time_comps, became_next_day, error_from_components
 
 # tuple[int, int, int] -> tuple[int, int, int] version of date.fromisocalendar
 def _isoweek_to_gregorian(year, week, day):
     # Year is bounded this way because 9999-12-31 is (9999, 52, 5)
     if not MINYEAR <= year <= MAXYEAR:
-        raise ValueError(f"Year is out of range: {year}")
+        raise ValueError(f"year must be in {MINYEAR}..{MAXYEAR}, not {year}")
 
     if not 0 < week < 53:
         out_of_range = True
@@ -502,7 +535,7 @@ def _isoweek_to_gregorian(year, week, day):
 def _check_tzname(name):
     if name is not None and not isinstance(name, str):
         raise TypeError("tzinfo.tzname() must return None or string, "
-                        "not '%s'" % type(name))
+                        f"not {type(name).__name__!r}")
 
 # name is the offset-producing method, "utcoffset" or "dst".
 # offset is what it returned.
@@ -515,24 +548,24 @@ def _check_utc_offset(name, offset):
     if offset is None:
         return
     if not isinstance(offset, timedelta):
-        raise TypeError("tzinfo.%s() must return None "
-                        "or timedelta, not '%s'" % (name, type(offset)))
+        raise TypeError(f"tzinfo.{name}() must return None "
+                        f"or timedelta, not {type(offset).__name__!r}")
     if not -timedelta(1) < offset < timedelta(1):
-        raise ValueError("%s()=%s, must be strictly between "
-                         "-timedelta(hours=24) and timedelta(hours=24)" %
-                         (name, offset))
+        raise ValueError("offset must be a timedelta "
+                         "strictly between -timedelta(hours=24) and "
+                         f"timedelta(hours=24), not {offset!r}")
 
 def _check_date_fields(year, month, day):
     year = _index(year)
     month = _index(month)
     day = _index(day)
     if not MINYEAR <= year <= MAXYEAR:
-        raise ValueError('year must be in %d..%d' % (MINYEAR, MAXYEAR), year)
+        raise ValueError(f"year must be in {MINYEAR}..{MAXYEAR}, not {year}")
     if not 1 <= month <= 12:
-        raise ValueError('month must be in 1..12', month)
+        raise ValueError(f"month must be in 1..12, not {month}")
     dim = _days_in_month(year, month)
     if not 1 <= day <= dim:
-        raise ValueError('day must be in 1..%d' % dim, day)
+        raise ValueError(f"day {day} must be in range 1..{dim} for month {month} in year {year}")
     return year, month, day
 
 def _check_time_fields(hour, minute, second, microsecond, fold):
@@ -541,24 +574,23 @@ def _check_time_fields(hour, minute, second, microsecond, fold):
     second = _index(second)
     microsecond = _index(microsecond)
     if not 0 <= hour <= 23:
-        raise ValueError('hour must be in 0..23', hour)
+        raise ValueError(f"hour must be in 0..23, not {hour}")
     if not 0 <= minute <= 59:
-        raise ValueError('minute must be in 0..59', minute)
+        raise ValueError(f"minute must be in 0..59, not {minute}")
     if not 0 <= second <= 59:
-        raise ValueError('second must be in 0..59', second)
+        raise ValueError(f"second must be in 0..59, not {second}")
     if not 0 <= microsecond <= 999999:
-        raise ValueError('microsecond must be in 0..999999', microsecond)
+        raise ValueError(f"microsecond must be in 0..999999, not {microsecond}")
     if fold not in (0, 1):
-        raise ValueError('fold must be either 0 or 1', fold)
+        raise ValueError(f"fold must be either 0 or 1, not {fold}")
     return hour, minute, second, microsecond, fold
 
 def _check_tzinfo_arg(tz):
     if tz is not None and not isinstance(tz, tzinfo):
-        raise TypeError("tzinfo argument must be None or of a tzinfo subclass")
-
-def _cmperror(x, y):
-    raise TypeError("can't compare '%s' to '%s'" % (
-                    type(x).__name__, type(y).__name__))
+        raise TypeError(
+            "tzinfo argument must be None or of a tzinfo subclass, "
+            f"not {type(tz).__name__!r}"
+        )
 
 def _divide_and_round(a, b):
     """divide a by b and round result to the nearest integer
@@ -612,7 +644,19 @@ class timedelta:
         # guide the C implementation; it's way more convoluted than speed-
         # ignoring auto-overflow-to-long idiomatic Python could be.
 
-        # XXX Check that all inputs are ints or floats.
+        for name, value in (
+            ("days", days),
+            ("seconds", seconds),
+            ("microseconds", microseconds),
+            ("milliseconds", milliseconds),
+            ("minutes", minutes),
+            ("hours", hours),
+            ("weeks", weeks)
+        ):
+            if not isinstance(value, (int, float)):
+                raise TypeError(
+                    f"unsupported type for timedelta {name} component: {type(value).__name__}"
+                )
 
         # Final values, all integer.
         # s and us fit in 32-bit signed ints; d isn't bounded.
@@ -713,9 +757,9 @@ class timedelta:
             args.append("microseconds=%d" % self._microseconds)
         if not args:
             args.append('0')
-        return "%s.%s(%s)" % (_get_class_module(self),
-                              self.__class__.__qualname__,
-                              ', '.join(args))
+        return "%s%s(%s)" % (_get_class_module(self),
+                             self.__class__.__qualname__,
+                             ', '.join(args))
 
     def __str__(self):
         mm, ss = divmod(self._seconds, 60)
@@ -912,6 +956,7 @@ class date:
     fromtimestamp()
     today()
     fromordinal()
+    strptime()
 
     Operators:
 
@@ -970,6 +1015,8 @@ class date:
     @classmethod
     def fromtimestamp(cls, t):
         "Construct a date from a POSIX timestamp (like time.time())."
+        if t is None:
+            raise TypeError("'NoneType' object cannot be interpreted as an integer")
         y, m, d, hh, mm, ss, weekday, jday, dst = _time.localtime(t)
         return cls(y, m, d)
 
@@ -992,8 +1039,12 @@ class date:
     @classmethod
     def fromisoformat(cls, date_string):
         """Construct a date from a string in ISO 8601 format."""
+
         if not isinstance(date_string, str):
-            raise TypeError('fromisoformat: argument must be str')
+            raise TypeError('Argument must be a str')
+
+        if not date_string.isascii():
+            raise ValueError('Argument must be an ASCII str')
 
         if len(date_string) not in (7, 8, 10):
             raise ValueError(f'Invalid isoformat string: {date_string!r}')
@@ -1010,6 +1061,12 @@ class date:
         This is the inverse of the date.isocalendar() function"""
         return cls(*_isoweek_to_gregorian(year, week, day))
 
+    @classmethod
+    def strptime(cls, date_string, format):
+        """Parse a date string according to the given format (like time.strptime())."""
+        import _strptime
+        return _strptime._strptime_datetime_date(cls, date_string, format)
+
     # Conversions to string
 
     def __repr__(self):
@@ -1019,11 +1076,11 @@ class date:
         >>> repr(d)
         'datetime.date(2010, 1, 1)'
         """
-        return "%s.%s(%d, %d, %d)" % (_get_class_module(self),
-                                      self.__class__.__qualname__,
-                                      self._year,
-                                      self._month,
-                                      self._day)
+        return "%s%s(%d, %d, %d)" % (_get_class_module(self),
+                                     self.__class__.__qualname__,
+                                     self._year,
+                                     self._month,
+                                     self._day)
     # XXX These shouldn't depend on time.localtime(), because that
     # clips the usable dates to [1970 .. 2038).  At least ctime() is
     # easily done without using strftime() -- that's better too because
@@ -1059,8 +1116,8 @@ class date:
         This is 'YYYY-MM-DD'.
 
         References:
-        - http://www.w3.org/TR/NOTE-datetime
-        - http://www.cl.cam.ac.uk/~mgk25/iso-time.html
+        - https://www.w3.org/TR/NOTE-datetime
+        - https://www.cl.cam.ac.uk/~mgk25/iso-time.html
         """
         return "%04d-%02d-%02d" % (self._year, self._month, self._day)
 
@@ -1108,35 +1165,38 @@ class date:
             day = self._day
         return type(self)(year, month, day)
 
+    __replace__ = replace
+
     # Comparisons of date objects with other.
 
     def __eq__(self, other):
-        if isinstance(other, date):
+        if isinstance(other, date) and not isinstance(other, datetime):
             return self._cmp(other) == 0
         return NotImplemented
 
     def __le__(self, other):
-        if isinstance(other, date):
+        if isinstance(other, date) and not isinstance(other, datetime):
             return self._cmp(other) <= 0
         return NotImplemented
 
     def __lt__(self, other):
-        if isinstance(other, date):
+        if isinstance(other, date) and not isinstance(other, datetime):
             return self._cmp(other) < 0
         return NotImplemented
 
     def __ge__(self, other):
-        if isinstance(other, date):
+        if isinstance(other, date) and not isinstance(other, datetime):
             return self._cmp(other) >= 0
         return NotImplemented
 
     def __gt__(self, other):
-        if isinstance(other, date):
+        if isinstance(other, date) and not isinstance(other, datetime):
             return self._cmp(other) > 0
         return NotImplemented
 
     def _cmp(self, other):
         assert isinstance(other, date)
+        assert not isinstance(other, datetime)
         y, m, d = self._year, self._month, self._day
         y2, m2, d2 = other._year, other._month, other._day
         return _cmp((y, m, d), (y2, m2, d2))
@@ -1191,7 +1251,7 @@ class date:
         The first week is 1; Monday is 1 ... Sunday is 7.
 
         ISO calendar algorithm taken from
-        http://www.phys.uu.nl/~vgent/calendar/isocalendar.htm
+        https://www.phys.uu.nl/~vgent/calendar/isocalendar.htm
         (used with permission)
         """
         year = self._year
@@ -1327,6 +1387,7 @@ class time:
     Constructors:
 
     __new__()
+    strptime()
 
     Operators:
 
@@ -1385,6 +1446,12 @@ class time:
         self._fold = fold
         return self
 
+    @classmethod
+    def strptime(cls, date_string, format):
+        """string, format -> new time parsed from a string (like time.strptime())."""
+        import _strptime
+        return _strptime._strptime_datetime_time(cls, date_string, format)
+
     # Read-only field accessors
     @property
     def hour(self):
@@ -1513,7 +1580,7 @@ class time:
             s = ", %d" % self._second
         else:
             s = ""
-        s= "%s.%s(%d, %d%s)" % (_get_class_module(self),
+        s = "%s%s(%d, %d%s)" % (_get_class_module(self),
                                 self.__class__.__qualname__,
                                 self._hour, self._minute, s)
         if self._tzinfo is not None:
@@ -1555,7 +1622,7 @@ class time:
         time_string = time_string.removeprefix('T')
 
         try:
-            return cls(*_parse_isoformat_time(time_string))
+            return cls(*_parse_isoformat_time(time_string)[0])
         except Exception:
             raise ValueError(f'Invalid isoformat string: {time_string!r}')
 
@@ -1633,6 +1700,8 @@ class time:
             fold = self._fold
         return type(self)(hour, minute, second, microsecond, tzinfo, fold=fold)
 
+    __replace__ = replace
+
     # Pickle support.
 
     def _getstate(self, protocol=3):
@@ -1680,7 +1749,7 @@ class datetime(date):
     The year, month and day arguments are required. tzinfo may be None, or an
     instance of a tzinfo subclass. The remaining arguments may be ints.
     """
-    __slots__ = date.__slots__ + time.__slots__
+    __slots__ = time.__slots__
 
     def __new__(cls, year, month=None, day=None, hour=0, minute=0, second=0,
                 microsecond=0, tzinfo=None, *, fold=0):
@@ -1867,10 +1936,27 @@ class datetime(date):
 
         if tstr:
             try:
-                time_components = _parse_isoformat_time(tstr)
+                time_components, became_next_day, error_from_components = _parse_isoformat_time(tstr)
             except ValueError:
                 raise ValueError(
                     f'Invalid isoformat string: {date_string!r}') from None
+            else:
+                if error_from_components:
+                    raise ValueError("minute, second, and microsecond must be 0 when hour is 24")
+
+                if became_next_day:
+                    year, month, day = date_components
+                    # Only wrap day/month when it was previously valid
+                    if month <= 12 and day <= (days_in_month := _days_in_month(year, month)):
+                        # Calculate midnight of the next day
+                        day += 1
+                        if day > days_in_month:
+                            day = 1
+                            month += 1
+                            if month > 12:
+                                month = 1
+                                year += 1
+                        date_components = [year, month, day]
         else:
             time_components = [0, 0, 0, 0, None]
 
@@ -1979,6 +2065,8 @@ class datetime(date):
         return type(self)(year, month, day, hour, minute, second,
                           microsecond, tzinfo, fold=fold)
 
+    __replace__ = replace
+
     def _local_timezone(self):
         if self.tzinfo is None:
             ts = self._mktime()
@@ -2040,7 +2128,7 @@ class datetime(date):
         By default, the fractional part is omitted if self.microsecond == 0.
 
         If self.tzinfo is not None, the UTC offset is also attached, giving
-        giving a full format of 'YYYY-MM-DD HH:MM:SS.mmmmmm+HH:MM'.
+        a full format of 'YYYY-MM-DD HH:MM:SS.mmmmmm+HH:MM'.
 
         Optional argument sep specifies the separator between date and
         time, default 'T'.
@@ -2068,9 +2156,9 @@ class datetime(date):
             del L[-1]
         if L[-1] == 0:
             del L[-1]
-        s = "%s.%s(%s)" % (_get_class_module(self),
-                           self.__class__.__qualname__,
-                           ", ".join(map(str, L)))
+        s = "%s%s(%s)" % (_get_class_module(self),
+                          self.__class__.__qualname__,
+                          ", ".join(map(str, L)))
         if self._tzinfo is not None:
             assert s[-1:] == ")"
             s = s[:-1] + ", tzinfo=%r" % self._tzinfo + ")"
@@ -2087,7 +2175,7 @@ class datetime(date):
     def strptime(cls, date_string, format):
         'string, format -> new datetime parsed from a string (like time.strptime()).'
         import _strptime
-        return _strptime._strptime_datetime(cls, date_string, format)
+        return _strptime._strptime_datetime_datetime(cls, date_string, format)
 
     def utcoffset(self):
         """Return the timezone offset as timedelta positive east of UTC (negative west of
@@ -2131,42 +2219,32 @@ class datetime(date):
     def __eq__(self, other):
         if isinstance(other, datetime):
             return self._cmp(other, allow_mixed=True) == 0
-        elif not isinstance(other, date):
-            return NotImplemented
         else:
-            return False
+            return NotImplemented
 
     def __le__(self, other):
         if isinstance(other, datetime):
             return self._cmp(other) <= 0
-        elif not isinstance(other, date):
-            return NotImplemented
         else:
-            _cmperror(self, other)
+            return NotImplemented
 
     def __lt__(self, other):
         if isinstance(other, datetime):
             return self._cmp(other) < 0
-        elif not isinstance(other, date):
-            return NotImplemented
         else:
-            _cmperror(self, other)
+            return NotImplemented
 
     def __ge__(self, other):
         if isinstance(other, datetime):
             return self._cmp(other) >= 0
-        elif not isinstance(other, date):
-            return NotImplemented
         else:
-            _cmperror(self, other)
+            return NotImplemented
 
     def __gt__(self, other):
         if isinstance(other, datetime):
             return self._cmp(other) > 0
-        elif not isinstance(other, date):
-            return NotImplemented
         else:
-            _cmperror(self, other)
+            return NotImplemented
 
     def _cmp(self, other, allow_mixed=False):
         assert isinstance(other, datetime)
@@ -2311,7 +2389,6 @@ datetime.resolution = timedelta(microseconds=1)
 
 def _isoweek1monday(year):
     # Helper to calculate the day number of the Monday starting week 1
-    # XXX This could be done more efficiently
     THURSDAY = 3
     firstday = _ymd2ord(year, 1, 1)
     firstweekday = (firstday + 6) % 7  # See weekday() above
@@ -2338,9 +2415,12 @@ class timezone(tzinfo):
         if not cls._minoffset <= offset <= cls._maxoffset:
             raise ValueError("offset must be a timedelta "
                              "strictly between -timedelta(hours=24) and "
-                             "timedelta(hours=24).")
+                             f"timedelta(hours=24), not {offset!r}")
         return cls._create(offset, name)
 
+    def __init_subclass__(cls):
+        raise TypeError("type 'datetime.timezone' is not an acceptable base type")
+
     @classmethod
     def _create(cls, offset, name=None):
         self = tzinfo.__new__(cls)
@@ -2375,12 +2455,12 @@ class timezone(tzinfo):
         if self is self.utc:
             return 'datetime.timezone.utc'
         if self._name is None:
-            return "%s.%s(%r)" % (_get_class_module(self),
-                                  self.__class__.__qualname__,
-                                  self._offset)
-        return "%s.%s(%r, %r)" % (_get_class_module(self),
-                                  self.__class__.__qualname__,
-                                  self._offset, self._name)
+            return "%s%s(%r)" % (_get_class_module(self),
+                                 self.__class__.__qualname__,
+                                 self._offset)
+        return "%s%s(%r, %r)" % (_get_class_module(self),
+                                 self.__class__.__qualname__,
+                                 self._offset, self._name)
 
     def __str__(self):
         return self.tzname(None)

Lib/_pydecimal.py

@@ -13,104 +13,7 @@
 # bug) and will be backported.  At this point the spec is stabilizing
 # and the updates are becoming fewer, smaller, and less significant.
 
-"""
-This is an implementation of decimal floating point arithmetic based on
-the General Decimal Arithmetic Specification:
-
-    http://speleotrove.com/decimal/decarith.html
-
-and IEEE standard 854-1987:
-
-    http://en.wikipedia.org/wiki/IEEE_854-1987
-
-Decimal floating point has finite precision with arbitrarily large bounds.
-
-The purpose of this module is to support arithmetic using familiar
-"schoolhouse" rules and to avoid some of the tricky representation
-issues associated with binary floating point.  The package is especially
-useful for financial applications or for contexts where users have
-expectations that are at odds with binary floating point (for instance,
-in binary floating point, 1.00 % 0.1 gives 0.09999999999999995 instead
-of 0.0; Decimal('1.00') % Decimal('0.1') returns the expected
-Decimal('0.00')).
-
-Here are some examples of using the decimal module:
-
->>> from decimal import *
->>> setcontext(ExtendedContext)
->>> Decimal(0)
-Decimal('0')
->>> Decimal('1')
-Decimal('1')
->>> Decimal('-.0123')
-Decimal('-0.0123')
->>> Decimal(123456)
-Decimal('123456')
->>> Decimal('123.45e12345678')
-Decimal('1.2345E+12345680')
->>> Decimal('1.33') + Decimal('1.27')
-Decimal('2.60')
->>> Decimal('12.34') + Decimal('3.87') - Decimal('18.41')
-Decimal('-2.20')
->>> dig = Decimal(1)
->>> print(dig / Decimal(3))
-0.333333333
->>> getcontext().prec = 18
->>> print(dig / Decimal(3))
-0.333333333333333333
->>> print(dig.sqrt())
-1
->>> print(Decimal(3).sqrt())
-1.73205080756887729
->>> print(Decimal(3) ** 123)
-4.85192780976896427E+58
->>> inf = Decimal(1) / Decimal(0)
->>> print(inf)
-Infinity
->>> neginf = Decimal(-1) / Decimal(0)
->>> print(neginf)
--Infinity
->>> print(neginf + inf)
-NaN
->>> print(neginf * inf)
--Infinity
->>> print(dig / 0)
-Infinity
->>> getcontext().traps[DivisionByZero] = 1
->>> print(dig / 0)
-Traceback (most recent call last):
-  ...
-  ...
-  ...
-decimal.DivisionByZero: x / 0
->>> c = Context()
->>> c.traps[InvalidOperation] = 0
->>> print(c.flags[InvalidOperation])
-0
->>> c.divide(Decimal(0), Decimal(0))
-Decimal('NaN')
->>> c.traps[InvalidOperation] = 1
->>> print(c.flags[InvalidOperation])
-1
->>> c.flags[InvalidOperation] = 0
->>> print(c.flags[InvalidOperation])
-0
->>> print(c.divide(Decimal(0), Decimal(0)))
-Traceback (most recent call last):
-  ...
-  ...
-  ...
-decimal.InvalidOperation: 0 / 0
->>> print(c.flags[InvalidOperation])
-1
->>> c.flags[InvalidOperation] = 0
->>> c.traps[InvalidOperation] = 0
->>> print(c.divide(Decimal(0), Decimal(0)))
-NaN
->>> print(c.flags[InvalidOperation])
-1
->>>
-"""
+"""Python decimal arithmetic module"""
 
 __all__ = [
     # Two major classes
@@ -135,13 +38,16 @@ __all__ = [
     'ROUND_FLOOR', 'ROUND_UP', 'ROUND_HALF_DOWN', 'ROUND_05UP',
 
     # Functions for manipulating contexts
-    'setcontext', 'getcontext', 'localcontext',
+    'setcontext', 'getcontext', 'localcontext', 'IEEEContext',
 
     # Limits for the C version for compatibility
-    'MAX_PREC',  'MAX_EMAX', 'MIN_EMIN', 'MIN_ETINY',
+    'MAX_PREC',  'MAX_EMAX', 'MIN_EMIN', 'MIN_ETINY', 'IEEE_CONTEXT_MAX_BITS',
 
-    # C version: compile time choice that enables the thread local context
-    'HAVE_THREADS'
+    # C version: compile time choice that enables the thread local context (deprecated, now always true)
+    'HAVE_THREADS',
+
+    # C version: compile time choice that enables the coroutine local context
+    'HAVE_CONTEXTVAR'
 ]
 
 __xname__ = __name__    # sys.modules lookup (--without-threads)
@@ -156,7 +62,7 @@ import sys
 
 try:
     from collections import namedtuple as _namedtuple
-    DecimalTuple = _namedtuple('DecimalTuple', 'sign digits exponent')
+    DecimalTuple = _namedtuple('DecimalTuple', 'sign digits exponent', module='decimal')
 except ImportError:
     DecimalTuple = lambda *args: args
 
@@ -172,14 +78,17 @@ ROUND_05UP = 'ROUND_05UP'
 
 # Compatibility with the C version
 HAVE_THREADS = True
+HAVE_CONTEXTVAR = True
 if sys.maxsize == 2**63-1:
     MAX_PREC = 999999999999999999
     MAX_EMAX = 999999999999999999
     MIN_EMIN = -999999999999999999
+    IEEE_CONTEXT_MAX_BITS = 512
 else:
     MAX_PREC = 425000000
     MAX_EMAX = 425000000
     MIN_EMIN = -425000000
+    IEEE_CONTEXT_MAX_BITS = 256
 
 MIN_ETINY = MIN_EMIN - (MAX_PREC-1)
 
@@ -190,7 +99,7 @@ class DecimalException(ArithmeticError):
 
     Used exceptions derive from this.
     If an exception derives from another exception besides this (such as
-    Underflow (Inexact, Rounded, Subnormal) that indicates that it is only
+    Underflow (Inexact, Rounded, Subnormal)) that indicates that it is only
     called if the others are present.  This isn't actually used for
     anything, though.
 
@@ -238,7 +147,7 @@ class InvalidOperation(DecimalException):
     x ** (+-)INF
     An operand is invalid
 
-    The result of the operation after these is a quiet positive NaN,
+    The result of the operation after this is a quiet positive NaN,
     except when the cause is a signaling NaN, in which case the result is
     also a quiet NaN, but with the original sign, and an optional
     diagnostic information.
@@ -431,82 +340,40 @@ _rounding_modes = (ROUND_DOWN, ROUND_HALF_UP, ROUND_HALF_EVEN, ROUND_CEILING,
 ##### Context Functions ##################################################
 
 # The getcontext() and setcontext() function manage access to a thread-local
-# current context.  Py2.4 offers direct support for thread locals.  If that
-# is not available, use threading.current_thread() which is slower but will
-# work for older Pythons.  If threads are not part of the build, create a
-# mock threading object with threading.local() returning the module namespace.
+# current context.
 
-try:
-    import threading
-except ImportError:
-    # Python was compiled without threads; create a mock object instead
-    class MockThreading(object):
-        def local(self, sys=sys):
-            return sys.modules[__xname__]
-    threading = MockThreading()
-    del MockThreading
+import contextvars
 
-try:
-    threading.local
+_current_context_var = contextvars.ContextVar('decimal_context')
 
-except AttributeError:
+_context_attributes = frozenset(
+    ['prec', 'Emin', 'Emax', 'capitals', 'clamp', 'rounding', 'flags', 'traps']
+)
 
-    # To fix reloading, force it to create a new context
-    # Old contexts have different exceptions in their dicts, making problems.
-    if hasattr(threading.current_thread(), '__decimal_context__'):
-        del threading.current_thread().__decimal_context__
+def getcontext():
+    """Returns this thread's context.
 
-    def setcontext(context):
-        """Set this thread's context to context."""
-        if context in (DefaultContext, BasicContext, ExtendedContext):
-            context = context.copy()
-            context.clear_flags()
-        threading.current_thread().__decimal_context__ = context
+    If this thread does not yet have a context, returns
+    a new context and sets this thread's context.
+    New contexts are copies of DefaultContext.
+    """
+    try:
+        return _current_context_var.get()
+    except LookupError:
+        context = Context()
+        _current_context_var.set(context)
+        return context
 
-    def getcontext():
-        """Returns this thread's context.
+def setcontext(context):
+    """Set this thread's context to context."""
+    if context in (DefaultContext, BasicContext, ExtendedContext):
+        context = context.copy()
+        context.clear_flags()
+    _current_context_var.set(context)
 
-        If this thread does not yet have a context, returns
-        a new context and sets this thread's context.
-        New contexts are copies of DefaultContext.
-        """
-        try:
-            return threading.current_thread().__decimal_context__
-        except AttributeError:
-            context = Context()
-            threading.current_thread().__decimal_context__ = context
-            return context
+del contextvars        # Don't contaminate the namespace
 
-else:
-
-    local = threading.local()
-    if hasattr(local, '__decimal_context__'):
-        del local.__decimal_context__
-
-    def getcontext(_local=local):
-        """Returns this thread's context.
-
-        If this thread does not yet have a context, returns
-        a new context and sets this thread's context.
-        New contexts are copies of DefaultContext.
-        """
-        try:
-            return _local.__decimal_context__
-        except AttributeError:
-            context = Context()
-            _local.__decimal_context__ = context
-            return context
-
-    def setcontext(context, _local=local):
-        """Set this thread's context to context."""
-        if context in (DefaultContext, BasicContext, ExtendedContext):
-            context = context.copy()
-            context.clear_flags()
-        _local.__decimal_context__ = context
-
-    del threading, local        # Don't contaminate the namespace
-
-def localcontext(ctx=None):
+def localcontext(ctx=None, **kwargs):
     """Return a context manager for a copy of the supplied context
 
     Uses a copy of the current context if no context is specified
@@ -542,8 +409,35 @@ def localcontext(ctx=None):
     >>> print(getcontext().prec)
     28
     """
-    if ctx is None: ctx = getcontext()
-    return _ContextManager(ctx)
+    if ctx is None:
+        ctx = getcontext()
+    ctx_manager = _ContextManager(ctx)
+    for key, value in kwargs.items():
+        if key not in _context_attributes:
+            raise TypeError(f"'{key}' is an invalid keyword argument for this function")
+        setattr(ctx_manager.new_context, key, value)
+    return ctx_manager
+
+
+def IEEEContext(bits, /):
+    """
+    Return a context object initialized to the proper values for one of the
+    IEEE interchange formats.  The argument must be a multiple of 32 and less
+    than IEEE_CONTEXT_MAX_BITS.
+    """
+    if bits <= 0 or bits > IEEE_CONTEXT_MAX_BITS or bits % 32:
+        raise ValueError("argument must be a multiple of 32, "
+                         f"with a maximum of {IEEE_CONTEXT_MAX_BITS}")
+
+    ctx = Context()
+    ctx.prec = 9 * (bits//32) - 2
+    ctx.Emax = 3 * (1 << (bits//16 + 3))
+    ctx.Emin = 1 - ctx.Emax
+    ctx.rounding = ROUND_HALF_EVEN
+    ctx.clamp = 1
+    ctx.traps = dict.fromkeys(_signals, False)
+
+    return ctx
 
 
 ##### Decimal class #######################################################
@@ -553,7 +447,7 @@ def localcontext(ctx=None):
 # numbers.py for more detail.
 
 class Decimal(object):
-    """Floating point class for decimal arithmetic."""
+    """Floating-point class for decimal arithmetic."""
 
     __slots__ = ('_exp','_int','_sign', '_is_special')
     # Generally, the value of the Decimal instance is given by
@@ -711,6 +605,21 @@ class Decimal(object):
 
         raise TypeError("Cannot convert %r to Decimal" % value)
 
+    @classmethod
+    def from_number(cls, number):
+        """Converts a real number to a decimal number, exactly.
+
+        >>> Decimal.from_number(314)              # int
+        Decimal('314')
+        >>> Decimal.from_number(0.1)              # float
+        Decimal('0.1000000000000000055511151231257827021181583404541015625')
+        >>> Decimal.from_number(Decimal('3.14'))  # another decimal instance
+        Decimal('3.14')
+        """
+        if isinstance(number, (int, Decimal, float)):
+            return cls(number)
+        raise TypeError("Cannot convert %r to Decimal" % number)
+
     @classmethod
     def from_float(cls, f):
         """Converts a float to a decimal number, exactly.
@@ -993,7 +902,7 @@ class Decimal(object):
             if self.is_snan():
                 raise TypeError('Cannot hash a signaling NaN value.')
             elif self.is_nan():
-                return _PyHASH_NAN
+                return object.__hash__(self)
             else:
                 if self._sign:
                     return -_PyHASH_INF
@@ -1674,13 +1583,13 @@ class Decimal(object):
 
     __trunc__ = __int__
 
+    @property
     def real(self):
         return self
-    real = property(real)
 
+    @property
     def imag(self):
         return Decimal(0)
-    imag = property(imag)
 
     def conjugate(self):
         return self
@@ -2260,10 +2169,16 @@ class Decimal(object):
             else:
                 return None
 
-            if xc >= 10**p:
+            # An exact power of 10 is representable, but can convert to a
+            # string of any length. But an exact power of 10 shouldn't be
+            # possible at this point.
+            assert xc > 1, self
+            assert xc % 10 != 0, self
+            strxc = str(xc)
+            if len(strxc) > p:
                 return None
             xe = -e-xe
-            return _dec_from_triple(0, str(xc), xe)
+            return _dec_from_triple(0, strxc, xe)
 
         # now y is positive; find m and n such that y = m/n
         if ye >= 0:
@@ -2272,7 +2187,7 @@ class Decimal(object):
             if xe != 0 and len(str(abs(yc*xe))) <= -ye:
                 return None
             xc_bits = _nbits(xc)
-            if xc != 1 and len(str(abs(yc)*xc_bits)) <= -ye:
+            if len(str(abs(yc)*xc_bits)) <= -ye:
                 return None
             m, n = yc, 10**(-ye)
             while m % 2 == n % 2 == 0:
@@ -2285,7 +2200,7 @@ class Decimal(object):
         # compute nth root of xc*10**xe
         if n > 1:
             # if 1 < xc < 2**n then xc isn't an nth power
-            if xc != 1 and xc_bits <= n:
+            if xc_bits <= n:
                 return None
 
             xe, rem = divmod(xe, n)
@@ -2313,13 +2228,18 @@ class Decimal(object):
             return None
         xc = xc**m
         xe *= m
-        if xc > 10**p:
+        # An exact power of 10 is representable, but can convert to a string
+        # of any length. But an exact power of 10 shouldn't be possible at
+        # this point.
+        assert xc > 1, self
+        assert xc % 10 != 0, self
+        str_xc = str(xc)
+        if len(str_xc) > p:
             return None
 
         # by this point the result *is* exactly representable
         # adjust the exponent to get as close as possible to the ideal
         # exponent, if necessary
-        str_xc = str(xc)
         if other._isinteger() and other._sign == 0:
             ideal_exponent = self._exp*int(other)
             zeros = min(xe-ideal_exponent, p-len(str_xc))
@@ -2543,12 +2463,12 @@ class Decimal(object):
 
         return ans
 
-    def __rpow__(self, other, context=None):
+    def __rpow__(self, other, modulo=None, context=None):
         """Swaps self/other and returns __pow__."""
         other = _convert_other(other)
         if other is NotImplemented:
             return other
-        return other.__pow__(self, context=context)
+        return other.__pow__(self, modulo, context=context)
 
     def normalize(self, context=None):
         """Normalize- strip trailing 0s, change anything equal to 0 to 0e0"""
@@ -3420,7 +3340,10 @@ class Decimal(object):
         return opa, opb
 
     def logical_and(self, other, context=None):
-        """Applies an 'and' operation between self and other's digits."""
+        """Applies an 'and' operation between self and other's digits.
+
+        Both self and other must be logical numbers.
+        """
         if context is None:
             context = getcontext()
 
@@ -3437,14 +3360,20 @@ class Decimal(object):
         return _dec_from_triple(0, result.lstrip('0') or '0', 0)
 
     def logical_invert(self, context=None):
-        """Invert all its digits."""
+        """Invert all its digits.
+
+        The self must be logical number.
+        """
         if context is None:
             context = getcontext()
         return self.logical_xor(_dec_from_triple(0,'1'*context.prec,0),
                                 context)
 
     def logical_or(self, other, context=None):
-        """Applies an 'or' operation between self and other's digits."""
+        """Applies an 'or' operation between self and other's digits.
+
+        Both self and other must be logical numbers.
+        """
         if context is None:
             context = getcontext()
 
@@ -3461,7 +3390,10 @@ class Decimal(object):
         return _dec_from_triple(0, result.lstrip('0') or '0', 0)
 
     def logical_xor(self, other, context=None):
-        """Applies an 'xor' operation between self and other's digits."""
+        """Applies an 'xor' operation between self and other's digits.
+
+        Both self and other must be logical numbers.
+        """
         if context is None:
             context = getcontext()
 
@@ -3837,6 +3769,10 @@ class Decimal(object):
         # represented in fixed point; rescale them to 0e0.
         if not self and self._exp > 0 and spec['type'] in 'fF%':
             self = self._rescale(0, rounding)
+        if not self and spec['no_neg_0'] and self._sign:
+            adjusted_sign = 0
+        else:
+            adjusted_sign = self._sign
 
         # figure out placement of the decimal point
         leftdigits = self._exp + len(self._int)
@@ -3867,7 +3803,7 @@ class Decimal(object):
 
         # done with the decimal-specific stuff;  hand over the rest
         # of the formatting to the _format_number function
-        return _format_number(self._sign, intpart, fracpart, exp, spec)
+        return _format_number(adjusted_sign, intpart, fracpart, exp, spec)
 
 def _dec_from_triple(sign, coefficient, exponent, special=False):
     """Create a decimal instance directly, without any validation,
@@ -5677,8 +5613,6 @@ class _WorkRep(object):
     def __repr__(self):
         return "(%r, %r, %r)" % (self.sign, self.int, self.exp)
 
-    __str__ = __repr__
-
 
 
 def _normalize(op1, op2, prec = 0):
@@ -6174,7 +6108,7 @@ _parser = re.compile(r"""        # A numeric string consists of:
         (?P<diag>\d*)            # with (possibly empty) diagnostic info.
     )
 #    \s*
-    \Z
+    \z
 """, re.VERBOSE | re.IGNORECASE).match
 
 _all_zeros = re.compile('0*$').match
@@ -6187,7 +6121,7 @@ _exact_half = re.compile('50*$').match
 #
 # A format specifier for Decimal looks like:
 #
-#   [[fill]align][sign][#][0][minimumwidth][,][.precision][type]
+#   [[fill]align][sign][z][#][0][minimumwidth][,][.precision][type]
 
 _parse_format_specifier_regex = re.compile(r"""\A
 (?:
@@ -6195,13 +6129,18 @@ _parse_format_specifier_regex = re.compile(r"""\A
    (?P<align>[<>=^])
 )?
 (?P<sign>[-+ ])?
+(?P<no_neg_0>z)?
 (?P<alt>\#)?
 (?P<zeropad>0)?
-(?P<minimumwidth>(?!0)\d+)?
-(?P<thousands_sep>,)?
-(?:\.(?P<precision>0|(?!0)\d+))?
+(?P<minimumwidth>\d+)?
+(?P<thousands_sep>[,_])?
+(?:\.
+    (?=[\d,_])  # lookahead for digit or separator
+    (?P<precision>\d+)?
+    (?P<frac_separators>[,_])?
+)?
 (?P<type>[eEfFgGn%])?
-\Z
+\z
 """, re.VERBOSE|re.DOTALL)
 
 del re
@@ -6292,6 +6231,9 @@ def _parse_format_specifier(format_spec, _localeconv=None):
         format_dict['grouping'] = [3, 0]
         format_dict['decimal_point'] = '.'
 
+    if format_dict['frac_separators'] is None:
+        format_dict['frac_separators'] = ''
+
     return format_dict
 
 def _format_align(sign, body, spec):
@@ -6411,6 +6353,11 @@ def _format_number(is_negative, intpart, fracpart, exp, spec):
 
     sign = _format_sign(is_negative, spec)
 
+    frac_sep = spec['frac_separators']
+    if fracpart and frac_sep:
+        fracpart = frac_sep.join(fracpart[pos:pos + 3]
+                                 for pos in range(0, len(fracpart), 3))
+
     if fracpart or spec['alt']:
         fracpart = spec['decimal_point'] + fracpart
 

Lib/_pyio.py

@@ -16,15 +16,16 @@ else:
     _setmode = None
 
 import io
-from io import (__all__, SEEK_SET, SEEK_CUR, SEEK_END)
+from io import (__all__, SEEK_SET, SEEK_CUR, SEEK_END, Reader, Writer)  # noqa: F401
 
 valid_seek_flags = {0, 1, 2}  # Hardwired values
 if hasattr(os, 'SEEK_HOLE') :
     valid_seek_flags.add(os.SEEK_HOLE)
     valid_seek_flags.add(os.SEEK_DATA)
 
-# open() uses st_blksize whenever we can
-DEFAULT_BUFFER_SIZE = 8 * 1024  # bytes
+# open() uses max(min(blocksize, 8 MiB), DEFAULT_BUFFER_SIZE)
+# when the device block size is available.
+DEFAULT_BUFFER_SIZE = 128 * 1024  # bytes
 
 # NOTE: Base classes defined here are registered with the "official" ABCs
 # defined in io.py. We don't use real inheritance though, because we don't want
@@ -33,11 +34,8 @@ DEFAULT_BUFFER_SIZE = 8 * 1024  # bytes
 # Rebind for compatibility
 BlockingIOError = BlockingIOError
 
-# Does io.IOBase finalizer log the exception if the close() method fails?
-# The exception is ignored silently by default in release build.
-_IOBASE_EMITS_UNRAISABLE = (hasattr(sys, "gettotalrefcount") or sys.flags.dev_mode)
 # Does open() check its 'errors' argument?
-_CHECK_ERRORS = _IOBASE_EMITS_UNRAISABLE
+_CHECK_ERRORS = (hasattr(sys, "gettotalrefcount") or sys.flags.dev_mode)
 
 
 def text_encoding(encoding, stacklevel=2):
@@ -126,10 +124,10 @@ def open(file, mode="r", buffering=-1, encoding=None, errors=None,
     the size of a fixed-size chunk buffer.  When no buffering argument is
     given, the default buffering policy works as follows:
 
-    * Binary files are buffered in fixed-size chunks; the size of the buffer
-      is chosen using a heuristic trying to determine the underlying device's
-      "block size" and falling back on `io.DEFAULT_BUFFER_SIZE`.
-      On many systems, the buffer will typically be 4096 or 8192 bytes long.
+   * Binary files are buffered in fixed-size chunks; the size of the buffer
+     is max(min(blocksize, 8 MiB), DEFAULT_BUFFER_SIZE)
+     when the device block size is available.
+     On most systems, the buffer will typically be 128 kilobytes long.
 
     * "Interactive" text files (files for which isatty() returns True)
       use line buffering.  Other text files use the policy described above
@@ -241,18 +239,11 @@ def open(file, mode="r", buffering=-1, encoding=None, errors=None,
     result = raw
     try:
         line_buffering = False
-        if buffering == 1 or buffering < 0 and raw.isatty():
+        if buffering == 1 or buffering < 0 and raw._isatty_open_only():
             buffering = -1
             line_buffering = True
         if buffering < 0:
-            buffering = DEFAULT_BUFFER_SIZE
-            try:
-                bs = os.fstat(raw.fileno()).st_blksize
-            except (OSError, AttributeError):
-                pass
-            else:
-                if bs > 1:
-                    buffering = bs
+            buffering = max(min(raw._blksize, 8192 * 1024), DEFAULT_BUFFER_SIZE)
         if buffering < 0:
             raise ValueError("invalid buffering size")
         if buffering == 0:
@@ -416,18 +407,12 @@ class IOBase(metaclass=abc.ABCMeta):
         if closed:
             return
 
-        if _IOBASE_EMITS_UNRAISABLE:
-            self.close()
-        else:
-            # The try/except block is in case this is called at program
-            # exit time, when it's possible that globals have already been
-            # deleted, and then the close() call might fail.  Since
-            # there's nothing we can do about such failures and they annoy
-            # the end users, we suppress the traceback.
-            try:
-                self.close()
-            except:
-                pass
+        if dealloc_warn := getattr(self, "_dealloc_warn", None):
+            dealloc_warn(self)
+
+        # If close() fails, the caller logs the exception with
+        # sys.unraisablehook. close() must be called at the end at __del__().
+        self.close()
 
     ### Inquiries ###
 
@@ -632,16 +617,15 @@ class RawIOBase(IOBase):
         n = self.readinto(b)
         if n is None:
             return None
+        if n < 0 or n > len(b):
+            raise ValueError(f"readinto returned {n} outside buffer size {len(b)}")
         del b[n:]
         return bytes(b)
 
     def readall(self):
         """Read until EOF, using multiple read() call."""
         res = bytearray()
-        while True:
-            data = self.read(DEFAULT_BUFFER_SIZE)
-            if not data:
-                break
+        while data := self.read(DEFAULT_BUFFER_SIZE):
             res += data
         if res:
             return bytes(res)
@@ -666,8 +650,6 @@ class RawIOBase(IOBase):
         self._unsupported("write")
 
 io.RawIOBase.register(RawIOBase)
-from _io import FileIO
-RawIOBase.register(FileIO)
 
 
 class BufferedIOBase(IOBase):
@@ -874,6 +856,10 @@ class _BufferedIOMixin(BufferedIOBase):
         else:
             return "<{}.{} name={!r}>".format(modname, clsname, name)
 
+    def _dealloc_warn(self, source):
+        if dealloc_warn := getattr(self.raw, "_dealloc_warn", None):
+            dealloc_warn(source)
+
     ### Lower-level APIs ###
 
     def fileno(self):
@@ -949,22 +935,22 @@ class BytesIO(BufferedIOBase):
         return self.read(size)
 
     def write(self, b):
-        if self.closed:
-            raise ValueError("write to closed file")
         if isinstance(b, str):
             raise TypeError("can't write str to binary stream")
         with memoryview(b) as view:
+            if self.closed:
+                raise ValueError("write to closed file")
+
             n = view.nbytes  # Size of any bytes-like object
-        if n == 0:
-            return 0
-        pos = self._pos
-        if pos > len(self._buffer):
-            # Inserts null bytes between the current end of the file
-            # and the new write position.
-            padding = b'\x00' * (pos - len(self._buffer))
-            self._buffer += padding
-        self._buffer[pos:pos + n] = b
-        self._pos += n
+            if n == 0:
+                return 0
+
+            pos = self._pos
+            if pos > len(self._buffer):
+                # Pad buffer to pos with null bytes.
+                self._buffer.resize(pos)
+            self._buffer[pos:pos + n] = view
+            self._pos += n
         return n
 
     def seek(self, pos, whence=0):
@@ -1478,6 +1464,17 @@ class BufferedRandom(BufferedWriter, BufferedReader):
         return BufferedWriter.write(self, b)
 
 
+def _new_buffersize(bytes_read):
+    # Parallels _io/fileio.c new_buffersize
+    if bytes_read > 65536:
+        addend = bytes_read >> 3
+    else:
+        addend = 256 + bytes_read
+    if addend < DEFAULT_BUFFER_SIZE:
+        addend = DEFAULT_BUFFER_SIZE
+    return bytes_read + addend
+
+
 class FileIO(RawIOBase):
     _fd = -1
     _created = False
@@ -1502,6 +1499,7 @@ class FileIO(RawIOBase):
         """
         if self._fd >= 0:
             # Have to close the existing file first.
+            self._stat_atopen = None
             try:
                 if self._closefd:
                     os.close(self._fd)
@@ -1511,6 +1509,11 @@ class FileIO(RawIOBase):
         if isinstance(file, float):
             raise TypeError('integer argument expected, got float')
         if isinstance(file, int):
+            if isinstance(file, bool):
+                import warnings
+                warnings.warn("bool is used as a file descriptor",
+                              RuntimeWarning, stacklevel=2)
+                file = int(file)
             fd = file
             if fd < 0:
                 raise ValueError('negative file descriptor')
@@ -1569,24 +1572,22 @@ class FileIO(RawIOBase):
                     if not isinstance(fd, int):
                         raise TypeError('expected integer from opener')
                     if fd < 0:
-                        raise OSError('Negative file descriptor')
+                        # bpo-27066: Raise a ValueError for bad value.
+                        raise ValueError(f'opener returned {fd}')
                 owned_fd = fd
                 if not noinherit_flag:
                     os.set_inheritable(fd, False)
 
             self._closefd = closefd
-            fdfstat = os.fstat(fd)
+            self._stat_atopen = os.fstat(fd)
             try:
-                if stat.S_ISDIR(fdfstat.st_mode):
+                if stat.S_ISDIR(self._stat_atopen.st_mode):
                     raise IsADirectoryError(errno.EISDIR,
                                             os.strerror(errno.EISDIR), file)
             except AttributeError:
                 # Ignore the AttributeError if stat.S_ISDIR or errno.EISDIR
                 # don't exist.
                 pass
-            self._blksize = getattr(fdfstat, 'st_blksize', 0)
-            if self._blksize <= 1:
-                self._blksize = DEFAULT_BUFFER_SIZE
 
             if _setmode:
                 # don't translate newlines (\r\n <=> \n)
@@ -1603,17 +1604,17 @@ class FileIO(RawIOBase):
                     if e.errno != errno.ESPIPE:
                         raise
         except:
+            self._stat_atopen = None
             if owned_fd is not None:
                 os.close(owned_fd)
             raise
         self._fd = fd
 
-    def __del__(self):
+    def _dealloc_warn(self, source):
         if self._fd >= 0 and self._closefd and not self.closed:
             import warnings
-            warnings.warn('unclosed file %r' % (self,), ResourceWarning,
+            warnings.warn(f'unclosed file {source!r}', ResourceWarning,
                           stacklevel=2, source=self)
-            self.close()
 
     def __getstate__(self):
         raise TypeError(f"cannot pickle {self.__class__.__name__!r} object")
@@ -1632,6 +1633,17 @@ class FileIO(RawIOBase):
             return ('<%s name=%r mode=%r closefd=%r>' %
                     (class_name, name, self.mode, self._closefd))
 
+    @property
+    def _blksize(self):
+        if self._stat_atopen is None:
+            return DEFAULT_BUFFER_SIZE
+
+        blksize = getattr(self._stat_atopen, "st_blksize", 0)
+        # WASI sets blsize to 0
+        if not blksize:
+            return DEFAULT_BUFFER_SIZE
+        return blksize
+
     def _checkReadable(self):
         if not self._readable:
             raise UnsupportedOperation('File not open for reading')
@@ -1643,7 +1655,13 @@ class FileIO(RawIOBase):
     def read(self, size=None):
         """Read at most size bytes, returned as bytes.
 
-        Only makes one system call, so less data may be returned than requested
+        If size is less than 0, read all bytes in the file making
+        multiple read calls. See ``FileIO.readall``.
+
+        Attempts to make only one system call, retrying only per
+        PEP 475 (EINTR). This means less data may be returned than
+        requested.
+
         In non-blocking mode, returns None if no data is available.
         Return an empty bytes object at EOF.
         """
@@ -1659,45 +1677,57 @@ class FileIO(RawIOBase):
     def readall(self):
         """Read all data from the file, returned as bytes.
 
-        In non-blocking mode, returns as much as is immediately available,
-        or None if no data is available.  Return an empty bytes object at EOF.
+        Reads until either there is an error or read() returns size 0
+        (indicates EOF). If the file is already at EOF, returns an
+        empty bytes object.
+
+        In non-blocking mode, returns as much data as could be read
+        before EAGAIN. If no data is available (EAGAIN is returned
+        before bytes are read) returns None.
         """
         self._checkClosed()
         self._checkReadable()
-        bufsize = DEFAULT_BUFFER_SIZE
+        if self._stat_atopen is None or self._stat_atopen.st_size <= 0:
+            bufsize = DEFAULT_BUFFER_SIZE
+        else:
+            # In order to detect end of file, need a read() of at least 1
+            # byte which returns size 0. Oversize the buffer by 1 byte so the
+            # I/O can be completed with two read() calls (one for all data, one
+            # for EOF) without needing to resize the buffer.
+            bufsize = self._stat_atopen.st_size + 1
+
+            if self._stat_atopen.st_size > 65536:
+                try:
+                    pos = os.lseek(self._fd, 0, SEEK_CUR)
+                    if self._stat_atopen.st_size >= pos:
+                        bufsize = self._stat_atopen.st_size - pos + 1
+                except OSError:
+                    pass
+
+        result = bytearray(bufsize)
+        bytes_read = 0
         try:
-            pos = os.lseek(self._fd, 0, SEEK_CUR)
-            end = os.fstat(self._fd).st_size
-            if end >= pos:
-                bufsize = end - pos + 1
-        except OSError:
-            pass
-
-        result = bytearray()
-        while True:
-            if len(result) >= bufsize:
-                bufsize = len(result)
-                bufsize += max(bufsize, DEFAULT_BUFFER_SIZE)
-            n = bufsize - len(result)
-            try:
-                chunk = os.read(self._fd, n)
-            except BlockingIOError:
-                if result:
-                    break
+            while n := os.readinto(self._fd, memoryview(result)[bytes_read:]):
+                bytes_read += n
+                if bytes_read >= len(result):
+                    result.resize(_new_buffersize(bytes_read))
+        except BlockingIOError:
+            if not bytes_read:
                 return None
-            if not chunk: # reached the end of the file
-                break
-            result += chunk
 
+        assert len(result) - bytes_read >= 1, \
+            "os.readinto buffer size 0 will result in erroneous EOF / returns 0"
+        result.resize(bytes_read)
         return bytes(result)
 
-    def readinto(self, b):
+    def readinto(self, buffer):
         """Same as RawIOBase.readinto()."""
-        m = memoryview(b).cast('B')
-        data = self.read(len(m))
-        n = len(data)
-        m[:n] = data
-        return n
+        self._checkClosed()
+        self._checkReadable()
+        try:
+            return os.readinto(self._fd, buffer)
+        except BlockingIOError:
+            return None
 
     def write(self, b):
         """Write bytes b to file, return number written.
@@ -1747,6 +1777,7 @@ class FileIO(RawIOBase):
         if size is None:
             size = self.tell()
         os.ftruncate(self._fd, size)
+        self._stat_atopen = None
         return size
 
     def close(self):
@@ -1756,8 +1787,9 @@ class FileIO(RawIOBase):
         called more than once without error.
         """
         if not self.closed:
+            self._stat_atopen = None
             try:
-                if self._closefd:
+                if self._closefd and self._fd >= 0:
                     os.close(self._fd)
             finally:
                 super().close()
@@ -1794,6 +1826,21 @@ class FileIO(RawIOBase):
         self._checkClosed()
         return os.isatty(self._fd)
 
+    def _isatty_open_only(self):
+        """Checks whether the file is a TTY using an open-only optimization.
+
+        TTYs are always character devices. If the interpreter knows a file is
+        not a character device when it would call ``isatty``, can skip that
+        call. Inside ``open()``  there is a fresh stat result that contains that
+        information. Use the stat result to skip a system call. Outside of that
+        context TOCTOU issues (the fd could be arbitrarily modified by
+        surrounding code).
+        """
+        if (self._stat_atopen is not None
+            and not stat.S_ISCHR(self._stat_atopen.st_mode)):
+            return False
+        return os.isatty(self._fd)
+
     @property
     def closefd(self):
         """True if the file descriptor will be closed by close()."""
@@ -2018,8 +2065,7 @@ class TextIOWrapper(TextIOBase):
             raise ValueError("invalid encoding: %r" % encoding)
 
         if not codecs.lookup(encoding)._is_text_encoding:
-            msg = ("%r is not a text encoding; "
-                   "use codecs.open() to handle arbitrary codecs")
+            msg = "%r is not a text encoding"
             raise LookupError(msg % encoding)
 
         if errors is None:
@@ -2527,9 +2573,12 @@ class TextIOWrapper(TextIOBase):
                 size = size_index()
         decoder = self._decoder or self._get_decoder()
         if size < 0:
+            chunk = self.buffer.read()
+            if chunk is None:
+                raise BlockingIOError("Read returned None.")
             # Read everything.
             result = (self._get_decoded_chars() +
-                      decoder.decode(self.buffer.read(), final=True))
+                      decoder.decode(chunk, final=True))
             if self._snapshot is not None:
                 self._set_decoded_chars('')
                 self._snapshot = None
@@ -2649,6 +2698,10 @@ class TextIOWrapper(TextIOBase):
     def newlines(self):
         return self._decoder.newlines if self._decoder else None
 
+    def _dealloc_warn(self, source):
+        if dealloc_warn := getattr(self.buffer, "_dealloc_warn", None):
+            dealloc_warn(source)
+
 
 class StringIO(TextIOWrapper):
     """Text I/O implementation using an in-memory buffer.

Lib/_pylong.py

@@ -45,10 +45,16 @@ except ImportError:
 #
 # and `mycache[lo]` replaces `base**lo` in the inner function.
 #
-# While this does give minor speedups (a few percent at best), the primary
-# intent is to simplify the functions using this, by eliminating the need for
-# them to craft their own ad-hoc caching schemes.
-def compute_powers(w, base, more_than, show=False):
+# If an algorithm wants the powers of ceiling(w/2) instead of the floor,
+# pass keyword argument `need_hi=True`.
+#
+# While this does give minor speedups (a few percent at best), the
+# primary intent is to simplify the functions using this, by eliminating
+# the need for them to craft their own ad-hoc caching schemes.
+#
+# See code near end of file for a block of code that can be enabled to
+# run millions of tests.
+def compute_powers(w, base, more_than, *, need_hi=False, show=False):
     seen = set()
     need = set()
     ws = {w}
@@ -58,40 +64,70 @@ def compute_powers(w, base, more_than, show=False):
             continue
         seen.add(w)
         lo = w >> 1
-        # only _need_ lo here; some other path may, or may not, need hi
-        need.add(lo)
-        ws.add(lo)
-        if w & 1:
-            ws.add(lo + 1)
+        hi = w - lo
+        # only _need_ one here; the other may, or may not, be needed
+        which = hi if need_hi else lo
+        need.add(which)
+        ws.add(which)
+        if lo != hi:
+            ws.add(w - which)
+
+    # `need` is the set of exponents needed. To compute them all
+    # efficiently, possibly add other exponents to `extra`. The goal is
+    # to ensure that each exponent can be gotten from a smaller one via
+    # multiplying by the base, squaring it, or squaring and then
+    # multiplying by the base.
+    #
+    # If need_hi is False, this is already the case (w can always be
+    # gotten from w >> 1 via one of the squaring strategies). But we do
+    # the work anyway, just in case ;-)
+    #
+    # Note that speed is irrelevant. These loops are working on little
+    # ints (exponents) and go around O(log w) times. The total cost is
+    # insignificant compared to just one of the bigint multiplies.
+    cands = need.copy()
+    extra = set()
+    while cands:
+        w = max(cands)
+        cands.remove(w)
+        lo = w >> 1
+        if lo > more_than and w-1 not in cands and lo not in cands:
+            extra.add(lo)
+            cands.add(lo)
+    assert need_hi or not extra
 
     d = {}
-    if not need:
-        return d
-    it = iter(sorted(need))
-    first = next(it)
-    if show:
-        print("pow at", first)
-    d[first] = base ** first
-    for this in it:
-        if this - 1 in d:
+    for n in sorted(need | extra):
+        lo = n >> 1
+        hi = n - lo
+        if n-1 in d:
             if show:
-                print("* base at", this)
-            d[this] = d[this - 1] * base # cheap
-        else:
-            lo = this >> 1
-            hi = this - lo
-            assert lo in d
+                print("* base", end="")
+            result = d[n-1] * base # cheap!
+        elif lo in d:
+            # Multiplying a bigint by itself is about twice as fast
+            # in CPython provided it's the same object.
             if show:
-                print("square at", this)
-            # Multiplying a bigint by itself (same object!) is about twice
-            # as fast in CPython.
-            sq = d[lo] * d[lo]
+                print("square", end="")
+            result = d[lo] * d[lo] # same object
             if hi != lo:
-                assert hi == lo + 1
                 if show:
-                    print("    and * base")
-                sq *= base
-            d[this] = sq
+                    print(" * base", end="")
+                assert 2 * lo + 1 == n
+                result *= base
+        else: # rare
+            if show:
+                print("pow", end='')
+            result = base ** n
+        if show:
+            print(" at", n, "needed" if n in need else "extra")
+        d[n] = result
+
+    assert need <= d.keys()
+    if excess := d.keys() - need:
+        assert need_hi
+        for n in excess:
+            del d[n]
     return d
 
 _unbounded_dec_context = decimal.getcontext().copy()
@@ -211,6 +247,145 @@ def _str_to_int_inner(s):
     return inner(0, len(s))
 
 
+# Asymptotically faster version, using the C decimal module. See
+# comments at the end of the file. This uses decimal arithmetic to
+# convert from base 10 to base 256. The latter is just a string of
+# bytes, which CPython can convert very efficiently to a Python int.
+
+# log of 10 to base 256 with best-possible 53-bit precision. Obtained
+# via:
+#    from mpmath import mp
+#    mp.prec = 1000
+#    print(float(mp.log(10, 256)).hex())
+_LOG_10_BASE_256 = float.fromhex('0x1.a934f0979a371p-2') # about 0.415
+
+# _spread is for internal testing. It maps a key to the number of times
+# that condition obtained in _dec_str_to_int_inner:
+#     key 0 - quotient guess was right
+#     key 1 - quotient had to be boosted by 1, one time
+#     key 999 - one adjustment wasn't enough, so fell back to divmod
+from collections import defaultdict
+_spread = defaultdict(int)
+del defaultdict
+
+def _dec_str_to_int_inner(s, *, GUARD=8):
+    # Yes, BYTELIM is "large". Large enough that CPython will usually
+    # use the Karatsuba _str_to_int_inner to convert the string. This
+    # allowed reducing the cutoff for calling _this_ function from 3.5M
+    # to 2M digits. We could almost certainly do even better by
+    # fine-tuning this and/or using a larger output base than 256.
+    BYTELIM = 100_000
+    D = decimal.Decimal
+    result = bytearray()
+    # See notes at end of file for discussion of GUARD.
+    assert GUARD > 0 # if 0, `decimal` can blow up - .prec 0 not allowed
+
+    def inner(n, w):
+        #assert n < D256 ** w # required, but too expensive to check
+        if w <= BYTELIM:
+            # XXX Stefan Pochmann discovered that, for 1024-bit ints,
+            # `int(Decimal)` took 2.5x longer than `int(str(Decimal))`.
+            # Worse, `int(Decimal) is still quadratic-time for much
+            # larger ints. So unless/until all that is repaired, the
+            # seemingly redundant `str(Decimal)` is crucial to speed.
+            result.extend(int(str(n)).to_bytes(w)) # big-endian default
+            return
+        w1 = w >> 1
+        w2 = w - w1
+        if 0:
+            # This is maximally clear, but "too slow". `decimal`
+            # division is asymptotically fast, but we have no way to
+            # tell it to reuse the high-precision reciprocal it computes
+            # for pow256[w2], so it has to recompute it over & over &
+            # over again :-(
+            hi, lo = divmod(n, pow256[w2][0])
+        else:
+            p256, recip = pow256[w2]
+            # The integer part will have a number of digits about equal
+            # to the difference between the log10s of `n` and `pow256`
+            # (which, since these are integers, is roughly approximated
+            # by `.adjusted()`). That's the working precision we need,
+            ctx.prec = max(n.adjusted() - p256.adjusted(), 0) + GUARD
+            hi = +n * +recip # unary `+` chops back to ctx.prec digits
+            ctx.prec = decimal.MAX_PREC
+            hi = hi.to_integral_value() # lose the fractional digits
+            lo = n - hi * p256
+            # Because we've been uniformly rounding down, `hi` is a
+            # lower bound on the correct quotient.
+            assert lo >= 0
+            # Adjust quotient up if needed. It usually isn't. In random
+            # testing on inputs through 5 billion digit strings, the
+            # test triggered once in about 200 thousand tries.
+            count = 0
+            if lo >= p256:
+                count = 1
+                lo -= p256
+                hi += 1
+                if lo >= p256:
+                    # Complete correction via an exact computation. I
+                    # believe it's not possible to get here provided
+                    # GUARD >= 3. It's tested by reducing GUARD below
+                    # that.
+                    count = 999
+                    hi2, lo = divmod(lo, p256)
+                    hi += hi2
+            _spread[count] += 1
+            # The assert should always succeed, but way too slow to keep
+            # enabled.
+            #assert hi, lo == divmod(n, pow256[w2][0])
+        inner(hi, w1)
+        del hi # at top levels, can free a lot of RAM "early"
+        inner(lo, w2)
+
+    # How many base 256 digits are needed?. Mathematically, exactly
+    # floor(log256(int(s))) + 1. There is no cheap way to compute this.
+    # But we can get an upper bound, and that's necessary for our error
+    # analysis to make sense. int(s) < 10**len(s), so the log needed is
+    # < log256(10**len(s)) = len(s) * log256(10). However, using
+    # finite-precision floating point for this, it's possible that the
+    # computed value is a little less than the true value. If the true
+    # value is at - or a little higher than - an integer, we can get an
+    # off-by-1 error too low. So we add 2 instead of 1 if chopping lost
+    # a fraction > 0.9.
+
+    # The "WASI" test platform can complain about `len(s)` if it's too
+    # large to fit in its idea of "an index-sized integer".
+    lenS = s.__len__()
+    log_ub = lenS * _LOG_10_BASE_256
+    log_ub_as_int = int(log_ub)
+    w = log_ub_as_int + 1 + (log_ub - log_ub_as_int > 0.9)
+    # And what if we've plain exhausted the limits of HW floats? We
+    # could compute the log to any desired precision using `decimal`,
+    # but it's not plausible that anyone will pass a string requiring
+    # trillions of bytes (unless they're just trying to "break things").
+    if w.bit_length() >= 46:
+        # "Only" had < 53 - 46 = 7 bits to spare in IEEE-754 double.
+        raise ValueError(f"cannot convert string of len {lenS} to int")
+    with decimal.localcontext(_unbounded_dec_context) as ctx:
+        D256 = D(256)
+        pow256 = compute_powers(w, D256, BYTELIM, need_hi=True)
+        rpow256 = compute_powers(w, 1 / D256, BYTELIM, need_hi=True)
+        # We're going to do inexact, chopped arithmetic, multiplying by
+        # an approximation to the reciprocal of 256**i. We chop to get a
+        # lower bound on the true integer quotient. Our approximation is
+        # a lower bound, the multiplication is chopped too, and
+        # to_integral_value() is also chopped.
+        ctx.traps[decimal.Inexact] = 0
+        ctx.rounding = decimal.ROUND_DOWN
+        for k, v in pow256.items():
+            # No need to save much more precision in the reciprocal than
+            # the power of 256 has, plus some guard digits to absorb
+            # most relevant rounding errors. This is highly significant:
+            # 1/2**i has the same number of significant decimal digits
+            # as 5**i, generally over twice the number in 2**i,
+            ctx.prec = v.adjusted() + GUARD + 1
+            # The unary "+" chops the reciprocal back to that precision.
+            pow256[k] = v, +rpow256[k]
+        del rpow256 # exact reciprocals no longer needed
+        ctx.prec = decimal.MAX_PREC
+        inner(D(s), w)
+    return int.from_bytes(result)
+
 def int_from_string(s):
     """Asymptotically fast version of PyLong_FromString(), conversion
     of a string of decimal digits into an 'int'."""
@@ -219,7 +394,10 @@ def int_from_string(s):
     # and underscores, and stripped leading whitespace.  The input can still
     # contain underscores and have trailing whitespace.
     s = s.rstrip().replace('_', '')
-    return _str_to_int_inner(s)
+    func = _str_to_int_inner
+    if len(s) >= 2_000_000 and _decimal is not None:
+        func = _dec_str_to_int_inner
+    return func(s)
 
 def str_to_int(s):
     """Asymptotically fast version of decimal string to 'int' conversion."""
@@ -352,7 +530,7 @@ def int_divmod(a, b):
     Its time complexity is O(n**1.58), where n = #bits(a) + #bits(b).
     """
     if b == 0:
-        raise ZeroDivisionError
+        raise ZeroDivisionError('division by zero')
     elif b < 0:
         q, r = int_divmod(-a, -b)
         return q, -r
@@ -361,3 +539,191 @@ def int_divmod(a, b):
         return ~q, b + ~r
     else:
         return _divmod_pos(a, b)
+
+
+# Notes on _dec_str_to_int_inner:
+#
+# Stefan Pochmann worked up a str->int function that used the decimal
+# module to, in effect, convert from base 10 to base 256. This is
+# "unnatural", in that it requires multiplying and dividing by large
+# powers of 2, which `decimal` isn't naturally suited to. But
+# `decimal`'s `*` and `/` are asymptotically superior to CPython's, so
+# at _some_ point it could be expected to win.
+#
+# Alas, the crossover point was too high to be of much real interest. I
+# (Tim) then worked on ways to replace its division with multiplication
+# by a cached reciprocal approximation instead, fixing up errors
+# afterwards. This reduced the crossover point significantly,
+#
+# I revisited the code, and found ways to improve and simplify it. The
+# crossover point is at about 3.4 million digits now.
+#
+# About .adjusted()
+# -----------------
+# Restrict to Decimal values x > 0. We don't use negative numbers in the
+# code, and I don't want to have to keep typing, e.g., "absolute value".
+#
+# For convenience, I'll use `x.a` to mean `x.adjusted()`. x.a doesn't
+# look at the digits of x, but instead returns an integer giving x's
+# order of magnitude. These are equivalent:
+#
+# - x.a is the power-of-10 exponent of x's most significant digit.
+# - x.a = the infinitely precise floor(log10(x))
+# - x can be written in this form, where f is a real with 1 <= f < 10:
+#    x = f * 10**x.a
+#
+# Observation; if x is an integer, len(str(x)) = x.a + 1.
+#
+# Lemma 1: (x * y).a = x.a + y.a, or one larger
+#
+# Proof: Write x = f * 10**x.a and y = g * 10**y.a, where f and g are in
+# [1, 10). Then x*y = f*g * 10**(x.a + y.a), where 1 <= f*g < 100. If
+# f*g < 10, (x*y).a is x.a+y.a. Else divide f*g by 10 to bring it back
+# into [1, 10], and add 1 to the exponent to compensate. Then (x*y).a is
+# x.a+y.a+1.
+#
+# Lemma 2: ceiling(log10(x/y)) <= x.a - y.a + 1
+#
+# Proof: Express x and y as in Lemma 1. Then x/y = f/g * 10**(x.a -
+# y.a), where 1/10 < f/g < 10. If 1 <= f/g, (x/y).a is x.a-y.a. Else
+# multiply f/g by 10 to bring it back into [1, 10], and subtract 1 from
+# the exponent to compensate. Then (x/y).a is x.a-y.a-1. So the largest
+# (x/y).a can be is x.a-y.a. Since that's the floor of log10(x/y). the
+# ceiling is at most 1 larger (with equality iff f/g = 1 exactly).
+#
+# GUARD digits
+# ------------
+# We only want the integer part of divisions, so don't need to build
+# the full multiplication tree. But using _just_ the number of
+# digits expected in the integer part ignores too much. What's left
+# out can have a very significant effect on the quotient. So we use
+# GUARD additional digits.
+#
+# The default 8 is more than enough so no more than 1 correction step
+# was ever needed for all inputs tried through 2.5 billion digits. In
+# fact, I believe 3 guard digits are always enough - but the proof is
+# very involved, so better safe than sorry.
+#
+# Short course:
+#
+# If prec is the decimal precision in effect, and we're rounding down,
+# the result of an operation is exactly equal to the infinitely precise
+# result times 1-e for some real e with 0 <= e < 10**(1-prec). In
+#
+#     ctx.prec = max(n.adjusted() - p256.adjusted(), 0) + GUARD
+#     hi = +n * +recip # unary `+` chops to ctx.prec digits
+#
+# we have 3 visible chopped operations, but there's also a 4th:
+# precomputing a truncated `recip` as part of setup.
+#
+# So the computed product is exactly equal to the true product times
+# (1-e1)*(1-e2)*(1-e3)*(1-e4); since the e's are all very small, an
+# excellent approximation to the second factor is 1-(e1+e2+e3+e4) (the
+# 2nd and higher order terms in the expanded product are too tiny to
+# matter). If they're all as large as possible, that's
+#
+# 1 - 4*10**(1-prec). This, BTW, is all bog-standard FP error analysis.
+#
+# That implies the computed product is within 1 of the true product
+# provided prec >= log10(true_product) + 1.602.
+#
+# Here are telegraphic details, rephrasing the initial condition in
+# equivalent ways, step by step:
+#
+# prod - prod * (1 - 4*10**(1-prec)) <= 1
+# prod - prod + prod * 4*10**(1-prec)) <= 1
+# prod * 4*10**(1-prec)) <= 1
+# 10**(log10(prod)) * 4*10**(1-prec)) <= 1
+# 4*10**(1-prec+log10(prod))) <= 1
+# 10**(1-prec+log10(prod))) <= 1/4
+# 1-prec+log10(prod) <= log10(1/4) = -0.602
+# -prec <= -1.602 - log10(prod)
+# prec >= log10(prod) + 1.602
+#
+# The true product is the same as the true ratio n/p256. By Lemma 2
+# above, n.a - p256.a + 1 is an upper bound on the ceiling of
+# log10(prod). Then 2 is the ceiling of 1.602. so n.a - p256.a + 3 is an
+# upper bound on the right hand side of the inequality. Any prec >= that
+# will work.
+#
+# But since this is just a sketch of a proof ;-), the code uses the
+# empirically tested 8 instead of 3. 5 digits more or less makes no
+# practical difference to speed - these ints are huge. And while
+# increasing GUARD above 3 may not be necessary, every increase cuts the
+# percentage of cases that need a correction at all.
+#
+# On Computing Reciprocals
+# ------------------------
+# In general, the exact reciprocals we compute have over twice as many
+# significant digits as needed. 1/256**i has the same number of
+# significant decimal digits as 5**i. It's a significant waste of RAM
+# to store all those unneeded digits.
+#
+# So we cut exact reciprocals back to the least precision that can
+# be needed so that the error analysis above is valid,
+#
+# [Note: turns out it's very significantly faster to do it this way than
+# to compute  1 / 256**i  directly to the desired precision, because the
+# power method doesn't require division. It's also faster than computing
+# (1/256)**i directly to the desired precision - no material division
+# there, but `compute_powers()` is much smarter about _how_ to compute
+# all the powers needed than repeated applications of `**` - that
+# function invokes `**` for at most the few smallest powers needed.]
+#
+# The hard part is that chopping back to a shorter width occurs
+# _outside_ of `inner`. We can't know then what `prec` `inner()` will
+# need. We have to pick, for each value of `w2`, the largest possible
+# value `prec` can become when `inner()` is working on `w2`.
+#
+# This is the `prec` inner() uses:
+#     max(n.a - p256.a, 0) + GUARD
+# and what setup uses (renaming its `v` to `p256` - same thing):
+#     p256.a + GUARD + 1
+#
+# We need that the second is always at least as large as the first,
+# which is the same as requiring
+#
+#     n.a - 2 * p256.a <= 1
+#
+# What's the largest n can be? n < 255**w = 256**(w2 + (w - w2)). The
+# worst case in this context is when w ix even. and then w = 2*w2, so
+# n < 256**(2*w2) = (256**w2)**2 = p256**2. By Lemma 1, then, n.a
+# is at most p256.a + p256.a + 1.
+#
+# So the most n.a - 2 * p256.a can be is
+# p256.a + p256.a + 1 - 2 * p256.a = 1. QED
+#
+# Note: an earlier version of the code split on floor(e/2) instead of on
+# the ceiling. The worst case then is odd `w`, and a more involved proof
+# was needed to show that adding 4 (instead of 1) may be necessary.
+# Basically because, in that case, n may be up to 256 times larger than
+# p256**2. Curiously enough, by splitting on the ceiling instead,
+# nothing in any proof here actually depends on the output base (256).
+
+# Enable for brute-force testing of compute_powers(). This takes about a
+# minute, because it tries millions of cases.
+if 0:
+    def consumer(w, limit, need_hi):
+        seen = set()
+        need = set()
+        def inner(w):
+            if w <= limit:
+                return
+            if w in seen:
+                return
+            seen.add(w)
+            lo = w >> 1
+            hi = w - lo
+            need.add(hi if need_hi else lo)
+            inner(lo)
+            inner(hi)
+        inner(w)
+        exp = compute_powers(w, 1, limit, need_hi=need_hi)
+        assert exp.keys() == need
+
+    from itertools import chain
+    for need_hi in (False, True):
+        for limit in (0, 1, 10, 100, 1_000, 10_000, 100_000):
+            for w in chain(range(1, 100_000),
+                           (10**i for i in range(5, 30))):
+                consumer(w, limit, need_hi)

Lib/_strptime.py

@@ -10,10 +10,13 @@ FUNCTIONS:
     strptime -- Calculates the time struct represented by the passed-in string
 
 """
+import os
 import time
 import locale
 import calendar
+import re
 from re import compile as re_compile
+from re import sub as re_sub
 from re import IGNORECASE
 from re import escape as re_escape
 from datetime import (date as datetime_date,
@@ -27,6 +30,41 @@ def _getlang():
     # Figure out what the current language is set to.
     return locale.getlocale(locale.LC_TIME)
 
+def _findall(haystack, needle):
+    # Find all positions of needle in haystack.
+    if not needle:
+        return
+    i = 0
+    while True:
+        i = haystack.find(needle, i)
+        if i < 0:
+            break
+        yield i
+        i += len(needle)
+
+def _fixmonths(months):
+    yield from months
+    # The lower case of 'İ' ('\u0130') is 'i\u0307'.
+    # The re module only supports 1-to-1 character matching in
+    # case-insensitive mode.
+    for s in months:
+        if 'i\u0307' in s:
+            yield s.replace('i\u0307', '\u0130')
+
+lzh_TW_alt_digits = (
+    # 〇:一:二:三:四:五:六:七:八:九
+    '\u3007', '\u4e00', '\u4e8c', '\u4e09', '\u56db',
+    '\u4e94', '\u516d', '\u4e03', '\u516b', '\u4e5d',
+    # 十:十一:十二:十三:十四:十五:十六:十七:十八:十九
+    '\u5341', '\u5341\u4e00', '\u5341\u4e8c', '\u5341\u4e09', '\u5341\u56db',
+    '\u5341\u4e94', '\u5341\u516d', '\u5341\u4e03', '\u5341\u516b', '\u5341\u4e5d',
+    # 廿:廿一:廿二:廿三:廿四:廿五:廿六:廿七:廿八:廿九
+    '\u5eff', '\u5eff\u4e00', '\u5eff\u4e8c', '\u5eff\u4e09', '\u5eff\u56db',
+    '\u5eff\u4e94', '\u5eff\u516d', '\u5eff\u4e03', '\u5eff\u516b', '\u5eff\u4e5d',
+    # 卅:卅一
+    '\u5345', '\u5345\u4e00')
+
+
 class LocaleTime(object):
     """Stores and handles locale-specific information related to time.
 
@@ -70,6 +108,7 @@ class LocaleTime(object):
         self.__calc_weekday()
         self.__calc_month()
         self.__calc_am_pm()
+        self.__calc_alt_digits()
         self.__calc_timezone()
         self.__calc_date_time()
         if _getlang() != self.lang:
@@ -101,53 +140,184 @@ class LocaleTime(object):
         am_pm = []
         for hour in (1, 22):
             time_tuple = time.struct_time((1999,3,17,hour,44,55,2,76,0))
-            am_pm.append(time.strftime("%p", time_tuple).lower())
+            # br_FR has AM/PM info (' ',' ').
+            am_pm.append(time.strftime("%p", time_tuple).lower().strip())
         self.am_pm = am_pm
 
+    def __calc_alt_digits(self):
+        # Set self.LC_alt_digits by using time.strftime().
+
+        # The magic data should contain all decimal digits.
+        time_tuple = time.struct_time((1998, 1, 27, 10, 43, 56, 1, 27, 0))
+        s = time.strftime("%x%X", time_tuple)
+        if s.isascii():
+            # Fast path -- all digits are ASCII.
+            self.LC_alt_digits = ()
+            return
+
+        digits = ''.join(sorted(set(re.findall(r'\d', s))))
+        if len(digits) == 10 and ord(digits[-1]) == ord(digits[0]) + 9:
+            # All 10 decimal digits from the same set.
+            if digits.isascii():
+                # All digits are ASCII.
+                self.LC_alt_digits = ()
+                return
+
+            self.LC_alt_digits = [a + b for a in digits for b in digits]
+            # Test whether the numbers contain leading zero.
+            time_tuple2 = time.struct_time((2000, 1, 1, 1, 1, 1, 5, 1, 0))
+            if self.LC_alt_digits[1] not in time.strftime("%x %X", time_tuple2):
+                self.LC_alt_digits[:10] = digits
+            return
+
+        # Either non-Gregorian calendar or non-decimal numbers.
+        if {'\u4e00', '\u4e03', '\u4e5d', '\u5341', '\u5eff'}.issubset(s):
+            # lzh_TW
+            self.LC_alt_digits = lzh_TW_alt_digits
+            return
+
+        self.LC_alt_digits = None
+
     def __calc_date_time(self):
-        # Set self.date_time, self.date, & self.time by using
-        # time.strftime().
+        # Set self.LC_date_time, self.LC_date, self.LC_time and
+        # self.LC_time_ampm by using time.strftime().
 
         # Use (1999,3,17,22,44,55,2,76,0) for magic date because the amount of
         # overloaded numbers is minimized.  The order in which searches for
         # values within the format string is very important; it eliminates
         # possible ambiguity for what something represents.
         time_tuple = time.struct_time((1999,3,17,22,44,55,2,76,0))
-        date_time = [None, None, None]
-        date_time[0] = time.strftime("%c", time_tuple).lower()
-        date_time[1] = time.strftime("%x", time_tuple).lower()
-        date_time[2] = time.strftime("%X", time_tuple).lower()
-        replacement_pairs = [('%', '%%'), (self.f_weekday[2], '%A'),
-                    (self.f_month[3], '%B'), (self.a_weekday[2], '%a'),
-                    (self.a_month[3], '%b'), (self.am_pm[1], '%p'),
-                    ('1999', '%Y'), ('99', '%y'), ('22', '%H'),
-                    ('44', '%M'), ('55', '%S'), ('76', '%j'),
-                    ('17', '%d'), ('03', '%m'), ('3', '%m'),
-                    # '3' needed for when no leading zero.
-                    ('2', '%w'), ('10', '%I')]
-        replacement_pairs.extend([(tz, "%Z") for tz_values in self.timezone
-                                                for tz in tz_values])
-        for offset,directive in ((0,'%c'), (1,'%x'), (2,'%X')):
-            current_format = date_time[offset]
-            for old, new in replacement_pairs:
+        time_tuple2 = time.struct_time((1999,1,3,1,1,1,6,3,0))
+        replacement_pairs = []
+
+        # Non-ASCII digits
+        if self.LC_alt_digits or self.LC_alt_digits is None:
+            for n, d in [(19, '%OC'), (99, '%Oy'), (22, '%OH'),
+                         (44, '%OM'), (55, '%OS'), (17, '%Od'),
+                         (3, '%Om'), (2, '%Ow'), (10, '%OI')]:
+                if self.LC_alt_digits is None:
+                    s = chr(0x660 + n // 10) + chr(0x660 + n % 10)
+                    replacement_pairs.append((s, d))
+                    if n < 10:
+                        replacement_pairs.append((s[1], d))
+                elif len(self.LC_alt_digits) > n:
+                    replacement_pairs.append((self.LC_alt_digits[n], d))
+                else:
+                    replacement_pairs.append((time.strftime(d, time_tuple), d))
+        replacement_pairs += [
+            ('1999', '%Y'), ('99', '%y'), ('22', '%H'),
+            ('44', '%M'), ('55', '%S'), ('76', '%j'),
+            ('17', '%d'), ('03', '%m'), ('3', '%m'),
+            # '3' needed for when no leading zero.
+            ('2', '%w'), ('10', '%I'),
+        ]
+
+        date_time = []
+        for directive in ('%c', '%x', '%X', '%r'):
+            current_format = time.strftime(directive, time_tuple).lower()
+            current_format = current_format.replace('%', '%%')
+            # The month and the day of the week formats are treated specially
+            # because of a possible ambiguity in some locales where the full
+            # and abbreviated names are equal or names of different types
+            # are equal. See doc of __find_month_format for more details.
+            lst, fmt = self.__find_weekday_format(directive)
+            if lst:
+                current_format = current_format.replace(lst[2], fmt, 1)
+            lst, fmt = self.__find_month_format(directive)
+            if lst:
+                current_format = current_format.replace(lst[3], fmt, 1)
+            if self.am_pm[1]:
                 # Must deal with possible lack of locale info
                 # manifesting itself as the empty string (e.g., Swedish's
                 # lack of AM/PM info) or a platform returning a tuple of empty
                 # strings (e.g., MacOS 9 having timezone as ('','')).
-                if old:
-                    current_format = current_format.replace(old, new)
+                current_format = current_format.replace(self.am_pm[1], '%p')
+            for tz_values in self.timezone:
+                for tz in tz_values:
+                    if tz:
+                        current_format = current_format.replace(tz, "%Z")
+            # Transform all non-ASCII digits to digits in range U+0660 to U+0669.
+            if not current_format.isascii() and self.LC_alt_digits is None:
+                current_format = re_sub(r'\d(?<![0-9])',
+                                        lambda m: chr(0x0660 + int(m[0])),
+                                        current_format)
+            for old, new in replacement_pairs:
+                current_format = current_format.replace(old, new)
             # If %W is used, then Sunday, 2005-01-03 will fall on week 0 since
             # 2005-01-03 occurs before the first Monday of the year.  Otherwise
             # %U is used.
-            time_tuple = time.struct_time((1999,1,3,1,1,1,6,3,0))
-            if '00' in time.strftime(directive, time_tuple):
+            if '00' in time.strftime(directive, time_tuple2):
                 U_W = '%W'
             else:
                 U_W = '%U'
-            date_time[offset] = current_format.replace('11', U_W)
+            current_format = current_format.replace('11', U_W)
+            date_time.append(current_format)
         self.LC_date_time = date_time[0]
         self.LC_date = date_time[1]
         self.LC_time = date_time[2]
+        self.LC_time_ampm = date_time[3]
+
+    def __find_month_format(self, directive):
+        """Find the month format appropriate for the current locale.
+
+        In some locales (for example French and Hebrew), the default month
+        used in __calc_date_time has the same name in full and abbreviated
+        form.  Also, the month name can by accident match other part of the
+        representation: the day of the week name (for example in Morisyen)
+        or the month number (for example in Japanese).  Thus, cycle months
+        of the year and find all positions that match the month name for
+        each month,  If no common positions are found, the representation
+        does not use the month name.
+        """
+        full_indices = abbr_indices = None
+        for m in range(1, 13):
+            time_tuple = time.struct_time((1999, m, 17, 22, 44, 55, 2, 76, 0))
+            datetime = time.strftime(directive, time_tuple).lower()
+            indices = set(_findall(datetime, self.f_month[m]))
+            if full_indices is None:
+                full_indices = indices
+            else:
+                full_indices &= indices
+            indices = set(_findall(datetime, self.a_month[m]))
+            if abbr_indices is None:
+                abbr_indices = set(indices)
+            else:
+                abbr_indices &= indices
+            if not full_indices and not abbr_indices:
+                return None, None
+        if full_indices:
+            return self.f_month, '%B'
+        if abbr_indices:
+            return self.a_month, '%b'
+        return None, None
+
+    def __find_weekday_format(self, directive):
+        """Find the day of the week format appropriate for the current locale.
+
+        Similar to __find_month_format().
+        """
+        full_indices = abbr_indices = None
+        for wd in range(7):
+            time_tuple = time.struct_time((1999, 3, 17, 22, 44, 55, wd, 76, 0))
+            datetime = time.strftime(directive, time_tuple).lower()
+            indices = set(_findall(datetime, self.f_weekday[wd]))
+            if full_indices is None:
+                full_indices = indices
+            else:
+                full_indices &= indices
+            if self.f_weekday[wd] != self.a_weekday[wd]:
+                indices = set(_findall(datetime, self.a_weekday[wd]))
+            if abbr_indices is None:
+                abbr_indices = set(indices)
+            else:
+                abbr_indices &= indices
+            if not full_indices and not abbr_indices:
+                return None, None
+        if full_indices:
+            return self.f_weekday, '%A'
+        if abbr_indices:
+            return self.a_weekday, '%a'
+        return None, None
 
     def __calc_timezone(self):
         # Set self.timezone by using time.tzname.
@@ -181,12 +351,14 @@ class TimeRE(dict):
         else:
             self.locale_time = LocaleTime()
         base = super()
-        base.__init__({
+        mapping = {
             # The " [1-9]" part of the regex is to make %c from ANSI C work
             'd': r"(?P<d>3[0-1]|[1-2]\d|0[1-9]|[1-9]| [1-9])",
             'f': r"(?P<f>[0-9]{1,6})",
-            'H': r"(?P<H>2[0-3]|[0-1]\d|\d)",
-            'I': r"(?P<I>1[0-2]|0[1-9]|[1-9])",
+            'H': r"(?P<H>2[0-3]|[0-1]\d|\d| \d)",
+            'k': r"(?P<H>2[0-3]|[0-1]\d|\d| \d)",
+            'I': r"(?P<I>1[0-2]|0[1-9]|[1-9]| [1-9])",
+            'l': r"(?P<I>1[0-2]|0[1-9]|[1-9]| [1-9])",
             'G': r"(?P<G>\d\d\d\d)",
             'j': r"(?P<j>36[0-6]|3[0-5]\d|[1-2]\d\d|0[1-9]\d|00[1-9]|[1-9]\d|0[1-9]|[1-9])",
             'm': r"(?P<m>1[0-2]|0[1-9]|[1-9])",
@@ -198,25 +370,60 @@ class TimeRE(dict):
             'V': r"(?P<V>5[0-3]|0[1-9]|[1-4]\d|\d)",
             # W is set below by using 'U'
             'y': r"(?P<y>\d\d)",
-            #XXX: Does 'Y' need to worry about having less or more than
-            #     4 digits?
             'Y': r"(?P<Y>\d\d\d\d)",
             'z': r"(?P<z>[+-]\d\d:?[0-5]\d(:?[0-5]\d(\.\d{1,6})?)?|(?-i:Z))",
             'A': self.__seqToRE(self.locale_time.f_weekday, 'A'),
             'a': self.__seqToRE(self.locale_time.a_weekday, 'a'),
-            'B': self.__seqToRE(self.locale_time.f_month[1:], 'B'),
-            'b': self.__seqToRE(self.locale_time.a_month[1:], 'b'),
+            'B': self.__seqToRE(_fixmonths(self.locale_time.f_month[1:]), 'B'),
+            'b': self.__seqToRE(_fixmonths(self.locale_time.a_month[1:]), 'b'),
             'p': self.__seqToRE(self.locale_time.am_pm, 'p'),
             'Z': self.__seqToRE((tz for tz_names in self.locale_time.timezone
                                         for tz in tz_names),
                                 'Z'),
-            '%': '%'})
-        base.__setitem__('W', base.__getitem__('U').replace('U', 'W'))
-        base.__setitem__('c', self.pattern(self.locale_time.LC_date_time))
-        base.__setitem__('x', self.pattern(self.locale_time.LC_date))
-        base.__setitem__('X', self.pattern(self.locale_time.LC_time))
+            '%': '%'}
+        if self.locale_time.LC_alt_digits is None:
+            for d in 'dmyCHIMS':
+                mapping['O' + d] = r'(?P<%s>\d\d|\d| \d)' % d
+            mapping['Ow'] = r'(?P<w>\d)'
+        else:
+            mapping.update({
+                'Od': self.__seqToRE(self.locale_time.LC_alt_digits[1:32], 'd',
+                                     '3[0-1]|[1-2][0-9]|0[1-9]|[1-9]'),
+                'Om': self.__seqToRE(self.locale_time.LC_alt_digits[1:13], 'm',
+                                     '1[0-2]|0[1-9]|[1-9]'),
+                'Ow': self.__seqToRE(self.locale_time.LC_alt_digits[:7], 'w',
+                                     '[0-6]'),
+                'Oy': self.__seqToRE(self.locale_time.LC_alt_digits, 'y',
+                                     '[0-9][0-9]'),
+                'OC': self.__seqToRE(self.locale_time.LC_alt_digits, 'C',
+                                     '[0-9][0-9]'),
+                'OH': self.__seqToRE(self.locale_time.LC_alt_digits[:24], 'H',
+                                     '2[0-3]|[0-1][0-9]|[0-9]'),
+                'OI': self.__seqToRE(self.locale_time.LC_alt_digits[1:13], 'I',
+                                     '1[0-2]|0[1-9]|[1-9]'),
+                'OM': self.__seqToRE(self.locale_time.LC_alt_digits[:60], 'M',
+                                     '[0-5][0-9]|[0-9]'),
+                'OS': self.__seqToRE(self.locale_time.LC_alt_digits[:62], 'S',
+                                     '6[0-1]|[0-5][0-9]|[0-9]'),
+            })
+        mapping.update({
+            'e': mapping['d'],
+            'Oe': mapping['Od'],
+            'P': mapping['p'],
+            'Op': mapping['p'],
+            'W': mapping['U'].replace('U', 'W'),
+        })
+        mapping['W'] = mapping['U'].replace('U', 'W')
 
-    def __seqToRE(self, to_convert, directive):
+        base.__init__(mapping)
+        base.__setitem__('T', self.pattern('%H:%M:%S'))
+        base.__setitem__('R', self.pattern('%H:%M'))
+        base.__setitem__('r', self.pattern(self.locale_time.LC_time_ampm))
+        base.__setitem__('X', self.pattern(self.locale_time.LC_time))
+        base.__setitem__('x', self.pattern(self.locale_time.LC_date))
+        base.__setitem__('c', self.pattern(self.locale_time.LC_date_time))
+
+    def __seqToRE(self, to_convert, directive, altregex=None):
         """Convert a list to a regex string for matching a directive.
 
         Want possible matching values to be from longest to shortest.  This
@@ -232,8 +439,9 @@ class TimeRE(dict):
         else:
             return ''
         regex = '|'.join(re_escape(stuff) for stuff in to_convert)
-        regex = '(?P<%s>%s' % (directive, regex)
-        return '%s)' % regex
+        if altregex is not None:
+            regex += '|' + altregex
+        return '(?P<%s>%s)' % (directive, regex)
 
     def pattern(self, format):
         """Return regex pattern for the format string.
@@ -242,21 +450,36 @@ class TimeRE(dict):
         regex syntax are escaped.
 
         """
-        processed_format = ''
         # The sub() call escapes all characters that might be misconstrued
         # as regex syntax.  Cannot use re.escape since we have to deal with
         # format directives (%m, etc.).
-        regex_chars = re_compile(r"([\\.^$*+?\(\){}\[\]|])")
-        format = regex_chars.sub(r"\\\1", format)
-        whitespace_replacement = re_compile(r'\s+')
-        format = whitespace_replacement.sub(r'\\s+', format)
-        while '%' in format:
-            directive_index = format.index('%')+1
-            processed_format = "%s%s%s" % (processed_format,
-                                           format[:directive_index-1],
-                                           self[format[directive_index]])
-            format = format[directive_index+1:]
-        return "%s%s" % (processed_format, format)
+        format = re_sub(r"([\\.^$*+?\(\){}\[\]|])", r"\\\1", format)
+        format = re_sub(r'\s+', r'\\s+', format)
+        format = re_sub(r"'", "['\u02bc]", format)  # needed for br_FR
+        year_in_format = False
+        day_of_month_in_format = False
+        def repl(m):
+            format_char = m[1]
+            match format_char:
+                case 'Y' | 'y' | 'G':
+                    nonlocal year_in_format
+                    year_in_format = True
+                case 'd':
+                    nonlocal day_of_month_in_format
+                    day_of_month_in_format = True
+            return self[format_char]
+        format = re_sub(r'%[-_0^#]*[0-9]*([OE]?\\?.?)', repl, format)
+        if day_of_month_in_format and not year_in_format:
+            import warnings
+            warnings.warn("""\
+Parsing dates involving a day of month without a year specified is ambiguous
+and fails to parse leap day. The default behavior will change in Python 3.15
+to either always raise an exception or to use a different default year (TBD).
+To avoid trouble, add a specific year to the input & format.
+See https://github.com/python/cpython/issues/70647.""",
+                          DeprecationWarning,
+                          skip_file_prefixes=(os.path.dirname(__file__),))
+        return format
 
     def compile(self, format):
         """Return a compiled re object for the format string."""
@@ -319,14 +542,13 @@ def _strptime(data_string, format="%a %b %d %H:%M:%S %Y"):
             # \\, in which case it was a stray % but with a space after it
             except KeyError as err:
                 bad_directive = err.args[0]
-                if bad_directive == "\\":
-                    bad_directive = "%"
                 del err
+                bad_directive = bad_directive.replace('\\s', '')
+                if not bad_directive:
+                    raise ValueError("stray %% in format '%s'" % format) from None
+                bad_directive = bad_directive.replace('\\', '', 1)
                 raise ValueError("'%s' is a bad directive in format '%s'" %
                                     (bad_directive, format)) from None
-            # IndexError only occurs when the format string is "%"
-            except IndexError:
-                raise ValueError("stray %% in format '%s'" % format) from None
             _regex_cache[format] = format_regex
     found = format_regex.match(data_string)
     if not found:
@@ -348,6 +570,15 @@ def _strptime(data_string, format="%a %b %d %H:%M:%S %Y"):
     # values
     weekday = julian = None
     found_dict = found.groupdict()
+    if locale_time.LC_alt_digits:
+        def parse_int(s):
+            try:
+                return locale_time.LC_alt_digits.index(s)
+            except ValueError:
+                return int(s)
+    else:
+        parse_int = int
+
     for group_key in found_dict.keys():
         # Directives not explicitly handled below:
         #   c, x, X
@@ -355,30 +586,34 @@ def _strptime(data_string, format="%a %b %d %H:%M:%S %Y"):
         #   U, W
         #      worthless without day of the week
         if group_key == 'y':
-            year = int(found_dict['y'])
-            # Open Group specification for strptime() states that a %y
-            #value in the range of [00, 68] is in the century 2000, while
-            #[69,99] is in the century 1900
-            if year <= 68:
-                year += 2000
+            year = parse_int(found_dict['y'])
+            if 'C' in found_dict:
+                century = parse_int(found_dict['C'])
+                year += century * 100
             else:
-                year += 1900
+                # Open Group specification for strptime() states that a %y
+                #value in the range of [00, 68] is in the century 2000, while
+                #[69,99] is in the century 1900
+                if year <= 68:
+                    year += 2000
+                else:
+                    year += 1900
         elif group_key == 'Y':
             year = int(found_dict['Y'])
         elif group_key == 'G':
             iso_year = int(found_dict['G'])
         elif group_key == 'm':
-            month = int(found_dict['m'])
+            month = parse_int(found_dict['m'])
         elif group_key == 'B':
             month = locale_time.f_month.index(found_dict['B'].lower())
         elif group_key == 'b':
             month = locale_time.a_month.index(found_dict['b'].lower())
         elif group_key == 'd':
-            day = int(found_dict['d'])
+            day = parse_int(found_dict['d'])
         elif group_key == 'H':
-            hour = int(found_dict['H'])
+            hour = parse_int(found_dict['H'])
         elif group_key == 'I':
-            hour = int(found_dict['I'])
+            hour = parse_int(found_dict['I'])
             ampm = found_dict.get('p', '').lower()
             # If there was no AM/PM indicator, we'll treat this like AM
             if ampm in ('', locale_time.am_pm[0]):
@@ -394,9 +629,9 @@ def _strptime(data_string, format="%a %b %d %H:%M:%S %Y"):
                 if hour != 12:
                     hour += 12
         elif group_key == 'M':
-            minute = int(found_dict['M'])
+            minute = parse_int(found_dict['M'])
         elif group_key == 'S':
-            second = int(found_dict['S'])
+            second = parse_int(found_dict['S'])
         elif group_key == 'f':
             s = found_dict['f']
             # Pad to always return microseconds.
@@ -548,18 +783,40 @@ def _strptime_time(data_string, format="%a %b %d %H:%M:%S %Y"):
     tt = _strptime(data_string, format)[0]
     return time.struct_time(tt[:time._STRUCT_TM_ITEMS])
 
-def _strptime_datetime(cls, data_string, format="%a %b %d %H:%M:%S %Y"):
-    """Return a class cls instance based on the input string and the
+def _strptime_datetime_date(cls, data_string, format="%a %b %d %Y"):
+    """Return a date instance based on the input string and the
+    format string."""
+    tt, _, _ = _strptime(data_string, format)
+    args = tt[:3]
+    return cls(*args)
+
+def _parse_tz(tzname, gmtoff, gmtoff_fraction):
+    tzdelta = datetime_timedelta(seconds=gmtoff, microseconds=gmtoff_fraction)
+    if tzname:
+        return datetime_timezone(tzdelta, tzname)
+    else:
+        return datetime_timezone(tzdelta)
+
+def _strptime_datetime_time(cls, data_string, format="%H:%M:%S"):
+    """Return a time instance based on the input string and the
+    format string."""
+    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
+    tzname, gmtoff = tt[-2:]
+    args = tt[3:6] + (fraction,)
+    if gmtoff is None:
+        return cls(*args)
+    else:
+        tz = _parse_tz(tzname, gmtoff, gmtoff_fraction)
+        return cls(*args, tz)
+
+def _strptime_datetime_datetime(cls, data_string, format="%a %b %d %H:%M:%S %Y"):
+    """Return a datetime instance based on the input string and the
     format string."""
     tt, fraction, gmtoff_fraction = _strptime(data_string, format)
     tzname, gmtoff = tt[-2:]
     args = tt[:6] + (fraction,)
-    if gmtoff is not None:
-        tzdelta = datetime_timedelta(seconds=gmtoff, microseconds=gmtoff_fraction)
-        if tzname:
-            tz = datetime_timezone(tzdelta, tzname)
-        else:
-            tz = datetime_timezone(tzdelta)
-        args += (tz,)
-
-    return cls(*args)
+    if gmtoff is None:
+        return cls(*args)
+    else:
+        tz = _parse_tz(tzname, gmtoff, gmtoff_fraction)
+        return cls(*args, tz)

Lib/_weakrefset.py

@@ -8,69 +8,29 @@ from types import GenericAlias
 __all__ = ['WeakSet']
 
 
-class _IterationGuard:
-    # This context manager registers itself in the current iterators of the
-    # weak container, such as to delay all removals until the context manager
-    # exits.
-    # This technique should be relatively thread-safe (since sets are).
-
-    def __init__(self, weakcontainer):
-        # Don't create cycles
-        self.weakcontainer = ref(weakcontainer)
-
-    def __enter__(self):
-        w = self.weakcontainer()
-        if w is not None:
-            w._iterating.add(self)
-        return self
-
-    def __exit__(self, e, t, b):
-        w = self.weakcontainer()
-        if w is not None:
-            s = w._iterating
-            s.remove(self)
-            if not s:
-                w._commit_removals()
-
-
 class WeakSet:
     def __init__(self, data=None):
         self.data = set()
+
         def _remove(item, selfref=ref(self)):
             self = selfref()
             if self is not None:
-                if self._iterating:
-                    self._pending_removals.append(item)
-                else:
-                    self.data.discard(item)
+                self.data.discard(item)
+
         self._remove = _remove
-        # A list of keys to be removed
-        self._pending_removals = []
-        self._iterating = set()
         if data is not None:
             self.update(data)
 
-    def _commit_removals(self):
-        pop = self._pending_removals.pop
-        discard = self.data.discard
-        while True:
-            try:
-                item = pop()
-            except IndexError:
-                return
-            discard(item)
-
     def __iter__(self):
-        with _IterationGuard(self):
-            for itemref in self.data:
-                item = itemref()
-                if item is not None:
-                    # Caveat: the iterator will keep a strong reference to
-                    # `item` until it is resumed or closed.
-                    yield item
+        for itemref in self.data.copy():
+            item = itemref()
+            if item is not None:
+                # Caveat: the iterator will keep a strong reference to
+                # `item` until it is resumed or closed.
+                yield item
 
     def __len__(self):
-        return len(self.data) - len(self._pending_removals)
+        return len(self.data)
 
     def __contains__(self, item):
         try:
@@ -83,21 +43,15 @@ class WeakSet:
         return self.__class__, (list(self),), self.__getstate__()
 
     def add(self, item):
-        if self._pending_removals:
-            self._commit_removals()
         self.data.add(ref(item, self._remove))
 
     def clear(self):
-        if self._pending_removals:
-            self._commit_removals()
         self.data.clear()
 
     def copy(self):
         return self.__class__(self)
 
     def pop(self):
-        if self._pending_removals:
-            self._commit_removals()
         while True:
             try:
                 itemref = self.data.pop()
@@ -108,18 +62,12 @@ class WeakSet:
                 return item
 
     def remove(self, item):
-        if self._pending_removals:
-            self._commit_removals()
         self.data.remove(ref(item))
 
     def discard(self, item):
-        if self._pending_removals:
-            self._commit_removals()
         self.data.discard(ref(item))
 
     def update(self, other):
-        if self._pending_removals:
-            self._commit_removals()
         for element in other:
             self.add(element)
 
@@ -136,8 +84,6 @@ class WeakSet:
     def difference_update(self, other):
         self.__isub__(other)
     def __isub__(self, other):
-        if self._pending_removals:
-            self._commit_removals()
         if self is other:
             self.data.clear()
         else:
@@ -151,8 +97,6 @@ class WeakSet:
     def intersection_update(self, other):
         self.__iand__(other)
     def __iand__(self, other):
-        if self._pending_removals:
-            self._commit_removals()
         self.data.intersection_update(ref(item) for item in other)
         return self
 
@@ -184,8 +128,6 @@ class WeakSet:
     def symmetric_difference_update(self, other):
         self.__ixor__(other)
     def __ixor__(self, other):
-        if self._pending_removals:
-            self._commit_removals()
         if self is other:
             self.data.clear()
         else:

Lib/argparse.py

@@ -18,11 +18,12 @@ command-line and writes the result to a file::
         'integers', metavar='int', nargs='+', type=int,
         help='an integer to be summed')
     parser.add_argument(
-        '--log', default=sys.stdout, type=argparse.FileType('w'),
+        '--log',
         help='the file where the sum should be written')
     args = parser.parse_args()
-    args.log.write('%s' % sum(args.integers))
-    args.log.close()
+    with (open(args.log, 'w') if args.log is not None
+          else contextlib.nullcontext(sys.stdout)) as log:
+        log.write('%s' % sum(args.integers))
 
 The module contains the following public classes:
 
@@ -39,7 +40,8 @@ The module contains the following public classes:
 
     - FileType -- A factory for defining types of files to be created. As the
         example above shows, instances of FileType are typically passed as
-        the type= argument of add_argument() calls.
+        the type= argument of add_argument() calls. Deprecated since
+        Python 3.14.
 
     - Action -- The base class for parser actions. Typically actions are
         selected by passing strings like 'store_true' or 'append_const' to
@@ -159,18 +161,21 @@ class HelpFormatter(object):
     provided by the class are considered an implementation detail.
     """
 
-    def __init__(self,
-                 prog,
-                 indent_increment=2,
-                 max_help_position=24,
-                 width=None):
-
+    def __init__(
+        self,
+        prog,
+        indent_increment=2,
+        max_help_position=24,
+        width=None,
+        color=True,
+    ):
         # default setting for width
         if width is None:
             import shutil
             width = shutil.get_terminal_size().columns
             width -= 2
 
+        self._set_color(color)
         self._prog = prog
         self._indent_increment = indent_increment
         self._max_help_position = min(max_help_position,
@@ -187,6 +192,16 @@ class HelpFormatter(object):
         self._whitespace_matcher = _re.compile(r'\s+', _re.ASCII)
         self._long_break_matcher = _re.compile(r'\n\n\n+')
 
+    def _set_color(self, color):
+        from _colorize import can_colorize, decolor, get_theme
+
+        if color and can_colorize():
+            self._theme = get_theme(force_color=True).argparse
+            self._decolor = decolor
+        else:
+            self._theme = get_theme(force_no_color=True).argparse
+            self._decolor = lambda text: text
+
     # ===============================
     # Section and indentation methods
     # ===============================
@@ -225,7 +240,11 @@ class HelpFormatter(object):
             if self.heading is not SUPPRESS and self.heading is not None:
                 current_indent = self.formatter._current_indent
                 heading_text = _('%(heading)s:') % dict(heading=self.heading)
-                heading = '%*s%s\n' % (current_indent, '', heading_text)
+                t = self.formatter._theme
+                heading = (
+                    f'{" " * current_indent}'
+                    f'{t.heading}{heading_text}{t.reset}\n'
+                )
             else:
                 heading = ''
 
@@ -262,7 +281,7 @@ class HelpFormatter(object):
         if action.help is not SUPPRESS:
 
             # find all invocations
-            get_invocation = self._format_action_invocation
+            get_invocation = lambda x: self._decolor(self._format_action_invocation(x))
             invocation_lengths = [len(get_invocation(action)) + self._current_indent]
             for subaction in self._iter_indented_subactions(action):
                 invocation_lengths.append(len(get_invocation(subaction)) + self._current_indent)
@@ -296,42 +315,39 @@ class HelpFormatter(object):
                         if part and part is not SUPPRESS])
 
     def _format_usage(self, usage, actions, groups, prefix):
+        t = self._theme
+
         if prefix is None:
             prefix = _('usage: ')
 
         # if usage is specified, use that
         if usage is not None:
-            usage = usage % dict(prog=self._prog)
+            usage = (
+                t.prog_extra
+                + usage
+                % {"prog": f"{t.prog}{self._prog}{t.reset}{t.prog_extra}"}
+                + t.reset
+            )
 
         # if no optionals or positionals are available, usage is just prog
         elif usage is None and not actions:
-            usage = '%(prog)s' % dict(prog=self._prog)
+            usage = f"{t.prog}{self._prog}{t.reset}"
 
         # if optionals and positionals are available, calculate usage
         elif usage is None:
             prog = '%(prog)s' % dict(prog=self._prog)
 
-            # split optionals from positionals
-            optionals = []
-            positionals = []
-            for action in actions:
-                if action.option_strings:
-                    optionals.append(action)
-                else:
-                    positionals.append(action)
-
+            parts, pos_start = self._get_actions_usage_parts(actions, groups)
             # build full usage string
-            format = self._format_actions_usage
-            action_usage = format(optionals + positionals, groups)
-            usage = ' '.join([s for s in [prog, action_usage] if s])
+            usage = ' '.join(filter(None, [prog, *parts]))
 
             # wrap the usage parts if it's too long
             text_width = self._width - self._current_indent
-            if len(prefix) + len(usage) > text_width:
+            if len(prefix) + len(self._decolor(usage)) > text_width:
 
                 # break usage into wrappable parts
-                opt_parts = self._get_actions_usage_parts(optionals, groups)
-                pos_parts = self._get_actions_usage_parts(positionals, groups)
+                opt_parts = parts[:pos_start]
+                pos_parts = parts[pos_start:]
 
                 # helper for wrapping lines
                 def get_lines(parts, indent, prefix=None):
@@ -343,12 +359,13 @@ class HelpFormatter(object):
                     else:
                         line_len = indent_length - 1
                     for part in parts:
-                        if line_len + 1 + len(part) > text_width and line:
+                        part_len = len(self._decolor(part))
+                        if line_len + 1 + part_len > text_width and line:
                             lines.append(indent + ' '.join(line))
                             line = []
                             line_len = indent_length - 1
                         line.append(part)
-                        line_len += len(part) + 1
+                        line_len += part_len + 1
                     if line:
                         lines.append(indent + ' '.join(line))
                     if prefix is not None:
@@ -356,8 +373,9 @@ class HelpFormatter(object):
                     return lines
 
                 # if prog is short, follow it with optionals or positionals
-                if len(prefix) + len(prog) <= 0.75 * text_width:
-                    indent = ' ' * (len(prefix) + len(prog) + 1)
+                prog_len = len(self._decolor(prog))
+                if len(prefix) + prog_len <= 0.75 * text_width:
+                    indent = ' ' * (len(prefix) + prog_len + 1)
                     if opt_parts:
                         lines = get_lines([prog] + opt_parts, indent, prefix)
                         lines.extend(get_lines(pos_parts, indent))
@@ -380,97 +398,120 @@ class HelpFormatter(object):
                 # join lines into usage
                 usage = '\n'.join(lines)
 
-        # prefix with 'usage:'
-        return '%s%s\n\n' % (prefix, usage)
+            usage = usage.removeprefix(prog)
+            usage = f"{t.prog}{prog}{t.reset}{usage}"
 
-    def _format_actions_usage(self, actions, groups):
-        return ' '.join(self._get_actions_usage_parts(actions, groups))
+        # prefix with 'usage:'
+        return f'{t.usage}{prefix}{t.reset}{usage}\n\n'
+
+    def _is_long_option(self, string):
+        return len(string) > 2
 
     def _get_actions_usage_parts(self, actions, groups):
-        # find group indices and identify actions in groups
-        group_actions = set()
-        inserts = {}
+        """Get usage parts with split index for optionals/positionals.
+
+        Returns (parts, pos_start) where pos_start is the index in parts
+        where positionals begin.
+        This preserves mutually exclusive group formatting across the
+        optionals/positionals boundary (gh-75949).
+        """
+        actions = [action for action in actions if action.help is not SUPPRESS]
+        # group actions by mutually exclusive groups
+        action_groups = dict.fromkeys(actions)
         for group in groups:
-            if not group._group_actions:
-                raise ValueError(f'empty group {group}')
-
-            if all(action.help is SUPPRESS for action in group._group_actions):
-                continue
-
-            try:
-                start = actions.index(group._group_actions[0])
-            except ValueError:
-                continue
-            else:
-                end = start + len(group._group_actions)
-                if actions[start:end] == group._group_actions:
-                    group_actions.update(group._group_actions)
-                    inserts[start, end] = group
+            for action in group._group_actions:
+                if action in action_groups:
+                    action_groups[action] = group
+        # positional arguments keep their position
+        positionals = []
+        for action in actions:
+            if not action.option_strings:
+                group = action_groups.pop(action)
+                if group:
+                    group_actions = [
+                        action2 for action2 in group._group_actions
+                        if action2.option_strings and
+                           action_groups.pop(action2, None)
+                    ] + [action]
+                    positionals.append((group.required, group_actions))
+                else:
+                    positionals.append((None, [action]))
+        # the remaining optional arguments are sorted by the position of
+        # the first option in the group
+        optionals = []
+        for action in actions:
+            if action.option_strings and action in action_groups:
+                group = action_groups.pop(action)
+                if group:
+                    group_actions = [action] + [
+                        action2 for action2 in group._group_actions
+                        if action2.option_strings and
+                           action_groups.pop(action2, None)
+                    ]
+                    optionals.append((group.required, group_actions))
+                else:
+                    optionals.append((None, [action]))
 
         # collect all actions format strings
         parts = []
-        for action in actions:
+        t = self._theme
+        pos_start = None
+        for i, (required, group) in enumerate(optionals + positionals):
+            start = len(parts)
+            if i == len(optionals):
+                pos_start = start
+            in_group = len(group) > 1
+            for action in group:
+                # produce all arg strings
+                if not action.option_strings:
+                    default = self._get_default_metavar_for_positional(action)
+                    part = self._format_args(action, default)
+                    # if it's in a group, strip the outer []
+                    if in_group:
+                        if part[0] == '[' and part[-1] == ']':
+                            part = part[1:-1]
+                    part = t.summary_action + part + t.reset
 
-            # suppressed arguments are marked with None
-            if action.help is SUPPRESS:
-                part = None
-
-            # produce all arg strings
-            elif not action.option_strings:
-                default = self._get_default_metavar_for_positional(action)
-                part = self._format_args(action, default)
-
-                # if it's in a group, strip the outer []
-                if action in group_actions:
-                    if part[0] == '[' and part[-1] == ']':
-                        part = part[1:-1]
-
-            # produce the first way to invoke the option in brackets
-            else:
-                option_string = action.option_strings[0]
-
-                # if the Optional doesn't take a value, format is:
-                #    -s or --long
-                if action.nargs == 0:
-                    part = action.format_usage()
-
-                # if the Optional takes a value, format is:
-                #    -s ARGS or --long ARGS
+                # produce the first way to invoke the option in brackets
                 else:
-                    default = self._get_default_metavar_for_optional(action)
-                    args_string = self._format_args(action, default)
-                    part = '%s %s' % (option_string, args_string)
+                    option_string = action.option_strings[0]
+                    if self._is_long_option(option_string):
+                        option_color = t.summary_long_option
+                    else:
+                        option_color = t.summary_short_option
 
-                # make it look optional if it's not required or in a group
-                if not action.required and action not in group_actions:
-                    part = '[%s]' % part
+                    # if the Optional doesn't take a value, format is:
+                    #    -s or --long
+                    if action.nargs == 0:
+                        part = action.format_usage()
+                        part = f"{option_color}{part}{t.reset}"
 
-            # add the action string to the list
-            parts.append(part)
+                    # if the Optional takes a value, format is:
+                    #    -s ARGS or --long ARGS
+                    else:
+                        default = self._get_default_metavar_for_optional(action)
+                        args_string = self._format_args(action, default)
+                        part = (
+                            f"{option_color}{option_string} "
+                            f"{t.summary_label}{args_string}{t.reset}"
+                        )
 
-        # group mutually exclusive actions
-        inserted_separators_indices = set()
-        for start, end in sorted(inserts, reverse=True):
-            group = inserts[start, end]
-            group_parts = [item for item in parts[start:end] if item is not None]
-            group_size = len(group_parts)
-            if group.required:
-                open, close = "()" if group_size > 1 else ("", "")
-            else:
-                open, close = "[]"
-            group_parts[0] = open + group_parts[0]
-            group_parts[-1] = group_parts[-1] + close
-            for i, part in enumerate(group_parts[:-1], start=start):
-                # insert a separator if not already done in a nested group
-                if i not in inserted_separators_indices:
-                    parts[i] = part + ' |'
-                    inserted_separators_indices.add(i)
-            parts[start + group_size - 1] = group_parts[-1]
-            for i in range(start + group_size, end):
-                parts[i] = None
+                    # make it look optional if it's not required or in a group
+                    if not (action.required or required or in_group):
+                        part = '[%s]' % part
 
-        # return the usage parts
-        return [item for item in parts if item is not None]
+                # add the action string to the list
+                parts.append(part)
+
+            if in_group:
+                parts[start] = ('(' if required else '[') + parts[start]
+                for i in range(start, len(parts) - 1):
+                    parts[i] += ' |'
+                parts[-1] += ')' if required else ']'
+
+        if pos_start is None:
+            pos_start = len(parts)
+        return parts, pos_start
 
     def _format_text(self, text):
         if '%(prog)' in text:
@@ -486,6 +527,7 @@ class HelpFormatter(object):
         help_width = max(self._width - help_position, 11)
         action_width = help_position - self._current_indent - 2
         action_header = self._format_action_invocation(action)
+        action_header_no_color = self._decolor(action_header)
 
         # no help; start on same line and add a final newline
         if not action.help:
@@ -493,9 +535,15 @@ class HelpFormatter(object):
             action_header = '%*s%s\n' % tup
 
         # short action name; start on the same line and pad two spaces
-        elif len(action_header) <= action_width:
-            tup = self._current_indent, '', action_width, action_header
+        elif len(action_header_no_color) <= action_width:
+            # calculate widths without color codes
+            action_header_color = action_header
+            tup = self._current_indent, '', action_width, action_header_no_color
             action_header = '%*s%-*s  ' % tup
+            # swap in the colored header
+            action_header = action_header.replace(
+                action_header_no_color, action_header_color
+            )
             indent_first = 0
 
         # long action name; start on the next line
@@ -528,23 +576,42 @@ class HelpFormatter(object):
         return self._join_parts(parts)
 
     def _format_action_invocation(self, action):
+        t = self._theme
+
         if not action.option_strings:
             default = self._get_default_metavar_for_positional(action)
-            return ' '.join(self._metavar_formatter(action, default)(1))
+            return (
+                t.action
+                + ' '.join(self._metavar_formatter(action, default)(1))
+                + t.reset
+            )
 
         else:
 
+            def color_option_strings(strings):
+                parts = []
+                for s in strings:
+                    if self._is_long_option(s):
+                        parts.append(f"{t.long_option}{s}{t.reset}")
+                    else:
+                        parts.append(f"{t.short_option}{s}{t.reset}")
+                return parts
+
             # if the Optional doesn't take a value, format is:
             #    -s, --long
             if action.nargs == 0:
-                return ', '.join(action.option_strings)
+                option_strings = color_option_strings(action.option_strings)
+                return ', '.join(option_strings)
 
             # if the Optional takes a value, format is:
             #    -s, --long ARGS
             else:
                 default = self._get_default_metavar_for_optional(action)
-                args_string = self._format_args(action, default)
-                return ', '.join(action.option_strings) + ' ' + args_string
+                option_strings = color_option_strings(action.option_strings)
+                args_string = (
+                    f"{t.label}{self._format_args(action, default)}{t.reset}"
+                )
+                return ', '.join(option_strings) + ' ' + args_string
 
     def _metavar_formatter(self, action, default_metavar):
         if action.metavar is not None:
@@ -590,16 +657,19 @@ class HelpFormatter(object):
         return result
 
     def _expand_help(self, action):
+        help_string = self._get_help_string(action)
+        if '%' not in help_string:
+            return help_string
         params = dict(vars(action), prog=self._prog)
         for name in list(params):
-            if params[name] is SUPPRESS:
+            value = params[name]
+            if value is SUPPRESS:
                 del params[name]
-        for name in list(params):
-            if hasattr(params[name], '__name__'):
-                params[name] = params[name].__name__
+            elif hasattr(value, '__name__'):
+                params[name] = value.__name__
         if params.get('choices') is not None:
             params['choices'] = ', '.join(map(str, params['choices']))
-        return self._get_help_string(action) % params
+        return help_string % params
 
     def _iter_indented_subactions(self, action):
         try:
@@ -669,11 +739,14 @@ class ArgumentDefaultsHelpFormatter(HelpFormatter):
         if help is None:
             help = ''
 
-        if '%(default)' not in help:
-            if action.default is not SUPPRESS:
-                defaulting_nargs = [OPTIONAL, ZERO_OR_MORE]
-                if action.option_strings or action.nargs in defaulting_nargs:
-                    help += _(' (default: %(default)s)')
+        if (
+            '%(default)' not in help
+            and action.default is not SUPPRESS
+            and not action.required
+        ):
+            defaulting_nargs = (OPTIONAL, ZERO_OR_MORE)
+            if action.option_strings or action.nargs in defaulting_nargs:
+                help += _(' (default: %(default)s)')
         return help
 
 
@@ -844,22 +917,16 @@ class Action(_AttributeHolder):
         return self.option_strings[0]
 
     def __call__(self, parser, namespace, values, option_string=None):
-        raise NotImplementedError(_('.__call__() not defined'))
+        raise NotImplementedError('.__call__() not defined')
 
 
-# FIXME: remove together with `BooleanOptionalAction` deprecated arguments.
-_deprecated_default = object()
-
 class BooleanOptionalAction(Action):
     def __init__(self,
                  option_strings,
                  dest,
                  default=None,
-                 type=_deprecated_default,
-                 choices=_deprecated_default,
                  required=False,
                  help=None,
-                 metavar=_deprecated_default,
                  deprecated=False):
 
         _option_strings = []
@@ -867,38 +934,19 @@ class BooleanOptionalAction(Action):
             _option_strings.append(option_string)
 
             if option_string.startswith('--'):
+                if option_string.startswith('--no-'):
+                    raise ValueError(f'invalid option name {option_string!r} '
+                                     f'for BooleanOptionalAction')
                 option_string = '--no-' + option_string[2:]
                 _option_strings.append(option_string)
 
-        # We need `_deprecated` special value to ban explicit arguments that
-        # match default value. Like:
-        #   parser.add_argument('-f', action=BooleanOptionalAction, type=int)
-        for field_name in ('type', 'choices', 'metavar'):
-            if locals()[field_name] is not _deprecated_default:
-                import warnings
-                warnings._deprecated(
-                    field_name,
-                    "{name!r} is deprecated as of Python 3.12 and will be "
-                    "removed in Python {remove}.",
-                    remove=(3, 14))
-
-        if type is _deprecated_default:
-            type = None
-        if choices is _deprecated_default:
-            choices = None
-        if metavar is _deprecated_default:
-            metavar = None
-
         super().__init__(
             option_strings=_option_strings,
             dest=dest,
             nargs=0,
             default=default,
-            type=type,
-            choices=choices,
             required=required,
             help=help,
-            metavar=metavar,
             deprecated=deprecated)
 
 
@@ -1180,6 +1228,7 @@ class _SubParsersAction(Action):
         self._name_parser_map = {}
         self._choices_actions = []
         self._deprecated = set()
+        self._color = True
 
         super(_SubParsersAction, self).__init__(
             option_strings=option_strings,
@@ -1195,23 +1244,30 @@ class _SubParsersAction(Action):
         if kwargs.get('prog') is None:
             kwargs['prog'] = '%s %s' % (self._prog_prefix, name)
 
+        # set color
+        if kwargs.get('color') is None:
+            kwargs['color'] = self._color
+
         aliases = kwargs.pop('aliases', ())
 
         if name in self._name_parser_map:
-            raise ArgumentError(self, _('conflicting subparser: %s') % name)
+            raise ValueError(f'conflicting subparser: {name}')
         for alias in aliases:
             if alias in self._name_parser_map:
-                raise ArgumentError(
-                    self, _('conflicting subparser alias: %s') % alias)
+                raise ValueError(f'conflicting subparser alias: {alias}')
 
         # create a pseudo-action to hold the choice help
         if 'help' in kwargs:
             help = kwargs.pop('help')
             choice_action = self._ChoicesPseudoAction(name, aliases, help)
             self._choices_actions.append(choice_action)
+        else:
+            choice_action = None
 
         # create the parser and add it to the map
         parser = self._parser_class(**kwargs)
+        if choice_action is not None:
+            parser._check_help(choice_action)
         self._name_parser_map[name] = parser
 
         # make parser available under aliases also
@@ -1276,7 +1332,7 @@ class _ExtendAction(_AppendAction):
 # ==============
 
 class FileType(object):
-    """Factory for creating file object types
+    """Deprecated factory for creating file object types
 
     Instances of FileType are typically passed as type= arguments to the
     ArgumentParser add_argument() method.
@@ -1293,6 +1349,12 @@ class FileType(object):
     """
 
     def __init__(self, mode='r', bufsize=-1, encoding=None, errors=None):
+        import warnings
+        warnings.warn(
+            "FileType is deprecated. Simply open files after parsing arguments.",
+            category=PendingDeprecationWarning,
+            stacklevel=2
+        )
         self._mode = mode
         self._bufsize = bufsize
         self._encoding = encoding
@@ -1396,7 +1458,7 @@ class _ActionsContainer(object):
         self._defaults = {}
 
         # determines whether an "option" looks like a negative number
-        self._negative_number_matcher = _re.compile(r'^-\d+$|^-\d*\.\d+$')
+        self._negative_number_matcher = _re.compile(r'-\.?\d')
 
         # whether or not there are any optionals that look like negative
         # numbers -- uses a list so it can be shared and edited
@@ -1449,7 +1511,8 @@ class _ActionsContainer(object):
         chars = self.prefix_chars
         if not args or len(args) == 1 and args[0][0] not in chars:
             if args and 'dest' in kwargs:
-                raise ValueError('dest supplied twice for positional argument')
+                raise TypeError('dest supplied twice for positional argument,'
+                                ' did you mean metavar?')
             kwargs = self._get_positional_kwargs(*args, **kwargs)
 
         # otherwise, we're adding an optional argument
@@ -1465,27 +1528,34 @@ class _ActionsContainer(object):
                 kwargs['default'] = self.argument_default
 
         # create the action object, and add it to the parser
+        action_name = kwargs.get('action')
         action_class = self._pop_action_class(kwargs)
         if not callable(action_class):
-            raise ValueError('unknown action "%s"' % (action_class,))
+            raise ValueError(f'unknown action {action_class!r}')
         action = action_class(**kwargs)
 
+        # raise an error if action for positional argument does not
+        # consume arguments
+        if not action.option_strings and action.nargs == 0:
+            raise ValueError(f'action {action_name!r} is not valid for positional arguments')
+
         # raise an error if the action type is not callable
         type_func = self._registry_get('type', action.type, action.type)
         if not callable(type_func):
-            raise ValueError('%r is not callable' % (type_func,))
+            raise TypeError(f'{type_func!r} is not callable')
 
         if type_func is FileType:
-            raise ValueError('%r is a FileType class object, instance of it'
-                             ' must be passed' % (type_func,))
+            raise TypeError(f'{type_func!r} is a FileType class object, '
+                            f'instance of it must be passed')
 
         # raise an error if the metavar does not match the type
-        if hasattr(self, "_get_formatter"):
+        if hasattr(self, "_get_validation_formatter"):
+            formatter = self._get_validation_formatter()
             try:
-                self._get_formatter()._format_args(action, None)
+                formatter._format_args(action, None)
             except TypeError:
                 raise ValueError("length of metavar tuple does not match nargs")
-
+        self._check_help(action)
         return self._add_action(action)
 
     def add_argument_group(self, *args, **kwargs):
@@ -1529,8 +1599,8 @@ class _ActionsContainer(object):
             if group.title in title_group_map:
                 # This branch could happen if a derived class added
                 # groups with duplicated titles in __init__
-                msg = _('cannot merge actions - two groups are named %r')
-                raise ValueError(msg % (group.title))
+                msg = f'cannot merge actions - two groups are named {group.title!r}'
+                raise ValueError(msg)
             title_group_map[group.title] = group
 
         # map each action to its group
@@ -1571,13 +1641,15 @@ class _ActionsContainer(object):
     def _get_positional_kwargs(self, dest, **kwargs):
         # make sure required is not specified
         if 'required' in kwargs:
-            msg = _("'required' is an invalid argument for positionals")
+            msg = "'required' is an invalid argument for positionals"
             raise TypeError(msg)
 
         # mark positional arguments as required if at least one is
         # always required
         nargs = kwargs.get('nargs')
-        if nargs not in [OPTIONAL, ZERO_OR_MORE, REMAINDER, SUPPRESS, 0]:
+        if nargs == 0:
+            raise ValueError('nargs for positionals must be != 0')
+        if nargs not in [OPTIONAL, ZERO_OR_MORE, REMAINDER, SUPPRESS]:
             kwargs['required'] = True
 
         # return the keyword arguments with no option strings
@@ -1590,11 +1662,9 @@ class _ActionsContainer(object):
         for option_string in args:
             # error on strings that don't start with an appropriate prefix
             if not option_string[0] in self.prefix_chars:
-                args = {'option': option_string,
-                        'prefix_chars': self.prefix_chars}
-                msg = _('invalid option string %(option)r: '
-                        'must start with a character %(prefix_chars)r')
-                raise ValueError(msg % args)
+                raise ValueError(
+                    f'invalid option string {option_string!r}: '
+                    f'must start with a character {self.prefix_chars!r}')
 
             # strings starting with two prefix characters are long options
             option_strings.append(option_string)
@@ -1610,8 +1680,8 @@ class _ActionsContainer(object):
                 dest_option_string = option_strings[0]
             dest = dest_option_string.lstrip(self.prefix_chars)
             if not dest:
-                msg = _('dest= is required for options like %r')
-                raise ValueError(msg % option_string)
+                msg = f'dest= is required for options like {option_string!r}'
+                raise TypeError(msg)
             dest = dest.replace('-', '_')
 
         # return the updated keyword arguments
@@ -1627,8 +1697,8 @@ class _ActionsContainer(object):
         try:
             return getattr(self, handler_func_name)
         except AttributeError:
-            msg = _('invalid conflict_resolution value: %r')
-            raise ValueError(msg % self.conflict_handler)
+            msg = f'invalid conflict_resolution value: {self.conflict_handler!r}'
+            raise ValueError(msg)
 
     def _check_conflict(self, action):
 
@@ -1667,10 +1737,26 @@ class _ActionsContainer(object):
             if not action.option_strings:
                 action.container._remove_action(action)
 
+    def _check_help(self, action):
+        if action.help and hasattr(self, "_get_validation_formatter"):
+            formatter = self._get_validation_formatter()
+            try:
+                formatter._expand_help(action)
+            except (ValueError, TypeError, KeyError) as exc:
+                raise ValueError('badly formed help string') from exc
+
 
 class _ArgumentGroup(_ActionsContainer):
 
     def __init__(self, container, title=None, description=None, **kwargs):
+        if 'prefix_chars' in kwargs:
+            import warnings
+            depr_msg = (
+                "The use of the undocumented 'prefix_chars' parameter in "
+                "ArgumentParser.add_argument_group() is deprecated."
+            )
+            warnings.warn(depr_msg, DeprecationWarning, stacklevel=3)
+
         # add any missing keyword arguments by checking the container
         update = kwargs.setdefault
         update('conflict_handler', container.conflict_handler)
@@ -1702,14 +1788,7 @@ class _ArgumentGroup(_ActionsContainer):
         self._group_actions.remove(action)
 
     def add_argument_group(self, *args, **kwargs):
-        import warnings
-        warnings.warn(
-            "Nesting argument groups is deprecated.",
-            category=DeprecationWarning,
-            stacklevel=2
-        )
-        return super().add_argument_group(*args, **kwargs)
-
+        raise ValueError('argument groups cannot be nested')
 
 class _MutuallyExclusiveGroup(_ArgumentGroup):
 
@@ -1720,7 +1799,7 @@ class _MutuallyExclusiveGroup(_ArgumentGroup):
 
     def _add_action(self, action):
         if action.required:
-            msg = _('mutually exclusive arguments must be optional')
+            msg = 'mutually exclusive arguments must be optional'
             raise ValueError(msg)
         action = self._container._add_action(action)
         self._group_actions.append(action)
@@ -1730,14 +1809,29 @@ class _MutuallyExclusiveGroup(_ArgumentGroup):
         self._container._remove_action(action)
         self._group_actions.remove(action)
 
-    def add_mutually_exclusive_group(self, *args, **kwargs):
-        import warnings
-        warnings.warn(
-            "Nesting mutually exclusive groups is deprecated.",
-            category=DeprecationWarning,
-            stacklevel=2
-        )
-        return super().add_mutually_exclusive_group(*args, **kwargs)
+    def add_mutually_exclusive_group(self, **kwargs):
+        raise ValueError('mutually exclusive groups cannot be nested')
+
+def _prog_name(prog=None):
+    if prog is not None:
+        return prog
+    arg0 = _sys.argv[0]
+    try:
+        modspec = _sys.modules['__main__'].__spec__
+    except (KeyError, AttributeError):
+        # possibly PYTHONSTARTUP or -X presite or other weird edge case
+        # no good answer here, so fall back to the default
+        modspec = None
+    if modspec is None:
+        # simple script
+        return _os.path.basename(arg0)
+    py = _os.path.basename(_sys.executable)
+    if modspec.name != '__main__':
+        # imported module or package
+        modname = modspec.name.removesuffix('.__main__')
+        return f'{py} -m {modname}'
+    # directory or ZIP file
+    return f'{py} {arg0}'
 
 
 class ArgumentParser(_AttributeHolder, _ActionsContainer):
@@ -1760,6 +1854,9 @@ class ArgumentParser(_AttributeHolder, _ActionsContainer):
         - allow_abbrev -- Allow long options to be abbreviated unambiguously
         - exit_on_error -- Determines whether or not ArgumentParser exits with
             error info when an error occurs
+        - suggest_on_error - Enables suggestions for mistyped argument choices
+            and subparser names (default: ``False``)
+        - color - Allow color output in help messages (default: ``False``)
     """
 
     def __init__(self,
@@ -1775,19 +1872,18 @@ class ArgumentParser(_AttributeHolder, _ActionsContainer):
                  conflict_handler='error',
                  add_help=True,
                  allow_abbrev=True,
-                 exit_on_error=True):
-
+                 exit_on_error=True,
+                 *,
+                 suggest_on_error=False,
+                 color=True,
+                 ):
         superinit = super(ArgumentParser, self).__init__
         superinit(description=description,
                   prefix_chars=prefix_chars,
                   argument_default=argument_default,
                   conflict_handler=conflict_handler)
 
-        # default setting for prog
-        if prog is None:
-            prog = _os.path.basename(_sys.argv[0])
-
-        self.prog = prog
+        self.prog = _prog_name(prog)
         self.usage = usage
         self.epilog = epilog
         self.formatter_class = formatter_class
@@ -1795,6 +1891,11 @@ class ArgumentParser(_AttributeHolder, _ActionsContainer):
         self.add_help = add_help
         self.allow_abbrev = allow_abbrev
         self.exit_on_error = exit_on_error
+        self.suggest_on_error = suggest_on_error
+        self.color = color
+
+        # Cached formatter for validation (avoids repeated _set_color calls)
+        self._cached_formatter = None
 
         add_group = self.add_argument_group
         self._positionals = add_group(_('positional arguments'))
@@ -1844,7 +1945,7 @@ class ArgumentParser(_AttributeHolder, _ActionsContainer):
 
     def add_subparsers(self, **kwargs):
         if self._subparsers is not None:
-            raise ArgumentError(None, _('cannot have multiple subparser arguments'))
+            raise ValueError('cannot have multiple subparser arguments')
 
         # add the parser class to the arguments if it's not present
         kwargs.setdefault('parser_class', type(self))
@@ -1859,15 +1960,19 @@ class ArgumentParser(_AttributeHolder, _ActionsContainer):
         # prog defaults to the usage message of this parser, skipping
         # optional arguments and with no "usage:" prefix
         if kwargs.get('prog') is None:
-            formatter = self._get_formatter()
+            # Create formatter without color to avoid storing ANSI codes in prog
+            formatter = self.formatter_class(prog=self.prog)
+            formatter._set_color(False)
             positionals = self._get_positional_actions()
             groups = self._mutually_exclusive_groups
-            formatter.add_usage(self.usage, positionals, groups, '')
+            formatter.add_usage(None, positionals, groups, '')
             kwargs['prog'] = formatter.format_help().strip()
 
         # create the parsers action and add it to the positionals list
         parsers_class = self._pop_action_class(kwargs, 'parsers')
         action = parsers_class(option_strings=[], **kwargs)
+        action._color = self.color
+        self._check_help(action)
         self._subparsers._add_action(action)
 
         # return the created parsers action
@@ -2498,7 +2603,6 @@ class ArgumentParser(_AttributeHolder, _ActionsContainer):
                 value = action.default
             if isinstance(value, str) and value is not SUPPRESS:
                 value = self._get_value(action, value)
-                self._check_value(action, value)
 
         # when nargs='*' on a positional, if there were no command-line
         # args, use the default if it is anything other than None
@@ -2506,11 +2610,8 @@ class ArgumentParser(_AttributeHolder, _ActionsContainer):
               not action.option_strings):
             if action.default is not None:
                 value = action.default
-                self._check_value(action, value)
             else:
-                # since arg_strings is always [] at this point
-                # there is no need to use self._check_value(action, value)
-                value = arg_strings
+                value = []
 
         # single argument or optional argument produces a single value
         elif len(arg_strings) == 1 and action.nargs in [None, OPTIONAL]:
@@ -2543,8 +2644,7 @@ class ArgumentParser(_AttributeHolder, _ActionsContainer):
     def _get_value(self, action, arg_string):
         type_func = self._registry_get('type', action.type, action.type)
         if not callable(type_func):
-            msg = _('%r is not callable')
-            raise ArgumentError(action, msg % type_func)
+            raise TypeError(f'{type_func!r} is not callable')
 
         # convert the value to the appropriate type
         try:
@@ -2568,14 +2668,27 @@ class ArgumentParser(_AttributeHolder, _ActionsContainer):
     def _check_value(self, action, value):
         # converted value must be one of the choices (if specified)
         choices = action.choices
-        if choices is not None:
-            if isinstance(choices, str):
-                choices = iter(choices)
-            if value not in choices:
-                args = {'value': str(value),
-                        'choices': ', '.join(map(str, action.choices))}
-                msg = _('invalid choice: %(value)r (choose from %(choices)s)')
-                raise ArgumentError(action, msg % args)
+        if choices is None:
+            return
+
+        if isinstance(choices, str):
+            choices = iter(choices)
+
+        if value not in choices:
+            args = {'value': str(value),
+                    'choices': ', '.join(map(str, action.choices))}
+            msg = _('invalid choice: %(value)r (choose from %(choices)s)')
+
+            if self.suggest_on_error and isinstance(value, str):
+                if all(isinstance(choice, str) for choice in action.choices):
+                    import difflib
+                    suggestions = difflib.get_close_matches(value, action.choices, 1)
+                    if suggestions:
+                        args['closest'] = suggestions[0]
+                        msg = _('invalid choice: %(value)r, maybe you meant %(closest)r? '
+                                '(choose from %(choices)s)')
+
+            raise ArgumentError(action, msg % args)
 
     # =======================
     # Help-formatting methods
@@ -2611,7 +2724,16 @@ class ArgumentParser(_AttributeHolder, _ActionsContainer):
         return formatter.format_help()
 
     def _get_formatter(self):
-        return self.formatter_class(prog=self.prog)
+        formatter = self.formatter_class(prog=self.prog)
+        formatter._set_color(self.color)
+        return formatter
+
+    def _get_validation_formatter(self):
+        # Return cached formatter for read-only validation operations
+        # (_expand_help and _format_args). Avoids repeated slow _set_color calls.
+        if self._cached_formatter is None:
+            self._cached_formatter = self._get_formatter()
+        return self._cached_formatter
 
     # =====================
     # Help-printing methods

Lib/asyncio/__init__.py

@@ -10,6 +10,7 @@ from .coroutines import *
 from .events import *
 from .exceptions import *
 from .futures import *
+from .graph import *
 from .locks import *
 from .protocols import *
 from .runners import *
@@ -27,6 +28,7 @@ __all__ = (base_events.__all__ +
            events.__all__ +
            exceptions.__all__ +
            futures.__all__ +
+           graph.__all__ +
            locks.__all__ +
            protocols.__all__ +
            runners.__all__ +
@@ -45,3 +47,28 @@ if sys.platform == 'win32':  # pragma: no cover
 else:
     from .unix_events import *  # pragma: no cover
     __all__ += unix_events.__all__
+
+def __getattr__(name: str):
+    import warnings
+
+    match name:
+        case "AbstractEventLoopPolicy":
+            warnings._deprecated(f"asyncio.{name}", remove=(3, 16))
+            return events._AbstractEventLoopPolicy
+        case "DefaultEventLoopPolicy":
+            warnings._deprecated(f"asyncio.{name}", remove=(3, 16))
+            if sys.platform == 'win32':
+                return windows_events._DefaultEventLoopPolicy
+            return unix_events._DefaultEventLoopPolicy
+        case "WindowsSelectorEventLoopPolicy":
+            if sys.platform == 'win32':
+                warnings._deprecated(f"asyncio.{name}", remove=(3, 16))
+                return windows_events._WindowsSelectorEventLoopPolicy
+            # Else fall through to the AttributeError below.
+        case "WindowsProactorEventLoopPolicy":
+            if sys.platform == 'win32':
+                warnings._deprecated(f"asyncio.{name}", remove=(3, 16))
+                return windows_events._WindowsProactorEventLoopPolicy
+            # Else fall through to the AttributeError below.
+
+    raise AttributeError(f"module {__name__!r} has no attribute {name!r}")

Lib/asyncio/__main__.py

@@ -1,41 +1,53 @@
+import argparse
 import ast
 import asyncio
-import code
+import asyncio.tools
 import concurrent.futures
+import contextvars
 import inspect
+import os
+import site
 import sys
 import threading
 import types
 import warnings
 
+from _colorize import get_theme
+from _pyrepl.console import InteractiveColoredConsole
+
 from . import futures
 
 
-class AsyncIOInteractiveConsole(code.InteractiveConsole):
+class AsyncIOInteractiveConsole(InteractiveColoredConsole):
 
     def __init__(self, locals, loop):
-        super().__init__(locals)
+        super().__init__(locals, filename="<stdin>")
         self.compile.compiler.flags |= ast.PyCF_ALLOW_TOP_LEVEL_AWAIT
 
         self.loop = loop
+        self.context = contextvars.copy_context()
 
     def runcode(self, code):
+        global return_code
         future = concurrent.futures.Future()
 
         def callback():
+            global return_code
             global repl_future
-            global repl_future_interrupted
+            global keyboard_interrupted
 
             repl_future = None
-            repl_future_interrupted = False
+            keyboard_interrupted = False
 
             func = types.FunctionType(code, self.locals)
             try:
                 coro = func()
-            except SystemExit:
-                raise
+            except SystemExit as se:
+                return_code = se.code
+                self.loop.stop()
+                return
             except KeyboardInterrupt as ex:
-                repl_future_interrupted = True
+                keyboard_interrupted = True
                 future.set_exception(ex)
                 return
             except BaseException as ex:
@@ -47,39 +59,72 @@ class AsyncIOInteractiveConsole(code.InteractiveConsole):
                 return
 
             try:
-                repl_future = self.loop.create_task(coro)
+                repl_future = self.loop.create_task(coro, context=self.context)
                 futures._chain_future(repl_future, future)
             except BaseException as exc:
                 future.set_exception(exc)
 
-        loop.call_soon_threadsafe(callback)
+        self.loop.call_soon_threadsafe(callback, context=self.context)
 
         try:
             return future.result()
-        except SystemExit:
-            raise
+        except SystemExit as se:
+            return_code = se.code
+            self.loop.stop()
+            return
         except BaseException:
-            if repl_future_interrupted:
-                self.write("\nKeyboardInterrupt\n")
+            if keyboard_interrupted:
+                if not CAN_USE_PYREPL:
+                    self.write("\nKeyboardInterrupt\n")
             else:
                 self.showtraceback()
-
+            return self.STATEMENT_FAILED
 
 class REPLThread(threading.Thread):
 
     def run(self):
+        global return_code
+
         try:
             banner = (
                 f'asyncio REPL {sys.version} on {sys.platform}\n'
                 f'Use "await" directly instead of "asyncio.run()".\n'
                 f'Type "help", "copyright", "credits" or "license" '
                 f'for more information.\n'
-                f'{getattr(sys, "ps1", ">>> ")}import asyncio'
             )
 
-            console.interact(
-                banner=banner,
-                exitmsg='exiting asyncio REPL...')
+            console.write(banner)
+
+            if startup_path := os.getenv("PYTHONSTARTUP"):
+                sys.audit("cpython.run_startup", startup_path)
+
+                import tokenize
+                with tokenize.open(startup_path) as f:
+                    startup_code = compile(f.read(), startup_path, "exec")
+                    exec(startup_code, console.locals)
+
+            ps1 = getattr(sys, "ps1", ">>> ")
+            if CAN_USE_PYREPL:
+                theme = get_theme().syntax
+                ps1 = f"{theme.prompt}{ps1}{theme.reset}"
+            console.write(f"{ps1}import asyncio\n")
+
+            if CAN_USE_PYREPL:
+                from _pyrepl.simple_interact import (
+                    run_multiline_interactive_console,
+                )
+                try:
+                    run_multiline_interactive_console(console)
+                except SystemExit:
+                    # expected via the `exit` and `quit` commands
+                    pass
+                except BaseException:
+                    # unexpected issue
+                    console.showtraceback()
+                    console.write("Internal error, ")
+                    return_code = 1
+            else:
+                console.interact(banner="", exitmsg="")
         finally:
             warnings.filterwarnings(
                 'ignore',
@@ -88,8 +133,56 @@ class REPLThread(threading.Thread):
 
             loop.call_soon_threadsafe(loop.stop)
 
+    def interrupt(self) -> None:
+        if not CAN_USE_PYREPL:
+            return
+
+        from _pyrepl.simple_interact import _get_reader
+        r = _get_reader()
+        if r.threading_hook is not None:
+            r.threading_hook.add("")  # type: ignore
+
 
 if __name__ == '__main__':
+    parser = argparse.ArgumentParser(
+        prog="python3 -m asyncio",
+        description="Interactive asyncio shell and CLI tools",
+        color=True,
+    )
+    subparsers = parser.add_subparsers(help="sub-commands", dest="command")
+    ps = subparsers.add_parser(
+        "ps", help="Display a table of all pending tasks in a process"
+    )
+    ps.add_argument("pid", type=int, help="Process ID to inspect")
+    pstree = subparsers.add_parser(
+        "pstree", help="Display a tree of all pending tasks in a process"
+    )
+    pstree.add_argument("pid", type=int, help="Process ID to inspect")
+    args = parser.parse_args()
+    match args.command:
+        case "ps":
+            asyncio.tools.display_awaited_by_tasks_table(args.pid)
+            sys.exit(0)
+        case "pstree":
+            asyncio.tools.display_awaited_by_tasks_tree(args.pid)
+            sys.exit(0)
+        case None:
+            pass  # continue to the interactive shell
+        case _:
+            # shouldn't happen as an invalid command-line wouldn't parse
+            # but let's keep it for the next person adding a command
+            print(f"error: unhandled command {args.command}", file=sys.stderr)
+            parser.print_usage(file=sys.stderr)
+            sys.exit(1)
+
+    sys.audit("cpython.run_stdin")
+
+    if os.getenv('PYTHON_BASIC_REPL'):
+        CAN_USE_PYREPL = False
+    else:
+        from _pyrepl.main import CAN_USE_PYREPL
+
+    return_code = 0
     loop = asyncio.new_event_loop()
     asyncio.set_event_loop(loop)
 
@@ -102,14 +195,31 @@ if __name__ == '__main__':
     console = AsyncIOInteractiveConsole(repl_locals, loop)
 
     repl_future = None
-    repl_future_interrupted = False
+    keyboard_interrupted = False
 
     try:
         import readline  # NoQA
     except ImportError:
-        pass
+        readline = None
 
-    repl_thread = REPLThread()
+    interactive_hook = getattr(sys, "__interactivehook__", None)
+
+    if interactive_hook is not None:
+        sys.audit("cpython.run_interactivehook", interactive_hook)
+        interactive_hook()
+
+    if interactive_hook is site.register_readline:
+        # Fix the completer function to use the interactive console locals
+        try:
+            import rlcompleter
+        except:
+            pass
+        else:
+            if readline is not None:
+                completer = rlcompleter.Completer(console.locals)
+                readline.set_completer(completer.complete)
+
+    repl_thread = REPLThread(name="Interactive thread")
     repl_thread.daemon = True
     repl_thread.start()
 
@@ -117,9 +227,13 @@ if __name__ == '__main__':
         try:
             loop.run_forever()
         except KeyboardInterrupt:
+            keyboard_interrupted = True
             if repl_future and not repl_future.done():
                 repl_future.cancel()
-                repl_future_interrupted = True
+            repl_thread.interrupt()
             continue
         else:
             break
+
+    console.write('exiting asyncio REPL...\n')
+    sys.exit(return_code)

Lib/asyncio/base_events.py

@@ -17,7 +17,6 @@ import collections
 import collections.abc
 import concurrent.futures
 import errno
-import functools
 import heapq
 import itertools
 import os
@@ -279,7 +278,9 @@ class Server(events.AbstractServer):
                  ssl_handshake_timeout, ssl_shutdown_timeout=None):
         self._loop = loop
         self._sockets = sockets
-        self._active_count = 0
+        # Weak references so we don't break Transport's ability to
+        # detect abandoned transports
+        self._clients = weakref.WeakSet()
         self._waiters = []
         self._protocol_factory = protocol_factory
         self._backlog = backlog
@@ -292,14 +293,13 @@ class Server(events.AbstractServer):
     def __repr__(self):
         return f'<{self.__class__.__name__} sockets={self.sockets!r}>'
 
-    def _attach(self):
+    def _attach(self, transport):
         assert self._sockets is not None
-        self._active_count += 1
+        self._clients.add(transport)
 
-    def _detach(self):
-        assert self._active_count > 0
-        self._active_count -= 1
-        if self._active_count == 0 and self._sockets is None:
+    def _detach(self, transport):
+        self._clients.discard(transport)
+        if len(self._clients) == 0 and self._sockets is None:
             self._wakeup()
 
     def _wakeup(self):
@@ -348,9 +348,17 @@ class Server(events.AbstractServer):
             self._serving_forever_fut.cancel()
             self._serving_forever_fut = None
 
-        if self._active_count == 0:
+        if len(self._clients) == 0:
             self._wakeup()
 
+    def close_clients(self):
+        for transport in self._clients.copy():
+            transport.close()
+
+    def abort_clients(self):
+        for transport in self._clients.copy():
+            transport.abort()
+
     async def start_serving(self):
         self._start_serving()
         # Skip one loop iteration so that all 'loop.add_reader'
@@ -422,6 +430,8 @@ class BaseEventLoop(events.AbstractEventLoop):
         self._clock_resolution = time.get_clock_info('monotonic').resolution
         self._exception_handler = None
         self.set_debug(coroutines._is_debug_mode())
+        # The preserved state of async generator hooks.
+        self._old_agen_hooks = None
         # In debug mode, if the execution of a callback or a step of a task
         # exceed this duration in seconds, the slow callback/task is logged.
         self.slow_callback_duration = 0.1
@@ -448,26 +458,24 @@ class BaseEventLoop(events.AbstractEventLoop):
         """Create a Future object attached to the loop."""
         return futures.Future(loop=self)
 
-    def create_task(self, coro, *, name=None, context=None):
-        """Schedule a coroutine object.
+    def create_task(self, coro, **kwargs):
+        """Schedule or begin executing a coroutine object.
 
         Return a task object.
         """
         self._check_closed()
-        if self._task_factory is None:
-            task = tasks.Task(coro, loop=self, name=name, context=context)
-            if task._source_traceback:
-                del task._source_traceback[-1]
-        else:
-            if context is None:
-                # Use legacy API if context is not needed
-                task = self._task_factory(self, coro)
-            else:
-                task = self._task_factory(self, coro, context=context)
+        if self._task_factory is not None:
+            return self._task_factory(self, coro, **kwargs)
 
-            tasks._set_task_name(task, name)
-
-        return task
+        task = tasks.Task(coro, loop=self, **kwargs)
+        if task._source_traceback:
+            del task._source_traceback[-1]
+        try:
+            return task
+        finally:
+            # gh-128552: prevent a refcycle of
+            # task.exception().__traceback__->BaseEventLoop.create_task->task
+            del task
 
     def set_task_factory(self, factory):
         """Set a task factory that will be used by loop.create_task().
@@ -475,9 +483,10 @@ class BaseEventLoop(events.AbstractEventLoop):
         If factory is None the default task factory will be set.
 
         If factory is a callable, it should have a signature matching
-        '(loop, coro)', where 'loop' will be a reference to the active
-        event loop, 'coro' will be a coroutine object.  The callable
-        must return a Future.
+        '(loop, coro, **kwargs)', where 'loop' will be a reference to the active
+        event loop, 'coro' will be a coroutine object, and **kwargs will be
+        arbitrary keyword arguments that should be passed on to Task.
+        The callable must return a Task.
         """
         if factory is not None and not callable(factory):
             raise TypeError('task factory must be a callable or None')
@@ -624,29 +633,52 @@ class BaseEventLoop(events.AbstractEventLoop):
             raise RuntimeError(
                 'Cannot run the event loop while another loop is running')
 
-    def run_forever(self):
-        """Run until stop() is called."""
+    def _run_forever_setup(self):
+        """Prepare the run loop to process events.
+
+        This method exists so that custom event loop subclasses (e.g., event loops
+        that integrate a GUI event loop with Python's event loop) have access to all the
+        loop setup logic.
+        """
         self._check_closed()
         self._check_running()
         self._set_coroutine_origin_tracking(self._debug)
 
-        old_agen_hooks = sys.get_asyncgen_hooks()
-        try:
-            self._thread_id = threading.get_ident()
-            sys.set_asyncgen_hooks(firstiter=self._asyncgen_firstiter_hook,
-                                   finalizer=self._asyncgen_finalizer_hook)
+        self._old_agen_hooks = sys.get_asyncgen_hooks()
+        self._thread_id = threading.get_ident()
+        sys.set_asyncgen_hooks(
+            firstiter=self._asyncgen_firstiter_hook,
+            finalizer=self._asyncgen_finalizer_hook
+        )
 
-            events._set_running_loop(self)
+        events._set_running_loop(self)
+
+    def _run_forever_cleanup(self):
+        """Clean up after an event loop finishes the looping over events.
+
+        This method exists so that custom event loop subclasses (e.g., event loops
+        that integrate a GUI event loop with Python's event loop) have access to all the
+        loop cleanup logic.
+        """
+        self._stopping = False
+        self._thread_id = None
+        events._set_running_loop(None)
+        self._set_coroutine_origin_tracking(False)
+        # Restore any pre-existing async generator hooks.
+        if self._old_agen_hooks is not None:
+            sys.set_asyncgen_hooks(*self._old_agen_hooks)
+            self._old_agen_hooks = None
+
+    def run_forever(self):
+        """Run until stop() is called."""
+        self._run_forever_setup()
+        try:
             while True:
                 self._run_once()
                 if self._stopping:
                     break
         finally:
-            self._stopping = False
-            self._thread_id = None
-            events._set_running_loop(None)
-            self._set_coroutine_origin_tracking(False)
-            sys.set_asyncgen_hooks(*old_agen_hooks)
+            self._run_forever_cleanup()
 
     def run_until_complete(self, future):
         """Run until the Future is done.
@@ -803,7 +835,7 @@ class BaseEventLoop(events.AbstractEventLoop):
 
     def _check_callback(self, callback, method):
         if (coroutines.iscoroutine(callback) or
-                coroutines.iscoroutinefunction(callback)):
+                coroutines._iscoroutinefunction(callback)):
             raise TypeError(
                 f"coroutines cannot be used with {method}()")
         if not callable(callback):
@@ -840,7 +872,10 @@ class BaseEventLoop(events.AbstractEventLoop):
         self._check_closed()
         if self._debug:
             self._check_callback(callback, 'call_soon_threadsafe')
-        handle = self._call_soon(callback, args, context)
+        handle = events._ThreadSafeHandle(callback, args, self, context)
+        self._ready.append(handle)
+        if handle._source_traceback:
+            del handle._source_traceback[-1]
         if handle._source_traceback:
             del handle._source_traceback[-1]
         self._write_to_self()
@@ -981,39 +1016,43 @@ class BaseEventLoop(events.AbstractEventLoop):
         family, type_, proto, _, address = addr_info
         sock = None
         try:
-            sock = socket.socket(family=family, type=type_, proto=proto)
-            sock.setblocking(False)
-            if local_addr_infos is not None:
-                for lfamily, _, _, _, laddr in local_addr_infos:
-                    # skip local addresses of different family
-                    if lfamily != family:
-                        continue
-                    try:
-                        sock.bind(laddr)
-                        break
-                    except OSError as exc:
-                        msg = (
-                            f'error while attempting to bind on '
-                            f'address {laddr!r}: '
-                            f'{exc.strerror.lower()}'
-                        )
-                        exc = OSError(exc.errno, msg)
-                        my_exceptions.append(exc)
-                else:  # all bind attempts failed
-                    if my_exceptions:
-                        raise my_exceptions.pop()
-                    else:
-                        raise OSError(f"no matching local address with {family=} found")
-            await self.sock_connect(sock, address)
-            return sock
-        except OSError as exc:
-            my_exceptions.append(exc)
-            if sock is not None:
-                sock.close()
-            raise
+            try:
+                sock = socket.socket(family=family, type=type_, proto=proto)
+                sock.setblocking(False)
+                if local_addr_infos is not None:
+                    for lfamily, _, _, _, laddr in local_addr_infos:
+                        # skip local addresses of different family
+                        if lfamily != family:
+                            continue
+                        try:
+                            sock.bind(laddr)
+                            break
+                        except OSError as exc:
+                            msg = (
+                                f'error while attempting to bind on '
+                                f'address {laddr!r}: {str(exc).lower()}'
+                            )
+                            exc = OSError(exc.errno, msg)
+                            my_exceptions.append(exc)
+                    else:  # all bind attempts failed
+                        if my_exceptions:
+                            raise my_exceptions.pop()
+                        else:
+                            raise OSError(f"no matching local address with {family=} found")
+                await self.sock_connect(sock, address)
+                return sock
+            except OSError as exc:
+                my_exceptions.append(exc)
+                raise
         except:
             if sock is not None:
-                sock.close()
+                try:
+                    sock.close()
+                except OSError:
+                    # An error when closing a newly created socket is
+                    # not important, but it can overwrite more important
+                    # non-OSError error. So ignore it.
+                    pass
             raise
         finally:
             exceptions = my_exceptions = None
@@ -1107,11 +1146,18 @@ class BaseEventLoop(events.AbstractEventLoop):
                     except OSError:
                         continue
             else:  # using happy eyeballs
-                sock, _, _ = await staggered.staggered_race(
-                    (functools.partial(self._connect_sock,
-                                       exceptions, addrinfo, laddr_infos)
-                     for addrinfo in infos),
-                    happy_eyeballs_delay, loop=self)
+                sock = (await staggered.staggered_race(
+                    (
+                        # can't use functools.partial as it keeps a reference
+                        # to exceptions
+                        lambda addrinfo=addrinfo: self._connect_sock(
+                            exceptions, addrinfo, laddr_infos
+                        )
+                        for addrinfo in infos
+                    ),
+                    happy_eyeballs_delay,
+                    loop=self,
+                ))[0]  # can't use sock, _, _ as it keeks a reference to exceptions
 
             if sock is None:
                 exceptions = [exc for sub in exceptions for exc in sub]
@@ -1120,7 +1166,7 @@ class BaseEventLoop(events.AbstractEventLoop):
                         raise ExceptionGroup("create_connection failed", exceptions)
                     if len(exceptions) == 1:
                         raise exceptions[0]
-                    else:
+                    elif exceptions:
                         # If they all have the same str(), raise one.
                         model = str(exceptions[0])
                         if all(str(exc) == model for exc in exceptions):
@@ -1129,6 +1175,9 @@ class BaseEventLoop(events.AbstractEventLoop):
                         # the various error messages.
                         raise OSError('Multiple exceptions: {}'.format(
                             ', '.join(str(exc) for exc in exceptions)))
+                    else:
+                        # No exceptions were collected, raise a timeout error
+                        raise TimeoutError('create_connection failed')
                 finally:
                     exceptions = None
 
@@ -1254,8 +1303,8 @@ class BaseEventLoop(events.AbstractEventLoop):
                 read = await self.run_in_executor(None, file.readinto, view)
                 if not read:
                     return total_sent  # EOF
-                await proto.drain()
                 transp.write(view[:read])
+                await proto.drain()
                 total_sent += read
         finally:
             if total_sent > 0 and hasattr(file, 'seek'):
@@ -1474,6 +1523,7 @@ class BaseEventLoop(events.AbstractEventLoop):
             ssl=None,
             reuse_address=None,
             reuse_port=None,
+            keep_alive=None,
             ssl_handshake_timeout=None,
             ssl_shutdown_timeout=None,
             start_serving=True):
@@ -1545,8 +1595,13 @@ class BaseEventLoop(events.AbstractEventLoop):
                     if reuse_address:
                         sock.setsockopt(
                             socket.SOL_SOCKET, socket.SO_REUSEADDR, True)
-                    if reuse_port:
+                    # Since Linux 6.12.9, SO_REUSEPORT is not allowed
+                    # on other address families than AF_INET/AF_INET6.
+                    if reuse_port and af in (socket.AF_INET, socket.AF_INET6):
                         _set_reuseport(sock)
+                    if keep_alive:
+                        sock.setsockopt(
+                            socket.SOL_SOCKET, socket.SO_KEEPALIVE, True)
                     # Disable IPv4/IPv6 dual stack support (enabled by
                     # default on Linux) which makes a single socket
                     # listen on both address families.
@@ -1561,7 +1616,7 @@ class BaseEventLoop(events.AbstractEventLoop):
                     except OSError as err:
                         msg = ('error while attempting '
                                'to bind on address %r: %s'
-                               % (sa, err.strerror.lower()))
+                               % (sa, str(err).lower()))
                         if err.errno == errno.EADDRNOTAVAIL:
                             # Assume the family is not enabled (bpo-30945)
                             sockets.pop()
@@ -1619,8 +1674,7 @@ class BaseEventLoop(events.AbstractEventLoop):
             raise ValueError(
                 'ssl_shutdown_timeout is only meaningful with ssl')
 
-        if sock is not None:
-            _check_ssl_socket(sock)
+        _check_ssl_socket(sock)
 
         transport, protocol = await self._create_connection_transport(
             sock, protocol_factory, ssl, '', server_side=True,
@@ -1833,6 +1887,8 @@ class BaseEventLoop(events.AbstractEventLoop):
         - 'protocol' (optional): Protocol instance;
         - 'transport' (optional): Transport instance;
         - 'socket' (optional): Socket instance;
+        - 'source_traceback' (optional): Traceback of the source;
+        - 'handle_traceback' (optional): Traceback of the handle;
         - 'asyncgen' (optional): Asynchronous generator that caused
                                  the exception.
 
@@ -1943,8 +1999,11 @@ class BaseEventLoop(events.AbstractEventLoop):
             timeout = 0
         elif self._scheduled:
             # Compute the desired timeout.
-            when = self._scheduled[0]._when
-            timeout = min(max(0, when - self.time()), MAXIMUM_SELECT_TIMEOUT)
+            timeout = self._scheduled[0]._when - self.time()
+            if timeout > MAXIMUM_SELECT_TIMEOUT:
+                timeout = MAXIMUM_SELECT_TIMEOUT
+            elif timeout < 0:
+                timeout = 0
 
         event_list = self._selector.select(timeout)
         self._process_events(event_list)

Lib/asyncio/base_subprocess.py

@@ -1,6 +1,9 @@
 import collections
 import subprocess
 import warnings
+import os
+import signal
+import sys
 
 from . import protocols
 from . import transports
@@ -23,6 +26,7 @@ class BaseSubprocessTransport(transports.SubprocessTransport):
         self._pending_calls = collections.deque()
         self._pipes = {}
         self._finished = False
+        self._pipes_connected = False
 
         if stdin == subprocess.PIPE:
             self._pipes[0] = None
@@ -101,7 +105,12 @@ class BaseSubprocessTransport(transports.SubprocessTransport):
         for proto in self._pipes.values():
             if proto is None:
                 continue
-            proto.pipe.close()
+            # See gh-114177
+            # skip closing the pipe if loop is already closed
+            # this can happen e.g. when loop is closed immediately after
+            # process is killed
+            if self._loop and not self._loop.is_closed():
+                proto.pipe.close()
 
         if (self._proc is not None and
                 # has the child process finished?
@@ -115,7 +124,8 @@ class BaseSubprocessTransport(transports.SubprocessTransport):
 
             try:
                 self._proc.kill()
-            except ProcessLookupError:
+            except (ProcessLookupError, PermissionError):
+                # the process may have already exited or may be running setuid
                 pass
 
             # Don't clear the _proc reference yet: _post_init() may still run
@@ -141,17 +151,31 @@ class BaseSubprocessTransport(transports.SubprocessTransport):
         if self._proc is None:
             raise ProcessLookupError()
 
-    def send_signal(self, signal):
-        self._check_proc()
-        self._proc.send_signal(signal)
+    if sys.platform == 'win32':
+        def send_signal(self, signal):
+            self._check_proc()
+            self._proc.send_signal(signal)
 
-    def terminate(self):
-        self._check_proc()
-        self._proc.terminate()
+        def terminate(self):
+            self._check_proc()
+            self._proc.terminate()
 
-    def kill(self):
-        self._check_proc()
-        self._proc.kill()
+        def kill(self):
+            self._check_proc()
+            self._proc.kill()
+    else:
+        def send_signal(self, signal):
+            self._check_proc()
+            try:
+                os.kill(self._proc.pid, signal)
+            except ProcessLookupError:
+                pass
+
+        def terminate(self):
+            self.send_signal(signal.SIGTERM)
+
+        def kill(self):
+            self.send_signal(signal.SIGKILL)
 
     async def _connect_pipes(self, waiter):
         try:
@@ -190,6 +214,7 @@ class BaseSubprocessTransport(transports.SubprocessTransport):
         else:
             if waiter is not None and not waiter.cancelled():
                 waiter.set_result(None)
+            self._pipes_connected = True
 
     def _call(self, cb, *data):
         if self._pending_calls is not None:
@@ -233,6 +258,15 @@ class BaseSubprocessTransport(transports.SubprocessTransport):
         assert not self._finished
         if self._returncode is None:
             return
+        if not self._pipes_connected:
+            # self._pipes_connected can be False if not all pipes were connected
+            # because either the process failed to start or the self._connect_pipes task
+            # got cancelled. In this broken state we consider all pipes disconnected and
+            # to avoid hanging forever in self._wait as otherwise _exit_waiters
+            # would never be woken up, we wake them up here.
+            for waiter in self._exit_waiters:
+                if not waiter.cancelled():
+                    waiter.set_result(self._returncode)
         if all(p is not None and p.disconnected
                for p in self._pipes.values()):
             self._finished = True

Lib/asyncio/coroutines.py

@@ -18,7 +18,16 @@ _is_coroutine = object()
 
 
 def iscoroutinefunction(func):
+    import warnings
     """Return True if func is a decorated coroutine function."""
+    warnings._deprecated("asyncio.iscoroutinefunction",
+                         f"{warnings._DEPRECATED_MSG}; "
+                         "use inspect.iscoroutinefunction() instead",
+                         remove=(3,16))
+    return _iscoroutinefunction(func)
+
+
+def _iscoroutinefunction(func):
     return (inspect.iscoroutinefunction(func) or
             getattr(func, '_is_coroutine', None) is _is_coroutine)
 

Lib/asyncio/events.py

@@ -5,14 +5,18 @@
 # SPDX-FileCopyrightText: Copyright (c) 2015-2021 MagicStack Inc.  http://magic.io
 
 __all__ = (
-    'AbstractEventLoopPolicy',
-    'AbstractEventLoop', 'AbstractServer',
-    'Handle', 'TimerHandle',
-    'get_event_loop_policy', 'set_event_loop_policy',
-    'get_event_loop', 'set_event_loop', 'new_event_loop',
-    'get_child_watcher', 'set_child_watcher',
-    '_set_running_loop', 'get_running_loop',
-    '_get_running_loop',
+    "AbstractEventLoop",
+    "AbstractServer",
+    "Handle",
+    "TimerHandle",
+    "get_event_loop_policy",
+    "set_event_loop_policy",
+    "get_event_loop",
+    "set_event_loop",
+    "new_event_loop",
+    "_set_running_loop",
+    "get_running_loop",
+    "_get_running_loop",
 )
 
 import contextvars
@@ -22,6 +26,7 @@ import socket
 import subprocess
 import sys
 import threading
+import warnings
 
 from . import format_helpers
 
@@ -54,7 +59,8 @@ class Handle:
             info.append('cancelled')
         if self._callback is not None:
             info.append(format_helpers._format_callback_source(
-                self._callback, self._args))
+                self._callback, self._args,
+                debug=self._loop.get_debug()))
         if self._source_traceback:
             frame = self._source_traceback[-1]
             info.append(f'created at {frame[0]}:{frame[1]}')
@@ -90,7 +96,8 @@ class Handle:
             raise
         except BaseException as exc:
             cb = format_helpers._format_callback_source(
-                self._callback, self._args)
+                self._callback, self._args,
+                debug=self._loop.get_debug())
             msg = f'Exception in callback {cb}'
             context = {
                 'message': msg,
@@ -102,6 +109,34 @@ class Handle:
             self._loop.call_exception_handler(context)
         self = None  # Needed to break cycles when an exception occurs.
 
+# _ThreadSafeHandle is used for callbacks scheduled with call_soon_threadsafe
+# and is thread safe unlike Handle which is not thread safe.
+class _ThreadSafeHandle(Handle):
+
+    __slots__ = ('_lock',)
+
+    def __init__(self, callback, args, loop, context=None):
+        super().__init__(callback, args, loop, context)
+        self._lock = threading.RLock()
+
+    def cancel(self):
+        with self._lock:
+            return super().cancel()
+
+    def cancelled(self):
+        with self._lock:
+            return super().cancelled()
+
+    def _run(self):
+        # The event loop checks for cancellation without holding the lock
+        # It is possible that the handle is cancelled after the check
+        # but before the callback is called so check it again after acquiring
+        # the lock and return without calling the callback if it is cancelled.
+        with self._lock:
+            if self._cancelled:
+                return
+            return super()._run()
+
 
 class TimerHandle(Handle):
     """Object returned by timed callback registration methods."""
@@ -173,6 +208,14 @@ class AbstractServer:
         """Stop serving.  This leaves existing connections open."""
         raise NotImplementedError
 
+    def close_clients(self):
+        """Close all active connections."""
+        raise NotImplementedError
+
+    def abort_clients(self):
+        """Close all active connections immediately."""
+        raise NotImplementedError
+
     def get_loop(self):
         """Get the event loop the Server object is attached to."""
         raise NotImplementedError
@@ -282,7 +325,7 @@ class AbstractEventLoop:
 
     # Method scheduling a coroutine object: create a task.
 
-    def create_task(self, coro, *, name=None, context=None):
+    def create_task(self, coro, **kwargs):
         raise NotImplementedError
 
     # Methods for interacting with threads.
@@ -320,6 +363,7 @@ class AbstractEventLoop:
             *, family=socket.AF_UNSPEC,
             flags=socket.AI_PASSIVE, sock=None, backlog=100,
             ssl=None, reuse_address=None, reuse_port=None,
+            keep_alive=None,
             ssl_handshake_timeout=None,
             ssl_shutdown_timeout=None,
             start_serving=True):
@@ -358,6 +402,9 @@ class AbstractEventLoop:
         they all set this flag when being created. This option is not
         supported on Windows.
 
+        keep_alive set to True keeps connections active by enabling the
+        periodic transmission of messages.
+
         ssl_handshake_timeout is the time in seconds that an SSL server
         will wait for completion of the SSL handshake before aborting the
         connection. Default is 60s.
@@ -615,7 +662,7 @@ class AbstractEventLoop:
         raise NotImplementedError
 
 
-class AbstractEventLoopPolicy:
+class _AbstractEventLoopPolicy:
     """Abstract policy for accessing the event loop."""
 
     def get_event_loop(self):
@@ -638,18 +685,7 @@ class AbstractEventLoopPolicy:
         the current context, set_event_loop must be called explicitly."""
         raise NotImplementedError
 
-    # Child processes handling (Unix only).
-
-    def get_child_watcher(self):
-        "Get the watcher for child processes."
-        raise NotImplementedError
-
-    def set_child_watcher(self, watcher):
-        """Set the watcher for child processes."""
-        raise NotImplementedError
-
-
-class BaseDefaultEventLoopPolicy(AbstractEventLoopPolicy):
+class _BaseDefaultEventLoopPolicy(_AbstractEventLoopPolicy):
     """Default policy implementation for accessing the event loop.
 
     In this policy, each thread has its own event loop.  However, we
@@ -666,7 +702,6 @@ class BaseDefaultEventLoopPolicy(AbstractEventLoopPolicy):
 
     class _Local(threading.local):
         _loop = None
-        _set_called = False
 
     def __init__(self):
         self._local = self._Local()
@@ -676,28 +711,6 @@ class BaseDefaultEventLoopPolicy(AbstractEventLoopPolicy):
 
         Returns an instance of EventLoop or raises an exception.
         """
-        if (self._local._loop is None and
-                not self._local._set_called and
-                threading.current_thread() is threading.main_thread()):
-            stacklevel = 2
-            try:
-                f = sys._getframe(1)
-            except AttributeError:
-                pass
-            else:
-                # Move up the call stack so that the warning is attached
-                # to the line outside asyncio itself.
-                while f:
-                    module = f.f_globals.get('__name__')
-                    if not (module == 'asyncio' or module.startswith('asyncio.')):
-                        break
-                    f = f.f_back
-                    stacklevel += 1
-            import warnings
-            warnings.warn('There is no current event loop',
-                          DeprecationWarning, stacklevel=stacklevel)
-            self.set_event_loop(self.new_event_loop())
-
         if self._local._loop is None:
             raise RuntimeError('There is no current event loop in thread %r.'
                                % threading.current_thread().name)
@@ -706,7 +719,6 @@ class BaseDefaultEventLoopPolicy(AbstractEventLoopPolicy):
 
     def set_event_loop(self, loop):
         """Set the event loop."""
-        self._local._set_called = True
         if loop is not None and not isinstance(loop, AbstractEventLoop):
             raise TypeError(f"loop must be an instance of AbstractEventLoop or None, not '{type(loop).__name__}'")
         self._local._loop = loop
@@ -776,26 +788,35 @@ def _init_event_loop_policy():
     global _event_loop_policy
     with _lock:
         if _event_loop_policy is None:  # pragma: no branch
-            from . import DefaultEventLoopPolicy
-            _event_loop_policy = DefaultEventLoopPolicy()
+            if sys.platform == 'win32':
+                from .windows_events import _DefaultEventLoopPolicy
+            else:
+                from .unix_events import _DefaultEventLoopPolicy
+            _event_loop_policy = _DefaultEventLoopPolicy()
 
 
-def get_event_loop_policy():
+def _get_event_loop_policy():
     """Get the current event loop policy."""
     if _event_loop_policy is None:
         _init_event_loop_policy()
     return _event_loop_policy
 
+def get_event_loop_policy():
+    warnings._deprecated('asyncio.get_event_loop_policy', remove=(3, 16))
+    return _get_event_loop_policy()
 
-def set_event_loop_policy(policy):
+def _set_event_loop_policy(policy):
     """Set the current event loop policy.
 
     If policy is None, the default policy is restored."""
     global _event_loop_policy
-    if policy is not None and not isinstance(policy, AbstractEventLoopPolicy):
+    if policy is not None and not isinstance(policy, _AbstractEventLoopPolicy):
         raise TypeError(f"policy must be an instance of AbstractEventLoopPolicy or None, not '{type(policy).__name__}'")
     _event_loop_policy = policy
 
+def set_event_loop_policy(policy):
+    warnings._deprecated('asyncio.set_event_loop_policy', remove=(3,16))
+    _set_event_loop_policy(policy)
 
 def get_event_loop():
     """Return an asyncio event loop.
@@ -810,28 +831,17 @@ def get_event_loop():
     current_loop = _get_running_loop()
     if current_loop is not None:
         return current_loop
-    return get_event_loop_policy().get_event_loop()
+    return _get_event_loop_policy().get_event_loop()
 
 
 def set_event_loop(loop):
     """Equivalent to calling get_event_loop_policy().set_event_loop(loop)."""
-    get_event_loop_policy().set_event_loop(loop)
+    _get_event_loop_policy().set_event_loop(loop)
 
 
 def new_event_loop():
     """Equivalent to calling get_event_loop_policy().new_event_loop()."""
-    return get_event_loop_policy().new_event_loop()
-
-
-def get_child_watcher():
-    """Equivalent to calling get_event_loop_policy().get_child_watcher()."""
-    return get_event_loop_policy().get_child_watcher()
-
-
-def set_child_watcher(watcher):
-    """Equivalent to calling
-    get_event_loop_policy().set_child_watcher(watcher)."""
-    return get_event_loop_policy().set_child_watcher(watcher)
+    return _get_event_loop_policy().new_event_loop()
 
 
 # Alias pure-Python implementations for testing purposes.
@@ -861,7 +871,7 @@ if hasattr(os, 'fork'):
     def on_fork():
         # Reset the loop and wakeupfd in the forked child process.
         if _event_loop_policy is not None:
-            _event_loop_policy._local = BaseDefaultEventLoopPolicy._Local()
+            _event_loop_policy._local = _BaseDefaultEventLoopPolicy._Local()
         _set_running_loop(None)
         signal.set_wakeup_fd(-1)
 

Lib/asyncio/format_helpers.py

@@ -19,19 +19,26 @@ def _get_function_source(func):
     return None
 
 
-def _format_callback_source(func, args):
-    func_repr = _format_callback(func, args, None)
+def _format_callback_source(func, args, *, debug=False):
+    func_repr = _format_callback(func, args, None, debug=debug)
     source = _get_function_source(func)
     if source:
         func_repr += f' at {source[0]}:{source[1]}'
     return func_repr
 
 
-def _format_args_and_kwargs(args, kwargs):
+def _format_args_and_kwargs(args, kwargs, *, debug=False):
     """Format function arguments and keyword arguments.
 
     Special case for a single parameter: ('hello',) is formatted as ('hello').
+
+    Note that this function only returns argument details when
+    debug=True is specified, as arguments may contain sensitive
+    information.
     """
+    if not debug:
+        return '()'
+
     # use reprlib to limit the length of the output
     items = []
     if args:
@@ -41,10 +48,11 @@ def _format_args_and_kwargs(args, kwargs):
     return '({})'.format(', '.join(items))
 
 
-def _format_callback(func, args, kwargs, suffix=''):
+def _format_callback(func, args, kwargs, *, debug=False, suffix=''):
     if isinstance(func, functools.partial):
-        suffix = _format_args_and_kwargs(args, kwargs) + suffix
-        return _format_callback(func.func, func.args, func.keywords, suffix)
+        suffix = _format_args_and_kwargs(args, kwargs, debug=debug) + suffix
+        return _format_callback(func.func, func.args, func.keywords,
+                                debug=debug, suffix=suffix)
 
     if hasattr(func, '__qualname__') and func.__qualname__:
         func_repr = func.__qualname__
@@ -53,7 +61,7 @@ def _format_callback(func, args, kwargs, suffix=''):
     else:
         func_repr = repr(func)
 
-    func_repr += _format_args_and_kwargs(args, kwargs)
+    func_repr += _format_args_and_kwargs(args, kwargs, debug=debug)
     if suffix:
         func_repr += suffix
     return func_repr

Lib/asyncio/futures.py

@@ -2,6 +2,7 @@
 
 __all__ = (
     'Future', 'wrap_future', 'isfuture',
+    'future_add_to_awaited_by', 'future_discard_from_awaited_by',
 )
 
 import concurrent.futures
@@ -43,7 +44,6 @@ class Future:
     - This class is not compatible with the wait() and as_completed()
       methods in the concurrent.futures package.
 
-    (In Python 3.4 or later we may be able to unify the implementations.)
     """
 
     # Class variables serving as defaults for instance variables.
@@ -61,12 +61,15 @@ class Future:
     #   the Future protocol (i.e. is intended to be duck-type compatible).
     #   The value must also be not-None, to enable a subclass to declare
     #   that it is not compatible by setting this to None.
-    # - It is set by __iter__() below so that Task._step() can tell
+    # - It is set by __iter__() below so that Task.__step() can tell
     #   the difference between
-    #   `await Future()` or`yield from Future()` (correct) vs.
+    #   `await Future()` or `yield from Future()` (correct) vs.
     #   `yield Future()` (incorrect).
     _asyncio_future_blocking = False
 
+    # Used by the capture_call_stack() API.
+    __asyncio_awaited_by = None
+
     __log_traceback = False
 
     def __init__(self, *, loop=None):
@@ -116,6 +119,12 @@ class Future:
             raise ValueError('_log_traceback can only be set to False')
         self.__log_traceback = False
 
+    @property
+    def _asyncio_awaited_by(self):
+        if self.__asyncio_awaited_by is None:
+            return None
+        return frozenset(self.__asyncio_awaited_by)
+
     def get_loop(self):
         """Return the event loop the Future is bound to."""
         loop = self._loop
@@ -138,9 +147,6 @@ class Future:
             exc = exceptions.CancelledError()
         else:
             exc = exceptions.CancelledError(self._cancel_message)
-        exc.__context__ = self._cancelled_exc
-        # Remove the reference since we don't need this anymore.
-        self._cancelled_exc = None
         return exc
 
     def cancel(self, msg=None):
@@ -194,8 +200,7 @@ class Future:
         the future is done and has an exception set, this exception is raised.
         """
         if self._state == _CANCELLED:
-            exc = self._make_cancelled_error()
-            raise exc
+            raise self._make_cancelled_error()
         if self._state != _FINISHED:
             raise exceptions.InvalidStateError('Result is not ready.')
         self.__log_traceback = False
@@ -212,8 +217,7 @@ class Future:
         InvalidStateError.
         """
         if self._state == _CANCELLED:
-            exc = self._make_cancelled_error()
-            raise exc
+            raise self._make_cancelled_error()
         if self._state != _FINISHED:
             raise exceptions.InvalidStateError('Exception is not set.')
         self.__log_traceback = False
@@ -272,9 +276,13 @@ class Future:
             raise exceptions.InvalidStateError(f'{self._state}: {self!r}')
         if isinstance(exception, type):
             exception = exception()
-        if type(exception) is StopIteration:
-            raise TypeError("StopIteration interacts badly with generators "
-                            "and cannot be raised into a Future")
+        if isinstance(exception, StopIteration):
+            new_exc = RuntimeError("StopIteration interacts badly with "
+                                   "generators and cannot be raised into a "
+                                   "Future")
+            new_exc.__cause__ = exception
+            new_exc.__context__ = exception
+            exception = new_exc
         self._exception = exception
         self._exception_tb = exception.__traceback__
         self._state = _FINISHED
@@ -318,11 +326,9 @@ def _set_result_unless_cancelled(fut, result):
 def _convert_future_exc(exc):
     exc_class = type(exc)
     if exc_class is concurrent.futures.CancelledError:
-        return exceptions.CancelledError(*exc.args)
-    elif exc_class is concurrent.futures.TimeoutError:
-        return exceptions.TimeoutError(*exc.args)
+        return exceptions.CancelledError(*exc.args).with_traceback(exc.__traceback__)
     elif exc_class is concurrent.futures.InvalidStateError:
-        return exceptions.InvalidStateError(*exc.args)
+        return exceptions.InvalidStateError(*exc.args).with_traceback(exc.__traceback__)
     else:
         return exc
 
@@ -419,6 +425,49 @@ def wrap_future(future, *, loop=None):
     return new_future
 
 
+def future_add_to_awaited_by(fut, waiter, /):
+    """Record that `fut` is awaited on by `waiter`."""
+    # For the sake of keeping the implementation minimal and assuming
+    # that most of asyncio users use the built-in Futures and Tasks
+    # (or their subclasses), we only support native Future objects
+    # and their subclasses.
+    #
+    # Longer version: tracking requires storing the caller-callee
+    # dependency somewhere. One obvious choice is to store that
+    # information right in the future itself in a dedicated attribute.
+    # This means that we'd have to require all duck-type compatible
+    # futures to implement a specific attribute used by asyncio for
+    # the book keeping. Another solution would be to store that in
+    # a global dictionary. The downside here is that that would create
+    # strong references and any scenario where the "add" call isn't
+    # followed by a "discard" call would lead to a memory leak.
+    # Using WeakDict would resolve that issue, but would complicate
+    # the C code (_asynciomodule.c). The bottom line here is that
+    # it's not clear that all this work would be worth the effort.
+    #
+    # Note that there's an accelerated version of this function
+    # shadowing this implementation later in this file.
+    if isinstance(fut, _PyFuture) and isinstance(waiter, _PyFuture):
+        if fut._Future__asyncio_awaited_by is None:
+            fut._Future__asyncio_awaited_by = set()
+        fut._Future__asyncio_awaited_by.add(waiter)
+
+
+def future_discard_from_awaited_by(fut, waiter, /):
+    """Record that `fut` is no longer awaited on by `waiter`."""
+    # See the comment in "future_add_to_awaited_by()" body for
+    # details on implementation.
+    #
+    # Note that there's an accelerated version of this function
+    # shadowing this implementation later in this file.
+    if isinstance(fut, _PyFuture) and isinstance(waiter, _PyFuture):
+        if fut._Future__asyncio_awaited_by is not None:
+            fut._Future__asyncio_awaited_by.discard(waiter)
+
+
+_py_future_add_to_awaited_by = future_add_to_awaited_by
+_py_future_discard_from_awaited_by = future_discard_from_awaited_by
+
 try:
     import _asyncio
 except ImportError:
@@ -426,3 +475,7 @@ except ImportError:
 else:
     # _CFuture is needed for tests.
     Future = _CFuture = _asyncio.Future
+    future_add_to_awaited_by = _asyncio.future_add_to_awaited_by
+    future_discard_from_awaited_by = _asyncio.future_discard_from_awaited_by
+    _c_future_add_to_awaited_by = future_add_to_awaited_by
+    _c_future_discard_from_awaited_by = future_discard_from_awaited_by

Lib/asyncio/graph.py

@@ -0,0 +1,276 @@
+"""Introspection utils for tasks call graphs."""
+
+import dataclasses
+import io
+import sys
+import types
+
+from . import events
+from . import futures
+from . import tasks
+
+__all__ = (
+    'capture_call_graph',
+    'format_call_graph',
+    'print_call_graph',
+    'FrameCallGraphEntry',
+    'FutureCallGraph',
+)
+
+# Sadly, we can't re-use the traceback module's datastructures as those
+# are tailored for error reporting, whereas we need to represent an
+# async call graph.
+#
+# Going with pretty verbose names as we'd like to export them to the
+# top level asyncio namespace, and want to avoid future name clashes.
+
+
+@dataclasses.dataclass(frozen=True, slots=True)
+class FrameCallGraphEntry:
+    frame: types.FrameType
+
+
+@dataclasses.dataclass(frozen=True, slots=True)
+class FutureCallGraph:
+    future: futures.Future
+    call_stack: tuple["FrameCallGraphEntry", ...]
+    awaited_by: tuple["FutureCallGraph", ...]
+
+
+def _build_graph_for_future(
+    future: futures.Future,
+    *,
+    limit: int | None = None,
+) -> FutureCallGraph:
+    if not isinstance(future, futures.Future):
+        raise TypeError(
+            f"{future!r} object does not appear to be compatible "
+            f"with asyncio.Future"
+        )
+
+    coro = None
+    if get_coro := getattr(future, 'get_coro', None):
+        coro = get_coro() if limit != 0 else None
+
+    st: list[FrameCallGraphEntry] = []
+    awaited_by: list[FutureCallGraph] = []
+
+    while coro is not None:
+        if hasattr(coro, 'cr_await'):
+            # A native coroutine or duck-type compatible iterator
+            st.append(FrameCallGraphEntry(coro.cr_frame))
+            coro = coro.cr_await
+        elif hasattr(coro, 'ag_await'):
+            # A native async generator or duck-type compatible iterator
+            st.append(FrameCallGraphEntry(coro.cr_frame))
+            coro = coro.ag_await
+        else:
+            break
+
+    if future._asyncio_awaited_by:
+        for parent in future._asyncio_awaited_by:
+            awaited_by.append(_build_graph_for_future(parent, limit=limit))
+
+    if limit is not None:
+        if limit > 0:
+            st = st[:limit]
+        elif limit < 0:
+            st = st[limit:]
+    st.reverse()
+    return FutureCallGraph(future, tuple(st), tuple(awaited_by))
+
+
+def capture_call_graph(
+    future: futures.Future | None = None,
+    /,
+    *,
+    depth: int = 1,
+    limit: int | None = None,
+) -> FutureCallGraph | None:
+    """Capture the async call graph for the current task or the provided Future.
+
+    The graph is represented with three data structures:
+
+    * FutureCallGraph(future, call_stack, awaited_by)
+
+      Where 'future' is an instance of asyncio.Future or asyncio.Task.
+
+      'call_stack' is a tuple of FrameGraphEntry objects.
+
+      'awaited_by' is a tuple of FutureCallGraph objects.
+
+    * FrameCallGraphEntry(frame)
+
+      Where 'frame' is a frame object of a regular Python function
+      in the call stack.
+
+    Receives an optional 'future' argument. If not passed,
+    the current task will be used. If there's no current task, the function
+    returns None.
+
+    If "capture_call_graph()" is introspecting *the current task*, the
+    optional keyword-only 'depth' argument can be used to skip the specified
+    number of frames from top of the stack.
+
+    If the optional keyword-only 'limit' argument is provided, each call stack
+    in the resulting graph is truncated to include at most ``abs(limit)``
+    entries. If 'limit' is positive, the entries left are the closest to
+    the invocation point. If 'limit' is negative, the topmost entries are
+    left. If 'limit' is omitted or None, all entries are present.
+    If 'limit' is 0, the call stack is not captured at all, only
+    "awaited by" information is present.
+    """
+
+    loop = events._get_running_loop()
+
+    if future is not None:
+        # Check if we're in a context of a running event loop;
+        # if yes - check if the passed future is the currently
+        # running task or not.
+        if loop is None or future is not tasks.current_task(loop=loop):
+            return _build_graph_for_future(future, limit=limit)
+        # else: future is the current task, move on.
+    else:
+        if loop is None:
+            raise RuntimeError(
+                'capture_call_graph() is called outside of a running '
+                'event loop and no *future* to introspect was provided')
+        future = tasks.current_task(loop=loop)
+
+    if future is None:
+        # This isn't a generic call stack introspection utility. If we
+        # can't determine the current task and none was provided, we
+        # just return.
+        return None
+
+    if not isinstance(future, futures.Future):
+        raise TypeError(
+            f"{future!r} object does not appear to be compatible "
+            f"with asyncio.Future"
+        )
+
+    call_stack: list[FrameCallGraphEntry] = []
+
+    f = sys._getframe(depth) if limit != 0 else None
+    try:
+        while f is not None:
+            is_async = f.f_generator is not None
+            call_stack.append(FrameCallGraphEntry(f))
+
+            if is_async:
+                if f.f_back is not None and f.f_back.f_generator is None:
+                    # We've reached the bottom of the coroutine stack, which
+                    # must be the Task that runs it.
+                    break
+
+            f = f.f_back
+    finally:
+        del f
+
+    awaited_by = []
+    if future._asyncio_awaited_by:
+        for parent in future._asyncio_awaited_by:
+            awaited_by.append(_build_graph_for_future(parent, limit=limit))
+
+    if limit is not None:
+        limit *= -1
+        if limit > 0:
+            call_stack = call_stack[:limit]
+        elif limit < 0:
+            call_stack = call_stack[limit:]
+
+    return FutureCallGraph(future, tuple(call_stack), tuple(awaited_by))
+
+
+def format_call_graph(
+    future: futures.Future | None = None,
+    /,
+    *,
+    depth: int = 1,
+    limit: int | None = None,
+) -> str:
+    """Return the async call graph as a string for `future`.
+
+    If `future` is not provided, format the call graph for the current task.
+    """
+
+    def render_level(st: FutureCallGraph, buf: list[str], level: int) -> None:
+        def add_line(line: str) -> None:
+            buf.append(level * '    ' + line)
+
+        if isinstance(st.future, tasks.Task):
+            add_line(
+                f'* Task(name={st.future.get_name()!r}, id={id(st.future):#x})'
+            )
+        else:
+            add_line(
+                f'* Future(id={id(st.future):#x})'
+            )
+
+        if st.call_stack:
+            add_line(
+                f'  + Call stack:'
+            )
+            for ste in st.call_stack:
+                f = ste.frame
+
+                if f.f_generator is None:
+                    f = ste.frame
+                    add_line(
+                        f'  |   File {f.f_code.co_filename!r},'
+                        f' line {f.f_lineno}, in'
+                        f' {f.f_code.co_qualname}()'
+                    )
+                else:
+                    c = f.f_generator
+
+                    try:
+                        f = c.cr_frame
+                        code = c.cr_code
+                        tag = 'async'
+                    except AttributeError:
+                        try:
+                            f = c.ag_frame
+                            code = c.ag_code
+                            tag = 'async generator'
+                        except AttributeError:
+                            f = c.gi_frame
+                            code = c.gi_code
+                            tag = 'generator'
+
+                    add_line(
+                        f'  |   File {f.f_code.co_filename!r},'
+                        f' line {f.f_lineno}, in'
+                        f' {tag} {code.co_qualname}()'
+                    )
+
+        if st.awaited_by:
+            add_line(
+                f'  + Awaited by:'
+            )
+            for fut in st.awaited_by:
+                render_level(fut, buf, level + 1)
+
+    graph = capture_call_graph(future, depth=depth + 1, limit=limit)
+    if graph is None:
+        return ""
+
+    buf: list[str] = []
+    try:
+        render_level(graph, buf, 0)
+    finally:
+        # 'graph' has references to frames so we should
+        # make sure it's GC'ed as soon as we don't need it.
+        del graph
+    return '\n'.join(buf)
+
+def print_call_graph(
+    future: futures.Future | None = None,
+    /,
+    *,
+    file: io.Writer[str] | None = None,
+    depth: int = 1,
+    limit: int | None = None,
+) -> None:
+    """Print the async call graph for the current task or the provided Future."""
+    print(format_call_graph(future, depth=depth, limit=limit), file=file)

Lib/asyncio/locks.py

@@ -24,25 +24,23 @@ class Lock(_ContextManagerMixin, mixins._LoopBoundMixin):
     """Primitive lock objects.
 
     A primitive lock is a synchronization primitive that is not owned
-    by a particular coroutine when locked.  A primitive lock is in one
+    by a particular task when locked.  A primitive lock is in one
     of two states, 'locked' or 'unlocked'.
 
     It is created in the unlocked state.  It has two basic methods,
     acquire() and release().  When the state is unlocked, acquire()
     changes the state to locked and returns immediately.  When the
     state is locked, acquire() blocks until a call to release() in
-    another coroutine changes it to unlocked, then the acquire() call
+    another task changes it to unlocked, then the acquire() call
     resets it to locked and returns.  The release() method should only
     be called in the locked state; it changes the state to unlocked
     and returns immediately.  If an attempt is made to release an
     unlocked lock, a RuntimeError will be raised.
 
-    When more than one coroutine is blocked in acquire() waiting for
-    the state to turn to unlocked, only one coroutine proceeds when a
-    release() call resets the state to unlocked; first coroutine which
-    is blocked in acquire() is being processed.
-
-    acquire() is a coroutine and should be called with 'await'.
+    When more than one task is blocked in acquire() waiting for
+    the state to turn to unlocked, only one task proceeds when a
+    release() call resets the state to unlocked; successive release()
+    calls will unblock tasks in FIFO order.
 
     Locks also support the asynchronous context management protocol.
     'async with lock' statement should be used.
@@ -95,6 +93,8 @@ class Lock(_ContextManagerMixin, mixins._LoopBoundMixin):
         This method blocks until the lock is unlocked, then sets it to
         locked and returns True.
         """
+        # Implement fair scheduling, where thread always waits
+        # its turn. Jumping the queue if all are cancelled is an optimization.
         if (not self._locked and (self._waiters is None or
                 all(w.cancelled() for w in self._waiters))):
             self._locked = True
@@ -105,19 +105,22 @@ class Lock(_ContextManagerMixin, mixins._LoopBoundMixin):
         fut = self._get_loop().create_future()
         self._waiters.append(fut)
 
-        # Finally block should be called before the CancelledError
-        # handling as we don't want CancelledError to call
-        # _wake_up_first() and attempt to wake up itself.
         try:
             try:
                 await fut
             finally:
                 self._waiters.remove(fut)
         except exceptions.CancelledError:
+            # Currently the only exception designed be able to occur here.
+
+            # Ensure the lock invariant: If lock is not claimed (or about
+            # to be claimed by us) and there is a Task in waiters,
+            # ensure that the Task at the head will run.
             if not self._locked:
                 self._wake_up_first()
             raise
 
+        # assert self._locked is False
         self._locked = True
         return True
 
@@ -125,7 +128,7 @@ class Lock(_ContextManagerMixin, mixins._LoopBoundMixin):
         """Release a lock.
 
         When the lock is locked, reset it to unlocked, and return.
-        If any other coroutines are blocked waiting for the lock to become
+        If any other tasks are blocked waiting for the lock to become
         unlocked, allow exactly one of them to proceed.
 
         When invoked on an unlocked lock, a RuntimeError is raised.
@@ -139,7 +142,7 @@ class Lock(_ContextManagerMixin, mixins._LoopBoundMixin):
             raise RuntimeError('Lock is not acquired.')
 
     def _wake_up_first(self):
-        """Wake up the first waiter if it isn't done."""
+        """Ensure that the first waiter will wake up."""
         if not self._waiters:
             return
         try:
@@ -147,9 +150,7 @@ class Lock(_ContextManagerMixin, mixins._LoopBoundMixin):
         except StopIteration:
             return
 
-        # .done() necessarily means that a waiter will wake up later on and
-        # either take the lock, or, if it was cancelled and lock wasn't
-        # taken already, will hit this again and wake up a new waiter.
+        # .done() means that the waiter is already set to wake up.
         if not fut.done():
             fut.set_result(True)
 
@@ -179,8 +180,8 @@ class Event(mixins._LoopBoundMixin):
         return self._value
 
     def set(self):
-        """Set the internal flag to true. All coroutines waiting for it to
-        become true are awakened. Coroutine that call wait() once the flag is
+        """Set the internal flag to true. All tasks waiting for it to
+        become true are awakened. Tasks that call wait() once the flag is
         true will not block at all.
         """
         if not self._value:
@@ -191,7 +192,7 @@ class Event(mixins._LoopBoundMixin):
                     fut.set_result(True)
 
     def clear(self):
-        """Reset the internal flag to false. Subsequently, coroutines calling
+        """Reset the internal flag to false. Subsequently, tasks calling
         wait() will block until set() is called to set the internal flag
         to true again."""
         self._value = False
@@ -200,7 +201,7 @@ class Event(mixins._LoopBoundMixin):
         """Block until the internal flag is true.
 
         If the internal flag is true on entry, return True
-        immediately.  Otherwise, block until another coroutine calls
+        immediately.  Otherwise, block until another task calls
         set() to set the flag to true, then return True.
         """
         if self._value:
@@ -219,8 +220,8 @@ class Condition(_ContextManagerMixin, mixins._LoopBoundMixin):
     """Asynchronous equivalent to threading.Condition.
 
     This class implements condition variable objects. A condition variable
-    allows one or more coroutines to wait until they are notified by another
-    coroutine.
+    allows one or more tasks to wait until they are notified by another
+    task.
 
     A new Lock object is created and used as the underlying lock.
     """
@@ -247,45 +248,64 @@ class Condition(_ContextManagerMixin, mixins._LoopBoundMixin):
     async def wait(self):
         """Wait until notified.
 
-        If the calling coroutine has not acquired the lock when this
+        If the calling task has not acquired the lock when this
         method is called, a RuntimeError is raised.
 
         This method releases the underlying lock, and then blocks
         until it is awakened by a notify() or notify_all() call for
-        the same condition variable in another coroutine.  Once
+        the same condition variable in another task.  Once
         awakened, it re-acquires the lock and returns True.
+
+        This method may return spuriously,
+        which is why the caller should always
+        re-check the state and be prepared to wait() again.
         """
         if not self.locked():
             raise RuntimeError('cannot wait on un-acquired lock')
 
+        fut = self._get_loop().create_future()
         self.release()
         try:
-            fut = self._get_loop().create_future()
-            self._waiters.append(fut)
             try:
-                await fut
-                return True
-            finally:
-                self._waiters.remove(fut)
-
-        finally:
-            # Must reacquire lock even if wait is cancelled
-            cancelled = False
-            while True:
+                self._waiters.append(fut)
                 try:
-                    await self.acquire()
-                    break
-                except exceptions.CancelledError:
-                    cancelled = True
+                    await fut
+                    return True
+                finally:
+                    self._waiters.remove(fut)
 
-            if cancelled:
-                raise exceptions.CancelledError
+            finally:
+                # Must re-acquire lock even if wait is cancelled.
+                # We only catch CancelledError here, since we don't want any
+                # other (fatal) errors with the future to cause us to spin.
+                err = None
+                while True:
+                    try:
+                        await self.acquire()
+                        break
+                    except exceptions.CancelledError as e:
+                        err = e
+
+                if err is not None:
+                    try:
+                        raise err  # Re-raise most recent exception instance.
+                    finally:
+                        err = None  # Break reference cycles.
+        except BaseException:
+            # Any error raised out of here _may_ have occurred after this Task
+            # believed to have been successfully notified.
+            # Make sure to notify another Task instead.  This may result
+            # in a "spurious wakeup", which is allowed as part of the
+            # Condition Variable protocol.
+            self._notify(1)
+            raise
 
     async def wait_for(self, predicate):
         """Wait until a predicate becomes true.
 
-        The predicate should be a callable which result will be
-        interpreted as a boolean value.  The final predicate value is
+        The predicate should be a callable whose result will be
+        interpreted as a boolean value.  The method will repeatedly
+        wait() until it evaluates to true.  The final predicate value is
         the return value.
         """
         result = predicate()
@@ -295,20 +315,22 @@ class Condition(_ContextManagerMixin, mixins._LoopBoundMixin):
         return result
 
     def notify(self, n=1):
-        """By default, wake up one coroutine waiting on this condition, if any.
-        If the calling coroutine has not acquired the lock when this method
+        """By default, wake up one task waiting on this condition, if any.
+        If the calling task has not acquired the lock when this method
         is called, a RuntimeError is raised.
 
-        This method wakes up at most n of the coroutines waiting for the
-        condition variable; it is a no-op if no coroutines are waiting.
+        This method wakes up n of the tasks waiting for the condition
+         variable; if fewer than n are waiting, they are all awoken.
 
-        Note: an awakened coroutine does not actually return from its
+        Note: an awakened task does not actually return from its
         wait() call until it can reacquire the lock. Since notify() does
         not release the lock, its caller should.
         """
         if not self.locked():
             raise RuntimeError('cannot notify on un-acquired lock')
+        self._notify(n)
 
+    def _notify(self, n):
         idx = 0
         for fut in self._waiters:
             if idx >= n:
@@ -319,9 +341,9 @@ class Condition(_ContextManagerMixin, mixins._LoopBoundMixin):
                 fut.set_result(False)
 
     def notify_all(self):
-        """Wake up all threads waiting on this condition. This method acts
-        like notify(), but wakes up all waiting threads instead of one. If the
-        calling thread has not acquired the lock when this method is called,
+        """Wake up all tasks waiting on this condition. This method acts
+        like notify(), but wakes up all waiting tasks instead of one. If the
+        calling task has not acquired the lock when this method is called,
         a RuntimeError is raised.
         """
         self.notify(len(self._waiters))
@@ -357,6 +379,7 @@ class Semaphore(_ContextManagerMixin, mixins._LoopBoundMixin):
 
     def locked(self):
         """Returns True if semaphore cannot be acquired immediately."""
+        # Due to state, or FIFO rules (must allow others to run first).
         return self._value == 0 or (
             any(not w.cancelled() for w in (self._waiters or ())))
 
@@ -365,11 +388,12 @@ class Semaphore(_ContextManagerMixin, mixins._LoopBoundMixin):
 
         If the internal counter is larger than zero on entry,
         decrement it by one and return True immediately.  If it is
-        zero on entry, block, waiting until some other coroutine has
+        zero on entry, block, waiting until some other task has
         called release() to make it larger than 0, and then return
         True.
         """
         if not self.locked():
+            # Maintain FIFO, wait for others to start even if _value > 0.
             self._value -= 1
             return True
 
@@ -378,29 +402,34 @@ class Semaphore(_ContextManagerMixin, mixins._LoopBoundMixin):
         fut = self._get_loop().create_future()
         self._waiters.append(fut)
 
-        # Finally block should be called before the CancelledError
-        # handling as we don't want CancelledError to call
-        # _wake_up_first() and attempt to wake up itself.
         try:
             try:
                 await fut
             finally:
                 self._waiters.remove(fut)
         except exceptions.CancelledError:
-            if not fut.cancelled():
+            # Currently the only exception designed be able to occur here.
+            if fut.done() and not fut.cancelled():
+                # Our Future was successfully set to True via _wake_up_next(),
+                # but we are not about to successfully acquire(). Therefore we
+                # must undo the bookkeeping already done and attempt to wake
+                # up someone else.
                 self._value += 1
-                self._wake_up_next()
             raise
 
-        if self._value > 0:
-            self._wake_up_next()
+        finally:
+            # New waiters may have arrived but had to wait due to FIFO.
+            # Wake up as many as are allowed.
+            while self._value > 0:
+                if not self._wake_up_next():
+                    break  # There was no-one to wake up.
         return True
 
     def release(self):
         """Release a semaphore, incrementing the internal counter by one.
 
-        When it was zero on entry and another coroutine is waiting for it to
-        become larger than zero again, wake up that coroutine.
+        When it was zero on entry and another task is waiting for it to
+        become larger than zero again, wake up that task.
         """
         self._value += 1
         self._wake_up_next()
@@ -408,13 +437,15 @@ class Semaphore(_ContextManagerMixin, mixins._LoopBoundMixin):
     def _wake_up_next(self):
         """Wake up the first waiter that isn't done."""
         if not self._waiters:
-            return
+            return False
 
         for fut in self._waiters:
             if not fut.done():
                 self._value -= 1
                 fut.set_result(True)
-                return
+                # `fut` is now `done()` and not `cancelled()`.
+                return True
+        return False
 
 
 class BoundedSemaphore(Semaphore):
@@ -454,7 +485,7 @@ class Barrier(mixins._LoopBoundMixin):
     def __init__(self, parties):
         """Create a barrier, initialised to 'parties' tasks."""
         if parties < 1:
-            raise ValueError('parties must be > 0')
+            raise ValueError('parties must be >= 1')
 
         self._cond = Condition() # notify all tasks when state changes
 

Lib/asyncio/proactor_events.py

@@ -63,7 +63,7 @@ class _ProactorBasePipeTransport(transports._FlowControlMixin,
         self._called_connection_lost = False
         self._eof_written = False
         if self._server is not None:
-            self._server._attach()
+            self._server._attach(self)
         self._loop.call_soon(self._protocol.connection_made, self)
         if waiter is not None:
             # only wake up the waiter when connection_made() has been called
@@ -167,7 +167,7 @@ class _ProactorBasePipeTransport(transports._FlowControlMixin,
             self._sock = None
             server = self._server
             if server is not None:
-                server._detach()
+                server._detach(self)
                 self._server = None
             self._called_connection_lost = True
 
@@ -460,6 +460,8 @@ class _ProactorWritePipeTransport(_ProactorBaseWritePipeTransport):
 class _ProactorDatagramTransport(_ProactorBasePipeTransport,
                                  transports.DatagramTransport):
     max_size = 256 * 1024
+    _header_size = 8
+
     def __init__(self, loop, sock, protocol, address=None,
                  waiter=None, extra=None):
         self._address = address
@@ -487,9 +489,6 @@ class _ProactorDatagramTransport(_ProactorBasePipeTransport,
             raise TypeError('data argument must be bytes-like object (%r)',
                             type(data))
 
-        if not data:
-            return
-
         if self._address is not None and addr not in (None, self._address):
             raise ValueError(
                 f'Invalid address: must be None or {self._address}')
@@ -502,7 +501,7 @@ class _ProactorDatagramTransport(_ProactorBasePipeTransport,
 
         # Ensure that what we buffer is immutable.
         self._buffer.append((bytes(data), addr))
-        self._buffer_size += len(data)
+        self._buffer_size += len(data) + self._header_size
 
         if self._write_fut is None:
             # No current write operations are active, kick one off
@@ -529,7 +528,7 @@ class _ProactorDatagramTransport(_ProactorBasePipeTransport,
                 return
 
             data, addr = self._buffer.popleft()
-            self._buffer_size -= len(data)
+            self._buffer_size -= len(data) + self._header_size
             if self._address is not None:
                 self._write_fut = self._loop._proactor.send(self._sock,
                                                             data)
@@ -724,6 +723,8 @@ class BaseProactorEventLoop(base_events.BaseEventLoop):
         return await self._proactor.sendto(sock, data, 0, address)
 
     async def sock_connect(self, sock, address):
+        if self._debug and sock.gettimeout() != 0:
+            raise ValueError("the socket must be non-blocking")
         return await self._proactor.connect(sock, address)
 
     async def sock_accept(self, sock):

Lib/asyncio/queues.py

@@ -1,4 +1,11 @@
-__all__ = ('Queue', 'PriorityQueue', 'LifoQueue', 'QueueFull', 'QueueEmpty')
+__all__ = (
+    'Queue',
+    'PriorityQueue',
+    'LifoQueue',
+    'QueueFull',
+    'QueueEmpty',
+    'QueueShutDown',
+)
 
 import collections
 import heapq
@@ -18,6 +25,11 @@ class QueueFull(Exception):
     pass
 
 
+class QueueShutDown(Exception):
+    """Raised when putting on to or getting from a shut-down Queue."""
+    pass
+
+
 class Queue(mixins._LoopBoundMixin):
     """A queue, useful for coordinating producer and consumer coroutines.
 
@@ -41,6 +53,7 @@ class Queue(mixins._LoopBoundMixin):
         self._finished = locks.Event()
         self._finished.set()
         self._init(maxsize)
+        self._is_shutdown = False
 
     # These three are overridable in subclasses.
 
@@ -81,6 +94,8 @@ class Queue(mixins._LoopBoundMixin):
             result += f' _putters[{len(self._putters)}]'
         if self._unfinished_tasks:
             result += f' tasks={self._unfinished_tasks}'
+        if self._is_shutdown:
+            result += ' shutdown'
         return result
 
     def qsize(self):
@@ -112,8 +127,12 @@ class Queue(mixins._LoopBoundMixin):
 
         Put an item into the queue. If the queue is full, wait until a free
         slot is available before adding item.
+
+        Raises QueueShutDown if the queue has been shut down.
         """
         while self.full():
+            if self._is_shutdown:
+                raise QueueShutDown
             putter = self._get_loop().create_future()
             self._putters.append(putter)
             try:
@@ -125,7 +144,7 @@ class Queue(mixins._LoopBoundMixin):
                     self._putters.remove(putter)
                 except ValueError:
                     # The putter could be removed from self._putters by a
-                    # previous get_nowait call.
+                    # previous get_nowait call or a shutdown call.
                     pass
                 if not self.full() and not putter.cancelled():
                     # We were woken up by get_nowait(), but can't take
@@ -138,7 +157,11 @@ class Queue(mixins._LoopBoundMixin):
         """Put an item into the queue without blocking.
 
         If no free slot is immediately available, raise QueueFull.
+
+        Raises QueueShutDown if the queue has been shut down.
         """
+        if self._is_shutdown:
+            raise QueueShutDown
         if self.full():
             raise QueueFull
         self._put(item)
@@ -150,8 +173,13 @@ class Queue(mixins._LoopBoundMixin):
         """Remove and return an item from the queue.
 
         If queue is empty, wait until an item is available.
+
+        Raises QueueShutDown if the queue has been shut down and is empty, or
+        if the queue has been shut down immediately.
         """
         while self.empty():
+            if self._is_shutdown and self.empty():
+                raise QueueShutDown
             getter = self._get_loop().create_future()
             self._getters.append(getter)
             try:
@@ -163,7 +191,7 @@ class Queue(mixins._LoopBoundMixin):
                     self._getters.remove(getter)
                 except ValueError:
                     # The getter could be removed from self._getters by a
-                    # previous put_nowait call.
+                    # previous put_nowait call, or a shutdown call.
                     pass
                 if not self.empty() and not getter.cancelled():
                     # We were woken up by put_nowait(), but can't take
@@ -176,8 +204,13 @@ class Queue(mixins._LoopBoundMixin):
         """Remove and return an item from the queue.
 
         Return an item if one is immediately available, else raise QueueEmpty.
+
+        Raises QueueShutDown if the queue has been shut down and is empty, or
+        if the queue has been shut down immediately.
         """
         if self.empty():
+            if self._is_shutdown:
+                raise QueueShutDown
             raise QueueEmpty
         item = self._get()
         self._wakeup_next(self._putters)
@@ -214,6 +247,36 @@ class Queue(mixins._LoopBoundMixin):
         if self._unfinished_tasks > 0:
             await self._finished.wait()
 
+    def shutdown(self, immediate=False):
+        """Shut-down the queue, making queue gets and puts raise QueueShutDown.
+
+        By default, gets will only raise once the queue is empty. Set
+        'immediate' to True to make gets raise immediately instead.
+
+        All blocked callers of put() and get() will be unblocked.
+
+        If 'immediate', the queue is drained and unfinished tasks
+        is reduced by the number of drained tasks.  If unfinished tasks
+        is reduced to zero, callers of Queue.join are unblocked.
+        """
+        self._is_shutdown = True
+        if immediate:
+            while not self.empty():
+                self._get()
+                if self._unfinished_tasks > 0:
+                    self._unfinished_tasks -= 1
+            if self._unfinished_tasks == 0:
+                self._finished.set()
+        # All getters need to re-check queue-empty to raise ShutDown
+        while self._getters:
+            getter = self._getters.popleft()
+            if not getter.done():
+                getter.set_result(None)
+        while self._putters:
+            putter = self._putters.popleft()
+            if not putter.done():
+                putter.set_result(None)
+
 
 class PriorityQueue(Queue):
     """A subclass of Queue; retrieves entries in priority order (lowest first).

Lib/asyncio/runners.py

@@ -3,6 +3,7 @@ __all__ = ('Runner', 'run')
 import contextvars
 import enum
 import functools
+import inspect
 import threading
 import signal
 from . import coroutines
@@ -84,10 +85,7 @@ class Runner:
         return self._loop
 
     def run(self, coro, *, context=None):
-        """Run a coroutine inside the embedded event loop."""
-        if not coroutines.iscoroutine(coro):
-            raise ValueError("a coroutine was expected, got {!r}".format(coro))
-
+        """Run code in the embedded event loop."""
         if events._get_running_loop() is not None:
             # fail fast with short traceback
             raise RuntimeError(
@@ -95,8 +93,19 @@ class Runner:
 
         self._lazy_init()
 
+        if not coroutines.iscoroutine(coro):
+            if inspect.isawaitable(coro):
+                async def _wrap_awaitable(awaitable):
+                    return await awaitable
+
+                coro = _wrap_awaitable(coro)
+            else:
+                raise TypeError('An asyncio.Future, a coroutine or an '
+                                'awaitable is required')
+
         if context is None:
             context = self._context
+
         task = self._loop.create_task(coro, context=context)
 
         if (threading.current_thread() is threading.main_thread()
@@ -168,6 +177,7 @@ def run(main, *, debug=None, loop_factory=None):
     running in the same thread.
 
     If debug is True, the event loop will be run in debug mode.
+    If loop_factory is passed, it is used for new event loop creation.
 
     This function always creates a new event loop and closes it at the end.
     It should be used as a main entry point for asyncio programs, and should

Lib/asyncio/selector_events.py

@@ -173,16 +173,20 @@ class BaseSelectorEventLoop(base_events.BaseEventLoop):
         # listening socket has triggered an EVENT_READ. There may be multiple
         # connections waiting for an .accept() so it is called in a loop.
         # See https://bugs.python.org/issue27906 for more details.
-        for _ in range(backlog):
+        for _ in range(backlog + 1):
             try:
                 conn, addr = sock.accept()
                 if self._debug:
                     logger.debug("%r got a new connection from %r: %r",
                                  server, addr, conn)
                 conn.setblocking(False)
-            except (BlockingIOError, InterruptedError, ConnectionAbortedError):
-                # Early exit because the socket accept buffer is empty.
-                return None
+            except ConnectionAbortedError:
+                # Discard connections that were aborted before accept().
+                continue
+            except (BlockingIOError, InterruptedError):
+                # Early exit because of a signal or
+                # the socket accept buffer is empty.
+                return
             except OSError as exc:
                 # There's nowhere to send the error, so just log it.
                 if exc.errno in (errno.EMFILE, errno.ENFILE,
@@ -265,22 +269,17 @@ class BaseSelectorEventLoop(base_events.BaseEventLoop):
             except (AttributeError, TypeError, ValueError):
                 # This code matches selectors._fileobj_to_fd function.
                 raise ValueError(f"Invalid file object: {fd!r}") from None
-        try:
-            transport = self._transports[fileno]
-        except KeyError:
-            pass
-        else:
-            if not transport.is_closing():
-                raise RuntimeError(
-                    f'File descriptor {fd!r} is used by transport '
-                    f'{transport!r}')
+        transport = self._transports.get(fileno)
+        if transport and not transport.is_closing():
+            raise RuntimeError(
+                f'File descriptor {fd!r} is used by transport '
+                f'{transport!r}')
 
     def _add_reader(self, fd, callback, *args):
         self._check_closed()
         handle = events.Handle(callback, args, self, None)
-        try:
-            key = self._selector.get_key(fd)
-        except KeyError:
+        key = self._selector.get_map().get(fd)
+        if key is None:
             self._selector.register(fd, selectors.EVENT_READ,
                                     (handle, None))
         else:
@@ -294,30 +293,27 @@ class BaseSelectorEventLoop(base_events.BaseEventLoop):
     def _remove_reader(self, fd):
         if self.is_closed():
             return False
-        try:
-            key = self._selector.get_key(fd)
-        except KeyError:
+        key = self._selector.get_map().get(fd)
+        if key is None:
             return False
+        mask, (reader, writer) = key.events, key.data
+        mask &= ~selectors.EVENT_READ
+        if not mask:
+            self._selector.unregister(fd)
         else:
-            mask, (reader, writer) = key.events, key.data
-            mask &= ~selectors.EVENT_READ
-            if not mask:
-                self._selector.unregister(fd)
-            else:
-                self._selector.modify(fd, mask, (None, writer))
+            self._selector.modify(fd, mask, (None, writer))
 
-            if reader is not None:
-                reader.cancel()
-                return True
-            else:
-                return False
+        if reader is not None:
+            reader.cancel()
+            return True
+        else:
+            return False
 
     def _add_writer(self, fd, callback, *args):
         self._check_closed()
         handle = events.Handle(callback, args, self, None)
-        try:
-            key = self._selector.get_key(fd)
-        except KeyError:
+        key = self._selector.get_map().get(fd)
+        if key is None:
             self._selector.register(fd, selectors.EVENT_WRITE,
                                     (None, handle))
         else:
@@ -332,24 +328,22 @@ class BaseSelectorEventLoop(base_events.BaseEventLoop):
         """Remove a writer callback."""
         if self.is_closed():
             return False
-        try:
-            key = self._selector.get_key(fd)
-        except KeyError:
+        key = self._selector.get_map().get(fd)
+        if key is None:
             return False
+        mask, (reader, writer) = key.events, key.data
+        # Remove both writer and connector.
+        mask &= ~selectors.EVENT_WRITE
+        if not mask:
+            self._selector.unregister(fd)
         else:
-            mask, (reader, writer) = key.events, key.data
-            # Remove both writer and connector.
-            mask &= ~selectors.EVENT_WRITE
-            if not mask:
-                self._selector.unregister(fd)
-            else:
-                self._selector.modify(fd, mask, (reader, None))
+            self._selector.modify(fd, mask, (reader, None))
 
-            if writer is not None:
-                writer.cancel()
-                return True
-            else:
-                return False
+        if writer is not None:
+            writer.cancel()
+            return True
+        else:
+            return False
 
     def add_reader(self, fd, callback, *args):
         """Add a reader callback."""
@@ -801,7 +795,7 @@ class _SelectorTransport(transports._FlowControlMixin,
         self._paused = False  # Set when pause_reading() called
 
         if self._server is not None:
-            self._server._attach()
+            self._server._attach(self)
         loop._transports[self._sock_fd] = self
 
     def __repr__(self):
@@ -878,6 +872,8 @@ class _SelectorTransport(transports._FlowControlMixin,
         if self._sock is not None:
             _warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
             self._sock.close()
+            if self._server is not None:
+                self._server._detach(self)
 
     def _fatal_error(self, exc, message='Fatal error on transport'):
         # Should be called from exception handler only.
@@ -916,7 +912,7 @@ class _SelectorTransport(transports._FlowControlMixin,
             self._loop = None
             server = self._server
             if server is not None:
-                server._detach()
+                server._detach(self)
                 self._server = None
 
     def get_write_buffer_size(self):
@@ -1054,8 +1050,8 @@ class _SelectorSocketTransport(_SelectorTransport):
 
     def write(self, data):
         if not isinstance(data, (bytes, bytearray, memoryview)):
-            raise TypeError(f'data argument must be a bytes-like object, '
-                            f'not {type(data).__name__!r}')
+            raise TypeError(f'data argument must be a bytes, bytearray, or memoryview '
+                            f'object, not {type(data).__name__!r}')
         if self._eof:
             raise RuntimeError('Cannot call write() after write_eof()')
         if self._empty_waiter is not None:
@@ -1178,20 +1174,31 @@ class _SelectorSocketTransport(_SelectorTransport):
             raise RuntimeError('unable to writelines; sendfile is in progress')
         if not list_of_data:
             return
+
+        if self._conn_lost:
+            if self._conn_lost >= constants.LOG_THRESHOLD_FOR_CONNLOST_WRITES:
+                logger.warning('socket.send() raised exception.')
+            self._conn_lost += 1
+            return
+
         self._buffer.extend([memoryview(data) for data in list_of_data])
         self._write_ready()
         # If the entire buffer couldn't be written, register a write handler
         if self._buffer:
             self._loop._add_writer(self._sock_fd, self._write_ready)
+            self._maybe_pause_protocol()
 
     def can_write_eof(self):
         return True
 
     def _call_connection_lost(self, exc):
-        super()._call_connection_lost(exc)
-        if self._empty_waiter is not None:
-            self._empty_waiter.set_exception(
-                ConnectionError("Connection is closed by peer"))
+        try:
+            super()._call_connection_lost(exc)
+        finally:
+            self._write_ready = None
+            if self._empty_waiter is not None:
+                self._empty_waiter.set_exception(
+                    ConnectionError("Connection is closed by peer"))
 
     def _make_empty_waiter(self):
         if self._empty_waiter is not None:
@@ -1206,13 +1213,13 @@ class _SelectorSocketTransport(_SelectorTransport):
 
     def close(self):
         self._read_ready_cb = None
-        self._write_ready = None
         super().close()
 
 
 class _SelectorDatagramTransport(_SelectorTransport, transports.DatagramTransport):
 
     _buffer_factory = collections.deque
+    _header_size = 8
 
     def __init__(self, loop, sock, protocol, address=None,
                  waiter=None, extra=None):
@@ -1251,8 +1258,6 @@ class _SelectorDatagramTransport(_SelectorTransport, transports.DatagramTranspor
         if not isinstance(data, (bytes, bytearray, memoryview)):
             raise TypeError(f'data argument must be a bytes-like object, '
                             f'not {type(data).__name__!r}')
-        if not data:
-            return
 
         if self._address:
             if addr not in (None, self._address):
@@ -1288,13 +1293,13 @@ class _SelectorDatagramTransport(_SelectorTransport, transports.DatagramTranspor
 
         # Ensure that what we buffer is immutable.
         self._buffer.append((bytes(data), addr))
-        self._buffer_size += len(data)
+        self._buffer_size += len(data) + self._header_size
         self._maybe_pause_protocol()
 
     def _sendto_ready(self):
         while self._buffer:
             data, addr = self._buffer.popleft()
-            self._buffer_size -= len(data)
+            self._buffer_size -= len(data) + self._header_size
             try:
                 if self._extra['peername']:
                     self._sock.send(data)
@@ -1302,7 +1307,7 @@ class _SelectorDatagramTransport(_SelectorTransport, transports.DatagramTranspor
                     self._sock.sendto(data, addr)
             except (BlockingIOError, InterruptedError):
                 self._buffer.appendleft((data, addr))  # Try again later.
-                self._buffer_size += len(data)
+                self._buffer_size += len(data) + self._header_size
                 break
             except OSError as exc:
                 self._protocol.error_received(exc)

Lib/asyncio/sslproto.py

@@ -101,7 +101,7 @@ class _SSLProtocolTransport(transports._FlowControlMixin,
         return self._ssl_protocol._app_protocol
 
     def is_closing(self):
-        return self._closed
+        return self._closed or self._ssl_protocol._is_transport_closing()
 
     def close(self):
         """Close the transport.
@@ -379,6 +379,9 @@ class SSLProtocol(protocols.BufferedProtocol):
             self._app_transport_created = True
         return self._app_transport
 
+    def _is_transport_closing(self):
+        return self._transport is not None and self._transport.is_closing()
+
     def connection_made(self, transport):
         """Called when the low-level connection is made.
 
@@ -542,7 +545,7 @@ class SSLProtocol(protocols.BufferedProtocol):
         # start handshake timeout count down
         self._handshake_timeout_handle = \
             self._loop.call_later(self._ssl_handshake_timeout,
-                                  lambda: self._check_handshake_timeout())
+                                  self._check_handshake_timeout)
 
         self._do_handshake()
 
@@ -623,7 +626,7 @@ class SSLProtocol(protocols.BufferedProtocol):
             self._set_state(SSLProtocolState.FLUSHING)
             self._shutdown_timeout_handle = self._loop.call_later(
                 self._ssl_shutdown_timeout,
-                lambda: self._check_shutdown_timeout()
+                self._check_shutdown_timeout
             )
             self._do_flush()
 
@@ -762,7 +765,7 @@ class SSLProtocol(protocols.BufferedProtocol):
                     else:
                         break
                 else:
-                    self._loop.call_soon(lambda: self._do_read())
+                    self._loop.call_soon(self._do_read)
         except SSLAgainErrors:
             pass
         if offset > 0:

Lib/asyncio/staggered.py

@@ -3,24 +3,15 @@
 __all__ = 'staggered_race',
 
 import contextlib
-import typing
 
 from . import events
 from . import exceptions as exceptions_mod
 from . import locks
 from . import tasks
+from . import futures
 
 
-async def staggered_race(
-        coro_fns: typing.Iterable[typing.Callable[[], typing.Awaitable]],
-        delay: typing.Optional[float],
-        *,
-        loop: events.AbstractEventLoop = None,
-) -> typing.Tuple[
-    typing.Any,
-    typing.Optional[int],
-    typing.List[typing.Optional[Exception]]
-]:
+async def staggered_race(coro_fns, delay, *, loop=None):
     """Run coroutines with staggered start times and take the first to finish.
 
     This method takes an iterable of coroutine functions. The first one is
@@ -73,14 +64,38 @@ async def staggered_race(
     """
     # TODO: when we have aiter() and anext(), allow async iterables in coro_fns.
     loop = loop or events.get_running_loop()
+    parent_task = tasks.current_task(loop)
     enum_coro_fns = enumerate(coro_fns)
     winner_result = None
     winner_index = None
+    unhandled_exceptions = []
     exceptions = []
-    running_tasks = []
+    running_tasks = set()
+    on_completed_fut = None
 
-    async def run_one_coro(
-            previous_failed: typing.Optional[locks.Event]) -> None:
+    def task_done(task):
+        running_tasks.discard(task)
+        futures.future_discard_from_awaited_by(task, parent_task)
+        if (
+            on_completed_fut is not None
+            and not on_completed_fut.done()
+            and not running_tasks
+        ):
+            on_completed_fut.set_result(None)
+
+        if task.cancelled():
+            return
+
+        exc = task.exception()
+        if exc is None:
+            return
+        unhandled_exceptions.append(exc)
+
+    async def run_one_coro(ok_to_start, previous_failed) -> None:
+        # in eager tasks this waits for the calling task to append this task
+        # to running_tasks, in regular tasks this wait is a no-op that does
+        # not yield a future. See gh-124309.
+        await ok_to_start.wait()
         # Wait for the previous task to finish, or for delay seconds
         if previous_failed is not None:
             with contextlib.suppress(exceptions_mod.TimeoutError):
@@ -96,9 +111,14 @@ async def staggered_race(
             return
         # Start task that will run the next coroutine
         this_failed = locks.Event()
-        next_task = loop.create_task(run_one_coro(this_failed))
-        running_tasks.append(next_task)
-        assert len(running_tasks) == this_index + 2
+        next_ok_to_start = locks.Event()
+        next_task = loop.create_task(run_one_coro(next_ok_to_start, this_failed))
+        futures.future_add_to_awaited_by(next_task, parent_task)
+        running_tasks.add(next_task)
+        next_task.add_done_callback(task_done)
+        # next_task has been appended to running_tasks so next_task is ok to
+        # start.
+        next_ok_to_start.set()
         # Prepare place to put this coroutine's exceptions if not won
         exceptions.append(None)
         assert len(exceptions) == this_index + 1
@@ -123,27 +143,37 @@ async def staggered_race(
             # up as done() == True, cancelled() == False, exception() ==
             # asyncio.CancelledError. This behavior is specified in
             # https://bugs.python.org/issue30048
-            for i, t in enumerate(running_tasks):
-                if i != this_index:
+            current_task = tasks.current_task(loop)
+            for t in running_tasks:
+                if t is not current_task:
                     t.cancel()
 
-    first_task = loop.create_task(run_one_coro(None))
-    running_tasks.append(first_task)
+    propagate_cancellation_error = None
     try:
-        # Wait for a growing list of tasks to all finish: poor man's version of
-        # curio's TaskGroup or trio's nursery
-        done_count = 0
-        while done_count != len(running_tasks):
-            done, _ = await tasks.wait(running_tasks)
-            done_count = len(done)
+        ok_to_start = locks.Event()
+        first_task = loop.create_task(run_one_coro(ok_to_start, None))
+        futures.future_add_to_awaited_by(first_task, parent_task)
+        running_tasks.add(first_task)
+        first_task.add_done_callback(task_done)
+        # first_task has been appended to running_tasks so first_task is ok to start.
+        ok_to_start.set()
+        propagate_cancellation_error = None
+        # Make sure no tasks are left running if we leave this function
+        while running_tasks:
+            on_completed_fut = loop.create_future()
+            try:
+                await on_completed_fut
+            except exceptions_mod.CancelledError as ex:
+                propagate_cancellation_error = ex
+                for task in running_tasks:
+                    task.cancel(*ex.args)
+            on_completed_fut = None
+        if __debug__ and unhandled_exceptions:
             # If run_one_coro raises an unhandled exception, it's probably a
             # programming error, and I want to see it.
-            if __debug__:
-                for d in done:
-                    if d.done() and not d.cancelled() and d.exception():
-                        raise d.exception()
+            raise ExceptionGroup("staggered race failed", unhandled_exceptions)
+        if propagate_cancellation_error is not None:
+            raise propagate_cancellation_error
         return winner_result, winner_index, exceptions
     finally:
-        # Make sure no tasks are left running if we leave this function
-        for t in running_tasks:
-            t.cancel()
+        del exceptions, propagate_cancellation_error, unhandled_exceptions, parent_task

Lib/asyncio/streams.py

@@ -201,7 +201,6 @@ class StreamReaderProtocol(FlowControlMixin, protocols.Protocol):
             # is established.
             self._strong_reader = stream_reader
         self._reject_connection = False
-        self._stream_writer = None
         self._task = None
         self._transport = None
         self._client_connected_cb = client_connected_cb
@@ -214,10 +213,8 @@ class StreamReaderProtocol(FlowControlMixin, protocols.Protocol):
             return None
         return self._stream_reader_wr()
 
-    def _replace_writer(self, writer):
+    def _replace_transport(self, transport):
         loop = self._loop
-        transport = writer.transport
-        self._stream_writer = writer
         self._transport = transport
         self._over_ssl = transport.get_extra_info('sslcontext') is not None
 
@@ -239,11 +236,8 @@ class StreamReaderProtocol(FlowControlMixin, protocols.Protocol):
             reader.set_transport(transport)
         self._over_ssl = transport.get_extra_info('sslcontext') is not None
         if self._client_connected_cb is not None:
-            self._stream_writer = StreamWriter(transport, self,
-                                               reader,
-                                               self._loop)
-            res = self._client_connected_cb(reader,
-                                            self._stream_writer)
+            writer = StreamWriter(transport, self, reader, self._loop)
+            res = self._client_connected_cb(reader, writer)
             if coroutines.iscoroutine(res):
                 def callback(task):
                     if task.cancelled():
@@ -405,9 +399,9 @@ class StreamWriter:
             ssl_handshake_timeout=ssl_handshake_timeout,
             ssl_shutdown_timeout=ssl_shutdown_timeout)
         self._transport = new_transport
-        protocol._replace_writer(self)
+        protocol._replace_transport(new_transport)
 
-    def __del__(self):
+    def __del__(self, warnings=warnings):
         if not self._transport.is_closing():
             if self._loop.is_closed():
                 warnings.warn("loop is closed", ResourceWarning)
@@ -596,20 +590,34 @@ class StreamReader:
         If the data cannot be read because of over limit, a
         LimitOverrunError exception  will be raised, and the data
         will be left in the internal buffer, so it can be read again.
+
+        The ``separator`` may also be a tuple of separators. In this
+        case the return value will be the shortest possible that has any
+        separator as the suffix. For the purposes of LimitOverrunError,
+        the shortest possible separator is considered to be the one that
+        matched.
         """
-        seplen = len(separator)
-        if seplen == 0:
+        if isinstance(separator, tuple):
+            # Makes sure shortest matches wins
+            separator = sorted(separator, key=len)
+        else:
+            separator = [separator]
+        if not separator:
+            raise ValueError('Separator should contain at least one element')
+        min_seplen = len(separator[0])
+        max_seplen = len(separator[-1])
+        if min_seplen == 0:
             raise ValueError('Separator should be at least one-byte string')
 
         if self._exception is not None:
             raise self._exception
 
         # Consume whole buffer except last bytes, which length is
-        # one less than seplen. Let's check corner cases with
-        # separator='SEPARATOR':
+        # one less than max_seplen. Let's check corner cases with
+        # separator[-1]='SEPARATOR':
         # * we have received almost complete separator (without last
         #   byte). i.e buffer='some textSEPARATO'. In this case we
-        #   can safely consume len(separator) - 1 bytes.
+        #   can safely consume max_seplen - 1 bytes.
         # * last byte of buffer is first byte of separator, i.e.
         #   buffer='abcdefghijklmnopqrS'. We may safely consume
         #   everything except that last byte, but this require to
@@ -622,26 +630,35 @@ class StreamReader:
         #   messages :)
 
         # `offset` is the number of bytes from the beginning of the buffer
-        # where there is no occurrence of `separator`.
+        # where there is no occurrence of any `separator`.
         offset = 0
 
-        # Loop until we find `separator` in the buffer, exceed the buffer size,
+        # Loop until we find a `separator` in the buffer, exceed the buffer size,
         # or an EOF has happened.
         while True:
             buflen = len(self._buffer)
 
-            # Check if we now have enough data in the buffer for `separator` to
-            # fit.
-            if buflen - offset >= seplen:
-                isep = self._buffer.find(separator, offset)
+            # Check if we now have enough data in the buffer for shortest
+            # separator to fit.
+            if buflen - offset >= min_seplen:
+                match_start = None
+                match_end = None
+                for sep in separator:
+                    isep = self._buffer.find(sep, offset)
 
-                if isep != -1:
-                    # `separator` is in the buffer. `isep` will be used later
-                    # to retrieve the data.
+                    if isep != -1:
+                        # `separator` is in the buffer. `match_start` and
+                        # `match_end` will be used later to retrieve the
+                        # data.
+                        end = isep + len(sep)
+                        if match_end is None or end < match_end:
+                            match_end = end
+                            match_start = isep
+                if match_end is not None:
                     break
 
                 # see upper comment for explanation.
-                offset = buflen + 1 - seplen
+                offset = max(0, buflen + 1 - max_seplen)
                 if offset > self._limit:
                     raise exceptions.LimitOverrunError(
                         'Separator is not found, and chunk exceed the limit',
@@ -650,7 +667,7 @@ class StreamReader:
             # Complete message (with full separator) may be present in buffer
             # even when EOF flag is set. This may happen when the last chunk
             # adds data which makes separator be found. That's why we check for
-            # EOF *ater* inspecting the buffer.
+            # EOF *after* inspecting the buffer.
             if self._eof:
                 chunk = bytes(self._buffer)
                 self._buffer.clear()
@@ -659,12 +676,12 @@ class StreamReader:
             # _wait_for_data() will resume reading if stream was paused.
             await self._wait_for_data('readuntil')
 
-        if isep > self._limit:
+        if match_start > self._limit:
             raise exceptions.LimitOverrunError(
-                'Separator is found, but chunk is longer than limit', isep)
+                'Separator is found, but chunk is longer than limit', match_start)
 
-        chunk = self._buffer[:isep + seplen]
-        del self._buffer[:isep + seplen]
+        chunk = self._buffer[:match_end]
+        del self._buffer[:match_end]
         self._maybe_resume_transport()
         return bytes(chunk)
 

Lib/asyncio/taskgroups.py

@@ -6,6 +6,7 @@ __all__ = ("TaskGroup",)
 
 from . import events
 from . import exceptions
+from . import futures
 from . import tasks
 
 
@@ -66,6 +67,20 @@ class TaskGroup:
         return self
 
     async def __aexit__(self, et, exc, tb):
+        tb = None
+        try:
+            return await self._aexit(et, exc)
+        finally:
+            # Exceptions are heavy objects that can have object
+            # cycles (bad for GC); let's not keep a reference to
+            # a bunch of them. It would be nicer to use a try/finally
+            # in __aexit__ directly but that introduced some diff noise
+            self._parent_task = None
+            self._errors = None
+            self._base_error = None
+            exc = None
+
+    async def _aexit(self, et, exc):
         self._exiting = True
 
         if (exc is not None and
@@ -73,14 +88,10 @@ class TaskGroup:
                 self._base_error is None):
             self._base_error = exc
 
-        propagate_cancellation_error = \
-            exc if et is exceptions.CancelledError else None
-        if self._parent_cancel_requested:
-            # If this flag is set we *must* call uncancel().
-            if self._parent_task.uncancel() == 0:
-                # If there are no pending cancellations left,
-                # don't propagate CancelledError.
-                propagate_cancellation_error = None
+        if et is not None and issubclass(et, exceptions.CancelledError):
+            propagate_cancellation_error = exc
+        else:
+            propagate_cancellation_error = None
 
         if et is not None:
             if not self._aborting:
@@ -126,51 +137,78 @@ class TaskGroup:
         assert not self._tasks
 
         if self._base_error is not None:
-            raise self._base_error
+            try:
+                raise self._base_error
+            finally:
+                exc = None
+
+        if self._parent_cancel_requested:
+            # If this flag is set we *must* call uncancel().
+            if self._parent_task.uncancel() == 0:
+                # If there are no pending cancellations left,
+                # don't propagate CancelledError.
+                propagate_cancellation_error = None
 
         # Propagate CancelledError if there is one, except if there
         # are other errors -- those have priority.
-        if propagate_cancellation_error and not self._errors:
-            raise propagate_cancellation_error
+        try:
+            if propagate_cancellation_error is not None and not self._errors:
+                try:
+                    raise propagate_cancellation_error
+                finally:
+                    exc = None
+        finally:
+            propagate_cancellation_error = None
 
-        if et is not None and et is not exceptions.CancelledError:
+        if et is not None and not issubclass(et, exceptions.CancelledError):
             self._errors.append(exc)
 
         if self._errors:
-            # Exceptions are heavy objects that can have object
-            # cycles (bad for GC); let's not keep a reference to
-            # a bunch of them.
+            # If the parent task is being cancelled from the outside
+            # of the taskgroup, un-cancel and re-cancel the parent task,
+            # which will keep the cancel count stable.
+            if self._parent_task.cancelling():
+                self._parent_task.uncancel()
+                self._parent_task.cancel()
             try:
-                me = BaseExceptionGroup('unhandled errors in a TaskGroup', self._errors)
-                raise me from None
+                raise BaseExceptionGroup(
+                    'unhandled errors in a TaskGroup',
+                    self._errors,
+                ) from None
             finally:
-                self._errors = None
+                exc = None
 
-    def create_task(self, coro, *, name=None, context=None):
+
+    def create_task(self, coro, **kwargs):
         """Create a new task in this group and return it.
 
         Similar to `asyncio.create_task`.
         """
         if not self._entered:
+            coro.close()
             raise RuntimeError(f"TaskGroup {self!r} has not been entered")
         if self._exiting and not self._tasks:
+            coro.close()
             raise RuntimeError(f"TaskGroup {self!r} is finished")
         if self._aborting:
+            coro.close()
             raise RuntimeError(f"TaskGroup {self!r} is shutting down")
-        if context is None:
-            task = self._loop.create_task(coro)
-        else:
-            task = self._loop.create_task(coro, context=context)
-        tasks._set_task_name(task, name)
-        # optimization: Immediately call the done callback if the task is
+        task = self._loop.create_task(coro, **kwargs)
+
+        futures.future_add_to_awaited_by(task, self._parent_task)
+
+        # Always schedule the done callback even if the task is
         # already done (e.g. if the coro was able to complete eagerly),
-        # and skip scheduling a done callback
-        if task.done():
-            self._on_task_done(task)
-        else:
-            self._tasks.add(task)
-            task.add_done_callback(self._on_task_done)
-        return task
+        # otherwise if the task completes with an exception then it will cancel
+        # the current task too early. gh-128550, gh-128588
+        self._tasks.add(task)
+        task.add_done_callback(self._on_task_done)
+        try:
+            return task
+        finally:
+            # gh-128552: prevent a refcycle of
+            # task.exception().__traceback__->TaskGroup.create_task->task
+            del task
 
     # Since Python 3.8 Tasks propagate all exceptions correctly,
     # except for KeyboardInterrupt and SystemExit which are
@@ -190,6 +228,8 @@ class TaskGroup:
     def _on_task_done(self, task):
         self._tasks.discard(task)
 
+        futures.future_discard_from_awaited_by(task, self._parent_task)
+
         if self._on_completed_fut is not None and not self._tasks:
             if not self._on_completed_fut.done():
                 self._on_completed_fut.set_result(True)

Lib/asyncio/tasks.py

@@ -15,8 +15,8 @@ import contextvars
 import functools
 import inspect
 import itertools
+import math
 import types
-import warnings
 import weakref
 from types import GenericAlias
 
@@ -25,6 +25,7 @@ from . import coroutines
 from . import events
 from . import exceptions
 from . import futures
+from . import queues
 from . import timeouts
 
 # Helper to generate new task names
@@ -47,39 +48,11 @@ def all_tasks(loop=None):
     # capturing the set of eager tasks first, so if an eager task "graduates"
     # to a regular task in another thread, we don't risk missing it.
     eager_tasks = list(_eager_tasks)
-    # Looping over the WeakSet isn't safe as it can be updated from another
-    # thread, therefore we cast it to list prior to filtering. The list cast
-    # itself requires iteration, so we repeat it several times ignoring
-    # RuntimeErrors (which are not very likely to occur).
-    # See issues 34970 and 36607 for details.
-    scheduled_tasks = None
-    i = 0
-    while True:
-        try:
-            scheduled_tasks = list(_scheduled_tasks)
-        except RuntimeError:
-            i += 1
-            if i >= 1000:
-                raise
-        else:
-            break
-    return {t for t in itertools.chain(scheduled_tasks, eager_tasks)
+
+    return {t for t in itertools.chain(_scheduled_tasks, eager_tasks)
             if futures._get_loop(t) is loop and not t.done()}
 
 
-def _set_task_name(task, name):
-    if name is not None:
-        try:
-            set_name = task.set_name
-        except AttributeError:
-            warnings.warn("Task.set_name() was added in Python 3.8, "
-                      "the method support will be mandatory for third-party "
-                      "task implementations since 3.13.",
-                      DeprecationWarning, stacklevel=3)
-        else:
-            set_name(name)
-
-
 class Task(futures._PyFuture):  # Inherit Python Task implementation
                                 # from a Python Future implementation.
 
@@ -137,7 +110,7 @@ class Task(futures._PyFuture):  # Inherit Python Task implementation
             self.__eager_start()
         else:
             self._loop.call_soon(self.__step, context=self._context)
-            _register_task(self)
+            _py_register_task(self)
 
     def __del__(self):
         if self._state == futures._PENDING and self._log_destroy_pending:
@@ -267,42 +240,44 @@ class Task(futures._PyFuture):  # Inherit Python Task implementation
         """
         if self._num_cancels_requested > 0:
             self._num_cancels_requested -= 1
+            if self._num_cancels_requested == 0:
+                self._must_cancel = False
         return self._num_cancels_requested
 
     def __eager_start(self):
-        prev_task = _swap_current_task(self._loop, self)
+        prev_task = _py_swap_current_task(self._loop, self)
         try:
-            _register_eager_task(self)
+            _py_register_eager_task(self)
             try:
                 self._context.run(self.__step_run_and_handle_result, None)
             finally:
-                _unregister_eager_task(self)
+                _py_unregister_eager_task(self)
         finally:
             try:
-                curtask = _swap_current_task(self._loop, prev_task)
+                curtask = _py_swap_current_task(self._loop, prev_task)
                 assert curtask is self
             finally:
                 if self.done():
                     self._coro = None
                     self = None  # Needed to break cycles when an exception occurs.
                 else:
-                    _register_task(self)
+                    _py_register_task(self)
 
     def __step(self, exc=None):
         if self.done():
             raise exceptions.InvalidStateError(
-                f'_step(): already done: {self!r}, {exc!r}')
+                f'__step(): already done: {self!r}, {exc!r}')
         if self._must_cancel:
             if not isinstance(exc, exceptions.CancelledError):
                 exc = self._make_cancelled_error()
             self._must_cancel = False
         self._fut_waiter = None
 
-        _enter_task(self._loop, self)
+        _py_enter_task(self._loop, self)
         try:
             self.__step_run_and_handle_result(exc)
         finally:
-            _leave_task(self._loop, self)
+            _py_leave_task(self._loop, self)
             self = None  # Needed to break cycles when an exception occurs.
 
     def __step_run_and_handle_result(self, exc):
@@ -347,6 +322,7 @@ class Task(futures._PyFuture):  # Inherit Python Task implementation
                         self._loop.call_soon(
                             self.__step, new_exc, context=self._context)
                     else:
+                        futures.future_add_to_awaited_by(result, self)
                         result._asyncio_future_blocking = False
                         result.add_done_callback(
                             self.__wakeup, context=self._context)
@@ -381,6 +357,7 @@ class Task(futures._PyFuture):  # Inherit Python Task implementation
             self = None  # Needed to break cycles when an exception occurs.
 
     def __wakeup(self, future):
+        futures.future_discard_from_awaited_by(future, self)
         try:
             future.result()
         except BaseException as exc:
@@ -389,7 +366,7 @@ class Task(futures._PyFuture):  # Inherit Python Task implementation
         else:
             # Don't pass the value of `future.result()` explicitly,
             # as `Future.__iter__` and `Future.__await__` don't need it.
-            # If we call `_step(value, None)` instead of `_step()`,
+            # If we call `__step(value, None)` instead of `__step()`,
             # Python eval loop would use `.send(value)` method call,
             # instead of `__next__()`, which is slower for futures
             # that return non-generator iterators from their `__iter__`.
@@ -409,20 +386,13 @@ else:
     Task = _CTask = _asyncio.Task
 
 
-def create_task(coro, *, name=None, context=None):
+def create_task(coro, **kwargs):
     """Schedule the execution of a coroutine object in a spawn task.
 
     Return a Task object.
     """
     loop = events.get_running_loop()
-    if context is None:
-        # Use legacy API if context is not needed
-        task = loop.create_task(coro)
-    else:
-        task = loop.create_task(coro, context=context)
-
-    _set_task_name(task, name)
-    return task
+    return loop.create_task(coro, **kwargs)
 
 
 # wait() and as_completed() similar to those in PEP 3148.
@@ -437,8 +407,6 @@ async def wait(fs, *, timeout=None, return_when=ALL_COMPLETED):
 
     The fs iterable must not be empty.
 
-    Coroutines will be wrapped in Tasks.
-
     Returns two sets of Future: (done, pending).
 
     Usage:
@@ -530,6 +498,7 @@ async def _wait(fs, timeout, return_when, loop):
     if timeout is not None:
         timeout_handle = loop.call_later(timeout, _release_waiter, waiter)
     counter = len(fs)
+    cur_task = current_task()
 
     def _on_completion(f):
         nonlocal counter
@@ -542,9 +511,11 @@ async def _wait(fs, timeout, return_when, loop):
                 timeout_handle.cancel()
             if not waiter.done():
                 waiter.set_result(None)
+        futures.future_discard_from_awaited_by(f, cur_task)
 
     for f in fs:
         f.add_done_callback(_on_completion)
+        futures.future_add_to_awaited_by(f, cur_task)
 
     try:
         await waiter
@@ -580,62 +551,125 @@ async def _cancel_and_wait(fut):
         fut.remove_done_callback(cb)
 
 
-# This is *not* a @coroutine!  It is just an iterator (yielding Futures).
-def as_completed(fs, *, timeout=None):
-    """Return an iterator whose values are coroutines.
+class _AsCompletedIterator:
+    """Iterator of awaitables representing tasks of asyncio.as_completed.
 
-    When waiting for the yielded coroutines you'll get the results (or
-    exceptions!) of the original Futures (or coroutines), in the order
-    in which and as soon as they complete.
-
-    This differs from PEP 3148; the proper way to use this is:
-
-        for f in as_completed(fs):
-            result = await f  # The 'await' may raise.
-            # Use result.
-
-    If a timeout is specified, the 'await' will raise
-    TimeoutError when the timeout occurs before all Futures are done.
-
-    Note: The futures 'f' are not necessarily members of fs.
+    As an asynchronous iterator, iteration yields futures as they finish. As a
+    plain iterator, new coroutines are yielded that will return or raise the
+    result of the next underlying future to complete.
     """
-    if futures.isfuture(fs) or coroutines.iscoroutine(fs):
-        raise TypeError(f"expect an iterable of futures, not {type(fs).__name__}")
+    def __init__(self, aws, timeout):
+        self._done = queues.Queue()
+        self._timeout_handle = None
 
-    from .queues import Queue  # Import here to avoid circular import problem.
-    done = Queue()
-
-    loop = events.get_event_loop()
-    todo = {ensure_future(f, loop=loop) for f in set(fs)}
-    timeout_handle = None
-
-    def _on_timeout():
+        loop = events.get_event_loop()
+        todo = {ensure_future(aw, loop=loop) for aw in set(aws)}
         for f in todo:
-            f.remove_done_callback(_on_completion)
-            done.put_nowait(None)  # Queue a dummy value for _wait_for_one().
-        todo.clear()  # Can't do todo.remove(f) in the loop.
+            f.add_done_callback(self._handle_completion)
+        if todo and timeout is not None:
+            self._timeout_handle = (
+                loop.call_later(timeout, self._handle_timeout)
+            )
+        self._todo = todo
+        self._todo_left = len(todo)
 
-    def _on_completion(f):
-        if not todo:
-            return  # _on_timeout() was here first.
-        todo.remove(f)
-        done.put_nowait(f)
-        if not todo and timeout_handle is not None:
-            timeout_handle.cancel()
+    def __aiter__(self):
+        return self
 
-    async def _wait_for_one():
-        f = await done.get()
+    def __iter__(self):
+        return self
+
+    async def __anext__(self):
+        if not self._todo_left:
+            raise StopAsyncIteration
+        assert self._todo_left > 0
+        self._todo_left -= 1
+        return await self._wait_for_one()
+
+    def __next__(self):
+        if not self._todo_left:
+            raise StopIteration
+        assert self._todo_left > 0
+        self._todo_left -= 1
+        return self._wait_for_one(resolve=True)
+
+    def _handle_timeout(self):
+        for f in self._todo:
+            f.remove_done_callback(self._handle_completion)
+            self._done.put_nowait(None)  # Sentinel for _wait_for_one().
+        self._todo.clear()  # Can't do todo.remove(f) in the loop.
+
+    def _handle_completion(self, f):
+        if not self._todo:
+            return  # _handle_timeout() was here first.
+        self._todo.remove(f)
+        self._done.put_nowait(f)
+        if not self._todo and self._timeout_handle is not None:
+            self._timeout_handle.cancel()
+
+    async def _wait_for_one(self, resolve=False):
+        # Wait for the next future to be done and return it unless resolve is
+        # set, in which case return either the result of the future or raise
+        # an exception.
+        f = await self._done.get()
         if f is None:
-            # Dummy value from _on_timeout().
+            # Dummy value from _handle_timeout().
             raise exceptions.TimeoutError
-        return f.result()  # May raise f.exception().
+        return f.result() if resolve else f
 
-    for f in todo:
-        f.add_done_callback(_on_completion)
-    if todo and timeout is not None:
-        timeout_handle = loop.call_later(timeout, _on_timeout)
-    for _ in range(len(todo)):
-        yield _wait_for_one()
+
+def as_completed(fs, *, timeout=None):
+    """Create an iterator of awaitables or their results in completion order.
+
+    Run the supplied awaitables concurrently. The returned object can be
+    iterated to obtain the results of the awaitables as they finish.
+
+    The object returned can be iterated as an asynchronous iterator or a plain
+    iterator. When asynchronous iteration is used, the originally-supplied
+    awaitables are yielded if they are tasks or futures. This makes it easy to
+    correlate previously-scheduled tasks with their results:
+
+        ipv4_connect = create_task(open_connection("127.0.0.1", 80))
+        ipv6_connect = create_task(open_connection("::1", 80))
+        tasks = [ipv4_connect, ipv6_connect]
+
+        async for earliest_connect in as_completed(tasks):
+            # earliest_connect is done. The result can be obtained by
+            # awaiting it or calling earliest_connect.result()
+            reader, writer = await earliest_connect
+
+            if earliest_connect is ipv6_connect:
+                print("IPv6 connection established.")
+            else:
+                print("IPv4 connection established.")
+
+    During asynchronous iteration, implicitly-created tasks will be yielded for
+    supplied awaitables that aren't tasks or futures.
+
+    When used as a plain iterator, each iteration yields a new coroutine that
+    returns the result or raises the exception of the next completed awaitable.
+    This pattern is compatible with Python versions older than 3.13:
+
+        ipv4_connect = create_task(open_connection("127.0.0.1", 80))
+        ipv6_connect = create_task(open_connection("::1", 80))
+        tasks = [ipv4_connect, ipv6_connect]
+
+        for next_connect in as_completed(tasks):
+            # next_connect is not one of the original task objects. It must be
+            # awaited to obtain the result value or raise the exception of the
+            # awaitable that finishes next.
+            reader, writer = await next_connect
+
+    A TimeoutError is raised if the timeout occurs before all awaitables are
+    done. This is raised by the async for loop during asynchronous iteration or
+    by the coroutines yielded during plain iteration.
+    """
+    if inspect.isawaitable(fs):
+        raise TypeError(
+            f"expects an iterable of awaitables, not {type(fs).__name__}"
+        )
+
+    return _AsCompletedIterator(fs, timeout)
 
 
 @types.coroutine
@@ -656,6 +690,9 @@ async def sleep(delay, result=None):
         await __sleep0()
         return result
 
+    if math.isnan(delay):
+        raise ValueError("Invalid delay: NaN (not a number)")
+
     loop = events.get_running_loop()
     future = loop.create_future()
     h = loop.call_later(delay,
@@ -764,10 +801,19 @@ def gather(*coros_or_futures, return_exceptions=False):
         outer.set_result([])
         return outer
 
-    def _done_callback(fut):
+    loop = events._get_running_loop()
+    if loop is not None:
+        cur_task = current_task(loop)
+    else:
+        cur_task = None
+
+    def _done_callback(fut, cur_task=cur_task):
         nonlocal nfinished
         nfinished += 1
 
+        if cur_task is not None:
+            futures.future_discard_from_awaited_by(fut, cur_task)
+
         if outer is None or outer.done():
             if not fut.cancelled():
                 # Mark exception retrieved.
@@ -824,7 +870,6 @@ def gather(*coros_or_futures, return_exceptions=False):
     nfuts = 0
     nfinished = 0
     done_futs = []
-    loop = None
     outer = None  # bpo-46672
     for arg in coros_or_futures:
         if arg not in arg_to_fut:
@@ -837,12 +882,13 @@ def gather(*coros_or_futures, return_exceptions=False):
                 # can't control it, disable the "destroy pending task"
                 # warning.
                 fut._log_destroy_pending = False
-
             nfuts += 1
             arg_to_fut[arg] = fut
             if fut.done():
                 done_futs.append(fut)
             else:
+                if cur_task is not None:
+                    futures.future_add_to_awaited_by(fut, cur_task)
                 fut.add_done_callback(_done_callback)
 
         else:
@@ -862,6 +908,25 @@ def gather(*coros_or_futures, return_exceptions=False):
     return outer
 
 
+def _log_on_exception(fut):
+    if fut.cancelled():
+        return
+
+    exc = fut.exception()
+    if exc is None:
+        return
+
+    context = {
+        'message':
+        f'{exc.__class__.__name__} exception in shielded future',
+        'exception': exc,
+        'future': fut,
+    }
+    if fut._source_traceback:
+        context['source_traceback'] = fut._source_traceback
+    fut._loop.call_exception_handler(context)
+
+
 def shield(arg):
     """Wait for a future, shielding it from cancellation.
 
@@ -902,11 +967,16 @@ def shield(arg):
     loop = futures._get_loop(inner)
     outer = loop.create_future()
 
+    if loop is not None and (cur_task := current_task(loop)) is not None:
+        futures.future_add_to_awaited_by(inner, cur_task)
+    else:
+        cur_task = None
+
+    def _clear_awaited_by_callback(inner):
+        futures.future_discard_from_awaited_by(inner, cur_task)
+
     def _inner_done_callback(inner):
         if outer.cancelled():
-            if not inner.cancelled():
-                # Mark inner's result as retrieved.
-                inner.exception()
             return
 
         if inner.cancelled():
@@ -918,10 +988,16 @@ def shield(arg):
             else:
                 outer.set_result(inner.result())
 
-
     def _outer_done_callback(outer):
         if not inner.done():
             inner.remove_done_callback(_inner_done_callback)
+            # Keep only one callback to log on cancel
+            inner.remove_done_callback(_log_on_exception)
+            inner.add_done_callback(_log_on_exception)
+
+    if cur_task is not None:
+        inner.add_done_callback(_clear_awaited_by_callback)
+
 
     inner.add_done_callback(_inner_done_callback)
     outer.add_done_callback(_outer_done_callback)
@@ -970,9 +1046,9 @@ def create_eager_task_factory(custom_task_constructor):
         used. E.g. `loop.set_task_factory(asyncio.eager_task_factory)`.
         """
 
-    def factory(loop, coro, *, name=None, context=None):
+    def factory(loop, coro, *, eager_start=True, **kwargs):
         return custom_task_constructor(
-            coro, loop=loop, name=name, context=context, eager_start=True)
+            coro, loop=loop, eager_start=eager_start, **kwargs)
 
     return factory
 
@@ -1044,14 +1120,13 @@ _py_unregister_eager_task = _unregister_eager_task
 _py_enter_task = _enter_task
 _py_leave_task = _leave_task
 _py_swap_current_task = _swap_current_task
-
+_py_all_tasks = all_tasks
 
 try:
     from _asyncio import (_register_task, _register_eager_task,
                           _unregister_task, _unregister_eager_task,
                           _enter_task, _leave_task, _swap_current_task,
-                          _scheduled_tasks, _eager_tasks, _current_tasks,
-                          current_task)
+                          current_task, all_tasks)
 except ImportError:
     pass
 else:
@@ -1063,3 +1138,4 @@ else:
     _c_enter_task = _enter_task
     _c_leave_task = _leave_task
     _c_swap_current_task = _swap_current_task
+    _c_all_tasks = all_tasks

Lib/asyncio/timeouts.py

@@ -1,7 +1,6 @@
 import enum
 
 from types import TracebackType
-from typing import final, Optional, Type
 
 from . import events
 from . import exceptions
@@ -23,14 +22,13 @@ class _State(enum.Enum):
     EXITED = "finished"
 
 
-@final
 class Timeout:
     """Asynchronous context manager for cancelling overdue coroutines.
 
     Use `timeout()` or `timeout_at()` rather than instantiating this class directly.
     """
 
-    def __init__(self, when: Optional[float]) -> None:
+    def __init__(self, when: float | None) -> None:
         """Schedule a timeout that will trigger at a given loop time.
 
         - If `when` is `None`, the timeout will never trigger.
@@ -39,15 +37,15 @@ class Timeout:
         """
         self._state = _State.CREATED
 
-        self._timeout_handler: Optional[events.TimerHandle] = None
-        self._task: Optional[tasks.Task] = None
+        self._timeout_handler: events.TimerHandle | None = None
+        self._task: tasks.Task | None = None
         self._when = when
 
-    def when(self) -> Optional[float]:
+    def when(self) -> float | None:
         """Return the current deadline."""
         return self._when
 
-    def reschedule(self, when: Optional[float]) -> None:
+    def reschedule(self, when: float | None) -> None:
         """Reschedule the timeout."""
         if self._state is not _State.ENTERED:
             if self._state is _State.CREATED:
@@ -96,10 +94,10 @@ class Timeout:
 
     async def __aexit__(
         self,
-        exc_type: Optional[Type[BaseException]],
-        exc_val: Optional[BaseException],
-        exc_tb: Optional[TracebackType],
-    ) -> Optional[bool]:
+        exc_type: type[BaseException] | None,
+        exc_val: BaseException | None,
+        exc_tb: TracebackType | None,
+    ) -> bool | None:
         assert self._state in (_State.ENTERED, _State.EXPIRING)
 
         if self._timeout_handler is not None:
@@ -109,10 +107,16 @@ class Timeout:
         if self._state is _State.EXPIRING:
             self._state = _State.EXPIRED
 
-            if self._task.uncancel() <= self._cancelling and exc_type is exceptions.CancelledError:
+            if self._task.uncancel() <= self._cancelling and exc_type is not None:
                 # Since there are no new cancel requests, we're
                 # handling this.
-                raise TimeoutError from exc_val
+                if issubclass(exc_type, exceptions.CancelledError):
+                    raise TimeoutError from exc_val
+                elif exc_val is not None:
+                    self._insert_timeout_error(exc_val)
+                    if isinstance(exc_val, ExceptionGroup):
+                        for exc in exc_val.exceptions:
+                            self._insert_timeout_error(exc)
         elif self._state is _State.ENTERED:
             self._state = _State.EXITED
 
@@ -125,8 +129,18 @@ class Timeout:
         # drop the reference early
         self._timeout_handler = None
 
+    @staticmethod
+    def _insert_timeout_error(exc_val: BaseException) -> None:
+        while exc_val.__context__ is not None:
+            if isinstance(exc_val.__context__, exceptions.CancelledError):
+                te = TimeoutError()
+                te.__context__ = te.__cause__ = exc_val.__context__
+                exc_val.__context__ = te
+                break
+            exc_val = exc_val.__context__
 
-def timeout(delay: Optional[float]) -> Timeout:
+
+def timeout(delay: float | None) -> Timeout:
     """Timeout async context manager.
 
     Useful in cases when you want to apply timeout logic around block
@@ -146,7 +160,7 @@ def timeout(delay: Optional[float]) -> Timeout:
     return Timeout(loop.time() + delay if delay is not None else None)
 
 
-def timeout_at(when: Optional[float]) -> Timeout:
+def timeout_at(when: float | None) -> Timeout:
     """Schedule the timeout at absolute time.
 
     Like timeout() but argument gives absolute time in the same clock system

Lib/asyncio/tools.py

@@ -0,0 +1,276 @@
+"""Tools to analyze tasks running in asyncio programs."""
+
+from collections import defaultdict, namedtuple
+from itertools import count
+from enum import Enum
+import sys
+from _remote_debugging import RemoteUnwinder, FrameInfo
+
+class NodeType(Enum):
+    COROUTINE = 1
+    TASK = 2
+
+
+class CycleFoundException(Exception):
+    """Raised when there is a cycle when drawing the call tree."""
+    def __init__(
+            self,
+            cycles: list[list[int]],
+            id2name: dict[int, str],
+        ) -> None:
+        super().__init__(cycles, id2name)
+        self.cycles = cycles
+        self.id2name = id2name
+
+
+
+# ─── indexing helpers ───────────────────────────────────────────
+def _format_stack_entry(elem: str|FrameInfo) -> str:
+    if not isinstance(elem, str):
+        if elem.lineno == 0 and elem.filename == "":
+            return f"{elem.funcname}"
+        else:
+            return f"{elem.funcname} {elem.filename}:{elem.lineno}"
+    return elem
+
+
+def _index(result):
+    id2name, awaits, task_stacks = {}, [], {}
+    for awaited_info in result:
+        for task_info in awaited_info.awaited_by:
+            task_id = task_info.task_id
+            task_name = task_info.task_name
+            id2name[task_id] = task_name
+
+            # Store the internal coroutine stack for this task
+            if task_info.coroutine_stack:
+                for coro_info in task_info.coroutine_stack:
+                    call_stack = coro_info.call_stack
+                    internal_stack = [_format_stack_entry(frame) for frame in call_stack]
+                    task_stacks[task_id] = internal_stack
+
+            # Add the awaited_by relationships (external dependencies)
+            if task_info.awaited_by:
+                for coro_info in task_info.awaited_by:
+                    call_stack = coro_info.call_stack
+                    parent_task_id = coro_info.task_name
+                    stack = [_format_stack_entry(frame) for frame in call_stack]
+                    awaits.append((parent_task_id, stack, task_id))
+    return id2name, awaits, task_stacks
+
+
+def _build_tree(id2name, awaits, task_stacks):
+    id2label = {(NodeType.TASK, tid): name for tid, name in id2name.items()}
+    children = defaultdict(list)
+    cor_nodes = defaultdict(dict)  # Maps parent -> {frame_name: node_key}
+    next_cor_id = count(1)
+
+    def get_or_create_cor_node(parent, frame):
+        """Get existing coroutine node or create new one under parent"""
+        if frame in cor_nodes[parent]:
+            return cor_nodes[parent][frame]
+
+        node_key = (NodeType.COROUTINE, f"c{next(next_cor_id)}")
+        id2label[node_key] = frame
+        children[parent].append(node_key)
+        cor_nodes[parent][frame] = node_key
+        return node_key
+
+    # Build task dependency tree with coroutine frames
+    for parent_id, stack, child_id in awaits:
+        cur = (NodeType.TASK, parent_id)
+        for frame in reversed(stack):
+            cur = get_or_create_cor_node(cur, frame)
+
+        child_key = (NodeType.TASK, child_id)
+        if child_key not in children[cur]:
+            children[cur].append(child_key)
+
+    # Add coroutine stacks for leaf tasks
+    awaiting_tasks = {parent_id for parent_id, _, _ in awaits}
+    for task_id in id2name:
+        if task_id not in awaiting_tasks and task_id in task_stacks:
+            cur = (NodeType.TASK, task_id)
+            for frame in reversed(task_stacks[task_id]):
+                cur = get_or_create_cor_node(cur, frame)
+
+    return id2label, children
+
+
+def _roots(id2label, children):
+    all_children = {c for kids in children.values() for c in kids}
+    return [n for n in id2label if n not in all_children]
+
+# ─── detect cycles in the task-to-task graph ───────────────────────
+def _task_graph(awaits):
+    """Return {parent_task_id: {child_task_id, …}, …}."""
+    g = defaultdict(set)
+    for parent_id, _stack, child_id in awaits:
+        g[parent_id].add(child_id)
+    return g
+
+
+def _find_cycles(graph):
+    """
+    Depth-first search for back-edges.
+
+    Returns a list of cycles (each cycle is a list of task-ids) or an
+    empty list if the graph is acyclic.
+    """
+    WHITE, GREY, BLACK = 0, 1, 2
+    color = defaultdict(lambda: WHITE)
+    path, cycles = [], []
+
+    def dfs(v):
+        color[v] = GREY
+        path.append(v)
+        for w in graph.get(v, ()):
+            if color[w] == WHITE:
+                dfs(w)
+            elif color[w] == GREY:            # back-edge → cycle!
+                i = path.index(w)
+                cycles.append(path[i:] + [w])  # make a copy
+        color[v] = BLACK
+        path.pop()
+
+    for v in list(graph):
+        if color[v] == WHITE:
+            dfs(v)
+    return cycles
+
+
+# ─── PRINT TREE FUNCTION ───────────────────────────────────────
+def get_all_awaited_by(pid):
+    unwinder = RemoteUnwinder(pid)
+    return unwinder.get_all_awaited_by()
+
+
+def build_async_tree(result, task_emoji="(T)", cor_emoji=""):
+    """
+    Build a list of strings for pretty-print an async call tree.
+
+    The call tree is produced by `get_all_async_stacks()`, prefixing tasks
+    with `task_emoji` and coroutine frames with `cor_emoji`.
+    """
+    id2name, awaits, task_stacks = _index(result)
+    g = _task_graph(awaits)
+    cycles = _find_cycles(g)
+    if cycles:
+        raise CycleFoundException(cycles, id2name)
+    labels, children = _build_tree(id2name, awaits, task_stacks)
+
+    def pretty(node):
+        flag = task_emoji if node[0] == NodeType.TASK else cor_emoji
+        return f"{flag} {labels[node]}"
+
+    def render(node, prefix="", last=True, buf=None):
+        if buf is None:
+            buf = []
+        buf.append(f"{prefix}{'└── ' if last else '├── '}{pretty(node)}")
+        new_pref = prefix + ("    " if last else "│   ")
+        kids = children.get(node, [])
+        for i, kid in enumerate(kids):
+            render(kid, new_pref, i == len(kids) - 1, buf)
+        return buf
+
+    return [render(root) for root in _roots(labels, children)]
+
+
+def build_task_table(result):
+    id2name, _, _ = _index(result)
+    table = []
+
+    for awaited_info in result:
+        thread_id = awaited_info.thread_id
+        for task_info in awaited_info.awaited_by:
+            # Get task info
+            task_id = task_info.task_id
+            task_name = task_info.task_name
+
+            # Build coroutine stack string
+            frames = [frame for coro in task_info.coroutine_stack
+                     for frame in coro.call_stack]
+            coro_stack = " -> ".join(_format_stack_entry(x).split(" ")[0]
+                                   for x in frames)
+
+            # Handle tasks with no awaiters
+            if not task_info.awaited_by:
+                table.append([thread_id, hex(task_id), task_name, coro_stack,
+                            "", "", "0x0"])
+                continue
+
+            # Handle tasks with awaiters
+            for coro_info in task_info.awaited_by:
+                parent_id = coro_info.task_name
+                awaiter_frames = [_format_stack_entry(x).split(" ")[0]
+                                for x in coro_info.call_stack]
+                awaiter_chain = " -> ".join(awaiter_frames)
+                awaiter_name = id2name.get(parent_id, "Unknown")
+                parent_id_str = (hex(parent_id) if isinstance(parent_id, int)
+                               else str(parent_id))
+
+                table.append([thread_id, hex(task_id), task_name, coro_stack,
+                            awaiter_chain, awaiter_name, parent_id_str])
+
+    return table
+
+def _print_cycle_exception(exception: CycleFoundException):
+    print("ERROR: await-graph contains cycles - cannot print a tree!", file=sys.stderr)
+    print("", file=sys.stderr)
+    for c in exception.cycles:
+        inames = " → ".join(exception.id2name.get(tid, hex(tid)) for tid in c)
+        print(f"cycle: {inames}", file=sys.stderr)
+
+
+def exit_with_permission_help_text():
+    """
+    Prints a message pointing to platform-specific permission help text and exits the program.
+    This function is called when a PermissionError is encountered while trying
+    to attach to a process.
+    """
+    print(
+        "Error: The specified process cannot be attached to due to insufficient permissions.\n"
+        "See the Python documentation for details on required privileges and troubleshooting:\n"
+        "https://docs.python.org/3.14/howto/remote_debugging.html#permission-requirements\n"
+    )
+    sys.exit(1)
+
+
+def _get_awaited_by_tasks(pid: int) -> list:
+    try:
+        return get_all_awaited_by(pid)
+    except RuntimeError as e:
+        while e.__context__ is not None:
+            e = e.__context__
+        print(f"Error retrieving tasks: {e}")
+        sys.exit(1)
+    except PermissionError as e:
+        exit_with_permission_help_text()
+
+
+def display_awaited_by_tasks_table(pid: int) -> None:
+    """Build and print a table of all pending tasks under `pid`."""
+
+    tasks = _get_awaited_by_tasks(pid)
+    table = build_task_table(tasks)
+    # Print the table in a simple tabular format
+    print(
+        f"{'tid':<10} {'task id':<20} {'task name':<20} {'coroutine stack':<50} {'awaiter chain':<50} {'awaiter name':<15} {'awaiter id':<15}"
+    )
+    print("-" * 180)
+    for row in table:
+        print(f"{row[0]:<10} {row[1]:<20} {row[2]:<20} {row[3]:<50} {row[4]:<50} {row[5]:<15} {row[6]:<15}")
+
+
+def display_awaited_by_tasks_tree(pid: int) -> None:
+    """Build and print a tree of all pending tasks under `pid`."""
+
+    tasks = _get_awaited_by_tasks(pid)
+    try:
+        result = build_async_tree(tasks)
+    except CycleFoundException as e:
+        _print_cycle_exception(e)
+        sys.exit(1)
+
+    for tree in result:
+        print("\n".join(tree))

Lib/asyncio/transports.py

@@ -181,6 +181,8 @@ class DatagramTransport(BaseTransport):
         to be sent out asynchronously.
         addr is target socket address.
         If addr is None use target address pointed on transport creation.
+        If data is an empty bytes object a zero-length datagram will be
+        sent.
         """
         raise NotImplementedError
 

Lib/asyncio/unix_events.py

@@ -28,10 +28,7 @@ from .log import logger
 
 __all__ = (
     'SelectorEventLoop',
-    'AbstractChildWatcher', 'SafeChildWatcher',
-    'FastChildWatcher', 'PidfdChildWatcher',
-    'MultiLoopChildWatcher', 'ThreadedChildWatcher',
-    'DefaultEventLoopPolicy',
+    'EventLoop',
 )
 
 
@@ -63,6 +60,11 @@ class _UnixSelectorEventLoop(selector_events.BaseSelectorEventLoop):
     def __init__(self, selector=None):
         super().__init__(selector)
         self._signal_handlers = {}
+        self._unix_server_sockets = {}
+        if can_use_pidfd():
+            self._watcher = _PidfdChildWatcher()
+        else:
+            self._watcher = _ThreadedChildWatcher()
 
     def close(self):
         super().close()
@@ -92,7 +94,7 @@ class _UnixSelectorEventLoop(selector_events.BaseSelectorEventLoop):
         Raise RuntimeError if there is a problem setting up the handler.
         """
         if (coroutines.iscoroutine(callback) or
-                coroutines.iscoroutinefunction(callback)):
+                coroutines._iscoroutinefunction(callback)):
             raise TypeError("coroutines cannot be used "
                             "with add_signal_handler()")
         self._check_signal(sig)
@@ -195,33 +197,22 @@ class _UnixSelectorEventLoop(selector_events.BaseSelectorEventLoop):
     async def _make_subprocess_transport(self, protocol, args, shell,
                                          stdin, stdout, stderr, bufsize,
                                          extra=None, **kwargs):
-        with warnings.catch_warnings():
-            warnings.simplefilter('ignore', DeprecationWarning)
-            watcher = events.get_child_watcher()
-
-        with watcher:
-            if not watcher.is_active():
-                # Check early.
-                # Raising exception before process creation
-                # prevents subprocess execution if the watcher
-                # is not ready to handle it.
-                raise RuntimeError("asyncio.get_child_watcher() is not activated, "
-                                "subprocess support is not installed.")
-            waiter = self.create_future()
-            transp = _UnixSubprocessTransport(self, protocol, args, shell,
-                                            stdin, stdout, stderr, bufsize,
-                                            waiter=waiter, extra=extra,
-                                            **kwargs)
-            watcher.add_child_handler(transp.get_pid(),
-                                    self._child_watcher_callback, transp)
-            try:
-                await waiter
-            except (SystemExit, KeyboardInterrupt):
-                raise
-            except BaseException:
-                transp.close()
-                await transp._wait()
-                raise
+        watcher = self._watcher
+        waiter = self.create_future()
+        transp = _UnixSubprocessTransport(self, protocol, args, shell,
+                                        stdin, stdout, stderr, bufsize,
+                                        waiter=waiter, extra=extra,
+                                        **kwargs)
+        watcher.add_child_handler(transp.get_pid(),
+                                self._child_watcher_callback, transp)
+        try:
+            await waiter
+        except (SystemExit, KeyboardInterrupt):
+            raise
+        except BaseException:
+            transp.close()
+            await transp._wait()
+            raise
 
         return transp
 
@@ -283,7 +274,7 @@ class _UnixSelectorEventLoop(selector_events.BaseSelectorEventLoop):
             sock=None, backlog=100, ssl=None,
             ssl_handshake_timeout=None,
             ssl_shutdown_timeout=None,
-            start_serving=True):
+            start_serving=True, cleanup_socket=True):
         if isinstance(ssl, bool):
             raise TypeError('ssl argument must be an SSLContext or None')
 
@@ -339,6 +330,15 @@ class _UnixSelectorEventLoop(selector_events.BaseSelectorEventLoop):
                 raise ValueError(
                     f'A UNIX Domain Stream Socket was expected, got {sock!r}')
 
+        if cleanup_socket:
+            path = sock.getsockname()
+            # Check for abstract socket. `str` and `bytes` paths are supported.
+            if path[0] not in (0, '\x00'):
+                try:
+                    self._unix_server_sockets[sock] = os.stat(path).st_ino
+                except FileNotFoundError:
+                    pass
+
         sock.setblocking(False)
         server = base_events.Server(self, [sock], protocol_factory,
                                     ssl, backlog, ssl_handshake_timeout,
@@ -393,6 +393,9 @@ class _UnixSelectorEventLoop(selector_events.BaseSelectorEventLoop):
                 fut.set_result(total_sent)
                 return
 
+        # On 32-bit architectures truncate to 1GiB to avoid OverflowError
+        blocksize = min(blocksize, sys.maxsize//2 + 1)
+
         try:
             sent = os.sendfile(fd, fileno, offset, blocksize)
         except (BlockingIOError, InterruptedError):
@@ -456,6 +459,27 @@ class _UnixSelectorEventLoop(selector_events.BaseSelectorEventLoop):
                     self.remove_writer(fd)
         fut.add_done_callback(cb)
 
+    def _stop_serving(self, sock):
+        # Is this a unix socket that needs cleanup?
+        if sock in self._unix_server_sockets:
+            path = sock.getsockname()
+        else:
+            path = None
+
+        super()._stop_serving(sock)
+
+        if path is not None:
+            prev_ino = self._unix_server_sockets[sock]
+            del self._unix_server_sockets[sock]
+            try:
+                if os.stat(path).st_ino == prev_ino:
+                    os.unlink(path)
+            except FileNotFoundError:
+                pass
+            except OSError as err:
+                logger.error('Unable to clean up listening UNIX socket '
+                             '%r: %r', path, err)
+
 
 class _UnixReadPipeTransport(transports.ReadTransport):
 
@@ -830,93 +854,7 @@ class _UnixSubprocessTransport(base_subprocess.BaseSubprocessTransport):
                 stdin_w.close()
 
 
-class AbstractChildWatcher:
-    """Abstract base class for monitoring child processes.
-
-    Objects derived from this class monitor a collection of subprocesses and
-    report their termination or interruption by a signal.
-
-    New callbacks are registered with .add_child_handler(). Starting a new
-    process must be done within a 'with' block to allow the watcher to suspend
-    its activity until the new process if fully registered (this is needed to
-    prevent a race condition in some implementations).
-
-    Example:
-        with watcher:
-            proc = subprocess.Popen("sleep 1")
-            watcher.add_child_handler(proc.pid, callback)
-
-    Notes:
-        Implementations of this class must be thread-safe.
-
-        Since child watcher objects may catch the SIGCHLD signal and call
-        waitpid(-1), there should be only one active object per process.
-    """
-
-    def __init_subclass__(cls) -> None:
-        if cls.__module__ != __name__:
-            warnings._deprecated("AbstractChildWatcher",
-                             "{name!r} is deprecated as of Python 3.12 and will be "
-                             "removed in Python {remove}.",
-                              remove=(3, 14))
-
-    def add_child_handler(self, pid, callback, *args):
-        """Register a new child handler.
-
-        Arrange for callback(pid, returncode, *args) to be called when
-        process 'pid' terminates. Specifying another callback for the same
-        process replaces the previous handler.
-
-        Note: callback() must be thread-safe.
-        """
-        raise NotImplementedError()
-
-    def remove_child_handler(self, pid):
-        """Removes the handler for process 'pid'.
-
-        The function returns True if the handler was successfully removed,
-        False if there was nothing to remove."""
-
-        raise NotImplementedError()
-
-    def attach_loop(self, loop):
-        """Attach the watcher to an event loop.
-
-        If the watcher was previously attached to an event loop, then it is
-        first detached before attaching to the new loop.
-
-        Note: loop may be None.
-        """
-        raise NotImplementedError()
-
-    def close(self):
-        """Close the watcher.
-
-        This must be called to make sure that any underlying resource is freed.
-        """
-        raise NotImplementedError()
-
-    def is_active(self):
-        """Return ``True`` if the watcher is active and is used by the event loop.
-
-        Return True if the watcher is installed and ready to handle process exit
-        notifications.
-
-        """
-        raise NotImplementedError()
-
-    def __enter__(self):
-        """Enter the watcher's context and allow starting new processes
-
-        This function must return self"""
-        raise NotImplementedError()
-
-    def __exit__(self, a, b, c):
-        """Exit the watcher's context"""
-        raise NotImplementedError()
-
-
-class PidfdChildWatcher(AbstractChildWatcher):
+class _PidfdChildWatcher:
     """Child watcher implementation using Linux's pid file descriptors.
 
     This child watcher polls process file descriptors (pidfds) to await child
@@ -928,21 +866,6 @@ class PidfdChildWatcher(AbstractChildWatcher):
     recent (5.3+) kernels.
     """
 
-    def __enter__(self):
-        return self
-
-    def __exit__(self, exc_type, exc_value, exc_traceback):
-        pass
-
-    def is_active(self):
-        return True
-
-    def close(self):
-        pass
-
-    def attach_loop(self, loop):
-        pass
-
     def add_child_handler(self, pid, callback, *args):
         loop = events.get_running_loop()
         pidfd = os.pidfd_open(pid)
@@ -967,386 +890,7 @@ class PidfdChildWatcher(AbstractChildWatcher):
         os.close(pidfd)
         callback(pid, returncode, *args)
 
-    def remove_child_handler(self, pid):
-        # asyncio never calls remove_child_handler() !!!
-        # The method is no-op but is implemented because
-        # abstract base classes require it.
-        return True
-
-
-class BaseChildWatcher(AbstractChildWatcher):
-
-    def __init__(self):
-        self._loop = None
-        self._callbacks = {}
-
-    def close(self):
-        self.attach_loop(None)
-
-    def is_active(self):
-        return self._loop is not None and self._loop.is_running()
-
-    def _do_waitpid(self, expected_pid):
-        raise NotImplementedError()
-
-    def _do_waitpid_all(self):
-        raise NotImplementedError()
-
-    def attach_loop(self, loop):
-        assert loop is None or isinstance(loop, events.AbstractEventLoop)
-
-        if self._loop is not None and loop is None and self._callbacks:
-            warnings.warn(
-                'A loop is being detached '
-                'from a child watcher with pending handlers',
-                RuntimeWarning)
-
-        if self._loop is not None:
-            self._loop.remove_signal_handler(signal.SIGCHLD)
-
-        self._loop = loop
-        if loop is not None:
-            loop.add_signal_handler(signal.SIGCHLD, self._sig_chld)
-
-            # Prevent a race condition in case a child terminated
-            # during the switch.
-            self._do_waitpid_all()
-
-    def _sig_chld(self):
-        try:
-            self._do_waitpid_all()
-        except (SystemExit, KeyboardInterrupt):
-            raise
-        except BaseException as exc:
-            # self._loop should always be available here
-            # as '_sig_chld' is added as a signal handler
-            # in 'attach_loop'
-            self._loop.call_exception_handler({
-                'message': 'Unknown exception in SIGCHLD handler',
-                'exception': exc,
-            })
-
-
-class SafeChildWatcher(BaseChildWatcher):
-    """'Safe' child watcher implementation.
-
-    This implementation avoids disrupting other code spawning processes by
-    polling explicitly each process in the SIGCHLD handler instead of calling
-    os.waitpid(-1).
-
-    This is a safe solution but it has a significant overhead when handling a
-    big number of children (O(n) each time SIGCHLD is raised)
-    """
-
-    def __init__(self):
-        super().__init__()
-        warnings._deprecated("SafeChildWatcher",
-                             "{name!r} is deprecated as of Python 3.12 and will be "
-                             "removed in Python {remove}.",
-                              remove=(3, 14))
-
-    def close(self):
-        self._callbacks.clear()
-        super().close()
-
-    def __enter__(self):
-        return self
-
-    def __exit__(self, a, b, c):
-        pass
-
-    def add_child_handler(self, pid, callback, *args):
-        self._callbacks[pid] = (callback, args)
-
-        # Prevent a race condition in case the child is already terminated.
-        self._do_waitpid(pid)
-
-    def remove_child_handler(self, pid):
-        try:
-            del self._callbacks[pid]
-            return True
-        except KeyError:
-            return False
-
-    def _do_waitpid_all(self):
-
-        for pid in list(self._callbacks):
-            self._do_waitpid(pid)
-
-    def _do_waitpid(self, expected_pid):
-        assert expected_pid > 0
-
-        try:
-            pid, status = os.waitpid(expected_pid, os.WNOHANG)
-        except ChildProcessError:
-            # The child process is already reaped
-            # (may happen if waitpid() is called elsewhere).
-            pid = expected_pid
-            returncode = 255
-            logger.warning(
-                "Unknown child process pid %d, will report returncode 255",
-                pid)
-        else:
-            if pid == 0:
-                # The child process is still alive.
-                return
-
-            returncode = waitstatus_to_exitcode(status)
-            if self._loop.get_debug():
-                logger.debug('process %s exited with returncode %s',
-                             expected_pid, returncode)
-
-        try:
-            callback, args = self._callbacks.pop(pid)
-        except KeyError:  # pragma: no cover
-            # May happen if .remove_child_handler() is called
-            # after os.waitpid() returns.
-            if self._loop.get_debug():
-                logger.warning("Child watcher got an unexpected pid: %r",
-                               pid, exc_info=True)
-        else:
-            callback(pid, returncode, *args)
-
-
-class FastChildWatcher(BaseChildWatcher):
-    """'Fast' child watcher implementation.
-
-    This implementation reaps every terminated processes by calling
-    os.waitpid(-1) directly, possibly breaking other code spawning processes
-    and waiting for their termination.
-
-    There is no noticeable overhead when handling a big number of children
-    (O(1) each time a child terminates).
-    """
-    def __init__(self):
-        super().__init__()
-        self._lock = threading.Lock()
-        self._zombies = {}
-        self._forks = 0
-        warnings._deprecated("FastChildWatcher",
-                             "{name!r} is deprecated as of Python 3.12 and will be "
-                             "removed in Python {remove}.",
-                              remove=(3, 14))
-
-    def close(self):
-        self._callbacks.clear()
-        self._zombies.clear()
-        super().close()
-
-    def __enter__(self):
-        with self._lock:
-            self._forks += 1
-
-            return self
-
-    def __exit__(self, a, b, c):
-        with self._lock:
-            self._forks -= 1
-
-            if self._forks or not self._zombies:
-                return
-
-            collateral_victims = str(self._zombies)
-            self._zombies.clear()
-
-        logger.warning(
-            "Caught subprocesses termination from unknown pids: %s",
-            collateral_victims)
-
-    def add_child_handler(self, pid, callback, *args):
-        assert self._forks, "Must use the context manager"
-
-        with self._lock:
-            try:
-                returncode = self._zombies.pop(pid)
-            except KeyError:
-                # The child is running.
-                self._callbacks[pid] = callback, args
-                return
-
-        # The child is dead already. We can fire the callback.
-        callback(pid, returncode, *args)
-
-    def remove_child_handler(self, pid):
-        try:
-            del self._callbacks[pid]
-            return True
-        except KeyError:
-            return False
-
-    def _do_waitpid_all(self):
-        # Because of signal coalescing, we must keep calling waitpid() as
-        # long as we're able to reap a child.
-        while True:
-            try:
-                pid, status = os.waitpid(-1, os.WNOHANG)
-            except ChildProcessError:
-                # No more child processes exist.
-                return
-            else:
-                if pid == 0:
-                    # A child process is still alive.
-                    return
-
-                returncode = waitstatus_to_exitcode(status)
-
-            with self._lock:
-                try:
-                    callback, args = self._callbacks.pop(pid)
-                except KeyError:
-                    # unknown child
-                    if self._forks:
-                        # It may not be registered yet.
-                        self._zombies[pid] = returncode
-                        if self._loop.get_debug():
-                            logger.debug('unknown process %s exited '
-                                         'with returncode %s',
-                                         pid, returncode)
-                        continue
-                    callback = None
-                else:
-                    if self._loop.get_debug():
-                        logger.debug('process %s exited with returncode %s',
-                                     pid, returncode)
-
-            if callback is None:
-                logger.warning(
-                    "Caught subprocess termination from unknown pid: "
-                    "%d -> %d", pid, returncode)
-            else:
-                callback(pid, returncode, *args)
-
-
-class MultiLoopChildWatcher(AbstractChildWatcher):
-    """A watcher that doesn't require running loop in the main thread.
-
-    This implementation registers a SIGCHLD signal handler on
-    instantiation (which may conflict with other code that
-    install own handler for this signal).
-
-    The solution is safe but it has a significant overhead when
-    handling a big number of processes (*O(n)* each time a
-    SIGCHLD is received).
-    """
-
-    # Implementation note:
-    # The class keeps compatibility with AbstractChildWatcher ABC
-    # To achieve this it has empty attach_loop() method
-    # and doesn't accept explicit loop argument
-    # for add_child_handler()/remove_child_handler()
-    # but retrieves the current loop by get_running_loop()
-
-    def __init__(self):
-        self._callbacks = {}
-        self._saved_sighandler = None
-        warnings._deprecated("MultiLoopChildWatcher",
-                             "{name!r} is deprecated as of Python 3.12 and will be "
-                             "removed in Python {remove}.",
-                              remove=(3, 14))
-
-    def is_active(self):
-        return self._saved_sighandler is not None
-
-    def close(self):
-        self._callbacks.clear()
-        if self._saved_sighandler is None:
-            return
-
-        handler = signal.getsignal(signal.SIGCHLD)
-        if handler != self._sig_chld:
-            logger.warning("SIGCHLD handler was changed by outside code")
-        else:
-            signal.signal(signal.SIGCHLD, self._saved_sighandler)
-        self._saved_sighandler = None
-
-    def __enter__(self):
-        return self
-
-    def __exit__(self, exc_type, exc_val, exc_tb):
-        pass
-
-    def add_child_handler(self, pid, callback, *args):
-        loop = events.get_running_loop()
-        self._callbacks[pid] = (loop, callback, args)
-
-        # Prevent a race condition in case the child is already terminated.
-        self._do_waitpid(pid)
-
-    def remove_child_handler(self, pid):
-        try:
-            del self._callbacks[pid]
-            return True
-        except KeyError:
-            return False
-
-    def attach_loop(self, loop):
-        # Don't save the loop but initialize itself if called first time
-        # The reason to do it here is that attach_loop() is called from
-        # unix policy only for the main thread.
-        # Main thread is required for subscription on SIGCHLD signal
-        if self._saved_sighandler is not None:
-            return
-
-        self._saved_sighandler = signal.signal(signal.SIGCHLD, self._sig_chld)
-        if self._saved_sighandler is None:
-            logger.warning("Previous SIGCHLD handler was set by non-Python code, "
-                           "restore to default handler on watcher close.")
-            self._saved_sighandler = signal.SIG_DFL
-
-        # Set SA_RESTART to limit EINTR occurrences.
-        signal.siginterrupt(signal.SIGCHLD, False)
-
-    def _do_waitpid_all(self):
-        for pid in list(self._callbacks):
-            self._do_waitpid(pid)
-
-    def _do_waitpid(self, expected_pid):
-        assert expected_pid > 0
-
-        try:
-            pid, status = os.waitpid(expected_pid, os.WNOHANG)
-        except ChildProcessError:
-            # The child process is already reaped
-            # (may happen if waitpid() is called elsewhere).
-            pid = expected_pid
-            returncode = 255
-            logger.warning(
-                "Unknown child process pid %d, will report returncode 255",
-                pid)
-            debug_log = False
-        else:
-            if pid == 0:
-                # The child process is still alive.
-                return
-
-            returncode = waitstatus_to_exitcode(status)
-            debug_log = True
-        try:
-            loop, callback, args = self._callbacks.pop(pid)
-        except KeyError:  # pragma: no cover
-            # May happen if .remove_child_handler() is called
-            # after os.waitpid() returns.
-            logger.warning("Child watcher got an unexpected pid: %r",
-                           pid, exc_info=True)
-        else:
-            if loop.is_closed():
-                logger.warning("Loop %r that handles pid %r is closed", loop, pid)
-            else:
-                if debug_log and loop.get_debug():
-                    logger.debug('process %s exited with returncode %s',
-                                 expected_pid, returncode)
-                loop.call_soon_threadsafe(callback, pid, returncode, *args)
-
-    def _sig_chld(self, signum, frame):
-        try:
-            self._do_waitpid_all()
-        except (SystemExit, KeyboardInterrupt):
-            raise
-        except BaseException:
-            logger.warning('Unknown exception in SIGCHLD handler', exc_info=True)
-
-
-class ThreadedChildWatcher(AbstractChildWatcher):
+class _ThreadedChildWatcher:
     """Threaded child watcher implementation.
 
     The watcher uses a thread per process
@@ -1363,18 +907,6 @@ class ThreadedChildWatcher(AbstractChildWatcher):
         self._pid_counter = itertools.count(0)
         self._threads = {}
 
-    def is_active(self):
-        return True
-
-    def close(self):
-        pass
-
-    def __enter__(self):
-        return self
-
-    def __exit__(self, exc_type, exc_val, exc_tb):
-        pass
-
     def __del__(self, _warn=warnings.warn):
         threads = [thread for thread in list(self._threads.values())
                    if thread.is_alive()]
@@ -1392,15 +924,6 @@ class ThreadedChildWatcher(AbstractChildWatcher):
         self._threads[pid] = thread
         thread.start()
 
-    def remove_child_handler(self, pid):
-        # asyncio never calls remove_child_handler() !!!
-        # The method is no-op but is implemented because
-        # abstract base classes require it.
-        return True
-
-    def attach_loop(self, loop):
-        pass
-
     def _do_waitpid(self, loop, expected_pid, callback, args):
         assert expected_pid > 0
 
@@ -1439,62 +962,11 @@ def can_use_pidfd():
     return True
 
 
-class _UnixDefaultEventLoopPolicy(events.BaseDefaultEventLoopPolicy):
-    """UNIX event loop policy with a watcher for child processes."""
+class _UnixDefaultEventLoopPolicy(events._BaseDefaultEventLoopPolicy):
+    """UNIX event loop policy"""
     _loop_factory = _UnixSelectorEventLoop
 
-    def __init__(self):
-        super().__init__()
-        self._watcher = None
-
-    def _init_watcher(self):
-        with events._lock:
-            if self._watcher is None:  # pragma: no branch
-                if can_use_pidfd():
-                    self._watcher = PidfdChildWatcher()
-                else:
-                    self._watcher = ThreadedChildWatcher()
-
-    def set_event_loop(self, loop):
-        """Set the event loop.
-
-        As a side effect, if a child watcher was set before, then calling
-        .set_event_loop() from the main thread will call .attach_loop(loop) on
-        the child watcher.
-        """
-
-        super().set_event_loop(loop)
-
-        if (self._watcher is not None and
-                threading.current_thread() is threading.main_thread()):
-            self._watcher.attach_loop(loop)
-
-    def get_child_watcher(self):
-        """Get the watcher for child processes.
-
-        If not yet set, a ThreadedChildWatcher object is automatically created.
-        """
-        if self._watcher is None:
-            self._init_watcher()
-
-        warnings._deprecated("get_child_watcher",
-                            "{name!r} is deprecated as of Python 3.12 and will be "
-                            "removed in Python {remove}.", remove=(3, 14))
-        return self._watcher
-
-    def set_child_watcher(self, watcher):
-        """Set the watcher for child processes."""
-
-        assert watcher is None or isinstance(watcher, AbstractChildWatcher)
-
-        if self._watcher is not None:
-            self._watcher.close()
-
-        self._watcher = watcher
-        warnings._deprecated("set_child_watcher",
-                            "{name!r} is deprecated as of Python 3.12 and will be "
-                            "removed in Python {remove}.", remove=(3, 14))
-
 
 SelectorEventLoop = _UnixSelectorEventLoop
-DefaultEventLoopPolicy = _UnixDefaultEventLoopPolicy
+_DefaultEventLoopPolicy = _UnixDefaultEventLoopPolicy
+EventLoop = SelectorEventLoop

Lib/asyncio/windows_events.py

@@ -29,8 +29,8 @@ from .log import logger
 
 __all__ = (
     'SelectorEventLoop', 'ProactorEventLoop', 'IocpProactor',
-    'DefaultEventLoopPolicy', 'WindowsSelectorEventLoopPolicy',
-    'WindowsProactorEventLoopPolicy',
+    '_DefaultEventLoopPolicy', '_WindowsSelectorEventLoopPolicy',
+    '_WindowsProactorEventLoopPolicy', 'EventLoop',
 )
 
 
@@ -315,24 +315,25 @@ class ProactorEventLoop(proactor_events.BaseProactorEventLoop):
             proactor = IocpProactor()
         super().__init__(proactor)
 
-    def run_forever(self):
-        try:
-            assert self._self_reading_future is None
-            self.call_soon(self._loop_self_reading)
-            super().run_forever()
-        finally:
-            if self._self_reading_future is not None:
-                ov = self._self_reading_future._ov
-                self._self_reading_future.cancel()
-                # self_reading_future always uses IOCP, so even though it's
-                # been cancelled, we need to make sure that the IOCP message
-                # is received so that the kernel is not holding on to the
-                # memory, possibly causing memory corruption later. Only
-                # unregister it if IO is complete in all respects. Otherwise
-                # we need another _poll() later to complete the IO.
-                if ov is not None and not ov.pending:
-                    self._proactor._unregister(ov)
-                self._self_reading_future = None
+    def _run_forever_setup(self):
+        assert self._self_reading_future is None
+        self.call_soon(self._loop_self_reading)
+        super()._run_forever_setup()
+
+    def _run_forever_cleanup(self):
+        super()._run_forever_cleanup()
+        if self._self_reading_future is not None:
+            ov = self._self_reading_future._ov
+            self._self_reading_future.cancel()
+            # self_reading_future always uses IOCP, so even though it's
+            # been cancelled, we need to make sure that the IOCP message
+            # is received so that the kernel is not holding on to the
+            # memory, possibly causing memory corruption later. Only
+            # unregister it if IO is complete in all respects. Otherwise
+            # we need another _poll() later to complete the IO.
+            if ov is not None and not ov.pending:
+                self._proactor._unregister(ov)
+            self._self_reading_future = None
 
     async def create_pipe_connection(self, protocol_factory, address):
         f = self._proactor.connect_pipe(address)
@@ -890,12 +891,13 @@ class _WindowsSubprocessTransport(base_subprocess.BaseSubprocessTransport):
 SelectorEventLoop = _WindowsSelectorEventLoop
 
 
-class WindowsSelectorEventLoopPolicy(events.BaseDefaultEventLoopPolicy):
+class _WindowsSelectorEventLoopPolicy(events._BaseDefaultEventLoopPolicy):
     _loop_factory = SelectorEventLoop
 
 
-class WindowsProactorEventLoopPolicy(events.BaseDefaultEventLoopPolicy):
+class _WindowsProactorEventLoopPolicy(events._BaseDefaultEventLoopPolicy):
     _loop_factory = ProactorEventLoop
 
 
-DefaultEventLoopPolicy = WindowsProactorEventLoopPolicy
+_DefaultEventLoopPolicy = _WindowsProactorEventLoopPolicy
+EventLoop = ProactorEventLoop

Lib/base64.py

@@ -1,12 +1,9 @@
-#! /usr/bin/env python3
-
 """Base16, Base32, Base64 (RFC 3548), Base85 and Ascii85 data encodings"""
 
 # Modified 04-Oct-1995 by Jack Jansen to use binascii module
 # Modified 30-Dec-2003 by Barry Warsaw to add full RFC 3548 support
 # Modified 22-May-2007 by Guido van Rossum to use bytes everywhere
 
-import re
 import struct
 import binascii
 
@@ -286,7 +283,7 @@ def b16decode(s, casefold=False):
     s = _bytes_from_decode_data(s)
     if casefold:
         s = s.upper()
-    if re.search(b'[^0-9A-F]', s):
+    if s.translate(None, delete=b'0123456789ABCDEF'):
         raise binascii.Error('Non-base16 digit found')
     return binascii.unhexlify(s)
 
@@ -465,9 +462,12 @@ def b85decode(b):
     # Delay the initialization of tables to not waste memory
     # if the function is never called
     if _b85dec is None:
-        _b85dec = [None] * 256
+        # we don't assign to _b85dec directly to avoid issues when
+        # multiple threads call this function simultaneously
+        b85dec_tmp = [None] * 256
         for i, c in enumerate(_b85alphabet):
-            _b85dec[c] = i
+            b85dec_tmp[c] = i
+        _b85dec = b85dec_tmp
 
     b = _bytes_from_decode_data(b)
     padding = (-len(b)) % 5
@@ -604,7 +604,14 @@ def main():
         with open(args[0], 'rb') as f:
             func(f, sys.stdout.buffer)
     else:
-        func(sys.stdin.buffer, sys.stdout.buffer)
+        if sys.stdin.isatty():
+            # gh-138775: read terminal input data all at once to detect EOF
+            import io
+            data = sys.stdin.buffer.read()
+            buffer = io.BytesIO(data)
+        else:
+            buffer = sys.stdin.buffer
+        func(buffer, sys.stdout.buffer)
 
 
 if __name__ == '__main__':

Lib/bdb.py

@@ -2,7 +2,10 @@
 
 import fnmatch
 import sys
+import threading
 import os
+import weakref
+from contextlib import contextmanager
 from inspect import CO_GENERATOR, CO_COROUTINE, CO_ASYNC_GENERATOR
 
 __all__ = ["BdbQuit", "Bdb", "Breakpoint"]
@@ -14,6 +17,166 @@ class BdbQuit(Exception):
     """Exception to give up completely."""
 
 
+E = sys.monitoring.events
+
+class _MonitoringTracer:
+    EVENT_CALLBACK_MAP = {
+        E.PY_START: 'call',
+        E.PY_RESUME: 'call',
+        E.PY_THROW: 'call',
+        E.LINE: 'line',
+        E.JUMP: 'jump',
+        E.PY_RETURN: 'return',
+        E.PY_YIELD: 'return',
+        E.PY_UNWIND: 'unwind',
+        E.RAISE: 'exception',
+        E.STOP_ITERATION: 'exception',
+        E.INSTRUCTION: 'opcode',
+    }
+
+    GLOBAL_EVENTS = E.PY_START | E.PY_RESUME | E.PY_THROW | E.PY_UNWIND | E.RAISE
+    LOCAL_EVENTS = E.LINE | E.JUMP | E.PY_RETURN | E.PY_YIELD | E.STOP_ITERATION
+
+    def __init__(self):
+        self._tool_id = sys.monitoring.DEBUGGER_ID
+        self._name = 'bdbtracer'
+        self._tracefunc = None
+        self._disable_current_event = False
+        self._tracing_thread = None
+        self._enabled = False
+
+    def start_trace(self, tracefunc):
+        self._tracefunc = tracefunc
+        self._tracing_thread = threading.current_thread()
+        curr_tool = sys.monitoring.get_tool(self._tool_id)
+        if curr_tool is None:
+            sys.monitoring.use_tool_id(self._tool_id, self._name)
+        elif curr_tool == self._name:
+            sys.monitoring.clear_tool_id(self._tool_id)
+        else:
+            raise ValueError('Another debugger is using the monitoring tool')
+        E = sys.monitoring.events
+        all_events = 0
+        for event, cb_name in self.EVENT_CALLBACK_MAP.items():
+            callback = self.callback_wrapper(getattr(self, f'{cb_name}_callback'), event)
+            sys.monitoring.register_callback(self._tool_id, event, callback)
+            if event != E.INSTRUCTION:
+                all_events |= event
+        self.update_local_events()
+        sys.monitoring.set_events(self._tool_id, self.GLOBAL_EVENTS)
+        self._enabled = True
+
+    def stop_trace(self):
+        self._enabled = False
+        self._tracing_thread = None
+        curr_tool = sys.monitoring.get_tool(self._tool_id)
+        if curr_tool != self._name:
+            return
+        sys.monitoring.clear_tool_id(self._tool_id)
+        sys.monitoring.free_tool_id(self._tool_id)
+
+    def disable_current_event(self):
+        self._disable_current_event = True
+
+    def restart_events(self):
+        if sys.monitoring.get_tool(self._tool_id) == self._name:
+            sys.monitoring.restart_events()
+
+    def callback_wrapper(self, func, event):
+        import functools
+
+        @functools.wraps(func)
+        def wrapper(*args):
+            if self._tracing_thread != threading.current_thread():
+                return
+            try:
+                frame = sys._getframe().f_back
+                ret = func(frame, *args)
+                if self._enabled and frame.f_trace:
+                    self.update_local_events()
+                if (
+                    self._disable_current_event
+                    and event not in (E.PY_THROW, E.PY_UNWIND, E.RAISE)
+                ):
+                    return sys.monitoring.DISABLE
+                else:
+                    return ret
+            except BaseException:
+                self.stop_trace()
+                sys._getframe().f_back.f_trace = None
+                raise
+            finally:
+                self._disable_current_event = False
+
+        return wrapper
+
+    def call_callback(self, frame, code, *args):
+        local_tracefunc = self._tracefunc(frame, 'call', None)
+        if local_tracefunc is not None:
+            frame.f_trace = local_tracefunc
+            if self._enabled:
+                sys.monitoring.set_local_events(self._tool_id, code, self.LOCAL_EVENTS)
+
+    def return_callback(self, frame, code, offset, retval):
+        if frame.f_trace:
+            frame.f_trace(frame, 'return', retval)
+
+    def unwind_callback(self, frame, code, *args):
+        if frame.f_trace:
+            frame.f_trace(frame, 'return', None)
+
+    def line_callback(self, frame, code, *args):
+        if frame.f_trace and frame.f_trace_lines:
+            frame.f_trace(frame, 'line', None)
+
+    def jump_callback(self, frame, code, inst_offset, dest_offset):
+        if dest_offset > inst_offset:
+            return sys.monitoring.DISABLE
+        inst_lineno = self._get_lineno(code, inst_offset)
+        dest_lineno = self._get_lineno(code, dest_offset)
+        if inst_lineno != dest_lineno:
+            return sys.monitoring.DISABLE
+        if frame.f_trace and frame.f_trace_lines:
+            frame.f_trace(frame, 'line', None)
+
+    def exception_callback(self, frame, code, offset, exc):
+        if frame.f_trace:
+            if exc.__traceback__ and hasattr(exc.__traceback__, 'tb_frame'):
+                tb = exc.__traceback__
+                while tb:
+                    if tb.tb_frame.f_locals.get('self') is self:
+                        return
+                    tb = tb.tb_next
+            frame.f_trace(frame, 'exception', (type(exc), exc, exc.__traceback__))
+
+    def opcode_callback(self, frame, code, offset):
+        if frame.f_trace and frame.f_trace_opcodes:
+            frame.f_trace(frame, 'opcode', None)
+
+    def update_local_events(self, frame=None):
+        if sys.monitoring.get_tool(self._tool_id) != self._name:
+            return
+        if frame is None:
+            frame = sys._getframe().f_back
+        while frame is not None:
+            if frame.f_trace is not None:
+                if frame.f_trace_opcodes:
+                    events = self.LOCAL_EVENTS | E.INSTRUCTION
+                else:
+                    events = self.LOCAL_EVENTS
+                sys.monitoring.set_local_events(self._tool_id, frame.f_code, events)
+            frame = frame.f_back
+
+    def _get_lineno(self, code, offset):
+        import dis
+        last_lineno = None
+        for start, lineno in dis.findlinestarts(code):
+            if offset < start:
+                return last_lineno
+            last_lineno = lineno
+        return last_lineno
+
+
 class Bdb:
     """Generic Python debugger base class.
 
@@ -28,11 +191,24 @@ class Bdb:
     is determined by the __name__ in the frame globals.
     """
 
-    def __init__(self, skip=None):
+    def __init__(self, skip=None, backend='settrace'):
         self.skip = set(skip) if skip else None
         self.breaks = {}
         self.fncache = {}
+        self.frame_trace_lines_opcodes = {}
         self.frame_returning = None
+        self.trace_opcodes = False
+        self.enterframe = None
+        self.cmdframe = None
+        self.cmdlineno = None
+        self.code_linenos = weakref.WeakKeyDictionary()
+        self.backend = backend
+        if backend == 'monitoring':
+            self.monitoring_tracer = _MonitoringTracer()
+        elif backend == 'settrace':
+            self.monitoring_tracer = None
+        else:
+            raise ValueError(f"Invalid backend '{backend}'")
 
         self._load_breaks()
 
@@ -53,6 +229,18 @@ class Bdb:
             self.fncache[filename] = canonic
         return canonic
 
+    def start_trace(self):
+        if self.monitoring_tracer:
+            self.monitoring_tracer.start_trace(self.trace_dispatch)
+        else:
+            sys.settrace(self.trace_dispatch)
+
+    def stop_trace(self):
+        if self.monitoring_tracer:
+            self.monitoring_tracer.stop_trace()
+        else:
+            sys.settrace(None)
+
     def reset(self):
         """Set values of attributes as ready to start debugging."""
         import linecache
@@ -60,6 +248,12 @@ class Bdb:
         self.botframe = None
         self._set_stopinfo(None, None)
 
+    @contextmanager
+    def set_enterframe(self, frame):
+        self.enterframe = frame
+        yield
+        self.enterframe = None
+
     def trace_dispatch(self, frame, event, arg):
         """Dispatch a trace function for debugged frames based on the event.
 
@@ -84,24 +278,28 @@ class Bdb:
 
         The arg parameter depends on the previous event.
         """
-        if self.quitting:
-            return # None
-        if event == 'line':
-            return self.dispatch_line(frame)
-        if event == 'call':
-            return self.dispatch_call(frame, arg)
-        if event == 'return':
-            return self.dispatch_return(frame, arg)
-        if event == 'exception':
-            return self.dispatch_exception(frame, arg)
-        if event == 'c_call':
+
+        with self.set_enterframe(frame):
+            if self.quitting:
+                return # None
+            if event == 'line':
+                return self.dispatch_line(frame)
+            if event == 'call':
+                return self.dispatch_call(frame, arg)
+            if event == 'return':
+                return self.dispatch_return(frame, arg)
+            if event == 'exception':
+                return self.dispatch_exception(frame, arg)
+            if event == 'c_call':
+                return self.trace_dispatch
+            if event == 'c_exception':
+                return self.trace_dispatch
+            if event == 'c_return':
+                return self.trace_dispatch
+            if event == 'opcode':
+                return self.dispatch_opcode(frame, arg)
+            print('bdb.Bdb.dispatch: unknown debugging event:', repr(event))
             return self.trace_dispatch
-        if event == 'c_exception':
-            return self.trace_dispatch
-        if event == 'c_return':
-            return self.trace_dispatch
-        print('bdb.Bdb.dispatch: unknown debugging event:', repr(event))
-        return self.trace_dispatch
 
     def dispatch_line(self, frame):
         """Invoke user function and return trace function for line event.
@@ -110,9 +308,17 @@ class Bdb:
         self.user_line(). Raise BdbQuit if self.quitting is set.
         Return self.trace_dispatch to continue tracing in this scope.
         """
-        if self.stop_here(frame) or self.break_here(frame):
+        # GH-136057
+        # For line events, we don't want to stop at the same line where
+        # the latest next/step command was issued.
+        if (self.stop_here(frame) or self.break_here(frame)) and not (
+            self.cmdframe == frame and self.cmdlineno == frame.f_lineno
+        ):
             self.user_line(frame)
+            self.restart_events()
             if self.quitting: raise BdbQuit
+        elif not self.get_break(frame.f_code.co_filename, frame.f_lineno):
+            self.disable_current_event()
         return self.trace_dispatch
 
     def dispatch_call(self, frame, arg):
@@ -128,12 +334,18 @@ class Bdb:
             self.botframe = frame.f_back # (CT) Note that this may also be None!
             return self.trace_dispatch
         if not (self.stop_here(frame) or self.break_anywhere(frame)):
-            # No need to trace this function
+            # We already know there's no breakpoint in this function
+            # If it's a next/until/return command, we don't need any CALL event
+            # and we don't need to set the f_trace on any new frame.
+            # If it's a step command, it must either hit stop_here, or skip the
+            # whole module. Either way, we don't need the CALL event here.
+            self.disable_current_event()
             return # None
         # Ignore call events in generator except when stepping.
         if self.stopframe and frame.f_code.co_flags & GENERATOR_AND_COROUTINE_FLAGS:
             return self.trace_dispatch
         self.user_call(frame, arg)
+        self.restart_events()
         if self.quitting: raise BdbQuit
         return self.trace_dispatch
 
@@ -147,16 +359,25 @@ class Bdb:
         if self.stop_here(frame) or frame == self.returnframe:
             # Ignore return events in generator except when stepping.
             if self.stopframe and frame.f_code.co_flags & GENERATOR_AND_COROUTINE_FLAGS:
+                # It's possible to trigger a StopIteration exception in
+                # the caller so we must set the trace function in the caller
+                self._set_caller_tracefunc(frame)
                 return self.trace_dispatch
             try:
                 self.frame_returning = frame
                 self.user_return(frame, arg)
+                self.restart_events()
             finally:
                 self.frame_returning = None
             if self.quitting: raise BdbQuit
             # The user issued a 'next' or 'until' command.
             if self.stopframe is frame and self.stoplineno != -1:
                 self._set_stopinfo(None, None)
+            # The previous frame might not have f_trace set, unless we are
+            # issuing a command that does not expect to stop, we should set
+            # f_trace
+            if self.stoplineno != -1:
+                self._set_caller_tracefunc(frame)
         return self.trace_dispatch
 
     def dispatch_exception(self, frame, arg):
@@ -173,6 +394,7 @@ class Bdb:
             if not (frame.f_code.co_flags & GENERATOR_AND_COROUTINE_FLAGS
                     and arg[0] is StopIteration and arg[2] is None):
                 self.user_exception(frame, arg)
+                self.restart_events()
                 if self.quitting: raise BdbQuit
         # Stop at the StopIteration or GeneratorExit exception when the user
         # has set stopframe in a generator by issuing a return command, or a
@@ -182,10 +404,26 @@ class Bdb:
                 and self.stopframe.f_code.co_flags & GENERATOR_AND_COROUTINE_FLAGS
                 and arg[0] in (StopIteration, GeneratorExit)):
             self.user_exception(frame, arg)
+            self.restart_events()
             if self.quitting: raise BdbQuit
 
         return self.trace_dispatch
 
+    def dispatch_opcode(self, frame, arg):
+        """Invoke user function and return trace function for opcode event.
+        If the debugger stops on the current opcode, invoke
+        self.user_opcode(). Raise BdbQuit if self.quitting is set.
+        Return self.trace_dispatch to continue tracing in this scope.
+
+        Opcode event will always trigger the user callback. For now the only
+        opcode event is from an inline set_trace() and we want to stop there
+        unconditionally.
+        """
+        self.user_opcode(frame)
+        self.restart_events()
+        if self.quitting: raise BdbQuit
+        return self.trace_dispatch
+
     # Normally derived classes don't override the following
     # methods, but they may if they want to redefine the
     # definition of stopping and breakpoints.
@@ -249,9 +487,25 @@ class Bdb:
         raise NotImplementedError("subclass of bdb must implement do_clear()")
 
     def break_anywhere(self, frame):
-        """Return True if there is any breakpoint for frame's filename.
+        """Return True if there is any breakpoint in that frame
         """
-        return self.canonic(frame.f_code.co_filename) in self.breaks
+        filename = self.canonic(frame.f_code.co_filename)
+        if filename not in self.breaks:
+            return False
+        for lineno in self.breaks[filename]:
+            if self._lineno_in_frame(lineno, frame):
+                return True
+        return False
+
+    def _lineno_in_frame(self, lineno, frame):
+        """Return True if the line number is in the frame's code object.
+        """
+        code = frame.f_code
+        if lineno < code.co_firstlineno:
+            return False
+        if code not in self.code_linenos:
+            self.code_linenos[code] = set(lineno for _, _, lineno in code.co_lines())
+        return lineno in self.code_linenos[code]
 
     # Derived classes should override the user_* methods
     # to gain control.
@@ -272,7 +526,24 @@ class Bdb:
         """Called when we stop on an exception."""
         pass
 
-    def _set_stopinfo(self, stopframe, returnframe, stoplineno=0):
+    def user_opcode(self, frame):
+        """Called when we are about to execute an opcode."""
+        pass
+
+    def _set_trace_opcodes(self, trace_opcodes):
+        if trace_opcodes != self.trace_opcodes:
+            self.trace_opcodes = trace_opcodes
+            frame = self.enterframe
+            while frame is not None:
+                frame.f_trace_opcodes = trace_opcodes
+                if frame is self.botframe:
+                    break
+                frame = frame.f_back
+            if self.monitoring_tracer:
+                self.monitoring_tracer.update_local_events()
+
+    def _set_stopinfo(self, stopframe, returnframe, stoplineno=0, opcode=False,
+                      cmdframe=None, cmdlineno=None):
         """Set the attributes for stopping.
 
         If stoplineno is greater than or equal to 0, then stop at line
@@ -285,6 +556,21 @@ class Bdb:
         # stoplineno >= 0 means: stop at line >= the stoplineno
         # stoplineno -1 means: don't stop at all
         self.stoplineno = stoplineno
+        # cmdframe/cmdlineno is the frame/line number when the user issued
+        # step/next commands.
+        self.cmdframe = cmdframe
+        self.cmdlineno = cmdlineno
+        self._set_trace_opcodes(opcode)
+
+    def _set_caller_tracefunc(self, current_frame):
+        # Issue #13183: pdb skips frames after hitting a breakpoint and running
+        # step commands.
+        # Restore the trace function in the caller (that may not have been set
+        # for performance reasons) when returning from the current frame, unless
+        # the caller is the botframe.
+        caller_frame = current_frame.f_back
+        if caller_frame and not caller_frame.f_trace and caller_frame is not self.botframe:
+            caller_frame.f_trace = self.trace_dispatch
 
     # Derived classes and clients can call the following methods
     # to affect the stepping state.
@@ -299,24 +585,22 @@ class Bdb:
 
     def set_step(self):
         """Stop after one line of code."""
-        # Issue #13183: pdb skips frames after hitting a breakpoint and running
-        # step commands.
-        # Restore the trace function in the caller (that may not have been set
-        # for performance reasons) when returning from the current frame.
-        if self.frame_returning:
-            caller_frame = self.frame_returning.f_back
-            if caller_frame and not caller_frame.f_trace:
-                caller_frame.f_trace = self.trace_dispatch
-        self._set_stopinfo(None, None)
+        # set_step() could be called from signal handler so enterframe might be None
+        self._set_stopinfo(None, None, cmdframe=self.enterframe,
+                           cmdlineno=getattr(self.enterframe, 'f_lineno', None))
+
+    def set_stepinstr(self):
+        """Stop before the next instruction."""
+        self._set_stopinfo(None, None, opcode=True)
 
     def set_next(self, frame):
         """Stop on the next line in or below the given frame."""
-        self._set_stopinfo(frame, None)
+        self._set_stopinfo(frame, None, cmdframe=frame, cmdlineno=frame.f_lineno)
 
     def set_return(self, frame):
         """Stop when returning from the given frame."""
         if frame.f_code.co_flags & GENERATOR_AND_COROUTINE_FLAGS:
-            self._set_stopinfo(frame, None, -1)
+            self._set_stopinfo(frame, frame, -1)
         else:
             self._set_stopinfo(frame.f_back, frame)
 
@@ -325,15 +609,21 @@ class Bdb:
 
         If frame is not specified, debugging starts from caller's frame.
         """
+        self.stop_trace()
         if frame is None:
             frame = sys._getframe().f_back
         self.reset()
-        while frame:
-            frame.f_trace = self.trace_dispatch
-            self.botframe = frame
-            frame = frame.f_back
-        self.set_step()
-        sys.settrace(self.trace_dispatch)
+        with self.set_enterframe(frame):
+            while frame:
+                frame.f_trace = self.trace_dispatch
+                self.botframe = frame
+                self.frame_trace_lines_opcodes[frame] = (frame.f_trace_lines, frame.f_trace_opcodes)
+                # We need f_trace_lines == True for the debugger to work
+                frame.f_trace_lines = True
+                frame = frame.f_back
+            self.set_stepinstr()
+            self.enterframe = None
+        self.start_trace()
 
     def set_continue(self):
         """Stop only at breakpoints or when finished.
@@ -344,11 +634,16 @@ class Bdb:
         self._set_stopinfo(self.botframe, None, -1)
         if not self.breaks:
             # no breakpoints; run without debugger overhead
-            sys.settrace(None)
+            self.stop_trace()
             frame = sys._getframe().f_back
             while frame and frame is not self.botframe:
                 del frame.f_trace
                 frame = frame.f_back
+            for frame, (trace_lines, trace_opcodes) in self.frame_trace_lines_opcodes.items():
+                frame.f_trace_lines, frame.f_trace_opcodes = trace_lines, trace_opcodes
+            if self.backend == 'monitoring':
+                self.monitoring_tracer.update_local_events()
+            self.frame_trace_lines_opcodes = {}
 
     def set_quit(self):
         """Set quitting attribute to True.
@@ -358,7 +653,7 @@ class Bdb:
         self.stopframe = self.botframe
         self.returnframe = None
         self.quitting = True
-        sys.settrace(None)
+        self.stop_trace()
 
     # Derived classes and clients can call the following methods
     # to manipulate breakpoints.  These methods return an
@@ -387,6 +682,14 @@ class Bdb:
             return 'Line %s:%d does not exist' % (filename, lineno)
         self._add_to_breaks(filename, lineno)
         bp = Breakpoint(filename, lineno, temporary, cond, funcname)
+        # After we set a new breakpoint, we need to search through all frames
+        # and set f_trace to trace_dispatch if there could be a breakpoint in
+        # that frame.
+        frame = self.enterframe
+        while frame:
+            if self.break_anywhere(frame):
+                frame.f_trace = self.trace_dispatch
+            frame = frame.f_back
         return None
 
     def _load_breaks(self):
@@ -578,6 +881,16 @@ class Bdb:
             s += f'{lprefix}Warning: lineno is None'
         return s
 
+    def disable_current_event(self):
+        """Disable the current event."""
+        if self.backend == 'monitoring':
+            self.monitoring_tracer.disable_current_event()
+
+    def restart_events(self):
+        """Restart all events."""
+        if self.backend == 'monitoring':
+            self.monitoring_tracer.restart_events()
+
     # The following methods can be called by clients to use
     # a debugger to debug a statement or an expression.
     # Both can be given as a string, or a code object.
@@ -595,14 +908,14 @@ class Bdb:
         self.reset()
         if isinstance(cmd, str):
             cmd = compile(cmd, "<string>", "exec")
-        sys.settrace(self.trace_dispatch)
+        self.start_trace()
         try:
             exec(cmd, globals, locals)
         except BdbQuit:
             pass
         finally:
             self.quitting = True
-            sys.settrace(None)
+            self.stop_trace()
 
     def runeval(self, expr, globals=None, locals=None):
         """Debug an expression executed via the eval() function.
@@ -615,14 +928,14 @@ class Bdb:
         if locals is None:
             locals = globals
         self.reset()
-        sys.settrace(self.trace_dispatch)
+        self.start_trace()
         try:
             return eval(expr, globals, locals)
         except BdbQuit:
             pass
         finally:
             self.quitting = True
-            sys.settrace(None)
+            self.stop_trace()
 
     def runctx(self, cmd, globals, locals):
         """For backwards-compatibility.  Defers to run()."""
@@ -637,7 +950,7 @@ class Bdb:
         Return the result of the function call.
         """
         self.reset()
-        sys.settrace(self.trace_dispatch)
+        self.start_trace()
         res = None
         try:
             res = func(*args, **kwds)
@@ -645,7 +958,7 @@ class Bdb:
             pass
         finally:
             self.quitting = True
-            sys.settrace(None)
+            self.stop_trace()
         return res
 
 

Lib/bz2.py

@@ -10,20 +10,20 @@ __all__ = ["BZ2File", "BZ2Compressor", "BZ2Decompressor",
 __author__ = "Nadeem Vawda <nadeem.vawda@gmail.com>"
 
 from builtins import open as _builtin_open
+from compression._common import _streams
 import io
 import os
-import _compression
 
 from _bz2 import BZ2Compressor, BZ2Decompressor
 
 
-_MODE_CLOSED   = 0
+# Value 0 no longer used
 _MODE_READ     = 1
 # Value 2 no longer used
 _MODE_WRITE    = 3
 
 
-class BZ2File(_compression.BaseStream):
+class BZ2File(_streams.BaseStream):
 
     """A file object providing transparent bzip2 (de)compression.
 
@@ -54,7 +54,7 @@ class BZ2File(_compression.BaseStream):
         """
         self._fp = None
         self._closefp = False
-        self._mode = _MODE_CLOSED
+        self._mode = None
 
         if not (1 <= compresslevel <= 9):
             raise ValueError("compresslevel must be between 1 and 9")
@@ -88,7 +88,7 @@ class BZ2File(_compression.BaseStream):
             raise TypeError("filename must be a str, bytes, file or PathLike object")
 
         if self._mode == _MODE_READ:
-            raw = _compression.DecompressReader(self._fp,
+            raw = _streams.DecompressReader(self._fp,
                 BZ2Decompressor, trailing_error=OSError)
             self._buffer = io.BufferedReader(raw)
         else:
@@ -100,7 +100,7 @@ class BZ2File(_compression.BaseStream):
         May be called more than once without error. Once the file is
         closed, any other operation on it will raise a ValueError.
         """
-        if self._mode == _MODE_CLOSED:
+        if self.closed:
             return
         try:
             if self._mode == _MODE_READ:
@@ -115,13 +115,21 @@ class BZ2File(_compression.BaseStream):
             finally:
                 self._fp = None
                 self._closefp = False
-                self._mode = _MODE_CLOSED
                 self._buffer = None
 
     @property
     def closed(self):
         """True if this file is closed."""
-        return self._mode == _MODE_CLOSED
+        return self._fp is None
+
+    @property
+    def name(self):
+        self._check_not_closed()
+        return self._fp.name
+
+    @property
+    def mode(self):
+        return 'wb' if self._mode == _MODE_WRITE else 'rb'
 
     def fileno(self):
         """Return the file descriptor for the underlying file."""
@@ -240,7 +248,7 @@ class BZ2File(_compression.BaseStream):
 
         Line separators are not added between the written byte strings.
         """
-        return _compression.BaseStream.writelines(self, seq)
+        return _streams.BaseStream.writelines(self, seq)
 
     def seek(self, offset, whence=io.SEEK_SET):
         """Change the file position.

Lib/calendar.py

@@ -428,6 +428,7 @@ class TextCalendar(Calendar):
             headers = (header for k in months)
             a(formatstring(headers, colwidth, c).rstrip())
             a('\n'*l)
+
             # max number of weeks for this row
             height = max(len(cal) for cal in row)
             for j in range(height):
@@ -646,6 +647,117 @@ class LocaleHTMLCalendar(HTMLCalendar):
         with different_locale(self.locale):
             return super().formatmonthname(theyear, themonth, withyear)
 
+
+class _CLIDemoCalendar(TextCalendar):
+    def __init__(self, highlight_day=None, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.highlight_day = highlight_day
+
+    def formatweek(self, theweek, width, *, highlight_day=None):
+        """
+        Returns a single week in a string (no newline).
+        """
+        if highlight_day:
+            from _colorize import get_colors
+
+            ansi = get_colors()
+            highlight = f"{ansi.BLACK}{ansi.BACKGROUND_YELLOW}"
+            reset = ansi.RESET
+        else:
+            highlight = reset = ""
+
+        return ' '.join(
+            (
+                f"{highlight}{self.formatday(d, wd, width)}{reset}"
+                if d == highlight_day
+                else self.formatday(d, wd, width)
+            )
+            for (d, wd) in theweek
+        )
+
+    def formatmonth(self, theyear, themonth, w=0, l=0):
+        """
+        Return a month's calendar string (multi-line).
+        """
+        if (
+            self.highlight_day
+            and self.highlight_day.year == theyear
+            and self.highlight_day.month == themonth
+        ):
+            highlight_day = self.highlight_day.day
+        else:
+            highlight_day = None
+        w = max(2, w)
+        l = max(1, l)
+        s = self.formatmonthname(theyear, themonth, 7 * (w + 1) - 1)
+        s = s.rstrip()
+        s += '\n' * l
+        s += self.formatweekheader(w).rstrip()
+        s += '\n' * l
+        for week in self.monthdays2calendar(theyear, themonth):
+            s += self.formatweek(week, w, highlight_day=highlight_day).rstrip()
+            s += '\n' * l
+        return s
+
+    def formatyear(self, theyear, w=2, l=1, c=6, m=3):
+        """
+        Returns a year's calendar as a multi-line string.
+        """
+        w = max(2, w)
+        l = max(1, l)
+        c = max(2, c)
+        colwidth = (w + 1) * 7 - 1
+        v = []
+        a = v.append
+        a(repr(theyear).center(colwidth*m+c*(m-1)).rstrip())
+        a('\n'*l)
+        header = self.formatweekheader(w)
+        for (i, row) in enumerate(self.yeardays2calendar(theyear, m)):
+            # months in this row
+            months = range(m*i+1, min(m*(i+1)+1, 13))
+            a('\n'*l)
+            names = (self.formatmonthname(theyear, k, colwidth, False)
+                     for k in months)
+            a(formatstring(names, colwidth, c).rstrip())
+            a('\n'*l)
+            headers = (header for k in months)
+            a(formatstring(headers, colwidth, c).rstrip())
+            a('\n'*l)
+
+            if (
+                self.highlight_day
+                and self.highlight_day.year == theyear
+                and self.highlight_day.month in months
+            ):
+                month_pos = months.index(self.highlight_day.month)
+            else:
+                month_pos = None
+
+            # max number of weeks for this row
+            height = max(len(cal) for cal in row)
+            for j in range(height):
+                weeks = []
+                for k, cal in enumerate(row):
+                    if j >= len(cal):
+                        weeks.append('')
+                    else:
+                        day = (
+                            self.highlight_day.day if k == month_pos else None
+                        )
+                        weeks.append(
+                            self.formatweek(cal[j], w, highlight_day=day)
+                        )
+                a(formatstring(weeks, colwidth, c).rstrip())
+                a('\n' * l)
+        return ''.join(v)
+
+
+class _CLIDemoLocaleCalendar(LocaleTextCalendar, _CLIDemoCalendar):
+    def __init__(self, highlight_day=None, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.highlight_day = highlight_day
+
+
 # Support for old module level interface
 c = TextCalendar()
 
@@ -698,7 +810,7 @@ def timegm(tuple):
 
 def main(args=None):
     import argparse
-    parser = argparse.ArgumentParser()
+    parser = argparse.ArgumentParser(color=True)
     textgroup = parser.add_argument_group('text only arguments')
     htmlgroup = parser.add_argument_group('html only arguments')
     textgroup.add_argument(
@@ -765,6 +877,7 @@ def main(args=None):
         sys.exit(1)
 
     locale = options.locale, options.encoding
+    today = datetime.date.today()
 
     if options.type == "html":
         if options.month:
@@ -781,23 +894,23 @@ def main(args=None):
         optdict = dict(encoding=encoding, css=options.css)
         write = sys.stdout.buffer.write
         if options.year is None:
-            write(cal.formatyearpage(datetime.date.today().year, **optdict))
+            write(cal.formatyearpage(today.year, **optdict))
         else:
             write(cal.formatyearpage(options.year, **optdict))
     else:
         if options.locale:
-            cal = LocaleTextCalendar(locale=locale)
+            cal = _CLIDemoLocaleCalendar(highlight_day=today, locale=locale)
         else:
-            cal = TextCalendar()
+            cal = _CLIDemoCalendar(highlight_day=today)
         cal.setfirstweekday(options.first_weekday)
         optdict = dict(w=options.width, l=options.lines)
         if options.month is None:
             optdict["c"] = options.spacing
             optdict["m"] = options.months
-        if options.month is not None:
+        else:
             _validate_month(options.month)
         if options.year is None:
-            result = cal.formatyear(datetime.date.today().year, **optdict)
+            result = cal.formatyear(today.year, **optdict)
         elif options.month is None:
             result = cal.formatyear(options.year, **optdict)
         else:

Lib/cmd.py

@@ -5,16 +5,16 @@ Interpreters constructed with this class obey the following conventions:
 1. End of file on input is processed as the command 'EOF'.
 2. A command is parsed out of each line by collecting the prefix composed
    of characters in the identchars member.
-3. A command `foo' is dispatched to a method 'do_foo()'; the do_ method
+3. A command 'foo' is dispatched to a method 'do_foo()'; the do_ method
    is passed a single argument consisting of the remainder of the line.
 4. Typing an empty line repeats the last command.  (Actually, it calls the
-   method `emptyline', which may be overridden in a subclass.)
-5. There is a predefined `help' method.  Given an argument `topic', it
-   calls the command `help_topic'.  With no arguments, it lists all topics
+   method 'emptyline', which may be overridden in a subclass.)
+5. There is a predefined 'help' method.  Given an argument 'topic', it
+   calls the command 'help_topic'.  With no arguments, it lists all topics
    with defined help_ functions, broken into up to three topics; documented
    commands, miscellaneous help topics, and undocumented commands.
-6. The command '?' is a synonym for `help'.  The command '!' is a synonym
-   for `shell', if a do_shell method exists.
+6. The command '?' is a synonym for 'help'.  The command '!' is a synonym
+   for 'shell', if a do_shell method exists.
 7. If completion is enabled, completing commands will be done automatically,
    and completing of commands args is done by calling complete_foo() with
    arguments text, line, begidx, endidx.  text is string we are matching
@@ -23,31 +23,34 @@ Interpreters constructed with this class obey the following conventions:
    indexes of the text being matched, which could be used to provide
    different completion depending upon which position the argument is in.
 
-The `default' method may be overridden to intercept commands for which there
+The 'default' method may be overridden to intercept commands for which there
 is no do_ method.
 
-The `completedefault' method may be overridden to intercept completions for
+The 'completedefault' method may be overridden to intercept completions for
 commands that have no complete_ method.
 
-The data member `self.ruler' sets the character used to draw separator lines
+The data member 'self.ruler' sets the character used to draw separator lines
 in the help messages.  If empty, no ruler line is drawn.  It defaults to "=".
 
-If the value of `self.intro' is nonempty when the cmdloop method is called,
+If the value of 'self.intro' is nonempty when the cmdloop method is called,
 it is printed out on interpreter startup.  This value may be overridden
 via an optional argument to the cmdloop() method.
 
-The data members `self.doc_header', `self.misc_header', and
-`self.undoc_header' set the headers used for the help function's
+The data members 'self.doc_header', 'self.misc_header', and
+'self.undoc_header' set the headers used for the help function's
 listings of documented functions, miscellaneous topics, and undocumented
 functions respectively.
 """
 
-import inspect, string, sys
+import sys
 
 __all__ = ["Cmd"]
 
 PROMPT = '(Cmd) '
-IDENTCHARS = string.ascii_letters + string.digits + '_'
+IDENTCHARS = ('ABCDEFGHIJKLMNOPQRSTUVWXYZ'
+              'abcdefghijklmnopqrstuvwxyz'
+              '0123456789'
+              '_')
 
 class Cmd:
     """A simple framework for writing line-oriented command interpreters.
@@ -270,7 +273,7 @@ class Cmd:
             endidx = readline.get_endidx() - stripped
             if begidx>0:
                 cmd, args, foo = self.parseline(line)
-                if cmd == '':
+                if not cmd:
                     compfunc = self.completedefault
                 else:
                     try:
@@ -303,9 +306,11 @@ class Cmd:
             try:
                 func = getattr(self, 'help_' + arg)
             except AttributeError:
+                from inspect import cleandoc
+
                 try:
                     doc=getattr(self, 'do_' + arg).__doc__
-                    doc = inspect.cleandoc(doc)
+                    doc = cleandoc(doc)
                     if doc:
                         self.stdout.write("%s\n"%str(doc))
                         return

Lib/code.py

@@ -5,6 +5,7 @@
 # Inspired by similar code by Jeff Epler and Fredrik Lundh.
 
 
+import builtins
 import sys
 import traceback
 from codeop import CommandCompiler, compile_command
@@ -24,10 +25,10 @@ class InteractiveInterpreter:
     def __init__(self, locals=None):
         """Constructor.
 
-        The optional 'locals' argument specifies the dictionary in
-        which code will be executed; it defaults to a newly created
-        dictionary with key "__name__" set to "__console__" and key
-        "__doc__" set to None.
+        The optional 'locals' argument specifies a mapping to use as the
+        namespace in which code will be executed; it defaults to a newly
+        created dictionary with key "__name__" set to "__console__" and
+        key "__doc__" set to None.
 
         """
         if locals is None:
@@ -63,7 +64,7 @@ class InteractiveInterpreter:
             code = self.compile(source, filename, symbol)
         except (OverflowError, SyntaxError, ValueError):
             # Case 1
-            self.showsyntaxerror(filename)
+            self.showsyntaxerror(filename, source=source)
             return False
 
         if code is None:
@@ -93,7 +94,7 @@ class InteractiveInterpreter:
         except:
             self.showtraceback()
 
-    def showsyntaxerror(self, filename=None):
+    def showsyntaxerror(self, filename=None, **kwargs):
         """Display the syntax error that just occurred.
 
         This doesn't display a stack trace because there isn't one.
@@ -105,29 +106,14 @@ class InteractiveInterpreter:
         The output is written by self.write(), below.
 
         """
-        type, value, tb = sys.exc_info()
-        sys.last_exc = value
-        sys.last_type = type
-        sys.last_value = value
-        sys.last_traceback = tb
-        if filename and type is SyntaxError:
-            # Work hard to stuff the correct filename in the exception
-            try:
-                msg, (dummy_filename, lineno, offset, line) = value.args
-            except ValueError:
-                # Not the format we expect; leave it alone
-                pass
-            else:
-                # Stuff in the right filename
-                value = SyntaxError(msg, (filename, lineno, offset, line))
-                sys.last_exc = sys.last_value = value
-        if sys.excepthook is sys.__excepthook__:
-            lines = traceback.format_exception_only(type, value)
-            self.write(''.join(lines))
-        else:
-            # If someone has set sys.excepthook, we let that take precedence
-            # over self.write
-            sys.excepthook(type, value, tb)
+        try:
+            typ, value, tb = sys.exc_info()
+            if filename and issubclass(typ, SyntaxError):
+                value.filename = filename
+            source = kwargs.pop('source', "")
+            self._showtraceback(typ, value, None, source)
+        finally:
+            typ = value = tb = None
 
     def showtraceback(self):
         """Display the exception that just occurred.
@@ -137,19 +123,46 @@ class InteractiveInterpreter:
         The output is written by self.write(), below.
 
         """
-        sys.last_type, sys.last_value, last_tb = ei = sys.exc_info()
-        sys.last_traceback = last_tb
-        sys.last_exc = ei[1]
         try:
-            lines = traceback.format_exception(ei[0], ei[1], last_tb.tb_next)
-            if sys.excepthook is sys.__excepthook__:
-                self.write(''.join(lines))
-            else:
-                # If someone has set sys.excepthook, we let that take precedence
-                # over self.write
-                sys.excepthook(ei[0], ei[1], last_tb)
+            typ, value, tb = sys.exc_info()
+            self._showtraceback(typ, value, tb.tb_next, "")
         finally:
-            last_tb = ei = None
+            typ = value = tb = None
+
+    def _showtraceback(self, typ, value, tb, source):
+        sys.last_type = typ
+        sys.last_traceback = tb
+        value = value.with_traceback(tb)
+        # Set the line of text that the exception refers to
+        lines = source.splitlines()
+        if (source and typ is SyntaxError
+                and not value.text and value.lineno is not None
+                and len(lines) >= value.lineno):
+            value.text = lines[value.lineno - 1]
+        sys.last_exc = sys.last_value = value
+        if sys.excepthook is sys.__excepthook__:
+            self._excepthook(typ, value, tb)
+        else:
+            # If someone has set sys.excepthook, we let that take precedence
+            # over self.write
+            try:
+                sys.excepthook(typ, value, tb)
+            except SystemExit:
+                raise
+            except BaseException as e:
+                e.__context__ = None
+                e = e.with_traceback(e.__traceback__.tb_next)
+                print('Error in sys.excepthook:', file=sys.stderr)
+                sys.__excepthook__(type(e), e, e.__traceback__)
+                print(file=sys.stderr)
+                print('Original exception was:', file=sys.stderr)
+                sys.__excepthook__(typ, value, tb)
+
+    def _excepthook(self, typ, value, tb):
+        # This method is being overwritten in
+        # _pyrepl.console.InteractiveColoredConsole
+        lines = traceback.format_exception(typ, value, tb)
+        self.write(''.join(lines))
 
     def write(self, data):
         """Write a string.
@@ -169,7 +182,7 @@ class InteractiveConsole(InteractiveInterpreter):
 
     """
 
-    def __init__(self, locals=None, filename="<console>"):
+    def __init__(self, locals=None, filename="<console>", *, local_exit=False):
         """Constructor.
 
         The optional locals argument will be passed to the
@@ -181,6 +194,7 @@ class InteractiveConsole(InteractiveInterpreter):
         """
         InteractiveInterpreter.__init__(self, locals)
         self.filename = filename
+        self.local_exit = local_exit
         self.resetbuffer()
 
     def resetbuffer(self):
@@ -205,12 +219,17 @@ class InteractiveConsole(InteractiveInterpreter):
         """
         try:
             sys.ps1
+            delete_ps1_after = False
         except AttributeError:
             sys.ps1 = ">>> "
+            delete_ps1_after = True
         try:
-            sys.ps2
+            _ps2 = sys.ps2
+            delete_ps2_after = False
         except AttributeError:
             sys.ps2 = "... "
+            delete_ps2_after = True
+
         cprt = 'Type "help", "copyright", "credits" or "license" for more information.'
         if banner is None:
             self.write("Python %s on %s\n%s\n(%s)\n" %
@@ -219,29 +238,72 @@ class InteractiveConsole(InteractiveInterpreter):
         elif banner:
             self.write("%s\n" % str(banner))
         more = 0
-        while 1:
-            try:
-                if more:
-                    prompt = sys.ps2
-                else:
-                    prompt = sys.ps1
-                try:
-                    line = self.raw_input(prompt)
-                except EOFError:
-                    self.write("\n")
-                    break
-                else:
-                    more = self.push(line)
-            except KeyboardInterrupt:
-                self.write("\nKeyboardInterrupt\n")
-                self.resetbuffer()
-                more = 0
-        if exitmsg is None:
-            self.write('now exiting %s...\n' % self.__class__.__name__)
-        elif exitmsg != '':
-            self.write('%s\n' % exitmsg)
 
-    def push(self, line):
+        # When the user uses exit() or quit() in their interactive shell
+        # they probably just want to exit the created shell, not the whole
+        # process. exit and quit in builtins closes sys.stdin which makes
+        # it super difficult to restore
+        #
+        # When self.local_exit is True, we overwrite the builtins so
+        # exit() and quit() only raises SystemExit and we can catch that
+        # to only exit the interactive shell
+
+        _exit = None
+        _quit = None
+
+        if self.local_exit:
+            if hasattr(builtins, "exit"):
+                _exit = builtins.exit
+                builtins.exit = Quitter("exit")
+
+            if hasattr(builtins, "quit"):
+                _quit = builtins.quit
+                builtins.quit = Quitter("quit")
+
+        try:
+            while True:
+                try:
+                    if more:
+                        prompt = sys.ps2
+                    else:
+                        prompt = sys.ps1
+                    try:
+                        line = self.raw_input(prompt)
+                    except EOFError:
+                        self.write("\n")
+                        break
+                    else:
+                        more = self.push(line)
+                except KeyboardInterrupt:
+                    self.write("\nKeyboardInterrupt\n")
+                    self.resetbuffer()
+                    more = 0
+                except SystemExit as e:
+                    if self.local_exit:
+                        self.write("\n")
+                        break
+                    else:
+                        raise e
+        finally:
+            # restore exit and quit in builtins if they were modified
+            if _exit is not None:
+                builtins.exit = _exit
+
+            if _quit is not None:
+                builtins.quit = _quit
+
+            if delete_ps1_after:
+                del sys.ps1
+
+            if delete_ps2_after:
+                del sys.ps2
+
+            if exitmsg is None:
+                self.write('now exiting %s...\n' % self.__class__.__name__)
+            elif exitmsg != '':
+                self.write('%s\n' % exitmsg)
+
+    def push(self, line, filename=None, _symbol="single"):
         """Push a line to the interpreter.
 
         The line should not have a trailing newline; it may have
@@ -257,7 +319,9 @@ class InteractiveConsole(InteractiveInterpreter):
         """
         self.buffer.append(line)
         source = "\n".join(self.buffer)
-        more = self.runsource(source, self.filename)
+        if filename is None:
+            filename = self.filename
+        more = self.runsource(source, filename, symbol=_symbol)
         if not more:
             self.resetbuffer()
         return more
@@ -276,8 +340,22 @@ class InteractiveConsole(InteractiveInterpreter):
         return input(prompt)
 
 
+class Quitter:
+    def __init__(self, name):
+        self.name = name
+        if sys.platform == "win32":
+            self.eof = 'Ctrl-Z plus Return'
+        else:
+            self.eof = 'Ctrl-D (i.e. EOF)'
 
-def interact(banner=None, readfunc=None, local=None, exitmsg=None):
+    def __repr__(self):
+        return f'Use {self.name} or {self.eof} to exit'
+
+    def __call__(self, code=None):
+        raise SystemExit(code)
+
+
+def interact(banner=None, readfunc=None, local=None, exitmsg=None, local_exit=False):
     """Closely emulate the interactive Python interpreter.
 
     This is a backwards compatible interface to the InteractiveConsole
@@ -290,14 +368,15 @@ def interact(banner=None, readfunc=None, local=None, exitmsg=None):
     readfunc -- if not None, replaces InteractiveConsole.raw_input()
     local -- passed to InteractiveInterpreter.__init__()
     exitmsg -- passed to InteractiveConsole.interact()
+    local_exit -- passed to InteractiveConsole.__init__()
 
     """
-    console = InteractiveConsole(local)
+    console = InteractiveConsole(local, local_exit=local_exit)
     if readfunc is not None:
         console.raw_input = readfunc
     else:
         try:
-            import readline
+            import readline  # noqa: F401
         except ImportError:
             pass
     console.interact(banner, exitmsg)
@@ -306,7 +385,7 @@ def interact(banner=None, readfunc=None, local=None, exitmsg=None):
 if __name__ == "__main__":
     import argparse
 
-    parser = argparse.ArgumentParser()
+    parser = argparse.ArgumentParser(color=True)
     parser.add_argument('-q', action='store_true',
                        help="don't print version and copyright messages")
     args = parser.parse_args()

Lib/codecs.py

@@ -111,6 +111,9 @@ class CodecInfo(tuple):
                 (self.__class__.__module__, self.__class__.__qualname__,
                  self.name, id(self))
 
+    def __getnewargs__(self):
+        return tuple(self)
+
 class Codec:
 
     """ Defines the interface for stateless encoders/decoders.
@@ -615,7 +618,7 @@ class StreamReader(Codec):
             method and are included in the list entries.
 
             sizehint, if given, is ignored since there is no efficient
-            way to finding the true end-of-line.
+            way of finding the true end-of-line.
 
         """
         data = self.read()
@@ -706,13 +709,13 @@ class StreamReaderWriter:
 
         return self.reader.read(size)
 
-    def readline(self, size=None):
+    def readline(self, size=None, keepends=True):
 
-        return self.reader.readline(size)
+        return self.reader.readline(size, keepends)
 
-    def readlines(self, sizehint=None):
+    def readlines(self, sizehint=None, keepends=True):
 
-        return self.reader.readlines(sizehint)
+        return self.reader.readlines(sizehint, keepends)
 
     def __next__(self):
 
@@ -881,7 +884,6 @@ class StreamRecoder:
 ### Shortcuts
 
 def open(filename, mode='r', encoding=None, errors='strict', buffering=-1):
-
     """ Open an encoded file using the given mode and return
         a wrapped version providing transparent encoding/decoding.
 
@@ -909,8 +911,11 @@ def open(filename, mode='r', encoding=None, errors='strict', buffering=-1):
         .encoding which allows querying the used encoding. This
         attribute is only available if an encoding was specified as
         parameter.
-
     """
+    import warnings
+    warnings.warn("codecs.open() is deprecated. Use open() instead.",
+                  DeprecationWarning, stacklevel=2)
+
     if encoding is not None and \
        'b' not in mode:
         # Force opening of the file in binary mode
@@ -1106,24 +1111,15 @@ def make_encoding_map(decoding_map):
 
 ### error handlers
 
-try:
-    strict_errors = lookup_error("strict")
-    ignore_errors = lookup_error("ignore")
-    replace_errors = lookup_error("replace")
-    xmlcharrefreplace_errors = lookup_error("xmlcharrefreplace")
-    backslashreplace_errors = lookup_error("backslashreplace")
-    namereplace_errors = lookup_error("namereplace")
-except LookupError:
-    # In --disable-unicode builds, these error handler are missing
-    strict_errors = None
-    ignore_errors = None
-    replace_errors = None
-    xmlcharrefreplace_errors = None
-    backslashreplace_errors = None
-    namereplace_errors = None
+strict_errors = lookup_error("strict")
+ignore_errors = lookup_error("ignore")
+replace_errors = lookup_error("replace")
+xmlcharrefreplace_errors = lookup_error("xmlcharrefreplace")
+backslashreplace_errors = lookup_error("backslashreplace")
+namereplace_errors = lookup_error("namereplace")
 
 # Tell modulefinder that using codecs probably needs the encodings
 # package
 _false = 0
 if _false:
-    import encodings
+    import encodings  # noqa: F401

Lib/codeop.py

@@ -44,9 +44,10 @@ __all__ = ["compile_command", "Compile", "CommandCompiler"]
 # Caveat emptor: These flags are undocumented on purpose and depending
 # on their effect outside the standard library is **unsupported**.
 PyCF_DONT_IMPLY_DEDENT = 0x200
+PyCF_ONLY_AST = 0x400
 PyCF_ALLOW_INCOMPLETE_INPUT = 0x4000
 
-def _maybe_compile(compiler, source, filename, symbol):
+def _maybe_compile(compiler, source, filename, symbol, flags):
     # Check for source consisting of only blank lines and comments.
     for line in source.split("\n"):
         line = line.strip()
@@ -60,10 +61,10 @@ def _maybe_compile(compiler, source, filename, symbol):
     with warnings.catch_warnings():
         warnings.simplefilter("ignore", (SyntaxWarning, DeprecationWarning))
         try:
-            compiler(source, filename, symbol)
+            compiler(source, filename, symbol, flags=flags)
         except SyntaxError:  # Let other compile() errors propagate.
             try:
-                compiler(source + "\n", filename, symbol)
+                compiler(source + "\n", filename, symbol, flags=flags)
                 return None
             except _IncompleteInputError as e:
                 return None
@@ -73,24 +74,13 @@ def _maybe_compile(compiler, source, filename, symbol):
 
     return compiler(source, filename, symbol, incomplete_input=False)
 
-def _is_syntax_error(err1, err2):
-    rep1 = repr(err1)
-    rep2 = repr(err2)
-    if "was never closed" in rep1 and "was never closed" in rep2:
-        return False
-    if rep1 == rep2:
-        return True
-    return False
-
-def _compile(source, filename, symbol, incomplete_input=True):
-    flags = 0
+def _compile(source, filename, symbol, incomplete_input=True, *, flags=0):
     if incomplete_input:
         flags |= PyCF_ALLOW_INCOMPLETE_INPUT
         flags |= PyCF_DONT_IMPLY_DEDENT
     return compile(source, filename, symbol, flags)
 
-
-def compile_command(source, filename="<input>", symbol="single"):
+def compile_command(source, filename="<input>", symbol="single", flags=0):
     r"""Compile a command and determine whether it is incomplete.
 
     Arguments:
@@ -109,7 +99,7 @@ def compile_command(source, filename="<input>", symbol="single"):
       syntax error (OverflowError and ValueError can be produced by
       malformed literals).
     """
-    return _maybe_compile(_compile, source, filename, symbol)
+    return _maybe_compile(_compile, source, filename, symbol, flags)
 
 class Compile:
     """Instances of this class behave much like the built-in compile
@@ -119,12 +109,14 @@ class Compile:
     def __init__(self):
         self.flags = PyCF_DONT_IMPLY_DEDENT | PyCF_ALLOW_INCOMPLETE_INPUT
 
-    def __call__(self, source, filename, symbol, **kwargs):
-        flags = self.flags
+    def __call__(self, source, filename, symbol, flags=0, **kwargs):
+        flags |= self.flags
         if kwargs.get('incomplete_input', True) is False:
             flags &= ~PyCF_DONT_IMPLY_DEDENT
             flags &= ~PyCF_ALLOW_INCOMPLETE_INPUT
         codeob = compile(source, filename, symbol, flags, True)
+        if flags & PyCF_ONLY_AST:
+            return codeob  # this is an ast.Module in this case
         for feature in _features:
             if codeob.co_flags & feature.compiler_flag:
                 self.flags |= feature.compiler_flag
@@ -159,4 +151,4 @@ class CommandCompiler:
           syntax error (OverflowError and ValueError can be produced by
           malformed literals).
         """
-        return _maybe_compile(self.compiler, source, filename, symbol)
+        return _maybe_compile(self.compiler, source, filename, symbol, flags=self.compiler.flags)

Lib/collections/__init__.py

@@ -29,6 +29,9 @@ __all__ = [
 import _collections_abc
 import sys as _sys
 
+_sys.modules['collections.abc'] = _collections_abc
+abc = _collections_abc
+
 from itertools import chain as _chain
 from itertools import repeat as _repeat
 from itertools import starmap as _starmap
@@ -46,19 +49,19 @@ else:
     _collections_abc.MutableSequence.register(deque)
 
 try:
-    from _collections import _deque_iterator
+    # Expose _deque_iterator to support pickling deque iterators
+    from _collections import _deque_iterator  # noqa: F401
 except ImportError:
     pass
 
 try:
     from _collections import defaultdict
 except ImportError:
-    # FIXME: try to implement defaultdict in collections.rs rather than in Python
-    # I (coolreader18) couldn't figure out some class stuff with __new__ and
-    # __init__ and __missing__ and subclassing built-in types from Rust, so I went
-    # with this instead.
+    # TODO: RUSTPYTHON - implement defaultdict in Rust
     from ._defaultdict import defaultdict
 
+heapq = None  # Lazily imported
+
 
 ################################################################################
 ### OrderedDict
@@ -461,7 +464,7 @@ def namedtuple(typename, field_names, *, rename=False, defaults=None, module=Non
     def _replace(self, /, **kwds):
         result = self._make(_map(kwds.pop, field_names, self))
         if kwds:
-            raise ValueError(f'Got unexpected field names: {list(kwds)!r}')
+            raise TypeError(f'Got unexpected field names: {list(kwds)!r}')
         return result
 
     _replace.__doc__ = (f'Return a new {typename} object replacing specified '
@@ -499,6 +502,7 @@ def namedtuple(typename, field_names, *, rename=False, defaults=None, module=Non
         '_field_defaults': field_defaults,
         '__new__': __new__,
         '_make': _make,
+        '__replace__': _replace,
         '_replace': _replace,
         '__repr__': __repr__,
         '_asdict': _asdict,
@@ -592,7 +596,7 @@ class Counter(dict):
     # References:
     #   http://en.wikipedia.org/wiki/Multiset
     #   http://www.gnu.org/software/smalltalk/manual-base/html_node/Bag.html
-    #   http://www.demo2s.com/Tutorial/Cpp/0380__set-multiset/Catalog0380__set-multiset.htm
+    #   http://www.java2s.com/Tutorial/Cpp/0380__set-multiset/Catalog0380__set-multiset.htm
     #   http://code.activestate.com/recipes/259174/
     #   Knuth, TAOCP Vol. II section 4.6.3
 
@@ -632,7 +636,10 @@ class Counter(dict):
             return sorted(self.items(), key=_itemgetter(1), reverse=True)
 
         # Lazy import to speedup Python startup time
-        import heapq
+        global heapq
+        if heapq is None:
+            import heapq
+
         return heapq.nlargest(n, self.items(), key=_itemgetter(1))
 
     def elements(self):
@@ -642,7 +649,8 @@ class Counter(dict):
         >>> sorted(c.elements())
         ['A', 'A', 'B', 'B', 'C', 'C']
 
-        # Knuth's example for prime factors of 1836:  2**2 * 3**3 * 17**1
+        Knuth's example for prime factors of 1836:  2**2 * 3**3 * 17**1
+
         >>> import math
         >>> prime_factors = Counter({2: 2, 3: 3, 17: 1})
         >>> math.prod(prime_factors.elements())
@@ -683,7 +691,7 @@ class Counter(dict):
 
         '''
         # The regular dict.update() operation makes no sense here because the
-        # replace behavior results in the some of original untouched counts
+        # replace behavior results in some of the original untouched counts
         # being mixed-in with all of the other counts for a mismash that
         # doesn't have a straight-forward interpretation in most counting
         # contexts.  Instead, we implement straight-addition.  Both the inputs
@@ -1018,7 +1026,7 @@ class ChainMap(_collections_abc.MutableMapping):
         return self.__missing__(key)            # support subclasses that define __missing__
 
     def get(self, key, default=None):
-        return self[key] if key in self else default
+        return self[key] if key in self else default    # needs to make use of __contains__
 
     def __len__(self):
         return len(set().union(*self.maps))     # reuses stored hash values if possible
@@ -1030,7 +1038,10 @@ class ChainMap(_collections_abc.MutableMapping):
         return iter(d)
 
     def __contains__(self, key):
-        return any(key in m for m in self.maps)
+        for mapping in self.maps:
+            if key in mapping:
+                return True
+        return False
 
     def __bool__(self):
         return any(self.maps)
@@ -1040,9 +1051,9 @@ class ChainMap(_collections_abc.MutableMapping):
         return f'{self.__class__.__name__}({", ".join(map(repr, self.maps))})'
 
     @classmethod
-    def fromkeys(cls, iterable, *args):
-        'Create a ChainMap with a single dict created from the iterable.'
-        return cls(dict.fromkeys(iterable, *args))
+    def fromkeys(cls, iterable, value=None, /):
+        'Create a new ChainMap with keys from iterable and values set to value.'
+        return cls(dict.fromkeys(iterable, value))
 
     def copy(self):
         'New ChainMap or subclass with a new copy of maps[0] and refs to maps[1:]'
@@ -1485,6 +1496,8 @@ class UserString(_collections_abc.Sequence):
         return self.data.format_map(mapping)
 
     def index(self, sub, start=0, end=_sys.maxsize):
+        if isinstance(sub, UserString):
+            sub = sub.data
         return self.data.index(sub, start, end)
 
     def isalpha(self):
@@ -1553,6 +1566,8 @@ class UserString(_collections_abc.Sequence):
         return self.data.rfind(sub, start, end)
 
     def rindex(self, sub, start=0, end=_sys.maxsize):
+        if isinstance(sub, UserString):
+            sub = sub.data
         return self.data.rindex(sub, start, end)
 
     def rjust(self, width, *args):

Lib/collections/abc.py

@@ -1,3 +0,0 @@
-from _collections_abc import *
-from _collections_abc import __all__
-from _collections_abc import _CallableGenericAlias

Lib/compileall.py

@@ -317,7 +317,9 @@ def main():
     import argparse
 
     parser = argparse.ArgumentParser(
-        description='Utilities to support installing Python libraries.')
+        description='Utilities to support installing Python libraries.',
+        color=True,
+    )
     parser.add_argument('-l', action='store_const', const=0,
                         default=None, dest='maxlevels',
                         help="don't recurse into subdirectories")

Lib/compression/_common/_streams.py

@@ -1,4 +1,4 @@
-"""Internal classes used by the gzip, lzma and bz2 modules"""
+"""Internal classes used by compression modules"""
 
 import io
 import sys

Lib/compression/bz2.py

@@ -0,0 +1,5 @@
+import bz2
+__doc__ = bz2.__doc__
+del bz2
+
+from bz2 import *

Lib/compression/gzip.py

@@ -0,0 +1,5 @@
+import gzip
+__doc__ = gzip.__doc__
+del gzip
+
+from gzip import *

Lib/compression/lzma.py

@@ -0,0 +1,5 @@
+import lzma
+__doc__ = lzma.__doc__
+del lzma
+
+from lzma import *

Lib/compression/zlib.py

@@ -0,0 +1,5 @@
+import zlib
+__doc__ = zlib.__doc__
+del zlib
+
+from zlib import *

Lib/compression/zstd/__init__.py

@@ -0,0 +1,242 @@
+"""Python bindings to the Zstandard (zstd) compression library (RFC-8878)."""
+
+__all__ = (
+    # compression.zstd
+    'COMPRESSION_LEVEL_DEFAULT',
+    'compress',
+    'CompressionParameter',
+    'decompress',
+    'DecompressionParameter',
+    'finalize_dict',
+    'get_frame_info',
+    'Strategy',
+    'train_dict',
+
+    # compression.zstd._zstdfile
+    'open',
+    'ZstdFile',
+
+    # _zstd
+    'get_frame_size',
+    'zstd_version',
+    'zstd_version_info',
+    'ZstdCompressor',
+    'ZstdDecompressor',
+    'ZstdDict',
+    'ZstdError',
+)
+
+import _zstd
+import enum
+from _zstd import (ZstdCompressor, ZstdDecompressor, ZstdDict, ZstdError,
+                   get_frame_size, zstd_version)
+from compression.zstd._zstdfile import ZstdFile, open, _nbytes
+
+# zstd_version_number is (MAJOR * 100 * 100 + MINOR * 100 + RELEASE)
+zstd_version_info = (*divmod(_zstd.zstd_version_number // 100, 100),
+                     _zstd.zstd_version_number % 100)
+"""Version number of the runtime zstd library as a tuple of integers."""
+
+COMPRESSION_LEVEL_DEFAULT = _zstd.ZSTD_CLEVEL_DEFAULT
+"""The default compression level for Zstandard, currently '3'."""
+
+
+class FrameInfo:
+    """Information about a Zstandard frame."""
+
+    __slots__ = 'decompressed_size', 'dictionary_id'
+
+    def __init__(self, decompressed_size, dictionary_id):
+        super().__setattr__('decompressed_size', decompressed_size)
+        super().__setattr__('dictionary_id', dictionary_id)
+
+    def __repr__(self):
+        return (f'FrameInfo(decompressed_size={self.decompressed_size}, '
+                f'dictionary_id={self.dictionary_id})')
+
+    def __setattr__(self, name, _):
+        raise AttributeError(f"can't set attribute {name!r}")
+
+
+def get_frame_info(frame_buffer):
+    """Get Zstandard frame information from a frame header.
+
+    *frame_buffer* is a bytes-like object. It should start from the beginning
+    of a frame, and needs to include at least the frame header (6 to 18 bytes).
+
+    The returned FrameInfo object has two attributes.
+    'decompressed_size' is the size in bytes of the data in the frame when
+    decompressed, or None when the decompressed size is unknown.
+    'dictionary_id' is an int in the range (0, 2**32). The special value 0
+    means that the dictionary ID was not recorded in the frame header,
+    the frame may or may not need a dictionary to be decoded,
+    and the ID of such a dictionary is not specified.
+    """
+    return FrameInfo(*_zstd.get_frame_info(frame_buffer))
+
+
+def train_dict(samples, dict_size):
+    """Return a ZstdDict representing a trained Zstandard dictionary.
+
+    *samples* is an iterable of samples, where a sample is a bytes-like
+    object representing a file.
+
+    *dict_size* is the dictionary's maximum size, in bytes.
+    """
+    if not isinstance(dict_size, int):
+        ds_cls = type(dict_size).__qualname__
+        raise TypeError(f'dict_size must be an int object, not {ds_cls!r}.')
+
+    samples = tuple(samples)
+    chunks = b''.join(samples)
+    chunk_sizes = tuple(_nbytes(sample) for sample in samples)
+    if not chunks:
+        raise ValueError("samples contained no data; can't train dictionary.")
+    dict_content = _zstd.train_dict(chunks, chunk_sizes, dict_size)
+    return ZstdDict(dict_content)
+
+
+def finalize_dict(zstd_dict, /, samples, dict_size, level):
+    """Return a ZstdDict representing a finalized Zstandard dictionary.
+
+    Given a custom content as a basis for dictionary, and a set of samples,
+    finalize *zstd_dict* by adding headers and statistics according to the
+    Zstandard dictionary format.
+
+    You may compose an effective dictionary content by hand, which is used as
+    basis dictionary, and use some samples to finalize a dictionary. The basis
+    dictionary may be a "raw content" dictionary. See *is_raw* in ZstdDict.
+
+    *samples* is an iterable of samples, where a sample is a bytes-like object
+    representing a file.
+    *dict_size* is the dictionary's maximum size, in bytes.
+    *level* is the expected compression level. The statistics for each
+    compression level differ, so tuning the dictionary to the compression level
+    can provide improvements.
+    """
+
+    if not isinstance(zstd_dict, ZstdDict):
+        raise TypeError('zstd_dict argument should be a ZstdDict object.')
+    if not isinstance(dict_size, int):
+        raise TypeError('dict_size argument should be an int object.')
+    if not isinstance(level, int):
+        raise TypeError('level argument should be an int object.')
+
+    samples = tuple(samples)
+    chunks = b''.join(samples)
+    chunk_sizes = tuple(_nbytes(sample) for sample in samples)
+    if not chunks:
+        raise ValueError("The samples are empty content, can't finalize the "
+                         "dictionary.")
+    dict_content = _zstd.finalize_dict(zstd_dict.dict_content, chunks,
+                                       chunk_sizes, dict_size, level)
+    return ZstdDict(dict_content)
+
+
+def compress(data, level=None, options=None, zstd_dict=None):
+    """Return Zstandard compressed *data* as bytes.
+
+    *level* is an int specifying the compression level to use, defaulting to
+    COMPRESSION_LEVEL_DEFAULT ('3').
+    *options* is a dict object that contains advanced compression
+    parameters. See CompressionParameter for more on options.
+    *zstd_dict* is a ZstdDict object, a pre-trained Zstandard dictionary. See
+    the function train_dict for how to train a ZstdDict on sample data.
+
+    For incremental compression, use a ZstdCompressor instead.
+    """
+    comp = ZstdCompressor(level=level, options=options, zstd_dict=zstd_dict)
+    return comp.compress(data, mode=ZstdCompressor.FLUSH_FRAME)
+
+
+def decompress(data, zstd_dict=None, options=None):
+    """Decompress one or more frames of Zstandard compressed *data*.
+
+    *zstd_dict* is a ZstdDict object, a pre-trained Zstandard dictionary. See
+    the function train_dict for how to train a ZstdDict on sample data.
+    *options* is a dict object that contains advanced compression
+    parameters. See DecompressionParameter for more on options.
+
+    For incremental decompression, use a ZstdDecompressor instead.
+    """
+    results = []
+    while True:
+        decomp = ZstdDecompressor(options=options, zstd_dict=zstd_dict)
+        results.append(decomp.decompress(data))
+        if not decomp.eof:
+            raise ZstdError('Compressed data ended before the '
+                            'end-of-stream marker was reached')
+        data = decomp.unused_data
+        if not data:
+            break
+    return b''.join(results)
+
+
+class CompressionParameter(enum.IntEnum):
+    """Compression parameters."""
+
+    compression_level = _zstd.ZSTD_c_compressionLevel
+    window_log = _zstd.ZSTD_c_windowLog
+    hash_log = _zstd.ZSTD_c_hashLog
+    chain_log = _zstd.ZSTD_c_chainLog
+    search_log = _zstd.ZSTD_c_searchLog
+    min_match = _zstd.ZSTD_c_minMatch
+    target_length = _zstd.ZSTD_c_targetLength
+    strategy = _zstd.ZSTD_c_strategy
+
+    enable_long_distance_matching = _zstd.ZSTD_c_enableLongDistanceMatching
+    ldm_hash_log = _zstd.ZSTD_c_ldmHashLog
+    ldm_min_match = _zstd.ZSTD_c_ldmMinMatch
+    ldm_bucket_size_log = _zstd.ZSTD_c_ldmBucketSizeLog
+    ldm_hash_rate_log = _zstd.ZSTD_c_ldmHashRateLog
+
+    content_size_flag = _zstd.ZSTD_c_contentSizeFlag
+    checksum_flag = _zstd.ZSTD_c_checksumFlag
+    dict_id_flag = _zstd.ZSTD_c_dictIDFlag
+
+    nb_workers = _zstd.ZSTD_c_nbWorkers
+    job_size = _zstd.ZSTD_c_jobSize
+    overlap_log = _zstd.ZSTD_c_overlapLog
+
+    def bounds(self):
+        """Return the (lower, upper) int bounds of a compression parameter.
+
+        Both the lower and upper bounds are inclusive.
+        """
+        return _zstd.get_param_bounds(self.value, is_compress=True)
+
+
+class DecompressionParameter(enum.IntEnum):
+    """Decompression parameters."""
+
+    window_log_max = _zstd.ZSTD_d_windowLogMax
+
+    def bounds(self):
+        """Return the (lower, upper) int bounds of a decompression parameter.
+
+        Both the lower and upper bounds are inclusive.
+        """
+        return _zstd.get_param_bounds(self.value, is_compress=False)
+
+
+class Strategy(enum.IntEnum):
+    """Compression strategies, listed from fastest to strongest.
+
+    Note that new strategies might be added in the future.
+    Only the order (from fast to strong) is guaranteed,
+    the numeric value might change.
+    """
+
+    fast = _zstd.ZSTD_fast
+    dfast = _zstd.ZSTD_dfast
+    greedy = _zstd.ZSTD_greedy
+    lazy = _zstd.ZSTD_lazy
+    lazy2 = _zstd.ZSTD_lazy2
+    btlazy2 = _zstd.ZSTD_btlazy2
+    btopt = _zstd.ZSTD_btopt
+    btultra = _zstd.ZSTD_btultra
+    btultra2 = _zstd.ZSTD_btultra2
+
+
+# Check validity of the CompressionParameter & DecompressionParameter types
+_zstd.set_parameter_types(CompressionParameter, DecompressionParameter)

Lib/compression/zstd/_zstdfile.py

@@ -0,0 +1,345 @@
+import io
+from os import PathLike
+from _zstd import ZstdCompressor, ZstdDecompressor, ZSTD_DStreamOutSize
+from compression._common import _streams
+
+__all__ = ('ZstdFile', 'open')
+
+_MODE_CLOSED = 0
+_MODE_READ = 1
+_MODE_WRITE = 2
+
+
+def _nbytes(dat, /):
+    if isinstance(dat, (bytes, bytearray)):
+        return len(dat)
+    with memoryview(dat) as mv:
+        return mv.nbytes
+
+
+class ZstdFile(_streams.BaseStream):
+    """A file-like object providing transparent Zstandard (de)compression.
+
+    A ZstdFile can act as a wrapper for an existing file object, or refer
+    directly to a named file on disk.
+
+    ZstdFile provides a *binary* file interface. Data is read and returned as
+    bytes, and may only be written to objects that support the Buffer Protocol.
+    """
+
+    FLUSH_BLOCK = ZstdCompressor.FLUSH_BLOCK
+    FLUSH_FRAME = ZstdCompressor.FLUSH_FRAME
+
+    def __init__(self, file, /, mode='r', *,
+                 level=None, options=None, zstd_dict=None):
+        """Open a Zstandard compressed file in binary mode.
+
+        *file* can be either an file-like object, or a file name to open.
+
+        *mode* can be 'r' for reading (default), 'w' for (over)writing, 'x' for
+        creating exclusively, or 'a' for appending. These can equivalently be
+        given as 'rb', 'wb', 'xb' and 'ab' respectively.
+
+        *level* is an optional int specifying the compression level to use,
+        or COMPRESSION_LEVEL_DEFAULT if not given.
+
+        *options* is an optional dict for advanced compression parameters.
+        See CompressionParameter and DecompressionParameter for the possible
+        options.
+
+        *zstd_dict* is an optional ZstdDict object, a pre-trained Zstandard
+        dictionary. See train_dict() to train ZstdDict on sample data.
+        """
+        self._fp = None
+        self._close_fp = False
+        self._mode = _MODE_CLOSED
+        self._buffer = None
+
+        if not isinstance(mode, str):
+            raise ValueError('mode must be a str')
+        if options is not None and not isinstance(options, dict):
+            raise TypeError('options must be a dict or None')
+        mode = mode.removesuffix('b')  # handle rb, wb, xb, ab
+        if mode == 'r':
+            if level is not None:
+                raise TypeError('level is illegal in read mode')
+            self._mode = _MODE_READ
+        elif mode in {'w', 'a', 'x'}:
+            if level is not None and not isinstance(level, int):
+                raise TypeError('level must be int or None')
+            self._mode = _MODE_WRITE
+            self._compressor = ZstdCompressor(level=level, options=options,
+                                              zstd_dict=zstd_dict)
+            self._pos = 0
+        else:
+            raise ValueError(f'Invalid mode: {mode!r}')
+
+        if isinstance(file, (str, bytes, PathLike)):
+            self._fp = io.open(file, f'{mode}b')
+            self._close_fp = True
+        elif ((mode == 'r' and hasattr(file, 'read'))
+                or (mode != 'r' and hasattr(file, 'write'))):
+            self._fp = file
+        else:
+            raise TypeError('file must be a file-like object '
+                            'or a str, bytes, or PathLike object')
+
+        if self._mode == _MODE_READ:
+            raw = _streams.DecompressReader(
+                self._fp,
+                ZstdDecompressor,
+                zstd_dict=zstd_dict,
+                options=options,
+            )
+            self._buffer = io.BufferedReader(raw)
+
+    def close(self):
+        """Flush and close the file.
+
+        May be called multiple times. Once the file has been closed,
+        any other operation on it will raise ValueError.
+        """
+        if self._fp is None:
+            return
+        try:
+            if self._mode == _MODE_READ:
+                if getattr(self, '_buffer', None):
+                    self._buffer.close()
+                    self._buffer = None
+            elif self._mode == _MODE_WRITE:
+                self.flush(self.FLUSH_FRAME)
+                self._compressor = None
+        finally:
+            self._mode = _MODE_CLOSED
+            try:
+                if self._close_fp:
+                    self._fp.close()
+            finally:
+                self._fp = None
+                self._close_fp = False
+
+    def write(self, data, /):
+        """Write a bytes-like object *data* to the file.
+
+        Returns the number of uncompressed bytes written, which is
+        always the length of data in bytes. Note that due to buffering,
+        the file on disk may not reflect the data written until .flush()
+        or .close() is called.
+        """
+        self._check_can_write()
+
+        length = _nbytes(data)
+
+        compressed = self._compressor.compress(data)
+        self._fp.write(compressed)
+        self._pos += length
+        return length
+
+    def flush(self, mode=FLUSH_BLOCK):
+        """Flush remaining data to the underlying stream.
+
+        The mode argument can be FLUSH_BLOCK or FLUSH_FRAME. Abuse of this
+        method will reduce compression ratio, use it only when necessary.
+
+        If the program is interrupted afterwards, all data can be recovered.
+        To ensure saving to disk, also need to use os.fsync(fd).
+
+        This method does nothing in reading mode.
+        """
+        if self._mode == _MODE_READ:
+            return
+        self._check_not_closed()
+        if mode not in {self.FLUSH_BLOCK, self.FLUSH_FRAME}:
+            raise ValueError('Invalid mode argument, expected either '
+                             'ZstdFile.FLUSH_FRAME or '
+                             'ZstdFile.FLUSH_BLOCK')
+        if self._compressor.last_mode == mode:
+            return
+        # Flush zstd block/frame, and write.
+        data = self._compressor.flush(mode)
+        self._fp.write(data)
+        if hasattr(self._fp, 'flush'):
+            self._fp.flush()
+
+    def read(self, size=-1):
+        """Read up to size uncompressed bytes from the file.
+
+        If size is negative or omitted, read until EOF is reached.
+        Returns b'' if the file is already at EOF.
+        """
+        if size is None:
+            size = -1
+        self._check_can_read()
+        return self._buffer.read(size)
+
+    def read1(self, size=-1):
+        """Read up to size uncompressed bytes, while trying to avoid
+        making multiple reads from the underlying stream. Reads up to a
+        buffer's worth of data if size is negative.
+
+        Returns b'' if the file is at EOF.
+        """
+        self._check_can_read()
+        if size < 0:
+            # Note this should *not* be io.DEFAULT_BUFFER_SIZE.
+            # ZSTD_DStreamOutSize is the minimum amount to read guaranteeing
+            # a full block is read.
+            size = ZSTD_DStreamOutSize
+        return self._buffer.read1(size)
+
+    def readinto(self, b):
+        """Read bytes into b.
+
+        Returns the number of bytes read (0 for EOF).
+        """
+        self._check_can_read()
+        return self._buffer.readinto(b)
+
+    def readinto1(self, b):
+        """Read bytes into b, while trying to avoid making multiple reads
+        from the underlying stream.
+
+        Returns the number of bytes read (0 for EOF).
+        """
+        self._check_can_read()
+        return self._buffer.readinto1(b)
+
+    def readline(self, size=-1):
+        """Read a line of uncompressed bytes from the file.
+
+        The terminating newline (if present) is retained. If size is
+        non-negative, no more than size bytes will be read (in which
+        case the line may be incomplete). Returns b'' if already at EOF.
+        """
+        self._check_can_read()
+        return self._buffer.readline(size)
+
+    def seek(self, offset, whence=io.SEEK_SET):
+        """Change the file position.
+
+        The new position is specified by offset, relative to the
+        position indicated by whence. Possible values for whence are:
+
+            0: start of stream (default): offset must not be negative
+            1: current stream position
+            2: end of stream; offset must not be positive
+
+        Returns the new file position.
+
+        Note that seeking is emulated, so depending on the arguments,
+        this operation may be extremely slow.
+        """
+        self._check_can_read()
+
+        # BufferedReader.seek() checks seekable
+        return self._buffer.seek(offset, whence)
+
+    def peek(self, size=-1):
+        """Return buffered data without advancing the file position.
+
+        Always returns at least one byte of data, unless at EOF.
+        The exact number of bytes returned is unspecified.
+        """
+        # Relies on the undocumented fact that BufferedReader.peek() always
+        # returns at least one byte (except at EOF)
+        self._check_can_read()
+        return self._buffer.peek(size)
+
+    def __next__(self):
+        if ret := self._buffer.readline():
+            return ret
+        raise StopIteration
+
+    def tell(self):
+        """Return the current file position."""
+        self._check_not_closed()
+        if self._mode == _MODE_READ:
+            return self._buffer.tell()
+        elif self._mode == _MODE_WRITE:
+            return self._pos
+
+    def fileno(self):
+        """Return the file descriptor for the underlying file."""
+        self._check_not_closed()
+        return self._fp.fileno()
+
+    @property
+    def name(self):
+        self._check_not_closed()
+        return self._fp.name
+
+    @property
+    def mode(self):
+        return 'wb' if self._mode == _MODE_WRITE else 'rb'
+
+    @property
+    def closed(self):
+        """True if this file is closed."""
+        return self._mode == _MODE_CLOSED
+
+    def seekable(self):
+        """Return whether the file supports seeking."""
+        return self.readable() and self._buffer.seekable()
+
+    def readable(self):
+        """Return whether the file was opened for reading."""
+        self._check_not_closed()
+        return self._mode == _MODE_READ
+
+    def writable(self):
+        """Return whether the file was opened for writing."""
+        self._check_not_closed()
+        return self._mode == _MODE_WRITE
+
+
+def open(file, /, mode='rb', *, level=None, options=None, zstd_dict=None,
+         encoding=None, errors=None, newline=None):
+    """Open a Zstandard compressed file in binary or text mode.
+
+    file can be either a file name (given as a str, bytes, or PathLike object),
+    in which case the named file is opened, or it can be an existing file object
+    to read from or write to.
+
+    The mode parameter can be 'r', 'rb' (default), 'w', 'wb', 'x', 'xb', 'a',
+    'ab' for binary mode, or 'rt', 'wt', 'xt', 'at' for text mode.
+
+    The level, options, and zstd_dict parameters specify the settings the same
+    as ZstdFile.
+
+    When using read mode (decompression), the options parameter is a dict
+    representing advanced decompression options. The level parameter is not
+    supported in this case. When using write mode (compression), only one of
+    level, an int representing the compression level, or options, a dict
+    representing advanced compression options, may be passed. In both modes,
+    zstd_dict is a ZstdDict instance containing a trained Zstandard dictionary.
+
+    For binary mode, this function is equivalent to the ZstdFile constructor:
+    ZstdFile(filename, mode, ...). In this case, the encoding, errors and
+    newline parameters must not be provided.
+
+    For text mode, an ZstdFile object is created, and wrapped in an
+    io.TextIOWrapper instance with the specified encoding, error handling
+    behavior, and line ending(s).
+    """
+
+    text_mode = 't' in mode
+    mode = mode.replace('t', '')
+
+    if text_mode:
+        if 'b' in mode:
+            raise ValueError(f'Invalid mode: {mode!r}')
+    else:
+        if encoding is not None:
+            raise ValueError('Argument "encoding" not supported in binary mode')
+        if errors is not None:
+            raise ValueError('Argument "errors" not supported in binary mode')
+        if newline is not None:
+            raise ValueError('Argument "newline" not supported in binary mode')
+
+    binary_file = ZstdFile(file, mode, level=level, options=options,
+                           zstd_dict=zstd_dict)
+
+    if text_mode:
+        return io.TextIOWrapper(binary_file, encoding, errors, newline)
+    else:
+        return binary_file

Lib/concurrent/futures/__init__.py

@@ -23,6 +23,7 @@ __all__ = (
     'ALL_COMPLETED',
     'CancelledError',
     'TimeoutError',
+    'InvalidStateError',
     'BrokenExecutor',
     'Future',
     'Executor',
@@ -50,4 +51,4 @@ def __getattr__(name):
         ThreadPoolExecutor = te
         return te
 
-    raise AttributeError(f"module {__name__} has no attribute {name}")
+    raise AttributeError(f"module {__name__!r} has no attribute {name!r}")

Lib/concurrent/futures/_base.py

@@ -50,9 +50,7 @@ class CancelledError(Error):
     """The Future was cancelled."""
     pass
 
-class TimeoutError(Error):
-    """The operation exceeded the given deadline."""
-    pass
+TimeoutError = TimeoutError  # make local alias for the standard exception
 
 class InvalidStateError(Error):
     """The operation is not allowed in this state."""
@@ -284,7 +282,7 @@ def wait(fs, timeout=None, return_when=ALL_COMPLETED):
         A named 2-tuple of sets. The first set, named 'done', contains the
         futures that completed (is finished or cancelled) before the wait
         completed. The second set, named 'not_done', contains uncompleted
-        futures. Duplicate futures given to *fs* are removed and will be 
+        futures. Duplicate futures given to *fs* are removed and will be
         returned only once.
     """
     fs = set(fs)
@@ -312,6 +310,18 @@ def wait(fs, timeout=None, return_when=ALL_COMPLETED):
     done.update(waiter.finished_futures)
     return DoneAndNotDoneFutures(done, fs - done)
 
+
+def _result_or_cancel(fut, timeout=None):
+    try:
+        try:
+            return fut.result(timeout)
+        finally:
+            fut.cancel()
+    finally:
+        # Break a reference cycle with the exception in self._exception
+        del fut
+
+
 class Future(object):
     """Represents the result of an asynchronous computation."""
 
@@ -386,7 +396,7 @@ class Future(object):
             return self._state in [CANCELLED, CANCELLED_AND_NOTIFIED, FINISHED]
 
     def __get_result(self):
-        if self._exception:
+        if self._exception is not None:
             try:
                 raise self._exception
             finally:
@@ -606,9 +616,9 @@ class Executor(object):
                 while fs:
                     # Careful not to keep a reference to the popped future
                     if timeout is None:
-                        yield fs.pop().result()
+                        yield _result_or_cancel(fs.pop())
                     else:
-                        yield fs.pop().result(end_time - time.monotonic())
+                        yield _result_or_cancel(fs.pop(), end_time - time.monotonic())
             finally:
                 for future in fs:
                     future.cancel()