RustPython/DEVELOPMENT.md

RustPython Development Guide and Tips

RustPython attracts developers with interest and experience in Rust, Python, or WebAssembly. Whether you are familiar with Rust, Python, or WebAssembly, the goal of this Development Guide is to give you the basics to get set up for developing RustPython and contributing to this project.

The contents of the Development Guide include:

• Setting up a development environment
• Code style
• Testing
• Profiling
• Code organization
• Understanding internals
• Questions

Setting up a development environment

RustPython requires the following:

• Rust 1.36 or higher
  • To check Rust version: rustc --version
  • If you have rustup on your system, enter to update to the latest stable version: rustup update stable
  • If you do not have Rust installed, use rustup to do so.
• CPython version 3.7.4 or higher
  • CPython can be installed by your operating system's package manager, from the Python website, or using a third-party distribution, such as Anaconda.
• [Optional] The Python package, pytest, is used for testing Python code snippets. To install, enter python3 -m pip install pytest.

Code style

The Rust code style used is the default rustfmt codestyle. Please format your code accordingly. We also use clippy to detect rust code issues.

Python code should follow the PEP 8 style.

Testing

To test RustPython's functionality, a collection of Python snippets are located in the tests/snippets directory and can be run using pytest:

$ cd tests
$ pytest -v

Rust unit tests can be run with cargo:

$ cargo test --all

Profiling

To profile RustPython, build it in release mode with the flame-it feature. This will generate a file flamescope.json, which can be viewed at https://speedscope.app.

$ cargo run --release --features flame-it script.py

To display the default output json file:

$ cat flamescope.json
{<json>}

You can specify another file name other than the default by using the --output-file option to specify a file name (or stdout if you specify -). The --output-format option determines the format of the output file. The speedscope json format (default), text, or raw html can be passed. There exists a raw html viewer which is currently broken, and we welcome a PR to fix it.

Code organization

Understanding a new codebase takes time. Here's a brief view of the repository's structure:

• bytecode/src: python bytecode representation in rust structures
• compiler/src: python compilation to bytecode
• derive/src: Rust language extensions and macros specific to rustpython
• parser/src: python lexing, parsing and ast
• Lib: Carefully selected / copied files from CPython sourcecode. This is the python side of the standard library.
• vm/src: python virtual machine
  • builtins.rs: Builtin functions
  • compile.rs: the python compiler from ast to bytecode
  • obj: python builtin types
  • stdlib: Standard library parts implemented in rust.
• src: using the other subcrates to bring rustpython to life.
• docs: documentation (work in progress)
• py_code_object: CPython bytecode to rustpython bytecode converter (work in progress)
• wasm: Binary crate and resources for WebAssembly build
• tests: integration test snippets

Understanding Internals

Rust crates rustpython: top level crate rustpython_parser rustpython-vm (source is found in vm/src)

Lexer, Parser, AST (Abstract Syntax Tree)

• Lexer: parser/lexer.rs converts Python source code into tokens
• Parser: Takes the tokens generated by the lexer and parses the tokens into an AST (Abstract Syntax Tree) where the nodes of the syntax tree are Rust structs and enums
  • The Parser relies on LALRPOP, a Rust parser generator framework
  • More information on parsers and a tutorial can be found in the LALRPOP book.

Compiler Generates bytecode from the AST

VM (Virtual Machine) Fetch and dispatch loop

Import system

Builtin Objects

Questions

Have you tried these steps and have a question, please chat with us on gitter.