Add subsections to Understanding Internals

This commit is contained in:
Carol Willing
2019-08-16 09:36:20 +09:00
parent 2426440c00
commit cc99c38678

View File

@@ -41,7 +41,8 @@ code accordingly. We also use [clippy](https://github.com/rust-lang/rust-clippy)
to detect rust code issues.
Python code should follow the
[PEP 8](https://www.python.org/dev/peps/pep-0008/) style.
[PEP 8](https://www.python.org/dev/peps/pep-0008/) style. We also use
[flake8](http://flake8.pycqa.org/en/latest/) to check Python code style.
## Testing
@@ -107,22 +108,46 @@ repository's structure:
## Understanding Internals
Rust crates
rustpython: top level crate
rustpython_parser
rustpython-vm (source is found in `vm/src`)
The RustPython package is the `rustpython` top-level crate. The `Cargo.toml`
file in the root of the repo provide configuration of the crate and the
implementation is found in the `src` directory (specifically,
`src/main.rs`).
Lexer, Parser, AST (Abstract Syntax Tree)
The top-level `rustpython` depends on several lower-level crates including:
- `rustpython-parser` (source: `parser/src`)
- `rustpython-compiler` (source:`compiler/src`)
- `rustpython-vm` (source: `vm/src`)
Together, these crates provide the functions of a programming language and
enable a line of code to go through a series of steps:
- parse the line of source code into tokens
- determine if the tokens are valid syntax
- create an Abstract Syntax Tree (AST)
- compile the AST into bytecode
- execute the bytecode in the virtual machine (VM).
### rustpython-parser
This crate contains the lexer and parser to convert a line of code to
an Abstract Syntax Tree (AST):
- Lexer: `parser/lexer.rs` converts Python source code into tokens
- Parser: Takes the tokens generated by the lexer and parses the tokens
into an AST (Abstract Syntax Tree) where the nodes of the syntax tree
are Rust structs and enums
are Rust structs and enums.
- The Parser relies on `LALRPOP`, a Rust parser generator framework
- More information on parsers and a tutorial can be found in the
[LALRPOP book](https://lalrpop.github.io/lalrpop/README.html).
- AST: `parser/ast.rs` implements Python types and expressions
represented in by the AST nodes in Rust.
### rustpython-compiler
Compiler
Generates bytecode from the AST
(source is found in `compiler/src`) is primarily used to transform AST (Abstract Syntax Tree to bytecode)
### rustpython-vm
VM (Virtual Machine)
Fetch and dispatch loop