fd2e0aa41e
* Initial implementation of named expression and import according CPython tests * added new instruction with inversed evaluation order for dict comprehension, in other cases use regular evaluation order * added further aspects to implementation, cleaned up, imported test from CPython * implemented first parts of scoping enhancement and extended checks * completion of name resolution ongoing, now more test passing, still warinings and cleanup required * further optimization of name resolution in nested scopes * Initialize the vm with imports from _io instead of io * Add the OpenBSD support that I am smart enough to * adapted grammer to full support, most test are passing now, esp. all invalids are passed. Cleaned up in symboltable * more conditional compiling, this time for errors * rustfmt was not pleased * premature push, whoops * Add expected_failure result type to jsontests * Initial implementation of named expression and import according CPython tests * added new instruction with inversed evaluation order for dict comprehension, in other cases use regular evaluation order * added further aspects to implementation, cleaned up, imported test from CPython * implemented first parts of scoping enhancement and extended checks * completion of name resolution ongoing, now more test passing, still warinings and cleanup required * further optimization of name resolution in nested scopes * adapted grammer to full support, most test are passing now, esp. all invalids are passed. Cleaned up in symboltable * Fixed nameing convention violation and removed unnecessary information from symbol resolution. Added some more comments. * Fixed nameing convention violation and removed unnecessary information from symbol resolution. Added some more comments. Co-authored-by: Noah <33094578+coolreader18@users.noreply.github.com> Co-authored-by: Reuben Staley <lighthousemaniac@gmail.com>
RustPython/parser
This directory has the code for python lexing, parsing and generating Abstract Syntax Trees (AST).
The steps are:
- Lexical analysis: splits the source code into tokens.
- Parsing and generating the AST: transforms those tokens into an AST. Uses
LALRPOP, a Rust parser generator framework.
This crate is published on https://docs.rs/rustpython-parser.
We wrote a blog post with screenshots and an explanation to help you understand the steps by seeing them in action.
For more information on LALRPOP, here is a link to the LALRPOP book.
There is a readme in the src folder with the details of each file.
Directory content
build.rs: The build script.
Cargo.toml: The config file.
The src directory has:
lib.rs
This is the crate's root.
lexer.rs
This module takes care of lexing python source text. This means source code is translated into separate tokens.
parser.rs
A python parsing module. Use this module to parse python code into an AST. There are three ways to parse python code. You could parse a whole program, a single statement, or a single expression.
ast.rs
Implements abstract syntax tree (AST) nodes for the python language. Roughly equivalent to the python AST.
python.lalrpop
Python grammar.
token.rs
Different token definitions. Loosely based on token.h from CPython source.
errors.rs
Define internal parse error types. The goal is to provide a matching and a safe error API, masking errors from LALR.
fstring.rs
Format strings.
function.rs
Collection of functions for parsing parameters, arguments.
location.rs
Datatypes to support source location information.
mode.rs
Execution mode check. Allowed modes are exec, eval or single.
How to use
For example, one could do this:
use rustpython_parser::{parser, ast};
let python_source = "print('Hello world')";
let python_ast = parser::parse_expression(python_source).unwrap();