A query tool for Python abstract syntax trees
Project description
A command-line utility for grepping Python files using XPath syntax against the Python AST (Abstract Syntax Tree).
In other words, this allows you to search Python code against specific syntax elements (function definitions, arguments, assignments, variables etc), instead of grepping for string matches.
The interface and behaviour is designed to match grep and ripgrep as far as it makes sense to do so.
Status: usable, with test suite, but still cleaning things up and implementing some options
No PyPI package yet.
Installation
Python 3.7+ required.
We recommend pipx to install it conveniently in an isolated environment:
pipx install pyastgrep
You can also use pip:
pip install pyastgrep
Understanding the XML structure
To get started, you’ll need some understanding of how Python AST is structured, and how that is mapped to XML. Some methods for doing that are below:
Use Python AST Explorer to play around with what AST looks like.
Dump out the AST and/or XML structure of the top-level statements in a Python file. The top-level XML elements are <Module><body>, and don’t correspond to actual source lines. To get the statements within the body, you can use an XPath expression /Module/body/* or ./*/*:
$ pyastgrep --xml --ast './*/*' myfile.py myfile.py:1:1:import os Import( lineno=1, col_offset=0, end_lineno=1, end_col_offset=9, names=[alias(lineno=1, col_offset=7, end_lineno=1, end_col_offset=9, name='os', asname=None)], ) ... <Import lineno="1" col_offset="0"> <names> <alias lineno="1" col_offset="7" type="str" name="os"/> </names> </Import> ...
Note that the XML format is a very direct translation of the Python AST as produced by the ast module (with some small additions made to improve usability for a few cases). This AST is not stable across Python versions, so the XML is not stable either. Normally changes in the AST correspond to new syntax that is added to Python, but in some cases a new Python version will make significant changes made to the AST generated for the same code.
You can also pipe specific Python fragments using - to specify stdin as the input file:
$ echo 'a + b' | pyastgrep --xml './*/*' -
<stdin>:1:1:a + b
<Expr lineno="1" col_offset="0">
<value>
<BinOp lineno="1" col_offset="0">
<left>
<Name lineno="1" col_offset="0" type="str" id="a">
<ctx>
<Load/>
</ctx>
</Name>
</left>
<op>
<Add/>
</op>
...
You’ll also need some understanding of how to write XPath expressions (see links below), but the examples below should get you started.
Examples
Find all usages of a function called open:
$ pyastgrep ".//Call/func/Name[@id='open']"
src/pyastgrep/search.py:88:18: with open(path) as f:
Find all literal numbers (Python 3.8+):
$ pyastgrep './/Constant[@type="int" or @type="float"]'
tests/examples/test_xml/everything.py:5:20: assigned_int = 123
tests/examples/test_xml/everything.py:6:22: assigned_float = 3.14
Names longer than 42 characters:
$ pyastgrep './/Name[string-length(@id) > 42]'
except clauses that raise a different exception class than they catch:
$ pyastgrep "//ExceptHandler[body//Raise/exc//Name and not(contains(body//Raise/exc//Name/@id, type/Name/@id))]"
Classes whose name matches a regular expression:
$ pyastgrep ".//ClassDef[re:match('M.*', @name)]"
src/pyastgrep/search.py:18:1:class Match:
Tips
To get pyastgrep to print absolute paths in results, pass the current absolute path as the directory to search:
pyastgrep "..." $(pwd)
Limitations
pyastgrep is useful for grepping Python code at a fairly low level. It can be used for various refactoring or linting tasks. Some linting tasks require higher level understanding of a code base. For example, to detect use of a certain function, you need to cope with various ways that the function may be imported and used, and avoid detecting a function with the same name but from a different module. For these kinds of tasks, you might be interested in:
Use as a library
pyastgrep is structured internally to make it easy to use a library as well as a CLI. However, while we will try to break things without good reason, at this point we are not documenting or guaranteeing API stability on these functions.
Contributing
Get test suite running:
pip install -r requirements-test.txt pytest
Run against all versions:
pip install tox tox
Install
Links
Green tree snakes - a very readable overview of Python ASTs.
Python AST Explorer for worked examples of ASTs.
A brief guide to XPath. See also the XPath Axes guide which can be very helpful for matching related AST nodes.
History
This project was forked from https://github.com/hchasestevens/astpath by H. Chase Stevens. Main changes: * Many bugs fixed * Significant rewrite of parts of code * Changes to match grep/ripgrep
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.