Functional Analysis Description Language - Backend AST Manipulation Packages
Project description
func_adl
Construct hierarchical data queries using SQL-like concepts in python.
func_adl Uses an SQL like language, and extracts data and computed values from a ROOT file or an ATLAS xAOD file
and returns them in a columnar format. It is currently used as a central part of two of the ServiceX transformers.
This is the base package that has the backend-agnostic code to query hierarchical data. In all likelihood you will want to install one of the following packages:
- func_adl_xAOD: for running on an ATLAS & CMS experiment xAOD file hosted in ServiceX
- func_adl_uproot: for running on flat root files
- func_adl.xAOD.backend: for running on a local file using docker
See the documentation for more information on what expressions and capabilities are possible in each of these backends.
Extensibility
There are two several extensibility points:
EventDatasetshould be sub-classed to provide an executor.EventDatasetcan use Python's type system to allow for editors and other intelligent typing systems to type check expressions. The more type data present, the more the system can help.- It is possible to insert a call back at a function or method call site that will allow for modification of the
ObjectStreamor the call site'sast.
EventDataSet
An example EventDataSet:
class events(EventDataset):
async def execute_result_async(self, a: ast.AST, title: Optional[str] = None):
await asyncio.sleep(0.01)
return a
and some func_adl code that uses it:
r = (events()
.SelectMany(lambda e: e.Jets('jets'))
.Select(lambda j: j.eta())
.value())
- When the
.value()method is invoked, theexecute_result_asyncwith a completeastrepresenting the query is called. This is the point that one would send it to the backend to actually be processed. - Normally, the constructor of
eventswould take in the name of the dataset to be processed, which could then be used inexecute_result_async.
Typing EventDataset
A minor change to the declaration above, and no change to the query:
class dd_jet:
def pt(self) -> float:
...
def eta(self) -> float:
...
class dd_event:
def Jets(self, bank: str) -> Iterable[dd_jet]:
...
def EventNumber(self, bank='default') -> int
...
class events(EventDataset[dd_event]):
async def execute_result_async(self, a: ast.AST, title: Optional[str] = None):
await asyncio.sleep(0.01)
return a
This is not required, but when this is done:
- Editors that use types to give one a list of options/guesses will now light up as long as they have reasonable type-checking built in.
- If a required argument is missed, an error will be generated
- If a default argument is missed, it will be automatically filled in.
It should be noted that the type and expression follower is not very sophisticated! While it can follow method calls, it won't follow much else!
Type-based callbacks
By adding a function and a reference in the type system, arbitrary code can be executed during the traversing of the func_adl. Keeping the query the same and the events definition the same, we can add the info directly to the python type declarations:
def add_md_for_type(s: ObjectStream[T], a: ast.Call) -> Tuple[ObjectStream[T], ast.AST]:
return s.MetaData({'hi': 'there'}), a
class dd_event:
_func_adl_type_info = add_md_for_type
def Jets(self, bank: str) -> Iterable[dd_jet]:
...
- When the
.Jets()method is processed, theadd_md_for_typeis called with the current object stream and the ast. add_md_for_typehere adds metadata and returns the updated stream and ast.- Nothing prevents the function from parsing the AST, removing or adding arguments, adding more complex metadata, or doing any of this depending on the arguments in the call site.
Development
After a new release has been built and passes the tests you can release it by creating a new release on github. An action that runs when a release is "created" will send it to pypi.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file func_adl.ast-2.4b4.tar.gz.
File metadata
- Download URL: func_adl.ast-2.4b4.tar.gz
- Upload date:
- Size: 13.0 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | a8ec21395be42fd44f90fc0123ac47a309fe7effc674495f25eb893b72cf2a44 |
|
| MD5 | 51fd3c85f20ed76f06f4136f719a6a78 |
|
| BLAKE2b-256 | 0a14fc0e4f66bfc148a979b7cb3ae586bf5b9d945ed5c1e6506718bbe60186fe |
File details
Details for the file func_adl.ast-2.4b4-py3-none-any.whl.
File metadata
- Download URL: func_adl.ast-2.4b4-py3-none-any.whl
- Upload date:
- Size: 25.4 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | a33a69252a9b77667793ac81625bdcccbd95431da3bf769cd022bbe12f923ed5 |
|
| MD5 | 238af0050a23dd47e51ca12cd904f697 |
|
| BLAKE2b-256 | 5fe07c755653c2c4f3f49b56700a6589d1a5bb6c304a8120fd16c75550d73999 |
