Front-end for the ServiceX Data Server
Project description
ServiceX_frontend
Client access library for ServiceX
Introduction
Given you have a selection string, this library will manage submitting it to a ServiceX instance and retreiving the data locally for you. The selection string is often generated by another front-end library, for example:
- func_adl.xAOD (for ATLAS xAOD's)
- func_adl.XXX (for flat ntuples)
- xxx for columns
These libraries are just coming up now, so this list is just an outline.
Prerequisites
Before you install this library you'll need:
- An environment based on python 3.7 or later
- A ServiceX end-point. For example,
http://localhost:5000/servicex.
Usage
The following lines will return a pandas.DataFrame containing all the jet pT's from an ATLAS xAOD file containing Z->ee Monte Carlo:
import servicex
query = "(call ResultTTree (call Select (call SelectMany (call EventDataset (list 'localds:bogus')) (lambda (list e) (call (attr e 'Jets') 'AntiKt4EMTopoJets'))) (lambda (list j) (/ (call (attr j 'pt')) 1000.0))) (list 'JetPt') 'analysis' 'junk.root')"
dataset = "mc15_13TeV:mc15_13TeV.361106.PowhegPythia8EvtGen_AZNLOCTEQ6L1_Zee.merge.DAOD_STDM3.e3601_s2576_s2132_r6630_r6264_p2363_tid05630052_00"
r = servicex.get_data(query , dataset, servicex_endpoint=endpoint)
print(r)
And the output in a terminal window from running the above script (takes about 1-2 minutes to complete):
python scripts\run_test.py http://localhost:5000/servicex
JetPt
entry
0 38.065707
1 31.967096
2 7.881337
3 6.669581
4 5.624053
... ...
710183 42.926141
710184 30.815709
710185 6.348002
710186 5.472711
710187 5.212714
[11355980 rows x 1 columns]
If your query is badly formed or there is an other problem with the backend, an exception will be thrown.
If you'd like to be able to submit multiple queries and have them run on the ServiceX back end in parallel, it may be best to use the asyncio interface, which has the identical signature, but is called get_data_async.
Features
Implemented:
- Accepts a
qastleformatted query - Exceptions are used to report back errors of all sorts from the service to the user's code.
- Data is return as a
pandas.DataFrameor aawkwardarray (see thedata_typeparameter) - Complete returned data must fit in the process' memory
- Run in an async or a non-async environment and non-async methods will accomodate automatically (including
jupyternotebooks). - Support up to 100 simultanious queries from a laptop-like front end without overwhelming the local machine (hopefully ServiceX will be overwhelmed!)
- Start downloading files as soon as they are ready (before ServiceX is done with the complete transform).
Comming:
- Data is returned as a list of ROOT files located in a specified directory
- Make it easy to submit the same query for 100 different datasets
Testing
This code has been tested in several environments:
- Windows, Linux, MacOS
- Python 3.6, 3.7, 3.8
- 3.8.0 and 3.8.1 only. Unfortunately, 3.8.2 has caused
nest_asyncioto fail. Until that package is updated we are stuck at 3.8.1.
- 3.8.0 and 3.8.1 only. Unfortunately, 3.8.2 has caused
- Jupyter Notebooks (not automated), regular python command-line invoked source files
Development
For any changes please feel free to submit pull requests!
To do development please setup your environment with the following steps:
- A python 3.7 development environment
- Pull down this package, XX
python -m pip install -e .[test]- Run the tests to make sure everything is good:
pytest.
Then add tests as you develop. When you are done, submit a pull request with any required changes to the documentation and the online tests will run.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file servicex-1.0.0b3.tar.gz.
File metadata
- Download URL: servicex-1.0.0b3.tar.gz
- Upload date:
- Size: 7.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.42.1 CPython/3.7.7
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 220e237785d03e926ea9d739edda59a0ded40b502a228112476865afa33d4e5d |
|
| MD5 | 546c5c58b3566e50cb6405250492ef71 |
|
| BLAKE2b-256 | ddb4f542b66df192a7fe22363e9cad17d4f1194b941dcf82cfa6d80d1a015aeb |
File details
Details for the file servicex-1.0.0b3-py3-none-any.whl.
File metadata
- Download URL: servicex-1.0.0b3-py3-none-any.whl
- Upload date:
- Size: 7.2 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.42.1 CPython/3.7.7
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 466862008767a3ffd5028167d743ca5688762fa39ee38bed2f90b0747c96604d |
|
| MD5 | c24c8b61830dbc8c4dc1e3effeb65b85 |
|
| BLAKE2b-256 | 422628e8051746562d156731c472511cdf00d22871914ffcd4f7181282c9fa66 |
