The ATLAS Metadata Interface
Project description
For more information see the full pyAMI documentation
pyAMI 4 is a major upgrade of the AMI python client. The command line syntax has been rationalized, with improved help functions. pyAMI is available in the ATLAS software release and also installed centrally on lxplus. pyAMI can be installed standalone on a laptop. Instructions are given below for linux and windows.
All commands are simply subcommands of ami.
List AOD datasets matching a pattern:
ami list datasets --type AOD data11_7TeV
list files in a dataset:
ami list files -lhc dataset.name
list latest NTUP_TAUMEDIUM data datasets (and number of events in each one) originating from AOD in periods L1 and L2 that are contained in a good runs list (GRL):
ami list data --project data11_7TeV --type NTUP_TAUMEDIUM --parent-type AOD \ --periods L1,L2 --stream physics_JetTauEtmiss --latest --fields events \ --grl mygrl.xml
list runs in a period or multiple periods:
ami list runs B M
display dataset metadata:
ami dataset info dataset.name
and query projects, data types, dataset provenance, etc. pyAMI also provides an API allowing you to perform all of the same queries from within your own Python program.
LXPLUS and CVMFS
pyAMI is installed centrally on LXPLUS at CERN. To begin using pyAMI simply:
source /afs/cern.ch/atlas/software/tools/atlasmeta/setup.sh
then authenticate yourself if you haven’t already (see below).
pyAMI is also available through CVMFS:
source /cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/user/atlasLocalSetup.sh localSetupPyAMI
Installation
pyAMI requires at least Python 2.4. lxml is optional but required for XSL transformations. libxml2 and libxslt (dev packages) must be installed to build lxml.
Automatic Installation
Automatically install the latest version of pyAMI with pip:
pip install --user pyAMI
or with easy_install:
easy_install --user pyAMI
Omit the --user for a system-wide installation (requires root privileges). Add ${HOME}/.local/bin to your ${PATH} if using --user and if it is not there already (put this in your .bashrc):
export PATH=${HOME}/.local/bin${PATH:+:$PATH}
To upgrade an existing installation use the -U option in the pip or easy_install commands above.
Manual Installation
Get the latest tarball on PyPI
Untar and install (replace X appropriately):
tar -zxvf pyAMI-X.tar.gz cd pyAMI-X
pyAMI uses distribute to install but you may revert to a basic distutils install by setting the environment variable:
export PYAMI_NO_DISTRIBUTE=1
One advantage of using distribute is that all dependencies are automatically downloaded and installed for you. To install pyAMI into your home directory if using at least Python 2.6:
python setup.py install --user
or with older Python versions:
python setup.py install --prefix=~/.local
Add ${HOME}/.local/bin to your ${PATH} if it is not there already (put this in your .bashrc):
export PATH=${HOME}/.local/bin${PATH:+:$PATH}
If you are unable to satisfy the dependency on lxml (only used for XSL transformations) then you may disable this dependency before installation with:
export PYAMI_NO_LXML=1
Installation on Windows
Install python, for example Python 2.7.3
Install distribute (for easy installation of dependences). For 64 bit machines there may be problems, see bugs.python.org/issue6792.
Download pip-1.1.tar.gz.
Install pip. Open a “cmd” terminal in windows. Assuming that you installed pip below Python:
cd C:\Python27\pip-1.1 C:\Python27\python setup.py install
Install pyAMI. pip.exe should be in C:\Python27\Scripts:
cd C:\Python27\Scripts pip install pyAMI
Install lxml to enable XSLT. If you don’t install lxml, pyAMI will still work, but you will only be able to obtain XML output on the command line. However if you only want to use the API of pyAMI you may skip this step. (lxml 3.0.1 seems to install with pip automatically):
easy_install --allow-hosts=lxml.de,*.python.org lxml==2.2.8
Now so that you can use pyAMI conveniently you must adjust your paths in the Windows environment. If you do not know how to do this follow the instructions here www.java.com/en/download/help/path.xml. Add to the path:
PATH C:\PYTHON27;C:\PYTHON27\Scripts
Lastly explain to Windows that a python script can be executed. Add to the PATHEXT variable:
PATHEXT .PY
Then change the name of the file ami in C:\Python27\Scripts to ami.py so that script ami.py can be executed just by typing ami.
Authentication
VOMS authentication is supported:
voms-proxy-init -voms atlas
The alternative option (if gLite is not available) is to send your username and password along with each AMI command:
ami auth
This will prompt you for your AMI username and password. You will only need to do this once since your credentials are stored in ~/.pyami/ami.conf for later use. If your credentials change just run ami auth again. If you would like to keep your pyAMI configuration in a directory other than ~/.pyami then set the environment variable PYAMI_CONFIG_DIR.
Version 4.1.1 (13/2/2013)
Upgrade to buildout 2.0.0
Fix a few import issues in pyAMI.extern.ZSI
Version 4.1.0 (4/2/2013)
A patched version of ZSI is now included in pyAMI.extern that removes dependency on PyXML.
argparse is now included in pyAMI.extern.
A minimal pyAMI installation (no lxml) now does not depend on any external non-standard library packages.
Added the -F and –from-file to the “ami list datasets” command where the pattern is interpreted as a file name which will be read for patterns (one per line, ignoring blank lines or lines beginning with #).
Added the option to perform literal matches with “ami list datasets” using the new optional flags -L or –literal-match. In this case the pattern is interpreted as a literal string and the results must match exactly. A literal match query is much faster than a patterned match and can greatly speed up commands where you want to query information for known elements. This option pairs nicely with the new –from-file feature mentioned above for files listing many complete dataset names that should be interpreted literally and not as patterns.
AMI errors are now printed correctly.
Version 4.0.4 (6/11/2012)
Fix globbing issue. Only surround pattern with % if the pattern didn’t have % on both ends originally. (it needed more fixing!)
Version 4.0.3 (11/10/2012)
Skip version 4.0.2 to sync with ATLAS SVN tag version
Fix setup.py for py2.4 compatibility
Make ami-status=VALID default
Fix globbing issue. Only surround pattern with % if the pattern didn’t have % on both ends originally.
Version 4.0.1 (20/9/2012)
This is pyAMI-04-00-01 in the ATLAS release
Minor syntax changes in the API
Added Windows installation to the doc
Expanded doc section on API
Version 4.0.0a3 (17/4/2012)
Make distribute the default install method
Include patched urllib2 module in pyAMI.backports
Version 4.0.0a2 (10/4/2012)
Second prerelease of the new pyAMI
Support for connection through a proxy (BNL)
Other minor improvements
Version 4.0.0a1 (28/3/2012)
Prerelease of the new pyAMI
Package renamed to pyAMI
Speed improvements
VOMS authentication
Fixed permissions of ~/.pyami/ami.cfg
Removed intermediate ami package Everything is now under pyAMI
atlasmeta versions below:
Version 0.4.0 (4/2/2012)
Now compatible with Python 2.4
Install with distutils by default and make installation with distribute optional
Improved parent-type feature in dataset query
Ability to specify a good runs list (GRL) in ‘ami list data’ and only the runs contained in the GRL are shown
Installed on LXPLUS at CERN
XSL transformations reimplemented
Version 0.3.1 (10/1/2012)
Fix bug in file listing where size/events is NULL in database
Version 0.3 (20/12/2011)
Add ability to query datasets by parent type with –parent-type option
list files in dataset with ‘ami list files’ and optionally display total size and events
new schema module replicating AMI schema
query files, projects, subprojects, datatypes, subdatatypes, tags, nomenclatures, production steps
Version 0.2 (17/12/2011)
Query and display additional dataset fields (i.e. datatype, number of events)
Version 0.1 (1/12/2011)
Forked pyAMI
Major code rewrite
better command-line interface
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
File details
Details for the file pyAMI-4.1.1.tar.gz
.
File metadata
- Download URL: pyAMI-4.1.1.tar.gz
- Upload date:
- Size: 233.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | b5374d0d72897b2f330fc56e71beee53d27813513b0d4d7f958f31aab69b56c4 |
|
MD5 | b8b8a11becdc9944c5f6eb830d5bc665 |
|
BLAKE2b-256 | b57e71330f98b0658d99152d2dda88012f7f2cfc78297029625e5675486c5cd2 |