drive-ami 0.9.0

Rationale

From a radio astronomy point of view:

This package makes it trivial to script reduction of raw AMI data from python. What’s more, it provides tools to group the raw files into datasets, outputting the UVFITS for each dataset under a single folder. It does this by extracting the pointing information from the raw data, resulting in fairly reliable groupings (although you can edit these manually, see later).

When processing the data, all output from reduce is saved to an accompanying log-file, retaining all information that would normally be available to the user from the interactive interface. Meanwhile, all emulated commands passed to reduce are recorded in a separate log for each file processed, so it’s easy to re-run the script manually and tinker with the reduction process.

Additionally, when running commands listed in a script the interface quietly parses key information such as flagging percentages, rain modulation, and estimated noise, from the reduce output. These are then stored to disk alongside the UVFITs in easily machine readable JSON format. (These may also be added to the UVFITS header in future.)

From a software engineering point of view:

Written in Fortran, with an interactive terminal interface, the reduce pipeline takes a little coaxing to co-operate with calling scripts, making automated processing of large numbers of files tricky. Fortunately, the python library pexpect provides an easy mechanism to emulate human interaction, upon which I’ve built an interface class. Limitations such as a maximum path length of ~32 chars are circumvented with a few careful hacks. The python logging libraries then allow us provide the user with minimal progress information, whilst retaining all possible information for debugging and scientific evaluation.

Installation

Requirements:

• You will need a working installation of AMI-reduce (naturally)

• pexpect For interfacing with AMI-reduce. (Installed automatically as part of the python setup.)

• astropy Used for calculating co-ordinate distances, etc. (Installed automatically as part of the python setup.)

From the command line (preferably within a virtualenv):

git clone git://github.com/timstaley/drive-ami.git
cd drive-ami
pip install numpy #Workaround for buggy scipy/numpy combined install.
pip install .

Usage

Command-line scripts are installed along with the package. Their sourcefiles can be found at https://github.com/timstaley/drive-ami/tree/master/bin. For full details, run e.g.:

driveami_list_rawfiles.py -h

Where -h is short for ‘help’.

Typical usage is to run driveami_list_rawfiles.py to build a full listing of available data, followed by driveami_filter_rawfile_listing.py to extract the entries on a desired target. Finally, driveami_calibrate_rawfiles.py actually does the processing using AMI-REDUCE.