A relaxed wrapper for FISS and dalmatian
Project description
Lapdog
A relaxed wrapper for dalmatian and FISS
Prerequisites
- Lapdog requires MacOS or Linux. Windows is not officially supported
- Lapdog requires Python >= 3.3
- Lapdog requires the Google Cloud SDK, which can be installed here.
- Your
gcloud --versionshould be at least252.0.0. If it is not, please rungcloud components update
- Your
- Lapdog uses your Gcloud application-default credentials
- Use
gcloud auth application-default loginandgcloud config set account {account}before running Lapdog
- Use
- Lapdog requires the Alpha and Beta suite for Gcloud
- Alpha and Beta suite can be installed with
gcloud components install alpha beta
- Alpha and Beta suite can be installed with
- (Optional) The Lapdog User Interface requires node and npm, which can be installed using your system's package manager
- MacOS:
brew install node npm - Linux: Install through system package manager or from source
- Lapdog requires
node --version>= 10.15.1 - Lapdog requires
npm --version>= 6.4.1
- MacOS:
Installing
- Install lapdog via pip:
pip install lapdog- If you already have lapdog installed, you can upgrade it with
pip install --upgrade lapdog - If you encounter an error with PyYAML see the PyYaml Note below
- If you already have lapdog installed, you can upgrade it with
- (Optional) Enable the Lapdog User Interface:
- The UI runs locally by default. If you are installing Lapdog on a server, you'll need to set up an SSH tunnel for ports 4200 and 4201
- Install
nodeandnpmif you don't already have them installed- If you're on Mac OS, run
brew install node npm
- If you're on Mac OS, run
- Run
lapdog ui --install. This may take a while
Usage
lapdogmay be imported within python as a drop-in replacement fordalmatian- lapdog presents a superset of features available in dalmatian
WorkspaceManagers in lapdog cache data when communicating with Firecloud. If Firecloud experiences an intermittent failure, theWorkspaceManagermay be able to continue running in offline mode. CallingWorkspaceManager.sync()will reconnect to Firecloud, pushing out any data updates that were queued while in offline modeWorkspaceManagers in lapdog present the execution api viaWorkspaceManager.execute(). Executions differ from submissions in that they run directly on Google and results are uploaded back to Firecloud afterwards
lapdogmay be used as a command line tool.- The tool provides the necessary functions to create a workspace, fill it with data, import or upload methods and configurations, and submit jobs (or execute them directly)
- Run
lapdog --helpto get the list of available commands
lapdogmay be used via an interactive user interface which serves to run and monitor lapdog executions- Run
lapdog uito launch the user interface
- Run
Job Execution
Lapdog executes jobs through dedicated Google Projects ("Engines") for each FireCloud Namespace.
A Lapdog Engine can only be initialized for a given Namespace by a billing account admin.
To initialize a new Engine, contact your Namespace admin and ask them to run lapdog initialize-project.
After an Engine is initialized, you will have to register with it:
- The Lapdog User Interface will automatically register you when you load a workspace in a namespace that you're not registered to
- The Lapdog python module supports manual registration
- When you create a
WorkspaceManagerin an unregistered Namespace, you will get a warning - You can also check your registration status by checking the value of
WorkspaceManager.gateway.registered - You can then register by using
WorkspaceManager.gateway.register() - If registration fails due to any FireCloud errors, simply wait a few minutes
then try calling
register()again
- When you create a
- The Lapdog CLI does not support registration. You can register through the UI or python module
Workspace Permissions
In the UI, at the bottom of every page, you will find a firecloud.org email.
This is a proxy group email which contains you, and all your service accounts.
To allow the Lapdog Engine to run jobs, that proxy group email must be granted
WRITE access to FireCloud workspaces where jobs will run. You may grant the group
READ access to workspaces where data will be read from, but jobs cannot execute
in workspaces without WRITE permissions. The proxy group email can be found by
calling lapdog.cloud.proxy_group_for_user(YOUR_EMAIL).
NOTE: Due to a bug in FireCloud, permissions will not be granted if the group was already granted access to a workspace before you registered to that namespace's Lapdog Engine. If your proxy email definitely was granted access to a workspace, but your jobs are still failing with permissions errors, try removing access and then re-granting it. You can see FireCloud's response to this bug report here
Roadmap
See the milestones page to read the development roadmap.
Pro/Con with Firecloud
Pros
- Each submission has a dedicated Cromwell instance. Your jobs will never queue, unless you hit a Google usage quota
- Lapdog supports Requester Pays buckets and GPUs
- Workspace cache: Lapdog caches most data received from Firecloud.
- In the event of a Firecloud error, Lapdog will attempt to keep running by using it's cached data. Any data updates will by pushed back to Firecloud when the workspace is synced
- Data caches: The Lapdog API caches data sent to the UI and read from Google
- These caches greatly improve UI performance by storing results whenever possible
- Streamlined UI: The Lapdog UI was built with efficiency in mind
- Quality of life features:
- Save time updating methods. Set
methodRepoMethod.methodVersionto "latest" and let Lapdog figure out what the snapshot ID is - Easy data uploads. Call
prepare_entity_dfon a DataFrame before uploading to Firecloud. Any local filepaths will be uploaded to the workspace's bucket in the background and a new DataFrame will be returned containing the newgs://paths - Automatic reference uploads. When you call
update_attributes, any values which refer to local filepaths will be uploaded in the background (just likeprepare_entity_df).update_attributesnow returns a dictionary containing the attributes exactly as uploaded
- Save time updating methods. Set
Cons
- You pay an additional 5c/hour fee for each submission to run the Cromwell server
- Submission results must be manually uploaded to Firecloud by clicking the
Upload Resultsbutton in the UI. - There are small overhead costs billed to the Lapdog Engine for operation. These costs are for calls to the API and for storage of metadata, both of which should be very cheap
PyYAML Note
Often, when installing Lapdog, the installation fails when upgrading PyYAML because it is unable to uninstall the current PyYAML version.
This is because some older versions of PyYAML were distributed through distutils
which prevents packages from being uninstalled. New versions of PyYAML are distributed
through setuptools which can be successfully uninstalled.
To resolve this issue, navigate to your Python site-packages directory. The site-packages
directory usually ends with .../lib/python{version}/site-packages/. The exact location
depends on your platform, environment manager, and python configuration.
- For anaconda, this path is within anaconda's installation folder:
/{path to anaconda}/envs/{environment}/lib/python{version}/site-packages - For virtualenvs, this path is within the environment's installation folder:
/{path to environment}/lib/python{version}/site-packages - For system python on Unix systems:
/usr/local/lib/python{version}/site-packages
An easy way to locate this folder is to open python and then:
import yaml
yaml
This expression will evaluate and print the path to the yaml module, which will be
within your site-packages directory.
After locating the site-packages directory, you must remove the following two directories:
.../site-packages/yaml/.../site-packages/PyYAML*.egg-info
Once removed, you can try the installation command for lapdog again.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file lapdog-0.18.9.tar.gz.
File metadata
- Download URL: lapdog-0.18.9.tar.gz
- Upload date:
- Size: 282.4 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.6.0 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.7.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 7d6ef932dd0e0f45eb065395c5706db0bb8564a9ea3a7c3801749468cf8e72e6 |
|
| MD5 | a030e4701389d834663016aa1dc92e71 |
|
| BLAKE2b-256 | 049fc1032cae830f3929fa1b7322ced9d7a128124c6e73c3dccf240c90c8858d |
File details
Details for the file lapdog-0.18.9-py3-none-any.whl.
File metadata
- Download URL: lapdog-0.18.9-py3-none-any.whl
- Upload date:
- Size: 138.2 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.6.0 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.7.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 4d14f14b7f7922a4878aee7a8d15bc89c5b1ccad342fc0c60fea401ea1cea650 |
|
| MD5 | 7333d52aa57f3f54c8ee3bd5fe7a0d35 |
|
| BLAKE2b-256 | 15ca842ce802ec420bb507d41ae62f31e938130d8581c297522d6f9594dd816f |
