BioMAJ download service
Project description
About
Microservice to manage the downloads of biomaj.
A protobuf interface is available in biomaj_download/message/message_pb2.py to exchange messages between BioMAJ and the download service. Messages go through RabbitMQ (to be installed).
Protobuf
To compile protobuf, in biomaj_download/message:
protoc --python_out=. downmessage.proto
Development
flake8 biomaj_download/\*.py biomaj_download/download
Test
To run the test suite, use:
nosetests -a '!local_irods' tests/biomaj_tests.py
This command skips the test that need a local iRODS server.
Some test might fail due to network connection. You can skip them with:
nosetests -a '!network' tests/biomaj_tests.py
(To skip the local iRODS test and the network tests, use -a '!network,!local_irods').
Run
Message consumer:
export BIOMAJ_CONFIG=path_to_config.yml
python bin/biomaj_download_consumer.py
Web server
If package is installed via pip, you need a file named gunicorn_conf.py containing somehwhere on local server:
def worker_exit(server, worker):
from prometheus_client import multiprocess
multiprocess.mark_process_dead(worker.pid)
If you cloned the repository and installed it via python setup.py install, just refer to the gunicorn_conf.py in the cloned repository.
export BIOMAJ_CONFIG=path_to_config.yml
rm -rf ..path_to/prometheus-multiproc
mkdir -p ..path_to/prometheus-multiproc
export prometheus_multiproc_dir=..path_to/prometheus-multiproc
gunicorn -c gunicorn_conf.py biomaj_download.biomaj_download_web:app
Web processes should be behind a proxy/load balancer, API base url /api/download
Prometheus endpoint metrics are exposed via /metrics on web server
3.1.1: #17 Support MDTM command in directftp 3.1.0: #16 Don't change name after download in DirectHTTPDownloader PR #7 Refactor downloaders (WARNING breaks API) 3.0.27: Fix previous release broken with a bug in direct protocols 3.0.26: Change default download timeout to 1h #12 Allow FTPS protocol #14 Add mechanism for protocol specific options 3.0.25: Allow to use hardlinks in LocalDownload 3.0.24: Remove debug logs 3.0.23: Support spaces in remote file names 3.0.22: Fix */ remote.files parsing 3.0.21: Fix traefik labels 3.0.20: Update pika dependency release Add tags for traefik support 3.0.19: Check archives after download Fix python regexps syntax (deprecation) 3.0.18: Rename protobuf and use specific package to avoid conflicts 3.0.17: Regenerate protobuf message desc, failing on python3 3.0.16: Add missing req in setup.py 3.0.15: Fix progress download control where could have infinite loop Add irods download
3.0.14: Allow setup of local_endpoint per service, else use default local_endpoint
3.0.13: In rate limiting, add progress vs total of download Fix rate limiting submission
3.0.12: Add retry in case of session creation failure disable web thread logging
3.0.11: Display progress of download by percent of downloads In case of contact error in downloadclient, retry connection
3.0.10: Feature #3: Add rate limiting option to limit number of parallel downloads for a client
3.0.9: Add host in prometheus stats Fix #2: allow setting http.group.file.size or http.group.file.date to -1 if not avalaible in http(s) page for regexp
3.0.8: Fix prometheus stats Add consul supervision
3.0.7: Change size type to int64
3.0.6: Fix download_or_copy to avoid downloading a file existing in a previous production directory
3.0.4: Fixes on messages
3.0.3: Fix management of timeout leading to a crash when using biomaj.download parameter.
3.0.2: set rabbitmq parameter optional
3.0.1: add missing README etc.. in package
3.0.0: move download management out of biomaj main package
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file biomaj_download-3.1.1.tar.gz.
File metadata
- Download URL: biomaj_download-3.1.1.tar.gz
- Upload date:
- Size: 30.0 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.15.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/42.0.1 requests-toolbelt/0.9.1 tqdm/4.39.0 CPython/2.7.15
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 92ada9e1c2c461137853bf36cf86da9af8b24d2d7bbbac40b5df75292a8ce275 |
|
| MD5 | 7dea2ae5a4282c03b54ac589b54c1612 |
|
| BLAKE2b-256 | f604967e264f4356ae6c18da2730feaf1179f8bfd0e0c2b7d8753400e9332b67 |
File details
Details for the file biomaj_download-3.1.1-py2.py3-none-any.whl.
File metadata
- Download URL: biomaj_download-3.1.1-py2.py3-none-any.whl
- Upload date:
- Size: 47.6 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.15.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/42.0.1 requests-toolbelt/0.9.1 tqdm/4.39.0 CPython/2.7.15
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 403e85895c5ce098eb8dbf3a090fcbbce64734bcad47978262b4a42ad0148e15 |
|
| MD5 | 98a43d67534d850460affdc900991285 |
|
| BLAKE2b-256 | b0a4f27ebb70080c2f14a3666cda217d4b2b1a4087ebcf6942c9b800c325db19 |
