BioMAJ download service
Project description
# About
Microservice to manage the downloads of biomaj.
A protobuf interface is available in biomaj_download/message/message_pb2.py to exchange messages between BioMAJ and the download service.
Messages go through RabbitMQ (to be installed).
# Protobuf
To compile protobuf, in biomaj_download/message:
protoc --python_out=. message.proto
# Development
flake8 biomaj_download/\*.py biomaj_download/download
# Run
## Message consumer:
export BIOMAJ_CONFIG=path_to_config.yml
python bin/biomaj_download_consumer.py
## Web server
If package is installed via pip, you need a file named *gunicorn_conf.py* containing somehwhere on local server:
def worker_exit(server, worker):
from prometheus_client import multiprocess
multiprocess.mark_process_dead(worker.pid)
If you cloned the repository and installed it via python setup.py install, just refer to the *gunicorn_conf.py* in the cloned repository.
export BIOMAJ_CONFIG=path_to_config.yml
rm -rf ..path_to/prometheus-multiproc
mkdir -p ..path_to/prometheus-multiproc
export prometheus_multiproc_dir=..path_to/prometheus-multiproc
gunicorn -c gunicorn_conf.py biomaj_download.biomaj_download_web:app
Web processes should be behind a proxy/load balancer, API base url /api/download
Prometheus endpoint metrics are exposed via /metrics on web server
3.0.24:
Remove debug logs
3.0.23:
Support spaces in remote file names
3.0.22:
Fix **/* remote.files parsing
3.0.21:
Fix traefik labels
3.0.20:
Update pika dependency release
Add tags for traefik support
3.0.19:
Check archives after download
Fix python regexps syntax (deprecation)
3.0.18:
Rename protobuf and use specific package to avoid conflicts
3.0.17:
Regenerate protobuf message desc, failing on python3
3.0.16:
Add missing req in setup.py
3.0.15:
Fix progress download control where could have infinite loop
Add irods download
3.0.14:
Allow setup of local_endpoint per service, else use default local_endpoint
3.0.13:
In rate limiting, add progress vs total of download
Fix rate limiting submission
3.0.12:
Add retry in case of session creation failure
disable web thread logging
3.0.11:
Display progress of download by percent of downloads
In case of contact error in downloadclient, retry connection
3.0.10:
Feature #3: Add rate limiting option to limit number of parallel downloads for a client
3.0.9:
Add host in prometheus stats
Fix #2: allow setting http.group.file.size or http.group.file.date to -1 if not avalaible in http(s) page for regexp
3.0.8:
Fix prometheus stats
Add consul supervision
3.0.7:
Change size type to int64
3.0.6:
Fix download_or_copy to avoid downloading a file existing in a previous production directory
3.0.4:
Fixes on messages
3.0.3:
Fix management of timeout leading to a crash when using biomaj.download parameter.
3.0.2:
set rabbitmq parameter optional
3.0.1:
add missing README etc.. in package
3.0.0:
move download management out of biomaj main package
Microservice to manage the downloads of biomaj.
A protobuf interface is available in biomaj_download/message/message_pb2.py to exchange messages between BioMAJ and the download service.
Messages go through RabbitMQ (to be installed).
# Protobuf
To compile protobuf, in biomaj_download/message:
protoc --python_out=. message.proto
# Development
flake8 biomaj_download/\*.py biomaj_download/download
# Run
## Message consumer:
export BIOMAJ_CONFIG=path_to_config.yml
python bin/biomaj_download_consumer.py
## Web server
If package is installed via pip, you need a file named *gunicorn_conf.py* containing somehwhere on local server:
def worker_exit(server, worker):
from prometheus_client import multiprocess
multiprocess.mark_process_dead(worker.pid)
If you cloned the repository and installed it via python setup.py install, just refer to the *gunicorn_conf.py* in the cloned repository.
export BIOMAJ_CONFIG=path_to_config.yml
rm -rf ..path_to/prometheus-multiproc
mkdir -p ..path_to/prometheus-multiproc
export prometheus_multiproc_dir=..path_to/prometheus-multiproc
gunicorn -c gunicorn_conf.py biomaj_download.biomaj_download_web:app
Web processes should be behind a proxy/load balancer, API base url /api/download
Prometheus endpoint metrics are exposed via /metrics on web server
3.0.24:
Remove debug logs
3.0.23:
Support spaces in remote file names
3.0.22:
Fix **/* remote.files parsing
3.0.21:
Fix traefik labels
3.0.20:
Update pika dependency release
Add tags for traefik support
3.0.19:
Check archives after download
Fix python regexps syntax (deprecation)
3.0.18:
Rename protobuf and use specific package to avoid conflicts
3.0.17:
Regenerate protobuf message desc, failing on python3
3.0.16:
Add missing req in setup.py
3.0.15:
Fix progress download control where could have infinite loop
Add irods download
3.0.14:
Allow setup of local_endpoint per service, else use default local_endpoint
3.0.13:
In rate limiting, add progress vs total of download
Fix rate limiting submission
3.0.12:
Add retry in case of session creation failure
disable web thread logging
3.0.11:
Display progress of download by percent of downloads
In case of contact error in downloadclient, retry connection
3.0.10:
Feature #3: Add rate limiting option to limit number of parallel downloads for a client
3.0.9:
Add host in prometheus stats
Fix #2: allow setting http.group.file.size or http.group.file.date to -1 if not avalaible in http(s) page for regexp
3.0.8:
Fix prometheus stats
Add consul supervision
3.0.7:
Change size type to int64
3.0.6:
Fix download_or_copy to avoid downloading a file existing in a previous production directory
3.0.4:
Fixes on messages
3.0.3:
Fix management of timeout leading to a crash when using biomaj.download parameter.
3.0.2:
set rabbitmq parameter optional
3.0.1:
add missing README etc.. in package
3.0.0:
move download management out of biomaj main package
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
biomaj_download-3.0.24.tar.gz
(26.8 kB
view details)
Built Distribution
File details
Details for the file biomaj_download-3.0.24.tar.gz.
File metadata
- Download URL: biomaj_download-3.0.24.tar.gz
- Upload date:
- Size: 26.8 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.6.3
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | b3d0d8881d3ea573ee3bf434e71e379d92aa79f9ecd7888004a93859a3c0d375 |
|
| MD5 | c08aeffdb104481c16a741b39778a0be |
|
| BLAKE2b-256 | 2e7d506e1219838f54426c5bd3154ca246aa1e4306514573bb4064e6a9b245a1 |
File details
Details for the file biomaj_download-3.0.24-py2.py3-none-any.whl.
File metadata
- Download URL: biomaj_download-3.0.24-py2.py3-none-any.whl
- Upload date:
- Size: 47.0 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.6.3
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 266c692d34647ffaee2ffda87ac003abebdf303adc934e68286dc8bb2297bcf9 |
|
| MD5 | 3aa3752b5cf9c83cc59c36253621d883 |
|
| BLAKE2b-256 | c94a004f4ae827d2ae8a2a12c8d850c847ccd2fbb0e03ab41c9d7179abf6ece0 |
