BioMAJ download service
Project description
# About
Microservice to manage the downloads of biomaj.
A protobuf interface is available in biomaj_download/message/message_pb2.py to exchange messages between BioMAJ and the download service. Messages go through RabbitMQ (to be installed).
# Protobuf
To compile protobuf, in biomaj_download/message:
protoc –python_out=. message.proto
# Development
flake8 biomaj_download/*.py biomaj_download/download
# Run
## Message consumer:
export BIOMAJ_CONFIG=path_to_config.yml python bin/biomaj_download_consumer.py
## Web server
If package is installed via pip, you need a file named gunicorn_conf.py containing somehwhere on local server:
- def worker_exit(server, worker):
from prometheus_client import multiprocess multiprocess.mark_process_dead(worker.pid)
If you cloned the repository and installed it via python setup.py install, just refer to the gunicorn_conf.py in the cloned repository.
export BIOMAJ_CONFIG=path_to_config.yml rm -rf ..path_to/prometheus-multiproc mkdir -p ..path_to/prometheus-multiproc export prometheus_multiproc_dir=..path_to/prometheus-multiproc gunicorn -c gunicorn_conf.py biomaj_download.biomaj_download_web:app
Web processes should be behind a proxy/load balancer, API base url /api/download
Prometheus endpoint metrics are exposed via /metrics on web server
- 3.0.21:
Fix traefik labels
- 3.0.20:
Update pika dependency release Add tags for traefik support
- 3.0.19:
Check archives after download Fix python regexps syntax (deprecation)
- 3.0.18:
Rename protobuf and use specific package to avoid conflicts
- 3.0.17:
Regenerate protobuf message desc, failing on python3
- 3.0.16:
Add missing req in setup.py
- 3.0.15:
Fix progress download control where could have infinite loop Add irods download
- 3.0.14:
Allow setup of local_endpoint per service, else use default local_endpoint
- 3.0.13:
In rate limiting, add progress vs total of download Fix rate limiting submission
- 3.0.12:
Add retry in case of session creation failure disable web thread logging
- 3.0.11:
Display progress of download by percent of downloads In case of contact error in downloadclient, retry connection
- 3.0.10:
Feature #3: Add rate limiting option to limit number of parallel downloads for a client
- 3.0.9:
Add host in prometheus stats Fix #2: allow setting http.group.file.size or http.group.file.date to -1 if not avalaible in http(s) page for regexp
- 3.0.8:
Fix prometheus stats Add consul supervision
- 3.0.7:
Change size type to int64
- 3.0.6:
Fix download_or_copy to avoid downloading a file existing in a previous production directory
- 3.0.4:
Fixes on messages
- 3.0.3:
Fix management of timeout leading to a crash when using biomaj.download parameter.
- 3.0.2:
set rabbitmq parameter optional
- 3.0.1:
add missing README etc.. in package
- 3.0.0:
move download management out of biomaj main package
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file biomaj_download-3.0.21.tar.gz.
File metadata
- Download URL: biomaj_download-3.0.21.tar.gz
- Upload date:
- Size: 26.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.4.6
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | d55a30bae9ea0b3f5d4cb192b6d6610fa75e7fad75b1658cd06166b8c45429d9 |
|
| MD5 | ed6823f37699f54f085c7ce84d906881 |
|
| BLAKE2b-256 | 3b08a90355104d88dd1ddcf6636d3c891dc4eafa6b4c685989bdd8bbe7ddf0e7 |
File details
Details for the file biomaj_download-3.0.21-py2.py3-none-any.whl.
File metadata
- Download URL: biomaj_download-3.0.21-py2.py3-none-any.whl
- Upload date:
- Size: 46.9 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.4.6
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 8a0bb85b8cd0d7e96569e8028c1f8a3bc330d2252c2f8e39f3da208cddf44cc2 |
|
| MD5 | 5c2a5e8ff25e3cd35e60a88d673ac79c |
|
| BLAKE2b-256 | 83affeafbcfab5be5191e55cab2cba238e519d6c4b3af7f9c2e4a93b4baac353 |
