Skip to main content

Downloads and caches files for knowledge graph ETL

Project description

KG-Hub Downloader

| Documentation | Repository | PyPI |

Overview

This is a configuration based file caching downloader with initial support for http requests & queries against elasticsearch.

Installation

KGHub Downloader is available to install via pip:

pip install kghub-downloader

Configure

The downloader requires a YAML file which contains a list of target URLs to download, and local names to save those downloads.
For an example, see example/download.yaml

Available options are:

  • *url: The URL to download from. Currently supported:

    • http(s)
    • ftp
      • with glob: option to download files with specific extensions (only with ftp as of now and looks recursively).
    • Google Cloud Storage (gs://)
    • Google Drive (gdrive:// or https://drive.google.com/...). The file must be publicly accessible.
    • Amazon AWS S3 bucket (s3://)
    • GitHub Release Assets (git://RepositoryOwner/RepositoryName)

    If the URL includes a name in {CURLY_BRACES}, it will be expanded from environment variables.

  • glob: An optional glob pattern to limit downloading files (FTP only)

  • local_name: The name to save the file as locally

  • tag: A tag to use to filter downloads

  • api: The API to use to download the file. Currently supported: elasticsearch

  • elastic search options

    • query_file: The file containing the query to run against the index
    • index: The elastic search index for query

* Note:
Google Cloud Storage URLs require that you have set up your credentials as described here. You must:

Mirorring local files to Amazon AWS S3 bucket requires the following:

You can also include any secrets like API keys you have set as environment variables using {VARIABLE_NAME}, for example:

---
- url: "https://example.com/myfancyfile.json?key={YOUR_SECRET}"
  localname: myfancyfile.json

Note: YOUR_SECRET MUST as an environment variable, and be sure to include the {curly braces} in the url string.

Usage

Downloader can be used directly in Python or via command line

In Python

from kghub_downloader.download_utils import download_from_yaml

download_from_yaml(yaml_file="download.yaml", output_dir="data")

Command Line

To download files listed in a download.yaml file:

$ downloader [OPTIONS] ARGS
OPTIONS
yaml_file Path to the download.yaml file, to be parsed for things to download.
Defaults to ./download.yaml
ignore_cache Ignore cache and download files even if they exist (Default False)
snippet_only Downloads only the first 5 kB of each uncompressed source, for testing and file checks
tags Limit to only downloads with this tag
mirror Remote storage URL to upload downloaded files to.
Supported buckets: Google Cloud Storage
ARGUMENTS
output_dir Where to save downloaded files.

Examples:

$ downloader --output_dir example_output --tags zfin_gene_to_phenotype example.yaml
$ downloader --output_dir example_output --mirror gs://your-bucket/desired/directory

# Note that if your YAML file is named `download.yaml`,
# the argument can be omitted from the CLI call.
$ downloader --output_dir example_output

Development

Install

git clone https://github.com/monarch-initiative/kghub-downloader.git
cd kghub-downloader
poetry install

Run tests

poetry run pytest

NOTE: The tests require gcloud credentials to be set up as described above, using the Monarch github actions service account.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

kghub_downloader-0.3.11.tar.gz (11.4 kB view details)

Uploaded Source

Built Distribution

kghub_downloader-0.3.11-py3-none-any.whl (12.3 kB view details)

Uploaded Python 3

File details

Details for the file kghub_downloader-0.3.11.tar.gz.

File metadata

  • Download URL: kghub_downloader-0.3.11.tar.gz
  • Upload date:
  • Size: 11.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.8.20

File hashes

Hashes for kghub_downloader-0.3.11.tar.gz
Algorithm Hash digest
SHA256 db022532ad99d599036d62793d1fd6c3592182bfdfcf3795628272397eee5967
MD5 3a6238b2a80a9e945df2edded467894f
BLAKE2b-256 9c58c7da77829e3b27292d3bead2e625abd66284cecc60b6465fa3f635ed656d

See more details on using hashes here.

File details

Details for the file kghub_downloader-0.3.11-py3-none-any.whl.

File metadata

File hashes

Hashes for kghub_downloader-0.3.11-py3-none-any.whl
Algorithm Hash digest
SHA256 21777a901ce9479f73732e3c624ab2e6d5f9f915b33dfa259abcdda09b1b8cfc
MD5 8b33bf39c5280dbe127922632d2fccec
BLAKE2b-256 a0bc4151dc73a9b90f0071176740fb8312ec7bb814165246af158cbff20ea3f4

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page