Skip to main content

Secure, cache-friendly access to large binary blobs for builds and tests

Project description

# Tooltool

This is tooltool. Tooltool is a program that helps make downloading large binaries easier in a CI environment. The program creates a json based manifest that is small compared to the binaries. That manifest is transmitted to the machine that needs the binary somehow (checked in, included in tarball, etc) where the machine will run tooltool to download.

When using the fetch mode, the program will check to see if the file exists locally. If this file does not exist locally the program will try to fetch from one of the base URLs provided. The API that tooltool uses to fetch files is exceedingly simple. the API is that each file request will look for an http resource that is a combination of an arbitrary base url, a directory that is named as the hashing algorithm used and the hashing results of each file stored.

Example, using base url of “http://localhost:8080/tooltool”, algorithm of “sha512” and a file that hashes to “abcedf0123456789”, tooltool would look for the file at “http://localhost:8080/tooltool/sha512/abcdef0123456789”. If there is a local file that has the filename specified in the manifest already, tooltool will not overwrite by default. In this case, tooltool will exit with a non-0 exit value. If overwrite mode is enabled, tooltool will overwrite the local file with the file specified in the manifest.

## Structure

This repository contains both the tooltool client – tooltool.py – and the tooltool server component, which is a [RelengAPI](https://github.com/mozilla/build-relengapi) blueprint.

If you want to use the client, just copy out tooltool.py – it has no dependencies.

## Development

Hack on tooltool as you would any RelengAPI blueprint. That means, roughly, creating a virtualenv and running pip install -e .[test] to install the blueprint and all of its dependences, including those for running tests.

Send pull requests through GitHub.

Both the client and the server components are covered by Travis, via the validate.sh script which you can run yourself.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

relengapi-tooltool-1.0.0.tar.gz (128.6 kB view details)

Uploaded Source

File details

Details for the file relengapi-tooltool-1.0.0.tar.gz.

File metadata

File hashes

Hashes for relengapi-tooltool-1.0.0.tar.gz
Algorithm Hash digest
SHA256 933036031bd3fe879e5abeaae666348eb99d0eb09c380db46032d7de4dd221a8
MD5 c1b5b5f7aa9b2931a4b85696f422f1e4
BLAKE2b-256 5bbef65236b093ce647bcf0c0553bfbbae4ac5f2aaad7acb1d94189bcd838774

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page