Client library to download and publish models on the huggingface.co hub
Project description
huggingface_hub
Client library to download and publish models and other files on the huggingface.co hub
Do you have an open source ML library? We're looking to partner with a small number of other cool open source ML libraries to provide model hosting + versioning. https://twitter.com/julien_c/status/1336374565157679104 https://twitter.com/mnlpariente/status/1336277058062852096
Advantages are:
- versioning is built-in (as hosting is built around git and git-lfs), no lock-in, you can just
git cloneaway.- anyone can upload a new model for your library, just need to add the corresponding tag for the model to be discoverable – no more need for a hardcoded list in your code
- Fast downloads! We use Cloudfront (a CDN) to geo-replicate downloads so they're blazing fast from anywhere on the globe
- Usage stats and more features to come
Ping us if interested 😎
♻️ Partial list of implementations in third party libraries:
- http://github.com/asteroid-team/asteroid [initial PR 👀]
- https://github.com/pyannote/pyannote-audio [initial PR 👀]
- https://github.com/flairNLP/flair [work-in-progress, initial PR 👀]
- https://github.com/espnet/espnet [initial PR 👀]
Download files from the huggingface.co hub
Integration inside a library is super simple. We expose two functions, hf_hub_url() and cached_download().
hf_hub_url
hf_hub_url() takes:
- a repo id (e.g. a model id like
julien-c/EsperBERTo-smalli.e. a user or organization name and a repo name, separated by/), - a filename (like
pytorch_model.bin), - and an optional git revision id (can be a branch name, a tag, or a commit hash)
and returns the url we'll use to download the actual files: https://huggingface.co/julien-c/EsperBERTo-small/resolve/main/pytorch_model.bin
If you check out this URL's headers with a HEAD http request (which you can do from the command line with curl -I) for a few different files, you'll see that:
- small files are returned directly
- large files (i.e. the ones stored through git-lfs) are returned via a redirect to a Cloudfront URL. Cloudfront is a Content Delivery Network, or CDN, that ensures that downloads are as fast as possible from anywhere on the globe.
cached_download
cached_download() takes the following parameters, downloads the remote file, stores it to disk (in a versioning-aware way) and returns its local file path.
Parameters:
- a remote
url - your library's name and version (
library_nameandlibrary_version), which will be added to the HTTP requests' user-agent so that we can provide some usage stats. - a
cache_dirwhich you can specify if you want to control where on disk the files are cached.
Check out the source code for all possible params (we'll create a real doc page in the future).
Publish models to the huggingface.co hub
Uploading a model to the hub is super simple too:
- create a model repo directly from the website, at huggingface.co/new (models can be public or private, and are namespaced under either a user or an organization)
- clone it with git
- download and install git lfs if you don't already have it on your machine (you can check by running a simple
git lfs) - add, commit and push your files, from git, as you usually do.
We are intentionally not wrapping git too much, so that you can go on with the workflow you’re used to and the tools you already know.
👀 To see an example of how we document the model sharing process in
transformers, check out https://huggingface.co/transformers/model_sharing.html
Users add tags into their README.md model cards (e.g. your library_name, a domain tag like audio, etc.) to make sure their models are discoverable.
Documentation about the model hub itself is at https://huggingface.co/docs
API utilities in hf_api.py
You don't need them for the standard publishing workflow, however, if you need a programmatic way of creating a repo, deleting it (⚠️ caution), or listing models from the hub, you'll find helpers in hf_api.py.
We also have an API to query models by specific tags (e.g. if you want to list models compatible to your library)
huggingface-cli
Those API utilities are also exposed through a CLI:
huggingface-cli login
huggingface-cli logout
huggingface-cli whoami
huggingface-cli repo create
Need to upload large (>5GB) files?
To upload large files (>5GB 🔥), you need to install the custom transfer agent for git-lfs, bundled in this package.
To install, just run:
$ huggingface-cli lfs-enable-largefiles
This should be executed once for each model repo that contains a model file >5GB. If you just try to push a file bigger than 5GB without running that command, you will get an error with a message reminding you to run it.
Finally, there's a huggingface-cli lfs-multipart-upload command but that one is internal (called by lfs directly) and is not meant to be called by the user.
Visual integration into the huggingface.co hub
Finally, we'll implement a few tweaks to improve the UX for your models on the website – let's use Asteroid as an example:
Model authors add an asteroid tag to their model card and they get the advantages of model versioning built-in
We add a custom "Use in Asteroid" button.
When clicked you get a library-specific code sample that you'll be able to specify. 🔥
Feedback (feature requests, bugs, etc.) is super welcome 💙💚💛💜♥️🧡
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file huggingface_hub-0.0.3rc1.tar.gz.
File metadata
- Download URL: huggingface_hub-0.0.3rc1.tar.gz
- Upload date:
- Size: 22.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.57.0 CPython/3.9.1
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | aee5605787021df80eb8cd2d2954a59d7f6c58950968e3f9b7ca39cfa23deef2 |
|
| MD5 | ac0ce69a24ee5edb12b5f4120cf5942e |
|
| BLAKE2b-256 | bf193e2bbd312942ccc40442c6e4a01952a9eb89dfaf59deccfb3d315744302c |
File details
Details for the file huggingface_hub-0.0.3rc1-py3-none-any.whl.
File metadata
- Download URL: huggingface_hub-0.0.3rc1-py3-none-any.whl
- Upload date:
- Size: 27.7 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.57.0 CPython/3.9.1
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 91c3c56266ab8c016205ed2e6d94b638e705ef03e795fbe399e45b2b74c7f65a |
|
| MD5 | 531b3c9e2fce9e082a7f9de6ea92cd95 |
|
| BLAKE2b-256 | 0cef3709ffcc2b7f873b1ef23009e6ce4002f6f841af047575e927d42524cce9 |
