A set of tools to compress gensim fasttext models
Project description
Compress-fasttext
This Python 3 package allows to compress fastText models (from the gensim package) by orders of magnitude,
without seriously affecting their quality.
Use it like this:
import gensim
import compress_fasttext
big_model = gensim.models.fasttext.FastTextKeyedVectors.load('path-to-original-model')
small_model = compress_fasttext.prune_ft_freq(big_model, pq=True)
small_model.save('path-to-new-model')
Different compression methods include:
- matrix decomposition (
svd_ft) - product quantization (
quantize_ft) - optimization of feature hashing (
prune_ft) - feature selection (
prune_ft_freq)
The recommended approach is combination of feature selection and quantization
(the function prune_ft_freq with pq=True).
This code is heavily based on the navec package by Alexander Kukushkin and the blogpost by Andrey Vasnetsov about shrinking fastText embeddings.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
File details
Details for the file compress-fasttext-0.0.1.tar.gz.
File metadata
- Download URL: compress-fasttext-0.0.1.tar.gz
- Upload date:
- Size: 6.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/44.0.0 requests-toolbelt/0.9.1 tqdm/4.40.2 CPython/3.6.8
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 4181db4b38c8510dbc6b303b9dfcffacb46c2e1ba71ddb8ebd5d76c6f1176c42 |
|
| MD5 | 78843a8e55668f46c92b79185edb157d |
|
| BLAKE2b-256 | ab9c2599e00937b162aa599e24946ebbb5a8a79f815c715a1888954ecfec73b6 |
