Text utilities and datasets for PyTorch
Project description
torchtext
This repository consists of:
torchtext.datasets: The raw text iterators for common NLP datasets
torchtext.data: Some basic NLP building blocks (tokenizers, metrics, functionals etc.)
torchtext.nn: NLP related modules
torchtext.vocab: Vocab and Vectors related classes and factory functions
examples: Example NLP workflows with PyTorch and torchtext library.
Note: The legacy code discussed in torchtext v0.7.0 release note has been retired to torchtext.legacy folder. Those legacy code will not be maintained by the development team, and we plan to fully remove them in the future release. See torchtext.legacy folder for more details.
Installation
We recommend Anaconda as a Python package management system. Please refer to pytorch.org for the details of PyTorch installation. The following are the corresponding torchtext versions and supported Python versions.
PyTorch version |
torchtext version |
Supported Python version |
---|---|---|
nightly build |
main |
>=3.7, <=3.9 |
1.10.0 |
0.11.0 |
>=3.6, <=3.9 |
1.9.1 |
0.10.1 |
>=3.6, <=3.9 |
1.9 |
0.10 |
>=3.6, <=3.9 |
1.8.2 |
0.9.2 |
>=3.6, <=3.9 |
1.8.1 |
0.9.1 |
>=3.6, <=3.9 |
1.8 |
0.9 |
>=3.6, <=3.9 |
1.7.1 |
0.8.1 |
>=3.6, <=3.9 |
1.7 |
0.8 |
>=3.6, <=3.8 |
1.6 |
0.7 |
>=3.6, <=3.8 |
1.5 |
0.6 |
>=3.5, <=3.8 |
1.4 |
0.5 |
2.7, >=3.5, <=3.8 |
0.4 and below |
0.2.3 |
2.7, >=3.5, <=3.8 |
Using conda:
conda install -c pytorch torchtext
Using pip:
pip install torchtext
Optional requirements
If you want to use English tokenizer from SpaCy, you need to install SpaCy and download its English model:
pip install spacy python -m spacy download en_core_web_sm
Alternatively, you might want to use the Moses tokenizer port in SacreMoses (split from NLTK). You have to install SacreMoses:
pip install sacremoses
For torchtext 0.5 and below, sentencepiece:
conda install -c powerai sentencepiece
Building from source
To build torchtext from source, you need git, CMake and C++11 compiler such as g++.:
git clone https://github.com/pytorch/text torchtext cd torchtext git submodule update --init --recursive # Linux python setup.py clean install # OSX MACOSX_DEPLOYMENT_TARGET=10.9 CC=clang CXX=clang++ python setup.py clean install # or ``python setup.py develop`` if you are making modifications.
Note
When building from source, make sure that you have the same C++ compiler as the one used to build PyTorch. A simple way is to build PyTorch from source and use the same environment to build torchtext. If you are using the nightly build of PyTorch, checkout the environment it was built with conda (here) and pip (here).
Documentation
Find the documentation here.
Datasets
The datasets module currently contains:
Language modeling: WikiText2, WikiText103, PennTreebank, EnWik9
Machine translation: IWSLT2016, IWSLT2017, Multi30k
Sequence tagging (e.g. POS/NER): UDPOS, CoNLL2000Chunking
Question answering: SQuAD1, SQuAD2
Text classification: AG_NEWS, SogouNews, DBpedia, YelpReviewPolarity, YelpReviewFull, YahooAnswers, AmazonReviewPolarity, AmazonReviewFull, IMDB
For example, to access the raw text from the AG_NEWS dataset:
>>> from torchtext.datasets import AG_NEWS >>> train_iter = AG_NEWS(split='train') >>> # Iterate with for loop >>> for (label, line) in train_iter: >>> print(label, line) >>> # Or send to DataLoader >>> from torch.utils.data import DataLoader >>> train_iter = AG_NEWS(split='train') >>> dataloader = DataLoader(train_iter, batch_size=8, shuffle=False)
Tutorials
To get started with torchtext, users may refer to the following tutorials available on PyTorch website.
[BC Breaking] Legacy
In the v0.9.0 release, we moved the following legacy code to torchtext.legacy. This is part of the work to revamp the torchtext library and the motivation has been discussed in Issue #664:
torchtext.legacy.data.field
torchtext.legacy.data.batch
torchtext.legacy.data.example
torchtext.legacy.data.iterator
torchtext.legacy.data.pipeline
torchtext.legacy.datasets
We have a migration tutorial to help users switch to the torchtext datasets in v0.9.0 release. For the users who still want the legacy components, they can add legacy to the import path.
In the v0.10.0 release, we retire the Vocab class to torchtext.legacy. Users can still access the legacy Vocab via torchtext.legacy.vocab. This class has been replaced by a Vocab module that is backed by efficient C++ implementation and provides common functional APIs for NLP workflows.
Disclaimer on Datasets
This is a utility library that downloads and prepares public datasets. We do not host or distribute these datasets, vouch for their quality or fairness, or claim that you have license to use the dataset. It is your responsibility to determine whether you have permission to use the dataset under the dataset’s license.
If you’re a dataset owner and wish to update any part of it (description, citation, etc.), or do not want your dataset to be included in this library, please get in touch through a GitHub issue. Thanks for your contribution to the ML community!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
File details
Details for the file torchtext-0.12.0-cp310-cp310-win_amd64.whl
.
File metadata
- Download URL: torchtext-0.12.0-cp310-cp310-win_amd64.whl
- Upload date:
- Size: 1.5 MB
- Tags: CPython 3.10, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.7.1 importlib_metadata/3.10.1 pkginfo/1.8.2 requests/2.27.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 391dd25cc426ade637da7297e6d72eaf5efc5fa13a4e25cf2fa5d93119bb176b |
|
MD5 | 82a5e5c094f5ed662784c1b05106d417 |
|
BLAKE2b-256 | 70a804ea5b84b866a5fa3b9f0391006c2c0a11d2bca3090f897b861989830361 |
Provenance
File details
Details for the file torchtext-0.12.0-cp310-cp310-manylinux2014_aarch64.whl
.
File metadata
- Download URL: torchtext-0.12.0-cp310-cp310-manylinux2014_aarch64.whl
- Upload date:
- Size: 12.7 MB
- Tags: CPython 3.10
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.7.1 importlib_metadata/3.10.1 pkginfo/1.8.2 requests/2.27.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 36a8c9d6ddbfb70904f15caade91f8deb28b35c2693347853f1bea7f65f00b40 |
|
MD5 | d1b62452f0d1786af7063c128719f994 |
|
BLAKE2b-256 | c41fd96e850c97d6453e8637ac2177619d9700f906d0f87abde4006bb276d87f |
Provenance
File details
Details for the file torchtext-0.12.0-cp310-cp310-manylinux1_x86_64.whl
.
File metadata
- Download URL: torchtext-0.12.0-cp310-cp310-manylinux1_x86_64.whl
- Upload date:
- Size: 10.4 MB
- Tags: CPython 3.10
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.7.1 importlib_metadata/3.10.1 pkginfo/1.8.2 requests/2.27.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 174bc6b33d77d9eebbae473f98389183a0c269c68416372ca614b12cfa326969 |
|
MD5 | 0af1a044a1f0d14340f5bb7c3d2af61d |
|
BLAKE2b-256 | 967ad2440cca2e3e8191d4af9f7f027d515cd829aa760255111ef36d861787dd |
Provenance
File details
Details for the file torchtext-0.12.0-cp310-cp310-macosx_11_0_arm64.whl
.
File metadata
- Download URL: torchtext-0.12.0-cp310-cp310-macosx_11_0_arm64.whl
- Upload date:
- Size: 1.6 MB
- Tags: CPython 3.10, macOS 11.0+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.7.1 importlib_metadata/3.10.1 pkginfo/1.8.2 requests/2.27.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | af04961985c419579662e703243fa592d74fc44e2863a44c23214b377cf426d8 |
|
MD5 | 69bb2ffeb0190c94b522058d3d3aed7e |
|
BLAKE2b-256 | a147ef5e94a227220c86337fe75009affe40d2ac307aaefc2e994280db8dd9b5 |
Provenance
File details
Details for the file torchtext-0.12.0-cp310-cp310-macosx_10_9_x86_64.whl
.
File metadata
- Download URL: torchtext-0.12.0-cp310-cp310-macosx_10_9_x86_64.whl
- Upload date:
- Size: 1.7 MB
- Tags: CPython 3.10, macOS 10.9+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.7.1 importlib_metadata/3.10.1 pkginfo/1.8.2 requests/2.27.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 66792b1e6d3929944a8d47ad371c71de07a4023f5692385ba7e50c61e111f2f2 |
|
MD5 | 76831dbc447a09ca909fb2217621d1a5 |
|
BLAKE2b-256 | ebf37f2f9247a469325b0ea39e43f40a976cbcc145e8635835f88911eba545b3 |
Provenance
File details
Details for the file torchtext-0.12.0-cp39-cp39-win_amd64.whl
.
File metadata
- Download URL: torchtext-0.12.0-cp39-cp39-win_amd64.whl
- Upload date:
- Size: 1.5 MB
- Tags: CPython 3.9, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.7.1 importlib_metadata/3.10.1 pkginfo/1.8.2 requests/2.27.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 59912472689e5734ccdd134a0352e46bb36cecd3d59b323e097474f4c4e4bf9a |
|
MD5 | ea9786a2d2a6c6ee19265129e2713fb0 |
|
BLAKE2b-256 | ef8248a6d58ed8cc3bfaab5d560acbe4b4856d40545bfdda49985c0dc60a65fa |
Provenance
File details
Details for the file torchtext-0.12.0-cp39-cp39-manylinux2014_aarch64.whl
.
File metadata
- Download URL: torchtext-0.12.0-cp39-cp39-manylinux2014_aarch64.whl
- Upload date:
- Size: 12.7 MB
- Tags: CPython 3.9
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.7.1 importlib_metadata/3.10.1 pkginfo/1.8.2 requests/2.27.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1845c480fe47c641816d770efac3a32fb927673d46b00c062a2ec377bc312cdc |
|
MD5 | 996c0950041c1bbe13b7afce00b2a29c |
|
BLAKE2b-256 | b8e057d1b3b68d25f7fdfd4ab9cb290a6670edf373565922f79b2cdd17497b3d |
Provenance
File details
Details for the file torchtext-0.12.0-cp39-cp39-manylinux1_x86_64.whl
.
File metadata
- Download URL: torchtext-0.12.0-cp39-cp39-manylinux1_x86_64.whl
- Upload date:
- Size: 10.4 MB
- Tags: CPython 3.9
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.7.1 importlib_metadata/3.10.1 pkginfo/1.8.2 requests/2.27.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8bbf5649e4dc691535920437c09017fcab7cfed7125ed0cfe1e1a3bbe792ab93 |
|
MD5 | 0e66c1091b46be0283cd0b4f76449c28 |
|
BLAKE2b-256 | 3ffbe93d421e96a0cdf69ad7cccfde4c6be48be9ba971077d3d3fbcad0e58a43 |
Provenance
File details
Details for the file torchtext-0.12.0-cp39-cp39-macosx_11_0_arm64.whl
.
File metadata
- Download URL: torchtext-0.12.0-cp39-cp39-macosx_11_0_arm64.whl
- Upload date:
- Size: 1.6 MB
- Tags: CPython 3.9, macOS 11.0+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.7.1 importlib_metadata/3.10.1 pkginfo/1.8.2 requests/2.27.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | b7fa7557fe5542b58c09abb4bf14e0c6ed6afe653606e318080ba0718d3e1eda |
|
MD5 | 78f343697b5d5f75ff336c9c2d867a45 |
|
BLAKE2b-256 | 09cf9cab047be367ddf32f4a1a84b1883a0754ed69f0430d8e6c330d25e88318 |
Provenance
File details
Details for the file torchtext-0.12.0-cp39-cp39-macosx_10_9_x86_64.whl
.
File metadata
- Download URL: torchtext-0.12.0-cp39-cp39-macosx_10_9_x86_64.whl
- Upload date:
- Size: 1.6 MB
- Tags: CPython 3.9, macOS 10.9+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.7.1 importlib_metadata/3.10.1 pkginfo/1.8.2 requests/2.27.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2ea13e79b02d226adb122ecc8f9648c768e0b683e32fd8b3c0c0c22e8661ed6f |
|
MD5 | 76a68382fa8b7d494c24bd0776dc336c |
|
BLAKE2b-256 | b9629962efcdc1a8adf93b4c7c1634b930f89d6def3e32a6233f6767b67460ac |
Provenance
File details
Details for the file torchtext-0.12.0-cp38-cp38-win_amd64.whl
.
File metadata
- Download URL: torchtext-0.12.0-cp38-cp38-win_amd64.whl
- Upload date:
- Size: 1.5 MB
- Tags: CPython 3.8, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.7.1 importlib_metadata/3.10.1 pkginfo/1.8.2 requests/2.27.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 24b34a1b857aeef8d0c6e7b3ed2d9230d0a36df5bdb9dd5262b018675b739306 |
|
MD5 | c2c39d05e52c063b4e59a7d731da1ab6 |
|
BLAKE2b-256 | 5b229653d652c7ddda62d18f96525815fa725b0563ec3f0374a7271767eacc7c |
Provenance
File details
Details for the file torchtext-0.12.0-cp38-cp38-manylinux2014_aarch64.whl
.
File metadata
- Download URL: torchtext-0.12.0-cp38-cp38-manylinux2014_aarch64.whl
- Upload date:
- Size: 12.8 MB
- Tags: CPython 3.8
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.7.1 importlib_metadata/3.10.1 pkginfo/1.8.2 requests/2.27.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2ecd9a32064ea5caf78c7e4843719ecabe100066b32df8d228a6f2ba6bc5ea8b |
|
MD5 | db929ced4c6e90ddad503be87ebffbf7 |
|
BLAKE2b-256 | 7618285cdf1e083cf80a7c96b252bd2c2afbeeeb31f08839845f095ccd32c307 |
Provenance
File details
Details for the file torchtext-0.12.0-cp38-cp38-manylinux1_x86_64.whl
.
File metadata
- Download URL: torchtext-0.12.0-cp38-cp38-manylinux1_x86_64.whl
- Upload date:
- Size: 10.4 MB
- Tags: CPython 3.8
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.7.1 importlib_metadata/3.10.1 pkginfo/1.8.2 requests/2.27.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 365e5f640e29d2fc924cc274f024ee02e49e680b5a6d10d4d7ccdd1665b73120 |
|
MD5 | 9043af2cc9ef7a7e73dbcabfe8a8e501 |
|
BLAKE2b-256 | 577cfda80107f0326ea48a0d16889b121c8178c0698e41458e7b1e546b48e374 |
Provenance
File details
Details for the file torchtext-0.12.0-cp38-cp38-macosx_11_0_arm64.whl
.
File metadata
- Download URL: torchtext-0.12.0-cp38-cp38-macosx_11_0_arm64.whl
- Upload date:
- Size: 1.6 MB
- Tags: CPython 3.8, macOS 11.0+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.7.1 importlib_metadata/3.10.1 pkginfo/1.8.2 requests/2.27.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | bce2a8f6f3a82974b950ee526640b3f747e7bade32b92d0747687bfe76af8d81 |
|
MD5 | 01c9ed2d77f70245f23bc3952b039f2f |
|
BLAKE2b-256 | 5c9bff86c4d49262a3489f02d53da7e8b895e8529fb76d1e0865dc313627beb2 |
Provenance
File details
Details for the file torchtext-0.12.0-cp38-cp38-macosx_10_9_x86_64.whl
.
File metadata
- Download URL: torchtext-0.12.0-cp38-cp38-macosx_10_9_x86_64.whl
- Upload date:
- Size: 1.7 MB
- Tags: CPython 3.8, macOS 10.9+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.7.1 importlib_metadata/3.10.1 pkginfo/1.8.2 requests/2.27.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 87530abac557e9702d41edd89c94ccbc4871d43c0f479af97ae3342836e875ec |
|
MD5 | 3c6a43d55390a93d79598838ab7f89e6 |
|
BLAKE2b-256 | 0844b01eca9e5be36af256d2898158a63fcce4d620228f531b6245718e26cc93 |
Provenance
File details
Details for the file torchtext-0.12.0-cp37-cp37m-win_amd64.whl
.
File metadata
- Download URL: torchtext-0.12.0-cp37-cp37m-win_amd64.whl
- Upload date:
- Size: 1.5 MB
- Tags: CPython 3.7m, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.7.1 importlib_metadata/3.10.1 pkginfo/1.8.2 requests/2.27.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 09a047a90615febb5280bb39f69e89de515212d8d0f75b85e5ad474221f1744a |
|
MD5 | 71383013d6f2a73ce5795d913dee770d |
|
BLAKE2b-256 | 028e65d49d3c0e46200cc6442a6bac3cd65e6a04c3c84bbfc6a5fdf56f7babae |
Provenance
File details
Details for the file torchtext-0.12.0-cp37-cp37m-manylinux2014_aarch64.whl
.
File metadata
- Download URL: torchtext-0.12.0-cp37-cp37m-manylinux2014_aarch64.whl
- Upload date:
- Size: 12.9 MB
- Tags: CPython 3.7m
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.7.1 importlib_metadata/3.10.1 pkginfo/1.8.2 requests/2.27.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 21b6b5a00000ad9dfa09d984a1a7b7cdd79c8310c2a780e672cdf95a2bc26c1b |
|
MD5 | 7f6f0bc2dfd59616e382a31504c6d9e6 |
|
BLAKE2b-256 | 52aa210df5ea0eaa0bbdbed16ccb0931164cb820b74714edcf5bace68cb260f3 |
Provenance
File details
Details for the file torchtext-0.12.0-cp37-cp37m-manylinux1_x86_64.whl
.
File metadata
- Download URL: torchtext-0.12.0-cp37-cp37m-manylinux1_x86_64.whl
- Upload date:
- Size: 10.4 MB
- Tags: CPython 3.7m
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.7.1 importlib_metadata/3.10.1 pkginfo/1.8.2 requests/2.27.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1db04814d0289429bebd776707ea58d251166f56cc8298956101534a20f0a5cd |
|
MD5 | 1a4942039d4206cd91f0a5be0bed3af5 |
|
BLAKE2b-256 | c753b2a874be405142508c7d0fe3dfa6971e1a12fe7c3449e87736a5fa99ec73 |
Provenance
File details
Details for the file torchtext-0.12.0-cp37-cp37m-macosx_10_9_x86_64.whl
.
File metadata
- Download URL: torchtext-0.12.0-cp37-cp37m-macosx_10_9_x86_64.whl
- Upload date:
- Size: 1.7 MB
- Tags: CPython 3.7m, macOS 10.9+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.7.1 importlib_metadata/3.10.1 pkginfo/1.8.2 requests/2.27.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.12
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 63d9b6e94af6092c472c77e26cf466599eccc1e3a0facc59c87d7611b5afcda6 |
|
MD5 | 30f98afea6d68a83eeb2ce97b5d1c993 |
|
BLAKE2b-256 | 01eba69f6d776822fd6ca18c29036884f1d1ea4542c44a4f53b317289fa86d86 |