Fast and Customizable Tokenizers

These details have been verified by PyPI

Maintainers

ArthurZucker McPotato Nicolas.Patry xn1t0x

These details have not been verified by PyPI

Project links

Homepage

Project description

Tokenizers

Provides an implementation of today's most used tokenizers, with a focus on performance and versatility.

Bindings over the Rust implementation. If you are interested in the High-level design, you can go check it there.

Otherwise, let's dive in!

Main features:

Train new vocabularies and tokenize using 4 pre-made tokenizers (Bert WordPiece and the 3 most common BPE versions).
Extremely fast (both training and tokenization), thanks to the Rust implementation. Takes less than 20 seconds to tokenize a GB of text on a server's CPU.
Easy to use, but also extremely versatile.
Designed for research and production.
Normalization comes with alignments tracking. It's always possible to get the part of the original sentence that corresponds to a given token.
Does all the pre-processing: Truncate, Pad, add the special tokens your model needs.

Installation

With pip:

pip install tokenizers

From sources:

To use this method, you need to have the Rust installed:

# Install with:
curl https://sh.rustup.rs -sSf | sh -s -- -y
export PATH="$HOME/.cargo/bin:$PATH"

Once Rust is installed, you can compile doing the following

git clone https://github.com/huggingface/tokenizers
cd tokenizers/bindings/python

# Create a virtual env (you can use yours as well)
python -m venv .env
source .env/bin/activate

# Install `tokenizers` in the current virtual env
pip install setuptools_rust
python setup.py install

Load a pretrained tokenizer from the Hub

from tokenizers import Tokenizer

tokenizer = Tokenizer.from_pretrained("bert-base-cased")

Using the provided Tokenizers

We provide some pre-build tokenizers to cover the most common cases. You can easily load one of these using some vocab.json and merges.txt files:

from tokenizers import CharBPETokenizer

# Initialize a tokenizer
vocab = "./path/to/vocab.json"
merges = "./path/to/merges.txt"
tokenizer = CharBPETokenizer(vocab, merges)

# And then encode:
encoded = tokenizer.encode("I can feel the magic, can you?")
print(encoded.ids)
print(encoded.tokens)

And you can train them just as simply:

from tokenizers import CharBPETokenizer

# Initialize a tokenizer
tokenizer = CharBPETokenizer()

# Then train it!
tokenizer.train([ "./path/to/files/1.txt", "./path/to/files/2.txt" ])

# Now, let's use it:
encoded = tokenizer.encode("I can feel the magic, can you?")

# And finally save it somewhere
tokenizer.save("./path/to/directory/my-bpe.tokenizer.json")

Provided Tokenizers

CharBPETokenizer: The original BPE
ByteLevelBPETokenizer: The byte level version of the BPE
SentencePieceBPETokenizer: A BPE implementation compatible with the one used by SentencePiece
BertWordPieceTokenizer: The famous Bert tokenizer, using WordPiece

All of these can be used and trained as explained above!

Build your own

Whenever these provided tokenizers don't give you enough freedom, you can build your own tokenizer, by putting all the different parts you need together. You can check how we implemented the provided tokenizers and adapt them easily to your own needs.

Building a byte-level BPE

Here is an example showing how to build your own byte-level BPE by putting all the different pieces together, and then saving it to a single file:

from tokenizers import Tokenizer, models, pre_tokenizers, decoders, trainers, processors

# Initialize a tokenizer
tokenizer = Tokenizer(models.BPE())

# Customize pre-tokenization and decoding
tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=True)
tokenizer.decoder = decoders.ByteLevel()
tokenizer.post_processor = processors.ByteLevel(trim_offsets=True)

# And then train
trainer = trainers.BpeTrainer(
    vocab_size=20000,
    min_frequency=2,
    initial_alphabet=pre_tokenizers.ByteLevel.alphabet()
)
tokenizer.train([
    "./path/to/dataset/1.txt",
    "./path/to/dataset/2.txt",
    "./path/to/dataset/3.txt"
], trainer=trainer)

# And Save it
tokenizer.save("byte-level-bpe.tokenizer.json", pretty=True)

Now, when you want to use this tokenizer, this is as simple as:

from tokenizers import Tokenizer

tokenizer = Tokenizer.from_file("byte-level-bpe.tokenizer.json")

encoded = tokenizer.encode("I can feel the magic, can you?")

Project details

These details have been verified by PyPI

Maintainers

ArthurZucker McPotato Nicolas.Patry xn1t0x

These details have not been verified by PyPI

Project links

Homepage

Release history Release notifications | RSS feed

0.20.3

Nov 5, 2024

0.20.3rc0 pre-release

Nov 15, 2024

0.20.2

Nov 4, 2024

0.20.1

Oct 10, 2024

0.20.1rc1 pre-release

Oct 10, 2024

0.20.0

Aug 8, 2024

0.20.0rc1 pre-release

Aug 8, 2024

0.19.1

Apr 17, 2024

0.19.0

Apr 17, 2024

0.15.2

Feb 12, 2024

0.15.2rc1 pre-release

Feb 11, 2024

0.15.1

Jan 22, 2024

0.15.0

Nov 14, 2023

0.14.1

Oct 6, 2023

0.14.0

Sep 7, 2023

0.13.3

Apr 5, 2023

0.13.2

Nov 7, 2022

0.13.1

Oct 6, 2022

0.13.0

Sep 21, 2022

0.12.1

Apr 13, 2022

0.12.0 yanked

Mar 31, 2022

Reason this release was yanked:

Breaking change with unexpected effect

This version

0.11.6

Feb 28, 2022

0.11.5

Feb 16, 2022

0.11.4

Jan 17, 2022

0.11.3

Jan 17, 2022

0.11.2

Jan 4, 2022

0.11.1

Dec 28, 2021

0.11.0

Dec 23, 2021

0.10.3

May 24, 2021

0.10.2

Apr 5, 2021

0.10.1

Feb 8, 2021

0.10.1rc1 pre-release

Feb 4, 2021

0.10.0

Jan 12, 2021

0.10.0rc1 pre-release

Dec 8, 2020

0.9.4

Nov 10, 2020

0.9.3

Oct 26, 2020

0.9.2

Oct 15, 2020

0.9.1

Oct 13, 2020

0.9.0

Oct 9, 2020

0.9.0rc2 pre-release

Oct 6, 2020

0.9.0rc1 pre-release

Sep 29, 2020

0.9.0.dev4 pre-release

Sep 24, 2020

0.9.0.dev3 pre-release

Sep 18, 2020

0.9.0.dev2 pre-release

Sep 14, 2020

0.9.0.dev1 pre-release

Sep 14, 2020

0.9.0.dev0 pre-release

Aug 21, 2020

0.8.1

Jul 20, 2020

0.8.1rc2 pre-release

Jul 17, 2020

0.8.1rc1 pre-release

Jul 6, 2020

0.8.0

Jun 26, 2020

0.8.0rc4 pre-release

Jun 26, 2020

0.8.0rc3 pre-release

Jun 22, 2020

0.8.0rc2 pre-release

Jun 19, 2020

0.8.0rc1 pre-release

Jun 11, 2020

0.8.0.dev2 pre-release

Jun 3, 2020

0.8.0.dev1 pre-release

May 27, 2020

0.8.0.dev0 pre-release

May 21, 2020

0.7.0

Apr 17, 2020

0.7.0rc7 pre-release

Apr 16, 2020

0.7.0rc6 pre-release

Apr 16, 2020

0.7.0rc5 pre-release

Apr 9, 2020

0.7.0rc4 pre-release

Apr 8, 2020

0.7.0rc3 pre-release

Mar 31, 2020

0.7.0rc2 pre-release

Mar 27, 2020

0.7.0rc1 pre-release

Mar 26, 2020

0.6.0

Mar 2, 2020

0.5.2

Feb 24, 2020

0.5.1

Feb 24, 2020

0.5.0

Feb 19, 2020

0.4.2

Feb 11, 2020

0.4.1

Feb 11, 2020

0.4.0

Feb 10, 2020

0.3.0

Feb 5, 2020

0.2.1

Jan 22, 2020

0.2.0

Jan 20, 2020

0.1.1

Jan 12, 2020

0.1.0

Jan 10, 2020

0.0.13

Jan 8, 2020

0.0.12

Jan 7, 2020

0.0.11

Dec 27, 2019

0.0.10

Dec 26, 2019

0.0.9

Dec 23, 2019

0.0.8

Dec 20, 2019

0.0.7

Dec 17, 2019

0.0.6

Dec 17, 2019

0.0.5

Dec 13, 2019

0.0.4

Dec 10, 2019

0.0.3

Dec 3, 2019

0.0.2

Dec 2, 2019

0.0.1

Nov 1, 2019

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tokenizers-0.11.6.tar.gz (221.1 kB view details)

Uploaded Feb 28, 2022 Source

Built Distributions

tokenizers-0.11.6-cp310-cp310-win_amd64.whl (3.2 MB view details)

Uploaded Feb 28, 2022 CPython 3.10 Windows x86-64

tokenizers-0.11.6-cp310-cp310-win32.whl (2.9 MB view details)

Uploaded Feb 28, 2022 CPython 3.10 Windows x86

tokenizers-0.11.6-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl (7.5 MB view details)

Uploaded Feb 28, 2022 CPython 3.10 manylinux: glibc 2.17+ s390x

tokenizers-0.11.6-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (7.9 MB view details)

Uploaded Feb 28, 2022 CPython 3.10 manylinux: glibc 2.17+ ppc64le

tokenizers-0.11.6-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (6.8 MB view details)

Uploaded Feb 28, 2022 CPython 3.10 manylinux: glibc 2.17+ ARM64

tokenizers-0.11.6-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (6.5 MB view details)

Uploaded Feb 28, 2022 CPython 3.10 manylinux: glibc 2.12+ x86-64

tokenizers-0.11.6-cp310-cp310-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded Mar 1, 2022 CPython 3.10 macOS 11.0+ ARM64

tokenizers-0.11.6-cp310-cp310-macosx_10_11_x86_64.whl (3.6 MB view details)

Uploaded Feb 28, 2022 CPython 3.10 macOS 10.11+ x86-64

tokenizers-0.11.6-cp39-cp39-win_amd64.whl (3.2 MB view details)

Uploaded Feb 28, 2022 CPython 3.9 Windows x86-64

tokenizers-0.11.6-cp39-cp39-win32.whl (2.9 MB view details)

Uploaded Feb 28, 2022 CPython 3.9 Windows x86

tokenizers-0.11.6-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl (7.5 MB view details)

Uploaded Feb 28, 2022 CPython 3.9 manylinux: glibc 2.17+ s390x

tokenizers-0.11.6-cp39-cp39-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (7.9 MB view details)

Uploaded Feb 28, 2022 CPython 3.9 manylinux: glibc 2.17+ ppc64le

tokenizers-0.11.6-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (6.8 MB view details)

Uploaded Feb 28, 2022 CPython 3.9 manylinux: glibc 2.17+ ARM64

tokenizers-0.11.6-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (6.5 MB view details)

Uploaded Feb 28, 2022 CPython 3.9 manylinux: glibc 2.12+ x86-64

tokenizers-0.11.6-cp39-cp39-macosx_12_0_arm64.whl (3.4 MB view details)

Uploaded Mar 1, 2022 CPython 3.9 macOS 12.0+ ARM64

tokenizers-0.11.6-cp39-cp39-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded Mar 1, 2022 CPython 3.9 macOS 11.0+ ARM64

tokenizers-0.11.6-cp39-cp39-macosx_10_11_x86_64.whl (3.6 MB view details)

Uploaded Feb 28, 2022 CPython 3.9 macOS 10.11+ x86-64

tokenizers-0.11.6-cp38-cp38-win_amd64.whl (3.2 MB view details)

Uploaded Feb 28, 2022 CPython 3.8 Windows x86-64

tokenizers-0.11.6-cp38-cp38-win32.whl (2.9 MB view details)

Uploaded Feb 28, 2022 CPython 3.8 Windows x86

tokenizers-0.11.6-cp38-cp38-manylinux_2_17_s390x.manylinux2014_s390x.whl (7.5 MB view details)

Uploaded Feb 28, 2022 CPython 3.8 manylinux: glibc 2.17+ s390x

tokenizers-0.11.6-cp38-cp38-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (7.9 MB view details)

Uploaded Feb 28, 2022 CPython 3.8 manylinux: glibc 2.17+ ppc64le

tokenizers-0.11.6-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (6.8 MB view details)

Uploaded Feb 28, 2022 CPython 3.8 manylinux: glibc 2.17+ ARM64

tokenizers-0.11.6-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (6.5 MB view details)

Uploaded Feb 28, 2022 CPython 3.8 manylinux: glibc 2.12+ x86-64

tokenizers-0.11.6-cp38-cp38-macosx_11_0_arm64.whl (3.4 MB view details)

Uploaded Mar 1, 2022 CPython 3.8 macOS 11.0+ ARM64

tokenizers-0.11.6-cp38-cp38-macosx_10_11_x86_64.whl (3.6 MB view details)

Uploaded Feb 28, 2022 CPython 3.8 macOS 10.11+ x86-64

tokenizers-0.11.6-cp37-cp37m-win_amd64.whl (3.2 MB view details)

Uploaded Feb 28, 2022 CPython 3.7m Windows x86-64

tokenizers-0.11.6-cp37-cp37m-win32.whl (2.9 MB view details)

Uploaded Feb 28, 2022 CPython 3.7m Windows x86

tokenizers-0.11.6-cp37-cp37m-manylinux_2_17_s390x.manylinux2014_s390x.whl (7.5 MB view details)

Uploaded Feb 28, 2022 CPython 3.7m manylinux: glibc 2.17+ s390x

tokenizers-0.11.6-cp37-cp37m-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (7.9 MB view details)

Uploaded Feb 28, 2022 CPython 3.7m manylinux: glibc 2.17+ ppc64le

tokenizers-0.11.6-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (6.8 MB view details)

Uploaded Feb 28, 2022 CPython 3.7m manylinux: glibc 2.17+ ARM64

tokenizers-0.11.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (6.5 MB view details)

Uploaded Feb 28, 2022 CPython 3.7m manylinux: glibc 2.12+ x86-64

tokenizers-0.11.6-cp37-cp37m-macosx_10_11_x86_64.whl (3.6 MB view details)

Uploaded Feb 28, 2022 CPython 3.7m macOS 10.11+ x86-64

tokenizers-0.11.6-cp36-cp36m-manylinux_2_17_s390x.manylinux2014_s390x.whl (7.5 MB view details)

Uploaded Feb 28, 2022 CPython 3.6m manylinux: glibc 2.17+ s390x

tokenizers-0.11.6-cp36-cp36m-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (7.9 MB view details)

Uploaded Feb 28, 2022 CPython 3.6m manylinux: glibc 2.17+ ppc64le

tokenizers-0.11.6-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (6.8 MB view details)

Uploaded Feb 28, 2022 CPython 3.6m manylinux: glibc 2.17+ ARM64

tokenizers-0.11.6-cp36-cp36m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (6.5 MB view details)

Uploaded Feb 28, 2022 CPython 3.6m manylinux: glibc 2.12+ x86-64

tokenizers-0.11.6-cp36-cp36m-macosx_12_0_arm64.whl (3.4 MB view details)

Uploaded Mar 1, 2022 CPython 3.6m macOS 12.0+ ARM64

File details

Details for the file tokenizers-0.11.6.tar.gz.

File metadata

Download URL: tokenizers-0.11.6.tar.gz
Upload date: Feb 28, 2022
Size: 221.1 kB
Tags: Source
Uploaded using Trusted Publishing? No
Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.63.0 importlib-metadata/4.11.2 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.3 CPython/3.10.2

File hashes

Hashes for tokenizers-0.11.6.tar.gz
Algorithm	Hash digest
SHA256	`562b2022faf0882586c915385620d1f11798fc1b32bac55353a530132369a6d0`
MD5	`6d8a7d0799d234df817141253452bf7b`
BLAKE2b-256	`c6adf1d539784a41747e76f3d543bedcb3b255ff677b4f113e362743ede6384f`

Algorithm	Hash digest
SHA256	`3c1ad5d230bdd6a3f63bbffd16a9fdea6a049ceb6d225e4d70a2664853f40aaf`
MD5	`7e68cb55338866304579860f1e150a21`
BLAKE2b-256	`919b34a4135b4ffe71b8a9d2b3550c6d7919e628b0fbc1551392e634724165fe`

Algorithm	Hash digest
SHA256	`6bc5cc6d5b17bf8222049a2c97bcc117974793e37fb2e42b8fb04b2ef984d165`
MD5	`aea51082fbe27c872878c04ae1d25ccd`
BLAKE2b-256	`a29f0ad48cf1e9fb61b50937d9b0bebd60bd420238fa6125cb5d336a3e2547c1`

Algorithm	Hash digest
SHA256	`46a763d1f43f46448e41884356f105a2f067b6e573c7e0c67d8f93512304b22c`
MD5	`2ade9adbe9632e30e6fa3a1195205838`
BLAKE2b-256	`8a6109c5224ab66fbc62978a703e1ae7e51feac2c83633db4c8e9c60ecf908a3`

Algorithm	Hash digest
SHA256	`3fde9a1e7d18caddff3ac13baf4e31235688b0db2ba42bbc8179dc878327560a`
MD5	`86602f515a0555ef149f2c23c664ac61`
BLAKE2b-256	`99c727f45928247dd01af8fe3a2f02f339a13d2f62c71cde78fb027114736b76`

Algorithm	Hash digest
SHA256	`c7d8ea3a05b593e5744c4ad0e6e2cbba6f82588c302d663855316c1861c09557`
MD5	`635c5554d9f3a957c4103ead8952c178`
BLAKE2b-256	`d95506835c5c45f18fa5965dd5aac66ca5472f7acb279969b45c409dadaf3e49`

Algorithm	Hash digest
SHA256	`6f7b82aaedb24e0a4dcd8fe77a79de9a0acf43db8ae173cdb4eca1e767566b47`
MD5	`adeee300f6fe2d825bed557a73a980a5`
BLAKE2b-256	`297636c05a1082749a8daaef9044e9be9ad52b048691599e7d93e6d3ff59fcbb`

Algorithm	Hash digest
SHA256	`1c5a786fe12a4c1782337abc818fc48ca84e07f8cb0eeab263a27fcd30f7fc6f`
MD5	`fcbcff2b78de30cdbf81fe986918cc41`
BLAKE2b-256	`27938214a4670a743b5a6d78877c1751ef705b39104f0c1e1c0a552fbae4e625`

Algorithm	Hash digest
SHA256	`c24f3e0e69edf015efab6bea0a24d45eb19f477106d00a739c19d2a02f6085fc`
MD5	`87aa0381a359b7a9ddbd004fe398370d`
BLAKE2b-256	`2f2cb577c144a05e03a5803ac5b82381f67529734917f10a40dc154d85161d7f`

Algorithm	Hash digest
SHA256	`b28966c68a2cdecd5120f4becea159eebe0335b8202e21e292eb381031026edc`
MD5	`f703956d3c01f1c8bf72675ea51d189e`
BLAKE2b-256	`77a52f6f2c25fb9f3a237aa326a8f3b3fe7a7aae6331136325cf2418dcbc9b89`

Algorithm	Hash digest
SHA256	`468c3e3f414a532924fa277a58807e127a27d2a56b04113540ea755fc5ca9ba8`
MD5	`6e8c7d00a2654332237b4505857b776d`
BLAKE2b-256	`add1cb3f0171c43759b543004b45144c1d8b71f1b9a580faff3d6a4f253f55e6`

Algorithm	Hash digest
SHA256	`0d4f67758f62d64410e86d548c2c38ca5e59310ca38c18e8239637213c49e2fb`
MD5	`66ffc79c59239e3e2b11702308123ffb`
BLAKE2b-256	`59987c9b4cf43d8fb2d732d4c3a935e48d42da59d484d145f03c901b38fb38ff`

Algorithm	Hash digest
SHA256	`47878a49211f8df63f7ec7de62f13b1c65e68be501f6a98b28f4bbb8d3390f25`
MD5	`83fda1ae962554de06f189d8c2533a01`
BLAKE2b-256	`24a961291aa2ca0ee1ccee928da42945ce0d44980b3467396174a7d990d617de`

Algorithm	Hash digest
SHA256	`8afc3c76268cdef4567f84a19d9c06fdd697b22f5142d1af18254ec70703db7b`
MD5	`70c39330c45921d8c3d7e58be4874434`
BLAKE2b-256	`736c93d95d9834b0ea429daddce5f2bcc52c51f494312f46e4cf8733cee97a1b`

Algorithm	Hash digest
SHA256	`e75efa0275665ca30536c75dc5b4d33492fc40ae40c90d9085bdcdb99e060044`
MD5	`fbcbf1c8ccc2a318badb645983cc7f1c`
BLAKE2b-256	`1347f7c1c0f523597d885b95aeeed4bfb05466cad8f0ac12f65b66c04e880c9d`

Algorithm	Hash digest
SHA256	`b9a34bd33d866862f45bfc4a409563132a0b6af5951e08f3ccfde36152cd7683`
MD5	`db1c2b94bb591cd4943a5d8b5abf1347`
BLAKE2b-256	`b8dd89cceb1acb9cf30eec49e89afde082cca97edd26bf467fbf54a9f87aacbb`

Algorithm	Hash digest
SHA256	`5b3829ad386747760e7d805c9ffd5cb5a9309679467029eea3162fb76c8c89dc`
MD5	`28848016a91521e1436479c0faf2d014`
BLAKE2b-256	`2b68c4b394c83695c0a09c97ab06380a0021c102639b1fbd70b6c6c980db64e4`

Algorithm	Hash digest
SHA256	`809b506a6e9f2f6cba86cfe642c1d1e82cbd758574bdc1207efe07229d6fa4d4`
MD5	`5e9353d2adedbb5640c9a18f5b8efe26`
BLAKE2b-256	`5f90c2e1432fa89637265668473dc4a54e0a2108624b58a563cfdc6788793f11`

Algorithm	Hash digest
SHA256	`3c1c6e10f655e0f57b9bb9a64ac4b7ea70d20a73f8013f8bd38d21d64d45d96a`
MD5	`a645bd1722653013d9c505c7e036ddfd`
BLAKE2b-256	`3eb1762427feb3a942eb16c88b2b45f6f934d63616ae01d887078819c1495c5d`

Algorithm	Hash digest
SHA256	`fd3fccc801b7621eca8dfd876494fad0b00bd43fc73afdc1c11d6be3c9136f78`
MD5	`08b3a9fad9fa353060e01f0fa9b95aa4`
BLAKE2b-256	`a696b7bdac79da2d6fd05f3803821ae3408680c427229e591e7507ad3f2ae23c`

Algorithm	Hash digest
SHA256	`251978daef3b57257bd313df7ef705aacc6cd5831644f6e2e8a42e2c7b7c4a30`
MD5	`5d54de5084f18b56e17536f2aad6ec06`
BLAKE2b-256	`28b5dd212939e998f1ae38753326b33a40d07926002c258f3f2d4a61730ad561`

tokenizers 0.11.6

Navigation

Verified details

Maintainers

Unverified details

Project links

Meta

Classifiers

Project description

Tokenizers

Main features:

Installation

With pip:

From sources:

Load a pretrained tokenizer from the Hub

Using the provided Tokenizers

Provided Tokenizers

Build your own

Building a byte-level BPE

Project details

Verified details

Maintainers

Unverified details

Project links

Meta

Classifiers

Release history Release notifications | RSS feed

Download files

Source Distribution

Built Distributions

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

Algorithm	Hash digest
SHA256	`d393eb6b79ab972e6ebede2aa330903913cf380f6fe975d64a92fca15b5d8579`
MD5	`d9b943e2792366173e0287164eeb6b69`
BLAKE2b-256	`af70beee8246820866fa9ccb02eea72f1001d764bf0c5da77c46d5f0c9d7ce81`

Algorithm	Hash digest
SHA256	`5c5ae264b75b7355db2bc3f34e8e3eb608fcca799ff9a109f3d61e4d9614f947`
MD5	`f74bef2113516bd1688871bcdfe07007`
BLAKE2b-256	`81fc2bcdf379c0e0ab0231cb31d2bec20d1574d5e79f40cbb487ba6ca6e2de96`

Algorithm	Hash digest
SHA256	`7074162348a7784faccbea18cf814138483ce0c47eb17dc482cbc4bbbde8a00c`
MD5	`0bc714d0770dd3fd812bcc37832fe0a7`
BLAKE2b-256	`519c60efcefec8b6ef7c30207d4fd6141495debc29cae7f2577f1ece9f7b0dc2`

Algorithm	Hash digest
SHA256	`e6d02076160d60966d6b2b320744affe6b846ee10a37d1c0222b6ca9e640bdb8`
MD5	`5e1b25884360ed8869911dd371362108`
BLAKE2b-256	`8bf8ebeb4d6b65b026cd39384f0dbd5d20c8e10180dd826b069a4a50827a844a`

Algorithm	Hash digest
SHA256	`216745a6e92eb52d99b56123e6fece59e0dcf7bd1444f42bee09e1f02c89bbec`
MD5	`62e7426fd971544b16de90d42383fade`
BLAKE2b-256	`abc234e886390a065b463bfa435426d990a50d2184fedab4e034a2dc8e37329d`

Algorithm	Hash digest
SHA256	`6840554a8cac1196db627d42edbf77f4b5decf6ce6fa9ca073efaaa2cff887a6`
MD5	`02f23c2c80332287596e4b18f3c491e6`
BLAKE2b-256	`cbfd0fbfd3f0444a3eecbe5537d64ea45a78b111a80bae49b95eaa5f19d6c8ea`

Algorithm	Hash digest
SHA256	`7ed928ac19a3397af6fe3716b313fb13dcaf54978f0fc159eeabaff8e35e5c92`
MD5	`f0f75cd791b528c5222bb35d685fd126`
BLAKE2b-256	`50926082f70b458f9d109b219d03d946265b1c63ef1a8d15c16db7a7c8587661`

Algorithm	Hash digest
SHA256	`79074927fc9efaf13b3accd53246e50ade37550920077411ab55bd5ed4944153`
MD5	`2f01fa6aa2c5d152032540d3bbbc747a`
BLAKE2b-256	`96c560098e3f262dc80ccffa59eaf08efe7cfbabcafdf1181b1df8a5bdef05d7`

Algorithm	Hash digest
SHA256	`6259833c189e36c29e84a853b9503028324a3b176a2ae3980c815456d326b652`
MD5	`1428625e691ce089000ae76c15ed08ea`
BLAKE2b-256	`577412152600275cfb8d8dcfb45cf3776dc86736872241e9f53f2b6278700b5a`

Algorithm	Hash digest
SHA256	`6c72383a918e9fef9c2bf4666a961f3b312361fb027165b4446ff333441ebf91`
MD5	`f4f70621a8741cfbcd6c02d12d4e7e16`
BLAKE2b-256	`a6c460c8a13245adbd86efce68fface5b6ef26195e82492a931fdf1f05c5b45d`