Skip to main content

Fast and Customizable Tokenizers

Project description



Build GitHub


Tokenizers

Provides an implementation of today's most used tokenizers, with a focus on performance and versatility.

Bindings over the Rust implementation. If you are interested in the High-level design, you can go check it there.

Otherwise, let's dive in!

Main features:

  • Train new vocabularies and tokenize using 4 pre-made tokenizers (Bert WordPiece and the 3 most common BPE versions).
  • Extremely fast (both training and tokenization), thanks to the Rust implementation. Takes less than 20 seconds to tokenize a GB of text on a server's CPU.
  • Easy to use, but also extremely versatile.
  • Designed for research and production.
  • Normalization comes with alignments tracking. It's always possible to get the part of the original sentence that corresponds to a given token.
  • Does all the pre-processing: Truncate, Pad, add the special tokens your model needs.

Installation

With pip:

pip install tokenizers

From sources:

To use this method, you need to have the Rust installed:

# Install with:
curl https://sh.rustup.rs -sSf | sh -s -- -y
export PATH="$HOME/.cargo/bin:$PATH"

Once Rust is installed, you can compile doing the following

git clone https://github.com/huggingface/tokenizers
cd tokenizers/bindings/python

# Create a virtual env (you can use yours as well)
python -m venv .env
source .env/bin/activate

# Install `tokenizers` in the current virtual env
pip install setuptools_rust
python setup.py install

Using the provided Tokenizers

We provide some pre-build tokenizers to cover the most common cases. You can easily load one of these using some vocab.json and merges.txt files:

from tokenizers import CharBPETokenizer

# Initialize a tokenizer
vocab = "./path/to/vocab.json"
merges = "./path/to/merges.txt"
tokenizer = CharBPETokenizer(vocab, merges)

# And then encode:
encoded = tokenizer.encode("I can feel the magic, can you?")
print(encoded.ids)
print(encoded.tokens)

And you can train them just as simply:

from tokenizers import CharBPETokenizer

# Initialize a tokenizer
tokenizer = CharBPETokenizer()

# Then train it!
tokenizer.train([ "./path/to/files/1.txt", "./path/to/files/2.txt" ])

# Now, let's use it:
encoded = tokenizer.encode("I can feel the magic, can you?")

# And finally save it somewhere
tokenizer.save("./path/to/directory/my-bpe.tokenizer.json")

Provided Tokenizers

  • CharBPETokenizer: The original BPE
  • ByteLevelBPETokenizer: The byte level version of the BPE
  • SentencePieceBPETokenizer: A BPE implementation compatible with the one used by SentencePiece
  • BertWordPieceTokenizer: The famous Bert tokenizer, using WordPiece

All of these can be used and trained as explained above!

Build your own

Whenever these provided tokenizers don't give you enough freedom, you can build your own tokenizer, by putting all the different parts you need together. You can check how we implemented the provided tokenizers and adapt them easily to your own needs.

Building a byte-level BPE

Here is an example showing how to build your own byte-level BPE by putting all the different pieces together, and then saving it to a single file:

from tokenizers import Tokenizer, models, pre_tokenizers, decoders, trainers, processors

# Initialize a tokenizer
tokenizer = Tokenizer(models.BPE())

# Customize pre-tokenization and decoding
tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=True)
tokenizer.decoder = decoders.ByteLevel()
tokenizer.post_processor = processors.ByteLevel(trim_offsets=True)

# And then train
trainer = trainers.BpeTrainer(vocab_size=20000, min_frequency=2)
tokenizer.train([
	"./path/to/dataset/1.txt",
	"./path/to/dataset/2.txt",
	"./path/to/dataset/3.txt"
], trainer=trainer)

# And Save it
tokenizer.save("byte-level-bpe.tokenizer.json", pretty=True)

Now, when you want to use this tokenizer, this is as simple as:

from tokenizers import Tokenizer

tokenizer = Tokenizer.from_file("byte-level-bpe.tokenizer.json")

encoded = tokenizer.encode("I can feel the magic, can you?")

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tokenizers-0.10.0.tar.gz (209.4 kB view details)

Uploaded Source

Built Distributions

tokenizers-0.10.0-cp39-cp39-win_amd64.whl (2.0 MB view details)

Uploaded CPython 3.9 Windows x86-64

tokenizers-0.10.0-cp39-cp39-win32.whl (1.8 MB view details)

Uploaded CPython 3.9 Windows x86

tokenizers-0.10.0-cp39-cp39-manylinux2014_s390x.whl (3.7 MB view details)

Uploaded CPython 3.9

tokenizers-0.10.0-cp39-cp39-manylinux2010_x86_64.whl (3.2 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.12+ x86-64

tokenizers-0.10.0-cp39-cp39-manylinux1_x86_64.whl (3.2 MB view details)

Uploaded CPython 3.9

tokenizers-0.10.0-cp39-cp39-macosx_10_11_x86_64.whl (2.2 MB view details)

Uploaded CPython 3.9 macOS 10.11+ x86-64

tokenizers-0.10.0-cp38-cp38-win_amd64.whl (2.0 MB view details)

Uploaded CPython 3.8 Windows x86-64

tokenizers-0.10.0-cp38-cp38-win32.whl (1.8 MB view details)

Uploaded CPython 3.8 Windows x86

tokenizers-0.10.0-cp38-cp38-manylinux2014_s390x.whl (3.7 MB view details)

Uploaded CPython 3.8

tokenizers-0.10.0-cp38-cp38-manylinux2010_x86_64.whl (3.2 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.12+ x86-64

tokenizers-0.10.0-cp38-cp38-manylinux1_x86_64.whl (3.2 MB view details)

Uploaded CPython 3.8

tokenizers-0.10.0-cp38-cp38-macosx_10_11_x86_64.whl (2.2 MB view details)

Uploaded CPython 3.8 macOS 10.11+ x86-64

tokenizers-0.10.0-cp37-cp37m-win_amd64.whl (2.0 MB view details)

Uploaded CPython 3.7m Windows x86-64

tokenizers-0.10.0-cp37-cp37m-win32.whl (1.8 MB view details)

Uploaded CPython 3.7m Windows x86

tokenizers-0.10.0-cp37-cp37m-manylinux2014_s390x.whl (3.7 MB view details)

Uploaded CPython 3.7m

tokenizers-0.10.0-cp37-cp37m-manylinux2010_x86_64.whl (3.2 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.12+ x86-64

tokenizers-0.10.0-cp37-cp37m-manylinux1_x86_64.whl (3.2 MB view details)

Uploaded CPython 3.7m

tokenizers-0.10.0-cp37-cp37m-macosx_10_11_x86_64.whl (2.2 MB view details)

Uploaded CPython 3.7m macOS 10.11+ x86-64

tokenizers-0.10.0-cp36-cp36m-win_amd64.whl (2.0 MB view details)

Uploaded CPython 3.6m Windows x86-64

tokenizers-0.10.0-cp36-cp36m-win32.whl (1.8 MB view details)

Uploaded CPython 3.6m Windows x86

tokenizers-0.10.0-cp36-cp36m-manylinux2014_s390x.whl (3.7 MB view details)

Uploaded CPython 3.6m

tokenizers-0.10.0-cp36-cp36m-manylinux2010_x86_64.whl (3.2 MB view details)

Uploaded CPython 3.6m manylinux: glibc 2.12+ x86-64

tokenizers-0.10.0-cp36-cp36m-manylinux1_x86_64.whl (3.2 MB view details)

Uploaded CPython 3.6m

tokenizers-0.10.0-cp36-cp36m-macosx_10_11_x86_64.whl (2.2 MB view details)

Uploaded CPython 3.6m macOS 10.11+ x86-64

tokenizers-0.10.0-cp35-cp35m-win_amd64.whl (2.0 MB view details)

Uploaded CPython 3.5m Windows x86-64

tokenizers-0.10.0-cp35-cp35m-win32.whl (1.8 MB view details)

Uploaded CPython 3.5m Windows x86

tokenizers-0.10.0-cp35-cp35m-manylinux2014_s390x.whl (3.7 MB view details)

Uploaded CPython 3.5m

tokenizers-0.10.0-cp35-cp35m-manylinux2010_x86_64.whl (3.2 MB view details)

Uploaded CPython 3.5m manylinux: glibc 2.12+ x86-64

tokenizers-0.10.0-cp35-cp35m-manylinux1_x86_64.whl (3.2 MB view details)

Uploaded CPython 3.5m

tokenizers-0.10.0-cp35-cp35m-macosx_10_11_x86_64.whl (2.2 MB view details)

Uploaded CPython 3.5m macOS 10.11+ x86-64

File details

Details for the file tokenizers-0.10.0.tar.gz.

File metadata

  • Download URL: tokenizers-0.10.0.tar.gz
  • Upload date:
  • Size: 209.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0.tar.gz
Algorithm Hash digest
SHA256 13e9971af25b39e6f355835dc192bcc6b6091c045869af4a99838cbbd4c8cb6c
MD5 f9e227dfde5afd77a4783b605e294bfe
BLAKE2b-256 609aec2ac1728154f6ab91032c03901b4321c8a263c2c901a2afddc0d1f36786

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 2.0 MB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 db1b33b3b7d40742f1447d3d03205eb4924c79b66cc69f63eafe4a22b513bd4c
MD5 3554aeecc12530c5a91256ce1b95c2d0
BLAKE2b-256 686fd09f7e862b8be0c80803e18a3549e410c972a40d53e369b7dafe112293fc

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp39-cp39-win32.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp39-cp39-win32.whl
  • Upload date:
  • Size: 1.8 MB
  • Tags: CPython 3.9, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp39-cp39-win32.whl
Algorithm Hash digest
SHA256 e750eb8876bfa1d69af44dde308de6a6d888e41f82876f65c8e69441088c4dac
MD5 bc8fa526438eead05d8fbcacee5a616c
BLAKE2b-256 3e730b6e9b24d665a11e91d663d0aa924b490ab3fa850a04d7a48ec26a3a5c28

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp39-cp39-manylinux2014_s390x.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp39-cp39-manylinux2014_s390x.whl
  • Upload date:
  • Size: 3.7 MB
  • Tags: CPython 3.9
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp39-cp39-manylinux2014_s390x.whl
Algorithm Hash digest
SHA256 d2a60c309707e992e250c4636837a25339b0f5d1ce9a18ed192fc69c102bf0b0
MD5 216a1f0b21fd518c8eb42b2cfc64e207
BLAKE2b-256 225e6e451529eac8f2ff430a33621fa9af9f4410598d8e94d2ad6a80af462c4d

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp39-cp39-manylinux2014_ppc64le.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp39-cp39-manylinux2014_ppc64le.whl
  • Upload date:
  • Size: 3.6 MB
  • Tags: CPython 3.9
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp39-cp39-manylinux2014_ppc64le.whl
Algorithm Hash digest
SHA256 df325bdf4d785ebd5952e396de2242c4754fd23ed38ddc0d5bc3cd33e078b000
MD5 c10ec009d8d72c7f8a16042f58cbcf29
BLAKE2b-256 27ff84cb1cbe2cf2126d52596cbaf1590c4728dc14815bb8fdbe24dcb176aa73

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp39-cp39-manylinux2014_aarch64.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp39-cp39-manylinux2014_aarch64.whl
  • Upload date:
  • Size: 3.1 MB
  • Tags: CPython 3.9
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp39-cp39-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 63087fdd819650905a394fbc560598a602911bed090f523cb8b11f650785ff37
MD5 cd81dba7f05fc289512197950a975f2a
BLAKE2b-256 3c1251e5c1f4b1fc4e9d427e31a8b6777167b2f4a5f3f89908c16a1baf43baf3

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp39-cp39-manylinux2010_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp39-cp39-manylinux2010_x86_64.whl
  • Upload date:
  • Size: 3.2 MB
  • Tags: CPython 3.9, manylinux: glibc 2.12+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp39-cp39-manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 36951ef070bd4dbdfe19d9555b86c85cd4bb63aa8e9e9cdaff1444e86823851e
MD5 b8a492e6d397c49ddbd3739d35e9fcb3
BLAKE2b-256 50ac72b645a501a56b988cdc7475e6b2097cd27432fdca49a50a13bfb41508f2

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp39-cp39-manylinux1_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp39-cp39-manylinux1_x86_64.whl
  • Upload date:
  • Size: 3.2 MB
  • Tags: CPython 3.9
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp39-cp39-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 2696e0e2a1b9a0a52938cca1c2347ac21c4bc49926e61744a29cff7a28bbe1af
MD5 4d03ca4ccd56e4853ecdd69d9ff2dd5d
BLAKE2b-256 d2a960f45d394e46de30d01b5ed7f8841887d0123551ba6c779a63d89c40ef7c

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp39-cp39-macosx_10_11_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp39-cp39-macosx_10_11_x86_64.whl
  • Upload date:
  • Size: 2.2 MB
  • Tags: CPython 3.9, macOS 10.11+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp39-cp39-macosx_10_11_x86_64.whl
Algorithm Hash digest
SHA256 0052ee882afab3883cc1afd6f4ab6a18e6ace5f95651536a76c0f30c2487abd7
MD5 be3bd2d665f470a3f8d11b0b3c739c75
BLAKE2b-256 11ee4e8e693d696f98953076638ab47f54304dc90641d105e81e59eecf5b41e7

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 2.0 MB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 84024dcb0a81f1910deaac011bfc6d3113ebdd782f11d0189dc06955e2608276
MD5 fb35d06ec851a2198005d00c23044f36
BLAKE2b-256 c793bdc4ec68997aef9ced30a900ca2bea4aaa4e0bdc5444370f34de48e490e5

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp38-cp38-win32.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp38-cp38-win32.whl
  • Upload date:
  • Size: 1.8 MB
  • Tags: CPython 3.8, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp38-cp38-win32.whl
Algorithm Hash digest
SHA256 0e0ddc892bc354e86a244459a2210356c57753927088bd84dc1c4fa24ddf98d9
MD5 688436435371117e46368841489309a5
BLAKE2b-256 fb8e51336d52a9545671f99244d770d8ef4c2986cdafd49e4d5f71d4907d2d8f

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp38-cp38-manylinux2014_s390x.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp38-cp38-manylinux2014_s390x.whl
  • Upload date:
  • Size: 3.7 MB
  • Tags: CPython 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp38-cp38-manylinux2014_s390x.whl
Algorithm Hash digest
SHA256 8bb39ea0a2ae04521e17fd58a30c6e6c452f814969178b771ac836b7f1254b0c
MD5 1bfc06465516639ad4c97cb27059163b
BLAKE2b-256 3e1a738e1e7972cb3f28cadc577530b65d81bfdf77d74d8c542604f9ecaae1ee

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp38-cp38-manylinux2014_ppc64le.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp38-cp38-manylinux2014_ppc64le.whl
  • Upload date:
  • Size: 3.6 MB
  • Tags: CPython 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp38-cp38-manylinux2014_ppc64le.whl
Algorithm Hash digest
SHA256 02ee33a636b6389aea8410a8115c03de7207a8a8cccbe3e3d010a11249e51adc
MD5 436a7d5513f3109cae64f60f1c6d9103
BLAKE2b-256 f9447adf50e9d9b034dc25a87b7fd6805d6e917c7dcfe90592b634b9d202f3b9

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp38-cp38-manylinux2014_aarch64.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp38-cp38-manylinux2014_aarch64.whl
  • Upload date:
  • Size: 3.1 MB
  • Tags: CPython 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp38-cp38-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 c0046cad036e0eab09149296d515dd026bbc936b1bbd1a3d05a1aea2836d2839
MD5 98e216215031075668f46991713b29a5
BLAKE2b-256 9e7faa9fd30bcb1c6a2a9cfbd559dfdd20978d9e71a89b9684c8da38cbdd140e

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp38-cp38-manylinux2010_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp38-cp38-manylinux2010_x86_64.whl
  • Upload date:
  • Size: 3.2 MB
  • Tags: CPython 3.8, manylinux: glibc 2.12+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp38-cp38-manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 b2a120fd79ca8453f8419389408132ad0ba682c40e2f2d4d30a375dbe849d79f
MD5 a4f0a1b207b1f53f6b5302343a96fef5
BLAKE2b-256 0ddeaa8ed5030ac7bdb06114bb56e3b9aefaba38d939636abca4b0a461f85a9c

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp38-cp38-manylinux1_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp38-cp38-manylinux1_x86_64.whl
  • Upload date:
  • Size: 3.2 MB
  • Tags: CPython 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp38-cp38-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 cbaf9612a274e96a01dea0b620759242d429a25de9b9b3deac86381547cac492
MD5 391450ae6bd1875a780299a93598fc1f
BLAKE2b-256 1d8f585ea05ea23f9e3fd5b17af34b917846d5960d9185305b3ee4c7567442b2

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp38-cp38-macosx_10_11_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp38-cp38-macosx_10_11_x86_64.whl
  • Upload date:
  • Size: 2.2 MB
  • Tags: CPython 3.8, macOS 10.11+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp38-cp38-macosx_10_11_x86_64.whl
Algorithm Hash digest
SHA256 7d5c059d28530ee270d6c1317f6e83b2c5b89fd923c68be24770b28a9041eba4
MD5 428c1f5c7725951b0987303bd78d6316
BLAKE2b-256 70659e9bba4bc5c6264159d9ed86f20902f8d54ca00fd09b17e8605fe463f6fc

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 2.0 MB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 2b894be94fc89ffb336d759dc6cc51c3e530d524a9b84c22e9f5dd4e4a7af486
MD5 ab839968bf0355ca4d1610505485e8e5
BLAKE2b-256 630e63b6dfdb07ca4640a1742186fcd6b988eb40d3946536486670f00e010fcc

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp37-cp37m-win32.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp37-cp37m-win32.whl
  • Upload date:
  • Size: 1.8 MB
  • Tags: CPython 3.7m, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp37-cp37m-win32.whl
Algorithm Hash digest
SHA256 d94465e5b01d08d31ec610f3f6a614b2ec5d3561b589fb98f0326d038b286032
MD5 e81299504a50dda2fadbafb51ca090bd
BLAKE2b-256 4bafaf44e4f448e77900396c64a995fe29b1a43749ad792c2139ee6250d2aba0

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp37-cp37m-manylinux2014_s390x.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp37-cp37m-manylinux2014_s390x.whl
  • Upload date:
  • Size: 3.7 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp37-cp37m-manylinux2014_s390x.whl
Algorithm Hash digest
SHA256 d7fad5b89f10a64fe167bca195d17e27e090da6975209503204846fb9725c87e
MD5 d3036e1df748ec49b1fd0852dd14eeb2
BLAKE2b-256 a8bf22c30980e25846cb248a4ca47aa908cae348381c2ec747a341cd28f85dd7

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp37-cp37m-manylinux2014_ppc64le.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp37-cp37m-manylinux2014_ppc64le.whl
  • Upload date:
  • Size: 3.6 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp37-cp37m-manylinux2014_ppc64le.whl
Algorithm Hash digest
SHA256 53b64db1228de6bde758e6cf14fefd0e5e51681ebcae6a7052491d281f30e54e
MD5 62116dacd45867eb7921cf6ef0918523
BLAKE2b-256 bf9085ab0fc5a13b7a56ae8ee0fbde76f6ec8d09818672a77b57bc717f8b2f3a

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp37-cp37m-manylinux2014_aarch64.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp37-cp37m-manylinux2014_aarch64.whl
  • Upload date:
  • Size: 3.1 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp37-cp37m-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 8cdb3aa80cf0c9e9e2aee9fa1595398bb7cc9c800947d980c4604204cf60c408
MD5 d40eee54a5d1f558b7cef917ad10680b
BLAKE2b-256 3505f5ab7a39ca3dd489e146017ffa68ba673bc654ed51df51e209590c9b8381

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp37-cp37m-manylinux2010_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp37-cp37m-manylinux2010_x86_64.whl
  • Upload date:
  • Size: 3.2 MB
  • Tags: CPython 3.7m, manylinux: glibc 2.12+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp37-cp37m-manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 aa7b6acdc31a6b9b9c7a066543704d6e3533b39e1777db9bf9cb48701b24760a
MD5 60ef2f55ae316580ff24fe8221e358cc
BLAKE2b-256 8078db38bf74b6ff3dcc5cb3974c250ee3fdfcb98640a5d8b3629f4c06d929ef

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp37-cp37m-manylinux1_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp37-cp37m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 3.2 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp37-cp37m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 d8bcd57d3a6ff4689e897782b77687b104e1a412a94702662f2948caf4fee978
MD5 c390662a49ac70fd4c3bcf157b803b92
BLAKE2b-256 5b90e6c5b48a9fb45ecbfc7c23ab4bbb582654d2b91b07721ded336332548bd9

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp37-cp37m-macosx_10_11_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp37-cp37m-macosx_10_11_x86_64.whl
  • Upload date:
  • Size: 2.2 MB
  • Tags: CPython 3.7m, macOS 10.11+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp37-cp37m-macosx_10_11_x86_64.whl
Algorithm Hash digest
SHA256 af9171fc86da257f6cbe75e2569f5edac6e85039506bdb0fb5b575a7cdf1b60b
MD5 953494bf11b2c569c61d6aebc8400bd3
BLAKE2b-256 82679689440512612495d0594815cabdc1a52653de88201979793a9d5ecf4510

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 2.0 MB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 2abf4e131556670ea0e420f9cd32928a6590206e5ff262321b4a5474be1b4dcc
MD5 a8694b0d519c9033b99c2d01964f5c89
BLAKE2b-256 92f0303d80efc7bb54420b86328237fccdad7f5b15899351ab265dd3bb435815

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp36-cp36m-win32.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp36-cp36m-win32.whl
  • Upload date:
  • Size: 1.8 MB
  • Tags: CPython 3.6m, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp36-cp36m-win32.whl
Algorithm Hash digest
SHA256 32f36aea2a4345114a1cbe718680acd9258ac49b5cedbdb9802bb2af77159cb5
MD5 e53a35a0ad8a8115cffa18e63f31cfc9
BLAKE2b-256 49afc3ad58be0a3e1598fa0ee6791859cd77b6c2d266bb4ac684f403291cb366

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp36-cp36m-manylinux2014_s390x.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp36-cp36m-manylinux2014_s390x.whl
  • Upload date:
  • Size: 3.7 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp36-cp36m-manylinux2014_s390x.whl
Algorithm Hash digest
SHA256 adfbb053848e06a8199afd18d17c83f8977a5e12881882a03c325909b75cdef7
MD5 2aba356f54c36cb309b8bcb54bfb56d4
BLAKE2b-256 b3a7c812230405fb88030c987efab1eaed4cde4c40ac9a5e1bb86526e62a86bd

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp36-cp36m-manylinux2014_ppc64le.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp36-cp36m-manylinux2014_ppc64le.whl
  • Upload date:
  • Size: 3.6 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp36-cp36m-manylinux2014_ppc64le.whl
Algorithm Hash digest
SHA256 cd636389f4de084d7cd4c0a872799dbe60e1c4df4af9e85bb2020e491357422e
MD5 4c7547d9e2aa2c4edf905c50e8597902
BLAKE2b-256 ff0e9cbb7a669c96f238c06ab4ae25183bf8a9d87b9ce4ef0a984a888d1b6fe5

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp36-cp36m-manylinux2014_aarch64.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp36-cp36m-manylinux2014_aarch64.whl
  • Upload date:
  • Size: 3.1 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp36-cp36m-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 966b7b9de277bd142a14329c9af9e51e5c30ae5cb75a96114b5e894b5f382953
MD5 d912a3d77aae9fa266abfd7b06c7a330
BLAKE2b-256 eb92d718910fab0c36a7ed25d0e849c4d16de20d39474129fbc05ba0983c2d53

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp36-cp36m-manylinux2010_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp36-cp36m-manylinux2010_x86_64.whl
  • Upload date:
  • Size: 3.2 MB
  • Tags: CPython 3.6m, manylinux: glibc 2.12+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp36-cp36m-manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 90cb3a219b56794f05a4c947b6e9b6e5fe6bfc8786a96f8b9f8f5b737de1f119
MD5 9bf5e0ee190f768b14ac42c379b486ec
BLAKE2b-256 193d7d02fdd8ff736fc171b8a6e81edba23defa2e3d37672ece1809b7feea819

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp36-cp36m-manylinux1_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp36-cp36m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 3.2 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp36-cp36m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 877cebaca36eea1c8ac41cda7deddfe9e43e8423de86a72eff5e06a2cf804c62
MD5 44679a5f9d6178873bbeb8961c77347a
BLAKE2b-256 88c19d27c200820d2999b347c4e3f7e519d1cbdbda8e8eeea77d84bfeb02371c

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp36-cp36m-macosx_10_11_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp36-cp36m-macosx_10_11_x86_64.whl
  • Upload date:
  • Size: 2.2 MB
  • Tags: CPython 3.6m, macOS 10.11+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp36-cp36m-macosx_10_11_x86_64.whl
Algorithm Hash digest
SHA256 7ddcf6e01a0cb9b32a0916c87fb28707220fd76885f73d936cb3778a11fff29e
MD5 5aafd812e82c3f4470965c62c965e483
BLAKE2b-256 7b172345a43d031da8f9ba10eca3782c2319190143e1e2759ed478ca1a82ed01

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp35-cp35m-win_amd64.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp35-cp35m-win_amd64.whl
  • Upload date:
  • Size: 2.0 MB
  • Tags: CPython 3.5m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp35-cp35m-win_amd64.whl
Algorithm Hash digest
SHA256 3c4e731b9eaf0c8ab609e611cf954f66406681861056358bab4df81a01a4eaf3
MD5 e029e85c9cd936909ca68eb605fddf50
BLAKE2b-256 46aaeacbef6f945ef45977b6ada88be2594017a3769d177a8fb4d3fd81a87b3c

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp35-cp35m-win32.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp35-cp35m-win32.whl
  • Upload date:
  • Size: 1.8 MB
  • Tags: CPython 3.5m, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp35-cp35m-win32.whl
Algorithm Hash digest
SHA256 32e4a8097276fcfba07acfb5038b64068cacef46b0bc6f79f9e484a43ac19922
MD5 80692eb099d261775d7f5253eca69917
BLAKE2b-256 d184d08f69d9c4622fb2bcbde02cd3d6058606d04c6d5c74ea032d03f9709629

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp35-cp35m-manylinux2014_s390x.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp35-cp35m-manylinux2014_s390x.whl
  • Upload date:
  • Size: 3.7 MB
  • Tags: CPython 3.5m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp35-cp35m-manylinux2014_s390x.whl
Algorithm Hash digest
SHA256 641a8df7274945e5eb4c8af2c077db3aba0e6726453923dda30fe0849fa552bf
MD5 f6b2e990339bda59ff8574cf81647a6d
BLAKE2b-256 7b647d765e48d0b2b06378a0912014369eb70433344ad6a4886a0a2ed6620fe7

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp35-cp35m-manylinux2014_ppc64le.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp35-cp35m-manylinux2014_ppc64le.whl
  • Upload date:
  • Size: 3.6 MB
  • Tags: CPython 3.5m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp35-cp35m-manylinux2014_ppc64le.whl
Algorithm Hash digest
SHA256 a1eb5df6cc3af0418c39cb432cdc5f95da2e6c628ffad9c46653324d8d39462b
MD5 398fbacc17860b15f221d623e912cd52
BLAKE2b-256 f632f2833c4e5de1019038f509259ec2ff1d3edd93c2c5b8d37261b73ed62341

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp35-cp35m-manylinux2014_aarch64.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp35-cp35m-manylinux2014_aarch64.whl
  • Upload date:
  • Size: 3.1 MB
  • Tags: CPython 3.5m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp35-cp35m-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 3cc95b44c1f6f79c73b3685ab92841a8ce1dd6a9ef22df811f80e400c09b54c8
MD5 23ef1e86270adf1e89b550e629074693
BLAKE2b-256 7dd9d33225d17807847ba9f708680f3949d2e30881d6a9b28f7bf14851357835

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp35-cp35m-manylinux2010_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp35-cp35m-manylinux2010_x86_64.whl
  • Upload date:
  • Size: 3.2 MB
  • Tags: CPython 3.5m, manylinux: glibc 2.12+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp35-cp35m-manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 bbe6bb111cb011a97a246f6ab530dadd92d9edaa1e0fdc09a0ce472172a7dd78
MD5 bf58776b75369d74524c843b1c71e9bc
BLAKE2b-256 08ba213c5a7a7ebd122e693d0472f9b9d258de0257aecf2dbc8165d2394d13e1

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp35-cp35m-manylinux1_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp35-cp35m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 3.2 MB
  • Tags: CPython 3.5m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp35-cp35m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 18dcbaca0517f5bfcfdb7ea6175d6a4afc0ef430e59dc738a842106d320d001f
MD5 bdf7f5af11436e0c8c4d100f73de4d32
BLAKE2b-256 26c751e48695c332333720668737656939b8d692caf74a621c010c9e04fe170e

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.0-cp35-cp35m-macosx_10_11_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.10.0-cp35-cp35m-macosx_10_11_x86_64.whl
  • Upload date:
  • Size: 2.2 MB
  • Tags: CPython 3.5m, macOS 10.11+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.0-cp35-cp35m-macosx_10_11_x86_64.whl
Algorithm Hash digest
SHA256 bca56bce9b0466dbb4a69995c4b9098434879ab20caf7aec3a065119a26df31a
MD5 7d6f61e6c42d72d834555fd174184593
BLAKE2b-256 91146d81f7535bc9e9ec073df40c5be4dedc57c93cc1618d96072678e6962668

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page