Skip to main content

Fast and Customizable Tokenizers

Project description



Build GitHub


Tokenizers

Provides an implementation of today's most used tokenizers, with a focus on performance and versatility.

Bindings over the Rust implementation. If you are interested in the High-level design, you can go check it there.

Otherwise, let's dive in!

Main features:

  • Train new vocabularies and tokenize using 4 pre-made tokenizers (Bert WordPiece and the 3 most common BPE versions).
  • Extremely fast (both training and tokenization), thanks to the Rust implementation. Takes less than 20 seconds to tokenize a GB of text on a server's CPU.
  • Easy to use, but also extremely versatile.
  • Designed for research and production.
  • Normalization comes with alignments tracking. It's always possible to get the part of the original sentence that corresponds to a given token.
  • Does all the pre-processing: Truncate, Pad, add the special tokens your model needs.

Installation

With pip:

pip install tokenizers

From sources:

To use this method, you need to have the Rust installed:

# Install with:
curl https://sh.rustup.rs -sSf | sh -s -- -y
export PATH="$HOME/.cargo/bin:$PATH"

Once Rust is installed, you can compile doing the following

git clone https://github.com/huggingface/tokenizers
cd tokenizers/bindings/python

# Create a virtual env (you can use yours as well)
python -m venv .env
source .env/bin/activate

# Install `tokenizers` in the current virtual env
pip install setuptools_rust
python setup.py install

Load a pretrained tokenizer from the Hub

from tokenizers import Tokenizer

tokenizer = Tokenizer.from_pretrained("bert-base-cased")

Using the provided Tokenizers

We provide some pre-build tokenizers to cover the most common cases. You can easily load one of these using some vocab.json and merges.txt files:

from tokenizers import CharBPETokenizer

# Initialize a tokenizer
vocab = "./path/to/vocab.json"
merges = "./path/to/merges.txt"
tokenizer = CharBPETokenizer(vocab, merges)

# And then encode:
encoded = tokenizer.encode("I can feel the magic, can you?")
print(encoded.ids)
print(encoded.tokens)

And you can train them just as simply:

from tokenizers import CharBPETokenizer

# Initialize a tokenizer
tokenizer = CharBPETokenizer()

# Then train it!
tokenizer.train([ "./path/to/files/1.txt", "./path/to/files/2.txt" ])

# Now, let's use it:
encoded = tokenizer.encode("I can feel the magic, can you?")

# And finally save it somewhere
tokenizer.save("./path/to/directory/my-bpe.tokenizer.json")

Provided Tokenizers

  • CharBPETokenizer: The original BPE
  • ByteLevelBPETokenizer: The byte level version of the BPE
  • SentencePieceBPETokenizer: A BPE implementation compatible with the one used by SentencePiece
  • BertWordPieceTokenizer: The famous Bert tokenizer, using WordPiece

All of these can be used and trained as explained above!

Build your own

Whenever these provided tokenizers don't give you enough freedom, you can build your own tokenizer, by putting all the different parts you need together. You can check how we implemented the provided tokenizers and adapt them easily to your own needs.

Building a byte-level BPE

Here is an example showing how to build your own byte-level BPE by putting all the different pieces together, and then saving it to a single file:

from tokenizers import Tokenizer, models, pre_tokenizers, decoders, trainers, processors

# Initialize a tokenizer
tokenizer = Tokenizer(models.BPE())

# Customize pre-tokenization and decoding
tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=True)
tokenizer.decoder = decoders.ByteLevel()
tokenizer.post_processor = processors.ByteLevel(trim_offsets=True)

# And then train
trainer = trainers.BpeTrainer(
    vocab_size=20000,
    min_frequency=2,
    initial_alphabet=pre_tokenizers.ByteLevel.alphabet()
)
tokenizer.train([
    "./path/to/dataset/1.txt",
    "./path/to/dataset/2.txt",
    "./path/to/dataset/3.txt"
], trainer=trainer)

# And Save it
tokenizer.save("byte-level-bpe.tokenizer.json", pretty=True)

Now, when you want to use this tokenizer, this is as simple as:

from tokenizers import Tokenizer

tokenizer = Tokenizer.from_file("byte-level-bpe.tokenizer.json")

encoded = tokenizer.encode("I can feel the magic, can you?")

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tokenizers-0.11.5.tar.gz (220.0 kB view details)

Uploaded Source

Built Distributions

tokenizers-0.11.5-cp310-cp310-win_amd64.whl (3.3 MB view details)

Uploaded CPython 3.10 Windows x86-64

tokenizers-0.11.5-cp310-cp310-win32.whl (3.0 MB view details)

Uploaded CPython 3.10 Windows x86

tokenizers-0.11.5-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl (7.8 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ s390x

tokenizers-0.11.5-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (8.1 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ppc64le

tokenizers-0.11.5-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (7.0 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ARM64

tokenizers-0.11.5-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (6.8 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.12+ x86-64

tokenizers-0.11.5-cp310-cp310-macosx_10_11_x86_64.whl (3.7 MB view details)

Uploaded CPython 3.10 macOS 10.11+ x86-64

tokenizers-0.11.5-cp39-cp39-win_amd64.whl (3.3 MB view details)

Uploaded CPython 3.9 Windows x86-64

tokenizers-0.11.5-cp39-cp39-win32.whl (3.0 MB view details)

Uploaded CPython 3.9 Windows x86

tokenizers-0.11.5-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl (7.8 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ s390x

tokenizers-0.11.5-cp39-cp39-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (8.1 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ppc64le

tokenizers-0.11.5-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (7.0 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64

tokenizers-0.11.5-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (6.8 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.12+ x86-64

tokenizers-0.11.5-cp39-cp39-macosx_10_11_x86_64.whl (3.7 MB view details)

Uploaded CPython 3.9 macOS 10.11+ x86-64

tokenizers-0.11.5-cp38-cp38-win_amd64.whl (3.3 MB view details)

Uploaded CPython 3.8 Windows x86-64

tokenizers-0.11.5-cp38-cp38-win32.whl (3.0 MB view details)

Uploaded CPython 3.8 Windows x86

tokenizers-0.11.5-cp38-cp38-manylinux_2_17_s390x.manylinux2014_s390x.whl (7.8 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ s390x

tokenizers-0.11.5-cp38-cp38-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (8.1 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ppc64le

tokenizers-0.11.5-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (7.0 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARM64

tokenizers-0.11.5-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (6.8 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.12+ x86-64

tokenizers-0.11.5-cp38-cp38-macosx_10_11_x86_64.whl (3.7 MB view details)

Uploaded CPython 3.8 macOS 10.11+ x86-64

tokenizers-0.11.5-cp37-cp37m-win_amd64.whl (3.3 MB view details)

Uploaded CPython 3.7m Windows x86-64

tokenizers-0.11.5-cp37-cp37m-win32.whl (3.0 MB view details)

Uploaded CPython 3.7m Windows x86

tokenizers-0.11.5-cp37-cp37m-manylinux_2_17_s390x.manylinux2014_s390x.whl (7.8 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ s390x

tokenizers-0.11.5-cp37-cp37m-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (8.1 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ ppc64le

tokenizers-0.11.5-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (7.0 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ ARM64

tokenizers-0.11.5-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (6.8 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.12+ x86-64

tokenizers-0.11.5-cp37-cp37m-macosx_10_11_x86_64.whl (3.7 MB view details)

Uploaded CPython 3.7m macOS 10.11+ x86-64

tokenizers-0.11.5-cp36-cp36m-manylinux_2_17_s390x.manylinux2014_s390x.whl (7.8 MB view details)

Uploaded CPython 3.6m manylinux: glibc 2.17+ s390x

tokenizers-0.11.5-cp36-cp36m-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (8.1 MB view details)

Uploaded CPython 3.6m manylinux: glibc 2.17+ ppc64le

tokenizers-0.11.5-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (7.0 MB view details)

Uploaded CPython 3.6m manylinux: glibc 2.17+ ARM64

tokenizers-0.11.5-cp36-cp36m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (6.8 MB view details)

Uploaded CPython 3.6m manylinux: glibc 2.12+ x86-64

File details

Details for the file tokenizers-0.11.5.tar.gz.

File metadata

  • Download URL: tokenizers-0.11.5.tar.gz
  • Upload date:
  • Size: 220.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.3 CPython/3.10.2

File hashes

Hashes for tokenizers-0.11.5.tar.gz
Algorithm Hash digest
SHA256 8ec3f22c0bcad5d2cfe295adff70981b7a3c09bbd577f467718f2edaf0f391c9
MD5 d7ae6ec648695412fdd76bcda1df7b59
BLAKE2b-256 676479189b3d1d91211fd966561aa73cff186c3217a7c65b4bc460fe1a7276d3

See more details on using hashes here.

File details

Details for the file tokenizers-0.11.5-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: tokenizers-0.11.5-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 3.3 MB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.3 CPython/3.10.2

File hashes

Hashes for tokenizers-0.11.5-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 8c6fc9db82a366262ea1cb085fbc25871b089d816caa3f1e1aa5e1e7eb9115eb
MD5 dff836f6fefbed0121e72a2a48fe9d07
BLAKE2b-256 d32e6e69b329df29af40e847e2384f6b6d063fea9a2142ca95ffe64b5f8c7adf

See more details on using hashes here.

File details

Details for the file tokenizers-0.11.5-cp310-cp310-win32.whl.

File metadata

  • Download URL: tokenizers-0.11.5-cp310-cp310-win32.whl
  • Upload date:
  • Size: 3.0 MB
  • Tags: CPython 3.10, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.3 CPython/3.10.2

File hashes

Hashes for tokenizers-0.11.5-cp310-cp310-win32.whl
Algorithm Hash digest
SHA256 b9bbad492390d80c912f5b26d81c8cb6dd34bc08d701a9b48a5e7de6fa508c67
MD5 5928eee3aaccfeb9fd0455d772b35044
BLAKE2b-256 107531db91e9cce354c0a5836047eccc3fd53a041eda4a394c20042fe57d4527

See more details on using hashes here.

File details

Details for the file tokenizers-0.11.5-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl.

File metadata

  • Download URL: tokenizers-0.11.5-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl
  • Upload date:
  • Size: 7.8 MB
  • Tags: CPython 3.10, manylinux: glibc 2.17+ s390x
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.3 CPython/3.10.2

File hashes

Hashes for tokenizers-0.11.5-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm Hash digest
SHA256 c8dea772f46119de0a520449140d215e2e4d52d4dfa828cc1410712c86f2733b
MD5 df2245bc5541e96dbfe80c34dbf97154
BLAKE2b-256 fe37009a5f7c5d16fb994b435368ca8804a547c33b87ffdbe1bf8c3e19bffff0

See more details on using hashes here.

File details

Details for the file tokenizers-0.11.5-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl.

File metadata

  • Download URL: tokenizers-0.11.5-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
  • Upload date:
  • Size: 8.1 MB
  • Tags: CPython 3.10, manylinux: glibc 2.17+ ppc64le
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.3 CPython/3.10.2

File hashes

Hashes for tokenizers-0.11.5-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm Hash digest
SHA256 6b82536f6ede23bdcb659a452c6742ce433431c791506e4c8162b7001b78f53f
MD5 0c8feb5e910c4c8650d6c03a77398372
BLAKE2b-256 5ca6253bbc7b431df335b119181a542a9d29658c4737d30f71cd9c7a3b72dc0e

See more details on using hashes here.

File details

Details for the file tokenizers-0.11.5-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

  • Download URL: tokenizers-0.11.5-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
  • Upload date:
  • Size: 7.0 MB
  • Tags: CPython 3.10, manylinux: glibc 2.17+ ARM64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.3 CPython/3.10.2

File hashes

Hashes for tokenizers-0.11.5-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 c7b38a72685375d0aafb339ff8f45ee905d9e51b230fac7935d85839a5c7febe
MD5 bd741a330ef097df299d24763bde4e81
BLAKE2b-256 ee15ac3dfd94a3fff27817a7cef309b8c49557e55286a9348dbba454b4d7e67d

See more details on using hashes here.

File details

Details for the file tokenizers-0.11.5-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.11.5-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
  • Upload date:
  • Size: 6.8 MB
  • Tags: CPython 3.10, manylinux: glibc 2.12+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.3 CPython/3.10.2

File hashes

Hashes for tokenizers-0.11.5-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 2f3419390f9a61cbbcbdb68ef679ef11e15f8c7335fbd5918275b53c05505bb4
MD5 f27deb6d9ba2dae9ab89f4d604e8a80f
BLAKE2b-256 74c7df928632fe7dfbe931ab35b9cf293b348ba03f92f6f87572f0893c3a8953

See more details on using hashes here.

File details

Details for the file tokenizers-0.11.5-cp310-cp310-macosx_10_11_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.11.5-cp310-cp310-macosx_10_11_x86_64.whl
  • Upload date:
  • Size: 3.7 MB
  • Tags: CPython 3.10, macOS 10.11+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.3 CPython/3.10.2

File hashes

Hashes for tokenizers-0.11.5-cp310-cp310-macosx_10_11_x86_64.whl
Algorithm Hash digest
SHA256 cd4c23fdb0ec608a109d7a246abc00ee1a87eee5d5ed5b40bd5c7d9edc2f242f
MD5 7374b46fc76477f16d26599e3270de7c
BLAKE2b-256 550bc150b25d3aa083e648d6a7f9b545f20ecb30f076ec679cb8db0be266d687

See more details on using hashes here.

File details

Details for the file tokenizers-0.11.5-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: tokenizers-0.11.5-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 3.3 MB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.3 CPython/3.10.2

File hashes

Hashes for tokenizers-0.11.5-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 2dadbac4356ccb389a996fde1ed2933653d26698dc6dec49a163e168e3e95baf
MD5 fa8dcb1830f62cadf11fd6ef0a616667
BLAKE2b-256 f186607c6b46f3cf1e58b2b0cde22e737cb6cd80c085d8a5856e2094e64d9a0c

See more details on using hashes here.

File details

Details for the file tokenizers-0.11.5-cp39-cp39-win32.whl.

File metadata

  • Download URL: tokenizers-0.11.5-cp39-cp39-win32.whl
  • Upload date:
  • Size: 3.0 MB
  • Tags: CPython 3.9, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.3 CPython/3.10.2

File hashes

Hashes for tokenizers-0.11.5-cp39-cp39-win32.whl
Algorithm Hash digest
SHA256 08e0c6c423cab65c1462a2d82af42fe2220a4b9e202f422c4a2c9280d6a3bd4c
MD5 7b16b484939a8157a3807dbee0dfbdd7
BLAKE2b-256 aaf50f406af1e1c857c1e5f91c5a5240fbcf80df79417e4a5a4a715a3759cf8b

See more details on using hashes here.

File details

Details for the file tokenizers-0.11.5-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl.

File metadata

  • Download URL: tokenizers-0.11.5-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl
  • Upload date:
  • Size: 7.8 MB
  • Tags: CPython 3.9, manylinux: glibc 2.17+ s390x
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.3 CPython/3.10.2

File hashes

Hashes for tokenizers-0.11.5-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm Hash digest
SHA256 ad2902602c5518fd5e065eb6185736dddc775be5133a61a392a798062f9c1b8d
MD5 9e84581941d45fa2d6687d40a6f313a1
BLAKE2b-256 6a802aa140a221f603af4ff3a53da7c174bfd9666997835be4606c7705c81f78

See more details on using hashes here.

File details

Details for the file tokenizers-0.11.5-cp39-cp39-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl.

File metadata

  • Download URL: tokenizers-0.11.5-cp39-cp39-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
  • Upload date:
  • Size: 8.1 MB
  • Tags: CPython 3.9, manylinux: glibc 2.17+ ppc64le
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.3 CPython/3.10.2

File hashes

Hashes for tokenizers-0.11.5-cp39-cp39-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm Hash digest
SHA256 95e71dcd1322c09ce5b0c85d2a47442d1a92b735e20bacf6b36191876fe5c156
MD5 111dc9fe2ce1a41093680b87b475d3d0
BLAKE2b-256 41cc12dd137df9705ec8cf9c5b40567ec6fc87b64a4130586611550181980532

See more details on using hashes here.

File details

Details for the file tokenizers-0.11.5-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

  • Download URL: tokenizers-0.11.5-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
  • Upload date:
  • Size: 7.0 MB
  • Tags: CPython 3.9, manylinux: glibc 2.17+ ARM64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.3 CPython/3.10.2

File hashes

Hashes for tokenizers-0.11.5-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 f53f7dc79720ce8aac006269c18698fa431b9b15d3437bc18e307650f012c19f
MD5 c5fc3f896425b8795baa573eb3536b60
BLAKE2b-256 cdbe242ae33aa30d2ab85d69289a94f259fd0f43e6eb41881b6364498331e35b

See more details on using hashes here.

File details

Details for the file tokenizers-0.11.5-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.11.5-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
  • Upload date:
  • Size: 6.8 MB
  • Tags: CPython 3.9, manylinux: glibc 2.12+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.3 CPython/3.10.2

File hashes

Hashes for tokenizers-0.11.5-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 f670895357c7134efc4b44d2384e6dcac719cc2047848767d2fdaf013a746357
MD5 d9ff893c68addfd1b3b7d4657ac1dcd5
BLAKE2b-256 bdd068e11a1968b05f7fbb1ede6e09ec49c3dcacfe8d10755b75dc5dd3fbf33f

See more details on using hashes here.

File details

Details for the file tokenizers-0.11.5-cp39-cp39-macosx_10_11_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.11.5-cp39-cp39-macosx_10_11_x86_64.whl
  • Upload date:
  • Size: 3.7 MB
  • Tags: CPython 3.9, macOS 10.11+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.3 CPython/3.10.2

File hashes

Hashes for tokenizers-0.11.5-cp39-cp39-macosx_10_11_x86_64.whl
Algorithm Hash digest
SHA256 e1886178704f20a628ff754abe3100079698323cfb093c21b40f7f7847bbd2f0
MD5 ffdb830d6302d1206e2540a85228909d
BLAKE2b-256 058108a29133b697df55c2901dbfd570e507632f4f3351ab029b2f670858f73d

See more details on using hashes here.

File details

Details for the file tokenizers-0.11.5-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: tokenizers-0.11.5-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 3.3 MB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.3 CPython/3.10.2

File hashes

Hashes for tokenizers-0.11.5-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 8cd3d05e9c4340a176a37a49aed60b348a2068a4cf21bf1f6a1cdc6bcb4bebdf
MD5 7fa953e491db7d3aebd4f5d3db5c3188
BLAKE2b-256 36263e224f704d2d9d6ca7c7d9ead35c77c7a4c3333cd0484212f449d082b142

See more details on using hashes here.

File details

Details for the file tokenizers-0.11.5-cp38-cp38-win32.whl.

File metadata

  • Download URL: tokenizers-0.11.5-cp38-cp38-win32.whl
  • Upload date:
  • Size: 3.0 MB
  • Tags: CPython 3.8, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.3 CPython/3.10.2

File hashes

Hashes for tokenizers-0.11.5-cp38-cp38-win32.whl
Algorithm Hash digest
SHA256 4377d4995c6d2da4f6c657e9cb6edcc5a8fb71e278c20a3082724b4ebf35ba55
MD5 f4258cceac99aa2db6d11da69db26091
BLAKE2b-256 98b94c853cef7d58a177fe63d4acb7430e85def68729be3457fdecf1fca6528f

See more details on using hashes here.

File details

Details for the file tokenizers-0.11.5-cp38-cp38-manylinux_2_17_s390x.manylinux2014_s390x.whl.

File metadata

  • Download URL: tokenizers-0.11.5-cp38-cp38-manylinux_2_17_s390x.manylinux2014_s390x.whl
  • Upload date:
  • Size: 7.8 MB
  • Tags: CPython 3.8, manylinux: glibc 2.17+ s390x
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.3 CPython/3.10.2

File hashes

Hashes for tokenizers-0.11.5-cp38-cp38-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm Hash digest
SHA256 728f32efa6ae9ba3f0fe4f2d4618ac1162618e1cfca0e649aff7c7f2605d967c
MD5 cae38581c6bdfb348fb0497339530040
BLAKE2b-256 2be462e638405fd1cf4fb2672cb59053f9995af27bcf219a2468697b0ff14c79

See more details on using hashes here.

File details

Details for the file tokenizers-0.11.5-cp38-cp38-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl.

File metadata

  • Download URL: tokenizers-0.11.5-cp38-cp38-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
  • Upload date:
  • Size: 8.1 MB
  • Tags: CPython 3.8, manylinux: glibc 2.17+ ppc64le
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.3 CPython/3.10.2

File hashes

Hashes for tokenizers-0.11.5-cp38-cp38-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm Hash digest
SHA256 1df50ff10547629ff829eb2d90b1967f77f0432f24ef655730fd25969642053d
MD5 68885e0286abe76a07b4a0bf619b7d51
BLAKE2b-256 d832b29b3bdfbe57416890afef52e03ba844f8a13c405520e65f3e8cd53b8e75

See more details on using hashes here.

File details

Details for the file tokenizers-0.11.5-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

  • Download URL: tokenizers-0.11.5-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
  • Upload date:
  • Size: 7.0 MB
  • Tags: CPython 3.8, manylinux: glibc 2.17+ ARM64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.3 CPython/3.10.2

File hashes

Hashes for tokenizers-0.11.5-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 8c005425bf73863cb907f594850e8fbbf402328f852201aa0deeed5eea679299
MD5 d8e621a18aab2be79636c7d352788374
BLAKE2b-256 813738236d34a9658eaee0cfc87a4bdfaccbdc551e4cd48c82ad43b8ddeca8db

See more details on using hashes here.

File details

Details for the file tokenizers-0.11.5-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.11.5-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
  • Upload date:
  • Size: 6.8 MB
  • Tags: CPython 3.8, manylinux: glibc 2.12+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.3 CPython/3.10.2

File hashes

Hashes for tokenizers-0.11.5-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 8d5256e65aff6e3f882eba4f763526a1621167f96e8fbdec0b2a8d850dc5057c
MD5 ae3adc69572bd66d5d9534ecd34cbdf8
BLAKE2b-256 6105b03b5149df78fbee63b9ecb294819d80b454b22f7f583c091f8e581d2f91

See more details on using hashes here.

File details

Details for the file tokenizers-0.11.5-cp38-cp38-macosx_10_11_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.11.5-cp38-cp38-macosx_10_11_x86_64.whl
  • Upload date:
  • Size: 3.7 MB
  • Tags: CPython 3.8, macOS 10.11+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.3 CPython/3.10.2

File hashes

Hashes for tokenizers-0.11.5-cp38-cp38-macosx_10_11_x86_64.whl
Algorithm Hash digest
SHA256 be0a05707cf4a973a2dd2cff97ef80c11c0c383c9c56b80baf79ba5d4538b141
MD5 633af55e3a327f7d061ecdaf6d465c2b
BLAKE2b-256 7428b2fbc6e0cb1fb8ddfdc30ee85ea3c1173b7845b94613c8aeb614390b5f4c

See more details on using hashes here.

File details

Details for the file tokenizers-0.11.5-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: tokenizers-0.11.5-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 3.3 MB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.3 CPython/3.10.2

File hashes

Hashes for tokenizers-0.11.5-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 d17f5d1341267b58d2277a717678f97053671b0417c1bf0077fbbc6006b8f1ea
MD5 fd56bc90d81c7ca5b56b04a314e30c48
BLAKE2b-256 f9faf859df5f4e186ee5ad2d0e81d3da671efe81f0deb2ec2bb25528f5b07810

See more details on using hashes here.

File details

Details for the file tokenizers-0.11.5-cp37-cp37m-win32.whl.

File metadata

  • Download URL: tokenizers-0.11.5-cp37-cp37m-win32.whl
  • Upload date:
  • Size: 3.0 MB
  • Tags: CPython 3.7m, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.3 CPython/3.10.2

File hashes

Hashes for tokenizers-0.11.5-cp37-cp37m-win32.whl
Algorithm Hash digest
SHA256 411af5b673c6fc1734bf6692bc15eef5fd70f4931cd94b35ea4c23ad4a22e87e
MD5 9c5cf7e148efe63b68c2b39cbddda064
BLAKE2b-256 206e2f0adc079076bb7c2796286b679c92fcc974c077854ab231bc22e34ad9d4

See more details on using hashes here.

File details

Details for the file tokenizers-0.11.5-cp37-cp37m-manylinux_2_17_s390x.manylinux2014_s390x.whl.

File metadata

  • Download URL: tokenizers-0.11.5-cp37-cp37m-manylinux_2_17_s390x.manylinux2014_s390x.whl
  • Upload date:
  • Size: 7.8 MB
  • Tags: CPython 3.7m, manylinux: glibc 2.17+ s390x
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.3 CPython/3.10.2

File hashes

Hashes for tokenizers-0.11.5-cp37-cp37m-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm Hash digest
SHA256 e61bc01acf6ae5a41992b638260522a3775ad826f95bcc6cac38a0d7bc5d1fdd
MD5 1b3a84d9aa9a609080a47a1971f1c93f
BLAKE2b-256 b7586a4b4bb2e4f6bc302b6738d31e0f3628f40a410de2ccaae3c5ec5166c814

See more details on using hashes here.

File details

Details for the file tokenizers-0.11.5-cp37-cp37m-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl.

File metadata

  • Download URL: tokenizers-0.11.5-cp37-cp37m-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
  • Upload date:
  • Size: 8.1 MB
  • Tags: CPython 3.7m, manylinux: glibc 2.17+ ppc64le
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.3 CPython/3.10.2

File hashes

Hashes for tokenizers-0.11.5-cp37-cp37m-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm Hash digest
SHA256 d35d3355c30d18e09d29b36cb765588358b5c6a0b2461c7de5fa93ec20fb1faa
MD5 0f37fc22a11dedb16a416b09bfaf2e9b
BLAKE2b-256 1a0584b549292a4e4aab7e196b4d103c9c31b0651fa932d73c5747d46dd61ce9

See more details on using hashes here.

File details

Details for the file tokenizers-0.11.5-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

  • Download URL: tokenizers-0.11.5-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
  • Upload date:
  • Size: 7.0 MB
  • Tags: CPython 3.7m, manylinux: glibc 2.17+ ARM64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.3 CPython/3.10.2

File hashes

Hashes for tokenizers-0.11.5-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 5bc324040a9441fc59607641e0ca92d6393750e57cc7804478c815c1c27295af
MD5 4d27012b3dbd5034de30d7e7170ce29d
BLAKE2b-256 80a345f53b55f4ac8d07a13fba54137e6ffedcd2a299b37370b5512278f021fa

See more details on using hashes here.

File details

Details for the file tokenizers-0.11.5-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.11.5-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
  • Upload date:
  • Size: 6.8 MB
  • Tags: CPython 3.7m, manylinux: glibc 2.12+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.3 CPython/3.10.2

File hashes

Hashes for tokenizers-0.11.5-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 f3b4416486b75903f4760dd4560dd85c649e95f2a160d0f2d2ecec69ac94c684
MD5 2bb6920e1cea9b0a4aa2d35e315345e1
BLAKE2b-256 274d161a612daa2cdbf9a6b0024c2f30bfaaa0f54c4b387a4b4093323522cea0

See more details on using hashes here.

File details

Details for the file tokenizers-0.11.5-cp37-cp37m-macosx_10_11_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.11.5-cp37-cp37m-macosx_10_11_x86_64.whl
  • Upload date:
  • Size: 3.7 MB
  • Tags: CPython 3.7m, macOS 10.11+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.3 CPython/3.10.2

File hashes

Hashes for tokenizers-0.11.5-cp37-cp37m-macosx_10_11_x86_64.whl
Algorithm Hash digest
SHA256 e9c196aa0e555b78390c08b8a244dbc66697ac88167f6af88cca0c0f90a8dbc3
MD5 5e1638d475e47a7ee8a6c808cf442bd4
BLAKE2b-256 a4c064dc2d5111972cffeea6f89bfac80789d3b87b54db5e6c403dd339777915

See more details on using hashes here.

File details

Details for the file tokenizers-0.11.5-cp36-cp36m-manylinux_2_17_s390x.manylinux2014_s390x.whl.

File metadata

  • Download URL: tokenizers-0.11.5-cp36-cp36m-manylinux_2_17_s390x.manylinux2014_s390x.whl
  • Upload date:
  • Size: 7.8 MB
  • Tags: CPython 3.6m, manylinux: glibc 2.17+ s390x
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.3 CPython/3.10.2

File hashes

Hashes for tokenizers-0.11.5-cp36-cp36m-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm Hash digest
SHA256 c5b6289282ff5b1c6868e3b57c02168f8858a6deb584b50dc0e01830d7db7081
MD5 6cfebbfcf66a432874ec59fb3d4c428c
BLAKE2b-256 39d776d8182a63499102baa2d23d51d7a6dcadccf4baf975f3c19990c83bd8fc

See more details on using hashes here.

File details

Details for the file tokenizers-0.11.5-cp36-cp36m-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl.

File metadata

  • Download URL: tokenizers-0.11.5-cp36-cp36m-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
  • Upload date:
  • Size: 8.1 MB
  • Tags: CPython 3.6m, manylinux: glibc 2.17+ ppc64le
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.3 CPython/3.10.2

File hashes

Hashes for tokenizers-0.11.5-cp36-cp36m-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm Hash digest
SHA256 24d51c00202c950233049487db625dd4bae1f631719c5f7b7781c90db0f30a30
MD5 dff210e7033524a17736a1aec1f69fe5
BLAKE2b-256 b12aca0a567300437ead22131dc50d6ccc19c1bb384d972eb41ec4147a54d7b0

See more details on using hashes here.

File details

Details for the file tokenizers-0.11.5-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

  • Download URL: tokenizers-0.11.5-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
  • Upload date:
  • Size: 7.0 MB
  • Tags: CPython 3.6m, manylinux: glibc 2.17+ ARM64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.3 CPython/3.10.2

File hashes

Hashes for tokenizers-0.11.5-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 372cb851d2a95f6573fe40e74fafaa97fe48883e97cca588311a4da89772924e
MD5 a80aa4450da3003d37cc7bd8b3b5a75b
BLAKE2b-256 adc3ad730269f0aafe84c724a7808473c7cf322d4ca546a7eb2eee64815f9dda

See more details on using hashes here.

File details

Details for the file tokenizers-0.11.5-cp36-cp36m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.11.5-cp36-cp36m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
  • Upload date:
  • Size: 6.8 MB
  • Tags: CPython 3.6m, manylinux: glibc 2.12+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/32.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.62.3 importlib-metadata/4.11.1 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.3 CPython/3.10.2

File hashes

Hashes for tokenizers-0.11.5-cp36-cp36m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 0d749400bb91c614e32d56c27840a51a12cda8bc62f258e7666840380a65ae6f
MD5 7988594f2f1d83d7d204b3b946b22f3e
BLAKE2b-256 bae5f32f0f8c652d1e676b14fffcc7562c3207b3def0b7591cf047191ed0bbf8

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page