Skip to main content

Fast and Customizable Tokenizers

Project description



Build GitHub


Tokenizers

Provides an implementation of today's most used tokenizers, with a focus on performance and versatility.

Bindings over the Rust implementation. If you are interested in the High-level design, you can go check it there.

Otherwise, let's dive in!

Main features:

  • Train new vocabularies and tokenize using 4 pre-made tokenizers (Bert WordPiece and the 3 most common BPE versions).
  • Extremely fast (both training and tokenization), thanks to the Rust implementation. Takes less than 20 seconds to tokenize a GB of text on a server's CPU.
  • Easy to use, but also extremely versatile.
  • Designed for research and production.
  • Normalization comes with alignments tracking. It's always possible to get the part of the original sentence that corresponds to a given token.
  • Does all the pre-processing: Truncate, Pad, add the special tokens your model needs.

Installation

With pip:

pip install tokenizers

From sources:

To use this method, you need to have the Rust installed:

# Install with:
curl https://sh.rustup.rs -sSf | sh -s -- -y
export PATH="$HOME/.cargo/bin:$PATH"

Once Rust is installed, you can compile doing the following

git clone https://github.com/huggingface/tokenizers
cd tokenizers/bindings/python

# Create a virtual env (you can use yours as well)
python -m venv .env
source .env/bin/activate

# Install `tokenizers` in the current virtual env
pip install setuptools_rust
python setup.py install

Using the provided Tokenizers

We provide some pre-build tokenizers to cover the most common cases. You can easily load one of these using some vocab.json and merges.txt files:

from tokenizers import CharBPETokenizer

# Initialize a tokenizer
vocab = "./path/to/vocab.json"
merges = "./path/to/merges.txt"
tokenizer = CharBPETokenizer(vocab, merges)

# And then encode:
encoded = tokenizer.encode("I can feel the magic, can you?")
print(encoded.ids)
print(encoded.tokens)

And you can train them just as simply:

from tokenizers import CharBPETokenizer

# Initialize a tokenizer
tokenizer = CharBPETokenizer()

# Then train it!
tokenizer.train([ "./path/to/files/1.txt", "./path/to/files/2.txt" ])

# Now, let's use it:
encoded = tokenizer.encode("I can feel the magic, can you?")

# And finally save it somewhere
tokenizer.save("./path/to/directory/my-bpe.tokenizer.json")

Provided Tokenizers

  • CharBPETokenizer: The original BPE
  • ByteLevelBPETokenizer: The byte level version of the BPE
  • SentencePieceBPETokenizer: A BPE implementation compatible with the one used by SentencePiece
  • BertWordPieceTokenizer: The famous Bert tokenizer, using WordPiece

All of these can be used and trained as explained above!

Build your own

Whenever these provided tokenizers don't give you enough freedom, you can build your own tokenizer, by putting all the different parts you need together. You can check how we implemented the provided tokenizers and adapt them easily to your own needs.

Building a byte-level BPE

Here is an example showing how to build your own byte-level BPE by putting all the different pieces together, and then saving it to a single file:

from tokenizers import Tokenizer, models, pre_tokenizers, decoders, trainers, processors

# Initialize a tokenizer
tokenizer = Tokenizer(models.BPE())

# Customize pre-tokenization and decoding
tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=True)
tokenizer.decoder = decoders.ByteLevel()
tokenizer.post_processor = processors.ByteLevel(trim_offsets=True)

# And then train
trainer = trainers.BpeTrainer(vocab_size=20000, min_frequency=2)
tokenizer.train([
	"./path/to/dataset/1.txt",
	"./path/to/dataset/2.txt",
	"./path/to/dataset/3.txt"
], trainer=trainer)

# And Save it
tokenizer.save("byte-level-bpe.tokenizer.json", pretty=True)

Now, when you want to use this tokenizer, this is as simple as:

from tokenizers import Tokenizer

tokenizer = Tokenizer.from_file("byte-level-bpe.tokenizer.json")

encoded = tokenizer.encode("I can feel the magic, can you?")

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tokenizers-0.10.1.tar.gz (210.7 kB view details)

Uploaded Source

Built Distributions

tokenizers-0.10.1-cp39-cp39-win_amd64.whl (2.0 MB view details)

Uploaded CPython 3.9 Windows x86-64

tokenizers-0.10.1-cp39-cp39-win32.whl (1.8 MB view details)

Uploaded CPython 3.9 Windows x86

tokenizers-0.10.1-cp39-cp39-manylinux2010_x86_64.whl (3.2 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.12+ x86-64

tokenizers-0.10.1-cp39-cp39-manylinux1_x86_64.whl (3.2 MB view details)

Uploaded CPython 3.9

tokenizers-0.10.1-cp39-cp39-macosx_10_11_x86_64.whl (2.2 MB view details)

Uploaded CPython 3.9 macOS 10.11+ x86-64

tokenizers-0.10.1-cp38-cp38-win_amd64.whl (2.0 MB view details)

Uploaded CPython 3.8 Windows x86-64

tokenizers-0.10.1-cp38-cp38-win32.whl (1.8 MB view details)

Uploaded CPython 3.8 Windows x86

tokenizers-0.10.1-cp38-cp38-manylinux2010_x86_64.whl (3.2 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.12+ x86-64

tokenizers-0.10.1-cp38-cp38-manylinux1_x86_64.whl (3.2 MB view details)

Uploaded CPython 3.8

tokenizers-0.10.1-cp38-cp38-macosx_10_11_x86_64.whl (2.2 MB view details)

Uploaded CPython 3.8 macOS 10.11+ x86-64

tokenizers-0.10.1-cp37-cp37m-win_amd64.whl (2.0 MB view details)

Uploaded CPython 3.7m Windows x86-64

tokenizers-0.10.1-cp37-cp37m-win32.whl (1.8 MB view details)

Uploaded CPython 3.7m Windows x86

tokenizers-0.10.1-cp37-cp37m-manylinux2014_s390x.whl (3.7 MB view details)

Uploaded CPython 3.7m

tokenizers-0.10.1-cp37-cp37m-manylinux2010_x86_64.whl (3.2 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.12+ x86-64

tokenizers-0.10.1-cp37-cp37m-manylinux1_x86_64.whl (3.2 MB view details)

Uploaded CPython 3.7m

tokenizers-0.10.1-cp37-cp37m-macosx_10_11_x86_64.whl (2.2 MB view details)

Uploaded CPython 3.7m macOS 10.11+ x86-64

tokenizers-0.10.1-cp36-cp36m-win_amd64.whl (2.0 MB view details)

Uploaded CPython 3.6m Windows x86-64

tokenizers-0.10.1-cp36-cp36m-win32.whl (1.8 MB view details)

Uploaded CPython 3.6m Windows x86

tokenizers-0.10.1-cp36-cp36m-manylinux2014_s390x.whl (3.7 MB view details)

Uploaded CPython 3.6m

tokenizers-0.10.1-cp36-cp36m-manylinux2010_x86_64.whl (3.2 MB view details)

Uploaded CPython 3.6m manylinux: glibc 2.12+ x86-64

tokenizers-0.10.1-cp36-cp36m-manylinux1_x86_64.whl (3.2 MB view details)

Uploaded CPython 3.6m

tokenizers-0.10.1-cp36-cp36m-macosx_10_11_x86_64.whl (2.2 MB view details)

Uploaded CPython 3.6m macOS 10.11+ x86-64

tokenizers-0.10.1-cp35-cp35m-win_amd64.whl (2.0 MB view details)

Uploaded CPython 3.5m Windows x86-64

tokenizers-0.10.1-cp35-cp35m-win32.whl (1.8 MB view details)

Uploaded CPython 3.5m Windows x86

tokenizers-0.10.1-cp35-cp35m-manylinux2014_s390x.whl (3.7 MB view details)

Uploaded CPython 3.5m

tokenizers-0.10.1-cp35-cp35m-manylinux2010_x86_64.whl (3.2 MB view details)

Uploaded CPython 3.5m manylinux: glibc 2.12+ x86-64

tokenizers-0.10.1-cp35-cp35m-manylinux1_x86_64.whl (3.2 MB view details)

Uploaded CPython 3.5m

tokenizers-0.10.1-cp35-cp35m-macosx_10_11_x86_64.whl (2.2 MB view details)

Uploaded CPython 3.5m macOS 10.11+ x86-64

File details

Details for the file tokenizers-0.10.1.tar.gz.

File metadata

  • Download URL: tokenizers-0.10.1.tar.gz
  • Upload date:
  • Size: 210.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1.tar.gz
Algorithm Hash digest
SHA256 81c35b4bc9238c0b5d0af91a719e732a60ee0d87d8bf76615bfec8f3e3ba8f15
MD5 b42de0eed9b46b671f9082c911dae5bc
BLAKE2b-256 01ee206ac679a6e0e5d8fd685d9a16e8f23229b39a3733c57ec0851efe65f2c6

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 2.0 MB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 3e7e526fafaae5d7360570be0ec6f1331e3e962bafa262f6a357ba6bd850f8df
MD5 5c672999cbd0afd3277109e4df4f4a18
BLAKE2b-256 6fed22ebc93377cd324c3f4fd869487facd53b17f8380266714aff9e68c2539c

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp39-cp39-win32.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp39-cp39-win32.whl
  • Upload date:
  • Size: 1.8 MB
  • Tags: CPython 3.9, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp39-cp39-win32.whl
Algorithm Hash digest
SHA256 484939ddea9689454156f4d2ee96032ebefddff1dc174caafa9a28c919f3c85d
MD5 69a7d4e029fc91d1512f4a5ee2a8c289
BLAKE2b-256 69b54ebb8ea757b3415567f04140bbec10c9d18f7cd791f9de3b4f2f1995df95

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp39-cp39-manylinux2014_s390x.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp39-cp39-manylinux2014_s390x.whl
  • Upload date:
  • Size: 3.7 MB
  • Tags: CPython 3.9
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp39-cp39-manylinux2014_s390x.whl
Algorithm Hash digest
SHA256 bcb2c07cc052f45419a1aac1bfef96de69cd0677c53f901f5dcba5bfa4361575
MD5 090d21aa8fcd397ed9019eea5f91b98f
BLAKE2b-256 5c159416fdb71fea00b2a8e97ea36f26b849071f0c201d608c09c022d2ceb7f6

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp39-cp39-manylinux2014_ppc64le.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp39-cp39-manylinux2014_ppc64le.whl
  • Upload date:
  • Size: 3.6 MB
  • Tags: CPython 3.9
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp39-cp39-manylinux2014_ppc64le.whl
Algorithm Hash digest
SHA256 f405c9f0c0f77f9f6500b7c8587eacc122d1a0256ff55bcbef219ed94201b8f8
MD5 9aa3a8c0835da05303ac707eed193671
BLAKE2b-256 345590364e0aeb340b6c45ba7aff4dc56ece9df74a97f533cc541eb4ebb3c943

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp39-cp39-manylinux2014_aarch64.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp39-cp39-manylinux2014_aarch64.whl
  • Upload date:
  • Size: 3.1 MB
  • Tags: CPython 3.9
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp39-cp39-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 296c741df0ceec93946f030e6b94c6fa975f17823cdf0bdf0989654eee277b54
MD5 89f7dfe1dc063d55f39daf85ec777aab
BLAKE2b-256 603eeb20b245e24855e048989b1dfee83049c2aaf20342bdbe06c05bb3c69163

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp39-cp39-manylinux2010_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp39-cp39-manylinux2010_x86_64.whl
  • Upload date:
  • Size: 3.2 MB
  • Tags: CPython 3.9, manylinux: glibc 2.12+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp39-cp39-manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 c96b0c985ab6073c901767df3d77e96dc208ab391ad94d5a0b9b5211d6e1729c
MD5 aa4cdf6e34c7f4059c3285f99eff23db
BLAKE2b-256 8fdb421bd02b74270961edc81e43f30f900e087e964b9273607ca4166ec6981c

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp39-cp39-manylinux1_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp39-cp39-manylinux1_x86_64.whl
  • Upload date:
  • Size: 3.2 MB
  • Tags: CPython 3.9
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp39-cp39-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 2739e9d44cfbd9b3e4339a279e7037c572757763bb09faa9727ae35c93f79623
MD5 622ca6b0dd21d41a666c79f0f2141a03
BLAKE2b-256 f4ed5504da389aaf6b335a75ae093b6c32c85939327b485fa38d2b7e712be867

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp39-cp39-macosx_10_11_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp39-cp39-macosx_10_11_x86_64.whl
  • Upload date:
  • Size: 2.2 MB
  • Tags: CPython 3.9, macOS 10.11+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp39-cp39-macosx_10_11_x86_64.whl
Algorithm Hash digest
SHA256 eabea1bdf9d525ca316e72bfc94e41c8e9b42b8fe8e4cbd01d0a4f520e25276f
MD5 39e2235c114a988e3f204cfe354a10fc
BLAKE2b-256 60f66e28ff0a62d3b5a622885f8b8f1ed77cab0730dd6cc71f78419cc634073b

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 2.0 MB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 62c21682a4a8dd8cc8eecf68d3a1c058b5c912f0b792a5e0507e0afd7524f56c
MD5 b71173c1ae72647592436f8c7d75d863
BLAKE2b-256 f0d6d316d2c676a1f3522dc4c76091b51d66383ddf71a3a72d8468d25e6cad30

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp38-cp38-win32.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp38-cp38-win32.whl
  • Upload date:
  • Size: 1.8 MB
  • Tags: CPython 3.8, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp38-cp38-win32.whl
Algorithm Hash digest
SHA256 3a343b703bd4e174de3b126702418c7924bd68a95a25b43cbb590d1c2e7a4034
MD5 8286379419600ea08e98b47576cd84ca
BLAKE2b-256 8257de815c7a8a4de588f743890b0508611b068f8c92dad284109a2e4eb42e4d

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp38-cp38-manylinux2014_s390x.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp38-cp38-manylinux2014_s390x.whl
  • Upload date:
  • Size: 3.7 MB
  • Tags: CPython 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp38-cp38-manylinux2014_s390x.whl
Algorithm Hash digest
SHA256 069e30baedbfb098390e03af1701c95c87d024055c82842e4b2d699a0d49d3db
MD5 74d8210999342284784b970f1ba71dc9
BLAKE2b-256 e4e211de096e10503e696131895cbb28665a12a75d60f9501da0a5dd9b43a7d4

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp38-cp38-manylinux2014_ppc64le.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp38-cp38-manylinux2014_ppc64le.whl
  • Upload date:
  • Size: 3.6 MB
  • Tags: CPython 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp38-cp38-manylinux2014_ppc64le.whl
Algorithm Hash digest
SHA256 d212515cb0c44ef884009169fb9b7c6a3efa540b3fa497620a0915624a7d21b6
MD5 aff630360a4e2bd4d8776bac3cc26389
BLAKE2b-256 7bb7ffa973bbf5b9794bb94a6db72bd691d1531f2717aab627a4b69caee084bb

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp38-cp38-manylinux2014_aarch64.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp38-cp38-manylinux2014_aarch64.whl
  • Upload date:
  • Size: 3.1 MB
  • Tags: CPython 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp38-cp38-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 60a2f5a33e7846817ce54a5ddd2f61108e95aaccf579caeaea0bba7b6a0a9222
MD5 104f957b39d4307f7b309c68f1d8f663
BLAKE2b-256 c4d7f096d49dad963bed27948c77d04bff002ef6899e27e786d8efc23128389b

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp38-cp38-manylinux2010_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp38-cp38-manylinux2010_x86_64.whl
  • Upload date:
  • Size: 3.2 MB
  • Tags: CPython 3.8, manylinux: glibc 2.12+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp38-cp38-manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 07e321f46df4ecf8f7919b260f67d4df18151e50eddfaaa8ef7ba80ed473a79a
MD5 2777329a224e20b91801c2d90d4d17ec
BLAKE2b-256 05d197b23255d1d6d02e467e2701a345f74848fcd68242bac84923cf9ddd7388

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp38-cp38-manylinux1_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp38-cp38-manylinux1_x86_64.whl
  • Upload date:
  • Size: 3.2 MB
  • Tags: CPython 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp38-cp38-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 95d59780e202abf43f150225dc1fd09ac67fb49cc8d93e437a27f3e4593fe64c
MD5 2da996d1f76734068afc2039764f75c8
BLAKE2b-256 e171b6b9a97cf03f99b917386d34e8341aee3534fb0e45583c4f04d0eb3069ae

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp38-cp38-macosx_10_11_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp38-cp38-macosx_10_11_x86_64.whl
  • Upload date:
  • Size: 2.2 MB
  • Tags: CPython 3.8, macOS 10.11+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp38-cp38-macosx_10_11_x86_64.whl
Algorithm Hash digest
SHA256 d2bc2eab13002f06fb987be9d762654acf3c0909a7a8c4e822fe7ebaa7a2a09d
MD5 eebe543cba75eb2b6ae213e1e28123cc
BLAKE2b-256 b0a5f52ed85a4712e29d7885132c5d306270a99aa52123be94642c830faa5a73

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 2.0 MB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 5dd957cfd1109e56ff895dfce14d1be16bd9399ce98999b60cf0e8e21243d34c
MD5 ae6b48217ebc8aed5c65633cc2ee3327
BLAKE2b-256 cc3a261b99d6815f6afdd2d35b6420d524305daa17bf430c420dca0115bd1c6c

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp37-cp37m-win32.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp37-cp37m-win32.whl
  • Upload date:
  • Size: 1.8 MB
  • Tags: CPython 3.7m, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp37-cp37m-win32.whl
Algorithm Hash digest
SHA256 1c5aef76b8c9f1fed979a1943cfde2a1707a396a24dc3c09d510e06b3b8a3f3d
MD5 a086dab4525c6c5a3dc66c468f95bfed
BLAKE2b-256 a23c510a772003bd634aa5cf1bc49e211ca06384c47769e9307a921deabe982a

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp37-cp37m-manylinux2014_s390x.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp37-cp37m-manylinux2014_s390x.whl
  • Upload date:
  • Size: 3.7 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp37-cp37m-manylinux2014_s390x.whl
Algorithm Hash digest
SHA256 dd399f58c26d18d2ee6ffd647c5ec9365c910bed9004e5ddaea5842ff533bcec
MD5 306a4b74306288031f867f10d76bc5a9
BLAKE2b-256 ca074202e7ea30ff9a7fbc19ac90182cf720e6f9f29ff604d1c780fa8f31938d

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp37-cp37m-manylinux2014_ppc64le.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp37-cp37m-manylinux2014_ppc64le.whl
  • Upload date:
  • Size: 3.6 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp37-cp37m-manylinux2014_ppc64le.whl
Algorithm Hash digest
SHA256 ed236e0ea8c3369cb3bb65a206c2b186e50a3779b03cfca1a92e0ce0b272caab
MD5 90e52f72859cd8636dc547b6df78edf2
BLAKE2b-256 8de64bd87d92d4a08c6bb58ca2b15d2e1a7c5b134784d103f94405af82461b45

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp37-cp37m-manylinux2014_aarch64.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp37-cp37m-manylinux2014_aarch64.whl
  • Upload date:
  • Size: 3.1 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp37-cp37m-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 57c37a312ba56060129f2ad075f852e83bae205df1a5f34db50e6da2e0432805
MD5 f39a5afc268dd6055f505f65282a59bd
BLAKE2b-256 35b23d00b0725d3893a2d40b0c65224ef74003ee5eca8d99b801ba826f08e23a

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp37-cp37m-manylinux2010_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp37-cp37m-manylinux2010_x86_64.whl
  • Upload date:
  • Size: 3.2 MB
  • Tags: CPython 3.7m, manylinux: glibc 2.12+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp37-cp37m-manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 4f42453c86dcabbc3c19c3ec166d693843ffa80543982e3150209e3ca274a322
MD5 4995870452a91e573078eb34bcf0879d
BLAKE2b-256 71232ddc317b2121117bf34dd00f5b0de194158f2a44ee2bf5e47c7166878a97

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp37-cp37m-manylinux1_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp37-cp37m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 3.2 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp37-cp37m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 eea64d50897234f2df0301c53412d6dc5810688ad73b75e661e53fe698d33e71
MD5 6e6fc0533e18ababd4538c331dc0903e
BLAKE2b-256 711860eb3eb2b6ed3c524ad79f87e3145f84d725ef05a249d04badbc755cc1a2

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp37-cp37m-macosx_10_11_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp37-cp37m-macosx_10_11_x86_64.whl
  • Upload date:
  • Size: 2.2 MB
  • Tags: CPython 3.7m, macOS 10.11+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp37-cp37m-macosx_10_11_x86_64.whl
Algorithm Hash digest
SHA256 917157a63c919c44888bf9d13f8aae43a8b409d1dba831caafd47b4aee0ea917
MD5 dee2a8ac0a764581b018a3ead3f109b5
BLAKE2b-256 ac8175ac49a3566f51f85d5fd3a7a7b587fa6cfe37cb9fc19ce3de48763b7295

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 2.0 MB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 6b70f446bf57c4447d7a192ffaa4f5a883fee99524d76c694025ae05942065a1
MD5 9a087d8058fc089d734ca8f6db59ca76
BLAKE2b-256 9bda55dd69f54ecd790aef1bcd058ec0619931d93835072e6f608e75d93cdc17

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp36-cp36m-win32.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp36-cp36m-win32.whl
  • Upload date:
  • Size: 1.8 MB
  • Tags: CPython 3.6m, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp36-cp36m-win32.whl
Algorithm Hash digest
SHA256 9c4aedbd763dbf929a27073bb6e569bd454bf79c409f3f6920fa0f4db12abe51
MD5 6b171a0524ea306e67f8635cb3275d27
BLAKE2b-256 440c3bf00064a39644ce59d99d58ba03f4d1d54aa8a03eab562dd997a8b2eff4

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp36-cp36m-manylinux2014_s390x.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp36-cp36m-manylinux2014_s390x.whl
  • Upload date:
  • Size: 3.7 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp36-cp36m-manylinux2014_s390x.whl
Algorithm Hash digest
SHA256 a7a4c5c59f9896601467807a92b32eb64970b3210bed6d8f1e87c14f1da17496
MD5 f068782595519a4cb9a9230608f14097
BLAKE2b-256 096176562c84f633ee11e84d2781cede913309accfdc8198c0e992102840f71e

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp36-cp36m-manylinux2014_ppc64le.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp36-cp36m-manylinux2014_ppc64le.whl
  • Upload date:
  • Size: 3.6 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp36-cp36m-manylinux2014_ppc64le.whl
Algorithm Hash digest
SHA256 3da95c8f9bea55e261376972d0d461bd6a4bd02913c5b3942c087801c8631b65
MD5 9f183097c51670d1555b7a722f39ae9e
BLAKE2b-256 3c3285be67aea7695b121b1f6be45cdb7ad955876c3ff721fc72a95afe1caa12

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp36-cp36m-manylinux2014_aarch64.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp36-cp36m-manylinux2014_aarch64.whl
  • Upload date:
  • Size: 3.1 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp36-cp36m-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 2ef2ac8e03b9cdc4b7d3be8a84dc411ce264abdb694cfc198e6ba9d22f5c1381
MD5 802c0837aa9be41283bc5335c8dcbccd
BLAKE2b-256 97789615a7ee27a1452fa03b3161300af3270709240292a76f3e482fa384cd20

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp36-cp36m-manylinux2010_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp36-cp36m-manylinux2010_x86_64.whl
  • Upload date:
  • Size: 3.2 MB
  • Tags: CPython 3.6m, manylinux: glibc 2.12+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp36-cp36m-manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 fecfcb95c23ce6ebb4834156e78cb6a4cf883b37102c8dd46bc8bd3b5244c3d5
MD5 fe04b46f0d231bfd6166728055d320c7
BLAKE2b-256 fd5b44baae602e0a30bcc53fbdbc60bd940c15e143d252d658dfdefce736ece5

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp36-cp36m-manylinux1_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp36-cp36m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 3.2 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp36-cp36m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 6f9f5ac1a1c67dec947e0a28905ea017a826926faa9a827eff1792012632a2f5
MD5 88f998e1c40b696aba91f8b51d234853
BLAKE2b-256 b334bb758c20fad7295eb8e2d0227e3b6860a8b1ecc85ab30be4883b18235c15

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp36-cp36m-macosx_10_11_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp36-cp36m-macosx_10_11_x86_64.whl
  • Upload date:
  • Size: 2.2 MB
  • Tags: CPython 3.6m, macOS 10.11+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp36-cp36m-macosx_10_11_x86_64.whl
Algorithm Hash digest
SHA256 8f5d2a1d6d4a502ead87af60aaec9fdd943e3f298dfa302fe139c0c357f553e5
MD5 863c2e7e2a30245fa011906d5b70950f
BLAKE2b-256 8408b8bd450b671e8a4d340ebfcad7cbb1d04e266709fc8b049df3f544c0688e

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp35-cp35m-win_amd64.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp35-cp35m-win_amd64.whl
  • Upload date:
  • Size: 2.0 MB
  • Tags: CPython 3.5m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp35-cp35m-win_amd64.whl
Algorithm Hash digest
SHA256 adb5566b37fae9a01bd6ecd86a85e71f421df4ab481822b9dd6dae446d6f37d1
MD5 23d0d062c6a6e94ae0206b3b7b067f7d
BLAKE2b-256 7e7bd0eeca5047542230878448c6e87bd96002ae235887e408a822e424386c1b

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp35-cp35m-win32.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp35-cp35m-win32.whl
  • Upload date:
  • Size: 1.8 MB
  • Tags: CPython 3.5m, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp35-cp35m-win32.whl
Algorithm Hash digest
SHA256 eae53cbdd4ce50d7fe37e9d4e49019023205065ceac6f373bb656038937e42ec
MD5 2644a2d9aed57b17b6c6141b99ea6b31
BLAKE2b-256 aad3a758158feccb5645e78a070066412589fe25d2241ecb72d6d0bd8e757ce5

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp35-cp35m-manylinux2014_s390x.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp35-cp35m-manylinux2014_s390x.whl
  • Upload date:
  • Size: 3.7 MB
  • Tags: CPython 3.5m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp35-cp35m-manylinux2014_s390x.whl
Algorithm Hash digest
SHA256 99eb29009bb23548edd98f01fabb6df24c99ce55456f1c0910204996c7211920
MD5 06ac6f51624c05f37e4bef9d81d7dec5
BLAKE2b-256 bc43c273ac109f8b0a6d1357e43da9fcfa63244ae29b145de7bd6faea4cf78b0

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp35-cp35m-manylinux2014_ppc64le.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp35-cp35m-manylinux2014_ppc64le.whl
  • Upload date:
  • Size: 3.6 MB
  • Tags: CPython 3.5m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp35-cp35m-manylinux2014_ppc64le.whl
Algorithm Hash digest
SHA256 e380e87600430aa9e5b32757e3ffcfa25744ea14d010585d8ca817097bd1f4f7
MD5 9410ee9309e343b46059838e271b74aa
BLAKE2b-256 85e7550b388a8ba3b7cfcfed2a3ba8a586ceb806b90fa80aef4cbd505ca543c6

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp35-cp35m-manylinux2014_aarch64.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp35-cp35m-manylinux2014_aarch64.whl
  • Upload date:
  • Size: 3.1 MB
  • Tags: CPython 3.5m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp35-cp35m-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 e035deb888de33bf7bce3f004c9f815090f74498fe867d8a4ce4d33964db4068
MD5 02ae98a71044ca46cd397d9b281813cd
BLAKE2b-256 5c30bf98280f43adb4e31eaca0f55e5e81da13736a1c0c01ccbb08c4a20ffc2c

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp35-cp35m-manylinux2010_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp35-cp35m-manylinux2010_x86_64.whl
  • Upload date:
  • Size: 3.2 MB
  • Tags: CPython 3.5m, manylinux: glibc 2.12+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp35-cp35m-manylinux2010_x86_64.whl
Algorithm Hash digest
SHA256 290eceb99780d5ef5de89968bb6a9bcf912d6ea79578107e9c8bc1386f490389
MD5 34fed08ca729842e2c01399083a8e69c
BLAKE2b-256 61c281648220d2a9b5f75c72153df1356e48b7e10a6c1e8945708a985ea41df6

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp35-cp35m-manylinux1_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp35-cp35m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 3.2 MB
  • Tags: CPython 3.5m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp35-cp35m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 cd045b3b79e04c0278c3d5f02ca2bbab2fe351fdb489e8441b231da12bea41de
MD5 00c250a595c4d906781ee4fb1f90f951
BLAKE2b-256 e943166574ff9b699c565cc589ce944a4509f39be48ea8d1d9be320af2145955

See more details on using hashes here.

File details

Details for the file tokenizers-0.10.1-cp35-cp35m-macosx_10_11_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.10.1-cp35-cp35m-macosx_10_11_x86_64.whl
  • Upload date:
  • Size: 2.2 MB
  • Tags: CPython 3.5m, macOS 10.11+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.2.1 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.9.1

File hashes

Hashes for tokenizers-0.10.1-cp35-cp35m-macosx_10_11_x86_64.whl
Algorithm Hash digest
SHA256 729954ba7f650a3965f1b3e481c57da1c372b9d3fe1d70b0a25c711543e48300
MD5 1b1bdb3ab2bd5ec5c05e164cc374df9b
BLAKE2b-256 338b342d2e31fdf7c53e1408d295c054af6b99ae9b33242f1ccbcb814bd0f99f

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page