Fast and Customizable Tokenizers

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for ArthurZucker from gravatar.com ArthurZucker Avatar for McPotato from gravatar.com McPotato Avatar for Nicolas.Patry from gravatar.com Nicolas.Patry Avatar for xn1t0x from gravatar.com xn1t0x

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: Apache Software License (Apache License 2.0)
  • Author: Anthony MOI
  • Tags NLP, tokenizer, BPE, transformer, deep, learning
Classifiers

Project description



Build GitHub


Tokenizers

Provides an implementation of today's most used tokenizers, with a focus on performance and versatility.

Bindings over the Rust implementation. If you are interested in the High-level design, you can go check it there.

Otherwise, let's dive in!

Main features:

  • Train new vocabularies and tokenize using 4 pre-made tokenizers (Bert WordPiece and the 3 most common BPE versions).
  • Extremely fast (both training and tokenization), thanks to the Rust implementation. Takes less than 20 seconds to tokenize a GB of text on a server's CPU.
  • Easy to use, but also extremely versatile.
  • Designed for research and production.
  • Normalization comes with alignments tracking. It's always possible to get the part of the original sentence that corresponds to a given token.
  • Does all the pre-processing: Truncate, Pad, add the special tokens your model needs.

Installation

With pip:

pip install tokenizers

From sources:

To use this method, you need to have the Rust installed:

# Install with:
curl https://sh.rustup.rs -sSf | sh -s -- -y
export PATH="$HOME/.cargo/bin:$PATH"

Once Rust is installed, you can compile doing the following

git clone https://github.com/huggingface/tokenizers
cd tokenizers/bindings/python

# Create a virtual env (you can use yours as well)
python -m venv .env
source .env/bin/activate

# Install `tokenizers` in the current virtual env
pip install setuptools_rust
python setup.py install

Using the provided Tokenizers

Using a pre-trained tokenizer is really simple:

from tokenizers import CharBPETokenizer

# Initialize a tokenizer
vocab = "./path/to/vocab.json"
merges = "./path/to/merges.txt"
tokenizer = CharBPETokenizer(vocab, merges)

# And then encode:
encoded = tokenizer.encode("I can feel the magic, can you?")
print(encoded.ids)
print(encoded.tokens)

And you can train yours just as simply:

from tokenizers import CharBPETokenizer

# Initialize a tokenizer
tokenizer = CharBPETokenizer()

# Then train it!
tokenizer.train([ "./path/to/files/1.txt", "./path/to/files/2.txt" ])

# And you can use it
encoded = tokenizer.encode("I can feel the magic, can you?")

# And finally save it somewhere
tokenizer.save("./path/to/directory", "my-bpe")

Provided Tokenizers

  • CharBPETokenizer: The original BPE
  • ByteLevelBPETokenizer: The byte level version of the BPE
  • SentencePieceBPETokenizer: A BPE implementation compatible with the one used by SentencePiece
  • BertWordPieceTokenizer: The famous Bert tokenizer, using WordPiece

All of these can be used and trained as explained above!

Build your own

You can also easily build your own tokenizers, by putting all the different parts you need together:

Use a pre-trained tokenizer

from tokenizers import Tokenizer, models, pre_tokenizers, decoders, processors

# Load a BPE Model
vocab = "./path/to/vocab.json"
merges = "./path/to/merges.txt"
bpe = models.BPE(vocab, merges)

# Initialize a tokenizer
tokenizer = Tokenizer(bpe)

# Customize pre-tokenization and decoding
tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=True)
tokenizer.decoder = decoders.ByteLevel()
tokenizer.post_processor = processors.ByteLevel(trim_offsets=True)

# And then encode:
encoded = tokenizer.encode("I can feel the magic, can you?")
print(encoded.ids)
print(encoded.tokens)

# Or tokenize multiple sentences at once:
encoded = tokenizer.encode_batch([
	"I can feel the magic, can you?",
	"The quick brown fox jumps over the lazy dog"
])
print(encoded)

Train a new tokenizer

from tokenizers import Tokenizer, models, pre_tokenizers, decoders, trainers, processors

# Initialize a tokenizer
tokenizer = Tokenizer(models.BPE())

# Customize pre-tokenization and decoding
tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=True)
tokenizer.decoder = decoders.ByteLevel()
tokenizer.post_processor = processors.ByteLevel(trim_offsets=True)

# And then train
trainer = trainers.BpeTrainer(vocab_size=20000, min_frequency=2)
tokenizer.train(trainer, [
	"./path/to/dataset/1.txt",
	"./path/to/dataset/2.txt",
	"./path/to/dataset/3.txt"
])

# Now we can encode
encoded = tokenizer.encode("I can feel the magic, can you?")
print(encoded)

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for ArthurZucker from gravatar.com ArthurZucker Avatar for McPotato from gravatar.com McPotato Avatar for Nicolas.Patry from gravatar.com Nicolas.Patry Avatar for xn1t0x from gravatar.com xn1t0x

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: Apache Software License (Apache License 2.0)
  • Author: Anthony MOI
  • Tags NLP, tokenizer, BPE, transformer, deep, learning
Classifiers

Release history Release notifications | RSS feed

0.20.3

0.20.3rc0 pre-release

0.20.2

0.20.1

0.20.1rc1 pre-release

0.20.0

0.20.0rc1 pre-release

0.19.1

0.19.0

0.15.2

0.15.2rc1 pre-release

0.15.1

0.15.0

0.14.1

0.14.0

0.13.3

0.13.2

0.13.1

0.13.0

0.12.1

0.12.0 yanked

Reason this release was yanked:

Breaking change with unexpected effect

0.11.6

0.11.5

0.11.4

0.11.3

0.11.2

0.11.1

0.11.0

0.10.3

0.10.2

0.10.1

0.10.1rc1 pre-release

0.10.0

0.10.0rc1 pre-release

0.9.4

0.9.3

0.9.2

0.9.1

0.9.0

0.9.0rc2 pre-release

0.9.0rc1 pre-release

0.9.0.dev4 pre-release

0.9.0.dev3 pre-release

0.9.0.dev2 pre-release

0.9.0.dev1 pre-release

0.9.0.dev0 pre-release

0.8.1

0.8.1rc2 pre-release

0.8.1rc1 pre-release

0.8.0

0.8.0rc4 pre-release

0.8.0rc3 pre-release

0.8.0rc2 pre-release

0.8.0rc1 pre-release

0.8.0.dev2 pre-release

0.8.0.dev1 pre-release

0.8.0.dev0 pre-release

0.7.0

0.7.0rc7 pre-release

0.7.0rc6 pre-release

This version

0.7.0rc5 pre-release

0.7.0rc4 pre-release

0.7.0rc3 pre-release

0.7.0rc2 pre-release

0.7.0rc1 pre-release

0.6.0

0.5.2

0.5.1

0.5.0

0.4.2

0.4.1

0.4.0

0.3.0

0.2.1

0.2.0

0.1.1

0.1.0

0.0.13

0.0.12

0.0.11

0.0.10

0.0.9

0.0.8

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

0.0.1

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tokenizers-0.7.0rc5.tar.gz (79.6 kB view details)

Uploaded Source

Built Distributions

tokenizers-0.7.0rc5-cp38-cp38-win_amd64.whl (1.1 MB view details)

Uploaded CPython 3.8 Windows x86-64

tokenizers-0.7.0rc5-cp38-cp38-win32.whl (970.0 kB view details)

Uploaded CPython 3.8 Windows x86

tokenizers-0.7.0rc5-cp38-cp38-manylinux1_x86_64.whl (7.5 MB view details)

Uploaded CPython 3.8

tokenizers-0.7.0rc5-cp38-cp38-macosx_10_10_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.8 macOS 10.10+ x86-64

tokenizers-0.7.0rc5-cp37-cp37m-win_amd64.whl (1.1 MB view details)

Uploaded CPython 3.7m Windows x86-64

tokenizers-0.7.0rc5-cp37-cp37m-win32.whl (969.8 kB view details)

Uploaded CPython 3.7m Windows x86

tokenizers-0.7.0rc5-cp37-cp37m-manylinux1_x86_64.whl (5.7 MB view details)

Uploaded CPython 3.7m

tokenizers-0.7.0rc5-cp37-cp37m-macosx_10_10_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.7m macOS 10.10+ x86-64

tokenizers-0.7.0rc5-cp36-cp36m-win_amd64.whl (1.1 MB view details)

Uploaded CPython 3.6m Windows x86-64

tokenizers-0.7.0rc5-cp36-cp36m-win32.whl (969.9 kB view details)

Uploaded CPython 3.6m Windows x86

tokenizers-0.7.0rc5-cp36-cp36m-manylinux1_x86_64.whl (3.8 MB view details)

Uploaded CPython 3.6m

tokenizers-0.7.0rc5-cp36-cp36m-macosx_10_10_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.6m macOS 10.10+ x86-64

tokenizers-0.7.0rc5-cp35-cp35m-win_amd64.whl (1.1 MB view details)

Uploaded CPython 3.5m Windows x86-64

tokenizers-0.7.0rc5-cp35-cp35m-win32.whl (969.9 kB view details)

Uploaded CPython 3.5m Windows x86

tokenizers-0.7.0rc5-cp35-cp35m-manylinux1_x86_64.whl (1.9 MB view details)

Uploaded CPython 3.5m

tokenizers-0.7.0rc5-cp35-cp35m-macosx_10_10_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.5m macOS 10.10+ x86-64

File details

Details for the file tokenizers-0.7.0rc5.tar.gz.

File metadata

  • Download URL: tokenizers-0.7.0rc5.tar.gz
  • Upload date:
  • Size: 79.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2

File hashes

Hashes for tokenizers-0.7.0rc5.tar.gz
Algorithm Hash digest
SHA256 b160ffb926a801208c8bd1b7bc06a6fb3ef531be9c104354ffb00ae4b72ad3da
MD5 2e5afabff4c1e24ef1f4d07722a8d06b
BLAKE2b-256 503e0880f10c7307efb44870cf54dbe16fcb1d6f498ec8bc55d81ef4f3e6b8f9

See more details on using hashes here.

File details

Details for the file tokenizers-0.7.0rc5-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: tokenizers-0.7.0rc5-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 1.1 MB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2

File hashes

Hashes for tokenizers-0.7.0rc5-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 0f53dfa934440bee2678355631d1b3384f61bc1e347b66520eff5e0279b31aae
MD5 1c4ad2f7060a48291aed8a88c75d1d4d
BLAKE2b-256 afc97976e5335171ac6354266f3156f09455deb4452ee0fe10911a60927ee098

See more details on using hashes here.

File details

Details for the file tokenizers-0.7.0rc5-cp38-cp38-win32.whl.

File metadata

  • Download URL: tokenizers-0.7.0rc5-cp38-cp38-win32.whl
  • Upload date:
  • Size: 970.0 kB
  • Tags: CPython 3.8, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2

File hashes

Hashes for tokenizers-0.7.0rc5-cp38-cp38-win32.whl
Algorithm Hash digest
SHA256 93998e3aee5377648fdb96bd1fea6d624514cc9d6617eae4db74ef33b1db27ad
MD5 dbb21da645489fcf91d0a2bfb9a108dd
BLAKE2b-256 b7c473e895cb3dc33b4d6f9e7e71d55e493260c9401d336303ec49f9e544ff3c

See more details on using hashes here.

File details

Details for the file tokenizers-0.7.0rc5-cp38-cp38-manylinux1_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.7.0rc5-cp38-cp38-manylinux1_x86_64.whl
  • Upload date:
  • Size: 7.5 MB
  • Tags: CPython 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2

File hashes

Hashes for tokenizers-0.7.0rc5-cp38-cp38-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 f224b971cc55181d9d39a38a9102ee11df7a6c829b7396652440d6e636a51ad0
MD5 737c89fc6538b966af1408997d1f6638
BLAKE2b-256 8f79fdec0aa438147606b80a5e2fd2b1cd946cb3f766471b0ac3e649ce4521f9

See more details on using hashes here.

File details

Details for the file tokenizers-0.7.0rc5-cp38-cp38-macosx_10_10_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.7.0rc5-cp38-cp38-macosx_10_10_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.8, macOS 10.10+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2

File hashes

Hashes for tokenizers-0.7.0rc5-cp38-cp38-macosx_10_10_x86_64.whl
Algorithm Hash digest
SHA256 4184aee61fa9e450a67fed215d8f2249fde5f4376aed627e2e7f1e6a84566850
MD5 a66db1425026384cb97ac3a7a53b6541
BLAKE2b-256 641334ce46bb3af2c910d77c427ed804f7d25b39cf6f82da4351b2d8eca17f35

See more details on using hashes here.

File details

Details for the file tokenizers-0.7.0rc5-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: tokenizers-0.7.0rc5-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 1.1 MB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2

File hashes

Hashes for tokenizers-0.7.0rc5-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 e10cabd700a1ce7c56741a8d587ac5dba5eaa0249b082a62e6e063c0f4cb3d58
MD5 396730e94c269bd9f3e4b9f8e61fed1f
BLAKE2b-256 8ade41e172800d7c2f7eba73aa32779ba742d7cfbc704068eff6d7a34ee1cc7d

See more details on using hashes here.

File details

Details for the file tokenizers-0.7.0rc5-cp37-cp37m-win32.whl.

File metadata

  • Download URL: tokenizers-0.7.0rc5-cp37-cp37m-win32.whl
  • Upload date:
  • Size: 969.8 kB
  • Tags: CPython 3.7m, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2

File hashes

Hashes for tokenizers-0.7.0rc5-cp37-cp37m-win32.whl
Algorithm Hash digest
SHA256 586956e774136022102349e58f76c821f1fce382dda80d6b2c802eb4f1d03791
MD5 f1af55d6513d0657e4a1b09157e549d1
BLAKE2b-256 78030672db058b74da947493d71ff11597119d66c63f33ddf2f914b05a03b462

See more details on using hashes here.

File details

Details for the file tokenizers-0.7.0rc5-cp37-cp37m-manylinux1_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.7.0rc5-cp37-cp37m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 5.7 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2

File hashes

Hashes for tokenizers-0.7.0rc5-cp37-cp37m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 f3b9d16baa2a40d3352377ded25006004acbce9aaffd2d0f5ef6aaef3ce6029a
MD5 56a4ad221f46a8109d93485dd95e2f0c
BLAKE2b-256 88677ba87cecd81b8dcd3aa12649bd62d2e4b83b7c447721376b1c8db8e68d48

See more details on using hashes here.

File details

Details for the file tokenizers-0.7.0rc5-cp37-cp37m-macosx_10_10_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.7.0rc5-cp37-cp37m-macosx_10_10_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.7m, macOS 10.10+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2

File hashes

Hashes for tokenizers-0.7.0rc5-cp37-cp37m-macosx_10_10_x86_64.whl
Algorithm Hash digest
SHA256 8d5e635126ff0552941d466baad396297c6d847830ad26920675e6460ab29676
MD5 4ea8ecdff1b42290c476e39dea23885e
BLAKE2b-256 b760e687655d196e2094ffd1bdeec3a961dd1b3837b389df9db19bbd28675688

See more details on using hashes here.

File details

Details for the file tokenizers-0.7.0rc5-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: tokenizers-0.7.0rc5-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 1.1 MB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2

File hashes

Hashes for tokenizers-0.7.0rc5-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 5ed427e4743b47e5c38d532ab26802ab50ef882c1ec0f0db8d4b7d003e25f712
MD5 fabf1d2e7badb21bc57d705e4bd145ed
BLAKE2b-256 782b0f6467bbdef7c5a9d120a089a2367f213325b3779cab2c87cedcd6239a3c

See more details on using hashes here.

File details

Details for the file tokenizers-0.7.0rc5-cp36-cp36m-win32.whl.

File metadata

  • Download URL: tokenizers-0.7.0rc5-cp36-cp36m-win32.whl
  • Upload date:
  • Size: 969.9 kB
  • Tags: CPython 3.6m, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2

File hashes

Hashes for tokenizers-0.7.0rc5-cp36-cp36m-win32.whl
Algorithm Hash digest
SHA256 1206a039e81e49f6049e6a3e416dbdfd2c474ddb608e6a8326885f654a1d3aa5
MD5 c571901da2e74fcddcaa66846fe89ff1
BLAKE2b-256 b27019de24c8d5c96d28dae35f589c2940bc65e5340b55fe3a6b1af5a74d5b41

See more details on using hashes here.

File details

Details for the file tokenizers-0.7.0rc5-cp36-cp36m-manylinux1_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.7.0rc5-cp36-cp36m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 3.8 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2

File hashes

Hashes for tokenizers-0.7.0rc5-cp36-cp36m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 af4d3258c715ce63b799cdbd7c89c09e494c3c801108afab0dacc5c54478ede4
MD5 0c95d155ffc40b92294fcca7dacf2fee
BLAKE2b-256 a422ee2cfbc36fcf8db25ab1cbb987df5db1988df837403c724720ce7e01f1bb

See more details on using hashes here.

File details

Details for the file tokenizers-0.7.0rc5-cp36-cp36m-macosx_10_10_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.7.0rc5-cp36-cp36m-macosx_10_10_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.6m, macOS 10.10+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2

File hashes

Hashes for tokenizers-0.7.0rc5-cp36-cp36m-macosx_10_10_x86_64.whl
Algorithm Hash digest
SHA256 b0411893a10d62ba1b1a5a2ba2dceaf23fb42ea54099cf9952c8e8c2099f3db6
MD5 5436f3e50291eb661ab79248747c9617
BLAKE2b-256 fcd9f64776efc991067b1bda40dbbb92abaa7cb103323040ce5cb9e83f12e140

See more details on using hashes here.

File details

Details for the file tokenizers-0.7.0rc5-cp35-cp35m-win_amd64.whl.

File metadata

  • Download URL: tokenizers-0.7.0rc5-cp35-cp35m-win_amd64.whl
  • Upload date:
  • Size: 1.1 MB
  • Tags: CPython 3.5m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2

File hashes

Hashes for tokenizers-0.7.0rc5-cp35-cp35m-win_amd64.whl
Algorithm Hash digest
SHA256 c825e487babca768158722caed5731bfe49802220d0bfe154d2fa9fa6fac3925
MD5 dde7cb5d32779b27927ce097ca810a80
BLAKE2b-256 895de42c6ed33b593d76ad09210a3125a5f190d0ef1ab982954f1dc4244dd5c6

See more details on using hashes here.

File details

Details for the file tokenizers-0.7.0rc5-cp35-cp35m-win32.whl.

File metadata

  • Download URL: tokenizers-0.7.0rc5-cp35-cp35m-win32.whl
  • Upload date:
  • Size: 969.9 kB
  • Tags: CPython 3.5m, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2

File hashes

Hashes for tokenizers-0.7.0rc5-cp35-cp35m-win32.whl
Algorithm Hash digest
SHA256 a5fd82e00e2e80bb42be64bcfd4f822b2d8dd1355d99b52dde185f5cf27495d5
MD5 ecf5407a3207d446e8257902c5b69c05
BLAKE2b-256 d21b2988bbf829c50e7c1bb97011b314c964059bf83ffdcc2c0fd0f2e7914647

See more details on using hashes here.

File details

Details for the file tokenizers-0.7.0rc5-cp35-cp35m-manylinux1_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.7.0rc5-cp35-cp35m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.9 MB
  • Tags: CPython 3.5m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2

File hashes

Hashes for tokenizers-0.7.0rc5-cp35-cp35m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 1014e1857a25997b7713af9271285dbbe438948a58fd8231806fc47168fc3faa
MD5 063460835214ce04572f8490d0fa1ce7
BLAKE2b-256 8cbdeba55cc23283b884c556c67aa21a25d5fcc69c6794a536a89fedc487bc26

See more details on using hashes here.

File details

Details for the file tokenizers-0.7.0rc5-cp35-cp35m-macosx_10_10_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.7.0rc5-cp35-cp35m-macosx_10_10_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.5m, macOS 10.10+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2

File hashes

Hashes for tokenizers-0.7.0rc5-cp35-cp35m-macosx_10_10_x86_64.whl
Algorithm Hash digest
SHA256 6705e9cd4654072e080bd7311ca9d42bad5dca1fcd19400b080aabaf50c5c056
MD5 da21cdd81b21d9e608bcf7200a28d7c2
BLAKE2b-256 11b61892cdd132f303621f2e14481bfd27f1b0bd49bb86415531239890287de5

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page