Fast and Customizable Tokenizers

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for ArthurZucker from gravatar.com ArthurZucker Avatar for McPotato from gravatar.com McPotato Avatar for Nicolas.Patry from gravatar.com Nicolas.Patry Avatar for xn1t0x from gravatar.com xn1t0x

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: Apache Software License (Apache License 2.0)
  • Author: Anthony MOI
  • Tags NLP, tokenizer, BPE, transformer, deep, learning
Classifiers

Project description



Build GitHub


Tokenizers

Provides an implementation of today's most used tokenizers, with a focus on performance and versatility.

Bindings over the Rust implementation. If you are interested in the High-level design, you can go check it there.

Otherwise, let's dive in!

Main features:

  • Train new vocabularies and tokenize using 4 pre-made tokenizers (Bert WordPiece and the 3 most common BPE versions).
  • Extremely fast (both training and tokenization), thanks to the Rust implementation. Takes less than 20 seconds to tokenize a GB of text on a server's CPU.
  • Easy to use, but also extremely versatile.
  • Designed for research and production.
  • Normalization comes with alignments tracking. It's always possible to get the part of the original sentence that corresponds to a given token.
  • Does all the pre-processing: Truncate, Pad, add the special tokens your model needs.

Installation

With pip:

pip install tokenizers

From sources:

To use this method, you need to have the Rust installed:

# Install with:
curl https://sh.rustup.rs -sSf | sh -s -- -y
export PATH="$HOME/.cargo/bin:$PATH"

Once Rust is installed, you can compile doing the following

git clone https://github.com/huggingface/tokenizers
cd tokenizers/bindings/python

# Create a virtual env (you can use yours as well)
python -m venv .env
source .env/bin/activate

# Install `tokenizers` in the current virtual env
pip install setuptools_rust
python setup.py install

Using the provided Tokenizers

Using a pre-trained tokenizer is really simple:

from tokenizers import CharBPETokenizer

# Initialize a tokenizer
vocab = "./path/to/vocab.json"
merges = "./path/to/merges.txt"
tokenizer = CharBPETokenizer(vocab, merges)

# And then encode:
encoded = tokenizer.encode("I can feel the magic, can you?")
print(encoded.ids)
print(encoded.tokens)

And you can train yours just as simply:

from tokenizers import CharBPETokenizer

# Initialize a tokenizer
tokenizer = CharBPETokenizer()

# Then train it!
tokenizer.train([ "./path/to/files/1.txt", "./path/to/files/2.txt" ])

# And you can use it
encoded = tokenizer.encode("I can feel the magic, can you?")

# And finally save it somewhere
tokenizer.save("./path/to/directory", "my-bpe")

Provided Tokenizers

  • CharBPETokenizer: The original BPE
  • ByteLevelBPETokenizer: The byte level version of the BPE
  • SentencePieceBPETokenizer: A BPE implementation compatible with the one used by SentencePiece
  • BertWordPieceTokenizer: The famous Bert tokenizer, using WordPiece

All of these can be used and trained as explained above!

Build your own

You can also easily build your own tokenizers, by putting all the different parts you need together:

Use a pre-trained tokenizer

from tokenizers import Tokenizer, models, pre_tokenizers, decoders, processors

# Load a BPE Model
vocab = "./path/to/vocab.json"
merges = "./path/to/merges.txt"
bpe = models.BPE(vocab, merges)

# Initialize a tokenizer
tokenizer = Tokenizer(bpe)

# Customize pre-tokenization and decoding
tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=True)
tokenizer.decoder = decoders.ByteLevel()
tokenizer.post_processor = processors.ByteLevel(trim_offsets=True)

# And then encode:
encoded = tokenizer.encode("I can feel the magic, can you?")
print(encoded.ids)
print(encoded.tokens)

# Or tokenize multiple sentences at once:
encoded = tokenizer.encode_batch([
	"I can feel the magic, can you?",
	"The quick brown fox jumps over the lazy dog"
])
print(encoded)

Train a new tokenizer

from tokenizers import Tokenizer, models, pre_tokenizers, decoders, trainers, processors

# Initialize a tokenizer
tokenizer = Tokenizer(models.BPE())

# Customize pre-tokenization and decoding
tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=True)
tokenizer.decoder = decoders.ByteLevel()
tokenizer.post_processor = processors.ByteLevel(trim_offsets=True)

# And then train
trainer = trainers.BpeTrainer(vocab_size=20000, min_frequency=2)
tokenizer.train(trainer, [
	"./path/to/dataset/1.txt",
	"./path/to/dataset/2.txt",
	"./path/to/dataset/3.txt"
])

# Now we can encode
encoded = tokenizer.encode("I can feel the magic, can you?")
print(encoded)

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for ArthurZucker from gravatar.com ArthurZucker Avatar for McPotato from gravatar.com McPotato Avatar for Nicolas.Patry from gravatar.com Nicolas.Patry Avatar for xn1t0x from gravatar.com xn1t0x

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: Apache Software License (Apache License 2.0)
  • Author: Anthony MOI
  • Tags NLP, tokenizer, BPE, transformer, deep, learning
Classifiers

Release history Release notifications | RSS feed

0.20.3

0.20.3rc0 pre-release

0.20.2

0.20.1

0.20.1rc1 pre-release

0.20.0

0.20.0rc1 pre-release

0.19.1

0.19.0

0.15.2

0.15.2rc1 pre-release

0.15.1

0.15.0

0.14.1

0.14.0

0.13.3

0.13.2

0.13.1

0.13.0

0.12.1

0.12.0 yanked

Reason this release was yanked:

Breaking change with unexpected effect

0.11.6

0.11.5

0.11.4

0.11.3

0.11.2

0.11.1

0.11.0

0.10.3

0.10.2

0.10.1

0.10.1rc1 pre-release

0.10.0

0.10.0rc1 pre-release

0.9.4

0.9.3

0.9.2

0.9.1

0.9.0

0.9.0rc2 pre-release

0.9.0rc1 pre-release

0.9.0.dev4 pre-release

0.9.0.dev3 pre-release

0.9.0.dev2 pre-release

0.9.0.dev1 pre-release

0.9.0.dev0 pre-release

0.8.1

0.8.1rc2 pre-release

0.8.1rc1 pre-release

0.8.0

0.8.0rc4 pre-release

0.8.0rc3 pre-release

0.8.0rc2 pre-release

0.8.0rc1 pre-release

0.8.0.dev2 pre-release

0.8.0.dev1 pre-release

0.8.0.dev0 pre-release

0.7.0

0.7.0rc7 pre-release

0.7.0rc6 pre-release

0.7.0rc5 pre-release

This version

0.7.0rc4 pre-release

0.7.0rc3 pre-release

0.7.0rc2 pre-release

0.7.0rc1 pre-release

0.6.0

0.5.2

0.5.1

0.5.0

0.4.2

0.4.1

0.4.0

0.3.0

0.2.1

0.2.0

0.1.1

0.1.0

0.0.13

0.0.12

0.0.11

0.0.10

0.0.9

0.0.8

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

0.0.2

0.0.1

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tokenizers-0.7.0rc4.tar.gz (79.7 kB view details)

Uploaded Source

Built Distributions

tokenizers-0.7.0rc4-cp38-cp38-win_amd64.whl (1.1 MB view details)

Uploaded CPython 3.8 Windows x86-64

tokenizers-0.7.0rc4-cp38-cp38-win32.whl (962.0 kB view details)

Uploaded CPython 3.8 Windows x86

tokenizers-0.7.0rc4-cp38-cp38-manylinux1_x86_64.whl (7.5 MB view details)

Uploaded CPython 3.8

tokenizers-0.7.0rc4-cp38-cp38-macosx_10_10_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.8 macOS 10.10+ x86-64

tokenizers-0.7.0rc4-cp37-cp37m-win_amd64.whl (1.1 MB view details)

Uploaded CPython 3.7m Windows x86-64

tokenizers-0.7.0rc4-cp37-cp37m-win32.whl (961.2 kB view details)

Uploaded CPython 3.7m Windows x86

tokenizers-0.7.0rc4-cp37-cp37m-manylinux1_x86_64.whl (5.6 MB view details)

Uploaded CPython 3.7m

tokenizers-0.7.0rc4-cp37-cp37m-macosx_10_10_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.7m macOS 10.10+ x86-64

tokenizers-0.7.0rc4-cp36-cp36m-win_amd64.whl (1.1 MB view details)

Uploaded CPython 3.6m Windows x86-64

tokenizers-0.7.0rc4-cp36-cp36m-win32.whl (961.1 kB view details)

Uploaded CPython 3.6m Windows x86

tokenizers-0.7.0rc4-cp36-cp36m-manylinux1_x86_64.whl (3.8 MB view details)

Uploaded CPython 3.6m

tokenizers-0.7.0rc4-cp36-cp36m-macosx_10_10_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.6m macOS 10.10+ x86-64

tokenizers-0.7.0rc4-cp35-cp35m-win_amd64.whl (1.1 MB view details)

Uploaded CPython 3.5m Windows x86-64

tokenizers-0.7.0rc4-cp35-cp35m-win32.whl (961.2 kB view details)

Uploaded CPython 3.5m Windows x86

tokenizers-0.7.0rc4-cp35-cp35m-manylinux1_x86_64.whl (1.9 MB view details)

Uploaded CPython 3.5m

tokenizers-0.7.0rc4-cp35-cp35m-macosx_10_10_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.5m macOS 10.10+ x86-64

File details

Details for the file tokenizers-0.7.0rc4.tar.gz.

File metadata

  • Download URL: tokenizers-0.7.0rc4.tar.gz
  • Upload date:
  • Size: 79.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2

File hashes

Hashes for tokenizers-0.7.0rc4.tar.gz
Algorithm Hash digest
SHA256 e7e98b8c200051f0c0fa393f172b6fae903c4225ee93e47b47db9bc58ae26a2b
MD5 1ae65bd31581477cc0feb90813b64123
BLAKE2b-256 37816b6266ab6e636ac32dc42d1ae9bf83dc04193b2488c3d87d603a925ac615

See more details on using hashes here.

File details

Details for the file tokenizers-0.7.0rc4-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: tokenizers-0.7.0rc4-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 1.1 MB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2

File hashes

Hashes for tokenizers-0.7.0rc4-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 2c193267c528323fdfd36fd1a6f5e9f8a97bb320d56c4a032bae7b7f19b419ea
MD5 7f7ba8ff45b73a47c0ce198457c37421
BLAKE2b-256 c7d98b754a319c29fbe236cab11425851971e818c6cf987ff89047212628fc80

See more details on using hashes here.

File details

Details for the file tokenizers-0.7.0rc4-cp38-cp38-win32.whl.

File metadata

  • Download URL: tokenizers-0.7.0rc4-cp38-cp38-win32.whl
  • Upload date:
  • Size: 962.0 kB
  • Tags: CPython 3.8, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2

File hashes

Hashes for tokenizers-0.7.0rc4-cp38-cp38-win32.whl
Algorithm Hash digest
SHA256 9d834cd9232224fed0cbad3bed2bc05e80c1f89d09921bd73d293138cac59062
MD5 59376f1b3747b9919ecdb2ea58974fb6
BLAKE2b-256 a4c7b9626abd5af3b0a5aaba24e5e753a2024b984a671eb648d9990d6adbe83c

See more details on using hashes here.

File details

Details for the file tokenizers-0.7.0rc4-cp38-cp38-manylinux1_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.7.0rc4-cp38-cp38-manylinux1_x86_64.whl
  • Upload date:
  • Size: 7.5 MB
  • Tags: CPython 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2

File hashes

Hashes for tokenizers-0.7.0rc4-cp38-cp38-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 d65f89232875fe7e115c0e2cf714a978b289f09a35b527a8a0cec97306667579
MD5 b4495dd30629cba4ace81b83fcf9e041
BLAKE2b-256 db7c2b3b39529cf08e83452c2424484bd1afe40dd2b276ed073563dc985d3025

See more details on using hashes here.

File details

Details for the file tokenizers-0.7.0rc4-cp38-cp38-macosx_10_10_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.7.0rc4-cp38-cp38-macosx_10_10_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.8, macOS 10.10+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2

File hashes

Hashes for tokenizers-0.7.0rc4-cp38-cp38-macosx_10_10_x86_64.whl
Algorithm Hash digest
SHA256 13572311aeb8f55b31fc8e40bc160abe6846093930103675b0ec58e13a0fe9bb
MD5 0585fb88e8e99401a0c091e920135ed4
BLAKE2b-256 dc0886761688ac7785ebe478e394d597677e2c7744806968f278f36015985f1e

See more details on using hashes here.

File details

Details for the file tokenizers-0.7.0rc4-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: tokenizers-0.7.0rc4-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 1.1 MB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2

File hashes

Hashes for tokenizers-0.7.0rc4-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 fb274ee8f1e82edbe1631c020033fb9dfc8c5ea997baa6c30a40f834e24e4ecd
MD5 274bb8c4fe2f9fe9790db3c4f439bfbc
BLAKE2b-256 5aacd35c80e976e743b00ffb9e5f4c1ba9d64cfd06b9ea98bf98f99942fddd65

See more details on using hashes here.

File details

Details for the file tokenizers-0.7.0rc4-cp37-cp37m-win32.whl.

File metadata

  • Download URL: tokenizers-0.7.0rc4-cp37-cp37m-win32.whl
  • Upload date:
  • Size: 961.2 kB
  • Tags: CPython 3.7m, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2

File hashes

Hashes for tokenizers-0.7.0rc4-cp37-cp37m-win32.whl
Algorithm Hash digest
SHA256 09ea6ffb15ade5ab1e27dafed95a8f4d7b91105a05291d46f099c41a90410601
MD5 eceff3165046e8eb3a9308010414dbf5
BLAKE2b-256 b580cb081cacea2dff9a1dc89622ca8bf5dbd346f1e2066e46adc6cbca025f68

See more details on using hashes here.

File details

Details for the file tokenizers-0.7.0rc4-cp37-cp37m-manylinux1_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.7.0rc4-cp37-cp37m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 5.6 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2

File hashes

Hashes for tokenizers-0.7.0rc4-cp37-cp37m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 e45a37bed1b1d76b89a236986127f9cd3671c4b4c3b60a9d1ca302b6600ed301
MD5 6cacea0b984abf8b7004fd07ddd751f5
BLAKE2b-256 729ff9ce9154819e733843dba863dc1dd2992a0739afddbe9eb1bbbed12ca27f

See more details on using hashes here.

File details

Details for the file tokenizers-0.7.0rc4-cp37-cp37m-macosx_10_10_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.7.0rc4-cp37-cp37m-macosx_10_10_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.7m, macOS 10.10+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2

File hashes

Hashes for tokenizers-0.7.0rc4-cp37-cp37m-macosx_10_10_x86_64.whl
Algorithm Hash digest
SHA256 110a79d8db65bda0b6af8ec4803be7211690e883e929b51068ba83927e145fbe
MD5 b3a7dc0f3d71870b690f0c974f0e5645
BLAKE2b-256 b3fa750cd10bbcf823cd0cee5d6ba4ce40252ea9a0ae0d195c802d22c2ffc670

See more details on using hashes here.

File details

Details for the file tokenizers-0.7.0rc4-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: tokenizers-0.7.0rc4-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 1.1 MB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2

File hashes

Hashes for tokenizers-0.7.0rc4-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 ae2dae3384f60f1f9e04f102c4e144fb80d2fcb8feb6b134de19611da93a5bf8
MD5 0bf55c5281f5bd48ad9148d35c57a1fa
BLAKE2b-256 f02d8c5b4049502095b27b7850b84f8af848f741d99c57107cb2e35776d0c644

See more details on using hashes here.

File details

Details for the file tokenizers-0.7.0rc4-cp36-cp36m-win32.whl.

File metadata

  • Download URL: tokenizers-0.7.0rc4-cp36-cp36m-win32.whl
  • Upload date:
  • Size: 961.1 kB
  • Tags: CPython 3.6m, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2

File hashes

Hashes for tokenizers-0.7.0rc4-cp36-cp36m-win32.whl
Algorithm Hash digest
SHA256 b6d1398e803f30d3076fa311db45508c0cd3b08b9ff68c8cc0a32dd2691e9351
MD5 1ef5851848fdd1c9d7cb23267e3430b3
BLAKE2b-256 9741da25b98ca5754f45d46b4a3454c15dbbcb83f3583044876227ba7c148e68

See more details on using hashes here.

File details

Details for the file tokenizers-0.7.0rc4-cp36-cp36m-manylinux1_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.7.0rc4-cp36-cp36m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 3.8 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2

File hashes

Hashes for tokenizers-0.7.0rc4-cp36-cp36m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 247563f34126cd6b52b2fe45e85451c68930a8eb51681684415edf4dec5128e5
MD5 8a242fb66b305b0e3fcfac8b915ad400
BLAKE2b-256 3c6b495a8375725f1bcd14eef25f89d80bd242831cb48bbacf18e97f762990ed

See more details on using hashes here.

File details

Details for the file tokenizers-0.7.0rc4-cp36-cp36m-macosx_10_10_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.7.0rc4-cp36-cp36m-macosx_10_10_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.6m, macOS 10.10+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2

File hashes

Hashes for tokenizers-0.7.0rc4-cp36-cp36m-macosx_10_10_x86_64.whl
Algorithm Hash digest
SHA256 64291f121a5a865616144f6eff55d5e333b913d777f873d3ebc5527b5c2fc6d1
MD5 4a2c36938d8024574fbe494b3f9aba7f
BLAKE2b-256 33083b5b2275b20b997628a18641bf1194ce26154a0d67320edcb2e395532269

See more details on using hashes here.

File details

Details for the file tokenizers-0.7.0rc4-cp35-cp35m-win_amd64.whl.

File metadata

  • Download URL: tokenizers-0.7.0rc4-cp35-cp35m-win_amd64.whl
  • Upload date:
  • Size: 1.1 MB
  • Tags: CPython 3.5m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2

File hashes

Hashes for tokenizers-0.7.0rc4-cp35-cp35m-win_amd64.whl
Algorithm Hash digest
SHA256 ec245076393d818e8a469c5d20110b76724939a634974d57430f5a551da739f6
MD5 097e7a22c7ef40ed7f75bb3b7264f705
BLAKE2b-256 6ed9ad6d2c0e6c4c9e9aeb680b9b4be092727d51c16b3c6bd39e528161b28b25

See more details on using hashes here.

File details

Details for the file tokenizers-0.7.0rc4-cp35-cp35m-win32.whl.

File metadata

  • Download URL: tokenizers-0.7.0rc4-cp35-cp35m-win32.whl
  • Upload date:
  • Size: 961.2 kB
  • Tags: CPython 3.5m, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2

File hashes

Hashes for tokenizers-0.7.0rc4-cp35-cp35m-win32.whl
Algorithm Hash digest
SHA256 0845cf8b9ec1202358465c7749d7d396b754e1e662a7812c68c4beb8b022b098
MD5 dc13fa55a54f3008bad0a5aba258bf9e
BLAKE2b-256 5dad340553d92bebf067e786ef2150676d8a03fbfd3cb494461c2f195636ae80

See more details on using hashes here.

File details

Details for the file tokenizers-0.7.0rc4-cp35-cp35m-manylinux1_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.7.0rc4-cp35-cp35m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.9 MB
  • Tags: CPython 3.5m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2

File hashes

Hashes for tokenizers-0.7.0rc4-cp35-cp35m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 3eaf6cc8c83a069aaa698bd796009f1d7a8b3c7c2ef772970323f072733e8683
MD5 ce3bc527a2a8da8e3223c027b823145b
BLAKE2b-256 562b5e01e89f03b40755ef9dc57406fe4211180bdf508371454d1205d1a17c15

See more details on using hashes here.

File details

Details for the file tokenizers-0.7.0rc4-cp35-cp35m-macosx_10_10_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.7.0rc4-cp35-cp35m-macosx_10_10_x86_64.whl
  • Upload date:
  • Size: 1.2 MB
  • Tags: CPython 3.5m, macOS 10.10+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.2

File hashes

Hashes for tokenizers-0.7.0rc4-cp35-cp35m-macosx_10_10_x86_64.whl
Algorithm Hash digest
SHA256 f742287ca9b4ff0b29caf39562d967999394dbd11e801a18a4bab79210113201
MD5 fb35d4d4790c6085174e9a097569a4c8
BLAKE2b-256 edb81a1e61935d7c6ef72764261de91cd408d8f5f072b3a4c5080001df3c4e1e

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page