Fast and Customizable Tokenizers

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for ArthurZucker from gravatar.com ArthurZucker Avatar for McPotato from gravatar.com McPotato Avatar for Nicolas.Patry from gravatar.com Nicolas.Patry Avatar for xn1t0x from gravatar.com xn1t0x

Unverified details

These details have not been verified by PyPI
Project links
Meta

Project description

The author of this package has not provided a project description

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for ArthurZucker from gravatar.com ArthurZucker Avatar for McPotato from gravatar.com McPotato Avatar for Nicolas.Patry from gravatar.com Nicolas.Patry Avatar for xn1t0x from gravatar.com xn1t0x

Unverified details

These details have not been verified by PyPI
Project links
Meta

Release history Release notifications | RSS feed

0.20.3

0.20.3rc0 pre-release

0.20.2

0.20.1

0.20.1rc1 pre-release

0.20.0

0.20.0rc1 pre-release

0.19.1

0.19.0

0.15.2

0.15.2rc1 pre-release

0.15.1

0.15.0

0.14.1

0.14.0

0.13.3

0.13.2

0.13.1

0.13.0

0.12.1

0.12.0 yanked

Reason this release was yanked:

Breaking change with unexpected effect

0.11.6

0.11.5

0.11.4

0.11.3

0.11.2

0.11.1

0.11.0

0.10.3

0.10.2

0.10.1

0.10.1rc1 pre-release

0.10.0

0.10.0rc1 pre-release

0.9.4

0.9.3

0.9.2

0.9.1

0.9.0

0.9.0rc2 pre-release

0.9.0rc1 pre-release

0.9.0.dev4 pre-release

0.9.0.dev3 pre-release

0.9.0.dev2 pre-release

0.9.0.dev1 pre-release

0.9.0.dev0 pre-release

0.8.1

0.8.1rc2 pre-release

0.8.1rc1 pre-release

0.8.0

0.8.0rc4 pre-release

0.8.0rc3 pre-release

0.8.0rc2 pre-release

0.8.0rc1 pre-release

0.8.0.dev2 pre-release

0.8.0.dev1 pre-release

0.8.0.dev0 pre-release

0.7.0

0.7.0rc7 pre-release

0.7.0rc6 pre-release

0.7.0rc5 pre-release

0.7.0rc4 pre-release

0.7.0rc3 pre-release

0.7.0rc2 pre-release

0.7.0rc1 pre-release

0.6.0

0.5.2

0.5.1

0.5.0

0.4.2

0.4.1

0.4.0

0.3.0

0.2.1

0.2.0

0.1.1

0.1.0

0.0.13

0.0.12

0.0.11

0.0.10

0.0.9

0.0.8

0.0.7

0.0.6

0.0.5

0.0.4

0.0.3

This version

0.0.2

0.0.1

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tokenizers-0.0.2.tar.gz (18.3 kB view details)

Uploaded Source

Built Distributions

tokenizers-0.0.2-cp38-cp38-win_amd64.whl (591.3 kB view details)

Uploaded CPython 3.8 Windows x86-64

tokenizers-0.0.2-cp38-cp38-manylinux1_x86_64.whl (5.2 MB view details)

Uploaded CPython 3.8

tokenizers-0.0.2-cp38-cp38-macosx_10_13_x86_64.whl (658.7 kB view details)

Uploaded CPython 3.8 macOS 10.13+ x86-64

tokenizers-0.0.2-cp37-cp37m-win_amd64.whl (591.3 kB view details)

Uploaded CPython 3.7m Windows x86-64

tokenizers-0.0.2-cp37-cp37m-manylinux1_x86_64.whl (3.9 MB view details)

Uploaded CPython 3.7m

tokenizers-0.0.2-cp37-cp37m-macosx_10_13_x86_64.whl (658.6 kB view details)

Uploaded CPython 3.7m macOS 10.13+ x86-64

tokenizers-0.0.2-cp36-cp36m-win_amd64.whl (591.7 kB view details)

Uploaded CPython 3.6m Windows x86-64

tokenizers-0.0.2-cp36-cp36m-manylinux1_x86_64.whl (2.6 MB view details)

Uploaded CPython 3.6m

tokenizers-0.0.2-cp36-cp36m-macosx_10_13_x86_64.whl (658.8 kB view details)

Uploaded CPython 3.6m macOS 10.13+ x86-64

tokenizers-0.0.2-cp35-cp35m-win_amd64.whl (591.7 kB view details)

Uploaded CPython 3.5m Windows x86-64

tokenizers-0.0.2-cp35-cp35m-manylinux1_x86_64.whl (1.3 MB view details)

Uploaded CPython 3.5m

tokenizers-0.0.2-cp35-cp35m-macosx_10_13_x86_64.whl (658.8 kB view details)

Uploaded CPython 3.5m macOS 10.13+ x86-64

File details

Details for the file tokenizers-0.0.2.tar.gz.

File metadata

  • Download URL: tokenizers-0.0.2.tar.gz
  • Upload date:
  • Size: 18.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.40.0 CPython/3.8.0

File hashes

Hashes for tokenizers-0.0.2.tar.gz
Algorithm Hash digest
SHA256 5b79fa284e0a01c34da9b3fae8a45f460b287526015182ee4b1c73e2676bd3c7
MD5 dbe4ca8927202bd28d7b7ff32d0faf4b
BLAKE2b-256 4b6feaf987d8fd647ee2b738c130006b2ccded394ee966199cf3f432bbb8d1d0

See more details on using hashes here.

File details

Details for the file tokenizers-0.0.2-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: tokenizers-0.0.2-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 591.3 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.40.0 CPython/3.8.0

File hashes

Hashes for tokenizers-0.0.2-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 7621c4c2e24168a99e248bec139588c444ec5d3fa24709fd66a07910c3650b83
MD5 d3eabc2fbdb57a1d454153371f173f4b
BLAKE2b-256 8313b4893104f9cf38995064bfe998e572b2d1a701a185ed8724269dfb419057

See more details on using hashes here.

File details

Details for the file tokenizers-0.0.2-cp38-cp38-manylinux1_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.0.2-cp38-cp38-manylinux1_x86_64.whl
  • Upload date:
  • Size: 5.2 MB
  • Tags: CPython 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.40.0 CPython/3.8.0

File hashes

Hashes for tokenizers-0.0.2-cp38-cp38-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 5587d6b12a159bd3ac44a150d9039fb19bb721b3c8bc45a92fec11cf84876f4a
MD5 790e2dc3cb3b92670ff6ceebfe80974a
BLAKE2b-256 7a8af1e0b4995a94bf7a1989d58915e4da390ed8cb9135ac7caff56f73fede50

See more details on using hashes here.

File details

Details for the file tokenizers-0.0.2-cp38-cp38-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.0.2-cp38-cp38-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 658.7 kB
  • Tags: CPython 3.8, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.40.0 CPython/3.8.0

File hashes

Hashes for tokenizers-0.0.2-cp38-cp38-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 d78e818be071c8234d196b75701f0c61cf1ab7edff690c7eb081f74a2cd2fc51
MD5 536392c724b174d73d640536be514606
BLAKE2b-256 1272477115beee1e51a4a526b34086456f59c7e14dae15045c76be2aa6fa3046

See more details on using hashes here.

File details

Details for the file tokenizers-0.0.2-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: tokenizers-0.0.2-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 591.3 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.40.0 CPython/3.8.0

File hashes

Hashes for tokenizers-0.0.2-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 5af14c7572388e6139ec27a59b0bea80a4fede5e02bf35c3cda1f0b8795edecc
MD5 6505019364607ce83ed6125244ced620
BLAKE2b-256 3adb4db2f0d1cb90d293dadc091a37efcf5849ac1f185ed82c18dd4ae2f330ee

See more details on using hashes here.

File details

Details for the file tokenizers-0.0.2-cp37-cp37m-manylinux1_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.0.2-cp37-cp37m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 3.9 MB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.40.0 CPython/3.8.0

File hashes

Hashes for tokenizers-0.0.2-cp37-cp37m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 fc773c5ace59398ffe628ff50e36a550467e31486fd1cd54cee8d4c16ee7cf9d
MD5 f7353495b84b3a3dc7ca191b475aaaa2
BLAKE2b-256 e9fc3112b7092c0472960d5274467b6d2587835ff4c6e9e7406c600143e22d4a

See more details on using hashes here.

File details

Details for the file tokenizers-0.0.2-cp37-cp37m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.0.2-cp37-cp37m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 658.6 kB
  • Tags: CPython 3.7m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.40.0 CPython/3.8.0

File hashes

Hashes for tokenizers-0.0.2-cp37-cp37m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 fdcf5518126f31e2793475b900b8130c1ec0eb5c813fea9a6429d010468a66d4
MD5 032ea971b49974ec68243edf152dee95
BLAKE2b-256 b1871c255a9b2d42fc75bdf7f637378b051eb9dd851378427bf54948dec399e2

See more details on using hashes here.

File details

Details for the file tokenizers-0.0.2-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: tokenizers-0.0.2-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 591.7 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.40.0 CPython/3.8.0

File hashes

Hashes for tokenizers-0.0.2-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 c80bac8cef483f255f0ca1764e991592535ae220d0c10091ea9eedbc8644cb94
MD5 aa8e62e661cea90bb1170d297f98ef46
BLAKE2b-256 e07fbab0970798da098a8103cadd0d61b868ddb0e855943b9bb2bdef1166ed39

See more details on using hashes here.

File details

Details for the file tokenizers-0.0.2-cp36-cp36m-manylinux1_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.0.2-cp36-cp36m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 2.6 MB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.40.0 CPython/3.8.0

File hashes

Hashes for tokenizers-0.0.2-cp36-cp36m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 8b8a9430acaa61efe6048a51606b2b998e98fcd3a343aa3180734760672ab2a5
MD5 5fb6e00ed17f3327ad62509e08bcd8b9
BLAKE2b-256 f196cfdf66bef9408f697feb2205702b5c6e6f9aacb742a7b8734eb36ca0503f

See more details on using hashes here.

File details

Details for the file tokenizers-0.0.2-cp36-cp36m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.0.2-cp36-cp36m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 658.8 kB
  • Tags: CPython 3.6m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.40.0 CPython/3.8.0

File hashes

Hashes for tokenizers-0.0.2-cp36-cp36m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 371b33615a8eb3637e41b766cb5d0abbb530c2d5dc2aa8bd9a8e720b412b3e27
MD5 57564af795342a216576ae557a728527
BLAKE2b-256 9fa833f0c7723b84ed26688b8afe92dca43d16085656f4745cf9e4262b2c4795

See more details on using hashes here.

File details

Details for the file tokenizers-0.0.2-cp35-cp35m-win_amd64.whl.

File metadata

  • Download URL: tokenizers-0.0.2-cp35-cp35m-win_amd64.whl
  • Upload date:
  • Size: 591.7 kB
  • Tags: CPython 3.5m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.40.0 CPython/3.8.0

File hashes

Hashes for tokenizers-0.0.2-cp35-cp35m-win_amd64.whl
Algorithm Hash digest
SHA256 d7df6ccafe28e41addfe612e7f0b8ff41c4fb15bf6f040e84c49a7d801d8c9af
MD5 5a908537be2acb6f80cae71432b02332
BLAKE2b-256 da957037760fdcfbb2c1834901f10cb4961f6f94d02e3bea544119aeef52435a

See more details on using hashes here.

File details

Details for the file tokenizers-0.0.2-cp35-cp35m-manylinux1_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.0.2-cp35-cp35m-manylinux1_x86_64.whl
  • Upload date:
  • Size: 1.3 MB
  • Tags: CPython 3.5m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.40.0 CPython/3.8.0

File hashes

Hashes for tokenizers-0.0.2-cp35-cp35m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 21a992a740ab6d98e2d9e80fbe6e476fd57f95583d5181451aa5f4ab03932785
MD5 8f5f8ab851b48d8cfd26f6ae227f5e72
BLAKE2b-256 4ca1170f7695a977bdaae9ce59c4a9071ae1057b35f75c33a0e31cca336853c5

See more details on using hashes here.

File details

Details for the file tokenizers-0.0.2-cp35-cp35m-macosx_10_13_x86_64.whl.

File metadata

  • Download URL: tokenizers-0.0.2-cp35-cp35m-macosx_10_13_x86_64.whl
  • Upload date:
  • Size: 658.8 kB
  • Tags: CPython 3.5m, macOS 10.13+ x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.40.0 CPython/3.8.0

File hashes

Hashes for tokenizers-0.0.2-cp35-cp35m-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 de7f34c0bdd068c5ecb44d8b6b0bf2641849804333e6c8a267d3d8717370cd00
MD5 d00336fe0c69743d150f6a782abbe5a6
BLAKE2b-256 f43f2649acc41f9564722c8175defd57fb2943dd58cd262d14727e8f07dc9dad

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page