Skip to main content

tiktoken is a fast BPE tokeniser for use with OpenAI's models

Project description

⏳ tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

import tiktoken
enc = tiktoken.get_encoding("o200k_base")
assert enc.decode(enc.encode("hello world")) == "hello world"

# To get the tokeniser corresponding to a specific model in the OpenAI API:
enc = tiktoken.encoding_for_model("gpt-4o")

The open source version of tiktoken can be installed from PyPI:

pip install tiktoken

The tokeniser API is documented in tiktoken/core.py.

Example code using tiktoken can be found in the OpenAI Cookbook.

Performance

tiktoken is between 3-6x faster than a comparable open source tokeniser:

image

Performance measured on 1GB of text using the GPT-2 tokeniser, using GPT2TokenizerFast from tokenizers==0.13.2, transformers==4.24.0 and tiktoken==0.2.0.

Getting help

Please post questions in the issue tracker.

If you work at OpenAI, make sure to check the internal documentation or feel free to contact @shantanu.

What is BPE anyway?

Language models don't see text like you and I, instead they see a sequence of numbers (known as tokens). Byte pair encoding (BPE) is a way of converting text into tokens. It has a couple desirable properties:

  1. It's reversible and lossless, so you can convert tokens back into the original text
  2. It works on arbitrary text, even text that is not in the tokeniser's training data
  3. It compresses the text: the token sequence is shorter than the bytes corresponding to the original text. On average, in practice, each token corresponds to about 4 bytes.
  4. It attempts to let the model see common subwords. For instance, "ing" is a common subword in English, so BPE encodings will often split "encoding" into tokens like "encod" and "ing" (instead of e.g. "enc" and "oding"). Because the model will then see the "ing" token again and again in different contexts, it helps models generalise and better understand grammar.

tiktoken contains an educational submodule that is friendlier if you want to learn more about the details of BPE, including code that helps visualise the BPE procedure:

from tiktoken._educational import *

# Train a BPE tokeniser on a small amount of text
enc = train_simple_encoding()

# Visualise how the GPT-4 encoder encodes text
enc = SimpleBytePairEncoding.from_tiktoken("cl100k_base")
enc.encode("hello world aaaaaaaaaaaa")

Extending tiktoken

You may wish to extend tiktoken to support new encodings. There are two ways to do this.

Create your Encoding object exactly the way you want and simply pass it around.

cl100k_base = tiktoken.get_encoding("cl100k_base")

# In production, load the arguments directly instead of accessing private attributes
# See openai_public.py for examples of arguments for specific encodings
enc = tiktoken.Encoding(
    # If you're changing the set of special tokens, make sure to use a different name
    # It should be clear from the name what behaviour to expect.
    name="cl100k_im",
    pat_str=cl100k_base._pat_str,
    mergeable_ranks=cl100k_base._mergeable_ranks,
    special_tokens={
        **cl100k_base._special_tokens,
        "<|im_start|>": 100264,
        "<|im_end|>": 100265,
    }
)

Use the tiktoken_ext plugin mechanism to register your Encoding objects with tiktoken.

This is only useful if you need tiktoken.get_encoding to find your encoding, otherwise prefer option 1.

To do this, you'll need to create a namespace package under tiktoken_ext.

Layout your project like this, making sure to omit the tiktoken_ext/__init__.py file:

my_tiktoken_extension
├── tiktoken_ext
│   └── my_encodings.py
└── setup.py

my_encodings.py should be a module that contains a variable named ENCODING_CONSTRUCTORS. This is a dictionary from an encoding name to a function that takes no arguments and returns arguments that can be passed to tiktoken.Encoding to construct that encoding. For an example, see tiktoken_ext/openai_public.py. For precise details, see tiktoken/registry.py.

Your setup.py should look something like this:

from setuptools import setup, find_namespace_packages

setup(
    name="my_tiktoken_extension",
    packages=find_namespace_packages(include=['tiktoken_ext*']),
    install_requires=["tiktoken"],
    ...
)

Then simply pip install ./my_tiktoken_extension and you should be able to use your custom encodings! Make sure not to use an editable install.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tiktoken-0.8.0.tar.gz (35.1 kB view details)

Uploaded Source

Built Distributions

tiktoken-0.8.0-cp313-cp313-win_amd64.whl (883.8 kB view details)

Uploaded CPython 3.13 Windows x86-64

tiktoken-0.8.0-cp313-cp313-musllinux_1_2_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.13 musllinux: musl 1.2+ x86-64

tiktoken-0.8.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.13 manylinux: glibc 2.17+ x86-64

tiktoken-0.8.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.1 MB view details)

Uploaded CPython 3.13 manylinux: glibc 2.17+ ARM64

tiktoken-0.8.0-cp313-cp313-macosx_11_0_arm64.whl (982.8 kB view details)

Uploaded CPython 3.13 macOS 11.0+ ARM64

tiktoken-0.8.0-cp313-cp313-macosx_10_13_x86_64.whl (1.0 MB view details)

Uploaded CPython 3.13 macOS 10.13+ x86-64

tiktoken-0.8.0-cp312-cp312-win_amd64.whl (883.8 kB view details)

Uploaded CPython 3.12 Windows x86-64

tiktoken-0.8.0-cp312-cp312-musllinux_1_2_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.12 musllinux: musl 1.2+ x86-64

tiktoken-0.8.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.12 manylinux: glibc 2.17+ x86-64

tiktoken-0.8.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.1 MB view details)

Uploaded CPython 3.12 manylinux: glibc 2.17+ ARM64

tiktoken-0.8.0-cp312-cp312-macosx_11_0_arm64.whl (982.6 kB view details)

Uploaded CPython 3.12 macOS 11.0+ ARM64

tiktoken-0.8.0-cp312-cp312-macosx_10_13_x86_64.whl (1.0 MB view details)

Uploaded CPython 3.12 macOS 10.13+ x86-64

tiktoken-0.8.0-cp311-cp311-win_amd64.whl (884.5 kB view details)

Uploaded CPython 3.11 Windows x86-64

tiktoken-0.8.0-cp311-cp311-musllinux_1_2_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.11 musllinux: musl 1.2+ x86-64

tiktoken-0.8.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

tiktoken-0.8.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.1 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ ARM64

tiktoken-0.8.0-cp311-cp311-macosx_11_0_arm64.whl (982.4 kB view details)

Uploaded CPython 3.11 macOS 11.0+ ARM64

tiktoken-0.8.0-cp311-cp311-macosx_10_9_x86_64.whl (1.0 MB view details)

Uploaded CPython 3.11 macOS 10.9+ x86-64

tiktoken-0.8.0-cp310-cp310-win_amd64.whl (884.2 kB view details)

Uploaded CPython 3.10 Windows x86-64

tiktoken-0.8.0-cp310-cp310-musllinux_1_2_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.10 musllinux: musl 1.2+ x86-64

tiktoken-0.8.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

tiktoken-0.8.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.1 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ARM64

tiktoken-0.8.0-cp310-cp310-macosx_11_0_arm64.whl (982.4 kB view details)

Uploaded CPython 3.10 macOS 11.0+ ARM64

tiktoken-0.8.0-cp310-cp310-macosx_10_9_x86_64.whl (1.0 MB view details)

Uploaded CPython 3.10 macOS 10.9+ x86-64

tiktoken-0.8.0-cp39-cp39-win_amd64.whl (884.8 kB view details)

Uploaded CPython 3.9 Windows x86-64

tiktoken-0.8.0-cp39-cp39-musllinux_1_2_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.9 musllinux: musl 1.2+ x86-64

tiktoken-0.8.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

tiktoken-0.8.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.1 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64

tiktoken-0.8.0-cp39-cp39-macosx_11_0_arm64.whl (983.8 kB view details)

Uploaded CPython 3.9 macOS 11.0+ ARM64

tiktoken-0.8.0-cp39-cp39-macosx_10_9_x86_64.whl (1.0 MB view details)

Uploaded CPython 3.9 macOS 10.9+ x86-64

File details

Details for the file tiktoken-0.8.0.tar.gz.

File metadata

  • Download URL: tiktoken-0.8.0.tar.gz
  • Upload date:
  • Size: 35.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.11.9

File hashes

Hashes for tiktoken-0.8.0.tar.gz
Algorithm Hash digest
SHA256 9ccbb2740f24542534369c5635cfd9b2b3c2490754a78ac8831d99f89f94eeb2
MD5 89c2416ed3ccd09f84952fa7dd29c84b
BLAKE2b-256 3702576ff3a6639e755c4f70997b2d315f56d6d71e0d046f4fb64cb81a3fb099

See more details on using hashes here.

Provenance

File details

Details for the file tiktoken-0.8.0-cp313-cp313-win_amd64.whl.

File metadata

  • Download URL: tiktoken-0.8.0-cp313-cp313-win_amd64.whl
  • Upload date:
  • Size: 883.8 kB
  • Tags: CPython 3.13, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.11.9

File hashes

Hashes for tiktoken-0.8.0-cp313-cp313-win_amd64.whl
Algorithm Hash digest
SHA256 18228d624807d66c87acd8f25fc135665617cab220671eb65b50f5d70fa51f69
MD5 b85c5b17fa82afa5ff3225920ee827d2
BLAKE2b-256 405914b20465f1d1cb89cfbc96ec27e5617b2d41c79da12b5e04e96d689be2a7

See more details on using hashes here.

Provenance

File details

Details for the file tiktoken-0.8.0-cp313-cp313-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for tiktoken-0.8.0-cp313-cp313-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 5376b6f8dc4753cd81ead935c5f518fa0fbe7e133d9e25f648d8c4dabdd4bad7
MD5 d538ad7ad01f80be3e0bfa79140b4cb7
BLAKE2b-256 19eb5989e16821ee8300ef8ee13c16effc20dfc26c777d05fbb6825e3c037b81

See more details on using hashes here.

Provenance

File details

Details for the file tiktoken-0.8.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for tiktoken-0.8.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 4177faa809bd55f699e88c96d9bb4635d22e3f59d635ba6fd9ffedf7150b9953
MD5 dbd0b4a7ea66abd96017dfc47eda504e
BLAKE2b-256 abd3155d2d4514f3471a25dc1d6d20549ef254e2aa9bb5b1060809b1d3b03d3a

See more details on using hashes here.

Provenance

File details

Details for the file tiktoken-0.8.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for tiktoken-0.8.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 6b231f5e8982c245ee3065cd84a4712d64692348bc609d84467c57b4b72dcbc5
MD5 433605d4e8ede8344f67f59a5970024a
BLAKE2b-256 e4f00ecf79a279dfa41fc97d00adccf976ecc2556d3c08ef3e25e45eb31f665b

See more details on using hashes here.

Provenance

File details

Details for the file tiktoken-0.8.0-cp313-cp313-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for tiktoken-0.8.0-cp313-cp313-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 c94ff53c5c74b535b2cbf431d907fc13c678bbd009ee633a2aca269a04389f9a
MD5 17bdc613f6224b7e7c89d403bda50f21
BLAKE2b-256 b1da24cdbfc302c98663fbea66f5866f7fa1048405c7564ab88483aea97c3b1a

See more details on using hashes here.

Provenance

File details

Details for the file tiktoken-0.8.0-cp313-cp313-macosx_10_13_x86_64.whl.

File metadata

File hashes

Hashes for tiktoken-0.8.0-cp313-cp313-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 02be1666096aff7da6cbd7cdaa8e7917bfed3467cd64b38b1f112e96d3b06a24
MD5 5b16dd9c6b67824e50f3da283c2656e8
BLAKE2b-256 e338802e79ba0ee5fcbf240cd624143f57744e5d411d2e9d9ad2db70d8395986

See more details on using hashes here.

Provenance

File details

Details for the file tiktoken-0.8.0-cp312-cp312-win_amd64.whl.

File metadata

  • Download URL: tiktoken-0.8.0-cp312-cp312-win_amd64.whl
  • Upload date:
  • Size: 883.8 kB
  • Tags: CPython 3.12, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.11.9

File hashes

Hashes for tiktoken-0.8.0-cp312-cp312-win_amd64.whl
Algorithm Hash digest
SHA256 d8f3192733ac4d77977432947d563d7e1b310b96497acd3c196c9bddb36ed9db
MD5 d81d475b5c4ecfcd8e2777c1a9f26c80
BLAKE2b-256 45e239d4aa02a52bba73b2cd21ba4533c84425ff8786cc63c511d68c8897376e

See more details on using hashes here.

Provenance

File details

Details for the file tiktoken-0.8.0-cp312-cp312-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for tiktoken-0.8.0-cp312-cp312-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 294440d21a2a51e12d4238e68a5972095534fe9878be57d905c476017bff99fc
MD5 61d139037c1137d24aa175bb5675222a
BLAKE2b-256 c789926b66e9025b97e9fbabeaa59048a736fe3c3e4530a204109571104f921c

See more details on using hashes here.

Provenance

File details

Details for the file tiktoken-0.8.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for tiktoken-0.8.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 d2908c0d043a7d03ebd80347266b0e58440bdef5564f84f4d29fb235b5df3b04
MD5 00db86a186bb6a277ae637c178b9c95c
BLAKE2b-256 2632e0e3a859136e95c85a572e4806dc58bf1ddf651108ae8b97d5f3ebe1a244

See more details on using hashes here.

Provenance

File details

Details for the file tiktoken-0.8.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for tiktoken-0.8.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 9a58deb7075d5b69237a3ff4bb51a726670419db6ea62bdcd8bd80c78497d7ab
MD5 b0af0d1ed4dd23868e660b5b0ee5f8ea
BLAKE2b-256 b3a179846e5ef911cd5d75c844de3fa496a10c91b4b5f550aad695c5df153d72

See more details on using hashes here.

Provenance

File details

Details for the file tiktoken-0.8.0-cp312-cp312-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for tiktoken-0.8.0-cp312-cp312-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 fe9399bdc3f29d428f16a2f86c3c8ec20be3eac5f53693ce4980371c3245729b
MD5 8c7dec7d4caa26e0807148807e6f9a3a
BLAKE2b-256 04d2c793cf49c20f5855fd6ce05d080c0537d7418f22c58e71f392d5e8c8dbf7

See more details on using hashes here.

Provenance

File details

Details for the file tiktoken-0.8.0-cp312-cp312-macosx_10_13_x86_64.whl.

File metadata

File hashes

Hashes for tiktoken-0.8.0-cp312-cp312-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 881839cfeae051b3628d9823b2e56b5cc93a9e2efb435f4cf15f17dc45f21586
MD5 ea48da6beef9dbd760a1ec7736b9db07
BLAKE2b-256 c12234b2e136a6f4af186b6640cbfd6f93400783c9ef6cd550d9eab80628d9de

See more details on using hashes here.

Provenance

File details

Details for the file tiktoken-0.8.0-cp311-cp311-win_amd64.whl.

File metadata

  • Download URL: tiktoken-0.8.0-cp311-cp311-win_amd64.whl
  • Upload date:
  • Size: 884.5 kB
  • Tags: CPython 3.11, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.11.9

File hashes

Hashes for tiktoken-0.8.0-cp311-cp311-win_amd64.whl
Algorithm Hash digest
SHA256 326624128590def898775b722ccc327e90b073714227175ea8febbc920ac0a99
MD5 54b32ad2d47f1ed2ca180f6d8dcece2e
BLAKE2b-256 1e86eea2309dc258fb86c7d9b10db536434fc16420feaa3b6113df18b23db7c2

See more details on using hashes here.

Provenance

File details

Details for the file tiktoken-0.8.0-cp311-cp311-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for tiktoken-0.8.0-cp311-cp311-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 56edfefe896c8f10aba372ab5706b9e3558e78db39dd497c940b47bf228bc419
MD5 c120ba324494e3d6b559126edc8cea7c
BLAKE2b-256 f8a3ef984e976822cd6c2227c854f74d2e60cf4cd6fbfca46251199914746f78

See more details on using hashes here.

Provenance

File details

Details for the file tiktoken-0.8.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for tiktoken-0.8.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 9fb0e352d1dbe15aba082883058b3cce9e48d33101bdaac1eccf66424feb5b47
MD5 c52ccd945dcf1223058bf94f49d534ae
BLAKE2b-256 01c4c4a4360de845217b6aa9709c15773484b50479f36bb50419c443204e5de9

See more details on using hashes here.

Provenance

File details

Details for the file tiktoken-0.8.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for tiktoken-0.8.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 5637e425ce1fc49cf716d88df3092048359a4b3bbb7da762840426e937ada06d
MD5 e05ee5f4a0c8ef02adf5a263cc461b36
BLAKE2b-256 ac3c2b95391d9bd520a73830469f80a96e3790e6c0a5ac2444f80f20b4b31051

See more details on using hashes here.

Provenance

File details

Details for the file tiktoken-0.8.0-cp311-cp311-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for tiktoken-0.8.0-cp311-cp311-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 2efaf6199717b4485031b4d6edb94075e4d79177a172f38dd934d911b588d54a
MD5 7cd767a4e56d41beb5809e16cde92ff6
BLAKE2b-256 8cf8f0101d98d661b34534769c3818f5af631e59c36ac6d07268fbfc89e539ce

See more details on using hashes here.

Provenance

File details

Details for the file tiktoken-0.8.0-cp311-cp311-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for tiktoken-0.8.0-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 d622d8011e6d6f239297efa42a2657043aaed06c4f68833550cac9e9bc723ef1
MD5 a188930ffb921e5417a5c504b4e8c3bd
BLAKE2b-256 f61eca48e7bfeeccaf76f3a501bd84db1fa28b3c22c9d1a1f41af9fb7579c5f6

See more details on using hashes here.

Provenance

File details

Details for the file tiktoken-0.8.0-cp310-cp310-win_amd64.whl.

File metadata

  • Download URL: tiktoken-0.8.0-cp310-cp310-win_amd64.whl
  • Upload date:
  • Size: 884.2 kB
  • Tags: CPython 3.10, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.11.9

File hashes

Hashes for tiktoken-0.8.0-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 d8c2d0e5ba6453a290b86cd65fc51fedf247e1ba170191715b049dac1f628005
MD5 839c5099634f564dc521137e5565b538
BLAKE2b-256 dcda8d1cc3089a83f5cf11c2e489332752981435280285231924557350523a59

See more details on using hashes here.

Provenance

File details

Details for the file tiktoken-0.8.0-cp310-cp310-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for tiktoken-0.8.0-cp310-cp310-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 6b2ddbc79a22621ce8b1166afa9f9a888a664a579350dc7c09346a3b5de837d9
MD5 7d512a41b30acf0ee3272b636c9e0204
BLAKE2b-256 57818a5be305cbd39d4e83a794f9e80c7f2c84b524587b7feb27c797b2046d51

See more details on using hashes here.

Provenance

File details

Details for the file tiktoken-0.8.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for tiktoken-0.8.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 f13d13c981511331eac0d01a59b5df7c0d4060a8be1e378672822213da51e0a2
MD5 6efd63d522ccede3dd93a2bdfe1caa30
BLAKE2b-256 2e28cf3633018cbcc6deb7805b700ccd6085c9a5a7f72b38974ee0bffd56d311

See more details on using hashes here.

Provenance

File details

Details for the file tiktoken-0.8.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for tiktoken-0.8.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 25e13f37bc4ef2d012731e93e0fef21dc3b7aea5bb9009618de9a4026844e560
MD5 0233bd5c801f1b63d4b0a4bb8692be19
BLAKE2b-256 e99818ec4a8351a6cf4537e40cd6e19a422c10cce1ef00a2fcb716e0a96af58b

See more details on using hashes here.

Provenance

File details

Details for the file tiktoken-0.8.0-cp310-cp310-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for tiktoken-0.8.0-cp310-cp310-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 9269348cb650726f44dd3bbb3f9110ac19a8dcc8f54949ad3ef652ca22a38e21
MD5 c14bfccdf31500e62b31c8e4955cd430
BLAKE2b-256 910513dab8fd7460391c387b3e69e14bf1e51ff71fe0a202cd2933cc3ea93fb6

See more details on using hashes here.

Provenance

File details

Details for the file tiktoken-0.8.0-cp310-cp310-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for tiktoken-0.8.0-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 b07e33283463089c81ef1467180e3e00ab00d46c2c4bbcef0acab5f771d6695e
MD5 e9aec9936e07a62306d763e0f8261175
BLAKE2b-256 c9baa35fad753bbca8ba0cc1b0f3402a70256a110ced7ac332cf84ba89fc87ab

See more details on using hashes here.

Provenance

File details

Details for the file tiktoken-0.8.0-cp39-cp39-win_amd64.whl.

File metadata

  • Download URL: tiktoken-0.8.0-cp39-cp39-win_amd64.whl
  • Upload date:
  • Size: 884.8 kB
  • Tags: CPython 3.9, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.11.9

File hashes

Hashes for tiktoken-0.8.0-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 1473cfe584252dc3fa62adceb5b1c763c1874e04511b197da4e6de51d6ce5a02
MD5 214c5ae25ef5c4671f8a45773a9bf45d
BLAKE2b-256 d53b7c8812952ca55e1bab08afc1dda3c5991804c71b550b9402e82a082ab795

See more details on using hashes here.

Provenance

File details

Details for the file tiktoken-0.8.0-cp39-cp39-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for tiktoken-0.8.0-cp39-cp39-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 845287b9798e476b4d762c3ebda5102be87ca26e5d2c9854002825d60cdb815d
MD5 97df403acb979b1725396683d68f5753
BLAKE2b-256 3e6b3ae00f0bff5d0b6925bf6370cf0ff606f56daed76210c2b0a156017b78dc

See more details on using hashes here.

Provenance

File details

Details for the file tiktoken-0.8.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for tiktoken-0.8.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 b591fb2b30d6a72121a80be24ec7a0e9eb51c5500ddc7e4c2496516dd5e3816b
MD5 5e05e735cfb04bb2b90f6bcf02108329
BLAKE2b-256 c2e16c7a772e0200131e960e3381f1d7b26406bc5612c70677989c1498af2a60

See more details on using hashes here.

Provenance

File details

Details for the file tiktoken-0.8.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for tiktoken-0.8.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 6adc8323016d7758d6de7313527f755b0fc6c72985b7d9291be5d96d73ecd1e1
MD5 ae48187a7f78a6cb4f58b8f4bc2195dc
BLAKE2b-256 f7011483856d84827c5fe541cb160f07914c6b063b8d961146e9c3557c4730c0

See more details on using hashes here.

Provenance

File details

Details for the file tiktoken-0.8.0-cp39-cp39-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for tiktoken-0.8.0-cp39-cp39-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 886f80bd339578bbdba6ed6d0567a0d5c6cfe198d9e587ba6c447654c65b8edc
MD5 7a656dbf90440cefe47598ccecd9a5e7
BLAKE2b-256 427a914bd98100449422778f9222d00b3a4ee654211c40784e57541fa46311ab

See more details on using hashes here.

Provenance

File details

Details for the file tiktoken-0.8.0-cp39-cp39-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for tiktoken-0.8.0-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 7e17807445f0cf1f25771c9d86496bd8b5c376f7419912519699f3cc4dc5c12e
MD5 fe4a59beae9047d9ae25eef1d586fd3c
BLAKE2b-256 08f38a8ba9329e6b426d822c974d58fc6477f3f7b3b8deef651813d275cbe75f

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page