ONNX Runtime generate() API

Project description

ONNX Runtime generate() API

Run SLMs/LLMs and multi modal models on-device and in the cloud with ONNX Runtime.

Model architectures supported so far (and more coming soon): Gemma, Llama, Mistral, Phi (language and vision).

For more details, see: docs https://onnxruntime.ai/docs/genai and repo: https://github.com/microsoft/onnxruntime-genai

Project details

Release history Release notifications | RSS feed

This version

0.5.2

Nov 25, 2024

0.5.1

Nov 13, 2024

0.5.0

Nov 7, 2024

0.4.0

Aug 21, 2024

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

onnxruntime_genai_cuda-0.5.2-cp312-cp312-win_amd64.whl (14.4 MB view details)

Uploaded Nov 25, 2024 CPython 3.12 Windows x86-64

onnxruntime_genai_cuda-0.5.2-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (15.1 MB view details)

Uploaded Nov 25, 2024 CPython 3.12 manylinux: glibc 2.27+ x86-64 manylinux: glibc 2.28+ x86-64

onnxruntime_genai_cuda-0.5.2-cp311-cp311-win_amd64.whl (14.4 MB view details)

Uploaded Nov 25, 2024 CPython 3.11 Windows x86-64

onnxruntime_genai_cuda-0.5.2-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (15.1 MB view details)

Uploaded Nov 25, 2024 CPython 3.11 manylinux: glibc 2.27+ x86-64 manylinux: glibc 2.28+ x86-64

onnxruntime_genai_cuda-0.5.2-cp310-cp310-win_amd64.whl (14.4 MB view details)

Uploaded Nov 25, 2024 CPython 3.10 Windows x86-64

onnxruntime_genai_cuda-0.5.2-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (15.1 MB view details)

Uploaded Nov 25, 2024 CPython 3.10 manylinux: glibc 2.27+ x86-64 manylinux: glibc 2.28+ x86-64

File details

Details for the file onnxruntime_genai_cuda-0.5.2-cp312-cp312-win_amd64.whl.

File metadata

Download URL: onnxruntime_genai_cuda-0.5.2-cp312-cp312-win_amd64.whl
Upload date: Nov 25, 2024
Size: 14.4 MB
Tags: CPython 3.12, Windows x86-64
Uploaded using Trusted Publishing? No
Uploaded via: RestSharp/106.13.0.0

File hashes

Hashes for onnxruntime_genai_cuda-0.5.2-cp312-cp312-win_amd64.whl
Algorithm	Hash digest
SHA256	`8d113cd7ceb282391b0beec09e34e6653659298f199831f862340050b40d5d04`
MD5	`868030f81c5935ca2a5177d5e97f721a`
BLAKE2b-256	`dd64f177338aac2c39c833063061c9f8d2fcf94a976a9ce75ca8eed519c1b017`

See more details on using hashes here.

File details

Details for the file onnxruntime_genai_cuda-0.5.2-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.

File metadata

Download URL: onnxruntime_genai_cuda-0.5.2-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
Upload date: Nov 25, 2024
Size: 15.1 MB
Tags: CPython 3.12, manylinux: glibc 2.27+ x86-64, manylinux: glibc 2.28+ x86-64
Uploaded using Trusted Publishing? No
Uploaded via: RestSharp/106.13.0.0

File hashes

Hashes for onnxruntime_genai_cuda-0.5.2-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
Algorithm	Hash digest
SHA256	`f12570e0221772281c4edb0e5d8890b63b8da5bd09b54222fec0d401e7623c2c`
MD5	`c716726d0ef1f90c6f2bbee18c5d8b44`
BLAKE2b-256	`f75dccd7e1ea364e67b9be65d94a4651664b70e9739de6cd4393d4cb21709524`

See more details on using hashes here.

File details

Details for the file onnxruntime_genai_cuda-0.5.2-cp311-cp311-win_amd64.whl.

File metadata

Download URL: onnxruntime_genai_cuda-0.5.2-cp311-cp311-win_amd64.whl
Upload date: Nov 25, 2024
Size: 14.4 MB
Tags: CPython 3.11, Windows x86-64
Uploaded using Trusted Publishing? No
Uploaded via: RestSharp/106.13.0.0

File hashes

Hashes for onnxruntime_genai_cuda-0.5.2-cp311-cp311-win_amd64.whl
Algorithm	Hash digest
SHA256	`7dfd3fc9692789030155e95e6c3d167fba8e0f29240317b8ec366b6b9fc7e008`
MD5	`98f0ba794ffb69f2545261d472505869`
BLAKE2b-256	`3d0ba4b0cb35892e7e5d4906451f1be68441fcacdaf32dec7ff22f2ea2ea50f2`

See more details on using hashes here.

File details

Details for the file onnxruntime_genai_cuda-0.5.2-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.

File metadata

Download URL: onnxruntime_genai_cuda-0.5.2-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
Upload date: Nov 25, 2024
Size: 15.1 MB
Tags: CPython 3.11, manylinux: glibc 2.27+ x86-64, manylinux: glibc 2.28+ x86-64
Uploaded using Trusted Publishing? No
Uploaded via: RestSharp/106.13.0.0

File hashes

Hashes for onnxruntime_genai_cuda-0.5.2-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
Algorithm	Hash digest
SHA256	`255909f0faa82b98ec9e52d00c8f568f9714c047513802857d1e550463a25613`
MD5	`814246b1c0efd00c8ec2e8e032cc5446`
BLAKE2b-256	`c20a7f2dcbd8cdd4548991c52ed2e5eb7ef75ca1d82381646163f9f79cdf3959`

See more details on using hashes here.

File details

Details for the file onnxruntime_genai_cuda-0.5.2-cp310-cp310-win_amd64.whl.

File metadata

Download URL: onnxruntime_genai_cuda-0.5.2-cp310-cp310-win_amd64.whl
Upload date: Nov 25, 2024
Size: 14.4 MB
Tags: CPython 3.10, Windows x86-64
Uploaded using Trusted Publishing? No
Uploaded via: RestSharp/106.13.0.0

File hashes

Hashes for onnxruntime_genai_cuda-0.5.2-cp310-cp310-win_amd64.whl
Algorithm	Hash digest
SHA256	`54d1830840af178dd14f7ded138852f065ea40834a99f31ab98170d8187bfa51`
MD5	`d00383ad20a8689d6abce775717d8b7f`
BLAKE2b-256	`8f011fef3571f0e9b7bc5f15dce82af0290b1a0845db9110942b031f6f4c9f70`

See more details on using hashes here.

File details

Details for the file onnxruntime_genai_cuda-0.5.2-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.

File metadata

Download URL: onnxruntime_genai_cuda-0.5.2-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
Upload date: Nov 25, 2024
Size: 15.1 MB
Tags: CPython 3.10, manylinux: glibc 2.27+ x86-64, manylinux: glibc 2.28+ x86-64
Uploaded using Trusted Publishing? No
Uploaded via: RestSharp/106.13.0.0

File hashes

Hashes for onnxruntime_genai_cuda-0.5.2-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
Algorithm	Hash digest
SHA256	`af2daa4c3adab26fdebacdbae1eb00e166e15456aad35dd6976db37164364ff5`
MD5	`693360b3adae8b0ecec5e704be5cff8b`
BLAKE2b-256	`6650c8fcf08185b4414e0191923f1dea7159c760451c3f5e5317ed77feccd428`