Skip to main content

A video decoder for PyTorch

Project description

Installation | Simple Example | Detailed Example | Documentation | Contributing | License

TorchCodec

TorchCodec is a Python library for decoding videos into PyTorch tensors. It aims to be fast, easy to use, and well integrated into the PyTorch ecosystem. If you want to use PyTorch to train ML models on videos, TorchCodec is how you turn those videos into data.

We achieve these capabilities through:

  • Pythonic APIs that mirror Python and PyTorch conventions.
  • Relying on FFmpeg to do the decoding. TorchCodec uses the version of FFmpeg you already have installed. FFmpeg is a mature library with broad coverage available on most systems. It is, however, not easy to use. TorchCodec abstracts FFmpeg's complexity to ensure it is used correctly and efficiently.
  • Returning data as PyTorch tensors, ready to be fed into PyTorch transforms or used directly to train models.

[!NOTE] ������ TorchCodec is still in early development stage and some APIs may be updated in future versions without a deprecation cycle, depending on user feedback. If you have any suggestions or issues, please let us know by opening an issue!

Using TorchCodec

Here's a condensed summary of what you can do with TorchCodec. For a more detailed example, check out our documentation!

from torchcodec.decoders import SimpleVideoDecoder

decoder = SimpleVideoDecoder("path/to/video.mp4")

decoder.metadata
# VideoStreamMetadata:
#   num_frames: 250
#   duration_seconds: 10.0
#   bit_rate: 31315.0
#   codec: h264
#   average_fps: 25.0
#   ... (truncated output)

len(decoder)  # == decoder.metadata.num_frames!
# 250
decoder.metadata.average_fps  # Note: instantaneous fps can be higher or lower
# 25.0

# Simple Indexing API
decoder[0]  # uint8 tensor of shape [C, H, W]
decoder[0 : -1 : 20]  # uint8 stacked tensor of shape [N, C, H, W]


# Iterate over frames:
for frame in decoder:
    pass

# Indexing, with PTS and duration info
decoder.get_frame_at(len(decoder) - 1)
# Frame:
#   data (shape): torch.Size([3, 400, 640])
#   pts_seconds: 9.960000038146973
#   duration_seconds: 0.03999999910593033

decoder.get_frames_at(start=10, stop=30, step=5)
# FrameBatch:
#   data (shape): torch.Size([4, 3, 400, 640])
#   pts_seconds: tensor([0.4000, 0.6000, 0.8000, 1.0000])
#   duration_seconds: tensor([0.0400, 0.0400, 0.0400, 0.0400])

# Time-based indexing with PTS and duration info
decoder.get_frame_displayed_at(pts_seconds=2)
# Frame:
#   data (shape): torch.Size([3, 400, 640])
#   pts_seconds: 2.0
#   duration_seconds: 0.03999999910593033

You can use the following snippet to generate a video with FFmpeg and tryout TorchCodec:

fontfile=/usr/share/fonts/dejavu-sans-mono-fonts/DejaVuSansMono-Bold.ttf
output_video_file=/tmp/output_video.mp4

ffmpeg -f lavfi -i \
    color=size=640x400:duration=10:rate=25:color=blue \
    -vf "drawtext=fontfile=${fontfile}:fontsize=30:fontcolor=white:x=(w-text_w)/2:y=(h-text_h)/2:text='Frame %{frame_num}'" \
    ${output_video_file}

Installing TorchCodec

Note: if you're on MacOS, you'll need to build from source. Instructions below assume you're on Linux.

First install the latest stable version of PyTorch following the official instructions.

Then:

pip install torchcodec

You will also need FFmpeg installed on your system, and TorchCodec decoding capabilities are determined by your underlying FFmpeg installation. There are different options to install FFmpeg e.g.:

conda install ffmpeg
# or
conda install ffmpeg -c conda-forge

Your Linux distribution probably comes with FFmpeg pre-installed as well. TorchCodec supports all major FFmpeg version in [4, 7].

Planned future work

We are actively working on the following features:

Let us know if you have any feature requests by opening an issue!

Contributing

We welcome contributions to TorchCodec! Please see our contributing guide for more details.

License

TorchCodec is released under the BSD 3 license.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

TorchCodec-0.0.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (657.5 kB view details)

Uploaded CPython 3.12 manylinux: glibc 2.17+ x86-64

TorchCodec-0.0.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (659.1 kB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

TorchCodec-0.0.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (655.1 kB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

TorchCodec-0.0.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (655.5 kB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

File details

Details for the file TorchCodec-0.0.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for TorchCodec-0.0.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 3775bb4c27f1de3106722f8a59b9a5236aa664c89f547659c3ecaf01e058b614
MD5 20fc5cbab46adee234a9bbd4c0318509
BLAKE2b-256 68a3bcd49271800d33a1f94496dea78649887d19658ea4d14984be4c2e9a1096

See more details on using hashes here.

Provenance

File details

Details for the file TorchCodec-0.0.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for TorchCodec-0.0.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 5ad63f29f711e83ac084d6fd36a906cad5eb30e5df0d6b0894590127a567bf69
MD5 5d727da766e1c97aa31bac328ca49dfb
BLAKE2b-256 2ca73175936b9604fe64a94f8321958838a4d0c7c347c8287d43782453f50f99

See more details on using hashes here.

Provenance

File details

Details for the file TorchCodec-0.0.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for TorchCodec-0.0.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 2d597d09f5a36ab30ed6f3d7d16e62353cf822f831db19757c85281967d0e898
MD5 4d20d63a107c5db002172f147b231f03
BLAKE2b-256 5426f9cccdeaaa37bcf03eac7a2d03a74eef57a8f01ed30686209f87c7f79c39

See more details on using hashes here.

Provenance

File details

Details for the file TorchCodec-0.0.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for TorchCodec-0.0.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 e7add1c6f304f8b2ad159603bada40c8605915ca9f149a21b5e056c52a91587c
MD5 eb2946d82b0e939040da23b6152b7e2a
BLAKE2b-256 fd0b26c244a02916213e8c627af2e4f0b6f2f96717e391884ec51c3d497bced6

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page