PyTorch extension for handling deeply nested sequences of variable length

Project description

Tests

FoldedTensor: PyTorch extension for handling deeply nested sequences of variable length

foldedtensor is a PyTorch extension that provides efficient handling of tensors containing deeply nested sequences variable sizes. It enables the flattening/unflattening (or unfolding/folding) of data dimensions based on a inner structure of sequence lengths. This library is particularly useful when working with data that can be split in different ways and enables you to avoid choosing a fixed representation.

Installation

The library can be installed with pip:

pip install foldedtensor

Features

Support for arbitrary numbers of nested dimensions
No computational overhead when dealing with already padded tensors
Dynamic re-padding (or refolding) of data based on stored inner lengths
Automatic mask generation and updating whenever the tensor is refolded
C++ optimized code for fast data loading from Python lists and refolding
Flexibility in data representation, making it easy to switch between different layouts when needed

Examples

At its simplest, foldedtensor can be used to convert nested Python lists into a PyTorch tensor:

from foldedtensor import as_folded_tensor

ft = as_folded_tensor(
    [
        [0, 1, 2],
        [3],
    ],
)
# FoldedTensor([[0, 1, 2],
#               [3, 0, 0]])

You can also specify names and flattened/unflattened dimensions at the time of creation:

import torch
from foldedtensor import as_folded_tensor

# Creating a folded tensor from a nested list
# There are 2 samples, the first with 5 lines, the second with 1 line.
# Each line contain between 1 and 2 words.
ft = as_folded_tensor(
    [
        [[1], [], [], [], [2, 3]],
        [[4, 3]],
    ],
    data_dims=("samples", "words"),
    full_names=("samples", "lines", "words"),
    dtype=torch.long,
)
print(ft)
# FoldedTensor([[1, 2, 3],
#               [4, 3, 0]])

Once created, you can change the shape of the tensor by refolding it:

# Refold on the lines and words dims (flatten the samples dim)
print(ft.refold(("lines", "words")))
# FoldedTensor([[1, 0],
#               [0, 0],
#               [0, 0],
#               [0, 0],
#               [2, 3],
#               [4, 3]])

# Refold on the words dim only: flatten everything
print(ft.refold(("words",)))
# FoldedTensor([1, 2, 3, 4, 3])

The tensor can be further used with standard PyTorch operations:

# Working with PyTorch operations
embedder = torch.nn.Embedding(10, 16)
embedding = embedder(ft.refold(("words",)))
print(embedding.shape)
# torch.Size([5, 16]) # 5 words total, 16 dims

refolded_embedding = embedding.refold(("samples", "words"))
print(refolded_embedding.shape)
# torch.Size([2, 5, 16]) # 2 samples, 5 words max, 16 dims

Benchmarks

View the comparisons of foldedtensor against various alternatives here: docs/benchmarks.

Comparison with alternatives

Unlike other ragged or nested tensor implementations, a FoldedTensor does not enforce a specific structure on the nested data, and does not require padding all dimensions. This provides the user with greater flexibility when working with data that can be arranged in multiple ways depending on the data transformation. Moreover, the C++ optimization ensures high performance, making it ideal for handling deeply nested tensors efficiently.

Here is a comparison with other common implementations for handling nested sequences of variable length:

Feature	NestedTensor	MaskedTensor	FoldedTensor
Inner data structure	Flat	Padded	Arbitrary
Max nesting level	1	1	∞
From nested python lists	No	No	Yes
Layout conversion	To padded	No	Any
Reduction ops w/o padding	Yes	No	No

Project details

Release history Release notifications | RSS feed

0.3.5

Sep 16, 2024

This version

0.3.4

May 12, 2024

0.3.3

Feb 14, 2024

0.3.2

Oct 12, 2023

0.3.1

Aug 30, 2023

0.3.0

Jul 7, 2023

0.2.2

Jun 5, 2023

0.2.1.post0

May 23, 2023

0.2.1

May 23, 2023

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

foldedtensor-0.3.4.tar.gz (18.0 kB view details)

Uploaded May 12, 2024 Source

Built Distributions

foldedtensor-0.3.4-cp312-cp312-win_amd64.whl (82.0 kB view details)

Uploaded May 12, 2024 CPython 3.12 Windows x86-64

foldedtensor-0.3.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (119.9 kB view details)

Uploaded May 12, 2024 CPython 3.12 manylinux: glibc 2.17+ x86-64

foldedtensor-0.3.4-cp312-cp312-macosx_11_0_arm64.whl (82.5 kB view details)

Uploaded May 12, 2024 CPython 3.12 macOS 11.0+ ARM64

foldedtensor-0.3.4-cp312-cp312-macosx_10_9_x86_64.whl (85.9 kB view details)

Uploaded May 12, 2024 CPython 3.12 macOS 10.9+ x86-64

foldedtensor-0.3.4-cp311-cp311-win_amd64.whl (81.7 kB view details)

Uploaded May 12, 2024 CPython 3.11 Windows x86-64

foldedtensor-0.3.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (120.0 kB view details)

Uploaded May 12, 2024 CPython 3.11 manylinux: glibc 2.17+ x86-64

foldedtensor-0.3.4-cp311-cp311-macosx_11_0_arm64.whl (84.0 kB view details)

Uploaded May 12, 2024 CPython 3.11 macOS 11.0+ ARM64

foldedtensor-0.3.4-cp311-cp311-macosx_10_9_x86_64.whl (86.8 kB view details)

Uploaded May 12, 2024 CPython 3.11 macOS 10.9+ x86-64

foldedtensor-0.3.4-cp310-cp310-win_amd64.whl (80.5 kB view details)

Uploaded May 12, 2024 CPython 3.10 Windows x86-64

foldedtensor-0.3.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (118.4 kB view details)

Uploaded May 12, 2024 CPython 3.10 manylinux: glibc 2.17+ x86-64

foldedtensor-0.3.4-cp310-cp310-macosx_11_0_arm64.whl (82.7 kB view details)

Uploaded May 12, 2024 CPython 3.10 macOS 11.0+ ARM64

foldedtensor-0.3.4-cp310-cp310-macosx_10_9_x86_64.whl (85.4 kB view details)

Uploaded May 12, 2024 CPython 3.10 macOS 10.9+ x86-64

foldedtensor-0.3.4-cp39-cp39-win_amd64.whl (80.5 kB view details)

Uploaded May 12, 2024 CPython 3.9 Windows x86-64

foldedtensor-0.3.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (118.5 kB view details)

Uploaded May 12, 2024 CPython 3.9 manylinux: glibc 2.17+ x86-64

foldedtensor-0.3.4-cp39-cp39-macosx_11_0_arm64.whl (82.9 kB view details)

Uploaded May 12, 2024 CPython 3.9 macOS 11.0+ ARM64

foldedtensor-0.3.4-cp39-cp39-macosx_10_9_x86_64.whl (85.5 kB view details)

Uploaded May 12, 2024 CPython 3.9 macOS 10.9+ x86-64

foldedtensor-0.3.4-cp38-cp38-win_amd64.whl (80.6 kB view details)

Uploaded May 12, 2024 CPython 3.8 Windows x86-64

foldedtensor-0.3.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (118.3 kB view details)

Uploaded May 12, 2024 CPython 3.8 manylinux: glibc 2.17+ x86-64

foldedtensor-0.3.4-cp38-cp38-macosx_11_0_arm64.whl (82.7 kB view details)

Uploaded May 12, 2024 CPython 3.8 macOS 11.0+ ARM64

foldedtensor-0.3.4-cp38-cp38-macosx_10_9_x86_64.whl (85.3 kB view details)

Uploaded May 12, 2024 CPython 3.8 macOS 10.9+ x86-64

foldedtensor-0.3.4-cp37-cp37m-win_amd64.whl (81.3 kB view details)

Uploaded May 12, 2024 CPython 3.7m Windows x86-64

foldedtensor-0.3.4-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (120.3 kB view details)

Uploaded May 12, 2024 CPython 3.7m manylinux: glibc 2.17+ x86-64

foldedtensor-0.3.4-cp37-cp37m-macosx_10_9_x86_64.whl (85.1 kB view details)

Uploaded May 12, 2024 CPython 3.7m macOS 10.9+ x86-64

File details

Details for the file foldedtensor-0.3.4.tar.gz.

File metadata

Download URL: foldedtensor-0.3.4.tar.gz
Upload date: May 12, 2024
Size: 18.0 kB
Tags: Source
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/5.0.0 CPython/3.12.3

File hashes

Hashes for foldedtensor-0.3.4.tar.gz
Algorithm	Hash digest
SHA256	`0630804b8b252607cd2d1a5a79787c7bb29e9a46b088651ad927b89b1b32ebf4`
MD5	`dcf0abc4064c4cf95187381339dee6d0`
BLAKE2b-256	`2af02e1f6bee28e06ba9193c893a157e0bf01bfe6b8c117ecdaa40239461fd40`

Algorithm	Hash digest
SHA256	`abc6b8b81e753f54808481c754ccfac03b3d9f417991b885d58b86f441ff5476`
MD5	`af42a6b2537ea8857718211d3f99ffbe`
BLAKE2b-256	`d8a97e7c0152e52ef7c2febfb957c517c42771ffc83f9a1b9d6d926b61ec101f`

Algorithm	Hash digest
SHA256	`bf9b0f9796775c8e58e5775bb1231273c635374cd35957f24864856034335631`
MD5	`a98821aa3cd83db4b496733a1a6793af`
BLAKE2b-256	`cb8d053d6120c6d73d390b0556c94cb3898652534fe5ad3959b398f09926aaca`

Algorithm	Hash digest
SHA256	`c17cdf47bf1e4173c5fa146f933bf93d5b5cd11f5107ea47018da43d4aa65599`
MD5	`4d831e57f4574159671b66c10118bbc7`
BLAKE2b-256	`00b4a5e65581e13f004eacf5600f4610c58a2909e6798ca754efbf75865bf8ec`

Algorithm	Hash digest
SHA256	`8d58f517693aca8a8e743bf7cc93d49dea415b56c19b2fae7869bdd134a8009c`
MD5	`3a841335963095e0112758c623418bd5`
BLAKE2b-256	`a619a0669958bcff298fc9514e0025dc811664f35299c67e30a26802a73bdf3e`

Algorithm	Hash digest
SHA256	`ffb31353e49b5ac8cfe23d2b598863c9442d5299d786feadfcfa207b14a372aa`
MD5	`1a368fcab0e77e43acf52aa9735be6c4`
BLAKE2b-256	`b9e3af32f2017bc54c6eb9c2e0a72cfe38ecb215bec34861962ce69eea61bfec`

Algorithm	Hash digest
SHA256	`12d70cbea70926d255cee0ed0d3289fbf49c39323e3c34f49b3d25f9bc29b895`
MD5	`052c6060a3bf1e06cbb30e7bb3524a5a`
BLAKE2b-256	`99e0f508999b6375c8b65a01b759683123e0e31f70cbb996080a5555f7631f21`

Algorithm	Hash digest
SHA256	`33ff3f4ada2c8ddb1c729c8ee0aaeccc5f8c3ac2229eb30d31bcc84fc1fa2be2`
MD5	`494a638c67829be418e756cbb25aa8ce`
BLAKE2b-256	`c0c94d4c9029716826057ec0c571d81c9e9eb8d5f3d5665341fc69cda0ca7776`

Algorithm	Hash digest
SHA256	`aadbb508c8a9611d2a744c9643418064f04d6a6cc1257e0cc1abfbfe243d4f2c`
MD5	`57e3af0f2fbad36470a2cea1ee999630`
BLAKE2b-256	`2449eb299b332f2ab6be42ca4b68f8d6798f7f42e9f0e47c7d0e3e98e7ddca07`

Algorithm	Hash digest
SHA256	`c3c081303ab1353a06e82fdebde24737379217a0e1b48923c59b46b093b2c61b`
MD5	`f47e42b0c74665ea6fad29732a0a8de6`
BLAKE2b-256	`fab61aaac150f3521821575f1eeb7f00ff4a14aaf8fd127ea6fe98d2158d968e`

Algorithm	Hash digest
SHA256	`381cf303e993eca5e1cb9adb2fe216665867284b2f5ad59cabd5d6b0ae602e60`
MD5	`6285c0f6892ab18dfd129a857dbc23d5`
BLAKE2b-256	`2210d617cd1d427f8ee8cbfe144a14bf1f23b496c9a03d97c2f241b73bc99459`

Algorithm	Hash digest
SHA256	`81793e96a8cbde44af759e42ef8d04412ec4d2f8b37da798f22c3ab73e22d23b`
MD5	`c884a631928d60e681269d0046137dac`
BLAKE2b-256	`4cd9cbeaf345ac399cfe42645c9d6b969f8150bb7ec30bcdb00539656a0f3663`

Algorithm	Hash digest
SHA256	`443c4ae53e7cf652bbec0a7f4c3c36ee36f6fde5d1d19b2eb5e0bc089ce6ca83`
MD5	`fcf128f9d1fef39dbd5de2d44a4fa52c`
BLAKE2b-256	`84a731bcbd6a1fc8093ab8d1352ca3dcf09f2a2520b2fc3492f4452bb03a53f6`

Algorithm	Hash digest
SHA256	`304c8e1065e4183255457d2c61448645164e32f2f1580bd7baeb6ed2c79e1798`
MD5	`47f3f1a2becf62704860f62dc1cda7cf`
BLAKE2b-256	`555c1ba62014d18962dd9006440d69670e552fd19394bd242f8a04659c90abf5`

Algorithm	Hash digest
SHA256	`70a3903c4805169f435e17f1708db8df867512cfc431d4211f166e1c5ac76f19`
MD5	`cbdb946e32eebf7fa1ffb6c860705871`
BLAKE2b-256	`509a496ab41e509d38baa46838e211e93f0d0d05cc486fe14dd35f86e314a9f0`

Algorithm	Hash digest
SHA256	`c7e2c627003762d5f91db2341890f254dcb45778b0fca12a5967b0ea9b87075d`
MD5	`febe96a9ff152f56dee075bf7c3a2269`
BLAKE2b-256	`a70ffe4697193c023e6c4f69110e2335996aaa6406d707f6ba838019ead7b86c`

Algorithm	Hash digest
SHA256	`c28408557493388b19d05ebc3e51e0dbdc0ad578ac6e4d301c5595dad5f7ebc8`
MD5	`f810cb8cce4e9fc45e3ec136139f3a52`
BLAKE2b-256	`73044a1ecbfd5b91d6adbd08c089da5c5464f5e9a02e46d3c63132f9ca0c7870`

Algorithm	Hash digest
SHA256	`99e364199bd5b56de104b115fb21a1160ff6c6555ee6e372c0c22f4ef65c6368`
MD5	`4f3355b517a49660638413125963c30f`
BLAKE2b-256	`cab395042c5a5da7505029b04f5ba54fcaa5caeb1d8adde3b7b84e976dcb454e`

Algorithm	Hash digest
SHA256	`b37c66b811923f1f34f9568b75e051a9cfef61f36cfd6a1294f36e60ebb1b6c7`
MD5	`6a0e7d9359de6697bf6478864e64f665`
BLAKE2b-256	`95af31de7d759f4238da0ee2e03662e34be5d207986f91d8cfaea3568875f014`

Algorithm	Hash digest
SHA256	`29de45621146fc0c571b044f879792be812b8ae1398b0bc2b63a4974e3303589`
MD5	`104ac1c553e3edf307b846e7e852e1e6`
BLAKE2b-256	`ea18cbec47440d7c07a68efafc6f11e6b8c5db8fb0082dda839573905a8fdc6f`

Algorithm	Hash digest
SHA256	`66954c52852892ff5d1c25a29f019c7f9802e16b8498539e6df32f02d9906d60`
MD5	`728723890faff7a2a78545c44ac9fcc7`
BLAKE2b-256	`3a0756c927f5fa3903edc1ea738eaf5465f2cfc4edffac60109e3f9f548cec1e`

foldedtensor 0.3.4

Navigation

Verified details

Maintainers

Unverified details

Project links

Meta

Project description

FoldedTensor: PyTorch extension for handling deeply nested sequences of variable length

Installation

Features

Examples

Benchmarks

Comparison with alternatives

Project details

Verified details

Maintainers

Unverified details

Project links

Meta

Release history Release notifications | RSS feed

Download files

Source Distribution

Built Distributions

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

Algorithm	Hash digest
SHA256	`969994b75ec845e190e8616f77987532682e7cd19fe7b2b2fb7d17a88916b2a9`
MD5	`592292fc6fd8136d6f8e8b998269d964`
BLAKE2b-256	`ae75c7044e291e04efc3d4c22d2a2d7c59991cc7d06e907ee1f4b1b8861c8a1a`

Algorithm	Hash digest
SHA256	`821c964852b8401747acb69c5b3d5f124c01ae21b73b01349998663fd40b911d`
MD5	`5fac020e07056d2d491761b61cbaf3f2`
BLAKE2b-256	`20165cd664b484b52c66c424f62e628b8941af7a8aa978b322044aa53b4fe813`

Algorithm	Hash digest
SHA256	`9e9c7aa81fbfc1003d8a51d40e9f8c729e99117faa26380c6c6c8cfc4f3e1dde`
MD5	`32eb7bc8452218084dd6ea8c755a37c5`
BLAKE2b-256	`823dacac7d68d56c5a96005259602e984c1c1fd85f4b1a48860a012504d8401b`