PyTorch extension for handling deeply nested sequences of variable length

Project description

Tests

FoldedTensor: PyTorch extension for handling deeply nested sequences of variable length

foldedtensor is a PyTorch extension that provides efficient handling of tensors containing deeply nested sequences variable sizes. It enables the flattening/unflattening (or unfolding/folding) of data dimensions based on a inner structure of sequence lengths. This library is particularly useful when working with data that can be split in different ways and enables you to avoid choosing a fixed representation.

Installation

The library can be installed with pip:

pip install foldedtensor

Features

Support for arbitrary numbers of nested dimensions
No computational overhead when dealing with already padded tensors
Dynamic re-padding (or refolding) of data based on stored inner lengths
Automatic mask generation and updating whenever the tensor is refolded
C++ optimized code for fast data loading from Python lists and refolding
Flexibility in data representation, making it easy to switch between different layouts when needed

Example

import torch
from foldedtensor import as_folded_tensor

# Creating a folded tensor from a nested list
# There are 2 samples, the first with 5 lines, the second with 1 line.
# Each line contain between 1 and 2 words.
ft = as_folded_tensor(
    [
        [[1], [], [], [], [2, 3]],
        [[4, 3]],
    ],
    data_dims=("samples", "words"),
    full_names=("samples", "lines", "words"),
    dtype=torch.long,
)
print(ft)
# FoldedTensor([[1, 2, 3],
#               [4, 3, 0]])

# Refold on the lines and words dims (flatten the samples dim)
print(ft.refold(("lines", "words")))
# FoldedTensor([[1, 0],
#               [0, 0],
#               [0, 0],
#               [0, 0],
#               [2, 3],
#               [4, 3]])

# Refold on the words dim only: flatten everything
print(ft.refold(("words",)))
# FoldedTensor([1, 2, 3, 4, 3])

# Working with PyTorch operations
embedder = torch.nn.Embedding(10, 16)
embedding = embedder(ft.refold(("words",)))
print(embedding.shape)
# torch.Size([5, 16]) # 5 words total, 16 dims

refolded_embedding = embedding.refold(("samples", "words"))
print(refolded_embedding.shape)
# torch.Size([2, 5, 16]) # 2 samples, 5 words max, 16 dims

Comparison with alternatives

Unlike other ragged or nested tensor implementations, a FoldedTensor does not enforce a specific structure on the nested data, and does not require padding all dimensions. This provides the user with greater flexibility when working with data that can be arranged in multiple ways depending on the data transformation. Moreover, the C++ optimization ensures high performance, making it ideal for handling deeply nested tensors efficiently.

Here is a comparison with other common implementations for handling nested sequences of variable length:

Feature	NestedTensor	MaskedTensor	FoldedTensor
Inner data structure	Flat	Padded	Arbitrary
Max nesting level	1	1	∞
From nested python lists	No	No	Yes
Layout conversion	To padded	No	Any
Reduction ops w/o padding	Yes	No	No

Project details

Release history Release notifications | RSS feed

0.3.5

Sep 16, 2024

0.3.4

May 12, 2024

0.3.3

Feb 14, 2024

0.3.2

Oct 12, 2023

This version

0.3.1

Aug 30, 2023

0.3.0

Jul 7, 2023

0.2.2

Jun 5, 2023

0.2.1.post0

May 23, 2023

0.2.1

May 23, 2023

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

foldedtensor-0.3.1.tar.gz (14.3 kB view details)

Uploaded Aug 30, 2023 Source

Built Distributions

foldedtensor-0.3.1-cp311-cp311-win_amd64.whl (76.5 kB view details)

Uploaded Aug 30, 2023 CPython 3.11 Windows x86-64

foldedtensor-0.3.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (113.7 kB view details)

Uploaded Aug 30, 2023 CPython 3.11 manylinux: glibc 2.17+ x86-64

foldedtensor-0.3.1-cp311-cp311-macosx_11_0_arm64.whl (77.5 kB view details)

Uploaded Aug 30, 2023 CPython 3.11 macOS 11.0+ ARM64

foldedtensor-0.3.1-cp311-cp311-macosx_10_9_x86_64.whl (81.0 kB view details)

Uploaded Aug 30, 2023 CPython 3.11 macOS 10.9+ x86-64

foldedtensor-0.3.1-cp310-cp310-win_amd64.whl (75.5 kB view details)

Uploaded Aug 30, 2023 CPython 3.10 Windows x86-64

foldedtensor-0.3.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (112.1 kB view details)

Uploaded Aug 30, 2023 CPython 3.10 manylinux: glibc 2.17+ x86-64

foldedtensor-0.3.1-cp310-cp310-macosx_11_0_arm64.whl (76.2 kB view details)

Uploaded Aug 30, 2023 CPython 3.10 macOS 11.0+ ARM64

foldedtensor-0.3.1-cp310-cp310-macosx_10_9_x86_64.whl (79.6 kB view details)

Uploaded Aug 30, 2023 CPython 3.10 macOS 10.9+ x86-64

foldedtensor-0.3.1-cp39-cp39-win_amd64.whl (75.3 kB view details)

Uploaded Aug 30, 2023 CPython 3.9 Windows x86-64

foldedtensor-0.3.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (112.2 kB view details)

Uploaded Aug 30, 2023 CPython 3.9 manylinux: glibc 2.17+ x86-64

foldedtensor-0.3.1-cp39-cp39-macosx_11_0_arm64.whl (76.4 kB view details)

Uploaded Aug 30, 2023 CPython 3.9 macOS 11.0+ ARM64

foldedtensor-0.3.1-cp39-cp39-macosx_10_9_x86_64.whl (79.7 kB view details)

Uploaded Aug 30, 2023 CPython 3.9 macOS 10.9+ x86-64

foldedtensor-0.3.1-cp38-cp38-win_amd64.whl (75.4 kB view details)

Uploaded Aug 30, 2023 CPython 3.8 Windows x86-64

foldedtensor-0.3.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (112.0 kB view details)

Uploaded Aug 30, 2023 CPython 3.8 manylinux: glibc 2.17+ x86-64

foldedtensor-0.3.1-cp38-cp38-macosx_11_0_arm64.whl (76.2 kB view details)

Uploaded Aug 30, 2023 CPython 3.8 macOS 11.0+ ARM64

foldedtensor-0.3.1-cp38-cp38-macosx_10_9_x86_64.whl (79.5 kB view details)

Uploaded Aug 30, 2023 CPython 3.8 macOS 10.9+ x86-64

foldedtensor-0.3.1-cp37-cp37m-win_amd64.whl (76.3 kB view details)

Uploaded Aug 30, 2023 CPython 3.7m Windows x86-64

foldedtensor-0.3.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (113.4 kB view details)

Uploaded Aug 30, 2023 CPython 3.7m manylinux: glibc 2.17+ x86-64

foldedtensor-0.3.1-cp37-cp37m-macosx_10_9_x86_64.whl (79.5 kB view details)

Uploaded Aug 30, 2023 CPython 3.7m macOS 10.9+ x86-64

File details

Details for the file foldedtensor-0.3.1.tar.gz.

File metadata

Download URL: foldedtensor-0.3.1.tar.gz
Upload date: Aug 30, 2023
Size: 14.3 kB
Tags: Source
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/4.0.2 CPython/3.11.5

File hashes

Hashes for foldedtensor-0.3.1.tar.gz
Algorithm	Hash digest
SHA256	`cebda944ab60a3cba888248dbb445e9e05701be7d693212dfe54895799698902`
MD5	`a75d59715763491704dedb281dfbea4e`
BLAKE2b-256	`be76e4bfae357c68e123b331b4afe32ca75fa772f95605393a7349e57471663d`

Algorithm	Hash digest
SHA256	`f393a8fba36143e65d5b40f8bcc6e0ba8ec5173964cb575ddd48fb4e95a06c19`
MD5	`3ca787e328957d6c40a3670c4c745ec5`
BLAKE2b-256	`80fe9e04b3c970130d075b58793e054dc07078296f67c8848b9953b72bede64b`

Algorithm	Hash digest
SHA256	`1d0d536a5414df255618b8852bc693ff5c393f1184f486e4c0c0dec323df46c8`
MD5	`7b01d2e59d4bae3eb18b56fcffb1e5a8`
BLAKE2b-256	`39380993e8e011d950a5bf2da6cf62a789c913447e4a83071f217a6a9cdd5a5f`

Algorithm	Hash digest
SHA256	`1a98477e2ec88ad885c7b69b45e4ece53a12e0e0dda78574d1b8adc063f5e9e6`
MD5	`2193fc4a009b95f7d60f1252f99fa2e4`
BLAKE2b-256	`8f56bb5973351964b707af71ae3fff3fb8a268b7999c826791b76824bb45aac2`

Algorithm	Hash digest
SHA256	`6bb7eb557bf56c2ae98d1ad8a0435b5748d7343bbf90be7bcc8dc668659b34dc`
MD5	`0dde610f44992c8d4a671295cf8489cf`
BLAKE2b-256	`b1b05bc534708b6b92f231a3ac0527c600015aefc6b5ddb1ce8d5212650ad35c`

Algorithm	Hash digest
SHA256	`40ded8fc0d85c16434869ccfe214189ddcbdb61beb96ea56abe4b8b747b3a66d`
MD5	`298e2a330c0f32e9526a2d2b40533371`
BLAKE2b-256	`46333ca0f1015cdfb1c548ac31954def986b6c6d2082a077d21685c668e627fc`

Algorithm	Hash digest
SHA256	`f129747fd1860a642206c26657bde2c9c5386794ee05eaf37711e3b77f3eec56`
MD5	`4b226687a94be9b777640dec1c31bc1a`
BLAKE2b-256	`0acc1f5cb58765b3ede0f97d83fbf50aa5b5e5501e5c22f9617e1c197a6d8762`

Algorithm	Hash digest
SHA256	`9f2faf04e21e481bcb606bd2a8cfc0b74a0466905e24b5d4fc187862b39f2ee6`
MD5	`2df113d82e132bf36ddd5e9021e01086`
BLAKE2b-256	`d1b8f00e703d14114f8718153490b9b2a7d058e8f637c2ea01287a2ca00086f0`

Algorithm	Hash digest
SHA256	`fcc7985049811000e596c32239f2caf06fbce86c7fe5b7cb84409aa9ccfa83e7`
MD5	`e9b4999a5d4d156a321e1ddca9db925c`
BLAKE2b-256	`43da30a1469df17fa58438f5df83bfa0d1204c6c4668c215785514c3e61b219f`

Algorithm	Hash digest
SHA256	`b41751bb3fa328445eab23ed0b40d353d6f2394c629cc984dac20b85f8ffae7e`
MD5	`cedabd56c280846d0a9478dd083661e5`
BLAKE2b-256	`65cdca5665889258dae03a11e92d1d255d18bfa3da46cc7094e392f472ed03f0`

Algorithm	Hash digest
SHA256	`36662e7c2bcd55b74656944974ae6ed29fd02406ac3d348cd10d1565a6c22a1b`
MD5	`ee579ad6fc56e6f75ac806e6d78c2f01`
BLAKE2b-256	`7ffcb1d94167633f33147fb8af32e85d4b723db21dc903a06eccf082b11446b8`

Algorithm	Hash digest
SHA256	`8d3a7966e77f8ded51ab9a96826e62fb0bf870b29d55ec546e82ee835feee5c5`
MD5	`8d7832b78ae8bb1d715165253b8fce5b`
BLAKE2b-256	`e89b084910ac1d9031031f4ca05c58879b5ad0e4768566ca1294e7adc64f756c`

Algorithm	Hash digest
SHA256	`b304c14c18fc51bb61817b16749b4c024734049dc05bf35fa4af14f16c4f8658`
MD5	`f2e790236b836608a5fb188ce50e0ef8`
BLAKE2b-256	`9ff03f7364fed5ff383d23f585a11f08b58534beb8695852a0875d6f56bdec5d`

Algorithm	Hash digest
SHA256	`476e78574771fe058eaf8e5a1023457ab8e9f9428e061a9314589130a69dc751`
MD5	`9fe242c3ddd2ff00959507226e64acdd`
BLAKE2b-256	`9807880a64eff6570ba8afd6f86618fd9c7b34ce3ba61503931dbd454c90aac2`

Algorithm	Hash digest
SHA256	`ae18803bb3f4a7162d846daf9376eddb93bf91f2265b8bdb9639b3becd0b49fc`
MD5	`525671ecdaa22e95263e5c68106dffcc`
BLAKE2b-256	`597bff2f5f46fbf1f74db5d603eee482a3a496b2b75e4d964e7930ee28a2331a`

Algorithm	Hash digest
SHA256	`4218cc97024bc073e38486532546e2dca43c450e99c3192a65077fbfe02ac587`
MD5	`f73ebe394b8b414479562b804cbe59f1`
BLAKE2b-256	`5a635eca9621122ca02c1e9d8c8b488bbb177adffbd4acc43128e0ac73bc9f50`

Algorithm	Hash digest
SHA256	`a601935d281414ac2179446976ea2bebe9949f4b0f6fb15c9bd31e8285f4776d`
MD5	`c5af3d4d4c7fb6b36b76bd4366274c7e`
BLAKE2b-256	`3d6aa660f57b13520810deeaba867d47b34aef552b221db09cc223cc528430c8`

Algorithm	Hash digest
SHA256	`7fded12f5d2564c59683f8840caabab5fc64f024fee7a5ee86149b5c119475aa`
MD5	`584e58d2d163c55f837b8c88b6e186c2`
BLAKE2b-256	`f30825e4ee5538e95049a62053197a6311475d4be96d60c63220127f88b67314`

Algorithm	Hash digest
SHA256	`fb9dde62469eca3d9bc5f08acf16a632dd48bd930a47a5c66d3bfaaa1b3b9d85`
MD5	`3f6d9bcc00a8f868df27a6ca8ea71996`
BLAKE2b-256	`604eea9f91fdc9f8cad24c70c7fd1f540bd720b313cf4ecccf24f02ee8be705e`

Algorithm	Hash digest
SHA256	`b5a6e996d5cc68edab5504bf4868da653ebe7e51dbcb05ea621d8cbeba69dc37`
MD5	`25f1147c3b21287faa5e596976bbfedc`
BLAKE2b-256	`7e779bd80201477992ae8b4c5e065104e8f91992f533e86aa474a6a5a5022311`

foldedtensor 0.3.1

Navigation

Verified details

Maintainers

Unverified details

Project links

Meta

Project description

FoldedTensor: PyTorch extension for handling deeply nested sequences of variable length

Installation

Features

Example

Comparison with alternatives

Project details

Verified details

Maintainers

Unverified details

Project links

Meta

Release history Release notifications | RSS feed

Download files

Source Distribution

Built Distributions

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details