Skip to main content

PyTorch extension for handling deeply nested sequences of variable length

Project description



Tests PyPI Coverage License

FoldedTensor: PyTorch extension for handling deeply nested sequences of variable length

foldedtensor is a PyTorch extension that provides efficient handling of tensors containing deeply nested sequences variable sizes. It enables the flattening/unflattening (or unfolding/folding) of data dimensions based on a inner structure of sequence lengths. This library is particularly useful when working with data that can be split in different ways and enables you to avoid choosing a fixed representation.

Installation

The library can be installed with pip:

pip install foldedtensor

Features

  • Support for arbitrary numbers of nested dimensions
  • No computational overhead when dealing with already padded tensors
  • Dynamic re-padding (or refolding) of data based on stored inner lengths
  • Automatic mask generation and updating whenever the tensor is refolded
  • C++ optimized code for fast data loading from Python lists and refolding
  • Flexibility in data representation, making it easy to switch between different layouts when needed

Examples

At its simplest, foldedtensor can be used to convert nested Python lists into a PyTorch tensor:

from foldedtensor import as_folded_tensor

ft = as_folded_tensor(
    [
        [0, 1, 2],
        [3],
    ],
)
# FoldedTensor([[0, 1, 2],
#               [3, 0, 0]])

You can also specify names and flattened/unflattened dimensions at the time of creation:

import torch
from foldedtensor import as_folded_tensor

# Creating a folded tensor from a nested list
# There are 2 samples, the first with 5 lines, the second with 1 line.
# Each line contain between 1 and 2 words.
ft = as_folded_tensor(
    [
        [[1], [], [], [], [2, 3]],
        [[4, 3]],
    ],
    data_dims=("samples", "words"),
    full_names=("samples", "lines", "words"),
    dtype=torch.long,
)
print(ft)
# FoldedTensor([[1, 2, 3],
#               [4, 3, 0]])

Once created, you can change the shape of the tensor by refolding it:

# Refold on the lines and words dims (flatten the samples dim)
print(ft.refold(("lines", "words")))
# FoldedTensor([[1, 0],
#               [0, 0],
#               [0, 0],
#               [0, 0],
#               [2, 3],
#               [4, 3]])

# Refold on the words dim only: flatten everything
print(ft.refold(("words",)))
# FoldedTensor([1, 2, 3, 4, 3])

The tensor can be further used with standard PyTorch operations:

# Working with PyTorch operations
embedder = torch.nn.Embedding(10, 16)
embedding = embedder(ft.refold(("words",)))
print(embedding.shape)
# torch.Size([5, 16]) # 5 words total, 16 dims

refolded_embedding = embedding.refold(("samples", "words"))
print(refolded_embedding.shape)
# torch.Size([2, 5, 16]) # 2 samples, 5 words max, 16 dims

Benchmarks

View the comparisons of foldedtensor against various alternatives here: docs/benchmarks.

Comparison with alternatives

Unlike other ragged or nested tensor implementations, a FoldedTensor does not enforce a specific structure on the nested data, and does not require padding all dimensions. This provides the user with greater flexibility when working with data that can be arranged in multiple ways depending on the data transformation. Moreover, the C++ optimization ensures high performance, making it ideal for handling deeply nested tensors efficiently.

Here is a comparison with other common implementations for handling nested sequences of variable length:

Feature NestedTensor MaskedTensor FoldedTensor
Inner data structure Flat Padded Arbitrary
Max nesting level 1 1
From nested python lists No No Yes
Layout conversion To padded No Any
Reduction ops w/o padding Yes No No

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

foldedtensor-0.3.5.tar.gz (18.4 kB view details)

Uploaded Source

Built Distributions

foldedtensor-0.3.5-cp312-cp312-win_amd64.whl (83.4 kB view details)

Uploaded CPython 3.12 Windows x86-64

foldedtensor-0.3.5-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (120.3 kB view details)

Uploaded CPython 3.12 manylinux: glibc 2.17+ x86-64

foldedtensor-0.3.5-cp312-cp312-macosx_11_0_arm64.whl (83.2 kB view details)

Uploaded CPython 3.12 macOS 11.0+ ARM64

foldedtensor-0.3.5-cp312-cp312-macosx_10_9_x86_64.whl (86.4 kB view details)

Uploaded CPython 3.12 macOS 10.9+ x86-64

foldedtensor-0.3.5-cp311-cp311-win_amd64.whl (83.1 kB view details)

Uploaded CPython 3.11 Windows x86-64

foldedtensor-0.3.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (120.7 kB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

foldedtensor-0.3.5-cp311-cp311-macosx_11_0_arm64.whl (84.6 kB view details)

Uploaded CPython 3.11 macOS 11.0+ ARM64

foldedtensor-0.3.5-cp311-cp311-macosx_10_9_x86_64.whl (87.2 kB view details)

Uploaded CPython 3.11 macOS 10.9+ x86-64

foldedtensor-0.3.5-cp310-cp310-win_amd64.whl (81.9 kB view details)

Uploaded CPython 3.10 Windows x86-64

foldedtensor-0.3.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (119.1 kB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

foldedtensor-0.3.5-cp310-cp310-macosx_11_0_arm64.whl (83.3 kB view details)

Uploaded CPython 3.10 macOS 11.0+ ARM64

foldedtensor-0.3.5-cp310-cp310-macosx_10_9_x86_64.whl (85.8 kB view details)

Uploaded CPython 3.10 macOS 10.9+ x86-64

foldedtensor-0.3.5-cp39-cp39-win_amd64.whl (81.8 kB view details)

Uploaded CPython 3.9 Windows x86-64

foldedtensor-0.3.5-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (119.1 kB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

foldedtensor-0.3.5-cp39-cp39-macosx_11_0_arm64.whl (83.4 kB view details)

Uploaded CPython 3.9 macOS 11.0+ ARM64

foldedtensor-0.3.5-cp39-cp39-macosx_10_9_x86_64.whl (85.9 kB view details)

Uploaded CPython 3.9 macOS 10.9+ x86-64

foldedtensor-0.3.5-cp38-cp38-win_amd64.whl (81.9 kB view details)

Uploaded CPython 3.8 Windows x86-64

foldedtensor-0.3.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (119.0 kB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

foldedtensor-0.3.5-cp38-cp38-macosx_11_0_arm64.whl (83.2 kB view details)

Uploaded CPython 3.8 macOS 11.0+ ARM64

foldedtensor-0.3.5-cp38-cp38-macosx_10_9_x86_64.whl (85.6 kB view details)

Uploaded CPython 3.8 macOS 10.9+ x86-64

foldedtensor-0.3.5-cp37-cp37m-win_amd64.whl (82.7 kB view details)

Uploaded CPython 3.7m Windows x86-64

foldedtensor-0.3.5-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (121.0 kB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64

foldedtensor-0.3.5-cp37-cp37m-macosx_10_9_x86_64.whl (85.4 kB view details)

Uploaded CPython 3.7m macOS 10.9+ x86-64

File details

Details for the file foldedtensor-0.3.5.tar.gz.

File metadata

  • Download URL: foldedtensor-0.3.5.tar.gz
  • Upload date:
  • Size: 18.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/5.1.1 CPython/3.12.6

File hashes

Hashes for foldedtensor-0.3.5.tar.gz
Algorithm Hash digest
SHA256 d53c09fd076ba88896b88915ba4c9dcd2b99a092ab33cbb9f6309861cacdf304
MD5 4694840757b959e0091e3c264d0f5aec
BLAKE2b-256 0f77d24fd67a10a9efcb9f671eb76598215b372e9c239703da3cb3935d826c7d

See more details on using hashes here.

Provenance

File details

Details for the file foldedtensor-0.3.5-cp312-cp312-win_amd64.whl.

File metadata

File hashes

Hashes for foldedtensor-0.3.5-cp312-cp312-win_amd64.whl
Algorithm Hash digest
SHA256 07a7306a501820147c14d6ea561571ed27014ba2736f8b25e0a31befbf93d591
MD5 69ac8a652c8fc294635e1b11bc5b9e22
BLAKE2b-256 8e78dd945afda9912c25f59858593402cd0207eda218d7bd32cd06f931cafa24

See more details on using hashes here.

Provenance

File details

Details for the file foldedtensor-0.3.5-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for foldedtensor-0.3.5-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 5bcf77019f273daa263165ebf4d8ad10a405f27b3a35065affb20da514fc59f2
MD5 36a7ff4665dfd8747cfcfaa79de6019d
BLAKE2b-256 2dba2d14dbd0d57c2f53709d461158c7602a45ecce914566f3cb2f231dd06813

See more details on using hashes here.

Provenance

File details

Details for the file foldedtensor-0.3.5-cp312-cp312-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for foldedtensor-0.3.5-cp312-cp312-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 1c8c473e5be655b817800da0acb719c6439bdd972c8d09a8332de7a013f7e429
MD5 3541a492f78bdc2ab8902ae01e44b481
BLAKE2b-256 1d4f84d78ac4a4a9fa0aaa9e7083f44f932578734bbca0c0fed40a60ffb7e9d7

See more details on using hashes here.

Provenance

File details

Details for the file foldedtensor-0.3.5-cp312-cp312-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for foldedtensor-0.3.5-cp312-cp312-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 51ee0f94f59ae6a10f5ed0b832a1a0cb269dac7e496c9277ca073839ed5c377d
MD5 9afadd560b40d76eec62d1f0c5a2fe30
BLAKE2b-256 4cf48b01106c7d18ed7c8ac620d2775c2874eb1e722fda02254f7aea7eef05e3

See more details on using hashes here.

Provenance

File details

Details for the file foldedtensor-0.3.5-cp311-cp311-win_amd64.whl.

File metadata

File hashes

Hashes for foldedtensor-0.3.5-cp311-cp311-win_amd64.whl
Algorithm Hash digest
SHA256 43f588db05201f5d512e0e37ace39bc592d8ee4912e5f3fc8c7e629b5c8c5d1f
MD5 44cc06103de846d1d7ce8f3719e7c04d
BLAKE2b-256 593e348a82dfda81e2a65ae6386826d8251756a109051e4830cf5a1e59149d49

See more details on using hashes here.

Provenance

File details

Details for the file foldedtensor-0.3.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for foldedtensor-0.3.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 46c29a22d43fdc44f6e5f68f86a841ca22c8b7fcf84148e2b06186e08f17596f
MD5 f0f93fb656e09fdbb31e320a142ae7d0
BLAKE2b-256 bf79c22809cb7b43b3a685c9e63ae95f1d17fff3714cae18356d881057dd3959

See more details on using hashes here.

Provenance

File details

Details for the file foldedtensor-0.3.5-cp311-cp311-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for foldedtensor-0.3.5-cp311-cp311-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 b41b559792b3336c5eec345197715d7de27b4e4dfffc29673a7603fe05492e1c
MD5 62bf6442246aa0667392fc0ddb3d29bf
BLAKE2b-256 207c22b17bd38c1b8a90c3afcb9d38c7f23ec42814a0e8cc8b4117b4e282c17a

See more details on using hashes here.

Provenance

File details

Details for the file foldedtensor-0.3.5-cp311-cp311-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for foldedtensor-0.3.5-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 9d74caaf4f1992d1d05ddd82f19de82d829c49c6e5120554844c9619ce10ab64
MD5 617fd1874adcbc6310d3368b4ef29713
BLAKE2b-256 456dd04a397de0c9f55c0590051c9abc9365f74f210b5d51f9938bd369b75c01

See more details on using hashes here.

Provenance

File details

Details for the file foldedtensor-0.3.5-cp310-cp310-win_amd64.whl.

File metadata

File hashes

Hashes for foldedtensor-0.3.5-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 c864a3e4ad59709da8e5e1f830c78fce8cf1e75ff2ebfeeac150592ac7052a9a
MD5 08a896b77d7df0c5a0c441f2febf2386
BLAKE2b-256 de978b541b14e5578a58d461088620aa2dd2fb82a4bc94bdfcb5bea61685a411

See more details on using hashes here.

Provenance

File details

Details for the file foldedtensor-0.3.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for foldedtensor-0.3.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 89ea9a24b8a67e0ddd8d37578367eeb809b9d83dde8a2b9e6e2b01d1290ecb43
MD5 734799093e981c9c4fd6e454c4b04a28
BLAKE2b-256 fb668371d82b6b3ac29705fffcb24884de97e2a0b3bb192ea33cd557ab93db4d

See more details on using hashes here.

Provenance

File details

Details for the file foldedtensor-0.3.5-cp310-cp310-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for foldedtensor-0.3.5-cp310-cp310-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 cacb506377f381c7fbe9f7015fa0425941d180cd41d8f54ab9fe7ddda6d4bd84
MD5 8d31f097cff975308189b1fb734576d7
BLAKE2b-256 091fbb472f08915b23647d3835b57bf9f7191f7102217339689254fd740eeb71

See more details on using hashes here.

Provenance

File details

Details for the file foldedtensor-0.3.5-cp310-cp310-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for foldedtensor-0.3.5-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 53aa70fa00670fa8d093f6bed15d48b16787cc463d432d1f7d0c1face68f2ff7
MD5 54a0c876e958e36cde69191a5519680d
BLAKE2b-256 58b7ffad3f13dc4c802fc9a4424df6925503f1d89710b42a512f17a81e48e807

See more details on using hashes here.

Provenance

File details

Details for the file foldedtensor-0.3.5-cp39-cp39-win_amd64.whl.

File metadata

File hashes

Hashes for foldedtensor-0.3.5-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 632376de6b435da0d6bac8428e44b14527724197e6109c4515fa3d0e4e981a0a
MD5 b9019a876fceae8b9d77e798e0661a7d
BLAKE2b-256 56f65ff95e3e255d398dace9ca8c38f87f114e872d83b22bbbd09beca8aa18bc

See more details on using hashes here.

Provenance

File details

Details for the file foldedtensor-0.3.5-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for foldedtensor-0.3.5-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 7d4162492c8a35e230e61c7adda219719b3797d7c0a16efcbbbbde83e1b2a905
MD5 3655e057b7de3bb59ca3b0872cf4edf4
BLAKE2b-256 520e48a61b2ced61feaabce36b59ec31c02a628a3386612e21ce97c5a1fe7e48

See more details on using hashes here.

Provenance

File details

Details for the file foldedtensor-0.3.5-cp39-cp39-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for foldedtensor-0.3.5-cp39-cp39-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 d498072cddf34da0ad3edd6eb908618cb536068a6021ad937d097c13025241f4
MD5 5ef81cedfab77d16a81d480600731e59
BLAKE2b-256 3645598e1eaf6fe49122080d45874957e9fb5c5592cb20396511e23f09d68baa

See more details on using hashes here.

Provenance

File details

Details for the file foldedtensor-0.3.5-cp39-cp39-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for foldedtensor-0.3.5-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 781b8c145459a2d3cd5f43dbc4b469aad55626f21475853397fb6608217beafe
MD5 47b0ab29a5bbb990c10fe314dbe5fe5d
BLAKE2b-256 22712655d64c94b2ee96f019ff01bbf9413dd8a67b3bad3570e936ee6cb91863

See more details on using hashes here.

Provenance

File details

Details for the file foldedtensor-0.3.5-cp38-cp38-win_amd64.whl.

File metadata

File hashes

Hashes for foldedtensor-0.3.5-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 e5e706ba011447e18dcd079cbb62e269433e05fa80d12cda5152b1d054328776
MD5 04ce76e5f61a3310d4891ee6aa5f09b7
BLAKE2b-256 bbe34fdb40f2cf5ec657d5be1359e2ce99209e1fe784987c1be46e88ad11a2e4

See more details on using hashes here.

Provenance

File details

Details for the file foldedtensor-0.3.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for foldedtensor-0.3.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 b2239b18d398f6708378fff65ad0b307ae276120d939aede3d3a419047801232
MD5 b95ee17defa16137119dc8e4ecbe1166
BLAKE2b-256 adf83696d8dbd2faad8bdcafebb128820fe545bd7d99637df920fc74c893706e

See more details on using hashes here.

Provenance

File details

Details for the file foldedtensor-0.3.5-cp38-cp38-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for foldedtensor-0.3.5-cp38-cp38-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 aad0c26c2f86765cc4f400fc9e7d04c6f9ed59f32506821941925b1172e65c32
MD5 5d3926112f3da9e624c925c8b715067d
BLAKE2b-256 5a5c180df1f1b270a5c3c3f8a61618ec75601b12b8f31db4cad631e2172e3c0d

See more details on using hashes here.

Provenance

File details

Details for the file foldedtensor-0.3.5-cp38-cp38-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for foldedtensor-0.3.5-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 f8c66a3c701263293f892d14662a1373b94157cb4e5bd44c42f7650cd4d01ff9
MD5 3559fed1086b057843e1aa93e590acb5
BLAKE2b-256 d5d528f88c90d8d573a0ab6a77137e50849e94b95b52f0c00a81f9acd9e1b31c

See more details on using hashes here.

Provenance

File details

Details for the file foldedtensor-0.3.5-cp37-cp37m-win_amd64.whl.

File metadata

File hashes

Hashes for foldedtensor-0.3.5-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 a3f9eedf10a8afde2852b57ad07bd9227f64f1403c711f33853ae880b348976e
MD5 d5bc8f03f54ff34fdd470a43302028a9
BLAKE2b-256 32a4311689b46ec80b6bef44235d2d1d4989e7d41058ff49f086cab1ad689bce

See more details on using hashes here.

Provenance

File details

Details for the file foldedtensor-0.3.5-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for foldedtensor-0.3.5-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 4b0254ee39ecea79571c9c246c9c98d36a4861857c724c72c30a8e7f26bb6373
MD5 c11ea1a6665e66f8568b67a02127ff4f
BLAKE2b-256 fbf32bfb2df3d1672b3176ebe7b297f2bf2fb925aefa60c1b42f91d999887040

See more details on using hashes here.

Provenance

File details

Details for the file foldedtensor-0.3.5-cp37-cp37m-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for foldedtensor-0.3.5-cp37-cp37m-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 d009ca99e4ad64c26ab9e11d52b0536c45d8e793760b37ba297dd0a11b19c11c
MD5 765c3dc985b6f4405b53aa4a173b90b6
BLAKE2b-256 1c262c8b315378044ae4eeecef2569280e40286049541df2c20e9cc3274dfd9b

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page