Skip to main content

The kwarray module

Project description

GitlabCIPipeline GitlabCICoverage Appveyor Pypi Downloads ReadTheDocs

Read the docs

https://kwarray.readthedocs.io

Gitlab (main)

https://gitlab.kitware.com/computer-vision/kwarray

Github (mirror)

https://github.com/Kitware/kwarray

Pypi

https://pypi-hypernode.com/project/kwarray

The main webpage for this project is: https://gitlab.kitware.com/computer-vision/kwarray

The kwarray module implements a small set of pure-python extensions to numpy and torch.

The kwarray module started as extensions for numpy + a simplified pandas-like DataFrame object with much faster item row and column access. But it also include an ArrayAPI, which is a wrapper that allows 100% interoperability between torch and numpy. It also contains a few algorithms like setcover and mincost_assignment.

The top-level API is:

from kwarray.arrayapi import ArrayAPI, dtype_info
from .algo_assignment import (maxvalue_assignment, mincost_assignment,
                              mindist_assignment,)
from .algo_setcover import (setcover,)
from .dataframe_light import (DataFrameArray, DataFrameLight, LocLight,)
from .fast_rand import (standard_normal, standard_normal32, standard_normal64,
                        uniform, uniform32,)
from .util_averages import (RunningStats, stats_dict,)
from .util_groups import (apply_grouping, group_consecutive,
                          group_consecutive_indices, group_indices,
                          group_items,)
from .util_misc import (FlatIndexer,)
from .util_numpy import (arglexmax, argmaxima, argminima, atleast_nd, boolmask,
                         isect_flags, iter_reduce_ufunc, normalize,)
from .util_random import (ensure_rng, random_combinations, random_product,
                          seed_global, shuffle,)
from .util_slices import (embed_slice, padded_slice,)
from .util_slider import (SlidingWindow, Stitcher,)
from .util_torch import (one_hot_embedding, one_hot_lookup,)

The ArrayAPI

On of the most useful features in kwarray is the kwarray.ArrayAPI — a class that helps bridge between numpy and torch. This class consists of static methods that implement part of the numpy API and operate equivalently on either torch.Tensor or numpy.ndarray objects.

This works because every function call checks if the input is a torch tensor or a numpy array and then takes the appropriate action.

As you can imagine, it can be slow to validate your inputs on each function call. Therefore the recommended way of using the array API is via the kwarray.ArrayAPI.impl function. This function does the check once and then returns another object that directly performs the correct operations on subsequent data items of the same type.

The following example demonstrates both modes of usage.

import torch
import numpy as np
data1 = torch.rand(10, 10)
data2 = data1.numpy()
# Method 1: grab the appropriate sub-impl
impl1 = ArrayAPI.impl(data1)
impl2 = ArrayAPI.impl(data2)
result1 = impl1.sum(data1, axis=0)
result2 = impl2.sum(data2, axis=0)
assert np.all(impl1.numpy(result1) == impl2.numpy(result2))
# Method 2: choose the impl on the fly
result1 = ArrayAPI.sum(data1, axis=0)
result2 = ArrayAPI.sum(data2, axis=0)
assert np.all(ArrayAPI.numpy(result1) == ArrayAPI.numpy(result2))

Other Notes:

The kwarray.ensure_rng function helps you properly maintain and control local seeded random number generation. This means that you wont clobber the random state of another library / get your random state clobbered.

DataFrameArray and DataFrameLight implement a subset of the pandas API. They are less powerful, but orders of magnitude faster. The main drawback is that you lose loc, but iloc is available.

uniform32 and standard_normal32 are faster 32-bit random number generators (compared to their 64-bit numpy counterparts).

mincost_assignment is the Munkres / Hungarian algorithm. It solves the assignment problem.

setcover - solves the minimum weighted set cover problem using either an approximate or an exact solution.

one_hot_embedding is a fast numpy / torch way to perform the often needed OHE deep-learning trick.

group_items is a fast way to group a numpy array by another numpy array. For fine grained control we also expose group_indices, which groups the indices of a numpy array, and apply_grouping, which partitions a numpy array by those indices.

boolmask effectively inverts np.where.

Usefulness:

This is the frequency that I’ve used various components of this library with in my projects:

Function name

Usefulness

kwarray.ensure_rng

239

kwarray.ArrayAPI

148

kwarray.atleast_nd

50

kwarray.DataFrameArray

43

kwarray.group_indices

40

kwarray.stats_dict

34

kwarray.normalize

28

kwarray.embed_slice

21

kwarray.shuffle

17

kwarray.padded_slice

14

kwarray.SlidingWindow

14

kwarray.isect_flags

12

kwarray.RunningStats

12

kwarray.standard_normal

10

kwarray.setcover

8

kwarray.robust_normalize

7

kwarray.boolmask

7

kwarray.one_hot_embedding

7

kwarray.uniform

6

kwarray.find_robust_normalizers

6

kwarray.Stitcher

6

kwarray.apply_grouping

6

kwarray.group_consecutive

5

kwarray.argmaxima

4

kwarray.seed_global

4

kwarray.FlatIndexer

3

kwarray.group_items

3

kwarray.arglexmax

2

kwarray.DataFrameLight

2

kwarray.group_consecutive_indices

1

kwarray.equal_with_nan

1

kwarray.dtype_info

1

kwarray.unique_rows

0

kwarray.uniform32

0

kwarray.standard_normal64

0

kwarray.standard_normal32

0

kwarray.random_product

0

kwarray.random_combinations

0

kwarray.one_hot_lookup

0

kwarray.mindist_assignment

0

kwarray.mincost_assignment

0

kwarray.maxvalue_assignment

0

kwarray.iter_reduce_ufunc

0

kwarray.generalized_logistic

0

kwarray.argminima

0

kwarray.apply_embedded_slice

0

kwarray.NoSupportError

0

kwarray.LocLight

0

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

kwarray-0.7.1.tar.gz (112.9 kB view details)

Uploaded Source

Built Distribution

kwarray-0.7.1-py3-none-any.whl (108.9 kB view details)

Uploaded Python 3

File details

Details for the file kwarray-0.7.1.tar.gz.

File metadata

  • Download URL: kwarray-0.7.1.tar.gz
  • Upload date:
  • Size: 112.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.11.10

File hashes

Hashes for kwarray-0.7.1.tar.gz
Algorithm Hash digest
SHA256 80c76a27516a1f371d53c783938e1ebc19588167c8e31725939e6e43bb69cead
MD5 109be1584c02ad339ae7a80c7666e9a7
BLAKE2b-256 51090e555288cdedf3eac3738dfc8e3b6bfe07a01021e34024b49bb9ab42c6d9

See more details on using hashes here.

Provenance

File details

Details for the file kwarray-0.7.1-py3-none-any.whl.

File metadata

  • Download URL: kwarray-0.7.1-py3-none-any.whl
  • Upload date:
  • Size: 108.9 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.11.10

File hashes

Hashes for kwarray-0.7.1-py3-none-any.whl
Algorithm Hash digest
SHA256 378eacfeeae27ee967eaa15c49f5141c8ce1f0cf869c1aea8139bf216c6d7f4b
MD5 013d4915d66ff002863941c0e97cb126
BLAKE2b-256 b2a289e0bcbaa742950581cb6213a33b0f96f3d3e8e7e6186db2b98afb16ec38

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page