Skip to main content

Chunk large QuerySets into small chunks, and iterate over them without killing your RAM.

Project description

Chunk large QuerySets into small chunks, and iterate over them without killing your RAM.

https://travis-ci.org/peopledoc/django-chunkator.svg

Tested with all the combinations of:

  • Python: 3.5, 3.6, 3.7, 3.8

  • Django: 2, 2.1, 2.2, 3.0, master

Usage

from chunkator import chunkator
for item in chunkator(LargeModel.objects.all(), 200):
    do_something(item)

This tool is intended to work on Django querysets.

Your model must define a pk field (this is done by default, but sometimes it can be overridden) and this pk has to be unique. django- chunkator has been tested with PostgreSQL and SQLite, using regular PKs and UUIDs as primary keys.

You can also use values():

from chunkator import chunkator
for item in chunkator(LargeModel.objects.values('pk', 'name'), 200):
    do_something(item)

If you want to manipulate the pages directly, you can use chunkator_page:

from chunkator import chunkator_page
queryset = LargeModel.objects.all().values('pk')
for page in chunkator_page(queryset, 200):
    launch_some_task([item['pk'] for item in page])

License

MIT License.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

django-chunkator-2.0.0.tar.gz (3.6 kB view details)

Uploaded Source

Built Distribution

django_chunkator-2.0.0-py3-none-any.whl (3.3 kB view details)

Uploaded Python 3

File details

Details for the file django-chunkator-2.0.0.tar.gz.

File metadata

  • Download URL: django-chunkator-2.0.0.tar.gz
  • Upload date:
  • Size: 3.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.31.1 CPython/3.6.5

File hashes

Hashes for django-chunkator-2.0.0.tar.gz
Algorithm Hash digest
SHA256 57d705966762e6ba6879c2344a1b9288045f64c638fa47cbe00467f0eeeadfaf
MD5 8619a53f456be3a6ce786ee54deeeda5
BLAKE2b-256 acc3b3dd9b543068294cb6cae2b965aba927ac71b8adf17e5255ec2f0632cf78

See more details on using hashes here.

Provenance

File details

Details for the file django_chunkator-2.0.0-py3-none-any.whl.

File metadata

  • Download URL: django_chunkator-2.0.0-py3-none-any.whl
  • Upload date:
  • Size: 3.3 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/39.2.0 requests-toolbelt/0.8.0 tqdm/4.31.1 CPython/3.6.5

File hashes

Hashes for django_chunkator-2.0.0-py3-none-any.whl
Algorithm Hash digest
SHA256 210713faf05d68035d067de9b015868b3d902d5405865401384a3c134f64c43a
MD5 4eb33cdff127c777d6d7f080e97d0904
BLAKE2b-256 1e20c3b02663d901ddbdbc89a7b98de4cdae768214b0931921a14953ef058eb0

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page