Opinionated JSON to CSV converter

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for kindly from gravatar.com kindly

Unverified details

These details have not been verified by PyPI
Meta

Project description

Full Documentation

Introduction

An opinionated JSON to CSV/XLSX/SQLITE/PARQUET converter which tries to make a useful relational output for data analysis.

Web playgroud of CSV/XLSX conversions

Rationale

When receiving a JSON file where the structure is deeply nested or not well specified, it is hard to determine what the data contains. Also, even after knowing the JSON structure, it requires a lot of time to work out how to flatten the JSON into a relational structure to do data analysis on and to be part of a data pipeline.

Flatterer aims to be the first tool to go to when faced with the above problem. It may not be the tool that you end up using to flatten the JSON in your data pipeline, as hand written flattening may be required, but it could be. It has many benefits over most hand written approaches:

  • It is fast, written in rust but with python bindings for ease of use. It can be 10x faster than hand written python flattening.
  • Memory efficient. Uses a custom streaming JSON parser to mean that long list of objects nested with the JSON will be streamed, so not much data needs to be loaded into memory at once.
  • Fast memory efficient output to CSV/XLSX/SQLITE/PARQUET
  • Uses best practice that has been learnt from flattening JSON countless times, such as generating keys to link one-to-many tables to their parents.

Install

pip install flatterer

Flatterer requires Python 3.6 or greater. It is written as a python extension in Rust but has binaries (wheels) for linux (x64), macos (x64 and universal) and windows (x64, x86). On other platforms a rust toolchain will need to be installed.

Example JSON

Say you have a JSON data like this named games.json:

[
  {
    "id": 1,
    "title": "A Game",
    "releaseDate": "2015-01-01",
    "platforms": [
      {"name":"Xbox"},
      {"name":"Playstation"}
    ],
    "rating": {
      "code": "E",
      "name": "Everyone"
    }
  },
  {
    "id": 2,
    "title": "B Game",
    "releaseDate": "2016-01-01",
    "platforms": [
      {"name":"PC"}
    ],
    "rating": {
      "code": "E",
      "name": "Everyone"
    }
  }
]

Running Flatterer

Run the above file with flatterer.

flatterer games.json games_dir

Output Files

By running the above you will get the following files:

tree games_dir

games_dir/
├── csv
│   ├── games.csv
│   └── platforms.csv
├── datapackage.json
├── fields.csv
└── ...

Main Table

games.csv contains:

_link _link_games id rating_code rating_name releaseDate title
1 1 1 E Everyone 2015-01-01 A Game
2 2 2 E Everyone 2016-01-01 B Game

Special column _link is generated. _link is the primary key there unique per game.

Also the rating sub-object is promoted to this table it has a one-to-one relationship with games. Sub-object properties are separated by '_'.

One To Many Table

platforms is an array so is a one-to-many with games therefore needs its own table: platforms.csv contains:

_link _link_games name
1.platforms.0 1 Xbox
1.platforms.1 1 Playstation
2.platforms.0 2 PC

Link Fields

_link is the primary key for the platforms table too. Every table except games table, contains a _link_games field to easily join to the main games table.

If there was a sub-array of platforms then that would have _link, _link_games and _link_platforms fields.

To generalize this the _link__<table_name> fields joins to the _link field of <table_name> i.e the _link__<table_name> are the foreign keys refrencing <table_name>._link.

Fields CSV

fields.csv contains some metadata about the output tables:

table_name field_name field_type count field_title
platforms _link text 3 _link
platforms _link_games text 3 _link_games
platforms name text 3 name
games _link text 2 _link
games id number 2 id
games rating_code text 2 rating_code
games rating_name text 2 rating_name
games releaseDate date 2 releaseDate
games title text 2 title

The field_type column contains a type guess useful for inserting into a database. The field_title is the column heading in the CSV file or XLSX tab, which is initally the same as the field_name. After editing this file then you can rerun the transform:

flatterer games.json new_games_dir -f myfields.csv --only-fields

This can be useful for renameing columns, rearranging the field order or if you want to remove some fields the --only-fields flag will only include the fields in the edited file.

datapackage.json contains metadata in the Tabular Datapackge Spec

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for kindly from gravatar.com kindly

Unverified details

These details have not been verified by PyPI
Meta

Release history Release notifications | RSS feed

0.20.1

This version

0.20.0

0.19.19

0.19.18

0.19.17

0.19.16

0.19.15

0.19.13

0.19.12

0.19.10

0.18.2

0.17.1

0.17.0

0.16.3

0.15.0

0.13.1

0.12.12

0.11.0

0.10.1

0.9.0

0.8.0

0.3.1

0.3.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

flatterer-0.20.0-cp313-none-win_amd64.whl (19.9 MB view details)

Uploaded CPython 3.13 Windows x86-64

flatterer-0.20.0-cp313-none-win32.whl (17.8 MB view details)

Uploaded CPython 3.13 Windows x86

flatterer-0.20.0-cp313-cp313-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl (38.3 MB view details)

Uploaded CPython 3.13 macOS 10.9+ universal2 (ARM64, x86-64) macOS 10.9+ x86-64 macOS 11.0+ ARM64

flatterer-0.20.0-cp313-cp313-macosx_10_7_x86_64.whl (21.5 MB view details)

Uploaded CPython 3.13 macOS 10.7+ x86-64

flatterer-0.20.0-cp312-none-win_amd64.whl (19.9 MB view details)

Uploaded CPython 3.12 Windows x86-64

flatterer-0.20.0-cp312-none-win32.whl (17.8 MB view details)

Uploaded CPython 3.12 Windows x86

flatterer-0.20.0-cp312-cp312-manylinux_2_28_x86_64.whl (25.0 MB view details)

Uploaded CPython 3.12 manylinux: glibc 2.28+ x86-64

flatterer-0.20.0-cp312-cp312-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl (38.4 MB view details)

Uploaded CPython 3.12 macOS 10.9+ universal2 (ARM64, x86-64) macOS 10.9+ x86-64 macOS 11.0+ ARM64

flatterer-0.20.0-cp312-cp312-macosx_10_7_x86_64.whl (21.5 MB view details)

Uploaded CPython 3.12 macOS 10.7+ x86-64

flatterer-0.20.0-cp311-none-win_amd64.whl (19.9 MB view details)

Uploaded CPython 3.11 Windows x86-64

flatterer-0.20.0-cp311-none-win32.whl (17.8 MB view details)

Uploaded CPython 3.11 Windows x86

flatterer-0.20.0-cp311-cp311-manylinux_2_28_x86_64.whl (25.0 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.28+ x86-64

flatterer-0.20.0-cp311-cp311-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl (38.4 MB view details)

Uploaded CPython 3.11 macOS 10.9+ universal2 (ARM64, x86-64) macOS 10.9+ x86-64 macOS 11.0+ ARM64

flatterer-0.20.0-cp311-cp311-macosx_10_7_x86_64.whl (21.5 MB view details)

Uploaded CPython 3.11 macOS 10.7+ x86-64

flatterer-0.20.0-cp310-none-win_amd64.whl (19.9 MB view details)

Uploaded CPython 3.10 Windows x86-64

flatterer-0.20.0-cp310-none-win32.whl (17.8 MB view details)

Uploaded CPython 3.10 Windows x86

flatterer-0.20.0-cp310-cp310-manylinux_2_28_x86_64.whl (25.0 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.28+ x86-64

flatterer-0.20.0-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl (38.4 MB view details)

Uploaded CPython 3.10 macOS 10.9+ universal2 (ARM64, x86-64) macOS 10.9+ x86-64 macOS 11.0+ ARM64

flatterer-0.20.0-cp310-cp310-macosx_10_7_x86_64.whl (21.5 MB view details)

Uploaded CPython 3.10 macOS 10.7+ x86-64

flatterer-0.20.0-cp37-cp37m-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl (38.3 MB view details)

Uploaded CPython 3.7m macOS 10.9+ universal2 (ARM64, x86-64) macOS 10.9+ x86-64 macOS 11.0+ ARM64

flatterer-0.20.0-cp37-cp37m-macosx_10_7_x86_64.whl (21.5 MB view details)

Uploaded CPython 3.7m macOS 10.7+ x86-64

File details

Details for the file flatterer-0.20.0-cp313-none-win_amd64.whl.

File metadata

File hashes

Hashes for flatterer-0.20.0-cp313-none-win_amd64.whl
Algorithm Hash digest
SHA256 36c15df02d240be4efb5c4da27bdde42866b676ed6d3826e377c941c6343df1a
MD5 369cd308e6642e169d0e628475239175
BLAKE2b-256 533851b4f578846a142b1b2362b9f5822f6451a1bc6be1a94fa2325d167fe81c

See more details on using hashes here.

File details

Details for the file flatterer-0.20.0-cp313-none-win32.whl.

File metadata

  • Download URL: flatterer-0.20.0-cp313-none-win32.whl
  • Upload date:
  • Size: 17.8 MB
  • Tags: CPython 3.13, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.9.20

File hashes

Hashes for flatterer-0.20.0-cp313-none-win32.whl
Algorithm Hash digest
SHA256 2457c609d0153a3623c0c62bab5f430fd518cff1fadfe46cfede4c18d668ce7a
MD5 3ff51869103cafa16851a3214cccddf9
BLAKE2b-256 dbe435a1a5c02ba12dd069d7b269caabed2d900d546493ea3b56cc4df69799eb

See more details on using hashes here.

File details

Details for the file flatterer-0.20.0-cp313-cp313-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl.

File metadata

File hashes

Hashes for flatterer-0.20.0-cp313-cp313-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm Hash digest
SHA256 b8496980665d3cf38b0a9c876d94bbd7c91fe391494904cd8bba5a8078b78d35
MD5 7e578a7018510e8ec1ccf636236fb1c5
BLAKE2b-256 d7d48df56001c96f1fddd7672eb9d9c649e51771f05495e722a508fcaaf9b139

See more details on using hashes here.

File details

Details for the file flatterer-0.20.0-cp313-cp313-macosx_10_7_x86_64.whl.

File metadata

File hashes

Hashes for flatterer-0.20.0-cp313-cp313-macosx_10_7_x86_64.whl
Algorithm Hash digest
SHA256 906eb483166308b8a2b2a752bac4e8efd0946eb3d54eece31c913de0d706db05
MD5 10e6b8f52f9a8261f6c5c261bb226ed3
BLAKE2b-256 45b6b0fc179fd0d3a359b53402cefc0ccf413919f6ff45ba9403d2a924905022

See more details on using hashes here.

File details

Details for the file flatterer-0.20.0-cp312-none-win_amd64.whl.

File metadata

File hashes

Hashes for flatterer-0.20.0-cp312-none-win_amd64.whl
Algorithm Hash digest
SHA256 8a8e38c70aff61a922f297169107335aed8b589a2f1f1b42df8285d23e80c260
MD5 d43a37fb6885be8fdc1e503b438d9db4
BLAKE2b-256 0ba463088252c3936b5261ba4e4015636007be112a6f9f65d9a1cb0d107b1bc6

See more details on using hashes here.

File details

Details for the file flatterer-0.20.0-cp312-none-win32.whl.

File metadata

  • Download URL: flatterer-0.20.0-cp312-none-win32.whl
  • Upload date:
  • Size: 17.8 MB
  • Tags: CPython 3.12, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.9.20

File hashes

Hashes for flatterer-0.20.0-cp312-none-win32.whl
Algorithm Hash digest
SHA256 f9491dc3e3c27e220a5ea0d6231b9d3267d3b05db8f2edd55c184f0012ed1d50
MD5 9f52a2327ced02a384e1df34a10bcf32
BLAKE2b-256 e18d5237a8127eb43bc6078fb926818f86344beb74e10d77715ee2e5a2ae0038

See more details on using hashes here.

File details

Details for the file flatterer-0.20.0-cp312-cp312-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for flatterer-0.20.0-cp312-cp312-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 bfc076c31a8254e027fc863dd99bf0b42f37bcb7c0021b050f8b886a1970dd71
MD5 f94455916416ee0b49070d6f6a31ab07
BLAKE2b-256 25bcb9f64bcbaba4f50bc95e9320b19c9a7ed6908bed3ca407a860a966573329

See more details on using hashes here.

File details

Details for the file flatterer-0.20.0-cp312-cp312-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl.

File metadata

File hashes

Hashes for flatterer-0.20.0-cp312-cp312-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm Hash digest
SHA256 e98ba40d18a682502b7c5e49d02e8629e9119f4efb864ad22592176ec98e53d3
MD5 00ea42a56bdded509771672ad1fb04dc
BLAKE2b-256 dea827d02ad7519844287e2787a69c36c192f0cc5b472c3adfc061bcb8ec8c5e

See more details on using hashes here.

File details

Details for the file flatterer-0.20.0-cp312-cp312-macosx_10_7_x86_64.whl.

File metadata

File hashes

Hashes for flatterer-0.20.0-cp312-cp312-macosx_10_7_x86_64.whl
Algorithm Hash digest
SHA256 b2c246fb4bedccfb589ea31896d5f8fdddf63a6a624464b8edc60aac1d32cf3e
MD5 65ee1a286fd6a9618f40f34c4f78a864
BLAKE2b-256 3f3c2824a359e3d4c02d58d66b1674c52f3795f452ccbe732a6d0c2f745c54fb

See more details on using hashes here.

File details

Details for the file flatterer-0.20.0-cp311-none-win_amd64.whl.

File metadata

File hashes

Hashes for flatterer-0.20.0-cp311-none-win_amd64.whl
Algorithm Hash digest
SHA256 6d86bd19c10829692a60300afa3c324d8df24cb378b8951dcbb13858d80f4ea2
MD5 913898f7af0cc18b87f5e4c7ed6e56b7
BLAKE2b-256 d96854bcbef87c8fe07dfe39cad0bf0899e167f6f8bf8ea6e8b65fff8439fa86

See more details on using hashes here.

File details

Details for the file flatterer-0.20.0-cp311-none-win32.whl.

File metadata

  • Download URL: flatterer-0.20.0-cp311-none-win32.whl
  • Upload date:
  • Size: 17.8 MB
  • Tags: CPython 3.11, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.9.20

File hashes

Hashes for flatterer-0.20.0-cp311-none-win32.whl
Algorithm Hash digest
SHA256 eb9e6cc488f90a25b25f58a633abc106e36621ff37a44605eb986ff6d9265468
MD5 8e72528bfb2b589708bcc292ac930292
BLAKE2b-256 33c011b0cb1a2bc91055d90f7fa994f83d4ecba0fc87c513cc177bdec5f6abf4

See more details on using hashes here.

File details

Details for the file flatterer-0.20.0-cp311-cp311-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for flatterer-0.20.0-cp311-cp311-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 c3f84ad1c12c2826c66d30ae12b5c26ff6d6b109ceb8b4f8d81174ce33740862
MD5 3f010a535d130bb8260d9c81d232279c
BLAKE2b-256 c5ddc9f9c92dd1f1b8ae7685e3f73a3d69424ac30b2c8480febc5c19e8b7f173

See more details on using hashes here.

File details

Details for the file flatterer-0.20.0-cp311-cp311-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl.

File metadata

File hashes

Hashes for flatterer-0.20.0-cp311-cp311-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm Hash digest
SHA256 8cf77eff8f5e8bed9139ca438319d8d2e0c1e67e8e7ebff9434698d11694380a
MD5 c3e74e45a09ecebb15830d3e74376194
BLAKE2b-256 19dbc4d90c16d6549e0a1df76200eab13e1ee02acf751173fb467c905d1a9130

See more details on using hashes here.

File details

Details for the file flatterer-0.20.0-cp311-cp311-macosx_10_7_x86_64.whl.

File metadata

File hashes

Hashes for flatterer-0.20.0-cp311-cp311-macosx_10_7_x86_64.whl
Algorithm Hash digest
SHA256 62eb8bd8fe2fc66d73ee18e326329ec7330d0da9e3c21041d1f14d1e19bbbf0c
MD5 5a4193a8302155a05f8f282cff393508
BLAKE2b-256 d2147598dc6a8f43ef164812293b61835aa6fa5370bf827407075420636cc5e7

See more details on using hashes here.

File details

Details for the file flatterer-0.20.0-cp310-none-win_amd64.whl.

File metadata

File hashes

Hashes for flatterer-0.20.0-cp310-none-win_amd64.whl
Algorithm Hash digest
SHA256 8ab48f3be90931203dbc4ed9f7426f50d5d9495eda0d473edb6b8724777dc8f6
MD5 934e5729d62821150c69c6bbef6d31e7
BLAKE2b-256 2f192014ccf49f788f9c8f21038f6526ab6878c7a79bb03694da0aa382dd802f

See more details on using hashes here.

File details

Details for the file flatterer-0.20.0-cp310-none-win32.whl.

File metadata

  • Download URL: flatterer-0.20.0-cp310-none-win32.whl
  • Upload date:
  • Size: 17.8 MB
  • Tags: CPython 3.10, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.9.20

File hashes

Hashes for flatterer-0.20.0-cp310-none-win32.whl
Algorithm Hash digest
SHA256 2436d21157b3678944a2a2f9ad441cfe8bbed2813d263cf055cf7cf8ef27c3a1
MD5 a52e1ce89759d069b26efc85ca13487a
BLAKE2b-256 b6ee156a63288d4597e785369c937741eda531527bfa0c2f3100ef089c838340

See more details on using hashes here.

File details

Details for the file flatterer-0.20.0-cp310-cp310-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for flatterer-0.20.0-cp310-cp310-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 6b00f4dd4d5c252878dbc22dfbc0e1627cf6fcd030fd4dc89633db4f98777f6b
MD5 f3242520aeeaaedd3a7b08d4599f05f5
BLAKE2b-256 cd19c60fa07b813922319c6fbb5bd41b6f98e125b4e4fc728eff58383065c523

See more details on using hashes here.

File details

Details for the file flatterer-0.20.0-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl.

File metadata

File hashes

Hashes for flatterer-0.20.0-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm Hash digest
SHA256 a7342a83c5deb7c7a6996051a9f275bc7df11db255e39af87902c6f9024404db
MD5 cf5da8d696559f5faeaed5c189b36473
BLAKE2b-256 9f36fbe51759bb3417c348d4a0caed88dd7a1d2f52b7549faf770fb12e3cb0ed

See more details on using hashes here.

File details

Details for the file flatterer-0.20.0-cp310-cp310-macosx_10_7_x86_64.whl.

File metadata

File hashes

Hashes for flatterer-0.20.0-cp310-cp310-macosx_10_7_x86_64.whl
Algorithm Hash digest
SHA256 3f1bc8c633e159160e92fdeb096d0e11cd8ab3e8134ee2517259734b8f40b090
MD5 0d22cdd727eff0b40c2e3cc052791b3c
BLAKE2b-256 a7297c1cc33c7d09dcba4fe3cc94a1c4206436ba55697ceaf9d5a780f7f03bbc

See more details on using hashes here.

File details

Details for the file flatterer-0.20.0-cp37-cp37m-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl.

File metadata

File hashes

Hashes for flatterer-0.20.0-cp37-cp37m-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm Hash digest
SHA256 498c6932bce468cb8b3bfed489afc288910b6636abf0f1a6e85f7beb28fbbc09
MD5 3cdbe7ea8ae168a33e9a49bbeeb93d7c
BLAKE2b-256 d48eefb82dfb078f39b52c832a3837ed49fdfa2fef22d6251c4ed22acf024c73

See more details on using hashes here.

File details

Details for the file flatterer-0.20.0-cp37-cp37m-macosx_10_7_x86_64.whl.

File metadata

File hashes

Hashes for flatterer-0.20.0-cp37-cp37m-macosx_10_7_x86_64.whl
Algorithm Hash digest
SHA256 0a9dff102c7cfbd20e8274cf2807775c0494465ad23f091a063dc8ce45560645
MD5 589e8b617556b7a33d03f9839efad24c
BLAKE2b-256 f3d0f89e53db16f7f08a4d281931bc003cca60bc2aebbb695bbccc005062ea47

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page