A platform for describing, extracting, transforming, loading and serving open data.
Project description
Spinta is a framework to describe, extract, transform and publish data (DETP Framework). It supports a great deal of data schemes and formats.
Purpose
Describe your data: You can automatically generate data structure description table (Manifest) from many different data sources.
Extract your data: Once you have your data structure in Manifest tables, you can extract data from multiple external data sources. Extracted data are validated and transformed using rules defined in Manifest table. Finally, data can be stored into internal database in order to provide fast and flexible access to data.
Transform your data: Data transformations are applied in real time, when reading data from source. This puts some limitations on transformation side, but allows data to be streamed in real time.
Publish your data: Once you have your data loaded into internal database, you can publish data using API. API is generated automatically using Manifest tables and provides extracted data in many different formats. For example if original data source was a CSV file, now you have a flexible API, that can talk JSON, RDF, SQL, CSV and other formats.
Features
Simple 15 column table format for describing data structures (you can use any spreadsheet software to manage metadata of your data)
Internal data storage with pluggable backends (PostgreSQL or Mongo)
Build-in async API server built on top of Starlette for data publishing
Simple web based data browser.
Convenient command-line interface
Public or restricted API access via OAuth protocol using build-in access management.
Simple DSL for querying, transforming and validating data.
Low memory consumption for data of any size
Support for many different data sources
Advanced data extraction even from dynamic API.
Compatible with DCAT and Frictionless Data Specifications.
Example
If you have an SQLite database:
$ sqlite3 sqlite.db <<EOF
CREATE TABLE COUNTRY (
NAME TEXT
);
EOFYou can get a limited API and simple web based data browser with a single command:
$ spinta run -r sql sqlite:///sqlite.dbThen you can generate metadata table (I call it manifest) like this:
$ spinta inspect -r sql sqlite:///sqlite.db
d | r | b | m | property | type | ref | source | prepare | level | access | uri | title | description
dataset | | | | | | | | |
| sql | sql | | sqlite:///sqlite.db | | | | | |
| | | | | | | | |
| | | Country | | | COUNTRY | | | | | |
| | | | name | string | | NAME | | 3 | open | | |Generated data structure table can be saved into a CSV file:
$ spinta inspect -r sql sqlite:///sqlite.db -o manifest.csvMissing peaces in metadata can be filled using any Spreadsheet software.
Once you done editing metadata, you can test it via web based data browser or API:
$ spinta run --mode external manifest.csvOnce you are satisfied with metadata, you can generate a new metadata table for publishing, removing all traces of original data source:
$ spinta copy --no-source --access open manifest.csv manifest-public.csvNow you have matadata for publishing, but all things about original data source are gone. In order to publish data, you need to copy external data to internal data store. To do that, first you need to initialize internal data store:
$ spinta config add backend my_backend postgresql postgresql://localhost/db
$ spinta config add manifest my_manifest tabular manifest-public.csv
$ spinta migrateOnce internal database is initialized, you can push external data into it:
$ spinta push --access open manifest.csvAnd now you can publish data via full featured API with a web based data browser:
$ spinta runYou can access your data like this:
$ http :8000/dataset/sql/Country
HTTP/1.1 200 OK
content-type: application/json
{
"_data": [
{
"_type": "dataset/sql/Country",
"_id": "abdd1245-bbf9-4085-9366-f11c0f737c1d",
"_rev": "16dabe62-61e9-4549-a6bd-07cecfbc3508",
"_txn": "792a5029-63c9-4c07-995c-cbc063aaac2c",
"name": "Vilnius"
}
]
}
$ http :8000/dataset/sql/Country/abdd1245-bbf9-4085-9366-f11c0f737c1d
HTTP/1.1 200 OK
content-type: application/json
{
"_type": "dataset/sql/Country",
"_id": "abdd1245-bbf9-4085-9366-f11c0f737c1d",
"_rev": "16dabe62-61e9-4549-a6bd-07cecfbc3508",
"_txn": "792a5029-63c9-4c07-995c-cbc063aaac2c",
"name": "Vilnius"
}
$ http :8000/dataset/sql/Country/abdd1245-bbf9-4085-9366-f11c0f737c1d?select(name)
HTTP/1.1 200 OK
content-type: application/json
{
"name": "Vilnius"
}Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file spinta-0.1.57.tar.gz.
File metadata
- Download URL: spinta-0.1.57.tar.gz
- Upload date:
- Size: 286.5 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.6.1 CPython/3.11.5 Linux/5.15.131-1-MANJARO
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 3e029a2b0463b1c123b0fa561b5c4819eba43f1841513783bcca063c822dda51 |
|
| MD5 | b0814e6c29caeb2a5a397bf4fec40f46 |
|
| BLAKE2b-256 | 01b0956fb9932854a5355e9f66a9f9564c72fe949e9984e5889ec6a754abfa0a |
File details
Details for the file spinta-0.1.57-py3-none-any.whl.
File metadata
- Download URL: spinta-0.1.57-py3-none-any.whl
- Upload date:
- Size: 408.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.6.1 CPython/3.11.5 Linux/5.15.131-1-MANJARO
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 1f77a2b46f9bd1d62de091380bcc4740d9ad20f3c846aa87952d552851ad0b7e |
|
| MD5 | 691bdfa5ef642d40ed2b1eaf6d62104c |
|
| BLAKE2b-256 | 0a38feda9541269e4517b5084eefb6fe9292cc1ab0356c4dddbf3e5a2a179284 |
