django-data-browser

Name	Default	Docs Section	Function
DATA_BROWSER_ALLOW_PUBLIC	False	Security	Allow selected saved views to be accessed without admin login in limited circumstances.
DATA_BROWSER_AUTH_USER_COMPAT	True	Performance	When calling get_fieldsets on a UserAdmin alwyas pass an instance of the associated model.
DATA_BROWSER_DEFAULT_ROW_LIMIT	1000		The default value for the row limit selector in the UI.
DATA_BROWSER_DEV	False	Development	Enable proxying frontend to JS dev server.
DATA_BROWSER_FE_DSN	None	Sentry	The DSN the frontend sentry should report to, disabled by default.

Most of the Django views in the Data Browser can only be accessed by Django “staff members”. These views support general querying of the database, checked against the admin permissions of the logged in user.

The only exception to this is “Public Saved Views” these are views which have been saved and marked as public. They can be accessed by anyone without needing a login but they can only be used to access a query that has been saved and made public and they have long random URL’s.

You can use the admin permission data_browser | view | Can make a saved view publically available to restrict who can make views public. To be public the view must be marked as public and owned by someone who has the permission.

Additionally the entire public views system is gated by the Django settings value DATA_BROWSER_ALLOW_PUBLIC.

Sentry

The frontend code has builtin Sentry support, it is disabled by default. To enable it set the Django settings value DATA_BROWSER_FE_DSN, for example to set it to the Data Browser project Sentry use:

DATA_BROWSER_FE_DSN = "https://af64f22b81994a0e93b82a32add8cb2b@o390136.ingest.sentry.io/5231151"

Customization and Performance

get_queryset

The Data Browser does it’s fetching in two stages.

First it does a single DB query to get the majority of the data. To construct the queryset for this it will call get_queryset on the ModelAdmin of the current Model. It uses .values() to fetch only the data it needs from the database and it will inline all referenced models to ensure it doesn’t do multiple queries.

Secondly for any calculated fields it will then fetch the complete objects that are needed for those calculated fields. To construct the querysets for these it will call get_queryset on their associated ModelAdmins. These calls are aggregated so it will only make one per model.

As a simple example. If you did a query against the Book model for the fields:

book.name
book.author.name
book.author.age
book.publisher.name

Where the author.age is actually a property on the Author Model then it would do the following two queries:

BookAdmin.get_queryset().values("name", "author__name", "author__id", "publisher__name")
AuthorAdmin.get_queryset().in_bulk(pks=...)

Where the pks passed to in_bulk in the second query came from author__id in the first.

When the Data Browser calls the admin get_queryset functions it will put some context in request.databrowser. This means you can test to see if the Data Browser is making the call as follows:

if request.databrowser:
    # Data Browser specific customization

This is particularly useful if you want to route the Data Browser to a DB replica.

The context includes a calculated_fields member that is set when doing the second stage requests for calculated fields. You can use this to do conditional prefetching or annotating to support those fields like this:

if not hasattr(request, "databrowser") or "my_field" in request.databrowser[``calculated_fields``]:
        # do prefetching and annotating associated with my_field

get_fieldsets

The Data Browser also calls get_fieldsets to find out what fields the current user can access.

As with get_queryset the Data Browser will set request.databrowser when calling get_fieldsets and you can test this to detect it and make Data Browser specific customizations.

The Django User Admin has code to change the fieldsets when adding a new user. To compensate for this, when calling get_fieldsets on a subclass of django.contrib.auth.admin.UserAdmin the Data Browser will pass a newly constructed instance of the relevant model. This behavior can be disabled by setting settings.DATA_BROWSER_AUTH_USER_COMPAT to False.

URL Format

The query URL format is query/<model>/<fields>.<format>?<filters>.

Model is a Django app and model name for example library.Book

Fields are a series of comma separated fields, where each field is the path to that field from the model with the parts separated by __, e.g. author__name. This path structure also includes aggregates and functions e.g. author__birthday__month__count. Fields can be pivoted (where appropriate) by prefixing them with &. And sorted by suffixing with a direction +/- and a priority e.g. author__birthday+1.

Filters use the same __ path format as fields including a lookup e.g. author__name__contains=Joe. Since filters always have a field and a lookup they always contain at least one __. Filters that don’t contain a __ are reserved, at the time of writing the only such filter is the row limit filter limit.

Format determines the returned data format, the currently available formats are:

Format	Details
html	Load the interactive Javascript frontend.
csv	Standard CSV format.
json	Standard JSON format, the JS frontend uses this for all data access.
ctx	See the JSON encoded config passed to the JS on page load.
query	See the parsed URL in JSON format, the JS frontend uses this to boostrap.

Development

The easiest way to develop this is against your existing client project.

The compiled Javascript is checked into the repo, so if only want to mess with the Python then it’s sufficient to:

Install the Data Browser in editable mode pip install -e <directory to your git clone>.

If you want to modify the Javascript then you also need to:

Enable proxying to the JS dev server by adding DATA_BROWSER_DEV = True to your settings.
Run the Javascript dev server with WDS_SOCKET_PORT=3000 PUBLIC_URL=data_browser npm start. The WDS_SOCKET_PORT is so the proxied JS can find it’s dev server. The PUBLIC_URL tells the JS dev server what path to serve from and should be the same as the URL you have mounted the Data Browser on in your urls file.

To run the Python tests, in the top level of your git clone run pip install -r requirements.txt then pytest.

There is also pre-commit config for lint etc to enable this run pip install pre-commit && pre-commit install then lint will run on git commit. The linting includes Black and isort autoformatting.

To build the JS, move the files around appropriately and recreate the wheels run build.sh.

During development it can be useful to look at the .ctx and .json views. The .ctx view will show you the initial context being passed to the Javascript on page load. The .json view is the actual API request the Javascript uses to fetch query results.

Structure

Terminology

Term	Meaning
aggregate	Corresponds to a Django aggregation function.
bound query	A query that has been validated against the config.
calculated field	A field that can not be sorted or filtered, generally a field whose value comes from a property or function on the Admin or Model.
concrete field	A field that can be sorted and filtered, generally anything that came directly from the ORM.
config	Information that doesn’t change based on the particular query, includes all the models and their fields.
field name	Just the name of the field e.g. created_time.
field path	Includes information on how to reach the model the field is on e.g. ["order","seller","created_time"].
function	Corresponds to a Django database function for transforming a value, e.g. ExtractYear.
model name	Fullstop separated app and model names e.g. myapp.MyModel, also includes synthetic ‘models’ for hosting aggregate and function fields.
model path	Like field path for the model the field is on.
model	In Python the actual model class, in Javascript the model name as above.
pretty…	User friendly field, and path values
query	The information that changes with the query being done, in the Javascript this also includes the results.
type	A data type, like string or number
view	A saved query.

Most of the code deals with “models” that have “fields” that have “types”. In this context a “model” is just anything which might have fields. An important consequence of this is that most types also have associated models which hold that types aggregate and function fields. The special meanings of foreignkeys, aggregates, functions and calculated fields is confined to orm.py and orm_fields.py.

Fields have 5 main properties.

Property	Meaning and impact
name	The only required one.
type	If set then this field can be added to a query and will return results of the specified type.
concrete	Can this field be sorted and filtered. Requires type to be set.
can_pivot	The field goes on the outside of a pivot table and as such can be pivoted.
model	If set then this field has additional nested fields that are detailed on the given model.

Version numbers

The Data Browser uses the standard Major.Minor.Patch version numbering scheme.

Patch versions may include bug fixes and minor features.

Minor versions are for significant new features.

Major versions are for major features, significant changes to existing functionality and breaking changes.

Patch and Minor versions should never contain breaking changes and should always be backward compatible. A breaking change is a change that makes backward incompatible changes to one or more of the following:

The query URL format.
The json, csv etc data formats.
request.databrowser.
Invalidates saved views.
Changes the URL’s of public saved views.

Release History

Version	Date	Summary
2.1.1	2020-07-06	Bug fixes. The representation of empty pivot cells has changed in the JSON.
2.1.0	2020-07-06	Bring views into the JS frontend. Implement row limits on results. All existing saved views will be limited to 1000 rows. Better loading and error status indication. Lock column headers.
2.0.5	2020-06-20	Bug fixes.
2.0.4	2020-06-18	Fix Py3.6 support.
2.0.3	2020-06-14	Improve filtering on aggregates when pivoted.
2.0.2	2020-06-14	Improve fonts and symbols.
2.0.1	2020-06-14	Improve sorting when pivoted.
2.0.0	2020-06-14	Pivot tables. All public view URL’s have changed. The JSON data format has changed.
1.2.6	2020-06-08	Bug fixes.
1.2.5	2020-06-08	Bug fixes.
1.2.4	2020-06-03	Calculated fields interact better with aggregation.
1.2.3	2020-06-02	JS error handling tweaks.
1.2.2	2020-06-01	Minor fix.
1.2.1	2020-05-31	Improved date handling.
1.2.0	2020-05-31	Support for date functions “year”, “month” etc and filtering based on “now”.
1.1.6	2020-05-24	Stronger sanitizing of URL strings.
1.1.5	2020-05-23	Fix bug aggregating time fields.
1.1.4	2020-05-23	Fix breaking bug with GenericInlineModelAdmin.
1.1.3	2020-05-23	Cosmetic fixes.
1.1.2	2020-05-22	Cosmetic fixes.
1.1.1	2020-05-20	Cosmetic fixes.
1.1.0	2020-05-20	Aggregate support.
1.0.2	2020-05-17	Py3.6 support.
1.0.1	2020-05-17	Small fixes.
1.0.0	2020-05-17	Initial version.

Project details

These details have not been verified by PyPI

Project links

Homepage

Release history Release notifications | RSS feed

4.2.10

Apr 9, 2024

4.2.9

Apr 2, 2024

4.2.8

Feb 11, 2024

4.2.7

Feb 4, 2024

4.2.6

Feb 4, 2024

4.2.5

Jul 20, 2023

4.2.4

Jul 2, 2023

4.2.3

Jun 15, 2023

4.2.2

Jun 8, 2023

4.2.1

May 21, 2023

4.1.1

May 7, 2023

4.1.0

May 1, 2023

4.0.17

Apr 25, 2023

4.0.16

Apr 16, 2023

4.0.15

Apr 11, 2023

4.0.14

Apr 5, 2023

4.0.13

Mar 6, 2023

4.0.12

May 3, 2022

4.0.11

Apr 12, 2022

4.0.10

Apr 12, 2022

4.0.9

Jan 4, 2022

4.0.8

Dec 12, 2021

4.0.7

Aug 16, 2021

4.0.6

Aug 10, 2021

4.0.5

Aug 9, 2021

4.0.4

Jul 4, 2021

4.0.3

Jun 18, 2021

4.0.2

Apr 12, 2021

4.0.1

Apr 11, 2021

4.0.0

Mar 13, 2021

3.3.0

Feb 19, 2021

3.2.5

Feb 7, 2021

3.2.4

Feb 2, 2021

3.2.3

Jan 11, 2021

3.2.2

Dec 30, 2020

3.2.1

Dec 30, 2020

3.2.0

Dec 30, 2020

3.1.4

Dec 19, 2020

3.1.3

Dec 13, 2020

3.1.2

Dec 9, 2020

3.1.1

Dec 1, 2020

3.1.0

Nov 29, 2020

3.0.4

Nov 28, 2020

3.0.3

Nov 22, 2020

3.0.2

Nov 18, 2020

3.0.1

Nov 12, 2020

3.0.0

Nov 9, 2020

2.2.21

Nov 2, 2020

2.2.20

Oct 22, 2020

2.2.19

Oct 19, 2020

2.2.18

Oct 18, 2020

2.2.17

Oct 15, 2020

2.2.16

Sep 28, 2020

2.2.15

Sep 27, 2020

2.2.14

Sep 20, 2020

2.2.13

Sep 13, 2020

2.2.12

Sep 9, 2020

2.2.11

Sep 6, 2020

2.2.10

Aug 31, 2020

2.2.9

Aug 25, 2020

2.2.8

Aug 23, 2020

2.2.7

Aug 22, 2020

2.2.6

Aug 16, 2020

2.2.5

Aug 1, 2020

2.2.4

Aug 1, 2020

2.2.3

Jul 31, 2020

2.2.2

Jul 26, 2020

2.2.1

Jul 25, 2020

2.2.0

Jul 21, 2020

2.1.2

Jul 11, 2020

This version

2.1.1

Jul 6, 2020

2.1.0

Jul 6, 2020

2.0.5

Jun 20, 2020

2.0.4

Jun 18, 2020

2.0.3

Jun 15, 2020

2.0.2

Jun 14, 2020

2.0.1

Jun 14, 2020

2.0.0

Jun 14, 2020

1.2.6

Jun 8, 2020

1.2.5

Jun 8, 2020

1.2.4

Jun 3, 2020

1.2.3

Jun 2, 2020

1.2.2

Jun 1, 2020

1.2.1

May 31, 2020

1.2.0

May 31, 2020

1.1.6

May 24, 2020

1.1.5

May 23, 2020

1.1.4

May 23, 2020

1.1.3

May 23, 2020

1.1.2

May 22, 2020

1.1.1

May 20, 2020

1.1.0

May 20, 2020

1.0.2

May 17, 2020

1.0.1

May 17, 2020

1.0.0

May 17, 2020

0.0.9

May 10, 2020

0.0.8

May 9, 2020

0.0.7

May 9, 2020

0.0.6

May 9, 2020

0.0.5

May 9, 2020

0.0.4

May 9, 2020

0.0.3

Apr 25, 2020

0.0.2

Apr 22, 2020

0.0.1

Apr 14, 2020

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

django-data-browser-2.1.1.tar.gz (414.7 kB view details)

Uploaded Jul 6, 2020 Source

Built Distribution

django_data_browser-2.1.1-py3-none-any.whl (412.6 kB view details)

Uploaded Jul 6, 2020 Python 3

File details

Details for the file django-data-browser-2.1.1.tar.gz.

File metadata

Download URL: django-data-browser-2.1.1.tar.gz
Upload date: Jul 6, 2020
Size: 414.7 kB
Tags: Source
Uploaded using Trusted Publishing? No
Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/49.1.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.0

File hashes

Hashes for django-data-browser-2.1.1.tar.gz
Algorithm	Hash digest
SHA256	`78d031f3beab00261070afda54cf620a2eb5905f36fec51bd3c96bc7f5cd0492`
MD5	`65dafe41e9c41de27656d466b4e3e641`
BLAKE2b-256	`90fcf362388fb84c6b9e8b5c2ed0607ca1e24461196684982c8bb4a515a9ecd5`

See more details on using hashes here.

File details

Details for the file django_data_browser-2.1.1-py3-none-any.whl.

File metadata

Download URL: django_data_browser-2.1.1-py3-none-any.whl
Upload date: Jul 6, 2020
Size: 412.6 kB
Tags: Python 3
Uploaded using Trusted Publishing? No
Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.23.0 setuptools/49.1.0 requests-toolbelt/0.9.1 tqdm/4.45.0 CPython/3.8.0

File hashes

Hashes for django_data_browser-2.1.1-py3-none-any.whl
Algorithm	Hash digest
SHA256	`c362e487055a30c54522e26e850862ad84f9d1976a2299c2cab3fddab2107013`
MD5	`8eaaffa93170f71a77f9ad52aa7933e1`
BLAKE2b-256	`297f88da1587a9bf87d0417816094d5c5c61fd59dbe4c74e4766ef38c7b20b91`