Web Client for Visualizing Pandas Objects
Project description
Getting Started
Setup/Activate your environment and install the egg
Python 3
# create a virtualenv, if you haven't already created one
$ python3 -m venv ~/pyenvs/dtale
$ source ~/pyenvs/dtale/bin/activate
# install dtale egg (important to use the "--upgrade" every time you install so it will grab the latest version)
$ pip install --upgrade dtalePython 2
# create a virtualenv, if you haven't already created one
$ python -m virtualenv ~/pyenvs/dtale
$ source ~/pyenvs/dtale/bin/activate
# install dtale egg (important to use the "--upgrade" every time you install so it will grab the latest version)
$ pip install --upgrade dtaleNow you will have to ability to use D-Tale from the command-line or within a python-enabled terminal
Command-line
Loading data from arctic
dtale --arctic-host mongodb://localhost:27027 --arctic-library jdoe.my_lib --arctic-node my_node --arctic-start 20130101 --arctic-end 20161231Loading data from CSV
dtale --csv-path /home/jdoe/my_csv.csv --csv-parse_dates dateLoading data from a Custom loader - Using the DTALE_CLI_LOADERS environment variable, specify a path to a location containing some python modules - Any python module containing the global variables LOADER_KEY & LOADER_PROPS will be picked up as a custom loader - LOADER_KEY: the key that will be associated with your loader. By default you are given arctic & csv (if you use one of these are your key it will override these) - LOADER_PROPS: the individual props available to be specified. - For example, with arctic we have host, library, node, start & end. - If you leave this property as an empty list your loader will be treated as a flag. For example, instead of using all the arctic properties we would simply specify --arctic (this wouldn’t work well in arctic’s case since it depends on all those properties) - You will also need to specify a function with the following signature def find_loader(kwargs) which returns a function that returns a dataframe or None - Here is an example of a custom loader:
from dtale.cli.clickutils import get_loader_options
'''
IMPORTANT!!! This global variable is required for building any customized CLI loader.
When find loaders on startup it will search for any modules containing the global variable LOADER_KEY.
'''
LOADER_KEY = 'testdata'
LOADER_PROPS = ['rows', 'columns']
def test_data(rows, columns):
import pandas as pd
import numpy as np
import random
from past.utils import old_div
from pandas.tseries.offsets import Day
from dtale.utils import dict_merge
import string
now = pd.Timestamp(pd.Timestamp('now').date())
dates = pd.date_range(now - Day(364), now)
num_of_securities = old_div(rows, len(dates))
securities = [
dict(security_id=100000 + sec_id, int_val=random.randint(1, 100000000000),
str_val=random.choice(string.ascii_letters) * 5)
for sec_id in range(num_of_securities)
]
data = pd.concat([
pd.DataFrame([dict_merge(dict(date=date), sd) for sd in securities])
for date in dates
], ignore_index=True)[['date', 'security_id', 'int_val', 'str_val']]
col_names = ['Col{}'.format(c) for c in range(columns)]
return pd.concat([data, pd.DataFrame(np.random.randn(len(data), columns), columns=col_names)], axis=1)
# IMPORTANT!!! This function is required for building any customized CLI loader.
def find_loader(kwargs):
test_data_opts = get_loader_options(LOADER_KEY, kwargs)
if len([f for f in test_data_opts.values() if f]):
def _testdata_loader():
return test_data(int(test_data_opts.get('rows', 1000500)), int(test_data_opts.get('columns', 96)))
return _testdata_loader
return None
In this example we simplying building a dataframe with some dummy data based on dimensions specified on the command-line: - --testdata-rows - --testdata-columns
Here’s how you would use this loader:
DTALE_CLI_LOADERS=./path_to_loaders bash -c 'dtale --testdata-rows 10 --testdata-columns 5'Python Terminal
This comes courtesy of PyCharm 
UI
Once you have kicked off your D-Tale session please copy & paste the
link on the last line of output in your browser
The information in the upper right-hand corner is similar to saslook
Selecting/Deselecting Columns - to select a column, simply click on the
column header (to deselect, click the column header again) - You’ll
notice that the columns you’ve selected will display in the top of your
browser
Menu functions w/ no columns selected
Menu
Describe: view all the columns & their data types as well as individual details of each column
Data Type |
Display |
Notes |
|---|---|---|
date |
||
string |
If you have less than or equal to 100 unique values they will be displayed at the bottom of your popup |
|
int |
Anything with standard numeric classifications (min, max, 25%, 50%, 75%) will have a nice boxplot with the mean (if it exists) displayed as an outlier if you look closely. |
|
float |
Filter: apply a simple pandas query to your data (link to pandas documentation included in popup)
Editing |
Result |
|---|---|
Coverage: check for coverage gaps on column(s) by way of other column(s) as group(s)
Select column(s) in “Group(s)” & “Col(s)”
date-type columns you can also specify a frequency of D, W, M, Q, Y
Select multiple values in “Cols(s)” and/or “Groups(s)” by holdings the SHIFT key as you click
Click “Load”
The output will be the counts of non-nan records in “Col(s)” grouped by your selections in “Group(s)”
Daily |
Daily Regional |
|---|---|
Correlations: shows a pearson correlation matrix of all numeric columns against all other numeric columns
By deafult, it will show a grid of pearson correlations
If you have a date-type column, you can click an individual cell and see a timeseries of pearson correlations for that column combination
Currently if you have multiple date-type columns you will have the ability to toggle between them by way of a drop-down
Furthermore, you can click on individual points in the timeseries to view the scatter plot of the points going into that correlation
Matrix |
Timeseries |
Scatter |
|---|---|---|
About: This will give you information about what version of D-Tale you’re running as well as if its out of date to whats on PyPi.
Up To Date |
Out Of Date |
|---|---|
Resize: mostly a fail-safe in the event that your columns are no longer lining up. Click this and should fix that
Shutdown: pretty self-explanatory, kills your D-Tale session (there is also an auto-kill process that will kill your D-Tale after an hour of inactivity)
Menu functions w/ one column is selected
Menu Single-Column
Move To Front: moves your column to the front of the “unlocked” columns
Lock: adds your column to “locked” columns
“locked” means that if you scroll horizontally these columns will stay pinned to the right-hand side
this is handy when you want to keep track of which date or security_id you’re looking at
by default, any index columns on the data passed to D-Tale will be locked
Unlock: removed column from “locked” columns
Sorting (Ascending/Descending/Clear): applies/removes sorting to the column selected
Important: as you add sorts they sort added will be added to the end of the multi-sort. For example:
Action |
Sort |
|---|---|
select “a” |
|
sort asc |
a (asc) |
deselect “a” |
a (asc) |
select “b” |
a (asc) |
sort desc |
a (asc), b(desc) |
select “a” |
a (asc), b(desc) |
clear sort |
b(desc) |
sort desc |
b(desc), a(desc) |
select “b” |
b(desc), a(desc) |
clear sort |
|
sort asc |
a (asc), b(asc) |
Formats: apply simple formats to numeric values in your grid
|Editing|Result| |——–|:——:| |
Format |
Output |
|---|---|
Precision (6) |
-123456.789000 |
Thousands Sep |
-123,456.789 |
Abbreviate |
-123k |
Exponent |
-1e+5 |
BPS |
-1234567890BPS |
Red Negatives |
-123457 |
Histogram: display histograms in bins of 5, 10, 20 or 50 for any numeric column
For Developers
Getting Started
Clone the code (git clone ssh://git@github.com:manahl/dtale.git), then start the backend server:
$ git clone ssh://git@github.com:manahl/dtale.git
# install the dependencies
$ python setup.py develop
# start the server
$ python dtale --csv-path /home/jdoe/my_csv.csv --csv-parse_dates dateYou can also run dtale from PyDev directly.
You will also want to import javascript dependencies and build the source:
$ npm install
# 1) a persistent server that serves the latest JS:
$ npm run watch
# 2) or one-off build:
$ npm run buildRunning tests
The usual npm test command works:
$ npm test
You can run individual test files:
$ TEST=static/__tests__/dtale/DataViewer-base-test.jsx npm run test-file
Linting
You can lint all the JS and CSS to confirm there’s nothing obviously wrong with it:
$ npm run lint -sYou can also lint individual JS files:
$ npm run lint-js-file -s -- static/dtale/DataViewer.jsxFormatting JS
You can auto-format code as follows:
$ npm run formatDocker development
You can build python 27-3 & run D-Tale as follows:
$ yarn run build
$ docker-compose build dtale_2_7
$ docker run -it --network host dtale_2_7:latest
$ python
>>> import pandas as pd
>>> df = pd.DataFrame([dict(a=1,b=2,c=3)])
>>> import dtale
>>> dtale.show(df)Then view your D-Tale instance in your browser using the link that gets printed
You can build python 36-1 & run D-Tale as follows:
$ yarn run build
$ docker-compose build dtale_3_6
$ docker run -it --network host dtale_3_6:latest
$ python
>>> import pandas as pd
>>> df = pd.DataFrame([dict(a=1,b=2,c=3)])
>>> import dtale
>>> dtale.show(df)Then view your D-Tale instance in your browser using the link that gets printed
Documentation
Have a look at the detailed documentation.
Requirements
D-Tale works with:
Back-end
arctic
Flask
Flask-Caching
Flask-Compress
flasgger
Pandas
scipy
six
Front-end
react-virtualized
chart.js
Acknowledgements
D-Tale has been under active development at Man Numeric since 2019.
Original concept and implementation: Andrew Schonfeld
Contributors:
Youssef Habchi - title font
… and many others …
Contributions welcome!
License
D-Tale is licensed under the GNU LGPL v2.1. A copy of which is included in LICENSE
Changelog
1.0.0 (2019-09-06)
Initial public release
1.1.0 (2019-10-08)
IE support
Describe & About popups
Custom CLI support
1.1.1 (2019-10-23)
#13: fix for auto-detection of column widths for strings and floats
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
File details
Details for the file dtale-1.1.1.tar.gz.
File metadata
- Download URL: dtale-1.1.1.tar.gz
- Upload date:
- Size: 4.3 MB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/2.0.0 pkginfo/None requests/2.22.0 setuptools/39.0.1 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.6.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 512a21fad9993cafc992cafbbbd7f12ead533fbd6a7f06e9984a17dfc92d8c0a |
|
| MD5 | 4307a17ea5c4dec48af3ddd0e9439c06 |
|
| BLAKE2b-256 | e0dcf166c13924d69710e0f90119ff45ec16538f28cea2ddc42a8db548370aea |
File details
Details for the file dtale-1.1.1-py3.6.egg.
File metadata
- Download URL: dtale-1.1.1-py3.6.egg
- Upload date:
- Size: 4.4 MB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/2.0.0 pkginfo/None requests/2.22.0 setuptools/39.0.1 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.6.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | beeeec0fdc883029fb2cdcb00ee9be48a4749717c780d3587df4265c3e0a262e |
|
| MD5 | 5d7b54faf3dedf18d280d21bf5c8eb8e |
|
| BLAKE2b-256 | b279dbd5265fc2b27beca6c71e42890c90a07cd5f78b58acd599aa0302a817e3 |
File details
Details for the file dtale-1.1.1-py2.py3-none-any.whl.
File metadata
- Download URL: dtale-1.1.1-py2.py3-none-any.whl
- Upload date:
- Size: 4.4 MB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/2.0.0 pkginfo/None requests/2.22.0 setuptools/39.0.1 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.6.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 71f1d9365fdc94384a59dda694978d31b84717d2bc3d1e6ef9cc0a5eeafc309f |
|
| MD5 | 346609e27e6af80a177ca0523210204e |
|
| BLAKE2b-256 | d96c81a3e2d371072d79827e816d3a9d3a488f3572ed39b928cbdd32b7a7ada7 |
File details
Details for the file dtale-1.1.1-py2.7.egg.
File metadata
- Download URL: dtale-1.1.1-py2.7.egg
- Upload date:
- Size: 4.4 MB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.15.0 pkginfo/None requests/2.22.0 setuptools/28.8.0 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/2.7.13
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 38dd0a79a7c9b60fde881c6ae02f7a7bad523f05bddf4d3ff0b05e5d4aec943f |
|
| MD5 | c344474065f127723fc08cf014eb8df3 |
|
| BLAKE2b-256 | 7e94e65ed85a1a03cb45b8cc9dd34d5b97ccd5fd9e25947ced2a46d73e302e61 |
