Treasure Data API library for Python
Project description
Treasure Data API library for Python
Treasure Data API library for Python
Requirements
td-client supports the following versions of Python.
- Python 2.7+
- Python 3.4+
- PyPy
Install
You can install the releases from PyPI.
$ pip install td-client
It'd be better to install certifi to enable SSL certificate verification.
$ pip install certifi
Examples
Please see also the examples at Treasure Data Documentation.
Listing jobs
Treasure Data API key will be read from environment variable TD_API_KEY, if none is given via apikey= argument passed to tdclient.Client.
Treasure Data API endpoint https://api.treasuredata.com is used by default. You can override this with environment variable TD_API_SERVER, which in turn can be overridden via endpoint= argument passed to tdclient.Client. List of available Treasure Data sites and corresponding API endpoints can be found here.
import tdclient
with tdclient.Client() as td:
for job in td.jobs():
print(job.job_id)
Running jobs
Running jobs on Treasure Data.
import tdclient
with tdclient.Client() as td:
job = td.query("sample_datasets", "SELECT COUNT(1) FROM www_access", type="hive")
job.wait()
for row in job.result():
print(repr(row))
Running jobs via DBAPI2
td-client-python implements PEP 0249 Python Database API v2.0. You can use td-client-python with external libraries which supports Database API such like pandas.
import pandas
import tdclient
def on_waiting(cursor):
print(cursor.job_status())
with tdclient.connect(db="sample_datasets", type="presto", wait_callback=on_waiting) as td:
data = pandas.read_sql("SELECT symbol, COUNT(1) AS c FROM nasdaq GROUP BY symbol", td)
print(repr(data))
We offer another package for pandas named pandas-td with some advanced features. You may prefer it if you need to do complicated things, such like exporting result data to Treasure Data, printing job's progress during long execution, etc.
Importing data
Importing data into Treasure Data in streaming manner, as similar as fluentd is doing.
import sys
import tdclient
with tdclient.Client() as td:
for file_name in sys.argv[:1]:
td.import_file("mydb", "mytbl", "csv", file_name)
Bulk import
Importing data into Treasure Data in batch manner.
from __future__ import print_function
import sys
import tdclient
import time
import warnings
if len(sys.argv) <= 1:
sys.exit(0)
with tdclient.Client() as td:
session_name = "session-%d" % (int(time.time()),)
bulk_import = td.create_bulk_import(session_name, "mydb", "mytbl")
try:
for file_name in sys.argv[1:]:
part_name = "part-%s" % (file_name,)
bulk_import.upload_file(part_name, "json", file_name)
bulk_import.freeze()
except:
bulk_import.delete()
raise
bulk_import.perform(wait=True)
if 0 < bulk_import.error_records:
warnings.warn("detected %d error records." % (bulk_import.error_records,))
if 0 < bulk_import.valid_records:
print("imported %d records." % (bulk_import.valid_records,))
else:
raise(RuntimeError("no records have been imported: %s" % (repr(bulk_import.name),)))
bulk_import.commit(wait=True)
bulk_import.delete()
Development
Running tests
Run tests.
$ python setup.py test
Running tests (tox)
You can run tests against all supported Python versions. I'd recommend you to install pyenv to manage Pythons.
$ pyenv shell system
$ for version in $(cat .python-version); do [ -d "$(pyenv root)/versions/${version}" ] || pyenv install "${version}"; done
$ pyenv shell --unset
Install tox.
$ pip install tox
Then, run tox.
$ tox
Release
Release to PyPI.
$ python setup.py bdist_wheel --universal sdist upload
Version History
See CHANGELOG.md.
License
Apache Software License, Version 2.0
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file td-client-0.14.0.tar.gz.
File metadata
- Download URL: td-client-0.14.0.tar.gz
- Upload date:
- Size: 48.1 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.21.0 setuptools/39.0.1 requests-toolbelt/0.9.1 tqdm/4.32.2 CPython/3.6.6
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | bd57e1605b8ca40aa77c9404ccd9f50e3d7885fa95aca7e34f30a0cef68bf46b |
|
| MD5 | 306313a655ce10df08b51164ee7df39a |
|
| BLAKE2b-256 | 273e952447dadc4451bad78d62147fd9e61ded4db39050059c91b5df06788e72 |
File details
Details for the file td_client-0.14.0-py2.py3-none-any.whl.
File metadata
- Download URL: td_client-0.14.0-py2.py3-none-any.whl
- Upload date:
- Size: 73.3 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.21.0 setuptools/39.0.1 requests-toolbelt/0.9.1 tqdm/4.32.2 CPython/3.6.6
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 7884703bb63e7a232c836e83439a41277bb49519c06558b64c3b318aae570b38 |
|
| MD5 | 94f6fe7b0b4d5cdfdd440cd676653750 |
|
| BLAKE2b-256 | 4753c3259c70e56563cebd36661dfdbd769f331f8d0f579964a2d9a4a26dd1e0 |
