Time series forecasting benchmarking
Project description
TSBenchmarks
Time-series benchmarking a service. TSBenchmarks is an SDK for benchmarking models on public datasets.
Since we take care of the whole infrastructure, benchmarking becomes as easy as running a line in your python nootebooks or calling an API.
Why?
We build TSBenchmarks because we wanted a standarized solution for benchmarking time-series forecasting models. We evaluate provided forecasts on well known public datasets against benchmark models.
Table of contents
Available Benchmark Datasets
- M4-Daily
Available Benchmark models
- Naive1
- Naive2
- ETS
- Theta
- ARIMA
- MLP
- RNN
- ESRNN
- FFORMA
- NBEATS
Available Metrics and Plots
- MASE
- sMAPE
- Average loss by time-series
- Average loss by timestamp
Install
pip install tsbenchmarks
How to use
You can use TSBenchmarks by either using a completely public S3 bucket or by uploading a file to your own S3 bucket provided by us.
Data Format
Currently we only support .csv
files. These files must include at least 3 columns, with a unique_id (identifier of each time-series) a date stamp and a value. The unique_id and ds must coincide with the test set of the selected benchmark dataset.
1. Request free trial
Request a free trial sending an email to: fede.garza.ramirez@gmail.com and get your BUCKET_NAME
, API_ID
and API_KEY
, AWS_ACCESS_KEY_ID
, AWS_SECRET_ACCESS_KEY
.
2. Run tsbenchmarks
on a private S3 Bucket
If you don´t want other people to potentially have access to your data you can run tsbenchmarks
on a private S3 Bucket. For that you have to upload your data to a private S3 Bucket that we will provide for you; you can do this inside of python.
2.1 Upload to S3 from python
You will need the AWS_ACCESS_KEY_ID
and AWS_SECRET_ACCESS_KEY
that we provided.
- Import and Instantiate
TSBenchmarks
introducing yourBUCKET_NAME
,API_ID
andAPI_KEY
,AWS_ACCESS_KEY_ID
,AWS_SECRET_ACCESS_KEY
.
from tsbenchmarks.core import TSBenchmarks
tsbenchmarks = TSBenchmarks(bucket_name='<BUCKET_NAME>',
api_id='<API_ID>',
api_key='<API_KEY>',
aws_access_key_id='<AWS_ACCESS_KEY_ID>',
aws_secret_access_key='<AWS_SECRET_ACCESS_KEY>')
- Upload your local file introducing its name.
s3_uri = tsbenchmarks.upload_to_s3(file='<YOUR FILE NAME>')
- Run the evaluation process
To run the process specify:
s3_uri
: S3 uri provided after callingtsbenchmarks.upload_to_s3()
.dataset
: Name of datasetds_column
: Name of the unique id column.y_column
: Name of the target column.
#Run Evaluation
response_tmp_ft = tsbenchmarks.evaluate_my_model(
s3_uri="<PRIVATE S3 URI HERE>",
dataset='M4-Daily',
unique_id_column="<NAME OF ID COLUMN>",
ds_column= "<NAME OF DATESTAMP COLUMN>",
y_column="<NAME OF TARGET COLUMN>"
)
response_tmp_ft
status | message | id_job | dest_url | |
---|---|---|---|---|
0 | 200 | Check job status at GET /tsbenchmarks/jobs/{jo... | d8d2ae2f-ac53-4b81-87f5-49520782365a | s3://ts-benchmarks-api-public/M4-Daily-benchma... |
|
- Monitor the process with the following code. Once it's done, access to your bucket to download the generated features.
job_id = response_tmp_ft['id_job'].item()
tsbenchmarks.get_status(job_id)
status | processing_time_seconds | |
---|---|---|
0 | InProgress | 3 |
2.2 Download your results from s3
Once the process is done you can explore and download the results from s3.
ToDos
- Optimizing writing and reading speed with Parquet files
- Nan Handling
- Check data integrity before Upload
- Informative error messages
- Informative Status
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for tsbenchmarks-0.0.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a4e53aeb6c72c66520c11cda845df4c20a02bc815b0cdf2157d515bc3f5dde9c |
|
MD5 | acda043c08d516b4a5608b86eca50a4e |
|
BLAKE2b-256 | 1991c0daceab8d3547cee52f693fdbf7f4500aa9bbf0b7f663b792ecdf8d8074 |