pandas-streaming 0.2.175

pandas_streaming: streaming API over pandas

pandas_streaming aims at processing big files with pandas, too big to hold in memory, too small to be parallelized with a significant gain. The module replicates a subset of pandas API and implements other functionalities for machine learning.

from pandas_streaming.df import StreamingDataFrame
sdf = StreamingDataFrame.read_csv("filename", sep="\t", encoding="utf-8")

for df in sdf:
    # process this chunk of data
    # df is a dataframe
    print(df)

The module can also stream an existing dataframe.

import pandas
df = pandas.DataFrame([dict(cf=0, cint=0, cstr="0"),
                       dict(cf=1, cint=1, cstr="1"),
                       dict(cf=3, cint=3, cstr="3")])

from pandas_streaming.df import StreamingDataFrame
sdf = StreamingDataFrame.read_df(df)

for df in sdf:
    # process this chunk of data
    # df is a dataframe
    print(df)

It contains other helpers to split datasets into train and test with some weird constraints.

Links:

• GitHub/pandas_streaming

• documentation

• Blog

current - 2020-08-06 - 0.00Mb

0.0.0 - 2020-08-06 - 0.00Mb

• 16: Unit tests failing with pandas 1.1.0. (2020-08-06)

• 15: implements parameter lines, flatten for read_json (2018-11-21)

• 14: implements fillna (2018-10-29)

• 13: implement concat for axis=0,1 (2018-10-26)

• 12: add groupby_streaming (2018-10-26)

• 11: add method add_column (2018-10-26)

• 10: plan B to bypass a bug in pandas about read_csv when iterator=True –> closed, pandas has a weird behaviour when names is too small compare to the number of columns (2018-10-26)

• 9: head is very slow (2018-10-26)

• 8: fix pandas_streaming for pandas 0.23.1 (2018-07-31)

• 7: implement read_json (2018-05-17)

• 6: add pandas_groupby_nan from pyensae (2018-05-17)

• 5: add random_state parameter to splitting functions (2018-02-04)

• 2: add method sample, resevoir sampling (2017-11-05)

• 3: method train_test_split for out-of-memory datasets (2017-10-21)

• 1: Excited for your project (2017-10-10)