pandas-streaming 0.1.87

README

pandas_streaming aims at processing big files with pandas, too big to hold in memory, too small to be parallelized with a significant gain. The module replicates a subset of pandas API and implements other functionalities for machine learning.

from pandas_streaming.df import StreamingDataFrame
sdf = StreamingDataFrame.read_csv("filename", sep="\t", encoding="utf-8")

for df in sdf:
    # process this chunk of data
    # df is a dataframe
    print(df)

The module can also stream an existing dataframe.

import pandas
df = pandas.DataFrame([dict(cf=0, cint=0, cstr="0"),
                       dict(cf=1, cint=1, cstr="1"),
                       dict(cf=3, cint=3, cstr="3")])

from pandas_streaming.df import StreamingDataFrame
sdf = StreamingDataFrame.read_df(df)

for df in sdf:
    # process this chunk of data
    # df is a dataframe
    print(df)

Links:

• GitHub/pandas_streaming

• documentation

• Blog

current - 2018-05-17 - 0.00Mb

• 6: add pandas_groupby_nan from pyensae (2018-05-17)

0.1.66 - 2018-02-05 - 0.02Mb

• 5: add random_state parameter to splitting functions (2018-02-04)

• 2: add method sample, resevoir sampling (2017-11-05)

• 3: method train_test_split for out-of-memory datasets (2017-10-21)

• 1: Excited for your project (2017-10-10)