ProxyStore

ProxyStore is a library that facilitates efficient data management in distributed Python applications. At the core of ProxyStore is the proxy object which acts as a transparent reference to an object living in a global object store. This pass-by-reference interface with just-in-time object resolution works across processes, machines, and sites, and enables data producers to change the low-level communication method dynamically without altering application code or behavior.

ProxyStore accelerates the development of dynamic task-based workflows, serverless applications, and more by (1) decoupling control flow from data flow, (2) abstracting low-level communication mechanisms, and (3) eliminating the need for shims, wrapper functions, and boilerplate code.

ProxyStore supports a diverse set of programming paradigms:

• Task-based Workflows
• Function-as-a-Service/Serverless Applications
• Distributed Futures
• Bulk Data Streaming
• and more!

ProxyStore provides support for many third-party mediated communication methods out-of-the-box including DAOS, Globus Transfer, Kafka, KeyDB, and Redis. Custom communication methods built on Mochi, UCX, WebRTC, and ZeroMQ are provided for high-performance and peer-to-peer applications.

Read more about ProxyStore's concepts here. Complete documentation for ProxyStore is available at docs.proxystore.dev.

Installation

The base ProxyStore package can be installed with pip.

pip install proxystore

Many features require dependencies that are not installed by default but can be enabled via extras installation options such as endpoints, kafka, or redis. All optional dependencies can be installed with:

pip install proxystore[all]

This will also install the proxystore-ex package which contains extension and experimental features. The extensions package can also be installed with pip using proxystore[extensions] or proxystore-ex.

See the Installation guide for more information about the available extras installation options. See the Contributing guide to get started for local development.

Example

Getting started with ProxyStore requires a few lines of code.

from proxystore.connectors.redis import RedisConnector
from proxystore.proxy import Proxy
from proxystore.store import register_store
from proxystore.store import Store

store = Store('my-store', RedisConnector('localhost', 6379))

# Store the object and get a proxy. The proxy acts
# like a reference to the object.
data = MyDataType(...)
proxy = store.proxy(data)
assert isinstance(proxy, Proxy)

def my_function(x: MyDataType) -> ...:
    # x is resolved my-store on first use transparently to the
    # function. Then x behaves as an instance of MyDataType.
    assert isinstance(x, MyDataType)

my_function(proxy)  # Succeeds

Check out the Get Started guide to learn more!

Citation

If you use ProxyStore or any of this code in your work, please cite our SC '23 paper.

@inproceedings{pauloski2023proxystore,
    author = {Pauloski, J. Gregory and Hayot-Sasson, Valerie and Ward, Logan and Hudson, Nathaniel and Sabino, Charlie and Baughman, Matt and Chard, Kyle and Foster, Ian},
    title = {{Accelerating Communications in Federated Applications with Transparent Object Proxies}},
    address = {New York, NY, USA},
    articleno = {59},
    booktitle = {Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis},
    doi = {10.1145/3581784.3607047},
    isbn = {9798400701092},
    location = {Denver, CO, USA},
    numpages = {15},
    publisher = {Association for Computing Machinery},
    series = {SC '23},
    url = {https://doi.org/10.1145/3581784.3607047},
    year = {2023}
}