ProxyStore

ProxyStore provides pass-by-reference semantics for distributed Python applications through transparent object proxies. Moving data via proxies (1) decouples control flow from data flow, (2) enables producers to unilaterally (i.e., without the agreement of or awareness by the consumer) choose the best storage and communication channel for the data, and (3) perform just-in-time data movement directly between producer and consumer.

ProxyStore's goals are to:

• Improve productivity. ProxyStore enables easy decoupling of communication from the rest of the code, allowing developers to focus on functionality and performance.
• Improve compatibility. Consumers of data can be agnostic to the communication method because object proxies handle the communication behind the scenes.
• Improve performance. Transport methods and object stores can be changed at runtime to optimal choices for the given data without the consumers being aware of the change.

ProxyStore provides support for many third-party mediated communication methods out-of-the-box including Globus Transfer, KeyDB, and Redis. Custom communication methods built on Mochi, UCX, WebRTC, and ZeroMQ are provided for high-performance and multi-site applications.

Read more about ProxyStore's concepts here. Complete documentation for ProxyStore is available at docs.proxystore.dev.

Installation

$ pip install proxystore
$ pip install proxystore[all]

See the Installation guide for more information about the available extra install options. For local development, see the Contributing guide.

Extended features are available in the proxystore-ex package.

Example

Getting started with ProxyStore requires a few lines of code.

from proxystore.connectors.redis import RedisConnector
from proxystore.proxy import Proxy
from proxystore.store import register_store
from proxystore.store import Store

store = Store('my-store', RedisConnector('localhost', 6379))

# Store the object and get a proxy. The proxy acts
# like a reference to the object.
data = MyDataType(...)
proxy = store.proxy(data)
assert isinstance(proxy, Proxy)

def my_function(x: MyDataType) -> ...:
    # x is resolved my-store on first use transparently to the
    # function. Then x behaves as an instance of MyDataType.
    assert isinstance(x, MyDataType)

my_function(proxy)  # Succeeds

Check out the Get Started guide to learn more!

Citation

If you use ProxyStore or any of this code in your work, please cite our SC '23 paper.

@inproceedings{pauloski2023proxystore,
    author = {Pauloski, J. Gregory and Hayot-Sasson, Valerie and Ward, Logan and Hudson, Nathaniel and Sabino, Charlie and Baughman, Matt and Chard, Kyle and Foster, Ian},
    title = {{Accelerating Communications in Federated Applications with Transparent Object Proxies}},
    address = {New York, NY, USA},
    articleno = {59},
    booktitle = {Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis},
    doi = {10.1145/3581784.3607047},
    isbn = {9798400701092},
    location = {Denver, CO, USA},
    numpages = {15},
    publisher = {Association for Computing Machinery},
    series = {SC '23},
    url = {https://doi.org/10.1145/3581784.3607047},
    year = {2023}
}