Backend.AI Storage Proxy
Project description
Backend.AI Storage Proxy
Backend.AI Storage Proxy is an RPC daemon to manage vfolders used in Backend.AI agent, with quota and storage-specific optimization support.
Package Structure
ai.backend.storage
server
: The agent daemon which communicates between Backend.AI Managerapi.client
: The client-facing API to handle tus.io server-side protocol for uploads and ranged HTTP queries for downloads.api.manager
: The manager-facing (internal) API to provide abstraction of volumes and separation of the hardware resources for volume and file operations.vfs
- The minimal fallback backend which only uses the standard Linux filesystem interfaces
xfs
- XFS-optimized backend with a small daemon to manage XFS project IDs for quota limits
agent
: Implementation ofAbstractVolumeAgent
with XFS support
purestorage
- PureStorage's FlashBlade-optimized backend with RapidFile Toolkit (formerly PureTools)
cephfs
(TODO)- CephFS-optimized backend with quota limit support
Installation
Prequisites
- Python 3.8 or higher with pyenv and pyenv-virtualenv (optional but recommneded)
Installation Process
First, prepare the source clone of this agent:
# git clone https://github.com/lablup/backend.ai-storage-agent
From now on, let's assume all shell commands are executed inside the virtualenv.
Now install dependencies:
# pip install -U -r requirements/dist.txt # for deployment
# pip install -U -r requirements/dev.txt # for development
Then, copy halfstack.toml to root of the project folder and edit to match your machine:
# cp config/sample.toml storage-proxy.toml
When done, start storage server:
# python -m ai.backend.storage.server
It will start Storage Proxy daemon bound at 127.0.0.1:6021
(client API) and
127.0.0.1:6022
(manager API).
NOTE: Depending on the backend, the server may require to be run as root.
Production Deployment
To get performance boosts by using OS-provided sendfile()
syscall
for file transfers, SSL termination should be handled by reverse-proxies
such as nginx and the storage proxy daemon itself should be run without SSL.
Filesystem Backends
VFS
Prerequisites
- User account permission to access for the given directory
XFS
Prerequisites
- Local device mounted under
/vfroot
- Native support for XFS filesystem
- Access to root shell
Creating virtual XFS device for testing
Create a virtual block device mounted to lo
(loopback) if you are the only one
to use the storage for testing:
- Create file with your desired size
# dd if=/dev/zero of=xfs_test.img bs=1G count=100
- Make file as XFS partition
# mkfs.xfs xfs_test.img
- Mount it to loopback
# export LODEVICE=$(losetup -f)
# losetup $LODEVICE xfs_test.img
- Create mount point and mount loopback device, with pquota option
# mkdir -p /vfroot/xfs
# mount -o loop -o pquota $LODEVICE /vfroot/xfs
PureStorage FlashBlade
Prerequisites
- NFSv3 export mounted under
/vfroot
- Purity API access
CephFS
Prerequisites
- FUSE export mounted unde
/vfroot
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for backend.ai-storage-proxy-20.9.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3005ad8f0394a4282c37e90d2d00c9993e6eac7d88701c03d556f5a3e0de2cbc |
|
MD5 | 7d1153afd9da7dfed7a0599468662921 |
|
BLAKE2b-256 | acf968859f57bb84690f5dd69a53973eb8342ff19af9cef57bb87dd1caf70bf6 |
Hashes for backend.ai_storage_proxy-20.9.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 89709e59345f7895aeb11faf342fc36edc5f17d9f3c02a9aaa7caba9f70e5544 |
|
MD5 | 1c1ab5bbbbd85178553479ca6c823fed |
|
BLAKE2b-256 | 73069ef5a12db0da570488b94522d3248302b6c23c49f99e93255714b49b47b6 |