Backend.AI Storage Proxy
Project description
Backend.AI Storage Proxy
Backend.AI Storage Proxy is an RPC daemon to manage vfolders used in Backend.AI agent, with quota and storage-specific optimization support.
Package Structure
ai.backend.storageserver: The agent daemon which communicates between Backend.AI Managerapi.client: The client-facing API to handle tus.io server-side protocol for uploads and ranged HTTP queries for downloads.api.manager: The manager-facing (internal) API to provide abstraction of volumes and separation of the hardware resources for volume and file operations.vfs- The minimal fallback backend which only uses the standard Linux filesystem interfaces
xfs- XFS-optimized backend with a small daemon to manage XFS project IDs for quota limits
agent: Implementation ofAbstractVolumeAgentwith XFS support
purestorage- PureStorage's FlashBlade-optimized backend with RapidFile Toolkit (formerly PureTools)
cephfs(TODO)- CephFS-optimized backend with quota limit support
Installation
Prequisites
- Python 3.8 or higher with pyenv and pyenv-virtualenv (optional but recommneded)
Installation Process
First, prepare the source clone of this agent:
# git clone https://github.com/lablup/backend.ai-storage-agent
From now on, let's assume all shell commands are executed inside the virtualenv.
Now install dependencies:
# pip install -U -r requirements/dist.txt # for deployment
# pip install -U -r requirements/dev.txt # for development
Then, copy halfstack.toml to root of the project folder and edit to match your machine:
# cp config/sample.toml storage-proxy.toml
When done, start storage server:
# python -m ai.backend.storage.server
It will start Storage Proxy daemon bound at 127.0.0.1:6021 (client API) and
127.0.0.1:6022 (manager API).
NOTE: Depending on the backend, the server may require to be run as root.
Production Deployment
To get performance boosts by using OS-provided sendfile() syscall
for file transfers, SSL termination should be handled by reverse-proxies
such as nginx and the storage proxy daemon itself should be run without SSL.
Filesystem Backends
VFS
Prerequisites
- User account permission to access for the given directory
XFS
Prerequisites
- Local device mounted under
/vfroot - Native support for XFS filesystem
- Access to root shell
Creating virtual XFS device for testing
Create a virtual block device mounted to lo (loopback) if you are the only one
to use the storage for testing:
- Create file with your desired size
# dd if=/dev/zero of=xfs_test.img bs=1G count=100
- Make file as XFS partition
# mkfs.xfs xfs_test.img
- Mount it to loopback
# export LODEVICE=$(losetup -f)
# losetup $LODEVICE xfs_test.img
- Create mount point and mount loopback device, with pquota option
# mkdir -p /vfroot/xfs
# mount -o loop -o pquota $LODEVICE /vfroot/xfs
PureStorage FlashBlade
Prerequisites
- NFSv3 export mounted under
/vfroot - Purity API access
CephFS
Prerequisites
- FUSE export mounted unde
/vfroot
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for backend.ai-storage-proxy-21.3.0.tar.gz
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | dba36a746fc1fb3f686b02c13ae804113f19a7f25bb28ca6d1f7c6f604ad637c |
|
| MD5 | 6abf9ec2b2700cb0b7a50a7398d5412b |
|
| BLAKE2b-256 | 35e256e02e94e9a6307fa566cf98b53e098852a8215fe738cc555f9a5d69e015 |
Hashes for backend.ai_storage_proxy-21.3.0-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | fcaabb483991d9ba394532b9e8d73b09d23bbc18fcf845ca27c6ddba8b74993f |
|
| MD5 | 1faa00ffa7369b281554038285f4b584 |
|
| BLAKE2b-256 | 30bfa18fe0dfbf1ade25c0711404babaa30680b00fcadb935a6258aca67c16eb |
