Backend.AI Storage Proxy
Project description
Backend.AI Storage Proxy
Backend.AI Storage Proxy is an RPC daemon to manage vfolders used in Backend.AI agent, with quota and storage-specific optimization support.
Package Structure
ai.backend.storageserver: The agent daemon which communicates between Backend.AI Managerapi.client: The client-facing API to handle tus.io server-side protocol for uploads and ranged HTTP queries for downloads.api.manager: The manager-facing (internal) API to provide abstraction of volumes and separation of the hardware resources for volume and file operations.vfs- The minimal fallback backend which only uses the standard Linux filesystem interfaces
xfs- XFS-optimized backend with a small daemon to manage XFS project IDs for quota limits
agent: Implementation ofAbstractVolumeAgentwith XFS support
purestorage- PureStorage's FlashBlade-optimized backend with RapidFile Toolkit (formerly PureTools)
cephfs(TODO)- CephFS-optimized backend with quota limit support
Installation
Prequisites
- Python 3.8 or higher with pyenv and pyenv-virtualenv (optional but recommneded)
Installation Process
First, prepare the source clone of this agent:
# git clone https://github.com/lablup/backend.ai-storage-agent
From now on, let's assume all shell commands are executed inside the virtualenv.
Now install dependencies:
# pip install -U -r requirements/dist.txt # for deployment
# pip install -U -r requirements/dev.txt # for development
Then, copy halfstack.toml to root of the project folder and edit to match your machine:
# cp config/sample.toml storage-proxy.toml
When done, start storage server:
# python -m ai.backend.storage.server
It will start Storage Proxy daemon bound at 127.0.0.1:6021 (client API) and
127.0.0.1:6022 (manager API).
NOTE: Depending on the backend, the server may require to be run as root.
Production Deployment
To get performance boosts by using OS-provided sendfile() syscall
for file transfers, SSL termination should be handled by reverse-proxies
such as nginx and the storage proxy daemon itself should be run without SSL.
Filesystem Backends
VFS
Prerequisites
- User account permission to access for the given directory
XFS
Prerequisites
- Local device mounted under
/vfroot - Native support for XFS filesystem
- Access to root shell
Creating virtual XFS device for testing
Create a virtual block device mounted to lo (loopback) if you are the only one
to use the storage for testing:
- Create file with your desired size
# dd if=/dev/zero of=xfs_test.img bs=1G count=100
- Make file as XFS partition
# mkfs.xfs xfs_test.img
- Mount it to loopback
# export LODEVICE=$(losetup -f)
# losetup $LODEVICE xfs_test.img
- Create mount point and mount loopback device, with pquota option
# mkdir -p /vfroot/xfs
# mount -o loop -o pquota $LODEVICE /vfroot/xfs
PureStorage FlashBlade
Prerequisites
- NFSv3 export mounted under
/vfroot - Purity API access
CephFS
Prerequisites
- FUSE export mounted unde
/vfroot
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for backend.ai-storage-proxy-20.9.2.tar.gz
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 4af8c4e8269d5a3a73383cf8f406e2551ea1227d5784daef4319bd7b8eca32d2 |
|
| MD5 | e4abb20981b9ebcbc037303f5897d59f |
|
| BLAKE2b-256 | 9201ebee333c701a8287cd9355c293326c4e553f116d944377d1d07c2d6d8d97 |
Hashes for backend.ai_storage_proxy-20.9.2-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 57db48671024a1dd2081d0a8e3f7e6fc7b41670f73bf2b55d6f81c3094251f58 |
|
| MD5 | 5198437bac291e5b6534be378aac9ae2 |
|
| BLAKE2b-256 | 68b4ba46ff524b99d0c72b6e24282f037778562f4a153a4f50e39d8aa3d07165 |
