Backend.AI Storage Proxy
Project description
Backend.AI Storage Proxy
Backend.AI Storage Proxy is an RPC daemon to manage vfolders used in Backend.AI agent, with quota and storage-specific optimization support.
Package Structure
ai.backend.storageserver: The agent daemon which communicates between Backend.AI Managerapi.client: The client-facing API to handle tus.io server-side protocol for uploads and ranged HTTP queries for downloads.api.manager: The manager-facing (internal) API to provide abstraction of volumes and separation of the hardware resources for volume and file operations.vfs- The minimal fallback backend which only uses the standard Linux filesystem interfaces
xfs- XFS-optimized backend with a small daemon to manage XFS project IDs for quota limits
agent: Implementation ofAbstractVolumeAgentwith XFS support
purestorage- PureStorage's FlashBlade-optimized backend with RapidFile Toolkit (formerly PureTools)
cephfs(TODO)- CephFS-optimized backend with quota limit support
Installation
Prequisites
- Python 3.8 or higher with pyenv and pyenv-virtualenv (optional but recommneded)
Installation Process
First, prepare the source clone of this agent:
# git clone https://github.com/lablup/backend.ai-storage-agent
From now on, let's assume all shell commands are executed inside the virtualenv.
Now install dependencies:
# pip install -U -r requirements/dist.txt # for deployment
# pip install -U -r requirements/dev.txt # for development
Then, copy halfstack.toml to root of the project folder and edit to match your machine:
# cp config/sample.toml storage-proxy.toml
When done, start storage server:
# python -m ai.backend.storage.server
It will start Storage Proxy daemon bound at 127.0.0.1:6021 (client API) and
127.0.0.1:6022 (manager API).
NOTE: Depending on the backend, the server may require to be run as root.
Production Deployment
To get performance boosts by using OS-provided sendfile() syscall
for file transfers, SSL termination should be handled by reverse-proxies
such as nginx and the storage proxy daemon itself should be run without SSL.
Filesystem Backends
VFS
Prerequisites
- User account permission to access for the given directory
XFS
Prerequisites
- Local device mounted under
/vfroot - Native support for XFS filesystem
- Access to root shell
Creating virtual XFS device for testing
Create a virtual block device mounted to lo (loopback) if you are the only one
to use the storage for testing:
- Create file with your desired size
# dd if=/dev/zero of=xfs_test.img bs=1G count=100
- Make file as XFS partition
# mkfs.xfs xfs_test.img
- Mount it to loopback
# export LODEVICE=$(losetup -f)
# losetup $LODEVICE xfs_test.img
- Create mount point and mount loopback device, with pquota option
# mkdir -p /vfroot/xfs
# mount -o loop -o pquota $LODEVICE /vfroot/xfs
PureStorage FlashBlade
Prerequisites
- NFSv3 export mounted under
/vfroot - Purity API access
CephFS
Prerequisites
- FUSE export mounted unde
/vfroot
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for backend.ai-storage-proxy-20.9.0a2.tar.gz
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 99a7c8f05fc1aae2292bf79ff07f4379ad29536e77b14d099424badaad0cd994 |
|
| MD5 | 8a6cdad4ea56fe76a32b48b09132ae1d |
|
| BLAKE2b-256 | ed7662899f74a3352901dbb6535913d37a1fd4b1e50a0a66f818d06f735d629a |
Hashes for backend.ai_storage_proxy-20.9.0a2-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 761b85d258f1a8d4f8977d1cf5af4ac72706f9db87223223fa6ff45387a1c351 |
|
| MD5 | 4ab2b760dc627b25e1fa49b1c1a51e4e |
|
| BLAKE2b-256 | 2fa53be90f19fa6fc92ae2a78ff751b631b3188784719a77a6a585b7e6868205 |
