Backend.AI Storage Proxy
Project description
Backend.AI Storage Proxy
Backend.AI Storage Proxy is an RPC daemon to manage vfolders used in Backend.AI agent, with quota and storage-specific optimization support.
Package Structure
ai.backend.storageserver: The agent daemon which communicates between Backend.AI Managerapi.client: The client-facing API to handle tus.io server-side protocol for uploads and ranged HTTP queries for downloads.api.manager: The manager-facing (internal) API to provide abstraction of volumes and separation of the hardware resources for volume and file operations.vfs- The minimal fallback backend which only uses the standard Linux filesystem interfaces
xfs- XFS-optimized backend with a small daemon to manage XFS project IDs for quota limits
agent: Implementation ofAbstractVolumeAgentwith XFS support
purestorage- PureStorage's FlashBlade-optimized backend with RapidFile Toolkit (formerly PureTools)
cephfs(TODO)- CephFS-optimized backend with quota limit support
Installation
Prequisites
- Python 3.8 or higher with pyenv and pyenv-virtualenv (optional but recommneded)
Installation Process
First, prepare the source clone of this agent:
# git clone https://github.com/lablup/backend.ai-storage-agent
From now on, let's assume all shell commands are executed inside the virtualenv.
Now install dependencies:
# pip install -U -r requirements/dist.txt # for deployment
# pip install -U -r requirements/dev.txt # for development
Then, copy halfstack.toml to root of the project folder and edit to match your machine:
# cp config/sample.toml storage-proxy.toml
When done, start storage server:
# python -m ai.backend.storage.server
It will start Storage Proxy daemon bound at 127.0.0.1:6021 (client API) and
127.0.0.1:6022 (manager API).
NOTE: Depending on the backend, the server may require to be run as root.
Production Deployment
To get performance boosts by using OS-provided sendfile() syscall
for file transfers, SSL termination should be handled by reverse-proxies
such as nginx and the storage proxy daemon itself should be run without SSL.
Filesystem Backends
VFS
Prerequisites
- User account permission to access for the given directory
XFS
Prerequisites
- Local device mounted under
/vfroot - Native support for XFS filesystem
- Access to root shell
Creating virtual XFS device for testing
Create a virtual block device mounted to lo (loopback) if you are the only one
to use the storage for testing:
- Create file with your desired size
# dd if=/dev/zero of=xfs_test.img bs=1G count=100
- Make file as XFS partition
# mkfs.xfs xfs_test.img
- Mount it to loopback
# export LODEVICE=$(losetup -f)
# losetup $LODEVICE xfs_test.img
- Create mount point and mount loopback device, with pquota option
# mkdir -p /vfroot/xfs
# mount -o loop -o pquota $LODEVICE /vfroot/xfs
PureStorage FlashBlade
Prerequisites
- NFSv3 export mounted under
/vfroot - Purity API access
CephFS
Prerequisites
- FUSE export mounted unde
/vfroot
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file backend.ai-storage-proxy-20.9.0rc1.tar.gz.
File metadata
- Download URL: backend.ai-storage-proxy-20.9.0rc1.tar.gz
- Upload date:
- Size: 25.0 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.2.0 pkginfo/1.6.0 requests/2.24.0 setuptools/51.0.0 requests-toolbelt/0.9.1 tqdm/4.50.2 CPython/3.8.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 34574ba02827ae5ae0c41f1f9f13833fa1c736b660d2043ed43987e82f11688e |
|
| MD5 | 8e50613422155de1c471840d12c72c1e |
|
| BLAKE2b-256 | 7c23242a96d02d937dca807143b137bf96522a0722587de1778ab54cb3b36949 |
Provenance
File details
Details for the file backend.ai_storage_proxy-20.9.0rc1-py3-none-any.whl.
File metadata
- Download URL: backend.ai_storage_proxy-20.9.0rc1-py3-none-any.whl
- Upload date:
- Size: 27.7 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.2.0 pkginfo/1.6.0 requests/2.24.0 setuptools/51.0.0 requests-toolbelt/0.9.1 tqdm/4.50.2 CPython/3.8.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 0e783da25a6c588091407fd023adf0c2bfd06a5c021e9322e3ff16c66b3dda0e |
|
| MD5 | b6c95eb05d74c4146730780383650f4d |
|
| BLAKE2b-256 | d7ab0bbc9489bd3295e78653bba8c6f2f007588281d139321367e4ec1f73edfc |
