Backend.AI Storage Proxy
Project description
Backend.AI Storage Proxy
Backend.AI Storage Proxy is an RPC daemon to manage vfolders used in Backend.AI agent, with quota and storage-specific optimization support.
Package Structure
ai.backend.storage
server
: The agent daemon which communicates between Backend.AI Managerapi.client
: The client-facing API to handle tus.io server-side protocol for uploads and ranged HTTP queries for downloads.api.manager
: The manager-facing (internal) API to provide abstraction of volumes and separation of the hardware resources for volume and file operations.vfs
- The minimal fallback backend which only uses the standard Linux filesystem interfaces
xfs
- XFS-optimized backend with a small daemon to manage XFS project IDs for quota limits
agent
: Implementation ofAbstractVolumeAgent
with XFS support
purestorage
- PureStorage's FlashBlade-optimized backend with RapidFile Toolkit (formerly PureTools)
cephfs
(TODO)- CephFS-optimized backend with quota limit support
Installation
Prequisites
- Python 3.8 or higher with pyenv and pyenv-virtualenv (optional but recommneded)
Installation Process
First, prepare the source clone of this agent:
# git clone https://github.com/lablup/backend.ai-storage-agent
From now on, let's assume all shell commands are executed inside the virtualenv.
Now install dependencies:
# pip install -U -r requirements/dist.txt # for deployment
# pip install -U -r requirements/dev.txt # for development
Then, copy halfstack.toml to root of the project folder and edit to match your machine:
# cp config/sample.toml storage-proxy.toml
When done, start storage server:
# python -m ai.backend.storage.server
It will start Storage Proxy daemon bound at 127.0.0.1:6021
(client API) and
127.0.0.1:6022
(manager API).
NOTE: Depending on the backend, the server may require to be run as root.
Filesystem Backends
VFS
Prerequisites
- User account permission to access for the given directory
XFS
Prerequisites
- Local device mounted under
/vfroot
- Native support for XFS filesystem
- Access to root shell
Creating virtual XFS device for testing
Create a virtual block device mounted to lo
(loopback) if you are the only one
to use the storage for testing:
- Create file with your desired size
# dd if=/dev/zero of=xfs_test.img bs=1G count=100
- Make file as XFS partition
# mkfs.xfs xfs_test.img
- Mount it to loopback
# export LODEVICE=$(losetup -f)
# losetup $LODEVICE xfs_test.img
- Create mount point and mount loopback device, with pquota option
# mkdir -p /vfroot/xfs
# mount -o loop -o pquota $LODEVICE /vfroot/xfs
PureStorage FlashBlade
Prerequisites
- NFSv3 export mounted under
/vfroot
- Purity API access
CephFS
Prerequisites
- FUSE export mounted unde
/vfroot
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for backend.ai-storage-proxy-20.9.0a1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 23c3f1f67c1d3e3094b6d9aec34e00136585a81cb5cc47c593046a5b62b68323 |
|
MD5 | 0364624938e5532b59ce0ffe2f1a6116 |
|
BLAKE2b-256 | 1ac9444bfeaa85d08a5371ce3fa2adab1ca1c3a4fa9a380b45a40847aa0deeea |
Hashes for backend.ai_storage_proxy-20.9.0a1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7767397c8238153400f86d155024a0ea094cd538b1335ad30dd37855d0b7d9c6 |
|
MD5 | cb3f28a087b74a930d09d534f748e18a |
|
BLAKE2b-256 | 05e61f2557bd68ea17799f9a7411bd62ee3c4739b6f7235a0c562657934f3c91 |