A development environment management tool for data scientists.
Project description
Development environment for AI/ML
What is envd?
envd (ɪnˈvdɪ) is a command-line tool that helps you create the container-based development environment for AI/ML.
Development environments are full of Python and system dependencies, CUDA, BASH scripts, Dockerfiles, SSH configurations, Kubernetes YAMLs, and many other clunky things that are always breaking. envd is to solve the problem:
- Declare the list of dependencies (CUDA, Python packages, your favorite IDE, and so on) in
build.envd - Simply run
envd up. - Develop in the isolated environment.
Why use envd?
Environments built with envd provide the following features out-of-the-box:
❤️ Knowledge reuse in your team
envd build functions can be reused. Use include function to import any Git repositories. No more copy/paste Dockerfile instructions, let's reuse them.
envdlib = include("https://github.com/tensorchord/envdlib")
def build():
base(os="ubuntu20.04", language="python")
envdlib.tensorboard(host_port=8888)
envdlib.tensorboard is defined in github.com/tensorchord/envdlib
def tensorboard(
envd_port=6006,
envd_dir="/home/envd/logs",
host_port=0,
host_dir="/tmp",
):
"""Configure TensorBoard.
Make sure you have permission for `host_dir`
Args:
envd_port (Optional[int]): port used by envd container
envd_dir (Optional[str]): log storage mount path in the envd container
host_port (Optional[int]): port used by the host, if not specified or equals to 0,
envd will randomly choose a free port
host_dir (Optional[str]): log storage mount path in the host
"""
install.python_packages(["tensorboard"])
runtime.mount(host_path=host_dir, envd_path=envd_dir)
runtime.daemon(
commands=[
[
"tensorboard",
"--logdir",
envd_dir,
"--port",
str(envd_port),
"--host",
"0.0.0.0",
],
]
)
runtime.expose(envd_port=envd_port, host_port=host_port, service="tensorboard")
⏱️ BuildKit-native, build up to 6x faster
BuildKit supports parallel builds and software cache (e.g. pip index cache and apt cache). You can enjoy the benefits without knowledge of it.
For example, the PyPI cache is shared across builds and thus the package will be cached if it has been downloaded before.
Getting Started 🚀
Requirements
- Docker (20.10.0 or above)
Install and bootstrap envd
envd can be installed with pip, or you can download the binary release directly. After the installation, please run envd bootstrap to bootstrap.
pip3 install --upgrade envd
After the installation, please run envd bootstrap to bootstrap:
envd bootstrap
Read the documentation for more alternative installation methods.
You can add
--dockerhub-mirroror-mflag when runningenvd bootstrap, to configure the mirror for docker.io registry:envd bootstrap --dockerhub-mirror https://docker.mirrors.sjtug.sjtu.edu.cn
Create an envd environment
Please clone the envd-quick-start:
git clone https://github.com/tensorchord/envd-quick-start.git
The build manifest build.envd looks like:
def build():
base(os="ubuntu20.04", language="python3")
# Configure the pip index if needed.
# config.pip_index(url = "https://pypi.tuna.tsinghua.edu.cn/simple")
install.python_packages(name = [
"numpy",
])
shell("zsh")
Note that we use Python here as an example but please check out examples for other languages such as R and Julia here.
Then please run the command below to set up a new environment:
cd envd-quick-start && envd up
$ cd envd-quick-start && envd up
[+] ⌚ parse build.envd and download/cache dependencies 2.8s ✅ (finished)
=> download oh-my-zsh 2.8s
[+] 🐋 build envd environment 18.3s (25/25) ✅ (finished)
=> create apt source dir 0.0s
=> local://cache-dir 0.1s
=> => transferring cache-dir: 5.12MB 0.1s
...
=> pip install numpy 13.0s
=> copy /oh-my-zsh /home/envd/.oh-my-zsh 0.1s
=> mkfile /home/envd/install.sh 0.0s
=> install oh-my-zsh 0.1s
=> mkfile /home/envd/.zshrc 0.0s
=> install shell 0.0s
=> install PyPI packages 0.0s
=> merging all components into one 0.3s
=> => merging 0.3s
=> mkfile /home/envd/.gitconfig 0.0s
=> exporting to oci image format 2.4s
=> => exporting layers 2.0s
=> => exporting manifest sha256:7dbe9494d2a7a39af16d514b997a5a8f08b637f 0.0s
=> => exporting config sha256:1da06b907d53cf8a7312c138c3221e590dedc2717 0.0s
=> => sending tarball 0.4s
envd-quick-start via Py v3.9.13 via 🅒 envd
⬢ [envd]❯ # You are in the container-based environment!
Set up Jupyter notebook
Please edit the build.envd to enable jupyter notebook:
def build():
base(os="ubuntu20.04", language="python3")
# Configure the pip index if needed.
# config.pip_index(url = "https://pypi.tuna.tsinghua.edu.cn/simple")
install.python_packages(name = [
"numpy",
])
shell("zsh")
config.jupyter()
You can get the endpoint of the running Jupyter notebook via envd envs ls.
$ envd up --detach
$ envd envs ls
NAME JUPYTER SSH TARGET CONTEXT IMAGE GPU CUDA CUDNN STATUS CONTAINER ID
envd-quick-start http://localhost:42779 envd-quick-start.envd /home/gaocegege/code/envd-quick-start envd-quick-start:dev false <none> <none> Up 54 seconds bd3f6a729e94
More on documentation 📝
See envd documentation.
Roadmap 🗂️
Please checkout ROADMAP.
Contribute 😊
We welcome all kinds of contributions from the open-source community, individuals, and partners.
- Join our discord community!
- To build from the source, please read our contributing documentation and development tutorial.
Contributors ✨
Thanks goes to these wonderful people (emoji key):
This project follows the all-contributors specification. Contributions of any kind welcome!
Star History
License 📋
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
File details
Details for the file envd-0.3.5.tar.gz.
File metadata
- Download URL: envd-0.3.5.tar.gz
- Upload date:
- Size: 9.4 MB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.8.10
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 4daefca0a4404fde1c2cb76580674cdee3d353d6502278a2b2143e1633cbf4af |
|
| MD5 | 9820744de132ef4803836582b7bce608 |
|
| BLAKE2b-256 | 647f2af6edda65337d0179fb9ed996aec64e2a59c00bda9ea9c0a80d9e06423f |
Provenance
File details
Details for the file envd-0.3.5-py2.py3-none-musllinux_1_1_x86_64.whl.
File metadata
- Download URL: envd-0.3.5-py2.py3-none-musllinux_1_1_x86_64.whl
- Upload date:
- Size: 9.2 MB
- Tags: Python 2, Python 3, musllinux: musl 1.1+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.8.10
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 1016ffab962f69c549a8a089bad508019307bc7ed9aca641860e5bcd0650c6f4 |
|
| MD5 | d15b389342f3e08fa8cc9259cabb73a0 |
|
| BLAKE2b-256 | a673a317e8e5a35e7fba3b04bca6fcd33b1b854045ac8a36840bed4a6b7b79b3 |
Provenance
File details
Details for the file envd-0.3.5-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.
File metadata
- Download URL: envd-0.3.5-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Upload date:
- Size: 9.2 MB
- Tags: Python 2, Python 3, manylinux: glibc 2.17+ x86-64, manylinux: glibc 2.5+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.8.10
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 982f30a5e2fcddbb4539104c7302c39dadb7745d5206c9737f36fbb86a6c384f |
|
| MD5 | 51655881741a27a43b230ed87d4ab656 |
|
| BLAKE2b-256 | 4affe4f5467fedbb89581790ec9ff2a3466598ef18942ca44131e92a3e032b7e |
Provenance
File details
Details for the file envd-0.3.5-py2.py3-none-macosx_11_0_arm64.whl.
File metadata
- Download URL: envd-0.3.5-py2.py3-none-macosx_11_0_arm64.whl
- Upload date:
- Size: 18.9 MB
- Tags: Python 2, Python 3, macOS 11.0+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.15.0 pkginfo/1.8.3 requests/2.27.1 setuptools/41.2.0 requests-toolbelt/0.10.1 tqdm/4.64.1 CPython/2.7.18
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | f065349a0ac9055563fa267370341bf66a980789371cb2322bcb8b7d7c215763 |
|
| MD5 | 14a19e43f96e5b912005ddc2d58ed55d |
|
| BLAKE2b-256 | 119d56c96b6fbba3f0875750a5976ec87992cbc1065a1dd52da202dc35f0ad6c |
Provenance
File details
Details for the file envd-0.3.5-py2.py3-none-macosx_10_9_x86_64.whl.
File metadata
- Download URL: envd-0.3.5-py2.py3-none-macosx_10_9_x86_64.whl
- Upload date:
- Size: 18.9 MB
- Tags: Python 2, Python 3, macOS 10.9+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.15.0 pkginfo/1.8.3 requests/2.27.1 setuptools/41.2.0 requests-toolbelt/0.10.1 tqdm/4.64.1 CPython/2.7.18
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 8dd5b4fa6c3418933bcf6b079a5a1b362e8548df7bb0b3f4a359dbecc5d5cbcf |
|
| MD5 | a83e2f884c837f9d97cf7d859e857c7f |
|
| BLAKE2b-256 | a6418867f640fe4d62fdc3db932a94994cd20f4032ce4f7996dca7d22bbe18d8 |
