tchotcho
Project description
TchoTcho
(tchotcho)OOOOOoo...
_____ oo
_______ ||_||__n_n__I_
|__T__|-|_T_|_________)>
oo oo o ()() ()() o\
Train on EC2 when you are ready with experimenting!
This tool contains bundle of methods to setup a EC2 instance for training.
Install
❯ pip install tchotcho
Requirements
- Setup a private/ public key (login to EC2, helpful for cloning private repo)
- Install rsync (copy project to EC2)
- Install ssh (execute commands remote on EC2)
- Setup your aws credentials profile via (
~/.aws/credentials
)
Actions
Available commands:
❯ tchotcho --help
Usage: tchotcho [OPTIONS] COMMAND [ARGS]...
Options:
--help Show this message and exit.
Commands:
info
key
shell
spot
stack
Info
Update
Retrieves list of EC2 instances from https://ec2instances.info/ and queries aws to find suitable ami imges for training. It caches the result locally for later usage.
Help:
❯ AWS_PROFILE=dev tchotcho info update --help
Usage: tchotcho info update [OPTIONS]
Update the GPU info json file
Options:
--ownerid TEXT Owner id used (we use ubuntu) [default: 898082745236;
required]
--namefilter TEXT AMI filter by name [default: Deep Learning AMI* 18.04*;
required]
--limit INTEGER Number of AMI image [default: 5; required]
--csv / --no-csv
--region TEXT List in region [required]
--help Show this message and exit.
Usage:
❯ AWS_PROFILE=dev env/bin/tchotcho info update
╒═══════════════╤═══════╤══════════════╤═══════════════════════╤═══════╤═══════════╤═════════════╤═══════════════╕
│ name │ gpu │ gpu_memory │ gpu_model │ cpu │ memory │ supported │ price │
╞═══════════════╪═══════╪══════════════╪═══════════════════════╪═══════╪═══════════╪═════════════╪═══════════════╡
│ p2.16xlarge │ 16 │ 192 │ NVIDIA Tesla K80 │ 64 │ 732 │ True │ 21.216 │
├───────────────┼───────┼──────────────┼───────────────────────┼───────┼───────────┼─────────────┼───────────────┤
│ p3dn.24xlarge │ 8 │ 256 │ NVIDIA Tesla V100 │ 96 │ 768 │ True │ Not available │
├───────────────┼───────┼──────────────┼───────────────────────┼───────┼───────────┼─────────────┼───────────────┤
│ p2.8xlarge │ 8 │ 96 │ NVIDIA Tesla K80 │ 32 │ 488 │ True │ 10.608 │
...
╒══════════════╤═══════════════════════════════════════════════╤══════════════════════════╤═══════════════════════╕
│ region │ name │ date │ ami │
╞══════════════╪═══════════════════════════════════════════════╪══════════════════════════╪═══════════════════════╡
│ eu-central-1 │ Deep Learning AMI (Ubuntu 18.04) Version 29.0 │ 2020-05-20T14:57:39.000Z │ ami-062a3145bcf312c71 │
├──────────────┼───────────────────────────────────────────────┼──────────────────────────┼───────────────────────┤
│ eu-central-1 │ Deep Learning AMI (Ubuntu 18.04) Version 28.1 │ 2020-05-03T19:46:44.000Z │ ami-061aaaac62de85935 │
├──────────────┼───────────────────────────────────────────────┼──────────────────────────┼───────────────────────┤
│ eu-central-1 │ Deep Learning AMI (Ubuntu 18.04) Version 28.0 │ 2020-04-29T09:32:42.000Z │ ami-0f162c7e9b0e7d6f1 │
├──────────────┼───────────────────────────────────────────────┼──────────────────────────┼───────────────────────┤
│ eu-central-1 │ Deep Learning AMI (Ubuntu 18.04) Version 27.0 │ 2020-03-03T03:13:27.000Z │ ami-09633db638556dc39 │
├──────────────┼───────────────────────────────────────────────┼──────────────────────────┼───────────────────────┤
│ eu-central-1 │ Deep Learning AMI (Ubuntu 18.04) Version 26.0 │ 2019-12-02T22:09:33.000Z │ ami-0dcdcc4bc9e75005f │
╘══════════════╧═══════════════════════════════════════════════╧══════════════════════════╧═══════════════════════╛
List
Use the cached result from update
command from ~/.tchotcho/gpu_info.json
.
Key
Function to manage keys on EC2.
❯ AWS_PROFILE=dev tchotcho key --help
Usage: tchotcho key [OPTIONS] COMMAND [ARGS]...
Options:
--help Show this message and exit.
Commands:
create
delete
fingerprint
import
list
Shell
Currently rsync and ssh are implemented.
Rsync
Sync your local experiment to EC2. The folder .git
is excluded and all rules
are applied from .gitignore
.
Usage:
❯ tchotcho shell rsync --src . --dst ubuntu@1.2.3.4:/home/ubuntu/project --privat-key ~/.ssh/id_rsa
Ssh
Execute remote command on EC2.
Usage:
❯ tchotcho shell ssh --host ubuntu@1.2.3.4 --privat-key ~/.ssh/id_rsa --cmd "pwd && date"
Spot
List spot price for given instances and regions.
❯ AWS_PROFILE=dev tchotcho spot list --help
Usage: tchotcho spot list [OPTIONS]
Options:
--region TEXT List spot prices in region
--gpu / --no-gpu Limit results to show only GPU instances
--inst TEXT List spot prices for instance type
--csv / --no-csv
--help Show this message and exit.
List
List spot price by default only supported GPU instances are shown.
Usage:
❯ AWS_PROFILE=dev tchotcho spot list --region eu-central-1
╒════════════════╤════════════════════╤═════════════╕
│ InstanceType │ AvailabilityZone │ SpotPrice │
╞════════════════╪════════════════════╪═════════════╡
│ g4dn.xlarge │ eu-central-1b │ 0.1974 │
├────────────────┼────────────────────┼─────────────┤
│ g4dn.xlarge │ eu-central-1a │ 0.1974 │
├────────────────┼────────────────────┼─────────────┤
│ g3.4xlarge │ eu-central-1b │ 0.4275 │
├────────────────┼────────────────────┼─────────────┤
│ g3.4xlarge │ eu-central-1a │ 0.4275 │
...
│ g3.16xlarge │ eu-central-1a │ 1.71 │
├────────────────┼────────────────────┼─────────────┤
│ g4dn.16xlarge │ eu-central-1b │ 1.8593 │
├────────────────┼────────────────────┼─────────────┤
│ p3.8xlarge │ eu-central-1b │ 4.5876 │
├────────────────┼────────────────────┼─────────────┤
│ p3.8xlarge │ eu-central-1a │ 4.5876 │
├────────────────┼────────────────────┼─────────────┤
│ p3.16xlarge │ eu-central-1b │ 9.1752 │
├────────────────┼────────────────────┼─────────────┤
│ p2.8xlarge │ eu-central-1b │ 10.608 │
├────────────────┼────────────────────┼─────────────┤
│ p2.8xlarge │ eu-central-1a │ 10.608 │
╘════════════════╧════════════════════╧═════════════╛
Stack
Core of this tool create a EC2 instance via cloudformation.
❯ AWS_PROFILE=dev tchotcho stack --help
Usage: tchotcho stack [OPTIONS] COMMAND [ARGS]...
Options:
--help Show this message and exit.
Commands:
create
delete
list
Create
Create a cloudformation stack. The name needs to be the same as the imported key in aws.
Usage:
❯ AWS_PROFILE=dev tchotcho stack create --help
Usage: tchotcho stack create [OPTIONS]
Options:
--name TEXT Name of stack to create && key [required]
--ami TEXT Name of ami to use [default: ami-061aaaac62de85935;
required]
--inst TEXT Name of the instance to use [required]
--security_group TEXT Name of the security group to use
--subnet TEXT Name of the subnet to use
--price FLOAT Name of the instance to use
--size INTEGER Size of the disk in GB
--dry / --no-dry Only print yaml no create
--help Show this message and exit.
Example:
❯ AWS_PROFILE=dev tchotcho stack create --name test-dl --inst t2.medium --price 0.02 --dry
User script
TODO Currently only via the api changeable. Explain:
- waiting for apt via flock
- signal to cloudformation so we wait until user script install is finished
Code style
We use flake8
to ensure code quality and black
to autoformat code.
Add it as a git-hook:
$ flake8 --install-hook git
$ git config --bool flake8.strict true
TODO
- Try paramiko + scp to be pure python
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.