Python Configuration and a command line Runner

Python configuration file and command line executable to run a script with.

Why a python file as configuration?

• Configuration of any complexity
• No need to serialize the configuration
• No need other meta-languages for the configuration

Usage

In the code

Configuration file (e.g. config.py):

num_epochs = 100
batch_size = 256

model = resnet18(10)
train_loader = get_train_loader("/path/to/dataset", batch_size=batch_size)
unsup_dataloader = get_train_unsup_loader("/path/to/unsup_dataset", batch_size=batch_size)
...

Script file (e.g. training.py):

from torch.utils.data import DataLoader
from py_config_runner import ConfigObject, TrainvalConfigSchema


class SSLTrainvalConfigSchema(TrainvalConfigSchema):

    unsup_dataloader: DataLoader



def training(config):
    # ...
    print(config.config_filepath)
    print(config.output_path)
    print(config.num_epochs)
    print(config.model)
    print(len(config.train_loader))


def main():

    config_filepath = "/path/to/config.py"
    config = ConfigObject(config_filepath)

    SSLTrainvalConfigSchema.validate(config)

    # Add more things at runtime    
    config.output_path = "/tmp/output"

    training(config)

With launcher

cd /path/to/my/project
py_config_runner scripts/training.py configs/train/baseline.py

or

cd /path/to/my/project
python -u -m py_config_runner scripts/training.py configs/train/baseline.py

or if your specific launcher requires only python script files:

cd /path/to/my/project
python -m special_launcher `py_config_runner_script` scripts/training.py configs/train/baseline.py

The only condition on the script file is it should contain run(config, **kwargs) callable method. Additionally, argument kwargs contains logger (e.g. kwargs['logger']) and local_rank (e.g. kwargs['logger']) for distributed computations.

No restrictions are applied on the configuration file. It is user's responsibility to provide the script file that can consume given configuration file. Provided configuration file is loaded as python module and exposed into the script as the module named config.

Examples

• Example for Machine/Deep Learning

Installation

pip install py-config-runner