Shlax: Pythonic automation tool

Shlax is a Python framework for system automation, initially with the purpose of replacing docker, docker-compose and ansible with a single tool with the purpose of code-reuse made possible by target abstraction.

Development status: Design state

I got the thing to work with an ugly PoC that I basically brute-forced, I'm currently rewriting the codebase with a proper design.

The stories are in development in this order:

• replacing docker build, that's in the state of polishing
• replacing docker-compose, not in use but the PoC works so far
• replacing ansible, also working in working PoC state, the shlax command line demonstrates

This project is supposed to unblock me from adding the CI feature to the Sentry/GitLab/Portainer implementation I'm doing in pure python on top of Django, CRUDLFA+ and Ryzom (isomorphic components in Python to replace templates). So, as you can see, I'm really deep in it with a strong determination.

Shlax builds its container itself, so check the shlaxfile.py of this repository to see what it currently looks like, and check the build job of the CI pipeline to see the output.

Design

The pattern resolves around two moving parts: Actions and Targets.

Action

An action is a function that takes a target argument, it may execute nested actions by passing over the target argument which collects the results.

Example:

async def hello_world(target):
    """Bunch of silly commands to demonstrate action programming."""
    await target.mkdir('foo')
    python = await target.which('python3', 'python')
    await target.exec(f'{python} --version > foo/test')
    version = target.exec('cat foo/test').output
    print('version')

Recursion

An action may call other actions recursively. There are two ways:

async def something(target):
    # just run the other action code
    hello_world(target)

    # or delegate the call to target
    target(hello_world)

In the first case, the resulting count of ran actions will remain 1: "something" action.

In the second case, the resulting count of ran actions will be 2: "something" and "hello_world".

Callable classes

Actually in practice, Actions are basic callable Python classes, here's a basic example to run a command:

class Run:
    def __init__(self, cmd):
        self.cmd = cmd

    async def __call__(self, target):
        return await target.exec(self.cmd)

This allows to create callable objects which may be called just like functions and as such be appropriate actions, instead of:

async def one(target):
    target.exec('one')

async def two(target):
    target.exec('two')

You can do:

one = Run('one')
two = Run('two')

Parallel execution

Actions may be executed in parallel with an action named ... Parallel. This defines an action that will execute three actions in parallel:

action = Parallel(
    hello_world,
    something,
    Run('echo hi'),
)

In this case, all actions must succeed for the parallel action to be considered a success.

Methods

An action may also be a method, as long as it just takes a target argument, for example:

class Thing:
    def start(self, target):
        """Starts thing"""

    def stop(self, target):
        """Stops thing"""

action = Thing().start

Cleaning

If an action defines a clean method, it will always be called wether or not the action succeeded. Example:

class Thing:
    def __call__(self, target):
        """Do some thing"""

    def clean(self, target):
        """Clean-up target after __call__"""

Colorful actions

If an action defines a colorize method, it will be called with the colorset as argument for every output, this allows to code custom output rendering.

Target

A Target is mainly an object providing an abstraction layer over the system we want to automate with actions. It defines functions to execute a command, mount a directory, copy a file, manage environment variables and so on.

Pre-configuration

A Target can be pre-configured with a list of Actions in which case calling the target without argument will execute its Actions until one fails by raising an Exception:

say_hello = Localhost(
    hello_world,
    Run('echo hi'),
)
await say_hello()

Results

Every time a target execute an action, it will set the "status" attribute on it to "success" or "failure", and add it to the "results" attribute:

say_hello = Localhost(Run('echo hi'))
await say_hello()
say_hello.results  # contains the action with status="success"

Targets as Actions: the nesting story

We've seen that any callable taking a target argument is good to be considered an action, and that targets are callables.

To make a Target runnable like any action, all we had to do is add the target keyword argument to Target.__call__.

But target() fills self.results, so nested action results would not propagate to the parent target.

That's why if Target receives a non-None target argument, it will has to set self.parent with it.

This allows nested targets to traverse parents and get to the root Target with target.caller, where it can then attach results to.

This opens the nice side effect that a target implementation may call the parent target if any, you could write a Docker target as such:

class Docker(Target):
    def __init__(self, *actions, name):
        self.name = name
        super().__init__(*actions)

    async def exec(self, *args):
        return await self.parent.exec(*['docker', 'exec', self.name] + args)

This also means that you always need a parent with an exec implementation, there are two:

• Localhost, executes on localhost
• Stub, for testing

The result of that design is that the following use cases are available:

# This action installs my favorite package on any distro
action = Packages('python3')

# Run it right here: apt install python3
Localhost()(action)

# Or remotely: ssh yourhost apt install python3
Ssh(host='yourhost')(action)

# Let's make a container build receipe with that action
build = Buildah(package)

# Run it locally: buildah exec apt install python3
Localhost()(build)

# Or on a server: ssh yourhost build exec apt install python3
Ssh(host='yourhost')(build)

# Or on a server behingh a bastion:
# ssh yourbastion ssh yourhost build exec apt install python3
Localhost()(Ssh(host='bastion')(Ssh(host='yourhost')(build))

# That's going to do the same
Localhost(Ssh(
    Ssh(
        build,
        host='yourhost'
    ),
    host='bastion'
))()

CLI

You can execute Shlax actions directly on the command line with the shlax CLI command.

For your own Shlaxfiles, you can build your CLI with your favorite CLI framework. If you decide to use cli2, then Shlax provides a thin layer on top of it: Group and Command objects made for Shlax objects.

For example:

yourcontainer = Container(
    build=Buildah(
        User('app', '/app', 1000),
        Packages('python', 'unzip', 'findutils'),
        Copy('setup.py', 'yourdir', '/app'),
        base='archlinux',
        commit='yourimage',
    ),
)


if __name__ == '__main__':
    print(Group(doc=__doc__).load(yourcontainer).entry_point())

The above will execute a cli2 command with each method of yourcontainer as a sub-command.