Dsconfig

This is a command line tool for managing configuration of Tango device servers. It runs on python 2.7 as well as 3.6 and up.

The goal of this project is to provide tools for configuring a Tango database in a convenient way. Right now the focus is on supporting Excel files as input ("xls2json"), but support for other formats should follow.

The main idea is that the input files are parsed and turned into an intermediate JSON format, specified by a schema. This file can then be given to the "json2tango" tool which then tries to make the database contents match, by adding, modifying or removing servers, devices and properties.

The JSON format is easy to create and supported by many tools and languages, so generating them from various sources should be straightforward. Once you have such a file, it should be a simple thing to configure the Tango database.

Making a release

When it's time to release a new package, the procedure is currently like this:

• Make sure all the new stuff is in main, and pipelines are green.
• Bump the version number in setup.py (try to use "semver" logic), and an entry in CHANGELOG.md.
• Create a tag in the repo, with the same version number.
• There should be a new build pipeline in CI, ending with a manually triggered step to upload to PyPI.

Done! Your new version should very shortly be available on PyPI.

Caveats

There are a few things to be aware of before using this tool.

• The basic idea of dsconfig is idempotence. This means that applying the same dsconfig file a second time should result in no changes at all. The intention is that it should be useful not only to update the configuration but also to be able to check if anything has changed since the last application. Therefore, the tool tries to figure out the smallest set of database operations needed to get to the intended state.

• TANGO is case insensitive for names, for example of devices and properties. But there are some cases where this causes confusing results. For example, TANGO keeps the case that was used when last written, which means that the same name may exist in different places with different cases. dsconfig tries to handle this gracefully, but it is complex (for example, all relevant string comparisons need to be done in a case insensitive way) and there are bound to be corner cases where the behavior is unexpected. Please report such cases if you run into them.

JSON format

This is an example of the format, with comments (comments are not actually supported by JSON so don't copy-paste this!):

{
    // these lines are meta information and are ignored so far
    "_version": 1,
    "_source": "ConfigInjectorDiag.xls",
    "_title": "MAX-IV Tango JSON intermediate format",
    "_date": "2014-11-03 17:45:04.258926",

    // here comes the actual Tango data
    // First, server instances and devices...
    "servers": {
        "some-server/instance": {
            "SomeDeviceClass": {
                "some/device/1": {
                    "properties": {
                        "someImportantProperty": [
                            "foo",
                            "bar"
                        ],
                        "otherProperty": ["7"]
                    },
                    "attribute_properties": {
                        "anAttribute": {
                            "min_value": ["-5"],
                            "unit": ["mV"]
                        }
                    }
                }
            }
        }
    },
    // Here you can list your class properties
    "classes": {
        "SomeDeviceClass": {
            "properties": {
                "aClassProperty": ["67.4"]
            }
        }
    }
}

Note that all properties are given as lists of strings. This is how the Tango DB represents them so it gets a lot easier to compare things if we do it too.

Note: the format is now more loosely defined; it is allowed to split the server and instance names into separate levels, like so:

    "servers": {
        "some-server": {
            "instance": {
                "SomeDeviceClass": ...

Note: Leaving out "properties" in a device will mean that the tool just ignores any existing properties when applying the configuration. Otherwise, properties not specified in the new configuration will get cleaned away. An empty properties object ("properties": {}) means that any existing properties will get cleaned up. Same goes for "attribute_properties".

Protected properties

Some properties are considered "protected", meaning they will not be removed if they are absent in the new config. This covers properties that are used by Tango for configuring internal features, such as polling, events and logging. The assumption is that these may be configured in other ways and should not cleaned up automatically. Individual protected properties can still be explicitly removed by specifying an empty list as value. There is also a flag, --cleanup-protected-props, which means protected properties are handled just like normal ones.

The lists of protected properties can be found in dsconfig/tangodb.py.

xls2json

This tool converts a excel file (of proper format) into a dsconfig JSON file.

The XLS format supported is almost identical to the dsgenerator format, with a few changes:

• It is now possible to spread server definitions over any number of pages, and to selectively use only a subset of these by giving their names to the xls2json tool.
• The column names (the first line of each column) are now significant, so that their order can be relaxed. There are a few differences to the "standard" sheet though; "ServerName" should be "Server", "Devices" should be "Device" and, in the "ParamConfig" tab, "Parameter" should now be "Attribute". These changes were made for consistency.
• A few features have been added for flexibility; see the example Excel file in "test/files/test.xls".

Converting an excel file is done like this:

xls2json config.xls

This will output the resulting JSON data to stdout. If there are errors or warnings, they will be printed on stderr. To save the JSON to a file, just redirect the output.

By default, all sheets are processed. If you want to only include some of them, include the sheet names as further arguments to the command:

xls2json config.xls sheet1 sheet2

The "Dynamics" and "ParamConfig" sheets are treated specially as they follow a different style. Some syntax checking is done on dynamic formulas, to make sure they compile. Failures are printed to stderr and the corresponding properties skipped, so be careful (see the -f flag to override this).

The command is quite verbose and it will by default happily skip lines that contain incomplete information. Make sure to check the stderr output for hints about this. At the end the command prints a line of statistics, listing the number of servers, etc, it has found. This is intended as a useful sanity check. Also look over the JSON result to see if it makes sense.

Useful flags (see --help):

• --fatal (-f) means that the command will treat any parsing failure as a fatal error and exit instead of skipping the line as normal. Use if you don't like the lenient default behavior.

json2tango

This tool reads a JSON file (or from stdout if no filename is given), validates it and, optionally, configures a Tango database accordingly. By default, it will only check the current DB state, compare, and print out what actions would be performed, without changing anything. This should always be the first step, in order to catch errors before they are permanently written to the DB.

json2tango config.json

Inspect the output of this command carefully. Things in red means removal, green additions and yellow changes. Note that properties are stored as lists of strings in the DB, so don't be confused by the fact that your numeric properties turn up as strings.

[Pro-tip: if you're unsure of what's going on, it's a good idea to inspect the output of the -d argument (see below) before doing any non-trivial changes. It's usually less readable than the normal diff output, but garanteed to be accurate.]

A summary of the numbers of different database operations is printed at the end. This should be useful to double check, usually you have a good idea of e.g. how many devices should be added, etc.

Once you're convinced that the actions are correct, add the "-w" flag to the command line (this can be at the end or anywhere). Now the command will actually perform the actions in the Tango DB.

For safety and convenience, the program also writes the previous DB state that was changed into a temp JSON file (this is the same as the output of the -d flag). It should, in principle, be possible to undo the changes made by swapping your input JSON file with the temp file. This is a new feature that is not tested for many cases so don't rely on it.

Note that the tool in principle only concerns itself with the server instances defined in your JSON file. All other servers in the DB are left untouched. The exception is if your JSON contains devices that already exist in the DB, but in different servers. The devices will be moved to the new servers, and if any of the original servers become empty of devices, they will be removed. There is currently no other way to remove a server with dsconfig.

Some useful flags (see --help for a complete list):

• --write (-w) is needed in order to actually do anything to the database. This means that the command will perform the actions needed to bring the DB into the described state. If the state is already correct, nothing is done.

• --update (-u) means that "nothing" (be careful, see caveats below) will be removed, only changed or added. Again the exception is any existing duplicates of your devices. Also, this only applies to whole properties, not individual lines. So if your JSON has lines removed from a property, the lines will be removed from the DB as the whole property is overwritten, regardless of the --update flag.

• --include (-i) [Experimental] lets you filter the configuration before applying it. You give a filter consisting of a "term" (server/class/device/property) and a regular expression, separated by colon. E.g. "--include=device:VAC/IP.*01". This will cause the command to only apply configuration that concerns those devices matching the regex. It is possible to add several includes, just tack more "--include=..." statements on.

• --exclude (-x) [Experimental] works like --include except it removes the matching parts from the config instead.

Some less useful flags:

• --no-validation (-v) skips the JSON validation step. If you know what you're doing, this may be useful as the validation is very strict, while the tool itself is more forgiving. Watch out for unexpected behavior though; you're on your own! It's probably a better idea to fix your JSON.

• --dbcalls (-d) prints out all the Tango database API calls that were, or would have been, made to perform the changes. This is mostly handy for debugging problems. Since this is the real list of commands that are performed, it is guaranteed to correspond to reality.

• --sleep (-s) tweaks the time to wait between db calls. The default is 0.01 s. This is intended to lighten the load on the Tango DB service a bit, but it can be set to 0 if you just want the config to be done as fast as possible.

• --input (-p) tells the command to simply print the configuration file, but after any filters have been applied. It can be useful in order to check the result of filtering. If no filters are used, it will just (pretty) print whatever file you gave as input. This flag skips all database operations so it can be used "offline".

• --json (-j) [Experimental] prints a JSON format representation of the diff, instead of the default "human friendly" output. This can be more convenient if the output is to be consumed by another program, as JSON is easy to parse. For an example of the JSON format, see the json2tango tests. Since this feature is considered "experimental" the format may change in future versions.

Other features

Dumping the database

It's often useful to be able to make a "snapshot" of the current state of the configuration, e.g. for safe keeping, or for usage in scripts. There is a module in dsconfig that allows this called dump.

python -m dsconfig.dump

It outputs the contents of the current TANGO database to stdout. There are some filtering functionality to allow only dumping selected parts (e.g. servers), like so:

python -m dsconfig.dump server:LimaCCDs/*

For more help, try the --help flag.

Viewing JSON files

Reading a large, nested JSON file can be painful, but dsconfig has a solution; a hierarchical, terminal based JSON viewer! If you install the python packages urwid and urwidtrees, you can interactively view any JSON file by running

python  -m dsconfig.viewer something.json

From the start, everything is "folded" but you can navigate the structure by using the arrow keys and return to fold/unfold nodes.