CommonBuild Dependency Manager

A module to manage Java dependencies across multiple Python packages.

Introduction

When using multiple packages (PyJapc, PyTimber, etc) inside a single script, they must share a single JVM instance. This means that the required jars for all packages must be added to the class path of the JVM that is started by the first instantiated package.

cmmnbuild-dep-manager provides a simple way to get a single list of jars for all installed packages. As packages are added/removed it automatically resolves and downloads the required jar files using the CBNG web service (within the CERN network) or Gradle (in a public network).

Usage

Packages should implement the variable __cmmnbuild_deps__ in their __init__.py containing a list of the required dependencies, for example:

__cmmnbuild_deps__ = [
    'japc',
    'japc-value',
    'japc-ext-cmwrda',
    'japc-ext-cmwrda3',
    'japc-ext-dirservice',
    {'product': 'inca-client',   'version': 'PRO'},
    {'product': 'slf4j-log4j12', 'local':   'true'},
    {'product': 'slf4j-api',     'local':   'true'},
    {'product': 'log4j',         'local':   'true'}
]

Each item in the list can be either a string containing the product name, or a dictionary with keys matching the CBNG product.xml file (more info).

With the variable in place, packages can be registered with cmmnbuild-dep-manager. Registration is automatic if your package defines an entry_point of the form "cmmnbuild_dep_manager": {package_name}={package_version}. For example, PyJapc might have an item in its setup.py such as:

entry_points={
    # Register with cmmnbuild_dep_manager.
    'cmmnbuild_dep_manager': ['pyjapc=2.3.1'],
},

Finally, from your code, you can start a JVM with the complete list of jars using:

mgr = cmmnbuild_dep_manager.Manager()
jpype = mgr.start_jpype_jvm()

At this stage, JPype is ready to use.

To enable the JPype (>= 0.7) import system, you can use the imports() context manager:

mgr = cmmnbuild_dep_manager.Manager()
with mgr.imports():
    from cern.some.service import Service
    # ...

Advanced usage

Re-downloading jars

The jars can be resolved and re-downloaded at any time by running the following shell command:

$ python -m cmmnbuild_dep_manager resolve

Jar inspection helper functions

Helper functions are provided to inspect the classes in the downloaded jars.

class_list()

Provides a listing of all classes contained within the jars:

mgr = cmmnbuild_dep_manager.Manager()
mgr.class_list()
['cern.accsoft.cals.extr.client.commandline.CommandLineException',
 'cern.accsoft.cals.extr.client.commandline.CommandLineServiceBuilder',
 'cern.accsoft.cals.extr.client.commandline.CommandMethod',
 'cern.accsoft.cals.extr.client.commandline.CommandOption',
 ...]

class_search()

Search for any class by name:

mgr = cmmnbuild_dep_manager.Manager()
mgr.class_search('ServiceBuilder')
['cern.accsoft.cals.extr.client.commandline.CommandLineServiceBuilder',
 'cern.accsoft.cals.extr.client.service.ServiceBuilder',
 'cern.cmw.rda3.client.service.ClientServiceBuilder',
 'cern.cmw.rda3.impl.client.service.ClientServiceBuilderImpl']

class_hints()

Provide a pasteable example of how to use a specific class from Python:

mgr = cmmnbuild_dep_manager.Manager()
mgr.class_hints('cern.accsoft.cals.extr.client.service.ServiceBuilder')
cern = jpype.JPackage('cern')
ServiceBuilder = cern.accsoft.cals.extr.client.service.ServiceBuilder

jpype = mgr.start_jpype_jvm()
cern = jpype.JPackage('cern')
ServiceBuilder = cern.accsoft.cals.extr.client.service.ServiceBuilder
ServiceBuilder
jpype._jclass.cern.accsoft.cals.extr.client.service.ServiceBuilder

Command-line interface

cmmnbuild-dep-manager has a basic command-line interface that allows access to any method of the class. It can be invoked with:

$ python -m cmmnbuild_dep_manager METHOD [ARG ...]

For example, to register the packages pytimber and pyjapc you could run:

$ python -m cmmnbuild_dep_manager register pytimber pyjapc
pytimber
pyjapc

Which is equivalent to the following Python code:

mgr = cmmnbuild_dep_manager.Manager()
mgr.register('pytimber', 'pyjapc')
('pytimber', 'pyjapc')

You can also see from this example that the return value is automatically split to one list item per line.

Manually installing a package

In certain circumstances during development it can be useful to manipulate the cmmnbuild_dep_manager manager directly. For example, manual installation of a package can be achieved with:

mgr = cmmnbuild_dep_manager.Manager()
mgr.install('pyjapc')

The install() function registers the package and resolves the dependencies automatically. It was previously recommended that this be done as part of the setup.py, but Python's move away from code execution at installation and towards a non-executable binary wheel format means that the declarative entry_points solution is now strongly recommended.

Dealing with read-only Python installations

For a typical installation, cmmnbuild-dep-manager downloads jars into a subdirectory of the global site-packages and this process happens when a package is installed. However, in some situations, the global site-packages is not writeable by the user (e.g. in a centrally managed distribution, such as SWAN). In this case, jars can be downloaded to each user's local site-packages instead.

To facilitate this, a package can automatically install itself by setting an optional parameter of its Manager() instance. For example:

class PyJapc:
    def __init__(self, ...):
        ...
        mgr = cmmnbuild_dep_manager.Manager('pyjapc')
        mgr.start_jpype_jvm()
        ...

Then the user executing:

import pyjapc
japc = pyjapc.PyJapc()

will cause the jars to be downloaded if they aren't already existing on disk or if the version of PyJapc has changed.

Usage outside of CERN

For deployments outside of the CERN network, packages should implement the variable __gradle_deps__ in their __init__.py containing a list of the required dependencies. In this case, cmmnbuild-dep-manager will use Gradle to query public Java product repositories: Maven Central and JCenter.

Note that __cmmnbuild_deps__ (see above) will be used within CERN and __gradle_deps__ outside of CERN. Packages used in both scenarios must implement both variables, and ensure consistency within the two.

__cmmnbuild_deps__ = [
    "accmodel-jmad-core",
    "accmodel-jmad-models-lhc",
    "accmodel-jmad-models-gsi",
    "accmodel-jmad-models-lhctransfer",
    "accsoft-steering-commons",
    "accmodel-jmad-gui",
    "slf4j-log4j12",
    "slf4j-api",
    "log4j"
]

__gradle_deps__ = [
    "jmad:jmad-core:+",
    "org.slf4j:slf4j-api:+",
    "org.slf4j:slf4j-log4j12:+",
    "log4j:log4j:1.2.17",
]

For __gradle_deps__, the list should consist of strings in the Gradle syntax groupId:artifactId:version, or a dictionary with keys groupId, product and version.

More information

For more information or to report problems, please contact P. Elson via the Acc-Py support mailing-list.