calmjs.rjs 2.0.1

Installing/using RequireJS with calmjs

To establish a development/build environment for a Python package with the support for r.js through calmjs.rjs in the current working directory (e.g. for a project), the following command may be executed:

$ calmjs npm --install calmjs.rjs

While running npm install requirejs (along with other related packages declared by calmjs.rjs that it needs from npm) will achieve the same effect, do note the Calmjs framework makes it possible for npm dependencies to be propagated down to dependent packages; such that if a Python package that have declared calmjs.rjs as a dependency (either through install_requires or an extras_require in its setup.py) may have its complete set of dependencies on npm be installed using the following command (assuming the package is named example.package:

$ calmjs npm --install example.package

All standard JavaScript and Node.js dependencies for example.package will now be installed into the current directory through the relevant tools. This process will also install all the other dependencies through npm or requirejs that other Python packages depended on by example.package have declared.

For further details about how this all works can be found in the documentation for calmjs. Otherwise, please continue on to the usage section.

Alternative installation methods (advanced users)

Development is still ongoing with calmjs.rjs, for the latest features and bug fixes, the development version can be installed through git like so:

$ pip install calmjs
$ pip install git+https://github.com/calmjs/calmjs.rjs.git#egg=calmjs.rjs

Alternatively, the git repository can be cloned directly and execute python setup.py develop while inside the root of the source directory.

Keep in mind that calmjs MUST be available before the setup.py within the calmjs.rjs source tree is executed, for it needs the package_json writing capabilities in calmjs. Alternatively, please execute python setup.py egg_info if any message about Unknown distribution option: is noted during the invocation of setup.py.

As calmjs is declared as both namespace and package, there are certain low-level setup that is required on the working Python environment to ensure that all modules within can be located correctly. However, versions of setuptools earlier than v31.0.0 does not create the required package namespace declarations when a package is installed using this development installation method when mixed with pip install within the same namespace. As a result, inconsistent import failures can happen for any modules under the calmjs namespace. As an example:

>>> import calmjs.rjs
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named 'calmjs.rjs'
>>> import calmjs.base
>>> import calmjs.rjs
>>>

If this behavior (and workaround) is undesirable, please ensure the installation of all calmjs related packages follow the same method (i.e. either python setup.py develop for all packages, or using the wheels acquired through pip), or upgrade setuptools to version 31 or greater and reinstall all affected packages.

Testing the installation

Finally, to verify for the successful installation of calmjs.rjs, the included tests may be executed through this command:

$ python -m unittest calmjs.rjs.tests.make_suite

However, if the steps to install external Node.js dependencies to the current directory was followed, the current directory may be specified as the CALMJS_TEST_ENV environment variable. Under POSIX compatible shells this may be executed instead from within that directory:

$ CALMJS_TEST_ENV=. python -m unittest calmjs.rjs.tests.make_suite

Do note that if the calmjs.dev package is unavailable, a number of tests will be skipped. To avoid this, either install that package separately, or install calmjs.rjs using its extras dependencies declaration like so:

$ pip install calmjs.rjs[dev]

Declaring JavaScript exports for Python

Any exposed JavaScript code through the calmjs.module registry will be picked up and compiled into a working RequireJS artifact. For example, given the following entry points for that registry defined by a package named example:

[calmjs.module]
example = example

This is the most basic declaration that works for packages that share the same name as the import location provided.

The following is am example for packages that have nested submodules (called example.lib and example.app):

[calmjs.module]
example.lib = example.lib
example.app = example.app

While the import locations declared looks exactly like a Python module (as per the rules of a Python entry point), the calmjs.module registry will present them using the CommonJS/ES6 style import paths (i.e. 'example/lib' and 'example/app'), so users of that need those JavaScript modules to be sure they require those strings.

Please also note that the default source extractor will extract all JavaScript files within those directories. Finally, as a consequence of how the imports are done, it is recommended that no relative imports are to be used.

If the package at hand does not directly declare its dependency on calmjs, an explicit calmjs_module_registry=['calmjs.module'] may need to be declared in the setup function for the package to ensure that this default module registry will be used to acquire the JavaScript sources from.

Putting this together, the setup.py file should contain the following:

setup(
    name='example',
    # ... plus other declarations
    # This is one of the recommended options, even though the
    # project will not be importing from Calmjs.
    license='gpl',
    install_requires=[
        'calmjs>=3.0.0,<4',
        'calmjs.rjs>=2.0.0,<3',
        # plus other installation requirements
    ],
    # If the usage of the GPL is impossible for the project, or
    # declaring a direct dependency on calmjs packages is impossible
    # for the project for whatever other reasons (even though the
    # project itself will NOT be required to include/import ANY code
    # from the calmjs namespace), setup_requires may be used instead
    # of install_requires, and the following should also be included
    # as well:
    package_json={
        "devDependencies": {
            "requirejs": "~2.1.17",
        }
    },
    calmjs_module_registry=['calmjs.module'],
    # the entry points are required to allow calmjs to pick this up
    entry_points="""
    [calmjs.module]
    example = example
    example.lib = example.lib
    example.app = example.app
    """,
)

For the construction of the RequireJS artifact, the command calmjs rjs will automatically extract all relevant source files from the dependencies of the selected Python package(s) into a temporary build directory, where the build manifest will also be generated for the invocation of r.js to construct the artifact. An example run:

$ calmjs rjs example

Tracing dependencies for: /home/user/example.js

/home/user/example.js
----------------
/tmp/tmp_build/build/example/lib/form.js
/tmp/tmp_build/build/example/lib/ui.js
/tmp/tmp_build/build/example/lib/main.js
/tmp/tmp_build/build/example/app/index.js

As the build process used by calmjs.rjs is done in a separate build directory, all imports through the Node.js module system must be declared as extras_calmjs. For instance, if example/app/index need to use the jquery and underscore modules like so:

var $ = require('jquery'),
    _ = require('underscore');

It will need to declare the target location sourced from npm plus the package_json for the dependencies, it will need to declare this in its setup.py:

setup(
    # ...
    package_json={
        "dependencies": {
            "jquery": "~3.1.0",
            "underscore": "~1.8.0",
        },
    },
    extras_calmjs = {
        'node_modules': {
            'jquery': 'jquery/dist/jquery.js',
            'underscore': 'underscore/underscore.js',
        },
    },
)

Once that is done, rerun python setup.py egg_info to write the freshly declared metadata into the package’s egg-info directory, so that it can be used from within the environment. calmjs npm --install can now be invoked to install the npm dependencies into the current directory; to permit calmjs.rjs to find the required files sourced from npm to put into the build directory for r.js to locate them.

The resulting calmjs run may then end up looking something like this:

$ calmjs rjs example

Tracing dependencies for: /home/user/example.js

/home/user/example.js
----------------
/tmp/tmp_build/build/jquery.js
/tmp/tmp_build/build/underscore.js
/tmp/tmp_build/build/example/lib/form.js
/tmp/tmp_build/build/example/lib/ui.js
/tmp/tmp_build/build/example/lib/main.js
/tmp/tmp_build/build/example/app/index.js

The transpiler will add the appropriate boilerplates and thus the require statements through requirejs will import from node_modules if the extras_calmjs have been declared. However, there are cases where the desired artifact should only contain the sources from the Python package without the extras or vice versa (due to the library being available via another deployed artifact), this is supported by the empty: scheme by r.js, and to enable it for calmjs rjs it can be done like so:

$ calmjs rjs example --bundlepath-method empty --export-filename main.js

Tracing dependencies for: /home/user/main.js

/home/user/main.js
----------------
/tmp/tmp_build/build/example/lib/form.js
/tmp/tmp_build/build/example/lib/ui.js
/tmp/tmp_build/build/example/lib/main.js
/tmp/tmp_build/build/example/app/index.js

$ calmjs rjs example --sourcepath-method empty --export-filename deps.js

Tracing dependencies for: /home/user/deps.js

/home/user/deps.js
----------------
/tmp/tmp_build/build/jquery.js
/tmp/tmp_build/build/underscore.js

The above example shows the generation of two separate artifacts, one containing just the sources from the Python package example that had been declared in the calmjs.module registry, and the other contains only the external extra sources.

If the above triggers a dependency trace error for r.js, there is a last resort --empty flag that can be applied; do note that this completely disables the trace functionality for r.js as this initiates a similar trace process to locate all the imported module names for stubbing them out with the empty: scheme within the generated configuration file. Ensure that the modules required by the resulting artifact has all its required modules provided elsewhere.

The explicit extras_calmjs declaration also supports the usage through bower (supported via calmjs.bower); instead of using node_modules as the key, bower_components should be used instead.

Alternative registries aside from calmjs.module can be specified with the --source-registry flag. Assuming there are registries in the current environment registered as myreg1 and myreg2 and the example package has registered sources to both of them, the command to build a bundle from both those registries into one artifact can be triggered like so:

$ calmjs rjs --source-registry=myreg1,myreg2 example

Handling of RequireJS loader plugins

The AMD system as defined by RequireJS has the concept of loader plugins, where the module name provided may be suffixed with a ! as arguments for handling by the plugin. As the string provided after is opaque to the requirejs system as a whole and thus handled directly by the preceding plugin, the resources that it need will be specific to the plugin itself. As it may load resources through the requirejs system, any paths that require configuration will need to be done.

To account for this issue, calmjs.rjs introduces the concept of loader plugin handlers and a registry system for dealing with this. A given RJSToolchain will have a default loader plugin registry assigned, but this can be overridden by specifying a custom identifier (overriding the default 'calmjs.rjs.loader_plugin') for the registry to be used, which will allow the handling of very customized loaders for a given project. Please refer to the calmjs.rjs.registry module for more details on how this is constructed and set up for usage.

By default, the text handler is registered to the default loader plugin registry, which should cover the most common use case encountered by the calmjs framework. Do note that packages are still required to declare their (dev)dependencies in their package_json to the plugin, ideally with a well constrained version, so to ensure a consistent build experience for all end users.

Testing standalone, finalized RequireJS artifacts

AMD artifacts generated using the standard calmjs rjs toolchain runtime may be tested using the calmjs karma runtime provided by the calmjs.dev package. Given a finalized example.js that implements the features provided by the example package, the artifact may be tested with the tests provided by the example package using the following command:

$ calmjs karma run \
    -t calmjs.rjs \
    --artifact=example.js \
    example

The above command invokes the standalone Karma runner using the calmjs.rjs settings to test against the example.js artifact file, using the tests provided by the example package. The test execution is similar to the one during the development process.

Declare prebuilt JavaScript artifacts for Python packages

Finally, to complete the Python package deployment story, the process should include the automatic generation and inclusion of the JavaScript artifacts in the resulting Python wheel. This can be achieved by specifying an entry in the calmjs.artifacts registry, with the key being the filename of the artifact and the value being the import location to a builder. A default builder function provided at calmjs.rjs.artifact:complete_rjs will enable the generation of a complete RequireJS artifact for the Python package. For example:

[calmjs.artifacts]
example.package.rjs.js = calmjs.rjs.artifact:complete_rjs

Once those entry points are installed, running calmjs artifact build example.package will make use of the RequireJS toolchain and build the artifact at example.package.rjs.js inside the calmjs_artifacts directory within the metadata directory for example.package. Alternatively, for solution more integrated with setuptools, the setup function in setup.py should also enable the build_calmjs_artifacts flag such that setup.py build will also trigger the building process. This is useful for automatically generating and including the artifact as part of the wheel building process. Consider this setup.py:

setup(
    name='example.package',
    # ... other required fields truncated
    build_calmjs_artifacts=True,
    entry_points="""
    # ... other entry points truncated
    [calmjs.module]
    example.package = example.package

    [calmjs.artifacts]
    example.package.rjs.js = calmjs.rjs.artifact:complete_rjs
    """,
)

Building the wheel using setup.py may result in something like this. Note that the execution of r.js was part of the process and that the metadata (egg-info) directory was then built into the wheel.

$ python setup.py bdist_wheel
running bdist_wheel
running build
...
running build_calmjs_artifacts
automatically picked registries ['calmjs.module'] for sourcepaths
using loaderplugin registry 'calmjs.rjs.loader_plugin'
...
/src/example.package.egg-info/calmjs_artifacts/example.package.rjs.js
----------------
/tmp/tmpm_2jf151/build/example/package/index.js
...
running install_egg_info
Copying src/example.package.egg-info to build/.../wheel/example.package...
running install_scripts
creating build/.../wheel/example.package-1.0.dist-info/WHEEL

For testing the package artifact, the following entry point should also be specified under the calmjs.artifacts.tests registry, such that running calmjs artifact karma example.package will execute the JavaScript tests declared by example.package against the artifacts that were declared in calmjs.artifacts.

[calmjs.artifacts.tests]
example.package.rjs.js = calmjs.rjs.artifact:test_complete_rjs

When calling calmjs rjs on a package, got ENOENT

Typically this is caused by source files from the source package or registry invoking require a JavaScript module that is not available in the build directory. One possible cause through the calmjs framework is that the Python package failed to declare extras_calmjs that it might require, or that explicit map method and/or source registry that was selected did not result in all required sources be loaded into the build directory.

If the missing source files are intended, applying the --empty or the -e flag to the rjs tool will stub out all the missing modules from the bundle; do note that this will result in the generated artifact bundle not having all the required modules for its execution. The resulting artifact bundle should be used in conjunction with the other artifact bundles that provide the result of the required dependencies.

RJSRuntimeError: unable to locate ‘r.js’

This means the current Node.js environment is missing the requirejs package from npm; either install it manually with it or through calmjs on this package. If a given Python package is required to use requirejs to generate the package, its package_json should declare that, or declare dependency on calmjs.rjs.

UserWarning: Unknown distribution option:

During setup and installation using the development method, if this warning message is shown, please ensure the egg metadata is correctly generated by running python setup.py egg_info in the source directory, as the package calmjs was not available when the setup script was initially executed.