astropy-helpers 3.1

astropy-helpers
===============

.. image:: https://travis-ci.org/astropy/astropy-helpers.svg
:target: https://travis-ci.org/astropy/astropy-helpers

.. image:: https://ci.appveyor.com/api/projects/status/rt9161t9mhx02xp7/branch/master?svg=true
:target: https://ci.appveyor.com/project/Astropy/astropy-helpers

.. image:: https://codecov.io/gh/astropy/astropy-helpers/branch/master/graph/badge.svg
:target: https://codecov.io/gh/astropy/astropy-helpers

**Warning:** Please note that version ``v3.0`` and later of ``astropy-helpers``
requires Python 3.5 or later. If you wish to maintain Python 2 support
for your package that uses ``astropy-helpers``, then do not upgrade the
helpers to ``v3.0+``. We will still provide Python 2.7 compatible
releases on the ``v2.0.x`` branch during the lifetime of the ``astropy``
core package LTS of ``v2.0.x``.

About
-----

This project provides a Python package, ``astropy_helpers``, which includes
many build, installation, and documentation-related tools used by the Astropy
project, but packaged separately for use by other projects that wish to
leverage this work. The motivation behind this package and details of its
implementation are in the accepted
`Astropy Proposal for Enhancement (APE) 4 <https://github.com/astropy/astropy-APEs/blob/master/APE4.rst>`_.

``astropy_helpers`` includes a special "bootstrap" module called
``ah_bootstrap.py`` which is intended to be used by a project's setup.py in
order to ensure that the ``astropy_helpers`` package is available for
build/installation.

As described in `APE4 <https://github.com/astropy/astropy-APEs/blob/master/APE4.rst>`_, the version
numbers for ``astropy_helpers`` follow the corresponding major/minor version of
the `astropy core package <http://www.astropy.org/>`_, but with an independent
sequence of micro (bugfix) version numbers. Hence, the initial release is 0.4,
in parallel with Astropy v0.4, which will be the first version of Astropy to
use ``astropy-helpers``.

For examples of how to implement ``astropy-helpers`` in a project,
see the ``setup.py`` and ``setup.cfg`` files of the
`Affiliated package template <https://github.com/astropy/package-template>`_.

What does astropy-helpers provide?
----------------------------------

Astropy-helpers' big-picture purpose is to provide customization to Python's
packaging infrastructure process in ways that the Astropy Project has found to
help simplifying the developint and releasing packages. This is primarily
build around ``setup.py`` commands, as outlined below, as well as code to help
manage version numbers and better control the build process of larger packages.

Custom setup.py commands
^^^^^^^^^^^^^^^^^^^^^^^^

The main part of astropy-helpers is to provide customized setuptools commands.
For example, in a package that uses astropy-helpers, the following command
will be available::

python setup.py build_docs

and this command is implemented in astropy-helpers. To use the custom
commands described here, add::

from astropy_helpers.setup_helpers import register_commands

to your ``setup.py`` file, then do::

cmdclassd = register_commands(NAME, VERSION, RELEASE)

where ``NAME`` is the name of your package, ``VERSION`` is the full version
string, and ``RELEASE`` is a boolean value indicating whether the version is
a stable released version (``True``) or a developer version (``False``).
Finally, pass ``cmdclassd`` to the ``setup`` function::

setup(...,
cmdclass=cmdclassd)

The commands we provide or customize are:

**python setup.py test**

This command will automatically build the package, install it to a temporary
directory, and run the tests using `pytest <http://pytest.org/>`_ on this
installed version. Note that the bulk of this command is actually defined
in ``astropy.tests.command.AstropyTest``, because that allows the test
machinery to operate outside a setuptools command. This, here we
simply define the custom
setuptools command.

**python setup.py sdist**

We redefine ``sdist`` to use the version from distutils rather than from
setuptools, as the setuptools version requires duplication of information
in ``MANIFEST.in``.

**python setup.py build_docs**

This command will automatically build the package, then run sphinx to build
the documentation. This makes development much easier because it ensures
sphinx extensions that use the package's code to make documentation are
actually using the in-development version of the code. Sphinx itself
provides a custom setuptools command, which we
expand with the following options:

* ``-w``: set the return code to 1 if there are any warnings during the build
process.

* ``-l``: completely clean previous builds, including files generated by
the sphinx-automodapi package (which creates API pages for different
functions/classes).

* ``-n``: disable the intersphinx option.

* ``-o``: open the documentation in a browser if a build finishes successfully.

In addition, ``build_docs`` will automatically download and temporarily install
sphinx-astropy (which is a meta-package that
provides standardized configuration and documentation dependencies for astropy
packages) if it isn't already installed. Temporary installation means that the
package will be installed into an ``.eggs`` directory in the current working
directory, and it will only be available for the duration of the call to
``build_docs``.

**python setup.py build_ext**

This is also used when running ``build`` or ``install``. We add several features
compared to the default ``build_ext`` command:

* For packages with C/Cython extensions, we create a ``packagename._compiler``
submodule that contains information about the compilers used.

* Packages that need to build C extensions using the Numpy C API, we allow
those packages to define the include path as ``'numpy'`` as opposed to having
to import Numpy and call ``get_include``. The goal is to solve the issue that
if one has to import Numpy to define extensions, then Numpy has to be
installed/available before the package is installed, which means that one
needs to install Numpy in a separate installation step.

* We detect broken compilers and replace them with other compilers on-the-fly
unless the compiler is explicitly specified with the ``CC`` environment
variable.

* If Cython is not installed, then we automatically check for generated C files
(which are normally present in the stable releases) and give a nice error
if these are not found.

Version helpers
^^^^^^^^^^^^^^^^

Another piece of functionality we provide in astropy-helpers is the ability
to generate a ``packagename.version`` file that includes functions that
automatically set the version string for developer versions, to e.g.
``3.2.dev22213`` so that each developer version has a unique number (although
note that branches an equal number of commits away from the master branch will
share the same version number). To use this, import::

from astropy_helpers.git_helpers import get_git_devstr

in your ``setup.py`` file, and you will then be able to use::

VERSION += get_git_devstr()

where ``VERSION`` is a version string without any developer version suffix.

We then also provide a function that generates a ``version.py`` file inside your
package (which can then be imported as ``packagename.version``) that contains
variables such as ``major``, ``minor``, and ``bugfix``, as well as
``version_info`` (a tuple of the previous three values), a ``release`` flag that
indicates whether we are using a stable release, and several other complementary
variables. To use this, import::

from astropy_helpers.version_helpers import generate_version_py

in your ``setup.py`` file, and call::

generate_version_py(NAME, VERSION, RELEASE, uses_git=not RELEASE)

where ``NAME`` is the name of your package, ``VERSION`` is the full version string
(including any developer suffix), ``RELEASE`` indicates whether the version is a
stable or developer version, and ``uses_git`` indicates whether we are in a git
repository (using ``not RELEASE`` is sensible since git is not available in a
stable release).

Collecting package information
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The ``setup`` function from setuptools can take a number of options that indicate
for example what extensions to build, and what package data to include. However,
for large packages this can become cumbersome. We therefore provide a mechanism
for defining extensions and package data inside individual sub-packages. To do
this, you can create ``setup_package.py`` files anywhere in your package, and
these files can include one or more of the following functions:

* ``get_package_data``:
This function, if defined, should return a dictionary mapping the name of
the subpackage(s) that need package data to a list of data file paths
(possibly including wildcards) relative to the path of the package's source
code. e.g. if the source distribution has a needed data file
``astropy/wcs/tests/data/3d_cd.hdr``, this function should return
``{'astropy.wcs.tests':['data/3d_cd.hdr']}``. See the ``package_data``
option of the :func:`distutils.core.setup` function.

It is recommended that all such data be in a directory named ``data`` inside
the package within which it is supposed to be used. This package data
should be accessed via the ``astropy.utils.data.get_pkg_data_filename`` and
``astropy.utils.data.get_pkg_data_fileobj`` functions.

* ``get_extensions``:
This provides information for building C or Cython extensions. If defined,
it should return a list of ``distutils.core.Extension`` objects.

* ``get_build_options``:
This function allows a package to add extra build options. It
should return a list of tuples, where each element has:

- *name*: The name of the option as it would appear on the
commandline or in the ``setup.cfg`` file.

- *doc*: A short doc string for the option, displayed by
``setup.py build --help``.

- *is_bool* (optional): When `True`, the option is a boolean
option and doesn't have an associated value.

Once an option has been added, its value can be looked up using
``astropy_helpers.setup_helpers.get_distutils_build_option``.

* ``get_external_libraries``:
This function declares that the package uses libraries that are
included in the astropy distribution that may also be distributed
elsewhere on the users system. It should return a list of library
names. For each library, a new build option is created,
``'--use-system-X'`` which allows the user to request to use the
system's copy of the library. The package would typically call
``astropy_helpers.setup_helpers.use_system_library`` from its
``get_extensions`` function to determine if the package should use
the system library or the included one.

* ``get_entry_points()``:
This function can returns a dict formatted as required by
the ``entry_points`` argument to ``setup()``.

With these files in place, you can then add the following to your ``setup.py``
file::

from astropy_helpers.setup_helpers import get_package_info

...

package_info = get_package_info()

...

setup(..., **package_info)

OpenMP helpers
^^^^^^^^^^^^^^

We provide a helper function ``add_openmp_flags_if_available`` that can be used
to automatically add OpenMP flags for C/Cython extensions, based on whether
OpenMP is available and produces executable code. To use this, edit the
``setup_package.py`` file where you define a C extension, import the helper
function::

from astropy_helpers.openmp_helpers import add_openmp_flags_if_available

then once you have defined the extension and before returning it, use it as::

extension = Extension(...)

add_openmp_flags_if_available(extension)

return [extension]

Using astropy-helpers
---------------------

The easiest way to get set up with astropy-helpers in a new package is to use
the `package-template <http://docs.astropy.org/projects/package-template>`_
that we provide. This template is specifically designed for use with the helpers,
so using it avoids some of the tedium of setting up the heleprs.

However, we now go through the steps of adding astropy-helpers
as a submodule to a package in case you wish to do so manually. First, add
astropy-helpers as a submodule at the root of your repository::

git submodule add git://github.com/astropy/astropy-helpers astropy_helpers

Then go inside the submodule and check out a stable version of astropy-helpers.
You can see the available versions by running::

$ cd astropy_helpers
$ git tag
...
v2.0.6
v2.0.7
...
v3.0.1
v3.0.2

If you want to support Python 2, pick the latest v2.0.x version (in the above
case ``v2.0.7``) and if you don't need to support Python 2, just pick the latest
stable version (in the above case ``v3.0.2``). Check out this version with e.g.::

$ git checkout v3.0.2

Then go back up to the root of your repository and copy the ``ah_bootstrap.py``
file from the submodule to the root of your repository::

$ cd ..
$ cp astropy_helpers/ah_bootstrap.py .

Finally, add::

import ah_bootstrap

at the top of your ``setup.py`` file. This will ensure that ``astropy_helpers``
is now available to use in your ``setup.py`` file. Finally, add then commit your
changes::

git add astropy_helpers ah_bootstrap.py setup.py
git commit -m "Added astropy-helpers"

Updating astropy-helpers
------------------------

If you would like the Astropy team to automatically open pull requests to update
astropy-helpers in your package, then see the instructions `here
<https://github.com/astropy/astropy-procedures/blob/master/update-packages/README.md>`_.

To instead update astropy-helpers manually, go inside the submodule and do::

cd astropy_helpers
git fetch origin

Then checkout the version you want to use, e.g.::

git checkout v3.0.3

Go back up to the root of the repository and update the ``ah_bootstap.py`` file
too, then add your changes::

cp astropy_helpers/ah_bootstrap.py .
git add astropy_helpers ah_bootstrap.py
git commit ...