fedfind 4.1.0

Installation and use

fedfind is packaged in the official Fedora and EPEL repositories: to install on Fedora run dnf install fedfind, on RHEL / CentOS with EPEL enabled, run yum install fedfind. You may need to enable the updates-testing repository to get the latest version.

You can visit the fedfind project page on Pagure, and clone with git clone https://pagure.io/fedora-qa/fedfind.git. Tarballs are released through PyPI.

You can use the fedfind CLI from the tarball without installing it, as ./fedfind.py from the root of the tarball (you will need cached_property and six, and argparse on Python 2.6). You can of course copy the Python module anywhere you like and use it in place. To install both CLI and module systemwide, run python setup.py install.

Bugs, pull requests etc.

You can file issues and pull requests on Pagure. Pull requests must be signed off (use the -s git argument). By signing off your pull request you are agreeing to the Developer’s Certificate of Origin:

Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

Release identification

Some usage of fedfind relies on understanding the ‘dist’, ‘release’, ‘milestone’, ‘compose’, ‘respin’ versioning concept, so here is a quick primer. Note that if you intend to use fedfind solely with compose IDs or URLs for modern Pungi 4-generated composes, this section will be of less interest to you and you can probably skip it.

In this section we will write release, milestone, compose, dist quints as (release, milestone, compose, respin, dist) with ’‘indicating an omitted value, e.g. (22, Beta, TC3,’‘, ’Fedora’) or (22, ’‘,’‘,’‘, ’Fedora’). Note ‘dist’ is usually ‘Fedora’, and this is its default value, used when it is not explicitly specified. Conceptually, ‘dist’ should come at the front, but in fact fedfind functions which accept these values tend to place it after the others, for historical reasons (it did not exist when fedfind’s versioning scheme was created).

• Dist is the term fedfind uses for what pungi and productmd refer to as the shortname; when fedfind was created this didn’t really exist, but now the Fedora project produces many more composes than it used to, some with different dists / shortnames. In the compose with compose ID ‘Fedora-Rawhide-20160301.n.0’, the dist / shortname is Fedora; in a compose with compose ID ‘Fedora-Modular-Rawhide-20170816.n.0’, the dist / shortname is Fedora-Modular. Fedora is the dist for all mainline composes, and this is the default value. Fedora-Atomic is the dist for the nightly ‘two week Atomic’ composes, produced once a day for each of the current stable releases, which at present are found only in their initial location as output by release engineering, and not in their mirrored locations. Similarly, Fedora-Docker and Fedora-Cloud are the dists for the nightly Docker and Cloud composes for the current stable releases. So e.g. (27, ’‘, 20180111, 0, ’Fedora-Atomic’) will find the (first) 2018-01-11 nightly two-week Atomic compose for Fedora 27. Fedora-Modular was the dist for modular composes during the Fedora 27 cycle; at the time, Modularity was under development, and for technical reasons, required a separate stream of composes. FedoraRespin is the dist for the current post-release live respin compose in the live-respins directory; there is only ever one of these at a time (and note that these are only semi-official builds, provided by volunteers as a courtesy, they do not have the status of official composes). Any release or compose passed with the dist FedoraRespin is used as a check: if the existing contents of the live-respins directory don’t match the expected release (number) or compose (date), fedfind will raise an exception instead of returning the RespinRelease instance.

• Release is usually a Fedora release number, e.g. 27, 15 or 1. The only non-integer value that is accepted is ‘Rawhide’, for Rawhide nightly composes. These do not, properly speaking, have a definite release number associated with them: Rawhide is a perpetually rolling tree. The canonical versioning for Rawhide nightly composes is (Rawhide, ’‘, YYYYMMDD, N) (where N is the respin number). Note that python-wikitcms uses almost the same versioning concept as fedfind, but Wikitcms ’validation events’ for Rawhide nightly composes do have a release number: this is a property of the validation event, not of the compose. Thus there may be a Wikitcms validation event (24, Rawhide, 20151012, 1, ‘Fedora’) for the fedfind compose (Rawhide, ’’, 20151012, 1). fedfind and python-wikitcms both recognize this case and will attempt to convert each other’s values for convenience.

• Milestone indicates the milestone or the type of nightly compose. Valid milestones for current releases are Beta, RC (or Final, which means the same), Branched, and Production. The Alpha milestone existed until Fedora 25, but Fedora 26 and later releases have no Alpha; no Alpha releases can be found any more, as the Fedora 25 and earlier Alphas are now removed. fedfind will accept Rawhide as a milestone and convert it to the release - so (’‘, Rawhide, YYYYMMDD, N, ’Fedora’) is not exactly valid but will be handled by the CLI and the get_release function and converted to (Rawhide, ’‘, YYYYMMDD, N, ’Fedora’). Stable releases do not have a milestone; (23, RC, ’‘,’‘, ’Fedora’) will be accepted by get_release and the CLI but is treated internally as (23, ’‘,’‘,’‘, ’Fedora’). The Production milestone indicates a so-called ‘production’ compose, which will usually also be an Alpha, Beta or Final ‘candidate’ compose - you may be able to find the same compose in two different places, for instance, with the Production milestone and a date-based compose and respin, or with the Alpha, Beta or Final milestone and a numeric compose and respin. It is approximately the same as the difference between searching for a production compose by compose ID and searching for it by compose label. See more on this in the fedfind / Wikitcms vs. Pungi / productmd versioning section below. Currently, the milestone value has no meaning for dists other than Fedora and Fedora-Modular; in future we may use it to distinguish between nightly and released two-week Atomic composes.

The values Atomic, Docker, Cloud and Respin are accepted for backwards compatibility purposes; they will be translated into a dist value of Fedora-Atomic, Fedora-Docker, Fedora-Cloud or FedoraRespin respectively, and a blank milestone value. Versions of fedfind before 4.0 overloaded the milestone concept to handle these dist values, rather than properly handling them as dists. This functionality may be removed in a future major release.

• Compose is the precise compose identifier (in cases where one is needed). For candidate composes it is always 1. For nightly composes it is a date in YYYYMMDD format. Stable releases and milestone releases do not have a compose.

• Respin is an integer that’s bumped any time a compose which would otherwise have the same version is repeated. The concept is taken from Pungi. If we attempt to build two Rawhide nightly composes on 2016-03-01, for instance, in fedfind’s versioning they are (Rawhide, ’‘, 20160301, 0) and (Rawhide,’‘, 20160301, 1). The corresponding Pungi / productmd ’compose ID’-style versioning is Fedora-Rawhide-20160301.n.0 and Fedora-Rawhide-20160301.n.1.

Note that, as a convenience, fedfind will attempt to detect when a ‘compose’ value is in fact a combined compose and respin, and split it up - so you can specify the compose as 1.7 (for compose 1, respin 7) or 20160330.0 or 20160330.n.0 (both of which will be treated as compose 20160330, respin 0).

Some examples:

• Stable release: (23, ’‘,’‘,’‘, ’Fedora’)

• Alpha release: (24, Alpha, ’‘,’‘, ’Fedora’)

• Candidate: (24, Alpha, 1, 7, ‘Fedora’)

• Branched nightly: (24, Branched, 20160330, 0, ‘Fedora’)

• Branched modular nightly: (24, Branched, 20160330, 0, ‘Fedora-Modular’)

• Rawhide nightly: (Rawhide, ’‘, 20160330, 1, ’Fedora’)

• Rawhide modular nightly: (Rawhide, ’‘, 20160330, 1, ’Fedora-Modular’)

• Two-week Atomic nightly: (25, 20160330, 2, ‘Fedora-Atomic’)

The test suite contains a bunch of tests for get_release() which incidentally may function as further examples of accepted usage. The fedfind CLI and get_release() are designed to guess omitted values in many cases, primarily to aid unattended usage, so e.g. a script can simply specify (’‘, Branched,’‘,’‘, ’Fedora’) to run on the date’s latest branched compose, without having to know what the current Branched release number is. More detailed information on various cases can be found in the get_release() function’s docstring.

CLI

The fedfind CLI command gives you URLs for a release’s images. For instance, fedfind images -r 25 will print the URLs to all Fedora 25 images. You can filter the results in various ways. For more information, use fedfind -h and fedfind images -h.

Python module

The Python module provides access to all fedfind’s capabilities.

import fedfind.release

comp = fedfind.release.get_release(release=27)
print comp.location
for img in comp.all_images:
    print(img['url'])

How it works (roughly)

fedfind has a bunch of knowledge about where Fedora keeps various composes wired in. For Pungi 4 compose types, finding the compose is about all fedfind has to do; it reads all the information about what images are in the compose out from the metadata and exposes it.

For non-Pungi 4 composes (old stable releases) and Pungi 4 composes that are modified and have their metadata stripped (current stable and milestone releases), fedfind uses the imagelist files present on dl.fedoraproject.org, which contain lists of every image file in the entire tree. It finds the images for the specific compose being queried, then produces a path relative to the top of the mirror tree from the result. It can then combine that with a known prefix to produce an HTTPS URL.

For metadata, fedfind first tries to see if the compose was originally produced with Pungi 4 and its metadata is available from PDC. It tries to guess the compose label from the image file names, and then a compose ID from the compose label, and then query PDC for metadata for that compose. If it is successful, it tries to match each image discovered from the imagelist file with an image dict from the original metadata, and combine the two so that the path information is correct but all other information for the image is taken from the original metadata, rather than synthesized by fedfind.

For any discovered image no matching original image dict is found for, and for composes where no original metadata is available at all, fedfind synthesizes productmd-style metadata by analyzing the file path, guessing the properties that can be guessed and omitting others.

In all cases, the result is that metadata and derived properties like all_images are as similar as possible for the different types of compose, so fedfind consumers can interact with non-Pungi 4 composes in the same way as Pungi 4 ones (to the extent of the metadata discovery and synthesis implementations, some stuff just isn’t covered by the synthesis).

Caveats

Credits

This is pretty much all my fault. Note that aside from its external deps, older versions of fedfind (up to 1.1.2) included a copy of the cached_property implementation maintained here by Daniel Greenfield. The bundled copy was dropped with version 1.1.3.

Licensing

Fedora Finder is available under the GPL, version 3 or any later version. A copy is included as COPYING.