feed-archiver 2.0.0b1

Detailed Description

Installation

Install using any tool for installing standard Python 3 distributions such as pip:

$ sudo pip3 install feed-archiver

Optional shell tab completion is available via argcomplete.

Or use the Docker image. See the example ./docker-compose.yml file for usage details.

Usage

Create a ./.feed-archiver.yml YAML file in a directory to serve as the root directory for all feeds to be archived. The YAML file must have a top-level defaults key whose value is an object defining default or global options. In particular, the base-url key in that section whose value must be a string which defines the external base URL at which the archive is served to clients and is used to assemble absolute URLs where relative URLs can’t be used. The file must also have a top-level feeds key whose value is an array or list of objects defining the remote feeds to archive in this directory. Each feed object must contain a remote-url key whose value is a string that contains the URL of an individual feed to archive. In the simplest form, this can just be a file like so:

defaults:
  base-url: "https://feeds.example.com"
feeds:
  - title: "Garply Podcast Title"
    remote-url: "\
    https://foo-username:secret@grault.example.com\
    /feeds/garply.rss?bar=qux%2Fbaz#corge"
...

Then run the $ feed-archiver command in that directory to update the archive from the current version of the feeds and write an HTML index with links to the archived feeds:

$ cd "/var/www/html/feeds/"
$ feed-archiver
INFO:Retrieving feed URL: https://foo-username:secret@grault.example.com/feeds/garply.rss
...
INFO:Writing HTML index: /var/www/html/feeds/index.html

See also the command-line help for details on options and arguments:

$ feed-archiver --help
usage: feed-archiver [-h] [--log-level {CRITICAL,FATAL,ERROR,WARN,WARNING,INFO,DEBUG,NOTSET}]
                     [--archive-dir [ARCHIVE_DIR]]
                     {update,relink} ...

Archive RSS/Atom syndication feeds and their enclosures and assets.

positional arguments:
  {update,relink}       sub-command
    update              Request the URL of each feed in the archive and update contents accordingly.
    relink              Re-link enclosures to the correct locations for the current configuration.

options:
  -h, --help            show this help message and exit
  --log-level {CRITICAL,FATAL,ERROR,WARN,WARNING,INFO,DEBUG,NOTSET}
                        Select logging verbosity. (default: INFO)
  --archive-dir [ARCHIVE_DIR], -a [ARCHIVE_DIR]
                        the archive root directory into which all feeds, their enclosures and assets
                        will be downloaded (default: .)

To link feed item enclosures into an alternate hierarchy, such as in a media library, add a enclosures key to the feed configuration whose value is an list/array of objects each defining one alternative path to link to the feed item enclosure. Any enclosures defined in the top-level defaults key will be used for all feeds. Configuration to be shared across multiple enclosures configurations may be placed in the corresponding defaults / plugins / enclosures / {plugin_name} object. The actual linking of enclosures is delegated to plugins.

When updating the archive from the remote feed URLs using the $ feed-archiver update sub-command, the enclosures of new items are linked as configured. If the enclosures configuration changes or any of the used plugins refer to external resources that may change, such as the with the sonarr plugin when Sonarr has upgraded or renamed the corresponding video files, use the $ feed-archiver relink command to update all existing links.

Plugins

How feed item enclosures are linked into a media library is delegated to plugins or add-ons. Specifically, the plugin key in a enclosures configuration must be a string which is the name of a Python entry point registered in the feedarchiver.enclosures group. The entry point object reference must point to a feedarchiver.enclosures.EnclosurePlugin subclass which accepts the following arguments when instantiated:

1. parent=dict

   The feedarchiver.archive.Archive if the plugin is configured in defaults for all feeds or the feedarchiver.feed.ArchiveFeed if defined for a specific feed.

2. config=dict

   The Python dictionary object from the de-serialized archive configuration YAML for this specific enclosure configuration.

and whose instances must be callable and accept the following arguments when called:

1. archive_feed=feedarchiver.feed.ArchiveFeed

   The object feedarchiver uses internally to represent an individual feed in the archive.

2. feed_elem=xml.etree.ElementTree.Element, item_elem=xml.etree.ElementTree.Element

   The Python XML element object representing the whole feed, for RSS this is the <channel> child element while for Atom this is the root <feed> element, and the a similar object representing the specific feed item.

3. feed_parsed=feedparser.util.FeedParserDict, item_parsed=feedparser.util.FeedParserDict

   The feedparser object representing the whole feed and the specific feed item.

4. url_result=lxml.etree._ElementUnicodeResult

   The lmxl special string object that contains the URL of the specific enclosure. Can be used to access the specific enclosure element.

5. enclosure_path=pathlib.Path

   The path to the enclosure in the archive as a Python pathlib.Path object with the best guess at the most correct file basename, including the suffix or extension, for the given enclosure. This suffix takes into account the suffix from the enclosure URL, the Content-Type header of the response to the enclosure URL request, and finally the value of any type attribute of the enclosure element XML.

6. match=re.Match

   The Python regular expression match object if the match-pattern matched the string expanded from the Python format string in the match-string key. Particularly useful to designate regular expression groups in the match-pattern and then use the parts of match-string that matched those groups in the format template. If the match-pattern doesn’t match then the enclosure will not be linked. If there are symbolic group names, e.g. (?P<foo_group_name>.*) in the pattern, then they are also available by name in the format string, e.g {foo_group_name.lower()}. If no match-string is provided a default is used combining the feed title, item title, and enclosure basename with extension:

   {utils.quote_sep(feed_parsed.feed.title).strip()}/{utils.quote_sep(item_parsed.title).strip()}{enclosure_path.suffix}

If the plugin returns a value, it must be a list of strings and will be used as the target paths at which to link the enclosure. Relative paths are resolved against the archive root. These paths are not escaped, so if escaping is needed it must be a part of the plugin configuration. If no plugins link a given enclosure, then any plugins whose fallback key is true will be applied. Here’s an example enclosures definition:

defaults:
  base-url: "https://feeds.example.com"
  plugins:
    enclosures:
      sonarr:
        url: "http://localhost:8989"
        api-key: "????????????????????????????????"
  enclosures:
    # Link all feed item enclosures into the media library under the podcasts
    # directory.  Link items into an album directory named by series title if
    # matching.
    - template: "\
      /media/Library/Music/Podcasts\
      /{utils.quote_sep(feed_parsed.feed.title).strip()}\
      /{series_title}\
      /{utils.quote_sep(item_parsed.title).strip()}{enclosure_path.suffix}"
      match-string: "{utils.quote_sep(item_parsed.title).strip()}"
      match-pattern: "\
      (?P<item_title>.+) \\((?P<series_title>.+) \
      (?P<season_number>[0-9])(?P<episode_numbers>[0-9]+[0-9Ee& -]*)\\)"
    # Otherwise link into "self-titled" album directories of the same name as the
    # feed.
    - template: "\
      /media/Library/Music/Podcasts\
      /{utils.quote_sep(feed_parsed.feed.title).strip()}\
      /{utils.quote_sep(feed_parsed.feed.title).strip()}\
      /{utils.quote_sep(item_parsed.title).strip()}{enclosure_path.suffix}"
      fallback: true
feeds:
  - remote-url: "\
    https://foo-username:secret@grault.example.com\
    /feeds/garply.rss?bar=qux%2Fbaz#corge"
    enclosures:
      # This particular feed is a podcast about a TV series/show.  Link enclosures
      # from feed items about an individual episode next to the episode video file as
      # an external audio track using a non-default plugin.
      - plugin: "sonarr"
        match-string: "{utils.quote_sep(item_parsed.title).strip()}"
        match-pattern: "\
        (?P<item_title>.+) \\((?P<series_title>.+) \
        (?P<season_number>[0-9])(?P<episode_numbers>[0-9]+[0-9Ee& -]*)\\)"
        stem-append: "-garply"
...

./Feeds/{utils.quote_sep(feed_parsed.feed.title).strip()}/{utils.quote_sep(item_parsed.title).strip()}{enclosure_path.suffix}

CONTRIBUTING

NOTE: This project is hosted on GitLab. There’s a mirror on GitHub but please use GitLab for reporting issues, submitting PRs/MRs and any other development or maintenance activity.

See the ./CONTRIBUTING.rst file for more details on how to get started with development.