cdnupload 1.0.2

Source

source is the source directory of your static files, for example /website/static. Use the optional --include and --exclude arguments, and other arguments described below, to control exactly which files are uploaded.

Destination and dest-args

destination is the destination directory to upload to, or an s3://static-bucket/prefix path for uploading to Amazon S3.

You can also specify a custom scheme for the destination (the scheme:// part of the URL), and cdnupload will try to import a module named cdnupload_scheme (which must be on the PYTHONPATH) and use that module’s Destination class along with the dest_args to create the destination instance.

For example, if you create your own uploader for Google Cloud Storage, you might use the prefix gcs:// and name your module cdnupload_gcs. Then you could use gcs://my/path as a destination, and cdnupload would instantiate the destination instance using cdnupload_gcs.Destination('gcs://bucket', **dest_args).

See the custom destination section for more details about custom Destination subclasses.

dest_args are destination-specific arguments passed as keyword arguments to the Destination class (for example, for s3:// destinations, useful dest args might be max-age=86400 or region-name=us-west-2). Note that hyphens in dest args are converted to underscores, so region-name=us-west-2 becomes region_name='us-west-2'.

For help on destination-specific args, use the dest-help action. For example, to show S3-specific destination args:

cdnupload source s3:// --action=dest-help

Common arguments

-h, --help

Show help about these command-line options and exit.

-a ACTION, --action ACTION

Specify action to perform (the default is to upload):

• upload: Upload files from the source to the destination (but only if they’re not already on the destination).

• delete: Delete unused files at the destination (files no longer present at the source). Be careful with deleting, and use --dry-run to test first!

• dest-help: Show help and available destination arguments for the given Destination class.

-d, --dry-run

Show what the script would upload or delete instead of actually doing it. This option is recommended before running with --action=delete, to ensure you’re not deleting more than you expect.

-e PATTERN, --exclude PATTERN

Exclude source files if their relative path matches the given pattern (according to globbing rules as per Python’s fnmatch). For example, *.txt to include all text files, or __pycache__/* to exclude everything under the pycache directory. This option may be specified multiple times to exclude more than one pattern.

Excludes take precedence over includes, so you can do --include=*.txt but then exclude a specific text file with --exclude=docs/README.txt.

-f, --force

If uploading, force all files to be uploaded even if destination files already exist (useful, for example, when updating headers on Amazon S3).

If deleting, allow the delete to occur even if all files on the destination would be deleted (the default is to prevent that to avoid rm -rf style mistakes).

-i PATTERN, --include PATTERN

If specified, only include source files if their relative path matches the given pattern (according to globbing rules as per Python’s fnmatch). For example, *.png to include all PNG images, or images/* to include everything under the images directory. This option may be specified multiple times to include more than one pattern.

Excludes take precedence over includes, so you can do --include=*.txt but then exclude a specific text file with --exclude=docs/README.txt.

-k FILENAME, --key-map FILENAME

Write key mapping to given file as JSON (but only after successful upload or delete). This file can be used by your web server to produce full CDN URLs for your static files.

Keys in the JSON object are the original paths (relative to the source root), and values in the object are the destination paths (relative to the destination root). For example, the JSON might look like {"script.js": "script_0beec7b5ea3f0fdb.js", ...}.

-l LEVEL, --log-level LEVEL

Set the verbosity of the log output. The level must be one of:

• debug: Most detailed output. Log even files that the script would skip uploading.

• verbose: Verbose output. Log when the script starts, finishes, and when uploads and deletes occur (or would occur if doing a --dry-run).

• default: Default level of log output. Only log when and if the script actually uploads or deletes files (no start or finish logs). If there’s nothing to do, don’t log anything.

• error: Only log errors.

• off: Turn all logging off completely.

-v, --version

Show cdnupload’s version number and exit.

Less common arguments

--continue-on-errors

Continue after upload or delete errors. The script will still log the errors, and it will also return a nonzero exit code if there is at least one error. The default is to stop on the first error.

--dot-names

Include source files and directories that start with . (dot). The default is to skip any files or directories that start with a dot.

--follow-symlinks

Follow symbolic links to directories when walking the source tree. The default is to skip any symbolic links to directories.

--hash-length N

Set the number of hexadecimal characters of the content hash to use for destination key. The default is 16.

--ignore-walk-errors

Ignore errors when walking the source tree (for example, permissions errors on a directory), except for an error when listing the source root directory.

Custom destination

The most likely reason you’ll need to extend cdnupload is to write a custom Destination subclass (if the built-in file or Amazon S3 destinations don’t work for you).

For example, if you’re using a CDN that connects to an origin server called “My Origin”, you might write a custom subclass for uploading to your origin. You’ll need to subclass cdnupload.Destination and implement an initalizer as well as the __str__, walk_keys, upload, and delete methods:

import cdnupload
import myorigin

class Destination(cdnupload.Destination):
    def __init__(self, url, foo='FOO', bar=None):
        """Initialize destination instance with given "My Origin" URL
        (which should be in form my://server/path).
        """
        self.url = url
        self.conn = myorigin.Connection(url, foo=foo, bar=bar)

    def __str__(self):
        """Return a human-readable string for this destination."""
        return self.url

    def walk_keys(self):
        """Yield keys (files) that are currently on the destination."""
        for file in self.conn.get_files():
            yield file.name

    def upload(self, key, source, rel_path):
        """Upload a single file from source at rel_path to destination
        at given key. Normally this function will use the with statement
        "with source.open(rel_path)" to open the source file object.
        """
        with source.open(rel_path) as source_file:
            self.conn.upload_file_obj(source_file, key)

    def delete(self, key):
        """Delete a single file on destination at given key."""
        self.conn.delete_file(key)

To use this custom destination, save your custom code to cdnupload_my.py and ensure the file is somewhere on your PYTHONPATH. Then if you run the cdnupload command-line tool with a destination starting with scheme my://, it will automatically import cdnupload_my and look for a class called Destination, passing the my://server/path URL and any additional destination arguments to your initializer.

Note that when the command-line tool passes additional dest_args to a custom destination, it always passes them as strings (or a list of strings if a dest arg is specified more than once). So if you need an integer or other type, you’ll need to convert it in your __init__ method.

Upload and delete

The top-level functions upload() and delete() drive cdnupload. You can create your own command-line entry point if you want to hook into cdnupload’s Python API. For example, you could make a myupload.py script as follows:

import cdnupload
import hashlib

source = cdnupload.FileSource('/path/to/my/statics',
                              hash_class=hashlib.md5)
destination = cdnupload.S3Destination('s3://bucket/path')
cdnupload.upload(source, destination)

Here we’re doing some light customization of FileSource’s hashing behaviour (changing it from SHA-1 to MD5) and then performing an upload.

The upload() function uploads files from a source to a destination, but only if they’re missing at the destination (according to destination.walk_keys).

The delete() function deletes files from the destination if they’re no longer present at the source (according to source.build_key_map).

Both upload and delete take the same set of arguments:

• source: the source object; either a FileSource instance (or object that implements the same interface), or a string in which case it gets converted to a source via FileSource(source)

• destination: the destination object; either an instance of a concrete Destination subclass, or a string in which case it gets converted to a destination via FileDestination(destination)

• force=False: if True, same as specifying the --force command line option

• dry_run=False: if True, same as specifying the --dry-run command line option

• continue_on_errors=False: if True, same as specifying the --continue-on-errors command line option

Both functions return a Result namedtuple, which has the following attributes:

• source_key_map: the source path to destination key mapping, the same dict returned by source.build_key_map()

• destination_keys: a set containing the destination keys, as returned by destination.walk_keys()

• num_scanned: total number of files scanned (source files when uploading, or destination keys when deleting)

• num_processed: number of files processed (actually uploaded or deleted)

• num_errors: number of errors (useful when continue_on_errors is true)

Custom source

You can also customize the source of the files. There’s currently only one source class, FileSource, which reads files from the filesystem and produces file hashes. You can pass options to the FileSource initializer to control which files it includes or excludes, as well as how it hashes their contents to produce the content-based hash.

The dot_names, include, exclude, ignore_walk_errors, follow_symlinks, and hash_length arguments correspond directly to the --dot-names, --include, --exclude, --ignore-walk-errors, --follow-symlinks, and --hash-length command line options.

Additionally, you can customize FileSource further with the hash_chunk_size and hash_class arguments. The file is read in hash_chunk_size-byte blocks when being hashed, and hash_class is instantiated to generate the hashes (must have a hashlib-style signature).

Or you can subclass FileSource if you want to customize advanced behaviour. For example, you could override FileSource.hash_file()’s handling of text and binary files to treat all files as binary:

from cdnupload import FileSource

class BinarySource(FileSource):
    def hash_file(self, rel_path):
        return FileSource.hash_file(self, rel_path, is_text=False)

To use a subclassed FileSource, you’ll need to call the upload() and delete() functions with your instance directly from Python. It’s not currently possibly to use a subclassed source via the cdnupload command line script.

Logging

cdnupload functions use standard Python logging to log all operations. The name of the logger is cdnupload, and you can control log output format and verbosity (log level) using the Python logging functions.

For example, to log all errors but turn debug-level logging on only for cdnupload logs, you could do this:

import logging

logging.basicConfig(level=logging.ERROR)
logging.getLogger('cdnupload').setLevel(logging.DEBUG)