websnap

Copies files retrieved from an API to a S3 bucket or a local machine.

Installation

pip install websnap

Quickstart

Websnap can be used as a function or as a CLI.

Click here to view a websnap overview diagram.

Function

from websnap import websnap

# Execute websnap using default arguments
websnap()

# Execute websnap passing arguments
websnap(file_logs=True, s3_uploader=True, backup_s3_count=7, early_exit=True)

CLI

To access CLI documentation in terminal execute:

websnap_cli --help

Function Parameters / CLI Options

Function Parameters

CLI Options

Description

Usage: S3 Bucket

Copy files retrieved from an API to a S3 bucket.

Uses the AWS SDK for Python (Boto3) to add and backup API files to a S3 bucket.

Examples

Function

# The s3_uploader argument must be passed as True to copy files to a S3 bucket
# Copies files to a S3 bucket using default argument values
websnap(s3_uploader=True)

# Copies files to a S3 bucket and repeat every 1440 minutes (24 hours), 
# file logs are enabled and only 3 backup files are allowed for each config section
websnap(file_logs=True, s3_uploader=True, backup_s3_count=3, repeat_minutes=1440)

CLI

• The following CLI option must be used to enable websnap to upload files to a S3 bucket: --s3_uploader

• Copies files to a S3 bucket using default argument values:

   websnap_cli --s3_uploader 
  

• Copies files to a S3 bucket and repeat every 1440 minutes (24 hours), file logs are enabled and only 3 backup files are allowed for each config section:

   websnap_cli --file_logs --s3_uploader --backup_s3_count 3 --repeat_minutes 1440
  

Configuration

• A valid .ini configuration file is required for both function and CLI usage.
• Websnap expects the config to be config.ini in the same directory as websnap package is being executed from.
  • However, this can be changed using the config function argument (or CLI --config option).
• S3 config example file: src/websnap/config_templates/s3_config_template.ini
• All keys in tables below are mandatory.

[DEFAULT] Section

Example S3 configuration [DEFAULT] section:

[DEFAULT]
endpoint_url=https://dreamycloud.com
aws_access_key_id=1234567abcdefg
aws_secret_access_key=hijklmn1234567

Other Sections (one per API URL endpoint)

• Each file retrieved from an API requires its own config section!
• The section name be anything, it is suggested to have a name that relates to the copied file.

Example S3 config section configuration with key prefix:

[resource]
url=https://www.example.com/api/resource
bucket=exampledata
key=subdirectory_resource/resource.xml

Example S3 config section configuration without key prefix:

[project]
url=https://www.example.com/api/project
bucket=exampledata
key=project.json

Usage: Local Machine

Copy files retrieved from an API to a local machine.

Examples

Function

# Write files retrieved from an API to local machine using default argument values
websnap()

# Write files retrieved from an API locally and repeats every 60 minutes (1 hour), 
# file logs are enabled
websnap(file_logs=True, repeat_minutes=60)

CLI

• Write copied files to local machine using default argument values:

   websnap_cli 
  

• Write copied files locally and repeats every 60 minutes (1 hour), file logs are enabled:

   websnap_cli --file_logs --repeat_minutes 60
  

Configuration

• A valid .ini configuration file is required for both function and CLI usage.
• Websnap expects the config to be config.ini in the same directory as websnap package is being executed from.
  • However, this can be changed using the config function argument (or CLI --config option).
• Local machine config example file: src/websnap/config_templates/config_template.ini
• Each file that will be retrieved from an API requires its own section.
• If the optional directory key/value pair is omitted then the file will be written in the directory that the program is executed from.

Example local machine configuration section:

[project]
url=https://www.example.com/api/project
file_name=project.json
directory=projectdata

Sections (one per API URL endpoint)

Logs

Websnap supports optional rotating file logs.

• The following CLI option must be used to enable websnap to support rotating file logs: --file_logs
  • In function usage the following argument must be passed to support rotating file logs: file_logs=True
• If log keys are not specified in the configuration [DEFAULT] section then default values in the table below will be used.
• log_when expects a value used by logging module TimedRotatingFileHandler.
• Click here for more information about how to use TimedRotatingFileHandler.
• The default values result in the file logs being rotated once every day and no removal of backup log files.

Configuration

Example log configuration:

[DEFAULT]
log_when=midnight
log_interval=1
log_backup_count=7

[DEFAULT] Section

Minimum Download Size

Websnap supports optionally specifying the minimum download size (in kilobytes) a file must be to copy it from the configured API URL endpoint.

• By default the minimum default minimum size is 0 kb.
  • Unless specified in the configuration this means that a file of any size can be downloaded by websnap.
• Configured minimum download size must be a non-negative integer.
• If the content from the API URL endpoint is less than the configured size:
  • An error will be logged and the program continues to the next config section.
  • If the CLI option --early_exit (or function argument early_exit=True) is enabled then the program will terminate early.

Configuration

Example minimum download size configuration:

[DEFAULT]
min_size_kb=1

[DEFAULT] Section

Author

Rebecca Kurup Buchholz

Purpose

This project was developed to facilitate EnviDat resiliency and support continuous operation during server maintenance.

EnviDat is the environmental data portal of the Swiss Federal Institute for Forest, Snow and Landscape Research WSL.

License

MIT License