dandi-s3-log-parser 0.1.0

DANDI S3 Log Parser

Simple reductions of consolidated S3 logs (consolidation step not included in this repository) into minimal information for public sharing and plotting.

Developed for the DANDI Archive.

A single line of a raw S3 log file can be between 400-1000+ bytes. Some of the busiest daily logs on the archive can have around 5,014,386 lines. As of summer 2024, there are more than 6 TB of log files collected.

This parser can reduce these to tens of GB of consolidated and anonymized usage data, which is much more manageable for sharing and plotting.

Usage

To iteratively parse all historical logs all at once (parallelization with 10-15 total GB recommended):

parse_all_dandi_raw_s3_logs \
  --base_raw_s3_log_folder_path < base log folder > \
  --parsed_s3_log_folder_path < output folder > \
  --excluded_log_files < any log files to skip> \
  --excluded_ips < comma-separated list of known IPs to exclude > \
  --maximum_number_of_workers < number of CPUs to use > \
  --maximum_buffer_size_in_bytes < approximate amount of RAM to use >

For example, on Drogon:

parse_all_dandi_raw_s3_logs \
  --base_raw_s3_log_folder_path /mnt/backup/dandi/dandiarchive-logs \
  --parsed_s3_log_folder_path /mnt/backup/dandi/dandiarchive-logs-cody/parsed_7_13_2024/GET_per_asset_id \
  --excluded_log_files /mnt/backup/dandi/dandiarchive-logs/stats/start-end.log \
  --excluded_ips < Drogons IP > \
  --maximum_number_of_workers 3 \
  --maximum_buffer_size_in_bytes 15000000000

To parse only a single log file at a time, such as in a CRON job:

parse_dandi_raw_s3_log \
  --raw_s3_log_file_path < s3 log file path > \
  --parsed_s3_log_folder_path < output folder > \
  --excluded_ips < comma-separated list of known IPs to exclude >

Submit line decoding errors

Please email line decoding errors collected from your local config file to the core maintainer before raising issues or submitting PRs contributing them as examples, to more easily correct any aspects that might require anonymization.

Developer notes

.log file suffixes should typically be ignored when working with Git, so when committing changes to the example log collection, you will have to forcibly include it with

git add -f <example file name>.log