Cloud Custodian - Cloud Watch Log S3 exporter
Project description
c7n-log-exporter: Cloud watch log exporter automation
A small serverless app to archive cloud logs across accounts to an archive bucket. It utilizes cloud log export to s3 feature for historical exports.
It also supports kinesis streams / firehose to move to realtime exports in the same format as the periodic historical exports.
Features
- Log group filtering by regex
- Incremental support based on previously synced dates
- Incremental support based on last log group write time
- Cross account via sts role assume
- Lambda and CLI support.
- Day based log segmentation (output keys look like $prefix/$account_id/$group/$year/$month/$day/$export_task_uuid/$stream/$log)
Assumptions
- The archive bucket has already has appropriate bucket policy permissions. For details see: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/S3ExportTasks.html#S3Permissions
- Default periodicity for log group archival into s3 is daily.
- Exporter is run with account credentials that have access to the archive s3 bucket.
- Catch up archiving is not run in lambda (do a cli run first)
Cli usage
make install
You can run on a single account / log group via the export subcommand
c7n-log-exporter export --help
Config format
To ease usage when running across multiple accounts, a config file can be specified, as an example.
destination:
bucket: custodian-log-archive
prefix: logs2
accounts:
- name: custodian-demo
role: "arn:aws:iam::111111111111:role/CloudCustodianRole"
groups:
- "/aws/lambda/*"
- "vpc-flow-logs"
Multiple accounts via cli
To run on the cli across multiple accounts, edit the config.yml to specify multiple accounts and log groups.
c7n-log-exporter run --config config.yml
Serverless Usage
Edit config.yml to specify the accounts, archive bucket, and log groups you want to use.
make install
make deploy
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for c7n_logexporter-0.4.5-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | b965b02bad44aadd8ada13bcf0d1e0bfff551156875132893a10bba389ae606f |
|
| MD5 | b8c8d0059b16e027ca8ce3c88e1a860c |
|
| BLAKE2b-256 | d7ba6ead4481c1b386fb91edb37593f84379e506e033121b60a1d8648d3daf0e |
