Skip to main content

A Python package and CLI for parsing aggregate and forensic DMARC reports

Project description

==========
parsedmarc
==========

|Build Status|

.. image:: https://raw.githubusercontent.com/domainaware/parsedmarc/master/docs/_static/screenshots/dmarc-summary-charts.png
:alt: A screenshot of DMARC summary charts in Kibana
:align: center
:scale: 50
:target: https://raw.githubusercontent.com/domainaware/parsedmarc/master/docs/_static/screenshots/dmarc-summary-charts.png

``parsedmarc`` is a Python module and CLI utility for parsing DMARC reports.
When used with Elasticsearch and Kibana (or Splunk), it works as a self-hosted
open source alternative to commercial DMARC report processing services such
as Agari, Dmarcian, OnDMARC, ProofPoint Email Fraud Defense.

Features
========

* Parses draft and 1.0 standard aggregate/rua reports
* Parses forensic/failure/ruf reports
* Can parse reports from an inbox over IMAP
* Transparently handles gzip or zip compressed reports
* Consistent data structures
* Simple JSON and/or CSV output
* Optionally email the results
* Optionally send the results to Elasticsearch and/or Splunk, for use with premade dashboards
* Optionally send reports to Apache Kafka

Resources
=========

* `Demystifying DMARC`_ - A complete guide to SPF, DKIM, and DMARC

CLI help
========

::

usage: parsedmarc [-h] [-o OUTPUT] [-n NAMESERVERS [NAMESERVERS ...]]
[-t TIMEOUT] [-H HOST] [-u USER] [-p PASSWORD]
[--imap-port IMAP_PORT] [--imap-no-ssl] [-r REPORTS_FOLDER]
[-a ARCHIVE_FOLDER] [-d]
[-E [ELASTICSEARCH_HOST [ELASTICSEARCH_HOST ...]]]
[--elasticsearch-index-prefix ELASTICSEARCH_INDEX_PREFIX]
[--elasticsearch-index-suffix ELASTICSEARCH_INDEX_SUFFIX]
[--hec HEC] [--hec-token HEC_TOKEN] [--hec-index HEC_INDEX]
[--hec-skip-certificate-verification] [--save-aggregate]
[--save-forensic] [-O OUTGOING_HOST] [-U OUTGOING_USER]
[-P OUTGOING_PASSWORD] [--outgoing-port OUTGOING_PORT]
[--outgoing-ssl OUTGOING_SSL] [-F OUTGOING_FROM]
[-T OUTGOING_TO [OUTGOING_TO ...]] [-S OUTGOING_SUBJECT]
[-A OUTGOING_ATTACHMENT] [-M OUTGOING_MESSAGE] [-w] [--test]
[-s] [--debug] [-v]
[file_path [file_path ...]]

Parses DMARC reports

positional arguments:
file_path one or more paths to aggregate or forensic report
files or emails

optional arguments:
-h, --help show this help message and exit
-o OUTPUT, --output OUTPUT
Write output files to the given directory
-n NAMESERVERS [NAMESERVERS ...], --nameservers NAMESERVERS [NAMESERVERS ...]
nameservers to query (Default is Cloudflare's)
-t TIMEOUT, --timeout TIMEOUT
number of seconds to wait for an answer from DNS
(default 2.0)
-H HOST, --host HOST IMAP hostname or IP address
-u USER, --user USER IMAP user
-p PASSWORD, --password PASSWORD
IMAP password
--imap-port IMAP_PORT
IMAP port
--imap-no-ssl Do not use SSL/TLS when connecting to IMAP
-r REPORTS_FOLDER, --reports-folder REPORTS_FOLDER
The IMAP folder containing the reports Default: INBOX
-a ARCHIVE_FOLDER, --archive-folder ARCHIVE_FOLDER
Specifies the IMAP folder to move messages to after
processing them Default: Archive
-d, --delete Delete the reports after processing them
-E [ELASTICSEARCH_HOST [ELASTICSEARCH_HOST ...]], --elasticsearch-host [ELASTICSEARCH_HOST [ELASTICSEARCH_HOST ...]]
A list of one or more Elasticsearch hostnames or URLs
to use (e.g. localhost:9200)
--elasticsearch-index-prefix ELASTICSEARCH_INDEX_PREFIX
Prefix to add in front of the dmarc_aggregate and
dmarc_forensic Elasticsearch index names, joined by _
--elasticsearch-index-suffix ELASTICSEARCH_INDEX_SUFFIX
Append this suffix to the dmarc_aggregate and
dmarc_forensic Elasticsearch index names, joined by _
--hec HEC URL to a Splunk HTTP Event Collector (HEC)
--hec-token HEC_TOKEN
The authorization token for a Splunk HTTP Event
Collector (HEC)
--hec-index HEC_INDEX
The index to use when sending events to the Splunk
HTTP Event Collector (HEC)
--hec-skip-certificate-verification
Skip certificate verification for Splunk HEC
--save-aggregate Save aggregate reports to search indexes
--save-forensic Save forensic reports to search indexes
-O OUTGOING_HOST, --outgoing-host OUTGOING_HOST
Email the results using this host
-U OUTGOING_USER, --outgoing-user OUTGOING_USER
Email the results using this user
-P OUTGOING_PASSWORD, --outgoing-password OUTGOING_PASSWORD
Email the results using this password
--outgoing-port OUTGOING_PORT
Email the results using this port
--outgoing-ssl OUTGOING_SSL
Use SSL/TLS instead of STARTTLS (more secure, and
required by some providers, like Gmail)
-F OUTGOING_FROM, --outgoing-from OUTGOING_FROM
Email the results using this from address
-T OUTGOING_TO [OUTGOING_TO ...], --outgoing-to OUTGOING_TO [OUTGOING_TO ...]
Email the results to these addresses
-S OUTGOING_SUBJECT, --outgoing-subject OUTGOING_SUBJECT
Email the results using this subject
-A OUTGOING_ATTACHMENT, --outgoing-attachment OUTGOING_ATTACHMENT
Email the results using this filename
-M OUTGOING_MESSAGE, --outgoing-message OUTGOING_MESSAGE
Email the results using this message
-w, --watch Use an IMAP IDLE connection to process reports as they
arrive in the inbox
--test Do not move or delete IMAP messages
-s, --silent Only print errors and warnings
--debug Print debugging information
-v, --version show program's version number and exit

SPF and DMARC record validation
===============================

If you are looking for SPF and DMARC record validation and parsing,
check out the sister project,
`checkdmarc <https://domainaware.github.io/checkdmarc/>`_.

Lookalike domains
=================

DMARC protects against domain spoofing, not lookalike domains. for open source
lookalike domain monitoring, check out `DomainAware <https://github.com/seanthegeek/domainaware>`_.

Sample aggregate report output
==============================

Here are the results from parsing the `example <https://dmarc.org/wiki/FAQ#I_need_to_implement_aggregate_reports.2C_what_do_they_look_like.3F>`_
report from the dmarc.org wiki. It's actually an older draft of the the 1.0
report schema standardized in
`RFC 7480 Appendix C <https://tools.ietf.org/html/rfc7489#appendix-C>`_.
This draft schema is still in wide use.

``parsedmarc`` produces consistent, normalized output, regardless of the report
schema.

JSON
----

.. code-block:: json

{
"xml_schema": "draft",
"report_metadata": {
"org_name": "acme.com",
"org_email": "noreply-dmarc-support@acme.com",
"org_extra_contact_info": "http://acme.com/dmarc/support",
"report_id": "9391651994964116463",
"begin_date": "2012-04-27 20:00:00",
"end_date": "2012-04-28 19:59:59",
"errors": []
},
"policy_published": {
"domain": "example.com",
"adkim": "r",
"aspf": "r",
"p": "none",
"sp": "none",
"pct": "100",
"fo": "0"
},
"records": [
{
"source": {
"ip_address": "72.150.241.94",
"country": "US",
"reverse_dns": "adsl-72-150-241-94.shv.bellsouth.net",
"base_domain": "bellsouth.net"
},
"count": 2,
"alignment": {
"spf": true,
"dkim": false,
"dmarc": true
},
"policy_evaluated": {
"disposition": "none",
"dkim": "fail",
"spf": "pass",
"policy_override_reasons": []
},
"identifiers": {
"header_from": "example.com",
"envelope_from": "example.com",
"envelope_to": null
},
"auth_results": {
"dkim": [
{
"domain": "example.com",
"selector": "none",
"result": "fail"
}
],
"spf": [
{
"domain": "example.com",
"scope": "mfrom",
"result": "pass"
}
]
}
}
]
}

CSV
---

::

xml_schema,org_name,org_email,org_extra_contact_info,report_id,begin_date,end_date,errors,domain,adkim,aspf,p,sp,pct,fo,source_ip_address,source_country,source_reverse_dns,source_base_domain,count,disposition,dkim_alignment,spf_alignment,policy_override_reasons,policy_override_comments,envelope_from,header_from,envelope_to,dkim_domains,dkim_selectors,dkim_results,spf_domains,spf_scopes,spf_results
draft,acme.com,noreply-dmarc-support@acme.com,http://acme.com/dmarc/support,9391651994964116463,2012-04-27 20:00:00,2012-04-28 19:59:59,,example.com,r,r,none,none,100,0,72.150.241.94,US,adsl-72-150-241-94.shv.bellsouth.net,bellsouth.net,2,none,fail,pass,,,example.com,example.com,,example.com,none,fail,example.com,mfrom,pass


Sample forensic report output
=============================

I don't have a sample I can share for privacy reasons. If you have a sample
forensic report that you can share publicly, please contact me!

Installation
============

``parsedmarc`` works with Python 2 or 3, but Python 3 is preferred.

On Debian or Ubuntu systems, run:

.. code-block:: bash

$ sudo apt-get install python3-pip


Python 3 installers for Windows and macOS can be found at
https://www.python.org/downloads/

To install or upgrade to the latest stable release of ``parsedmarc`` on
macOS or Linux, run

.. code-block:: bash

$ sudo -H pip3 install -U parsedmarc

Or, install the latest development release directly from GitHub:

.. code-block:: bash

$ sudo -H pip3 install -U git+https://github.com/domainaware/parsedmarc.git

.. note::

On Windows, ``pip3`` is ``pip``, even with Python 3. So on Windows, simply
substitute ``pip`` as an administrator in place of ``sudo pip3``, in the
above commands.

Installation using pypy3
------------------------

For the best possible processing speed, consider using `parsedmarc` inside a ``pypy3``
virtualenv. First, `download the latest version of pypy3`_. Extract it to
``/opt/pypy3`` (``sudo mkdir /opt`` if ``/opt`` does not exist), then create a
symlink:

.. code-block:: bash

$ sudo ln -s /opt/pypy3/bin/pypy3 /usr/local/bin/pypy3

Install ``virtualenv`` on your system:

.. code-block:: bash

$ sudo apt-get install python3-pip
$ sudo -H pip3 install -U virtualenv

Uninstall any instance of ``parsedmarc`` that you may have installed globally

.. code-block:: bash

$ sudo -H pip3 uninstall -y parsedmarc

Next, create a ``pypy3`` virtualenv for parsedmarc


.. code-block:: bash

$ sudo mkdir /opt/venvs
$ cd /opt/venvs
$ sudo -H pip3 install -U virtualenv
$ sudo virtualenv --download -p /usr/local/bin/pypy3 parsedmarc
$ sudo -H /opt/venvs/parsedmarc/bin/pip3 install -U parsedmarc
$ sudo ln -s /opt/venvs/parsedmarc/bin/parsedmarc /usr/local/bin/parsedmarc

To upgrade ``parsedmarc`` inside the virtualenv, run:


.. code-block:: bash

$ sudo -H /opt/venvs/parsedmarc/bin/pip3 install -U parsedmarc

Or, install the latest development release directly from GitHub:

.. code-block:: bash

$ sudo -H /opt/venvs/parsedmarc/bin/pip3 install -U git+https://github.com/domainaware/parsedmarc.git

Optional dependencies
---------------------

If you would like to be able to parse emails saved from Microsoft Outlook
(i.e. OLE .msg files), install ``msgconvert``:

On Debian or Ubuntu systems, run:

.. code-block:: bash

$ sudo apt-get install libemail-outlook-message-perl

DNS performance
---------------

You can often improve performance by providing one or more local nameservers
to the CLI or function calls, as long as those nameservers return the same
records as the public DNS.


.. note::

If you do not specify any nameservers, Cloudflare's public nameservers are
used by default, **not the system's default nameservers**.

This is done to avoid a situation where records in a local nameserver do
not match records in the public DNS.

Testing multiple report analyzers
---------------------------------

If you would like to test parsedmarc and another report processing solution
at the same time, you can have up to two mailto URIs each in the rua and ruf
tags tgs in your DMARC record, separated by commas.

Documentation
=============

https://domainaware.github.io/parsedmarc

Bug reports
===========

Please report bugs on the GitHub issue tracker

https://github.com/domainaware/parsedmarc/issues

.. |Build Status| image:: https://travis-ci.org/domainaware/parsedmarc.svg?branch=master
:target: https://travis-ci.org/domainaware/parsedmarc

.. _Demystifying DMARC: https://seanthegeek.net/459/demystifying-dmarc/

.. _download the latest version of pypy3: https://pypy.org/download.html#default-with-a-jit-compiler


Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

parsedmarc-4.2.1.tar.gz (33.3 kB view details)

Uploaded Source

Built Distribution

parsedmarc-4.2.1-py3-none-any.whl (33.5 kB view details)

Uploaded Python 3

File details

Details for the file parsedmarc-4.2.1.tar.gz.

File metadata

  • Download URL: parsedmarc-4.2.1.tar.gz
  • Upload date:
  • Size: 33.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/40.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.6.6

File hashes

Hashes for parsedmarc-4.2.1.tar.gz
Algorithm Hash digest
SHA256 d2e923ce4eaedb8ca91f5e5b9afe54c47929e0f051992d0273d9dcd8cf3f8adb
MD5 e5d2741a4ef1a564b03156e801b599a8
BLAKE2b-256 65f230e2a0264695affbcfe952aceb128f2c81707794e6409fe5623acc6a810d

See more details on using hashes here.

File details

Details for the file parsedmarc-4.2.1-py3-none-any.whl.

File metadata

  • Download URL: parsedmarc-4.2.1-py3-none-any.whl
  • Upload date:
  • Size: 33.5 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/40.2.0 requests-toolbelt/0.8.0 tqdm/4.23.4 CPython/3.6.6

File hashes

Hashes for parsedmarc-4.2.1-py3-none-any.whl
Algorithm Hash digest
SHA256 c738564bb786444ac501253a2fb41fe74d4253a01102d45ba2ba7abda8dd15f8
MD5 9d3754ee2820d72cebd45dbc985f391a
BLAKE2b-256 95583b2f85c0f0c539de07a769877f908670d9ee2892513bef0d5ae93b684fad

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page