Skip to main content

Simple tool to compare directory contents.

Project description

Version Build status Coverage License Documentation status

Directory Content Difference

This project provides simple tools to compare the content of a directory against a reference directory.

This is useful to check the results of a process that generate several files, like a luigi workflow for example.

Installation

This package should be installed using pip:

pip install dir-content-diff

Usage

The dir-content-diff package introduces a framework to compare two directories. A comparator is associated to each file extension and then each file in the reference directory is compared to the file with the same relative path in the compared directory. By default, a few comparators are provided for usual files but others can be associated to new file extensions or can even replace the default ones. The comparators should be able to report the differences between two files accurately, reporting which elements are different among the data. When an extension has no comparator associated, a default comparator is used which just compares the whole binary data of the files, so it is not able to report which values are different.

Compare two directories

If one wants to compare two directories with the following structures:

└── reference_dir
    ├── sub_dir_1
    |   ├── sub_file_1.a
    |   └── sub_file_2.b
    └── file_1.c
└── compared_dir
    ├── sub_dir_1
    |   ├── sub_file_1.a
    |   └── sub_file_2.b
    |   └── sub_file_3.b
    └── file_1.c

These two directories can be compared with the following code:

import dir_content_diff

dir_content_diff.compare_trees("reference_dir", "compared_dir")

This code will return an empty dictionary because no difference was detected.

If reference_dir/file_1.c is the following JSON-like file:

{
    "a": 1,
    "b": [1, 2]
}

And compared_dir/file_1.c is the following JSON-like file:

{
    "a": 2,
    "b": [10, 2, 0]
}

The following code registers the JsonComparator for the file extension .c and compares the two directories:

import dir_content_diff

dir_content_diff.register_comparator(".c", dir_content_diff.JsonComparator())
dir_content_diff.compare_trees("reference_dir", "compared_dir")

The previous code will output the following dictionary:

{
    'file_1.c': (
        'The files \'reference_dir/file_1.c\' and \'compared_dir/file_1.c\' are different:\n'
        'Added the value(s) \'{"2": 0}\' in the \'[b]\' key.\n'
        'Changed the value of \'[a]\' from 1 to 2.\n'
        'Changed the value of \'[b][0]\' from 1 to 10.'
    )
}

It is also possible to check whether the two directories are equal or not with the following code:

import dir_content_diff

dir_content_diff.register_comparator(".c", dir_content_diff.JsonComparator())
dir_content_diff.assert_equal_trees("reference_dir", "compared_dir")

Which will output the following AssertionError:

AssertionError: The files 'reference_dir/file_1.c' and 'compared_dir/file_1.c' are different:
Added the value(s) '{"2": 0}' in the '[b]' key.
Changed the value of '[a]' from 1 to 2.
Changed the value of '[b][0]' from 1 to 10.

Finally, the comparators have parameters that can be passed either to be used for all files of a given extension or only for a specific file:

import dir_content_diff

# Get the default comparators
comparators = dir_content_diff.get_comparators()

# Replace the comparators for JSON files to perform the comparison with a given tolerance
comparators[".json"] = dir_content_diff.JsonComparator(default_diff_kwargs={"tolerance": 0.1})

# Use a specific tolerance for the file ``sub_dir_1/sub_file_1.a``
# In this case, the kwargs are used to compute the difference by default, except the following
# specific kwargs: ``return_raw_diffs``, ``load_kwargs``, ``format_data_kwargs``, ``filter_kwargs``,
# ``format_diff_kwargs``, ``sort_kwargs``, ``concat_kwargs`` and ``report_kwargs``.
specific_args = {"sub_dir_1/sub_file_1.a": {"tolerance": 0.5}}

dir_content_diff.assert_equal_trees(
    "reference_dir",
    "compared_dir",
    comparators=comparators,
    specific_args=specific_args,
)

Each comparator has different arguments that are detailed in the documentation.

Export formatted data

Some comparators have to format the data before comparing them. For example, if one wants to compare data with file paths inside, it's likely that only a relative part of these paths are relevant, not the entire absolute paths. To do this, a specific comparator can be defined with a custom format_data() method which is automatically called after the data are loaded but before the data are compared. It is then possible to export the data just after they have been formatted for check purpose for example. To do this, the export_formatted_files argument of the dir_content_diff.compare_trees and dir_content_diff.assert_equal_trees functions can be set to True. Thus all the files processed by a comparator with a save() method will be exported to a new directory. This new directory is the same as the compared directory to which a suffix is added. By default, the suffix is _FORMATTED, but it can be overridden by passing a non-empty string to the export_formatted_files argument.

Pytest plugin

This package can be used as a pytest plugin. When pytest is run and dir-content-diff is installed, it is automatically detected and registered as a plugin. It is then possible to trigger the export of formatted data with the following pytest option: --dcd-export-formatted-data. It is also possible to define a custom suffix for the new directory with the following option: --dcd-export-suffix.

Funding & Acknowledgment

The development of this software was supported by funding to the Blue Brain Project, a research center of the École polytechnique fédérale de Lausanne (EPFL), from the Swiss government’s ETH Board of the Swiss Federal Institutes of Technology.

For license and authors, see LICENSE.txt and AUTHORS.md respectively.

Copyright © 2021-2022 Blue Brain Project/EPFL

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dir-content-diff-1.4.0.tar.gz (46.8 kB view details)

Uploaded Source

Built Distribution

dir_content_diff-1.4.0-py3-none-any.whl (20.9 kB view details)

Uploaded Python 3

File details

Details for the file dir-content-diff-1.4.0.tar.gz.

File metadata

  • Download URL: dir-content-diff-1.4.0.tar.gz
  • Upload date:
  • Size: 46.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.11.2

File hashes

Hashes for dir-content-diff-1.4.0.tar.gz
Algorithm Hash digest
SHA256 ed8358c79a8758349f83c503d03e1244a36e7ac84c3c63311c5f5c76ba2a421e
MD5 57c877df6a9b39f791017f9b4f06cc30
BLAKE2b-256 73f801a5b09548079ee6631f19b5be12442e8dca56f1a5e3426700b12058707f

See more details on using hashes here.

Provenance

File details

Details for the file dir_content_diff-1.4.0-py3-none-any.whl.

File metadata

File hashes

Hashes for dir_content_diff-1.4.0-py3-none-any.whl
Algorithm Hash digest
SHA256 9864bebbea8c5aed4ea78fb75df52161944ebd04de22deea2f150f46aa7f2043
MD5 e1a5e22e59e349a1c943ed9857b1c7d1
BLAKE2b-256 8a64495690185496c28c1f59d91d14e23c876ffe66be81d21d6954b88341d348

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page