fuzzycat: bibliographic fuzzy matching for fatcat.wiki

This Python library contains routines for finding near-duplicate bibliographic entities (primarily research papers), and estimating whether two metadata records describe the same work (or variations of the same work). Some routines are designed to work "offline" with batches of billions of sorted metadata records, and others are designed to work "online" making queries against hosted web services and catalogs.

fuzzycat was originally developed by Martin Czygan at the Internet Archive, and is used in the construction of a citation graph and to identify duplicate records in the fatcat.wiki catalog and scholar.archive.org search index.

DISCLAIMER: this tool is still under development, as indicated by the "0" major version. The interface, semantics, and behavior are likely to be tweaked.

Quickstart

Inside a virtualenv (or similar), install with pip:

pip install fuzzycat

The fuzzycat.simple module contains high-level helpers which query Internet Archive hosted services:

import elasticsearch
from fuzzycat.simple import *

es_client = elasticsearch.Elasticsearch("https://search.fatcat.wiki:443")

# parses reference using GROBID (at https://grobid.qa.fatcat.wiki),
# then queries Elasticsearch (at https://search.fatcat.wiki),
# then scores candidates against latest catalog record fetched from
#  https://api.fatcat.wiki
best_match = closest_fuzzy_unstructured_match(
    """Cunningham HB, Weis JJ, Taveras LR, Huerta S. Mesh migration following abdominal hernia repair: a comprehensive review. Hernia. 2019 Apr;23(2):235-243. doi: 10.1007/s10029-019-01898-9. Epub 2019 Jan 30. PMID: 30701369.""",
    es_client=es_client)

print(best_match)
# FuzzyReleaseMatchResult(status=<Status.EXACT: 'exact'>, reason=<Reason.DOI: 'doi'>, release={...})

# same as above, but without the GROBID parsing, and returns multiple results
matches = close_fuzzy_biblio_matches(
    dict(
        title="Mesh migration following abdominal hernia repair: a comprehensive review",
        first_author="Cunningham",
        year=2019,
        journal="Hernia",
    ),
    es_client=es_client,
)

A CLI tool is included for processing records in UNIX stdin/stdout pipelines:

# print usage
python -m fuzzycat

Features and Use-Cases

The refcat project builds on top of this library to build a citation graph by processing billions of structured and unstructured reference records extracted from scholarly papers (note: jfor performance critical parts, some code has been ported to Go, albeit the test suite is shared between the Python and Go implementations).

Automated imports of metadata records into the fatcat catalog use fuzzycat to filter new metadata which look like duplicates of existing records from other sources.

In conjunction with standard command-line tools (like sort), fatcat bulk metadata snapshots can be clustered and reduced into groups to flag duplicate records for merging.

Extracted reference strings from any source (webpages, books, papers, wikis, databases, etc) can be resolved against the fatcat catalog of scholarly papers.

Support and Acknowledgements

Work on this software received support from the Andrew W. Mellon Foundation through multiple phases of the "Ensuring the Persistent Access of Open Access Journal Literature" project (see original announcement).

Additional acknowledgements at fatcat.wiki.