Skip to main content

Command-line evaluation tools for named entity linking and (cross-document) coreference resolution

Project description

Entity linking evaluation
=========================

Python command-line evaluation scripts for `TAC <http://www.nist.gov/tac/>`__
entity linking and related wikification, named entity disambiguation, and
within- and cross-document coreference tasks.

|version| |licence| |py-versions|

|issues| |build| |docs| |coverage|

It aims for **fast** and **flexible** coreference resolution and
**sophisticated** named entity recognition evaluation, such as partial scores
for partial overlap between gold and system mentions. CEAF, in particular, is
much faster to calculate here than in the `CoNLL-11/12 scorer
<https://github.com/conll/reference-coreference-scorers>`__. It boasts features
such as configurable metrics; accounting for or ignoring cross-document
coreference (see the ``evaluate --by-doc`` flag); plotting to compare
evaluation by system, measure and corpus subset; and bootstrap-based confidence
interval calculation for document-wise evaluation metrics.

Requires that ``python`` (2.7, with Py3k support experimental/partial)
be installed on your system with ``numpy`` (and preferably ``scipy`` for
fast CEAF calculation) and ``joblib``. ``matplotlib`` is required for
the ``plot-systems`` command.

See a list of commands with:

.. code:: bash

./nel --help

Or install onto your Python path (e.g. with
``pip install git+https://github.com/wikilinks/neleval``) then

.. code:: bash

python -m neleval --help

TAC-KBP 2014 EDL quickstart
===========================

.. code:: bash

./scripts/run_tac14_evaluation.sh \
/path/to/gold.xml \ # TAC14 gold standard queries/mentions
/path/to/gold.tab \ # TAC14 gold standard link and nil annotations
/system/output/directory \ # directory containing (only) TAC14 system output files
/script/output/directory \ # directory to which results are written
number_of_jobs # number of jobs for parallel mode

Each file in in the system output directory is scored against gold.tab.

Similar facility is available for TAC-KBP'15 EDL.

More details
============

See `the documentation <https://neleval.readthedocs.io>`__ for more
details.


.. |py-versions| image:: https://img.shields.io/pypi/pyversions/neleval.svg
:alt: Python versions supported

.. |version| image:: https://badge.fury.io/py/neleval.svg
:alt: Latest version on PyPi
:target: https://badge.fury.io/py/neleval

.. |build| image:: https://travis-ci.org/wikilinks/neleval.svg?branch=master
:alt: Travis CI build status
:scale: 100%
:target: https://travis-ci.org/wikilinks/neleval

.. |issues| image:: https://img.shields.io/github/issues/wikilinks/neleval.svg
:alt: Issue tracker
:target: https://github.com/wikilinks/neleval

.. |coverage| image:: https://coveralls.io/repos/github/wikilinks/neleval/badge.svg
:alt: Test coverage
:target: https://coveralls.io/github/wikilinks/neleval

.. |docs| image:: https://readthedocs.org/projects/neleval/badge/?version=latest
:alt: Documentation Status
:scale: 100%
:target: https://neleval.readthedocs.io/en/latest/?badge=latest

.. |licence| image:: https://img.shields.io/badge/Licence-Apache%202.0-blue.svg
:target: https://opensource.org/licenses/Apache-2.0

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

neleval-3.1.1.tar.gz (51.6 kB view details)

Uploaded Source

File details

Details for the file neleval-3.1.1.tar.gz.

File metadata

  • Download URL: neleval-3.1.1.tar.gz
  • Upload date:
  • Size: 51.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.32.1 CPython/3.7.3

File hashes

Hashes for neleval-3.1.1.tar.gz
Algorithm Hash digest
SHA256 fc396945f308f033a7d2bd5201c896c6d7aa11f7ef18aaa9e771b59264390b20
MD5 09395d672c651c681a1dbf5c6c65b103
BLAKE2b-256 aaf9c55bd10a392bbc2e9d4199497b2797b0a7dae08bb84a9e6407d39018937e

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page