factgenie

Annotate LLM outputs with a lightweight, self-hosted web application 🌈

📢 Changelog

• [1.0.0] - 2024-11-13: The first official release 🎉

👉️ How can factgenie help you?

Outputs from large language models (LLMs) may contain errors: semantic, factual, and lexical.

With factgenie, you can have the error spans annotated:

• From LLMs through an API.
• From humans through a crowdsourcing service.

Factgenie can provide you:

1. A user-friendly website for collecting annotations from human crowdworkers.
2. API calls for collecting equivalent annotations from LLM-based evaluators.
3. A visualization interface for visualizing the data and inspecting the annotated outputs.

What does factgenie not help with is collecting the data (we assume that you already have these), starting the crowdsourcing campaign (for that, you need to use a service such as Prolific.com) or running the LLM evaluators (for that, you need a local framework such as Ollama or a proprietary API).

🏃 Quickstart

Make sure you have Python >=3.9 installed.

If you want to quickly try out factgenie, you can install the package from PyPI:

pip install factgenie

However, the recommended approach for using factgenie is using an editable package:

git clone https://github.com/ufal/factgenie.git
cd factgenie
pip install -e .[dev,deploy]

This approach will allow you to manually modify configuration files, write your own data classes and access generated files.

After installing factgenie, use the following command to run the server on your local computer:

factgenie run --host=127.0.0.1 --port 8890

More information on how to set up factgenie is on Github wiki.

💡 Usage guide

See the following wiki pages that that will guide you through various use-cases of factgenie:

🔥 Tutorials

We also provide step-by-step walkthroughs showing how to employ factgenie on the the dataset from the Shared Task in Evaluating Semantic Accuracy:

💬 Cite us

Our paper was published at INLG 2024 System Demonstrations!

You can also find the paper on arXiv.

For citing us, please use the following BibTeX entry:

@inproceedings{kasner2024factgenie,
    title = "factgenie: A Framework for Span-based Evaluation of Generated Texts",
    author = "Kasner, Zden{\v{e}}k  and
      Platek, Ondrej  and
      Schmidtova, Patricia  and
      Balloccu, Simone  and
      Dusek, Ondrej",
    editor = "Mahamood, Saad  and
      Minh, Nguyen Le  and
      Ippolito, Daphne",
    booktitle = "Proceedings of the 17th International Natural Language Generation Conference: System Demonstrations",
    year = "2024",
    address = "Tokyo, Japan",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.inlg-demos.5",
    pages = "13--15",
}

Acknowledgements

This work was co-funded by the European Union (ERC, NG-NLG, 101039303).