Empirical evaluation of retrieval-augmented instruction-following models.
Project description
Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering
Quick Start
Installation
Make sure you have Python 3.7+ installed. It is also a good idea to use a virtual environment.
Show instructions for creating a Virtual Environment
python3 -m venv instruct-qa-venv
source instruct-qa-venv/bin/activate
You can install the library via pip
:
# Install the latest release
pip3 install instruct-qa
# Install the latest version from GitHub
pip3 install git+https://github.com/McGill-NLP/instruct-qa
For development, you can install it in editable mode with:
git clone https://github.com/McGill-NLP/instruct-qa
cd instruct-qa/
pip3 install -e .
Usage
Here is a simple example to get started. Using this library, use can easily leverage retrieval-augmented instruction-following models for question-answering in ~25 lines of code. The source file for this example is examples/get_started.py.
from instruct_qa.collections.utils import load_collection
from instruct_qa.retrieval.utils import load_retriever, load_index
from instruct_qa.prompt.utils import load_template
from instruct_qa.generation.utils import load_model
from instruct_qa.response_runner import ResponseRunner
collection = load_collection("dpr_wiki_collection")
index = load_index("dpr-nq-multi-hnsw")
retriever = load_retriever("facebook-dpr-question_encoder-multiset-base", index)
model = load_model("flan-t5-xxl")
prompt_template = load_template("qa")
queries = ["what is haleys comet"]
runner = ResponseRunner(
model=model,
retriever=retriever,
document_collection=collection,
prompt_template=prompt_template,
queries=queries,
)
responses = runner()
print(responses[0]["response"])
# Halley's Comet Halley's Comet or Comet Halley, officially designated 1P/Halley, is a short-period comet visible from Earth every 75–76 years. Halley is the only known short-period comet that is regularly visible to the naked eye from Earth, and the only naked-eye comet that might appear twice in a human lifetime. Halley last appeared...
You can also check the input prompt given to the instruction-sollowing model that contains the instruction and the retrieved passages.
print(responses[0]["prompt"])
"""
Please answer the following question given the following passages:
- Title: Bill Haley
then known as Bill Haley's Saddlemen...
- Title: C/2016 R2 (PANSTARRS)
(CO) with a blue coma. The blue color...
...
Question: what is haleys comet
Answer:
"""
Data and Resources (Coming soon!)
We plan to release data and resources soon! Stay tuned!
License
This work is licensed under the Apache 2 license. See LICENSE for details.
Citation
To cite this work, please use the following citation:
@article{adlakha2023evaluating,
title={Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering},
author={Vaibhav Adlakha and Parishad BehnamGhader and Xing Han Lu and Nicholas Meade and Siva Reddy},
year={2023},
journal={arXiv:2307.16877},
}
Contact
For queries and clarifications please contact vaibhav.adlakha (at) mila (dot) quebec
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for instruct_qa-0.0.1.post1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 559b2a0c76d93fba8c608e41f8c1a7fcf76e29a432727ed6cd5994a359592692 |
|
MD5 | 1f73d84742f162cd0c32b7e0f8caf67e |
|
BLAKE2b-256 | 93d89d02d413ff2a109c8260aa117c07a4ee04d3c797f5d86ca3b2b6ef95770f |