Empirical evaluation of retrieval-augmented instruction-following models.
Project description
Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering
Quick Start
Installation
You can install the library via pip.
pip install -e .
Usage
Here is a simple example to get started. Using this library, use can easily leverage retrieval-augmented instruction-following models for question-answering in ~25 lines of code. The source file for this example is examples/get_started.py.
from instruct_qa.collections.utils import load_collection
from instruct_qa.retrieval.utils import load_retriever, load_index
from instruct_qa.prompt.utils import load_template
from instruct_qa.generation.utils import load_model
from instruct_qa.response_runner import ResponseRunner
collection = load_collection("dpr_wiki_collection")
index = load_index("dpr-nq-multi-hnsw")
retriever = load_retriever("facebook-dpr-question_encoder-multiset-base", index)
model = load_model("flan-t5-xxl")
prompt_template = load_template("qa")
queries = ["what is haleys comet"]
runner = ResponseRunner(
model=model,
retriever=retriever,
document_collection=collection,
prompt_template=prompt_template,
queries=queries,
)
responses = runner()
print(responses[0]["response"])
# Halley's Comet Halley's Comet or Comet Halley, officially designated 1P/Halley, is a short-period comet visible from Earth every 75–76 years. Halley is the only known short-period comet that is regularly visible to the naked eye from Earth, and the only naked-eye comet that might appear twice in a human lifetime. Halley last appeared...
You can also check the input prompt given to the instruction-sollowing model that contains the instruction and the retrieved passages.
print(responses[0]["prompt"])
"""
Please answer the following question given the following passages:
- Title: Bill Haley
then known as Bill Haley's Saddlemen...
- Title: C/2016 R2 (PANSTARRS)
(CO) with a blue coma. The blue color...
...
Question: what is haleys comet
Answer:
"""
Data and Resources (Coming soon!)
We plan to release data and resources soon! Stay tuned!
License
This work is licensed under the Apache 2 license. See LICENSE for details.
Citation
To cite this work, please use the following citation:
@article{adlakha2023evaluating,
title={Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering},
author={Vaibhav Adlakha and Parishad BehnamGhader and Xing Han Lu and Nicholas Meade and Siva Reddy},
year={2023},
journal={arXiv:2307.16877},
}
Contact
For queries and clarifications please contact vaibhav.adlakha (at) mila (dot) quebec
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for instruct_qa-0.0.1rc6-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8db3e51b28e671b58df457f7fedbbf366b64c9564ca8df0dff49b38fc8005dbd |
|
MD5 | bef9e9f6ca771a7edb3d608e2f414679 |
|
BLAKE2b-256 | e0dcb4b4ac3c9ae29f7ddaf71cf4ae13b056c7bb262bb3424fd294c580bc0d02 |