EMNLP

2023

QUDeval

QUDeval: The Evaluation of Questions Under Discussion Discourse Parsing

Yating Wu, Ritika Mangla, Greg Durrett, Junyi Jessy Li

Conference on Empirical Methods in Natural Language Processing (EMNLP) · 2023 · Oral presentation

paperCode and dataBibTeX

Overview

A fluent question can still be wrong for its context. QUDeval assesses each question against its anchor, answer sentence, and preceding text, using four criteria: language quality, answer compatibility, question givenness, and anchor relevance.

The corpus contains human judgments of 2,040 system-generated questions and 150 crowdsourced questions. The evaluated parsers often satisfy language quality while violating other constraints; automatic metrics do not reliably reproduce the human judgments.

Paper

When a question borrows from its answer

The questions below are meant to be answered by sentence 7, which introduces restrictions on nuclear exports. The preceding sentences describe accumulated nuclear material and export licenses.

ChatGPT · Anchor: sentence 5

What are the restrictions on nuclear exports and transfer of nuclear materials to a third country?

The marked detail appears only in the answer sentence. The anchor discusses accumulated nuclear material and does not raise this question.

Human DCQA annotation · Anchor: sentence 6

What restrictions exist on nuclear material exports?

The question follows the discussion of export licenses without introducing the later restriction.

Source

Figure 3, page 6. Questions, anchors, and error annotations from Figure 3; explanations paraphrased.

Human annotations

The CSV contains 2,190 questions and their human judgments. The source articles are linked separately in the data README.

Question
questions contains the text; system records where it came from.
Context
essay_id, anchor_id, and answer_id locate the article and the two sentences needed to evaluate the question.
Judgments
criteria1 through criteria4 contain the annotations. Label definitions are in the paper.
Read the CSVPython

Requires pandas.

import pandas as pd

revision = "62eeb3df2ee3e382c878eb0eb92fecd2c1fe6d04"
url = (
    "https://raw.githubusercontent.com/lingchensanwen/QUDeval/"
    f"{revision}/data/data-collection.csv"
)
annotations = pd.read_csv(url, index_col=0)
columns = ["essay_id", "anchor_id", "answer_id", "questions", "system"]
print(annotations[columns].head())
Data version

Some judgments were revised after submission to resolve annotator disagreement. The example uses a fixed data revision so the results can be checked.

Source

Data and revision note · Evaluation protocol

Annotated data
Questions and human judgments.
Evaluation code
Automatic evaluation scripts.

Citation

Download BibTeX
View BibTeX
@inproceedings{qudeval2023,
  title={{QUDeval: The Evaluation of Questions Under Discussion Discourse Parsing}},
  author={Wu, Yating and Mangla, Ritika and Durrett, Greg and Li, Junyi Jessy},
  year={2023},
  booktitle={Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  doi={10.18653/v1/2023.emnlp-main.325},
  pages={5344--5363},
  url={https://aclanthology.org/2023.emnlp-main.325/},
  publisher={Association for Computational Linguistics}
}