EMNLP

2024

QSalience

Which questions should I answer? Salience Prediction of Inquisitive Questions

Yating Wu†, Ritika Mangla†, Alexandros G. Dimakis, Greg Durrett, Junyi Jessy Li

Conference on Empirical Methods in Natural Language Processing (EMNLP) · 2024 · Oral presentation Outstanding Paper Award

† Equal contribution.

paperCode and dataBibTeX

Overview

We rate questions by how much their answers would help a reader understand the preceding text. QSalience learns these judgments from 1,766 context–question pairs annotated by linguists.

More salient questions are more likely to be answered later in the article. In a summary-expansion experiment, readers also prefer summaries that answer more salient questions.

Paper

Questions after a cease-fire violation

A news report describes artillery fire in Grozny during a cease-fire. Annotators rated these two questions differently.

Q3 · Original question

5/ 5
What was the reason behind the artillery fire in Grozny on Tuesday despite the agreed cease-fire?

An explanation for the violation would help the reader assess the peace talks. A later sentence says the negotiators lack control over some fighters.

Q4 · Paraphrased question

3/ 5

What further details are available about the reported missile attacks?

More detail about the attack adds less to the reader’s understanding of the peace talks.

Source

Figure 1, page 2. Human ratings of GPT-4 questions. Q3 is quoted; Q4 and the rationales are paraphrased.

Try QSalience

Start with the example files in the repository, or supply your own questions and article.

Questions
A CSV with two columns: question and sentence_id.
Article
A text file with sentences numbered from 1. Only the text through sentence_id is passed to the model.
Output
T5_predictions.csv contains one predicted salience score per question, in input order. The scale runs from 1 to 5.
Run with T5Terminal

In a local checkout, install the README dependencies, then run:

cd code/preprocess
python preprocess.py \
  --question_csv_path example_question.csv \
  --article_txt_path example_article.txt

cd ..
python predict_salience.py \
  --model_name t5 \
  --input_file preprocess/example.json

Preprocessing writes example.json. The second command loads the released T5 model and writes the predictions.

Source

Preprocessing · Prediction code

Dataset
Salience annotations and data splits.
Code and Colab
Setup instructions and example notebook.
Models
Mistral, Llama 2, T5, and TinyLlama checkpoints.

Citation

Download BibTeX
View BibTeX
@inproceedings{qsalience2024,
  title={{Which questions should I answer? Salience Prediction of Inquisitive Questions}},
  author={Wu, Yating and Mangla, Ritika and Dimakis, Alexandros G. and Durrett, Greg and Li, Junyi Jessy},
  year={2024},
  booktitle={Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  doi={10.18653/v1/2024.emnlp-main.1114},
  pages={19969--19987},
  url={https://aclanthology.org/2024.emnlp-main.1114/},
  publisher={Association for Computational Linguistics}
}