COLM

2025

QUDsim

QUDsim: Quantifying Discourse Similarities in LLM-Generated Text

Ramya Namuduri, Yating Wu, Anshun Asher Zheng, Manya Wadhwa, Greg Durrett, Junyi Jessy Li

Conference on Language Modeling (COLM) · 2025

paperCode and dataBibTeX

Overview

Two texts can develop their ideas in similar ways while discussing different subjects. QUDsim compares them through the questions their passages answer, measuring answerability in both directions.

We find more repetition in the discourse structures of LLM writing than in human writing. The questions make it possible to inspect which passages share a discourse role.

Paper

Different explanations, the same discourse role

Two models receive the same fictional story prompt. In both stories, a leader questions a secret and another character defends keeping it.

Question answered by both passages

What is the reason given for the deception?

Gemini Flash 1.5

Exposing the secret would threaten social stability.

The character defends concealment as protection against upheaval.

GPT-4o

A shared belief unites people and makes progress possible.

The character defends concealment as a way to maintain cooperation.

Source

Figure 1 and Section 2, page 2. Question from Figure 1. The two story passages are paraphrased.

Scores and document pairs

Read the published scores below, or use the demo notebook to compare your own texts.

Scores
dataset/similarity.json has 650 rows. Each row includes document IDs, segment scores, and a Similarity value.
Your own texts
The notebook accepts JSON or CSV with Document 1, Document 2, Model 1, and Model 2 columns.
Questions
Level 0 keeps questions specific to the content; level 1 abstracts it. New comparisons require the repository’s API setup.
Read the scoresPython

Requires pandas.

import pandas as pd

revision = "f146127b870e866c661ab67a04da5122f9b6805a"
url = (
    "https://raw.githubusercontent.com/AlliteraryAlligator/QUDsim/"
    f"{revision}/dataset/similarity.json"
)
scores = pd.read_json(url)
print(scores[["Source ID", "Target ID", "Similarity"]].head())
Command-line setup

In the linked CLI version, --unique and args.with_replacement refer to different names. Use demo.ipynb until this is corrected.

Source

Dataset schema · Demo notebook · CLI source

Demo notebook
Compare a pair of documents.
Setup and input format
Dependencies, API setup, and input format.
Data and annotations
QUDs, human alignments, and published scores.

Citation

Download BibTeX
View BibTeX
@inproceedings{qudsim2025,
  title={{QUDsim: Quantifying Discourse Similarities in LLM-Generated Text}},
  author={Namuduri, Ramya and Wu, Yating and Zheng, Anshun Asher and Wadhwa, Manya and Durrett, Greg and Li, Junyi Jessy},
  year={2025},
  booktitle={Second Conference on Language Modeling},
  url={https://openreview.net/forum?id=zFz1BJu211}
}