Using AI model disagreements to guide revisions to annotation rules

A post shares a paper about targeting expert effort where large language models disagree during revisions to annotation codebooks.

1 Source, 2h ago, first seen 2h ago

Large-scale text annotation brings expert insight to millions of documents, often through a codebook that AI annotators follow. Developing a robust codebook, however, takes months. Large language models (LLMs) could speed this process by applying an early codebook to the data, surfacing cases with strong LLM disagreement, and eliciting expert feedback to address them. We examined three ways experts can provide feedback for LLM codebook revision: (i) editing LLM-generated revisions driven by cross-LLM disagreement (Codebook Verifying), (ii) answering questions about LLM disagreements (Question Answering), and (iii) labeling disagreement cases with rationales (Rationale Labeling). Experiments on thousands of tutoring-session transcripts show that Rationale Labeling yielded the highest LLM-labeling accuracy (64.9%) against expert labels, outperforming the expert-revised codebook (57.8%). The best Question Answering setting also outperformed it (60.5%). Our work shows that LLMs can be used

TLDR

A post links to “Experts Rise Where LLMs Disagree,” a paper on using disagreement across large language models to focus expert effort in codebook revision. Codebooks provide the rules for labeling data; the paper’s stated focus is large-scale annotation.

Featured Source

Using AI model disagreements to guide revisions to annotation rules

TLDR

1 Source

Combined views

Sentiment

Summary

Using AI model disagreements to guide revisions to annotation rules

TLDR

Combined views

Sentiment

Summary

Sentiment

Summary

1 Source

1 Source

1 Source