Report

Some user simulators matched human-like overall accuracy but missed human error patterns

The researchers found that people misremembered facts while models forgot them, producing different error patterns.

2 Sources, 1d ago, first seen 1d ago

Above: a figure illustrating the four experimental conditions in this study. In SayOnce, each fact that is tested is said once; in SayTwice, each fact gets the original version and an additional rephrased mention. In SayThrice, each fact gets mentioned three times. In the 'Interfere' condition, after each fact is mentioned once, competing facts about a different entity are mentioned.

Underneath that figure are examples of what counts as a mention of a fact. Examples of three restatements of the same fact: The first mention is "The mirelon's flesh is a vivid saffron-yellow due to a pigment called nivellin." The second mention is "Nivellin is what gives the mirelon its signature eyllow color." The third mention is "Nivellin is heat stable, so the mirelon stays golden even when cooked."

Examples of mentions of different facts: a filler attribute is "It's the kind of thing that's hard to forget once you've actually held one." An additional fact about the same entity is "The mirelon's bot

TLDR

Researchers tested whether memory-constrained user simulators respond to repeated facts and competing information as people do. They report that some simulators achieved human-like overall accuracy and replicated the magnitude of those effects, but none matched people's accuracy on particular questions. The team released its data as a challenge benchmark for future work.

Today's Rank

—

Not ranked yet

Combined views

2 Sources, first seen 1d ago