We release the datasets and evaluation harness for everything to try. Our hope is that ContinuousBench will help measure and accelerate progress in DP synthetic data.

Check out the full paper for more! w/ Peihan Liu (@_peihanliu ), Lucas Rosenblatt (@lucas ros), Weiwei Kong, Natalia Ponomareva, Gautam Kamath (@thegautamkamath ), Rachel Cummings (@radcummings ), Roxana Geambasu, Yu Gan, Lillian Tsai (@tslilyai).

https://arxiv.org/abs/2606.01849 4/4

Peihan wrote an excellent, short blog on ContinuousBench: https://peihanliu.com/posts/continuousbench.html

In ContinuousBench, we curate datasets that are unsolvable without training: - Synthetic articles about fictional pokemon-inspired creatures - Newly scraped news from CommonCrawl

...and ask whether models trained on DP synthetic versions of these datasets can reliably answer questions about the content.

2/n

Our core result is that while non-DP synthetic versions of these datasets transfer factual knowledge, DP synthesis does not.

Gains are marginal compared to to the base model, and fall far short of training on the real data.