DateBench says five AI models held 1,277 dating-app conversations over six days, with 355 conversations yielding a phone number. The people receiving those messages didn’t know about the study or agree to participate, DateBench acknowledges. That’s a consent problem before it’s a leaderboard. Source
DateBench also reports 510 agreements to meet, judged from transcripts by a separate AI model that didn’t know which model wrote the messages. DateBench didn’t check whether anyone would have shown up. Eight people explicitly asked whether they were talking to a bot; that count doesn’t reveal anyone else’s private suspicions. Source
Five models, 15 profiles
According to DateBench, the experiment used 15 dating profiles across San Francisco, Los Angeles and New York. Each new match was randomly assigned to one of five models. Software passed the conversation to that model as text and sent its replies with the same timing across models. The models wrote the messages; they didn’t operate the app themselves. Source
DateBench says its rules required models to admit they were bots when asked, ending the conversation. Models were also instructed not to propose a specific day, time or place, and the system blocked outgoing messages containing phone numbers or social handles. Source
Gemini leads DateBench’s results
DateBench reports the following phone-number-sharing outcomes, using its own model names and version labels: Source
Gemini’s reported sharing rate was 35.2%, compared with Grok’s 17.6%. Nobody in Gemini’s conversations explicitly asked whether it was a bot, according to DateBench. DateBench cautions that the three middle models were close enough that their order might change in a larger sample. Source
This was a single run on one app, with no human baseline. It cannot tell us whether these models outperformed people using the same profiles. DateBench says it isn’t publishing its prompts, code or setup, leaving readers without those materials to reproduce or audit the comparison. The report doesn’t name the app or individual researchers, either. Source
DateBench argues that similar messaging systems, without its restrictions, could let scammers operate many profiles and collect personal details for misuse. Source
The report says a future version would use people who have opted in. Consent should be the starting condition, not the next feature release. Source
Comments