Aflah 🍉🕊️ @ EMNLP

@aflah02101

PhD-ing @mpi_sws_ & hanging-out @AiEleuther • Prev. @GoldmanSachs GSoC'22 @TensorFlow @lcs2lab • RT≠Endorsement #PERMANENTCEASEFIRENOW🕊️

following
followers

Three variables are claimed to predict verbatim memorization in language models

Christopher Potts says Jing Huang’s Hubble model suite experiments use frequency, recency and hardness; hardness has comparatively low predictive value.

15 Sources, 16d ago, updated 16d ago

Dot plot addressing whether memorized sequences can be predicted without training the full model. Headline: “Largely yes, with Frequency and Recency doing most of the work.” Four rows show AUC-ROC (0.5 = chance, 1.0 = perfect) for a linear predictor of verbatim extraction, one mark per Hubble model (1B or 8B params; 100B or 500B tokens). All three features: 0.82–0.94. Frequency alone: 0.81–0.93. Recency alone: 0.82–0.90. Hardness alone: 0.51–0.55, barely above chance. In every row the 1B/500B model scores lowest and the 1B/100B model highest. Footnote: held-out strings, mean of 5 seeds; Frequency = copies of the string, Recency = steps since last copy, Hardness = NLL on models that never saw it.

New MPI-SWS PhD student looks toward LLMs and AI for science

The student says @lasha_nlp and Krishna Gummadi will advise them. They’re looking forward to working on problems related to large language models and AI for science.

2 Sources, 40d ago, first seen 40d ago

Featured in

Three variables are claimed to predict verbatim memorization in language models

New MPI-SWS PhD student looks toward LLMs and AI for science

Featured in

Three variables are claimed to predict verbatim memorization in language models

New MPI-SWS PhD student looks toward LLMs and AI for science