A new blog post introduces Synthetic Persona Pretraining to embed desired values directly into pretraining data and reports 1.7 percent mean attack success on 1.7B models
SPP Token Zero beat unfiltered, filtered, and SafeLM baselines across five benchmarks.
#353Dylan HadfieldMenell@DHADFIELDMENELLThis is super cool work. Great to see open research on this topic!
Julian Minder@jkminderNew blog! Synthetic Persona Pretraining (SPP): Alignment from Token Zero Current alignment is shallow - values bolted on after pretraining can be routed around. To solve this, we wrote the desired persona directly into pretraining data. Early results, but we're very excited. ๐งต
#1019Valentina Pyatkin@VALENTINA__PYCheck out Julian and co's interesting blogpost on how to use synthetic personas during pretraining, for improved safety alignment:
Julian Minder@jkminderNew blog! Synthetic Persona Pretraining (SPP): Alignment from Token Zero Current alignment is shallow - values bolted on after pretraining can be routed around. To solve this, we wrote the desired persona directly into pretraining data. Early results, but we're very excited. ๐งต
#1753๐๐๐ โข@VIEMCCOYPersona research is an entire field unto itself! If you want diverse Persona options, you need to consider them from the very start of your stack! Lots to explore here.
(If you are working on this sort of thing independently, I'm very interested in hiring you!)
Julian Minder@jkminderNew blog! Synthetic Persona Pretraining (SPP): Alignment from Token Zero Current alignment is shallow - values bolted on after pretraining can be routed around. To solve this, we wrote the desired persona directly into pretraining data. Early results, but we're very excited. ๐งต