Teams attending include: 1) Preparedness, which tracks frontier risks and works on mitigations 2) Pretraining Safety, which builds safer base models 3) Honesty & Reliability, which tackles hallucination, uncertainty, and deception
https://bit.ly/icml-safety <- fill out this form!
Users are excited about meeting the AI safety and alignment teams at the ICML workshop because they describe the teams as cool and important for addressing frontier risks.
No Digg Deeper questions have been answered for this story yet.
4) Robustness team, focused on making models more resilient to adversarial pressure 5) Trustworthy AI, focused on collective alignment and safety work grounded in societal impact, and third-party assurances
Teams attending include: 1) Preparedness, which tracks frontier risks and works on mitigations 2) Pretraining Safety, which builds safer base models 3) Honesty & Reliability, which tackles hallucination, uncertainty, and deception
And at the training level, Alignment Training studies how durable behaviors emerge across pre-, mid-, and post-training. The team builds data, objectives, and evals to help models follow intent, reason reliably, express uncertainty, and act honestly in new situations.
A related question is whether we can understand what models are doing as they reason. The CoT Monitorability team is developing ways to tell when CoTs are faithful and legible, what improves or degrades that signal, and whether reasoning can reveal latent behavior.
I sadly won't personally be in Seoul but excited for y'all to meet these cool teams!
And at the training level, Alignment Training studies how durable behaviors emerge across pre-, mid-, and post-training. The team builds data, objectives, and evals to help models follow intent, reason reliably, express uncertainty, and act honestly in new situations.
A related question is whether we can understand what models are doing as they reason. The CoT Monitorability team is developing ways to tell when CoTs are faithful and legible, what improves or degrades that signal, and whether reasoning can reveal latent behavior.
Several teams attending are also focused on how we scale alignment as models become more capable. Alignment Scaling works on synthetic data, reasoning-based reward models, and automated red teaming, with the longer-term goal of training models to help with alignment research.