Minervini Says Model Faithfulness Depends Heavily on Environment
University of Edinburgh lecturer comments on aryopg et al research into model faithfulness.
Pasquale Minervini, tenured Lecturer in NLP/ML at the University of Edinburgh and affiliated with EdinburghNLP, posted that whether a model is faithful depends heavily on environmental factors. He stated that faithfulness, mechinterp, and safety research should take those factors into account. Minervini called the work by @aryopg et al amazing and linked to their post at twitter.com/aryopg/status/2094727222926155985. He is also Co-Founder and CTO of Miniml.AI.
Combined views
155.9K
5 posts, first seen 1d ago
Reactions
5 postsAP 
Aryo Pradipta Gema@aryopgCoT monitoring assumes reasoning traces record what shapes an answer. Yet, we found that models are less likely to verbalize a cue in its CoT when the cue comes from a tool return instead of the user message! 👀 🧵
- likes: 89
- replies: 2
- bookmarks: 36
- reposts: 12
PM Pasquale Minervini@PMinerviniWhether a model is faithful (and, e.g., whether it's relying on a cue) depends heavily on environmental factors, and faithfulness/mechinterp/safety research should take that into account! 🚀 Amazing work by @aryopg et al.
- likes: 15
- replies: 0
- bookmarks: 0
- reposts: 3
PM Pasquale Minervini@PMinervini@merettm Hi Jakub, you may be interested in @aryopg's latest work on CoT faithfulness and monitorability, it seems super relevant:
- likes: 3
- replies: 0
- bookmarks: 3
- reposts: 0
AP Aryo Pradipta Gema@aryopg@merettm sure cot monitorability is fragile, tho I’m still bullish it can work if the model developers actively maintain it. we also need evals that track how "environmental" factors (not just the architecture) influence monitorability!
- likes: 6
- replies: 0
- bookmarks: 0
- reposts: 1
AP Aryo Pradipta Gema@aryopg@merettm sure cot monitorability is fragile, tho I’m still bullish it can work if the model developers actively maintain it. we also need evals that track how "environmental" factors (not just the architecture) influence monitorability!
- likes: 6
- replies: 0
- bookmarks: 0
- reposts: 1