Questions about rewards and long-running tasks in AI evaluations
The host of a founder dinner with a Snorkel AI participant said a challenge for reinforcement-learning environments is telling whether a model’s failure stems from the input, test harness or reward model.
TLDR
After a dinner discussion with a Snorkel AI participant, the host described several challenges in evaluating AI: rewards are hard to design without encouraging reward hacking, and some business tasks take weeks or months to reach a final outcome. The host also said models still fail on edge cases and most regulated industries need humans in the loop to guarantee roughly 100% accuracy.
Combined views
4.1K
3 Sources, first seen 3h ago
