Mechanize podcast guests argue AI models struggle to build emulators because of low-quality training data

Youtube: https://www.youtube.com/watch?v=UpO70AJGdJ8 Substack: https://mechanizework.substack.com/p/how-bad-data-teaches-models-to-write Spotify: https://open.spotify.com/show/033krxvlEYnqkHhClcM39e

Our new podcast on evals, with Max Niederman, Ege Erdil, and Stephen Yang.

0:00:00 – What's an eval, and how's it different from an RL environment? 0:19:33 – Why are models bad at building an emulator when the task is fully verifiable? 0:42:00 – How does training on bad data teach models to write terrible code? 1:04:00 – Why is continual learning still so bad? 2:25:24 – Why haven't software engineers been replaced when coding is basically solved?

Listen to the Mechanize Podcast on YouTube, Spotify, etc. Enjoy!

@MechanizeWork Apple podcast link?

@MechanizeWork Enjoyed the commentary on RL environment scaling and software engineering. Thanks guys!

@MechanizeWork @tamaybes interesting podcast

@MechanizeWork

@TheShmanuel https://podcasts.apple.com/us/podcast/how-bad-data-teaches-models-to-write-terrible-code/id1896935098?i=1000773011935