Despite the record, GLM still trails closed-source models.
@natolambert @Zai_org GLM might not have tried to train for ARC-AGI at all.
Add more wins for GLM.
The model has some brittle characteristics, and is getting crushed by closed models here, but we should expect open models to be more jagged, and you use multiple of them depending on the task.
Congrats again to @Zai_org and am excited for the next one
Some users praised measured takes on the open-source GLM's ARC-AGI-2 record as fair and useful, while others warned against its provider and criticized the $100B valuation despite minimal revenue.
No Digg Deeper questions have been answered for this story yet.
@natolambert @Zai_org Warning folks who might see this and want to go use http://z.ai as a provider -
@natolambert @Zai_org do you have any high-level ideas on why open models tend to be more jagged?
Add more wins for GLM.
The model has some brittle characteristics, and is getting crushed by closed models here, but we should expect open models to be more jagged, and you use multiple of them depending on the task.
Congrats again to @Zai_org and am excited for the next one
This is the strongest ARC-AGI-2 performance to date by an open-source model.
@natolambert @Zai_org honestly this is the most measured take on open model performance ive seen
brittle but useful for the right tasks is a fair summary
@natolambert @Zai_org I am less impressed by demos now. I want tools that quietly become part of the day.
@natolambert @Zai_org A $100 billion dollar market cap with barely any actual revenue.
@natolambert @Zai_org The jaggedness is the interesting part. Open models aren't worse, they're uneven, and the unevenness is usually a map of where the post-training data was thin. You can almost read the gaps backwards to the dataset.
