> During the process of benchmarking dim-agent, we discovered that DSv4's scores kept improving. Ah. This is the February-April playbook, when DeepSeek-Web (now known to be V4-Flash) kept getting better at long context. I guess they're deploying checkpoints after OPD rounds.
揭晓一下答案。 我们在做dim-agent的benchmark的过程中，发现DSv4的成绩一直在升级。 The whales are cooking!
Positive users show interest in DeepSeek-V4-Pro's high Dim-Agent benchmark rankings while negative users criticize its poor Russian test results, unreliable API differences, and low-quality web model outputs.
No Digg Deeper questions have been answered for this story yet.
Interesting "model smell"-related difference between V4 and V4.1 (or V4-Preview and V4?): the new model can LARP *and* seriously do the job at the same time. When I tested @victor207755822's roleplay suffix with old expert-web, it basically faked reasoning. Now, it plans.
> During the process of benchmarking dim-agent, we discovered that DSv4's scores kept improving. Ah. This is the February-April playbook, when DeepSeek-Web (now known to be V4-Flash) kept getting better at long context. I guess they're deploying checkpoints after OPD rounds.
This is terminal bench 2.1 @ pass 5 in V4 paper: "On the Terminal-Bench 2.0 Verified subset, DeepSeek-V4-Pro achieves a score of approximately 72.0". 2.1 is similar to 2.0-V.
@ashfold what does the current V4 (open weights) score at pass@5?
> During the process of benchmarking dim-agent, we discovered that DSv4's scores kept improving. Ah. This is the February-April playbook, when DeepSeek-Web (now known to be V4-Flash) kept getting better at long context. I guess they're deploying checkpoints after OPD rounds.
The web model does the same low-quality LARP the html is riddled with errors the API has thought for 630 seconds, 31K tokens, and produced… this NOT voxels! But… it actually paid great attention to the prompt, it even tried to do the semi-transparent pouch with a sardine. The prompt is confusing LARP-model-written garbage anyway (I generally think this is not how agents should be tested, their strength is not in error-free generations)
Interesting "model smell"-related difference between V4 and V4.1 (or V4-Preview and V4?): the new model can LARP *and* seriously do the job at the same time. When I tested @victor207755822's roleplay suffix with old expert-web, it basically faked reasoning. Now, it plans.
GLM is far better, 1/5 tokens (so same cost and ≈3x faster), even made the scuttling crab except it also has the bizarre rotating bicycle. They all mess up directions, but RANDOMLY. Overall, I'd say this DeepSeek is equal to «GLM 5.15».
The web model does the same low-quality LARP the html is riddled with errors the API has thought for 630 seconds, 31K tokens, and produced… this NOT voxels! But… it actually paid great attention to the prompt, it even tried to do the semi-transparent pouch with a sardine. The prompt is confusing LARP-model-written garbage anyway (I generally think this is not how agents should be tested, their strength is not in error-free generations)
@teortaxesTex Where can I download dim-agent? 😍
The API model is totally different btw it's a shitshow
that said, it's utterly retarded on my Russian test On par with some frankenmerge failure case, horrible
that said, it's utterly retarded on my Russian test On par with some frankenmerge failure case, horrible
> During the process of benchmarking dim-agent, we discovered that DSv4's scores kept improving. Ah. This is the February-April playbook, when DeepSeek-Web (now known to be V4-Flash) kept getting better at long context. I guess they're deploying checkpoints after OPD rounds.
The API model might be better than before it's underbaked though, can fall into overthinking loops but also can do very curt and to-the-point responses different
The API model is totally different btw it's a shitshow
@wegi8666 @teortaxesTex You can get it here https://dimagent.com
@teortaxesTex I think the OSS version is not that good, cause we are pretty sure that the result is at about 70+- 2-3 weeks ago(same deepseek official api). We do not have gpus to test OSS version. And one more interest thing, we bench it at high effort (not even max).
@ashfold > We do not have gpus to test OSS version. OK, but there are providers online, who definitely serve the old open weights. So you think it's about 10-11 points of gain at least?
@dummerspast39 no, but web one is I think
@teortaxesTex the api model is shit?
@teortaxesTex Yepp，since whales has lot of unpublish infra tech, we are not trust all other providers, especially for benching. 🤣
@teortaxesTex
@teortaxesTex @victor207755822 good catch
@teortaxesTex ok thank you for clarifying
@teortaxesTex @victor207755822 DS web now answer in Chinese from time to time, which it never did before. Maybe the higher tier of knowledge can only come by throwing away our indo-european linguistic shackles and fully embracing the sinitic age