• Home
  • Technology
  • Gaming
  • Entertainment
  • World & Business
  • Science
  • Sports
  • AI
HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
AI

Jev reportedly performs near Claude Sonnet on classification at far lower cost

A user testing TypeSafe AI’s small classifier reports promising results, while flagging possible training-data overlap in public benchmarks and a lack of real B2B sales data.

1 Source, 20d ago, first seen 20d ago

TLDR

A user evaluating TypeSafe AI’s Jev reports classification performance roughly on Claude Sonnet’s level while costing a couple of orders of magnitude less to run. The user says Jev performed on par with Opus on three public benchmarks, which may be in its training data, and alongside Sonnet and Opus on a fresh, hand-written test based on one Wikipedia article. On older customer-service and negotiation datasets, the user puts Jev a bit above Claude Haiku and slightly below Sonnet 5. Those datasets weren’t from real B2B sales, so the user calls the findings directional rather than representative of work at Lightfield.

Combined views

—

1 Source, first seen 20d ago

— likes— comments— saves— reposts

Combined views

—

1 Source, first seen 20d ago

— likes— comments— saves— reposts

Sentiment

Positive——Negative

Summary

Not enough discussion yet.

No sentiment analysis available yet.

Featured Source

Sentiment

Positive——Negative

Summary

Not enough discussion yet.

No sentiment analysis available yet.

1 Source

Ves Stoyanov@vesko_stEveryone has been impressed by TypeSafe AI’s Jev — I wanted to see for myself how good it is — and how useful it could be for a product like @lightfld. I started with text understanding, because it is the foundation of everything a decision model does: it has to understand the input, and it has to understand and follow the instructions you give it. The short takeaway: TypeSafe's tiny "Jev" classifier is surprisingly good — roughly on the level of Claude Sonnet on classification tasks, while being a couple of orders of magnitude cheaper to run. I looked at two kinds of work. First, general text understanding: three widely used reading-comprehension and commonsense benchmarks, where Jev performed on par with Opus. Because those may well sit in Jev's training data, I also hand-authored a fresh benchmark over a single random Wikipedia article — every wrong option drawn from the text, so nothing could be memorized. There, Jev landed alongside Sonnet and Opus (it tops CommonsenseQA and MMLU-CF, ties Opus on RACE-H, and is ~150× cheaper and ~10× faster on long-context questions). Then I tried tasks closer to sales and customer relationships. There aren't many good public benchmarks, so I dug up older customer-service and negotiation datasets. On those, Jev is a bit better than Claude Haiku and slightly below Sonnet 5. Big caveat: none of these come from real B2B sales, so the findings are directional rather than a description of what we actually do at @lightfld . Overall, the results look good enough that I think small, inexpensive classification models like Jev will have real uses, including in our own product. Link to the full write-up below.20d
    • Home
    • Technology
    • Gaming
    • Entertainment
    • World & Business
    • Science
    • Sports
    • AI

    1 Source

    Ves Stoyanov@vesko_stEveryone has been impressed by TypeSafe AI’s Jev — I wanted to see for myself how good it is — and how useful it could be for a product like @lightfld. I started with text understanding, because it is the foundation of everything a decision model does: it has to understand the input, and it has to understand and follow the instructions you give it. The short takeaway: TypeSafe's tiny "Jev" classifier is surprisingly good — roughly on the level of Claude Sonnet on classification tasks, while being a couple of orders of magnitude cheaper to run. I looked at two kinds of work. First, general text understanding: three widely used reading-comprehension and commonsense benchmarks, where Jev performed on par with Opus. Because those may well sit in Jev's training data, I also hand-authored a fresh benchmark over a single random Wikipedia article — every wrong option drawn from the text, so nothing could be memorized. There, Jev landed alongside Sonnet and Opus (it tops CommonsenseQA and MMLU-CF, ties Opus on RACE-H, and is ~150× cheaper and ~10× faster on long-context questions). Then I tried tasks closer to sales and customer relationships. There aren't many good public benchmarks, so I dug up older customer-service and negotiation datasets. On those, Jev is a bit better than Claude Haiku and slightly below Sonnet 5. Big caveat: none of these come from real B2B sales, so the findings are directional rather than a description of what we actually do at @lightfld . Overall, the results look good enough that I think small, inexpensive classification models like Jev will have real uses, including in our own product. Link to the full write-up below.20d
    Today's Rank

    —

    Not ranked yet

    Today's Rank

    —

    Not ranked yet