Continual learning is widely discussed right now, but mostly as improving on the job or avoiding catastrophic forgetting. But it has a different, difficult, and already urgent form:
Given nothing but a corpus of documents, how should AI systems develop expertise in a new, unfamiliar domain? We call this problem Machine Studying.
Users are excited about the 'Machine Studying' definition because it offers a fresh framing of intelligence akin to skill acquisition efficiency along with practical production examples.
No Digg Deeper questions have been answered for this story yet.
Studying gives us a second curve: expertise as a function of study compute. You could consider its weighted area a notion of “intelligence”. An intelligent agent, for our purposes, is one that can acquire expertise in totally new domains really efficiently. (And by this token, it’s not obvious that even the most knowledgeable of current agents are very smart!)
@jacobli99 putting the link here for those that want to jump right into the long form:
https://jacobxli.com/blog/2026/machine-studying/
@jacobli99 @lateinteraction I’m not up to date on training methods, so this may be a dumb question, but if expertise is defined by the quality/cost curve, then why not optimize index and search?
So how should an agent actually study? We test the simplest version of three natural bets out there: (1) self-supervised objectives, e.g. continual pre-training over the corpus; (2) synthesizing your own training data, e.g. turning the corpus into Q&A pairs to fine-tune on; and (3) amortized context management, e.g. having the agent write itself a cheatsheet.
Our preliminary finding is that none of these reliably turns exposure to the corpus into as much expertise as we’d like yet. Neither memorization of the contents of the corpus nor the ability to retrieve things stands in as a substitute for deeper expertise.
Read the blog for the details on all three paradigms, and what we think it'll take to do better!
We instantiate this in StudyBench, a benchmark we’re building so that we and others can begin to ask questions about agents' ability to study. StudyBench consists of three tasks, each built on a corpus that defines a domain of expertise, and each paired with a hidden exam. Below is an example coding question and rubric from our DSPy exam.
Humans face this problem of learning new domains constantly, and one of our default answers is *studying*.
Before an exam, even an open-book one, we read the textbook or the literature, think out loud, quiz ourselves, and write our own notes. Much of the expertise comes from the active effort of reading and thinking itself.
In contrast, current agents mostly rely on inference compute to understand the corpus while they’re working. We’d love for AI systems to be able to study from nothing but a corpus of documents that exist naturally on some subject, and to do so as efficiently as a person who learns a programming library largely by reading the documentation and a few tutorials and only a little bit of practice.
To compare procedures for machine studying, we start by defining expertise.
The corpus is always available at test time anyway, so a sufficiently intelligent non-expert agent could in principle always study during the exam.
What distinguishes an expert from a smart novice is a shift of the entire quality/cost curve: higher accuracy at the same budget, or the same accuracy at a smaller budget. We call the (appropriately weighted) area “expertise”, and the goal of studying is to raise expertise.
Thanks to my great co-authors: Rick Battle, @lateinteraction
Post (with links to the dataset!): https://jacobxli.com/blog/2026/machine-studying/
Great question! Optimizing the index/search actually counts as a possible studying algorithm, as long as it happens before the downstream task is known. Studying isn’t limited to just updating weights or context. The reason I don’t think ‘just optimize search’ resolves the problem is that retrieval has two separable bottlenecks: finding relevant material, and second, recognizing what’s important once you have it. Section 2 covers the first challenge, showing how agents’ biases shape what they even try to search for. Section 8 is about the second: in the literature review task, both models can already retrieve most of the relevant papers, but the older model struggles to pick out the most related ones or synthesize them into a review, especially for more recent papers. That’s exactly how we land on one of the conclusions that retrieval is no substitute for expertise!
@jacobli99 @lateinteraction You should check this out
@jacobli99 love seeing intelligence defined like this -- equivalent to @fchollet's "skill acquisition efficiency"
@jacobli99 @lateinteraction interested in this problem framing, especially for uncontaminated data
@jacobli99 tagging the @rickbattlephoto himself here :D
@jacobli99 @lateinteraction And second, I don’t imagine experts as encyclopedias of information. You know one by what they do with that information, not that they have it. It seems like expertise hinges on applying that knowledge and not just knowing it, but disambiguating that with models is hard!
@jacobli99 @CShorten30 @lateinteraction Reframing continual learning away from catastrophic forgetting toward incremental knowledge incorporation is the more useful mental model for practitioners. Most real problems are about absorbing new information without disrupting what already works, not about pure forgetting.
@jacobli99 @a1zhang @lateinteraction Catastrophic forgetting always sounds like a solved problem until you try updating a model on new data and watch old behaviors degrade in ways that don't surface until a week into prod. The task boundary assumption is the first thing that breaks in real deployments.
@jacobli99 @lateinteraction because the hard part was never scaling knowledge, it's making that incremental progress count in real-world decisions. most approaches to cl are focused on tweaking what we already know, not truly acquiring new understanding
@jacobli99 @lateinteraction This is super interesting work and I’m excited to see where y’all go with it!
@lateinteraction @jacobli99 Thanks a lot :)
@lateinteraction @jacobli99 The Sonnet/Qwen example is perfect. Stale priors overriding correct search is exactly what happens in production with post-cutoff libraries. The model doesn't just not know. It 'knows' wrong.
