Continual learning is widely discussed right now, but mostly as improving on the job or avoiding catastrophic forgetting. But it has a different, difficult, and already urgent form:
Given nothing but a corpus of documents, how should AI systems develop expertise in a new, unfamiliar domain? We call this problem Machine Studying.
Users praise the 'Machine Studying' proposal for aptly naming AI's limits in gaining domain expertise from documents and for linking intelligence to skill acquisition efficiency.
No Digg Deeper questions have been answered for this story yet.
Been extremely excited about this work by @jacobli99!
We're disappointed in the current ways our agents develop expertise in new domains. Very shallow and hand-engineered!
Humans turn reading textbooks or documentation into deep expertise all the time. Why can’t our agents?!
Continual learning is widely discussed right now, but mostly as improving on the job or avoiding catastrophic forgetting. But it has a different, difficult, and already urgent form:
Given nothing but a corpus of documents, how should AI systems develop expertise in a new, unfamiliar domain? We call this problem Machine Studying.
one of the least expected side effects of this effort is that it gives us a measurable definition of agent “intelligence” that i’m happy with for the first time
Continual learning is widely discussed right now, but mostly as improving on the job or avoiding catastrophic forgetting. But it has a different, difficult, and already urgent form:
Given nothing but a corpus of documents, how should AI systems develop expertise in a new, unfamiliar domain? We call this problem Machine Studying.
"and by this token, it’s not obvious that even the most knowledgeable of current agents are very smart!"
Studying gives us a second curve: expertise as a function of study compute. You could consider its weighted area a notion of “intelligence”. An intelligent agent, for our purposes, is one that can acquire expertise in totally new domains really efficiently. (And by this token, it’s not obvious that even the most knowledgeable of current agents are very smart!)
@jacobli99 Read Jacob's thread or jump right into his blog post at https://jacobxli.com/blog/2026/machine-studying/
Been extremely excited about this work by @jacobli99!
We're disappointed in the current ways our agents develop expertise in new domains. Very shallow and hand-engineered!
Humans turn reading textbooks or documentation into deep expertise all the time. Why can’t our agents?!
and it appears that current agents are NOT very intelligent by that measure yet - but we might not be that far off from figuring out how to do it, now that’s it’s clearly defined and benchmarkable
one of the least expected side effects of this effort is that it gives us a measurable definition of agent “intelligence” that i’m happy with for the first time
Studying gives us a second curve: expertise as a function of study compute. You could consider its weighted area a notion of “intelligence”. An intelligent agent, for our purposes, is one that can acquire expertise in totally new domains really efficiently. (And by this token, it’s not obvious that even the most knowledgeable of current agents are very smart!)
So how should an agent actually study? We test the simplest version of three natural bets out there: (1) self-supervised objectives, e.g. continual pre-training over the corpus; (2) synthesizing your own training data, e.g. turning the corpus into Q&A pairs to fine-tune on; and (3) amortized context management, e.g. having the agent write itself a cheatsheet.
Our preliminary finding is that none of these reliably turns exposure to the corpus into as much expertise as we’d like yet. Neither memorization of the contents of the corpus nor the ability to retrieve things stands in as a substitute for deeper expertise.
Read the blog for the details on all three paradigms, and what we think it'll take to do better!
To compare procedures for machine studying, we start by defining expertise.
The corpus is always available at test time anyway, so a sufficiently intelligent non-expert agent could in principle always study during the exam.
What distinguishes an expert from a smart novice is a shift of the entire quality/cost curve: higher accuracy at the same budget, or the same accuracy at a smaller budget. We call the (appropriately weighted) area “expertise”, and the goal of studying is to raise expertise.
We instantiate this in StudyBench, a benchmark we’re building so that we and others can begin to ask questions about agents' ability to study. StudyBench consists of three tasks, each built on a corpus that defines a domain of expertise, and each paired with a hidden exam. Below is an example coding question and rubric from our DSPy exam.
Humans face this problem of learning new domains constantly, and one of our default answers is *studying*.
Before an exam, even an open-book one, we read the textbook or the literature, think out loud, quiz ourselves, and write our own notes. Much of the expertise comes from the active effort of reading and thinking itself.
In contrast, current agents mostly rely on inference compute to understand the corpus while they’re working. We’d love for AI systems to be able to study from nothing but a corpus of documents that exist naturally on some subject, and to do so as efficiently as a person who learns a programming library largely by reading the documentation and a few tutorials and only a little bit of practice.
@jacobli99 putting the link here for those that want to jump right into the long form:
https://jacobxli.com/blog/2026/machine-studying/
Thanks to my great co-authors: Rick Battle, @lateinteraction
Post (with links to the dataset!): https://jacobxli.com/blog/2026/machine-studying/
@jacobli99 @lateinteraction I’m not up to date on training methods, so this may be a dumb question, but if expertise is defined by the quality/cost curve, then why not optimize index and search?
@jacobli99 @lateinteraction You should check this out
Great question! Optimizing the index/search actually counts as a possible studying algorithm, as long as it happens before the downstream task is known. Studying isn’t limited to just updating weights or context. The reason I don’t think ‘just optimize search’ resolves the problem is that retrieval has two separable bottlenecks: finding relevant material, and second, recognizing what’s important once you have it. Section 2 covers the first challenge, showing how agents’ biases shape what they even try to search for. Section 8 is about the second: in the literature review task, both models can already retrieve most of the relevant papers, but the older model struggles to pick out the most related ones or synthesize them into a review, especially for more recent papers. That’s exactly how we land on one of the conclusions that retrieval is no substitute for expertise!
you want to read jacob’s work before the content vultures arrive.
omar’s lab n e v e r misses
Continual learning is widely discussed right now, but mostly as improving on the job or avoiding catastrophic forgetting. But it has a different, difficult, and already urgent form:
Given nothing but a corpus of documents, how should AI systems develop expertise in a new, unfamiliar domain? We call this problem Machine Studying.
@jacobli99 tagging the @rickbattlephoto himself here :D
@jacobli99 @lateinteraction interested in this problem framing, especially for uncontaminated data
@jacobli99 @lateinteraction And second, I don’t imagine experts as encyclopedias of information. You know one by what they do with that information, not that they have it. It seems like expertise hinges on applying that knowledge and not just knowing it, but disambiguating that with models is hard!
If we are ever to build machines that can operate in new domains like experts, either we must reduce each domain to a sufficiently verifiable environment, or we must develop machines that can study to gain expertise on their own. Machine Studying explores the second possibility!
Continual learning is widely discussed right now, but mostly as improving on the job or avoiding catastrophic forgetting. But it has a different, difficult, and already urgent form:
Given nothing but a corpus of documents, how should AI systems develop expertise in a new, unfamiliar domain? We call this problem Machine Studying.
