/Tech1h ago

Researchers Train LLMs for Inductive Reasoning Using Probabilistic Programs

4214121.6K

Original post

1/7 🚨 New preprint : Using Probabilistic Programs to Train Inductive Reasoning in Large Language Models 👥 in collaboration with/ @akjagadish, @lakebrenden, and @cocosci_lab

Most LLM reasoning post-training focuses on deductive reasoning problems: tasks with single verifiable answers.

However, much of real-world reasoning is inductive: sparse evidence, latent causes, uncertain beliefs.

Examples: forecasting, diagnosis, scientific inference, or everyday judgment.

Current LLMs perform poorly on inductive reasoning. How do we remedy that?

11:12 AM · Jun 16, 2026 · 1K Views

Sentiment

Sentiment building, check back later.

Cluster Engagement

Digg Deeper

No Digg Deeper questions have been answered for this story yet.

ARXIV.ORGVia

#407

Posts from X

Most Activity

VIEWS460BOOKMARKS7LIKES15RETWEETS3REPLIES1

Brenden Lake@LakeBrenden

Everyone's focused on post-training for deduction, but what about induction? New preprint from @LiyiZhang_Leo distills probabilistic programs into LLMs to boost inductive reasoning, and it generalizes across tasks and domains https://arxiv.org/abs/2606.09856

1h460157

Liyi Zhang@LiyiZhang_Leo

6/7 Main results:

PPT improves estimation on held-out inductive tasks, including unseen motifs and a held-out healthcare domain.

It aligns much more closely with human judgments. It also transfers to OpenEstimate and multiple-choice benchmarks, improving log-likelihood and calibration.

3h26

Liyi Zhang@LiyiZhang_Leo

4/7 Our key insight is that probabilistic programs capturing open-world knowledge can act as a bridge:

LLMs come with rich real-world knowledge and are adept at generating scenarios that mimic them.

LLM-based workflows can generate probabilistic programs that capture open-world reasoning; see Wong et al. 2025

Probabilistic inference provides grounded, distributional supervision.

Together, this provides us a way to scale inductive reasoning data without relying only on human labels or stronger LLM labels.

3h24

Liyi Zhang@LiyiZhang_Leo

2/7 Post-training offers a path forward if we have access to a large, well-curated dataset, but the challenge is that good labels are hard to collect, and the right answer is often not a point value but a distribution.

So we ask: how can we train LLMs directly on posterior beliefs, and do so at scale?

3h24

Liyi Zhang@LiyiZhang_Leo

5/7 We generate 10k+ natural-language scenarios and 50k+ queries across sports, healthcare, and general domains.

Then we fine-tune open-source LLMs on posterior labels and evaluate on held-out motifs, held-out domains, human judgments, and external benchmarks.

3h23

Liyi Zhang@LiyiZhang_Leo

3/7 We introduce Program-based Posterior Training (PPT). Pipeline:

1. Use an LLM to generate open-world scenarios

2. Translate each scenario into a Pyro probabilistic program

3. Run probabilistic inference algorithms such as MCMC to get posterior distributions

4. Fine-tune an LLM on these soft labels

3h22

Brenden Lake@LakeBrenden

with @akjagadish and @cocosci_lab!

Brenden Lake@LakeBrenden

1h14010

Liyi Zhang@LiyiZhang_Leo

7/7 A useful takeaway: probabilistic programs are not only test-time reasoning tools, they can also be scalable supervision sources for training LLMs to internalize uncertainty.

Paper: https://arxiv.org/pdf/2606.09856 Code: https://github.com/zhang-liyi/llm-inductive

3h17