/TECH

Berkeley Researchers Introduce Context Distillation for Language Models

WB

Combined views

8.8K

2 posts, first seen 3h ago

200 likes6 comments

Source Posts on X

Cluster sources

will brown@willccbbTECH

cool new approach to address the spike problem in on-policy self-distillation by letting the privileged information steer the student distribution during rollouts directly https://x.com/willccbb/status/2075830477928423734/photo/1

Combined views

8.8K

2 posts, first seen 3h ago

200 likes

198 saves

15 reposts

Useful Links

Learning by Distilling Context(arXiv.org)viawill brown

Digg Deeper

Ask a question below.

Published answers will appear here.

Source Posts on X

Cluster sources

will brown@willccbbTECH

cool new approach to address the spike problem in on-policy self-distillation by letting the privileged information steer the student distribution during rollouts directly https://x.com/willccbb/status/2075830477928423734/photo/1

will brown@willccbbTECH

https://arxiv.org/abs/2209.15189

will brown@willccbbTECH

https://arxiv.org/abs/2209.15189

6 comments

198 saves

15 reposts

Useful Links

Learning by Distilling Context(arXiv.org)viawill brown

Digg Deeper

Ask a question below.

Published answers will appear here.