Announcement
New speculative decoders aim to speed up open-source models on OpenDocRouter
LlamaIndex says models such as LightOnOCR-3-1B should see considerably faster inference with the additional decoders.
TLDR
LlamaIndex says it trained additional speculative decoders to reduce latency for open-source models on OpenDocRouter. It expects considerably faster inference with models such as LightOnOCR-3-1B. LlamaIndex measured tokens per second for single requests and at 64-request saturated throughput. TeleOCR saw less speedup at scale, which LlamaIndex attributes to its many tiny calls, where decoding latency matters less.
Combined views
2.5K
4 Sources, first seen ago
