Speculative decoders aim to cut latency for open-source models on OpenDocRouter

LlamaIndex said in an Oct. 19, 2026, post that it had trained additional speculative decoders to help reduce latency for open-source models on OpenDocRouter. It expects considerably faster inference with models including LightOnOCR-3-1B. The post does not give a numerical speedup for that model.

OpenDocRouter describes itself as an API that turns PDFs and images into markdown using a model the user chooses. Its site says pages run as individual model calls as capacity becomes available.

LlamaIndex said it measured tokens per second for single requests, which it called the best case, and for 64 requests at saturated throughput. It said TeleOCR saw less speedup at scale because it makes many tiny calls, where decoding latency matters less. The post gives no figures or baseline for assessing those comparisons.