SGLang’s multi-item scoring reportedly reduces decision-model latency

LMSYS describes SGLang features for decision models that score options for classification, ranking and agent action selection. It says the /v1/score endpoint returns scores for the exact labels requested, such as Yes/No, while multi-item scoring (MIS) computes shared context once and keeps each candidate isolated. LMSYS reports that MIS latency stayed nearly flat from 2 to 16 candidates. For 16 candidates on Qwen3-8B, it reports 95th-percentile latency of 20.6 ms with MIS, compared with 54.1 ms using Generate.