Announcement
Resin adds tiered storage for its KV cache
Resin's built-in CPU memory provider stores reusable checkpoints outside GPU memory; custom providers can implement other tiers.
TLDR
Resin's developer announced on X that the inference-engine library can use CPU memory to extend storage for its KV cache, with asynchronous fetches. The project's documentation says stored checkpoints can let Resin reuse model state after it leaves GPU memory. If a stored checkpoint is unavailable or fails to load, the request falls back to processing the prompt again.
Combined views
146
2 Sources, first seen ago
