Announcement
New Metal kernels for speculative decoding land in llama.cpp
Qvac says the Apple Silicon update makes speculative decoding up to 3.4 times faster than plain decoding on an M3 Ultra.
TLDR
Qvac says its pull request adding Metal kernels for speculative decoding on Apple Silicon was merged into llama.cpp. On an M3 Ultra, it reports 110 tokens per second for speculative decoding versus 32.1 for plain decoding. Qvac says speculative decoding on a Mac was slower than plain decoding before the change.
Combined views
34.3K
4 Sources, first seen ago
