Executive Summary
Primary issue: TheTom's current TurboQuant+ build shows a material slowdown on dense Qwen3.6 27B MTP with `K=q8_0, V=turbo3`. In our run, it decoded at about 25 tok/s, while QuinsZouls TurboQuant on the same 27B family previously reached about 55.6 tok/s, and mainline llama.cpp q8/q8 reached about 49.1 tok/s.
MoE behavior is different: Qwen3.6 35B-A3B does not show the same catastrophic slowdown. TheTom reached about 54-56 tok/s at 192k total ctx, while mainline q8/q8 reached 81-89 tok/s at 64k total ctx.
All reported runs kept the large model tensors in GPU memory. The logs show `CUDA0 model buffer size` for the main weights; CPU-mapped buffers were small metadata/host mappings rather than the model being served from system RAM.
Lineage note: TheTom's README describes `TheTom/llama-cpp-turboquant` as the llama.cpp TurboQuant+ runtime integration and a long-lived upstream-tracking feature branch. The QuinsZouls runtime used here is a separate downstream llama-cpp-turboquant fork with the same `turbo3` KV format support, pinned older than TheTom's later upstream MTP sync. I would describe QuinsZouls as derivative/downstream of the same TurboQuant+ work, but not as a direct proof of commit ancestry without a full merge-base comparison.
Result Matrix
| Runtime | Model | Context | KV | MTP | Wall s | Prefill tok/s | Decode tok/s | Accept | VRAM | Notes |
|---|---|---|---|---|---|---|---|---|---|---|
| QuinsZouls llama-cpp-turboquant 1e5a46d, older fork | Qwen3.6 27B Q3_K_XL MTP | 192k total, np=1 | K=q8_0, V=turbo3 | draft-mtp, n=2 | 10.19 | 648.45 | 55.64 | 74.9% | 10.59 GiB earlier net / 21.47 GiB used in loaded 128k single-slot session | Known-good TurboQuant baseline. Quins could not run MTP with np=3. |
| TheTom llama.cpp + TurboQuant+ b9418-2cbfdc62 | 27b-q3xl-mtp | 192k total, np=3, 64k/slot | K=q8_0, V=turbo3 | draft-mtp n=2 | 23.42 | 446.78 | 24.81 | 77.2% | 19949 MiB net; 3329 MiB free | Dense 27B path is anomalously slow; 35B-A3B is not similarly affected. |
| TheTom llama.cpp + TurboQuant+ b9418-2cbfdc62 | 35a3b-q3xl-mtp | 192k total, np=3, 64k/slot | K=q8_0, V=turbo3 | draft-mtp n=2 | 10.99 | 656.51 | 56.41 | 74.8% | 18558 MiB net; 4725 MiB free | MoE path behaves much better than dense 27B under same runtime. |
| TheTom llama.cpp + TurboQuant+ b9418-2cbfdc62 | 35a3b-q3xl-nonmtp | 192k total, np=3, 64k/slot | K=q8_0, V=turbo3 | none | 11.03 | 783.39 | 54.36 | n/a | 17543 MiB net; 5741 MiB free | MoE path behaves much better than dense 27B under same runtime. |
| Mainline llama.cpp CUDA b9305 CUDA build | 27b-q3xl-mtp | 64k total, np=3, 21.5k/slot | K=q8_0, V=q8_0 | draft-mtp n=2 | 12.80 | 571.94 | 49.14 | 75.7% | 17611 MiB net; 5552 MiB free | Mainline recovers expected 27B speed; 35B-A3B reaches 80-90 tok/s in sequential smoke test. |
| Mainline llama.cpp CUDA b9305 CUDA build | 35a3b-q3xl-mtp | 64k total, np=3, 21.5k/slot | K=q8_0, V=q8_0 | draft-mtp n=2 | 8.05 | 703.67 | 81.36 | 72.7% | 17785 MiB net; 5277 MiB free | Mainline recovers expected 27B speed; 35B-A3B reaches 80-90 tok/s in sequential smoke test. |
| Mainline llama.cpp CUDA b9305 CUDA build | 35a3b-q3xl-nonmtp | 64k total, np=3, 21.5k/slot | K=q8_0, V=q8_0 | none | 7.28 | 877.76 | 89.36 | n/a | 16862 MiB net; 6204 MiB free | Mainline recovers expected 27B speed; 35B-A3B reaches 80-90 tok/s in sequential smoke test. |
Key Comparisons
- TheTom 27B MTP TQ3 vs mainline 27B MTP q8/q8: 24.81 tok/s vs 49.14 tok/s. Mainline is roughly 1.98x faster despite using q8 value cache instead of turbo3.
- TheTom 27B MTP TQ3 vs QuinsZouls 27B MTP TQ3: 24.81 tok/s vs 55.64 tok/s. QuinsZouls is roughly 2.24x faster in the prior single-slot 192k-ish profile.
- Mainline 35B-A3B non-MTP vs MTP: 89.36 tok/s vs 81.36 tok/s. MTP=2 did not pay for itself in this short sequential smoke test.
- TheTom 35B-A3B MTP vs non-MTP: 56.41 tok/s vs 54.36 tok/s. MTP helped only slightly in the TheTom 192k profile.
VRAM And Allocation Evidence
The most important evidence that this is not RAM spill: each model reports large `CUDA0 model buffer size` values. The 27B dense model also carries much larger recurrent-state and KV cost than 35B-A3B.
TheTom 27B Q3XL MTP, 192k total, np=3, q8/turbo3
0.01.455.321 I load_tensors: CPU model buffer size = 0.00 MiB 0.01.455.322 I load_tensors: CUDA0 model buffer size = 0.00 MiB 0.01.475.668 I llama_kv_cache: CUDA0 KV buffer size = 0.00 MiB 0.01.512.217 I llama_memory_recurrent: CUDA0 RS buffer size = 1346.62 MiB 0.01.527.057 I sched_reserve: CUDA0 compute buffer size = 124.72 MiB 0.01.527.061 I sched_reserve: CUDA_Host compute buffer size = 36.54 MiB 0.05.507.470 I load_tensors: CPU_Mapped model buffer size = 682.03 MiB 0.05.507.474 I load_tensors: CUDA0 model buffer size = 13410.41 MiB 0.10.169.845 I llama_kv_cache: CUDA0 KV buffer size = 4359.38 MiB 0.10.459.982 I llama_memory_recurrent: CUDA0 RS buffer size = 1346.62 MiB 0.10.481.902 I sched_reserve: CUDA0 compute buffer size = 124.72 MiB 0.10.481.906 I sched_reserve: CUDA_Host compute buffer size = 36.54 MiB 0.10.637.248 I llama_kv_cache: CUDA0 KV buffer size = 272.46 MiB 0.10.662.026 I sched_reserve: CUDA0 compute buffer size = 124.72 MiB 0.10.662.030 I sched_reserve: CUDA_Host compute buffer size = 36.54 MiB
TheTom 35B-A3B Q3XL MTP, 192k total, np=3, q8/turbo3
0.01.441.897 I load_tensors: CUDA0 model buffer size = 0.00 MiB 0.01.441.898 I load_tensors: CUDA_Host model buffer size = 0.00 MiB 0.01.462.359 I llama_kv_cache: CUDA0 KV buffer size = 0.00 MiB 0.01.478.236 I llama_memory_recurrent: CUDA0 RS buffer size = 565.31 MiB 0.01.492.590 I sched_reserve: CUDA0 compute buffer size = 125.22 MiB 0.01.492.594 I sched_reserve: CUDA_Host compute buffer size = 33.51 MiB 0.06.028.657 I load_tensors: CPU_Mapped model buffer size = 515.31 MiB 0.06.028.659 I load_tensors: CUDA0 model buffer size = 15903.70 MiB 0.11.865.042 I llama_kv_cache: CUDA0 KV buffer size = 1362.30 MiB 0.11.964.267 I llama_memory_recurrent: CUDA0 RS buffer size = 565.31 MiB 0.11.984.683 I sched_reserve: CUDA0 compute buffer size = 125.22 MiB 0.11.984.686 I sched_reserve: CUDA_Host compute buffer size = 33.51 MiB 0.12.184.236 I llama_kv_cache: CUDA0 KV buffer size = 136.23 MiB 0.12.201.197 I sched_reserve: CUDA0 compute buffer size = 124.21 MiB 0.12.201.201 I sched_reserve: CUDA_Host compute buffer size = 33.51 MiB
TheTom 35B-A3B Q3XL non-MTP, 192k total, np=3, q8/turbo3
0.01.453.740 I load_tensors: CUDA0 model buffer size = 0.00 MiB 0.01.453.741 I load_tensors: CUDA_Host model buffer size = 0.00 MiB 0.01.474.356 I llama_kv_cache: CUDA0 KV buffer size = 0.00 MiB 0.01.480.261 I llama_memory_recurrent: CUDA0 RS buffer size = 188.44 MiB 0.01.494.824 I sched_reserve: CUDA0 compute buffer size = 125.22 MiB 0.01.494.828 I sched_reserve: CUDA_Host compute buffer size = 33.51 MiB 0.05.716.756 I load_tensors: CPU_Mapped model buffer size = 515.31 MiB 0.05.716.758 I load_tensors: CUDA0 model buffer size = 15539.34 MiB 0.11.402.050 I llama_kv_cache: CUDA0 KV buffer size = 1362.30 MiB 0.11.478.868 I llama_memory_recurrent: CUDA0 RS buffer size = 188.44 MiB 0.11.499.989 I sched_reserve: CUDA0 compute buffer size = 125.22 MiB 0.11.499.994 I sched_reserve: CUDA_Host compute buffer size = 33.51 MiB
mainline64k-27b-q3xl-mtp-np3-q8q8
0.02.454.777 I load_tensors: CPU_Mapped model buffer size = 682.03 MiB 0.02.454.781 I load_tensors: CUDA0 model buffer size = 13410.41 MiB 0.07.100.164 I llama_kv_cache: CUDA0 KV buffer size = 2142.00 MiB 0.07.299.042 I llama_memory_recurrent: CUDA0 RS buffer size = 1346.62 MiB 0.07.310.725 I sched_reserve: CUDA0 compute buffer size = 127.24 MiB 0.07.310.729 I sched_reserve: CUDA_Host compute buffer size = 15.89 MiB 0.07.705.568 I llama_kv_cache: CUDA0 KV buffer size = 133.88 MiB 0.07.720.020 I sched_reserve: CUDA0 compute buffer size = 140.61 MiB 0.07.720.024 I sched_reserve: CUDA_Host compute buffer size = 15.89 MiB 0.09.340.728 I sched_reserve: CUDA0 compute buffer size = 263.75 MiB 0.09.340.732 I sched_reserve: CUDA_Host compute buffer size = 15.89 MiB
mainline64k-35a3b-q3xl-mtp-np3-q8q8
0.04.447.608 I load_tensors: CPU_Mapped model buffer size = 515.31 MiB 0.04.447.609 I load_tensors: CUDA0 model buffer size = 15903.70 MiB 0.10.129.782 I llama_kv_cache: CUDA0 KV buffer size = 669.38 MiB 0.10.249.805 I llama_memory_recurrent: CUDA0 RS buffer size = 565.31 MiB 0.10.259.724 I sched_reserve: CUDA0 compute buffer size = 125.22 MiB 0.10.259.729 I sched_reserve: CUDA_Host compute buffer size = 12.87 MiB 0.10.536.756 I llama_kv_cache: CUDA0 KV buffer size = 66.94 MiB 0.10.548.193 I sched_reserve: CUDA0 compute buffer size = 125.22 MiB 0.10.548.197 I sched_reserve: CUDA_Host compute buffer size = 12.87 MiB 0.12.055.874 I sched_reserve: CUDA0 compute buffer size = 248.37 MiB 0.12.055.879 I sched_reserve: CUDA_Host compute buffer size = 12.87 MiB
mainline64k-35a3b-q3xl-nonmtp-np3-q8q8
0.03.963.331 I load_tensors: CPU_Mapped model buffer size = 515.31 MiB 0.03.963.333 I load_tensors: CUDA0 model buffer size = 15539.34 MiB 0.09.725.542 I llama_kv_cache: CUDA0 KV buffer size = 669.38 MiB 0.09.771.316 I llama_memory_recurrent: CUDA0 RS buffer size = 188.44 MiB 0.09.780.799 I sched_reserve: CUDA0 compute buffer size = 125.22 MiB 0.09.780.804 I sched_reserve: CUDA_Host compute buffer size = 12.87 MiB
Reproduction Flags
Common TheTom profile: llama-server.exe -m MODEL.gguf --ctx-size 192000 --parallel 3 --gpu-layers 99 --flash-attn on --cache-type-k q8_0 --cache-type-v turbo3 --cache-ram 0 --batch-size 512 --ubatch-size 128 --no-webui MTP add-on: --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-type-k q8_0 --spec-draft-type-v turbo3 Common mainline profile: llama-server.exe -m MODEL.gguf --ctx-size 64000 --parallel 3 --gpu-layers 99 --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 --batch-size 512 --ubatch-size 128 --no-webui MTP add-on: --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-type-k q8_0 --spec-draft-type-v q8_0