⚡ Local Inference Recipes

English | 简体中文 | 日本語

Setups with RTX 5090: local inference speed (tok/s), sourced

As of 2026-11-14, this dataset has 106 recipes for RTX 5090. 60 were measured by the poster. Fastest measured single-device, single-stream text result: 538.1 tok/s with Qwen3.8-27B on TensorFold (1× RTX 5090; Measured, 2026-10-26; source). 85 entries have a date, from 2026-07-22 to 2026-11-14.

Filter these in the explorer →

ModelEngineHardwaretok/sMetricSource levelDateSource
Qwen3.8-27BTensorFold1× RTX 5090538.1harness decode on a fresh server: prose 278.3, code 538.1 (with DFlash2 drafts), structured 352.6; decode 222–225 after a 100K cold promptMeasured2026-10-26link
Qwen3.8-Flash-NextBasalt1× RTX 5090 32GB + 59GB RAM500.4Basalt v0.3.0 (named in the linked tree toolcall/basalt93, not inferred from the latest release at post time). Measured 2026-11-07. Headline is MTP-draft decode 500.4 (9 clean runs, 700 tokens each); no-draft 259.0 is separate. Same-weight llama.cpp comparison 98.3 / 89.1. Context is not named, so it is left null. Pre-registered rule and public tree: github.com/notwitcheer/llm-bench-rig/tree/main/toolcall/basalt93.Measured2026-11-07link
Qwen3.8-Flash-NextQEngine2× RTX 5090 32GB + Threadripper PRO 3975WX + 251GB RAM370club-3090 bench.sh: decode narrative / code 370 / 439 (KV in VRAM), 406 / 504 with FLASH_KV=ram; prefill about 5.7K at 10K and about 7.3K at 90K. Strata IQ3_S on the same machine 214 / 293. 8 streams at 64K: 124 per stream (RAM KV) vs 139 (VRAM KV)Measured2026-10-31link
LFM2.5-24B-A2Bllama.cppRTX 5090 32GB351.5351.5 tok/s; math problem 1.98 s; GSM8K 97%, MMLU 80%; same GGUF, same llama.cpp build; writing speed median of 5 runsMeasured2026-10-15link
NVIDIA Nemotron 3.5 Bolt 74B-A9Bllama.cppRTX 5090 32GB (Ubuntu, driver 570)343Single stream, median of 5 prompts × 3 runs (512 tokens, temp 0): drafter on 343 (303–354, 55% acceptance) / off 166; 64K window 342 / 166; f16 KV 313; VRAM 25.0 GiB (on) / 22.7 GiB (off); llama-bench pp512 5,166 / tg128 172Measured2026-10-20link
Qwen3.8-Flash-NextStrataRTX 5090 + Ryzen 9 9950X (Windows)270.7Q2_0 short 270.7, 4K 258.0, 16K 220.4; IQ3_S short 219.5, 4K 195.0, 16K 208.9; sanity 12/12Measured2026-10-26link
Qwen3.8-27BTensorFold1× RTX 5090 32GB268.2TensorFold 1.1.2 (BobClawblaw branch q35-prompt-int8 @18d93451), FP8 KV: decode code 268.2 / prose 238.7; 32K TTFT 7.7 s; 27.3 GiB at 262K context. The recipe serves 1 stream, so 8 requests total 204.6. Same card: NInfer MTP3 169.0, Mia EXL3 4.0bpw 154.5; repeated on 2026-11-01 with TensorFold 1.2.0: code 268.6 / prose 248.6, 32K prefill 3,954 (5.5–7.7% lower than 1.1.2), 8-request total 211.8 (x.com/niklaslenz_ai/status/2116829349278359748)Measured2026-10-25link
Nemotron 3.5 Bolt 74B-A9BTensorFold1× RTX 5090 32GB265.5Mia recipe: decode prose 265.5 / code 219.8; 500 words in about 2.8 s; first token at 128K 18.7 s. 2 slots (not the 16 slots of Mia), no vision. Median of 2 runs, 512 tokens, thinking offMeasured2026-11-03link
Qwen3.8-Flash-NextBasaltRTX 5090 32GB + Core Ultra 9 285K + 247GB DDR5242decode at 16K: basalt 242 / capy-flash 198 / ninfer-ext 92; basalt 216 at 128K; basalt prefill 8.3–9.2KMeasured2026-10-23link
Qwen3.8-Flash-NextStrata1× RTX 5090 32GB (PCIe Gen3 x16, shared with another job) + two Xeon Gold 5218 CPUs (no AVX-512)231.5Strata official Docker 0.2.13 (engine 0.1.59), configured 262,144, 1 slot. Two strata bench run standard-v1 runs, both complete, sanity 12/12. Headline is the first run (expert cache 3,090 slots): short decode 231.5. 4K 242.7, 16K 214.4. Prompt read 4K/16K: 2,373/5,148. The second run (14,570 slots after another job on the card stopped) gives short 242.5; the author says this is inside the spread of a 3-sample median. A 1-hour soak (not the standard protocol) completed 4,025 of 4,025 requests at 262-266 decode, note only. Earlier result on the same machine: engine 0.1.47 (Docker 0.2.2-cuda13), 1M context, --kv q4_0, --kv-resident 16384 gave single-request decode 82.5, and 93.8 with 5.5GB more free (https://github.com/Niko1221/Strata/issues/5852, no raw log, claim). No rack evidence for the dual Xeon box, so it is listed as a workstation. outputs.jsonl and results.json are in the PR.Measured2026-11-13link
Qwen3.8-Flash-NextStrataRTX 5090 32GB + 121GB DDR5227Strata 0.2.10 main value 227 (chat decode; same post: chat 152 → 227, +49%, first token 0.27 → 0.17 s; x.com/lyesbahi/status/2117545259358196187). History: Strata 0.2.3 (engine 0.1.48) single stream 240, 115 per user with 2 users; that method differs from the 0.2.10 chat decode, so do not compare them directly; author says no speed change on this card (the 2026-10-21 post used a different test: decode 185, peak 225, expert cache hit 98.5%, 9K prompt in about 1.4 s, x.com/lyesbahi/status/2112809633261969621); source post x.com/lyesbahi/status/2113602136051458520Measured2026-11-03link
Recursive Flint 40B-A2.5Bllama.cpp1× RTX 5090 32GB224.8llama.cpp b11803: code 224.8, prose 224.9; first token after 5.7 s for a 32K prompt; total for 1/2/8 requests at once 193.9 / 198.6 / 198.5 (1 slot only). Flint ignores the flag that turns thinking off. Same card: NInfer Qwen3.8-27B 169.0Measured2026-11-01link
MiniCPM-V 4.7 35B-A3Bllama.cppRTX 5090208Short-context decode ~208 tok/s; at 258K: decode 127, prefill 3.6K tok/s, TTFT 72 s (fresh prompt, median of 3); DSpark drafter slower at every length on llama.cppMeasured2026-10-15link
Qwen3.8-Flash-NextStrataRTX 5090 32GB + system RAM (partial offload)204Strata+MTP 204 tok/s (122 without MTP); writing the same city page 28.8 s; prefill median 4K ~3,790 / 32K ~5,230 tok/s; all 15 quality checks passed, only 1 of 27 repeats had identical text (tok_s is the speed of a single writing task; the post did not separate prose/code)Measured2026-10-14link
Qwen3.8-Flash-NextStrataRTX 5090 32GB (limited to 450W) + Ryzen 7 9800X3D + 60.5GB DDR5202.1bench.sh --long median: chat decode 202.1 / agent decode 223.8 tok/s; prefill 4K/16K/32K = 4,473 / 6,656 / 6,827 tok/sMeasured2026-10-13link
Qwen3.8-27BVoidInferRTX 5090 32GB1931 request, 1,024 output tokens, 3 runs per size: VoidInfer 3K 193 / 32K 177 / 97K 146 / 200K 127; NInfer 219 / 197 / 176 / 153. NInfer prefill is about 2–2.5× fasterMeasured2026-10-28link
Qwen3.8-Flash-NextStrataRTX 5090 32GB + RTX 4070 Ti SUPER 16GB (chipset PCIe x1) + i7-14700K + 32GB RAM186.3strata_bench short chat 186.3 (153.7–254.8); prefill 4K 2,446 / 16K 3,917; Strata v0.2.1 + 5 ExTV patchesMeasured2026-10-23link
Qwen3.8-Flash-NextStrataRTX 5090 32GB + Core Ultra 7 265K + 128GB RAM (Windows 11)176.3Strata 0.1.48.1 @7ddc17a3, engine 0.1.48, --spec 4 --mtp rt --kv int8, strata_bench suite 1: short chat decode 176.3 (142.3–221.4, median of 9); agent follow-up 188.7; prefill 4K 4,723 / 16K 6,442. 2026-11-13 repeat of the same combination by samykoka (RTX 5090 + Ryzen 9 7950X + 63.1GiB, Windows 11, Strata project 0.1.58 / engine 0.1.59, configured 262,144, strata_bench suite 1): short chat decode 173.6 (151.9-208.4, 9 runs), agent follow-up 222.9. This is 1.5% from the headline, a routine repeat, so the headline does not change; results.json is not attached to the issue. https://github.com/Niko1221/Strata/issues/5906Measured2026-10-25link
Qwen3.8-Flash-NextStrataRTX 5090 + 256GB RAM176176 tok/s decode; 162 at 64K; 5,247 tok/s prefillMeasured2026-09-30link
MiniCPM-V 4.7 35B-A3Bllama.cppRTX 5090 laptop176Q4_K_M: 176 tok/s; with DSpark draft code 272 tok/s (verified from video subtitles). Post also says the one run where Hermes Agent read back the disk tool screenshot was 171 tok/s (single task, not overall speed)Measured2026-10-15link
Qwen3.8-27BTensorFoldRTX 5090 32GB174code 370 | prose 174 | structured 634 tok/s; 32K prompt first token 12.4 s; 5 prompts×3 rounds interleaved, 450 W / 2200 MHz cap; each uses its own 4-bit format and draft (compares recipes, not quantization)Measured2026-10-13link
Qwen3.8-27B + Tencent DSpine K11NInfer1× RTX 5090 32GB (Windows 11)160.1NInfer Windows v1.2.5-dspine.1 (fba899b, headpiece747 port). Porter-author release table: English prose single-stream at 2,048 output tokens is 160.1 (147.7 at 256); Python single-stream at 2,048 is 515.0 and is separate. Eight-stream English total 684.4 is not single-stream. Same table, DFlash2 K9 English at 2,048 is 183.0 (DSpine about 12% slower on prose). The author's X post “1-stream replay” English 179.5 is another method and is not the headline. Context 8,192. Public fixtures and method.Measured2026-11-09link
Qwen3.8-27BQwentin1× RTX 5090158.3code 158.3, prose 139.2 (greedy); aggregate (separate) 8 requests 332.2Measured2026-10-26link
Qwen3.8-Flash-NextStrata1× RTX 5090157.8at 188K depth: Strata decode 157.8, FreeToken 55.0; prefill 7,290 vs 6,903 (median of 3, 512 output, temp 0)Measured2026-10-25link
Qwen3.6-35B-A3Bllama.cppRTX 5090 laptop157Q4: 157 tok/s (same card and task as x-063)Measured2026-10-15link
Qwen3.8-Flash-NextStrataRTX 5090 32GB (600W) + 128GB DDR5-6000 + Core Ultra 7 270K Plus (Windows 11)153.3Native medians (3 requests per context, 256 output, greedy, zero reuse): 4K 153.3 / 32K 155.1 / 128K 143.1 tok/s; same machine and day Infernix Dense8 133.6 / 144.0 / 147.2 (comm-046)Measured2026-10-18link
Qwen3.8-Flash-Next UncensoredInfernixRTX 5090 32GB + 189.6GB RAM + 9950X (Windows)149262K: decode 149.0 / cold prefill 3,548; 512K (YaRN 2): 137.0 / 3,497 tok/s (median of 3 runs)Measured2026-10-10link
Qwen3.8-27B-NVFP4-MTPllama.cppRTX 5090 32GB + Ryzen 7 9800X3D + 64GB RAM (Windows)148.3LOW + MTP (--spec-draft-n-max 4, KV q8_0, 192K): mean of 9 prompts 148.3 tok/s, peak 203, MTP acceptance 49.0%, VRAM 25.0 GiB; VERY-HIGH (BF16 head) + MTP mean 111.9 tok/s (peak 139)Measured2026-08-14link
Qwen3.8-27BvLLMRTX 5090 32GB147147 tok/s decode (mia-ai.net lists about 160)Measured2026-08-15link
MiniCPM-V 4.7 35B-A3BvLLMRTX 5090143Short-context decode ~143 tok/s; at 258K: decode 112, prefill 5.9K tok/s, TTFT 43 s; DSpark drafter 224 tok/s at 4K, slower than no drafter from ~16K, 8 tok/s at 192K (fresh prompt, median of 3; with drafter one 5090 fits only 192K)Measured2026-10-15link
Qwen3.8-Flash-NextInfernixRTX 5090 32GB (600W) + 128GB DDR5-6000 + Core Ultra 7 270K Plus (Windows 11)133.6Native medians (3 requests per context, 256 output, greedy, zero reuse): 4K 133.6 / 32K 144.0 / 128K 147.2 tok/s; first answer 0.4–0.9 s earlier than Strata NVFP4 (comm-045)Measured2026-10-18link
ThinkingCap 27B (BottleCap AI / Kearuga)ExLlama1× RTX 5090 32GB131.8Mia one-click kit @bbb11db + ExLlamaV3 1.6.0, DRAFT=mtp. Measured 2026-11-10. Headline is prose decode median of 2 runs, 131.8; code 140.2 is separate. Configured 262,144; tested to 131,072. Eight-request total 120.7 is not single-stream. The quant card says MTP quality is outside its text qualification; this page also does not qualify MTP. Method, commands and raw files are on the recipe page. In-window post https://x.com/niklaslenz_ai/status/2120626368757047667Measured2026-11-10link
Qwen3.8-Flash-NextStrataRTX 5090123.4123.4 tok/s live; full run 115+Measuredlink
Qwen3.8-Flash-Next UncensoredStrataRTX 5090 32GB + 189.6GB RAM + 9950X (Windows)120.6262K: decode 120.6 / cold prefill 1,550; 512K (YaRN 2): 126.3 / 3,045 tok/s (median of 3 runs)Measured2026-10-10link
Gemma 4 31BNInfernoRTX 5090 Laptop GPU (5090M 24GB)118.4Short-chat decode (greedy/sampled): Fast 118.4 / 114.0, Max-context 106.8 / 99.7, start point 33.2; prefill at 8K 4,154, at about 130K about 1,600Measured2026-10-29link
Nanbeige4.3-33BExLlama1× RTX 5090 32GB + Ryzen 9 9950X3D + about 128GB RAM (desktop running)116.92Fast preset (8,192 capacity, FP16 KV): output 116.92, prefill 3,130, first token 926 ms (2,842 in / 2,048 out, median of 3, temp 0, thinking off); maximum-context preset (180,224, Q4 KV): 115.14; 23.41 with a 179,705-token input, first token after 167.4 s. Five-prompt three-seed suite at temp 0.6: median 103.40. tok_s is the fast presetMeasured2026-10-27link
Qwen3.8-27Bllama.cppRTX 5090 32GB116code 186 | prose 116 | structured 200 tok/s; 32K prompt first token 12.5 s; 5 prompts×3 rounds interleaved, 450 W / 2200 MHz cap; each uses its own 4-bit format and draft (compares recipes, not quantization)Measured2026-10-13link
Qwen3.8-27BExLlamaRTX 5090 32GB114code 258 | prose 114 | structured 366 tok/s; 32K prompt first token 10.4 s; 5 prompts×3 rounds interleaved, 450 W / 2200 MHz cap; each uses its own 4-bit format and draft (compares recipes, not quantization)Measured2026-10-13link
Qwen3.8-Flash-Nextllama.cppRTX 5090 32GB + system RAM (partial offload)6868 tok/s (no draft model); same prompt 104.2 s; prefill median 4K ~1,270 / 32K ~1,355 tok/s; 27 of 27 repeats identical (same x-050 task)Measured2026-10-14link
Qwen3.8-27B-SLQ-EAR98vLLM1× RTX 509067.2code 67.2, prose 67.1; aggregate (separate) 8 requests 351.8; context 74,752Measured2026-10-26link
DeepSeek-V4.1-Flash-1017StrataRTX 5090 + 128GB DDR5 + i9-13900KF (Windows 11)40.66Strata Terminal 0.2.4.1: three runs 34.57 / 40.66 / 42.57, median 40.66 (higher than other reports on similar setups, but the 3 runs agree and a screenshot is attached)Measured2026-10-24link
DeepSeek-V4.1-Flash-1017StrataThreadripper 3960X + RTX 5090 + RTX 3080 Ti + 192GB DDR4 (Windows)38About 38 on average over a 6-turn coding conversation (37.4–38.7), up to 42.4 after warm-up; about 26–30 before the patch; stock defaults 19.4, settings only 29.1; prefill up to 772.1Measured2026-10-30link
DeepSeek-V4.1-Flash-1017Strata1× RTX 5090 32GB + Xeon Gold 6330 + 384GB RAM37strata-dsv41 0.1.55: median decode about 37 (32–42, including 100K–180K prompts); cold prefill 1,612 for a 139K promptMeasured2026-11-01link
DeepSeek-V4.1-Flash-1017Strata1× RTX 5090 32GB35Strata 0.2.10 main value 35 (prose decode 35–38, low end); history: Long-generation average about 50–51 (2.2K-token test, 256K context); prefill 1,350. The numbers come from the post text; the screenshot was not checked; Strata 0.2.10 (2026-11-03): prose decode 35–38, copy/agent 68; prefill 890–2,264 at 10–36K (x.com/lyesbahi/status/2117672208592220602)Measured2026-11-03link
DeepSeek-V4.1-FlashStrata (Strata-DS fork)RTX 5090 Laptop 24GB + Ryzen 9 9955HX3D + 64GB RAM + 2TB NVMe31.2Short prompts from a cold start 30.0–32.0 (median of 5; code prompt 31.2); through the API on a running server 33.2–39.3; 30 minutes non-stop 37.6; with 4K/32K/128K/256K prompts decode 26.1/25.6/25.2/24.4 and prefill 244/844/904/752 tok/s; skips on: HumanEval+ 90.2% (off 91.5%), GSM8K 98.0% (off 97.5%)Measured2026-10-16link
MiniMax-M3.1-FlashProject MayaRTX 5090 32GB + 92GiB RAM24Maya 1.1.50 default install, median of 2 runs of 512 tokens, thinking off: decode 24.0 (code 24.4 / prose 23.6); 18.7 with a 249K prompt. TTFT 36.6 s at 16K, 465.8 s at 128KMeasured2026-11-03link
GLM-5.3-Flash-Maya-S (321B)Unspecified1× RTX 5090 + 64GB DDR518Prefill of a 32K prompt 1,749 (the author says 1,107 on M5 Ultra); decode about 18–25 (the author says 31–41 on a 512GB M5 Ultra)Measured2026-10-30link
DeepSeek-V4.1-Flash-1017StrataRTX 5090 D v2 24GB + 128GB DDR5-4800 + i7-14700KF (Windows 10)8.6Strata 0.2.4 (DSV4.1 engine 0.1.49) defaults (--dspark auto): 256-token streaming, decode short 8.6 / 1.3K 7.9 / 4.9K 10.2, tok_s is short prompt; 4.9K prefill 240; RAM store grew only to about 50GiB, author suspects it is not at full speedMeasured2026-10-24link
Qwen-Image 2.1TensorFoldRTX 5090 @450W4.53 s / 6.94 s / 13.87 s (1024², 25 steps; NVFP4 peak 22.9GB; images not identical to Q6)Measuredlink
Qwen-Image 2.1UnspecifiedRTX 5090 32GB (455W power limit)Qwen-Image 2.1 base model, int8, 25 steps, 1MP: 5.2 s per image (median of the last 200 images), about 687 images per hour; power 399 W (nvidia-smi 60 s average); energy 0.58 Wh per imageMeasured2026-10-11link
Qwen-Image-2.1-Turbo (official 8 steps)UnspecifiedRTX 509029 s vs 108 s / image (2048²)Measuredlink
Z-Image Turbo vs Qwen-Image 2.1UnspecifiedRTX 5090Z-Image ~2× faster (no absolute times); quality: Qwen wins 9:6Measuredlink
Qwen3.8-27BSGLangRTX 5090 / 1× DGX Spark206 tok/s / 38 tok/sMeasuredlink
Qwen3.6-35B-A3BUnspecifiedRTX 5090No draft 275.5 → 312.4 (best 318.3)Measuredlink
Qwen3.8-27BoMLXMac Studio M5 Ultra 256GB / M3 Ultra 512GB / RTX 50906K prompt: 48 / 31 / 59 tok/s; PP 1,701 / 414 / 3,031; 256K: 24.3 / 15.0 / 30.0 (5090 needs 8-bit KV)Measuredlink
MiniMax H3 (base, 50-step class)ComfyUIRTX 50901280×736×5s: 242 s → 168 sMeasuredlink
MiniMax H3 vs FastH3 TurboUnspecifiedRTX 509015 s clip: 11 min 55 s vs 2 min 6 sMeasuredlink
LTX-2.3 (with lip sync)ComfyUIRTX 509010 s about 90 s (weaker GPU about 6×)Measuredlink
LTX-2.5 (t2v)UnspecifiedRTX 5090 (500W)720p/10 s: 61.11 s; 1080p/10 s: 189.90 sMeasuredlink
MiniMax H3ComfyUILaptop RTX 5090 24GB + 64GB RAMThree 5s chained clips: 17 min totalMeasured2026-09-22link
Qwen3.6-35B-A3BSGLang1× RTX 5090626sglang-elsa (author's fork, MTP + FR-Spec, thinking off): C=1 626 (stock SGLang 0.5.22 without MTP 310, vLLM 267 on the same card); aggregate (separate) C=4 1,343 / C=8 1,956Official2026-10-24link
Qwen3.6-35B-A3BsparkinferRTX 5090512512 tok/s (claimed 86% faster than llama.cpp)Claimedlink
Qwen3.8-Flash-NextBasalt1× RTX 5090 (400W limit) + 128GB RAM3547-engine comparison (greedy, thinking off, int8 KV, cold start per request, single run): short context Struct/Conv/Prose: Basalt 354/188/180, Infernix 349/170/163, capy-flash 292/165/152, Strata 204/146/123, llama.cpp 39 (MTP off); 200K depth Basalt 160; 100K prefill Basalt 8,666. tok_s is Basalt Struct. Conflict of interest: the poster is the Basalt developerOfficial2026-10-24link
gpt-oss-20bllama.cppRTX 5090 32GB (host EPYC 7543 + 945GB RAM, as reported by CLI)324.2chat-short decode 324.2 tok/s; prefill 586.8 (166-token prompt); TTFT 282.9 ms (suite-v1, batch=1; same run chat-long prefill 12,638)Claimed2026-10-17link
LFM2.5-24B-A2Bllama.cpp1× RTX 5090 32GB308.5Backfill outside the window (measured 2026-10-15, cited by @llmspeed inside the window: https://x.com/llmspeed/status/2121527998553948452). llm-speed signed run r_c8k6fleym09, llama.cpp commit a0e6dc7, llama-server, all layers on the GPU, 8K context. Headline is chat-short decode 308.5. chat-long 300.4 is separate. Peak VRAM 14.4 GiB. Another author's 5090 row is x-057. Builds and contexts differ between cards (the post says so). llm-speed signed-run pages are claims under the current rule.Claimed2026-10-15link
Qwen3-Coder-30B-A3B-Instructllama.cppRTX 5090 32GB (host EPYC 7543 + 945GB RAM, as reported by CLI)304.9chat-short decode 304.9 tok/s; prefill 514.5 (110-token prompt); TTFT 213.8 ms (suite-v1, batch=1; same run chat-long prefill 8,353)Claimed2026-10-17link
Qwen3.8-Flash-NextBasaltRTX 5090 32GB + RTX 5060 Ti 16GB + 128GB RAM300Basalt 0.2.0: prose about 300, structured about 600, prefill about 10K. NVFP4-DQ8 KLD against BF16 is 0.0561Claimed2026-11-03link
Qwen3.6-35B-A3B-Escha-W2SGLangRTX 5090 32GB283283 tok/s decodeClaimed2026-07-22link
Qwen3.8-27BFoundry Local1× RTX 5090 32GB266Official Microsoft blog chart (RTX 5090): Foundry Local generation speed 4K 265.82 / 16K 260.19 / 32K 234.26 / 65K 200.41 / 131K 212.36 / 260K 187.16; same-setting llama.cpp 150.60 / 164.45 / 151.73 / 146.91 / 109.68 / 96.11. Numbers read from the official chart images, not community relays. Microsoft's own comparison vs llama.cpp, not a third-party reproduction. tok_s uses the 4K value rounded to 266Official2026-11-05link
Qwen3.8-27BTensorFold1× RTX 5090 32GB188.1TensorFold 1.2.7 (stock). Headline is prose decode (default window) 188.1; code 195.2 is separate. The body only says default window; the configured context limit is not stated, left null. The post is truncated at “8 requests at once, 4 streams”; the unread multi-stream figure is not filled in. No raw log; claim. First first-hand 5090 figure for TensorFold 1.2.7 in this window.Claimed2026-11-12link
Qwen3.8-Flash-NextStrata1× RTX 5090 D 32GB187.2262K window at 99.2% full: one stream alone in a 4-slot server 187.2; aggregate (separate) 4 streams 314.0, 8 streams 405.5Claimed2026-11-01link
Qwen3.8-Flash-NextBasalt1× RTX 5090 32GB (PCIe 5.0 x16, 95% power limit) + i9-14900K + 160GB DDR5 (4 sticks, 5200 MT/s) + RAID0 NVMe (Windows)170Basalt commit d10ccd4bc2, community-1 preset, NVFP4, int8 KV. Headline is tg prose 170 from the table (65.1% acceptance). tg conv 172; tg struct 387 is structured output and not the headline. The community-1 cold decode of 297.7 (9,480 tokens in 31.8 s) uses a different basis and is separate. Prefill 16K / 64K: 5,727 / 7,487. The post attaches a community-row JSON (a one-line summary, not per-run data). Context and requests at once are not stated. Claim. Rows from other authors with this combination: comm-072, comm-086.Claimed2026-11-14link
Qwen3.8-27BvLLM2× RTX 5090 (P2P driver)160160 tok/s (comment, no further details)Claimed2026-10-15link
Qwen3.8-Flash-NextStrataRTX 5090 32GB (PCIe 5.0 x8) single-card engine + Ryzen 9 9950X + 96GB DDR5-6000 (two more RTX 3090 cards in the same box run one engine each; three engines share an expert arena of about 50GB in RAM)160Strata self-built v0.2.12 + PR #3341 + #5203 (engine 0.1.58.2), control 0.1.58 + #3565. Three engines share one expert arena in RAM (--shared-expert-arena, about 50GB), --ple-io direct. This row is the 5090 engine: 512K (YaRN x2, KV q4_0), --batch 2 --batch-mtp, 7 cores. Greedy short prompts: decode alone 160-205; the headline is the low end, 160. With all three engines busy: 139-186. Live traffic, 15.5K requests: decode median 164 (p90 192). Box-wide 4 streams give 371-392 aggregate, not a single-stream value. No raw logs, so this is a claim. The 3090 engines are comm-394.Claimed2026-11-13link
Qwen3.8-Flash-NextStrataRTX 5090 (RAM not stated)160160 tok/s (llama.cpp on the same card 15 tok/s)Relayedlink
Nemotron 3.5 Bolt 74B-A9Bllama.cpp1× RTX 5090 32GB (32K window)157.3llm-speed (llama.cpp) short-prompt: 5090@32K 157.3. Same model on TensorFold is x-229 (265.5). Same-suite 4K signed run r_z-l1yelv_67, chat-short decode 154.7, is a note only; the headline stays the original 32K 157.3 and remains a claim. https://llm-speed.com/r/r_z-l1yelv_67Claimed2026-11-05link
Qwen3.8-Flash-NextStrata (TCP layer split fork)2 PCs, 1× RTX 5090 each (14GB RAM per box, TCP layer split)155A Strata fork does the layer split across two machines over TCP, all experts in VRAM: about 155. Same post: Qwen3.8-27B AWQ on one 5090 with vLLM about 145Claimed2026-11-03link
Qwen3.8-Flash-NextStrataRTX 5090 + 64GB RAM151Strata 0.1.43.1: --resident-budget decode 151 tok/s vs --resident-experts 139 (the latter has 1.6× faster prefill)Claimed2026-10-14link
Qwen3.8-Flash-NextStrataRTX 5090 (power limited) + 96GB DDR5-6400150150–200 tok/s decode; 5–6K prefillRelayedlink
Qwen3.8-Flash-Next-GyroStrataRTX 5090 32GB140140.0 tok/s decode; 4883.0 tok/s prefillClaimed2026-09-30link
Edge0-120Bedge01× RTX 5090 D 24GB130.2Official Edge0 model card speed table. Headline is Q3_K chat decode 130.2; agent decode 144.6 is separate. The NVFP4 build on the same machine: chat 124.9 / agent 135.5 (https://huggingface.co/Edge0/Edge0-120B-NVFP4), note only. With the same card capped to 16GiB VRAM + 20GiB RAM (--vram-reserve-mib 8192, cgroup 20G), Q3_K gives chat 62.4 / agent 73.3 and NVFP4 chat 51.3 (a simulated budget, no separate row). The shipped model.json sets --max-context 131072, --spec 4 (MTP) and --kv int8. The card does not state the engine version or the concurrency. Official figures, no per-run raw data.Official2026-11-12link
Ternary Bonsai 2 27Bllama.cpp (PrismML fork)RTX 5090 32GB129.9129.9 tok/s decode; 3893 tok/s prefill; PTQ1_0 120.5 tok/s decode, 1805 tok/s prefillClaimed2026-09-16link
Qwen3.8-Flash-NextUnspecifiedRTX 5090 + 192GB RAM120About 120 tok/s; cold prefill 3,000–6,000 (comment, rough numbers)Claimed2026-10-14link
Qwen3.8-27Bllama.cppRTX 5090 32GB119119.0 tok/s decode. Other tiers on the same card: GPU-2 (2.88 bpw) 111, GPU-3 (3.01 bpw) 108, GPU-4 (3.23 bpw) 104, GPU-5 (3.84 bpw) 94. Qualification: with DFlash2 (7 drafts, text only) GPU-1 176 and GPU-5 167, not the headlineClaimed2026-08-18link
Qwen3.8-Flash-Next-Gyrollama.cpp (agentionai fork)RTX 5090 32GB113113.0 tok/s decode; 3000.0 tok/s prefillClaimed2026-09-30link
Qwen3.8-Flash-NextStrataRTX 3090 24GB (PCIe 4.0 x8) single-card engine + Ryzen 9 9950X + 96GB DDR5-6000 (same box: an RTX 5090 and a second RTX 3090 on OCuLink PCIe 4.0 x4 run one engine each and share the RAM expert arena)110Same machine and build as comm-393. This row is the 3090 engine on PCIe 4.0 x8: 256K, 1 slot, 5 cores. Greedy short prompts: decode alone 110-120; the headline is the low end, 110. With all three engines busy: 98-111. The 3090 engine on OCuLink x4 reads 113-127 alone and 90-125 with all busy; widened to 9 cores (8 workers) it reads 134-136 alone, but with both 3090 engines widened both fall to 27-47. Notes only. No raw logs, so this is a claim.Claimed2026-11-13link
Edge0-145Bedge01× RTX 5090 D 24GB99.7Official Edge0 model card speed table. Headline is Q3_K chat decode 99.7; agent decode 121.2 is separate. The NVFP4 build on the same machine: chat 79.6 / agent 98.3 (https://huggingface.co/Edge0/Edge0-145B-NVFP4), note only. With the same card capped to 16GiB VRAM + 20GiB RAM (--vram-reserve-mib 8192, cgroup 20G), Q3_K gives chat 36.8 / agent 44.9 (a simulated budget, no separate row). The shipped model.json sets --max-context 131072, --spec 4 (MTP) and --kv int8. The card does not state the engine version or the concurrency. Official figures, no per-run raw data.Official2026-11-12link
Qwen3.8-Flash-NextStrata1× RTX 5090 32GB + Ryzen 9 9950X3D + 96GB DDR5-6400 + 9100 PRO Gen5 (Ubuntu)85.5Strata engine 0.1.59 (linux-cuda-sm120) + Control 0.1.59.1, STRATA_ADAPT_FETCH=2. 24h telemetry single-stream decode 85.5–96.0 (headline is the lower bound 85.5; 176K context, REUSED=0). With two agents the main stream is 58–67 and is not single-stream. Configured 262,144, 8-bit KV, MTP 5. Screenshots and a written range only, no raw log; claim.Claimed2026-11-12link
Qwen3.8-27Bllama.cpp1× RTX 5090 32GB75.2Out-of-window makeup (measured 2026-10-15, cited in-window by @llmspeed). Signed llm-speed run r_qfrb2zmyk0k, suite-v1, llama.cpp b11669. chat-short decode 75.19, recorded as 75.2 (prefill 563.4, TTFT 273 ms, inter-token latency p50 13.2 ms); chat-long 74.57 is separate. The configured context limit is not stated, left null. No baseline row for llama.cpp × Q4_K_M × 5090 × Qwen3.8-27B. The llm-speed page shows only a summary and a signature, with no raw records and no full server settings, so this is a claim.Claimed2026-10-15link
Nanbeige4.3-33Blxe Rust/CUDA runner1× RTX 509070about 70; no context or prompt conditionsClaimed2026-10-26link
Qwen3.8-Flash-Nextllama.cppRTX 4090 / RTX 5090 + system RAM684090: 25→up to 41; 5090: 31→up to 68 tok/sOfficiallink
DeepSeek-V4.1-Flash-1017BasaltRTX 5090 32GB (400W) + RTX 5060 Ti 16GB (180W) + 128GB DDR563.55090 + 5060 Ti each computing experts: decode 63.5 / prefill 610 (6.2K prompt); 5090 only: 53.6 / 610 warm, 595 cold; same machine llama.cpp PR #30817 --fit about 20; the author has not published the experimental branchClaimed2026-10-21link
DeepSeek-V4.1-Flash-1017Strata1× RTX 5090 (inference only) + RTX 3070 Ti (display) + 128GB DDR5 + i9-13900KF (Windows 11)53.8Strata 0.2.11: median 53.8 (mean 52.0, min 41.5 max 57.8). Screenshot attached. tok_s uses the median 53.8Claimed2026-11-06link
Nanbeige4.3-33Bdsx1× RTX 509040Decode 40+; prefill 4,000+ tok/s; no speculative decoding; no context or prompt conditionsClaimed2026-10-27link
DeepSeek-V4.1-FlashFront_Eagle739 streaming stackMac Studio M3 Ultra 512GB + RTX 5090 PC (32GB RAM, RAID0 NVMe)35Prefill on long prompts about 3,800; decode 35 (prose) to 60 (code); Mac only about 900 / 60Claimed2026-10-30link
DeepSeek-V4.1-Flash-1017StrataRTX 5090 32GB + R9 9950X + 96GB DDR525Strata 0.2.4, 512K context: decode 25–40 (the lower bound, 25, is used; the author is still tuning); prefill not measuredClaimed2026-10-24link
DeepSeek-V4.1-FlashTandem-vLLM1× RTX 5090 32GB + i5-14600KF + 128GB DDR4-320017Same post, engine and quant as comm-371, hardware is a 5090 + i5-14600KF + 128GB DDR4-3200. The author says it runs, slowly: one user 17. Four-stream total 20 is not single-stream. The drafter switch is not stated and is not inferred. Context is unstated, left null. Claim.Claimed2026-11-12link
DeepSeek-V4.1-FlashFront_Eagle739 streaming stack1× RTX 5090 + 32GB DDR5 + RAID0 NVMe (about 25GB/s)7Normal decode 7; about 20 when the router prefers experts in VRAM (this changes the output); prefill up to 1,770Claimed2026-10-28link
DeepSeek-V4.1-FlashUnspecifiedsingle RTX 5090 32GB + 126GB RAM + NVMe5.1cold 5.1 tok/s; 21.4 after cache warmRelayedlink
Qwen-Image 2.1UnspecifiedRTX 5090 32GB5.93 s; NVFP4 6.7 s vs INT8 7.6 s vs BF16 15.6 s (1024²)Relayedlink
Qwen-Image 2.1UnspecifiedRTX 509013.5 s → 1.9 s / image (vendor claim)Claimedlink
Qwen3.8-27BTensorFoldRTX 5090 vs 1× DGX SparkNo draft 75 vs 14.4; speculative 215 vs 44 tok/sRelayedlink
FastH3 V2 (MiniMax H3 8-step distillation)UnspecifiedRTX 5090 32GB832×480×5s: 14.8 s; 1344×768×5s: 38.6 s (with audio, end-to-end)Official2026-10-06link
FastH3 Trim (42 blocks pruned, experimental)UnspecifiedRTX 5090480p 5s: 13.4 s; 768p: 35.4 sOfficiallink
Wan2.1 14B (SparkDiffusion)UnspecifiedRTX 5090Wan2.1-T2V-14B 720P: 50-step CFG dense baseline 4,769 s → 3-step CFG-free 18 s (paper claims 265× end to end)Officiallink
MiniMax H3LightX2VRTX 5090768×1344 15s video + audio ≈72 s (steady state, excl. MP4 export; baseline ~26 min)Official2026-10-10link

Numbers come from each source. This project did not measure them again. Models and settings differ between rows. Source levels: Official = project or vendor; Measured = the poster ran it (an engine or recipe author must also publish raw data and reproduction steps); Relayed = second-hand; Claimed = prediction or no reproduction details.