⚡ Local Inference Recipes

English | 简体中文 | 日本語

Bonsai: local inference speed (tok/s), sourced

As of 2026-11-14, this dataset has 28 recipes for Bonsai. 5 were measured by the poster. Fastest measured single-device, single-stream text result: 85 tok/s with Bonsai 1.7B on BitRun (Core i5-12600K (CPU only); Measured, 2026-10-16; source). 28 entries have a date, from 2026-03-18 to 2026-10-21.

Filter these in the explorer →

ModelEngineHardwaretok/sMetricSource levelDateSource
Bonsai 1.7BBitRunCore i5-12600K (CPU only)85generation 85 tok/s (~22 W)Measured2026-10-16link
Bonsai 1.7BBitRunApple M4 Pro (CPU only)73generation ~73 tok/s (~11.7 W, ~0.31 J/token)Measured2026-10-16link
Bonsai2 27BUnspecifiedRTX 5060 Ti 16GB6767 tok/s with MTP head; 262K context + MTP + vision loaded together use 15.2/15.9 GBMeasured2026-09-27link
Bonsai-27Bllama.cpp (pinkmanwest fork)GTX 1660 SUPER 6GB (Windows)56.6llama-bench tg512 at depth 2048: patched 56.6 / stock 19.0; perplexity 18.10 vs 18.11, greedy output byte-identical to stock; Turing (sm_75) and Q1_0 onlyMeasured2026-10-21link
Ternary Bonsai 2 27B (heretic, refusals removed)UnspecifiedRTX 3060 12GB5050 tok/s with self-built MTP/draft head (39 without draft); weights + head + 128K context use 10.7/12 GB total, KV q4 (another post 2110546197794468240 says 51 tok/s)Measured2026-10-15link
Bonsai-8Bllama.cpp (PrismML fork)RTX 4090368368 tok/s decode; 11809 tok/s prefillClaimed2026-03-18link
Bonsai-8BMLXMacBook M4 Pro 48GB131131 tok/s decode; 472 tok/s prefillClaimed2026-03-18link
Ternary Bonsai 2 27Bllama.cpp (PrismML fork)RTX 5090 32GB129.9129.9 tok/s decode; 3893 tok/s prefill; PTQ1_0 120.5 tok/s decode, 1805 tok/s prefillClaimed2026-09-16link
Ternary-Bonsai-2-27B-262kmirai-s-adaRTX 4070 12GB102102 tok/s decode; 1173 tok/s prefill; at 32K 78 (code 175), prefill 1,031; at the 258K window end 8.6 (code 12.1), prefill 436Claimed2026-10-07link
Bonsai-8Bllama.cpp (PrismML fork)MacBook M4 Pro 48GB8585 tok/s decode; 498 tok/s prefillClaimed2026-03-18link
Ternary-Bonsai-8BMLXMacBook M4 Pro 48GB8383 tok/s decode; 460 tok/s prefillClaimed2026-04-13link
Ternary Bonsai 2 27Bllama.cpp (PrismML fork)RTX 4090 24GB81.281.2 tok/s decode; 3124 tok/s prefill; PTQ1_0 91.1 tok/s decode, 1645 tok/s prefillClaimed2026-09-16link
Bonsai-8Bllama.cpp (PrismML fork)RTX 3060 Laptop8181 tok/s decode; 1871 tok/s prefillClaimed2026-03-18link
Bonsai 27B 1-bitllama.cpp (PrismML fork)MacBook M5 Max66.466.4 tok/s decode; 874 tok/s prefillClaimed2026-07-04link
Ternary-Bonsai-2-27B-MTPLX-Optimized-SpeedMTPLXMacBook Pro M5 Max 128GB64.464.4 tok/s decode; after a 16K prompt 57.1Claimed2026-09-22link
Ternary-Bonsai-2-27B-262kmirai-s-adaRTX 2060 SUPER 8GB51.251.2 tok/s decodeClaimed2026-10-07link
Ternary Bonsai 2 27Bllama.cppMacBook M5 Max4747.0 tok/s decode; 765 tok/s prefillClaimed2026-09-16link
Bonsai 27B 1-bitllama.cpp (PrismML fork)MacBook M5 Pro44.244.2 tok/s decode; 421 tok/s prefillClaimed2026-07-04link
Bonsai-8BMLX Swift (PrismML fork)iPhone 17 Pro Max4444 tok/s decode; 377 tok/s prefillClaimed2026-03-18link
Ternary-Bonsai-27Bllama.cpp (PrismML fork)MacBook M5 Max4444.0 tok/s decode; 830 tok/s prefillClaimed2026-07-04link
Ternary Bonsai 2 27Bllama.cpp (PrismML fork)MacBook M5 Pro28.128.1 tok/s decode; 387 tok/s prefillClaimed2026-09-16link
Ternary-Bonsai-8BMLX SwiftiPhone 17 Pro Max2727 tok/s decode; 363 tok/s prefillClaimed2026-04-13link
Ternary-Bonsai-27Bllama.cpp (PrismML fork)MacBook M5 Pro26.226.2 tok/s decode; 393 tok/s prefillClaimed2026-07-04link
Bonsai 27B 1-bitllama.cpp (PrismML fork)MacBook M4 Pro2626.0 tok/s decode; 133 tok/s prefillClaimed2026-07-04link
Bonsai-8Bllama.cppSamsung S25 Ultra19.619.6 tok/s decode; 30.4 tok/s prefillClaimed2026-03-18link
Ternary Bonsai 2 27Bllama.cppMacBook M4 Pro1818.0 tok/s decode; 125 tok/s prefillClaimed2026-09-16link
Ternary-Bonsai-27Bllama.cpp (PrismML fork)MacBook M4 Pro1818.0 tok/s decode; 125 tok/s prefillClaimed2026-07-04link
Bonsai 27B 1-bitMLX SwiftiPhone 17 Pro Max (A19 Pro)1111.0 tok/s decode; 111 tok/s prefillClaimed2026-07-04link

Numbers come from each source. This project did not measure them again. Models and settings differ between rows. Source levels: Official = project or vendor; Measured = the poster ran it (an engine or recipe author must also publish raw data and reproduction steps); Relayed = second-hand; Claimed = prediction or no reproduction details.