Bonsai: local inference speed (tok/s), sourced
As of 2026-11-14, this dataset has 28 recipes for Bonsai. 5 were measured by the poster. Fastest measured single-device, single-stream text result: 85 tok/s with Bonsai 1.7B on BitRun (Core i5-12600K (CPU only); Measured, 2026-10-16; source). 28 entries have a date, from 2026-03-18 to 2026-10-21.
Filter these in the explorer →
| Model | Engine | Hardware | tok/s | Metric | Source level | Date | Source |
|---|---|---|---|---|---|---|---|
| Bonsai 1.7B | BitRun | Core i5-12600K (CPU only) | 85 | generation 85 tok/s (~22 W) | Measured | 2026-10-16 | link |
| Bonsai 1.7B | BitRun | Apple M4 Pro (CPU only) | 73 | generation ~73 tok/s (~11.7 W, ~0.31 J/token) | Measured | 2026-10-16 | link |
| Bonsai2 27B | Unspecified | RTX 5060 Ti 16GB | 67 | 67 tok/s with MTP head; 262K context + MTP + vision loaded together use 15.2/15.9 GB | Measured | 2026-09-27 | link |
| Bonsai-27B | llama.cpp (pinkmanwest fork) | GTX 1660 SUPER 6GB (Windows) | 56.6 | llama-bench tg512 at depth 2048: patched 56.6 / stock 19.0; perplexity 18.10 vs 18.11, greedy output byte-identical to stock; Turing (sm_75) and Q1_0 only | Measured | 2026-10-21 | link |
| Ternary Bonsai 2 27B (heretic, refusals removed) | Unspecified | RTX 3060 12GB | 50 | 50 tok/s with self-built MTP/draft head (39 without draft); weights + head + 128K context use 10.7/12 GB total, KV q4 (another post 2110546197794468240 says 51 tok/s) | Measured | 2026-10-15 | link |
| Bonsai-8B | llama.cpp (PrismML fork) | RTX 4090 | 368 | 368 tok/s decode; 11809 tok/s prefill | Claimed | 2026-03-18 | link |
| Bonsai-8B | MLX | MacBook M4 Pro 48GB | 131 | 131 tok/s decode; 472 tok/s prefill | Claimed | 2026-03-18 | link |
| Ternary Bonsai 2 27B | llama.cpp (PrismML fork) | RTX 5090 32GB | 129.9 | 129.9 tok/s decode; 3893 tok/s prefill; PTQ1_0 120.5 tok/s decode, 1805 tok/s prefill | Claimed | 2026-09-16 | link |
| Ternary-Bonsai-2-27B-262k | mirai-s-ada | RTX 4070 12GB | 102 | 102 tok/s decode; 1173 tok/s prefill; at 32K 78 (code 175), prefill 1,031; at the 258K window end 8.6 (code 12.1), prefill 436 | Claimed | 2026-10-07 | link |
| Bonsai-8B | llama.cpp (PrismML fork) | MacBook M4 Pro 48GB | 85 | 85 tok/s decode; 498 tok/s prefill | Claimed | 2026-03-18 | link |
| Ternary-Bonsai-8B | MLX | MacBook M4 Pro 48GB | 83 | 83 tok/s decode; 460 tok/s prefill | Claimed | 2026-04-13 | link |
| Ternary Bonsai 2 27B | llama.cpp (PrismML fork) | RTX 4090 24GB | 81.2 | 81.2 tok/s decode; 3124 tok/s prefill; PTQ1_0 91.1 tok/s decode, 1645 tok/s prefill | Claimed | 2026-09-16 | link |
| Bonsai-8B | llama.cpp (PrismML fork) | RTX 3060 Laptop | 81 | 81 tok/s decode; 1871 tok/s prefill | Claimed | 2026-03-18 | link |
| Bonsai 27B 1-bit | llama.cpp (PrismML fork) | MacBook M5 Max | 66.4 | 66.4 tok/s decode; 874 tok/s prefill | Claimed | 2026-07-04 | link |
| Ternary-Bonsai-2-27B-MTPLX-Optimized-Speed | MTPLX | MacBook Pro M5 Max 128GB | 64.4 | 64.4 tok/s decode; after a 16K prompt 57.1 | Claimed | 2026-09-22 | link |
| Ternary-Bonsai-2-27B-262k | mirai-s-ada | RTX 2060 SUPER 8GB | 51.2 | 51.2 tok/s decode | Claimed | 2026-10-07 | link |
| Ternary Bonsai 2 27B | llama.cpp | MacBook M5 Max | 47 | 47.0 tok/s decode; 765 tok/s prefill | Claimed | 2026-09-16 | link |
| Bonsai 27B 1-bit | llama.cpp (PrismML fork) | MacBook M5 Pro | 44.2 | 44.2 tok/s decode; 421 tok/s prefill | Claimed | 2026-07-04 | link |
| Bonsai-8B | MLX Swift (PrismML fork) | iPhone 17 Pro Max | 44 | 44 tok/s decode; 377 tok/s prefill | Claimed | 2026-03-18 | link |
| Ternary-Bonsai-27B | llama.cpp (PrismML fork) | MacBook M5 Max | 44 | 44.0 tok/s decode; 830 tok/s prefill | Claimed | 2026-07-04 | link |
| Ternary Bonsai 2 27B | llama.cpp (PrismML fork) | MacBook M5 Pro | 28.1 | 28.1 tok/s decode; 387 tok/s prefill | Claimed | 2026-09-16 | link |
| Ternary-Bonsai-8B | MLX Swift | iPhone 17 Pro Max | 27 | 27 tok/s decode; 363 tok/s prefill | Claimed | 2026-04-13 | link |
| Ternary-Bonsai-27B | llama.cpp (PrismML fork) | MacBook M5 Pro | 26.2 | 26.2 tok/s decode; 393 tok/s prefill | Claimed | 2026-07-04 | link |
| Bonsai 27B 1-bit | llama.cpp (PrismML fork) | MacBook M4 Pro | 26 | 26.0 tok/s decode; 133 tok/s prefill | Claimed | 2026-07-04 | link |
| Bonsai-8B | llama.cpp | Samsung S25 Ultra | 19.6 | 19.6 tok/s decode; 30.4 tok/s prefill | Claimed | 2026-03-18 | link |
| Ternary Bonsai 2 27B | llama.cpp | MacBook M4 Pro | 18 | 18.0 tok/s decode; 125 tok/s prefill | Claimed | 2026-09-16 | link |
| Ternary-Bonsai-27B | llama.cpp (PrismML fork) | MacBook M4 Pro | 18 | 18.0 tok/s decode; 125 tok/s prefill | Claimed | 2026-07-04 | link |
| Bonsai 27B 1-bit | MLX Swift | iPhone 17 Pro Max (A19 Pro) | 11 | 11.0 tok/s decode; 111 tok/s prefill | Claimed | 2026-07-04 | link |
Numbers come from each source. This project did not measure them again. Models and settings differ between rows. Source levels: Official = project or vendor; Measured = the poster ran it (an engine or recipe author must also publish raw data and reproduction steps); Relayed = second-hand; Claimed = prediction or no reproduction details.