⚡ Local Inference Recipes

English | 简体中文 | 日本語

Mistral: local inference speed (tok/s), sourced

As of 2026-11-14, this dataset has 6 recipes for Mistral. 1 was measured by the poster. Fastest measured single-device, single-stream text result: 23.7 tok/s with Mistral Large 4 on llama.cpp (Mac Studio M5 Ultra 256GB; Measured, 2026-11-04; source). 6 entries have a date, from 2026-10-15 to 2026-11-12.

Filter these in the explorer →

ModelEngineHardwaretok/sMetricSource levelDateSource
Mistral Large 4llama.cppMac Studio M5 Ultra 256GB23.7llama.cpp b12020, median of 2 runs of 512 tokens: decode 23.7 (code 22.1 / prose 25.2). TTFT 9.1 s at 2K / 49.2 s at 8K / 242.3 s at 32K. 1T model, open weights since 10-29Measured2026-11-04link
Mistral Large 4 (1T-A52B)gr84× DGX Spark, QSFP ring (no switch required)30gr8 @069ec213 + 29 patches in patches/gr8 (commit stated in docs/PATCHES.md). Recipe README table: headline is single-request prose decode 30; code edit 35–40 and new code 31–34 are separate. 20.0 after a 24K prompt. Configured 65,536 (prompt and reply together; one pool for concurrent requests). Two/four-request totals 30 / 39 are not single-stream; with eight requests, four run and four wait, and the total stays 39. tools/bench.py, three fresh starts, recipe-author first-hand table, but with no per-run raw data, so this is a claim. In-window post https://x.com/GrubbyChicken/status/2120597958374912408Claimed2026-11-12link
Mistral Medium 3.5 128BMLX2× Mac Studio M5 Ultra 512GB (tensor)13.69decode 13.69 tok/s (Table 3, ~8K ctx, 128 gen)Official2026-10-15link
Mistral Medium 3.5 128BMLXMac Studio M5 Ultra 512GB7.7decode 7.7 tok/s (Table 3, ~8K ctx, 128 gen)Official2026-10-15link
Mistral Medium 3.5 128BMLXMac Studio M3 Ultra 512GB4.95decode 4.95 tok/s (Table 3, ~8K ctx, 128 gen)Official2026-10-15link
Mistral Large 4 (1T, A52B)MLX2× Mac Studio M3 Ultra (distributed)MLX-VLM distributed inference: 20 with a batch of 3 (no single-stream figure given)Claimed2026-11-03link

Numbers come from each source. This project did not measure them again. Models and settings differ between rows. Source levels: Official = project or vendor; Measured = the poster ran it (an engine or recipe author must also publish raw data and reproduction steps); Relayed = second-hand; Claimed = prediction or no reproduction details.