⚡ Local Inference Recipes

English | 简体中文 | 日本語

Gemma: local inference speed (tok/s), sourced

As of 2026-11-14, this dataset has 7 recipes for Gemma. 2 were measured by the poster. Fastest measured single-device, single-stream text result: 118.4 tok/s with Gemma 4 31B on NInferno (RTX 5090 Laptop GPU (5090M 24GB); Measured, 2026-10-29; source). 7 entries have a date, from 2026-10-13 to 2026-11-06.

Filter these in the explorer →

ModelEngineHardwaretok/sMetricSource levelDateSource
Gemma 4 31BNInfernoRTX 5090 Laptop GPU (5090M 24GB)118.4Short-chat decode (greedy/sampled): Fast 118.4 / 114.0, Max-context 106.8 / 99.7, start point 33.2; prefill at 8K 4,154, at about 130K about 1,600Measured2026-10-29link
Gemma 4 E4B-itSonicBoomlaptop RTX 3050 8GB51.65decode 51.65 tok/s (tg64; same machine/GGUF llama.cpp 45.20)Measured2026-10-13link
Gemma 4 31BRadiance4× Radeon AI PRO R9700 32GB445.81 user 445.8 (main chain 352.2); aggregate (separate) at 2/4/8/16 users about 376–965; the PR is not merged, so the numbers are claims onlyClaimed2026-10-25link
Gemma 4 31BRadiance4× Radeon AI PRO R9700 32GB377.71 user 377.7 (main chain 352.2, +7.2%); aggregate (separate) 16 users 959.6; the PR is not merged, so the numbers are claims onlyClaimed2026-10-25link
Gemma 4 26B-A4BoMLXMacBook / Mac Studio M5 Max 128GB83Same post, oMLX 0.7.0: about 83–86 tok/s. Method same as x-283. tok_s uses the floor 83Claimed2026-11-06link
Gemma 4 12BTensorFold1× DGX Spark (GB10)37single-stream decode 37; with its MTP assistant greedy chat goes 30→67 (2.2×, separate)Official2026-10-24link
Gemma 4 31BTensorFold1× DGX Spark (GB10)29.9greedy chat: 12.6 without drafter → 29.9 with z-lab DFlash (2.4×); tok_s is the DFlash valueOfficial2026-10-24link

Numbers come from each source. This project did not measure them again. Models and settings differ between rows. Source levels: Official = project or vendor; Measured = the poster ran it (an engine or recipe author must also publish raw data and reproduction steps); Relayed = second-hand; Claimed = prediction or no reproduction details.