LocalAI

See the speed before choosing hardware

Model setup recipes

Gemma 4 26B-A4B serving: Q4 on 24GB–32GB hardware and BF16 on a server

Gemma 4 26B-A4B uses only a small share of its parameters per token, but the full model still needs to be in memory. If it is slower than expected on 24GB hardware, check where the weights loaded before tuning computation settings. Start by checking Q4 loading, then adjust context length and generation speed.

Serving recipe

Settings for your hardware

For one user

Choose your hardware to see the launch command and settings you can adjust.

Available memory: 56GB / Bandwidth: 307GB/s

Profile

Apple MacBook Pro M5 Pro (64GB)

Memory fits

Recommended runtime: LM Studio (4-bit MLX (a different format from GGUF Q4_K_M)) · Documented runtime settings

For your first run or reliable everyday use

Profile context length

16,384 tokens

Based on runtime documentationMBP M5 Pro 64GB · Q4 · Request 1个

A starting setup based on the runtime documentation. Follow the steps below and compare speed as you make changes.

Estimated decode speed

26.9 ~ 65.4 tok/s

Estimate for one user

Estimated TTFT

8.8 ~ 21.5 s

Prefill: 765 ~ 1,861 tok/s

Estimated memory use

About 16.34GB / 56GB

Remaining margin: approx. 39.7GB

Server address

http://127.0.0.1:1234

Local only · 127.0.0.1

Estimates use the calculator’s recommended quantization and acceleration with 16,384tokens. They are not measurements of the configuration above.

Settings in this profile

ValueWhy this setting
Model weights4-bit MLX (a different format from GGUF Q4_K_M)Use an MLX model directory. This is a different file format from Q4_K_M GGUF.
Check input length16,384Check both the context-length setting in lms load and the context actually loaded.
KV cacheCheck model settingsThis launch command does not set the KV cache precision.
Prefill batch sizeCheck runtime settingsDo not use llama.cpp's batch-size and ubatch-size values as substitutes for MLX settings.
Concurrent requests1Measure the speed of one request, separately from the combined throughput of multiple requests.
Memory headroom2 GB or moreAfter loading the model, check actual memory use and whether the system is swapping.

MBP M5 Pro 64GB Run command (LM Studio)

LM Studio MLX engine · local 4-bit MLX model
# lms ls에서 확인한 MLX 4비트 모델 식별자를 입력
MODEL_ID=""
lms ls
: "${MODEL_ID:?MLX 모델 식별자를 입력하세요}"
lms load "$MODEL_ID" --gpu=max --context-length=16384
lms server start --port 1234
Record a baseline with this setup, then follow the tuning steps below, changing one setting at a time.

Checks after startup

  1. 01Check the startup log for completed model loading and the actual context length.
  2. 02After warming up, send one request at a time. Use three different inputs of the same length to record fresh-input processing time.
  3. 03Record repeated-input results separately as cache-reuse runs. Do not combine them with fresh-input results.
  4. 04Record prefill tok/s, decode tok/s and peak memory together, then change one setting at a time.

Trade-offs

  • Conservative context and concurrency settings may produce less than the hardware's maximum throughput.
See speed with recommended settings MBP M5 Pro 64GB View price, power, and sales configuration

Adjust one item at a time

Speed tuning, step by step

If several values are changed at once, it is difficult to find the cause.

  1. STEP 1

    Save a baseline

    Record first-input and cache-reuse runs separately, three times each. Compare prefill, decode and peak memory together.

    When to stop: Do not move on if there are errors or less than 2GB of memory headroom.

  2. STEP 2

    Tune prefill chunk size

    Increase in steps of 1,024 → 2,048 → 4,096, comparing TTFT and peak memory on long inputs.

    When to stop: Revert to the previous value if TTFT does not improve or peak memory rises sharply.

  3. STEP 3

    KV Cache Tuning

    Only when you need more context, compare Q8 cache against the F16/BF16 baseline using the same question.

    When to stop: Keep the default precision if the output changes or you already have enough context.

  4. STEP 4

    MTP/speculative decoding

    Enable it only for supported models. Test code and prose separately, measuring acceptance rates and actual decode tok/s.

    When to stop: Turn it off if the median of three runs does not improve on the baseline.

  5. STEP 5

    Context expansion

    Double context length at each step until you reach what you need. Check retrieval from the middle of the input and whether swapping occurs.

    When to stop: Reduce by one step if retrieval accuracy drops or swapping or memory compression starts.

Submit a measurement from my hardware

Import a JSON file with at least three runs under the same conditions. The file stays in this browser until you submit it.

Include only hardware, model, runtime settings and measurements. Do not include prompts, responses, raw logs or file paths.

If you do not have a measurement file, use the tool with your running local server. It requires Node.js 20 or later and does not upload results automatically.

Read the measurement instructionsDownload the measurement toolDownload the hardware and settings template

These are reviewed community submissions, not measurements made by this site.

Report a setup issue

Tell us where this setup failed. Only the site administrator can read your report.

Setup being reported · MBP M5 Pro 64GB · Gemma 4 26B-A4B (MoE) · Balanced

Distinguish active parameters from the weights you must download

This MoE activates about 3.8B parameters per token out of roughly 25.2B in total. You do not download and run only the active portion, so check the size of the full conversion. Even with 24GB VRAM or a large unified-memory pool, leave room for the cache and apps.

The official BF16 checkpoint and the Q4 desktop setup are different. The google/gemma-4-26B-A4B-it example is for an NVIDIA server with more memory. A machine that fits Q4 will not necessarily fit the official weights just by changing the model ID.

An app's general MoE support does not automatically cover a new architecture or quantization format. Check the runtime recommended for the file, then first make sure the model loads correctly.

Distinguish active parameters from the weights you must download
Distinguish active parameters from the weights you must download

Start with an 8K context and one request

Update the example below with the model identifier shown in LM Studio. If even 8K runs short of memory, reduce the context until loading succeeds. You do not need the maximum context setting to get useful performance from the model.

This example uses port 1234 for LM Studio and 8000 for vLLM. Match both the address and model name to the setup you ran. Verify local-only access, and configure authentication and network restrictions separately before making it available to other devices.

Check actual GPU memory use and which parts remain on the CPU. If some weights spill to the CPU, transfers can slow generation despite the low active parameter count. Do not treat this as equivalent to a run with the model fully loaded on the GPU.

Local serving with the LM Studio CLI (24GB–32GB desktops)
# 1. 로컬 모델 식별자 확인
lms ls

# 2. 호환 변환본을 8K 문맥으로 로드
lms load <gemma-26b-a4b-identifier> --gpu=max --context-length=8192

# 3. 로컬 서버 시작
lms server start --port 1234
Start the local daemon on 127.0.0.1 with a 24GB GPU or a Mac with 36GB+ memory.
Serve the official MoE checkpoint with vLLM (high-memory NVIDIA hardware)
vllm serve google/gemma-4-26B-A4B-it   --host 127.0.0.1   --port 8000   --max-model-len 8192   --gpu-memory-utilization 0.90
The official instruction-tuned checkpoint needs more memory than the Q4 desktop setup.

Use the same workload when comparing with a dense model

Once a short question completes correctly, try a document you normally work with. After warming up, record the median of three runs, keeping new-document and cached-input requests separate. Record time to first token separately from decode. Differences in token count or quantization must not be mistaken for differences in hardware performance.

It can be faster than a dense model, but not always. MoE kernels, memory access and cache conditions all matter. Use measured runtime results rather than trying to derive speed from the active parameter count alone.

Check that a fast answer has not skipped an important condition. Verifying answers to a few questions you regularly use is more useful for choosing a model than one peak tok/s figure.

Test an OpenAI-compatible chat completion request
curl -N http://127.0.0.1:1234/v1/chat/completions   -H "Content-Type: application/json"   -d '{
    "model": "<gemma-26b-a4b-identifier>",
    "messages": [
      {"role": "user", "content": "MoE 아키텍처의 장점을 짧게 정리해줘."}
    ],
    "max_tokens": 512,
    "stream": true
  }'
Measure response latency and generation speed at the local 127.0.0.1 endpoint.
Use the same workload when comparing with a dense model
Use the same workload when comparing with a dense model

If a longer context slows things down

The model weights may stay the same while the KV cache and working memory grow. Keep concurrency at one and increase gradually from 8K to the length you need. If speed drops suddenly, first check for swapping, memory pressure and CPU offload.

When testing a supported cache-compression option, leave weight precision unchanged. You need to know which change affected quality or speed. If problems appear, return to the last context length that worked and compare from there.

More memory is not just about getting the model to load once. Check whether there is still room for your usual long inputs and other apps. That is how to judge whether the larger configuration is worth it for you.

Revisit why you chose 26B

Think back to the questions where you needed better results than 12B, and check whether this model delivers that improvement. If it does not, staying with a smaller model and more memory headroom is an option. A larger model alone is no reason to accept an uncomfortable wait.

Once you have the results and speed you need, save the file, version, context length and cache settings. That turns a promising experiment into a setup you can use every day.

Revisit why you chose 26B
Revisit why you chose 26B

Running an NVFP4 checkpoint

The checkpoint currently documents TP=1. Check CUTLASS or Marlin support for the MoE backend.

vLLM · nvidia/Gemma-4-26B-A4B-NVFP4

Change log

These entries record changes to the site's guidance. They do not automatically check your installed engine or model version.

  1. Corrected model formats and runtime settings for Mac

    Separated the Mac MLX path from GGUF Q4_K_M and aligned the engine name, model format and command. KV-cache precision and prefill batching that the command does not specify are now marked for checking in the model or runtime settings.

    Also removed MLX speed profiles that implied a larger batch despite leaving the command unchanged.

  2. Separated uncached-input and cache-reuse records

    Updated the checks to record first-time inputs separately from repeated inputs. Cache effects are not folded into uncached prefill throughput or hardware differences.

  3. Added saving and reopening hardware profiles

    Runnable profiles can now be saved with their hardware selection in this browser, up to five entries, and reopened from the guide directory. A notice prompts you to recheck before running if the published settings have changed or the profile is no longer available.

  4. Added links to runtime explanations

    Engine names in the settings now link to the corresponding runtime guide. Paths without an identified engine are not linked to a guessed program.