Gemma 4 26B-A4B: a serving recipe for 24–32GB hardware

Hardware-specific speed tuning and serving recipe summary

Provides weights, KV cache, prefill batch, and concurrent request with acceleration functionality, separated for balanced performance, speed, and long-context profiling.

On the detail page, you can check the command per profile, applied value, stopping conditions, and measurement order. The execution command is displayed only for combinations of model file and execution program that have been verified.

Distinguish active parameters from the weights you must download

This model is a MoE with approximately 3.8B active per token out of a total of approximately 25.2B. Since we don't just take and run the active part, we check the size of the entire conversion. Even with 24GB of VRAM or large integrated memory, you still have to leave room for cache and apps.

The official BF16 checkpoint and Q4 desktop paths are different. The official google/gemma-4-26B-A4B-it example is for NVIDIA servers with larger memory. Just change the official model ID on the equipment that went into Q4 and don't expect the same capacity.

Even if the app supports MoE, support for the new model structure and quantization is separate. We start by checking the recommended runtime for the file and loading the model normally.

Starting with 8K context and one request

Check the actual identifier of LM Studio to match the example below. If there is not enough space even at 8K, lower the length and check loading first. Turning on the largest context does not mean that it will use all of the model's performance.

LM Studio uses port 1234 and vLLM uses port 8000 in this example. Please match the address and model name to the same path. Check your local-only access settings and prepare authentication and network restrictions separately if you plan to provide them externally.

Check actual GPU occupancy and CPU remaining. If some weights are exceeded, generation may be slow due to transfers even with low active computation. Do not compare this condition on the same terms as normal GPU loading results.

# 1. 로컬 모델 식별자 확인
lms ls

# 2. 호환 변환본을 8K 문맥으로 로드
lms load <gemma-26b-a4b-identifier> --gpu=max --context-length=8192

# 3. 로컬 서버 시작
lms server start --port 1234

vllm serve google/gemma-4-26B-A4B-it   --host 127.0.0.1   --port 8000   --max-model-len 8192   --gpu-memory-utilization 0.90

Use the same workload when comparing against a dense model

After the short question is completed successfully, I insert the usual document. After the preparation run, record the median of three identical inputs, and keep the first token time and decode separately. It is important not to read results with different token counts and quantizations as differences in equipment performance.

It can be faster than the Dense model, but not always. It is affected by MoE kernel, memory access, and cache conditions. Don't use the average activation number to calculate the actual speed, but instead use the runtime's execution results.

Also check to see if any important conditions are missing from the quick answer. Checking the answers to a few questions to write repeatedly is more helpful in selecting a model than just one best tok/s.

curl -N http://127.0.0.1:1234/v1/chat/completions   -H "Content-Type: application/json"   -d '{
    "model": "<gemma-26b-a4b-identifier>",
    "messages": [
      {"role": "user", "content": "MoE 아키텍처의 장점을 짧게 정리해줘."}
    ],
    "max_tokens": 512,
    "stream": true
  }'

If it becomes slow while increasing the context

Even if the model body remains the same, the KV cache and temporary space may increase. Have one concurrent request and step up from 8K to the required length. If it suddenly slows down, first check for swap, memory pressure, and CPU offload.

Even if you are testing supported cache compression, do not change weight precision at the same time. You need to know which changes affected quality or speed. If a problem occurs, return to the length that was operating normally just before and compare.

The value of high-level memory doesn't stop once you turn on the model. You can determine if this is an option you need by checking whether there is space left even when you input for a long time and use other apps.

Going back to the reason I chose 26B

Have you ever had a question where you needed better results than 12B, and see if this model really makes that difference. If there is no difference, it is possible to keep the smaller model with room to spare. Just because it's bigger doesn't mean you have to accept uncomfortable waiting.

Once you find the results and speed you want, save the file, version, context, and cache settings. This is the final step to move from checking the model's feasibility to becoming a tool for daily use.