LocalAI

See the speed before choosing hardware

Model setup recipes

Gemma 4 12B serving: Q4 on desktop and BF16 on a server

To try Gemma 4 12B on modest hardware, start with one text response rather than enabling every feature at once. Keep a setup that completes that answer, then add long documents, images and audio in turn. This makes it easier to see where memory use and waiting time increase.

Serving recipe

Settings for your hardware

For one user

Choose your hardware to see the launch command and settings you can adjust.

Available memory: 56GB / Bandwidth: 307GB/s

Profile

Apple MacBook Pro M5 Pro (64GB)

Memory fits

Recommended runtime: LM Studio (4-bit MLX (a different format from GGUF Q4_K_M)) · Documented runtime settings

For your first run or reliable everyday use

Profile context length

16,384 tokens

Based on runtime documentationMBP M5 Pro 64GB · Q4 · Request 1个

A starting setup based on the runtime documentation. Follow the steps below and compare speed as you make changes.

Estimated decode speed

31.3 ~ 34.8 tok/s

Estimate for one user

Estimated TTFT

21.3 ~ 46.3 s

Prefill: 354 ~ 769 tok/s

Estimated memory use

About 9.61GB / 56GB

Remaining margin: approx. 46.4GB

Server address

http://127.0.0.1:1234

Local only · 127.0.0.1

Estimates use the calculator’s recommended quantization and acceleration with 16,384tokens. They are not measurements of the configuration above.

Settings in this profile

ValueWhy this setting
Model weights4-bit MLX (a different format from GGUF Q4_K_M)Use an MLX model directory. This is a different file format from Q4_K_M GGUF.
Check input length16,384Check both the context-length setting in lms load and the context actually loaded.
KV cacheCheck model settingsThis launch command does not set the KV cache precision.
Prefill batch sizeCheck runtime settingsDo not use llama.cpp's batch-size and ubatch-size values as substitutes for MLX settings.
Concurrent requests1Measure the speed of one request, separately from the combined throughput of multiple requests.
Memory headroom2 GB or moreAfter loading the model, check actual memory use and whether the system is swapping.

MBP M5 Pro 64GB Run command (LM Studio)

LM Studio MLX engine · local 4-bit MLX model
# lms ls에서 확인한 MLX 4비트 모델 식별자를 입력
MODEL_ID=""
lms ls
: "${MODEL_ID:?MLX 모델 식별자를 입력하세요}"
lms load "$MODEL_ID" --gpu=max --context-length=16384
lms server start --port 1234
Record a baseline with this setup, then follow the tuning steps below, changing one setting at a time.

Checks after startup

  1. 01Check the startup log for completed model loading and the actual context length.
  2. 02After warming up, send one request at a time. Use three different inputs of the same length to record fresh-input processing time.
  3. 03Record repeated-input results separately as cache-reuse runs. Do not combine them with fresh-input results.
  4. 04Record prefill tok/s, decode tok/s and peak memory together, then change one setting at a time.

Trade-offs

  • Conservative context and concurrency settings may produce less than the hardware's maximum throughput.
See speed with recommended settings MBP M5 Pro 64GB View price, power, and sales configuration

Adjust one item at a time

Speed tuning, step by step

If several values are changed at once, it is difficult to find the cause.

  1. STEP 1

    Save a baseline

    Record first-input and cache-reuse runs separately, three times each. Compare prefill, decode and peak memory together.

    When to stop: Do not move on if there are errors or less than 2GB of memory headroom.

  2. STEP 2

    Tune prefill chunk size

    Increase in steps of 1,024 → 2,048 → 4,096, comparing TTFT and peak memory on long inputs.

    When to stop: Revert to the previous value if TTFT does not improve or peak memory rises sharply.

  3. STEP 3

    KV Cache Tuning

    Only when you need more context, compare Q8 cache against the F16/BF16 baseline using the same question.

    When to stop: Keep the default precision if the output changes or you already have enough context.

  4. STEP 4

    MTP/speculative decoding

    Enable it only for supported models. Test code and prose separately, measuring acceptance rates and actual decode tok/s.

    When to stop: Turn it off if the median of three runs does not improve on the baseline.

  5. STEP 5

    Context expansion

    Double context length at each step until you reach what you need. Check retrieval from the middle of the input and whether swapping occurs.

    When to stop: Reduce by one step if retrieval accuracy drops or swapping or memory compression starts.

Submit a measurement from my hardware

Import a JSON file with at least three runs under the same conditions. The file stays in this browser until you submit it.

Include only hardware, model, runtime settings and measurements. Do not include prompts, responses, raw logs or file paths.

If you do not have a measurement file, use the tool with your running local server. It requires Node.js 20 or later and does not upload results automatically.

Read the measurement instructionsDownload the measurement toolDownload the hardware and settings template

These are reviewed community submissions, not measurements made by this site.

Report a setup issue

Tell us where this setup failed. Only the site administrator can read your report.

Setup being reported · MBP M5 Pro 64GB · Gemma 4 12B · Balanced

Check whether you have a Q4 conversion or the official weights

For desktop use, choose a GGUF or MLX Q4 conversion supported by your app. What fits on a 16GB device still depends on the memory left for the operating system and cache. Check the actual download size and start with a short context.

The official google/gemma-4-12B-it BF16 weights are about 24GB. The vLLM example below is for a higher-memory NVIDIA system that can hold them, not the Q4 desktop setup. The shared model name does not mean the two setups have the same memory requirements.

Start with a short Korean question instead of an image or audio. Establish a text baseline before adding other supported inputs, so you can tell which feature increases the load.

Check whether you have a Q4 conversion or the official weights
Check whether you have a Q4 conversion or the official weights

Choose LM Studio or the official-checkpoint server setup

In LM Studio, use the model identifier shown in the list and start with the 8K context example. If memory is tight, lower it further until the model loads. --gpu=max requests full GPU offload; check the app and logs to see which parts actually loaded onto the GPU.

These examples use port 1234 for LM Studio and 8000 for vLLM. Do not mix the addresses from different commands. Check LM Studio's network-serving settings too, and configure authentication and a firewall before allowing other devices to connect.

A running app does not necessarily mean the model is ready. Confirm that loading has finished and check the actual context setting before sending a question. Sending overlapping requests while it is still getting ready makes the first run's memory use harder to understand.

Local serving with the LM Studio CLI (16GB–24GB desktops)
# 1. 모델 목록 확인
lms ls

# 2. GPU 최대 오프로드 및 8K 컨텍스트로 모델 로드
lms load <gemma-4-12b-identifier> --gpu=max --context-length=8192

# 3. 로컬 서버 데몬 구동 (127.0.0.1:1234)
lms server start --port 1234
After the model loads, check the server address and network access settings.
Serve the official text model with vLLM (NVIDIA GPU)
vllm serve google/gemma-4-12B-it   --host 127.0.0.1   --port 8000   --max-model-len 8192   --gpu-memory-utilization 0.90
Serve the official instruction-tuned weights at 127.0.0.1:8000 on an NVIDIA system with at least 32GB of memory.

Start with a short Korean answer

Replace the model name in the request below with the identifier you loaded. Check that the answer finishes and uses the requested language. Streaming lets you watch chunks arrive, but use the runtime's measurements for accurate tok/s.

After warming up, run the same question three times and record time to first token, generation speed and peak memory. Then try a document you would normally summarize. Completing a short self-introduction does not establish that long work documents will be handled well.

Check output quality too. The model may answer short questions well but miss dates or exceptions. Keep a document-based question with a verifiable answer for comparison after changing settings.

Test the OpenAI-compatible API
curl -N http://127.0.0.1:1234/v1/chat/completions   -H "Content-Type: application/json"   -d '{
    "model": "<gemma-4-12b-identifier>",
    "messages": [
      {"role": "user", "content": "한국어로 자기소개를 간단히 작성해줘."}
    ],
    "max_tokens": 512,
    "stream": true
  }'
Test a local API request at 127.0.0.1:1234.
Start with a short Korean answer
Start with a short Korean answer

Adding images and audio

First confirm that the model file and app support the input type. Add one type at a time and check the increase in memory use. Space left over during text-only use may be needed by the encoder and input processing.

If the model fails on limited memory, reduce context length and concurrent requests first. Change cache precision or offload settings one at a time. Selecting Q4 again for an already-Q4 model will not fix a memory shortage caused elsewhere.

If the inputs you need every day repeatedly cause problems, more memory is worth considering. But if text-only work already runs well, you may not need a larger machine just to cover the most demanding multimodal settings.

Staying with a smaller model is a valid choice

If 12B handles your usual documents well and the wait is acceptable, there is no need to move to a larger model. When a question exposes a weakness, send that same question to the next model to see whether it actually helps.

Save the model file, app version and input length that worked reliably. They give you a baseline to restore after a reinstall or update. Once a modest machine completes a useful task, it becomes easier to identify the performance you actually need next.

Staying with a smaller model is a valid choice
Staying with a smaller model is a valid choice

Running an NVFP4 checkpoint

Requires the SGLang branch with the Gemma 4 loader. FFN uses NVFP4; attention stays in BF16.

SGLang · AxionML/Gemma-4-12B-NVFP4

Change log

These entries record changes to the site's guidance. They do not automatically check your installed engine or model version.

  1. Corrected model formats and runtime settings for Mac

    Separated the Mac MLX path from GGUF Q4_K_M and aligned the engine name, model format and command. KV-cache precision and prefill batching that the command does not specify are now marked for checking in the model or runtime settings.

    Also removed MLX speed profiles that implied a larger batch despite leaving the command unchanged.

  2. Separated uncached-input and cache-reuse records

    Updated the checks to record first-time inputs separately from repeated inputs. Cache effects are not folded into uncached prefill throughput or hardware differences.

  3. Added saving and reopening hardware profiles

    Runnable profiles can now be saved with their hardware selection in this browser, up to five entries, and reopened from the guide directory. A notice prompts you to recheck before running if the published settings have changed or the profile is no longer available.

  4. Added links to runtime explanations

    Engine names in the settings now link to the corresponding runtime guide. Paths without an identified engine are not linked to a guessed program.