Gemma 4 12B: LM Studio and vLLM on 16GB hardware

Hardware-specific speed tuning and serving recipe summary

Provides weights, KV cache, prefill batch, and concurrent request with acceleration functionality, separated for balanced performance, speed, and long-context profiling.

On the detail page, you can check the command per profile, applied value, stopping conditions, and measurement order. The execution command is displayed only for combinations of model file and execution program that have been verified.

Check whether you have a Q4 conversion or the original weights

On desktop, select the GGUF or MLX Q4 rendition that the app currently supports. Even on 16GB devices, the execution range varies depending on the operating system and cache availability. Check the actual size of the file you will receive and start with a short context.

The official google/gemma-4-12B-it BF16 weight is approximately 24GB in size. The vLLM example below is the path to the larger NVIDIA environment that will house it, which is different from the Q4 desktop command. Do not mix the memory conditions of two paths because of the same model name.

At first, write short Korean questions instead of images or audio. Once you have the text baseline, you need to add supported inputs so you can tell which features are overburdened.

Choose between LM Studio and the official server

LM Studio starts with an 8K context example by inserting the actual model identifier from the list. If it is not enough, reduce it further and check for normal loading. --gpu=max is the requested setting, and what part is actually loaded on the GPU can be checked through the app's display and logs.

The server address for LM Studio is port 1234 here, and 8000 for the vLLM example. Do not mix addresses from different commands. Check the local access range in LM Studio's network provision settings, and prepare authentication and firewall first when disclosing to other devices.

Just because the program is turned on doesn't mean the model is ready. After confirming that the loading is complete and the actual context is set, send the question. Overlapping multiple requests while waiting makes it difficult to understand the memory state of the first run.

# 1. 모델 목록 확인
lms ls

# 2. GPU 최대 오프로드 및 8K 컨텍스트로 모델 로드
lms load <gemma-4-12b-identifier> --gpu=max --context-length=8192

# 3. 로컬 서버 데몬 구동 (127.0.0.1:1234)
lms server start --port 1234

vllm serve google/gemma-4-12B-it   --host 127.0.0.1   --port 8000   --max-model-len 8192   --gpu-memory-utilization 0.90

Start by checking a short Korean answer

Replace the model name in the request below with your actual identifier. First check to see if the sentence stops mid-sentence and if the answer is in the requested language. If you request streaming, you can also see the fragments arriving, but the exact tok/s is a separate runtime measure.

After executing the preparation, record the first token, generation rate, and peak memory for the same three questions. Next, change to the document you would normally summarize and see the difference. Just because it's a brief self-introduction doesn't mean that a lengthy business document is sufficient.

Also look at the quality of the generated results. Even if you answer short questions well, you may miss dates or exceptions. If you leave a question in the document where you can check the correct answer, it is good to compare after changing the settings.

curl -N http://127.0.0.1:1234/v1/chat/completions   -H "Content-Type: application/json"   -d '{
    "model": "<gemma-4-12b-identifier>",
    "messages": [
      {"role": "user", "content": "한국어로 자기소개를 간단히 작성해줘."}
    ],
    "max_tokens": 512,
    "stream": true
  }'

When adding images and audio

First check whether the input is supported by the current model file and app. Add one type at a time and see how much your memory increases. The space left over when writing only text can be used for encoders and input processing.

If it fails on small memory, lower the context and number of concurrent requests first. When changing cache precision or offload, compare items one by one. Choosing Q4 again in a model that is already Q4 will not solve other causes of the shortage.

If the problem recurs only on the inputs you need every day, there may be reason to look at larger memory. Conversely, if your text work is sufficient, you may not need to go all out for multimodal maximum conditions.

Continuing to use a smaller model is also an option.

If the 12B handles your usual documents well enough and you don't mind waiting, there's no need to move up to a larger model. When a question arises that is lacking, it is easier to see the difference by sending the question to the next model and comparing it.

Reliably save written files, app version, and input length. This will be the baseline to return to when reinstalling or updating. Once you've achieved one successful task on a small machine, you can be more specific about the performance you need next time.