DeepSeek-V4-Flash: distributed serving for a 304B model

Hardware-specific speed tuning and serving recipe summary

Provides weights, KV cache, prefill batch, and concurrent request with acceleration functionality, separated for balanced performance, speed, and long-context profiling.

On the detail page, you can check the command per profile, applied value, stopping conditions, and measurement order. The execution command is displayed only for combinations of model file and execution program that have been verified.

Distinguish between 13B active and 304B full

The active scale in this model describes the path taken by one token. That doesn't mean you can fit the entire weight onto one consumer GPU. The file and runtime conditions of the official deepseek-ai/DeepSeek-V4-Flash-0731 and the community compressed version are also different.

The official vLLM example uses a single 4×GB300 node and expert parallelism. This is not a path where you replace four random GPUs and then use the same command. You need to check not only the memory but also the kernel and connection conditions.

Experiments with compressed versions in large integrated memories should be reviewed separately from this official recipe. Loading estimates alone do not guarantee that the model structure will support execution.

Even on the official path, we start with a little context

The example below includes Data Parallel 4, Expert Parallel, and DSpark configurations. It is not a command to move option names one by one to another engine, but rather a bundle of supported server environments. The initial context is limited to 16K and the basic loading and answer are checked first.

The remote code execution option in the command can run code from the model repository, so you need to check which version to use and what it contains. Preserve the normal environment and leave behind a reproducible version. Just because you received the latest files may not be the same as your existing environment.

The server address remains 127.0.0.1. If you plan to open it to an external or organizational network, you will need to configure authentication, proxy, and firewall separately. The basic local execution example is not a replacement for configuring security on a shared server.

vllm serve deepseek-ai/DeepSeek-V4-Flash-0731   --host 127.0.0.1   --port 8000   --trust-remote-code   --data-parallel-size 4   --enable-expert-parallel   --kv-cache-dtype fp8   --block-size 256   --moe-backend deep_gemm_mega_moe   --attention-config '{"use_fp4_indexer_cache": true}'   --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"greedy"}'   --max-model-len 16384   --gpu-memory-utilization 0.90

Check the answer fragment and the actual amount generated

Send a short request to make sure the fragment arrives and the response exits gracefully at the end. After the preparation run, the first token time and decode of the same three inputs are recorded, along with the actual output length. Do not use only the briefly visible section on the screen as a measurement value.

Candidate acceptance by DSpark and actual timings are also confirmed. The length of verification and overall completion time are more important than the number of proposed candidates. Other questions may have different effects, so be sure to include a request to actually operate.

If you plan to handle multiple requests simultaneously, load test separately after a single request baseline. Total tok/s and each user's wait must be recorded separately so that you can select settings that suit your server operation purpose.

curl -N http://127.0.0.1:8000/v1/chat/completions   -H "Content-Type: application/json"   -d '{
    "model": "deepseek-ai/DeepSeek-V4-Flash-0731",
    "messages": [
      {"role": "user", "content": "DeepSeek-V4-Flash 분산 서빙 테스트."}
    ],
    "max_tokens": 512,
    "stream": true
  }'

Don't just look at memory totals

Check whether the weights, caches, and expert loads for each accelerator are placed as intended. Even if there is space left throughout, execution may fail if there is insufficient space on one side. Connection issues or waiting for specific accelerators may affect overall results.

Long contexts are increased in steps. The model's maximum supported length is not a setting that will be used directly on the actual server. It also measures peak memory and first token times and ensures that information in the middle of the document is accurately located.

If you encounter problems after changing the settings, go back to the default length and normal version to isolate the cause. Adding a new acceleration setting is not a solution to loading or distribution errors.

Know what you want to gain by running it yourself

At this scale, in addition to running the model, management and recovery, power and space are also considered. The reason for the equipment becomes clear when you decide for which tasks you will get better results than a smaller model and how many users you will need to handle.

This configuration is not the minimum for anyone looking to get started with local AI on a personal computer. With a smaller model, you can get your work done first. If you need a large server, bring the specific failure cases and request volume as the verification criteria for this recipe.