What server do you need to run Ling 3.1 Flash?

Ling 3.1 Flash generates answers from text prompts⁵ for coding and agent tasks. It has 560B total parameters, activates 25B per token, and supports up to a 1M-token context. The official FP8 checkpoint file listing totals about 574 GB. Model card and files

Context length⁶ is the maximum input a model can process at once. Longer documents also require more memory for the KV cache⁷ and inference⁸ engine. Check the available GPU⁹ memory and document length, then set the context to fit the task.

The official SGLang quick start uses TP8 and EP8 across eight H200 GPUs, or TP4 and EP4 across four GB300 GPUs. The model files total about 574 GB; inference also needs memory for the KV cache and runtime¹⁰, depending on context length. Check storage and GPU capacity, then choose a context length for your requests. SGLang setup

Tensor parallelism¹¹ (TP) splits computation across GPUs, while expert parallelism (EP) distributes MoE experts. Use the table to match these settings to the official server recipes, then allow for KV-cache and runtime memory at your actual context length.

First, pull the SGLang container as described in the model card. The command below targets 8×H200 and uses --host 127.0.0.1 instead of the official example's 0.0.0.0, keeping access on the same machine. Add network controls and authentication before allowing other computers to connect. Mount the Hugging Face cache on host storage so the roughly 574 GB checkpoint does not need to be downloaded again after each container removal. Cache locations

docker pull lmsysorg/sglang:dev-ling-3.1-flash
mkdir -p "$HOME/.cache/huggingface"

docker run --gpus all --network host --ipc=host --rm \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  lmsysorg/sglang:dev-ling-3.1-flash \
  python3 -m sglang.launch_server \
    --model-path inclusionAI/Ling-3.1-flash \
    --tp 8 --ep 8 \
    --speculative-algorithm NEXTN \
    --host 127.0.0.1 --port 30000

When the server is ready, send a request to http://127.0.0.1:30000/v1/chat/completions. The OpenAI-compatible API¹² uses the model ID Ling-3.1-flash; refer to the model card examples for enable_thinking, temperature, top-k, and top-p.

First send a short text prompt and confirm that a response arrives, then test your document length and tool calls¹³. If there is no response, inspect the container logs for model loading errors or insufficient GPU memory. Official setup

curl -s http://127.0.0.1:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Ling-3.1-flash","messages":[{"role":"user","content":"Write a one-sentence summary."}],"chat_template_kwargs":{"enable_thinking":true},"temperature":1.0,"top_k":20,"top_p":0.95,"stream":true}'

A checkpoint dated October 19 (Ling-3.1-flash-1019) is up. The architecture and window are unchanged and the post-training differs; 560B total with 25B active, a 1M window and the 8×H200 recipe all stay as they were.

The gains look like this: the Artificial Analysis intelligence index moves from 41 to 44, AA-Briefcase from about 1400 to 1524 Elo, AutomationBench from 62 to 68 percent, CritPt from 18 to 26 and SciCode from 54 to 57.

Dated checkpoints get mixed up with their predecessor: check which revision --model-path resolves to and which revision sits in your cache.

To check whether you need long context or coding and agent features, choose a representative task and repeat it with the same document and output length. Record time to first token¹⁴ and total completion time to see whether 1M context or reasoning options help your work. Published benchmark throughput¹⁵ applies to its test hardware and conditions.

Before providing the API to teammates, decide who can access it and how long request and server logs will be retained. For internal documents, also check the permitted scope of use.