memory and model
Last updated
How Much Long-Context Text Fits on a 64GB CMP 170HX?
A measured 18,976-token run exists, but 64GB alone does not determine your usable context length.
After loading the model into 64GB of memory, KV cache1 uses part of the remaining space to retain document and conversation context. The input length a CMP 170HX can handle depends on the engine and model setup. In the published Qwen3.8-27B test, open-jet completed an 18,976-token input, taking about 13.55 seconds to the first token2.
For long documents, time to the first answer matters alongside generation speed. Compare completion status and waiting time in the input-length table below, then check with documents you normally use.
Memory Capacity and Context Length Are Different Numbers
Model weights reside in GPU3 memory, and processing longer inputs also grows the KV cache that stores attention4 state for tokens. The prompt5 and answer share the configured context, so asking for a long answer leaves less room for input. The physical capacity of 64GB is one part of this memory budget; it is not the same as the maximum token count supported by an engine.
Document page count also cannot be converted directly into token count. The amount the model receives changes with charts, text density, and whether the PDF app sends the full document or only retrieved passages. Token counts in a recipe or measurement table describe the actual input length; they are not a page-count estimate.
Separate the Engine That Processed 18,976 Tokens from the One That Failed
open-jet completed inputs of 3,158, 9,472, and 18,976 tokens. vLLM completed the 3,163-token chat request, but exited with an error on the 9,472-token chat request. Start by finding your intended input length and engine in the table.
Scroll sideways to compare all columns.
| Engine and mode | Input tokens | Recorded result | TTFT |
|---|---|---|---|
| open-jet MTP d=4 | 3,158 | Completed | 2.066 s |
| open-jet MTP d=4 | 9,472 | Completed | 6.358 s |
| open-jet MTP d=4 | 18,976 | Completed | 13.550 s |
| vLLM DFlash2 k=7 | 3,163 | Completed | 1.651 s |
| vLLM DFlash2 k=7 | 9,472 | Engine exited with Xid 31 | Not measured |
| vLLM DFlash2 k=7 | 65,536 | Configured value; no input test | Not measured |
Test in Steps Under Consistent Conditions Before Loading a Long Document
First choose the model file and serving engine, then fix the instructions and conversation history that your real task will send. Check a short input, then increase the content and confirm whether each request completes. Record input token count, TTFT6, generation speed, and completion or error status separately at each length to see what changes as the document grows.
The open-jet measurement increased input length from 3,158 to 9,472 to 18,976 tokens on the same GPU setup. MTP7 d=4 TTFT rose from 2.066 to 6.358 to 13.550 seconds, while decode8 means were about 77.20, 76.74, and 75.39 tok/s. These are records from one device and engine; recheck with your own documents and answer length rather than applying them directly to another card or model quantization9.
If the engine stops as input length increases, do not call the context setting the maximum supported length. Record the token count and error, then retest at a shorter input. Splitting the document or retrieving relevant passages are also options, but they change the context sent to the model and should be distinguished from processing the entire document in one request.
Send relevant passages first when a document is too large.
vLLM also completed a 6,603-token input through /v1/completions. This used a different API10 and input from the chat requests in the table. Prepare documents around the length completed with your intended request type, rather than the server's configured limit of 65,536.
If the whole document is too large for one request, summarize it by chapter or section, then combine the summaries. For targeted questions, RAG11 can retrieve and send only relevant passages. Choose whether to send the whole document or selected parts based on the task.
Terminology notes
KV cache — Memory that stores attention keys and values from earlier tokens for reuse during later token generation. Its size depends on context length and batch size.
Back to the textToken — A unit into which a model divides input or output for processing.
Back to the textGPU — A processor designed to handle many calculations in parallel. It performs model computations during AI inference.
Back to the textAttention — A neural-network operation that calculates relationships between input tokens and combines information from each position into the current representation.
Back to the textPrompt — The input text sent to a model. For generative models it describes the desired result and strongly shapes the output.
Back to the textTime to First Token — The time from sending a request until its first output token arrives.
Back to the textMTP — A training objective or model component for predicting multiple future tokens at once. In speculative decoding, those predictions can be proposed as candidate tokens.
Back to the textToken decoding — The stage that generates successive output tokens using the input and the response produced so far.
Back to the textQuantization — A way to represent model weights or values with fewer bits, reducing storage and memory use. The coarser representation can also affect accuracy.
Back to the textAPI — A defined interface that lets other code call a program’s functions.
Back to the textRetrieval-Augmented Generation — A method that retrieves material relevant to a query, adds it to the model input, and generates an answer from that context.
Back to the text
Frequently asked questions
- What should you check first in this article?
- After loading the model into 64GB of memory, KV cache uses part of the remaining space to retain document and conversation context. The input length a CMP 170HX can handle depends on the engine and model setup. In the published Qwen3.8-27B test, open-jet completed an 18,976-token input, taking about 13.55 seconds to the first token.
- How much memory does it need?
- Model weights reside in GPU memory, and processing longer inputs also grows the KV cache that stores attention state for tokens. The prompt and answer share the configured context, so asking for a long answer leaves less room for input. The physical capacity of 64GB is one part of this memory budget; it is not the same as the maximum token count supported by an engine.