memory and model

Context length, RoPE and sliding-window attention

Documents, conversation, and answers share the context space.

The context window¹ is the token² limit an LLM³ can reference in one request. Instructions, questions, previous conversation, and the new answer share this space with the document text. Longer documents require planning both room for the answer and memory for the KV cache⁴.

For example, submit a contract, ask for the renewal date, and then request a memo including the exceptions. The second request may add the earlier question and answer to the contract. If the first request works but the follow-up stops midway, checking the actual input and output limit helps locate the cause.

The app sends more than the document.

An app may prepend system instructions, role markers, and tool descriptions. Apps that retain conversation include earlier questions and answers. A document app may send the full contract or only retrieved passages. Check the token display or execution logs because the actual request can be longer than the visible input.

Token counts cannot be precisely converted from character or PDF-page counts because Korean, numbers, tables, and code tokenize differently. Use the actual input-token count for the same document when comparing. For follow-up requests, also check how much previous conversation was included.

System instructions, conversation, documents, and output space sharing a bounded context strip — Leave room for the answer when adding documents.

Leave room for the answer in a 16,000-token window.

Suppose the total limit is 16,000 tokens. Instructions and the contract use 12,000, and previous conversation uses 3,000, leaving 1,000. To allow up to 2,000 answer tokens, reduce the input or increase the total within the supported range. Follow the app's request limits and output-reservation behavior.

When the length exceeds the limit, an app may reject the request or reduce older conversation. For errors, check the logged token limit; if an answer omits earlier conditions, inspect the actual model input. For a response cut off midway, check the configured output limit, generated length, and termination or truncation indicators.

An arithmetic example of input and output sharing one space.
Aspect	Tokens
Total limit	16,000
Instructions and contract	12,000
Previous conversation	3,000
Room left for the answer	1,000

Older conversation beyond the context window removed while recent conversation and a summary remain — If conversation was reduced, inspect the input the app actually sent.

Separate the supported maximum from the length you use.

A model card's 128K or 256K is its supported maximum context length. The runtime⁵ configures a length within that range and may reserve cache memory in advance or grow it as needed. Increasing that setting can therefore change available memory even for the same short question.

Long inputs add prefill⁶ work and space for the KV cache used to reference earlier tokens. This explains why execution memory grows while the weight file stays unchanged. Increase only context length while keeping the model, KV format, precision, and concurrency fixed, and record peak memory and time to first token⁷.

Q4 weights and KV-cache precision are separate settings. Adjust one at a time rather than both together. Use a default length that reliably fits your usual documents and answers, and consider retrieval or split summaries for occasional long documents.

Longer contexts need more memory — Qwen3.8-27B · Q4_K_M · MTP off · site memory estimate — Calculated weights, KV cache and runtime overhead. Leave additional memory for the operating system and other apps.

KV-cache drawers and first-output waiting time growing as input becomes longer — Long context can require more resources for KV cache and prefill.

Test whether the answer combines conditions from both ends.

Ask about the renewal date near the beginning and the termination exception near the end together. Request clause numbers or evidence sentences and compare them with the source. If the app retrieves only relevant passages, first check that both conditions appear in the retrieved input.

RoPE extension adjusts positional representations to extend context range. Follow the model and runtime's specified conditions, and test evidence from the beginning, middle, and end with the same questions. Sliding-window attention⁸ is a model architecture focused on nearby context; an app dropping old conversation is request construction. Knowing which stage removes information tells you whether to change settings or input.

Terminology notes

Context window — The token span of input and generated content a model can handle in one request. The supported limit and memory use depend on the model and runtime settings.
Back to the text
Token — A unit into which a model divides input or output for processing.
Back to the text
Large language model — A model trained on large text datasets to learn language patterns and process or generate text in context.
Back to the text
KV cache — Memory that stores attention keys and values from earlier tokens for reuse during later token generation. Its size depends on context length and batch size.
Back to the text
Runtime — The software environment that provides facilities needed while a program runs. In local AI, it can also refer to a model execution engine.
Back to the text
Prefill — The stage where an LLM reads the input prompt and computes representations for its tokens. Longer prompts contain more tokens to process.
Back to the text
Time to First Token — The time from sending a request until its first output token arrives.
Back to the text
Attention — A neural-network operation that calculates relationships between input tokens and combines information from each position into the current representation.
Back to the text

Frequently asked questions

What should you check first in this article?: The context window is the token limit an LLM can reference in one request. Instructions, questions, previous conversation, and the new answer share this space with the document text. Longer documents require planning both room for the answer and memory for the KV cache.
How much memory does it need?: A model card's 128K or 256K is its supported maximum context length. The runtime configures a length within that range and may reserve cache memory in advance or grow it as needed. Increasing that setting can therefore change available memory even for the same short question.