memory and model
KV cache: the memory your conversation needs
The model file remains the same, but the longer the conversation, the more memory it takes. I didn't even get a new model, so why do I need more space? This is because there is a record of the calculations that are kept for subsequent writing of the answer. Understanding this KV cache can also explain memory pressure during long conversations.
Avoiding the same calculation at every step
To create the next token, you will need to refer to what you have read and written previously. Recalculating the parts that have already been processed every time is very wasteful. The KV cache reduces this repetition by storing key and value states required for attention.
This is different from a file where the chat contents are copied as text. This is computational state used by multiple layers of the model and may require more space than a few lines of user-visible text. This is why you cannot immediately know the cache capacity just by looking at the number of characters remaining in the chat window.
Costs you didn't know about when you asked briefly
At first, there may have been some leeway because it was just one short question. As you insert long documents and keep receiving replies, the computational state of inputs and outputs increases. For the same model, memory usage varies depending on context length, structure, and cache precision.
If you run multiple conversations simultaneously, you'll also need the status of each request. Increasing the model weight once does not mean that the rest of the users will use very little space. If you are setting up a team server, you need to match the maximum number of contexts per person with the number of concurrent requests.
I received the model as Q4, but why is the cache the same?
Weight quantization and KV cache quantization are different settings. Even if you received the Q4 model, the cache may be stored with a different precision. If supported, the cache can be reduced to 8 or 4 bits to save space for long contexts.
Instead, there may be added conversion and computational costs or quality differences depending on the task. This cannot be considered a quick setting for short conversations with sufficient memory. Check the answers and speed of the same document and change only what you need.
Will it end if I delete old records?
Another approach is to place a cap on the cache and discard old state. Space may be limited, but early details may no longer be directly referenced. It is necessary to distinguish whether the model is originally structured to only look at nearby context, or whether the program arbitrarily truncates the record.
This difference is important if you are asking for conditions at the beginning of the conversation again later. Don't count it as a success just because memory usage is reduced, but make sure the answer retains the numbers or requests you initially passed.
What running out of memory tells you to change
If the cache is insufficient, you can check the input length, concurrent requests, and cache precision before changing the entire model. Conversely, if you always need long documents and don't get the quality you want even after compression, the reasons for larger memory become clear.
When choosing equipment, add ‘how long a conversation can last’ after saying that files will be included. The goal is not to find equipment that responds to your first greeting, but rather a configuration that can accompany you until the actual work is completed.