memory and model
Prompt caching: which part of the wait does it remove?
I asked a second question using the same document and he answered it much faster than the first time. Has the equipment suddenly become faster? You may have rewritten some of the calculations you made earlier. Prompt caches reduce the wait for repetitive tasks, but you need to know what must be the same so you can reuse them to remain effective.
Reuse the prefix computation, not the answer
Consider a tool that sends long, common instructions and documentation every time. Even if the last question is different, if you keep reprocessing the previous part, the same thing will be repeated. Prefix Cache finds and reuses the KV status of the common prefix section.
This is different from a response cache, which returns the previous response as is. New questions and answers still need to be calculated. The reduction is mainly in the processing time for common input, and generation after the first token is not accelerated by the same percentage.
Even if the meaning is the same, it will be different if the tokens are different.
The program usually looks at matching tokens from the beginning. Whitespace, changes in chat templates and tool definitions may also change the front part. Even if the document appears to be the same to a human, the cache may not be reused.
If you put a date or request information that changes every time, the matching section may not continue to the long common instructions at the end. Review the structure of placing fixed content at the front and variable questions at the back, but do not change the meaning of the instructions or the necessary information.
You also need space to store it
If there is only one common document, management is simple, but as the number of types increases, the memory occupied by the cache also increases. If you run out of space, it exports unused status, so questions that were quick before may take a long time again the next time.
When multiple people use the same server, only the state with the correct user, model, and settings should be reused. This is not a function to mix calculation records of different conditions. Part of the operation is determining at what granularity the app identifies and clears its cache.
Measure the first run and repeated runs separately
Try once using the document you are reading for the first time, and then change the questions to the same document and run it again. It records the time to the first token, newly processed input, and reused input, respectively. It is easier to understand the results if you also compare the medians of three identical conditions after the warm-up run.
If you mix up the numbers of devices with left and empty caches, it can look like hardware differences. Raw input processing performance and the experience of actual repetitive tasks are both useful, but different measures. What situations you compared should be left next to the results.
For repeated documents, check this before upgrading hardware
If you have an agent that sends the same rules every time, or a tool that asks multiple questions on a single document, caching can be helpful. Conversely, if the questions are short and different each time, the effect may be small. It cannot be reflected at the same scale for all purposes.
If you feel that your current equipment is slow, check whether common documents are being processed anew each time. It may be sufficient to simply rewrite what has already been calculated. However, since the model does not disappear until the first processing of new, unread data, you must separately check whether it is worth waiting for new documents as well.