memory and model
Context length, RoPE and sliding-window attention
I raised the context to the maximum to insert a long document, but it actually slowed down or stopped running. They say the model supports large contexts, but why? The length supported, the length my equipment will contain, and the length to properly utilize the content are not the same number.
Documents are not the only thing using the context budget
System instructions, previous conversations, searched documents, and current questions are all included in your input. The answers you generate also require space. If you fill the context you set with original text, you may not have enough room to create an answer of the desired length.
Tokens are not exactly equal to the number of characters. Rather than converting the number of pages of a Korean document into a fixed token, it is better to check the input length in the actual app. Tooltips and templates that are not visible on the chat screen may also be included.
A longer context setting needs more memory
Processing long inputs takes prefill time and also requires memory to hold the computation state. Depending on your program, you can reserve space for the maximum context in advance or increase it as needed. So even if the input is short, the setting value itself can affect whether or not it can be loaded.
First, start with the length you would normally put your documents and answers in. After a normal run, stretch to a longer document and look at peak memory and first token time. If you choose the maximum from the beginning, it is difficult to find out at what point you fall short.
Accepting a longer input does not mean using all of it well
There is a difference between allowing the input itself and accurately finding a small condition in the middle of the document. Checking for known numbers or exceptions before, in the middle, and at the end is a more useful check than simply whether the answer is natural.
RoPE extensions are a way to adjust the range of location representations and are not an option that automatically improves the understandability of the model. Non-recommended extension settings can affect even short questions. Follow the model's guiding scope and executable support first.
Throw away old parts or add only what you need
Sliding windows are a way of looking at the immediate context. This is different when it is included in the model design and when the app arbitrarily prunes old cache. Reducing the cache may make early details difficult to refer to directly later.
You can also search for relevant parts in the document. It can reduce typing and waiting, but ensure you don't miss out on anything you need during the search phase. There are points of failure between inserting the entire document and inserting by searching, so be sure to check the results.
How often do you cover your longest documents?
Consider whether you want to expand your entire equipment based on the longest documents you will use once a year, or whether you want to speed up everyday tasks. If infrequent tasks can wait or be split, the memory required may vary.
On the other hand, if you ask questions by linking long pieces of material every day, having more context becomes a real convenience. Please base your calculations on the length of your document and the answers required, not the maximum number advertised. Getting a bigger window is more important than not losing important content within it.