Long prompts can reverse a local LLM speed ranking

What do we want to finish quickly?

For example, let's say you are looking for equipment to organize long meeting minutes after work. This is something I put off because I didn't have time during the day. All you have to do is narrow down what has been decided in the document and what remains to be done. No need for a long answer. A few paragraphs to read again tomorrow morning will suffice.

What is needed at this time is a computer that can pour letters onto the screen the fastest? Or is it a computer that puts in a document and then produces the necessary organization within a moment? Although they may seem similar, it's a different question when comparing equipment.

There are no minutes in the performance sheet for us to process or any circumstances that need to be sorted out by tomorrow morning. There are only numbers measured under given conditions. I still have to read those numbers into my own context. Let’s check this difference before choosing the more expensive one.

Let me send you a short question first.

A and B in this article are virtual devices to illustrate the principle. This is not an actual measured value for a specific product. Let's assume that we use the same model to create an answer worth about 300 tokens to the same question.

A waits 4 seconds for the first token and then spends 60 tokens per second. It takes about 5 more seconds to write your answer, so the total is about 9 seconds. B is 6 seconds until the first token, then 30 tokens per second thereafter, for a total of about 16 seconds.

In this condition, A finishes first. However, even if the token generation speed is doubled, the overall time is not exactly halved. This is because there is a separate wait before responding.

Here, 60 tok/s and 30 tok/s are the generation speed after the answer starts, i.e. the decode speed. The time taken to reach the first token was added separately. Calculations are approximate and rounded for ease of understanding. It is not a single large number that you usually see in a table, but rather a count of the time the user waited from beginning to end.

What if you put a long document instead of a question?

This time, let's insert the same long document into both devices and set the answer length to about 300 tokens as before. Assume that processing the document results in A's first token occurring 40 seconds later and B's first token occurring 10 seconds later. The speed after the response starts remains the same.

Person A waits approximately 40 seconds and then answers for 5 seconds, for a total of 45 seconds. B waits 10 seconds, then answers 10 seconds, for a total of 20 seconds. A is still faster at using tokens, but B finishes the task first.

Equipment performance has not suddenly changed. What we were asked to do has changed. Input processing time, which is not evident in short questions, plays a large role in long documents.

The step that handles long input is called prefill. In addition to these steps, the time until the first token may include waiting for requests or preparing for execution. Here we compare those times together. This does not mean that B is always advantageous in actual equipment, but it means that the results of other tasks cannot be known just by the same decode rank.

So would equipment with a faster first token be better?

Again, you need to look at the length of the answer. If you're looking for just one date in a document, the answer is short, so waiting until the first token is important. Conversely, if you are asked to write a long draft, the time it takes to write a continuous response increases.

In other words, if you only choose between the first token and the decode, the same problem will repeat itself. You need to match the length of the input you frequently enter with the length of the response you want to receive. If we change the previous hypothetical example to a different length, the difference will be different.

You should also check your cache. Mixing results from reusing documents already read with results from documents read for the first time makes it difficult to distinguish between equipment differences and cache effects.

The amount of time you can wait varies depending on the task.

If you drop off dozens of documents at night and receive the results in the morning, one request starting a few seconds late may not be a big deal. What is more important is whether the entire task is completed on time and does not fail along the way.

On the other hand, asking for a small change in code, seeing the results, and asking again is different. While waiting for a response, the next judgment also stops. How many times a day you repeat a wait can have a greater impact on your equipment selection than a single wait.

However, it is difficult to set a passing line that works for everyone at a few seconds. Although it's fast enough to read and follow along, it can be slow for people who want to copy and write long results. You must first know where you feel comfortable with it so you can determine how much more you will have to pay for the difference in performance.

The tool you click on out of curiosity may be different from the tool you open again the next day. What you want to check before purchasing is not just how impressed you are on the first day, but also whether the wait will be bearable for repeated use.

Compare your own workload in both modes

First, in a short question, we only compare token generation speeds. Then, change it to a length closer to the document you would normally put in, and look at it again, including the prefill. See if the same device is ahead in both results.

The site's comparison screen allows you to change the input length and whether or not a prefill is included. What you see is an estimate and should be differentiated from the actual execution history, but you can see if the selection changes when you include the wait before answering.

The question you need to ask when deciding which equipment to buy doesn't end with 'which has a higher tok/s'. It should lead to ‘Which one can finish the tasks I often ask faster?’

Just because it's twice as fast doesn't mean you have to spend twice as much

Now let's go back to our original speed table. There is no mistake that A is 60 tokens per second and B is 30 tokens per second. However, for someone trying to organize meeting minutes, that fact alone was not enough to conclude a purchase.

I compared the wait time to insert documents and the cheaper one may be enough for my work. Conversely, if you have to wait until the end for a long answer each time, it may become clear why you'd spend money on the faster option. The important thing is that whatever the conclusion is, I confirmed that it was my job to do.

The content of the answer cannot be left out. If your quick summary misses important decisions, you should read the original again. This is why we use the same model when comparing speeds, and why we need to check whether the model handles our documents properly before deciding on a device.

Our initial goal was not to have the highest number, but to organize the minutes of meetings that had been delayed. If you can get the job done well enough, you don't have to buy more expensive equipment. On the other hand, if you find a block in the equipment you currently have, you can then know what to spend money on to change.

Choose two machines and compare them while processing the same document. If the results are similar than expected, that is also a helpful answer. There is no reason to spend the remaining budget on the next column of the performance table.