read first

Choosing local LLM hardware: memory, speed and power

You start looking for a computer to try local AI, and the more you read, the more your budget grows. A small Mac seemed enough at first. Then you see someone running a much larger model and begin to hesitate. How much do you really need to spend to avoid regretting it? Before looking at a product list, choose one job you want that computer to finish.

The first thing to decide is not the equipment

For example, let's say your goal is to find and summarize personal documents and occasionally fix code. What is needed at this time is not a device that can run all public models, but a model that can be entrusted with the task and an environment in which to run it comfortably. If the order is reversed, you buy the equipment first and then look for a model to fill that capacity.

Write down a few questions you would actually ask on a regular basis. Questions that allow you to check the results are good, such as finding specific content in a document, checking for errors in written code, or creating a draft of a long article. If you haven't decided on a model yet, narrow down the candidates that answer this question well.

Illustration comparing integrated memory computers and single/dual GPU equipment in terms of memory, speed, power, and cost.
Rather than the size of the equipment, the required memory, perceived speed, power consumption, and purchase cost should be viewed as a bundle.

The memory missing from “the model fits”

First, check whether the model file fits into memory. However, file size isn't everything. The workspace of the executable program and the KV cache required for input and response are added. If you're the type of person who's going to put in a long document, you shouldn't assume that the capacity is sufficient just because a short greeting is successful.

Mac's integrated memory is also used by the operating system and other apps. The VRAM on an external GPU is separate from system RAM. Therefore, do not calculate that the dedicated space for both models is the same just because they see the same 32GB notation. You also need to think about the space you have left when you have both the browser and document processing turned on.

Cabinet diagram showing how model weights, KV cache, and execution headroom fit into device memory.
In order to run stably, the operating system and KV cache must have enough room to use rather than using all of the display capacity.

Next, see how the wait feels

Even if you have enough space in your memory, if you reply late, you may end up not writing it often. Divide the time between inserting a long document and waiting for the first response, and the time from when the response starts to when it ends. Each is affected by the prefill that processes the input and the decode that continues the answer.

Don't just read the numbers in your speed trial, actually wait for that time. Whether you want to get a short summary or copy a long piece of code feels different, even at the same speed. It is a good idea to narrow down the candidates based on the expected experience of the site and then check the execution history of the same model and length.

Illustration of different tasks leading to different equipment choices, such as document summarization, dialogue, and code generation.
If you first determine the tasks to be performed frequently and the input/output length, the required memory and speed standards will naturally be narrowed down.

Graphics card prices and computer prices are different.

If you look at the GPU price alone, it may seem much cheaper than a Mac. If you already have a decent PC, this may actually be an advantageous choice. However, if you need a new one, you will have to add the cost of the body, power supply, storage and cooling. You also need to check how much space the card will fit and what power it will require.

Conversely, the higher initial price of the finished product may be worth the reduced assembly and setup burden. This cannot be converted to the same amount for everyone. The choice may be different for someone who enjoys fiddling with settings on the weekend and someone who only wants to turn it on when processing documents.

Memory you cannot upgrade, for tasks you have not chosen yet

For equipment where it is difficult to change the memory after purchase, it may be more convenient to configure it one size larger. However, if you have not decided which model to use in the future, there is no need to go up to the largest option just out of vague anxiety. Calculate the capacity for each thing you want to do now and the next thing you want to do.

Whether you want to outgrow all of the equipment you use every day for a big job that you only need occasionally is another issue to consider. For rare tasks, it may be acceptable to wait longer or split the processing into smaller models. On the other hand, if you are running out of memory every day, there is a clear reason to upgrade.

Write this down before asking for a recommendation

‘I will run this model with this equipment, wait this long, and process a document of this length.’ If you can write this far, the purchase conditions are quite specific. Comparing candidates that meet these criteria can help explain why more expensive equipment may or may not be needed.

While choosing equipment, I don't want you to forget what you were trying to do. If your goal is to organize documents, finishing your first document with a small model is a good start. If you find something truly lacking in the process, your next purchase can be made based on clear experience rather than vague expectations.