memory and model
How much VRAM or unified memory does a local LLM need?
The model file is 20GB, but the graphics card memory is 24GB. I have 4GB left, so I think that will be enough. However, if you have a long conversation, you may run out of memory. Let's start by looking at what more is needed other than space to store files.
Fitting the model file is only the start
The number 20GB is an illustrative example. The actual model will vary in size depending on the quantization method and file configuration. First of all, we need space to load this file into memory.
Added to this is the workspace of the executable program and the computational history of the conversation. This record is called the KV cache. This is why you shouldn't calculate that the entire remaining 4GB can be used for conversations.
But why did it work so well at first?
If the first question you sent was short, the cache may also be small. As you add longer documents or continue conversations, the number of calculation records to keep increases. The model file remains the same, but the memory may change during execution.
So, we need to distinguish between ‘I succeeded in loading the model’ and ‘I finished my work.’ Run it with the documents and answer lengths you usually put in, and check the most memory used at that time.
Will adding more RAM to my PC solve the problem?
If your PC has a dedicated graphics card, the system RAM and GPU's VRAM are separate spaces. Increasing RAM does not increase the VRAM capacity of the graphics card.
Some executable programs may split their models into RAM. However, it may take additional time to go back and forth between the two memories or for the CPU to calculate, so being executable and being able to use it quickly are two separate issues.
Mac's integrated memory is a structure in which the CPU and GPU use the same memory. Instead, the operating system and other apps also use that space. Therefore, you cannot compare 32GB of Mac and 32GB of graphics card as if the space dedicated to the models is the same.
Two GPUs do not simply become one larger memory pool
If you plug in two 24GB cards, the total physical memory is 48GB. However, the executable program must be able to split the model onto two cards to utilize that space.
You need to leave the necessary workspace for each card, and it takes time to exchange the divided calculation results. The number ‘48GB secured’ alone cannot be considered the same as a single 48GB GPU.
When choosing capacity, look in this order:
Determine the model file to receive and calculate the memory based on the actual input length to be used. This then leaves room for workspace and other apps to use while running. It is better to check the performance history of the combination than to have one margin ratio that works for all models.
If you already have enough space for your needs, simply adding more memory will not make the answer faster. From then on, compare the creation speed of the same model with the first token time. The choice becomes clear if you're purchasing to address memory constraints or reduce latency.
It may be tempting to go for a larger configuration from the start for fear of regretting lowering the capacity later. There is reason to be concerned, especially if the memory is difficult to change after purchase. However, if you choose the largest capacity first without even deciding on the model you want to use, it is difficult to know what you are paying more for.
Write down the model you want to use now and the next task you want to do. You can start by making sure both are included. You can't buy a machine today for every model that will come out one day, but you can at least reduce the chances of picking a machine that won't do what you set out to do.