In brief
KV here stands for key and value: two types of internal vectors that a transformer uses during attention. Attention helps the model process connections with previous input. The KV cache stores previously calculated keys and values so that they can be reused for the next token. In this LLM context, therefore, it does not refer to a general key-value database.
Why does that make a difference?
A language model typically constructs a response step by step. The next token depends in part on the previous tokens. Without reuse, it would have to recalculate many earlier intermediate results each time. The cache eliminates the need for some of that repetitive work. It is an optimisation of the model’s execution, not additional training.
The cache also uses memory
More retained context and more concurrent requests may require more cache memory. Exactly how much depends, amongst other things, on the architecture and the chosen caching strategy. Some models only retain a limited window in certain layers. A model file that fits into memory is therefore not enough: there must also be sufficient space for the active requests.
A practical example
A local assistant handles short queries smoothly. With a long document or multiple users at the same time, the same machine is more likely to become overloaded. The weights have not increased, but more working memory is required to handle the requests. You may then, for example, need to run fewer conversations simultaneously or select the context more carefully. You will need to test which solution works best on that particular installation.
KV-cache is not a permanent personal memory
The cache contains internal interim results for a specific context. It is not a standard list of facts and does not automatically alter the model weights. Saving conversation history, reusing repeated prompt sections and managing personal knowledge are other functions. Whilst they may be technically related, they should not all be described as the same type of memory in a user promise.
What does this mean for your setup?
Allocate capacity for model weights and context. Test latency and memory usage based on the number of users and document lengths you anticipate. There are various caching techniques, such as reduced precision or relocation to different memory. These can save space, but may also cause delays or differences in quality. You should therefore choose based on your specific workload.
From concept to practice
How AITJE applies this
When determining local capacity, we take into account not only model weights but also the memory required for active context. We test longer inputs and multiple requests, so that a short demo is not the only benchmark. If memory is the limiting factor, we look at more targeted document selection, concurrency and appropriate output settings. The aim is to create a stable environment for the intended use.
Installation and setupFrom explanation to application
Would you like to know what this means for your work?
