In brief
Quantisation means that numbers in an AI model are represented with less precision. As a result, they take up less space. In the case of LLMs, this often involves model weights, but other calculations and stored intermediate results can also be quantised. The aim is to make the application runnable with less memory and appropriate computational effort.
What do ‘16-bit’, ‘8-bit’ and ‘4-bit’ mean?
Bits are the units in which a number is stored. Fewer bits offer fewer possible values. A method must therefore determine how the original numbers are approximated. A model with 4-bit weights is not simply four times smaller than the full 16-bit installation: additional scaling data, software and the memory required for context also need to be taken into account.
A comparison to help you understand
Consider rounding a measured value with many decimal places down to fewer decimal places. For some applications, this makes little difference; for others, the loss of precision is significant. This is a simplified comparison. Quantisation uses various techniques to minimise error and retain important information as effectively as possible.
Why is this useful for local AI?
A model that does not fit into the available memory in its original form may still be usable in a different version. This can increase the choice of models on a Mac, PC or server. At the same time, you must reserve space for input, ongoing conversations and the KV cache. Therefore, file size alone does not determine whether the application will run smoothly.
What might you be contributing?
Quality may vary, for example when dealing with precise instructions, calculations or less common languages. Speed improvements are also not guaranteed: these depend on hardware and support within the execution software. You should therefore test the actual implementation using the same questions and documents. In particular, check whether errors occur in the areas that are important for your work.
Making a practical choice
Record the model version, quantisation and software used. Compare quality, memory and latency based on the expected number of users. Then choose the configuration that meets your requirements. A small file size in itself is not the aim; a stable application with sufficient quality and capacity is.
From concept to practice
How AITJE applies this
For on-premises installation, we compare available model variants with your device’s memory and processing power. A quantised version may be a suitable option, but we also assess its quality in relation to your specific tasks. The chosen variant, model version and configuration are documented. This ensures that any subsequent changes can be measured and that a smaller file size is not mistaken for better performance.
Installation and setupFrom explanation to application
Would you like to know what this means for your work?
