VRAM and local model-size planning

Why parameter count is not enough

Approximate weight memory depends on parameter count and bits per weight, but actual inference also needs room for the runtime, caches, context, and temporary tensors. A claim that a model fits exactly in a given memory size is not reliable without the exact model, quantization, and context.

• Check the exact model variant and file size.
• Account for KV-cache growth with context and concurrent requests.
• Leave GPU-memory headroom beyond the model weights.

Quantization and performance

Lower bit depth reduces weight memory but can change quality, speed, and tool support. Files described as Q4 can use different quantization methods and behave differently.

• Choose quantization together with quality and speed requirements.
• Confirm that the runtime supports the architecture and quantization.
• Test the real workflow rather than one short prompt benchmark.

GPU and system memory

When a whole model does not fit in GPU memory, some runtimes can place selected layers in system memory. That can make a model usable, but usually reduces speed and makes performance more sensitive to memory bandwidth.

• GPU memory defines the fastest fully GPU-resident path.
• System memory must cover the operating system, applications, data, and any offload.
• Apple Silicon unified memory is not directly equivalent to discrete GPU VRAM; macOS and applications also use it.

Practical pre-purchase check

• Record the exact model, quantization, desired context, and concurrent-user count.
• Define whether the priority is chat speed, long context, coding, document processing, or model adaptation.
• Check the selected runtime and retain memory headroom.
• Treat model classes shown on LLMLab.ee as planning guidance; the written quote confirms the exact configuration.