NL-2610.008Ongoing · Planned[nl.PF]
What does quantisation cost a coding model?
The Nook team
Opened 5 October 2026 · 1 min read
Quantised files are how a capable model fits on a small card: the same model, stored with fewer bits per weight. Every bit removed can cost accuracy, and download pages rarely say how much. We will run one coding model at four quantisation levels on an RTX 4060 with llama.cpp and measure file size, speed, memory, and the share of a fixed set of coding tasks each file completes, with tests deciding pass or fail.
- Stage
- Planned
- Comments
- RTX 4060 8 GB; llama.cpp, CUDA build.
- Subjects
- Performance (nl.PF); Memory (nl.MM)
- Cite as
NL-2610.008· usenook.ai/labs/quantisation-cost-coding.html
Full text
The question
Model pages offer the same model as Q8_0, Q5_K_M, Q4_K_M, Q3_K_M and more, and the common advice is that 4-bit is “nearly as good”. For coding, where one wrong token can break a build, is that true? And on an 8 GB card, where the 8-bit file may not fit at all, what do you give up by going smaller?
The plan
- Pick one coding model small enough that its 8-bit file fits on 8 GB, so every level runs on the same card without spilling.
- Run four files of it: 8, 5, 4 and 3 bits.
- For each: file size, load time, tokens a second while reading and while writing, and peak memory, as the median of five runs.
- Give each file the same set of small coding tasks, each with tests; a task passes when its tests pass. Same prompts, same settings, greedy sampling, so luck plays no part.
- Publish the tasks and the scripts with the results.
So far
Planned; no runs yet. Which coding model would you like us to use? Tell us at hello@usenook.ai.