On Windows, a model that does not fit in video memory does not fail. It crawls.

The Nook team

Submitted 5 October 2026 · 3 min read

Abstract

A translation in Nook sat at "Speaking · 0 of 1" for minutes, with no error. Whisper, the speech-to-text model, was still loaded when the voice model started, and together they did not fit in the card's 8 GB of video memory (VRAM). Instead of failing, the job spilled into the computer's ordinary memory and slowed to a crawl. Windows does this for programs that use Vulkan, which our voice engine did at the time, and NVIDIA's driver has done it for CUDA programs since version 536.40. Since Nook 0.5.5 the app unloads idle models before the voice speaks. This entry shows how to recognise a spill and, for CUDA, how to turn it back into an error.

Comments

RTX 4060 8 GB, Windows 11. Seen in normal use; not timed.

Subjects

Memory (nl.MM); Performance (nl.PF)

Cite as

NL-2610.002 · usenook.ai/labs/windows-vram-spill.html

Result

On Windows, running short of video memory often slows a GPU job down instead of stopping it. If a local model is suddenly far slower than usual, check whether it still fits before anything else.

What happened

In late September, while testing Nook's speech translator on an RTX 4060 with 8 GB, a translation stopped at “Speaking · 0 of 1” and stayed there for minutes. There was no error and no crash; the progress simply did not move. The speech had already been transcribed with Whisper and translated. The voice model was about to speak.

Why

Whisper was still loaded on the card when the voice model started, and the two together needed more than the card's 8 GB. On Windows, a program that asks for more video memory than is free is often not refused. It is handed shared memory instead: the computer's ordinary RAM, reached over the PCIe bus, many times slower than the card's own. The work carries on, very slowly.

Vulkan programs. Our voice engine was then audio.cpp's Vulkan build. For Vulkan programs, Windows' own memory manager does this, and we know of no switch to turn it off.
CUDA programs. For CUDA it is the NVIDIA driver's choice. Since driver 536.40 (2023), the Windows driver spills instead of returning an out-of-memory error. Since 546.01 there is a setting for it in the NVIDIA Control Panel.

How to spot it

Task Manager. Open Performance, then GPU. It shows Dedicated GPU memory and Shared GPU memory side by side. If dedicated memory is full and shared memory climbs while a model runs, the model does not fit.
nvidia-smi. It reports the card's own memory only. If it sits at the card's total while the job crawls, that is the same sign:

nvidia-smi --query-gpu=memory.used,memory.total --format=csv -l 1

What we changed

Since Nook 0.5.5, before the voice speaks, Nook unloads every engine that is idle and not pinned, so the card holds one large model at a time. The next job pays to load its model again, which costs far less than a spill.

What to do on your own machine

Run one large model at a time on cards with 8 GB or less, and unload what you are not using.
Close what else holds video memory. Browsers, games, video calls and screen recorders all do. On our own card, desktop apps were once holding about 4.7 GB before any model loaded (NL-2610.004).
For CUDA programs, choose an error over a crawl if you prefer one: NVIDIA Control Panel, Manage 3D Settings, CUDA – Sysmem Fallback Policy, then Prefer No Sysmem Fallback, for every program or just one. A job that does not fit then stops with an out-of-memory error, which is at least honest about the cause.

Limits

How sure we are. We saw this once in normal use and did not time it: “minutes instead of seconds” is all we can say about the size of the slowdown. How slow a spill is depends on the PCIe link and the system memory. The video models we are testing on 8 GB (NL-2610.007) will give us measured numbers.