a 4B model holds 256K of context on a single 12gb 3080 ti, and finds 40 codes out of 40

what i wanted to know

recent small models announce huge contexts, and i wanted to know whether a 4B could really use 256K tokens on a single 12gb card, the one a lot of people already have, and how fast it ran once the context was full.

the setup

Spark-X2.6-4B in its official Q8_0 version (4.07 GiB), with llama.cpp compiled the same day, flash attention on and the K/V cache in q8_0 to fit the 262K of context on the 3080 Ti. i also ran the same bench on the 3090 to compare.

the numbers

for the hidden codes, i took the text of War and Peace and slipped four codes into it at 10, 35, 60 and 85% depth, on prompts from 15.8K to ~245K tokens, two tries per length: it found all 40 codes. at ~245K, the full answer took 131 seconds.

what surprised me

the 3080 Ti beats the 3090 by 10 to 30% on this model, but only because i had capped the 3090 at 230 W versus 260 W for the 3080 Ti, which shows how much the power limit weighs.

the K/V cache in q8_0 also costs a lot at long context: at 128K, decode drops to 57 tok/s versus 78 in f16, so -27%. it's really only useful to go past ~170K tokens on a 12gb card.

and even though the model only has 9 full attention layers out of 36, decode still loses 74% between an empty context and 254K.

what didn't work

downloading the model took longer than testing it: curl topped out at 0.8 MB/s, and the Hugging Face client got stuck at 57% before starting over from zero. aria2c with 16 connections ran at ~7 MB/s. and you need a recent llama.cpp, a may 2026 version doesn't recognize the architecture.

what i keep

a 4B with 256K of context on a 12gb card really works for finding info in very long documents. for reasoning over long context, it'll take a real test. to install it at home, follow the llama.cpp guide.