your first local llm with llama.cpp, on a 12gb card

why llama.cpp

llama.cpp is the open source engine most local AI tools are built on (Ollama, LM Studio and many others). using it directly takes a bit more effort, but you see and tune everything, and you get new features as soon as they come out. it reads models in GGUF format, which you can find by the thousands on Hugging Face.

you'll need: an nvidia card (here a 3080 Ti 12gb, but a 3060 12gb will do the same, a bit slower), linux with the nvidia driver, the CUDA toolkit, git, cmake and a C++ compiler. on ubuntu, sudo apt install git cmake build-essential covers the last three.

1. compile llama.cpp for your card

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

-DGGML_CUDA=ON turns on compute on the nvidia card; without it, everything would run on the processor. compiling takes a few minutes, and the programs land in build/bin/. by default, llama.cpp is compiled for the card plugged in at compile time, which is exactly what we want.

keep it up to date: new models often need a recent version. on my machine, a may 2026 llama.cpp didn't recognize the architecture of a model released in october. to update, git pull then run the two cmake commands again.

2. pick and download the model

i'm taking Spark-X2.6-4B, a small recent model that accepts 256K tokens of context, in its official GGUF Q8_0 version (4.07 GiB). Q8_0 means "compressed to 8 bits": almost no loss compared to the original, and it fits easily in 12gb.

downloading a big file from Hugging Face gave me some bad surprises: curl alone topped out at 0.8 MB/s and the official client got stuck at 57% before starting over from zero. aria2c with several connections did the job at ~7 MB/s:

sudo apt install aria2
aria2c -x 16 -s 16 "<the address of the .gguf file on hugging face>"

(on the model page, "Files" tab, the download link of each file)

3. run it with 256K of context

./build/bin/llama-server \
  -m Spark-X2.6-4B-Q8_0.gguf \
  -ngl 99 \
  -fa on \
  -c 262144 \
  -ctk q8_0 -ctv q8_0 \
  -b 4096 -ub 1024 \
  --jinja

what each option does:

option	what it does
-m	the model file
-ngl 99	puts all the model's layers on the card (99 = "everything")
-fa on	turns on flash attention: faster and lighter on memory with long contexts
-c 262144	the context size in tokens, here 256K
-ctk q8_0 -ctv q8_0	compresses the context cache to 8 bits, otherwise 256K wouldn't fit in 12gb
-b 4096 -ub 1024	the size of the token batches when it reads your prompt; bigger = faster reading
--jinja	uses the chat template shipped with the model (required for tool calls)

once the server is running, open http://127.0.0.1:8080 in your browser: llama.cpp has its own chat page, enough to check right away that the model answers.

4. what it gives

on my 3080 Ti, measured with llama-bench (the bench tool that comes with it):

context already filled	prompt reading	generation
empty	7,925 tok/s	136 tok/s
32K	4,701 tok/s	106 tok/s
128K	1,993 tok/s	57 tok/s
254K	1,066 tok/s	35 tok/s

empty136.2 tok/s

32K106.4 tok/s

128K57.4 tok/s

254K34.9 tok/s

what to remember most is that speed goes down as the context fills up: 136 tok/s empty, 35 tok/s with 254K tokens. that's normal, the model rereads its whole context for every word. to measure your own card:

./build/bin/llama-bench -m Spark-X2.6-4B-Q8_0.gguf -fa 1 -ctk q8_0 -ctv q8_0 -p 2048 -n 128

5. the setting to understand: compress the cache or not

compressing the cache to q8_0 lets you go very far in context, but it costs a lot of speed: at 128K, i go from 78 tok/s (uncompressed cache, f16) to 57 tok/s (q8_0), so -27%. so if you don't need more than ~170K tokens on a 12gb card, remove -ctk q8_0 -ctv q8_0 and lower -c: you'll go noticeably faster.

what's next

• a model bigger than your card: for a MoE model (like the Qwen "A3B" ones), the --n-cpu-moe N option keeps the experts of the first N layers in RAM, and the rest on the card. that's what lets you run models of 30B and more on 12gb.
• plug it into your tools: llama-server speaks the OpenAI API on http://127.0.0.1:8080/v1, so most clients and agents connect to it directly.
• the details of my measurements and of the hidden code test up to 245K tokens are in the experiment.