# your first local llm with llama.cpp, on a 12gb card

compile llama.cpp for your nvidia card and run a recent model with 256K of context on 12gb, while understanding what each option does

- date: 2026-10-22
- type: guide (local ai)
- hardware: one 3080 Ti 12gb
- source: measurements made by the author on his own hardware
- html page: https://pkillsleep.dev/en/guides/llama-cpp-premier-llm-local/

## why llama.cpp

[llama.cpp](https://github.com/ggml-org/llama.cpp) is the open source engine most local AI tools are built on (Ollama, LM Studio and many others). using it directly takes a bit more effort, but you see and tune everything, and you get new features as soon as they come out. it reads models in **GGUF** format, which you can find by the thousands on Hugging Face.

[meme: Too Damn High]

**you'll need**: an nvidia card (here a 3080 Ti 12gb, but a 3060 12gb will do the same, a bit slower), linux with the nvidia driver, the **CUDA toolkit**, `git`, `cmake` and a C++ compiler. on ubuntu, `sudo apt install git cmake build-essential` covers the last three.

## 1. compile llama.cpp for your card

```sh
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j
```

`-DGGML_CUDA=ON` turns on compute on the nvidia card; without it, everything would run on the processor. compiling takes a few minutes, and the programs land in `build/bin/`. by default, llama.cpp is compiled for the card plugged in at compile time, which is exactly what we want.

**keep it up to date**: new models often need a recent version. on my machine, a may 2026 llama.cpp didn't recognize the architecture of a model released in october. to update, `git pull` then run the two `cmake` commands again.

## 2. pick and download the model

i'm taking **Spark-X2.6-4B**, a small recent model that accepts 256K tokens of context, in its official GGUF Q8_0 version (4.07 GiB). Q8_0 means "compressed to 8 bits": almost no loss compared to the original, and it fits easily in 12gb.

downloading a big file from Hugging Face gave me some bad surprises: curl alone topped out at 0.8 MB/s and the official client got stuck at 57% before starting over from zero. **aria2c** with several connections did the job at ~7 MB/s:

```sh
sudo apt install aria2
aria2c -x 16 -s 16 "<the address of the .gguf file on hugging face>"
```

(on the model page, "Files" tab, the download link of each file)

## 3. run it with 256K of context

```sh
./build/bin/llama-server \
  -m Spark-X2.6-4B-Q8_0.gguf \
  -ngl 99 \
  -fa on \
  -c 262144 \
  -ctk q8_0 -ctv q8_0 \
  -b 4096 -ub 1024 \
  --jinja
```

what each option does:

| option | what it does |
|---|---|
| `-m` | the model file |
| `-ngl 99` | puts all the model's layers on the card (99 = "everything") |
| `-fa on` | turns on flash attention: faster and lighter on memory with long contexts |
| `-c 262144` | the context size in tokens, here 256K |
| `-ctk q8_0 -ctv q8_0` | compresses the context cache to 8 bits, otherwise 256K wouldn't fit in 12gb |
| `-b 4096 -ub 1024` | the size of the token batches when it reads your prompt; bigger = faster reading |
| `--jinja` | uses the chat template shipped with the model (required for tool calls) |

once the server is running, open `http://127.0.0.1:8080` in your browser: llama.cpp has its own chat page, enough to check right away that the model answers.

## 4. what it gives

on my 3080 Ti, measured with `llama-bench` (the bench tool that comes with it):

| context already filled | prompt reading | generation |
|---|---:|---:|
| empty | 7,925 tok/s | 136 tok/s |
| 32K | 4,701 tok/s | 106 tok/s |
| 128K | 1,993 tok/s | 57 tok/s |
| 254K | 1,066 tok/s | 35 tok/s |

**generation depending on the context already filled**

| | tok/s |
|---|---:|
| empty | 136.2 |
| 32K | 106.4 |
| 128K | 57.4 |
| 254K | 34.9 |

what to remember most is that **speed goes down as the context fills up**: 136 tok/s empty, 35 tok/s with 254K tokens. that's normal, the model rereads its whole context for every word. to measure your own card:

```sh
./build/bin/llama-bench -m Spark-X2.6-4B-Q8_0.gguf -fa 1 -ctk q8_0 -ctv q8_0 -p 2048 -n 128
```

## 5. the setting to understand: compress the cache or not

compressing the cache to q8_0 lets you go very far in context, but it costs a lot of speed: at 128K, i go from 78 tok/s (uncompressed cache, f16) to 57 tok/s (q8_0), so -27%. so if you don't need more than ~170K tokens on a 12gb card, **remove `-ctk q8_0 -ctv q8_0`** and lower `-c`: you'll go noticeably faster.

## what's next

- **a model bigger than your card**: for a MoE model (like the Qwen "A3B" ones), the `--n-cpu-moe N` option keeps the experts of the first N layers in RAM, and the rest on the card. that's what lets you run models of 30B and more on 12gb.
- **plug it into your tools**: `llama-server` speaks the OpenAI API on `http://127.0.0.1:8080/v1`, so most clients and agents connect to it directly.
- the details of my measurements and of the hidden code test up to 245K tokens are in [the experiment](/en/labo/spark-4b-256k-3080ti/).