qwen 3.8 flash next on strata: a 125B at more than 110 tok/s on a 3090 + a 3080 ti

what strata is for

Strata is an open source engine built for a single family of models, Qwen 3.8 Flash Next, a MoE with 125 billion parameters. this kind of model normally runs on servers with hundreds of gigabytes of graphics memory, and Strata pulls it off on a regular PC thanks to a trick: in a MoE, each token only uses a small part of the "experts". the engine keeps in VRAM the ones that matter most for your conversation, leaves the others in RAM, and swaps them as it goes.

you'll need, according to the official docs: at least 32gb of RAM and VRAM combined (more is better), an nvidia RTX 20 card or newer with 12gb or more, and about 80gb of free disk, ideally on an SSD. with two cards, Strata can make them work together.

1. pick the model size

the same model comes in several compression levels. the more compressed it is, the faster and lighter it gets, and the slightly less smart. the table from the official docs is read by adding up your RAM and your VRAM:

your memory (RAM + VRAM)	recommended size
32gb	Gyro-S, or the Coder if you only do code
40gb	Swift 1.5 IQ2_XS or Gyro-M
48gb	IQ3_XXS (47gb)
64gb	IQ3_S (54.8gb)

on my machine, with 36gb of VRAM and 80gb of RAM, i use IQ3_XXS: it's the right balance between speed, quality and room for a lot of context.

2. install

the simplest way, on linux, is the official terminal. the script checks your PC, downloads the engine, verifies its sha256 hash and installs everything in your home folder, without admin rights:

curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh | sh

if you'd rather read the script before running it (it's a good habit), download it first with curl -fsSL <the address> -o install.sh, read it, then run sh install.sh.

3. the first launch

strata launch

it asks its questions one by one: the model, the context size, image reading, who is allowed to connect. then it downloads the engine suited to your card and the model (count a good hour for ~70gb, depending on your connection), then it serves the model on a local address compatible with the OpenAI API.

4. use both cards

with two cards, Strata cuts the model into two batches of layers: the first card does the first layers, the second one does the rest, and each one keeps the experts of its own layers in cache. so two cards keep about twice as many experts in VRAM as a single one, and that's where the speed comes from. no need for NVLink: going from one card to the other is light.

by default, Strata first tries the model on your biggest card and only offers to split it if it doesn't fit comfortably. to force it to use both cards:

strata launch --model iq3_xxs --gpus 0,1 --context 128k --parallel 2 --no-tui

--gpus 0,1 takes the cards as nvidia-smi numbers them (the first one is the main one), --parallel 2 opens two slots to serve two agents at the same time, and --no-tui serves the model then prints the address without an interactive screen.

5. speed solo

with a single agent or a single conversation, that's where the model impresses: on my box, Qwen 3.8 Flash Next generates at more than 110 tok/s (109.6 tok/s on the bench with short answers, and up to ~140 in real use). for a model with 125 billion parameters on two second-hand cards, that's API speed.

Strata 0.1.4591.7 tok/s

Strata 0.2.0107.6 tok/s

+ 260 W and calibration109.6 tok/s

two things matter to get there:

1. Strata 0.2.0 or newer: this version turns on a tuning table for RTX 30 cards, which takes generation from 91.7 to 107.6 tok/s on my setup (the measurements).
2. enough power for the cards: at 260 W per card with automatic calibration, i gained a few more tok/s. if you undervolt afterwards (i do, for the noise), you lose a bit of it again, it's all in this guide.

6. and with two agents at the same time

that's the second part, and it's trickier. with the default config, two agents working at the same time got in each other's way a lot: as soon as one read a file, the other dropped to 2 or 3 tok/s. in one evening, i went from 43.8 to 66.4 tok/s per agent, and only four things made the difference:

1. two fast slots rather than four slow ones (--parallel 2): on my box, four slots shared too little memory.
2. Strata 0.2.0, same as for solo.
3. the cards at 260 W instead of 220, with automatic calibration.
4. a PR that isn't merged yet (#2985), which lets the ahead-of-time token prediction (MTP) run in the shared slots. i compile it myself; if you don't want the hassle, skip this step, the first three already do most of it.

the full table, step by step, is in the two agents experiment.

7. go beyond 262K of context

the model natively accepts 262K tokens. you can stretch it twice with YaRN, by adding to the "args" list of the model config (a strata-*.json file):

"--max-context", "409600",
"--rope-scaling", "yarn",
"--rope-scale", "2.0",
"--kv", "int8"

--kv int8 halves the space the cache takes. do the RAM math first: on my machine, a session costs about 1.16gb of cache per 131K tokens, and i froze the whole machine the first time by asking for 512K on four slots (the story is here). with two slots at 400K, my peak is around 70gb. if your server runs under systemd, give it a memory cap (MemoryMax= in the service), so that it's the engine that stops if it goes over, not the whole machine.

what can get stuck

• it doesn't fit: Strata switches on its own to a "low RAM" mode, which is slower. take a more compressed size.
• two cards from different brands: impossible, Strata never splits a model between an nvidia card and an AMD card.
• the download stops: run strata launch again, it picks up where it left off.
• my agents don't finish faster despite more tok/s: that's normal, an agent spends most of its time reading prompts, not writing. i talk about it in the strata 0.2.0 experiment.