undervolting a 3090 or a 3080 ti on linux, without losing performance

before you start

on windows, you undervolt an nvidia card by dragging the curve in MSI Afterburner. on linux there's no equivalent tool, and the driver doesn't let you touch the voltage directly. so we'll get the same effect through a two-step detour:

1. we lock the card's maximum clock, so it never climbs into the zone where it draws a lot of power for very little gain;
2. we shift the whole voltage/frequency curve up. the card then reaches the locked clock while asking for less voltage than before, and that's what brings the power draw down.

it's different from simply lowering the power limit (nvidia-smi -pl), which forces the card to slow down as soon as it hits the cap. on my box, at equal power draw, the undervolt keeps clearly more throughput: the numbers are in the experiment.

you'll need: linux with the proprietary nvidia driver, root access, and python 3. it works on RTX 30 cards; i've only tested it on a 3090 and a 3080 Ti.

the risk: an undervolt that's too aggressive makes the card unstable (wrong calculations, driver crash). none of this survives a reboot as long as you haven't made it automatic, and we'll see how to undo everything at the end.

1. measure the stock state

before touching anything, write down how your card behaves under your real load (a model generating, a game, a render), otherwise you won't know whether you gained anything. in a terminal, while the load is running:

nvidia-smi --query-gpu=index,name,power.draw,clocks.gr,fan.speed,temperature.gpu --format=csv -l 2

keep the average power draw, the clock, the fan and your throughput (tok/s, fps…) aside. that's your baseline.

2. install the NVML library for python

the curve shift doesn't go through nvidia-smi, you have to call the NVML library directly. nvidia's official package is called nvidia-ml-py (it's imported as pynvml). i put it in a separate environment so i don't touch the system python:

sudo python3 -m venv /opt/gpu-undervolt/venv
sudo /opt/gpu-undervolt/venv/bin/pip install nvidia-ml-py

3. test the shift by hand

start with a single setting, by hand, to see what the card can take. in order:

# keep the driver loaded between two programs, otherwise the settings get lost
sudo nvidia-smi -pm 1

# shift the curve by +150 MHz on card 0
sudo /opt/gpu-undervolt/venv/bin/python -c "import pynvml as n; n.nvmlInit(); n.nvmlDeviceSetGpcClkVfOffset(n.nvmlDeviceGetHandleByIndex(0), 150)"

# lock the clock between 210 and 1500 MHz on the same card
sudo nvidia-smi -i 0 -lgc 210,1500

run your load again and watch the command from step 1. if all goes well, the clock tops out at 1500 MHz and the power draw drops clearly.

why these values: on my 3090, 1500 MHz with +150 gave the best ratio. raising the lock to 1650 MHz brought nothing in throughput, only more power draw and noise. on my 3080 Ti, i could keep 1800 MHz with the same +150. every card is different, so start from these values and adjust.

4. check that it's stable

an unstable card doesn't necessarily crash: it can also compute wrong, which is worse. let your real load run for at least fifteen minutes, and check that the results stay right. for a language model, a small series of math questions with a single right answer is enough; if the answers change or if the driver logs errors (sudo dmesg | grep -i xid), lower the shift (+100, then +50).

note: comparing generated texts word for word doesn't work as a test, because most engines don't give back exactly the same output from one run to another, even without undervolt.

5. make it automatic at boot

once you've found the right values, i put them in a small script started by systemd at boot. it identifies the cards by their UUID rather than their number, because the order of the cards can change. get the UUIDs with:

nvidia-smi -L

then create /opt/gpu-undervolt/apply.py with your UUIDs and your values:

#!/usr/bin/env python3
"""undervolt at boot: cards identified by UUID, not by number."""
import subprocess, sys, pynvml as n

CARDS = {  # uuid: (name, max clock in MHz, offset in MHz, power limit in W)
    "GPU-xxxxxxxx-...": ("RTX 3090", 1500, 150, 230),
    "GPU-yyyyyyyy-...": ("RTX 3080 Ti", 1800, 150, 260),
}

n.nvmlInit()
ok = True
for i in range(n.nvmlDeviceGetCount()):
    h = n.nvmlDeviceGetHandleByIndex(i)
    uuid = n.nvmlDeviceGetUUID(h)
    uuid = uuid.decode() if isinstance(uuid, bytes) else uuid
    if uuid not in CARDS:
        continue
    name, lock, offset, power = CARDS[uuid]
    try:
        subprocess.run(["nvidia-smi", "-i", uuid, "-pm", "1"], check=True, capture_output=True)
        subprocess.run(["nvidia-smi", "-i", uuid, "-pl", str(power)], check=True, capture_output=True)
        n.nvmlDeviceSetGpcClkVfOffset(h, offset)
        subprocess.run(["nvidia-smi", "-i", uuid, "-lgc", f"210,{lock}"], check=True, capture_output=True)
        print(f"{name}: locked at {lock} MHz, offset +{offset} MHz, {power} W")
    except Exception as e:
        ok = False
        print(f"{name}: FAILED {e}", file=sys.stderr)
sys.exit(0 if ok else 1)

and the service /etc/systemd/system/gpu-undervolt.service:

[Unit]
Description=GPU undervolt
After=nvidia-persistenced.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/opt/gpu-undervolt/venv/bin/python /opt/gpu-undervolt/apply.py

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now gpu-undervolt.service
systemctl status gpu-undervolt.service   # should show the line for each card

if a service for your AI server starts at boot, add Before=the-name-of-that-service.service in the [Unit] section so the undervolt is applied first.

6. undo everything

if something's off, these commands put the cards back to stock until the next reboot. the first two release the clock and remove the shift:

sudo nvidia-smi -rgc
sudo /opt/gpu-undervolt/venv/bin/python -c "import pynvml as n; n.nvmlInit(); [n.nvmlDeviceSetGpcClkVfOffset(n.nvmlDeviceGetHandleByIndex(i), 0) for i in range(n.nvmlDeviceGetCount())]"

for the power limit, ask each card for its factory value, then set it back card by card:

nvidia-smi --query-gpu=index,power.default_limit --format=csv
sudo nvidia-smi -i 0 -pl <the value of card 0>

and sudo systemctl disable gpu-undervolt.service so it doesn't come back at boot.

what it gave me

on my box, with two agents generating at the same time: 452 W → 358 W for both cards, the 3090 fan from 91% to 48%, and 62 tok/s per agent instead of 67. the details, the settings that didn't work and the limits of the measurement are in the experiment.

stock452 W

both undervolted358 W

what can get stuck

• the offset goes back to 0: the new per P-state offset API did that on my machine. only nvmlDeviceSetGpcClkVfOffset (the GPC offset) really took.
• the card still runs hot: normal, the undervolt mostly lowers power draw and noise. my 3090 still goes up to 84 °C.
• throughput drops a lot: you probably also have a power limit that's too low. below 200 W, my 3090 collapses (788 MHz at 180 W).