gpt-oss-120B Requirements: VRAM, RAM & How to Run It

Most guides give you one VRAM number for gpt-oss-120b and stop. The honest answer is a range, because it depends on your setup. The model ships natively quantized to MXFP4 — its Mixture-of-Experts weights are packed at ~4.25 bits per parameter — so the checkpoint is only about 60.8 GiB and fits on a single 80 GB GPU. And because it is MoE with just 5.1B of 116.8B parameters active per token, it runs far faster than a dense 120B would at the same size. This guide gives you the VRAM-by-setup table, the realistic consumer-GPU offload options, and the self-host-vs-API cost math — every number sourced to a primary reference.

gpt-oss-120b requirements at a glance

gpt-oss-120b needs an 80 GB GPU to run comfortably in its native MXFP4 format, but it is a Mixture-of-Experts model — so its real-world footprint and speed are unlike a dense 120B. Here are the load-bearing specs, each from a primary source:

The single most important fact: gpt-oss-120b ships already quantized. Its MoE weights are natively in MXFP4 at 4.25 bits per parameter, with attention layers, routers, and embeddings excluded. You are not applying a lossy after-market quant — this is the format OpenAI released and trained the deployment around.

VRAM by setup: the table competitors skip

There is no single VRAM number — it depends on whether you fit the whole model in GPU memory or offload experts to system RAM. Here is the honest breakdown, with each figure attributed to its source and confidence flagged:

Read this top to bottom and the strategy is obvious. The MXFP4 checkpoint is ~60.8 GiB, so with runtime and KV-cache overhead an 80GB GPU fits it whole — that is the clean, fast path OpenAI designed for. Drop below 80GB and you enter offload territory: the model still loads, but expert layers live in slower system RAM. And the ~240 GB BF16 figure is what makes MXFP4 remarkable — quantization is the difference between one GPU and a multi-GPU rig.

Why gpt-oss-120b is faster than a dense 120B

Because only 5.1B of its 116.8B parameters are active on any given token — a dense 120B activates all of them. That is the Mixture-of-Experts trick, and it is why the "120B" in the name is misleading about speed. Each of the model's 36 layers has 128 experts, of which a router picks the top 4 per token. So the memory cost is set by the total parameters (you must store all 128 experts), but the compute cost per token is set by the active parameters (you only run 4).

This decoupling is the whole point:

• Memory scales with total params (116.8B) — hence the ~60.8 GiB checkpoint you have to fit somewhere.
• Compute scales with active params (5.1B) — hence generation speed closer to a ~5B dense model than a 120B one.

The practical upshot: do not budget latency as if this were a dense 120B. Its throughput profile is much closer to a mid-size model, which is exactly why it is a viable open-weight workhorse. If you are picking among open models, our best open-source LLM roundup puts it in context against the field.

Running gpt-oss-120b on consumer GPUs

You can load it on a single 24GB consumer card by offloading MoE expert layers to system RAM — but expect a real speed penalty, and note no official source endorses a specific consumer config. The MoE architecture is what makes this even possible: because only a few experts fire per token, tools like llama.cpp can keep attention and hot layers on the GPU while parking the bulk of the expert weights in CPU/RAM. The GPU handles the always-on compute; RAM holds the rarely-touched experts.

What to actually expect:

• It loads, but it is slower. Every time a token needs an expert that lives in RAM, that weight moves over the PCIe bus. Unsloth is explicit that offload works but "inference slows significantly."
• Aim for the recommended floor. Unsloth suggests at least ~66 GB of combined VRAM+RAM for practical inference at 6+ tokens/sec. A 24GB GPU plus ample system RAM can clear that bar on total capacity — the bottleneck is bandwidth, not just capacity.
• Confidence is low here. The fact sheet flags that no primary or official source blesses a specific 24GB configuration. Treat consumer-GPU numbers as "it can work," not "it is supported."

If your card is smaller than that, the smaller sibling below is the better fit — or you skip the hardware question entirely with an API.

gpt-oss-20b: when the 120b is too big

gpt-oss-20b runs within 16GB of memory and keeps the same architecture family, context, and license — it is the model to reach for on a consumer GPU. Same MoE design, smaller everything:

A ~12.8 GiB checkpoint that runs within 16GB lands squarely inside a 16GB or 24GB consumer card — no offload gymnastics required. You trade capability for a much easier deployment. See its full model page for details, and the 120b page here when you want the bigger one via API.

How to run gpt-oss-120b locally

Pick a runtime that understands MXFP4 and the harmony format — vLLM, Ollama, or llama.cpp — and the rest is a download. Both models use the harmony response format and will not work correctly without it, so the runtime matters. The good news: mainstream runtimes handle harmony and MXFP4 for you.

A few non-negotiables when you set up:

1. Use a harmony-aware runtime. vLLM, Ollama, and llama.cpp apply harmony server-side. Only a raw inference loop forces you to format it yourself.
2. Budget for KV cache at 128k context. The 131,072-token context is powerful but its KV cache eats memory on top of the ~60.8 GiB of weights — size your GPU with headroom if you use long context.
3. Set reasoning effort. Both models support configurable low / medium / high reasoning effort and native function calling, so you can dial compute to the task.

The pragmatic alternative is to skip local hosting and hit the model through an OpenAI-compatible endpoint — same request shape you already use, no MXFP4 or harmony plumbing on your side. Point your existing client at a gateway base URL:

from openai import OpenAI

client = OpenAI(
    base_url="https://www.datallmlab.com/v1",
    api_key="YOUR_KEY",
)

resp = client.chat.completions.create(
    model="openai/gpt-oss-120b",
    messages=[{"role": "user", "content": "Explain MXFP4 in one line."}],
)
print(resp.choices[0].message.content)

The gateway applies harmony and MXFP4 server-side, so the same code runs the 20b by swapping the model string. No GPU required on your end.

Self-host vs API: the cost math

Self-hosting only wins at sustained high utilization; for bursty or low-volume traffic a per-token API is almost always cheaper. The trap is comparing token prices to zero. Self-hosting is not free — an 80GB GPU costs whether or not it is answering a request, and it sits idle between them. Here is the decision framework:

A worked way to think about it: an 80GB GPU running 24/7 has a fixed hourly cost regardless of throughput. If your traffic keeps that GPU near-continuously busy, dividing that fixed cost across a high token volume yields a low effective per-token rate — self-host wins. If your GPU is busy 10% of the day, you are paying full price for 100% of the time to serve 10% of the work, and a per-token API that charges only for the tokens you actually generate is dramatically cheaper. The break-even is a utilization question, not a price-per-token question.

Because gpt-oss-120b is Apache 2.0 and open-weight, you genuinely have both doors open — a rare luxury. The right move is usually to start on an API, measure your real utilization, and only build the self-host rig once the numbers prove a dedicated GPU stays busy. Our cheapest-LLM cost guide and best-LLM-API roundup lay out per-token options for that first phase.

When not to self-host at all

If your load is bursty, your team is small, or you need more than one model, self-hosting a single 80GB GPU is usually the wrong tool. Concrete signals to stay on an API:

• Spiky or low volume. A GPU you keep for occasional traffic is mostly idle burn. This is the single most common miscalculation.
• You want model choice. One GPU holds one model. If you route between gpt-oss-120b, the 20b, and other providers by task, a gateway is the natural fit — one key, many models, with routing and failover built in.
• You lack GPU ops appetite. Drivers, MXFP4 runtimes, KV-cache tuning, scaling, and on-call are real work. An API absorbs all of it.
• You are still validating. Prototyping against an API costs pennies and takes minutes; buying or reserving an 80GB GPU to test an idea is premature.

Self-hosting an open-weight model is a great option — but it is an optimization, earned by proven, steady demand, not a default. Until then, hitting gpt-oss-120b through an OpenAI-compatible gateway gives you the same open model with none of the hardware.

FAQ