GPU Cost Per Token: Why the Operator Decides

GPU cost per token is set by the operator, not the chip alone. SemiAnalysis reported that on the same B200 hardware, two serving stacks at the same interactivity target produced roughly 200 and 300 output tokens per second per user, so the faster stack delivered about 50% more usable output for the same hourly bill.

GPU Cost Per Token: What the Operator Changes

GPU cost per token measures how much acceptable output you get per unit of spend, and the operator's serving stack moves it as much as the hardware does. In an example published by SemiAnalysis in October 2026, two B200 deployments serving the same model at the same interactivity target produced about 200 and about 300 output tokens per second per user.

Oliver Kennon, a research analyst at SemiAnalysis, presented that comparison at an AI Engineer event. Both estimates used B200 GPUs, the same model, roughly 8,000 input tokens and 1,000 output tokens, and a P50 target of 100 output tokens per second for each user. The roughly 50% gap came from the serving stack: the software, configuration and number of concurrent requests.

SemiAnalysis builds on Nvidia's published specifications for the B200, a Blackwell-generation data center GPU. The comparison does not claim the chip performs differently. It claims the product around the chip does, and that the customer buys the product.

The arithmetic is straightforward. If two deployments carry the same hourly bill and one returns about 1.5 times the usable output, rental spend per acceptable token falls by roughly a third on the faster one. That figure is a derived illustration from the 1.5 ratio, not a separately measured benchmark.

Why GPU Hours Are an Input, Not an Output

GPU hours measure rented input, while intelligence delivered to a user is the output that matters. Dollar-per-hour pricing treats a unit of machine time as if it were a unit of work, and the two diverge sharply once software and operations enter the picture.

In the SemiAnalysis framing, hardware and electricity are the raw inputs. Software and operations act as a multiplier on them, and the end product is useful intelligence. The operator chooses and implements the software that serves the model and keeps requests moving through the system.

Useful intelligence has to clear three bars before it counts. It must be good enough for the task, arrive quickly enough to be useful, and stay within the access boundaries the buyer agreed to. The last requirement is rarely captured in chip performance metrics, which is why cluster security belongs in any honest assessment of an operator.

Some supply constraint currently hides these differences. Kennon estimated two to three years of visibility with demand outstripping supply, a period in which producers with very different cost structures can all find customers. That condition does not hold forever, and the differences become visible when customers gain alternatives.

How SemiAnalysis ClusterMAX Rates an Operator

ClusterMAX is SemiAnalysis's rating product for managed GPU clusters, and it scores operators on provisioning, reliability, monitoring and security. Kennon said the ratings were at version 3.0 in October 2026 and covered a couple of hundred clusters, with work underway toward a 4.0 release.

The rating rests on hands-on testing, provider documentation and customer feedback. The point is to see how a service runs when a machine fails, not only on a good day. Those four dimensions map directly onto the questions a buyer should ask before signing.

Dimension — What it tests — Why it affects cost per token — n — --- — --- — --- — n — Provisioning — Time and reliability of getting capacity — Idle waiting burns budget without producing output — n — Reliability — Behaviour under hardware failure — Outages stop acceptable responses — n — Monitoring — Visibility into logs and metrics — Failures go undetected and uncorrected — n — Security — Access boundaries between tenants — Leaked data makes output unusable

The scope limit matters more than the headline. ClusterMAX ratings apply to managed clusters and are relative to peers, with pricing and availability factored in. Kennon stressed that bare metal sits outside that assessment, because with bare metal more operating work falls to the customer and a managed-service rating does not automatically cover every product a provider sells.

SemiAnalysis has also published security findings on managed GPU environments. Kennon cited a case where monitoring access exposed another customer's logs and metrics. The GPUs in that case were fast, and the service still failed on a requirement that no throughput benchmark captures.

Production Cost vs Rental Price vs Your Bill

Rental price is revenue to the GPU owner and an input cost to the inference producer, and neither number is the producer's full production cost. Kennon treats these as separate quantities that get conflated whenever someone quotes a dollar-per-GPU-hour rate as the cost of intelligence.

The GPU owner pays for equipment, power, space and support, then rents out hours. The inference producer buys those hours and sells responses. Its full cost includes the rent plus the software, staff and other services it pays for beyond the rental. A producer that owns its own equipment instead spreads the equipment investment over its working life and adds running costs.

That gives a working formula: count all spending needed to deliver the service over a period, then divide by the output customers can actually use. The result is a cost per acceptable token for a comparable workload. Better operations lower it by widening the denominator, and the extra spending on testing and safe deployment has to be justified by the extra output.

Kennon was direct about the durability of that advantage. A repeatable operating process can support a lasting cost and price advantage, but one good benchmark does not establish it, and each new workload still needs checking. The advantage is earned on an ongoing basis rather than installed once.

Your Cost Curve Position Is Relative

Your position on a cost curve depends on what competitors can deliver, not only on your own improvement. Every producer on the curve sells comparable output, so the producer spending less to deliver the same usable work keeps more of the price when prices fall.

Kennon used a small hypothetical to make the trap visible. Your full cost per successful task falls from 10 to 8 over a period. Your competitor's falls from 6 to 12. Both improved on their own terms, and you started cheaper and ended more expensive.

That result gives the competitor more room to lower selling price, provided customers find the two services equally acceptable. Improvement without a relative measure can therefore coincide with losing ground.

Scarcity makes this easy to miss. When demand exceeds supply, strong revenue can make cost differences feel less urgent, and leadership attention naturally goes to building more capacity. Kennon's historical point is that every commodity has seen supply and demand swings, and not seeing the timing of the next one is not evidence against it. The practical advice is to build operating capability and cost advantage while capital is available and the sun is shining.

What Compute Futures Can and Cannot Hedge

Compute futures can hedge rental price risk, and they cannot hedge a failed deployment or an unpaid customer. A futures contract sets a standardized term for future delivery or cash settlement, and cash settlement depends on a price reference that has to be credible.

A rental price benchmark summarizes the price of GPU hours within its defined market. It is not the producer's production cost and it is not a measure of intelligence delivered. Service scope, contract terms, whose prices are sampled and how they are combined all shape the reference point, and operating quality can influence prices without the index measuring it separately.

Kennon drew a careful line around what an operator rating contributes. Rating the operator is a building block for a credible futures market, because it helps define which compute services are comparable. A rating does not by itself define a futures grade, establish a sound benchmark, or create enough trading to buy and sell easily.

Two more limits are worth keeping straight. Charging more than an index does not by itself make a hedge bad; what matters is whether changes in the reference track changes in your rent. And rent is only one component of an inference producer's cost, since the operator still has to convert compute into work a customer can use.

A Practical Checklist for Comparing GPU Offers

Compare GPU offers on delivered output at your requirements rather than on headline dollars per hour. The same hourly price can buy very different amounts of acceptable work, and the difference sits in software, settings, concurrency and support.

1. Fix the workload. State the model, the input and output token ratio, and the latency target you actually need, so two offers are measured on the same task.n2. Ask for tokens per second per user at your latency target, not peak throughput.n3. Ask who owns testing, deploying and rolling back serving-stack updates, and whether that party can do the work.n4. Check monitoring and security evidence, since exposed logs or metrics make output unusable regardless of speed.n5. Confirm the assessment covers the service you are buying, because managed-cluster ratings do not automatically extend to bare metal.n6. Recheck the numbers as your workload changes and competitors improve.

Teams inside semiAnalysis's public calculator can run that comparison themselves. Kennon pointed attendees to the InferenceX calculator at SemiAnalysis as a public tool for producing these estimates. Building an internal version of the same comparison is useful when your workload mix differs from the published presets.

Frequently Asked Questions

• What does GPU cost per token mean? It is total spending to deliver a service over a period divided by the output customers can actually use, measured for a comparable workload. SemiAnalysis treats it as a practical proxy for price per successfully completed task at a required quality level.

• Did the B200 example measure the chip or the operator? It measured the operator. Both deployments used B200 GPUs, the same model and the same latency target, and the roughly 50% output gap came from the serving stack, settings and number of concurrent requests.

• Is GPU cost per token the same as dollars per GPU hour? No. Dollars per GPU hour prices an input, while cost per token divides spend by usable output. Two providers can charge the same hourly rate and return very different amounts of acceptable work.

• What is ClusterMAX and what does it rate? ClusterMAX is SemiAnalysis's rating product for managed GPU clusters, scoring provisioning, reliability, monitoring and security. Kennon said the ratings were at version 3.0 in October 2026, covered a couple of hundred clusters and were relative to peers.

• Do ClusterMAX ratings apply to bare metal? No. Kennon stated explicitly that the ratings cover managed clusters, and that bare metal leaves more operating work with the customer. A managed-service rating does not automatically cover every product a provider sells.

• Can a compute futures contract guarantee usable output? No. A futures contract can offset rental price changes, and it does not fix a failed deployment or ensure a customer pays. Whether a hedge works depends on how closely changes in the reference track changes in your actual rent.

• Why does relative cost position matter during a supply shortage? Scarcity lets producers with very different cost structures all find customers, which hides the gap for a while. When customers have more choices and prices come under pressure, the producer with lower cost per usable output has more room to compete.

• How does the operator affect security? The operator implements the software and access boundaries around the cluster. SemiAnalysis reported a case where monitoring access exposed another customer's logs and metrics, which shows that fast hardware can still produce a service that fails on a core requirement.

• What is the takeaway for buyers of GPU capacity? Ask who can repeatedly meet your requirements at the lowest full cost per completed task, and check that the assessment covers the exact service you are buying. Hardware gives you potential; the operator decides how much of it becomes useful work.

Where This Leaves the Compute Market

The compute market rewards operators who can turn the same hardware into more usable output at a lower full cost, and keeps rewarding them only while they keep doing it. Kennon closed on the point that markets do not become informed on their own, and that rating operators is one building block toward comparable grades and credible price references.

For anyone holding a budget, the practical shift is small and consequential. Ask what an hour of capacity actually returns, how the provider handles failure and updates, and how its cost per acceptable token compares with alternatives rather than with last quarter's own number.

If you are documenting your own work on inference economics, Skalablog turns a recorded talk into a written draft you can review and publish, starting from a YouTube link and ending with a structured article you control.

Source video