TL;DR: Gross margin for an AI application is roughly (price − inference − retrieval and infrastructure − human fallback) ÷ price, computed per user or per unit sold. It varies sharply by user segment because usage is long-tailed. Falling model prices help margins only if pricing is anchored to customer value rather than passed through. Protect margin with routing, caching, allowances and tiered pricing, and track it by segment, not just on average.
The formula
GM = (P − (C_inference + C_retrieval_infra + C_human)) ÷ P
- P: price per unit sold (per seat per month, per task, per resolution).
- C_inference: model costs for all calls the unit triggers, including retries.
- C_retrieval_infra: vector search, storage, orchestration, logging, evaluation runs.
- C_human: human review, fallback handling and AI-related support.
Traditional SaaS often reports gross margins well above 70%, because serving another user costs little. AI applications pay for every interaction, so margins depend on usage.
A worked example
An AI add-on priced at $20 per user per month, with retrieval and infrastructure at $1 per user:
| Segment | Share of users | Inference per user | Margin for that segment |
|---|---|---|---|
| Light | 70% | $0.60 | 92% |
| Regular | 25% | $6 | 65% |
| Heavy | 4% | $30 | −55% |
| Extreme | 1% | $120 | −505% |
| Weighted | $4.32 | about 73% |
The average looks healthy; the tail does not. A small share of users can cost several times what they pay, subsidised by light users. If the product gets more popular with heavy users, margin falls.
Falling model prices
Model prices per unit of capability have fallen repeatedly, and providers offer wide price spreads between tiers (about 100 times between Anthropic's cheapest and most expensive models as of October 2026). What that does to margin depends on pricing:
- Price anchored to value (per seat, per outcome): cost falls, price holds, margin improves.
- Price passed through (per token, per credit tied to costs): savings go to customers; margin roughly flat.
- Competitive pressure: competitors get the same discount; if customers buy on price, some of the savings will be competed away.
Levers that protect margin
| Lever | Effect |
|---|---|
| Routing to smaller models for easy requests | Large drop in average inference cost |
| Prompt caching | Large discount on repeated prefixes |
| Usage allowances and fair-use limits | Caps the tail |
| Tiered pricing for heavy users | Aligns price with consumption |
| Batch processing for offline work | Discounted inference on eligible jobs |
| Reducing human fallback via higher resolution rates | Often the largest lever where humans are involved |
Human fallback is often the biggest cost
In support, document processing and other workflows where people handle what the AI cannot, human cost per unresolved item is usually many times the inference cost. Raising resolution rates improves margin more than cutting model prices. See effective cost with human fallback.
Tracking margin
Report margin by segment and plan monthly, watch the top 1% of users, model scenarios for price changes and usage growth, and include margin in launch decisions for new AI features. A feature that is popular but loses money on its heaviest users is a pricing problem waiting to happen.
In the interview
Margin questions ("is this profitable?", "the team wants a model three times more expensive") reward segmenting usage, computing margin per segment, explaining the effect of falling prices under different pricing models, and naming levers. Candidates who compute only the average margin miss the point interviewers are testing.
Next: pricing AI products · the moat ladder