An open model has a trillion parameters, 32 billion active. Engineering says it is cheap to self-host for our tool. Right?
Make your call first. Use the published answer and rubric to examine your reasoning when you are ready.
Case briefFacts and constraints for your answer
All figures are an illustrative planning fixture. An open mixture-of-experts model has one trillion parameters, 32 billion active per token. Engineering wants to self-host it for an internal tool.
| Traffic | 2,000 requests a day, each about 3,000 input and 500 output tokens. |
|---|---|
| Hardware | Accelerators hold 140 GB each and cost $3 per accelerator-hour. |
| Hosted option | An endpoint serving the same model charges $1 input and $4 output per million tokens. |
| Smaller option | A dense 32B model also exists. |
Your task. Decide whether self-hosting is cheaper, show the arithmetic, and say at what traffic the answer changes.
0 words