PRACTICE WORKSPACEhardPremium answer

An open model has a trillion parameters, 32 billion active. Engineering says it is cheap to self-host for our tool. Right?

Make your call first. Use the published answer and rubric to examine your reasoning when you are ready.

Case briefFacts and constraints for your answer

All figures are an illustrative planning fixture. An open mixture-of-experts model has one trillion parameters, 32 billion active per token. Engineering wants to self-host it for an internal tool.

Scenario inputs
Traffic2,000 requests a day, each about 3,000 input and 500 output tokens.
HardwareAccelerators hold 140 GB each and cost $3 per accelerator-hour.
Hosted optionAn endpoint serving the same model charges $1 input and $4 output per million tokens.
Smaller optionA dense 32B model also exists.

Your task. Decide whether self-hosting is cheaper, show the arithmetic, and say at what traffic the answer changes.

Elapsed time00:00
Think it through
0 words