api design

Rate limits protect capacity and fairness, and they shape the developer experience as much as any feature on the API. The interviewer wants limits by dimension, tiers that grow with trust, and clear behaviour when limits hit.

A dashboard shows settled charges while concurrent requests create new liabilities. Define what the cap promises, reconcile the exposure and decide how the product behaves when money is reserved.

Batch pricing sells your idle GPU hours. Promise a completion window rather than a latency, make the API shape unsuitable for interactive use, and don't charge for work you couldn't do.

Server-side state simplifies integration and can improve caching, but it means storing data and it does not make old tokens free. Offer it as an option with short retention and give zero-retention customers an equal stateless path.