Turn model behavior into product requirements

TL;DR: A language model generates a response from learned parameters and the context available on this request. Specify which facts must come from a current source, which properties code must enforce, and which behaviors need repeated evaluation. Fluency is poor evidence that any of those requirements have been met.

Where you are. The technical foundations section. Start here if you can describe a customer problem but struggle to explain why an AI system might fail it.

The sentence that costs you a customer

In this fictional Kitewire case, a support assistant writes: “Your parcel will arrive tomorrow. We have refunded the delivery fee.” The sentence reads well. The carrier has supplied no arrival estimate, and nobody has called the refund service.

There are two failures. The arrival promise lacks evidence. The refund claim describes an action that never happened. Improving the writing fixes neither.

A language model predicts tokens, the units its tokenizer uses to represent text and other supported inputs. Its learned parameters encode patterns from training. They are not a live order database. At inference time, the application supplies instructions and context; the model generates a response, possibly including a proposed tool call. The application decides whether that call is permitted and executes it. A model requesting a refund is different from a payment system confirming one.

Four mechanisms, four product decisions

Attention lets a model use relationships among tokens in its available context. That does not guarantee it will use every important fact correctly. A large context window answers “what can fit?”; your evaluation answers “what did the system actually use?”

Suppose your fictional request budget is 8,000 tokens. Instructions use 900, the conversation 1,500 and order records 2,600. Reserving 1,000 for the response leaves 2,000 for policy excerpts and other overhead. Treat this as a planning worksheet. Actual limits, reasoning-token accounting and tokenizer behavior depend on the selected system. Have engineering measure real requests, including Hindi or Tamil messages; an English word-to-token shortcut is not a cross-language estimate.

Explain the mechanism without claiming a fact lookup

Try a sixty-second explanation for Kitewire's operations lead. Tokens become vectors; attention combines information from allowed positions; repeated transformations produce scores for the next token. The chosen token then becomes context for later output. Keep two meanings of “weights” separate: learned parameters belong to the model, while attention scores are calculated for the current input.

Now explain why the wrong arrival date does not automatically justify retraining. The current carrier record may be absent. A correct record may be misinterpreted. A valid draft may be followed by an unexecuted refund. Each failure needs a different owner.

Use a fresh-page test: correct the delivery date in one conversation, then start another with no saved memory or carrier lookup. The model has not learned a durable account update merely because the first chat accepted the correction. Design the source integration before promising recall across sessions.

Write the requirement at the right layer

Kitewire's PM writes four acceptance statements:

• Delivery dates in the reply must come from a timestamped carrier record; absent that record, say the date is unavailable.
• A refund confirmation requires a successful transaction identifier returned by the refund service.
• The sending service rejects any recipient outside the authenticated retailer's order.
• On a held-out set, drafts must preserve the customer's actual request, including negation such as “do not cancel.”

The first two combine application checks with evaluation of the explanation. The recipient constraint belongs in code. The last criterion requires examples of language, including paraphrases and repeated trials. These distinctions tell engineering what to build and tell the PM what to inspect.

Reasoning models may spend additional computation on difficult tasks. That can be useful when resolving conflicting evidence, but it adds a cost and latency question. A stronger model still needs current order data. A multimodal model may read a damaged parcel photo; the photo alone cannot establish purchase ownership or refund eligibility. Keep those decisions tied to records and policy.

Do this before moving on

Write an acceptance table for an assistant that reads a delivery photo and drafts a damage claim. Include four columns: customer promise, required evidence, enforcement owner, test. Cover unreadable images, somebody else's parcel, missing order data and a valid damaged-item claim.

A defensible first version drafts a claim for the customer to inspect. It identifies visible damage without inventing its cause, requires an authenticated order match, and makes no payment promise. If an image is unreadable, ask for another image or route to support. A confident description is still wrong when it names an item that is not visible.

Check your table by removing the model. Which protections still operate? Authentication and payment authorization should survive that thought experiment. Then repeat the same ambiguous image several times; the variation tells you what the review interface must make easy to catch.

Go deeper

• Tokens and context windows explains the accounting behind the request budget you just wrote.
• Attention and learned weights gives the mechanism behind the explanation you just rehearsed.
• How language models generate connects token prediction to errors a PM must plan for.
• Deterministic and probabilistic behavior helps assign each acceptance criterion to the correct control.

Key takeaways

• Require current evidence for current facts.
• A proposed tool call is not a completed action.
• Budget and test context on the languages your users use.
• Separate enforceable constraints from behaviors measured across examples.

Check yourself

Answer out loud first, then reveal. Retrieving it yourself is what makes it stick; reading ours and agreeing does very little.

1. 1The reply says a refund succeeded, but there is no transaction ID. What should the product do?

   Answer it, then reveal

2. 2Will temperature zero guarantee that the delivery promise is true?

   Yes, because it removes uncertaintyNo; grounding and correctness are separate from sampling