Harvey LLM & RAG Fundamentals interview questions

Everyone knows models make things up. The interviewer wants to hear why, which kinds of hallucination exist, and a layered set of product defences, without promising to eliminate it.

Many enterprise RAG failures happen before retrieval: the system never read the document correctly. The interviewer wants you to find the parsing problem and fix it at the right layer.

Recommend an approach to a CEO who needs a product decision. Explain what retrieval and fine-tuning each change, then connect that difference to freshness, reliability and cost.

The first vocabulary word of AI product management. Tokens are the unit of cost, speed and memory, and a PM who can't reason in them can't price, budget or design an AI feature.

A fundamentals question that comes up when a stakeholder notices the assistant answered differently yesterday. The good answer explains sampling in one minute and then turns it into product decisions.

Retrieval quality decides most RAG products, and embeddings are where it starts. The interviewer wants the intuition, the failure modes, and the fixes a PM would push for.

A larger context window changes what fits in a request. Decide whether to use it by checking corpus size, freshness, permissions and cost per successful query, including cache misses.

Fine-tuning is sometimes the best decision a team makes and often a costly detour. The interviewer wants the test that tells them apart, with numbers.

A vocabulary check on one of the most important model developments of recent years. The good answer explains the trade (accuracy for time and money) and gives a rule for when it pays.

A fluency question about the request path. The interviewer does not need the maths; they want to know you can connect each stage to latency, cost and design decisions.

Retrieval failures look like model failures to users. This question tests whether you can locate the failure precisely and choose fixes in the order that pays, rather than swapping models.

A puzzle with a precise answer. Price per token is only comparable if a token means the same thing, and across model generations it often does not.

A small-sounding problem that becomes a big one at volume. The good answer explains why free-form generation produces malformed output and which controls make it near-impossible.

A design trade-off with cost, accuracy and latency all pulling differently. The interviewer wants a measured comparison on real receipts and a hybrid that sends each image down the cheapest path that works.

A trade-off question where both sides have serious arguments. The interviewer wants the dimensions that decide it, and a recommendation that depends on facts you would go and get.

A question about fairness and unit economics hiding in a tokenizer. The interviewer wants the mechanism, the arithmetic, and options that do not leave non-English users with a worse product.

A platform strategy question with hardware, privacy, cost and capability all in play. Principal-level answers design a split by task, with a privacy story users can understand and a plan for the devices that cannot run local models.