Original practice question. The scoring rubric is our teaching guide. How we research and score answers
TL;DR: An embedding turns text into a list of numbers that places similar meanings close together; vector search finds the stored passages whose numbers sit nearest to the query's. It is excellent at matching meaning across different wording and weak at exact terms (product codes, names, error numbers) and at negation. Production systems combine it with keyword search and a reranker, and measure retrieval separately.
How to approach it
Give an intuition first (meaning as coordinates on a map), then the mechanics in three steps, then where it fails and how teams fix it. Tie it to a product the interviewer can picture, such as search over a help center or a company wiki.
A strong answer
The intuition. Imagine placing every sentence on a huge map where sentences with similar meanings sit near each other: "how do I reset my password" next to "I forgot my login", far from "cancel my subscription". An embedding model produces those map coordinates, a list of hundreds or thousands of numbers per passage. Vector search finds the passages closest to the user's question on that map.
The mechanics.
- Index. Split documents into chunks (a paragraph or a section), compute each chunk's embedding, and store them in a vector index.
- Query. Embed the user's question with the same model.
- Search. Return the chunks whose embeddings are nearest the query's, typically the top 5 to 20, which then go to the language model as context.
Why it is powerful. It matches meaning even when the words differ. A user asking "Can I get my money back?" finds the article titled "Refund policy", even though they share no words.
Where it fails.
| Failure | Example | Why |
|---|---|---|
| Exact identifiers | "Error E-4021" returns articles about other errors | Codes have little meaning for the embedding model |
| Names and rare terms | A product name introduced last month | Embedding model never learned it |
| Negation | "Plans without a contract" returns contract plans | Similar words, near opposite meanings |
| Chunking problems | The answer spans two chunks; neither alone matches well | Splitting broke the context |
| Stale or duplicate content | Old and new policy both retrieved | Index not maintained |
| Permissions | Results from documents the user cannot see | Filtering not applied before search |
The fixes a PM should expect.
- Hybrid search. Combine vector search with keyword search (often a method called BM25), so exact codes and names still match.
- Reranking. Retrieve a larger set, say 50, then use a second model that reads the question and each passage together to reorder them and keep the best 5. Slower but more precise.
- Better chunking. Chunk by document structure (sections, headings), keep titles with chunks, and overlap slightly so answers are not split.
- Metadata filters. Filter by product, region, date and permissions before or during search.
- Index hygiene. Remove outdated versions; re-index when documents change.
The evidence behind those fixes. Anthropic's contextual retrieval write-up (September 2024) gives the exact-match case: a query for "Error code TS-999" can return content about error codes in general and miss TS-999 itself. On its test sets, adding context to chunks before embedding cut the top-20 retrieval failure rate from 5.7% to 3.7%, adding contextual BM25 brought it to 2.9%, and reranking to 1.9% (company-reported). The negation row has a benchmark too: NevIR (EACL 2024) found most retrieval models scored the same as or worse than random ranking on negation, with cross-encoders, the architecture rerankers use, doing best. That is a concrete argument for paying the reranker's latency. And Weller et al. (ICLR 2026) show a theoretical limit: the number of distinct top-k result sets a single embedding can return is bounded by its dimension, and state-of-the-art models failed their simple LIMIT test. Some queries cannot be fixed by a better embedding model; they need keyword search or a different retrieval design.
Measuring it. Recall@k: what fraction of labelled relevant passages are in the top k given to the model? Track query hit rate separately when several passages can be relevant. If recall is low, no amount of prompt work will fix the answers.
What interviewers probe next
"Do we need a dedicated vector database?" Not necessarily. Many teams start with vector search inside an existing database (pgvector in Postgres, for example) and move only when scale or features demand it.
"Can we switch embedding models later?" Yes, but every document must be re-embedded, and retrieval quality must be re-evaluated. Plan for it.
How it's scored
| Signal | Weak | Hire | Strong Hire |
|---|---|---|---|
| Explanation | Vague "AI search" | Correct mechanics | Clear intuition plus index, query, nearest-neighbour steps |
| Failure modes | None | Mentions inaccuracy | Exact terms, negation, chunking, staleness and permissions, each explained |
| Fixes | Better model | Hybrid search | Hybrid search, reranking, structural chunking, metadata filters, measured with recall@k, and knows some failures are limits of single-vector retrieval |
Common mistakes
Assuming vector search alone handles product codes and names.
Ignoring chunking, which often explains bad retrieval.
Evaluating final answers without measuring retrieval.