Original practice question. The scoring rubric is our teaching guide. How we research and score answers
TL;DR: Build RAG over connectors to each source system, with access-control lists captured at ingestion and enforced as a filter before retrieval, so the model never sees a document the user cannot open. Keep permissions fresh by syncing from sources, treat stale permissions as a severity-one incident, and answer with passage-level citations or an honest "not found". Measure retrieval recall, groundedness and permission correctness separately.
How to approach it
Clarify scale (employees, documents, sources such as email, chat, wikis, drives, ticketing), freshness needs, and the security bar. Then say up front that permissions are the core design constraint, not a feature: the system must never show content a user is not allowed to see, and the model cannot be trusted to enforce that.
A strong answer
Requirements. 20,000 employees, tens of millions of documents across a dozen systems, answers within a few seconds, content up to date within minutes to hours, and zero permission leaks.
Architecture.
1. Connectors and ingestion. Each source has a connector that pulls content, metadata and permissions: who can view each document, often through groups. Parse with layout awareness, chunk by structure, embed, and index text for keyword search. Every chunk carries its document's access list.
2. Permission enforcement. At query time, resolve the user's identity and group memberships, then filter the search to chunks the user can access before ranking. Three rules:
- Pre-filter, not post-filter. Filtering after retrieval can leak through counts, snippets or timing, and if the filter has a bug, the model already saw the content.
- The model is never the authorisation system. Instructions like "do not reveal confidential information" are not a control.
- Permissions must be fresh. When someone loses access to a document (they changed teams, a file was restricted), the index must reflect it quickly. Sync permission changes on a short cycle, and treat stale permissions as a severity-one incident class.
This is how shipped products describe it. Glean's security principles say connectors ingest each source's permission model and that the signed-in user's identity is evaluated against those mirrored permissions before any result is returned. Microsoft's Copilot privacy page makes the same promise: it surfaces only data the individual user can view.
3. Retrieval. Hybrid search (vector plus keyword, because employees search for project codenames, ticket numbers and people's names), metadata filters (department, date, source), and reranking down to the best few passages.
4. Generation. Answer only from retrieved passages, with passage-level citations linking back to the source document. If nothing relevant is accessible, say so; never imply that inaccessible information exists ("there is a document about layoffs you can't see" is itself a leak).
5. Freshness. Event-driven updates where sources support webhooks; scheduled syncs otherwise. Show document dates in citations so users can judge staleness.
6. Oversharing, the failure a perfect filter cannot catch. The access filter faithfully returns a file that someone shared with the whole company by mistake four years ago. Keyword search rarely exposed it because nobody knew its name; an assistant asked about "planned layoffs" finds it in seconds. Microsoft ships Restricted Content Discovery for exactly this: an admin hides a SharePoint site from organisation-wide search and Copilot without changing anyone's permissions, while its owners review who has access. The documentation, checked 6 November 2026, calls it a temporary control and warns that on sites with more than 500,000 items the change can take more than a week to reach search and Copilot. The product decisions that follow are yours. Launch with HR, finance and legal sources excluded or restricted. Run a scan for sensitive files shared company-wide before rollout and send owners the list. Give admins a discoverability switch that is separate from access, and show them how long it takes to apply.
Evaluation.
| Metric | How |
|---|---|
| Permission correctness | Automated tests with synthetic users in different groups querying restricted content: must be 100% |
| Retrieval recall@k | Labelled question-to-document pairs from real queries |
| Groundedness and correctness | Validated judges plus human audit |
| Freshness lag | Time from source change (content or permission) to index |
| Oversharing exposure | Sensitive-labelled files shared company-wide, counted before launch and tracked down |
| Usage outcomes | Searches that end in a click or no follow-up, self-reported time saved |
Scale and cost. Embedding tens of millions of chunks is a one-time cost plus incremental updates. Per-query cost is retrieval plus a model call with a few thousand tokens of context; cache the stable system prompt. Route simple lookups to a smaller model.
What interviewers probe next
"A user asks 'what is my manager's salary?' What happens?" If HR documents are correctly permissioned, retrieval returns nothing the user can see and the assistant says it cannot find that. The test suite should include exactly this kind of query.
"Permission sync from one source lags by a day. What do you do?" Exclude that source's restricted content until sync is reliable, or fetch permissions live at query time for it. A day-old access list is a leak waiting to happen.
"How do you handle content shared via a public link inside the company?" Respect the source system's semantics: if the link makes it company-wide, it is retrievable company-wide.
"Security wants every source on day one. Why are you holding HR back?" Because the filter enforces the permissions the source has, and nobody has audited those. Show the scan result: the number of sensitive files open to the whole company. Agree an exit criterion (owners have reviewed every flagged file) and a date, so the restriction does not become permanent by default.
How it's scored
| Signal | Weak | Hire | Strong Hire |
|---|---|---|---|
| Permissions | Model told not to reveal secrets | Filters results by access | ACLs captured at ingestion, pre-filtered at retrieval, freshness as an incident class, no hints about hidden content, and a launch plan for files that are overshared at the source |
| Retrieval | Vector search only | Hybrid search | Hybrid plus metadata filters and reranking, tuned for names and codes |
| Evaluation | Accuracy only | Retrieval and answer metrics | Adds automated permission tests with synthetic users and freshness lag |
Common mistakes
Relying on the prompt to protect confidential content.
Filtering after retrieval and ranking.
Treating permission sync as a nightly batch job.
Assuming source permissions are correct, so the first leak is a file the filter was right to return.
See permission-aware RAG and RAG explained.