Understand what your decisions depend on.
One idea per lesson, from evals and unit economics to agents and regulation. Each lesson ends in the decision a product manager owns, and links to the questions that test it.
Begin the curriculum →Type a term such as judge, routing or injection, or pick a track below.
Jump to a topic
AI Product Discovery
9 conceptsThe AI Fit Test
Five questions that decide whether a step needs a model, a rule, or a better form.
Cost of a Wrong Answer
Who absorbs the error, how it gets caught, and what undo exists decide the whole design.
Capability vs Reliability
A demo proves the model can do it once. Production needs it done right across segments, every day.
AI Product Archetypes
AI in an app, AI as the product, and AI inside a big company. What good looks like in each role.
Deterministic Plus Probabilistic
Rules for policy and money, models for messy input and language, people for exceptions.
Writing an AI PRD
The sections an AI spec needs: failure modes, thresholds, golden set, cost target, kill criteria.
Sizing an AI Opportunity
Count tasks, time and error cost bottom-up, then discount for quality and adoption.
The AI Product Lifecycle
Connect discovery, data, evaluation and launch with decisions and accountable owners.
Customer Segmentation
Separate user, buyer and affected people, then choose a task worth improving.
Designing for Uncertainty
7 conceptsAutonomy Levels
L0 suggest to L4 delegate. Autonomy is set per action and earned with evidence, not declared.
Risk and Reversibility
Low-risk reversible actions run freely; high-risk irreversible ones need confirmation and audit.
Designing for Uncertainty
Citations, abstention, clarifying questions, editable drafts, targeted flags and fallbacks.
Human Handoffs
Explicit triggers, two kinds of escalation error, and a handoff payload nobody has to re-ask.
Measuring Trust
Override, accept-without-edit, undo, re-contact. Falling override at stable quality is earned trust.
Perceived Latency
Streaming, progress narration, skeletons and classic results first. Perceived vs actual latency.
Discoverability and First Use
Users cannot see what a model can do. Show it at the moment of need, and only show what works.
Strategy & Moats
9 conceptsThe Moat Ladder
Prompt and UX, proprietary context, workflow, eval and data flywheel, distribution and trust.
Wrapper Gross Margin
GM = (price − inference − infra − human fallback) ÷ price, and what falling model prices do to it.
Build, Buy or Fine-Tune
API by default; fine-tuned open weights when evals and volume justify it; own model rarely.
Pricing AI Products
Seat, usage, credit and outcome pricing, the definition problem, and margin risk of flat seats.
Platform Risk and Lock-In
Providers moving into your space, and the portability order: evals > data > tools > prompts > SDK.
Data Flywheels
Labelled failures and outcome data compound; raw usage logs mostly sit in storage.
From Pilot to Production
Pilots stall without an owner, a baseline and a decision date. Set all three on day one.
Feedback and Exposure Bias
Record exposure and outcomes, distinguish missing labels, and verify changes before release.
Product Strategy and Distribution
Compare adoption, contribution and customer access before choosing a sales channel.
How LLMs Work, for PMs
11 conceptsTokens and Context Windows
What a token is, why input and output are priced differently, and what fits in a context window.
How LLMs Generate
Why outputs vary, why the first word is slow, and what each stage means for product design.
Embeddings and Vector Search
Meaning as coordinates, nearest-neighbour search, hybrid with keywords, and reranking.
RAG Explained
Retrieve, augment, generate: the pipeline, the choices at each stage, and where it breaks.
Fine-Tuning and LoRA
What fine-tuning changes, the data it needs, and when it is wasted effort.
RAG vs Fine-Tuning
Retrieval for knowledge that changes, fine-tuning for behaviour. Often both, in that order.
Reasoning Models
Thinking tokens buy accuracy on multi-step problems. When to pay, and how to route to them.
Classical ML Decisions
Choose models and thresholds from error costs, data quality and operating constraints.
Post-Training Objectives
Separate demonstrations, preferences and adapter methods before funding a training run.
Context Engineering
Choose current evidence, instructions and history within a tested context budget.
Attention and Model Weights
Trace a token through attention, then separate training from what happens during a request.
AI Architecture Patterns
7 conceptsPermission-Aware RAG
ACLs captured at ingestion, filtered before retrieval. The model is never the authorisation system.
Model Gateway and Routing
Provider-agnostic routing, fallback chains, budgets and logging, and routers evaluated per segment.
Latency Budgets
E2E = queue + retrieval + TTFT + tokens/TPS + tools + render. Budget at p95, cut the biggest bar.
Prompt and Prefix Caching
Stable prefixes, hit rates, write versus read pricing and TTLs, with the arithmetic.
Fallbacks and Degradation
Timeouts, circuit breakers, secondary models, classic paths and queues, decided per feature.
Model Migration
API portability is not behavioural portability. Baseline, adapt, compare, shadow, canary, ramp.
Document AI Pipelines
Parse, classify, extract, validate and cite. Where each stage fails and what it costs.
The Agent Lifecycle
10 conceptsWorkflow vs Agent
Workflows follow paths you design; agents choose their own. Use the least autonomy that works.
Reliability Compounding
P(task) = p^k. At 95% per step, ten steps finish 60% of the time. How to change the exponent.
The Tool Execution Envelope
The model proposes; code validates, authorises, classifies risk, approves, executes and checks.
Budgets and Loop Control
Step, token, cost and time caps enforced outside the model, plus what the user sees when one hits.
Single vs Multi-Agent
Start with one agent. Split only for separate permissions, real parallelism or independent review.
MCP and A2A
MCP connects agents to tools and data; A2A connects agents to agents. What each means for a PM.
Agent Memory
Four kinds of agent memory, and the rules for provenance, expiry and deletion.
Computer-Use Agents
Agents that operate screens reach systems without APIs, at a cost in speed, reliability and risk.
Prompts, Tools and Skills
Separate the task instruction, the executable capability and the procedure loaded at runtime.
Choosing Agent Orchestration
Choose abstractions from state, retries and approval needs, then test recovery after a restart.
Evals
10 conceptsError Analysis First
Read a hundred real traces and name the failures before picking any eval tool or metric.
Binary Pass/Fail Judgments
One yes-or-no question per failure mode, with a written reason, beats any quality score.
Validating LLM Judges
A judge that always says pass scores 95% agreement at a 5% failure rate. Measure what it catches.
Golden Sets
What goes in an eval set, where the cases come from, who owns it and how it avoids going stale.
Confidence Intervals for Pass Rates
How wide the error bars on a pass rate are, how many cases you need, and how to compare two models.
Regression Gates
The checks every model, prompt, retriever or tool change passes before it reaches all users.
Evaluating RAG
Score retrieval and generation separately, then confirm with what users do after the answer.
Trajectory Evals
A correct final state can hide an unauthorised call, a double charge or nine wasted minutes.
The AI Testing Pyramid
Keep cheap contract checks, repeated behavior evaluations and production evidence in one plan.
Agent Benchmark Evidence
Read trial consistency, missing runs and grader changes before trusting an agent score.
AI Unit Economics
10 conceptsCost per Successful Task
Price per token is an input. What a finished job costs, failures included, is the product number.
Effective Cost With Human Fallback
C_eff = C_AI + (1 − R) × C_human. In most AI workflows the human term is the bill.
Inference Cost Levers
Seven levers in the order to pull them, with the evidence for each and the way each one fails.
Latency Metrics
Which speed number maps to which experience, and why agent features live or die on p95.
The AI Metric Tree
Five layers that connect a model's outputs to the business, with the silent failures in view.
Metric Definitions
How AI metrics widen without anyone deciding to, and how to audit a number before trusting it.
Experimenting With AI Features
Novelty, shared workspaces, model changes mid-test and quality you cannot see in clicks.
Loss Functions and Product Costs
Translate prediction errors into consequences before choosing a training objective or model.
Estimating AI Product Demand
Define the unit, build a range and cross-check demand before estimating revenue.
Diagnosing Metric Drops
Reconcile counts and traffic mix before blaming a model release for a falling rate.
Security & Governance
7 conceptsPrompt Injection
Direct and indirect injection, why a stronger system prompt is not a control, and what is.
The Confused Deputy
An attacker persuades a trusted agent to use its permissions for them. Least privilege stops it.
OWASP LLM and Agent Risks
Compare OWASP editions and turn relevant risks into product controls with observable tests.
EU AI Act for PMs
Roles, risk tiers and the phase-in calendar, with dates as of October 2026.
US and India AI Rules
Federal posture vs state laws in the US, and India's phased DPDP rules. What PMs build either way.
Data Retention and Residency
Training use, retention, app storage, logging and residency are different questions. Answer each.
Copyright and IP Risk
Four sources of IP risk, the controls for each, and why indemnity terms must line up.
Models & Platforms
9 conceptsThe Model Lifecycle
Data, training, evals, red team, registry, canary, serving, monitoring, retirement.
Quantization for PMs
Lower precision changes memory, serving cost and quality. Measure each on the deployment workload.
Serving and Speculative Decoding
Batching trades latency for throughput; speculative decoding helps most at low load.
Developer Platform PM
Time to first call, SDKs, rate limits, pricing tiers, deprecations, and docs as product.
Launching a Model
Positioning, safety gates, staged access, pricing, capacity and retiring the predecessor.
Self-Host vs API
GPU cost, utilisation and the operating team versus API prices, plus residency and control.
Synthetic Data and Distillation
Cheap training data from a larger model, with contract, quality and evaluation checks.
GPU Economics for PMs
Connect parallel arithmetic, memory and utilization to the cost of a successful product task.
Data Readiness
Check purpose, provenance, labels, population coverage and serving-time availability before scaling.
Building with AI
10 conceptsThe Live Prototyping Round
5 minutes scoping, 20 building, 10 evaluating, 5 narrating. Graded on judgment, not polish.
Prompt-Matrix Evals
Run 50 golden cases across prompts × models in a notebook; a pass-rate table beats an opinion.
AI Research Synthesis
Use models to tag and cluster at speed, with every theme traceable to verbatim source quotes.
NL to SQL Safely
Semantic layer, read-only scoped access, limits, and answers that show their work.
Synthetic Personas
Good for edge cases, interview prep and copy tests. Never evidence of real demand.
Prototype to Production
Idea, clickable prototype, real-data stub, validated workflow, 20 to 100 cases, then engineering.
BDD Specs for AI Features
Exact scenarios for the rules, scenario outlines with a pass threshold for the model's behaviour.
AI Coding Agents for PMs
What a coding agent controls, the five settings a PM should know, and when to use which.
Spec-Driven Development
Specify, plan, break into tasks, then build and verify. The spec is what everyone reviews.
Prototype, Pilot or Product?
Separate demand, model capability and operating evidence before turning a demo into a commitment.
Leading AI Teams
8 conceptsSTAR-L Stories
Explain your own decision, the measured result and the change in your working practice.
Kill Criteria
Write the stop condition before the work starts, then honour it when the date arrives.
Capability vs Outcome Commitments
Commit to outcomes with fallbacks, not to model capabilities that do not exist yet.
Culture Rounds at AI Labs
Candour, changed minds, safety trade-offs: what values interviews test and how to prepare honestly.
Working With Research Teams
Shared evals, offline-to-online calibration, shadow launches, and safe failure.
AI Roadmaps
Commitments vs bets, reliability milestones, and model swaps as a line item.
Decision Memos for AI Bets
Decision, evidence, threshold, cost, risk and reversal on one page, with the uncertainty stated.
Evaluating an AI PM Role
Check decision rights, research dependencies and operating work before joining an AI team.