34Our assistant now personalises answers using each user's history. How do you evaluate quality when there's no single right answer?▼hardNewSpotifyMeta◆ premiumCompare the same request with and without the profile, judged by people who see the profile, and test the three failures users notice: ignoring a stated preference, using a stale one and over-personalising.Practise this question →Read answer →
36Model A answers everything, right 24% of the time. Model B declines half and is wrong far less. Which do we ship?▼hardNewOpenAIGoogle DeepMindPerplexity◆ premiumAccuracy alone rewards guessing. Set a penalty for wrong answers from what errors cost in your product, score both models under it, and the choice usually flips.Practise this question →Read answer →