34Our assistant now personalises answers using each user's history. How do you evaluate quality when there's no single right answer?▼hardNewSpotifyMeta◆ premiumCompare the same request with and without the profile, judged by people who see the profile, and test the three failures users notice: ignoring a stated preference, using a stale one and over-personalising.Practise this question →Read answer →
23A new model wins 64% of preference tests but expert users prefer the old one. Would you replace it?▼expertNewOpenAIAnthropicGoogle DeepMind◆ premiumA global preference win can conceal a regression for the people doing the hardest work. Reweight the evidence, inspect the behavior change and choose a release policy.Practise this question →Read answer →