Andrew NgAnalytical perspectiveAN-P02PRODUCT ADVISORY COUNCIL
Inspect outputs before designing evaluation
Andrew NgAnalytical perspectiveAN-P02The method
Inspect outputs before designing evaluation
A small prototype can reveal which failures matter to users.
When to use it
- Generative quality cannot be captured by one metric.
Ask yourself
- Should we build a huge evaluation set first?
Put it into action
- Inspect early outputs, classify important errors, and collect examples. Separate objective measures from subjective judgment and update the set as failures emerge.
Keep these limits in mind
- Small samples are a starting point, not proof of release reliability; model-based grading also needs calibration.