PRODUCT ADVISORY COUNCIL
Separate capability from presentation
The method
Separate capability from presentation
Task correctness and perceived interaction quality need separate tests.
When to use it
- AI interaction design or competitor evaluation.
Ask yourself
- Does a more logical-looking answer mean a better model?
Put it into action
- Measure success, usefulness of explanations, and user preference. Longer visible reasoning is not proof of stronger capability.
Keep these limits in mind
- Displayed reasoning is not necessarily a faithful record of internal mechanisms.