A method worth keeping close.

Ethan MollickAnalytical perspectiveETM-P03

PRODUCT ADVISORY COUNCIL

Interview models with real work

Choose with repeated examples of the work the model will actually perform.

Ethan MollickAnalytical perspectiveETM-P03

The method

Interview models with real work

Choose with repeated examples of the work the model will actually perform.

When to use it

• Organizational model selection or upgrades.

Ask yourself

• Is a leaderboard enough to choose a model?

Put it into action

• Use real job tasks and repeated runs, comparing models with an appropriate human baseline.
• Reassess periodically and record quality, risk preferences and obvious failures.
• Where feasible, hide model identity and have domain reviewers score against preset criteria to reduce brand and taste bias.

Keep these limits in mind

• Fun demonstrations do not prove reliability at scale. Check the evaluation itself for bias.

An analytical method adapted from the public skill package, not a reply or endorsement by the person represented.
View the public method source ↗