PRODUCT ADVISORY COUNCIL
Compare capabilities and costs fairly
The method
Compare capabilities and costs fairly
Compare the same task, required quality, and cost definitions.
When to use it
- Choosing a model or assessing cost announcements.
Ask yourself
- A new model is cheaper. Should we switch?
Put it into action
- Separate scale, equivalent-capability savings, and training changes. Record inference, training, research costs, and unverifiable claims separately.
Keep these limits in mind
- Cost claims may not be independently verifiable. Training, inference, and total company research spending are not interchangeable.