Model comparison

Model comparison answers a specific question: which candidate reaches the required quality on your data at the best cost and latency? It is not a generic provider benchmark.

Start from a trustworthy experiment

Create and validate a prompt experiment before opening Optimize Cost. The source experiment provides the AI Component, dataset cases, evaluators, prompt, and comparison baseline.

If the dataset or evaluator is weak, a model grid only produces a more precise comparison of the wrong thing.

Choose candidates

Select 2–8 model configurations. Thinking or reasoning levels are only applied when a model supports them; unsupported levels fall back to the model’s normal mode.

Keep the first comparison small. Include:

• the current production model;
• one plausible lower-cost model;
• one quality-oriented candidate;
• a reasoning configuration only when the task benefits from it.

What Promptic creates

The grid creates child experiments linked to the source experiment. Each child uses the same data and evaluators but targets a different model configuration. Children start independently, so partial enqueue failures can coexist with successfully started candidates.

Reading the comparison

Evaluate candidates in this order:

1. Exclude models that miss the quality threshold.
2. Inspect case-level regressions, not only the average.
3. Compare prediction latency on the held-out set.
4. Estimate production cost at a realistic monthly volume.
5. Check provider, region, and operational constraints.

The cheapest model is not a winner if it changes the product’s failure mode. Record the final quality floor and volume assumptions so the decision can be revisited.

Current API status

The current supported workflow starts from Optimize Cost in the dashboard. Model-grid metadata is visible on experiments, but programmatic creation should not be treated as available until the corresponding REST and SDK releases ship together.