How the benchmark works

DecisionBench evaluates Decision APIs: services that take a question and a closed set of options and return a probability for each option. Every provider gets the same cases, through the same endpoint, with the same settings.

Cases come from real-world work and are labeled by people or by verified real outcomes. There are no synthetic cases.

Two levels of categories

Every case is tagged on two independent levels, and results are reported for each:

• Modality: Vision, Text, Documents.
• Task domain: Finance, Legal, Health, Research, Development.

Metrics

Each provider's score is the share of correct decisions on a 0–100 scale, reported with a 95% bootstrap interval. Next to it we report calibration (expected calibration error and Brier score), consistency (how often a decision changes when the same case is asked again), cost per 1,000 decisions and median latency.

Closed by design

The construction of the benchmark stays private. We don't disclose the sources, the cases or the task definitions, so there is nothing for a provider to train against. The benchmark is not open to the public and cannot be run by third parties: only the DecisionBench team runs it, and only scores are published.

Independence

DecisionBench is not affiliated with any provider listed.