o3-mini
All 3 published results of this model on the leaderboard — every version, method and dataset — with the publications they come from. Results on different datasets are not directly comparable.
- Version
- 2025-04-16
- Approach
- Prompting
- Dataset
- Size
- 2025-04-16: ~100B · Estimate of a related model (see “About model sizes” on the leaderboard)
Publications
- https://doi.org/10.3233/SHTI251535 — 3 results on Zahn-50
Results
Results marked Platform team were produced by the people who run this platform — disclosed because the same team curates the data and verifies results.
| Version · badges | Approach | Dataset | Top-1 (diagnosis) | Top-3 (diagnosis) | Top-1 (group) | Top-3 (group) | Cohen's Kappa | Links |
|---|---|---|---|---|---|---|---|---|
| 2025-04-16 Verified Peer-reviewed publication ↗ Disclosure Produced by the platform teamThis result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out.
|
Criteria as text
Approach · Prompting Criteria as textThe case text, the candidate diagnoses and their ICHD-3 criteria as plain text in the prompt. Prompting approachesA general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.
|
External dataset · Zahn-50 Zahn et al. headache diagnosis dataset50 clinical headache case examples across 18 ICHD-3 diagnoses: 34 derived from real patients, 16 synthetic. The cases are not public. Accuracy values were read from the bar chart in Fig. 2 of the paper and are approximate. Results on this dataset are not directly comparable with results on a platform release or on another dataset.
|
63.07 % | 75.96 % | 83.97 % | — | — | Paper |
| 2025-04-16 Verified Peer-reviewed publication ↗ Disclosure Produced by the platform teamThis result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out.
|
Naive prompt
Approach · Prompting Naive promptThe case text and the list of candidate diagnoses, without any diagnostic criteria. Prompting approachesA general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.
|
External dataset · Zahn-50 Zahn et al. headache diagnosis dataset50 clinical headache case examples across 18 ICHD-3 diagnoses: 34 derived from real patients, 16 synthetic. The cases are not public. Accuracy values were read from the bar chart in Fig. 2 of the paper and are approximate. Results on this dataset are not directly comparable with results on a platform release or on another dataset.
|
58.88 % | 75.96 % | 80.14 % | — | — | Paper |
| 2025-04-16 Verified Peer-reviewed publication ↗ Disclosure Produced by the platform teamThis result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out.
|
Criteria as JSON
Approach · Prompting Criteria as JSONThe case text, the candidate diagnoses and their ICHD-3 criteria as structured JSON in the prompt. Prompting approachesA general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.
|
External dataset · Zahn-50 Zahn et al. headache diagnosis dataset50 clinical headache case examples across 18 ICHD-3 diagnoses: 34 derived from real patients, 16 synthetic. The cases are not public. Accuracy values were read from the bar chart in Fig. 2 of the paper and are approximate. Results on this dataset are not directly comparable with results on a platform release or on another dataset.
|
50.87 % | 71.08 % | 75.96 % | — | — | Paper |