Leaderboard
Externally produced LLM benchmark results on the published dataset releases — the platform hosts results, it does not run evaluations. Entries marked verified were evaluated by the platform team; peer-reviewed results come from a peer-reviewed publication (peer review checked the paper, not our reproduction of it); self-reported results are moderated before publication. Only results on datasets whose cases carry ICHD-3 reference diagnoses, scored against those diagnoses, are listed. Hover over a metric for its definition; select a model name for all of its results and the publications behind them.
- Prompting
- Retrieval-augmented
- Verified
- Unverified or as reported
Not shown (no size or no Top-1 (diagnosis) value): gpt-4.1 API, accessed 2025-11-10, Gradient boosting classifier Katsuki et al. 2023 model.
Size chart
About model sizes
Developers of proprietary models do not disclose parameter counts. The sizes are published third-party estimates; where none exists for a model, the estimate of a closely related model is used as an approximation. Read them as orders of magnitude.
| Model | Size | Basis | Source / estimation method |
|---|---|---|---|
| Gemma 3 12B IT google/gemma-3-12b-it | ~12B | Disclosed by the developer | Model card https://huggingface.co/google/gemma-3-12b-it |
| Ministral 3 14B Instruct mistralai/Ministral-3-14B-Instruct-2512 | ~14B | Disclosed by the developer | Model card https://huggingface.co/mistralai/Ministral-3-14B-Instruct-2512 |
| Qwen3-14B Qwen/Qwen3-14B | ~14.8B | Disclosed by the developer | Model card https://huggingface.co/Qwen/Qwen3-14B |
| Mistral Small 3.2 24B Instruct mistralai/Mistral-Small-3.2-24B-Instruct-2506 | ~24B | Disclosed by the developer | Model card https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506 |
| Gemma 3 27B IT google/gemma-3-27b-it | ~27B | Disclosed by the developer | Model card https://huggingface.co/google/gemma-3-27b-it |
| o3-mini 2025-04-16 | ~100B | Estimate of a related model | Estimate for o1-mini (~100B), no estimate exists for o3-mini: Abacha et al., MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes (Microsoft, 2024), model list with estimated sizes; the authors do not vouch for the figures. https://arxiv.org/abs/2412.19260 |
| Claude Sonnet 4.0 | ~175B | Estimate of a related model | Estimate for Claude 3.5 Sonnet (~175B), no estimate exists for Sonnet 4: Abacha et al., MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes (Microsoft, 2024), model list with estimated sizes; the authors do not vouch for the figures. https://arxiv.org/abs/2412.19260 For comparison, Li (2026) estimates Claude Sonnet 4.6 at ~766B. |
| gpt-3.5-turbo 2025-04-16 | ~175B | Third-party estimate | Listed as ChatGPT ~175B in Abacha et al., MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes (Microsoft, 2024), model list with estimated sizes; the authors do not vouch for the figures. https://arxiv.org/abs/2412.19260 Conflicting: a withdrawn Microsoft paper (CodeFusion, 2023) stated 20B. |
| Head.AI (GPT-4o + ICHD-3 knowledge base) GPT-4o, 2025 | ~200B | Third-party estimate | Size of the underlying GPT-4o, ~200B: Abacha et al., MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes (Microsoft, 2024), model list with estimated sizes; the authors do not vouch for the figures. https://arxiv.org/abs/2412.19260 |
| gpt-4-turbo 2025-04-16 | ~1.8T | Estimate of a related model | Estimate for GPT-4 (~1.76T), not GPT-4 Turbo: Abacha et al., MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes (Microsoft, 2024), model list with estimated sizes; the authors do not vouch for the figures. https://arxiv.org/abs/2412.19260 |
| Grok 3.0 | ~2.1T | Third-party estimate | Grok-3 ~2.1T [90% PI 642B–6.6T]: Li, Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity (2026), Table 13, combined curve; open-model-equivalent size, 90% prediction interval about ×/÷ 3.2. https://arxiv.org/abs/2604.24827 |
| gpt-4.1 2025-04-16 | ~2.2T | Third-party estimate | GPT-4.1 ~2.2T [90% PI 677B–6.9T]: Li, Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity (2026), Table 13, combined curve; open-model-equivalent size, 90% prediction interval about ×/÷ 3.2. https://arxiv.org/abs/2604.24827 |
| NotebookLM (Gemini 3.0 Pro) accessed 2026-02 | ~3T | Estimate of a related model | Estimate for the predecessor Gemini 2.5 Pro (~3.0T), used as an approximation for the Gemini 3.0 Pro behind NotebookLM: Li, Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity (2026), Table 13, combined curve; open-model-equivalent size, 90% prediction interval about ×/÷ 3.2. https://arxiv.org/abs/2604.24827 |
| Gemini 2.5 | ~3T | Estimate of a related model | Estimate for Gemini 2.5 Pro (~3.0T); the paper does not say whether Pro or Flash was used: Li, Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity (2026), Table 13, combined curve; open-model-equivalent size, 90% prediction interval about ×/÷ 3.2. https://arxiv.org/abs/2604.24827 |
Some results come from publications or team runs on external datasets rather than a platform release; they are shown as reported and are not directly comparable with results on a different dataset. Select a dataset name for its details; * marks a note on the result itself. Results marked Platform team were produced by the people who run this platform — disclosed because the same team curates the data and verifies results.
| Model | Approach | Dataset | Top-1 (diagnosis) Share of cases in which the model's first-ranked diagnosis matches the reference ICHD-3 diagnosis exactly (e.g. 1.1 Migraine without aura). | Top-3 (diagnosis) Share of cases in which the reference ICHD-3 diagnosis is among the model's three highest-ranked diagnoses. | Top-1 (group) Share of cases in which the model's first-ranked diagnosis falls into the same ICHD-3 diagnostic group as the reference (e.g. any 1.x for 1.1 Migraine without aura) — a coarser, more lenient measure. | Top-3 (group) Share of cases in which at least one of the model's three highest-ranked diagnoses falls into the reference ICHD-3 diagnostic group. | Cohen's Kappa Cohen's κ: agreement between the model's diagnoses and the expert reference, corrected for agreement expected by chance. 1 = perfect, 0 = chance level; the study targets κ ≥ 0.8. | Cost | Links | Submitter | Date |
|---|---|---|---|---|---|---|---|---|---|---|---|
|
o3-mini
2025-04-16 Verified Peer-reviewed publication ↗ Disclosure Produced by the platform teamThis result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out.
|
Criteria as JSON
Approach · Prompting Criteria as JSONThe case text, the candidate diagnoses and their ICHD-3 criteria as structured JSON in the prompt. Prompting approachesA general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.
|
External dataset · Zahn-50 Zahn et al. headache diagnosis dataset50 clinical headache case examples across 18 ICHD-3 diagnoses: 34 derived from real patients, 16 synthetic. The cases are not public. Accuracy values were read from the bar chart in Fig. 2 of the paper and are approximate. Results on this dataset are not directly comparable with results on a platform release or on another dataset.
|
50.87 % | 71.08 % | 75.96 % | — | — | — | Paper | — | 2026-10-22 |
|
o3-mini
2025-04-16 Verified Peer-reviewed publication ↗ Disclosure Produced by the platform teamThis result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out.
|
Naive prompt
Approach · Prompting Naive promptThe case text and the list of candidate diagnoses, without any diagnostic criteria. Prompting approachesA general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.
|
External dataset · Zahn-50 Zahn et al. headache diagnosis dataset50 clinical headache case examples across 18 ICHD-3 diagnoses: 34 derived from real patients, 16 synthetic. The cases are not public. Accuracy values were read from the bar chart in Fig. 2 of the paper and are approximate. Results on this dataset are not directly comparable with results on a platform release or on another dataset.
|
58.88 % | 75.96 % | 80.14 % | — | — | — | Paper | — | 2026-10-22 |
|
o3-mini
2025-04-16 Verified Peer-reviewed publication ↗ Disclosure Produced by the platform teamThis result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out.
|
Criteria as text
Approach · Prompting Criteria as textThe case text, the candidate diagnoses and their ICHD-3 criteria as plain text in the prompt. Prompting approachesA general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.
|
External dataset · Zahn-50 Zahn et al. headache diagnosis dataset50 clinical headache case examples across 18 ICHD-3 diagnoses: 34 derived from real patients, 16 synthetic. The cases are not public. Accuracy values were read from the bar chart in Fig. 2 of the paper and are approximate. Results on this dataset are not directly comparable with results on a platform release or on another dataset.
|
63.07 % | 75.96 % | 83.97 % | — | — | — | Paper | — | 2026-10-22 |
Download as CSV — current filters and sort applied.