Leaderboard
Externally produced LLM benchmark results on the published dataset releases — the platform hosts results, it does not run evaluations. Entries marked verified were evaluated by the platform team; peer-reviewed results come from a peer-reviewed publication (peer review checked the paper, not our reproduction of it); self-reported results are moderated before publication. Only results on datasets whose cases carry ICHD-3 reference diagnoses, scored against those diagnoses, are listed. Hover over a metric for its definition; select a model name for all of its results and the publications behind them.
- Prompting
- Retrieval-augmented
- Verified
- Unverified or as reported
Not shown (no size or no Top-1 (diagnosis) value): gpt-4.1 API, accessed 2025-11-10, Gradient boosting classifier Katsuki et al. 2023 model.
Size chart
About model sizes
Developers of proprietary models do not disclose parameter counts. The sizes are published third-party estimates; where none exists for a model, the estimate of a closely related model is used as an approximation. Read them as orders of magnitude.
| Model | Size | Basis | Source / estimation method |
|---|---|---|---|
| Gemma 3 12B IT google/gemma-3-12b-it | ~12B | Disclosed by the developer | Model card https://huggingface.co/google/gemma-3-12b-it |
| Ministral 3 14B Instruct mistralai/Ministral-3-14B-Instruct-2512 | ~14B | Disclosed by the developer | Model card https://huggingface.co/mistralai/Ministral-3-14B-Instruct-2512 |
| Qwen3-14B Qwen/Qwen3-14B | ~14.8B | Disclosed by the developer | Model card https://huggingface.co/Qwen/Qwen3-14B |
| Mistral Small 3.2 24B Instruct mistralai/Mistral-Small-3.2-24B-Instruct-2506 | ~24B | Disclosed by the developer | Model card https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506 |
| Gemma 3 27B IT google/gemma-3-27b-it | ~27B | Disclosed by the developer | Model card https://huggingface.co/google/gemma-3-27b-it |
| o3-mini 2025-04-16 | ~100B | Estimate of a related model | Estimate for o1-mini (~100B), no estimate exists for o3-mini: Abacha et al., MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes (Microsoft, 2024), model list with estimated sizes; the authors do not vouch for the figures. https://arxiv.org/abs/2412.19260 |
| Claude Sonnet 4.0 | ~175B | Estimate of a related model | Estimate for Claude 3.5 Sonnet (~175B), no estimate exists for Sonnet 4: Abacha et al., MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes (Microsoft, 2024), model list with estimated sizes; the authors do not vouch for the figures. https://arxiv.org/abs/2412.19260 For comparison, Li (2026) estimates Claude Sonnet 4.6 at ~766B. |
| gpt-3.5-turbo 2025-04-16 | ~175B | Third-party estimate | Listed as ChatGPT ~175B in Abacha et al., MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes (Microsoft, 2024), model list with estimated sizes; the authors do not vouch for the figures. https://arxiv.org/abs/2412.19260 Conflicting: a withdrawn Microsoft paper (CodeFusion, 2023) stated 20B. |
| Head.AI (GPT-4o + ICHD-3 knowledge base) GPT-4o, 2025 | ~200B | Third-party estimate | Size of the underlying GPT-4o, ~200B: Abacha et al., MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes (Microsoft, 2024), model list with estimated sizes; the authors do not vouch for the figures. https://arxiv.org/abs/2412.19260 |
| gpt-4-turbo 2025-04-16 | ~1.8T | Estimate of a related model | Estimate for GPT-4 (~1.76T), not GPT-4 Turbo: Abacha et al., MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes (Microsoft, 2024), model list with estimated sizes; the authors do not vouch for the figures. https://arxiv.org/abs/2412.19260 |
| Grok 3.0 | ~2.1T | Third-party estimate | Grok-3 ~2.1T [90% PI 642B–6.6T]: Li, Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity (2026), Table 13, combined curve; open-model-equivalent size, 90% prediction interval about ×/÷ 3.2. https://arxiv.org/abs/2604.24827 |
| gpt-4.1 2025-04-16 | ~2.2T | Third-party estimate | GPT-4.1 ~2.2T [90% PI 677B–6.9T]: Li, Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity (2026), Table 13, combined curve; open-model-equivalent size, 90% prediction interval about ×/÷ 3.2. https://arxiv.org/abs/2604.24827 |
| NotebookLM (Gemini 3.0 Pro) accessed 2026-02 | ~3T | Estimate of a related model | Estimate for the predecessor Gemini 2.5 Pro (~3.0T), used as an approximation for the Gemini 3.0 Pro behind NotebookLM: Li, Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity (2026), Table 13, combined curve; open-model-equivalent size, 90% prediction interval about ×/÷ 3.2. https://arxiv.org/abs/2604.24827 |
| Gemini 2.5 | ~3T | Estimate of a related model | Estimate for Gemini 2.5 Pro (~3.0T); the paper does not say whether Pro or Flash was used: Li, Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity (2026), Table 13, combined curve; open-model-equivalent size, 90% prediction interval about ×/÷ 3.2. https://arxiv.org/abs/2604.24827 |
Some results come from publications or team runs on external datasets rather than a platform release; they are shown as reported and are not directly comparable with results on a different dataset. Select a dataset name for its details; * marks a note on the result itself.
| Model | Approach | Dataset | Top-1 (diagnosis) Share of cases in which the model's first-ranked diagnosis matches the reference ICHD-3 diagnosis exactly (e.g. 1.1 Migraine without aura). | Top-3 (diagnosis) Share of cases in which the reference ICHD-3 diagnosis is among the model's three highest-ranked diagnoses. | Top-1 (group) Share of cases in which the model's first-ranked diagnosis falls into the same ICHD-3 diagnostic group as the reference (e.g. any 1.x for 1.1 Migraine without aura) — a coarser, more lenient measure. | Top-3 (group) Share of cases in which at least one of the model's three highest-ranked diagnoses falls into the reference ICHD-3 diagnostic group. | Cohen's Kappa Cohen's κ: agreement between the model's diagnoses and the expert reference, corrected for agreement expected by chance. 1 = perfect, 0 = chance level; the study targets κ ≥ 0.8. | Cost | Links | Submitter | Date |
|---|---|---|---|---|---|---|---|---|---|---|---|
|
Claude Sonnet
4.0 Peer-reviewed publication ↗ |
Chatbot default
Approach · Prompting Chatbot defaultThe public chatbot with its default settings, given the vignette as written. Prompting approachesA general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.
|
External dataset · HeadAI-pt-315 Head.AI validation vignettes (original, Brazilian Portuguese)315 expert-written clinical headache vignettes from the Head.AI validation study. Results as reported in the paper: correct diagnosis as the first hypothesis. Top-3 is omitted because the rank counts in the paper do not sum to n. Results on this dataset are not directly comparable with results on a platform release or on another dataset.
|
74.30 % | — | — | — | — | — | Paper | — | 2026-10-22 |
Download as CSV — current filters and sort applied.