Leaderboard
Externally produced LLM benchmark results on the published dataset releases — the platform hosts results, it does not run evaluations. Entries marked verified were evaluated by the platform team; peer-reviewed results come from a peer-reviewed publication (peer review checked the paper, not our reproduction of it); self-reported results are moderated before publication. Only results on datasets whose cases carry ICHD-3 reference diagnoses, scored against those diagnoses, are listed. Hover over a metric for its definition; select a model name for all of its results and the publications behind them.
- Prompting
- Retrieval-augmented
- Verified
- Unverified or as reported
Not shown (no size or no Top-1 (diagnosis) value): gpt-4.1 API, accessed 2025-11-10, Gradient boosting classifier Katsuki et al. 2023 model.
Size chart
About model sizes
Developers of proprietary models do not disclose parameter counts. The sizes are published third-party estimates; where none exists for a model, the estimate of a closely related model is used as an approximation. Read them as orders of magnitude.
| Model | Size | Basis | Source / estimation method |
|---|---|---|---|
| Gemma 3 12B IT google/gemma-3-12b-it | ~12B | Disclosed by the developer | Model card https://huggingface.co/google/gemma-3-12b-it |
| Ministral 3 14B Instruct mistralai/Ministral-3-14B-Instruct-2512 | ~14B | Disclosed by the developer | Model card https://huggingface.co/mistralai/Ministral-3-14B-Instruct-2512 |
| Qwen3-14B Qwen/Qwen3-14B | ~14.8B | Disclosed by the developer | Model card https://huggingface.co/Qwen/Qwen3-14B |
| Mistral Small 3.2 24B Instruct mistralai/Mistral-Small-3.2-24B-Instruct-2506 | ~24B | Disclosed by the developer | Model card https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506 |
| Gemma 3 27B IT google/gemma-3-27b-it | ~27B | Disclosed by the developer | Model card https://huggingface.co/google/gemma-3-27b-it |
| o3-mini 2025-04-16 | ~100B | Estimate of a related model | Estimate for o1-mini (~100B), no estimate exists for o3-mini: Abacha et al., MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes (Microsoft, 2024), model list with estimated sizes; the authors do not vouch for the figures. https://arxiv.org/abs/2412.19260 |
| Claude Sonnet 4.0 | ~175B | Estimate of a related model | Estimate for Claude 3.5 Sonnet (~175B), no estimate exists for Sonnet 4: Abacha et al., MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes (Microsoft, 2024), model list with estimated sizes; the authors do not vouch for the figures. https://arxiv.org/abs/2412.19260 For comparison, Li (2026) estimates Claude Sonnet 4.6 at ~766B. |
| gpt-3.5-turbo 2025-04-16 | ~175B | Third-party estimate | Listed as ChatGPT ~175B in Abacha et al., MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes (Microsoft, 2024), model list with estimated sizes; the authors do not vouch for the figures. https://arxiv.org/abs/2412.19260 Conflicting: a withdrawn Microsoft paper (CodeFusion, 2023) stated 20B. |
| Head.AI (GPT-4o + ICHD-3 knowledge base) GPT-4o, 2025 | ~200B | Third-party estimate | Size of the underlying GPT-4o, ~200B: Abacha et al., MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes (Microsoft, 2024), model list with estimated sizes; the authors do not vouch for the figures. https://arxiv.org/abs/2412.19260 |
| gpt-4-turbo 2025-04-16 | ~1.8T | Estimate of a related model | Estimate for GPT-4 (~1.76T), not GPT-4 Turbo: Abacha et al., MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes (Microsoft, 2024), model list with estimated sizes; the authors do not vouch for the figures. https://arxiv.org/abs/2412.19260 |
| Grok 3.0 | ~2.1T | Third-party estimate | Grok-3 ~2.1T [90% PI 642B–6.6T]: Li, Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity (2026), Table 13, combined curve; open-model-equivalent size, 90% prediction interval about ×/÷ 3.2. https://arxiv.org/abs/2604.24827 |
| gpt-4.1 2025-04-16 | ~2.2T | Third-party estimate | GPT-4.1 ~2.2T [90% PI 677B–6.9T]: Li, Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity (2026), Table 13, combined curve; open-model-equivalent size, 90% prediction interval about ×/÷ 3.2. https://arxiv.org/abs/2604.24827 |
| NotebookLM (Gemini 3.0 Pro) accessed 2026-02 | ~3T | Estimate of a related model | Estimate for the predecessor Gemini 2.5 Pro (~3.0T), used as an approximation for the Gemini 3.0 Pro behind NotebookLM: Li, Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity (2026), Table 13, combined curve; open-model-equivalent size, 90% prediction interval about ×/÷ 3.2. https://arxiv.org/abs/2604.24827 |
| Gemini 2.5 | ~3T | Estimate of a related model | Estimate for Gemini 2.5 Pro (~3.0T); the paper does not say whether Pro or Flash was used: Li, Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity (2026), Table 13, combined curve; open-model-equivalent size, 90% prediction interval about ×/÷ 3.2. https://arxiv.org/abs/2604.24827 |
Some results come from publications or team runs on external datasets rather than a platform release; they are shown as reported and are not directly comparable with results on a different dataset. Select a dataset name for its details; * marks a note on the result itself.
| Model | Approach | Dataset | Top-1 (diagnosis) Share of cases in which the model's first-ranked diagnosis matches the reference ICHD-3 diagnosis exactly (e.g. 1.1 Migraine without aura). | Top-3 (diagnosis) Share of cases in which the reference ICHD-3 diagnosis is among the model's three highest-ranked diagnoses. | Top-1 (group) Share of cases in which the model's first-ranked diagnosis falls into the same ICHD-3 diagnostic group as the reference (e.g. any 1.x for 1.1 Migraine without aura) — a coarser, more lenient measure. | Top-3 (group) Share of cases in which at least one of the model's three highest-ranked diagnoses falls into the reference ICHD-3 diagnostic group. | Cohen's Kappa Cohen's κ: agreement between the model's diagnoses and the expert reference, corrected for agreement expected by chance. 1 = perfect, 0 = chance level; the study targets κ ≥ 0.8. | Cost | Links | Submitter | Date |
|---|---|---|---|---|---|---|---|---|---|---|---|
|
NotebookLM (Gemini 3.0 Pro)
accessed 2026-02 Peer-reviewed publication ↗ |
NotebookLM over ICHD-3 + guidelines
Approach · Retrieval-augmented NotebookLM over ICHD-3 + guidelinesNotebookLM indexed with the full ICHD-3 and major headache treatment guidelines (PDF); each vignette in a fresh session, answers grounded in the uploaded sources. Retrieval-augmented approachesFor each case, relevant ICHD-3 knowledge is retrieved from a text collection and added to the prompt; the language model decides.
|
External dataset · Babaoglu-35 Babaoğlu headache case vignettes35 fictional case vignettes written for the study: 17 primary headaches, 15 secondary headaches, 3 neuropathic/facial pain syndromes; gold-standard diagnoses by two pain specialists. Very small set — read the 100 % as an upper bound, not a stable estimate. Results on this dataset are not directly comparable with results on a platform release or on another dataset. About this resultSubtype-level top-1 35/35 (95 % CI 90.0–100.0); secondary vs non-secondary 35/35.
|
100.00 % | — | — | — | 1.000 | — | Paper | — | 2026-10-22 |
Download as CSV — current filters and sort applied.