Leaderboard
Externally produced LLM benchmark results on the published dataset releases — the platform hosts results, it does not run evaluations. Entries marked verified were evaluated by the platform team; peer-reviewed results come from a peer-reviewed publication (peer review checked the paper, not our reproduction of it); self-reported results are moderated before publication. Only results on datasets whose cases carry ICHD-3 reference diagnoses, scored against those diagnoses, are listed. Hover over a metric for its definition; select a model name for all of its results and the publications behind them.
- Prompting
- Retrieval-augmented
- Verified
- Unverified or as reported
Not shown (no size or no Top-1 (diagnosis) value): gpt-4.1 API, accessed 2025-11-10, Gradient boosting classifier Katsuki et al. 2023 model.
Size chart
About model sizes
Developers of proprietary models do not disclose parameter counts. The sizes are published third-party estimates; where none exists for a model, the estimate of a closely related model is used as an approximation. Read them as orders of magnitude.
| Model | Size | Basis | Source / estimation method |
|---|---|---|---|
| Gemma 3 12B IT google/gemma-3-12b-it | ~12B | Disclosed by the developer | Model card https://huggingface.co/google/gemma-3-12b-it |
| Ministral 3 14B Instruct mistralai/Ministral-3-14B-Instruct-2512 | ~14B | Disclosed by the developer | Model card https://huggingface.co/mistralai/Ministral-3-14B-Instruct-2512 |
| Qwen3-14B Qwen/Qwen3-14B | ~14.8B | Disclosed by the developer | Model card https://huggingface.co/Qwen/Qwen3-14B |
| Mistral Small 3.2 24B Instruct mistralai/Mistral-Small-3.2-24B-Instruct-2506 | ~24B | Disclosed by the developer | Model card https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506 |
| Gemma 3 27B IT google/gemma-3-27b-it | ~27B | Disclosed by the developer | Model card https://huggingface.co/google/gemma-3-27b-it |
| o3-mini 2025-04-16 | ~100B | Estimate of a related model | Estimate for o1-mini (~100B), no estimate exists for o3-mini: Abacha et al., MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes (Microsoft, 2024), model list with estimated sizes; the authors do not vouch for the figures. https://arxiv.org/abs/2412.19260 |
| Claude Sonnet 4.0 | ~175B | Estimate of a related model | Estimate for Claude 3.5 Sonnet (~175B), no estimate exists for Sonnet 4: Abacha et al., MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes (Microsoft, 2024), model list with estimated sizes; the authors do not vouch for the figures. https://arxiv.org/abs/2412.19260 For comparison, Li (2026) estimates Claude Sonnet 4.6 at ~766B. |
| gpt-3.5-turbo 2025-04-16 | ~175B | Third-party estimate | Listed as ChatGPT ~175B in Abacha et al., MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes (Microsoft, 2024), model list with estimated sizes; the authors do not vouch for the figures. https://arxiv.org/abs/2412.19260 Conflicting: a withdrawn Microsoft paper (CodeFusion, 2023) stated 20B. |
| Head.AI (GPT-4o + ICHD-3 knowledge base) GPT-4o, 2025 | ~200B | Third-party estimate | Size of the underlying GPT-4o, ~200B: Abacha et al., MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes (Microsoft, 2024), model list with estimated sizes; the authors do not vouch for the figures. https://arxiv.org/abs/2412.19260 |
| gpt-4-turbo 2025-04-16 | ~1.8T | Estimate of a related model | Estimate for GPT-4 (~1.76T), not GPT-4 Turbo: Abacha et al., MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes (Microsoft, 2024), model list with estimated sizes; the authors do not vouch for the figures. https://arxiv.org/abs/2412.19260 |
| Grok 3.0 | ~2.1T | Third-party estimate | Grok-3 ~2.1T [90% PI 642B–6.6T]: Li, Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity (2026), Table 13, combined curve; open-model-equivalent size, 90% prediction interval about ×/÷ 3.2. https://arxiv.org/abs/2604.24827 |
| gpt-4.1 2025-04-16 | ~2.2T | Third-party estimate | GPT-4.1 ~2.2T [90% PI 677B–6.9T]: Li, Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity (2026), Table 13, combined curve; open-model-equivalent size, 90% prediction interval about ×/÷ 3.2. https://arxiv.org/abs/2604.24827 |
| NotebookLM (Gemini 3.0 Pro) accessed 2026-02 | ~3T | Estimate of a related model | Estimate for the predecessor Gemini 2.5 Pro (~3.0T), used as an approximation for the Gemini 3.0 Pro behind NotebookLM: Li, Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity (2026), Table 13, combined curve; open-model-equivalent size, 90% prediction interval about ×/÷ 3.2. https://arxiv.org/abs/2604.24827 |
| Gemini 2.5 | ~3T | Estimate of a related model | Estimate for Gemini 2.5 Pro (~3.0T); the paper does not say whether Pro or Flash was used: Li, Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity (2026), Table 13, combined curve; open-model-equivalent size, 90% prediction interval about ×/÷ 3.2. https://arxiv.org/abs/2604.24827 |
Some results come from publications or team runs on external datasets rather than a platform release; they are shown as reported and are not directly comparable with results on a different dataset. Select a dataset name for its details; * marks a note on the result itself. Results marked Platform team were produced by the people who run this platform — disclosed because the same team curates the data and verifies results.
| Model | Approach | Dataset | Top-1 (diagnosis) Share of cases in which the model's first-ranked diagnosis matches the reference ICHD-3 diagnosis exactly (e.g. 1.1 Migraine without aura). | Top-3 (diagnosis) Share of cases in which the reference ICHD-3 diagnosis is among the model's three highest-ranked diagnoses. | Top-1 (group) Share of cases in which the model's first-ranked diagnosis falls into the same ICHD-3 diagnostic group as the reference (e.g. any 1.x for 1.1 Migraine without aura) — a coarser, more lenient measure. | Top-3 (group) Share of cases in which at least one of the model's three highest-ranked diagnoses falls into the reference ICHD-3 diagnostic group. | Cohen's Kappa Cohen's κ: agreement between the model's diagnoses and the expert reference, corrected for agreement expected by chance. 1 = perfect, 0 = chance level; the study targets κ ≥ 0.8. | Cost | Links | Submitter | Date |
|---|---|---|---|---|---|---|---|---|---|---|---|
|
Ministral 3 14B Instruct
mistralai/Ministral-3-14B-Instruct-2512 Verified Peer-reviewed publication ↗ Disclosure Produced by the platform teamThis result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out.
|
RAG (ICHD-3 retrieval)
Approach · Retrieval-augmented RAG (ICHD-3 retrieval)For each case, the most similar ICHD-3 diagnosis entries are retrieved by embedding similarity and added to the prompt; the model decides. Retrieval-augmented approachesFor each case, relevant ICHD-3 knowledge is retrieved from a text collection and added to the prompt; the language model decides.
|
Platform release Release 1.0A published, checksummed release of this platform's community cases, validated by two independent reviewers. About this resultEnglish variants (n=305) of release 1.0; vLLM, temperature 0 (Qwen3: 0.6), runs March 2026. Top-3 from a separate top-3 run; group = ICHD-3 top-level group of the first diagnosis (computed from the paper's run data).
|
61.64 % | 69.20 % | 90.49 % | — | — | — | Paper | — | 2026-10-22 |
|
Ministral 3 14B Instruct
mistralai/Ministral-3-14B-Instruct-2512 Verified Peer-reviewed publication ↗ Disclosure Produced by the platform teamThis result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out.
|
Chain-of-thought
Approach · Prompting Chain-of-thoughtCase text and candidate list; the model reasons step by step before naming the diagnosis. Prompting approachesA general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.
|
Platform release Release 1.0A published, checksummed release of this platform's community cases, validated by two independent reviewers. About this resultEnglish variants (n=305) of release 1.0; vLLM, temperature 0 (Qwen3: 0.6), runs March 2026. Top-3 from a separate top-3 run; group = ICHD-3 top-level group of the first diagnosis (computed from the paper's run data).
|
67.54 % | 72.80 % | 92.79 % | — | — | — | Paper | — | 2026-10-22 |
|
Ministral 3 14B Instruct
mistralai/Ministral-3-14B-Instruct-2512 Verified Peer-reviewed publication ↗ Disclosure Produced by the platform teamThis result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out.
|
ICHD-3 criteria in context
Approach · Prompting ICHD-3 criteria in contextThe full ICHD-3 ontology (simplified criteria, descriptions, clinical features) is placed in the system prompt. Prompting approachesA general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.
|
Platform release Release 1.0A published, checksummed release of this platform's community cases, validated by two independent reviewers. About this resultEnglish variants (n=305) of release 1.0; vLLM, temperature 0 (Qwen3: 0.6), runs March 2026. Top-3 from a separate top-3 run; group = ICHD-3 top-level group of the first diagnosis (computed from the paper's run data).
|
67.87 % | 79.30 % | 92.46 % | — | — | — | Paper | — | 2026-10-22 |
|
Ministral 3 14B Instruct
mistralai/Ministral-3-14B-Instruct-2512 Verified Peer-reviewed publication ↗ Disclosure Produced by the platform teamThis result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out.
|
Zero-shot
Approach · Prompting Zero-shotCase text plus the list of candidate ICHD-3 codes; single best diagnosis, no examples. Prompting approachesA general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.
|
Platform release Release 1.0A published, checksummed release of this platform's community cases, validated by two independent reviewers. About this resultEnglish variants (n=305) of release 1.0; vLLM, temperature 0 (Qwen3: 0.6), runs March 2026. Top-3 from a separate top-3 run; group = ICHD-3 top-level group of the first diagnosis (computed from the paper's run data).
|
67.87 % | 78.30 % | 92.13 % | — | — | — | Paper | — | 2026-10-22 |
|
Ministral 3 14B Instruct
mistralai/Ministral-3-14B-Instruct-2512 Verified Peer-reviewed publication ↗ Disclosure Produced by the platform teamThis result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out.
|
Few-shot
Approach · Prompting Few-shotCase text, candidate list and worked example cases in the prompt. Prompting approachesA general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.
|
Platform release Release 1.0A published, checksummed release of this platform's community cases, validated by two independent reviewers. About this resultEnglish variants (n=305) of release 1.0; vLLM, temperature 0 (Qwen3: 0.6), runs March 2026. Top-3 from a separate top-3 run; group = ICHD-3 top-level group of the first diagnosis (computed from the paper's run data).
|
69.18 % | 77.20 % | 93.44 % | — | — | — | Paper | — | 2026-10-22 |
Download as CSV — current filters and sort applied.