Leaderboard

Externally produced LLM benchmark results on the published dataset releases — the platform hosts results, it does not run evaluations. Entries marked verified were evaluated by the platform team; peer-reviewed results come from a peer-reviewed publication (peer review checked the paper, not our reproduction of it); self-reported results are moderated before publication. Only results on datasets whose cases carry ICHD-3 reference diagnoses, scored against those diagnoses, are listed. Hover over a metric for its definition; select a model name for all of its results and the publications behind them.

Submit a result Request a benchmark

Top-1 (diagnosis) by model size

The chart has its own filters; the table's filters do not apply to it.

Metric

Approaches Prompting Retrieval-augmented Other

Datasets (none ticked = all) Release 1.0 Babaoglu-35 HeadAI-pt-315 Katsuki-1818 Zahn-50

Badges (untick to hide results carrying it) Verified Unverified Peer-reviewed Publication Platform team

Zoom

Enter a window, or drag a rectangle across the chart.

Size from (B) to (B) Top-1 (diagnosis) from to

Prompting
Retrieval-augmented
Verified
Unverified or as reported

Not shown (no size or no Top-1 (diagnosis) value): gpt-4.1 API, accessed 2025-11-10, Gradient boosting classifier Katsuki et al. 2023 model.

Size chart

About model sizes

Developers of proprietary models do not disclose parameter counts. The sizes are published third-party estimates; where none exists for a model, the estimate of a closely related model is used as an approximation. Read them as orders of magnitude.

Model	Size	Basis	Source / estimation method
Gemma 3 12B IT google/gemma-3-12b-it	~12B	Disclosed by the developer	Model card https://huggingface.co/google/gemma-3-12b-it
Ministral 3 14B Instruct mistralai/Ministral-3-14B-Instruct-2512	~14B	Disclosed by the developer	Model card https://huggingface.co/mistralai/Ministral-3-14B-Instruct-2512
Qwen3-14B Qwen/Qwen3-14B	~14.8B	Disclosed by the developer	Model card https://huggingface.co/Qwen/Qwen3-14B
Mistral Small 3.2 24B Instruct mistralai/Mistral-Small-3.2-24B-Instruct-2506	~24B	Disclosed by the developer	Model card https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506
Gemma 3 27B IT google/gemma-3-27b-it	~27B	Disclosed by the developer	Model card https://huggingface.co/google/gemma-3-27b-it
o3-mini 2025-04-16	~100B	Estimate of a related model	Estimate for o1-mini (~100B), no estimate exists for o3-mini: Abacha et al., MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes (Microsoft, 2024), model list with estimated sizes; the authors do not vouch for the figures. https://arxiv.org/abs/2412.19260
Claude Sonnet 4.0	~175B	Estimate of a related model	Estimate for Claude 3.5 Sonnet (~175B), no estimate exists for Sonnet 4: Abacha et al., MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes (Microsoft, 2024), model list with estimated sizes; the authors do not vouch for the figures. https://arxiv.org/abs/2412.19260 For comparison, Li (2026) estimates Claude Sonnet 4.6 at ~766B.
gpt-3.5-turbo 2025-04-16	~175B	Third-party estimate	Listed as ChatGPT ~175B in Abacha et al., MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes (Microsoft, 2024), model list with estimated sizes; the authors do not vouch for the figures. https://arxiv.org/abs/2412.19260 Conflicting: a withdrawn Microsoft paper (CodeFusion, 2023) stated 20B.
Head.AI (GPT-4o + ICHD-3 knowledge base) GPT-4o, 2025	~200B	Third-party estimate	Size of the underlying GPT-4o, ~200B: Abacha et al., MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes (Microsoft, 2024), model list with estimated sizes; the authors do not vouch for the figures. https://arxiv.org/abs/2412.19260
gpt-4-turbo 2025-04-16	~1.8T	Estimate of a related model	Estimate for GPT-4 (~1.76T), not GPT-4 Turbo: Abacha et al., MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes (Microsoft, 2024), model list with estimated sizes; the authors do not vouch for the figures. https://arxiv.org/abs/2412.19260
Grok 3.0	~2.1T	Third-party estimate	Grok-3 ~2.1T [90% PI 642B–6.6T]: Li, Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity (2026), Table 13, combined curve; open-model-equivalent size, 90% prediction interval about ×/÷ 3.2. https://arxiv.org/abs/2604.24827
gpt-4.1 2025-04-16	~2.2T	Third-party estimate	GPT-4.1 ~2.2T [90% PI 677B–6.9T]: Li, Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity (2026), Table 13, combined curve; open-model-equivalent size, 90% prediction interval about ×/÷ 3.2. https://arxiv.org/abs/2604.24827
NotebookLM (Gemini 3.0 Pro) accessed 2026-02	~3T	Estimate of a related model	Estimate for the predecessor Gemini 2.5 Pro (~3.0T), used as an approximation for the Gemini 3.0 Pro behind NotebookLM: Li, Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity (2026), Table 13, combined curve; open-model-equivalent size, 90% prediction interval about ×/÷ 3.2. https://arxiv.org/abs/2604.24827
Gemini 2.5	~3T	Estimate of a related model	Estimate for Gemini 2.5 Pro (~3.0T); the paper does not say whether Pro or Flash was used: Li, Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity (2026), Table 13, combined curve; open-model-equivalent size, 90% prediction interval about ×/÷ 3.2. https://arxiv.org/abs/2604.24827

Some results come from publications or team runs on external datasets rather than a platform release; they are shown as reported and are not directly comparable with results on a different dataset. Select a dataset name for its details; * marks a note on the result itself. Results marked Platform team were produced by the people who run this platform — disclosed because the same team curates the data and verifies results.

Model	Approach	Dataset	Top-1 (diagnosis)	Top-3 (diagnosis)	Top-1 (group)	Top-3 (group)	Cohen's Kappa	Cost	Links	Submitter	Date
gpt-3.5-turbo 2025-04-16 Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	Criteria as text Approach · Prompting Criteria as text The case text, the candidate diagnoses and their ICHD-3 criteria as plain text in the prompt. Model gpt-3.5-turbo 2025-04-16 Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	External dataset · Zahn-50 Zahn et al. headache diagnosis dataset 50 clinical headache case examples across 18 ICHD-3 diagnoses: 34 derived from real patients, 16 synthetic. The cases are not public. Accuracy values were read from the bar chart in Fig. 2 of the paper and are approximate. Cases 50 Cases available No, not public Source Zahn A, Strauss S, Zwanzig D. Towards Community-Based Evaluation of AI in Neurology: Development of a Headache Diagnosis Dataset for Large Language Models. Stud Health Technol Inform 332, 237–241 (2025). https://doi.org/10.3233/SHTI251535 Results on this dataset are not directly comparable with results on a platform release or on another dataset.	24.04 %	39.02 %	66.90 %	—	—	—	Paper	—	2026-10-22
gpt-3.5-turbo 2025-04-16 Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	Naive prompt Approach · Prompting Naive prompt The case text and the list of candidate diagnoses, without any diagnostic criteria. Model gpt-3.5-turbo 2025-04-16 Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	External dataset · Zahn-50 Zahn et al. headache diagnosis dataset 50 clinical headache case examples across 18 ICHD-3 diagnoses: 34 derived from real patients, 16 synthetic. The cases are not public. Accuracy values were read from the bar chart in Fig. 2 of the paper and are approximate. Cases 50 Cases available No, not public Source Zahn A, Strauss S, Zwanzig D. Towards Community-Based Evaluation of AI in Neurology: Development of a Headache Diagnosis Dataset for Large Language Models. Stud Health Technol Inform 332, 237–241 (2025). https://doi.org/10.3233/SHTI251535 Results on this dataset are not directly comparable with results on a platform release or on another dataset.	33.10 %	47.04 %	80.14 %	—	—	—	Paper	—	2026-10-22
gpt-3.5-turbo 2025-04-16 Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	Criteria as JSON Approach · Prompting Criteria as JSON The case text, the candidate diagnoses and their ICHD-3 criteria as structured JSON in the prompt. Model gpt-3.5-turbo 2025-04-16 Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	External dataset · Zahn-50 Zahn et al. headache diagnosis dataset 50 clinical headache case examples across 18 ICHD-3 diagnoses: 34 derived from real patients, 16 synthetic. The cases are not public. Accuracy values were read from the bar chart in Fig. 2 of the paper and are approximate. Cases 50 Cases available No, not public Source Zahn A, Strauss S, Zwanzig D. Towards Community-Based Evaluation of AI in Neurology: Development of a Headache Diagnosis Dataset for Large Language Models. Stud Health Technol Inform 332, 237–241 (2025). https://doi.org/10.3233/SHTI251535 Results on this dataset are not directly comparable with results on a platform release or on another dataset.	36.93 %	60.97 %	75.96 %	—	—	—	Paper	—	2026-10-22
Gemma 3 12B IT google/gemma-3-12b-it Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	Zero-shot Approach · Prompting Zero-shot Case text plus the list of candidate ICHD-3 codes; single best diagnosis, no examples. Model Gemma 3 12B IT google/gemma-3-12b-it Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	Platform release Release 1.0 A published, checksummed release of this platform's community cases, validated by two independent reviewers. Cases 610 case variants Published 2026-10-22 License CC BY-NC 4.0 Release page and downloads About this result English variants (n=305) of release 1.0; vLLM, temperature 0 (Qwen3: 0.6), runs March 2026. Top-3 from a separate top-3 run; group = ICHD-3 top-level group of the first diagnosis (computed from the paper's run data).	42.30 %	—	83.61 %	—	—	—	Paper	—	2026-10-22
gpt-4-turbo 2025-04-16 Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	Criteria as text Approach · Prompting Criteria as text The case text, the candidate diagnoses and their ICHD-3 criteria as plain text in the prompt. Model gpt-4-turbo 2025-04-16 Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	External dataset · Zahn-50 Zahn et al. headache diagnosis dataset 50 clinical headache case examples across 18 ICHD-3 diagnoses: 34 derived from real patients, 16 synthetic. The cases are not public. Accuracy values were read from the bar chart in Fig. 2 of the paper and are approximate. Cases 50 Cases available No, not public Source Zahn A, Strauss S, Zwanzig D. Towards Community-Based Evaluation of AI in Neurology: Development of a Headache Diagnosis Dataset for Large Language Models. Stud Health Technol Inform 332, 237–241 (2025). https://doi.org/10.3233/SHTI251535 Results on this dataset are not directly comparable with results on a platform release or on another dataset.	44.95 %	78.05 %	86.06 %	—	—	—	Paper	—	2026-10-22
Gemma 3 12B IT google/gemma-3-12b-it Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	Chain-of-thought Approach · Prompting Chain-of-thought Case text and candidate list; the model reasons step by step before naming the diagnosis. Model Gemma 3 12B IT google/gemma-3-12b-it Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	Platform release Release 1.0 A published, checksummed release of this platform's community cases, validated by two independent reviewers. Cases 610 case variants Published 2026-10-22 License CC BY-NC 4.0 Release page and downloads About this result English variants (n=305) of release 1.0; vLLM, temperature 0 (Qwen3: 0.6), runs March 2026. Top-3 from a separate top-3 run; group = ICHD-3 top-level group of the first diagnosis (computed from the paper's run data).	45.57 %	53.80 %	86.23 %	—	—	—	Paper	—	2026-10-22
Gemma 3 12B IT google/gemma-3-12b-it Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	ICHD-3 criteria in context Approach · Prompting ICHD-3 criteria in context The full ICHD-3 ontology (simplified criteria, descriptions, clinical features) is placed in the system prompt. Model Gemma 3 12B IT google/gemma-3-12b-it Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	Platform release Release 1.0 A published, checksummed release of this platform's community cases, validated by two independent reviewers. Cases 610 case variants Published 2026-10-22 License CC BY-NC 4.0 Release page and downloads About this result English variants (n=305) of release 1.0; vLLM, temperature 0 (Qwen3: 0.6), runs March 2026. Top-3 from a separate top-3 run; group = ICHD-3 top-level group of the first diagnosis (computed from the paper's run data).	45.90 %	43.60 %	81.31 %	—	—	—	Paper	—	2026-10-22
Gemma 3 12B IT google/gemma-3-12b-it Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	Few-shot Approach · Prompting Few-shot Case text, candidate list and worked example cases in the prompt. Model Gemma 3 12B IT google/gemma-3-12b-it Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	Platform release Release 1.0 A published, checksummed release of this platform's community cases, validated by two independent reviewers. Cases 610 case variants Published 2026-10-22 License CC BY-NC 4.0 Release page and downloads About this result English variants (n=305) of release 1.0; vLLM, temperature 0 (Qwen3: 0.6), runs March 2026. Top-3 from a separate top-3 run; group = ICHD-3 top-level group of the first diagnosis (computed from the paper's run data).	47.54 %	52.80 %	86.23 %	—	—	—	Paper	—	2026-10-22
Gemma 3 27B IT google/gemma-3-27b-it Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	ICHD-3 criteria in context Approach · Prompting ICHD-3 criteria in context The full ICHD-3 ontology (simplified criteria, descriptions, clinical features) is placed in the system prompt. Model Gemma 3 27B IT google/gemma-3-27b-it Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	Platform release Release 1.0 A published, checksummed release of this platform's community cases, validated by two independent reviewers. Cases 610 case variants Published 2026-10-22 License CC BY-NC 4.0 Release page and downloads About this result English variants (n=305) of release 1.0; vLLM, temperature 0 (Qwen3: 0.6), runs March 2026. Top-3 from a separate top-3 run; group = ICHD-3 top-level group of the first diagnosis (computed from the paper's run data).	49.18 %	58.00 %	87.87 %	—	—	—	Paper	—	2026-10-22
o3-mini 2025-04-16 Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	Criteria as JSON Approach · Prompting Criteria as JSON The case text, the candidate diagnoses and their ICHD-3 criteria as structured JSON in the prompt. Model o3-mini 2025-04-16 Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	External dataset · Zahn-50 Zahn et al. headache diagnosis dataset 50 clinical headache case examples across 18 ICHD-3 diagnoses: 34 derived from real patients, 16 synthetic. The cases are not public. Accuracy values were read from the bar chart in Fig. 2 of the paper and are approximate. Cases 50 Cases available No, not public Source Zahn A, Strauss S, Zwanzig D. Towards Community-Based Evaluation of AI in Neurology: Development of a Headache Diagnosis Dataset for Large Language Models. Stud Health Technol Inform 332, 237–241 (2025). https://doi.org/10.3233/SHTI251535 Results on this dataset are not directly comparable with results on a platform release or on another dataset.	50.87 %	71.08 %	75.96 %	—	—	—	Paper	—	2026-10-22
Qwen3-14B Qwen/Qwen3-14B Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	Zero-shot Approach · Prompting Zero-shot Case text plus the list of candidate ICHD-3 codes; single best diagnosis, no examples. Model Qwen3-14B Qwen/Qwen3-14B Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	Platform release Release 1.0 A published, checksummed release of this platform's community cases, validated by two independent reviewers. Cases 610 case variants Published 2026-10-22 License CC BY-NC 4.0 Release page and downloads About this result English variants (n=305) of release 1.0; vLLM, temperature 0 (Qwen3: 0.6), runs March 2026. Top-3 from a separate top-3 run; group = ICHD-3 top-level group of the first diagnosis (computed from the paper's run data).	52.79 %	64.30 %	87.21 %	—	—	—	Paper	—	2026-10-22
Gemma 3 27B IT google/gemma-3-27b-it Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	Chain-of-thought Approach · Prompting Chain-of-thought Case text and candidate list; the model reasons step by step before naming the diagnosis. Model Gemma 3 27B IT google/gemma-3-27b-it Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	Platform release Release 1.0 A published, checksummed release of this platform's community cases, validated by two independent reviewers. Cases 610 case variants Published 2026-10-22 License CC BY-NC 4.0 Release page and downloads About this result English variants (n=305) of release 1.0; vLLM, temperature 0 (Qwen3: 0.6), runs March 2026. Top-3 from a separate top-3 run; group = ICHD-3 top-level group of the first diagnosis (computed from the paper's run data).	53.44 %	56.40 %	90.16 %	—	—	—	Paper	—	2026-10-22
Gemma 3 27B IT google/gemma-3-27b-it Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	Zero-shot Approach · Prompting Zero-shot Case text plus the list of candidate ICHD-3 codes; single best diagnosis, no examples. Model Gemma 3 27B IT google/gemma-3-27b-it Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	Platform release Release 1.0 A published, checksummed release of this platform's community cases, validated by two independent reviewers. Cases 610 case variants Published 2026-10-22 License CC BY-NC 4.0 Release page and downloads About this result English variants (n=305) of release 1.0; vLLM, temperature 0 (Qwen3: 0.6), runs March 2026. Top-3 from a separate top-3 run; group = ICHD-3 top-level group of the first diagnosis (computed from the paper's run data).	53.77 %	58.40 %	89.84 %	—	—	—	Paper	—	2026-10-22
gpt-4-turbo 2025-04-16 Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	Criteria as JSON Approach · Prompting Criteria as JSON The case text, the candidate diagnoses and their ICHD-3 criteria as structured JSON in the prompt. Model gpt-4-turbo 2025-04-16 Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	External dataset · Zahn-50 Zahn et al. headache diagnosis dataset 50 clinical headache case examples across 18 ICHD-3 diagnoses: 34 derived from real patients, 16 synthetic. The cases are not public. Accuracy values were read from the bar chart in Fig. 2 of the paper and are approximate. Cases 50 Cases available No, not public Source Zahn A, Strauss S, Zwanzig D. Towards Community-Based Evaluation of AI in Neurology: Development of a Headache Diagnosis Dataset for Large Language Models. Stud Health Technol Inform 332, 237–241 (2025). https://doi.org/10.3233/SHTI251535 Results on this dataset are not directly comparable with results on a platform release or on another dataset.	55.05 %	80.14 %	86.06 %	—	—	—	Paper	—	2026-10-22
gpt-4.1 2025-04-16 Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	Naive prompt Approach · Prompting Naive prompt The case text and the list of candidate diagnoses, without any diagnostic criteria. Model gpt-4.1 2025-04-16 Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	External dataset · Zahn-50 Zahn et al. headache diagnosis dataset 50 clinical headache case examples across 18 ICHD-3 diagnoses: 34 derived from real patients, 16 synthetic. The cases are not public. Accuracy values were read from the bar chart in Fig. 2 of the paper and are approximate. Cases 50 Cases available No, not public Source Zahn A, Strauss S, Zwanzig D. Towards Community-Based Evaluation of AI in Neurology: Development of a Headache Diagnosis Dataset for Large Language Models. Stud Health Technol Inform 332, 237–241 (2025). https://doi.org/10.3233/SHTI251535 Results on this dataset are not directly comparable with results on a platform release or on another dataset.	55.05 %	75.96 %	88.15 %	—	—	—	Paper	—	2026-10-22
Gemma 3 12B IT google/gemma-3-12b-it Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	RAG (ICHD-3 retrieval) Approach · Retrieval-augmented RAG (ICHD-3 retrieval) For each case, the most similar ICHD-3 diagnosis entries are retrieved by embedding similarity and added to the prompt; the model decides. Model Gemma 3 12B IT google/gemma-3-12b-it Retrieval-augmented approaches For each case, relevant ICHD-3 knowledge is retrieved from a text collection and added to the prompt; the language model decides.	Platform release Release 1.0 A published, checksummed release of this platform's community cases, validated by two independent reviewers. Cases 610 case variants Published 2026-10-22 License CC BY-NC 4.0 Release page and downloads About this result English variants (n=305) of release 1.0; vLLM, temperature 0 (Qwen3: 0.6), runs March 2026. Top-3 from a separate top-3 run; group = ICHD-3 top-level group of the first diagnosis (computed from the paper's run data).	55.41 %	59.30 %	89.51 %	—	—	—	Paper	—	2026-10-22
Qwen3-14B Qwen/Qwen3-14B Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	RAG (ICHD-3 retrieval) Approach · Retrieval-augmented RAG (ICHD-3 retrieval) For each case, the most similar ICHD-3 diagnosis entries are retrieved by embedding similarity and added to the prompt; the model decides. Model Qwen3-14B Qwen/Qwen3-14B Retrieval-augmented approaches For each case, relevant ICHD-3 knowledge is retrieved from a text collection and added to the prompt; the language model decides.	Platform release Release 1.0 A published, checksummed release of this platform's community cases, validated by two independent reviewers. Cases 610 case variants Published 2026-10-22 License CC BY-NC 4.0 Release page and downloads About this result English variants (n=305) of release 1.0; vLLM, temperature 0 (Qwen3: 0.6), runs March 2026. Top-3 from a separate top-3 run; group = ICHD-3 top-level group of the first diagnosis (computed from the paper's run data).	56.72 %	64.60 %	85.57 %	—	—	—	Paper	—	2026-10-22
Mistral Small 3.2 24B Instruct mistralai/Mistral-Small-3.2-24B-Instruct-2506 Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	RAG (ICHD-3 retrieval) Approach · Retrieval-augmented RAG (ICHD-3 retrieval) For each case, the most similar ICHD-3 diagnosis entries are retrieved by embedding similarity and added to the prompt; the model decides. Model Mistral Small 3.2 24B Instruct mistralai/Mistral-Small-3.2-24B-Instruct-2506 Retrieval-augmented approaches For each case, relevant ICHD-3 knowledge is retrieved from a text collection and added to the prompt; the language model decides.	Platform release Release 1.0 A published, checksummed release of this platform's community cases, validated by two independent reviewers. Cases 610 case variants Published 2026-10-22 License CC BY-NC 4.0 Release page and downloads About this result English variants (n=305) of release 1.0; vLLM, temperature 0 (Qwen3: 0.6), runs March 2026. Top-3 from a separate top-3 run; group = ICHD-3 top-level group of the first diagnosis (computed from the paper's run data).	57.05 %	63.50 %	86.56 %	—	—	—	Paper	—	2026-10-22
Gemma 3 27B IT google/gemma-3-27b-it Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	Few-shot Approach · Prompting Few-shot Case text, candidate list and worked example cases in the prompt. Model Gemma 3 27B IT google/gemma-3-27b-it Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	Platform release Release 1.0 A published, checksummed release of this platform's community cases, validated by two independent reviewers. Cases 610 case variants Published 2026-10-22 License CC BY-NC 4.0 Release page and downloads About this result English variants (n=305) of release 1.0; vLLM, temperature 0 (Qwen3: 0.6), runs March 2026. Top-3 from a separate top-3 run; group = ICHD-3 top-level group of the first diagnosis (computed from the paper's run data).	57.05 %	61.60 %	91.15 %	—	—	—	Paper	—	2026-10-22
gpt-4-turbo 2025-04-16 Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	Naive prompt Approach · Prompting Naive prompt The case text and the list of candidate diagnoses, without any diagnostic criteria. Model gpt-4-turbo 2025-04-16 Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	External dataset · Zahn-50 Zahn et al. headache diagnosis dataset 50 clinical headache case examples across 18 ICHD-3 diagnoses: 34 derived from real patients, 16 synthetic. The cases are not public. Accuracy values were read from the bar chart in Fig. 2 of the paper and are approximate. Cases 50 Cases available No, not public Source Zahn A, Strauss S, Zwanzig D. Towards Community-Based Evaluation of AI in Neurology: Development of a Headache Diagnosis Dataset for Large Language Models. Stud Health Technol Inform 332, 237–241 (2025). https://doi.org/10.3233/SHTI251535 Results on this dataset are not directly comparable with results on a platform release or on another dataset.	57.14 %	78.05 %	86.06 %	—	—	—	Paper	—	2026-10-22
gpt-4.1 2025-04-16 Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	Criteria as JSON Approach · Prompting Criteria as JSON The case text, the candidate diagnoses and their ICHD-3 criteria as structured JSON in the prompt. Model gpt-4.1 2025-04-16 Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	External dataset · Zahn-50 Zahn et al. headache diagnosis dataset 50 clinical headache case examples across 18 ICHD-3 diagnoses: 34 derived from real patients, 16 synthetic. The cases are not public. Accuracy values were read from the bar chart in Fig. 2 of the paper and are approximate. Cases 50 Cases available No, not public Source Zahn A, Strauss S, Zwanzig D. Towards Community-Based Evaluation of AI in Neurology: Development of a Headache Diagnosis Dataset for Large Language Models. Stud Health Technol Inform 332, 237–241 (2025). https://doi.org/10.3233/SHTI251535 Results on this dataset are not directly comparable with results on a platform release or on another dataset.	57.14 %	75.96 %	89.89 %	—	—	—	Paper	—	2026-10-22
Gemma 3 27B IT google/gemma-3-27b-it Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	RAG (ICHD-3 retrieval) Approach · Retrieval-augmented RAG (ICHD-3 retrieval) For each case, the most similar ICHD-3 diagnosis entries are retrieved by embedding similarity and added to the prompt; the model decides. Model Gemma 3 27B IT google/gemma-3-27b-it Retrieval-augmented approaches For each case, relevant ICHD-3 knowledge is retrieved from a text collection and added to the prompt; the language model decides.	Platform release Release 1.0 A published, checksummed release of this platform's community cases, validated by two independent reviewers. Cases 610 case variants Published 2026-10-22 License CC BY-NC 4.0 Release page and downloads About this result English variants (n=305) of release 1.0; vLLM, temperature 0 (Qwen3: 0.6), runs March 2026. Top-3 from a separate top-3 run; group = ICHD-3 top-level group of the first diagnosis (computed from the paper's run data).	57.38 %	61.60 %	90.49 %	—	—	—	Paper	—	2026-10-22
o3-mini 2025-04-16 Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	Naive prompt Approach · Prompting Naive prompt The case text and the list of candidate diagnoses, without any diagnostic criteria. Model o3-mini 2025-04-16 Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	External dataset · Zahn-50 Zahn et al. headache diagnosis dataset 50 clinical headache case examples across 18 ICHD-3 diagnoses: 34 derived from real patients, 16 synthetic. The cases are not public. Accuracy values were read from the bar chart in Fig. 2 of the paper and are approximate. Cases 50 Cases available No, not public Source Zahn A, Strauss S, Zwanzig D. Towards Community-Based Evaluation of AI in Neurology: Development of a Headache Diagnosis Dataset for Large Language Models. Stud Health Technol Inform 332, 237–241 (2025). https://doi.org/10.3233/SHTI251535 Results on this dataset are not directly comparable with results on a platform release or on another dataset.	58.88 %	75.96 %	80.14 %	—	—	—	Paper	—	2026-10-22
gpt-4.1 2025-04-16 Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	Criteria as text Approach · Prompting Criteria as text The case text, the candidate diagnoses and their ICHD-3 criteria as plain text in the prompt. Model gpt-4.1 2025-04-16 Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	External dataset · Zahn-50 Zahn et al. headache diagnosis dataset 50 clinical headache case examples across 18 ICHD-3 diagnoses: 34 derived from real patients, 16 synthetic. The cases are not public. Accuracy values were read from the bar chart in Fig. 2 of the paper and are approximate. Cases 50 Cases available No, not public Source Zahn A, Strauss S, Zwanzig D. Towards Community-Based Evaluation of AI in Neurology: Development of a Headache Diagnosis Dataset for Large Language Models. Stud Health Technol Inform 332, 237–241 (2025). https://doi.org/10.3233/SHTI251535 Results on this dataset are not directly comparable with results on a platform release or on another dataset.	58.88 %	68.99 %	91.98 %	—	—	—	Paper	—	2026-10-22
Mistral Small 3.2 24B Instruct mistralai/Mistral-Small-3.2-24B-Instruct-2506 Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	Zero-shot Approach · Prompting Zero-shot Case text plus the list of candidate ICHD-3 codes; single best diagnosis, no examples. Model Mistral Small 3.2 24B Instruct mistralai/Mistral-Small-3.2-24B-Instruct-2506 Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	Platform release Release 1.0 A published, checksummed release of this platform's community cases, validated by two independent reviewers. Cases 610 case variants Published 2026-10-22 License CC BY-NC 4.0 Release page and downloads About this result English variants (n=305) of release 1.0; vLLM, temperature 0 (Qwen3: 0.6), runs March 2026. Top-3 from a separate top-3 run; group = ICHD-3 top-level group of the first diagnosis (computed from the paper's run data).	59.34 %	73.50 %	89.51 %	—	—	—	Paper	—	2026-10-22
Qwen3-14B Qwen/Qwen3-14B Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	Few-shot Approach · Prompting Few-shot Case text, candidate list and worked example cases in the prompt. Model Qwen3-14B Qwen/Qwen3-14B Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	Platform release Release 1.0 A published, checksummed release of this platform's community cases, validated by two independent reviewers. Cases 610 case variants Published 2026-10-22 License CC BY-NC 4.0 Release page and downloads About this result English variants (n=305) of release 1.0; vLLM, temperature 0 (Qwen3: 0.6), runs March 2026. Top-3 from a separate top-3 run; group = ICHD-3 top-level group of the first diagnosis (computed from the paper's run data).	60.00 %	70.50 %	89.18 %	—	—	—	Paper	—	2026-10-22
Ministral 3 14B Instruct mistralai/Ministral-3-14B-Instruct-2512 Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	RAG (ICHD-3 retrieval) Approach · Retrieval-augmented RAG (ICHD-3 retrieval) For each case, the most similar ICHD-3 diagnosis entries are retrieved by embedding similarity and added to the prompt; the model decides. Model Ministral 3 14B Instruct mistralai/Ministral-3-14B-Instruct-2512 Retrieval-augmented approaches For each case, relevant ICHD-3 knowledge is retrieved from a text collection and added to the prompt; the language model decides.	Platform release Release 1.0 A published, checksummed release of this platform's community cases, validated by two independent reviewers. Cases 610 case variants Published 2026-10-22 License CC BY-NC 4.0 Release page and downloads About this result English variants (n=305) of release 1.0; vLLM, temperature 0 (Qwen3: 0.6), runs March 2026. Top-3 from a separate top-3 run; group = ICHD-3 top-level group of the first diagnosis (computed from the paper's run data).	61.64 %	69.20 %	90.49 %	—	—	—	Paper	—	2026-10-22
Qwen3-14B Qwen/Qwen3-14B Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	Chain-of-thought Approach · Prompting Chain-of-thought Case text and candidate list; the model reasons step by step before naming the diagnosis. Model Qwen3-14B Qwen/Qwen3-14B Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	Platform release Release 1.0 A published, checksummed release of this platform's community cases, validated by two independent reviewers. Cases 610 case variants Published 2026-10-22 License CC BY-NC 4.0 Release page and downloads About this result English variants (n=305) of release 1.0; vLLM, temperature 0 (Qwen3: 0.6), runs March 2026. Top-3 from a separate top-3 run; group = ICHD-3 top-level group of the first diagnosis (computed from the paper's run data).	62.62 %	70.80 %	89.18 %	—	—	—	Paper	—	2026-10-22
Mistral Small 3.2 24B Instruct mistralai/Mistral-Small-3.2-24B-Instruct-2506 Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	ICHD-3 criteria in context Approach · Prompting ICHD-3 criteria in context The full ICHD-3 ontology (simplified criteria, descriptions, clinical features) is placed in the system prompt. Model Mistral Small 3.2 24B Instruct mistralai/Mistral-Small-3.2-24B-Instruct-2506 Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	Platform release Release 1.0 A published, checksummed release of this platform's community cases, validated by two independent reviewers. Cases 610 case variants Published 2026-10-22 License CC BY-NC 4.0 Release page and downloads About this result English variants (n=305) of release 1.0; vLLM, temperature 0 (Qwen3: 0.6), runs March 2026. Top-3 from a separate top-3 run; group = ICHD-3 top-level group of the first diagnosis (computed from the paper's run data).	62.95 %	71.50 %	92.46 %	—	—	—	Paper	—	2026-10-22
o3-mini 2025-04-16 Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	Criteria as text Approach · Prompting Criteria as text The case text, the candidate diagnoses and their ICHD-3 criteria as plain text in the prompt. Model o3-mini 2025-04-16 Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	External dataset · Zahn-50 Zahn et al. headache diagnosis dataset 50 clinical headache case examples across 18 ICHD-3 diagnoses: 34 derived from real patients, 16 synthetic. The cases are not public. Accuracy values were read from the bar chart in Fig. 2 of the paper and are approximate. Cases 50 Cases available No, not public Source Zahn A, Strauss S, Zwanzig D. Towards Community-Based Evaluation of AI in Neurology: Development of a Headache Diagnosis Dataset for Large Language Models. Stud Health Technol Inform 332, 237–241 (2025). https://doi.org/10.3233/SHTI251535 Results on this dataset are not directly comparable with results on a platform release or on another dataset.	63.07 %	75.96 %	83.97 %	—	—	—	Paper	—	2026-10-22
Mistral Small 3.2 24B Instruct mistralai/Mistral-Small-3.2-24B-Instruct-2506 Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	Chain-of-thought Approach · Prompting Chain-of-thought Case text and candidate list; the model reasons step by step before naming the diagnosis. Model Mistral Small 3.2 24B Instruct mistralai/Mistral-Small-3.2-24B-Instruct-2506 Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	Platform release Release 1.0 A published, checksummed release of this platform's community cases, validated by two independent reviewers. Cases 610 case variants Published 2026-10-22 License CC BY-NC 4.0 Release page and downloads About this result English variants (n=305) of release 1.0; vLLM, temperature 0 (Qwen3: 0.6), runs March 2026. Top-3 from a separate top-3 run; group = ICHD-3 top-level group of the first diagnosis (computed from the paper's run data).	64.59 %	72.10 %	93.44 %	—	—	—	Paper	—	2026-10-22
Mistral Small 3.2 24B Instruct mistralai/Mistral-Small-3.2-24B-Instruct-2506 Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	Few-shot Approach · Prompting Few-shot Case text, candidate list and worked example cases in the prompt. Model Mistral Small 3.2 24B Instruct mistralai/Mistral-Small-3.2-24B-Instruct-2506 Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	Platform release Release 1.0 A published, checksummed release of this platform's community cases, validated by two independent reviewers. Cases 610 case variants Published 2026-10-22 License CC BY-NC 4.0 Release page and downloads About this result English variants (n=305) of release 1.0; vLLM, temperature 0 (Qwen3: 0.6), runs March 2026. Top-3 from a separate top-3 run; group = ICHD-3 top-level group of the first diagnosis (computed from the paper's run data).	66.23 %	77.00 %	93.11 %	—	—	—	Paper	—	2026-10-22
Ministral 3 14B Instruct mistralai/Ministral-3-14B-Instruct-2512 Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	Chain-of-thought Approach · Prompting Chain-of-thought Case text and candidate list; the model reasons step by step before naming the diagnosis. Model Ministral 3 14B Instruct mistralai/Ministral-3-14B-Instruct-2512 Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	Platform release Release 1.0 A published, checksummed release of this platform's community cases, validated by two independent reviewers. Cases 610 case variants Published 2026-10-22 License CC BY-NC 4.0 Release page and downloads About this result English variants (n=305) of release 1.0; vLLM, temperature 0 (Qwen3: 0.6), runs March 2026. Top-3 from a separate top-3 run; group = ICHD-3 top-level group of the first diagnosis (computed from the paper's run data).	67.54 %	72.80 %	92.79 %	—	—	—	Paper	—	2026-10-22
Ministral 3 14B Instruct mistralai/Ministral-3-14B-Instruct-2512 Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	ICHD-3 criteria in context Approach · Prompting ICHD-3 criteria in context The full ICHD-3 ontology (simplified criteria, descriptions, clinical features) is placed in the system prompt. Model Ministral 3 14B Instruct mistralai/Ministral-3-14B-Instruct-2512 Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	Platform release Release 1.0 A published, checksummed release of this platform's community cases, validated by two independent reviewers. Cases 610 case variants Published 2026-10-22 License CC BY-NC 4.0 Release page and downloads About this result English variants (n=305) of release 1.0; vLLM, temperature 0 (Qwen3: 0.6), runs March 2026. Top-3 from a separate top-3 run; group = ICHD-3 top-level group of the first diagnosis (computed from the paper's run data).	67.87 %	79.30 %	92.46 %	—	—	—	Paper	—	2026-10-22
Ministral 3 14B Instruct mistralai/Ministral-3-14B-Instruct-2512 Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	Zero-shot Approach · Prompting Zero-shot Case text plus the list of candidate ICHD-3 codes; single best diagnosis, no examples. Model Ministral 3 14B Instruct mistralai/Ministral-3-14B-Instruct-2512 Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	Platform release Release 1.0 A published, checksummed release of this platform's community cases, validated by two independent reviewers. Cases 610 case variants Published 2026-10-22 License CC BY-NC 4.0 Release page and downloads About this result English variants (n=305) of release 1.0; vLLM, temperature 0 (Qwen3: 0.6), runs March 2026. Top-3 from a separate top-3 run; group = ICHD-3 top-level group of the first diagnosis (computed from the paper's run data).	67.87 %	78.30 %	92.13 %	—	—	—	Paper	—	2026-10-22
Ministral 3 14B Instruct mistralai/Ministral-3-14B-Instruct-2512 Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	Few-shot Approach · Prompting Few-shot Case text, candidate list and worked example cases in the prompt. Model Ministral 3 14B Instruct mistralai/Ministral-3-14B-Instruct-2512 Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	Platform release Release 1.0 A published, checksummed release of this platform's community cases, validated by two independent reviewers. Cases 610 case variants Published 2026-10-22 License CC BY-NC 4.0 Release page and downloads About this result English variants (n=305) of release 1.0; vLLM, temperature 0 (Qwen3: 0.6), runs March 2026. Top-3 from a separate top-3 run; group = ICHD-3 top-level group of the first diagnosis (computed from the paper's run data).	69.18 %	77.20 %	93.44 %	—	—	—	Paper	—	2026-10-22
Claude Sonnet 4.0 Peer-reviewed publication ↗	Chatbot default Approach · Prompting Chatbot default The public chatbot with its default settings, given the vignette as written. Model Claude Sonnet 4.0 Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	External dataset · HeadAI-pt-315 Head.AI validation vignettes (original, Brazilian Portuguese) 315 expert-written clinical headache vignettes from the Head.AI validation study. Results as reported in the paper: correct diagnosis as the first hypothesis. Top-3 is omitted because the rank counts in the paper do not sum to n. Cases 315 Language Portuguese (pt-BR) Cases available Yes, publicly Source de Andrade JBC et al. Validation of an AI-based platform for structured diagnosis of headache disorders using ICHD-3 criteria. BMC Neurology 25, 489 (2025). Dataset: Head.AI Platform, Mendeley Data V2, CC BY 4.0. https://doi.org/10.1186/s12883-025-04473-1 Results on this dataset are not directly comparable with results on a platform release or on another dataset.	74.30 %	—	—	—	—	—	Paper	—	2026-10-22
Gemini 2.5 Peer-reviewed publication ↗	Chatbot default Approach · Prompting Chatbot default The public chatbot with its default settings, given the vignette as written. Model Gemini 2.5 Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	External dataset · HeadAI-pt-315 Head.AI validation vignettes (original, Brazilian Portuguese) 315 expert-written clinical headache vignettes from the Head.AI validation study. Results as reported in the paper: correct diagnosis as the first hypothesis. Top-3 is omitted because the rank counts in the paper do not sum to n. Cases 315 Language Portuguese (pt-BR) Cases available Yes, publicly Source de Andrade JBC et al. Validation of an AI-based platform for structured diagnosis of headache disorders using ICHD-3 criteria. BMC Neurology 25, 489 (2025). Dataset: Head.AI Platform, Mendeley Data V2, CC BY 4.0. https://doi.org/10.1186/s12883-025-04473-1 Results on this dataset are not directly comparable with results on a platform release or on another dataset.	77.10 %	—	—	—	—	—	Paper	—	2026-10-22
Grok 3.0 Peer-reviewed publication ↗	Chatbot default Approach · Prompting Chatbot default The public chatbot with its default settings, given the vignette as written. Model Grok 3.0 Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	External dataset · HeadAI-pt-315 Head.AI validation vignettes (original, Brazilian Portuguese) 315 expert-written clinical headache vignettes from the Head.AI validation study. Results as reported in the paper: correct diagnosis as the first hypothesis. Top-3 is omitted because the rank counts in the paper do not sum to n. Cases 315 Language Portuguese (pt-BR) Cases available Yes, publicly Source de Andrade JBC et al. Validation of an AI-based platform for structured diagnosis of headache disorders using ICHD-3 criteria. BMC Neurology 25, 489 (2025). Dataset: Head.AI Platform, Mendeley Data V2, CC BY 4.0. https://doi.org/10.1186/s12883-025-04473-1 Results on this dataset are not directly comparable with results on a platform release or on another dataset.	80.30 %	—	—	—	—	—	Paper	—	2026-10-22
Head.AI (GPT-4o + ICHD-3 knowledge base) GPT-4o, 2025 Peer-reviewed publication ↗	ICHD-3 knowledge in prompt Approach · Prompting ICHD-3 knowledge in prompt The Head.AI platform: GPT-4o with engineered prompts that contain structured ICHD-3 knowledge as text. Model Head.AI (GPT-4o + ICHD-3 knowledge base) GPT-4o, 2025 Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	External dataset · HeadAI-pt-315 Head.AI validation vignettes (original, Brazilian Portuguese) 315 expert-written clinical headache vignettes from the Head.AI validation study. Results as reported in the paper: correct diagnosis as the first hypothesis. Top-3 is omitted because the rank counts in the paper do not sum to n. Cases 315 Language Portuguese (pt-BR) Cases available Yes, publicly Source de Andrade JBC et al. Validation of an AI-based platform for structured diagnosis of headache disorders using ICHD-3 criteria. BMC Neurology 25, 489 (2025). Dataset: Head.AI Platform, Mendeley Data V2, CC BY 4.0. https://doi.org/10.1186/s12883-025-04473-1 Results on this dataset are not directly comparable with results on a platform release or on another dataset.	89.50 %	—	—	—	—	—	Paper	—	2026-10-22
NotebookLM (Gemini 3.0 Pro) accessed 2026-02 Peer-reviewed publication ↗	NotebookLM over ICHD-3 + guidelines Approach · Retrieval-augmented NotebookLM over ICHD-3 + guidelines NotebookLM indexed with the full ICHD-3 and major headache treatment guidelines (PDF); each vignette in a fresh session, answers grounded in the uploaded sources. Model NotebookLM (Gemini 3.0 Pro) accessed 2026-02 Retrieval-augmented approaches For each case, relevant ICHD-3 knowledge is retrieved from a text collection and added to the prompt; the language model decides.	External dataset · Babaoglu-35 Babaoğlu headache case vignettes 35 fictional case vignettes written for the study: 17 primary headaches, 15 secondary headaches, 3 neuropathic/facial pain syndromes; gold-standard diagnoses by two pain specialists. Very small set — read the 100 % as an upper bound, not a stable estimate. Cases 35 Cases available No, not public Source Babaoğlu G. Evaluation of a guideline-anchored retrieval-augmented generation (RAG) system for headache diagnosis and management: A case-vignette study. Medicine Science 15(2), 593–601 (2026). https://doi.org/10.5455/medscience.2026.03.055 Results on this dataset are not directly comparable with results on a platform release or on another dataset. About this result Subtype-level top-1 35/35 (95 % CI 90.0–100.0); secondary vs non-secondary 35/35.	100.00 %	—	—	—	1.000	—	Paper	—	2026-10-22
Gradient boosting classifier Katsuki et al. 2023 model Peer-reviewed publication ↗	Supervised ML (questionnaire features) Approach · Other Supervised ML (questionnaire features) Supervised machine-learning classifier (gradient boosting, PyCaret) trained on 4,240 questionnaires of the same clinic; not a language model — listed as the study's comparator. Model Gradient boosting classifier Katsuki et al. 2023 model Other approaches An approach outside the categories above; see the method details.	External dataset · Katsuki-1818 Katsuki et al. headache clinic questionnaire test cohort Hold-out test cohort of 1,818 patients of a Japanese headache-specialty clinic: a 22-item structured questionnaire per patient, specialist-confirmed diagnosis. Five ICHD-3-based classes instead of single diagnoses: migraine and/or medication-overuse headache, tension-type headache, TACs, other primary headaches, secondary headaches. Real patients, strongly imbalanced (1,463 migraine/MOH). Results are therefore given at group level only; MOH is merged with migraine, so the classes do not match the ICHD-3 groups exactly. Cases 1818 Language Japanese Cases available No, not public Source Katsuki M, Moran K, O'Connor S, et al. Comparative diagnostic agreement of a supervised machine learning model and a general-purpose, zero-shot, non-domain-adapted large language model for classifying headache disorders using structured questionnaires. Cephalalgia 46(4) (2026). https://doi.org/10.1177/03331024261441574 Results on this dataset are not directly comparable with results on a platform release or on another dataset. About this result 5-class accuracy 1531/1818, computed from the confusion matrix (Table 2); macro sensitivity 48.5 %, macro F 54.3 %.	—	—	84.21 %	—	0.460	—	Paper	—	2026-10-22
gpt-4.1 API, accessed 2025-11-10 Peer-reviewed publication ↗	Zero-shot (structured questionnaire) Approach · Prompting Zero-shot (structured questionnaire) The 22 questionnaire answers as fixed Japanese text fields in one prompt; temperature 0, no examples, no fine-tuning; the model names one of the five classes. Model gpt-4.1 API, accessed 2025-11-10 Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	External dataset · Katsuki-1818 Katsuki et al. headache clinic questionnaire test cohort Hold-out test cohort of 1,818 patients of a Japanese headache-specialty clinic: a 22-item structured questionnaire per patient, specialist-confirmed diagnosis. Five ICHD-3-based classes instead of single diagnoses: migraine and/or medication-overuse headache, tension-type headache, TACs, other primary headaches, secondary headaches. Real patients, strongly imbalanced (1,463 migraine/MOH). Results are therefore given at group level only; MOH is merged with migraine, so the classes do not match the ICHD-3 groups exactly. Cases 1818 Language Japanese Cases available No, not public Source Katsuki M, Moran K, O'Connor S, et al. Comparative diagnostic agreement of a supervised machine learning model and a general-purpose, zero-shot, non-domain-adapted large language model for classifying headache disorders using structured questionnaires. Cephalalgia 46(4) (2026). https://doi.org/10.1177/03331024261441574 Results on this dataset are not directly comparable with results on a platform release or on another dataset. About this result 5-class accuracy 1236/1818, computed from the confusion matrix (Table 3); macro sensitivity 50.4 %, macro F 34.6 %.	—	—	67.99 %	—	0.260	—	Paper	—	2026-10-22

Download as CSV — current filters and sort applied.