gpt-4.1
All 4 published results of this model on the leaderboard — every version, method and dataset — with the publications they come from. Results on different datasets are not directly comparable.
- Versions
- 2025-04-16, API, accessed 2025-11-10
- Approach
- Prompting
- Datasets
- Size
- API, accessed 2025-11-10: ~2.2T · Third-party estimate (see “About model sizes” on the leaderboard)
- Size
- 2025-04-16: ~2.2T · Third-party estimate (see “About model sizes” on the leaderboard)
Publications
- https://doi.org/10.1177/03331024261441574 — 1 result on Katsuki-1818
- https://doi.org/10.3233/SHTI251535 — 3 results on Zahn-50
Results
Results marked Platform team were produced by the people who run this platform — disclosed because the same team curates the data and verifies results.
| Version · badges | Approach | Dataset | Top-1 (diagnosis) | Top-3 (diagnosis) | Top-1 (group) | Top-3 (group) | Cohen's Kappa | Links |
|---|---|---|---|---|---|---|---|---|
| API, accessed 2025-11-10 Peer-reviewed publication ↗ |
Zero-shot (structured questionnaire)
Approach · Prompting Zero-shot (structured questionnaire)The 22 questionnaire answers as fixed Japanese text fields in one prompt; temperature 0, no examples, no fine-tuning; the model names one of the five classes. Prompting approachesA general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.
|
External dataset · Katsuki-1818 Katsuki et al. headache clinic questionnaire test cohortHold-out test cohort of 1,818 patients of a Japanese headache-specialty clinic: a 22-item structured questionnaire per patient, specialist-confirmed diagnosis. Five ICHD-3-based classes instead of single diagnoses: migraine and/or medication-overuse headache, tension-type headache, TACs, other primary headaches, secondary headaches. Real patients, strongly imbalanced (1,463 migraine/MOH). Results are therefore given at group level only; MOH is merged with migraine, so the classes do not match the ICHD-3 groups exactly. Results on this dataset are not directly comparable with results on a platform release or on another dataset. About this result5-class accuracy 1236/1818, computed from the confusion matrix (Table 3); macro sensitivity 50.4 %, macro F 34.6 %.
|
— | — | 67.99 % | — | 0.260 | Paper |
| 2025-04-16 Verified Peer-reviewed publication ↗ Disclosure Produced by the platform teamThis result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out.
|
Criteria as text
Approach · Prompting Criteria as textThe case text, the candidate diagnoses and their ICHD-3 criteria as plain text in the prompt. Prompting approachesA general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.
|
External dataset · Zahn-50 Zahn et al. headache diagnosis dataset50 clinical headache case examples across 18 ICHD-3 diagnoses: 34 derived from real patients, 16 synthetic. The cases are not public. Accuracy values were read from the bar chart in Fig. 2 of the paper and are approximate. Results on this dataset are not directly comparable with results on a platform release or on another dataset.
|
58.88 % | 68.99 % | 91.98 % | — | — | Paper |
| 2025-04-16 Verified Peer-reviewed publication ↗ Disclosure Produced by the platform teamThis result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out.
|
Criteria as JSON
Approach · Prompting Criteria as JSONThe case text, the candidate diagnoses and their ICHD-3 criteria as structured JSON in the prompt. Prompting approachesA general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.
|
External dataset · Zahn-50 Zahn et al. headache diagnosis dataset50 clinical headache case examples across 18 ICHD-3 diagnoses: 34 derived from real patients, 16 synthetic. The cases are not public. Accuracy values were read from the bar chart in Fig. 2 of the paper and are approximate. Results on this dataset are not directly comparable with results on a platform release or on another dataset.
|
57.14 % | 75.96 % | 89.89 % | — | — | Paper |
| 2025-04-16 Verified Peer-reviewed publication ↗ Disclosure Produced by the platform teamThis result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out.
|
Naive prompt
Approach · Prompting Naive promptThe case text and the list of candidate diagnoses, without any diagnostic criteria. Prompting approachesA general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.
|
External dataset · Zahn-50 Zahn et al. headache diagnosis dataset50 clinical headache case examples across 18 ICHD-3 diagnoses: 34 derived from real patients, 16 synthetic. The cases are not public. Accuracy values were read from the bar chart in Fig. 2 of the paper and are approximate. Results on this dataset are not directly comparable with results on a platform release or on another dataset.
|
55.05 % | 75.96 % | 88.15 % | — | — | Paper |