Gemma 3 27B IT — Leaderboard — headache.bench

Version: google/gemma-3-27b-it
Approaches: Prompting, Retrieval-augmented
Dataset
Size: google/gemma-3-27b-it: ~27B · Disclosed by the developer (see “About model sizes” on the leaderboard)

Best Top-1 (diagnosis) 57.38 % RAG (ICHD-3 retrieval) · Release 1.0
Best Top-3 (diagnosis) 61.60 % RAG (ICHD-3 retrieval) · Release 1.0
Best Top-1 (group) 91.15 % Few-shot · Release 1.0

Publications

https://doi.org/10.3233/SHTI260922 — 5 results on Release 1.0

Results

Results marked Platform team were produced by the people who run this platform — disclosed because the same team curates the data and verifies results.

Version · badges	Approach	Dataset	Top-1 (diagnosis)	Top-3 (diagnosis)	Top-1 (group)	Top-3 (group)	Cohen's Kappa	Links
google/gemma-3-27b-it Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	RAG (ICHD-3 retrieval) Approach · Retrieval-augmented RAG (ICHD-3 retrieval) For each case, the most similar ICHD-3 diagnosis entries are retrieved by embedding similarity and added to the prompt; the model decides. Model Gemma 3 27B IT google/gemma-3-27b-it Retrieval-augmented approaches For each case, relevant ICHD-3 knowledge is retrieved from a text collection and added to the prompt; the language model decides.	Platform release Release 1.0 A published, checksummed release of this platform's community cases, validated by two independent reviewers. Cases 610 case variants Published 2026-10-22 License CC BY-NC 4.0 Release page and downloads About this result English variants (n=305) of release 1.0; vLLM, temperature 0 (Qwen3: 0.6), runs March 2026. Top-3 from a separate top-3 run; group = ICHD-3 top-level group of the first diagnosis (computed from the paper's run data).	57.38 %	61.60 %	90.49 %	—	—	Paper
google/gemma-3-27b-it Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	Few-shot Approach · Prompting Few-shot Case text, candidate list and worked example cases in the prompt. Model Gemma 3 27B IT google/gemma-3-27b-it Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	Platform release Release 1.0 A published, checksummed release of this platform's community cases, validated by two independent reviewers. Cases 610 case variants Published 2026-10-22 License CC BY-NC 4.0 Release page and downloads About this result English variants (n=305) of release 1.0; vLLM, temperature 0 (Qwen3: 0.6), runs March 2026. Top-3 from a separate top-3 run; group = ICHD-3 top-level group of the first diagnosis (computed from the paper's run data).	57.05 %	61.60 %	91.15 %	—	—	Paper
google/gemma-3-27b-it Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	Zero-shot Approach · Prompting Zero-shot Case text plus the list of candidate ICHD-3 codes; single best diagnosis, no examples. Model Gemma 3 27B IT google/gemma-3-27b-it Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	Platform release Release 1.0 A published, checksummed release of this platform's community cases, validated by two independent reviewers. Cases 610 case variants Published 2026-10-22 License CC BY-NC 4.0 Release page and downloads About this result English variants (n=305) of release 1.0; vLLM, temperature 0 (Qwen3: 0.6), runs March 2026. Top-3 from a separate top-3 run; group = ICHD-3 top-level group of the first diagnosis (computed from the paper's run data).	53.77 %	58.40 %	89.84 %	—	—	Paper
google/gemma-3-27b-it Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	Chain-of-thought Approach · Prompting Chain-of-thought Case text and candidate list; the model reasons step by step before naming the diagnosis. Model Gemma 3 27B IT google/gemma-3-27b-it Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	Platform release Release 1.0 A published, checksummed release of this platform's community cases, validated by two independent reviewers. Cases 610 case variants Published 2026-10-22 License CC BY-NC 4.0 Release page and downloads About this result English variants (n=305) of release 1.0; vLLM, temperature 0 (Qwen3: 0.6), runs March 2026. Top-3 from a separate top-3 run; group = ICHD-3 top-level group of the first diagnosis (computed from the paper's run data).	53.44 %	56.40 %	90.16 %	—	—	Paper
google/gemma-3-27b-it Verified Peer-reviewed publication ↗ Disclosure Produced by the platform team This result was produced by people who also run this platform and published in their own paper. The same team curates the case data, moderates submissions and verifies results, so its own results carry a potential conflict of interest. They are flagged so you can weigh them accordingly and filter them out. The publication	ICHD-3 criteria in context Approach · Prompting ICHD-3 criteria in context The full ICHD-3 ontology (simplified criteria, descriptions, clinical features) is placed in the system prompt. Model Gemma 3 27B IT google/gemma-3-27b-it Prompting approaches A general-purpose language model diagnoses from the case text, steered only by the prompt: zero- or few-shot, chain-of-thought, or with diagnosis lists or ICHD-3 criteria pasted into the prompt.	Platform release Release 1.0 A published, checksummed release of this platform's community cases, validated by two independent reviewers. Cases 610 case variants Published 2026-10-22 License CC BY-NC 4.0 Release page and downloads About this result English variants (n=305) of release 1.0; vLLM, temperature 0 (Qwen3: 0.6), runs March 2026. Top-3 from a separate top-3 run; group = ICHD-3 top-level group of the first diagnosis (computed from the paper's run data).	49.18 %	58.00 %	87.87 %	—	—	Paper

Show this model in the leaderboard