Performance and concordance of contemporary large language models on Royal Australasian College of Physicians examination questions: an exploratory analysis
Abstract We assessed six publicly available large language models (LLM) on Royal Australasian College of Physicians (RACP) adult medicine written examination questions. In a 30‐question set with answers, models achieved 93% correct responses overall, and accuracy was higher for text‐based than visual‐media questions. In a 170‐question exploratory set without an answer key, complete inter‐model agreement occurred in 86% of questions but was lower for visual‐media than text‐based questions and for scenario‐based than knowledge‐based questions. Contemporary LLM performed strongly on RACP‐style written questions, but interpretation of visual media and complex clinical vignettes remains a limitation.
Rohan Joshua Krishnaswamy, Joseph Julian Suttie. Performance and concordance of contemporary large language models on Royal Australasian College of Physicians examination questions: an exploratory analysis. // Internal Medicine Journal. – 2026. – DOI: 10.1111/imj.70664. – URL: https://onlinelibrary.wiley.com/doi/10.1111/imj.70664?af=R
Rohan Joshua Krishnaswamy & Joseph Julian Suttie (2026). Performance and concordance of contemporary large language models on Royal Australasian College of Physicians examination questions: an exploratory analysis. Internal Medicine Journal. https://doi.org/10.1111/imj.70664
Rohan Joshua Krishnaswamy and Joseph Julian Suttie. “Performance and concordance of contemporary large language models on Royal Australasian College of Physicians examination questions: an exploratory analysis.” Internal Medicine Journal, 2026. doi:10.1111/imj.70664.
@article{Krishnaswamy2026,
title = {Performance and concordance of contemporary large language models on Royal Australasian College of Physicians examination questions: an exploratory analysis},
author = {Krishnaswamy, Rohan Joshua and Suttie, Joseph Julian},
journal = {Internal Medicine Journal},
year = {2026},
doi = {10.1111/imj.70664},
url = {https://onlinelibrary.wiley.com/doi/10.1111/imj.70664?af=R}
}
Abstract Background Macrophage polarisation plays a crucial role in sepsis‐associated inflammatory responses; however, the underlying molecular regulatory mechanisms remain to be further explored. Aim This study aimed to explore the...