Researchers just tested ChatGPT on the same test questions as aspiring doctors – and found the AI was 'comfortably within the passing range'
Getty Images
- ChatGPT appeared capable of passing the US medical licensing examination in a research experiment.
- ChatGPT showed "moderate accuracy" and was "comfortably within the passing range," per the research.
- The research is still being peer-reviewed, Axios reported.
Open AI's artificial intelligence chatbot, ChatGPT, appears to be capable of passing all three parts of the United States medical licensing examination, according to researchers.
According to a new research experiment, ChatGPT showed "moderate accuracy" and was "comfortably within the passing range" in the exams.
The research is still being peer-reviewed, a process where professionals analyze their colleagues' work to ensure it's accurate and significant, Axios reported.
"ChatGPT performed at or near the passing threshold for all three exams without any specialized training or reinforcement," the researchers wrote in the paper.
Most of the authors work for Ansible Health, a startup based in Mountain View, California that's been researching ways of using AI to improve healthcare outcomes.
They said the tool was able to demonstrate "a high level of concordance and insight in its explanations" and the results "suggest that large language models may have the potential to assist with medical education, and potentially, clinical decision-making."
However, the researchers did exclude a set of "indeterminate" answers due to ChatGPT appearing to be programmed to avoid providing medical advice, per Axios.
"Those answers were so general that it was hard to say if they were right or wrong," said Morgan Cheatham, one of the authors who is studying medicine at Brown University and is a vice president focusing on healthcare at Bessemer Venture Partners, a venture capital firm.
The first step of the exam is typically taken by medical students who have completed two years of learning; the second by fourth-year medical students who have also completed up to two years of clinical rotations; and step three by postgraduate students, according to the study.
ChatGPT has impressed academics with its ability to produce high-quality essays and process complex subjects. However, some also point out there that the chatbot is susceptible to misinformation and lacks some depth of understanding.
Another AI, developed by AI safety and research firm Anthropic, has passed a university-level law and economics exam, according to an academic at Virginia's George Mason University.
The study is titled "Performance of ChatGPT on USMLE: Potential for AI-Assisted Medical Education Using Large Language Models" and published on medRxiv, an online archive for medical, clinical, and health sciences papers that have yet to be peer-reviewed.