OpenAI's LLM 4.0 passed the Clinical Neurology exam, answering 85% of questions correctly from the American Board of Psychiatry and Neurology. The study authors believe that with some improvements, LLMs could find widespread use in clinical neurology.

The results of the experiment, conducted by a team of researchers from Heidelberg University Hospital and the German Cancer Research Center, were published on December 7. The test, which took place on May 31, involved two LLM models - ChatGPT 3.5 and the more recent ChatGPT 4.0. For the experiment, the researchers used a neuroscience question bank from the American Board of Psychiatry and Neurology, as well as a small group of questions from the European Board.

While the older version of ChatGPT scored 66.8% correct, answering 1,306 out of 1,956 questions correctly, the newer model, ChatGPT 4.0, scored 85% with 1,662 correct answers. It is worth noting that ChatGPT 4.0 outperformed regular users in questions related to behavior, cognition and psychology, and passed the neuroscience exam, as 70% correct answers are considered a passing grade in educational institutions. However, both models showed lower performance in tasks requiring “higher order thinking.”

The researchers who conducted the experiment believe that these results support the promise of using LLMs in clinical neurology, with some modifications. However, they also note that there are still some limitations. The implementation of LLMs in clinical neurology as a documentation and decision support system requires caution, as they are still not mature enough to solve cognitive problems.