How reliable are medical recommendations from chatbots. Oxford study

The Oxford study revealed that people struggle to obtain useful health advice from chatbots.

Long queues and rising costs in overburdened healthcare systems are driving many to turn to AI chatbots, such as ChatGPT, for self-diagnosis. Approximately one in six American adults uses chatbots to get medical advice at least once a month.

However, excessive trust in such technologies can be risky. According to the study led by Oxford, users often find it difficult to determine what information to provide in order to receive accurate medical recommendations.

The study involved approximately 1,300 participants from the United Kingdom. Participants were given medical scenarios created by a group of doctors and asked to identify possible illnesses using both chatbots and their own methods.

The experiment used GPT-4o, Command R+ from Cohere, and Llama 3 from Meta, which was previously used in the Meta AI assistant. The chatbots not only reduced the likelihood of correctly identifying the disease but also increased the risk of underestimating its severity.

The study was published amid the active integration of AI into healthcare. Technology companies are promoting AI as a way to improve medical processes. For example, Apple is developing a tool that will provide advice on physical activity, nutrition, and sleep; Amazon is exploring the analysis of medical data to identify "social determinants of health"; and Microsoft is helping create AI that sorts patient messages for doctors.

However, the professional community and patients differ in their assessment of AI's readiness to perform more complex medical tasks. The American Medical Association recommends that doctors not use chatbots like ChatGPT for clinical decision-making, and major AI companies, including OpenAI, warn against using their technologies for self-diagnosis.

Existing methods for evaluating chatbots do not reflect the full complexity of their interaction with real users. Just as with clinical trials of new drugs, AI chatbots should be tested in real-world conditions before widespread adoption.

Adam Mahdi, Director of the Master's Program at the Oxford Internet Institute
The study identified problems in two-way communication. Those who used chatbots did not make better decisions compared to participants who relied on traditional methods, such as online searches or their own judgment.

Participants often omitted important details when consulting chatbots or received answers that were difficult to interpret.

The responses they received often contained both useful and erroneous recommendations. Existing methods for evaluating chatbots do not account for the full complexity of interaction with real users.

Source: TechCrunch