Generative artificial intelligence (AI): a key innovation or just hype in primary care settings?

Challenges and risks

Despite the potential benefits, integrating GenAI into clinical practice presents several challenges. One major concern is the risk of ‘AI hallucinations’, where the GenAI generates inaccurate or misleading information.1 Over-reliance on AI outputs can undermine clinical judgement, potentially leading to diagnostic or treatment errors.3 While GenAI can process vast amounts of data quickly, it cannot replace the critical thinking and expertise that clinicians provide. Balancing GenAI support with clinical oversight is essential to ensure patient safety.

Another challenge is algorithmic bias.7 AI models like GPT-4, though valuable for clinical decision support, can perpetuate racial and gender biases across healthcare tasks. A study by Zack showed significant bias in GPT-4’s clinical vignettes, differential diagnoses and treatment recommendations.7 To mitigate this, AI models must be developed and fine-tuned with diverse datasets that represent different demographic groups, promoting equity in healthcare delivery.8

Patient privacy and data security also major concerns. Many GenAI tools used in healthcare are driven by private, for-profit companies, raising questions about the handling and protection of sensitive health data.9 Without clear privacy regulations and transparent data policies, patient trust in AI-driven healthcare systems could erode, leading clinicians to be hesitant about adopting these technologies.

Finally, there is concern over how GenAI is used inconsistently across clinical practice. Some GPs may over-rely on AI tools, while others might avoid them due to uncertainty about their reliability. To address this, validation studies—including expert reviews, clinician-led trials and patient-centred feedback—should be prioritised. These studies will help build the evidence base needed to optimise GenAI use and ensure its safe integration into real-world clinical settings.

Back to top

Future directions

While the study by Blease et al offers valuable insights, it also has limitations that restrict the broader applicability of the findings. These limitations underscore the need for more comprehensive, real-world data. The study also highlights key risks but falls short of providing actionable recommendations to mitigate them. Future research should move beyond merely identifying risks and focus on developing frameworks that ensure the safe and ethical use of GenAI in healthcare.10 This includes training clinicians to effectively use GenAI as a support tool and conducting long-term evaluations of its impact on clinical outcomes and healthcare disparities. Further, regulatory authorities need to establish frameworks that incorporate principles of transparency, data security, risk management and clinical evaluation metrics to ensure any GenAI applications are safe to use and effective.11

Unsafe LLM-based applications, as noted by Blease et al, are already entering the market with limited validation. Without careful implementation, there is a risk that GenAI could be misused or relied on improperly. Ensuring the safe integration of GenAI into healthcare requires not only technical refinement but also ethical oversight and evidence-based practices. With responsible implementation, GenAI has the potential to revolutionise healthcare delivery, improving clinical outcomes and patient safety.1

Back to top