Last month I was reviewing a case from one of our ongoing projects in Bangladesh. A junior doctor in a rural upazila health complex typed a simple prompt into a general purpose LLM: "Summarise this patient's history and suggest next steps for suspected dengue." The output looked clean. It listed symptoms correctly. It even cited a plausible WHO guideline. But buried in the response was a quiet hallucination: the model recommended a drug the patient was allergic to, based on nothing in the chart. The doctor caught it in time. Many would not.
This is not a rare story. It is the everyday reality when we move large language models from demos to deployment in real clinics.
Everywhere I look in the literature right now, experts are publishing careful tutorials and comparative studies on prompt engineering. They show how chain of thought, few shot examples, or meta cognitive prompts can lift accuracy by 10 to 15 percent. That work matters. But after reading the latest papers, I keep coming back to the same uncomfortable truth: prompt engineering is a useful starting point, not a safety solution.
The Illusion of Control
Prompt engineering gives a sense of precision. You specify tone, structure, constraints, even reasoning steps. In controlled settings, this works surprisingly well. Tutorials for clinicians show improvements in clarity, relevance, and even reduced hallucination rates.
Yet these improvements are fragile.
Change the input slightly. Introduce ambiguity. Add real world noise. The same carefully crafted prompt can fail quietly. The model still produces confident outputs, but the guarantees are gone. This is not a bug in prompting. It is a property of how these models work. Prompting does not change the underlying uncertainty. It only reshapes how that uncertainty is expressed.
Change the input slightly. Introduce ambiguity. Add real world noise. The same carefully crafted prompt can fail quietly. The model still produces confident outputs, but the guarantees are gone. This is not a bug in prompting. It is a property of how these models work. Prompting does not change the underlying uncertainty. It only reshapes how that uncertainty is expressed.
What the Recent Research Actually Says
Workum and colleagues (Frontiers in AI, 2025):
They lay out a clear seven step pathway for safe LLM use in healthcare: protect privacy, adapt models with domain knowledge, tune hyperparameters, engineer prompts, separate clinical decision support from non decision support uses, evaluate outputs systematically, and put real governance in place. Notice where prompt engineering sits. It is step four. The authors treat it as necessary but never pretend it is sufficient. They introduce the ACUTE checklist (Accuracy, Consistency, semantically Unaltered, Traceable, Ethical) as the real test. Most prompt only experiments never reach that level of scrutiny.
They lay out a clear seven step pathway for safe LLM use in healthcare: protect privacy, adapt models with domain knowledge, tune hyperparameters, engineer prompts, separate clinical decision support from non decision support uses, evaluate outputs systematically, and put real governance in place. Notice where prompt engineering sits. It is step four. The authors treat it as necessary but never pretend it is sufficient. They introduce the ACUTE checklist (Accuracy, Consistency, semantically Unaltered, Traceable, Ethical) as the real test. Most prompt only experiments never reach that level of scrutiny.
Comparative analysis in BMC Medical Informatics and Decision Making:
Researchers tested three frontier models across six different prompting strategies on genuine clinical scenarios. The best strategy, meta cognitive prompting, improved ethical reasoning and cut safety incidents by almost half. Yet even then, critical safety problems still appeared in more than 11 percent of responses, especially in complex ethical cases. Empathy and communication scores stayed low across the board. The authors concluded that no amount of clever prompting can fix the underlying architectural tendencies toward hallucination, bias, and shallow reasoning. Those problems are baked in.
Researchers tested three frontier models across six different prompting strategies on genuine clinical scenarios. The best strategy, meta cognitive prompting, improved ethical reasoning and cut safety incidents by almost half. Yet even then, critical safety problems still appeared in more than 11 percent of responses, especially in complex ethical cases. Empathy and communication scores stayed low across the board. The authors concluded that no amount of clever prompting can fix the underlying architectural tendencies toward hallucination, bias, and shallow reasoning. Those problems are baked in.
Mental health chatbots and simulation training:
A recent conceptual framework for LLM based mental health tools builds a whole layered safety system: input risk detection, post generation ethical filters, therapist escalation protocols. The reason is simple. Static prompts alone cannot handle crisis signals or cultural nuance. In healthcare simulation training, another guide to prompt design warns that vague or uncalibrated prompts regularly produce fabricated patient histories and biased scenarios. Clinicians are told to verify everything. That advice is honest, but it also reveals the limit. If every output needs a second pair of trained eyes, we have not solved the safety problem. We have only moved it.
A recent conceptual framework for LLM based mental health tools builds a whole layered safety system: input risk detection, post generation ethical filters, therapist escalation protocols. The reason is simple. Static prompts alone cannot handle crisis signals or cultural nuance. In healthcare simulation training, another guide to prompt design warns that vague or uncalibrated prompts regularly produce fabricated patient histories and biased scenarios. Clinicians are told to verify everything. That advice is honest, but it also reveals the limit. If every output needs a second pair of trained eyes, we have not solved the safety problem. We have only moved it.
Why We Keep Acting Like Prompts Are Enough
Part of it is practical. Prompt engineering is cheap and fast. You can iterate in minutes without touching the model weights or setting up governance committees. In low resource settings like ours in Bangladesh, that speed feels like a lifeline. But speed without safeguards creates exactly the equity problems my own work tries to address. A model that performs well on well phrased English prompts from urban hospitals often fails quietly on Bangla mixed clinical notes or on patients from ethnic minorities. Prompt tweaks can mask the gap for a while. They cannot close it.
From My Research:
This is where my own research keeps pulling me. In our under review work on fairness aware representation learning for biosignals and on hybrid explainable systems for maternal health risk, we keep seeing the same pattern. Pure prompt tricks improve surface performance but leave the deeper failure modes untouched, especially when modalities are missing, when data is sparse, or when the stakes are highest. That is why we are exploring rule augmented neuro symbolic layers and clinician validated uncertainty signals. These are not replacements for good prompting. They are what comes after it.
This is where my own research keeps pulling me. In our under review work on fairness aware representation learning for biosignals and on hybrid explainable systems for maternal health risk, we keep seeing the same pattern. Pure prompt tricks improve surface performance but leave the deeper failure modes untouched, especially when modalities are missing, when data is sparse, or when the stakes are highest. That is why we are exploring rule augmented neuro symbolic layers and clinician validated uncertainty signals. These are not replacements for good prompting. They are what comes after it.
Real world evaluation studies:
A recent preprint on large scale LLM testing in healthcare calls for a new "RWE LLM" paradigm that treats deployment as an ongoing safety experiment rather than a one time prompt optimisation exercise. The authors argue we need continuous monitoring, traceable decision logs, and structured human oversight loops that persist long after the initial prompt has been tuned.
A recent preprint on large scale LLM testing in healthcare calls for a new "RWE LLM" paradigm that treats deployment as an ongoing safety experiment rather than a one time prompt optimisation exercise. The authors argue we need continuous monitoring, traceable decision logs, and structured human oversight loops that persist long after the initial prompt has been tuned.
What Actually Works
Practical Lessons for Researchers and Clinicians
- Treat prompt engineering as the entry ticket, not the destination. It gets you started but does not finish the job.
- Build in domain adaptation using retrieval augmented generation with verified local guidelines, not just web scrapes.
- Separate use cases: clinical decision support needs formal validation and regulatory thinking; administrative tools can be lighter but still require traceability.
- Implement structured evaluation frameworks like ACUTE or similar checklists every single time the model touches a patient record.
- Put governance in place before deployment: a small multidisciplinary team that owns monitoring, feedback, and model updates.
- Design for contestability. Every high stakes output should carry an easy path for a clinician to challenge it and feed that challenge back into the system.
None of this is glamorous. It will not produce flashy benchmark numbers next week. But it is the only path that respects the reality of clinical work. Patients are not prompts. Errors are not just statistical noise.
A Final Thought
I keep thinking about the doctor in that rural upazila who caught the allergy mistake. She succeeded not because the prompt was perfect, but because she never fully trusted the machine in the first place. Our job as researchers is to build systems that earn that careful trust instead of demanding it.
Prompt engineering gets us through the door. Rigorous validation, privacy preserving architectures, and continuous real world monitoring are what keep the door open safely.
Prompt engineering gets us through the door. Rigorous validation, privacy preserving architectures, and continuous real world monitoring are what keep the door open safely.
References
Workum, J. D., van de Sande, D., Gommers, D., & van Genderen, M. E. (2025). Bridging the gap: a practical step by step approach to warrant safe implementation of large language models in healthcare. Frontiers in Artificial Intelligence, 8, 1504805.
Esmaeilzadeh, P. (2025). Ethical implications of using general purpose LLMs in clinical settings: a comparative analysis of prompt engineering strategies and their impact on patient safety. BMC Medical Informatics and Decision Making, 25, 342.
Liu, J., Liu, F., Wang, C., & Liu, S. (2025). Prompt engineering in clinical practice: tutorial for clinicians. JMIR, 12, e72644.
Boit, S., & Patil, R. (2025). A prompt engineering framework for large language model based mental health chatbots: conceptual framework. JMIR Mental Health, 12, e75078.
Mackay, P. (2025). Compiling complex medical reports using a generic LLM: exploring hybrid prompt engineering strategies for language improvement and hallucination mitigation. [Thesis]. University of Lincoln.
Maaz, S., Palaganas, J. C., Palaganas, G., & Bajwa, M. (2025). A guide to prompt design: foundations and applications for healthcare simulationists. Frontiers in Medicine, 11, 1504532.
Bhimani, M., Miller, A., Agnew, J. D., et al. (2025). Real world evaluation of large language models in healthcare (RWE LLM): a new realm of AI safety and validation. medRxiv preprint.