With artificial intelligence being adopted rapidly, varied domains are increasingly looking for ways to implement the technology in order to improve efficiency and save time. We see a vast use of AI assistants for smoother processes. It has not been long since the FDA rolled out Elsa, its generative AI tool that was meant to be a forward-thinking move, to help improve drug and medical device approval workflows, but it seems like the platform is not as smooth as we would hope. Reports from the company's former and current employees are coming forward about the tool hallucinating and often misrepresenting real research.
FDA’s Elsa AI draws backlash for hallucinating studies, but is the real problem human oversight, and not the tech itself?
Just like the world at large is evolving, healthcare and regulatory tech are also inclined towards adopting generative AI to help streamline processes. The U.S. Food and Drug Administration introduced Elsa, an AI tool to help make drug and device approval seamless by cutting down on bureaucratic delays and enhancing overall efficiency by reducing administrative workloads. However, what was meant to be a technological leap turned out to be a deeply concerning move, raising questions regarding overdependence on technology and human negligence.
While everything looked revolutionary on paper, given how efficiently life-saving treatments could be reviewed, the internal staff at the FDA laid out the true details about the tool. According to the report by CNN, many former and current employees have expressed disquiet about the generative AI tool hallucinating, generating inaccurate summaries, and even fabricating entire clinical studies, as references to these papers do not even exist. At least three employees have revealed the troubling tendency of the tool and how it would go to the extent of distorting actual research findings. This seems especially concerning, given that the assistant's very purpose is to save time without having to be supervised.
FDA leadership did not show much urgency when the issue surrounding Elsa was brought to their attention. They were also quick to point out that the staff is not required to use Elsa and that implementing the tool is completely voluntary. The staff are not made to attend the training sessions unless they want to. This means no systems are in place to ensure the tool is used properly and safely.
AI hallucinations are not rare occurrences, and these glitches sometimes occur due to updates or changes. However, if the tool is being used in high-stakes settings, then proper safeguards need to be in place. The issue is not with some flaws coming up in the AI tool but rather its use without enough human supervision, especially in a field where small errors can have serious consequences for people's health.
Follow Wccftech on Google to get more of our news coverage in your feeds.
