An AI transcription app, also used by hospitals, 'hallucinates' and transcribes fictional sentences

The speech transcription application that works with the help of artificial intelligence Visper (Whisper) of the technology giant OpenAI tends to write entire fictional sentences into the text during operation.

More than a dozen software engineers, programmers and academic researchers say that the new tool, which OpenAI advertised at launch as working exactly like a human, "hallucinates" and inserts racial comments, violent rhetoric, even fictional medical treatments into the text.

Experts say such fabrications are problematic because Visper is used in a range of industries around the world to translate and transcribe interviews, generate text in popular consumer technologies and create subtitles for videos.

Even more troubling, they say, is the rush by medical centers to use such tools to transcribe patient consultations with doctors, despite OpenAI's warnings that the tool should not be used in "high-risk domains."

The full extent of the problem is hard to see, but researchers and engineers say they have often encountered "hallucinations" of the Wisper. One University of Michigan researcher who conducted the study said he found "hallucinations" in as many as 80 percent of the transcriptions he examined.

One machine learning engineer said he immediately detected "hallucinations" in half of the more than 100 hours of Visper transcriptions he analyzed, and one programmer found "hallucinations" in nearly every one of the 26.000 transcripts he made using Visper.

Problems persist even with well-recorded short audio samples. A recent study by computer scientists found 187 "hallucinations" in more than 13.000 clear audio clips they examined.

That trend would lead to tens of thousands of erroneous transcriptions across millions of recordings, the researchers said.

Such mistakes can have "serious consequences," especially in hospitals, said Alondra Nelson, who until last year led the White House Office of Science and Technology.

"Nobody wants a misdiagnosis," said Nelson, a professor at the Institute for Advanced Study in Princeton, New Jersey. "The criterion should be higher".

Visper is used to create subtitles for the deaf and hard of hearing, who are particularly at risk of mistranscriptions because they have no way to identify fabrications "hidden in texts," said Christian Vogler, who is deaf and directs Gallaudet University's Technology Access Program.

The prevalence of such "hallucinations" has led experts, activists and former OpenAI employees to call on the US government to consider AI regulations. At the very least, they said, OpenAI should solve the problem.

"It seems solvable if the company is willing to prioritize it," said William Saunders, a San Francisco-based research engineer who left OpenAI in February over concerns about the company's direction. "It is problematic if it is left, and people believe too much in what that tool can do and integrate it into all other systems," he believes.

An OpenAI spokesperson said the company is continuously studying how to reduce "hallucinations" and appreciates the researchers' findings, adding that OpenAI incorporates feedback into model updates.

While most developers assume that transcription tools misspell words or make other mistakes, engineers and researchers said they've never seen another AI transcription tool that "hallucinates" as much as Visper.

That tool is integrated into some versions of OpenAI's leading chatbot ChatGPT and built into the cloud offering of Oracle and Microsoft computing platforms that serve thousands of companies around the world. It is also used to transcribe and translate text into multiple languages.

In the last month alone, one recent version of Visper has been downloaded over 4,2 million times from the open source AI platform HuggingFace. Sanchit Gandhi, a machine learning engineer, said that Visper is the most popular open source speech recognition model and is embedded in everything from call centers to voice assistants.

Professors Allison Keneke of Cornell University and Mona Sloan of the University of Virginia examined thousands of short clips obtained from TalkBank, Carnegie Mellon University's research repository. They found that nearly 40 percent of the "hallucinations" were harmful or troubling because the speaker could be misinterpreted or misrepresented.

In the example they discovered, one speaker said, "He, the boy, wanted to—I'm not sure—take an umbrella," but the transcription software added, "He took a big part of the cross, a small, small part... Sure himself that he didn't have a knife for terrorism, so he killed a large number of people".

The speaker on the second recording described "two other girls and a lady", and Visper made up an additional comment about race, adding: "two other girls and a lady, um, who were black".

In the third transcription, Visper invented a drug called "hyperactivated antibiotics".

Researchers aren't sure why Visper and similar tools "hallucinate," but software developers say it's usually due to pauses in speech or mixing in background sounds or music.

OpenAI recommended on social media that Visper should not be used "in decision-making, where deficiencies in accuracy can lead to pronounced deficiencies in results."

However, that warning hasn't stopped hospitals and health centers from using speech-to-text models, including Visper, to transcribe what's being said during exams to free medical professionals from taking notes and writing reports.

Over 30.000 clinicians and 40 health systems, including the Mankato Clinic in Minnesota and Children's Hospital Los Angeles, use the Visper-based tool made by Nabla, which has offices in France and the US.

That tool was fine-tuned for medical parlance, said Nable's chief technology officer, Martin Reson.

Company officials said they are aware that Visper can "hallucinate" and are mitigating the problem.

The problem is that it's impossible to compare Nabla's AI-generated transcript with the original recording because Nabla deletes the original audio for "data security reasons," Reson said.

Nabla says her tool has been used to transcribe an estimated seven million conversations during medical appointments.

Saunders, a former OpenAI engineer, said the deletion of the original audio could be a concern if transcripts aren't double-checked or clinicians can't access the recording to confirm they're accurate.

Nabla says no model is perfect, and theirs currently requires doctors to quickly edit and approve transcribed notes, but that could change.

Since conversations between patients and doctors are confidential, it is difficult to know the impact of AI-generated transcripts.

California Rep. Rebecca Bauer-Kahan said she took one of her children to the doctor this year and refused to sign the health network's form asking for her permission to share recordings of conversations with the doctor with companies including Microsoft Azure, a "cloud" computing system. led by OpenAI's largest investor. Bauer-Kahan did not want intimate medical conversations shared with tech companies.