This copy is for your personal, non-commercial use only. Distribution and use of this material are governed by our Subscriber Agreement and by copyright law. For non-personal use or to order multiple copies, please contact The Epoch Times Reprints.
A report by Ontario’s auditor general found that artificial intelligence (AI) scribe systems used by doctors for note-taking demonstrated inaccuracies and hallucinations during testing.
The report, which examined the use of systems designed for the Ontario AI Scribe Program across the Ontario Public Service, found inaccuracies in the medical notes generated by most of the approved vendors’ AI Scribe systems. Errors included incorrect or incomplete information, and AI hallucinations—the phenomenon of AI systems generating information that appears to be credible but is partially or wholly inaccurate.
It also found that 11 of the 20 approved vendors did not submit required third-party audit reports, while five of the vendors did not submit threat risk or privacy impact assessments.
The report comes amid widespread concerns about AI hallucinations as institutions increasingly adopt AI systems to improve efficiency or overcome personnel shortages. In one recent instance, a Wall Street law firm submitted a motion that contained “inaccurate citations and other errors” caused in part by AI hallucinations. The motion was used in the proceedings of a fraud and money laundering case.
The Ontario program, which was officially launched on June 3, 2025, was developed through a partnership between the Ontario Ministry of Health, Ontario Health, OntarioMD, and Supply Ontario, with the aim of reducing the administrative burden on clinicians by using AI to automate clinical documentation.
OntarioMD (OMD) is a subsidiary of the Ontario Medical Association that supports community-based clinicians in adopting digital health technologies. OMD worked with Supply Ontario to procure and implement approved AI scribes.
Hallucinations and Inaccuracies
The audit report found that during testing, nine of the 20 vendors’ systems would make up information and make treatment suggestions not in line with the simulated recordings they were being tested on.
Seventeen of the systems would also miss key details about the patient in the test scenario.
The report said that, in a real-world scenario, such inaccuracies could result in inadequate or harmful treatment plans. It suggested further testing to ensure the quality of system-generated notes and to minimize inaccuracies.
Separately, the report found that bias risks in the systems were not properly evaluated. According to the report, bias can occur through misinterpretations of conversations, particularly if either speaker has a diverse accent.
The presence of such bias could impact the fairness, equity, and reliability of the system, the report said.
Testing
The report said vendors were not required to provide live demos of their systems and were instead allowed to generate structured notes offline.
This way of testing does not demonstrate the system’s actual capabilities, limitations, or real-time performance, the report said. The report recommended that Supply Ontario require mandatory live demonstrations for all future AI procurements in order to assess their real-word effectiveness.
