Fake Cases Raise Real Risks as AI Spreads in Canada’s Legal System

Experts warn that fake or mistaken case citations generated by so-called AI hallucinations can erode confidence in the justice system.Summit Art Creations/Shutterstock

In one Toronto court, a lawyer cited cases that simply don’t exist. In British Columbia, another lawyer used made-up citations to back a legal argument. In both cases, the judges delivered sharp warnings: Check your use of AI.

Artificial intelligence is moving rapidly into Canadian law offices and courtrooms, reshaping how legal work is done. Lawyers use AI to speed up research, summarize case law, draft filings, and review contracts, saving hours or even days of billable time. AI is lowering costs, accelerating work, and could improve access to legal services.

But those same tools are introducing new types of risks into a system that has depended on accuracy and accountability since the days of quills and ink bottles. Fake or mistaken case citations generated by so-called AI hallucinations can erode confidence in the justice system. Other risks include confidentiality breaches and AI’s ability to generate arguments faster than lawyers can verify them.

“We need to be careful in order to make sure that the fundamental ideas of the justice system are preserved in light of these new tools,” Katie Szilagyi, an associate member at the University of Ottawa Centre for Law, Technology and Society, told The Epoch Times in an interview.

High-Stakes Mistakes

Much of the concern centres on how AI systems produce information that appears authoritative even when it’s completely wrong.

“Large language models are trying to mimic human language, and are going to do that in as convincing a way as possible—without necessarily paying close attention to the facts,” Szilagyi said.

Platforms such as LexisNexis and Westlaw now use AI features to streamline legal research, draft documents, summarize case law, and generate preliminary arguments more quickly and efficiently.

But large language AI models, which underpin tools like ChatGPT, can produce convincing but misleading or even outright false information. This AI “hallucination” phenomenon has already resulted in a number of high-stakes errors in court.

Szilagyi compares AI tools to “a research assistant who is really eager and really smart... [but] when push comes to shove, they might make things up to try to impress you.”

AI hallucinations were a factor in two civil cases in 2024—Ko v. Li and Zhang v. Chen—where submissions to the courts included fabricated citations.

In Ko v. Li, parts of one party’s submissions cited case law that was found to possibly have been AI-generated and did not actually exist.

In Zhang v. Chen, fabricated citations were found in otherwise valid legal arguments, regarded as another early Canadian example of AI risks in litigation and the legal field.

Proceedings continued in both cases without being thrown out, but judges issued warnings in both cases and increased scrutiny.

The ChatGPT app seen in this photo illustration. Oleksii Pydsosonnii/The Epoch Times

Even small inaccuracies or slightly misleading citations can have a major impact in legal proceedings.

A wrong date, misplaced clause, or false citation can fundamentally invalidate or undermine a legal argument, contract, or court document.

Noah Waisberg, co-founder of the legal AI company Kira Systems, said the efficiency gains from AI are significant, reducing time for routine tasks such as reviewing contracts. However, he cautioned that time is needed for verification.

“I think the best practice right now is [that] AI is kind of imperfect in a lot of situations. And so to the extent to be perfect, then you should double check what the AI is doing,” Waisberg said in an interview.

Automation Bias

The University of Ottawa’s Szilagyi said some of the most significant risks of using AI in the legal profession are psychological.

She pointed to “automation bias” as a well-documented tendency for people to be overly trusting of output from AI and assume it is accurate.

“When we interact with computer automation, we are more likely to just take the results … at face value,” she said, adding that AI users often “allow the technology to drive” decision-making and conclusions.

This can lead to complacency, according to Szilagyi, who said users may stop critically evaluating the results AI gives them, especially if it usually produces accurate output.

This level of trust and dependency can lead to a longer-term erosion of legal skills and lapses in professional judgment, where lawyers and paralegals may not “properly vet the product” they get from AI, she said.

She added that this can particularly affect junior lawyers, since routine tasks formerly done by the newest members of the firm are increasingly outsourced to AI. That could lead to a weakening of foundational skills and knowledge that is usually built up over time and during in-person assessment, she said.

Responsibility

Courts have emphasized the obligation of lawyers to verify all citations, and judges have clarified that using AI does not diminish legal liability.

“If you’re a lawyer and you saw something wrong and it’s messed up, that’s on you,” Waisberg said.

Szilagyi echoed that point. While AI can speed up many tasks, she said, it is still entirely the responsibility of the individual using it in a legal or courtroom setting.

“If you’re using a technological tool that’s supposed to make your job better and easier, you really better understand how that tool works and be responsible for it,” she said.

And as AI develops rapidly, the types of risks could change with it.

Systemic risks to the justice system could include the possibility of fabricated evidence, AI-generated “deepfake” videos, and potential bias in AI systems, depending on what data they are trained on.

“We’re beginning to see discussion now about the dangers of fabrication of evidence with AI,” she added.

Judges bench at the Edmonton Law Courts building, in Edmonton in a file photo. The Canadian Press/Jason Franson

Advances in AI have made it easier to create highly convincing text, images, video, and audio, raising new concerns about the reliability of evidence. In areas such as family law, where cases often rely on text messages and other digital communications, verifying authenticity is critical. Judges and lawyers are increasingly required to scrutinize such evidence more closely, Szilagyi said.

She said bias in AI systems is another concern, since models are trained on datasets that can include biases and inaccuracies. This concern has been raised by many legal scholars, including the Governance Studies Media Office at the Brookings Institution.

An additional systemic issue is the potential for a two-tier justice system in which wealthier individuals can access human legal representation, while lower-income individuals may be more reliant on AI and free large language model tools.

Courts Respond

Canadian courts have started issuing guidance on AI use, including rules around verification and disclosure.

Szilagyi noted that there have been a variety of best-practice guidelines, with some saying lawyers don’t have to disclose every instance of AI use but should ensure they’re “always keeping a human in the loop.”

A Federal Court of Canada notice on Dec. 20, 2023, requires disclosure when AI is used to generate content in court filings, emphasizing that lawyers are responsible for verifying accuracy. However, the court’s chief justice later observed that only “three or four” of roughly 28,000 filings in 2024 disclosed use of AI.

Provincially, several courts have also responded. In 2023, the Court of King’s Bench of Manitoba, the Supreme Court of Yukon, and courts in Alberta and Nova Scotia issued guidelines emphasizing that all court submissions must be verified for accuracy, while stopping short of requiring disclosure of AI use.

More recently, Ontario’s Superior Court of Justice put out new AI rules late last year and again this February highlighting the need for “transparency, accuracy, and accountability” in the use of AI to help in court filings.

As the technology spreads, the challenge for courts and lawyers may be less about whether to use AI, and more about how to maintain trust in a system designed for human judgments.

Szilagyi said the stakes go beyond efficiency or error rates.

“Law is built by narratives that we tell about who we are and what we tolerate and what kind of society we want to have,” she said.