The Government of Newfoundland paid private consulting firm Deloitte $1.6 million for a Health and Human resources plan that contains fabricated sources—reminiscent of the hallucinations of Artificial Intelligence (AI).
This would be scandalous enough if it were a first-time offense, but it’s actually the second high profile incident where one of the big four consulting firms has been accused of such practices. Only one month ago, Deloitte was forced to refund the Australian government after admitting it used generative AI to produce a report “riddled with fake citations, phantom footnotes, and even a made-up quote from a Federal Court judgment.”
While critics have rightly used the incidents to call for greater oversight over the use of AI in government commissioned reports, perhaps a more important question is why governments are forced to rely on private consultants like Deloitte to research and design public policy in the first place.
Authors Chris Hurl and Leah Werner argue that governments increasingly find themselves in “the consulting trap.” Decades of down-sizing and outsourcing of public services—often on the advice of private consultants—have left governments bereft of the expertise and capacity required to design and deliver public policy on their own. Instead, these functions are outsourced to private consultants like Deloitte. As governments come to rely on these firms more and more for even basic functions like policy research, they can become ‘trapped’ in a cycle of dependence.
In comparison to our hollowed-out civil service, private consulting giants’ claim to possess insider knowledge and specialized expertise that governments simply cannot replicate. Consultants’ expert advice usually hinges on some carefully guarded formula or metric that simply cannot be revealed to the public lest these firms lose their commercial advantage over rivals. However, these metrics are often deeply flawed.
For example, consulting firms use secret ‘risk assessments’ to calculate how much governments should pay private firms for assuming the risk of cost-overruns in public-private-partnerships (P3). When actually revealed to the public via government audits, these amount to little more than “blue-sky” sessions where private sector bidders merely imagine any possible cost-overrun with no regard for evidence, inflating the risk premiums government must pay to the private partner. Similarly, the use of standardized, cookie-cutter reports by consultants that are shopped to different municipalities with no regard for local context or history is yet another example that there often isn’t much substantive research or analytical rigour behind the curtain.
The recent AI incidents are only further confirmation that consulting firms—for all their pretense of specialized knowledge and expertise—are more concerned with executing their government contracts in the cheapest, most cost-effective way possible, regardless of its impact on the quality of the final product. The deep irony here is that consulting firms like Deloitte are some of the biggest boosters of AI in government, with dedicated units offering their paid advice to transform public services and government operations through AI.
Yet these recent incidents in Newfoundland and Australia betrays a fundamental misunderstanding by Deloitte of what certain AI tools can and cannot do. The kind of large language model that Deloitte admitted to using in the Australian case (Azure OpenAI GPT – 40) is designed to predict the probability of words in sequence, not to distinguish between facts and non-facts. It is programmed to produce an answer, even if it doesn’t have strong evidence for the correct answer—hence the tendency to hallucinate or fabricate
People treat LLM’s like infallible search engines retrieving facts from their data repositories, rather than the word prediction machines they actually are. While we can excuse the ignorance of the general public as it grapples with this novel technology, it is difficult to be as sympathetic to a multinational consulting firm that has staked its future on supposedly knowing more than the rest of us about how AI can be deployed.
The only possible conclusion we can take from these incidents is either Deloitte doesn’t understand how LLM’s work and used it to create a government commissioned report without fact-checking, or it knew full well that there was the possibility that the LLM might fabricate parts of the report and it did not care to check. Either conclusion is equally damning for the consultancy.
Is this the kind of expertise we want our governments to rely on as we navigate a technologically uncertain future? Perhaps this incident will be the wake-up call needed for our governments to take a good look behind the curtain and break free from the consulting trap.
