A few days ago, a story broke that should put any company using AI without a clear process behind it on alert. South Africa suspended two senior officials from the Department of Home Affairs after discovering AI-generated hallucinations in the references list of the revised white paper on citizenship, immigration and refugee protection — the second cabinet-approved policy document caught with fabricated sources in less than a week.
This is not an isolated anecdote. It’s the tip of an iceberg that is going to hit many companies that are bringing AI into their processes without fully understanding what they’re doing.
Let’s break down what happened, why it happened, and — most importantly — what you, as a business leader, can learn from it so you don’t end up living your own version of the problem.
What exactly happened
The case is quite revealing. More than 100 fabricated references appeared in the revised white paper on citizenship, immigration and refugees. These were not typos. They were academic citations that simply did not exist — invented by a language model and pasted into an official document approved by cabinet.
The most interesting detail: according to the Department’s initial internal review, the problem appears to be linked to the reference list added to the Revised White Paper. It seems these references were generated and attached to the document afterwards, as they are not cited in the body of the text.
Translation: someone finished the document, asked an AI to generate a pretty bibliography, and pasted it in without checking. The AI did what it does best when it doesn’t have data: it made things up with total confidence.
The government’s response was decisive. Home Affairs hired two independent law firms — one to handle the disciplinary process and another to review every policy document the department has produced since November 30, 2022, the day OpenAI’s ChatGPT was released to the public. The department also committed to designing controls and AI disclosures in its internal approval processes.
And it wasn’t a one-off case. In the same week, the Department of Communications and Digital Technologies announced the immediate suspension of two officials, after withdrawing its long-awaited draft AI policy — for using academic citations that are widely believed to be AI hallucinations.
Four officials suspended. Two official documents withdrawn or challenged. All in one week. All for the same mistake.
What a "hallucination" is and why it happens
A hallucination is when an AI model generates information that sounds perfectly plausible but is false. It’s not a bug. It’s an inherent feature of how current language models work.
LLMs (models like ChatGPT, Claude, and Gemini) don’t “know” things. They predict the next most likely word based on statistical patterns from their training. When you ask them for a list of academic references on a topic, they don’t consult a database of real papers — they generate something that looks like an academic reference. Plausible authors, plausible titles, plausible journals, plausible years. Everything coherent. Everything invented.
The most common hallucinations in professional settings:
- Invented citations and references (the South Africa case)
- False numerical data presented with confidence
- Legal cases that do not exist (lawyers have already been sanctioned for this)
- Software features the system does not actually have
- Internal policies the model makes up when responding to a customer
- Fabricated addresses, phone numbers, or emails
What’s dangerous is not that AI makes mistakes. It’s that it makes them in an expert tone. An AI never says “I’m not sure” — it always sounds like it knows.
Why this matters for your company
You may be thinking: “Okay, but I don’t write white papers for the government.” Fair enough. But think for a moment about where you’re using AI today, or where you plan to use it:
| Use case | Hallucination risk |
|---|---|
| Customer support with a chatbot | The bot invents policies, deadlines, or products |
| Sales proposal generation | Fictional data, prices, or success stories |
| Meeting summaries | Agreements nobody actually made |
| Legal document analysis | Misread or invented clauses |
| Email replies | Commitments the company cannot fulfill |
| Lead research | Incorrect information about prospects |
Each of these cases has its own version of the South Africa scandal waiting to happen. And when it does, it’s not a public official who loses their job — it’s your company that loses a customer, a sale, or gets into legal trouble.
The right reaction is not “stop using AI”
The lesson from Home Affairs is not “AI is useless.” In fact, their own response makes that clear: the Department recognizes the benefits of AI and that it is being widely adopted across society. It is a transformative but disruptive technology that is changing how organizations operate in the private and public sectors. We must adapt to keep pace.
The lesson is something else: AI without a process is a time bomb. The problem in South Africa was not using AI. It was using it without verification, without checkpoints, without someone reviewing the output before it reached cabinet.
How to build AI automations that don’t blow up
This is where things get practical. These are the principles we apply when building AI automations for clients — and that any company should demand from whoever is implementing AI for them.
1. Generative AI should not be the source of truth
A model should not “remember” your business data. If your customer support bot needs to know your return policy, that policy should live in a document the bot retrieves — not in something the model “knows.” This is called RAG (Retrieval-Augmented Generation), and it’s the difference between a bot that answers with your real data and one that makes up an answer.
2. Every critical output needs validation
If AI is going to send an email to a customer, schedule a meeting, or generate a proposal with numbers, you have three options:
- Automated validation: cross-check the output against real data (does this price exist in the catalog? is this date available?)
- Human validation: a person approves it before it goes out
- Constraint-based validation: the AI can only choose from predefined options, not generate freely
3. Traceability: know what the bot did and why
A serious automation logs every decision. If something goes wrong, you need to be able to trace back and understand what happened. “The AI did it” is not an acceptable answer when a customer complains.
4. Clear boundaries on what it can and cannot do
A well-designed bot has a defined perimeter. It knows how to answer questions about hours, products, and prices. When asked something outside its scope, it hands off to a human instead of making up an answer.
5. Testing before production
You don’t launch automation to real customers without testing it against hundreds of cases, including the weird ones. Especially the weird ones — that’s where things fail.
The question you should be asking yourself
If you’re using AI in your business (or thinking about it), ask yourself this:
If tomorrow the AI says something completely false to a customer, who notices, when, and what happens next?
If the answer is “we wouldn’t know until the customer complains,” you have the same problem Home Affairs had before the scandal. The difference is that you still have time to fix it.
How we handle this at Studio SmartWork
We’re not impartial here, so let’s be direct: this is exactly the kind of problem we avoid when building automations for clients.
We don’t deliver “a ChatGPT with your data” and walk away. We design systems with verification, fallbacks, and human oversight at critical points. We use open-source tools like n8n so every step of the process is transparent and auditable — you know exactly what the bot does, what data it checks, and when it escalates to a person.
And when something doesn’t fit — when AI is not the right tool for part of the process — we say so. Sometimes the solution is not a language model. It’s a well-built integration between two systems you already have.
The takeaway
The South Africa scandal is not a story about bad AI. It’s a story about bad process. The AI did exactly what it always does: generate plausible text. The mistake was trusting that output without checking it and putting it into an official document.
The companies that win with AI in the coming years will not be the ones that use it the most. They will be the ones that use it best — with process, with verification, with oversight where it matters and autonomy where it makes sense. The difference between an automation that saves hours and one that creates a scandal lies in how it’s built, not in which model it uses underneath.
If you’re thinking about automating something in your business, don’t ask “which AI should I use?” Ask “how do I make sure that when it gets things wrong, it doesn’t blow up in my face?” That’s the question that separates someone who knows what they’re doing from someone playing with fire.