AI glitch or poor management?
Without human oversight, artificial intelligence can cause significant damage. This is evident from the AI glitches that have come to light in recent months. In his new book, Dr. Jens-Uwe Meyer, an expert in AI management, explains how companies can protect themselves against this.
Fictitious studies and court rulings, millions in losses due to inaccurate forecasts, erroneous wire transfers initiated by deepfake CFOs, and databases deleted by AI agents—at first glance, the most spectacular AI glitches of recent months all appear to be cases of technological failure. But a closer look reveals a different pattern. It is not artificial intelligence that is spiraling out of control; rather, people and companies are losing control over its use.
That is why AI management must evolve into a new leadership discipline, writes Dr. Jens-Uwe Meyer in his new book, *AI Management in Practice*. For this book, the CEO of DICIS AG in Leipzig examined real-world AI failures at companies. He was less interested in individual failures than in the patterns behind them. And his analysis revealed that the most spectacular cases can be traced back to a few recurring mistakes. That is why he concludes: «There are no AI failures. There is only poor AI management.»
Pattern 1: AI results are not adequately verified
According to Meyer, generative AI has an unpleasant trait: it can convincingly spout nonsense. This problem is now widely known. Nevertheless, incorrect AI outputs often still find their way, unchecked, into reports, legal briefs, customer communications, and other business-critical processes.
For example, as Meyer writes, the renowned auditing firm KPMG had to retract a report on the use of AI in companies after several organizations mentioned in it vehemently disputed the claims made therein. And the consulting firm Deloitte had to revise a report prepared for the Australian Department of Labor that, among other things, contained nonexistent sources and fabricated court citations. These are just two examples among many. The common thread in all these cases, according to Meyer, is this: it is not the AI’s “hallucinations” that are the real problem. What matters is that the result subsequently passed through a human review and approval process—without the “hallucinations” being detected.
In his book, Meyer describes the problem using a simple thought experiment: «Imagine a co-pilot who hits the runway perfectly nine times out of ten—but on the tenth approach, misses it so badly that it nearly causes an accident.»
No airline would let a co-pilot like that fly alone without supervision just because he’s usually right. With AI, however, that’s exactly what often happens—«even though AI is an excellent assistant, it’s no substitute for critical thinking, expert review, and human responsibility.».
Example 2: The AI is given too much freedom
AI agents give rise to a second category of risk. Modern AI no longer just generates text or images. It is increasingly capable of acting autonomously: processing databases, running programs, modifying files, sending emails, or executing entire process steps. This shifts the focus of management. It is no longer enough to ask: What answers is our AI allowed to give? Companies must ask themselves: What is it actually allowed to do?
According to Meyer, the 2026 case involving the software company PocketOS illustrates just how quickly problems can arise. An AI assistant was supposed to help with a task in a test environment. When a problem arose, the AI independently searched for a solution. In doing so, it used access credentials that allowed it to perform more extensive actions and accidentally deleted the production database—including the backup copies stored there. In his book, Meyer sums up the problem simply by stating that the crucial question is therefore: What is the AI actually technically allowed to do—and not just what we’ve told it it’s allowed to do?
According to Meyer, professional AI management thus «applies principles that companies take for granted when it comes to human employees to artificial intelligence.» Not every employee is granted administrator rights. Not everyone is authorized to approve payments of any amount. And not everyone is allowed to delete sensitive company data. The more autonomous AI becomes, the more important this principle becomes—including in the deployment of AI.
Pattern 3: Companies Underestimate the New Risks
The third pattern goes beyond classic AI errors. Artificial intelligence creates new possibilities—and with them, new forms of abuse. One particularly spectacular case described by Meyer involved the British engineering firm Arup. An employee there participated in a video conference and believed he was speaking with the CFO and colleagues. In reality, the people on the screen were deepfakes. The employee—acting on their behalf—initiated a total of 15 wire transfers totaling approximately 20 million pounds to the fraudsters. The case illustrates how quickly AI can render previously proven security mechanisms ineffective. In the past, a personal phone call or video conference could serve as an additional form of identity verification. Today, an executive’s voice and appearance can be artificially generated.
Added to this are entirely new methods of attack, emphasizes Meyer, who served as a police officer from 1982 to 1990, including at Hamburg’s Davidwache station and with the narcotics unit. In so-called “AI Search Poisoning,” for example, criminals attempt to place false phone numbers, links, or other information on the internet in such a way that AI systems pick them up and present them to users as supposedly trustworthy answers.
AI can also be misused within a company. Employees can create deepfakes, enter confidential information into inappropriate systems, or use AI for purposes that violate personal rights, copyrights, or internal guidelines. That’s why a one-time «AI license» for employees is no longer sufficient today, because: «Technology and risks are evolving too quickly for that.» For Meyer, AI management is therefore a new leadership discipline that «isn’t about constantly chasing after the latest AI model.» Rather, it is about creating an organization that can cope with this development.
Individual measures come together to form a management system
According to Meyer, the solution to these problems is not to ban AI or to require every AI-generated sentence to be approved through multiple levels of the organizational hierarchy. Professional AI management takes a different approach.
First, a company must know where AI is actually being used within the organization. It must then assess the risks associated with each individual application. There need to be guidelines for its use, clearly defined responsibilities, and employees who know what they are allowed to do and what they must monitor. Finally, it must be regularly verified whether these rules actually work in practice. «A management system is, at its core, something very simple: a combination of guidelines and controls,» Meyer writes in his book.
According to him, AI management is explicitly not about creating a new bureaucratic monster. An AI system that summarizes internal texts requires a different level of oversight than an AI that can independently communicate with customers, evaluate job applicants, support medical recommendations, or initiate payments. The principle is risk-based: the greater the potential impact, the stricter the rules and controls must be.
For Meyer, this is precisely where the difference lies between individual AI rules and an AI management system. A guideline alone does not prevent a failure. Only when responsibilities, risks, controls, training, and improvement processes work together does a system emerge. «It’s not AI that determines a company’s success. It’s the way it’s managed,» is one of the central messages of his book.
An international standard for AI management already exists
Companies don't have to invent this system themselves, Meyer emphasizes. According to the CEO of DICIS AG—which also conducts AI certifications—ISO/IEC 42001 already provides an international standard for AI management systems. Its logic is similar to that of established management systems for quality or information security. Among other things, ISO 42001 requires defining the scope of the AI management system, establishing responsibilities, assessing risks and opportunities, ensuring competence, monitoring the use of AI, and regularly reviewing and improving the management system. In addition, there are specific measures related to AI policies, resources, impact assessments, data, the lifecycle of AI systems, and interactions with external stakeholders.
The main advantage of such a standard is that not every company has to develop its own definition of «responsible AI use.» «ISO standards have a decisive advantage: they are recognized worldwide,» Meyer emphasizes. Consequently, a customer does not need to have the details explained to them regarding the in-house method a service provider uses to monitor its AI, because there is a common framework of reference.
ISO 42001 Certification as a New Mark of Trust
Meyer, who has already written several specialized books on the topics of AI and digitalization, does not view his new book, *AI Management in Practice*, merely as a book about the risks of artificial intelligence. Rather, it is intended to be a practical guide for companies seeking ISO 42001 certification. According to him, certification is particularly valuable «where companies not only want to promise their customers that they will use AI responsibly, but also want to demonstrate this through an independent audit.» This proof can build trust, especially among service providers and technology companies, because customers do not have to verify and assess for themselves whether rules, responsibilities, risk assessments, and controls are actually in place.
However, Meyer warns that the certificate does not guarantee that an AI failure will never occur again: «That would be unrealistic.» Rather, it demonstrates that a company has systematically organized its use of AI, assessed risks, defined responsibilities, implemented appropriate measures, and verified their effectiveness. In other words: ISO 42001 certification does not guarantee that a company’s AI will never make mistakes. Rather, it demonstrates that the company does not leave error prevention or quality management to chance, even when using AI.
Author:
Willy Laux (36) from Cologne works as a freelance journalist, primarily for specialized and industry magazines. His work focuses on technological change, climate adaptation, and the circular economy. Through his multimedia articles, he also covers digitalization and transformation in corporate management and the economy.
Book Recommendation
Dr. Jens-Uwe Meyer: ISO 42001 – AI Management Systems: Implementation and Certification: The Practical Guide to Artificial Intelligence in Business. Practical. Simple. Unbureaucratic. BusinessVillage Publishing, 2026, 256 pages, 1st edition, ISBN 978-3-86980-900-7.
This article originally appeared on m-q.ch - https://www.m-q.ch/de/ki-panne-oder-falsches-management/
