Blind spots in the productive use of AI

A prompt in, a fluid response back, and all in a matter of seconds: large language models (LLMs) such as ChatGPT have supposedly broken down the barriers to the use of AI in companies. However, it is precisely this low threshold that often obscures the actual bottleneck.

In practice, AI projects rarely fail due to the choice or size of the AI model. They fail because companies underestimate the hidden effort involved in transforming their own knowledge base into a form that machines can work with reliably. The real feat lies in understanding, structuring and contextualizing an organization's knowledge.

The actual value arises before the answer

Trust in AI can be broken down into specific questions in the corporate context: Can I trust the data on which the AI works? Is it up-to-date, complete and technically consistent? Is it clear why a result is produced? Was the relevant information consulted by the AI at all - or only what happened to be readily available?

This is precisely where LLMs reach their limits. They cannot bring order to data sets that are contradictory, incomplete or semantically fuzzy. The impressive ability of LLMs to formulate plausibly becomes more of a risk here - this is how inadequate input can be smoothed over.

Nevertheless, in many places AI is still measured by the quality of its output. For companies, the added value - and the effort - comes from qualifying data, building contexts and maintaining knowledge structures. This preparatory work determines whether AI results are reproducible, explainable and measurable.

This is particularly important if AI systems are to scale. Then it's all about consistency, traceability, quantitative evaluation and the ability to handle large volumes of documents in a controlled manner. It is precisely in this intersection of linguistic, technical and quantitative aspects that pure LLM setups regularly reach their limits.

AI-ready data is more than just available data

Just because a company has extensive databases does not automatically make them suitable for AI. Anyone who expects reliable results directly from unconnected documents, duplicates, free text and old data is confusing data access with data quality.

A robust framework is needed for recurring processes, automated decisions or regulatory-sensitive contexts. Is the information clear? Can sources be traced back? Are there contradictions between systems? Are key fields missing? Are documents properly keyworded? Is it clear which content is outdated, which is valid and which is only locally relevant?

Large models do not solve problems of meaning

In addition, document inventories in companies usually grow historically. Terms move between departments, systems and file formats. The same word can mean something different in a construction context than it does in insurance or technical documentation. People don't mind because they know «their» domain. For AI, it is a problem of semantics.

Consequently, raw data is not enough for a system to work reliably. It needs metadata, glossaries, reference structures and semantic classifications that make it clear what a term actually means in the respective context. This is the only way to create a working context in which «correct» means more than just linguistically plausible or statistically probable.

Specialist knowledge often lies where no machine can see it

And what if the relevant knowledge is not even available in data form? A not to be underestimated part of a company's wealth of knowledge lies in the routines, exceptions, implicit rules and empirical knowledge of the specialist departments. They know which source is reliable, which value is formally correct but remains practically irrelevant, or which formulation has a special meaning internally. This knowledge has often never been properly documented because it functions tacitly among people in everyday working life.

AI, on the other hand, thrives on explicit signals. It has to learn how an organization uses its language, which categories are common and which deviations are business-critical. This is precisely why «human in the loop» falls short if specialist departments only check AI results at the end. They need to be involved at an earlier stage: in clarifying terms, describing rules, evaluating sources and determining which contexts are essential for a reliable answer.

The difficult part of many AI projects begins with the fact that experts have to explain how they think. This is time-consuming. However, it is often the prerequisite for turning an eloquent system into a usable one.

Regardless of the LLM chosen: AI can condense, structure, prepare, compare and accelerate. It can penetrate large amounts of information and relieve experts of work that previously cost time, attention and nerves. What it cannot do: Take responsibility, heal contradictory data silently or understand the full organizational context off the cuff. The human being therefore remains in the process - as a technical and organizational authority.

To the author:

Heiko Beier is Managing Director of moresophy and Professor of Digital Media Communication and Artificial Intelligence with over 25 years of experience in AI-supported data analysis and automation of business processes. His expertise includes the development of explainable AI solutions that can be seamlessly integrated into business processes. www.moresophy.com

This article originally appeared on m-q.ch - https://www.m-q.ch/de/blinde-flecken-beim-produktiveinsatz-von-ki/