The Open-Source Paradox and the New Risks in the AI Supply Chain
AI repositories such as Hugging Face are becoming gateways for systemic risks. Dr. Kawin Boonyapredee, CISO Advisor at KnowBe4, explains how companies can address the growing threat posed by shadow AI and agent-based systems—and why humans must remain the central authority in this process.
The recent disclosure of critical vulnerabilities in major AI repositories such as Hugging Face highlights a fundamental problem: the very platforms that companies rely on to develop AI-powered applications are increasingly becoming a gateway to systemic risks. With the transition to agent-based systems, this problem is becoming increasingly urgent.
From Shadow IT to Shadow AI
For years, shadow IT—that is, the use of unauthorized hardware and software—has been a focus for many CISOs. This is now being replaced by a related, more elusive phenomenon: shadow AI. This refers to the unvetted use of public AI tools and open-source models by teams that, in their pursuit of efficiency, may unintentionally expose sensitive corporate data.
The incident at Hugging Face highlights the fragility of the AI supply chain. Anyone who obtains a model from a repository is not only importing code, but also a largely opaque logic that can be exploited through prompt injection, data poisoning, or integration into automated attacks.
Agent-Based Systems as a New Source of Risk
The trend is moving away from simple chatbots toward AI agents—that is, autonomous systems that execute transactions and make decisions on their own. While this progress promises efficiency gains, it also creates a significant governance gap.
Recent studies show that less than half of companies have an AI governance policy. This lack of oversight makes it easier for a vulnerability in a single model repository to affect the entire company. Compounding this is a lack of trust in agents already in production: Worldwide, only 34 percent of technical decision-makers at the director level and above trust the actions of their AI agents.
A roadmap with people as the central oversight body
Securing these areas requires more than just technical patches. It demands a fundamentally different approach to the human factor. Expert-led evaluation procedures should mandate a review of all AI models and providers before they are deployed, so that every integration is approved with full awareness. Control mechanisms based on the «human-in-the-loop» principle require human intervention at critical decision points to ensure that autonomous agents remain bound by organizational guidelines. Furthermore, security training should not only enable employees to recognize threats but also to actively monitor the AI agents they oversee—transforming them from passive users into active components of the security architecture.
Receive, Educate, Guide
The goal is not to slow down innovation, but to shape it responsibly. The transition must lead from a state of largely uncontrolled AI use to a practical approach that combines risk assessment and safe implementation. A simple framework can be summarized in three steps:
The first step is to promote innovation rather than hinder it. This involves taking stock of and evaluating the AI agents active within one’s own ecosystem. In the second step, the focus shifts from simply detecting phishing to managing the risks posed by the digital agents with which employees interact on a daily basis. Security awareness training should be tailored to the real threats users face. The third step centers on a thorough review: Before off-the-shelf AI solutions are integrated into workflows, strict procedures are needed to evaluate providers and models. The necessary transparency to understand the behavior of AI agents and ensure «human-in-the-loop» safeguards at all times is crucial.
The tools for managing agent-based risks are constantly evolving. The key question is no longer whether a company uses such technologies, but how it manages the associated risks. By combining email security, security awareness training, and defense strategies for AI agents, companies can ensure that their workforce remains an asset rather than a vulnerability.
Source: www.knowbe4.com
This article originally appeared on m-q.ch - https://www.m-q.ch/de/das-open-source-paradoxon-und-die-neuen-risiken-der-ki-lieferkette/
