AI Security and Trust Boundaries
Racing to raise new skyscrapers on the old foundations
In the last couple of months, we have seen the release of multiple AI browsers like Perlexity Comet and OpenAI Atlas, AI-based extensions, or even traditional browsers moving towards AI. From a business perspective, these might be rational attempts to gain access to more data, benefit from vertical integration, and challenge Google’s dominance in the evolving search space. From the users’ point of view, these are dangerous experiments that should be run only in controlled environments and scenarios that do not include any valuable or sensitive assets. AI browsers usually operate with the user’s full privileges across authenticated sessions, where traditional web security mitigations (same-origin policies, sandboxes) are insufficient, and new solutions are vulnerable to new systemic problems, like indirect prompt injections (not only in text). These are great examples of AI scenarios being pushed too early, too broadly, and too deeply, with threats that are not well understood, controls that are confirmed to be insufficient, missing regulations, an unfair imbalance of benefits vs risks, and security foundations that are not ready for the new requirements.
Applications like AI browsers, however, are only examples of a recent trend toward software becoming more agentic. Detailed definitions of AI agents are still under development, but we can think of them as applications that achieve a goal by observing the world and acting on it using the tools at their disposal. The world, in that context, will eventually expand into physical space as well, with the next big hopes around robotics. Still, currently, the efforts are focused on our digital ecosystems of connected devices and online services. That’s more than enough, as digital spaces host most modern activities, from daily information consumption, through most social interactions, to managing finances or health. Novel AI agents operating in those spaces rapidly become more complex, dynamic, and autonomous, despite implementation challenges and returns on investments that still need to be proven. Compared to AI models, AI agents are not limited to their training data, can interact with external systems, and have an orchestration layer that enables iterative reasoning and multi-step problem-solving. We should therefore not think of AI agents as isolated applications or websites responding to our prompts, but rather as software components closely integrated into our experiences, operating systems, infrastructure, and physical devices.
Adding AI components to our systems challenges existing security patterns and the effectiveness of controls and practices developed over the years.
The promises of AI focus on automating and augmenting our cognitive functions, including the increasing number of decisions and creative tasks. The marketing hopes behind transferring activities from humans to machines are to eliminate mundane tasks, make difficult ones easier, and make the impossible possible. With personal assistants, which are only a small subset of AI applications, AI agents, by design, need access to user data, local resources, and behavior, with the ability to interact with external services (or a network of agents) on the user’s behalf. Translating these functional requirements into a security context, we want non-deterministic black boxes, designed to be autonomous and continuously evolving, to operate with limited transparency, accessing sensitive and valuable data, and interacting with untrusted external entities. Such requirements fundamentally redefine the nature of personal computing and the relationships among users, data, services, and agents. Adding human-like AI components to our systems challenges existing security patterns and the effectiveness of controls and practices that have been incrementally developed over the years, often based on lessons from critical failures. They also impact the mental models required for a user-friendly experience, for understanding risks, and for making informed security decisions. We need to revisit the foundations for building security structures, policies, and guardrails, starting with basic components such as trust boundaries.
Trust boundaries can become obstacles to fulfilling the immediate promises of AI agents, which need to be deeply and broadly integrated into user experiences.
A Trust Boundary is one of the key concepts in practical cybersecurity. These are conceptual perimeters that separate areas with different levels of trust or security. Crossing a trust boundary is associated with specific requirements for authentication, authorization, or validation of data or commands. While using any computing device, we encounter multiple trust boundaries: the operating system kernel, user accounts and sessions, virtual machines or sandboxes, browsing sessions to authorized websites, or, last but not least, a trust boundary between local and remote systems. The trust boundaries are obviously not limited only to our personal devices but expand to complete digital ecosystems and organizations. They are not only associated with the final products but are also critical to software lifecycles, with special emphasis on the release process. In all those contexts, trust boundaries are essential for effective access control to data and resources, for identifying and describing attack surfaces that must be protected, or for conducting threat modeling and better understanding the risks. Trust boundaries are key elements of modern security frameworks, but they can easily become obstacles to fulfilling the immediate promises of AI agents and applications, which need to be deeply and broadly integrated into user experiences. And because of the perceived urgency of moving AI scenarios to production, there is little or no effort to address new security challenges, even those already known.
Access to DATA and USER Activity. Trust boundaries are crucial for access control, and AI needs access to as much data as possible for both training and inference. According to optimistic assumptions, which still seem to be the basis for most of the current business models, the more access to data, the more value AI can provide. Unfortunately, access to data is associated with high costs of required protection or potential damage in the event of a compromise. Running an AI browser in practice exposes all online user activity and data, including access to banking, health, or private emails. In the case of AI agents integrated with operating systems, limiting access to local known folders—such as Documents, Downloads, Desktop, or Pictures in most situations still means sharing most of the sensitive data and behavior with AI. Client systems are not equipped with robust mechanisms for personal data governance and classification, and the most critical piece of data is not necessarily a credit card number but something more contextually valuable. That can become a significant problem when personal data are used for training AI models, which is often communicated as improving product, or data sharing extended to generative AI models and other machine-learning technologies powering our services. Unfortunately, when personal information is used in training, it cannot easily be deleted from an ML model in the same way that it can from a database.
Transition between LOCAL and REMOTE processing. One of the key trust boundaries exists between a local system and remote resources. We have been continuously moving online - most tasks can now be performed in a browser, we have learnt to expect cross-device synchronization, and automatic backups are established best practices. That is not a problem, as long as such a transition online (including signing in to a browser) is an informed decision made by a user who understands the risks and consequences. However, recently, it’s become less about users’ intentions and more about a push from technology providers, who require an online account even for local usage or start saving all new documents in the cloud by default. AI in that context could become the final and ultimate excuse for moving the remaining users’ activity online. On-device AI capabilities are growing, but more advanced scenarios will require computing in data centers. When data are transferred to a remote system, they are always exposed to increased risks, even if they are never expected to be stored. Data leaving a local system are beyond users’ control, and their security becomes a shared responsibility, which might be a standard approach in corporate security but is much less suitable for individual scenarios. It is not a new problem, but in AI context, we might expect much more critical data transferred to AI data centers, and designing infrastructure with confidentiality and privacy in mind is not a common priority.
Attack Surface AND Privileged Access. According to the Principle of Least Privilege, a system should have the minimum permission required to complete a task. AI agents are supposed to thrive with broad access, leading to increased attack surfaces for the systems they operate in. On top of that, AI agents are a type of software that is very difficult to understand, control, or update, and are facing unique implementation challenges. Among the shared functional requirements for AI agents of accessing private data (1), processing untrusted input (2), and communicating externally (3), only two should be implemented at the same time within a session to mitigate risks of prompt injection (referred to as Lethal Trifecta or Agents Rule of Two). Solutions are not easy, since some of these new threats are inherently tied to AI capabilities, such as mixing data and commands that tell LLMs what to do. Old-school security strategies (e.g., input validation or output sanitization) are much less effective in open and unstructured interactions between users and AI. Despite intensive research, available defenses against jailbreaks and prompt injections are systematically vulnerable to adaptive and dynamically evolving attacks, also because of a focus on metrics rather than actual effectiveness. On top of that, broad access to local data and resources may lead to catastrophic consequences, not only from malicious attacks but also from non-security bugs or user mistakes (later apologies from AI agents are rarely useful).
External IDENTITY and Networks of Agents. One of the promised features of AI agents is their ability to act on behalf of users, both internally and externally. The intended consequence of that feature is AI agents’ participation in interactions with external services, other agents, or other humans. These can cover a wide range of activities, from managing personal information to agentic commerce, helping with personal finances, or assisting with complex interactions, like conflict resolution. All these scenarios involve strong technical requirements for identity, but existing systems were built for humans, not for artificial agents. Implementing the requirements of transparency, accountability, and non-repudiation is challenging even in simpler scenarios, let alone in dynamic, adaptive networks of AI agents and humans (with the latter expected to be much more numerous). As soon as we move beyond simple demos, we quickly run into liability issues related to actions performed by AI agents based on users’ intentions or the misinterpretation of their preferences. Regulations still need to catch up with these challenges, but in the meantime, AI agents could be qualified as electronic agents and form binding contracts between users and third parties. Balancing human-in-the-loop involvement in complex scenarios will require a complete rethink of our UX patterns and policies. Until then, such recommendations could be interpreted as shifting risks onto users’ shoulders, especially since the liabilities of AI providers are minimal, often with possible damages capped at subscription fees.
Trust boundaries are fundamentally different in systems consisting of humans and AI. They are different in strict technical terms, for implementing access to data or identity management, but also as the concepts essential for user experience and the perception of trust. We have a limited understanding of AI-specific threats (we keep getting surprised), clear evidence of gaps in existing controls, and serious doubts about security foundations designed for applications of a different nature. In that situation, it would be very rational to slow down and take each step carefully. Still, with the urgent need to justify unprecedented investments in AI infrastructure, the changes critical for users’ data and experiences are often implemented with disregard for costs and risks. Terms of service are updated to train AI on users’ data by default (sometimes without any notification), or to share data with affiliates for AI improvements and product personalization. In some cases, even the choice to opt out is taken away, like when an option to disable voice recordings from being stored online is removed. AI components are increasingly installed or enabled automatically on personal devices, and it may be confusing when their capabilities start to wander into areas currently considered experimental. Such UX dark patterns in AI and adjacent technologies should be very concerning, as they represent the control over personal data and individual behavior being taken away from users. The problem is not only that new AI capabilities are being pushed to users too early, too deeply, and too broadly, but also that they are being introduced with insufficient communication, unclear consent, and limited risk awareness.
Aggressively pushing untested and experimental AI features into personal systems with sensitive and private data ignores a perception boundary that is fundamental for building trust in new technology.
Mistakes and vulnerabilities might be explained as necessary costs of AI innovation, but decisions about accepting related risks must be intentional and informed. What is even more important, these decisions must be made by all entities that could ultimately be affected by the consequences of potential failures. Unfortunately, the benefits and costs of early AI adoption are not necessarily fairly distributed. If an AI application is successful, developers and providers of new capabilities can expect to benefit significantly from their investments. However, the costs and consequences of possible security failures, sensitive data compromises, the extraction of unique value, or unsanctioned behavior by AI agents will often be borne by individuals who might never have accepted such risks. In this context, aggressively pushing untested and experimental AI features into personal systems that store the most sensitive, private, and valuable data ignores a perception boundary that is fundamental for building trust in new technology. Failures in that scope may have serious long-term consequences for trust in specific companies and in types of technology (e.g., growing anti-AI resistance). In a more rational world, we could expect the necessary protections to be delivered (with some delay) by regulations and frameworks for new technologies. Unfortunately, we have failed to create adequate consumer protections in digital spaces, and those gaps are becoming even more damaging in the context of AI. Now would be the next best moment to comprehensively address these challenges and create long-term rules, as well as proper incentives for the new industries. Unfortunately, the current efforts are instead aimed at blocking any attempts to provide even limited forms of regulations.
Please note that in this post we focus specifically on individual users, but the same concerns are applicable to organizational and enterprise contexts. The challenges there are even bigger, as we are dealing with much more complex systems, full of technical debt, unclear dependencies and supply chains, and still struggling with digitalization, which is now an even more urgent necessity because of AI. Because relationships among users, data, and resources are also much more convoluted, the trust boundaries in organizations are often fuzzy, ignored, or enforced only in theory, with attack surfaces less clear and consistent. Such organizational systems are now expected to fully embrace the potential of AI and to effectively manage vast networks of humans and agents involved in high-risk or mission-critical applications (not always easily identified), which may very quickly gain much broader, and not always obvious, impact on customers and other humans. The previously mentioned challenges related to liability for actions by AI agents become even more severe in organizational contexts, due to strict requirements for data governance, non-repudiation, and the need for proof of due diligence. Organizational security is obviously different from protecting individuals, if only because of the dedicated cybersecurity professionals responsible for managing risks. There are some unique challenges (e.g., autonomous insider threats), but most of them are shared. If we struggle to secure much simpler systems, we will have problems fixing much bigger ones.
AI agents, browsers, and assistants are often presented as personal and private. At least for now, they are unfortunately usually neither. AI components expose user data and online behavior to misunderstood threats, insufficient guardrails, limited transparency, and missing regulations or accountabilities. These challenges cannot be fixed unless we improve existing systems, patterns, and frameworks (not just technical ones) to meet the requirements of AI applications and protect users from attackers and AI providers in desperate need of revenue streams. These efforts need to start with revisiting security foundations, including defining and supporting clear, robust, and implementable trust boundaries. Those we currently have are insufficient, intentionally bypassed, and the development of missing ones is definitely not a priority. Trust boundaries are fundamentally connected to privacy and control over our data and online behavior (soon also physical). In discussions about emerging threats, we can often get very concerned about AI gaining dangerous levels of autonomy. In the short term, maybe we should rather pay more attention to the risks of humans losing theirs?
Updated on February 27th, 2026, the diagram was temporarily removed.
This article is so timely. Trust boundaries are like Pilates core work.