Threat Modeling AI Use Context

We need to start with big pictures of threats and impacts related to AI applications

AI applications have become more complex, with multiple dependencies, operating on highly sensitive information and embedded in high-risk workflows. To effectively protect such solutions, we need to understand the threats and limitations of a complete system. Such a problem is not new for cybersecurity, as for years threat modeling has been used to help us with similar challenges. Threat modeling tools and methodologies need to be updated to address unique characteristics and requirements of AI components, different priorities, and possible future trends. These efforts are both a necessity and an opportunity for a more complete understanding of attack surfaces, requirements for technology providers, or the consequences of failures.

Applications of new technologies are associated not only with benefits but also with new ways in which they can be abused or misused. Developing new solutions usually focuses first on functionality, meeting users’ requirements, and delivering value. Security is often added later, which can be challenging, making it difficult to cover the complete product lifecycle (and increasing the costs of late fixes). Threat modeling is a structured approach to analyzing software design, architecture, and usage patterns from a security point of view. It is a process intended to help us understand how a system and its users could be attacked, evaluate if mitigations in place are sufficient, and identify vulnerabilities that should be addressed. By looking at the system from adversaries’ perspective and trying to anticipate their possible actions and impact, we can identify specific problems that otherwise could be missed and manifest only in real-life scenarios. Threat modeling can help us answer fundamental questions: what can go wrong, and if the system is ready.  

Different methodologies, frameworks, and tools have been developed for threat modeling, but on a high level, the process usually includes similar steps: 

• We start by scoping the system and understanding its goals, functional requirements, and intended and actual use scenarios.

• Based on that, we create a model of the system’s architecture, data flows, dependencies, external interactors, and trust boundaries.

• At this stage, we also enumerate the assets and properties of a system that need to be protected.

• We use that model (often a visual representation) to identify threats, which could be understood as possible adversaries’ actions against the system. 

• In the context of these threats, we evaluate current or available mitigations (controls and procedures) to determine if related requirements are met.

• Threats with missing or insufficient mitigations indicate potential vulnerabilities or gaps that should be further investigated (and always documented!).

A good threat model should provide a big picture of the application’s security, dependencies, trust relationships, control requirements, and action items needing attention. Such a big picture can be used as a frame of reference for managing security efforts throughout the application lifecycle, especially in planning and prioritization (e.g., identification of critical targets for penetration testing). It should help make informed decisions regarding applying specific solutions in particular situations, understanding value and limitations, and balancing risks and benefits. Despite attempts to standardize and automate threat modeling, the process usually still needs to be customized to domains or technologies and requires close cooperation with development teams. That’s almost guaranteed when application scenarios or technical capabilities are new.

Threat Modeling AI

Threat modeling AI is associated with new challenges, as AI components introduce new levels of complexity, dependencies, and unpredictability, which adversaries can target (not only in generative scenarios). When protecting AI applications, we must deal with new threats, like adversarial attacks during AI training and inference. Some other challenges are related to defining trust, as the input and output of AI components can be much harder to validate, and the sensitivity or value of exchanged data is more difficult to classify. The nature of user interactions with AI systems is also more open, which can lead to incorrect assumptions, unrealistic expectations, or risk of overdependency (individual and organizational!). On top of that, dependencies of AI applications on external software, services, or data can be complex, with often unclear responsibilities between technology creators, providers, and users. AI components can add a lot of unpredictability as they are frequently black boxes of fuzzy origins that are quickly implemented in scenarios where a lot is at stake. We should be concerned not only about vulnerabilities in these components but also their intended behavior, which might be aligned with the goals of 3rd parties involved in their development but not necessarily with end-users priorities.

In AI Security, we cannot talk only about specific technologies or isolated solutions outside the broad context of their practical usage. The value of AI lies in automation and augmentation of decision or creative processes, and AI components are parts of socio-technical systems that are potentially more closely integrated with individual or organizational workflows than other software. These systems combine technical, social, and business elements, and the scope of that integration may not always be thoroughly planned or identified. AI Security needs to go beyond protecting data, applications, or infrastructure and must also cover the INTEGRITY of processes and actions based on results from AI. Threat modeling efforts cannot be, therefore, limited to analysis of data flows between software components, direct users, and 3rd party systems. We should assume that there are many non-obvious ways in which AI components could be influenced by external entities, or their results could impact external entities. We need to look at the broad scope of the system’s usage, evaluate the role and provenance of AI components, and determine their impact on environments, individuals, groups, or organizations.

Threat Modeling AI Use Context

AI Security efforts can be approached using 3 Layers of Use Context, AI Models/Algorithms, and Platforms where they are operating. The threat modeling process should start with understanding the AI Use Context before we can dive into threats, define requirements, and evaluate mitigations. AI Use Context refers to all variables related to the intended and unintended use of AI components for specific scenarios and purposes. It includes elements like types of tasks supported by AI, characteristics of domains (any regulations?), and environments (digital vs physical?), as well as users, subjects, and other entities that AI can impact. Starting with AI Use Context should provide us with a proper structure for threat modeling of the complete system and help us understand its boundaries and external dependencies. The practical value of a threat model depends on the quality of the model part, and adding new variables increases the difficulty of the process. However, since AI threats can be unique or surprising, threat modeling efforts must identify all the unique assets that adversaries could target. To properly understand the impact and consequences of failures, we need to go beyond technology and include also non-technical elements.

Threat modeling should start with AI Use Context, but results from these efforts must be applicable at AI Model and Platform layers as they are closely interconnected. The output from threat modeling of AI Use Context works as input for similar efforts at lower levels but also can result in direct requirements for UX design, guardrails, and metrics of AI Model or operational best practices of AI Platform (all belonging to Secure Artificial Intelligence Lifecycle). Evaluation of mitigations implemented at different layers should lead to identifying gaps, documenting constraints and restrictions, developing proper guidelines, and overall determination of readiness for applying AI components in a specific scenario. To ensure that we don’t miss anything, we might consider using two simple patterns when threat modeling AI applications:

• FOCUSED analysis of AI components and all their dependencies. The threats specific to AI components cannot be analyzed for software products/services in isolation. The process must also cover all stages of their development, complete supply chains (software, data, services), and ongoing operational aspects emphasizing integration with local environments. That includes non-technical elements, like full transparency of the development process (stakeholders!), access to training data, or continuous availability of comprehensive logs for incident response (but also for possible challenges to results produced by AI). All those are critical for defining reasonable requirements for technology providers and clarifying shared responsibility for complex scenarios.  

• DOWNSTREAM analysis of the impact of results and interactions with AI. The focus is on the role of AI components, not only covering input/output but also downstream usage of the results of tasks that are supported with AI. It must be clear how critical AI components are, which specific workflows they can be integrated with, and what mitigations are to prevent that in all other cases. We need a holistic approach to predicting and documenting impact, especially potential harm, on users and other humans but also in broader organizational, social, or economic scope (a connecting point for other efforts). The results are essential for defining acceptable use policies, proper training, required controls, or data governance requirements for managing AI-tainted data. 

The use context of AI applications can be very dynamic, as AI capabilities may quickly improve and be easily adapted to new tasks and scenarios. That evolution may not always be entirely intentional as it may be driven only by benefits while missing the need to revisit security requirements. We should assume that most non-trivial changes to AI Use Contexts can impact the system's security by changing its attack surface, exposing it to new threats, making attacks easier or current controls insufficient. As mentioned previously, the AI Model could be fully suitable for a scenario at a release time but not acceptable after a modification or just after a while. Threat modeling of AI Use Context cannot be a one-time project. It should be a structured, repeatable, and continuous process of documenting, monitoring, and analyzing the use of AI applications. New security efforts should be automatically triggered when there is any change with a potential impact on the security of a system, like new types of attacks, a new data flow, a change in metrics (e.g., model drifts), or an incident reported by any technology providers. Our goal should be a threat model that gives us a big picture of our system, helps us understand and measure its security, and continuously validates its applicability in specific scenarios. 

Necessity and Opportunity

Many cybersecurity experiences can be valuable for AI applications, and we can see great efforts in using them, including threat modeling. MITRE has released a database of AI-focused adversarial tactics and techniques (Adversarial Threat Landscape for Artificial-Intelligence Systems) complementary to their ATT&CK framework. Guidelines and recommendations are developed by providers of products and platforms (e.g., Threat Modeling AI/ML Systems and Dependencies from Microsoft) or focused on specific types of AI components like LLMS (e.g., OWASP Top 10 for Large Language Model Applications). In practice, all these tools still need to be adapted to unique characteristics of AI, which may be challenging in advanced cases. There are, however, shared requirements for threat models to provide value. We need a threat model to be COMPLETE, cover all relevant elements of a socio-technical system, and capture all vulnerabilities or gaps that require attention. It needs to be CORRECT and accurately represent the system and its current state (i.e., up to date). Finally, it should be CONSISTENT and usable to enable smooth integration with other efforts focused on AI quality (e.g., fairness) and other partners (also external).

We could think about a threat model as documentation covering applicable threats and the readiness of controls and procedures to mitigate them. Such documentation can be useful for identifying high-risk and mission-critical scenarios, getting clarity around constraints and restrictions, or helping with understanding inherent and residual risks. However, in the context of practical AI Security, threat modeling becomes both a necessity and an opportunity. It is necessary to give us more control over the complexity and novelty of AI applications and help us protect them from AI-specific threats. It is also an opportunity to learn from cybersecurity experiences, incorporate security considerations early, and help to avoid mistakes that otherwise could be very expensive (not only in financial terms). Threat models can become centerpieces of AI Security Programs covering all essential elements, starting with awareness, policies, and training, through requirements for technical controls to operational procedures, continuous assurance, and incident response. A good threat model covering AI Use Context can be foundational for accountability and transparency and help build trust between stakeholders, internally and externally.

---

If we think about the goals of AI Security as Responsible Use of Trustworthy AI, threat modeling of AI Use Context is fundamental for the Responsible Use part (while trustworthiness is more related to AI Models and Platforms). In a way, AI Security starts and ends with AI Use Context, as this layer critically determines threat exposure and mitigation requirements. Threat modeling can be an effective tool for discovering and managing the complexity of modern AI applications. We need those complete, up-to-date, and usable big pictures to make more informed decisions about implementing AI capabilities in specific contexts. To understand how technology can be effectively attacked (or fail!), the scope of impact, and the possible consequences of failures. All these elements are necessary for building a strong AI Security culture in organizations, which is usually at least as crucial as implementing robust technical mitigations. We’ll look at more details and threat modeling examples in future posts. 

Updated on May 15th, 2024, with a revised diagram and minor fixes based on feedback.