Secure Artificial Intelligence Lifecycles
Value of lessons from the past for the security of AI applications in the future.
This post was intended to be very short – mainly a diagram with some annotations from a quick experiment of looking at Microsoft’s Secure Development Lifecycle (SDL) in the context of AI-based applications. There are unique challenges related to AI, but there are also lessons from the history of cybersecurity that can help us approach the complexity, changing nature, and unknowns of new technologies. These lessons should save us time and prevent us from repeating old mistakes so we can focus on discovering (and inventing) brand-new ones.
The importance of cybersecurity has been dramatically changing with our growing dependency on software and digital data. Before software development became big business, the primary focus was on delivering core functionality and the quality of products dependent mainly on the skills, awareness, and commitment of individual developers. Different ways of attacking computer systems had been discovered very early, but for a long time, they were considered curiosities with limited practical impact or subjects of academic divagations. In the context of competing priorities and the cost of software development, security rarely got much attention. That approach was not a critical problem with software in a box, disconnected, or operating in closed networks. That all changed with the arrival of the Internet, as we started to move our core activities and interactions online, and most systems got connected and exposed to attacks. Suddenly, much more was at stake, and software and data security became a top priority.
To meet requirements of security (but also privacy or reliability), software development had to become more predictable, repeatable, and manageable. Development processes needed engineering structure to ensure that known mistakes are avoided, correct requirements defined, and proper attention is paid to gaps and new challenges. Security Development Lifecycle (SDL), created at Microsoft in 2004, was one of the most successful attempts to develop such programs. This program implemented a holistic approach to software security, starting with training developers, including several defenses in depth (e.g., an accidental code defect should be caught by security testing and its impact contained by attack surface reduction), up to preparations for possible incidents. That approach not only changed technologies and processes of software development but also affected culture and security awareness. Today, we need a similar structure for creating and using applications with AI components to focus on the unique characteristics of these new technologies and pay attention not only to opportunities but also to risks and costs.
The diagram below includes an overview of the SDL process with some AI-related notes (click to enlarge). Please note that is just an exercise in analysis of established process, not an attempt to define a comprehensive lifecycle for AI-based solutions – there are interesting proposals of such frameworks covering secure design, development, deployment, and operations. For clarity of visualization, the original stages of SDL were used which didn’t cover agile and continuous software development. Again, the key goal here is to illustrate the additional elements that are important for applications with AI components.
Here are a few quick thoughts from the exercise:
Lifecycles of AI-based applications will be broader, more complex, and dynamic compared to classical software. AI components are still dynamically evolving; we don’t fully understand the details of their operations and struggle with measuring or testing them effectively. There are many more participants in their development, from business stakeholders to data scientists, with roles and responsibilities that are rarely clearly defined. Lifecycles of AI-based applications must be complete, and early stages are critical as some issues may be impossible to fix later (e.g., problematic training data may block the use of AI models in specific contexts). Also, the impact of an AI system may still exist even after it is decommissioned, e.g., no more organizational competencies, as AI successfully automated them.
Requirements and design of AI-based applications are not only about the technology but also their sociotechnical use contexts. Classical software started with a specification of goals and a list of prioritized scenarios that were translated into features, requirements, and development plans. AI-based solutions often begin with generic capabilities and opportunistic search for suitable applications without built-in guardrails or restrictions. The use of AI in specific scenarios must be well understood, including details of tasks and decisions that are automated/augmented, their possible impacts on users or other humans (also indirect), and applicable regulations. All that information must be captured (e.g., in an extended Threat Model), updated, and used along the lifecycle to coordinate all security and safety efforts.
Development and verification of AI-based applications should be continuous, covering Platform, Model, and Use Contexts. Security of AI is not only about software and its deployment but also depends on foundation models, data, and human feedback in training (e.g., RLHF). AI Models can be targets of attacks, misused or abused by users, and their behavior can change over time with unexpected consequences for security. AI supply chains are much more complex, with the need for specific requirements, analysis, and communication for any 3rd party integrations (even without sensitive data). On top of that, brand-new methodologies need to be created for pen-testing AI components (processing untrusted data in complex user interactions) and complete AI-based applications (covering downstream impact).
Continuous monitoring and assurance of applications need integration with requirements and up-to-date Threat Models. Logging and monitoring must cover the complete application, state, and behavior of specific components, all inputs and outputs, and the downstream impact of results. Detection should focus on changes, trends, and anomalies, including malicious activities, degraded performance, loss of observability, or misuse of AI components in an unapproved context (a big challenge for release management). Security infrastructure and workflows create unique opportunities to cover other AI-specific requirements (with the benefits of access to additional signals). The whole process should be integrated with threat modeling and security quality controls and be the foundation for data-driven transparency and accountability.
Incident response should be expanded to support new types of incidents and requirements related to dependency on AI capabilities. AI-related incidents and vulnerabilities may be complex for analysis (not readable code), with unclear root causes (changes of a model), and often with very difficult or expensive fixes (e.g., need to retrain a model). With the increasing use of AI in mission-critical scenarios, we will need to support new failure modes and plan for situations when AI components might be taken out of production for a longer time. Some failures may indirectly impact security and safety, e.g., performance drift or loss of specific properties could make AI components unacceptable for use in particular scenarios. The response process may also need to handle legal or regulatory challenges of decisions or products from AI by users or other affected individuals.
Modern AI-based applications are more complex, have more dependencies, involve more participants, and potentially have a much more significant impact on individuals and organizations. Security and safety requirements are critical for their broad and successful usage, even though they may not always be recognized. Similarly, like with cybersecurity 25 years ago. Existing tools and methodologies will not solve new AI-specific problems. SDL in its old form is insufficient (it is being replaced by Secure Future Initiative), but there is a lot of value in lessons from cybersecurity that we mustn’t dismiss. We must figure out what worked well, what needs to be updated, and where the gaps require proper research and investments.
We need security programs that provide complete coverage of the lifecycles of AI-based applications rather than focusing only on solving selected new problems (how would we know we chose the correct ones?). We need structure and foundations that can give us control and confidence while enabling effective research of new tools, practices, and patterns as we are dealing with an evolving and extremely dynamic threat landscape. Finally, we need to look at AI security and safety not only through technology but also consider the broader impact, organizational maturity, necessary changes to culture, and individual awareness about these new problems. Again, similarly to cybersecurity, we can expect the evolution of AI security from being outlier ideas to potential competitive advantage to core investment for anybody trying to use AI in high-risk domain or mission-critical scenarios. Except this time, that should happen much faster.
This is just the tip of the iceberg.
well said. AI security is still in its infancy and I find it fascinating just how much new threats and risks AI is generating. The industry is still playing catchup i feel !