As we start paying close attention to the responsible use of trustworthy AI, we cannot ignore applications created with malicious intent. Misused AI capabilities are changing old threats and creating brand-new ones that are applicable to information systems, users, enterprises, and societies. This is the first post in a series to discuss trends and concerns related to the Malicious Use of AI.
Recent progress in AI/ML gives promise to many great applications, and we can expect much more innovation as these technologies become broadly used. We are rightfully concerned about creating practical frameworks for the Responsible use of Trustworthy AI systems, but we cannot assume that all applications will follow that path. On the contrary, we should expect some applications of AI to consciously disregard rules and best practices, not only by taking shortcuts to break things faster. We must also be prepared for AI applications that are intentionally malicious or unethical and designed with explicit goals of causing disruption, harm, or abuse. Because the development and actual use of such systems will be outside of any control, they can be delivered faster and easier than any legitimate applications. AI can enable goals previously out of reach or too expensive for those with bad intentions. Before we learn about the actual value of AI in solving major problems, we should also be ready to face a variety of illegitimate and malicious applications. That is one of the key challenges for practical AI Security.
Even if we don’t plan to use AI in particular scenarios at the moment, we need to be ready for AI to be used to disrupt these scenarios.
Malicious or offensive use of AI capabilities has been broadly discussed in research for a while, but now these concerns are getting very practical relevance. Definitions of Malicious Use are different, but they generally include all practices that are intended to compromise the security of individuals, groups, or a society. Malicious Uses are also different from abuses of AI, as the former refers to using AI by bad actors to further their attack objectives rather than attacking AI systems directly. In other words, we are talking about automation or augmentation of intentionally malicious tasks with AI, where augmentation is likely to be achieved faster. Even though we should not attribute to malice that which can be explained by incompetence, in many cases, the absence of clear rules and regulations may create a comfortable gray zone for questionable decisions. The problem is that even if we don’t plan to use AI in particular scenarios at the moment, we need to be ready for AI to be used to disrupt these scenarios.
New threats and improved old ones
AI applications are changing existing threats. Possible attacks are scaling up and gaining much bigger reach while becoming more sophisticated and effective at the same time. AI automation and augmentation will enable faster, better, and cheaper attacks that can be targeted and personalized, taking individual contexts into account and going after the most vulnerable users or elements of a system. As a rule, new applications of technologies usually lead to new threats. With AI, everything will depend on the USE CONTEXT, types of decisions and tasks automated and augmented with AI, and new dependencies or interaction patterns. One unique property of the AI domain is a relatively low barrier of entry, at least compared to other advanced technologies. State-of-the-art AI research, tools, and models are widely available, as open source plays a critical role in this space. These free components can be used in many valuable applications, but similarly easily, they can be adapted to nefarious scenarios. And even if adversaries are not interested in getting into the technical details, we need to assume that AI capabilities are available through Crime-as-a-Service (CaaS) business model as products or continuously developed APIs.
AI-supported attacks against individuals and groups will unavoidably impact trust towards new and existing technologies and between their users.
Another property of AI applications is their operation in both digital and physical spaces. Serious concerns are related to increasing digital control over physical systems (e.g., taking control over a car) and Internet-of-Things (IoT) devices becoming a new attack vector (a problem for domains like healthcare). Not surprisingly, AI capabilities are also used in the actual weaponization of physical devices that are expected to operate with increasing autonomy due to possibly unreliable communication (see self-organizing swarms of drones). These scenarios will get much attention, but more damage might hide in less spectacular and more ubiquitous threats impacting our trust relationships. Deep fake voice capabilities are becoming a big challenge for our common authentication patterns - with overreliance on unreliable secrets (SSNs), assumed security of mobile devices, and voice phone calls often as the last backup mechanism. These practices were never robust, but now they are becoming unusable. We should expect that AI-supported attacks against individuals and groups on a massive scale will unavoidably impact trust towards new and existing technologies and between individuals and groups communicating with the help of technology.
Users, Enterprises and Societies
The most obvious area of offensive activities is AI automation and the augmentation of traditional attacks against information systems. However, the advent of Generative AI also enables new direct attacks against human elements, more sophisticated campaigns against enterprises, and unprecedented abilities to influence and disrupt societies and political systems.
Threats against INFORMATION SYSTEMS include using AI capabilities in attacks against software and infrastructure at different stages of the Cybersecurity Kill Chain.
Automation of the attack process can start with reconnaissance, attack surface analysis, and creating target profiles. Then AI can be used to generate custom payloads, simulate (ab)normal behavior, predict attack results, and facilitate concealment and later movement. AI can be particularly useful in bypassing security controls, both traditional and AI-enhanced, including CAPTCHA and various authentication methods (also biometric). AI-supported techniques can be fundamental for detecting and evading sandboxes or anomaly detection systems. Finally, we should expect broad usage of artificial agents with different levels of autonomy operating without Command-and-Control channels. These agents can imitate users’ behavior and be used for target selection, discovering critical data, and enabling exfiltration.
Threats against HUMANS and GROUPS include using AI elements to interact with users, simulate human behavior, or analyze and navigate social dynamics.
Automation and augmentation of social engineering will enable much more advanced attacks than old-school phishing. AI can be applied to create individual profiles of targets and drive complex interactions, moving from blind messages to conversations and building relationships. Special concerns are related to AI components interacting via social media, where they can impersonate individuals, groups, and organizations as parts of complex campaigns with generated believable background stories. Attacks against humans can also be executed beyond digital communication channels with conversational AI, voice recognition, and synthesis interacting over a phone. Language models can be trained with manipulation techniques and specifically used against the most vulnerable targets, such as Elders in emergency situations.
Threats against ORGANIZATIONS and ENTERPRISES include using AI applications in targeted and coordinated campaigns, often with specific objectives.
Advanced Persistent Threats (APT) can become easier and more effective, with automatically detected targets of a complete socio-technical system, custom content and payloads, and a continuous learning process. More sophisticated spear-phishing will take full advantage of growing generative AI capabilities. These can be especially effective when integrated with data about organizational dynamics, power structure, and communication patterns (e.g., using CEO’s deep-faked voice in a phone call). We should expect the next generation of attacks aimed at brand, reputation, and trust, as an effective simulation of human behavior enables new types of Denial-of-Service. They could be used in extorsion attempts without access to sensitive assets (similar to Ransom DDoS). Strategies like that could also be applied to individuals, especially in a political context.
Threats against SOCIETIES and POLITICAL SYSTEMS include using AI capabilities to affect opponents and competitors or to cause general disruptions.
Attacks using Generative AI with fake photos, videos, and audio produced using open-source models in political, business, or personal contexts. As in cybersecurity, there is a race between generating and detecting fake content, with growing pressure for accountability. Misinformation overload becomes a bigger problem as generating fake content becomes cheaper and faster while we struggle to balance technology and freedom of speech. Differentiating between real and forged content becomes more challenging and can reduce the impact of facts. Many new AI applications still have unknown consequences, e.g., AI surveillance applications are a big challenge for privacy. These applications are now available to private entities and individuals, and their deployment is met with opposition. Unfortunately, in many other cases, we may not even notice that something important is happening.
There may be subtle differences between accidentally irresponsible use, intentionally irresponsible use, or straight malicious applications.
We are not ready for all the changes that will be caused by attempts to build legitimate AI applications, and we are even less prepared for new or improved attacks supported by AI. AI will enable more sophisticated attack tactics and techniques to be implemented at a much larger scale. They will be cheaper and more effective, especially against vulnerable groups and individuals. With AI-based automation, existing vulnerabilities that attackers could currently miss will be more likely exploited (fortunately, that also means they should be easier to detect early). We need to assume that using AI technologies to achieve malicious goals gets easier and will be very well funded. Potential for big ROI exists, especially while defensive capabilities are insufficient or missing, and awareness about the threats is still limited. Temporarily, it is also a potentially comfortable situation, as in many cases, specific actions can be justified by novelty and lack of understanding. In practice, there may be subtle differences between accidentally irresponsible use (not understanding threats), intentionally irresponsible use (deciding to ignore problems), or straight malicious applications. Still, from a practical AI Security point of view, we should get ready to handle all threats related to such applications, regardless of the story generated behind them.
Updated on July 14th, 2023, based on received feedback, the content was shortened; some details will be covered in future posts.