AI Transparency, Accountability and Non-repudiation

With new threats, gaps, and requirements, we need to start with data and structure.

In the ongoing race to implement new AI capabilities in real-life scenarios, we need to deal with new threats, requirements, and control gaps. It will take some time for technical mitigations to mature. In the meantime, we need to focus on data, understanding new challenges, and making sure they are taken into consideration when accepting the risks. In this post, we look at transparency, accountability, and requirements like non-repudiation as foundations for the security and safety of AI applications.

AI solutions have been moving into consumer and enterprise spaces for years and are already used to process sensitive data and automate or augment critical decisions. Recently, that process started to accelerate with the race for applications of Generative AI, which are very new, with unique threats, gaps in controls, complex dependencies, and impact on HCI or value and access to data. We are not yet ready to implement these new capabilities on technical, organizational, and regulatory levels, especially in high-risk domains or mission-critical scenarios. Users trying to implement AI are struggling with making informed decisions about applications AI in particular use contexts that could have broad and often not fully understood impacts. These tasks are impossible to complete without sufficient information about the origins, internals, and operations of an AI system. AI applications must be safe and secure to be useful. And to be safe and secure, they require clear accountabilities, technical transparency, and focus on related requirements, such as non-repudiation. These elements start to get more attention, e.g., the Executive Order on Safe, Secure, and Trustworthy Artificial Intelligence requires developers of high-risk foundation models to share the results of all red-team safety tests. As with many other complex problems, AI security and safety also need to start with data. 

Accountability Structure

Implementing AI capabilities in real-life scenarios requires proper planning and execution to deal with unknowns, gaps, and surprises. We need adequate accountability structures in place to make sure that critical aspects of AI development and operations are not missed - or worse - intentionally omitted during those efforts¹. The possible mistakes include focusing only on benefits (we'll deal with risks later), making unreasonable assumptions (others would address that), or missing the bigger picture (we care only about our customers). At the end of the day, it's up to applications' owners to accept or block the implementation of AI in specific scenarios and take responsibility for all the consequences. They must understand the value for an organization, limitations, and constraints of technology and define requirements for AI use, model, and platform. That includes looking at complete lifecycles of AI applications, tracking how results are used in downstream decision processes, or how applications could be misused – especially with any functionality exposed to users.

On the technical level, developing, deploying, and operating AI applications is a shared responsibility of application owners and technology providers, similar to the pattern used for the cloud. The scope needs to be extended to address security concerns related to data used in training, more complex supply chains, limitations and constraints of models, or quality and characteristics of results produced by AI (e.g., including Content Development Lifecycle (PDF)). Technology providers must meet detailed security requirements, but until there are industrial standards and mature best practices, defining those requirements is the task of owners of AI applications. The owners must also understand and acknowledge the broad scope of impact and harm that can be caused by an AI application, not only limited to users or customers but also covering all potentially affected groups and individuals (may not be obvious, especially when looking only at parts of the system). All these tasks require a proper structure, clearly defined roles and responsibilities, and a lot of data. 

Transparency and Data

Transparency is generally mentioned in every document covering requirements for AI systems, usually with high priority and imprecise definitions. OECD Transparency Principle includes understanding AI systems, awareness of interactions with AI, understanding the outcome, and giving effective options to challenge a prediction, recommendation, or decision. The EU AI Act also places a lot of importance on documentation, traceability, and transparency, especially in the case of high-risk AI systems. When talking about the transparency of an AI system, we can look at its different elements, like data, models, or architecture and deployment.

• Data are essential during the training of AI systems and later during inference and their ongoing operations. We need to care about the quality of the data, their lineage, integrity, hygiene, and copyright status. Such information can be captured, for example, in Data sheets, and should be part of data governance when sensitive data are in scope.

• Models are created in a context, often with a specific intention, but in every case, with some limitations and constraints for their use. Such information should be captured, starting, for example, with Model Cards, which become common in practice. On top of that, there are technical fields related to the explainability of models and interpretability of results.

• Deployment and architecture of an AI application must be subject to regular security analysis of the platform but also cover additional elements. These include an extended view of supply chains (foundations models, data sources, software artifacts) and all integrations of AI applications while operating (e.g., supported plugins and external data flows).

In the security and safety context, we cannot rely only on data about a static snapshot of an AI application, but we need to look at its complete lifecycle. There are different types of data useful at different stages; during development, we care about intended use, training data, or logs from RLHF (e.g., in some LLMs), and during operations, we need details on how the system is used. Data about AI applications, including documentation, metrics, logs, and traces, should be available continuously and provide insights into the current state of technical components and the application’s use context. These data streams should enable building an up-to-date big picture of the AI system that includes its capabilities and limitations and all the issues that require attention and could impact performance and results. The data must also cover all external integrations and dependencies on technical or business levels. The data needs to be provided by (or requested from) technology providers involved in delivering an application, often multiple ones (e.g., different companies creating foundation models, tailoring it to specific scenarios, and hosting). Many details need to be clarified, such as the balance between openness and protecting intellectual property and giving more information to potential attackers. Still, having access to data is a required first step.

Non-repudiation and Other Requirements

Data have value if they can be used to meet specific technical requirements, help detect possible problems, and deliver appropriate solutions. Transparency must be operational and integrated with MLOps for models’ governance and monitoring (e.g., drift detection), but also with effective security monitoring. Security and safety of AI applications need to be covered by the Security Incident Response process to be sure that the impact of a vulnerability or a failure is reduced and required actions are quickly taken. That is especially important since fixing AI security problems may be much more complex than with regular software (e.g., the need for retraining a model vs recompiling and testing). The ability to conduct an in-depth analysis of results or behavior of AI components is also essential for any situation when results or downstream decisions are questioned, challenged, or just more explanation is needed (e.g., in healthcare). That is related to mentioned earlier areas of model explainability and results interpretability, but also requires extending observability (different in ML from monitoring) to enable tracing the complete decision process, the role of AI components, and all human involvements. All these capabilities need to be connected with specific technical requirements.  

Non-repudiation is one of the desired properties of a computing system (related to repudiation threats used in STRIDE threat modeling). This property refers to the inability of a user to deny (repudiate) performing some action or transaction (or dispute the validity of a contract). Non-repudiation requirements are expected to grow in importance as AI applications get more significance in decision-making, especially in domains like healthcare, transportation, or the military. In classic cases, these problems usually take the form of users denying that they did something. In complex scenarios of decisions automated and augmented with AI, users might claim that the AI component did OR didn’t do something. There are already interesting cases of claiming that a statement was a deep fake or using algorithmic output to justify unpopular business decisions, and access to data from incidents becomes critical for autonomous cars. Non-repudiation will require actionable data, including recording all inputs and outputs, detailed model metrics, or external integrations, to be transformed into a format understandable by humans. It will also require tracking all interactions in a decision process, including communication between humans and AI, between different AI components, and approving or rejecting AI outputs. Implementation of non-repudiation will be essential for effective human oversight in critical interactions with AI. 

---

In any application where the risks related to AI would not affect only their creators, novelty, ignorance, or carelessness are not valid excuses. There is a fundamental difference between AI experiments and production implementations with real-life consequences, and transparency and accountability are critical for the latter. We need proper due diligence when dealing with unknowns, and we need to ensure all gaps are identified, acknowledged, and covered by continuous analysis and research for mitigations. As we may have reasonable concerns about over-regulating AI, we shouldn’t be concerned about too much transparency or accountability, at least for a while – we might instead look at those as defense-in-depth covering for insufficient technical mitigations. Transparency and accountability should be built into the foundations and culture of the AI industry. They need to be recognized as substantial competitive advantages, and in high-risk domains or mission-critical scenarios – as requirements to compete and make a difference. Hopefully, with awareness (and likely some hard lessons), such perception might grow organically, independently from regulations. In every case, to evaluate the role of AI components in decision-making, we will need to analyze the existing workflows, and that can give us a chance to fix problems we didn’t even know existed before we tried to use AI as a solution.

1. Parts of this section were initially used in the previous post but were moved here after editing based on received feedback.