Safe AI Is Tested AI
By: Nikita Taratorin, AIDX Policy & Partnerships Lead
An AI system is only as safe as the evidence that it works, and that evidence has to be produced, not asserted. As AI moves from answering questions to taking actions inside core business functions, the question that matters is no longer whether your AI is safe, but whether you can show it.
Most large organisations now run AI inside functions that carry real consequences: lending and credit decisions, hiring, clinical triage, customer service, fraud detection. Ask their leaders whether the AI is safe and most will say yes. Ask how they know, and the answers thin out. There is usually a policy, perhaps a responsible-AI committee, a vendor’s assurances, a benchmark score from a launch deck. What there usually is not is current evidence, specific to the organisation’s own systems and data, that the AI does what it is supposed to do and not what it is not. McKinsey’s 2026 research on AI trust found that only about a third of organisations have reached meaningful maturity in AI strategy and governance, and that oversight of more autonomous systems lags furthest behind.
The argument of this piece is straightforward. For AI, safety is not a claim you make but a property you can measure, and the only way to measure it is to test, repeatedly, for the ways a system can fail. An untested system is not necessarily an unsafe one, but it is an unknown one, and unknown is a difficult position to defend in front of a regulator, a major customer, or your own board.
Why AI does not behave like the software you already know how to check
Traditional software does what someone wrote it to do. Give it the same input and it returns the same output, and when it goes wrong an engineer can usually trace the fault to a line of code. Decades of quality practice, code review, sign-off, version control, were built for systems that behave this way. AI does not. A model learns its behaviour from data rather than following written instructions, and that one difference breaks the familiar quality model in three places.
First, you cannot read it. There is no logic to inspect and no ledger to reconcile. The only way to learn what a model does in a given situation is to put it in that situation and observe the result.
Second, it fails unevenly. A model can be accurate on average and fail consistently for one group of customers or one kind of case. A credit model can look sound across a whole portfolio while mispricing a particular segment. A medical triage tool can be right most of the time and least reliable on exactly the rare presentations where being wrong is most dangerous. An average conceals this. Only testing aimed at the edges reveals it.
Third, it does not stay the same. The world shifts, and a model trained on last year’s data slowly loses accuracy, a problem known as drift. Applications built on a general-purpose model can change behaviour overnight when the provider updates that model underneath them, even though no one touched the application. And the newest systems do not only produce answers, they take actions: booking, paying, writing to records, calling other software. This shift from AI that suggests to AI that acts, often called agentic AI, brings failure modes with little precedent in ordinary software. One is prompt injection, where instructions hidden in a document, an email, or a web page are read by the system and quietly redirect what it does.
Taken together, these three properties point to one conclusion. A clean result at launch tells you little about the same system a quarter later. Testing cannot be a gate the system passes through once. It has to run before deployment, again at every release, and continuously while the system is live and the world around it keeps moving.
What counts as a real test
Not every exercise that produces a number deserves the name. A benchmark tells you how a model scores on a standardised exam, which is useful and nothing like the conditions of real use. A polished demonstration shows the system handling the cases it was built to handle. Neither answers the question a board is actually asking: what happens when the inputs are messy, the user is hostile, the case is rare, and the stakes are real.
A real test starts from the system’s actual job and the organisation’s own data and use cases. It probes the edges on purpose: unusual inputs, adversarial behaviour, the customer segments and failure types where being wrong costs the most. It produces results that can be reproduced, compared over time, and read by someone outside the team that built the system. And it is written down, because a result no one can see is not evidence.
That is the distinction in one line. A policy records what you intend. A test records what happened. When a regulator, a customer, or your own board asks how your AI behaves, only one of those is an answer.
This is already happening, and not because a regulator said so
It is tempting to read all of this as a compliance story, something forced on business from outside. That gets the order wrong. The pressure is coming first from inside the organisation.
AI has moved out of pilots and demos into the core of how companies operate, and anything that sits in the core has to be explained, defended, and scaled. A model that cannot be explained slows every decision that depends on it. A system that cannot be defended becomes a liability the moment something goes wrong and someone asks for the evidence. Capability that cannot be trusted does not get deployed at scale; it stays stuck in pilots. None of these are regulatory problems. They are operational ones, and they arrive the moment AI becomes business-critical, whether or not a regulator is watching.
The rules, where they exist, mostly confirm what operations already demand. The EU AI Act, the world’s first comprehensive AI law, in force since 2024 and arriving in phases, places its strictest obligations on high-risk systems, meaning AI used in areas such as hiring, lending, and law enforcement. In May 2026 the EU agreed to push the main deadlines back, to December 2027 for standalone high-risk systems and August 2028 for AI built into regulated products, pending formal adoption. The reason for the delay is the instructive part: the technical standards and testing tools companies needed in order to comply were not ready. The requirements did not change. High-risk systems will still have to show risk management, data quality, human oversight, and accuracy, and showing any of those, in practice, means testing for them.
The supporting machinery is being built in parallel. ISO/IEC 42001, published at the end of 2023, does for AI management what ISO 9001 did for quality and ISO 27001 did for information security: it sets out what a well-run programme looks like in a form an independent party can certify against. The NIST AI Risk Management Framework plays a similar role in the United States. In Singapore, the Global AI Assurance Sandbox, run by IMDA and the AI Verify Foundation, exists to build a working market for AI testing by pairing companies that deploy AI with specialist testers. An early pilot put the real applications of seventeen organisations, across roughly ten sectors including healthcare and finance, through structured testing, and the programme has since extended to agentic risks such as prompt injection and data leakage.
The market is following the same logic. Gartner expects spending on AI governance platforms to more than double between 2026 and 2030, from around 492 million to over a billion US dollars, while Forrester puts the broader category of AI governance software nearer 15.8 billion by 2030. The figures differ because the category is still taking shape, and even the large audit and advisory firms have begun building AI assurance practices. The common thread across all of it, operations, regulation, standards, and spending alike, is a demand for evidence that a system works, produced by someone other than the team that built it.
Tested is what safe means
Put the pieces together and the thesis resolves into something simple. For AI, tested is what safe means. Safety is not a property you can declare into existence with a policy or a principle; it is a property you establish by measuring it, and re-establish as the system and its world change. Governance that is not backed by evidence is documentation, and documentation has never made a single model behave.
This is the conviction AIDX is built on, and the layer of the problem we work on. We treat testing not as a one-off review but as continuous infrastructure: a modular, productised platform that checks AI systems and AI agents for the failures that actually occur, including inaccuracy, bias, brittleness under pressure, and unsafe action, from before deployment through every release and into live operation. The output is evidence of what the system actually did, not advice about what might go wrong, in a form the organisation can put in front of a regulator, a customer, or its own board, and act on. That is what turns a capable AI system into one a business can deploy, explain, and defend.
It is also where the hardest problems are moving. Testing a model that answers questions is difficult enough. Testing an agent that takes actions, chains tools together, and behaves a little differently each time it runs is a genuinely unsolved problem, and it is the one much of this field will be working on for the next several years.
For a leadership team, the place to start needs no technical depth. Three questions surface most of what matters. Which of our AI systems make decisions that could harm a customer or the business if they went wrong? When was each of them last tested, and against what? And if a regulator or a major customer asked us for the evidence tomorrow, could we produce it? Where the third answer is no, there is a testing gap, and testing gaps are the solvable kind.
If you are working out how to make AI testing a standing capability rather than a one-off exercise, that is the conversation we are built to have.