A banking application can pass thousands of automated tests and still behave unexpectedly when artificial intelligence is introduced into its operations. Consider an AI agent assigned to investigate a suspicious payment. It retrieves transaction records, examines customer activity, checks internal policies and recommends whether the payment should proceed. The system may perform each individual task correctly, yet arrive at the wrong conclusion because it misinterprets a policy, overlooks a relevant transaction or places too much confidence in incomplete information. Conventional software testing may confirm that the APIs work, the data is retrieved successfully and the application responds within the required time. What it cannot necessarily establish is whether the agent has made a sound decision. This distinction is becoming increasingly important as banks and their technology providers move beyond AI-powered chatbots towards systems capable of conducting investigations, preparing credit assessments, managing operational exceptions and executing increasingly complex workflows. The challenge is not that established testing methods have become obsolete, but that they were never designed to evaluate software capable of interpreting information and choosing its own course of action.
A Different Kind of Software Failure
Most banking software is developed around a relatively straightforward principle: given a particular input and a defined set of conditions, the application should produce an expected result. A payment processing system must calculate the correct amount, debit and credit the appropriate accounts, apply relevant checks and maintain accurate transaction records. If any of these steps produces an incorrect result, a test can identify the discrepancy.
The introduction of AI creates another category of failure that is considerably harder to detect. A system may be technically functioning as intended while producing an incorrect interpretation or recommendation. The problem may not originate in faulty code or a failed integration, but in the way an AI model understands an instruction, evaluates retrieved information or decides which action to take.
Take an AI application designed to support commercial lending. It may extract information from financial statements, compare performance across reporting periods and prepare an initial credit assessment. Traditional tests can verify whether the application retrieved the correct documents and whether numerical calculations were performed accurately. They may not establish whether the model has properly understood the relationship between declining revenue, rising receivables and a company’s ability to service its debt.
The distinction becomes more significant when AI systems are allowed to initiate subsequent actions. An incorrect interpretation could lead an agent to request additional information unnecessarily, recommend an inappropriate credit limit or escalate an otherwise routine case. Each individual software component may operate correctly while the overall workflow produces an undesirable outcome.
For banks, this raises an uncomfortable question about the meaning of a successful test. An application that returns a technically valid response is not necessarily an application that has reached the right conclusion.
Why the Same Test Can Produce Different Results
Traditional automated testing relies heavily on repeatability. Developers run the same tests after software changes to confirm that previously working functionality continues to behave correctly. This is the foundation of regression testing, an essential practice for banks maintaining complex applications that support millions of transactions.
Generative AI introduces variability into that process. Depending on the model, its configuration and the surrounding application, the same instruction may produce different responses. The wording may change, the reasoning presented may differ or the system may select another sequence of steps to complete the task.
This does not automatically indicate a defect. An AI assistant asked to summarise a customer’s financial position could produce two different summaries that are both accurate. A conventional test comparing the response against one predefined answer might incorrectly classify the alternative as a failure.
The opposite problem is more concerning. A response may resemble the expected answer closely enough to pass a superficial evaluation while containing an important factual omission or an unsupported conclusion.
Software testing teams therefore need ways to assess meaning and behaviour rather than relying exclusively on exact output comparisons. In a banking context, that could involve checking whether a credit summary accurately represents the underlying financial information, whether a compliance response is supported by the relevant policy and whether an agent has remained within its authorised responsibilities.
The problem becomes more complex when models are updated. A newer version may improve performance on common tasks while behaving differently in unusual situations. Even changes to system instructions, retrieval settings or external tools can affect outcomes without any modification to the underlying model.
Banks consequently need to treat AI testing as an ongoing activity rather than a process completed before an application enters production.
Agentic AI Introduces Risks That Individual Component Tests Cannot Capture
The growing interest in agentic AI is making this challenge more immediate. Unlike applications that simply generate answers, AI agents can use tools, retrieve information from multiple systems and determine the next steps required to complete an assigned task.
A banking operations agent might receive an instruction to investigate a failed corporate payment. It could query the payment platform, examine account information, check transaction limits, review internal procedures and prepare a recommendation. In a more automated environment, it might also initiate an approved corrective workflow.
Each connection can be tested independently. The bank can confirm that the payment API returns accurate information, the account service enforces access permissions and the workflow engine processes authorised requests correctly.
What becomes harder to validate is the sequence of decisions connecting these components.
An agent might retrieve the correct transaction but consult an outdated policy. It might interpret an ambiguous error message as evidence of insufficient funds rather than a temporary processing failure. It could repeat a tool call unnecessarily or attempt a corrective action before all required checks have been completed.
These failures may emerge only when several conditions occur together, making them difficult to reproduce through conventional testing scripts.
The use of the Model Context Protocol (MCP) and other tool-integration approaches adds another consideration. These technologies can make enterprise systems accessible to AI applications through standardised interfaces, but the ability to call a tool does not establish whether an agent should call it in a particular situation.
For banks, authorisation must remain independent of the model’s judgement. A system may recommend an action, but the underlying banking infrastructure must still enforce transaction limits, permissions, approval requirements and other controls.
This separation is particularly important for activities involving customer funds, sensitive information and regulated decisions. Testing must establish not only whether an agent completes its assigned task, but also whether it respects the boundaries within which that task is permitted.
The Testing Environment Needs to Reflect Real Banking Conditions
One reason AI applications can perform well during demonstrations but encounter difficulties in production is that controlled testing environments rarely capture the full complexity of banking operations.
A model trained or evaluated using carefully prepared documents may struggle when confronted with incomplete customer records, inconsistent terminology, unusual transaction patterns or information distributed across several legacy systems. A credit assessment application may work reliably with standard financial statements but misinterpret scanned documents, unusual accounting classifications or contradictory information from different sources.
The difficulty is not limited to data quality. Banking decisions often depend on context that is difficult to represent through simple test cases.
A transaction that appears suspicious in one customer account may be entirely normal for another. A temporary decline in cash flow may indicate financial distress for one SME but reflect a predictable seasonal pattern for another. A regulatory requirement may apply differently depending on the jurisdiction, customer classification or type of financial product.
Testing environments therefore need to include realistic variations rather than only examples that demonstrate the application’s intended functionality.
Synthetic data can help banks construct scenarios involving unusual customer behaviour, transaction failures and combinations of operational conditions that may be difficult to obtain from production records. However, synthetic data also has limitations. If it does not adequately reflect the complexity of actual banking activity, the resulting tests may create a misleading impression of model reliability.
Historical cases can provide another source of evaluation material, particularly where banks have verified outcomes and supporting documentation. These datasets require careful governance because they may contain sensitive customer information and reflect decisions or practices that are no longer appropriate.
A more robust approach combines controlled test scenarios, representative historical examples, simulated operational failures and carefully governed testing against live system behaviour.
The objective is to understand how an AI application responds when information is imperfect, circumstances are unusual and the correct course of action is not immediately obvious.
Why Banks Need More Than One Definition of Accuracy
Accuracy is often presented as the principal measure of AI performance, but it can be an incomplete indicator of whether a system is suitable for banking operations.
An AI application that correctly classifies 95% of test cases may appear effective. Yet the remaining 5% could contain errors with very different consequences. Incorrectly categorising a routine customer enquiry is not equivalent to overlooking a suspicious payment or producing an unsupported conclusion in a credit assessment.
Banks need evaluation frameworks that distinguish between the frequency of errors and their potential impact.
For an AI application assisting with financial crime investigations, relevant measures could include its ability to identify genuinely suspicious activity, avoid unnecessary alerts and provide explanations supported by transaction evidence. A system used in lending may require evaluation of financial interpretation, consistency across comparable cases, adherence to credit policy and the quality of information presented to human decision-makers.
Agentic applications introduce additional measures. Banks may need to examine whether an agent selected appropriate tools, followed the required approval sequence, avoided unnecessary actions and stopped when it encountered information outside its authority.
Testing should also distinguish between an incorrect recommendation and an unauthorised action. The first may require human review or model improvement. The second indicates a failure in the surrounding control architecture and may have more immediate operational consequences.
This suggests that banks will increasingly need risk-based evaluation frameworks tailored to the functions their AI applications perform. A customer service assistant, fraud investigation agent and autonomous payment operations tool should not be assessed against an identical set of performance criteria.
The Emerging Role of AI Evaluation Frameworks
Technology providers are developing tools designed to evaluate AI applications beyond traditional software testing. These approaches often involve running models against structured datasets, assessing the quality of generated responses and measuring how reliably systems follow instructions or use external tools.
For banks, one useful technique is to establish a collection of verified scenarios against which an AI application can be repeatedly evaluated. These may include genuine financial documents, carefully constructed transaction cases, policy interpretation exercises and operational exceptions.
The expected result does not always need to be a particular sentence or sequence of words. Instead, an evaluation can determine whether the response contains the necessary facts, reaches an acceptable conclusion, cites appropriate evidence and avoids prohibited actions.
Some frameworks use other AI models to assess generated responses, a technique commonly described as LLM-as-a-judge. This can help evaluate large volumes of outputs that would be impractical to review manually.
However, using one AI model to evaluate another introduces its own limitations. Automated judges can make mistakes, apply inconsistent standards or fail to recognise subtle financial errors. Banks cannot assume that an assessment becomes reliable simply because it is performed by another model.
For higher-risk applications, automated evaluation should be supported by deterministic checks, verified reference data and human review. The evaluation system itself also needs testing to establish whether its conclusions are dependable.
There is a further opportunity to use metamorphic testing, which examines whether an application’s behaviour remains appropriate when selected inputs are changed. For example, an AI credit assessment should not reach a materially different conclusion merely because the order of paragraphs in a financial report has changed. Similarly, an investigation agent should not disregard established risk indicators because a customer explanation is written more persuasively.
These techniques can expose weaknesses that conventional input-output tests may overlook.
Security Testing Must Extend to Instructions and Retrieved Information
AI introduces security risks that do not resemble conventional software vulnerabilities.
A traditional application may be compromised through insecure authentication, poorly protected APIs or weaknesses in its underlying code. AI systems face those risks as well, but they can also be influenced by malicious instructions embedded in information they are asked to process.
A banking AI assistant reviewing an uploaded document could encounter text instructing it to ignore previous directions, reveal confidential information or perform an unrelated action. This is commonly known as prompt injection.
The threat becomes more serious when an AI agent has access to tools that can retrieve sensitive records or initiate operational workflows. Malicious instructions may be placed in documents, messages, retrieved web content or other lower-trust material that the agent encounters while carrying out a legitimate task.
Banks therefore need to test whether their AI applications distinguish between authorised instructions and untrusted content. They must also verify that underlying systems enforce access controls even if the model attempts to perform an inappropriate action.
Security assessments should include attempts to manipulate the model’s behaviour, extract protected information, misuse connected tools and bypass approval requirements. These tests need to examine the complete application, including retrieval systems, prompts, APIs, identity controls and workflow permissions.
An AI application that resists malicious prompts in isolation may still be vulnerable when connected to a broader enterprise environment. The security of the model and the security of the surrounding architecture cannot be assessed separately.
Model Updates Are Creating a New Regression Testing Problem
Banks are accustomed to controlling software releases through established development and change management processes. A modification to a payment application can be reviewed, tested and approved before deployment.
AI introduces additional sources of change that may affect application behaviour without a conventional software release.
A model provider may introduce a new version, a retrieval database may receive updated documents or an AI application may be configured to use a different model for cost or performance reasons. Changes to prompts, tool descriptions and model parameters can also influence how the system responds.
An application that performed satisfactorily during its original evaluation may therefore behave differently after seemingly minor adjustments.
This creates a need for AI-specific regression testing. Banks should maintain representative evaluation datasets and rerun relevant tests whenever changes could affect the system’s behaviour. For critical workflows, comparisons should examine not only overall performance but also whether previously identified weaknesses have returned.
Version control becomes particularly important. Banks need to know which model, instructions, tools, retrieval sources and application configuration produced a particular result. Without this information, investigating an unexpected decision or reconstructing an operational incident can become difficult.
For technology providers delivering AI-enabled banking products, this could significantly change software maintenance responsibilities. Product updates may require evidence of behavioural testing alongside conventional functional, integration and security testing.
Production Monitoring Becomes Part of Software Quality Assurance
Even extensive pre-deployment testing cannot reproduce every situation an AI application may encounter in a live banking environment. Customer behaviour changes, financial products evolve, policies are updated and external systems introduce new information.
AI quality assurance therefore needs to continue after deployment.
Banks can monitor indicators such as the frequency of unsupported responses, failed tool calls, unusual workflow sequences, human overrides and deviations from expected performance. Changes in the characteristics of incoming data may also indicate that an application is operating under conditions different from those represented in its original testing.
For some systems, banks may initially choose a shadow deployment, where the AI produces recommendations alongside an existing process without influencing actual customer outcomes. Comparing these recommendations with established decisions can provide evidence about performance before broader automation is introduced.
Other applications may operate with human approval requirements, particularly where actions involve financial consequences or customer rights. The level of autonomy can be adjusted as the institution develops confidence in the system’s reliability and control environment.
Importantly, monitoring should not become an exercise in collecting large volumes of technical logs without understanding their significance. Banks need clear thresholds for investigation, defined responsibilities and mechanisms for suspending or restricting applications when unexpected behaviour emerges.
The ability to stop an AI agent safely may ultimately be as important as its ability to complete a task successfully.
Banking Technology Providers Face a Different Product Assurance Challenge
The implications extend well beyond banks’ internal technology teams. Core banking providers, payment technology companies, financial crime platforms and enterprise software vendors are increasingly incorporating AI capabilities into existing products.
These vendors may already operate mature software testing and certification processes. However, the introduction of generative AI and autonomous agents changes what customers need to understand before deploying a solution.
A bank evaluating an AI-enabled product will increasingly need evidence of how the vendor tests model behaviour, manages updates, validates tool permissions and handles unexpected outcomes. It may also need to understand whether the system uses proprietary models, third-party AI services or a combination of technologies.
This could make AI assurance an important area of competition among banking technology providers.
Vendors that can demonstrate reliable evaluation processes, reproducible testing, detailed audit records and effective controls over autonomous actions may have an advantage over those offering AI functionality without sufficient operational transparency.
There is also an opportunity for specialist technology companies providing AI testing infrastructure. Evaluation platforms, synthetic scenario generators, agent simulation environments, model monitoring tools and automated security testing could become increasingly important components of enterprise banking technology.
The opportunity is not simply to create more testing tools. It is to help banks establish confidence in systems whose behaviour cannot be exhaustively predicted through conventional software engineering methods.
What Changes When Banks Can No Longer Test Every Possible Outcome?
Banking technology has always involved managing complex systems and accepting that no testing programme can eliminate every possible failure. AI does not create that limitation, but it changes the nature of what must be evaluated.
The central issue is that intelligent applications can produce incorrect outcomes without encountering a conventional software defect. Their decisions may depend on incomplete information, changing model behaviour and sequences of actions that are difficult to anticipate during development.
Banks will continue to need the deterministic controls that underpin reliable financial infrastructure. Account balances, payment execution, access permissions and regulatory requirements cannot be left to probabilistic interpretation. AI can assist with decisions and workflows, but the systems surrounding it must establish clear boundaries and enforce them consistently.
This creates a different role for software testing. Rather than confirming only that an application performs its specified functions, testing teams must also evaluate how it behaves under uncertainty, how it handles conflicting information and whether its actions remain appropriate when circumstances change.
The banks that address this effectively will not necessarily be those with the most advanced AI models. They may be the institutions that develop the strongest engineering practices for understanding, controlling and validating how those models operate within real financial systems.
What this means for the industry
- Traditional testing remains essential but is no longer sufficient. Banks must continue validating deterministic financial processes while introducing additional methods to assess AI decisions, interpretations and autonomous behaviour.
- AI evaluation is becoming a distinct engineering discipline. Testing frameworks will need to measure factual accuracy, policy compliance, tool selection, decision consistency and the consequences of incorrect outcomes.
- Agentic AI increases the importance of end-to-end testing. Individual APIs may function correctly while an AI agent follows an inappropriate workflow or attempts an unauthorised action.
- Security testing must address prompt injection and tool misuse. Banks need to evaluate how AI applications respond to malicious instructions and ensure that permissions are enforced independently of model behaviour.
- Testing will increasingly continue into production. Model updates, changing data and evolving workflows require continuous monitoring, regression evaluation and controlled deployment practices.
- Banking software vendors face new assurance expectations. Demonstrable AI testing capabilities, auditability and operational controls could become important differentiators in enterprise technology procurement.
