Anti-Money Laundering (AML) Model Part 2: Testing & Monitoring Best Practices

NATE REGIMBAL: Welcome to the Risk and Accounting Advisory Podcast. I'm Nate Regimbal, Digital Advisory Leader at Cherry Bekaert.

NATE REGIMBAL: Today on the Risk in Review Podcast, we pick up the second part of our series discussing AML model validation and optimization.

NATE REGIMBAL: With me today is Sam Halaby, leader in our Risk and Data Analytics Practice, and Dan Gallagher, a leader in our Information Assurance and Cybersecurity Practice.

NATE REGIMBAL: In this series, we unpack insights into the importance of using a methodology, data capture, and execution within the context of AML optimization and validation.

NATE REGIMBAL: In today's episode, we'll be discussing how to conduct AML model testing. Last time, we discussed the significance of having a validation and optimization methodology for AML models. If you missed that episode, check it out at cbh.com/podcasts.

NATE REGIMBAL: Dan and Sam, thank you both for joining me today. Let's jump right in.

NATE REGIMBAL: Dan, could you share with us a good approach to conducting model testing?


DAN GALLAGHER: Model testing is a critical step to ensure that your AML application is providing the most effective alerting possible. It should be conducted in two distinct phases: initial setup and model validation.


NATE REGIMBAL: Thanks, Dan. Let's start at the beginning. What should financial institutions focus on during the initial setup phase?


DAN GALLAGHER: During the initial setup phase, it's important for financial institutions to invest significant effort in ensuring that the model and rules are adequately designed to address the specific risks and activities of the institution.

DAN GALLAGHER: This involves working closely with the vendor and performing parallel monitoring where both manual monitoring and the AML model are used to identify suspicious activity.

DAN GALLAGHER: It's also helpful to conduct pre- and post-implementation reviews to ensure that the model and rules are set up correctly from the beginning.


NATE REGIMBAL: What should the focus be for the model validation phase?


DAN GALLAGHER: Once the AML application has matured, it's recommended to conduct a model validation, normally around the six-month time frame.

DAN GALLAGHER: Model validation should also be performed whenever major changes occur within the financial institution, such as mergers, new product introductions, or system conversions.

DAN GALLAGHER: During model validation, there are two primary testing areas to consider: data imported from the core banking application or other feeder systems to the AML application, and the alert rule setup and output.


NATE REGIMBAL: Could you elaborate on the specific aspects to consider in each testing area?


DAN GALLAGHER: In terms of data validation, it's important to test the integrity and accuracy of the data imported from the core banking application into the AML application.

DAN GALLAGHER: This includes reviewing mapping of transaction activity and ensuring that all relevant transactions are present. Some institutions may have certain transactions, such as fees or ACH prenotifications, that may not be considered true transactions and therefore may not be mapped.

DAN GALLAGHER: For alert validations, it's crucial to review the alert setup, ensuring that it aligns with the institution's size, risk, and transactional activity.

DAN GALLAGHER: It's also important to examine the alert output over an extended period to verify that the rules are firing appropriately based on the defined rule set and sample transactions.

DAN GALLAGHER: Additionally, a sample of institutional alert reviews should be examined to ensure proper steps are taken for cases and appropriate note-taking to close out alerts.

DAN GALLAGHER: Procedures related to changes in alerts should also be addressed.


NATE REGIMBAL: Are there any other testing steps involved?


DAN GALLAGHER: Yes. In addition to the primary testing areas, it's important to conduct further tests during the validation process.

DAN GALLAGHER: This includes reviewing user access, the change management process around alert configurations, high-risk customers, and the policies and procedures in place to ensure the model is adequately monitored and maintained.

DAN GALLAGHER: By following this two-level approach, financial institutions can ensure that their data and alerting systems are adequately set up during the initial phase and that periodic testing is conducted to enable the most valuable AML monitoring possible.


NATE REGIMBAL: Dan, thank you. Sam, can you describe the additional ongoing monitoring processes and their importance?


SAM HALABY: Ongoing monitoring processes are crucial to maintaining the effectiveness of the model. They need to reconfirm the model's purpose within the context of current business activity and assess the model's effectiveness.

SAM HALABY: Reconfirming the purpose means asking whether the rules we have are still valid for our operation.

SAM HALABY: Business operations are seasonal and change with new customers and new profiles. For example, the advent of cannabis banking brings higher cash transactions that may be normal for those customers but not for traditional customers, which can lead to many false positives.

SAM HALABY: Understanding the model's effectiveness means recognizing that AML detection does not end when an alert is triggered.

SAM HALABY: Once an alert is triggered, investigative resources must be applied to clear it, and those resources can become overwhelmed.

SAM HALABY: Many banks set models to medium severity to err on the side of caution, which can generate excess false positives.

SAM HALABY: Investigators must clear alerts in a timely manner; otherwise, breaching SLAs can have regulatory consequences and adversely affect customer relationships.

SAM HALABY: Management must continually assess why there are so many false positives and adjust model rules accordingly.


NATE REGIMBAL: That was a helpful overview of the why. How are these periodic reviews performed?


SAM HALABY: The frequency of reviews is often a compliance matter driven by the regulator.

SAM HALABY: The tricky part is finding the optimal balance for accurate detection while reducing false positives. Most choose to err on the side of false positives because the risk of missed detections is higher than the cost of investigating false positives.

SAM HALABY: As an initial step, assess detection rules' effectiveness against alert dispositions to fine-tune thresholds and parameters.

SAM HALABY: Organize alert outcomes and run statistical methods to determine density and variance to learn what is causing those alerts and where they originate.

SAM HALABY: Once you identify potential adjustments, backtest the results.

SAM HALABY: For example, if a rule triggers on transactions over $10,000 but produces many false positives without detecting suspicious activity, incrementally test higher thresholds, $11,000, $12,000, $13,000, to see detection volumes and whether suspicious activity is still caught.

SAM HALABY: These incremental parameter changes allow you to move thresholds and potentially reduce false positives while maintaining detection of the same suspicious activity.

SAM HALABY: After determining appropriate parameters, work with compliance, IT management, and model management to introduce changes within model governance and change procedures.


NATE REGIMBAL: That helps illustrate the marriage between model sensitivity, configuration, and finding the organization's position on the risk continuum, balanced against the number of false positives you're comfortable dispositioning.

NATE REGIMBAL: What are the key limitations, and how do we solve these problems? The elements of AML all seem very tied together.


SAM HALABY: They're all tied together because suspicious activity detection is sensitive to technical, data, and operational factors.

SAM HALABY: Technically, most institutions use commercial platform models rather than building their own, as building a model is highly complex. Those vendor models vary in analytical methods, assumptions, and configurability.

SAM HALABY: On the data side, common issues include data entered in the wrong fields, such as a country name in the address field, notes in the second address field, or a beneficiary name in a comment field.

SAM HALABY: Models can't always look in every possible place for trigger words without generating many false positives.

SAM HALABY: Dirty or misaligned data leads to increased hits and false positives, which is costly. There is real ROI in cleaning up data, especially in banking and financial services.

SAM HALABY: Operational limits refer to the investigative team that must clear every alert. This requires human expert judgment and must occur within SLAs.

SAM HALABY: Resource constraints can lead to delayed reviews or inappropriate account actions if SLAs are breached.

SAM HALABY: Operational capacity must be considered alongside technology and data quality.


NATE REGIMBAL: The overall operation of technology, data, and process hinges upon data quality and currency of business rules, both aligned to your business model.

NATE REGIMBAL: There's a cascading effect, and work is required to keep it current, balanced, harmonized, and ahead of emerging threats.

NATE REGIMBAL: Let's get more forward-looking. As you know, I've got a background in artificial intelligence. It's now mainstream and can be incorporated into enterprise workflows and applications.

NATE REGIMBAL: Sam, what are clients asking us to help them solve?


SAM HALABY: We refer back to the three pillars: technical, data, and operational.

SAM HALABY: On the technical side, clients want better detection through advanced techniques using AI and machine learning, but they are limited by who owns the model and the model's internal rules.

SAM HALABY: If you didn't build your own model, vendor rules are often fixed.

SAM HALABY: What we can do is apply machine learning to the cases generated by those systems.

SAM HALABY: Post-alert analytics can apply metrics to estimate the likelihood that an alert is a false positive. In essence, the system raises the alert, and we can set up an "instant replay" to analyze the case post-alert.

SAM HALABY: On the data side, maintain data hygiene. Ensure data is accurate, complete, consistent, and properly aligned to the right attributes.

SAM HALABY: Model and data governance should cover this scope.

SAM HALABY: On the operational side, because institutions may have limited ability to change vendor model analytics, there is opportunity to optimize and continually improve the investigative process.

SAM HALABY: We can integrate with client systems and use technologies like robotic process automation to reduce time in alert triage.


NATE REGIMBAL: Can you talk more about RPA and driving efficiencies using technology?


SAM HALABY: I call it smart automation.

SAM HALABY: After an alert, investigative resources must collect relevant data from internal systems, transaction data, customer profiles, history, and external sources like LexisNexis and Google.

SAM HALABY: That data must be packaged and presented as an organized document.

SAM HALABY: RPA can gather that data, package it, and prepare a document ready for investigation.

SAM HALABY: RPA can also analyze the assembled content and provide a likelihood that the alert indicates suspicious activity or is a false positive.

SAM HALABY: This streamlines the investigator's work and supports decision-making.


NATE REGIMBAL: That's a great example: using RPA to automate data collection and analytics to indicate the content's likelihood of being a true alert, helping downstream investigators prioritize.


SAM HALABY: If you cannot modify the vendor model's internal analytics, RPA and smart automation can save significant time.

SAM HALABY: For example, a smaller client might manage over 2,000 false positive alerts a month. Clearing each alert can consume one to two hours per case, requiring many FTEs.

SAM HALABY: If RPA reduces that time to minutes, you've optimized resources and can repurpose staff for higher-value compliance work.

SAM HALABY: Our expertise in process and industry technology can reduce significant amounts of time for clients.


NATE REGIMBAL: This looks like a strong AI and machine learning use case.

NATE REGIMBAL: Not only can we analyze each case assembled by RPA, but we can also incorporate investigator dispositions as feedback, true signal or false positive, to train models.

NATE REGIMBAL: The outcome is flagging cases more likely to be real alerts and prioritizing investigative workload, while retaining a human in the loop.

NATE REGIMBAL: How should banks approach this, and how would they work with Cherry Bekaert?


SAM HALABY: We help clients strategically and tactically across the entire model validation spectrum, including validation, model review, and the investigative end to end.

SAM HALABY: This requires a holistic approach to address gaps in data quality, governance, and operating procedures and to brainstorm opportunities for innovative approaches, including automation and AI.


NATE REGIMBAL: Thank you both, Sam and Dan, for your insights.

NATE REGIMBAL: Thank you to our audience for listening. Please like, share, and subscribe to the Risk and Accounting Advisory Podcast.

NATE REGIMBAL: Visit us at cbh.com/risk to learn more. Thank you.