Model risk management (MRM) is the art of handling the inherent uncertainty related to mathematical modeling. We create algorithms for many different reasons. In the past, most models were built to study the evolution of dynamical systems (e.g. a credit risk model or a valuation model). Models were often created via a first-principles approach with analytical tractability in mind. Nowadays, ML models are everywhere, impacting both our individual behavior and changing the dynamics of entire societies. With such a persistent use of models, understanding the risks involved becomes mandatory since the consequences of model failure can be massive.
This is why there is a continuously growing pressure from governments and regulators to increase requirements for MRM and improve AI governance. Because of this evolution, financial organizations are looking at technology to address these challenges. In the current white paper we expand on this topic.
Data quality impacts model output in complex and deceitful ways. The common proverb “garbage in, garbage out” summarizes this, albeit in a rather trivializing fashion. Wrong data may produce wrong results, but this describes only the most straightforward case. Nonsensical data can produce incorrect results that are not obviously garbage: the effects of poor data are often surreptitiously difficult to detect. In addition, just as with any more common software bug, the effects of poor data sometimes only materialize far from the source. Since models can be chained, where the output of model X becomes the input of model Y (e.g. in market risk models) these effects propagate making the root cause very difficult to identify. In addition, models often depend on their data in a highly non-linear fashion, so that even minor data errors can produce arbitrarily serious problems.
After periods of hype followed by several “AI winters” during the past half century, we are experiencing an AI summer that might be here to stay. AI now drives many real-world applications in the financial sector, ranging from fraud detection to credit scoring. Embracing AI promises considerable benefits for businesses and economies through its contributions to productivity and innovation. At the same time, the potential challenges to adoption, including governance, workforce impacts, and other social concerns cannot be ignored. In the present post, we focus on these challenges from a model risk perspective. We will illustrate many aspects in the context of credit models.
A key challenge in model risk management is to quantify model uncertainty. The slide deck was used by Jos Gheerardyn in various courses on this topic. It highlights which ML techniques can be used to make quantification more efficient.