Prioritizing model validation tasks

Large banks have over 3000 models in production, and all of these algorithms require periodic review (re-validation). Given that an average validation typically takes around 4 – 6 weeks, this is a huge undertaking. Cycling through all models in a sequential manner might therefore not be the best solution because different models carry a different amount of model risk. A better strategy is the so-called risk-based prioritization.

In this approach, a validation team organizes the (periodic) review by taking into account the risk associated to each given model. This is most often implemented by using so-called model risk tiering. The tier of a model is determined through qualitative and quantitative measures such as materiality, risk exposure and regulatory impact. Typically, qualitative assessments such as model complexity are fairly constant over time. Quantitative metrics such as the amount of risk that is managed through the model can change quickly since those quantities are dependent on the context in which the model is used. This is the main driver for using quantitative methodologies as part of the model risk tiering.

When it comes to quantification, there are a few important concepts that have been introduced long ago by Knight. Let’s assume we have a quantity that we would like to model (such as the NPV of a swaption). This quantity has a (finite or infinite) number of potential outcomes. We talk about uncertainty when we do not know the probability of each of the outcomes, while we use the term risk when we do have such knowledge.

When it comes to building models, let us assume that we have a set of candidate models that can be used to describe the behavior of the quantity of interest. When we know the likelihood that a given model is the right one, we talk about model risk, while otherwise we use the concept of model uncertainty. We speak about model ambiguity whenever there are multiple probability measures available on the set of models P.

In case we know the likelihood of different outcomes, we can apply standard management practices like diversification to manage the risk. When a probability distribution however is not known, a typical approach is to work off the worst case and put aside a certain amount of capital as an insurance premium.

In the case of model risk quantification, a fairly standard approach is to use Bayesian analysis. In that case, we start by introducing for every model a prior distribution over the model parameters indicating the likelihood that any particular set of model parameters is correct. As we moreover have multiple models (each with their own set of parameters) we also have to assign a prior distribution indicating how likely it is that a given model is the correct one. This procedure allows for expert input since a model validator can quantify which model, according to her or his experience, is the right one.

As we then observe different outcomes of the quantity that is being modeled, we can compute the posterior probability on each candidate model which is the likelihood that the model is true, after having observed the outcomes x. This allows us to update our knowledge about the various models as we collect more observations. Using these Bayesian probability distributions, we can then compute model dependent quantities by taking the expectation over all available models and integrating over the distribution of the parameters. Concretely, by applying this approach we can compute the expected value of an option as it would be computed through a Black-Scholes formula, Heston model and a local volatility model. The same approach also allows us to compute uncertainty measures on this expected value such as the standard deviation on the NPV of an option which could serve as a measure of model risk.

When there are no probability distributions available, we can quantify model uncertainty using the so-called max min principle. Assume that an agent has several scenarios at his disposal. With the max-min principle, he will choose the scenario that maximizes his expected utility. For the expected utility, we use the worse case over all available models (the minimal outcome). Concretely, if we are long an option, and we have Heston, local volatility and Black-Scholes available, each with say three possible parameter choices, then according to the above, we should value the option by applying the formula (and parameters) that lead to the lowest possible NPV (let us call this NPV[w]). If moreover we have the possibility to sell the option at a price P, then using the max min principle, we will sell whenever P>NPV[w].

In conclusion, the need to organize models by their riskiness is driven by the fact that banks want to optimally allocate resources to the right models. Although such model tiering is driven both by qualitative and quantitative features, we have focused on the latter since these aspects tend to vary in a more dynamic fashion. We have highlighted two well-known methods. The Bayesian approach leads to more stable outcomes (due to model averaging) but is hard to implement (since assigning prior distributions is a difficult task). The max-min (i.e. worst case) approach is often encountered in the industry but due to its nature may lead to more volatile outcomes.