The Solution: Model Risk Management
Model Risk Management (MRM), with its blend of qualitative and quantitative aspects, can significantly mitigate such risks.
A cornerstone of MRM is to ensure that the model lifecycle process is well-designed and appropriately implemented. A typical lifecycle process is indicated in Fig. 1 showing various steps such as the ideation/prototyping phase, an independent validation phase after development, deployment, production and retirement. Each of these phases consist in itself of various subprocesses.
Fig.1: A typical model lifecycle process
One aspect of the lifecycle process is to identify clear conditions for when a new independent review of the algorithm is necessary. In the case of the lunar lander, a significant change in the flight path during approach qualifies as a material change warranting a new independent review.
Determining whether a change of the data is material, is a question that is frequently encountered in modelling, and depending on the field it is called a data representativity, a data drift, or a covariance shift issue.
To illustrate this in the case of the Hakuto-R Mission, the task at hand is to decide whether the time series generated by the sensors during approach of the new landing site would significantly deviate from time series corresponding to the previous landing site. One way to analyse this involves computing the statistical fingerprint of the time series of simulated approach data and conducting a clustering exercise to identify which types of paths have been tested. If the new landing site produces time series significantly different from the earlier ones, the clustering algorithm will assign the new data to a novel cluster.
This is illustrated in Fig. 2 where we have built a toy model to test with two types of trajectories (landing on a flat surface (Cluster 0) and landing over a region with a shallow well (Cluster 1)). The new trajectory (shown as Cluster 2) represents a cliff and is clearly different from the others.
Fig.2: Studying various landing trajectories