At Yields.io, we have not shared anything on the Covid-19 crisis for the simple fact that we are mathematicians, technologists and data scientists, not epidemiologists. And unless we start collaborating with experts in that field, we will follow Amanda Makulec’s advice and refrain from doing so – #vizresponsibly! However, models play an important role in understanding how we should react to such a crisis. As a model risk manager, dedicating my professional life to understanding the limitations of algorithms, I do want to contribute to that part of the equation.
In the face of exponential growth, we have to react fast which sometimes leads us to use information too quickly. A few examples: last week the results of preprints were disseminated by the press without passing the scientific rigor of peer review and policymakers in Belgium had to act on incomplete information leading to exactly opposing arguments whether or not to close schools. Understanding better the uncertainty around us, trying to quantify it, will lead to better risk management and this is exactly why model risk managers should contribute to the management of the current pandemic.
Model risk management is a discipline mostly encountered in the financial sector and attempts to control the inherent uncertainty when using mathematical models in decision making. Models can produce erroneous outputs because the algorithms contain mathematical mistakes, because we are using them on incomplete or false data, because the underlying assumptions do not match reality, etc. In order to control this risk, best practice suggests that every model has to be independently reviewed before it can be used. The depth and breadth of that review depends on the risks involved (the so-called model risk tier). In a way, independent model review is quite similar to peer review before your article is accepted for publication in a scientific journal. The art of model risk is to communicate clearly to the C-level the limitations and uncertainties in the models that drive the business forward.
The models that are being used to forecast the evolution of this epidemic are in the absolutely highest model risk tier – ever. We see data scientists developing ML algorithms on Kaggle and epidemiologists publishing complex network models based on small datasets. Before these results are used as a basis for policymaking and before the consequences impact health workers, these models should pass independent review. Even if this takes one additional day, the benefits are clear:
- Uncertainty on data and model assumptions would be clarified to – and better understood by – policymakers
- Quantifying the uncertainty would allow for better risk management
- If the uncertainty on our forecasts is clear, we can detect much faster which measures (such as different types of social distancing) work best
- We will be able to evolve our models faster, leading to more accurate predictions in the future which is critical when e.g. we have to decide when to stop lockdowns.
Conclusion
In conclusion, I would like to call on the modellers who are building the most critical algorithms of 2020 and the policymakers who use these models to reach out to model risk managers for help. I am sure that our entire community is ready to assist; we just need the dialogue to start.
Interested in learning more? Download here our white paper The Evolution of Model Risk Management.
