AI models can fail — not because they’re inaccurate, but because they’re ungoverned.
From biased healthcare tools to flawed public sector algorithms, the risks of poorly managed AI are real, measurable, and preventable. Data scientists and technical teams urgently need a structured, proven way to control these risks — not just comply with regulations.
In his session ‘Hands-on AI Management for Data Scientists’, Jos Gheerardyn (CEO of Yields) shares how Model Risk Management (MRM) frameworks from finance offer a practical and scalable foundation for AI governance.
Why managing AI Risk Governance matters
AI systems are increasingly embedded in high-stakes decisions: diagnosing patients, allocating welfare, detecting fraud. When these systems fail, the consequences are not just technical—they’re social, legal, and ethical.
Three real-world examples illustrate this:
- Melbourne Monolith – A generative model produced a surreal skyscraper design due to flawed geospatial input. While harmless, it revealed how data quality issues can derail entire systems.
- Pulse Oximeter Bias – Research in Nature (2022) exposed that pulse oximeters perform less accurately on darker skin tones, resulting in systemic healthcare disparities during the COVID-19 pandemic.
- Rotterdam Welfare Scandal – A Dutch algorithm misclassified vulnerable citizens as fraud suspects, largely due to biased assumptions baked into the model.
These are not failures of performance, but failures of governance—driven by weak validation, hidden biases, and poor accountability.
Compliance ≠ Risk Control
Meeting regulatory requirements isn’t the same as managing risk. A model can be fully documented and still harmful. This is especially true when systems reinforce existing biases or produce outcomes that are technically valid but socially unacceptable.
Regulatory frameworks like the EU AI Act and SR11-7 (U.S. Federal Reserve) are beginning to close this gap by creating consequences for ignoring risk. But real control starts internally—with process, structure, and ownership.
A Framework for Model Risk Management (MRM)
Model risk management, developed over decades in finance, provides a proven structure for AI governance. Its core components include:
- Model Definition: Clearly establish what qualifies as a “model.” For instance, an AI model predicting loan defaults is in scope, but a basic spreadsheet formula may not be.
- Risk Tiering: Classify models based on complexity and potential impact. A marketing recommendation engine poses very different risks compared to a medical device algorithm.
- Model Inventory: Maintain a comprehensive and auditable registry of all models, including ownership, assumptions, usage context, and development status.
- Process Standards: Enforce clear development and validation procedures. This includes defining success metrics before model development begins.
- Independent Review: Assign a separate validation team to review models for accuracy, documentation quality, and risk compliance, similar to peer review in science.
- Risk Appetite: Executive leadership must define acceptable risk thresholds. Visualization tools like heatmaps are often used to communicate this across teams.
Implementing Risk Management using MLOps
The principles above can be embedded into daily workflows through modern MLOps platforms. Key practices include:
- Data Quality Checks: Set rules to ensure datasets are representative and up to date, especially in dynamic environments like financial markets or pandemics.
- Data Versioning: Use tools such as deep clone to track how data samples evolve over time.
- Validation Recipes: Apply predefined sets of performance metrics and KPIs to standardize model evaluation.
- Model Registry Integration: Only register models once all assumptions, validations, and documentation are complete, ensuring accountability and traceability.
Addressing bias and fairness in AI
Fairness in AI is not just a question of removing sensitive attributes. Governance must address how models infer bias through proxies like location or behavior. A robust fairness strategy includes:
- Define Fairness: This should be tailored to the context. For example, in Belgium, fairness may include linguistic balance between Dutch and French speakers.
- Identify Proxy Variables: Removing sensitive features doesn’t guarantee fairness. Models may still infer protected attributes indirectly, for instance, through geographic data.
- Use Fairness Metrics: Standard measures like demographic parity, equal opportunity, and equalized odds help quantify fairness across different groups.
- Audit Both Data and Outputs: It’s not enough to check the inputs. Model predictions must also be evaluated for unintended consequences.
Conclusion
Governing AI effectively requires structure, not slogans. The financial industry’s approach to model risk offers tested, practical tools for data scientists and AI teams. Compliance is only the beginning—governance is what makes AI trustworthy at scale.
