OSFI to finalize risk management guidance covering the use of AI and machine learning within models

Financial institutions and technology vendors alike are anticipating the release by the Office of the Superintendent of Financial Institutions (OSFI) of its updated guidance on “model” risk management. Draft guidance published in the fourth quarter of 2023 significantly expands OSFI’s current guidance [PDF] (Guideline E-23) and has important implications for the use of artificial intelligence (AI) and machine learning (ML) by financial institutions.

The current version of Guideline E-23

The current version of Guideline E-23 outlines OSFI’s expectations for the establishment of an enterprise-wide model risk management framework by federally regulated financial institutions operating in Canada. The current guideline is noteworthy for its risk-based approach, with a clear emphasis on managing the use of models that create material risks to financial institutions. The current guideline applies only to deposit-taking institutions (i.e., banks, bank holding companies, federally regulated trust and loan companies and cooperative retail associations).

In the current guideline, the terms “model” and “model risk” are defined in a targeted manner. Model is defined to mean the application of certain techniques to process input data to generate quantitative estimates that are useful and meaningful to business lines and control functions.^[1] Model risk is defined as the risk of adverse financial and reputational consequences arising from the design, development, implementation or use of a model; and under the current guideline, financial institutions are expected to focus model risk management policies and processes on models that could materially impact their risk profile.

OSFI’s proposed changes to Guideline E-23

If adopted in its current form, OSFI’s updated version of Guideline E-23 (the draft guideline) will introduce a number of meaningful changes. In particular

• the draft guideline applies to all federally regulated financial institutions (FRFIs) (including authorized foreign bank branches, federally regulated insurers and federally regulated private pension plans)
• the draft guideline applies to all analytical models used by FRFIs, regardless of whether they require formal regulatory approval or not, including models for financial risks, pricing products and services, business optimization, climate risks, cyber and tech risks, and digital innovation risks
• the draft guideline defines “model” expansively to cover the application of certain techniques to process input data to generate results,^[2] without a requirement that the generated result provide a quantitative estimate. Additionally, AI/ML methods are explicitly called out as being within scope
• the draft guideline defines “model risk” to include risk of adverse financial, reputational or operational risk, materially expanding the definition from the current guideline, which limits the definition to a risk of both adverse financial and reputation consequences
• most significantly, the draft guideline includes an expectation that financial institutions deploy a detailed model-risk management framework that covers each stage of a model’s lifecycle, regardless of whether use of the model creates a material risk to the financial institution

Osler’s assessment

The changes to Guideline E-23 reflect OSFI’s increased focus on operational risk and are, understandably, directed at the recent, rapidly evolving reality that financial institutions are increasingly relying on models to support decision-making. However, as currently drafted, OSFI’s attempts to address this reality may go too far. Areas of particular concern include

• OSFI’s expectation that a financial institution applies a detailed model-risk management framework across its use of models generally, regardless of whether a particular use creates a material risk
• the expanded definitions of model and model risk, as described above, which are, as drafted, so broad as to capture activities within the financial institutions that would not present any meaningful risk
• OSFI’s inclusion of any AI/ML methods within the definition of the word model, without distinguishing between AI/ML models that make specific predictions or recommendations and AI/ML models that are used to generate content but not support decision-making.

We note that the draft guideline is not aligned with comparable guidance in other key markets. For example, guidance published by regulators in the United States[PDF] and the United Kingdom [PDF] apply to models used to make decisions in respect of key business functions.

Following its consultation, OSFI now has an opportunity to adjust the draft guideline when it issues a final version of Guideline E-23 (expected in July 2024). For many interested parties, the most material clarifications to be on the lookout for include

• a clarification that the objective of Guideline E-23 is to have a risk management framework applied to models used to support material decision-making
• a clarification of when Guideline E-23 does not apply, such as when an AI/ML model is used to generate content, rather than support material decision-making
• a clarification of OSFI’s expectation that a model risk management framework be applied only to uses of a model that create a material risk to the financial institution, rather than to all model uses

Interested parties will be watching closely in the hope that OSFI will make these important clarifications when it releases its updated draft. The ramifications if these clarifications are not introduced are significant, as an overly broad regulatory framework may make financial institutions in Canada less likely to adopt productivity-enhancing and risk-reducing technology than their competitors in other markets.

---

[1] Model is defined in the current guideline as follows: “A model generally refers to a methodology, system, and/or approach that applies theoretical and (expert) judgmental assumptions and statistical techniques to process input data in order to generate quantitative estimates. A model has three distinct components: i) a data input component that may also include relevant assumptions; ii) a processing component that translates the inputs into estimates; and iii) a result component that presents these estimates in a format that is useful and meaningful to business lines and control functions.

[2] Model is defined in the updated guideline as follows: “The application of theoretical, empirical, judgmental assumptions and/or statistical techniques, including AI/ML methods, which processes input data to generate results. A model has three distinct components: a. data input component that may also include relevant assumptions b. processing component that identifies relationship between inputs, and c. result component that presents outputs in a format that is useful and meaningful to business lines and control functions.