There are various types of climate models, including scenario expansion models for physical and transitional risk, hazard models, credit-impact-assessment models, and emission assessment models. Navigating the new climate model landscape creates significant challenges for financial institutions with regard to managing vendor climate models, assessing climate-specific model validation approaches, and competing for talent in the climate space. Many asset classes are impacted by climate change and, as the landscape evolves, model risk managers must grapple with numerous potential solutions. Moreover, regulatory pressure is rising, suggesting banks need to be comprehensive and robust in ensuring they are up to speed.
Climate change creates physical and transition risks that are complex, uncertain, and playing out in real time. To gauge the potential impacts on clients and portfolios, as well as the effects of mitigation measures, banks require new models, new documentation, and new model risk management (MRM) capabilities. With few precedents in hand, none of this is easy. And given the need for sector-specific methodologies, the industry is facing a significant talent deficit.
While climate change is playing out globally, US banks in particular face acute challenges in managing climate risk, due in part to investor and customer sentiment, as well as evolving regulations and legislation. Just 23 percent of US institutions have placed climate models under MRM oversight, compared with 67 percent of European institutions, according to the latest MRM survey by McKinsey Risk Dynamics. Much work remains to be done.
Some leading institutions are responding with strategies that can inform a successful validation. For many institutions, the task will be to consider these perspectives in the context of both their business priorities and the shifting impacts of climate change on the financial industry and beyond.
Climate model validation challenges
The scale of the challenge facing MRM teams with respect to climate change is reminiscent of the early days of the US Federal Reserve’s Comprehensive Capital Analysis and Review (CCAR), created in the wake of the financial crisis. Specifically, given the new class of models, MRM teams will need to work with model owners to test and verify the completeness of the model landscape. They will also need to put in place robust management of vendor climate models, design a tailored validation approach to address a lack of historical data and, not least of all, create strategies to compete effectively for climate talent. Here we discuss some of the key challenges that banks face with climate model validation.
1. Managing the risk of incomplete climate models that may miss key risk drivers and impacts
Climate models are a relatively recent addition to most banks’ model inventories. Typical types of climate models that banks and their clients will face are physical risk models that capture the impact of extreme heatwaves, flooding, or storms; transition risk models that capture the impact of climate-related policy changes, for example, carbon prices; impact models that translate physical and transition risk impact into credit risk, asset valuations, and more; and emission models to determine emission baselines and targets.
To gauge and assess these risks, banks will require either new models or significant repurposing of current models for new use cases. These models should be accompanied by fresh risk scenarios and mechanisms to translate model outputs into credit decisions. Banks will also need repurposed metrics, such as borrower climate risk scores and financed emissions analyses. Across the function, MRM teams will need to take on an array of new responsibilities.
The channels of climate impacts are uncertain, especially when it comes to second- or third-order effects, which are the main impact drivers. For example, while increased direct damage from physical hazards such as storms, flooding, or wild fires are the first-order impact for real estate, second-order impacts such as changes in investor perception or transition risk in particular geographies may be equally significant, or more so. Along with individual models, therefore, MRM teams need to understand interdependencies between models and the completeness of the model landscape to capture all material risks for the bank (see sidebar “Banks are forging a new climate model landscape focused on three use cases”).
2. Validating vendor climate models
Current climate analytics make use of a wide range of vendor models. These include climate systems models; models for individual climate hazards, such as flooding or wildfires; and impact translation models for both physical and transition risks. Many vendors in the space may be relatively new to banking analytics and have limited familiarity with applicable model risk management guidance such as SR 11-7.
One common challenge is misalignment with the bank’s definition of a model. In climate risk, this is particularly important given the use of qualitative/expert-based approaches (for example, scenario expansion or impact assessment mechanisms) that may not always be described as “models” by the vendor.
Also, there is often a lack of transparency on methodological choices and model specifications. Several climate model vendors are new to MRM requirements and need to develop model documentation that brings sufficient transparency for MRM teams to conduct an independent review and challenge. Specifically, initial vendor model documentation usually lacks the required level of information on the following:
- Model assumptions. These include identification of all key assumptions and rationale or evidence to support their soundness. Examples of model assumptions include (but are not limited to) assumptions to establish the relationship between climate change and productivity (for example, productivity curves providing the relationship between temperature and labor or agricultural productivity); assumptions to enable the estimation of physical risk hazard damages (for example, vulnerability curves estimating losses based on wind speeds, the expected evolution of insurance markups on premiums for real estate, and the geographic distribution of company assets); and the capital intensity requirements over time across industries and geographies.
- Model limitations. Limitations are usually not clearly identified or detailed due to limitations on data availability and expert-based approaches. These may include circumstances under which proxy, instead of actual data, is used (for example, the model falls back on sector averages as opposed to company-specific data), or cases when the model uses average physical impacts due to an inability to estimate and/or process tail risks; or cases where there are “known unknowns” (for example, the model does not capture demand growth for climate technologies that have not reached scale or achieved widespread deployment).
- Model testing. Models typically miss clear specification of test objectives and methods, as well as visibility on results and related analyses due to the complexity of model drivers and the limitations of historical data. For example, models may not include detailed impact analyses across sectors and climate risk components for a representative portfolio of companies, sensitivity analyses to understand how the model reacts across sectors to a wide range of scenarios and to shocks applied to key assumptions, and benchmarking to assess how model outcomes compare with other solutions in the market.
Finally, there can often be a lack of controls around model use, for example, in relation to model change management, technical implementation, contingency plans, and monitoring activities. Climate model vendors will need to work closely with banks to align on control requirements and implementation, including focusing on monitoring frequency and testing.
Designing specific model validation approaches for climate models
The immaturity of approaches, unprecedented risks, and long impact horizons create new challenges for MRM functions to validate the conceptual and analytical soundness of climate models. Due to the evolving nature of the practice, MRM teams will need to appropriately assess the risks of imperfect solutions and manage the associated model risks. Specifically, we see the following common challenges.
A variety of methodologies are unstandardized and lacking in maturity. MRM teams should concentrate their efforts on the most material model components for the use cases validated. For example, physical risk models are extremely complex but are typically sourced from well-established academic sources (and extensively peer reviewed), while transition risk models are much more recent, rather simple, and usually vendor-specific. Given that transition risks can have a higher impact than physical risks, the MRM team should design its validation approach accordingly to focus more on the conceptual soundness of transition risk models.
New data sources will be required for various climate model components, including geographic information with high granularity, detailed sectoral information, and full characterization of carbon footprints through value chains. A key MRM task will be to evaluate the appropriateness of data used in regard to the industries and/or geographies present in the bank’s portfolio, for example, company-specific data on Scope 1, 2, or 3 emissions, real-estate market data on Scope 1 and 2 emissions, abatement cost curves by geography and industry, or the location of company assets by geography.
Climate models will need to forecast impacts over longer time horizons, as much as 30 years for many scenarios, reflecting the potential impact of climate on today’s valuations of assets. While longer time horizons are required for long-lived assets (for example, housing, infrastructure, commercial real estate), they create challenges in assessing and validating model outputs. Instead of focusing validation on the accuracy of long-term credit or climate impacts, the MRM assessment should focus on how markets (for example, housing) will treat valuations in the near term based on market expectations of the medium and long term. Finally, depending on the use case of these models (for example, broader pricing), they should be able to reflect “Minsky moment” types of risk in a particular sector or geography.
Many aspects of climate models are expert driven, due to the absence of, or limitations in, data. Models may be significantly sensitive to expert assumptions, creating challenges for MRM. For example, MRM functions may need to opine on whether scenario expansions or impact assessment approaches (for example, in relation to the evolution of battery technology for electronic vehicles, commodity supply and demand, goods and services, cost/price elasticities, or eventual impacts on markets dynamics) are in line with the bank’s opinion and risk appetite.
MRM teams have typically relied on back-testing to assess model soundness; however, new approaches to outcome analysis are needed, as data unavailability may limit the use of this key tool. As a consequence, validation teams must leverage different approaches to test output quality, including benchmarks and challenger models, which may need to be sourced from vendors.
Climate is a new risk area where industry standards are still evolving and are not yet robust, for example, emission factors and channels of impact for scenarios. MRM teams need to apply critical thinking when using industry standards in model validation, and be iterative on expectations as the industry research evolves over time.
Tackling competition for talent
A key competitive differentiator in gearing up for climate risk impact will be talent. Indeed, banks that lead on climate capability building are likely to be the most successful in attracting people with deep climate expertise. For many institutions, it may make sense to take a pragmatic approach to capability building, taking into account the skill sets required and the pace of team building. Specifically, banks will need to consider whether they should develop capabilities through one of two options (or a combination of both).
The first option is to develop capabilities within existing MRM teams through dedicated training (for example, certifications or partnerships with vendors and academic institutions). The other option is to acquire external talent with climate expertise and provide training on model risk management.
Getting started
The task facing banks in establishing risk management frameworks for climate models is significant. Indeed, the cost of a suboptimal approach is likely to be high. Of the many tasks facing decision makers, we see five in particular that will lay the foundations for successful climate model risk management.
1. Model inventory completeness. Analytics leaders should develop a bankwide climate analytics strategy as well as an inventory of models to support climate decision making, including providing support for data acquisition, consolidating use cases (to limit the number of models being developed), and engaging early with model developers. This would require an update of relevant roles and responsibilities, as well as procedures for model inventory management. As a matter of urgency, banks should initiate a process to populate model inventories, including putting in place tiering and prioritization for model validation.
2. Climate-specific model validation standards. MRM teams should define new validation standards for climate models or, alternatively, enhance existing standards. They should work with model development teams to ensure consistency in development and validation protocols, ensuring that effective governance is integral to model development. A key part of the process should be to integrate MRM requirements into the vendor acquisition process. This means requiring vendors to provide sufficient transparency on their solutions. Finally, teams should translate standards into adapted test plans/code libraries. Given back-testing limitations, they may wish to focus on alternative outcome analyses such as sensitivity testing and benchmarking (see sidebar “Benchmarking and sensitivity analysis are best suited for assessing the assumptions and performance of a climate model”). The development of benchmark/challenger models is typically a time-consuming activity. MRM functions will need to work with model owners to strategize their benchmarking approaches by looking at a range of options, including partnerships with vendors that can rapidly deploy solutions.
3. Agile model validation approach. Amid a lack of standardized methodologies, climate modeling approaches are evolving fast. An agile model validation approach is an effective way to keep up as approaches evolve. Specifically, banks can design toll gates across specific stages of the model development process (for example, data, conceptual design, and outcome testing) in order for model validation to build an understanding of the models and share any potential red flags at an early stage.
4. Capacity management. Teams should create a book of work for climate model validation in line with deadlines for regulatory reporting and internal dependencies. This may mean, for example, focusing on credit models when scenarios are provided by regulators, starting with internal scenarios, before tackling risk models in other situations. Banks must have sufficient capacity across the three lines of defense (risk taking, risk oversight, and risk assurance) to ensure that climate model development as well as model risk management are in place.
5. Capabilities identification. Analytics leads should make it a priority to identify the skills and expertise required to effectively challenge all aspects of climate models. Both business knowledge and technical skills will be critical. A good place to start is with an assessment of skills gaps. Where necessary, banks should launch targeted efforts to close such gaps, including adding training, cross-staffing, and vendor collaboration ahead of model validation kick-off.
Through these combined efforts and the commitment of leadership, banks can start to meet the demanding requirements of effective climate model MRM. However, given the challenges of aligning with a pathway to net zero, and the certainty that risks will manifest in unpredictable ways, the construction of an effective capability should be seen as a process, rather than a task—and characterized by a flexible, risk based approach and an agile mindset.
