Blog: Understanding CalMTA’s Evaluation Framework for Market Transformation Initiatives

Careful evaluation is essential to the success of market transformation (MT) efforts. Accordingly, CalMTA is implementing strategically focused evaluation, measurement, and verification (EM&V) practices to ensure cost-effective energy savings for California’s ratepayers and system-wide benefits to the electrical grid. These evaluations inform adaptive management of market transformation initiatives (MTIs) and provide insight for ongoing investment decisions. Moreover, they support strong management accountability with visibility for stakeholders to understand how MTI implementation is progressing.  

Because MT is new in California, the CalMTA team developed an MTI Evaluation Framework that outlines the principles and processes we will use to evaluate MTIs when they are launched. These include:  

• Unambiguous MTI progress and impact goals and metrics, established at time of plan adoption 
• Theory-based evaluation (TBE), which relies on clear program theory, logic models, and associated market progress indicators to assess the market influence, progress, and causal impact of MTI interventions 
• Data-driven, transparent analysis methods to estimate market diffusion, cost-effectiveness, and incremental impact 
• Use of widely accepted best practices to develop and refine baseline market adoption forecast 
• Agreed-upon methodology to determine incremental impact of each MTI that supports California’s policy goals, optimal statewide collaboration, and decisions about future market transformation investments. 

Stakeholder review 

A draft of the framework was posted to the California Public Utilities Commission’s (CPUC) PDA website for comment last October. Those comments and ongoing discussions with our Market Transformation Advisory Board (MTAB) and the CPUC were then used to develop the final framework. 

Three notable revisions were made to the draft framework to directly address public comments: 

• Added content to address CalMTA’s commitment to equity and approach to equity accountability 
• Clarified approach to evaluating CalMTA incremental impact 
• Added a section on management and oversight of third-party evaluations. 

MT evaluation approach 

To start, the MT evaluation approach requires forecasting baseline market adoption (BMA)—the market adoption likely to occur absent an MTI and related interventions. In addition, this approach requires tracking total market adoption (TMA)—the actual market uptake that occurs over time. Assessing whether market interventions have in fact been the cause of observed outcomes is also required. 

Figure 1. Market Transformation accelerated adoption curve 

Elements of MTI evaluation

Several key elements are involved in evaluation of MTIs.

Theory-based Evaluation

CalMTA employs theory-based evaluation (TBE) right from the start. TBE requires each MTI to have a program theory that clearly identifies the specific intended market outcomes associated with the MTI strategic market interventions, along with their timing. Initial MTI theory development also identified strategies specifically designed to maximize opportunities to ensure equitable outcomes.

Market Progress Indicators

A set of clear, evaluable Market Progress Indicators (MPIs) will be established as one of the primary determinants of initiative performance. While judgments of market progress are often made on units of adoption, this metric can be misleading because market share and adoption typically increase slowly and accelerate only after addressing critical market barriers, such as product availability, quality and standardization, workforce capacity, and market perceptions. Therefore, evaluation includes short- and medium-term MPIs that align with a strategically designed logic model.

Causality Assessments

TBE also requires an assessment of the causal influence of the MTI interventions on observed outcomes. Causality assessment for market transformation programs is generally qualitative but can nonetheless be designed to provide reliable evaluation and verification of an MTI’s incremental impact. Best practices for causality assessment are based on a “preponderance of evidence” approach and are well-established in the market transformation evaluation literature, with methods typically including document review, in-depth interviews with market actors, decision makers and stakeholders, and historical tracing.

Equity Metrics

Proposed equity metrics are a key element of Phase II activities as well. Each of these components is then used to develop an Evaluation Plan that includes proposed market research, data collection, and analysis activities.

MTI Plan

In the development of an MTI Plan, which happens in Phase II and fully describes how an MTI will be implemented and evaluated, more robust forecasts of MTI incremental impacts and cost-effectiveness are developed.

As described in the graphic above, this involves carefully documented, data-driven BMA and TMA forecasting. CalMTA updates model assumptions when updated market data become available, and third-party impact evaluation activities in Phase III may include data collection activities and model review to inform and improve these estimates.

Third-party evaluators & ongoing evaluation

Once MTIs have been adopted into the CalMTA portfolio for full-scale market deployment (Phase III), CalMTA uses a competitive request for proposal (RFP) process to contract with third-party evaluators to perform ongoing evaluation for each MTI. EM&V activities focus on market progress evaluation per agreed-upon MPIs, assessing MTI causality, identifying adaptive management opportunities, and refining estimates of total and net (incremental) market impacts.

A management team comprising the CalMTA evaluation lead and a designated CPUC representative is then responsible for ongoing day-to-day management of third-party evaluations. An evaluation advisory group that includes the CalMTA management team and three other independent evaluation experts with relevant evaluation experience will review all third-party evaluation deliverables including, but not limited to, work plans, interim findings, and draft and final reports.

Ultimately, the evaluation processes, activities, and methods described in the Evaluation Framework enable well-informed, data-driven investment decisions regarding individual MTIs as well as continued investment in market transformation. These methods are consistent with well-established approaches for market transformation evaluation, but they differ from California’s established approaches for evaluating resource acquisition programs.

Please check out our answers to frequently asked questions about the framework. For more information on how the evaluation works at each stage, please read the MTI Evaluation Framework.