FAQ: Market Transformation Initiative Evaluation Framework

1. What is the purpose of the MTI Evaluation Framework?

The Market Transformation Initiative (MTI) Evaluation Framework summarizes the policies, principles, and high-level approaches CalMTA will use to assess its portfolio of MTIs. The approaches described in the framework are consistent with CPUC Decision D.19-12-021 and reference the substantial existing body of literature on the best practices and lessons learned for market transformation evaluation.

2. How is MT Evaluation different from evaluation for other energy efficiency programs?

Market transformation programs seek to increase and accelerate market adoption by inducing structural market changes that produce sustained market effects. CalMTA will adhere to the well-established approach to evaluating the impact of market transformation programs. First, the approach requires forecasting baseline market adoption (BMA), the counterfactual market adoption likely to occur absent the MTI and related utility-funded interventions. Second, the approach requires tracking total market adoption (TMA), the actual market uptake that occurs over time. This approach also requires rigorous evaluation of the MTI program theory, including a causality assessment. Figure 2 illustrates this market transformation impact: the difference between BMA and TMA. The area between these adoption curves represents MTI-induced market adoption—that is, the increase in market adoption above the baseline that results from the MTI, including other related utility–funded interventions.  

Figure 2. MTI-Induced Market Adoption

Evaluating the incremental impact of MTIs requires forecasting the naturally occurring baseline market adoption (BMA), tracking total market adoption (TMA), and multiplying the difference between the two (market transformation units) by unit energy impacts (UEI). That is, Total MTI Impacts = (TMA units – BMA units) * UEI.   

In addition to determining the Total MTI Impacts, the evaluator will conduct theory-based evaluation (TBE) to validate how and to what extent the MTI caused the observed market changes. TBE is widely accepted as a best practice for market transformation program evaluation and requires each MTI to have a program theory that clearly identifies the specific theorized market outcomes associated with the MTI strategic market interventions, along with their approximate timing.  

A set of clear, evaluable market progress indicators (MPIs) will be established at the time of MTI adoption and will serve as key determinants of MTI performance. In addition to rigorous assessment of MPIs by a third-party evaluator, TBE also requires an assessment of the causal influence of the MTI interventions on observed outcomes. Causality assessment for market transformation programs is generally qualitative but will be designed to provide reliable evaluation and verification of an MTI’s incremental impact. Best practices for causality assessment are based on a “preponderance of evidence” approach and are well-established in the market transformation evaluation literature, with methods typically including document review; in-depth interviews with market actors, decision makers, and stakeholders; and historical tracing. Effective causality assessment depends on an evaluation plan that anticipates the data and documentary sources that will be needed for this critical part of the evaluation. 

3. What method will be used to establish the baseline market adoption forecast?

CalMTA will develop an objective BMA forecast for each MTI in accordance with market transformation evaluation best practices. These best practices may include using published industry forecasts, econometric modeling such as the Bass Diffusion Model, and structured expert judging such as the Delphi technique. Given the significant implications of the BMA forecast, CalMTA will document the specific BMA forecasting sources and methods it uses for each MTI that is recommended for advancement from Phase II to Phase III, as part of the MTI Plan. 

4. What method will be used to attribute impacts to MTIs that target the same technologies as resource programs?

CalMTA will first assess the collective impact of all coordinated, collaborative program efforts (i.e., Total Incremental MTI Impacts, which equals [TMA units – BMA units] * UEI). Once that value is established, CalMTA will apply a rule-based, accounting approach to avoid double-counting resource impacts and MTI impacts: it will subtract all verified resource program savings associated with the MTI technologies or practices from Total Incremental MTI Impacts. The remainder will be referred to as Net Incremental MTI Impacts.

CalMTA adopted this approach because it is a simple way to avoid double-counting without expending resources to parse out the impacts attributable to individual programs that are working interdependently and which, arguably, cannot be separated.

5. What method will be used to attribute impacts to MTIs that target codes or standards that overlap the California Codes and Standards program?

The Decision that established the California Market Transformation Administrator (D.19-12-02) provided that under the adopted market transformation framework, an MTI may receive credit for savings achieved following the adoption of a code or standard as follows:

If the MTI results in the adoption of a code or standard, the savings credit afforded the MTI for the years following code adoption shall be equal to three times the savings claim made in the final year of MTI operations prior to the code or standard adoption. An MTI CE calculation may include projected C&S savings in accordance with the above approach for C&S savings recognition.

However, the Decision also asked the CAEECC Market Transformation Working Group (MTWG) to continue working on attribution issues and noted that “once the MTA is in place, that entity should formalize these coordination approaches and issues”. After considering the recommendations of the CAEECC MTWG, analyzing the advantages and disadvantages of the approach described in the Decision, and considering feedback from CalMTA stakeholders, CalMTA CalMTA and MTAB reached consensus on provisionally moving forward with the Decision’s prescribed approach, pending additional research and analysis that provides insight on how this approach could be improved. CalMTA recognizes the advantages of using this broad rule-based approach, particularly for initial forecasting. However, to better understand the possible unintended implications of using this rule, CalMTA will conduct and share additional research and analysis during Phase II of the MTI lifecycle.

6. How will stakeholders be certain that MTI investments are delivering incremental impact?

MTIs must demonstrably increase and/or accelerate market adoption beyond what would have happened absent the market transformation initiative. CalMTA will adhere to the well-established approaches to evaluating the total incremental impact of MTIs (that is, [TMA units – BMA units] * UEI), as well as rigorous theory-based evaluation to assess MTI causality.

7. How will CalMTA assess MTI equity?

The Decision included several high-level principles for MTIs as part of the adopted Market Transformation Framework. The first principle states that MTIs must drive incremental savings that achieve the state’s energy efficiency, equity, and GHG reduction goals. The Decision also states that MTIs should also meet a set of principles including this equity-specific one: Integrate strategies to maximize equity.¹  These principles necessitate corresponding evaluation.

CalMTA has committed to apply an equity lens in its approach to developing a portfolio of MTIs that will be designed to maximize equity and deliver benefits to environmental and social justice communities, as defined by the CPUC.² To track performance on this front, each MTI Plan will include proposed equity metrics and a plan for assessing them. 

8. When will CalMTA release RFPs for third-party evaluation?

Once MTIs have been approved by the CPUC for full-scale market development (Phase III), CalMTA will use a competitive request for proposal (RFP) process to contract with third-party evaluators to perform ongoing evaluation for each MTI. We expect to issue the first 1-3 of these RFPs in late-2025.

9. How will the CPUC assure appropriate independence of evaluation?

Per D.19-12-021, CalMTA will bid out the planned EM&V activities to third-party evaluators via competitive solicitations. CalMTA and the MTAB had extensive discussion regarding the appropriate roles of CalMTA versus CPUC Energy Division staff in MTI third-party evaluations, which are different than those established for IOU impact evaluations, as delineated in D.05-01-055. Specifically, Decision 19-12-021 recognizes the value of timely, ongoing evaluation to the success of market transformation programs and calls for CalMTA to administer third-party market progress evaluations, which include assessment of outcomes.  

To ensure appropriate independence and oversight, CPUC has approved the approach to management and oversight of third-party evaluation activities summarized in Table 5 and further described below.

Table 5. Management and Oversight of Third-Party Evaluations 

	Members	Roles
Evaluation Management Team	Energy Division Representative  CalMTA Evaluation Lead	Day-to-day management of third-party MTI evaluations  Energy Division Representative has final decision authority on all related matters  Seeks advice of Evaluation Advisory Group, as needed
Evaluation Advisory Group	Evaluation Management Team and three additional members	Review RFPs for third-party MTI evaluators  Review and score third-party MTI evaluator proposals  Review evaluation deliverables  Advise Evaluation Management Team, as requested

A management team comprising the CalMTA evaluation lead and a designated Energy Division representative will be responsible for ongoing day-to-day management of third-party evaluations. CalMTA will form an evaluation advisory group that includes the CalMTA management team and three other independent evaluation experts with relevant evaluation experience.³ Evaluation advisory group members will review RFP solicitations before they are finalized and score proposals according to an agreed upon set of criteria.  

Once MTI third-party evaluators are selected, based on advisory group scoring, the CalMTA evaluation lead will include the Energy Division representative in all communications and meetings with the third-party evaluator including, but not limited to, discussion of evaluation findings, analysis, and conclusions.  

Evaluation advisory group members will review all third-party evaluation deliverables including, but not limited to, work plans, interim findings, and draft and final reports. Third-party evaluators will determine the ultimate contents of final evaluation reports. CalMTA will ensure that all final evaluation reports are publicly available on its website and in the CALMAC database.

 

 

1. California Public Utilities Commission. Decision 19-12-021 December 5, 2019. “Order Instituting Rulemaking Concerning Energy Efficiency Rolling Portfolios, Policies, Programs, Evaluation, and Related Issues.” Date of issuance December 12, 2019. P.102. PDF (ca.gov)
2. For more about CalMTA’s commitment, see An equity lens – CalMTA
3. Once finalized, details on the selection, compensation, and charter for evaluation advisory group members will be posted on org. Evaluation advisory group members will be subject to the same conflict of interest policies documented on page 7 of the Market Transformation Advisory Board member COI requirements.