FAQ: Market Transformation Evaluation Framework

What is the purpose of the MT Evaluation Framework?

The Market Transformation (MT) Evaluation Framework summarizes the policies, principles, and high-level approaches CalMTA will use to assess its portfolio of MTIs. The approaches described in the framework are consistent with CPUC Decision D.19-12-021 and reference the substantial existing body of literature on the best practices and lessons learned for market transformation evaluation. CalMTA has introduced a draft Evaluation Framework detailed its proposed approach.

How is MT Evaluation different from evaluation for other energy efficiency programs?

Market transformation programs seek to increase and accelerate market adoption by inducing structural market changes that produce sustained market effects. CalMTA will adhere to the well-established approach to evaluating the impact of market transformation programs. First, the approach requires forecasting baseline market adoption (BMA), the counterfactual market adoption likely to occur absent the MTI and related utility-funded interventions. Second, the approach requires tracking total market adoption (TMA), the actual market uptake that occurs over time. This approach also requires rigorous evaluation of the MTI program theory, including a causality assessment. Figure 2 illustrates this market transformation impact: the difference between BMA and TMA. The area between these adoption curves represents MTI-induced market adoption—that is, the increase in market adoption above the baseline that results from the MTI, including other related utility-funded interventions.

Figure 2. MTI-Induced Market Adoption

Evaluating the incremental impact of MTIs requires forecasting the naturally occurring baseline market adoption (BMA), tracking total market adoption (TMA), and multiplying the difference between the two (market transformation units) by unit energy impacts (UEI). That is, Total MTI Impacts = (TMA units – BMA units) * UEI.

In addition to determining the Total MTI Impacts, the evaluator will conduct theory-based evaluation (TBE) to validate how and to what extent the MTI caused the observed market changes. TBE is widely accepted as a best practice for market transformation program evaluation and requires each MTI to have a program theory that clearly identifies the specific theorized market outcomes associated with the MTI strategic market interventions, along with their approximate timing.

A set of clear, evaluable market progress indicators (MPIs) will be established at the time of MTI adoption and will serve as key determinants of MTI performance. In addition to rigorous assessment of MPIs by a third-party evaluator, TBE also requires an assessment of the causal influence of the MTI interventions on observed outcomes. Causality assessment for market transformation programs is generally qualitative but will be designed to provide reliable evaluation and verification of an MTI’s incremental impact. Best practices for causality assessment are based on a “preponderance of evidence” approach and are well-established in the market transformation evaluation literature, with methods typically including document review; in-depth interviews with market actors, decision makers, and stakeholders; and historical tracing. Effective causality assessment depends on an evaluation plan that anticipates the data and documentary sources that will be needed for this critical part of the evaluation.

What method will be used to establish the baseline market adoption forecast?

CalMTA will develop an objective BMA forecast for each MTI in accordance with market transformation evaluation best practices. These best practices may include using published industry forecasts, econometric modeling such as the Bass Diffusion Model, and structured expert judging such as the Delphi technique. Given the significant implications of the BMA forecast, CalMTA will document the specific BMA forecasting sources and methods it uses for each MTI that is recommended for advancement from Phase II to Phase III, as part of the MTI Plan.

What method will be used to attribute impacts to MTIs that target the same technologies as resource programs?

CalMTA will first assess the collective impact of all coordinated, collaborative program efforts (i.e., Total MTI Impacts, which equals [TMA units – BMA units] * UEI). Once that value is established, CalMTA will apply a rule-based, accounting approach to avoid double-counting resource impacts and MTI impacts: it will subtract all verified resource program savings associated with the MTI technologies or practices from Total MTI Impacts. The remainder will be referred to as Net MTI Impacts.

CalMTA is proposing this approach because it is a simple way to avoid double-counting without expending resources to parse out the impacts attributable to individual programs that are working interdependently and which, arguably, cannot be separated.

What method will be used to attribute impacts to MTIs that target codes or standards that overlap the California Codes and Standards program?

The Decision that established the California Market Transformation Administrator (D.19-12-02) provided that under the adopted market transformation framework, an MTI may receive credit for savings achieved following the adoption of a code or standard as follows:

If the MTI results in the adoption of a code or standard, the savings credit afforded the MTI for the years following code adoption shall be equal to three times the savings claim made in the final year of MTI operations prior to the code or standard adoption. An MTI CE calculation may include projected C&S savings in accordance with the above approach for C&S savings recognition.

However, the Decision also asked the CAEECC Market Transformation Working Group (MTWG) to continue working on attribution issues and noted that “once the MTA is in place, that entity should formalize these coordination approaches and issues”. After considering the recommendations of the CAEECC MTWG and analyzing the advantages and disadvantages of different approaches, CalMTA concurred with the CAECC MTWG’s consensus recommendation – that CalMTA collaborate with the Codes and Standards program administrator to forecast MTI savings separate from those achieved by the Codes and Standards program administrator, which would result in de facto relative forecasted shares, to be updated as necessary. A detailed discussion of the approaches considered and rationale for the recommendation are included in the Framework.

How will stakeholders be certain that MTI investments are delivering incremental impact?

MTIs must demonstrably increase and/or accelerate market adoption beyond what would have happened absent the market transformation initiative. CalMTA will adhere to the well-established approaches to evaluating the total incremental impact of MTIs (that is, [TMA units – BMA units] * UEI), as well as rigorous theory-based evaluation to assess MTI causality.

When will CalMTA release RFPs for third-party evaluation?

Once MTIs have been approved by the CPUC for full-scale market development (Phase III), CalMTA will use a competitive request for proposal (RFP) process to contract with third-party evaluators to perform ongoing evaluation for each MTI. We expect to issue the first 1-3 of these RFPs in late-2025.