The Market Transformation Initiative (MTI) Evaluation Framework summarizes the policies, principles, and high-level approaches CalMTA will use to assess its portfolio of MTIs. The approaches described in the framework are consistent with California Public Utilities Commission (CPUC) Decision (D.)19-12-021 and reference the substantial existing body of literature on the best practices and lessons learned for market transformation evaluation.
Market transformation programs seek to increase and accelerate market adoption by inducing structural market changes that produce sustained market effects. CalMTA will adhere to the well-established approach to evaluating the impact of market transformation programs. First, the approach requires forecasting baseline market adoption (BMA), the counterfactual market adoption likely to occur absent the MTI and related utility-funded interventions. Second, the approach requires tracking total market adoption (TMA), the actual market uptake that occurs over time. This approach also requires rigorous evaluation of the MTI program theory, including a causality assessment.
Figure 1 illustrates this market transformation impact: the difference between BMA and TMA. The area between these adoption curves represents MTI-induced market adoption — that is, the increase in market adoption above the baseline that results from the MTI, including other related utility–funded interventions.
Evaluating the incremental impact of MTIs requires forecasting the naturally occurring BMA, tracking TMA, and multiplying the difference between the two (market transformation units) by unit energy impacts (UEI). That is, Total MTI Impacts = (TMA units – BMA units) * UEI.
In addition to determining the Total MTI Impacts, the evaluator will conduct theory-based evaluation (TBE) to validate how and to what extent the MTI caused the observed market changes. TBE is widely accepted as a best practice for market transformation program evaluation and requires each MTI to have a program theory that clearly identifies the specific theorized market outcomes associated with the MTI strategic market interventions, along with their approximate timing.
A set of clear, evaluable market progress indicators (MPIs) will be established at the time of MTI adoption and will serve as key determinants of MTI performance. In addition to rigorous assessment of MPIs by a third-party evaluator, TBE also requires an assessment of the causal influence of the MTI interventions on observed outcomes. Causality assessment for market transformation programs is generally qualitative but will be designed to provide reliable evaluation and verification of an MTI’s incremental impact. Best practices for causality assessment are based on a “preponderance of evidence” approach and are well-established in the market transformation evaluation literature, with methods typically including document review; in-depth interviews with market actors, decision makers, and stakeholders; and historical tracing. Effective causality assessment depends on an evaluation plan that anticipates the data and documentary sources that will be needed for this critical part of the evaluation.
CalMTA will develop an objective BMA forecast for each MTI in accordance with market transformation evaluation best practices. These best practices may include using published industry forecasts, econometric modeling such as the Bass Diffusion Model, and structured expert judging such as the Delphi technique. Given the significant implications of the BMA forecast, CalMTA will document the specific BMA forecasting sources and methods it uses for each MTI that is recommended for advancement from Phase II to Phase III, as part of the MTI Plan.
CalMTA will first assess the collective impact of all coordinated, collaborative program efforts (i.e., Total Incremental MTI Impacts, which equals [TMA units – BMA units] * UEI). Once that value is established, CalMTA will apply a rule-based, accounting approach to avoid double-counting resource impacts and MTI impacts: it will subtract all verified resource program savings associated with the MTI technologies or practices from Total Incremental MTI Impacts. The remainder will be referred to as Net Incremental MTI Impacts.
CalMTA adopted this approach because it is a simple way to avoid double-counting without expending resources to parse out the impacts attributable to individual programs that are working interdependently and which, arguably, cannot be separated.
The Decision that established the California Market Transformation Administrator (D.19-12-02) provided that under the adopted market transformation framework, an MTI may receive credit for savings achieved following the adoption of a code or standard as follows:
If the MTI results in the adoption of a code or standard, the savings credit afforded the MTI for the years following code adoption shall be equal to three times the savings claim made in the final year of MTI operations prior to the code or standard adoption. An MTI CE calculation may include projected C&S savings in accordance with the above approach for C&S savings recognition.
However, the Decision also asked the California Energy Efficiency Coordinating Committee (CAEECC) Market Transformation Working Group (MTWG) to continue working on attribution issues and noted that “once the MTA is in place, that entity should formalize these coordination approaches and issues.” After considering the recommendations of the CAEECC MTWG, analyzing the advantages and disadvantages of the approach described in the Decision, and considering feedback from CalMTA stakeholders, CalMTA and MTAB reached consensus on provisionally moving forward with the Decision’s prescribed approach, pending additional research and analysis that provides insight on how this approach could be improved. CalMTA recognizes the advantages of using this broad rule-based approach, particularly for initial forecasting. However, to better understand the possible unintended implications of using this rule, CalMTA will conduct and share additional research and analysis during Phase II of the MTI life cycle.
MTIs must demonstrably increase and/or accelerate market adoption beyond what would have happened absent the market transformation initiative. CalMTA will adhere to the well-established approaches to evaluating the total incremental impact of MTIs (that is, [TMA units – BMA units] * UEI), as well as rigorous theory-based evaluation to assess MTI causality.
CalMTA uses key performance indicators (KPIs) to measure its progress at the program-level and by market transformation initiative (MTI). The KPI scorecard is updated quarterly and tracks incremental benefits of MTIs including TSB, energy benefits, grid benefits, GHG reduction, co-created incremental TSB, and lifetime net incremental TSB.
The Decision included several high-level principles for MTIs as part of the adopted Market Transformation Framework. The first principle states that MTIs must drive incremental savings that achieve the State’s energy efficiency, equity, and GHG reduction goals. The Decision also states that MTIs should also meet a set of principles including this equity-specific one: Integrate strategies to maximize equity.[1] These principles necessitate corresponding evaluation.
CalMTA has committed to apply an equity lens in its approach to developing a portfolio of MTIs that will be designed to maximize equity and deliver benefits to environmental and social justice communities, as defined by the CPUC.[2] To track performance on this front, each MTI Plan will include proposed equity metrics and a plan for assessing them.
Once MTIs have been approved by the CPUC for full-scale market development (Phase III), CalMTA will use a competitive request for proposal (RFP) process to contract with third-party evaluators to perform ongoing evaluation for each MTI. We expect to issue the first one to three of these RFPs in late 2025.
CalMTA will bid out the planned EM&V activities to third-party evaluators via competitive solicitations. Prior to adopting its Market Transformation Initiative (MTI) Evaluation Framework, CalMTA and the Market Transformation Advisory Board (MTAB) had extensive discussion regarding the appropriate roles of CalMTA versus California Public Utilities Commission (CPUC) Energy Division staff in MTI third-party evaluations, which are different than those established for investor-owned utility (IOU) impact evaluations. Specifically, CPUC Decision (D.) 19-12-021 recognizes the value of timely, ongoing evaluation to the success of market transformation programs and calls for CalMTA to administer third-party market progress evaluations, which include assessment of outcomes.
To ensure appropriate independence and oversight, the CPUC has approved the approach to management and oversight of third-party evaluation activities summarized in the table and further described below.
| Members | Roles | |
| Evaluation Management Team | Energy Division Representative CalMTA Evaluation Lead |
|
| Evaluation Advisory Group | Evaluation Management Team and three additional members |
|
A management team comprising the CalMTA evaluation lead and a designated Energy Division representative will be responsible for ongoing day-to-day management of third-party evaluations. CalMTA will form an evaluation advisory group that includes the CalMTA management team and three other independent evaluation experts with relevant evaluation experience.3 Evaluation advisory group members will review RFP solicitations before they are finalized and score proposals according to an agreed upon set of criteria.
Once MTI third-party evaluators are selected, based on advisory group scoring, the CalMTA evaluation lead will include the Energy Division representative in all communications and meetings with the third-party evaluator including, but not limited to, discussion of evaluation findings, analysis, and conclusions.
Evaluation advisory group members will review all third-party evaluation deliverables including, but not limited to, work plans, interim findings, and draft and final reports. Third-party evaluators will determine the ultimate contents of final evaluation reports. CalMTA will ensure that all final evaluation reports are publicly available on its website and in the CALMAC database.
The California Market Transformation Administrator (CalMTA) develops and manages market transformation initiatives in the state to reduce energy use and reduce greenhouse gas (GHG) emissions.
CalMTA follows a rigorous process for reviewing, scoring, and then developing relevant, timely market transformation initiatives. The process supports market transformation initiative creation from concept to program development to market deployment, as well as the eventual exiting of the market.
We welcome your questions and suggestions.
Have questions or comments about CalMTA?
Use our contact form to connect with us, or reach out to:
CalMTA
Resource Innovations
719 Main Street, Suite A
Half Moon Bay, CA, 94019
(888) 217-0217
All Advisory Board meetings are open for public comment. If you’re unable to share your thoughts during an MTAB meeting, access our comment form to to provide your opinion.
Want to learn more? Sign up for our mailing list to receive twice-monthly updates about upcoming events, including Advisory Board meetings, informational webinars, important RFI deadlines, and more.