Evaluating market transformation progress
The California Market Transformation Framework, created via CPUC Decision 19-12-021, calls for setting clear savings goals and other market transformation initiative (MTI) metrics to ensure a high level of accountability, and ongoing evaluation to reduce program performance risk. To ensure this outcome, CalMTA has developed an Evaluation Framework that describes the policies, principles, and high-level approaches that CalMTA will use to assess its portfolio of MTIs. The approaches are consistent with the Decision and reference the substantial existing body of literature on the best practices and lessons learned for market transformation evaluation.
The draft Evaluation Framework was available for public comment Oct. 17-Nov. 2, 2023 on the California Public Utilities Commission (CPUC) public documents site. CalMTA has compiled a comprehensive list of comments received and provided a response to each comment in the Evaluation Framework Feedback Response Memo.
The critical role of evaluation
Evaluation is essential to ensure that market transformation programs are making progress and delivering cost-effective energy savings for California’s ratepayers and incremental system benefits to California’s electrical grid. CalMTA will oversee implementation of rigorous and strategically focused evaluation, measurement, and verification (EM&V) practices, which will allow us to gauge the performance of MTIs, verify incremental impacts, and improve the design and success of future work.
Evaluation informs effective evolution of initiatives and ongoing investment decisions and supports strong management accountability, which in turn can enhance stakeholder trust and collaboration. The graphic below illustrates the importance and purpose of evaluation.
“Real-time” market evaluation provides the continual feedback that MTI program managers and implementers need to understand whether the intended market changes are occurring and, if needed, pivot strategies to improve MTI performance. Tracking of near- and longer-term market progress indicators that are tightly aligned with the MTI’s market transformation theory will reduce MTI performance risk and support timely decisions regarding ongoing investment or termination of an MTI.
Foundational principles
CalMTA’s approach to MTI evaluation is grounded in the following principles:
- Unambiguous MTI progress and impact goals and metrics, established at time of adoption, along with a data collection plan that substantiates MTI evaluability
- Theory-based evaluation (TBE), which relies on clear program theory, logic models, and associated market progress indicators as the foundation upon which to establish the market influence, progress, and causal impact of MTI interventions
- Data-driven, transparent analysis methods to estimate market diffusion, cost-effectiveness, and incremental impact
- Use of widely accepted best practices to develop and refine baseline market adoption forecast
- Agreed-upon methodology to determine incremental impact of each MTI that supports California’s policy goals, optimal statewide collaboration, and decisions about future market transformation investments
Measuring incremental impact
MTIs seek to accelerate market adoption by driving market changes that increase and accelerate market adoption and deliver sustained market effects. To start, the approach requires forecasting baseline market adoption (BMA)—the market adoption likely to occur absent an MTI and related interventions. In addition, the approach requires tracking total market adoption (TMA)—the actual market uptake that occurs over time. Rigorous evaluation of the MTI program theory, including a causality assessment, is also required.
For successful measurement of incremental impacts, effective coordination and collaboration is needed between CalMTA and other energy efficiency program administrators and implementers, who are running programs in markets targeted by MTIs. CalMTA seeks to achieve synergistic results that maximize MTI effectiveness and improve collective outcomes for efficiency programs.
With collaboration, however, comes questions about how to fairly determine the impact that will be attributed to each program that is investing in the market. The general approach to MTI attribution is described in the Framework. The specific methods proposed to determine incremental impact for each MTI will be included in the MTI Plan developed during Phase II of the MTI lifecycle.
Watch the video
View our October 18, 2023 presentation to the Market Transformation Advisory Board (MTAB) on the Evaluation Framework:
You can also see the January 8, 2024 Evaluation Framework webinar, which provides additional clarification about the proposed evaluation approach, oversight, and management.
Learn more in our Evaluation Framework FAQ.
The path to finalizing the Evaluation Framework
The graphic below shows the process from initial draft through approval of the final Evaluation Framework, including a period of stakeholder outreach through October 2023, a two-week public comment period October 19–November 2, and final discussion at the November-December MTAB meeting.
