Blog: Understanding CalMTA’s Evaluation Framework for Market Transformation Initiatives

Blog

Careful evaluation is essential to the success of market transformation (MT) efforts. Accordingly, CalMTA is implementing strategically focused evaluation, measurement, and verification (EM&V) practices to ensure cost-effective energy savings for California’s ratepayers and system-wide benefits to the electrical grid. These evaluations inform adaptive management of market transformation initiatives (MTIs) and provide insight for ongoing investment decisions. Moreover, they support strong management accountability with visibility for stakeholders to understand how MTI implementation is progressing.  

Because MT is new in California, the CalMTA team developed an MTI Evaluation Framework that outlines the principles and processes we will use to evaluate MTIs when they are launched. These include:  

  • Unambiguous MTI progress and impact goals and metrics, established at time of plan adoption 
  • Theory-based evaluation (TBE), which relies on clear program theory, logic models, and associated market progress indicators to assess the market influence, progress, and causal impact of MTI interventions 
  • Data-driven, transparent analysis methods to estimate market diffusion, cost-effectiveness, and incremental impact 
  • Use of widely accepted best practices to develop and refine baseline market adoption forecast 
  • Agreed-upon methodology to determine incremental impact of each MTI that supports California’s policy goals, optimal statewide collaboration, and decisions about future market transformation investments. 

Stakeholder review 

A draft of the framework was posted to the California Public Utilities Commission’s (CPUC) PDA website for comment last October. Those comments and ongoing discussions with our Market Transformation Advisory Board (MTAB) and the CPUC were then used to develop the final framework. 

Three notable revisions were made to the draft framework to directly address public comments: 

  • Added content to address CalMTA’s commitment to equity and approach to equity accountability 
  • Clarified approach to evaluating CalMTA incremental impact 
  • Added a section on management and oversight of third-party evaluations. 

MT evaluation approach 

To start, the MT evaluation approach requires forecasting baseline market adoption (BMA)—the market adoption likely to occur absent an MTI and related interventions. In addition, this approach requires tracking total market adoption (TMA)—the actual market uptake that occurs over time. Assessing whether market interventions have in fact been the cause of observed outcomes is also required. 

The impact and savings opportunity of market transformation, shown as the classic s- or duck curve

Figure 1. Market Transformation accelerated adoption curve 

Elements of MTI evaluation

Several key elements are involved in evaluation of MTIs.

Theory-based Evaluation

CalMTA employs theory-based evaluation (TBE) right from the start. TBE requires each MTI to have a program theory that clearly identifies the specific intended market outcomes associated with the MTI strategic market interventions, along with their timing. Initial MTI theory development also identified strategies specifically designed to maximize opportunities to ensure equitable outcomes.

Market Progress Indicators

A set of clear, evaluable Market Progress Indicators (MPIs) will be established as one of the primary determinants of initiative performance. While judgments of market progress are often made on units of adoption, this metric can be misleading because market share and adoption typically increase slowly and accelerate only after addressing critical market barriers, such as product availability, quality and standardization, workforce capacity, and market perceptions. Therefore, evaluation includes short- and medium-term MPIs that align with a strategically designed logic model.

Causality Assessments

TBE also requires an assessment of the causal influence of the MTI interventions on observed outcomes. Causality assessment for market transformation programs is generally qualitative but can nonetheless be designed to provide reliable evaluation and verification of an MTI’s incremental impact. Best practices for causality assessment are based on a “preponderance of evidence” approach and are well-established in the market transformation evaluation literature, with methods typically including document review, in-depth interviews with market actors, decision makers and stakeholders, and historical tracing.

Equity Metrics

Proposed equity metrics are a key element of Phase II activities as well. Each of these components is then used to develop an Evaluation Plan that includes proposed market research, data collection, and analysis activities.

MTI Plan

In the development of an MTI Plan, which happens in Phase II and fully describes how an MTI will be implemented and evaluated, more robust forecasts of MTI incremental impacts and cost-effectiveness are developed.

Stage Gate Decisions and Related Evaluation Activities, by MTI Phase

As described in the graphic above, this involves carefully documented, data-driven BMA and TMA forecasting. CalMTA updates model assumptions when updated market data become available, and third-party impact evaluation activities in Phase III may include data collection activities and model review to inform and improve these estimates.

Third-party evaluators & ongoing evaluation

Once MTIs have been adopted into the CalMTA portfolio for full-scale market deployment (Phase III), CalMTA uses a competitive request for proposal (RFP) process to contract with third-party evaluators to perform ongoing evaluation for each MTI. EM&V activities focus on market progress evaluation per agreed-upon MPIs, assessing MTI causality, identifying adaptive management opportunities, and refining estimates of total and net (incremental) market impacts.

A management team comprising the CalMTA evaluation lead and a designated CPUC representative is then responsible for ongoing day-to-day management of third-party evaluations. An evaluation advisory group that includes the CalMTA management team and three other independent evaluation experts with relevant evaluation experience will review all third-party evaluation deliverables including, but not limited to, work plans, interim findings, and draft and final reports.

Ultimately, the evaluation processes, activities, and methods described in the Evaluation Framework enable well-informed, data-driven investment decisions regarding individual MTIs as well as continued investment in market transformation. These methods are consistent with well-established approaches for market transformation evaluation, but they differ from California’s established approaches for evaluating resource acquisition programs.

Please check out our answers to frequently asked questions about the framework. For more information on how the evaluation works at each stage, please read the MTI Evaluation Framework.

Blog: Summary Report of CalMTA Listening Sessions now available

Blog

Listening Sessions with ESJ Communities report coverCalifornia has a strong commitment to ensuring that communities facing a disproportionate energy burden and historical inequities of efficiency investment enjoy the benefits of energy efficiency. In support of these statewide goals, CalMTA is working to apply an “equity lens” in development of the state’s first portfolio of energy efficiency market transformation initiatives (MTIs).  To engage diverse communities in the early stages of market transformation (MT) idea development, CalMTA conducted a series of listening sessions with equity stakeholders and workforce education and training (WE&T) entities focused on energy equity. These five 90-minute listening sessions were held last fall and sought to understand past experiences and attitudes of environmental and social justice (ESJ) communities in relation to energy efficiency program engagement.

Stakeholders may now access a report, Listening Sessions with ESJ Communities: Key Findings and Market Transformation Recommendations, summarizing methodology and approach, session findings, and recommendations.

One of the things that we fear is going on right now is gentrification of environmental issues in this area, meaning that affluent communities are getting all the benefits of energy efficiency and poor communities must be on the waiting list.

Session findings

Key takeaways from the sessions, which focused either on Innovation & Technology or Workforce Development, highlighted the need for culturally appropriate, accessible, and credible outreach and education and hands-on experience with newer energy efficiency technology. Participants in all sessions related that this information is best received from trusted sources, reiterating the best practice of connecting with and leveraging existing outreach and education channels provided by CBOs and recognized community leaders.

On the question of WE&T, participants shared that any WE&T efforts must navigate structural barriers to successful training, identifying both structural and logistical barriers and suggesting many potential practices and solutions to address these barriers. Further, they reflected that training and education must connect ESJ participants to long-term, quality jobs. In addition to strategies for enhancing training and educational offerings, listening session conversations repeatedly called out the need to link these trainings with “high road” jobs that are well-paid and offer benefits and paths for advancement.

There are actually jobs out there that they never knew of -- it's not just auto mechanics or electrical, there's energy efficiency, heat pumps, weatherization that they can create a living off of

Participants identified other barriers to making the benefits of market transformation more accessible to ESJ communities, including high concerns about grid resiliency due to the impacts of rolling blackouts and other outages. Discussions also raised the challenge associated with suboptimal conditions of homes, noting that many homes in ESJ communities do not have the capacity to handle the increased electrical load required for some decarbonization measures without significant upgrades to panels, wiring, or other equipment.

Effectively addressing any of these barriers and opportunities requires that CalMTA engage ESJ community members in research, design, and evaluation to hear from community voices directly and ensure technologies and interventions accurately respond to needs and achieve energy equity goals of a particular MTI.

So how do we leverage environmental and EE organizations to utilize them as sounding boards, and advocates in the community...I would say, bring them along.

Recommendations

CalMTA proposes six recommendations based on the listening session takeaways including:

  • Working with trusted entities to help integrate the voices of ESJ communities into CalMTA’s work and support co-creation of culturally appropriate messaging and education.
  • Ensuring that MTIs include strategies designed to minimize potential risk or unintended negative impacts when promoting less proven energy efficiency technologies, or those the market is less able to support, to ESJ communities.
  • Leveraging upstream market interventions to share insights from ESJ communities with key market actors, demonstrating the potential for increased market share by making technologies more accessible to this customer segment.
  • Designing residential initiatives targeting ESJ communities with an understanding of specific housing stock characteristics and needs, including envelope upgrades that will optimize the performance of heat pump technology.
  • When MTI interventions target ESJ community member participation (e.g., workforce development), ensure strong partnership and collaboration with existing programs and work to incorporate wraparound services like childcare or transportation assistance.
  • As much as possible, embed equity considerations and create mechanisms to prioritize the needs of diverse participants in procurement protocols.

read the report, Listening Sessions with ESJ Communities

Blog: Stage 1 Disposition Report highlights initial actions to form market transformation in California

Blog

Outcomes of the 2023 RFI solicitation and ideas selection process summarized

CalMTA is excited to release its final report on the outcomes of our first Request for Ideas (RFI) held last summer. This Stage 1 Disposition Report provides details about the RFI process, insights into submissions received, the scoring and selection process used to identify and advance California’s first market transformation ideas, and the future work required to develop them into full initiatives in 2024 and 2025. This report is the first of two parts and summarizes the work completed through Stage 1 scoring.  

A second report, due to be completed in June 2024, will provide updates on Batch 1 Market Transformation Initiative (MTI) development and a detailed description of additional idea scoring and recommendations for a second batch of MTIs. These documents fulfill requirements of the California Public Utilities Commission (CPUC) Decision 19-12-021, which established the comprehensive statewide market transformation framework and created CalMTA as the state’s market transformation administrator. CalMTA is a program of the CPUC and is administered by Resource Innovations.  

Request for Ideas results

CalMTA’s first RFI, which lasted eight weeks and closed Aug. 18, 2023, resulted in 117 ideas submitted by stakeholders and market actors. Submissions ranged across sectors, but most fell into the residential category. Likewise, there were roughly three times the number of product-focused ideas than those targeting practices. Since then, CalMTA has conducted a rigorous review and scoring process to identify a first batch of MTIs for further development based on criteria that includes energy savings and grid benefits, product readiness, estimated cost/cost-effectiveness, equity considerations, and alignment with market transformation theory.  

Over one-third, 34%, of the submitted ideas prominently featured heat pumps. In some cases, these ideas promoted a heat pump as a stand-alone product for heating, cooling, and water heating. In others, they were included as part of broader strategies to improve the overall efficiency of a building or facilitate its decarbonization. There were also ideas geared toward workforce development and training to help accelerate heat pump adoption and create career opportunities for disadvantaged communities.  

Other end uses that generated significant interest include food storage and service and opportunities to create efficiency and mitigate the high global warming risk posed by refrigerants. Potential improvements in the efficiency of the cooking process itself were addressed for both residential and commercial kitchens including promotion of induction cooking. The building envelope was the focus of 17 ideas with several related to evolving window products and others that propose utilizing shading and insulating technologies. For commercial and industrial processes, 15 ideas involved process monitoring, controls, and efficient motors and pumps.  

Idea review & scoring

A multi-stage review process resulted in prioritization of submitted ideas beginning with an initial threshold review process to ensure that the submission was an appropriate fit for market transformation investment. The 92 ideas that passed threshold review moved into Stage 1 scoring, which ranked them against set criteria based on information that was included in the submission as well as our scoring team’s market expertise and some limited research.  

The figure shows the breakdown of how ideas were ultimately disposed through Stage 1 scoring including three that advanced for further research. The figure above shows the breakdown of how ideas were ultimately disposed through Stage 1 scoring including three that advanced for further research.

Through this process, 38 ideas addressing similar market segments, technologies, or practices were combined by our team to strengthen the overall MT concept. Of the 54 remaining ideas, 21 of the lowest scoring were archived, 14 were held for further research, and 19 of the top-  

ranked ideas were then slated to advance on to Stage 2 scoring, where our team conducts deeper secondary research, investigation, and analysis, including Total System Benefit (TSB) and other cost-effectiveness calculations. Three “frontrunner” Batch 1 ideas were recommended for expedited Advancement Plan development based on market transformation alignment, long-term value of California, and commercial readiness. Advancement Plans describe the research and investigation needed to develop these ideas into full MTI plans later this year.  

We are grateful to those who submitted their ideas for consideration and look forward to continued engagement as we move forward with development of future batches of market transformation ideas. This Stage 1 Disposition Report provides a foundational step in that work and CalMTA’s goal of creating a balanced market transformation portfolio that helps meet California’s energy, decarbonization, equity, and workforce development objectives.  

A recording of CalMTA’s presentation of the report at a Nov. 30-Dec. 1 meeting of our Market Transformation Advisory Board is also available on our website. The Disposition Report discussion began at the 24-minute mark on Day 1.  

Read the Disposition Report

 

Blog: RFI helps shape market transformation in California

Blog

A significant step toward establishing a market transformation portfolio for California concluded with the close of CalMTA’s first Request for Ideas (RFI) on August 18. The market transformation administrator received 117 submissions describing innovative technologies and practices that will help the state reach its energy efficiency and decarbonization goals through market transformation. RFI submittals are now being reviewed and scored by the CalMTA team, with the highest-ranked ideas selected for further development as California’s first full-scale market transformation initiatives (MTIs). The first set of ideas will be shared at the public Market Transformation Advisory Board meeting on Oct. 13.  

Summary of submitted ideas 

RFI responses encompass a diverse range of technologies, products, services, and market segments. The majority of submitted ideas fell into the commercial (43%) or residential (40%) sectors, although we also received industrial, agricultural, and cross-sector submittals. Process loads; building envelopes; heating, ventilation, and air conditioning (HVAC); and domestic hot water systems (DHW) were the most prevalent load types addressed by submittals, with additional ideas focused on foodservice technologies and consumer products. 

Results of CalMTA’s first RFI, showing distribution of submissions by type, load type, and target sector

Market response 

Stakeholders across California and nationwide showed great interest in the RFI process, with 63 unique submitters that included nonprofits, energy efficiency program implementers, consultants, manufacturers, utilities, and research firms.  

“We wanted to make sure the process was easy and accessible to anyone with a viable idea that could eventually become an MTI,” said Jeff Mitchell, principal of market transformation development. “The best part for me was meeting with submitters who signed up for office hours to discuss their ideas to increase market-level energy efficiency and decarbonization.”  

CalMTA held informational briefings with 32 interested organizations and distributed toolkits of easy-to-share RFI promotional materials to more than 60 ally organizations to increase awareness of the RFI. Two public webinars drew nearly 120 registrations. RFI submittal support services provided by CalMTA also proved popular, with 21 individuals scheduling office-hour meetings with our MTI team and 14 engaging with our online Q&A discussion board.  

Next steps 

An initial threshold review process will ensure that submitted ideas (1) save energy, (2) are commercially available or on track to be, and (3) include enough information to be scored. Top ideas will advance based on weighted scoring criteria that includes: 

  • total system benefit (TSB),  
  • commercial readiness,  
  • estimated MTI cost/cost-effectiveness,
  • non-energy impacts,  
  • beneficial impacts to environmental and social justice (ESJ) communities, and 
  • alignment with the market transformation framework used by CalMTA.  

Selected ideas will then be developed into Advancement Plans, which lay out the activities needed to develop full MTI Plans for implementation, such as market and technology research, pilots and demonstration projects, workforce development needs, and stakeholder engagement. The team will begin finalizing Advancement Plans for the first batch of MTIs in late 2023 and early 2024, as shown below. By the end of 2024, CalMTA will submit an application to the CPUC seeking funding approval for an initial set of MTIs.

MTI Development Timeline

This graphic shows the progress of MT ideas through the first two phases of the MTI development process, with key milestones indicated through the end of Phase II (click to enlarge).

CalMTA will develop Advancement Plans for additional MTI ideas throughout 2024 and beyond in a series of batches. Long term, the resulting market transformation portfolio will work to reduce energy consumption, promote decarbonization, increase grid health, and support equity and workforce development goals.

“There’s so much market transformation opportunity in California and I’m excited to see how MTIs drive long-lasting, equitable change across the energy landscape here,” Mitchell said. “These initiatives will help continue the state’s leadership in energy efficiency work and benefit the lives of Californians for decades to come.”  

Keep in touch 

To stay informed of development of CalMTA’s MTI portfolio, register to receive email updates, follow the CalMTA LinkedIn page, and join upcoming CalMTA Market Transformation Advisory Board (MTAB) meetings, which are open to the public.  

 

 

Blog: Catalyzing market change for California’s energy future

Since January, when the California Public Utilities Commission (CPUC) announced Resource Innovations as the state’s first-ever Market Transformation Administrator (CalMTA), a lot of behind-the-scenes work has been happening. Resource Innovations has staffed up CalMTA and has launched a key advisory board, known as the Market Transformation Advisory Board (MTAB). And now, the process of soliciting ideas to create the first initiatives in California’s market transformation portfolio begins. This “request for ideas” or RFI, which is open June 15 through August 18, is a way for everyone to participate in helping create market transformation initiatives (MTIs) for California. Once the MTIs are approved by the CPUC, CalMTA will lead a competitive solicitation to select the implementers.

Resource Innovations anticipates that the RFI will establish an arena to collect inventive and imaginative concepts, which can be leveraged to build a highly effective portfolio of market transformation initiatives in California.

Better Together

Market transformation is a proven strategic process of intervening in a market to create lasting change. In the context of CalMTA’s work, these changes work to deliver cost-effective energy efficiency and support California’s other goals on greenhouse-gas (GHG) reduction, workforce development, and equity.

The impact and savings opportunity of market transformation, shown as the classic s- or duck curve

 

Market-level interventions, which are a key component of the MTIs that CalMTA is developing, break down barriers to energy efficiency, such as lack of awareness, high costs, or low product availability. By creating a more competitive market for energy-efficient technologies and practices, market transformation can drive down costs and increase adoption, ultimately leading to greater energy savings and environmental benefits that remain long after active market interventions have ended.

CalMTA plans to align and collaborate with California’s current programs, utilities, and stakeholders to prioritize efforts that result in the highest-value market transformation, which delivers not only efficiency goals, but also decarbonization and demand flexibility. All MTI development will look specifically at how beneficial impacts can be delivered to environmental and social justice (ESJ) communities, which have historically been left out of efficiency investment.

Propose Your Idea

If you have an idea that will accelerate market adoption of a new or under-used energy efficiency technology or practice, then we want to hear about it! CalMTA is seeking ideas that will help California reach its energy goals and also advance state priorities for accessing hard-to-reach, low-to-moderate income customers, or disadvantaged communities. The Request for Ideas (RFI) officially opens on June 15, but you can learn more about the process on this website’s participate section or join the June 14 live briefing.

 

All ideas must result in energy savings but could also offer non-energy benefits and climate-friendly impacts that Californians value. If your idea is selected for advancement, it will then be developed into a plan for implementation. Once all the plans are designed and submitted, the process of bidding out the implementation will begin.

MTI development will be done in collaboration with the MTAB, which will provide expertise and unbiased, non-binding recommendations via public forums on the design, development, and deployment of market transformation initiatives.

Market transformation and energy efficiency are critical components of efforts to reduce greenhouse gas emissions, address climate change, and ensure a clean energy future for all.

A full summary of work to be completed by CalMTA this year is included in the 2023 Operations Plan.

You can find more information about CalMTA here, or join our email list to learn about meetings, input opportunities, and our work.
For more on RI’s MT capabilities, visit: resource-innovations.com/utility-services/market-transformation

Meet the Advisory Board

The newly formed advisory board includes representatives from investor-owned utilities, workforce development, ratepayer advocacy and environmental advocacy, as well as evaluation and efficiency experts.

Cyane Dandridge, Founder & Executive Director of Strategic Energy Initiatives

Hayley Goodson, Managing Attorney with TURN (The Utility Reform Network)

Fred Gordon, Director of Planning and Evaluation at The Energy Trust of Oregon

Jeff Harris, Chief Transformation Officer for the NEEA (Northwest Energy Efficiency Alliance)

Randall Higa, Codes and Standards Program Manager and lead for Zero Net Energy Strategies at Southern California Edison

Lujuana Medina, environmental initiatives sections manager for SoCalREN

Peter Miller, Senior Scientist with NRDC (Natural Resources Defense Council)

Christie Torok, Regulatory Analyst at the CPUC (California Public Utilities Commission)

Ky-Ann Tran, member of the CPUC’s Cal Advocates

This blog is by Margie Gardner, VP of Market Transformation, and was originally posted at www.resource-innovations.com/resources/ideas-action-catalyzing-market-change-californias-energy-future 

© 2024 CalMTA. All rights reserved.
Powered by Archie