Printer Friendly

Leveraging Smart Meter Data & Expanding Services.

Widely available interval data and recent advances in computing power are fostering the development of new technologies that streamline data collection and perform advanced analytics. The approach supports energy performance assessment by time of day. It underlies next generation measurement and verification (M&V) methods, which we refer to as advanced M&V. Advanced M&V has advantages over traditional regression-based billing analysis, including more rapid feedback, improved savings resolution, and increased model accuracy due to many more data points. In addition, the same data and analytics can be applied to identify operational abnormalities, which can support fruitful, longer-term customer engagement. These developments provide an opportunity for energy service providers to leverage smart meter data and advanced M&V to inform and expand building efficiency projects and services.

The application of advanced M&V methods is more complex than typical whole-building M&V methods using monthly billing data. Interval data M&V models include new forms of linear regression models that incorporate time indicator variables. In addition, as the time interval becomes smaller, the presence of autocorrelation in the data increases, which impacts savings uncertainty and necessitates its assessment. Applying advanced M&V methods can have added benefit if operational improvement savings can be determined with sufficient accuracy to be discernible using whole building data. In this article, we investigate these considerations through a case study.

M&V Methods

In the late 1980s, the International Performance Measurement and Verification Protocol (IPMVP) established an industry-accepted framework for M&V which includes four options for determining verified savings. Performing M&V using whole-building data to develop empirical models aligns with an IPMVP Option C approach. Option C involves developing an M&V model from baseline period whole building energy use data and independent variables, then projecting the baseline energy use into the post-install period conditions. Savings are determined by subtracting the measured post-installation period use from the adjusted baseline use. (1)

In the 1990s, ASHRAE began developing Guideline 14, Measurement of Energy, Demand, and Water Savings, to provide procedures for using measured data to verify savings. (2) At the same time, ASHRAE initiated Research Project 1050 (RP 1050), which created the inverse modeling toolkit (IMT). The IMT includes change-point models, which are piece-wise linear regression models with each segment meeting at a common point. (3) The IMT is commonly applied with Option C using monthly utility billing data and corresponding average monthly temperatures or heating and cooling degree days. It provides a simple and powerful approach for understanding the weather dependence of building energy use, but without temporal considerations.

Recent developments in interval data M&V modeling include new forms of linear regression models that incorporate time indicator variables. For example, the time-of-week and temperature (TOWT) model developed for demand response applications (4) considers the time of the week as well as the ambient temperature as influences on energy use. The TOWT model captures the nonlinear relationship between outdoor air temperature and load by dividing temperatures into many intervals and then fitting a piecewise linear and continuous temperature dependence. This means that for one year of hourly data, the TOWT model would include 168 (hours in a week) linked regressions; each developed from at least 52 data points (8,760 hours in a year).

A challenge for Option C applications is ensuring that the anticipated project savings are "discernable." Guideline 14 requires that savings be stated within [+ or -] 50% at the 68% confidence level to comply. Efficiency project stakeholders (owners, financiers, and service providers) would certainly desire more stringent criteria. One way to ensure savings meet uncertainty criteria is to develop accurate energy models for use in the analysis. This is one area where the interval data approach is advantageous over the monthly billing data approach.

Assessing Savings Uncertainty

Since actual savings can only be estimated and not compared with a directly measured value, we discuss the quality of our savings estimations in terms of their uncertainty. Uncertainty is an expression of the probability or confidence with which an estimate is within specified limits of the true value. As will be shown, the accuracy with which we can model the baseline period energy use has a direct effect on the uncertainty of our savings estimate.

Monthly models are usually developed using ordinary least squares regressions and 12 data points. Savings must be large to be discernable above the "noise" (random error of the regression model). For models developed from monthly data, this concern led to the rule-of-thumb that savings must be at least 10% or 15% of annual energy use.

Interval data converted to hourly or daily data provide orders of magnitude more data than monthly. As a general rule, the more data used to develop regression models, the more accurate the model. However, as the time interval becomes smaller, the presence of autocorrelation in the data increases. This reduces the independence of the data points and serves to increase the savings uncertainty, as discussed below.

Baseline model accuracy is assessed by metrics that quantify their random and bias error. The bias error shows how much the model under- or over-predicts the total actual energy consumed in the model training period, and should be zero, or extremely small. The random error indicates how well the model follows the usage patterns of the actual data; good models follow these patterns very closely. The metric most commonly used to quantify its random error is the root mean squared error (RMSE), or the coefficient of variation of the RMSE (CV[RMSE]), which is the RMSE divided by the average baseline energy use.

ASHRAE Guideline 14-2014 provides a simplified approach to estimate savings uncertainty for two cases: 1) weather-dependent linear models from data that are not serially correlated, and 2) weather-dependent linear models from data that are serially correlated. In the case of serial correlation, Equation 1 shows shows the relationship between the relative uncertainty and CV(RMSE), fraction of savings F, number of baseline and post-installation period data points n and m respectively, number of model parameters p, and confidence level (specified by the t-statistic t with a confidence level). Equation 1 accounts for the presence of autocorrelation (also known as serial correlation) in the data through the term n', which indicates the effective number of data points. For weather-dependent models without autocorrelation, the same form of Equation 1 is used, but n' is replaced by n. For example, monthly data are generally assumed to have no autocorrelation and n equals 12 for a one-year analysis. Guideline 14 provides details on how to determine [rho], the autocorrelation coefficient.

Equation 1


Equation 1 shows that the better the model goodness of fit (low MSE or CV(RMSE)), the more data points, and the higher the savings, the lower the savings uncertainty. The level of autocorrelation in the data (high [rho] values, which lower the effective n) tends to increase as the time interval becomes smaller.

This simplified relationship provides a convenient way to assess an advanced M&V approach in the planning stages of efficiency projects. For example, project stakeholders can establish how accurately savings must be reported. Based on the criterion, baseline data can be collected, a baseline model developed, and Equation 1 applied with assumptions made for savings and post-installation measurement period. Stakeholders can decide if the resulting savings uncertainty is acceptable. If not, another M&V approach can be developed.

This methodology is straightforward to apply; however, it was developed based on linear methods assuming normal distributions and independence of data. While the correction for autocorrelation helps, M&V models are being developed with complex algorithms that are nonlinear in nature. Advanced M&V algorithms include names such as vector machines, neural networks, and machine learning. Often these methods are proprietary. Methods for quantifying savings uncertainty using these algorithms are more complicated and are an area of current research.

Example Case Study

We applied the Guideline 14 methods to a portfolio subset that included whole-building interval data for 17 large commercial buildings located in climate zone 5a. Our client sought to understand the benefits of a streamlined advanced M&V approach applied across a large group of projects emphasizing operational improvements. The client wanted to know how precisely an Option C M&V model could determine savings prior to embarking on scaled implementation. Should the Option C approach not meet their accuracy requirements, they would explore a different M&V approach or consider a service-based business model instead of offering a performance guarantee.

We investigated the improvements in accuracy realized from an advanced M&V approach by developing three models for each site using monthly, daily, and hourly intervals for one year of data. We aimed to demonstrate a streamlined approach to evaluate the M&V model uncertainty and verified savings across the portfolio. The selected form of the monthly model was a four-parameter (4p) change point linear regression since it provided the best data fit overall for the sites. Commercial software that applies the IMT algorithms was used to develop the regressions. The selected daily and hourly data models were the LBNL TOWT (4) regressions developed using the Universal Translator 3.0 (UT3)tool. (5) We performed general quality assurance procedures but did not develop site-specific data filters to improve regression fit on a case-by-case basis. We determined the savings uncertainty for each model using Equation 1, assuming the same duration for the baseline and post-installation periods (m = n).

Figure 1 presents normalized monthly energy use for each site over the calendar year. The sites are ordered from largest to smallest (dark to light lines). Floor areas range from 3,500,000 [ft.sup.2] to 80,000 [ft.sup.2] (325 161 [m.sup.2] to 7432 [m.sup.2]). The figure indicates magnitude, range, and seasonal variations across the data set. Many of the buildings use some electric heat and a few receive cooling from a district chilled water loop, resulting in many of the buildings' winter month electric use exceeding summer month electric use.

Figure 2 shows the 4p regression plots developed using monthly data. They indicate the varying temperature dependence of electricity use for each site in the portfolio. The slopes on either side of the change point are indicative of the relative cooling and heating system efficiencies of the modeled building. The change point indicates the building balance point temperature above or below which space conditioning begins. No attempt was made to graph and present the annual data for the interval data or more complex TOWT models. However, their "energy signatures" can provide more detailed insights by capturing load shape trends, which may be best assessed through automated statistical methods or machine learning techniques due to voluminous data processing.

The regression statistics and auto-correlation values determined for the three models for each site are presented in Table 1. In general, the regression models developed with more granular data have larger residuals relative to the predicted value as indicated by their higher coefficient of variation (CV) and lower R-squared values. As indicated, the autocorrelation coefficient is zero for monthly data. For some sites, the level of auto-correlation was determined to be significant for the daily and hourly utility data, which results in the effective number of independent data points being much less than the actual number of data points. The uncertainty values, reported for the 90% confidence interval, are relative to the baseline annual energy consumption (determined by multiplying both sides of Equation 1 by F, the savings fraction). We refer to these values as the baseline energy use uncertainty fraction.

A comparison between the different interval data model fit is presented in Figure 3 for two sites. The Site 4 interval data models provided little or no improvement in accuracy due to high autocorrelation and actual-to-model data variance. As indicated in the chart, the hourly data model for Site 4 poorly predicted the actual data for swing season months possibly due to changes in seasonal equipment scheduling. The model accuracy improved the most with interval data for Site 9. For this site, the interval data had limited autocorrelation and the model had a relatively low CV.

Figure 4 presents the calculated uncertainty for a range of confidence intervals for the monthly, daily, and hourly regression models for each site. The values indicate the inaccuracies introduced by the M&V model in predicting performance. The results show that the uncertainty is reduced on average by 40% to 60% if using a daily data model instead of a monthly data model with the most impact seen at higher confidence intervals. Uncertainty is reduced further for hourly models--on average an additional 25% reduction relative to the daily model across all confidence intervals.

The baseline energy uncertainty fractions shown in Figure 4 can be compared against the anticipated project savings fraction to assess if the M&V approach is effective for verifying project savings using whole-building electric data. Since the energy services will be offered across a portfolio of buildings, it makes sense to quantify the aggregated uncertainty of the data set. Since each building is unique from the next, the fractional uncertainty across the portfolio is determined using Equation 2, where AE is the savings uncertainty and N is the number of buildings in the portfolio.

Equation 2.


Figure 5 shows the uncertainty expressed in terms of the anticipated savings. Our client estimates the range of project savings to fall between 5% and 15%. In the figure, the plot area upper bound is based on a 5% portfolio savings while the lower bound is based on 15% portfolio savings for a range of confidence intervals. The results show that the accuracy of verifying savings is noticeably improved using interval data.


In our analysis, Guideline 14 equations quantified the uncertainty associated with the baseline M&V models. Developing models using shorter time interval data did drive down uncertainty--a trend expected for good models. The presence of autocorrelation greatly reduced the number of independent data points. Instead of going from 12 to 365 to 8,760 data points, the effective number of data points was on average 12 to 150 to 700 (based on the average n in Table 1).

The treatment does not account for other primary sources of quantifiable uncertainty, including population sampling uncertainty and equipment measurement uncertainty. For the case study, no population sampling across the portfolio was attempted for this preliminary assessment. Also, measurement uncertainty is minimized by the use of revenue meter data. Other factors contributing to savings uncertainty that should be considered, although they can be difficult to quantify, include model mis-specification, lack of data on driving variables, and unaccounted for changes in load or operation conditions. Therefore, effective approaches for assessing accuracy must include quantified and methodological considerations. For instance, excluded driving factors can be checked by looking at residual plots for unexplained patterns, a standard best practice in regression modeling. The more we can account for uncertainty in determining savings, the better we can manage risk and have confidence in project results.

Although Guideline 14 and IPMVP methods represent current best practices for quantifying uncertainty, there are shortcomings to the methods, which are actively being discussed by industry experts. The methods are limited to linear models and how they account for the real effect of autocorrelation. The best-practice equations apply to regressions with one independent variable but methods are needed to account for more advanced equations with multiple variables. In addition, service providers and building owners would benefit from guidance on goodness of fit criteria. The Industry should specify these criteria and outline procedures for evaluating an M&V Plans including an Option C, advanced M&V approach.

Another important consideration of current M&V methods is the approach for accounting for non-routine events. Conditions and behaviors in buildings effecting energy loads are constantly changing. Conventional M&V practice includes creating a baseline model to account for routine adjustments and using engineering calculations on an as-needed basis to make non-routine adjustments. Generally, only large changes are tracked used to adjust savings. This represents a source for high savings uncertainty for smaller projects. These issues can be managed with new advanced M&V tool capabilities that automate the identification of unexpected changes in whole building energy use. The U.S. Department of Energy and leading utilities are sponsoring research for quantifying the need for non-routine adjustments, comparing the ability of different tools to detect unexplained performance changes, and evaluating simplified approaches for estimating uncertainty that is independent of model algorithm. These efforts, which support the development of a standardized methodology, will greatly help the industry.


The results from our portfolio-level analysis helped our client inform their offering and business model by indicating the potential savings risk introduced by the M&V model. The analysis showed that the models developed with interval data reduced the portfolio savings uncertainty by 50% or more. At the high confidence levels that were of interest to our client (e.g., 90% and greater), the interval data models performed significantly better than the monthly data models. The assessment indicated that they would need to use whole-building utility interval billing data for M&V to assess operational improvements in order to achieve their desired level of accuracy.

The evaluation demonstrates how IPMVP and Guideline 14 methods can be applied to interval data, as well as the order of magnitude of the resulting accuracy improvements that may be achieved. Using an advanced M&V approach can also provide additional benefits, including insights into building operations, near real-time savings assessment, and potentially shorter project monitoring periods. However, methods are still under development. Standardized and effective means to account for interval data autocorrelation are under discussion. Researchers are documenting and demonstrating advanced M&V tool capabilities and developing testing procedures. The potential for such methods to expand the scope and means for delivering services is something that is worth service provider consideration. Incorporating uncertainty analysis into M&V methods is a key first step.


(1.) EVO. 2016. "International Performance Measurement and Verification Protocol, Core Concepts." Efficiency Valuation Organization.

(2.) ASHRAE Guideline 14-2014, Measurement of Energy, Demand, and Water Savings.

(3.) ASHRAE. 2004. Research Project 1050, Inverse Modeling Toolkit.

(4.) Mathieu, J, P. Price, S. Kiliccote, M.A. Piette. 2011. "Quantifying Changes in Building Electricity Use with Application to Demand Response."

(5.) 2016. "Universal Translator 3 (UT3)." http://

Ellen Franconi, Ph.D., is energy analytics manager at the Rocky Mountain Institute in Boulder, Colo., and vice chair of SPC 209, Energy Simulation Aided Design for Buildings Except Low-Rise Residential Buildings. David Jump, Ph.D., PE., is project manager at kW Engineering in Oakland, Calif.

Caption: FIGURE 1 Monthly electricity consumption.

Caption: FIGURE 2 Monthly 4P change point models.

Caption: FIGURE 3 Comparison between Site 4 and Site 9 M&V model data fit.

Caption: FIGURE 4 Uncertainty associated with monthly, daily, and hourly baseline electricity regression models.

Caption: FIGURE 5 Portfolio-level uncertainty fraction associated with a 5% to 15% estimated savings range.
TABLE 1 M&V electricity models regression statistics and
example results.


          [R.sup.2]    CV    n   [rho]   n'         U
                                              (90% CI)

Site 1         0.9     2%   12      0    12      1.7%
Site 2        0.96     4%   12      0    12      3.2%
Site 3        0.71     9%   12      0    12      6.8%
Site 4        0.87     7%   12      0    12      4.8%
Site 5        0.94    12%   12      0    12      8.5%
Site 6        0.99     5%   12      0    12      3.9%
Site 7        0.82     9%   11      0    11      7.4%
Site 8        0.94    11%   12      0    12      8.1%
Site 9        0.84     8%   12      0    12      6.0%
Site 10       0.98     6%   12      0    12      4.3%
Site 11       0.89    10%   12      0    12      6.9%
Site 12       0.97     4%   12      0    12      2.8%
Site 13       0.96     8%   12      0    12      5.7%
Site 14       0.81    20%   12      0    12     14.2%
Site 15       0.98     9%   12      0    12      6.2%
Site 16       0.98     9%   12      0    12      6.2%
Site 17       0.95     7%   12      0    12      5.0%


          [R.sup.2]   CV      n   [rho]   n'          U
                                                (90% CI)

Site 1        0.87    10%   365   0.35    177       1.6%
Site 2        0.92     9%   364   0.20    242       1.2%
Site 3        0.66    14%   352   0.71     59       3.9%
Site 4        0.74    20%   365   0.71     61       5.4%
Site 5        0.89    21%   364   0.13    278       2.6%
Site 6        0.94    15%   364   0.34    180       2.3%
Site 7        0.57    26%   364   0.71     61       7.0%
Site 8        0.81    22%   366   0.71     61       5.9%
Site 9        0.90     9%   365   0.17    258       1.2%
Site 10       0.92    16%   365   0.13    281       2.0%
Site 11       0.70    23%   365   0.28    205       3.3%
Site 12       0.89    10%   364   0.31    192       1.5%
Site 13       0.86    17%   356   0.11    287       2.1%
Site 14       0.59    31%   364   0.72     59       8.6%
Site 15       0.86    24%   364   0.31    192       3.7%
Site 16       0.74    19%   364   0.38    162       3.2%
Site 17       0.80    17%   364   0.19    247       2.3%


          [R.sup.2]    CV      n    [rho]     n'          U
                                                    (90% CI)

Site 1        0.92    17%   8,782   0.87      588       1.4%
Site 2        0.84    19%   8,758   0.83      815       1.3%
Site 3        0.76    24%   8,471   0.95      233       3.3%
Site 4        0.78    33%   8,782   0.95      242       4.5%
Site 5        0.84    40%   8,603   0.77    1,104       2.5%
Site 6        0.85    24%   8,758   0.86      650       1.9%
Site 7        0.61    33%   8,758   0.91      395       3.4%
Site 8        0.77    28%   8,782   0.95      242       3.7%
Site 9        0.91    17%   8,782   0.81      918       1.2%
Site 10       0.80    32%   8,782   0.83      824       2.3%
Site 11       0.63
COPYRIGHT 2017 American Society of Heating, Refrigerating, and Air-Conditioning Engineers, Inc. (ASHRAE)
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2017 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Author:Franconi, Ellen; Jump, David
Publication:ASHRAE Journal
Geographic Code:1USA
Date:Sep 1, 2017
Previous Article:Living Wall Retrofit.
Next Article:Water-Energy Nexus: Heat Rejection Systems.

Terms of use | Privacy policy | Copyright © 2019 Farlex, Inc. | Feedback | For webmasters