Using strategic market research to inform policy decisions

Marianne Lourey and Brent Taylor

KPMG Regulators Conference Melbourne Australia 27 February 2003

The late 20th and early 21st centuries have been an era of great change in the way Australian utilities operate.  There has been a shift from Government ownership to private ownership, albeit incomplete, and there has been a transformation from a focus on a capital works program to serving consumers.  This shift has occurred in ports, rail, road, gas, water, sewage and electricity.

The policy issues underpinning this reform, and the policy decisions that continue to be made by regulators and governments with respect to these industries, are predicated on a range of assumptions that are made by policy makers.  Examples of the types of assumptions that are made are in relation to:

n       The reliability and quality of service that consumers value;

n       The responsiveness of consumers to price signals;

n       The responsiveness of consumers to environmental concerns; and

n       The extent to which consumers are willing to ‘cross subsidise’ other consumers.

This paper explores how strategic market research can be used to inform or challenge policy decisions that are based on these types of assumptions.  Whilst the rest of this paper will be focus on the electricity industry, the messages can be equally applied to the other industries.

The role of strategic market intelligence

The world is awash with market research data.  Most organisations, whether private or public sector, in a competitive market, natural or legislated monopoly, carry out market research.  Traditional types of market research include tracking research, brand research and customer relations research.  A major criticism of traditional research methodologies is the way in which they tend to reflect and entrench conventional wisdom about a problem or opportunity.  Traditional market research methodologies are less effective at exploring paradigm shifts in thinking and/or behaviour.  Furthermore, it usually produces “soft” data that cannot be used in financial and economic models.

Accordingly, traditional research methods are often not able to inform or challenge the assumptions which underpin many policy decisions.  As a result, these assumptions are generally formed in a vacuum, and policy decision makers are frequently oblivious to consumer preferences.

Increasingly policy decision makers, and parties that seek to influence policy makers, are turning to strategic market intelligence.  It is not the normal type of market research that asks feel good questions and ends up in the bottom drawer.  What differentiates strategic market intelligence from conventional market research is the ability to explore new paradigms and the direct applicability of its outputs.  Of course it uses some of the methods of conventional market research (such as stratified sample design, telephone interviewing etc) but it utilises additional tools to be able to explore “beyond the square”.

These additional tools include:

n       Development of perceptual measurement scales that translate complex industry terms in such a way that they are meaningful to consumers so that they are able to respond, but produce results that are meaningful to the industry.

n       Psycho-physical mapping of perceptual measures onto actual performance so that changes perceived by consumers can be translated to the actual change required by the industry to deliver that perceived change.

n       Identification of a full list of attributes that are meaningful to consumers, which may be different to a list of attributes that are assumed to be meaningful by industry.  

n       Identification of an appropriate questionnaire framework.  Questionnaire development appears easy; many people do it.  However, it is not easy to develop a questionnaire that ‘surrounds’ the problem, that is, it contains all the obvious questions to ask, plus enough apparently ‘wild card’ questions to capture the true drivers of behaviour.

n       Modelling – this is statistically linking a cause with an effect.  Identification of true business drivers guides the strategic allocation of resources.

KPMG has recently been engaged to undertake strategic market research that will inform the setting of standards for the distribution electricity businesses in South Australia and New South Wales.

Using strategic market research to inform the setting of standards

Consumers in fully competitive markets make their consumption decisions based on a variety of criteria – only one of which is cost and others that might be summarised as elements of quality or performance.  A competitive market responds to these signals, maximising efficiency in terms of both cost and output performance.  Arguably one of the objectives of regulation should be to provide the cost and performance incentives that mimic the manner in which competitive markets reward efficient businesses.

Cost and performance typically bear an inverse relationship to each other; increasing levels of performance typically results in higher costs, and decreasing costs typically leads to a reduction in performance[1].  Regulation aimed at promoting business efficiency (and hence desirable outcomes) must address both performance and cost efficiency to be effective. 

To date, industry regulators have tended to focus on cost based regulation.  However, as many regulators, commentators and regulated agencies have noted, the application of cost incentives alone creates an incentive environment that can actively promote decreased service standards.

The setting of these incentives requires the regulator to develop a detailed understanding on the value consumers place on differing levels of cost and performance, through the use of strategic market research. 

Willingness to Pay or valuation of consumer cost and performance preferences

Strategic market research into consumer preferences for quality and costs is a complex and intensive process.  In the context of regulated industries this research is made even more problematic due to the nature of the services under consideration for which many customers may not have consciously formed cost and performance preferences.  However, a number of research methodologies have been developed which enable such research to be undertaken[2].

Stated preference methodologies are considered to be the most appropriate methodologies for ascertaining consumer preferences and economic values for non-market goods and service.  Stated preference methodologies are based on respondents being asked to consider one or more hypothetical options and to state their preferences for the options.  It is argued that choice modelling, a stated preference methodology, results in measurements of consumer preferences that most closely match observed consumer behaviours. 

There is a robust discussion amongst the academic practitioners about the most appropriate method to use.  Irrespective of the advantages offered by the alternative stated preference methodologies, they are fiendishly difficult to implement properly.  Use of such methodologies requires absolute rigour in: selecting attributes; developing attribute levels; scenario specification; experimental design; sample selection; data collection; modelling; matching perception to reality and interpretation of the results in a meaningful way.  Not a method for the faint hearted!!

How good is good enough?

The electricity industry has adopted standard indicators for measuring reliability, refer Box 1.  These measures have generally been set based on historical performance, with an implicit objective to improve the measures year on year.  The strategic market research can be used to inform:

n       Whether these are the appropriate measures for measuring reliability performance; and

n       The setting of target reliability indexes based on a level of performance that consumers value – consumers may not value an improvement.

Box 1: Standard industry measures for reliability performance

n       SAIDI – System average interruption duration index, which is the sum of the duration of each sustained customer interruption[3] divided by the total number of customers;

n       SAIFI - System average interruption frequency index, which is the total number of sustained customer interruptions divided by the total number of customers;

n       CAIDI – Customer average interruption duration index, which is the sum of the duration of each sustained customer interruption, divided by the total number of sustained customer interruptions; and

n       MAIFI – Momentary average interruption frequency index, which is the total number of momentary customer interruptions[4] divided by the total number of customers

KPMG’s research indicates that consumers are generally realistic in their expectations of the network performance and will tolerate a small number of interruptions.  As a result, there appears to be a large proportion of consumers that are satisfied with their existing level of performance.  Mostly these consumers have “good” service and do not value improvements in performance. 

Conversely there are some consumers who experience very poor reliability of supply.  These consumers do value an improvement in supply.

We refer to the level where the proportion of consumers seeking (and willing to “trade” for) an improvement in reliability is equal to the proportion of consumers who, given the choice would “trade” for a lower standard of reliability, as the “threshold” level.  While the concept of “thresholds” is not new, the use of strategic market research to inform the setting of these “thresholds” is.



The objective would therefore appear to be to improve the reliability of those consumers whose current reliability is worse than the “threshold” level.  The critical measure is then the proportion of consumers whose current reliability is worse than the “threshold” level, rather than the average index that is currently used.

This is conceptually a simple idea, but it does imply a shift in emphasis in the way in which the reliability performance is measured and acted upon.  It essentially suggests that reliability should not be improved across the entire network; the performance of the network for consumers below the “threshold” can remain unchanged, at least until the performance of consumers currently above the “threshold” is improved to below the “threshold” level.


Development of service incentive schemes

In the late 1990s, several of our KPMG colleagues were involved in establishing the first service incentive scheme in Australia.  The service incentive scheme, known as a Performance Incentive or PI scheme, was developed for ETSA Utilities and ElectraNetSA as part of the process to reform the South Australian electricity industry.  When this scheme was established, the type of strategic market research that has recently been undertaken in South Australia was not available.  In the absence of such information, the scheme, was driven largely by traditional measures, refer Box 2.

Box 2: Current South Australian service incentive scheme for ETSA Utilities

Performance measures

n       SAIDI – System average interruption duration index, which is the sum of the duration of each sustained customer interruption[5] divided by the total number of customers;

n       SAIFI - System average interruption frequency index, which is the total number of sustained customer interruptions divided by the total number of customers;

n       CAIDI – Customer average interruption duration index, which is the sum of the duration of each sustained customer interruption, divided by the total number of sustained customer interruptions;

n       Time to restore supply to not less than 80% of interrupted customers; and

n       Average operating cost per customer.

Financial incentive

For each of these five measures, ETSA Utilities can earn up to 3 points and be deducted no more than 2 points depending on their performance relative to the base line target.  The scheme attributed a value of $300,000 for each point of performance for the 2000-01 regulatory year[6].  This implies a maximum reward of $900,000 and maximum penalty of $600,00 for each performance measure, and a total maximum reward of $4.5 million and a total maximum penalty of $3 million.

Structure

The service scheme specifies a base line target for each performance measure.  There is a deadband around the base line target.  The scheme is symmetric with positive points for performance above the base line targets and negative points for performance below the target.

The PI Scheme is subject to review as part of the 2005 Electricity Distribution Price Review.  The need for conducting strategic market intelligence to obtain definitive information on how consumers value changes in electricity service standards was identified during the initial development of the scheme.  Following completion of the consumer research, the Essential Services Commission of South Australia is now working through how this research will be applied to the service incentive scheme that will be effective during the period 2005 - 2010. 

As a result of our involvement in the process to date, we have some preliminary thoughts as to how the strategic market research may be applied to the service incentive scheme.

As discussed previously, a large proportion of consumers are satisfied with their current level of service – and, consequently, are not willing to pay for improvements to their current level of service.  There should therefore be no incentive to the distribution business to improve the service to these customers.  The focus should be on those customers who are currently receiving “poor” power.  We believe, therefore, that there should be a shift away from “average” measures to measures that focus on the worst served customers[7].  The standard industry measures of SAIFI, SAIDI, and CAIDI are average measures.  We would advocate that these be replaced with measures such as:

n       Percentage of customers who experience less than x interruptions per year;

n       Percentage of customers who experience less than y minutes of interruptions per annum; and

n       Percentage of customers who experience a longest interruption of less than z minutes.

The values of x, y and z can be set based on a combination of current performance, the cost of providing service and the results of the strategic market research.  The base line targets for the percentage of customers would necessarily be set based on historical performance.

The benefits from such a scheme will accrue to the distribution business, which will then pass on those benefits to all customers.  We also believe that the worst served customers should be compensated directly, in the form of a Guarantee Service Level (GSL) Payment, where the frequency and/or duration of interruptions exceeds a certain (higher) value.  The Regulator would be expected to use its discretion to determine the approximate proportion of customers to be so compensated.  The magnitude of the GSL payments would also be informed by the strategic market research that has been undertaken.

The results of the strategic market intelligence appear to indicate that consumers value the performance of the call centre when they need to ring about an interruption.  We believe that the remaining performance measures should measure the performance of the call centre.  Examples of the type of measures that could be incorporated are:

n       Percentage of calls to the distribution call centre answered by an operator within 30 seconds; and

n       Percentage of calls abandoned.

We continue to support the structure of the service incentive scheme as it was originally developed.  It is simple to administer, provides minimal opportunity for “gaming” and therefore is not a significant impost on either the regulator or the business.

Benefits associated with interval meters

Many policy decision makers are currently analysing the vexed question of whether interval meters should or should not be rolled out to all consumers.  It is our expectation that the significant benefits that may arise from the use of interval meters, will only occur with a paradigm shift in behaviour.  However to ensure that the electricity system continues to be sustainable in the longer term, then arguably such a paradigm shift is required.  The current lack of confidence in the benefits associated with interval meters is due to the assumptions underpinning the benefits which (we think) are generally uninformed.  Strategic market research has never been undertaken in this area to inform and challenge these assumptions.

Demab` d``2 `cap ap `hia`b paai dd add$b(`& #i2p$,[1]pah.4 ` $[1] ` b ap`%@s,,$iEhd"4r`ah h Be0@L)E c`xhd%$ @e!alba 0 ` ED"  It is also not current in terms of the appliances installed and is not relevant to Australian climatic conditions.

The increased uptake of air conditioning has created a needle peak of demand which, in turn, drives the need to augment the network and install peaking generation plant.  The imperative would therefore appear to be to reduce the demand for appliances that create the needle peak demand, rather than to reduce the base load.  This requires a sound understanding of the demand elasticity around times of peak demand.

The assumption is often made that consumers will not bear the risk of cost reflective pricing for electricity during these periods of peak demand.  As a result, the price signals are significantly blunted.  As one electricity network manager observed – “How can I convince my wife to turn off the air conditioner on a 45 degree day when the price to run it all day is about the same as a can of coke – if we trebled the cost it would make no difference”.

There is an obvious need for strategic market research, based on local climatic conditions, to provide a sound understanding of:

n       The extent to which consumers will bear the risk of significantly higher electricity tariffs on days where there is a reasonable probability that there will be a high demand for electricity; and

n       The extent to which consumers will respond to the significantly higher tariffs.

The objective would not be to detect changes in behaviour at the margins, such as raising the thermostat on the air conditioner[8], but to detect whether there would be a paradigm shift in behaviour, such as partaking in other activities so that the air conditioner is turned off.

Our research in a number of other areas indicates that consumers are willing to change their behaviour if they are given both a compelling reason to do so and the means to do so.  Mostly the means involve feeling that they can do something and that by doing something there is an overall benefit.  This benefit can be defined much in broader terms to consumer than just dollars.

The impact of demand management programs

KPMG’s research indicates that 60% of the population consider themselves to be “green”.  Despite this, the proportion of consumers that participate in demand management programs, beyond the use of off-peak electricity for hot water heating, is relatively small.  Similarly, the proportion of consumers that are on a “green” tariff or have installed energy efficient devices is low. 

So what will motivate consumers to respond to such environmental initiatives?  Our research has identified that consumers value information and being able to control outcomes.  “Green energy” and demand management programs need to be developed just like any other “product”.  This requires a sound knowledge of what motivates consumers, to inform the content and form of the information that may be required by consumers so that they will respond to such programs.  Strategic market research could provide such understanding.  Understanding demand drivers for different market segments would lead to efficient campaigns through targeted programs.

Our work in other markets has proven that the consumer markets are not homogeneous and that demographic factors are generally much weaker predictors of behaviour than attitudinal factors. 

While electricity is a commodity, with little differentiation in the market, the market for electricity is far from undifferentiated.  Different people use it in different ways, for different purposes with different importance in their lives.

A key factor in developing effective demand management programs would be to segment the market and then uncover which triggers are required by each of these segments to motivate them to conserve during times of needle peak demand.


Conservation is potentially one of a number of ‘mechanisms’ for driving demand management.  Australians have proven that they will respond to environmental messages if they are presented in an appropriate way.  The per capita consumption of water has reduced over the last ten years or so through the use of dual flush toilets, for example, and the current water restrictions have been effective.  The level of awareness of the level of dams would appear to be far greater than the level of awareness of the capacity of the electricity system.

Cost reflective pricing

Policy decision makers have been in a quandary over the benefits of cost reflective pricing and the impact that this may have on low income households.  It is often assumed that:

n       Low income households have fewer electrical devices and therefore have lower electricity bills. 

n       Low income households have a lower demand elasticity for electricity than high income households, and are not able to respond to price signals. 

n       Low income households do not understand their use of electricity and how they can save money if required.

n       Low income households therefore need to be protected from price signals.

Repeatedly, our research indicates a weak relationship between income and consumption. Low income households often consume more services, including electricity, than high income households and in some circumstances are more willing to pay for services than high income households.  Additionally, the Ombudsman of New South Wales recently expressed surprise at the understanding that low income households in Tasmania, with prepayment meters, have in relation to their electricity use and the cost implications of that electricity use[9].

Strategic market intelligence is therefore required to provide a sound understanding of:

n       Why low income households are often high consumers of electricity relative to their income; and

n       How the demand elasticity of low income households compares to the demand elasticity of high income households.

This may then inform policy decisions on, for example, whether to continue to provide concessions to low income households, or to provide incentives to improve the efficiency of appliances used by these low income households.

The role of strategic market research

With the increasing demand for electricity, policy decisions are required regarding the way in which this demand continues to be met.  There is an increasing recognition that resources are not unlimited, and tougher decisions are required.  In the absence of strategic market research, supply side thinking tends to limit initiatives to those that can be applied uniformly across the market.  But the market can be influenced in many ways and it is not homogeneous.  Many of the options that may be considered, such as changes in reliability standards, cost reflective pricing, and “green” initiatives, do not need to be applied uniformly across the market but to apply them non-uniformly requires a sound understanding of consumer behaviour. 

The answers to these questions will not be found in conventional market research.  Strategic market intelligence has developed to the stage that it may be used to inform these decisions.

Whilst this paper has focused on where strategic market intelligence may be used in the electricity industry, in a similar way, strategic market intelligence may also be used to inform policy decisions in areas as diverse as gas, ports, rail, roads, health and education.

Authors:

M Lourey was Director KPMG Regulation and Review Melbourne Australia

Brent Taylor

 



[1] The reduction of performance does not necessarily occur in the short-term; it may only become evident in the longer term.

[2] These methodologies are discussed further in the following reports: Electricity Tariffs and Security of Supply, SAIIR, Information Paper No. 1, June 2000 and Review of willingness to pay methodologies, Centre for International Economics, August 2001.

[3] Excluding momentary interruptions

[4] In some jurisdictions momentary interruptions are those of duration less than 30 seconds and in other jurisdictions momentary interruptions are those of duration less than 1 minute

[5] Excluding momentary interruptions

[6] This is adjusted annually by the March quarter CPI

[7] A component of the current PI scheme is the reliability of the 40 worst feeders.  However the effect of this incentive is small relative to the total scheme.

[8] Arguably on the days of needle peak demand, the temperature would be at levels where the air conditioner would not cycle on and off; the lowering of the thermostat would have little to no effect.

[9] Electricity purchases up close and pe~woomm{tmomwytawmuomeó*vepeyoenummot}{~<{t}/{poo~|/|{-m{>o~u~<o}>}osrsn~|or-jm~}/fmw~-omn}/w}w~mo*|ocdyo-{o|}l~o*