White Paper:

Contact Center Mystery

Shopping Best Practices



“You can expect what you inspect.”

No management philosophy is truer now than when this admonition was first put forward by W. Edwards Deming over 50 years ago. This management rule is particularly true with the customer experience. It is incumbent on customer experience researchers and managers to monitor the customer experience.

From call monitoring to post-call surveys, contact center managers have a variety of tools available to monitor the customer experience. Like different tools in a tool box, each tool has a specific purpose. Post-call surveys are an excellent tool to capture customer impressions, specifically how the customer felt about the experience and how they feel about their relationship with the brand overall. They measure the impression from the customer side of the brand-customer interface. Call monitoring, on the other hand, measures the customer experience from the employee side of the brand-customer interface. It is an excellent tool to monitor the customer experience in real life scenarios by listening to actual conversations between callers and agents.

Contact center mystery shopping is an additional tool for researchers and managers to monitor the customer experience from the employee side of the brand-customer interface – much like call monitoring. Mystery shopping aligns employee behaviors to the brand personality. Unlike call monitoring, in which the scenario of the call is dependent on the caller, mystery shopping allows managers to present employees with specific scenarios to test employee behaviors. Additionally, contact center mystery shopping allows managers to evaluate the influence of agent behaviors on important customer experience objectives, such as purchase intent or loyalty.

Best in class contact center mystery shopping focuses on empirical agent behaviors, testing agents with pre-determined scenarios. Best in class mystery shopping identifies the importance of agent behaviors by determining the relationship between specific agent behaviors and core customer experience objectives, such as purchase intent or loyalty.

Mystery Shopping Best Practices from Kinesis CEM, LLC


Types of Mystery Shopping

Contact center mystery shopping can be used to achieve a variety of objectives and shop a variety of scenarios. Among the types of mystery shopping are:

Mystery Shopping Best Practice: In-Bound Calls.

In-Bound Customer Experience:

In-bound mystery shops evaluate execution of sales and service behaviors during in-bound calls using predetermined scenarios.


Mystery Shopping Best Practice: Out-Bound Calls.

Out-Bound Customer Experience:

Out-bound mystery shopping is performed by seeding call lists with contact information for mystery shoppers. Agents then call the mystery shoppers. Service and sales execution and consistency are monitored, again, to both measure and motivate sales and service behaviors - but in an out-bound context.


Mystery Shopping Best Practice: Cross Channel Shops.

Cross-Channel Alignment:

For many brands the customer experience does not reside in one channel. Different channels with different management fulfill different functions. Yet customers expect the customer experience to be in alignment, regardless of the channel. Mystery shopping the customer experience across multiple channels to evaluate the cross-channel experience is an excellent tool to monitor the customer experience in a holistic and coherent manner across multiple channels.


Mystery Shopping Best Practice: Moments of Truth.

Moments of Truth:

Not all customer experiences are equal, some experiences are far more important than others. In every customer journey there are specific of “moments of truth” where customers form or change their opinion of the provider, either positively or negatively, based on their experience. Moments of truth can be quite varied. They can occur in a skilled sales presentation, when a shop owner stays open late help dad buy the perfect gift, or when a hold time is particularly long. They form a lasting impression on the customer in terms of how they perceive the brand, ultimately having a powerful effect on the customer relationship. These experiences are moments of truth. Moments of truth make or break the customer relationship. Contact center mystery shopping gives managers a unique tool to test the customer experience in these moments of truth.


Mystery Shopping Best Practice: IVR Usability.

IVR Usability:

While most mystery shopping is primarily used to evaluate employee behaviors. It is also a great tool to test the usability of Interactive Voice Response (IVR) systems. Interactive voice response systems weigh heavily on the call center customer experience. Usability contact center mystery shopping sends researchers through the IVR following different paths to test its configuration.


Mystery Shopping Best Practice: Life Cycle Shops.

Life Cycle:

As with cross-channel mystery shopping, the customer experience with a call seldom resides in a vacuum independent of the broader relationship with the brand. Life cycle shops use shoppers to interact with a brand over time through multiple channels. This yields valuable feedback about service performance throughout the customer life cycle and across multiple touch points.


Mystery Shopping Best Practice: Competitive Shops.

Competitive Shops:

Shopping competitors to benchmarks sales and service behaviors relative to their competitors.




Customer experience management is about alignment, alignment of the customer experience to both external messaging and customer expectations. Mystery shopping helps managers align agent behaviors with service standards and brand messaging.

The customer experience must align brand message, service standards, training content, agent behaviors, and incentives/rewards to customer expectations.

Align Brand Message with Customer Expectations

Brands have personality. Brand personality is often communicated to customers through external messaging. However, external messaging does not necessarily define the brand. The brand is in actuality defined by the customer experience. How customers perceive the brand in their interactions with agents defines the brand in stronger terms than external messaging. When the customer experience is in conflict with brand messaging, this conflict completely undermines the brand messaging and leaves the customer with a feeling they have been deceived.

Brand personality and external messaging supporting the brand must be aligned with customer expectations.

Align Customer Expectations with Service Standards

All brands need documented customer service standards. However, standards alone are not sufficient. Standards must be aligned with customer expectations. The problem is service standards tend to be products of management committees and tend to be management/process-centric, without necessarily reflecting customer expectations or needs. Service standards should be calibrated to customer expectations, needs and experiences.

Align Service Standards with Training Content

Training should obviously be aligned with service standards. Training managers should be included in the process of defining service standards to ensure training content will reflect standards.

Align Training Content with Agent Behaviors

Most training programs are evaluated on the participants’ ability to recall content after the fact. A better practice is to identify specific service delivery deficiencies and adjust training content appropriately. This is a perfect role for mystery shopping – identifying deficiencies in service delivery.

Align Agent Behaviors with Incentives and Rewards

Incentives for agent performance tend to be awarded on a quarterly basis. Depending on the data available, managers should consider rewards in a much more immediate cycle. For example, on a daily basis agents who perform well in a mystery shop can be given bonus points redeemable at a redemption site for rewards. As a result employees receive immediate reinforcement, and mystery shopping is positioned as a positive tool from which employees are rewarded not punished.

Mystery shopping helps align agent behaviors to brand message, service standards and customer expectations.

Define Objectives

Best in class mystery shop programs start with objectives. Defining objectives at the onset will ensure the mystery shop program starts on the right foot and produces useful results. Defining objectives is a fairly simple process. First, identify what is the primary business objective with respect to the customer experience. Ask yourself:

  • How do we want the customer to feel as a result of the customer experience?
  • What do we want the customer to do as a result of the customer experience?

Next, generate a list of specific behaviors you expect from agents during contacts with customers. This list should contain objective, empirical behaviors.

Identifying the primary business objective of the customer experience (how you want the customer to think, feel and act as a result of the experience), and listing specific behaviors expected from the agents is an important starting point. These two objective sets will form an analytical foundation for identifying the behaviors which are key drivers of the desired business outcome – identifying those behaviors which drive how the customer thinks, feels and acts as a result of the customer experience.

Ask yourself what service and sales behaviors you expect from agents. This list of behaviors will vary from industry-to-industry, and brand-to-brand. These questions may be as follows:

  • What specific service behaviors do we expect?
  • When greeting a customer, what specific behaviors do we expect from staff?
  • What specific hold/transfer procedures do we expect (for example asking to be placed on hold, informing customer of the destination of the transfer)?
  • Are there specific profiling questions we expect to be asked? – If so, what are they?
  • What closing behaviors do you expect? How do you want employees to ask for the business?
  • At the conclusion of the call, how do we want the employee to conclude the conversation or say goodbye?
  • Are there specific follow-up behaviors that we expect, such as getting contact information, or offering to call the customer?
  • What other specific behaviors do we expect?

Map Expectations to the Shop Questionnaire

Drafting a questionnaire is a natural progression after behavioral expectations have been identified. After a list of specific expected behaviors has been developed, the next step in the questionnaire design process is to map each expected behavior to a question or set of questions to be included into the mystery shop questionnaire.

Scenario Design

Similarly, scenario design is a natural off shoot of defining objectives. The objective definition process should include considering what types of customer service or sales integrations you want to measure. Some of the objectives to consider with respect to scenario include:

  • Do you want to test in-bound or out-bound calls?
  • What departments do you want to evaluate?
  • Are there any critical customer experiences (Months of Truth) you want to evaluate?

Mystery shopping allows managers to test their service experience in a controlled manner. Do you have a concern about how your employees respond to specific customer complaints or problems? - Send in a mystery shopper with that specific problem and evaluate the response. Are you concerned about cross-sell skills? - Send in a mystery shopper with an obvious cross-sell need and evaluate how it is handled. With mystery shoppers, managers can design controlled tests to evaluate how employees react when presented with specific moments of truth.

Questionnaire Design

Best in class research design always anticipates the analysis. Researchers must know in advance the analytical objectives and have a plan to achieve those objectives before questionnaire design. This ensures the appropriate data is collected to support the analytical plan. As the old adage says, “garbage in garbage out.”

Typically, Kinēsis uses what we call Key Driver Analysis in our mystery shop programs. Key Driver Analysis first asks what are the business objectives of each customer experience? How do you want customers to think, feel and act as a result of the interaction with the agent? These business objectives may vary by the type of experience. Typically, for service scenarios, the business objective typically is customer loyalty, or intention to maintain a relationship with the brand. For sales scenarios, the business objective typically is purchase intent. Next, we insert a scientifically designed question into the questionnaire to quantify how the mystery shop experience influenced their intention to maintain a relationship with the brand or purchase intent. In analytical terms, this key business objective question becomes the dependent variable - the variable whose value depends on others, and will serve as a basis to determine the relationship between the sales and service behaviors measured in the mystery shop and the key business objective.

Again, anticipating the analysis, questionnaire design must support the analytical plan

Three Pillars of Questionnaire Design

Kinēsis uses a three pillar approach to questionnaire design to support Key Driver Analysis.

Mystery Shopping Best Practice Questionnaire Design: What.

First, we ask what specific sales and service behaviors were observed. These are the expected empirical behaviors identified in the objective definition process.


Mystery Shopping Best Practice Questionnaire Design: How.

Next, we ask the Key Driver Analysis question designed to quantify the influence of the customer experience on the key business objective of the customer experience, again (typically, loyalty or purchase intent). This is the dependent variable we will use to quantify the relationship between each of the behaviors observed and loyalty or purchase intent. As we will discuss subsequently, this yields a basis from which to make judgments about the relative importance of each behavior in terms of driving this key business objective of the customer experience.


Mystery Shopping Best Practice Questionnaire Design: Why.

Married with this “how” question is an open-ended “why” question which asks the shopper to explain in their own words why they rated their loyalty or purchase intent as they did. This open-ended response gives excellent context into the shopper’s emotional reaction to the customer experience, and is woven into the analysis plan through a process of coding or classifying these open-ended comments into common themes, and comparing the frequency of these themes in shops with positive loyalty intent compared to those with negative loyalty intent.

Most clients score mystery shops, reducing the shop to either a single or set of scores. Scoring methodologies vary. However, most scores are a proportion of points earned over point possible expressed as a percentage of points possible.

Not all behaviors are equal, some are more important to either management or customers. As a result, scoring methodologies should include a weighting mechanism to assign more points possible to behaviors deemed more important.

Mystery Shop Scoring

What is a good mystery shop score? This is a common question I often hear.

Some providers glibly throw out scores like 90% or 95%, without any consideration of the context. Context is key. Given questionnaire construction is an outgrowth of each brand’s individual objectives and expectations and it varies from client to client, the underlying behaviors measured, therefore, will vary dramatically as well. As a result, what constitutes a good score will vary dramatically from client to client. Again, context is key.

A better solution for determining what is a good score is to compare each shop to the other shops by calculating the percentile rank of the shop score.

The percentile rank of the shop’s score is the proportion of the shops with scores below it. For example, if a shop’s percentile rank is 80%, then 80% of the shops fall below it. This ranks the shops relative to each other and automatically controls for questionnaire difficulty.

Ultimately, the objective in mystery shop scoring is to achieve a distribution of scores where low scores are skewed to the left. Tight distributions of mystery shop scores provide little opportunity to identify areas for improvement, as all the shops are very similar to each other. Distributions skewed to the low end make it easier to identify low performing shops and areas for improvement. If your mystery shop questionnaire produces a tight distribution you should consider redesigning the questionnaire or scoring system.

Sample Size & Statistical Reliability

Decisions regarding the number of shops are primarily driven by the size of the contact center, the budgetary resources available and the level of statistical reliability required.

Large Contact Centers

For large contact centers the best measure of statistical reliability are statistical calculations of margin of error. The following curve displays the margin of error for sample sizes of shops from 30 to 800.

Margin of Error cruve sample size 300 - 800.

At 30 shops the margin of error is +/- 17.9%, while at 800 shops the margin of error is +/- 3.6% (both at a 95% confidence level). This means the results have a 95% probability of coming within 17.9% and 3.6%, respectively, of the results had a census of all calls been taken.

As this curve illustrates the margin of error starts to flatten out at about 200 shops (+/- 7.1%). At 400 shops (+/- 4.9%) there is little to be gained in terms of statistical reliability by adding more shops.

Small Contact Centers

For small contact centers, the most appropriate measure of reliability is maximum possible shop distortion (MPSD) at the agent level.

Given that shops are snapshots of specific moments in time, it is possible for unique events to influence the outcome of any one shop - that the experience observed by the mystery shopper is not representative of what normally happens.

Maximum possible shop distortion is the maximum influence any unique event can have on a set of shops. With one shop to a given agent, we only have one data point, we do not know if the experience is typical or not, so the MPSD is 100%. It is possible the experience is not representative of what is typical. With two shops, the MPSD is 50%. If there are discrepancies within the shops, we do not know which shop is typical and which shop is the outlier. With three shops, we now have potentially two shops to point to the outlier (MPSD 33%). The MPSD continues to decline with each additional shop.

Mystery ShoppingBest Practice: Maximum Possible Shop Distortion.

As this graph illustrates, maximum possible shop distortion begins to flatten out relative to the incremental program cost as we approach 3 to 4 shops per agent.

Recording of Shops

Contact center shops offer the opportunity to record the calls for further review and analysis. The technology is relatively simple. However, the legal issues are not. The legality of call recording varies from state to state. While federal law allows for the recording of calls if one party gives consent, at the state level it is different. In some states, all parties are required to consent to the recording. If recording of shops is part of the plan, Kinēsis recommends that all agents sign a consent form to recording mystery shops prior to shopping.

Program Launch

When call center mystery shopping programs fail it is typically because the frontline employees (the agents and their supervisors) are not bought in to the program. Proper introduction to the program is essential. Without it, there may be internal resistance to the program. Make sure agents are fully informed and bought into the program before shopping begins. Pre-launch communication should include: the specific behaviors expected from agents (including giving them a copy of the questionnaire), training on how to read mystery shop reports, use the information effectively and how to set goals for improvement.

It is a best practice in mystery shopping to provide an adequate amount of internal administration. Anticipate the amount of administration required. Select a strong administrator to keep the contact center focused on the objectives of the program.

Part of the mystery shop process is disputed shops. Again, mystery shops are just a snap shot in time. As a result, extenuating circumstances may exist that need to be fairly addressed. Also, mystery shoppers are not perfect. If questions about the quality of a shopper’s work product exist, they need to be addressed in both a fair and firm process. Fairness is an absolute requirement for employee buy-in and morale. Firmness is required to keep the number of shop disputes in check, and cut down on frivolous score disputes.

Call to Action Analysis

Research without a call to action may be interesting, but not particularly useful. As discussed previously in study design, best in class mystery shop programs build clear call to action elements into study design to ensure that the research is useful.

Kinēsis uses what we refer to as Key Driver Analysis. Key Driver Analysis identifies an overall objective for the customer experience in terms of customer feelings or actions as a result of the experience. It then links behaviors observed in the program to this overall objective to identify which behaviors are key drivers of the business objective. For example, let’s assume the overall business objective of the customer experience is to increase customer loyalty as a result of the call. Key Driver Analysis determines the relationship of the various agent behaviors measured in the mystery shop to determine which behaviors are key drivers of loyalty.

Mystery ShoppingBest Practice: Key Driver Analysis.

This provides a basis from which to make judgments on the relative importance of each behavior in terms of driving customer loyalty.

Again, anticipating the analysis during questionnaire design is a best practice. Earlier we described three main design pillars: what, how and why. Here is how these design elements work in Key Driver Analysis:

Mystery Shopping Best Practice Analysis: How. Again, assuming the objective of the customer experience is to increase customer loyalty, shoppers are asked if they had been an actual customer, how the experience would influence their intent to maintain a relationship with the brand (loyalty intent). Comparing positive and negative loyalty intent identifies how the behavioral performance of agents varies in shops with high loyalty intent compared to those with low loyalty intent. The result of this analysis is a ranking of agent behaviors in terms of their importance to driving customer loyalty.


Mystery Shopping Best Practice Analysis: Why. Paired with the loyalty intent rating is an open-ended question asking why the shopper rated their loyalty intent as they did. The responses to this open-ended question are then classified or grouped by theme, giving a frequency of themes contained in the shopper comments. Comparing themes in shops with high loyalty intent to those with low loyalty intent produces a highly qualitative picture of specifically what drives customer loyalty.


Mystery Shopping Best Practice Analysis: What. Finally, agent behaviors with the highest potential for return on investment in terms of driving customer loyalty are identified using gap analysis. Gap analysis is a quadrant-based analysis using two dimensions: 1) the importance of each behavior (defined by the strength of its relationship to loyalty) and 2) the performance of each behavior (which is the frequency with which it is observed in the mystery shops). Graphing each behavior in this quadrant chart, like the one to the left, gives managers a means of identifying agent behaviors with the highest potential for return on investment in terms of driving customer loyalty.


Mystery ShoppingBest Practice: Gap Analysis.


Gap analysis gives managers direction in terms of prioritizing investments in training, coaching, incentives and rewards - prioritizing investments in behaviors with the highest potential for return on investment (behaviors with high importance and low performance).

Taking Action

Best in class mystery shop programs integrate mystery shop results in with other customer experience metrics as part of a balanced score card and broader incentive plan. The specific nature of the incentive plan and balanced score card should depend on the company culture. Balanced score cards typically combine financial, internal business process (such as handle time), employee satisfaction, innovation and learning along with customer experience metrics (mystery shopping and post-call surveys). Within these five general measurements, Kinēsis recommends managers select the specific metrics (such as handle time, unit profit and loss statement, mystery shop scores, and customer satisfaction), which will best measure performance relative to company goals.

It is a best practice in mystery shopping to identify agents who need coaching. Event-triggered reports identify agents performing below standards and push these reports to management. Consider the following example, assume cross-selling and up-selling are important to the brand. A coaching report can be programmed to identify agents with low performance in these key behaviors. Managers consult the coaching report and identify employee who would benefit from coaching.

Plan for Change

All processes require periodic adjustment – mystery shopping is no different. Scores flatten out, or cluster together, and as a result the value of the program diminishes as a tool. Periodic reviews of the program should be conducted to evaluate both the questionnaire and scenarios used to ensure the program remains relevant and useful.

As part of this review service standards should be realigned to customer expectations through a feedback loop between customer surveys and mystery shopping. Customer surveys define expectations while mystery shopping measures service standards against those expectations. This feedback loop will ensure that the mystery shop program is constantly aligned with customer expectations.

Cross-Channel Alignment

Different channels have different management and different roles within the organization, and as a result experience a silo effect where the customer experience evolves independently - often in different directions. Customers, however, see the brand as a whole and expect a consistent customer experience across channels. Mystery shopping is an excellent tool to test for cross-channel consistency and alignment.

Cross-channel mystery shopping starts with a definition of the brand and the customer experience in terms of attributes which should be common across all channels. Once the common customer experience has been defined, as set of specific service behaviors for each channel can be defined appropriate to each channel.

For example, assume a brand defines “establishing trust” as a brand attribute, and identifies six in-person and five contact center behaviors that establish trust:



Contact Center

Maintain eye contact
Speak clearly Speak clearly
Maintain smile Sound as if they were smiling through the phone
Thank for business Thank for business
Ask “What else may we assist you with today?” Ask “What else may we assist you with today?”
Encourage future business Encourage future business


Note this behavioral map assigns behaviors based on their appropriateness to each channel. So, for example, while the in-personal channel may be expected to maintain eye contact, obviously that would not apply for the contact center. Or the in-person channel may be expected to maintain a smile, while for the contact center this behavior may be modified for the phone channel to sounding as if they are smiling through the phone.

A cross-channel mystery shop program allows managers to compare performance across channels in common behaviors.

Provider Selection

Most mystery shop companies seek to differentiate themselves with process centered differentiation. They claim they have the best shoppers, process, etc., in reality mystery shop data collection is for the most part a commodity. All mystery shop providers who have been in business for any period of time can deliver on the process of mystery shopping. For the most part, all mystery shop providers use the same pool of shoppers, and use very similar technology to collect the data.

The real source of differentiation is the ability of the provider to be a partner in your process improvement, and provide an analytical framework for identifying specific actions improve agent performance and the customer experience.

When researching mystery shop providers carefully identify the providers that have the knowledge, analytical framework, and commitment to help build a truly valuable program – a program which will identify and motivate the behaviors that mater – behaviors that drive loyalty or whatever your customer experience objective is.


Contact center mystery shopping is an excellent tool to monitor the customer experience from the employee side of the brand-customer interface. It aligns service behaviors with the brand personality, and provides an opportunity to test agent behaviors in specific scenarios.

By linking agent behaviors observed in the mystery shop to a overall objective of the customer experience such as loyalty or purchase intent, contact center mystery shopping provides a unique opportunity to gauge the importance of each behavior in terms of driving customer loyalty or purchase intent. Overlaying the importance of each behavior to its performance proves to be an excellent road map for identifying the behaviors which have the most potential for yielding return on investment (ROI) in terms of driving purchase intent or loyalty.




Eric Larse is co-founder of Seattle-based Kinesis, which helps companies plan and execute their customer experience strategies. Mr. Larse can be reached at elarse@kinesis-cem.com.


Mystery shopping programs, when administered in accordance with certain mystery shopping best practices, not only test for the presence of service behaviors, but identify which sales and service behaviors matter most. These behaviors – the ones that matter most - are those which drive either purchase intent or customer loyalty.