Analyzing and Reporting on Respondent Scores Surveys Research

Market researchers agree it's important to communicate survey results to audiences with clarity. It's never wasted effort to explain in layman’s language how the survey results were analyzed and what the reporting conventions mean. This is particularly true when survey results are reported as statistics; the analysis and reporting of survey results deserves as much care as survey construction.

Infographics are a format used for reporting survey data that can make an enormous difference in how accessible the information is to various audiences. When used correctly, such visuals can provide a readily interpretable format for reporting survey outcomes. Infographics are intended to convey complex statistical information in a visually compelling way, with enhanced appeal for various audiences. The work of Dr. Edward Tufte is a prime example of infographic art.

Using Top-Box Reporting to Simplify Survey Findings

The top-box scores indicate the highest rating points on a survey scale. For example, if the survey participants were asked to respond to the survey questions by using a 5-point Likert scale, each point on the scale would be associated with a descriptive phrase or term. It helps to think of the scale as being vertically arranged—like a stack of children’s alphabet blocks—with the most positive possible response on the top and the most negative response on the bottom. The top-box is typically assigned the number “5” by the market researchers and is the most positive of the responses, and “4” is the second most positive of the possible responses. If the survey participant marks either one of these responses, they have given a so-called top-box response.

Most people look for simple patterns in the data. Constructing an executive summary that reports top-box scores will facilitate this natural human tendency. If an executive summary is provided to the market research audience, reporting on the top-box, cumulative frequency of survey responses can be attention-getting without being misleading. For example, if 82% of the responses on a survey question item were marked either “5” (for extremely satisfied) or “4” (for very satisfied), ​the market researchers can report that 82% of the survey respondents were very to extremely satisfied.

Certainly, the body of the summary survey report can elaborate on what the top-box figures mean and how they were calculated, but it's the top-box scores that most audience members will remember and understand.

The tendency is to focus attention on the frequency or percentage of survey responses in the top-box. You should also pay special attention to the frequency of responses in the bottom two boxes, as well. A high percentage score in the top-box range should not be allowed to eclipse bottom box scores altogether. One way to address this split analysis is to place a ceiling on the frequency or percentage of responses that are within the bottom box, just as a certain frequency or percentage in the top-box range is designated as the level to aim for either quarterly or annually.

Top-Box and Mean Customer Survey Scores

Data interpretation is made stronger when the frequency distribution and the cumulative frequency distribution are also provided. The frequency distribution shows the percent of responses for each question that correspond to the points on the rating scale. Cumulative percentages show the percent of responses up to and including all the previous points on the rating scale.

For year-to-year comparisons of surveys research conducted annually, the central tendency of the frequency distribution is one of the most valuable statistical tools. The mean, or arithmetic average, which may require weighting to be accurate, provides the best overall statistic of the typical rating given by survey respondents. In fact, it can be informative to overlay the frequency distributions of survey results from several years in order to compare the mean, median, skewness, and kurtosis of the distribution. This can be accomplished digitally by using Excel or the embedded capacity of a survey software application.

The hazard of using top-box reporting is that audiences will have limited visibility into the frequency distribution shape. Ostensibly, this is of greater interest to market researchers and other internal clients because a business development goal remains to move customers from the second highest top-box to the highest top-box, as well as moving customers from that sitting-on-the-fence position of “3” or neutral on the Likert scale. In fact, top-box score reporting and mean score reporting do not produce identical results.

A good way to demonstrate this for a customer or client is to arrange the responses to survey questions in rank order, creating two rows—one with mean scores and the other with top-box scores. The rank ordering will differ for the two methods. This difference can be particularly important when survey outcomes feed into employee performance evaluations or when the surveys are used to identify clients who are potentially at-risk of terminating their relationship with the organization.

Customer Satisfaction Is a Special Case

Surveys that measure customer satisfaction pose particular challenges to market researchers. Customer satisfaction surveys are purposefully designed to identify strengths and weaknesses in a company or organization from a consumer’s perspective. An associated challenge is that the results from customer satisfaction surveys are sometimes used to measure the performance of employees, which is not what the survey is designed to do.