Dr. Bruce Isaacson, President, MMR Strategy Group

In intellectual property matters, surveys are used to examine a variety of issues, including likelihood of confusion, secondary meaning, deceptive advertising, and consumer response to marketplace practices such as prices or packaging.  Surveys are often accepted (and even expected) as evidence in venues such as Federal Court, The Trademark Trial and Appeal Board (TTAB), and the National Advertising Division (NAD) of the Better Business Bureau. 

One requirement of surveys is that they replicate marketplace conditions – that is, the conditions under which a typical consumer might commonly encounter a product or service in the marketplace.  The issue is important to courts and others who must decide whether a survey is reliable;  as Professor McCarthy has written, “…the closer the survey methods mirror the situation in which the ordinary person would encounter the trademark, the greater the evidentiary weight of the survey results.”[1]

However, it is also widely recognized that no survey can perfectly replicate marketplace conditions, and the challenge of replicating marketplace conditions represents one of the more complex issues in designing or evaluating a survey. All surveys represent an abstraction from the marketplace, in part because surveys are designed to eliminate the effect of some factors in order to measure how consumers respond to variables of interest. The context of a survey is not the same as the actual marketplace, creating a unique environment with its own set of roles, expectations, and communication methods. As a recent article in The Trademark Reporter stated, “One critical way that surveys deviate from reality is that respondents know they are taking part in a survey rather than an actual purchase situation. Finding ways to minimize such ‘survey artifact’ effects is often the key to effective content design.”[2]

Given these considerations, the issue of marketplace conditions presents a dilemma to the survey researcher. On the one hand, a survey should provide some level of replication of marketplace conditions. On the other hand, it is not possible to match marketplace conditions exactly. This article discusses the considerations inherent in this tradeoff. We will first describe how surveys provide a unique context for communications, and then will describe how this context affects considerations regarding core survey design factors such as research mode, research method, and selection of respondents.  

The Unique Context of a Survey Interview

As we consider the challenges of replicating marketplace conditions in litigation surveys, it is first worth noting the characteristics that make the survey interview a unique environment.  One source has described survey responses as “…simply responses evoked in an artificial situation contrived by the researcher”.^[3]  Several characteristics make the responses provided by survey respondents different from everyday communications.  

One characteristic involves the mode of communications available. In a survey interview, a live interviewer or a computer-based interviewing program provides questions that the respondent answers based on the information provided in the survey. Most surveys, including those used in intellectual property matters, do not allow two-way feedback, do not allow the respondent to ask clarifying questions, and prohibit the respondent from consulting other sources during the interview. While these rules follow generally-accepted practices for conducting interviews, they differ from a typical marketplace, where a consumer who wants more information can make inquiries of the seller or can seek additional information.  

A second characteristic involves the consequences of the respondent’s actions. In the marketplace, a prospective buyer considering a purchase must decide whether to pay money in exchange for a product or service of interest, or whether to keep the money and forgo the purchase. This is true in both consumer and business contexts, even if in a business context the buyer spends money that may belong to an organization rather than to them personally. 

By contrast, in a typical intellectual property survey, the respondent does not make a purchase decision, but instead answers questions about the items that are the subject of the survey. While some surveys used in intellectual property matters inquire about purchase intent or purchase likelihood, even these questions do not typically ask respondent to spend their own money.  Instead, surveys typically ask the respondent to make judgments about what their attitudes and actions might involve given the situation described in the survey.  Simply put, the stakes are different in a survey.    

A third characteristic is the unusual nature of the task facing the respondent.  During the survey interview, respondents face a series of questions for which they are asked to provide answers.  No matter what instructions are provided, some survey respondents may view survey questions as a type of test in which some answers are more correct or acceptable to the interviewer. Some respondents may try to use clues from the survey interview to identify the correct or most acceptable answers. In the best case scenario, where such behavior is randomly directed and dispersed across the survey, the survey results will contain more “noise” and have lower reliability. In the worst case scenario, where the behavior systematically varies in direction or across survey stimuli or questions, the survey results can be biased and thus have lower validity. 

Considered together, these and other differences mean that the survey provides an environment in which the respondent cannot ask clarifying questions, is not in an actual purchase situation, has limited information upon which to make decisions, and may try to infer clues from the survey interview. However, these are challenges that can and should be properly managed by the survey expert.  When these and other issues are addressed, the survey has a number of inherent advantages that make it a highly effective way to produce probative evidence on issues critical to intellectual property matters.   

I will next describe how considerations relating to marketplace conditions affect the decisions researchers make regarding specific elements of research design, using three specific examples:  research mode, research stimuli, and respondent selection. 

Marketplace Conditions and Research Design

Research Mode

Research mode refers to the manner in which data is gathered during a survey. Today, most litigation research is conducted either in-person or online. In-person research is often conducted by live interviewers at dedicated facilities located in or near shopping malls, while online research is typically conducted using self-administered surveys conducted over the Internet.

When considering marketplace conditions, researchers often (but not always) match the mode of data gathering to the manner in which the product is shopped for and purchased. If a product is shopped for, inspected, or purchased in person, then the research will often be conducted in person. Similarly, if a product is shopped for or purchased online, then the research is more likely to be conducted online.

While online surveys are increasingly used in litigation research,[4] litigation surveys have lagged commercial (non-litigation) surveys in the prevalence of using online surveys. Today, the majority of commercial research is conducted online due to a variety of factors, including the difficulty of reaching respondents by other modes such as telephone or in person interviews, and the availability of visuals and online research tools and methods that are difficult to replicate in alternative modes.  Commercial research organizations are also more comfortable accepting the results of online research, even for products that are sold in person, because their standards and processes for evaluating and accepting research are different than litigation environments.

As a result, many surveys that would be conducted online in commercial contexts are conducted in person in litigation contexts.  Many survey experts start their consideration of marketplace conditions by making sure that the research mode matches the manner in which the product is purchased. 

Choice of Research Design

The consideration of marketplace conditions also affects the choice of research design. Research design refers to, among other considerations, the overall flow of questions involved in a survey and the means in which product is shown to respondents. 

For example, likelihood of confusion studies are commonly conducted using either a lineup format or an Ever-Ready format.[5]   In the Ever-Ready format, the respondent typically sees only the junior product, and is asked (among other things) who makes the product; responses that reference the senior product are taken to reflect confusion. By contrast, in a lineup study, the respondent sees a lineup of products. For example, one common implementation of a lineup study is that the respondent sees either the junior or senior product first, then sees a lineup consisting of the other product plus additional products from the category. 

Marketplace conditions are important considerations in the selection between these two research designs. For example, the Ever-Ready format is typically used only with matters involving a senior brand that is already known by respondents, while a lineup is used only “where the stimuli proximately tested in the format appear, in fact, proximately in the marketplace.”[6]  When brands do not appear in immediate physical proximity in the marketplace, and are not sufficiently well-known as to support a Ever-Ready design, researchers (under certain conditions) may choose to use a variant of a lineup study, such as a design in which products are presented sequentially rather than side-by-side in a lineup. On occasion, the same matter may involve both a lineup study and a sequential study.[7]

The decision between Ever-Ready and lineup formats provides one example of the host of considerations relating to marketplace conditions in research design. These considerations help courts, experts, and attorneys evaluate whether a survey “…sufficiently simulated the actual marketplace conditions in which consumers encounters the parties’ products.”[8]  The design of litigation surveys commonly involves a series of considerations about research design and presentation of stimuli, all of which are typically intended to create a survey that replicates marketplace conditions.  

Selection of Respondents

Considerations relating to marketplace conditions also arise in selecting and qualifying respondents to ensure that the proper universe is selected for the survey.

For example, the researcher must decide who should be included in a consumer sample. For example, Professor McCarthy, in discussing secondary meaning surveys, has stressed the importance of the “reasonably informed shopper” who has “that amount of basic knowledge about the product that most people would have from news and advertising,” but is not an expert.[9]  A researcher designing a survey must consider whether marketplace conditions are better reflected if the sample includes gatekeepers, such as dealers, retail buyers, or retail salespeople.

Similarly, the researcher must also decide whether marketplace conditions are better reflected if the survey sample includes past customers, prospective customers, or both. The answers depend in part on the marketplace and in part on the nature of the matter. For example, courts have indicated that including past customers in a sample for a survey on secondary meaning renders results of little value because it is meaningless to ask customers which company they bought from, as they know who they bought from.[10]  As one source stated, “One’s own customers do not a fair sample make.”[11]  In other matters, asking different types of questions, past customers may represent an entirely appropriate universe of respondents. For example, in a class-action matter, members of the proposed class may be the most relevant respondents if a survey is to reflect the marketplace. 

Conclusions
The challenge of replicating marketplace conditions is an important one in survey design, and is too complex to completely address in a brief article. The examples provided in this article do not necessarily reflect all possible considerations for a particular case. Rather, it is our hope that these examples illustrate in a useful way the inherent trade-offs between the objective of replicating marketplace conditions and preserving the reliability and validity of the obtained survey results.

About MMR Strategy Group

MMR Strategy Group (MMR Strategy) is a full-service market research-based consulting firm.  We help our clients grow by leveraging customer insight to develop marketing and sales strategies.  In order to support critical business decisions, we combine the data gathering capabilities of a research firm with the business analytics of a strategic consulting firm.

For more information, please contact us at 818.464.2400 or email info@mmrstrategy.com.

Authors[i]: 

• Dr. Bruce Isaacson,  President, MMR Strategy Group
• Professor Jonathan D. Hibbard, Professor of Marketing, Boston University School of Management
• Professor Scott D. Swain, Assistant Professor of Marketing, Northeastern University College of Business Administration

Surveys are increasingly common in intellectual property matters as an accepted and sometimes expected manner to measure consumer attitudes and behaviors.  At their core, many intellectual property disputes involve theories about the attitudes and behaviors of consumers, customers, or the general public; these theories are translated into concepts such as secondary meaning, likelihood of confusion, deceptive advertising, and genericness.  Surveys provide a reliable and scientific way to measure the attitudes and behaviors that form the basis for these theories and concepts.  In some matters, trying to bring a complaint without providing a survey can be viewed negatively by authorities,[2] who may look to the survey for evidence that there is indeed something about which to complain.  As a result, surveys have become commonly-submitted evidence in cases involving intellectual property issues.[3]

As an example, consider likelihood of confusion, an issue central for claims of trademark infringement and dilution.  Likelihood of confusion is measured by a variety of factors that vary by circuit.  For example, courts in the Ninth Circuit often rely on the so-called “Sleekcraft factors”[4]:

1. Strength of the mark,
2. Proximity of the goods,
3. Similarity of the marks,
4. Degree of care exercised by purchasers,    
5. Evidence of actual confusion,
6. Similarity of the party’s marketing channels,  
7. Defendant’s intent in selecting the mark, and          
8. Likelihood that the products will expand and overlap.   

Other courts may consider similar but slightly different lists of factors, including the Helene Curtis Factors (Seventh Circuit), the Polaroid Factors (Second Circuit), and the DuPont Factors (Federal Circuit).  Surveys can provide evidence relevant to these factors.  Using Sleekcraft as an example shows how surveys can measure confusion among customers of either party and help assess whether a mark calls to mind the other party’s trademarks or brands (the 5^th factor above).  Survey evidence can also measure whether consumers view the marks as similar (the 3^rd factor), the degree of care that purchasers exercise (the 4^th factor), the strength of the marks (the 1^st factor), and the proximity of the goods, i.e., whether they are seen as substitutes and/or share a common market (the 2^nd factor).  In fact, of the eight Sleekcraft factors, six address consumer attitudes and behaviors and can be measured by a survey, while only two (the defendant’s intent and the likelihood of expansion) are unlikely to be measured by a consumer survey.

Surveys measure these factors in straightforward ways.  As an example, imagine a hypothetical situation where an infringing company uses a name similar to that of a well-known senior brand.  In our example, the senior brand has alleged that unwary consumers may believe they are encountering the well-known senior brand, when in fact they are encountering the knockoff, or junior brand.   Different survey questions can measure different types of confusion.  To measure source confusion, a survey researcher may show consumers products made by the senior and/or the junior user, and ask whether the respondent believes that the two items come from the same company or from different companies.[5]  To measure confusion as to affiliation or sponsorship, a follow-up question may ask whether the respondent believes that the items are made by companies that are affiliated, connected, or associated in some manner.

Surveys proffered in intellectual property matters are held to high standards and are subject to review by a gauntlet of prepared and often-critical parties, including opposing counsel, outside rebuttal experts, and judges.  The researcher operating in this environment must adapt the standard survey process to the requirements of legal proceedings.  For example, past precedent may drive reliance upon specific wording for certain questions, even when the researcher might otherwise prefer alternative phrasing.  Even the procedural norms that guide the survey process are unique to the litigation context.  Many commercial or academic surveys follow an iterative process of survey development and refinement, including the use of preliminary qualitative research, pretests, pilot studies, and/or procedural calibrations.  By contrast, the timelines and discovery process in many legal cases may require the researcher to use a process that relies more upon precedent and provides a more certain timeline. 

There is a lot at stake in this process.  Typically, a party may submit a survey only once, and judges often have wide discretion during the process to admit or not admit a survey, and give it more or less consideration in the proceedings.  A survey assigned less consideration (known as “low weight”) by a judge may not be given a second opportunity to be updated or improved,[6] placing a heavy burden on the researcher to get the research right the first time. 

Given that surveys can be central to intellectual property matters, and given that their reliability and validity are important and subject to intense scrutiny from a variety of parties, this article discusses three questions that can help when evaluating survey design and execution.  Our experience suggests that a reliable survey will, among other criteria, successfully pass the test provided by three specific questions: 

1. Does the survey identify and sample the relevant population of consumers?
2. Does the survey use appropriate survey items?  
3. Does the survey employ appropriate controls?  

Does the survey identify and sample the relevant population of consumers?

One critical issue in intellectual property surveys is identifying and sampling the appropriate respondent population.  When conducting a survey, the researcher gathers a sample of respondents from a larger universe or population and then makes projections as to the makeup of the underlying population.  It is important that the researcher select respondents who are representative of the larger population. The alternative to sampling is to conduct a census by including everyone in the population, but this is prohibitively expensive and impractical in most contexts.  

The relevant population of consumers depends at least in part upon the issue that the survey is designed to measure. For example, the relevant population may be different for a survey measuring trademark confusion than for a survey measuring secondary meaning, and it may be still different for a survey measuring other issues.  Also, a survey measuring confusion in a trademark matter may include the customers of the senior user, the junior user, or both. 

For example, consider a hypothetical forward confusion case where a junior user makes items using a trademark that is accused of being confusingly similar to the senior user’s mark.  As a more concrete example, imagine a new watch company making “Rolvex” watches, a name similar to the well-known “Rolex” brand.  The right universe to survey depends at least in part upon one’s theory of the matter. One theory may state that consumers could buy the junior knockoff brand and think they are buying Rolex.  Another theory may posit that Rolex customers may see the knock-off brand and make negative inferences about Rolex based on the outlets where the knock-off is sold or based on the quality of the knock-off brand.  Thus, in this example, different theories of the case may lead the researcher to different decisions regarding whether to survey customers of the junior user, senior user, or both. 

Once the researcher has identified the characteristics of the relevant population to reach, they must then develop specific guidelines about how to reach that type of consumer. Is that consumer more likely to be male or female?  What age group?  What ethnic background?  Does the consumer use the Internet?  Where can we locate them?  Can we identify them by attitude or lifestyle?  To identify these and other attributes of the relevant customer base, the survey researcher may need to examine records from the litigants such as customer orders, marketing and advertising plans, strategic planning documents, advertising, and analysts’ reports.  These documents, some of which might be available through normal channels of discovery, can be very helpful in establishing whether the expert has surveyed the appropriate population. 

Does the survey use survey items that are non-leading and appropriately crafted? 

Every survey relies on what researchers call “survey items” – the questions that respondents are asked to answer. Developing good survey items requires experience and good judgment, and relies upon a number of core principles relating to considerations such as the phrasing of items, the order in which items are presented, the scales respondents use to provide answers and other elements of research design. 

Below we provide a few of the many guidelines that underlie the construction of appropriate survey items. Specifically, survey items should be:

1. Non-biased and non-leading: Survey items should clearly inquire about the variable of interest in a manner that is neither leading nor biased, so that the respondent does not think the researcher would like to obtain a particular answer.
2. Clear and unambiguous:  Survey items should use clear and direct language, so that survey respondents know what the question means that they are asked to answer.  The wording should be appropriate to the population.   For example, the language one would use for a survey of primary grocery shoppers might be different than the language one would use for a survey of physicians.
3. Directed at a single construct: A compound question is one that asks about two concepts. Responses to compound questions can be very difficult to interpret, because it can be unclear which concept the respondent has answered.
4. Not subject to guessing:  To avoid bias, good survey items reduce any possible temptation or rewards from guessing.  For example, as much as possible, the respondent should not be able to guess what answers will qualify them for the study.  Also, respondents should be told that if they don’t know an answer to a question, “don’t know” is an acceptable response and that they should not guess.  Also, some type of “don’t know” option should be provided to nearly all questions, except those for which a don’t know answer would be nonsense (such as respondent gender). 
5. Presented in random order:  The order of items and responses within items should be rotated or randomized. This reduces the “order bias” that would otherwise be present if, for example, items or responses presented first (or last) were more likely to be selected. 

For additional information on these and other considerations, a good resource is “The Reference Guide on Survey Research,” written by Shari Seidman Diamond, published by the Federal Judicial Center in The Reference Manual on Scientific Evidence and available at www.fjc.gov.  Some of the items in the list above are adapted from that resource, and they provide examples of guidelines to develop appropriate survey items.

Does the survey employ proper controls?

Another important matter in litigation surveys is establishing proper controls.  Consumer attitudes and behaviors are often measured relative to a control product.  For example, in a confusion survey, some percentage of consumers typically indicate that the trademarks are put out by the same company or are connected by affiliation or sponsorship, even if the products have no relationship with each other and if the product categories were as different as automobiles and ice cream.  To account for such random or background noise, researchers typically present a separate group of consumers with a control stimulus, which is a product that is similar to the test product but with the potentially-objectionable elements removed.  Thus, the control reflects the background level of confusion that exists in the marketplace.

The researcher subtracts the confusion associated with the control product from that of the test product.  In this way, the use of a control allows a survey to quantify the net or “true” level of confusion that is caused by the presence of the infringing elements. The process is analogous to using a placebo medicine or treatment in a medical trial to determine the true effect of the medicine or treatment tested. 

Sometimes, a suitable control stimulus can be located in the marketplace by using a product that is available to consumers. The researcher also has the option of using products from the marketplace modified to have as many elements in common with the test product as possible (without infringement).  Alternatively, researchers may use fictitious products specifically developed for the research.

For example, the authors recently worked on a study of possible confusion between two makers of casual apparel, where the junior user used a trademark that was accused of being too similar to the trademark of the senior user.  For this study, a control product was used that was similar to the potential infringer’s products in all respects, but logos were altered on labels and hangtags to eliminate the potentially infringing elements.  In that study, as in most studies, controls use the elements of the test product in their natural context – the same color scheme, the same font, and a name for the control product that conveyed the essential function and meaning of the test product name.[7]  

A control is not always required. Specifically, controls are not necessary in cases where one is measuring whether a particular attribute exists, or where the existence of that attribute is irrelevant to any baseline conditions.  For example, secondary meaning either exists or does not, and many studies measuring secondary meaning do not use controls except in particular cases where calibration of the results relative to a known name would be useful. 

Conclusion

These three questions, relating to selecting the appropriate population, crafting survey items, and using a proper control, are important and challenging matters in intellectual property surveys.  We find that sufficient consideration of these items before any survey is launched can save effort and budget later in the process, while insufficient consideration to these issues up front can leave the expert subject to difficult depositions and skeptical courts. 

About MMR Strategy Group

MMR Strategy Group (MMR Strategy) is a full-service market research-based consulting firm.  We help our clients grow by leveraging customer insight to develop marketing and sales strategies.  In order to support critical business decisions, we combine the data gathering capabilities of a research firm with the business analytics of a strategic consulting firm.

For more information, please contact us at 818.464.2400 or email info@mmrstrategy.com.

Dr. Bruce Isaacson, President of MMR Strategy Group

“What are the standards for a claim substantiation survey?”  Clients often ask this question when they are contemplating a claim substantiation survey in general.  The question comes up during discussions about advertising claim substantiation or about claims used in packaging, on point of sale materials, or during sales presentations.  I am also asked this question when discussing litigation matters involving issues such as false or deceptive advertising. 

Often, the question is asked in search of a clear standard that will describe exactly how to conduct a claim substantiation survey or a litigation survey.  This article presents some of the sources I consult more commonly for matters involving claim substantiation, substantiation surveys, and litigation surveys on topics such as deceptive advertising.[1]  After listing the sources, I will provide some general thoughts regarding the standards for designing and executing claim substantiation surveys and litigation surveys.   

The Federal Trade Commission

The FTC has issued a number of policy statements relating to claim substantiation. For example, see the “FTC Policy Statement Regarding Advertising Substantiation.” This statement, among other things, states that advertisers must “…have a reasonable basis for advertising claims before they are disseminated.”  The “reasonable basis standard” is well-known and depends on a variety of context specific factors, such as “the type of claim, the product, the consequences of a false claim, the benefits of a truthful claim, the cost of developing substantiation for the claim, and the amount of substantiation experts in the field believe is reasonable.”  The FTC has also published other materials relevant for claim substantiation.  For example, see “Substantiation:  The Science of Compliance” or “Advertising Substantiation Principles.”

The Federal Judicial Center

The Federal Judicial Center has published the Reference Manual on Scientific Evidence. The manual includes the “Reference Guide on Survey Research.”  Written by Professor Shari Seidman Diamond, the Reference Guide outlines a series of criteria for evaluating survey research that is used in court.  The document raises a series of questions to consider relating to topics such as the purpose and design of the survey, population definition and sampling, and survey questions and structure.  The document provides helpful guidance on survey-specific issues, including broad questions such as the purpose of the survey, and detailed issues of survey execution, such as validation or the rotation of question and response choices.

McCarthy on Trademarks

Published by Westlaw, McCarthy on Trademarks and Unfair Competition analyzes trademark law, including issues relating to topics such as deceptive advertising and infringement.  This is a multi-volume sourcebook, frequently cited and well-respected.  Sections present citations of articles and precedents on specific topics, including how the surveys were conducted and how the results were interpreted or accepted by courts. 

American Society for Testing and Measurement 

The ASTM publishes the “Standard Guide for Sensory Claim Substantiation,” which is available for purchase. This is a technical guide that describes specific practices to substantiate sensory or perceptual claims.  The guide addresses both comparative and non-comparative claims, and is particularly helpful in describing the types of statistical tests that should be used in claims.   The guide is written with claim substantiation in mind.  As the ASTM’s description states, “The objective of this guide is to disseminate good sensory and consumer testing practices. Validation of claims should be made more defendable if the essence of this guide is followed.” 

The Trademark Reporter

The Trademark Reporter is a peer-reviewed journal that frequently publishes articles relevant to substantiation surveys and litigation surveys.  Some of the articles describe when certain kinds of surveys should be used.  TMJ is published by INTA, the International Trademark Association.  As INTA states on its website, The Trademark Reporter is “…the world’s only scholarly journal dedicated solely to trademark law and related topics.” 

There are many other sources that could be listed here, as I have presented only a small sample of the many sources relevant to substantiation surveys and litigation surveys.  There are also a variety of sources maintained or published by private sources; for example, many large marketing companies have policies that govern the use of words such as “free”, “most”, “better”, or “leading”. 

As I look across these sources and others that I often consult, I’d offer these general thoughts: 

1. Claim substantiation surveys:  Claim substantiation surveys are different from the kind of marketing surveys that are used for purposes such as tracking brand progress or measuring brand image.  Substantiation surveys are typically more tightly focused on a smaller number of key issues, and are designed with certain elements specifically needed for claim substantiation.  
2. Research standards:  None of these sources tells you exactly how many surveys to conduct, but they do present standards and guidelines that may be relevant in setting sample size, selecting respondents, using controls, or other issues.  Often, these guidelines must be interpreted by a qualified expert or experienced attorney. 
3. Statistical testing:  One thing all sourcebooks agree upon is that statistical testing for claim substantiation should be done at the 95% level of confidence.

About MMR Strategy Group

MMR Strategy Group (MMR Strategy) is a full-service market research-based consulting firm.  We help our clients grow by leveraging customer insight to develop marketing and sales strategies.  In order to support critical business decisions, we combine the data gathering capabilities of a research firm with the business analytics of a strategic consulting firm.

For more information, please contact us at 818.464.2400 or email info@mmrstrategy.com.

Dr. Bruce Isaacson, President of MMR Strategy Group

For many marketers and consumer insights departments, a claim substantiation study can be new ground and a complicated topic.  Marketing researchers and marketers tend to be more familiar with typical research projects, such as tracking studies, brand image studies, and concept tests.  However, the survey that is used to substantiate a statement on an ad, a package, a point of sale display, or a sales presentation, provides a unique challenge because these surveys are not conducted on a regular basis. 

There are typically two ways to gather data to substantiate claims, either through scientific measurement or claims substantiation surveys. This article focuses on data which is gathered by means of a claim substantiation survey, which in general, gathers data about consumer perceptions, such as which product tastes best, or which shampoo leaves your hair feeling the silkiest.  Although a claim substantiation survey covers the familiar ground of consumer surveys, they are different from traditional marketing research and present their own unique challenges. 

Anyone contemplating a claim substantiation study should expect that it will likely differ substantially from traditional marketing research.  These differences are summarized in the table and described below.

Differences between Marketing Research and Claim Substantiation

1. The potential stakeholders are different.  The stakeholders for a marketing research study often include an internal client, such as a brand manager, marketer, or strategic planner, as well as the marketing research group.  The stakeholders for a claim substantiation study include these same internal clients, who need to know what positioning the product can support, as well as in-house counsel and trademark administrators.  External counsel may often be involved as well. 

2. The intended audience is different.  The intended audience for a claim substantiation study includes the internal stakeholders described above, as well as an external audience. This external audience may include potential challengers to a claim or authorities who might be in a position to evaluate whether the claim is substantiated, such as Federal Court or the National Advertising Division of the Better Business Bureau.  Often, companies undertaking a claim substantiation study aren’t sure exactly who will be the intended audience because they don’t know which specific competitor may challenge a claim, and what specific context or venue will be used for the challenge. 

3. The data and analysis are different.  The data gathered in a traditional marketing research report may be quantitative or qualitative.  It may be analyzed by a wide variety of techniques, from summarizing themes in a focus group to advanced analytics for a large-scale quantitative study.  By contrast, the data for a claim substantiation study is always quantitative, never qualitative, and always analyzed with specific statistical techniques intended for substantiation tests.   

4. The data gathering is different.  The questionnaires used in traditional marketing research studies can be long and include a variety of question types for thorough analysis, such as buying behavior, product usage, and brand ratings.  Compared with traditional marketing research, a claim substantiation survey is much shorter, more focused, and requires elements of execution not present in typical marketing research studies.  For example, the questionnaire in a claim substantiation study is more likely to require a third-party validation, and likely has more extensive rotation of questions and responses to avoid order bias.  This reason is why the claim substantiation study needs to stand alone, and not be combined with another research effort, such as a tracking study. 

5. The report is different.  The report for a marketing research study is often displayed in presentation format, providing results in an easily-scanned layout. The report for a claim substantiation study is written as an expert report, and provides much more detail about the research methods. 

6.  The key output is different.  Key elements of the output from many marketing research reports are the conclusions and recommendations, which describe the main findings implied by the data, and also the implications for the brand or product that is the subject of the research.  The researcher’s role focuses on the task of identifying conclusions and making recommendations.  In contrast, the key element of output from a claim substantiation study is the substantiation (or lack of substantiation) for the various claims that were tested.  The researcher’s role focuses on applying statistics and experienced judgment to evaluate the degree of support for the claims tested, and to consider alternative claims that might fit the data. 

This article lists only a few of the many differences between traditional marketing research and a claim substantiation study.  There are others, but hopefully the process of claim substantiation is now less foreign and more familiar.

About MMR Strategy Group

MMR Strategy Group (MMR Strategy) is a full-service market research-based consulting firm.  We help our clients grow by leveraging customer insight to develop marketing and sales strategies.  In order to support critical business decisions, we combine the data gathering capabilities of a research firm with the business analytics of a strategic consulting firm.

For more information, please contact us at 818.464.2400 or email info@mmrstrategy.com.

For such a complex, high profile, and expensive activity, market segmentation has a success rate that leaves much to be desired.  A company may undertake a market segmentation project with enthusiasm, only to find that the report ends up as a thick volume sitting on a shelf.   Other times, the segmentation is undertaken by a function or department that is particularly facile with research, such as the consumer insights group, but is never adopted by the rest of the organization.

Here are five common pitfalls of market segmentation, and how to avoid them.

1. The segmentation was not developed with the end goals in mind. 

   Market segmentation is the search for ways to divide a market into parts, called segments.  It is an exercise that focuses upon choices, such as selecting which segments are worth focusing on and which segments are unlikely to be productive.  For the segments chosen as the focus of the company’s efforts and resources, a proper segmentation also includes recommendations about how to address those segments, including important elements such as which positioning to adopt, how to go to market, and how to deploy vital functions such as sales and advertising.

    

   There is no one correct answer to any segmentation, but the best answer will be appropriate to the specific clients who intend to use the results.  For example, the segmentation that is best for use in selecting advertising may be different than that which will be used by the salesforce.

    

   Some clients prefer a simpler solution that is perhaps not as sophisticated, while others may seek something more sophisticated, even if it is more complex.   If the segmentation is to be used by relatively independent arms of a company, such as franchisees, agents, or distributors, that will also affect the “right” solution to the segmentation.

    

   Before undertaking a market segmentation study, an organization must think through its goals, addressing issues such as who will use the segmentation and for what activities.

 

2. The segmentation uses the wrong variables.

   There are a variety of ways to segment markets and each depends on the selection of the right variables.

    

   Demographic segmentation uses demographics to divide a market into segments.  For example, a company may target upper-income consumers, or moms with children, consumers who live in urban markets, or dog owners.  All of these segments are based on demographic variables.

    

   A second type of segmentation uses behavioral variables. For example, a company may target consumers who eat fast food at least weekly, or travel at least once a week on business, or who use financial advisors.  These types of variables create behavioral market segmentation because they rely on behavioral variables.

    

   A third type of segmentation, attitudinal segmentation, is based on consumer attitudes.  This is the most complex segmentation but can be the most powerful.  Two consumers may share similar behavioral and demographic characteristics, but maintain very different attitudes towards the product in question.  For example, a provider of home insurance may find that two neighbors live side-by-side in homes of similar value, but one believes in insurance and likes to be well-covered, while the other wishes to have the minimum amount of coverage.  Similarly, one consumer may prefer to have the advice of an agent, while the other may prefer self-service or an online purchase method.

   There are trade-offs across using these different types of variables.  Demographics are the easiest to identify and to target. They can be the focus of direct mail campaigns and are easy to communicate to the salesforce, yet don’t reflect more fundamental differences between consumers.

    

   Behaviors are somewhat more difficult to target, but tend to be more powerful because they reflect what people actually do.

    

   The most powerful variables in terms of generating segments are attitudes, but they are also the most difficult to analyze and understand.

    

   The trade-offs between these three types of variables must be considered and designed at the beginning of a segmentation project to avoid choosing the wrong type of variable. The research design must also consider whether other variables should be included, such as those that are used for buying media, locating stores, sending direct mail, or other data-driven activities.

 

3. The segments aren’t meaningfully different from each other

   A third common problem with segmentation is that the segments are not meaningfully different from each other.

    

   A common saying is that segmentation should produce segments that are “homogeneous within” and “heterogeneous between”.  In this type of segmentation, all the consumers within a segment are similar to each other, but the consumers in one segment are meaningfully different from consumers in another segment.

    

   As we mentioned earlier, segmentation is about choices.  It is undertaken so that an organization can select one segment over another, choosing the opportunity with the greatest potential.   It can also help a company select methods for going to market that match the needs of different parts of the market.

    

   If the segments are too similar to each other, there is no reason to select one segment over another.  For segmentation to be productive, it must help a company either select which parts of the market merit focus, or it must help a company select the right methods to address different segments.  Ideally, it will do both.

 

4. The segmentation is not linked to profits.

   If segmentation is about choices and about the search for opportunities where resources such as sales or advertising can be applied to yield the greatest opportunity, then the segmentation must yield a way to evaluate the different opportunities.  One segment may be more difficult and more expensive to reach, or may use products that have a higher cost and lower margins.  These considerations must be taken into account as you build a research strategy that maximizes your resources.  Any segmentation should link not only to revenues but also to profits, so that the choices about which segments to address and how to address them, can be made in a manner that boosts the bottom line.

 

5. The market has shifted. 

   Market segmentation, like any other type of research, ages over time.  Consumers change, markets shift, distribution channels evolve, and products reach new generations.  As these and other dynamics in a market change, the segmentation gradually ages.  To avoid becoming obsolete, any segmentation must be updated over time to stay current with the changing marketplace.

About MMR Strategy Group

MMR Strategy Group (MMR Strategy) is a full-service market research-based consulting firm.  We help our clients grow by leveraging customer insight to develop marketing and sales strategies.  In order to support critical business decisions, we combine the data gathering capabilities of a research firm with the business analytics of a strategic consulting firm.

Dr. Bruce Isaacson, President of MMR Strategy Group

For more information, please contact us at 818.464.2400 or email info@mmrstrategy.com.

Dr. Bruce Isaacson, President of MMR Strategy Group
Debbie Lesnick, Senior Vice President & Head of Research of MMR Strategy Group

Marketers who wish to make a claim must consider the close ties between claims and the evidence that supports them.  Whether the claim is placed in an advertisement, on a package, or at the point of sale, there are some consistent rules that govern the relationship between claims and evidence.  With some additional forethought about the phrasing of a claim, marketers can greatly reduce the costs of developing the evidence to substantiate the claim. 

To understand why, consider the relationship between the claim that is made and the evidence that is required to support it.  Although the way they interpret these requirements can vary, courts or regulatory authorities often have expectations similar to these three simple rules: 

1. the evidence is in place before the claim is made,
2. the evidence directly and clearly supports or substantiates the claim,  and
3. the basis for the evidence, including the means by which it was developed, is clear to a reasonable or typical consumer. 

As an example, let’s imagine a peanut butter manufacturer that wants to substantiate a claim that its peanut butter is “preferred by kids”.  If we follow the three simple rules listed above, then we can expect that supporting evidence was gathered before the statement was put on a label or in an ad.  This evidence might be in the form of a survey asking which peanut butter the kids prefer. 

We can also expect that, since the claim is “preferred by kids”, the survey evidence was gathered directly from children, as opposed to other methods, such as asking parents what their children prefer.  (If the claim was “parents say we are preferred by kids”, then we could gather the survey data directly from parents.)

Finally, for a claim of “preferred by kids”, we could also expect that our survey evidence shows that children preferred this particular peanut butter.  Many authorities might interpret this claim to suggest that the children actually tasted the peanut butter, as opposed to simply indicating their preference for a particular brand. 

Because the evidence for a claim must closely match the claim, the requirements for evidence can vary widely according to the claims itself.  We commonly encounter examples when minor differences in the phrasing of a claim can make big differences in the likelihood, difficulty, and cost of generating evidence to support the claim.   Often, changes in the phrasing of a claim that seem relatively minor can make big differences for substantiation. 

This is particularly true when considering the difference between claims that imply that one brand is “unsurpassed” by another brand, versus those that imply superiority.  The unsurpassed claim might state, “Our product is as good as the other brand.”, while the superiority claim states, “Our product is better than the other brand.”  Unsurpassed claims are an underused opportunity, particularly when a smaller or newer brand faces competitors that are better-known, longer on the market, or higher share. 

Unsurpassed is an odd phrase that means simply that your product is as good as the other on that particular attribute.  It can be easier to show that a product is as good as another, than it is to show that the same product is better than the other.  This is particularly true when the newer, smaller-share product is as good for one or two attributes or usage occasions that resonate with customers.    

Here are three examples of situations when a product might consider an unsurpassed claim:

1. A mainstream brand compares favorably to a premium brand:  For example, a shampoo that is sold in drugstores at a mainstream price leaves hair feeling just as clean or just as manageable as a brand that is sold in expensive hair salons at three times the price.  Even if the drugstore brand is not preferred overall, it may do just as well on attributes that are important to consumers. 
2. A value brand wants to make comparisons to the market leader.  For example, a store brand of diced tomatoes may work as well in recipes as the best-known brand, or may be indistinguishable from the better-known brand for certain types of cooking.  Even if the best-known brand is preferred overall, this type of claim can highlight the occasions when the consumer might accept the value brand. 
3. A new brand wants to make comparisons to an older brand.  For example, a new type of car handles as well as expensive European brands.  If the new brand performs as well on certain qualities important to consumers, claims that reference those qualities may increase awareness or consideration of the new entry.

From a statistical standpoint, the burden of survey evidence to show that a brand is unsurpassed by another is much lower than the burden required to show that a brand is better than another.  Marketers should keep in mind that unsurpassed claims are less expensive to substantiate than superiority claims because they require fewer respondents, which lowers research costs.  They are particularly helpful if your brand is chasing a larger-share, better-known, higher-priced market leader. 

About MMR Strategy Group

MMR Strategy Group (MMR Strategy) is a full-service market research-based consulting firm.  We help our clients grow by leveraging customer insight to develop marketing and sales strategies.  In order to support critical business decisions, we combine the data gathering capabilities of a research firm with the business analytics of a strategic consulting firm.

For more information, please contact us at 818.464.2400 or email info@mmrstrategy.com.

Dr. Bruce Isaacson, President of MMR Strategy Group

“Claim substantiation” refers to an activity that is very important for marketers, and very straightforward in concept:  providing evidence that the statements made or implied about products or services are, in fact, true. 

Marketers and sales activities routinely make all sorts of claims.  As consumers, we encounter claims every day on items such as product packages, product labels, advertisements, sales brochures, public relations materials, billboards, point of sale materials, and other locations.

Across these many locations, different types of claims require different types of data for substantiation.  Some claims can be substantiated by engineering or scientific data, such as the number of seconds it takes a car to accelerate from zero to sixty, the hard drive capacity of a new computer, or the square footage of an apartment.  Such claims are measured relatively objectively, using standard units of measurement such as seconds, gigabytes, and feet. 

Measurements are relatively objective because even if the units of measurement are standard, sometimes how those units are presented leaves interpretation up to the consumer.  For example, does the square footage of an apartment reflect only that particular apartment, or does it also include an allocation for common areas, such as elevators and stairwells?  Do the gas mileage ratings of a car reflect the actual mileage an average driver can expect to receive, or rather mileage in specific situations or certain conditions? 

Questions about the interpretation of claims can be simple, but have complex answers.  For many authorities who adjudicate such matters, the answer to the question should focus on what the statement means to the average consumer.  Claim substantiation surveys can provide key evidence and support. 

 
Other claims require non-scientific, subjective data, based on consumer attitudes and perceptions about the taste or performance or quality of a product.  For example, which peanut butter tastes crunchiest?  Which yogurt is preferred by children?  Which shampoo leaves your hair most shiny?  Which skin lotion has the most appealing scent?  These questions are often answered not with engineering tests, but rather with substantiation surveys in which consumers try and rate products.

 
A wide range of agencies and authorities have a say, either directly or indirectly, in making sure that claims have substantiation.  For example, a company that makes claims may find that it needs to substantiate or defend claims in the federal court system, or before agencies such as the Food and Drug Administration or the Federal Trade Commission. The Better Business Bureau operates a voluntary entity, called the National Advertising Division, which allows companies to resolve disputes in a system that can be faster, less adversarial, and easier than federal court. 

In short, claim substantiation matters.  Here are examples of some high-profile claim substantiation cases that have appeared in the news recently.  The examples demonstrate the obligations of marketers to clearly communicate facts backed by real evidence, and show how disputes can arise regarding what a statement means to the average consumer. 

• Do certain foods provide health benefits?  Lately, many food manufacturers have made claims related to the health content of the foods they make.  For example, some foods claim to improve brain health, reduce cholesterol, provide essential vitamins, lower sodium, enhance digestion, or provide other benefits.  As food marketers have increasingly seen “functional foods” as an opportunity and a priority, regulators at agencies such as the FDA and the FTC have taken more interest, as have the competitors of companies making claims. Click for The New York Times article.

• How much skin protection does sunscreen provide?  For a simple product, sunscreen can be complicated for consumers at the retail shelf.  There is a wide range of sunscreens on the market, positioned for uses such as a day at the beach, outdoor sports, swimming, everyday protection, protection for children, and others.  After 33 years, the FDA recently established new rules to clearly define the testing protocols that can support certain types of claims.  Along the way, familiar words such as “sunblock”, “waterproof”, and “sweatproof” have been banned.  Click for The New York Times article.

• Does water prevent dehydration?  The answer to this seemingly simple question has been quite complicated in Europe.  The European Commission was asked if makers of bottled water could claim on their label statements including that “regular consumption of significant amounts of water can reduce the risk of development of dehydration.  The European Foods Standard Authority concluded that the statement was misleading, in part because having too little water in the body “was a symptom of dehydration and not something that drinking water could subsequently control.”   The dispute hinges upon consumer perceptions, such as whether the statement suggests that bottled water provides special benefits against dehydration.  Click for The Telegraph article.

• Which wireless carrier has the best coverage?  In the United States, AT&T and Verizon have spent millions of dollars on advertising that discusses which company’s coverage is best and how its coverage compares to the other carrier.  For example, Verizon ran a series of commercials called “There’s a Map for That”; AT&T argued that the commercials unfairly depicted AT&T’s coverage.  Much of the dispute hinged on considerations of what claims consumers inferred from the commercials.  Click for The New York Times article.

•  Who has the best riding mowers?   Last year, Sears found itself trailing rivals such as Home Depot in the important category of garden care and home repair. They decided to fight back by providing reasons for consumers to shop at Sears rather than Home Depot or Lowes.  Such “comparative ads” are increasingly commonplace in the marketplace, as companies directly tell consumers the reasons why they should shop them rather than their rivals.  Click for The New York Times article.

About MMR Strategy Group

MMR Strategy Group (MMR Strategy) is a full-service market research-based consulting firm.  We help our clients grow by leveraging customer insight to develop marketing and sales strategies.  In order to support critical business decisions, we combine the data gathering capabilities of a research firm with the business analytics of a strategic consulting firm.

For more information, please contact us at 818.464.2400 or email info@mmrstrategy.com.