Chapter 2 - Quantitative Research
At the end of Phase I, EPA, in consultation with CLI Partner and Task Force members, concurred with the recommendation that quantitative research in Phase II would be valuable to better understand consumers' preference for, comprehension of, and satisfaction with current product labels. A quantitative approach was favored because it was necessary to obtain statistically sound data to support the findings from the earlier qualitative research. Furthermore, unlike qualitative research data, quantitative research data are representative of the study population and projectable to the entire population. Quantitative research was also used to determine the prevalence of particular opinions on a given issue expressed in the qualitative interviews. Additionally, quantitative research was appropriate for measuring both attitudes and behavior of consumers to current and new product labels. Demonstrating their support for this concept, the CLI Partners volunteered to fund and direct this research, which they felt would be of use even beyond the CLI. Quantitative research also provides a baseline that can be surveyed periodically to determine changes in attitude and behavior.
The Phase II quantitative consumer research was designed to assess consumer comprehension, attitudes, behavior and satisfaction with labeling and to evaluate labeling alternatives (for both registered and non-registered products) in the outdoor pesticide, indoor insecticide, and hard surface cleaner categories. The quantitative survey was organized along the six learning objectives identified by the CLI Partner and Task Force members at the beginning of Phase II. These learning objectives are as follows:
Each learning objective was intended to generate research findings that would enable the EPA and CLI Stakeholders to take immediate and short-term steps toward label improvements. Some changes, such as revised guidance and regulations, are almost entirely under the purview of the EPA. Other changes are entirely within the purview of the product marketers but may be subject to EPA label approval. Others, such as consumer education, involve many Stakeholders and would be implemented over a longer time period. The results of the quantitative research were expected to lead to certain actionable steps, such as:
The design and implementation plan of the quantitative research was developed by the Research Core Group, consisting of EPA personnel, industry and trade association Partners, people from other federal and state agencies, and other interested CLI Stakeholders. The Core Group began, by addressing the learning objectives identified at the beginning of Phase II by CLI Partner and Task Force members, to develop the quantitative screening and survey questionnaires. Several of the members of the research group were market researchers in their own organizations and, therefore, had extensive experience with survey design. The quantitative research was voluntarily undertaken and funded by industry and trade association Partners of CLI including: AgrEvo Environmental Health; American Cyanamid (American Home Products); Bayer Corporation; the Chemical Specialties Manufacturers Association (CSMA); Dow AgroSciences; FMC; Reckitt & Colman; S.C. Johnson & Son, Inc.; The Procter and Gamble Company; The Clorox Company; Purcell Industries, Inc.; Riverdale Chemical Co.; SC Johnson; The Andersons, Inc.; The Scotts Co.; Solaris (Monsanto); United Industries Corporation; and the RISE (Responsible Industry for a Sound Environment). This group of companies hired an independent survey research firm, National Family Opinion (NFO) Research, Inc. to implement the study.
During Phase II, the Core Group met on a weekly basis via telephone conference calls, and occasionally in ad hoc face-to-face meetings, to discuss the development of the survey instruments, the implementation of the survey itself, and interpretation of the data once the results of the survey were available. In July 1998, a smaller subgroup of the Core Group met in Washington, D.C., to discuss the survey data in detail and establish some of the preliminary findings from the survey results. This smaller group consisted of EPA Task Force members, and market researchers from Amway Corporation; Bayer Corporation; S.C. Johnson and Son, Inc.; and the Procter and Gamble Company. In August, the subgroup finalized the preliminary findings and prepared data tables to illustrate these conclusions. In September 1998, the subgroup presented these results at the Partner and Task Force meeting in Alexandria, VA.
The quantitative study consisted of three parts: an initial screening (to identify potential study participants), followed by telephone interviews and a self-administered mail questionnaire among those selected to participate in the main portion of the quantitative study.
Screening to Identify Product Category Users for Use in the Study
In the first part of the quantitative phase of the study, a postcard with a very short screening questionnaire (screener) was mailed to members of the NFO Panel.(1)
The screener contained questions to identify consumers eligible for participation in the main portion of the quantitative study (and to eliminate those consumers not eligible for participation). Screener questions asked respondents the following:Whether any household member used a household cleaner in the past 12 months. For those who indicated usage of a household cleaner, the age and gender of the household member who is the primary user of household cleaners;
Whether any household member used an indoor insecticide in the past 12 months. For those who indicated usage of an indoor insecticide, the age and gender of the household member who is the primary user of indoor insecticides;
Whether any household member used an outdoor pesticide in the past 12 months. For those who indicated usage of an outdoor pesticide, the age and gender of the household member who is the primary user of outdoor pesticides; and
Whether the respondent had gone to the store to purchase each of the three types of products, but did not because of information contained on the label of the product.
In March 1998, the screening postcard was mailed out to a total of 10,000 NFO consumer panel households. The distribution of recipients who received this postcard was balanced to be representative of the U.S. population as a whole on age and gender of the head of household, geographic region, household size, market size, and household income. An additional 2,250 postcards were mailed out to households from three low incidence groups of interest (minority, lower formal educational level, lower income) on NFO's panel of 550,000, to ensure adequate representation in the final survey results. These low incidence groups were also balanced to be representative of their counterparts in the overall U.S. population. In April 1998, returns were closed out and the returns tabulated. A total of 8,447 households returned the postcard (69% of the number sent out). These results were then used to determine which households and which individuals to include in the main portion of the CLI quantitative study (i.e., phone and mail questionnaires) for appropriate demographic representation. Appendix 2-1 contains the screening questionnaire.
As stated above, non-users (in the past 12 months) were excluded from the main portion of the quantitative study. It must be noted that among the group of consumers who said on the screener that they had not used the specific products in the past 12 months (and were thus ineligible for inclusion in the main portion of the study), a small number also indicated on the screener that they went to the store to buy such a product, but did not purchase it because of information on the package (6% of those who did not purchase household cleaners, 7% of those who did not purchase indoor insecticides, and 5% of those who did not purchase outdoor pesticides). The information on the package cited as the reason consumers did not buy the product was not specified. It cannot be determined, therefore, what biasing impact, if any, was created by excluding these consumers from the study. Based on the low number of consumers who were excluded (between 5% and 7% of non-users for each category), it is unlikely that any such biases would alter the survey findings in any meaningful way.
Sample for the Telephone Interviews and Mail Questionnaire
For each product category, a group was formed of participants who indicated that they had used that type of product in the past 12 months. Additionally, supplemental samples of low-income households (i.e., those making less than $10,000 per year), less educated heads of household (i.e., those with less than high school education), and minorities were drawn for all three categories, and a supplemental sample of fogger users was also drawn for the indoor insecticides category. These additional samples were needed because the overall incidence of these groups in the U.S. population is so low that there would not be enough members of these groups in the nationally representative sample to allow for meaningful quantitative analysis of these particular groups.
These supplemental groups (i.e., supplemental samples) were included only for analyses that looked specifically at the group for which the supplemental sample was pulled. For example, the respondents who were part of the supplemental group for low-income households were included only in the separate analysis of consumers from low-income households. Excluding these special supplemental groups of respondents from other groups (e.g., the nationally representative sample) prevented the creation of an unnatural skew toward over-representing consumers from those groups for which a supplemental sample was pulled. It is important to note that, due to random selection, there are still members among the nationally representative sample who fall into the demographic groups for which supplemental samples were pulled.
The samples for each product category were balanced to be representative of the portion of the U.S. population that uses that particular category (i.e., household cleaners, indoor insecticides, outdoor pesticides). The samples were balanced on the following demographic variables:
The self-administered mail questionnaires were mailed out to a total of 6,438 households, broken down as follows:
When survey returns were closed in early June 1998, a total of 3,234 consumers (50% of the total sent out) completed both the telephone and mail portions of the study, with approximately 850 to 900 being nationally representative users of each of the three product categories. As appropriate, the remainder of returns were used to supplement the various low incidence groups.
The main portion of the CLI quantitative study was composed of 1) a telephone interview, followed by 2) a self-administered 8-page mail questionnaire. The telephone interview was used to collect information that would have been difficult to collect without direct interaction with an interviewer (e.g., having the respondent state where certain label sections were located). Telephone interviewers also allowed for clarifications and follow-up probing of responses regarding comprehension. Questions on the phone survey were rotated so that any order bias or "question fatigue" would be avoided. The mail questionnaire was used to collect a large amount of detailed information that could not be collected over the telephone due to time (i.e., length of interview) considerations. The telephone interview also asked consumers for "top of mind" responses to mimic actual consumer behavior (e.g., exercise of choices and capabilities) when they encounter the label both in the store and at home.
There were three different versions of the survey: one for household cleaning products, one for indoor insecticides, and one for outdoor pesticides, with the bulk of questions being identical on all three. In April 1998, participants were sent one version of the questionnaire booklet, along with a letter of instruction and a "mock" label (for use in both the phone and mail portions of the study). The mock label was representative of a typical product label for the product category for which respondents were selected. Participants were instructed to await a telephone call before completing the self-administered mail questionnaire. After allowing time for mail delivery, respondents were contacted by phone in early May and asked to complete a 10-minute telephone interview (average time), with responses collected using a computerized telephone questionnaire. After completion of the telephone survey, respondents were then instructed to complete the 8-page mail questionnaire and return it to NFO Research. After one month for completion and return of the self-administered mail questionnaire, returns were closed in early June 1998, and all results from the telephone and mail surveys were then tabulated. Only results from those completing both the telephone and the mail portions of the survey were included in the final results.
Survey Questionnaires and Learning Objectives
The telephone and mail survey instruments were designed by the Core Group (quantitative research group) to address the learning objectives outlined at the beginning of this chapter. The learning objectives, questions from each questionnaire relating to that learning objective, and the potential action steps emerging from these questions are provided in Table 2-1.
In addition to the learning objectives, the Core Group developed the survey instruments to investigate consumer attitudes, behaviors, and understanding related to specific areas and issues, including:
The telephone interview questionnaire used "mock" labels to ask questions related to consumers' comprehension of and ease of finding information on the labels. More specifically, the telephone questionnaire tested respondents' ability to locate key sections of the label, the accuracy with which respondents were able to locate these sections, and their opinions on the ease of finding these sections. Respondents also were asked what they thought certain language on the label meant, including specific key words and phrases. Finally, the telephone survey asked several demographic questions. (See Appendix 2-2 for copies of the phone questionnaires, and Appendix 2-3 for the mock labels.) Each interview was conducted by trained interviewers from NFO Research, Inc., and lasted approximately 10-12 minutes. At the conclusion of the telephone interview, the interviewer instructed the respondent to complete the written questionnaire in his or her own time and mail it back to NFO Research, Inc., once completed.
Mail Questionnaire Outline
The mail questionnaires (see Appendix 2-4) were designed to address the following specific questions:
National Family Opinion Research completed collection of the survey responses and data tabulation during the months of June and early July(3). In the final count, the total number of responses received for the mail and the telephone surveys were as follows:
Statistical Testing of Data
When comparing different groups of data quantitatively, statistical tests are needed to help determine which data are meaningful and which are not. A two-tailed t-test, which compares the percentages or means of interest and the sample sizes, was used to determine whether differences existing among groups are significant on a statistical level.
This type of statistical testing is done based on the level of significance desired. Data are most frequently tested for significance at levels between 80% and 95%. The higher the level of statistical testing performed, the more likely it is that data differences detected in the study reliably reflect differences in the "real world." If a significant difference between two data points at the 95% confidence interval is found to exist, this means that the same study, if conducted 100 times, would show a significant difference reflected in its data at least 95 of those times. For the CLI study, data were tested at the 95% confidence level. In the raw data tables, significance was routinely tested. For each question asked, the mean, standard deviation, and standard error are also shown for each type of respondent.
Breakdown of CLI Data
The Core Group determined that it would be important to investigate whether significant differences existed among various groups of respondents. To this end, the raw data were broken down by various demographic categories and by ways in which respondents answered several key questions. These breakdowns were necessary so that analysis and comparisons could be made among different groups that responded to the questionnaire. For example, the gender category allowed the Core Group to determine if there is any significant difference between the numbers of males and females who read information on product labels. A total of seven demographic categories were made for the CLI study as follows:
In addition, seven categories were made to compare the ways in which respondents answered key questions of interest for the Core Group's analysis, as follows:
Based on a standard statistical measure for sample sizes of about 850 to 900 respondents, the data for the nationally representative sample of users for each of the three product categories are accurate to + 3½% at the 95% confidence interval. This means that if the study were conducted 100 times and 50% of respondents gave a certain response, 95 out of those 100 tests would yield a result for that response if given by between 46.5% and 53.5% of respondents. As percentages move towards the extremes (i.e., closer to 0% and 100%), the precision of these data points will actually be higher. It is important to note that these precision measures refer to specific data points, and not to differences between data points. Precision for groups with smaller sample sizes will be lower.
1. A complete set of the quantitative data may be found in the EPA's Public Docket, Administrative Record AR-139. The availability of the data for public review was announced in a Federal Register (FR) notice (63 FR 57298, October 27, 1998).
1. Consumers were screened from NFO Research's consumer panel of 550,000 households. The panel of 550,000 was randomly chosen from the population as a whole. The NFO panel consumers have agreed in advance to participate in marketing research studies. When households become members of the NFO panel, they provide a large amount of demographic information about their household (e.g., age and gender of household members, household income, household size, education and employment information on the male and female heads of household, and many other types of information). This large database of pre-recruited households allows NFO Research to:
2. Pesticides, disinfectants, and antimicrobial cleaners are subject to labeling requirements under FIFRA. Other products (i.e., in the case of products covered by CLI, non-disinfectant and/or antimicrobial household cleaners), are governed by other authorities. In the cleaner category, therefore, product labels are markedly different, depending on whether FIFRA or a different statute applies, even through the products in the bottle may be similar. For the CLI quantitative research, respondents in the household cleaners category were presented with a FIFRA and a non-FIFRA label to determine how each was perceived.
3. A complete set of the quantitative data may be found in the EPA's Public Docket, Administrative Record AR-139. The availability of the data for public review was announced in a Federal Register (FR) notice (63 FR 57298, October 27, 1998).