Jump to main content.

Chapter 11: Peer Review Comments on the Phase II Report Draft


The use of qualitative and quantitative research with a large number of consumers to determine consumer behavior and opinion is a relatively new and unique approach for EPA. The Consumer Labeling Initiative (CLI) had its Phase I Report peer reviewed in 1996 and found the reviewers' comments to be quite informative and helpful. Given the potential magnitude and impact of the recommendations deriving from the CLI's Phase II research, the EPA and CLI participants wanted to determine whether we had gone about our research appropriately and whether independent researchers believed the recommendations were supported by the research. With those goals in mind, a peer review of the Phase II Draft Report was undertaken.

Document Reviewed

The document reviewed was the Consumer Labeling Initiative Phase II Report - Draft, July 1, 1999. The Report contained the following major sections: 1) Executive Summary; 2) Overview of Phase II of the CLI; 3) Quantitative Research; 4) Qualitative Research; 5) Quantitative and Qualitative Research Conclusions; 6) First Aid - Qualitative Research; 7) Phase II Sub-groups; 8) Partner and Task Force Meetings; 9) Stakeholder Interactions and Comments; 10) CLI Phase II Recommendations; and finally many appendices supporting the research efforts. Appendices included: 1) Lists of participants; 2) Quantitative, qualitative, and first aid research screening documents, discussion guides, questionnaires, and mock labels; 3) notes of all major meetings; and, 4) a list of stakeholders who had submitted comments. The stakeholder comments were not included in the reviewed draft, and were included in the subsequent revision.

Peer Reviewers

The peer review was conducted by four independent reviewers not associated with either the United States Environmental Protection Agency, or the Consumer Labeling Initiative project. Reviewers were selected based on their expertise or experience in the fields of consumer behavior, consumer opinion, risk and hazard communications, consumer research and testing, and consumer education. Reviewers included: Dr. J. Stanley Black, Community Response Analyst, Office of Community Relations, Illinois Environmental Protection Agency; Dr. Albert J. Ignatowski, Principal, HazCom Consulting, and Senior Fellow, Wharton School, Risk Management and Decision Processes Center, University of Pennsylvania; Dr. Sidney I. Lirtzman, Dean, Zicklin School of Business, and Emanuel Saxe Professor of Management, Baruch College, City University of New York; and Beth Resnick, Associate Director, Division of Public Health Practice, National Association of City and County Health Officials.

Charge to Reviewers

Reviewers were asked to respond to 26 questions in five categories: Study Design; Study Implementation--quantitative, qualitative, and research groups; Study Results and Recommendations; the Peer Review Process; and any other comments not falling into those categories. The questions, which are included later in this chapter, asked, for example, about the appropriateness of the study methodologies and statistical methods chosen, the adequacy of the screening and survey instruments used, whether the key learning objectives were represented sufficiently in the research; whether the findings and recommendations were clearly supported by the research, etc.

Summary of Reviewers' Comments

Generally all of the reviewers' responses to the review questions were quite positive. However, there were some specific criticisms which are mentioned below. The most negative comments concerned the length and complexity of the written questionnaire. Comments which were submitted concerning specifics in the report itself have been addressed in the final version and so are not addressed here.

Study Design

All of the reviewers agreed the methodologies used in the study were appropriate and addressed the key learning objectives. One reviewer suggested that presenting randomly selected householders with a set of varied label formats might have provided more realistic results than the mail or phone surveys, but he also said the cost and logistical complications of that approach might not have been warranted by the increased value of the information obtained. Another reviewer believed that more valid information with respect to comprehension would have been obtained using personal interviews. Another reviewer didn't think enough focus was given to label alternatives for low-level readers and non-English speakers.

Qualitative Study Implementation


Generally the reviewers reported the recruitment procedures to be adequate. However, one reviewer stated it would have been better to focus on non-purchasers of products, and use the purchasers as a control group. Another reviewer said the selection criteria for the focus groups seemed quite inexact, but went on to say there were no claims that the groups were representative but only aimed for a reasonable variability.

Discussion Guides and Learning Objectives

All reviewers said the guides seemed sufficient and the objectives were represented. One reviewer suggested he would have asked participants for their preferences regarding label formats before showing any mock samples.

Mock Label Adequacy

All the reviewers agreed the labels were appropriate. One reviewer believed that there was too much emphasis on designing labels that mimicked existing FIFRA label design requirements, and then asking respondents if they liked them. The reviewer went on to say he would have preferred even more emphasis been devoted to isolating some key features of label design and presenting them in a manner to elicit respondent preferences.    Another was concerned there appeared to have been too many labels.

Quantitative Study Implementation

Screeners and Discussion Guides

All reviewers generally agreed the recruitment screeners and discussion guides were appropriate and adequate.

Written Questionnaire

All of the reviewers felt the written questionnaire was entirely too long. They had concerns about its complexity, smallness of type, and dense format. They were concerned the length could have lead to "question fatigue" and at least one reviewer expressed some concern about projecting the results because of that fatigue. Another reviewer suggested it would have been better if the items in the questionnaire were divided among subgroups of the study population, with appropriate redundancy for checking constancy.

Statistical Methods

Generally the reviewers were satisfied, but one reviewer said the tables were primitive (only percentages are reported) and he couldn't tell if tests of significance were performed routinely or not. He went on to say the size of the quantitative sample is large enough that some of the results have to be considered very important.

Learning Objectives

All agreed the learning objectives were adequately represented in the mail and phone questionnaires.

One reviewer did not think it appropriate that industry funded the quantitative research.

Research Groups

When asked "did the work of the groups appear to reflect what was being learned in the qualitative and quantitative research" all but one reviewer claimed they were unable to answer the question because of its vagueness. One reviewer did say the work of the groups was consistent with the gist of the results from the quantitative research.

Study Results and Recommendations

Findings Supported by Research

All the reviewers agreed the findings were supported by the research. One said, however, there was no attempt to qualify or moderate the findings based on the quite divergent results of the subgroup of respondents, namely the less-well educated, lower-income, and minority populations. Another reviewer expressed concern about whether we actually can determine consumers' current comprehension of the label language; although, he goes on to say "if one looks at the results of the preference data it is possible to draw the inference there is significant lack of comprehension of the standard label language because of preference for language which uses simpler words, phrases, and is active and directive toward specific goals." He later states it is only in the interviews on the first aid statements one is able to find reports of consumer confusion as to the meaning of words and phrases. This particular reviewer believed only personal interviews should be used to determine comprehension.

Use of Quotes

While the reviewers said the discussion and recommendations seemed relevant in relation to the quotes used, most said a wider sampling of quotes would have given them more confidence in the quotes selected.

Enough Raw Data Presented

All the reviewers agreed there was enough data presented. One reviewer said it should only be construed to represent consumer opinion and not actual behavior. He went on to say that while demographic information was obtained for all respondents, the tables are not broken down by these groups so the impact, if any, can be directly assessed. Another reviewer said "it is a very rich resource for evaluating consumer responses in this area."

All the reviewers agreed the conclusions and recommendations were supported by the findings and data.

Other Comments

Does the Report Adequately Explain the Project

All said yes, although one did say it was repetitive.

Are Stakeholder Concerns Adequately Represented/Addressed

Some reviewers felt stakeholder concerns were adequately represented, while others expressed some confusion or dissatisfaction. Limited stakeholder comments appeared in the version which was given to the peer reviewers. Significant additional stakeholder comments were included in the final version. One reviewer said more consumers and state and local agency representatives should have been included in the planning and steering groups and that increased retailer participation would have been helpful as well.

Storage and Disposal

One reviewer believed the extensive information on waste and container disposal was not well incorporated into consideration of the label design. Recommendations for including this information on labels seem "weak." The input for the various stakeholder groups was interesting but not directly germane to the study purpose. Another reviewer said it would have been more objective if both industry and the state and local organizations had presented reports or papers, rather than providing information differently.

Consumer Education

One reviewer suggested the education campaign should include references to source reduction and other alternative products and that retailers should be included since they will most likely play a large part in this effort.

One reviewer commented the study could have been significantly strengthened if more of the "interested parties" were professional hazard communicators. The reviewer went on to say he did not wish to diminish the value and import of much of what was learned; he found many of the conclusions immediately useful.

Peer Review Process

The reviewers all agreed this type of review should be done for similar efforts. One reviewer wrote the review procedure was commendable and long overdue. All reviewers agreed allotting more time to do the review would have been helpful. All agreed the materials provided to do the review were sufficient, but could have been organized better to facilitate the review; for example, the order of appendix materials, clearer labeling of appendix materials, references to the appropriate sections included in the questions, etc. One reviewer said the materials were unwieldy and offered several suggestions on how to improve the report.

Questions to the Peer Reviewers

Study Design

1.   Were appropriate methodologies chosen to conduct the study?

2.   Were appropriate methodologies chosen to address the key learning objectives?

Study Implementation


3.   Were the recruitment screeners appropriate to acquire the type of consumers needed to conduct this study?

4.   Were the questions asked in the discussion guides appropriate and/or sufficient to acquire the necessary consumer opinions about labels?

5.   Were the key learning objectives represented in the discussion guides?

6.   Did the mock labels/samples appear to be adequate for the participants?


7.   Were the recruitment screeners and practices appropriate to acquire the type and quantity of consumers needed to conduct the quantitative survey?

8.   Was the telephone interview outline adequate for its purpose?

9.   Was the length, structure and content of the written questionnaire appropriate?

10.   Were appropriate statistical methods and processes used to compile and evaluate the data from the surveys?

11.   Were the key learning objectives adequately represented by the questions on the mail and phone surveys?

Research Groups:

12.   Did the work of the groups appear to reflect what was being learned in the qualitative and quantitative research?

Study Results and Recommendations

13.   Are the findings supported by the research?

14.   Are the implications reasonable, based on the findings?

15.   Based on the quotes provided in the text from the focus groups, do the discussion and recommendations seem relevant?

16.   Is enough raw data presented to provide the reader with a clear picture of consumer behavior/opinions regarding labels?

17.   Are the conclusions supported by the findings and data?

18.   Do the recommendations appear supported by the research findings?

19.   Do the report findings/recommendations concerning the consumer education campaign, storage and disposal, standardized information, etc. appear to be supported by the research?

Peer Review Process

20.   Should the Agency consider this type of review for similar research efforts? If not, why not?

21.   Were the materials sufficient for your review? If not, what additional materials would you like to have seen included in the package.

22.   Was the time allotment adequate for review of the material and preparation of comments? If not, how much time do you believe is reasonably required to perform this review?

23.   What changes would you suggest to improve the process?


24.   Does the report adequately explain the goals, process, and accomplishments of the project?

25.   Are stakeholder concerns adequately represented/addressed?

26.   Are there any additional areas you would like to address or comments you would like to include?

Publications | Glossary | A-Z Index | Jobs

Local Navigation

Jump to main content.