Jump to main content or area navigation.

Contact Us

facebook iconLike Us on Facebook 
You are here: EPA Home » DfE » Appendix A

Appendix A

Office of Prevention, Pesticides and Toxic Substances

"Standard Operating Procedures for Peer Review

of Major Scientific and Technical Documents,

October 1, 1996 - September 30, 1997"

Office Of Prevention, Pesticides and Toxic Substances

STANDARD OPERATING PROCEDURES

FOR PEER REVIEW OF MAJOR

SCIENTIFIC AND TECHNICAL DOCUMENTS

OCTOBER 1, 1996 - SEPTEMBER 30, 1997


PREFACE

Overview

These Standard Operating Procedures (SOPs) provide guidance to all staff and managers in OPPTS on the organization and conduct of peer reviews pursuant to the Administrator's June 7, 1994 Peer Review Policy statement (Appendix A).

The guidance provides information and outlines procedures in several different areas:

  • basic principles and definitions, including distinctions between peer review and public comment, and between peer involvement and peer review;

  • preparing for peer review, including identifying work products, identifying appropriate peer review mechanisms, and identifying qualified experts; and,

  • conducting and completing peer reviews, including materials required for peer review, creating a peer review record, and utilizing peer review comments.

In addition, appendices provide information on OPPTS staff and management responsible for peer review activities and resources for the upcoming fiscal year, work products selected for peer review during the upcoming fiscal year, and detailed information on budget, procurement, and legal considerations.

This SOP does not address, nor does it supersede, established peer review practices or procedures of the selected peer review mechanism. In fact, the SOP seeks to incorporate existing mechanisms (e.g., professional journals for research papers, Science Advisory Board and FIFRA Scientific Advisory Panel, and research grant applications) as well as instituting new ones.

Background

Peer review at the USEPA takes many different forms depending on the nature of the work product, relevant statutory requirements, and office-specific policies and practices, among other things. In January 1993, responding to recommendations in the report Safeguarding the Future: Credible Science, Credible Decisions, Administrator William Reilly issued an Agency-wide policy for peer review. Administrator Carol Browner confirmed and reissued the policy on June 7, 1994 and instituted an Agency-wide implementation program. These SOPs constitute implementation guidance for OPPTS.

These procedures are based on the central themes set forth in the Administrator's policy statement:

    Major scientifically and technically based work products related to Agency decisions normally should be peer reviewed. Agency managers within Headquarters, Regions, laboratories, and field components determine and are accountable for the decision whether to employ peer review in particular instances and, if so, its character, scope, and timing. These decisions are made in conformance with program goals and priorities, resource constraints, and statutory or court-ordered deadlines. For those work products that are intended to support the most important decisions or that have special importance in their own right, external peer review is the procedure of choice. Peer review is not restricted to the penultimate version of work products; in fact, peer review at the planning stage can often be extremely beneficial.

As seen in the policy statement, one important task in implementing the Peer Review Policy entails the identification of "major scientific and technical work products." This decision is based on several considerations and is discussed in Section II. Another important task is using a peer review mechanism that is suitable for the work product under review.

The goal of the Peer Review Policy is to ensure that scientific and technical work products receive appropriate levels of critical scrutiny from scientific and technical experts as part of the overall decision making process. Generally, this technical review will precede the customary, more broadly based public review of the total decision.

OPPTS Peer Review Policy Overview

OPPTS recognizes that science and social science form the basis for protection of the environment and public health. The purpose of this SOP is to ensure that peer review is used appropriately to enhance the credibility of OPPTS' decisions while ensuring that OPPTS' peer review policies are in conformity with Agency guidance. In general, OPPTS has traditionally utilized peer review as well as peer involvement procedures with much emphasis placed upon involving experts and interested parties early and often in the development of a scientific or technical work product. This allows an open exchange of data, insights, and ideas throughout the life of a project, working toward building consensus on the technical aspects of the work.

Peer review mechanisms are typically selected based upon the nature of the scientific or technical work product. In general, scientific or technical work products will undergo an internal peer review. However, for work products that are "major", potentially costly, controversial, novel approaches, or have cross-Agency implications, external peer review is recommended.


TABLE OF CONTENTS

Preface

i

I. Guiding Principles of Peer Review

A. General Principles

B. Definitions

1

1

3

II. Identifying Work Products for Peer Review

A. The Selection Process

B. Mechanisms for Peer Review

C. Categories of Office Products

5

5

7

9

III. Planning and Conducting a Peer Review

A. Selecting Peer Reviewers

B. Scheduling Peer Reviews

C. Materials

11

12

14

15

IV. Completing a Peer Review

A. Evaluating Comments and Recommendations

B. The Final Work Product

C. Maintaining the Peer Review Record

17

17

17

18

V. Accountability and Responsibility in the Office

A. Line Management

B. Office Coordination

C. Budget

D. Annual Reviews

17

17

17

18

Appendix A: Agency Peer Review Policy

A-1

Appendix B: Key Personnel

I. Individuals Involved in the Peer Review

    A. Line Management

    B. Office Coordination

II. Legal Advice

B-1

B-1

B-1

B-2

B-3

Appendix C: Listing of Representative Office Products in Each Category Over the Past 3 Years

C-1

Appendix D: Office Candidates for Peer Review in Upcoming Fiscal Year

D-1

Appendix E: Resources for Peer Review Services

I. Acquisition of Peer Review Services

    A. Voluntary Services

    B. Contracts

    C. Small Purchases

II. Travel
    A. Contracts and Purchase Orders

    B. Special Government Employees

E-1

E-1

E-1

E-1

E-7

E-9

E-9

E-10


 

I. GUIDING PRINCIPLES OF PEER REVIEW

This section offers some guiding principles for peer review, defining certain terms, with emphasis on distinguishing between several closely related concepts. In particular, this section discusses the interrelationship between the broader concept of peer involvement and its components: peer input and peer review.

A. General Principles

The purpose of peer review is to uncover any technical problems or unresolved issues for use in revising a preliminary product so that the final work product will reflect sound technical information and analyses. It should be noted that peer review is a process for enhancing the scientific or technical work product.

Peer involvement occurs at different stages, involving several related but different activities in the overall decision-making process (e.g., an initial research protocol or work plan, a preliminary review of an interim draft, and/or a later review of a draft final work product). Peer involvement refers to both peer input (ongoing discussions during the development of the work product) and peer review (the critical, and usually, final evaluation of the work product). Peer review can occur during the early stages of project or methods selection, or as usually defined, as part of the culmination of the work product, ensuring that the final product is technically sound.

Subject matter experts who participate in the overall peer involvement process can be expected to undertake one of three related but different roles. First, they may work as paid or unpaid consultants with a significant role as author or advisor in developing a work product. Second, OPPTS may ask independent experts to provide peer input by participating in early developmental reviews or discussions of well-developed but unfinished work products. In this case, the agency is inviting and expecting expert suggestions that could lead to substantial changes based on reviewer recommendations in the final product. Third, experts may be asked to serve as peer reviewers, providing critical evaluation and comments on work products nearing completion.

Depending on the situation, peer review can be an iterative process or a single event. Subject matter experts may review several revisions of the same document, or they may comment only once on the document. Agency managers need to be aware of the role a subject matter expert has played in earlier reviews and make a determination -- on a case by case basis -- as to that expert's independence for any further reviews.

The importance of the peer reviewer's independence and technical expertise cannot be overemphasized as factors influencing the value and credibility of any peer review. Independence -- freedom from institutional, ideological, or technical bias -- as to the issues under review is necessary for objective, fair, and responsible evaluation of the work product under review. However, if reviewers are selected to represent any particular institution or technical perspective, balancing the review with representatives of other institutions or perspectives becomes a critical objective for the peer review. Such perspectives and/or potential conflicts-of-interest (real or perceived) should be fully identified to ensure a balanced and credible peer review. Technical bias should also be considered. An expert who serves as an author/advisor or early participant in developmental reviews or discussions may also not be considered independent and unbiased for a given work product.

The importance of scientific and technical expertise in the subject matter is obvious, but there are several dimensions. Knowledge in the subject matter is not equivalent to expertise in the subject matter. For agency decisions, a multi-disciplinary group of experts corresponding to the disciplines that contribute to complex agency decisions is often necessary for a full and complete peer review. For example, a risk assessment that relies on both animal and human data often requires experts in both areas for a complete review.

Peer review and public comment are not synonymous. Public comment solicited from the general public through the Federal Register or by other means is often required by the Administrative Procedures Act, relevant statutes or both. Public comment can also be solicited for policy purposes. The USEPA takes public comment on some strictly scientific products and almost all regulatory decisions. Public commentors usually include a broad array of people with an interest in the technical analysis or the regulatory decision; some are scientific experts, some are experts in other areas, and some are interested non-experts. The critical distinction is that public comment does not necessarily draw the kind of independent, expert information and analyses expected from the peer review process. Public comment is open to all issues, whereas, the peer review process is limited to consideration of technical issues. While an important component of the review process, public comment does not substitute for peer review.

As part of each peer review, OPPTS must formulate a clear, focused charge that identifies recognized problem areas and invites comments or assistance. This request signals the Agency's awareness of potential problems and its receptivity to expert recommendations. The charge to peer reviewers usually makes two general requests. First, it focuses the review by presenting specific questions and concerns that the Agency expects the reviewers to address. Secondly, it invites general comments on the entire work product. Focused questions also greatly simplify the task of collating, analyzing and synthesizing peer review comments on a topical basis. Moreover, written responses to these questions by peer reviewers help the Agency create a peer review record.

Careful attention to all of these elements, singly and together, assures a credible peer review process. Conversely, inattention can nullify the peer review attempt. A well-planned peer review applied to a reasonable quality starting work product, followed by responsible, visible utilization of peer review suggestions in the final product assures a credible product for use in Agency decision-making.

B. Definitions

1. Peer Involvement - Peer involvement refers to independent expert participation in the development and/or review of a scientific and technical work product. Peer involvement constitutes active outreach to and participation of the broad scientific, engineering, and economics communities beyond the USEPA (external) as well as within the USEPA (internal). Typically, peer involvement takes two general forms, peer input and peer review:

    a. Peer Input - Peer input generally connotes an interaction during the development of an evolving Agency work product, providing an open exchange of data, insights, and ideas. Peer input is characterized by a continued and iterative interaction with scientific experts during the early stages of peer involvement.

    b. Peer Review - Peer review is an objective, critical review of a specific Agency major scientific and technical work product by an independent peer reviewer or reviewers. Peer review can occur at several discrete points during the peer involvement process. It is characterized by a one-time interaction or a limited number of interactions by independent peer reviewers.

2. Independent Peer Reviewer - An independent peer reviewer is an expert who was not associated with the generation of the specific work product either directly by substantial contribution to its development or indirectly by consultation during the development of the specific product. The independent peer reviewer, thus, can be objectively judgmental. This individual has expertise in the subject matter required for the review function.

3. Major Scientific and Technical Work Product - Scientific, engineering, and economic documents or positions that are used to support a research agenda, regulatory program, policy position or other Agency decision that meet one or more of the criteria found in Table 1 (in Section II.).

4. Peer Review Leader - A person who organizes and oversees the peer review process for individual, specific work products. Sometimes this individual is also the Project Manager for the work product.

5. Peer Review Coordinator - The individual designated by the AA to coordinate and monitor peer review activities.

6. Peer Review Group - The individuals who form the working group that supports the Peer Review Coordinator.

Back to Appendix A Contents
Back to Main Contents

 

II. IDENTIFYING WORK PRODUCTS FOR PEER REVIEW

The principle underlying the Peer Review Policy is that all major scientific and technical work products (MSTWPs) normally should be peer reviewed. The process for identifying major products for review and then determining the mechanism of review will take into account various criteria. The decision maker(s) for peer review should consider the full field of possible work products that could benefit from peer review as well as the full spectrum of peer review mechanisms for each product.

A. The Selection Process

1. Major Scientific and Technical Work Product (MSTWP)

a) The determination that a scientific or technical product is a "major" work product is based upon a thorough consideration of a series of criteria. Work products that are used to support a regulatory program or policy position and that meet one or more of the criteria found in Table 1 are candidates for peer review:

TABLE 1.

SOME CRITERIA FOR IDENTIFYING MAJOR
SCIENTIFIC AND TECHNICAL WORK PRODUCTS

1. Supports major regulatory decisions or policy/guidance of major impact

2. Establishes a significant precedent, model, or methodology

3. Addresses controversial issues

4. Focuses on significant emerging issues

5. Has significant cross-Agency/inter-agency implications

6. Involves a significant investment of Agency resources

7. Considers an innovative approach for a previously defined problem/process/methodology

8. Satisfies a statutory or other legal mandate for peer review

There are two specific circumstances when there does not appear to be a rational for improving the work product with peer review. First, additional peer review is not required for a particular product that has a known peer review record by a recognized expert or expert body. For example, a cancer risk assessment methodology or an exposure modeling technique that had been the subject of earlier peer review, would not require additional review, even if the product supported a significant Agency decision. Second, additional review is not required if a new application of an adequately peer reviewed work product does not depart significantly from its scientific or technical approach.

b) Scientific or technical work products that do not meet these criteria for "major" may be also considered candidates for peer review depending upon the nature of these products, specific program needs/goals, and potential benefits of the peer review process (i.e., is value added to the work product?).

c) The need for and timing of peer review at various stages in the development of the work product may greatly benefit that product. The criteria in Table 1 may be used to help determine if peer review is appropriate at a particular stage of work product development (e.g., help an office/region decide which direction/ methodology/technique is most appropriate from two or more choices).

d) Appendix D lists scientific/technical activities/products identified in each category by OPPTS as candidates for peer review for the upcoming fiscal year. Also listed is the projected mechanism of peer review for each work product.

2. Special Circumstances Which May Constrain the Use of Peer Review

a) Exploratory Analyses and Voluntary Risk Reduction: OPPTS is involved with a number of activities that involve exploratory scientific and technical analyses. For example, the Pollution Prevention Program may collaborate with stakeholders, including chemical manufacturers, formulators, users and others to characterize the hazards, uses, exposure(s) and risks of a substance, as well as economic considerations, to identify pollution prevention opportunities. In other cases, alternative chemicals or use practices (i.e., OPP's Special Review process) may be considered that present lower risks or exploratory analyses may be used to set priorities for additional testing or information gathering. When such scientific and technical work products are developed in anticipation of collaborative activities with stakeholders, these scientific and technical work products would not usually require peer review. If, however, OPPTS were to determine that unilateral action is appropriate, scientific and technical work products supporting such action would be candidates for peer review.

b) Constraints: Statutory and court ordered deadlines and other time constraints may limit or preclude peer review. Rarely, resource availability may also limit peer review. OPPTS will evaluate these circumstances on a case by case basis; decisions will be based on consultations involving line management and the Peer Review Coordinator. If no peer review is conducted, then written justification to the peer review archive is needed to explain the circumstance. The AA will notify the Science Policy Council of the decision.

B. Mechanisms for Peer Review

1. Mechanisms List

Peer review for OPPTS takes many different forms. In OPPTS, the most commonly used mechanisms are the FIFRA Scientific Advisory Panel, the Biotechnology Science Advisory Committee, workshops or symposia, outside experts, and the Science Advisory Board. Table 2 lists 13 different types of peer review, ranging from a single Agency expert to a twenty-person panel of external experts. Familiarity with the method, advantages, resource requirements, and logistics of the different forms, and their variations, is an important aspect of peer review planning.

TABLE 2.

SOME MECHANISMS FOR OBTAINING PEER REVIEW
OF SCIENTIFIC AND TECHNICAL WORK PRODUCTS

INTERNAL PEER REVIEW

1. An independent expert from within the Agency
e.g., an ORD expert on non-cancer effects reviews a draft article on benchmark dose.

2. An ad hoc panel of independent experts from (mostly) within the Agency
e.g., a group is convened to examine the case for the classification for carcinogenicity for a chemical.

3. Office work group product reviewed for technical merit by scientist in Agency laboratory
e.g., initial review of regional WTI risk assessment plan by RTP scientists.

EXTERNAL PEER REVIEW

1. Independent expert(s) from outside the Agency
e.g., a peer reviewer for a journal to whom an Agency scientist has submitted a paper for publication; a letter review by one or more reviewers

2. An ad hoc panel of independent experts from (mostly) outside the Agency
e.g., a group is convened to reach consensus on the carcinogenicity of formaldehyde.

3. Agency (including Risk Assessment Forum)-sponsored Peer Review Workshops
e.g., review of indicators of ecosystem damage.

4. An Agency-based Federal Advisory Committee (other than SAB)
e.g., the Biotechnology Science Advisory Committee (BSAC) meets to review technical aspects of the release of bioengineered organisms.

5. Agency-appointed special Board or Commission
e.g., review of risk issued by the CAA Commission on Risk Assessment.

6. Agency Science Advisory Board (SAB)
e.g., review of drinking water criteria document for arsenic.

7. Interagency committee
e.g., review of research plans by Committee on the Environment and Natural Resources (CENR) coordinated by the White House.

8. Committee of another agency
e.g., review of "dioxin" reassessment by HHS Committee to Coordinate Environmentally Related Programs (CCERP).

9. Non USEPA-based groups
e.g., Society of Risk Analysis review of cancer guidelines.

10. National Academy of Science/National Research Council
e.g., review of children and pesticides.

 

2. Selection of Appropriate Mechanism(s)

a) Determining an appropriate mechanism of peer review is generally subjective. Science staff are responsible for recommending and OPPTS managers for selecting peer review options based upon the importance and impact of the work product, the resources available (staff, budget, reviewers), the time investment, and previous review history. The choice of mechanism is based upon several criteria, as seen in Table 1. Also, the extent of previous peer involvement and/or peer review helps determine an appropriate mechanism of peer review.

b) Generally, the more nearly novel or complex the science or technology, the greater the cost implications of the impending decision, and the more controversial the issue, then the stronger the indication for a more extensive and involved peer review and for external peer review in particular. For example, certain work products will clearly lend themselves to extensive external peer review; generally these will be products with large impacts. Other major work products may not need a large scale external peer review and may utilize a less involved, less resource intensive review. The peer review of some products may be better served with some form of internal peer review or a combination of internal and external peer review. Again, the choice is subjective and will generally be dependent upon the experience and assessment of the decision maker(s) dealing with peer review issues. It is important to make this choice at the time that the work is planned so that peer review costs and time can be budgeted into the work plan.

C. Categories of Office Products

OPPTS is involved in a wide variety of activities that have strong scientific and technical components. These activities include hazard, dose-response, exposure, and risk assessments, economic analyses and valuations, development of new assessment methods, and surveys. Some activities (e.g., evaluations of new chemical substances) are performed routinely using standardized assessment methodologies.

Other activities involve breaking new ground or the use of novel techniques or analyses (e.g., formaldehyde risk assessment, acute dietary risk assessment methodology, assessment of biological control agents). All OPPTS scientific and technical activities benefit from peer involvement. The level of peer involvement will differ depending on the scope of the effort, and the significance of the potential outcome, among other factors.

For purposes of discussion, major OPPTS scientific and technical activities can be grouped into several categories.

Category #1: Products that support major regulatory decisions.

    1) Risk assessments (including associated hazard, dose-response and exposure analyses)

    2) Benefits and cost (economic) assessments relevant to some unreasonable risk determination under TSCA or FIFRA

    3) Data requirements for pesticide registration

    4) Technical guidance documents for the regulated community

Category #2: New approaches to the assessment of chemicals or products of biotechnology. Issues having a major national impact on public health and or ecosystem protection. Examples include:

    1) issues associated with release of genetically engineered organisms into the environment

    2) application of new scientific principles in risk assessment, e.g., the consideration of biological mechanisms for induction of cancer in risk assessment

    3) risk assessment methods development, e.g., risk assessment methods concerning children exposed to pesticide residues

    4) risk mitigation proposals for eco-risk reduction

    5) efficacy testing procedures for hospital disinfectants and sterilants

Category #3: Products that result in generation of large quantities of data or unusual data. Examples include

    1) major surveys, e.g.,

      * national survey of lead-based paint in housing

      * survey of pesticides in drinking water wells

      * national food consumption survey

    2) monitoring of chemicals in environmental components

Category #4: New predictive tools or models of significance, generic assessment methodologies to be used routinely in risk assessment, novel approaches to economic valuation. Examples include:

    1) expert systems and quantitative techniques designed to help predict hazards, chemical fate, etc., from chemical structure, use, or toxicity/exposure data

    2) significant models and other techniques designed to predict exposure, simulate transport, etc.

    3) precedent-setting methods for economic valuation

    4) priority-setting tools, e.g., OPPT Use Cluster Scoring System

Appendix C lists completed peer reviews initially listed in prior Appendicies D plus representative scientific/technical activities/work products peer reviewed prior to FY-95.

Back to Appendix A Contents
Back to Main Contents

III. PLANNING AND CONDUCTING A PEER REVIEW

The success and usefulness of any peer review depends on the quality of the peer review draft, the care given to the statement of the issues or "charge", the match between the peer review draft and the form of peer review, the match between the peer review draft and the scientific/technical expertise of the reviewers, and Agency use of peer review comments in the final product. It is not enough simply to conduct a peer review; each of the foregoing elements requires serious attention.

A. Selecting Peer Reviewers

1. Sources for Peer Reviewers

a) Recommendations for potential peer reviewers can be identified from a number of organizations. These include external groups such as the affected party(ies); special interest groups; public interest groups; environmental groups; trade or business associations; state organizations or agencies; the National Research Council; and other Federal agencies with an involvement in or familiarity with the issue. Internal groups include the staff of the Science Advisory Board (SAB), BSAC, or the Scientific Advisory Panel (SAP); relevant ORD scientific staff; and other Program or Regional experts.

b) In certain circumstances, existing peer review organizations such as the SAB, BSAC or SAP may be used to conduct a peer review. These groups establish their own criteria for accepting work and coordination must be made directly with them. Both conduct formal, public, external peer reviews.

c) Occasionally, a member of the scientific community will offer his/her services for peer review during an ongoing peer review. Disposition of these offers will be handled on a case by case basis in consultation with the Peer Review Coordinator and appropriate decision makers.

2. Selecting Peer Reviewers

a) Peer reviewers should be selected for independence and scientific/technical expertise. The emphasis on independence and expertise applies equally to government experts and experts from the larger scientific community.

b) Ideally, peer reviewers should be free of real or perceived conflicts-of-interest or there should be a balancing of interests among peer reviewers.

c) Some peer reviews can be conducted with one or two reviewers; others involve panels of peer reviewers. In either case each of the peer reviewers should have recognized technical expertise that bears on the subject matter under discussion. Taken as a whole, the peer reviewers of a work product should represent a balanced range of technically legitimate points of view. In the face of equally qualified experts, cultural diversity and "address" (e.g., industrial, academic, or environmental community) are secondary factors that can play a role.

3. Internal vs. External Peer Reviewers

Generally, external peer reviewers are preferred. For some work products, either external or internal peer review may be appropriate. Selection of internal peer reviewers should be based upon technical expertise, available time and "address"; that is, they should not come from the immediate office or group producing the product or have any other connection with the product or document being peer reviewed.

4. Disciplinary Mix

a) A peer review panel or group can number from just a few individuals to ten or more, depending on the issue, the time and resources available and the broad spectrum of expertise required to treat the range of issues/questions in the charge (see Section III.C. below). Naturally, experts whose understanding of the specific technical area(s) being evaluated are critical; nevertheless, it is also important to include a broad enough spectrum of other related experts to completely evaluate the relevant impacts on other less obvious concerns. For example, for health related peer reviews, experts in such fields as ecology and economics may provide very useful insights.

b) There is usually a continuum of views on any issue. To the extent possible or practicable, selected experts should have views that fall to either side of the centrist position along the continuum, but not too far to either extreme. This will help maintain a balanced viewpoint, while allowing all views to be expressed and discussed. A balanced panel will allow consensus building. As a general rule, experts who have made public pronouncements on an issue (e.g., those who have clearly "taken sides") have difficulty in reaching consensus positions and should be avoided.

5. Constraints in Selecting Peer Reviewers

a) Peer review is not free. Regardless of the type of peer review chosen, there is some cost to the Agency. This cost can range from the time invested by a few Agency staff during an internal peer review, to the total costs associated with a full, formal, public external peer review; e.g., SAB review.

b) Sometimes the need for a peer review is accelerated due to a court-ordered deadline or other time-sensitive requirements. In such cases, it is difficult, if not impossible to conduct a full external peer review. It may even be impossible to conduct a small scale internal peer review using just a few individuals. Mechanisms for identifying and using a small number of peer reviewers should be included for OPPTS so that quick, effective peer review can be included for even the most rapidly moving products.

c) Care must be taken to reduce the possibility for real or apparent conflicts of interest between the reviewers and the OPPTS work product under review. Various tools are available to identify and limit conflicts of interest (e.g., attention to the employment, financial, and professional affiliations of the participants; filing Confidential Financial Disclosure Forms (SF-450) in the case of members of Federal Advisory Committees; exploring directly the issue with each of the participants before the review process takes place; and disclosing publicly at the beginning of meetings any previous involvement with the issue). The established peer review groups such as the Science Advisory Board and the Scientific Advisory Panel provide useful models for addressing balance and conflict-of-interest issues. Assistance in determining legal conflicts of interest can be obtained through the Office of the General Counsel. In addition, additional advice can be obtained from the Designated Agency Ethics Officials. Appendix E has further discussion on conflict of interest.

d) To evaluate OPPTS-generated studies properly, some peer reviewers may need access to confidential business information (CBI). However, unless the reviewers are Federal employees, it is unlikely that the USEPA has the authority to disclose CBI to them. Therefore, whenever contemplating the use of outside peer reviewers, Agency staff should determine whether the reviewers will need access to CBI. If so, the Office of the General Counsel should be consulted on whether it is practical to obtain the consent of CBI submitters to disclose the information to peer reviewers.

e) Offices need to be aware of the requirements of the Federal Advisory Committee Act (FACA) when establishing peer review mechanisms. Federal advisory committees are subject to chartering by the General Services Administration, hold meetings that are open to the public, and have balanced membership requirements. The Office of the General Counsel should be consulted regarding the applicability of FACA to peer review panels, see appendix E for additional information.

 

B. Scheduling Peer Reviews

The peer review schedule is a critical feature of the process. The schedule must take into account the availability of a peer review quality draft work product, availability of appropriate experts, time available for using peer review comments, deadlines for the final work product, and logistical aspects of the peer review (e.g., contracting procedures).

The schedule for peer review should take into account the overall rulemaking (or other decision-making) schedule. Peer review sometimes leads to new information and analyses, or recommendations for new research that would alter the work product and thus modify the scientific/technical basis for the action. For this reason, it is usually advisable to complete the peer review before taking public comment, or at least before the close of the public comment period.

C. Materials

1. Information Provided to Peer Reviewers

a) Essential documentation for each peer reviewer includes:

    1) A current copy of the work product to be peer reviewed with associated background material. The work product needs to be of the best possible scientific/technical quality to ensure an adequate and useful peer review.

    2) A clear charge or statement of work seeking informed comment on identified issues to properly focus the efforts of the peer reviewers and ensure that their individual efforts can be merged.

    3) Some information concerning the process that OPPTS is using for the peer review, including due date of reviewer comments and the format of those responses. Responses should be written and submitted to OPPTS by an agreed upon deadline. In certain rare cases, oral commentary may be sufficient. However, in such cases, a follow-up written response for the record is required.

    4) OPPTS will make clear to the peer reviewers their responsibilities. One of the major responsibilities of peer reviewers is to ensure confidentiality of the peer reviewed work product. Each peer reviewer must be informed of the need for confidentiality with regard to the release of OPPTS products that are stamped as "DRAFT" or "DRAFT - Do Not Cite, Quote, or Release." Premature release of draft Agency products, views, or positions is inappropriate and can be damaging to the credibility of the Agency or the peer reviewer. While not having legal authority, such language will be included in the charge to the peer reviewers. Other mechanisms to use in discouraging premature release include a disclaimer that appears in a separate section at the front of the document and creating the document with watermarks clearly delineating DRAFT status (or a header or footer that states DRAFT status) on every page. In addition, in any solicitation for peer reviewers, the necessity for confidentiality and the non-release of materials shall be emphasized.

b) Useful, but not critical materials that may be sent to peer reviewers include:

    1) Unless a "Delphi" type process is being undertaken, it is also useful for each peer reviewer to have the name, address, and phone, fax, and/or Internet numbers of each peer reviewer working on the specific review.

    2) Any particularly relevant scientific articles from the literature;

    3) It is often more practical to provide comments on a product that has line numbering added in the margin.

c) Peer Reviewers should be given what is needed to complete their task -- they should not be overburdened with excess material.

2. Peer Reviewer Responsibilities

The Peer Review Leader is responsible for ensuring that peer reviewers understand and comply with these responsibilities:

a) Advise the USEPA of any real or perceived conflicts-of-interest.

b) Provide written comments in specified format by the specified deadline.

c) Comply with the request for not disclosing draft work products to the public.

Back to Appendix A Contents
Back to Main Contents

IV. COMPLETING A PEER REVIEW

Performance of the formal peer review is not the final stage in the development of the work product. Rather, it is an important stage in developing the work product, with the final work product representing the true end of the peer review. As a result, the peer review process closes with three major activities: evaluating comments and recommendations, utilizing peer review comments for completing the final work product, and organizing and maintaining a record of the peer review.

A. Evaluating Comments and Recommendations

1. OPPTS must carefully evaluate and analyze all peer review comments and recommendations. As discussed in Section III., a carefully crafted charge to the peer reviewers simplifies organizing and analyzing comments. Also, any other issues that are raised need to be identified and evaluated.

2. The validity and objectivity of the comments need to be evaluated. Analyses include consultation with other experts/staff within the Office and/or Agency.

3. Comments that have significant impact on time, budgetary, and/or resource requirements need to be evaluated in consultation with management. These comments may lead to allocation of additional resources and a revised schedule for the completion of the work product.

B. The Final Work Product

1. OPPTS must utilize technically sound and responsible peer review comments and recommendations in completing the final work product. The major issues raised by the peer reviewers need to be identified in the front of the final work product, along with information on how they were used or not used in the final work product.

2. In the final work product, reference the location of the peer review record where all comments and related peer review information can be found. The peer review record should be placed in any associated established public docket in addition to the Office archive.

 

C. Maintaining the Peer Review Record

1. The Peer Review Leader will collect and maintain the following materials for the peer review record, including at least:

  • the draft work product submitted for peer review
  • materials and information given to the peer reviewer(s)
  • comments, information, and materials received from the peer reviewer(s)
  • information about the peer reviewer(s) (e.g., names, affiliations, etc.)
  • any logistical information (e.g., times; locations; duration, etc.)
  • the final work product

2. The peer review record must be indexed and maintained in an office archive.

All peer review comments should be carefully evaluated and used to revise work products where appropriate. As discussed in III. C. (above) it is particularly useful to craft specific questions for peer reviewers so that subsequent comment will be focussed. Focussing reviewers attention on issues of interest will facilitate comment evaluation.

In some cases, preparation of a document that responds to each comment may be appropriate. In other circumstances, comments may be addressed in a more general manner. In any case, a clear record must be maintained of the peer review process employed, as well as the specific comments received. Furthermore, the product itself must include some acknowledgment of the peer review process.

The Agency may or may not agree with comments received. However, it is important to reflect the nature of comments received and the extent to which these comments affect the scientific or technical product. In addition to revising the technical product, if appropriate, it is particularly important to communicate significant results of peer review to the decision maker(s) as well as to others who may not be versed in the technical discipline at hand. For example, a decision maker might find a peer review summary very helpful. The summary could present, in general terms, the scope of the peer review process and the thrust of the comments received. A brief statement discussing the kinds of comments received, how the comments were reflected in the work product, and the impact on the scientific or technical conclusion should be included. For example, a hypothetical summary might include language such as:

    Reviewers commented that the exposure analysis assumed no chemical degradation as a result of treatment in a POTW. Reviewers felt this was overly conservative and, as a result, risks may be overestimated.

    No information is available on the fate of the chemical in POTWs. However, based on analogous chemicals for which data are available, it is likely that less than 10% degradation would occur. The risk characterization has been changed to reflect risk as a range, thereby addressing the uncertainty associated with our lack of information on fate of this chemical in a POTW.

Back to Appendix A Contents
Back to Main Contents

V. ACCOUNTABILITY AND RESPONSIBILITY IN THE OFFICE

Under the June 7, 1994 Peer Review Policy, the Administrator has designated the Assistant Administrators and Regional Administrators to be accountable for implementing the Policy in their respective offices. This section provides information on OPPTS' delegations of responsibility to (a) line management for individual peer reviews, and (b) an office-wide coordination group for general assistance and advice.

A. Line Management

1. Decision maker(s)

a) The Assistant Administrator is accountable for the decisions regarding the identification of major scientific and technical work products and the mechanism(s) of peer review utilized for each of the products. The AA designates the OPPTS Division Directors, in conjunction with the Deputy Office Directors, as decision makers. The AA and their designee(s) are also responsible for ensuring that the peer reviews are performed and fully completed. See Appendix B for the names (or positions) of the designated decision maker(s).

b) Specific responsibilities of the decision maker(s) are:

  • Designating a Peer Review Leader to organize the peer review

  • Providing advice, guidance, and support to the Peer Review Leader in the preparation, conduct, and completion of the peer review

  • Establishing a realistic peer review schedule (see Section III)

  • Designating the stage(s) of product development where peer review is appropriate

  • Ensuring that the results of peer review are carried forward in the final work product

2. Peer Review Leader

a) The Peer Review Leader organizes and oversees the peer review for a specific work product. This person(s) can be the decision maker(s), but will usually be someone who is authorized by the decision maker to prepare, perform, and bring to completion the peer review. The Peer Review Leader will obtain the assistance and support of the Peer Review Coordinator and Peer Review Group (see below) as well as any others within the Agency to help perform the peer review. The Leader will be chosen on a case by case basis depending on the work product needing peer review.

b) Specific responsibilities of the Peer Review Leader are:

  • Coordinating the peer review of their assigned work product with the Peer Review Coordinator and Peer Review Group

  • Organizing, conducting, and completing the peer review following the procedures outlined in this SOP, particularly Sections III. and IV.

  • Selecting the peer reviewers in consultation with others involved with the peer review (e.g., Peer Review Coordinator)

  • Advising peer reviewers of their responsibilities

B. Office Coordination

1. Peer Review Coordinator

a) The Assistant Administrator for OPPTS will designate one person to coordinate or lead coordination of peer review activities for this office. The Peer Review Coordinator is listed in Appendix B.

b) Specific responsibilities of the Peer Review Coordinator are:

  • Leading the Peer Review Group as well as OPPTS peer review activities

  • Reporting peer review activities to the AA

  • Liaison with the Science Policy Council (SPC) and Science Advisor:

      1) Representing OPPTS before the SPC

      2) Advising the SPC of any changes in SPC-reviewed list of work products and peer review mechanisms in Appendix D

      3) Participating in SPC peer review training, workshops, etc., as requested

      4) Interfacing with the Administrator's Science Advisor on peer review matters

  • Submit information on OPPTS peer review candidates for each fiscal year on June 15 of the preceding year via Appendix D reporting

  • Submit any SOP revisions on July 1 of each year for review for incorporation in the SOP for the next fiscal year

  • Provide advice, guidance, and support to the various Peer Review Leaders for the performance of the peer reviews

  • Distribute Agency-wide peer review guidance and materials to appropriate OPPTS personnel, as requested

2. OPPTS Peer Review Advisory Group

a) Each year, the Assistant Administrator may also designate other persons and support staff to constitute the Peer Review Advisory Group. The Group will assist the Peer Review Coordinator either as special assignments or as a designated part of their normal duties. The group members and support staff are listed in Appendix B.

b) Specific responsibilities for the Peer Review Group are:

  • Assisting the Peer Review Coordinator in providing advice involving peer review activities in OPPTS

  • Assisting with logistical and technical peer review needs in the office, e.g., acquisition and distribution of resources

  • If needed, providing advice to decision maker(s) and/or help them with the identification of major work products and selection of appropriate peer review mechanism

  • Periodic review of the SOP and how it is being implemented

  • Provide advice during the planning and management of OPPTS' peer review program, e.g., integrating peer review plans into office workplans

3. Legal Advice

OPPTS staff and management work regularly with individual OGC staff assigned to Agency activities. Peer Review Leaders should continue to initially consult with their customary OGC advisors for legal advice or referral. Headquarters attorneys have specialties in specific areas and can be consulted as needed. Appendix B lists these contacts for OPPTS.

C. Budget

Both program offices within OPPTS (OPP and OPPT) have rigorous planning processes. The planning process is the appropriate forum to ensure that peer review-related activities are appropriately budgeted, and that sufficient resources will be available to effect a completed peer review.

The planning process articulates priority activities for the coming year and provides opportunities for periodic evaluation of project status, including opportunities for redirecting program priorities. The planning process facilitates development of project plans for priority projects, including identification of scientific and technical products necessary to complete priority activities.

The project planning process provides a natural forum for discussing the nature of scientific and technical products that will be developed to support various projects. This forum provides an opportunity to discuss the mechanism of peer involvement and/or peer review needed and how these peer activities will be achieved. Discussing the scope of peer review during the planing process provides the added benefit of ensuring that timing and resource requirements associated with peer review are included in the planning process and highlighted for senior management attention.

D. Annual Reviews

1. The Peer Review Coordinator will organize an annual preview of OPPTS' expected work products for the next fiscal year and submit this information to the SPC by June 15 of each year as required for Appendix D.

2. The Peer Review Coordinator will organize an annual review to assess the function of the SOP in practice. The Coordinator will consult with the SPC regarding any proposed changes by June 15 for the next fiscal year.

Back to Appendix A Contents
Back to Main Contents

 

APPENDIX A

U.S. ENVIRONMENTAL PROTECTION AGENCY PEER REVIEW POLICY

PEER REVIEW AND PEER INVOLVEMENT
AT THE U. S. ENVIRONMENTAL PROTECTION AGENCY

This document establishes the policy of the United States Environmental Protection Agency (EPA) for peer review of scientifically and technically based work products that are intended to support Agency decisions. Peer review is presented in the context of the broader concept, peer involvement.

BACKGROUND

The report "Safeguarding the Future: Credible Science, Credible Decisions" [1 EPA/600/9-91/050, March 1992] focused on the state of science at EPA. The panel of experts who prepared the report emphasized the importance of peer review, especially external peer review, and the need for broader and more systematic use of it at EPA to evaluate scientific and technical work products. Their specific recommendation regarding peer review reads as follows:

    "Quality assurance and peer review should be applied to the planning and results of all scientific and technical efforts to obtain data used for guidance and decisions at EPA, including such efforts in the program and regional offices. Such a requirement is essential if EPA is to be perceived as a credible, unbiased source of environmental and health information, both in the United States and throughout the world."

In response to this recommendation, then-Administrator Reilly directed staff to develop an EPA-wide policy statement, which he issued in January, 1993. The paragraphs below preserve the core of that earlier statement while updating it to specify the role of the Science Policy Council in guiding further implementation of the policy. Effective use of peer review is indispensable for fulfilling the EPA mission and therefore deserves high-priority attention from program managers and scientists within all pertinent Headquarters and Regional Offices.

PEER INVOLVEMENT AND PEER REVIEW

EPA strives to ensure that the scientific and technical underpinnings of its decisions meet two important criteria: they should be based upon the best current knowledge from science, engineering, and other domains of technical expertise; and they should be judged credible by those who deal with the Agency. EPA staff therefore frequently rely upon peer involvement -- that is, they augment their capabilities by inviting relevant subject-matter experts from outside the program to become involved in one or more aspects of the development of the work products that support policies and actions.

One particularly important type of peer involvement occurs when scientifically and technically based work products undergo peer review -- that is, when they are evaluated by relevant experts from outside the program who are peers of the program staff, consultants, and/or contractor personnel who prepared the product. Properly applied, peer review not only enriches the quality of work products but also adds a degree of credibility that cannot be achieved in any other way. Further, peer review early in the development of work products in some cases may conserve future resources by steering the development along the most efficacious course.

Peer review generally takes one of two forms. The review team may consist primarily of relevant experts from within EPA, albeit individuals who have no other involvement with respect to the work product that is to be evaluated (internal peer review). Or the review team may consist primarily of independent experts from outside EPA (external peer review).

POLICY STATEMENT

Major scientifically and technically based work products related to Agency decisions normally should be peer-reviewed. Agency managers within Headquarters, Regions, laboratories, and field components determine and are accountable for the decision whether to employ peer review in particular instances and, if so, its character, scope, and timing. These decisions are made in conformance with program goals and priorities, resource constraints, and statutory or court-ordered deadlines. For those work products that are intended to support the most important decisions or that have special importance in their own right, external peer review is the procedure of choice. Peer review is not restricted to the penultimate version of work products; in fact, peer review at the planning stage can often be extremely beneficial.

SCOPE

Agency managers routinely make regulatory and other decisions that necessarily involve many different considerations. This policy applies to major work products that are primarily scientific and technical in nature and may contribute to the basis for policy or regulatory decisions. By contrast, this policy does not apply to nonmajor or nontechnical matters that Agency managers consider as they make decisions. Similarly, this policy does not apply to these ultimate decisions.

This policy applies where appropriate, as determined by the National and Regional Program Managers, to major scientifically and technically based work products initiated subsequent to the date of issuance. Peer review should be employed to the extent reasonable to relevant work products that currently are under development. This policy does not apply to the bases for past decisions, unless and until the relevant scientific and technical issues are considered anew in the Agency's decision-making processes.

Except where it is required by law, formal peer review (as distinguished from the Agency's normal internal review procedures) should be conducted in a manner that will not cause EPA to miss or need extension of a statutory or court-ordered deadline. Agency managers still may undertake peer review if it can be conducted concurrently with necessary rulemaking steps.

LEGAL EFFECT

This policy statement does not establish or affect legal rights or obligations. Rather, it confirms the importance of peer review where appropriate, outlines relevant principles, and identifies factors Agency staff should consider in implementing the policy. On a continuing basis, Agency management is expected to evaluate the policy as well as the results of its application throughout the Agency and undertake revisions as necessary. Therefore, the policy does not stand alone; nor does it establish a binding norm that is finally determinative of the issues addressed. Minor variations in its application from one instance to another are appropriate and expected; they thus are not a legitimate basis for delaying or complicating action on otherwise satisfactory scientific, technical, and regulatory products.

Except where provided otherwise by law, peer review is not a formal part of or substitute for notice and comment rulemaking or adjudicative procedures. EPA's decision whether to conduct peer review in any particular case is wholly within the Agency's discretion. Similarly, nothing in this policy creates a legal requirement that EPA respond to peer reviewers. However, to the extent that EPA decisions rely on scientific and technical work products that have been subjected to peer review, the remarks of peer reviewers should be included in the decision record.

IMPLEMENTATION

The Science Policy Council is responsible for overseeing Agency-wide implementation. Its responsibilities include promoting consistent interpretation, assessing Agency-wide progress, and developing recommendations for revisions of the policy as necessary.

The Science Policy Council will oversee a peer-review work group, which will include representatives from program units throughout EPA to effect a consistent, workable implementation of the policy. The work group will assist the programs in (1) formulating and, as necessary, revising standard operating procedures (SOPs) for peer review consistent with this policy; (2) identifying work products that are subject to review; and (3) for each major work product, selecting an appropriate level and timing of peer review.

In assisting the programs, the work group will take into account statutory and court deadlines, resource implications, and availability of disinterested peer reviewers. The group will work closely with Headquarters offices and the Regional Offices toward ensuring effective, efficient uses of peer review in supporting their mission objectives. However, the Assistant Administrators and Regional Administrators remain ultimately responsible for developing SOPs, identifying work products subject to peer review, determining the type and timing of such review, documenting the process and outcome of each peer review, and otherwise implementing the policy within their organizational units.

Because peer review can be time-consuming and expensive, Agency managers within Headquarters, Regions, laboratories, and field components are expected to plan carefully with respect to its use -- taking account of program priorities, resource considerations, and any other relevant constraints as well as the policy goal of achieving high-quality, credible underpinnings for decisions. External peer reviewers should be chosen carefully to ensure an independent and objective evaluation. The affiliations of peer reviewers should be identified on the public record, so as to avoid undercutting the credibility of the peer-review process by conflicts of interest.

This policy is effective immediately. The peer-review work group mentioned above will identify the focal point to whom comments and questions should be addressed and, from time to time, will provide further information about implementation activities.

Back to Appendix A Contents
Back to Main Contents

 

APPENDIX B

 

KEY PERSONNEL

 

I. Individuals Involved in the Peer Review

A. Line Management

1. Decision maker(s): OPPTS Division Directors in conjunction with the Deputy Office Directors:

OPP Division Directors

Director, Program Management and Support Division

Director, Biological and Economic Analysis Division

Director, Biopesticides and Pollution Prevention Division

Director, Environmental Fate and Effects Division

Director, Field Operations Division

Director, Health Effects Division

Director, Registration Division

Director, Special Review and Reregistration Division

OPPT Division Directors

Director, Chemical Control Division

Director, Economics, Exposure and Technology Division

Director, Environmental Assistance Division

Director, Chemical Screening and Risk Assessment Division

Director, Chemical Management Division

Director, Health and Environmental Review Division

Director, Information Management Division

Director, Pollution Prevention Division

 

OPPTS Deputy Office Directors

Deputy Office Director, Office of Pesticide Programs

Deputy Office Director, Office of Pollution Prevention and Toxics

B. Office Coordination

2. Peer Review Coordinator

    Name: Michael Firestone
    Position: Science Advisor/OPPTS
    Telephone: (202) 260-2897
    Term of service: continuing

3. Peer Review Group

Individuals serving on the Peer Review Group in FY95:

    Chair: Susan H. Wayland
    Position: Deputy Assistant Administrator/
    OPPTS
    Telephone: (202) 260-2910
    Term of service: continuing

    Co-chair: Penelope Fenner-Crisp
    Position: Deputy Office Director/OPP
    Telephone: (703) 305-7090
    Term of service: continuing

    Co-chair: Joseph S. Carra
    Position: Deputy Office Director/OPPT
    Telephone: (202) 260-1815
    Term of service: continuing

    Member: Michael Firestone
    Position: Science Advisor/OPPTS
    Telephone: (202) 260-2897
    Term of service: continuing

    Member: Robert B. Jaeger
    Position: Designated Federal Official
    for FIFRA Scientific Advisory Panel
    Telephone: (703) 305-5369
    Term of service: continuing

    Member: Elizabeth Milewski
    Position: Designated Federal Official for
    Biotechnology Science Advisory Committee
    Telephone: (202) 260-6900
    Term of service: continuing

    Member: Amy Rispin
    Position: Senior Science Advisor/PSPS/OPP
    Telephone: (703) 308-2738
    Term of service: continuing

    Member: William T. Waugh
    Position: Deputy Director
    Chemical Screening and Risk Assessment Division
    Telephone: (202) 260-3489
    Term of service: continuing

II. Legal Advice

OPPTS works closely with OGC staff. Peer Review Leaders should continue to initially consult with their customary OGC advisers for legal advice or referral. The following individual has been identified as the OPPTS contact in OGC for matters relating to this SOP:

    Robert Perlis
    Office of General Counsel

The headquarters attorneys listed below specialize in the identified areas and may be consulted, as appropriate.

    Donald Sadowski
    Confidential Business Information
    Office of General Counsel

    Richard Feldman
    Conflicts of Interest Disclosure
    Office of General Counsel

    Hale Hawbecker
    FACA Issues
    Office of General Counsel

Back to Appendix A Contents
Back to Main Contents

 

Appendix C and D tables (203KB PDF)

 

Back to Appendix A Contents
Back to Main Contents

APPENDIX E

SOURCES FOR PEER REVIEW SERVICES

I. ACQUISITION OF PEER REVIEW SERVICES

A range of peer review services are available to the USEPA including internal, external (voluntary, purchase order, contractor employee), and Special Government Employee (SGE). The mechanism selected is generally based on the nature of the scientific or technical work product.

A. Voluntary Services

As a general matter, the USEPA can ask outside experts to peer review Agency products without compensation. Several environmental statutes authorize the USEPA to engage in cooperative activities with "institutions, organizations, and individuals." Accordingly, such groups may cooperate with the USEPA in research and informational activities related to the following statutes: Clean Air Act at 42 U.S.C. S7402; Clean Water Act at 33 U.S.C. S1254; Resource Conservation and Recovery Act at 42 U.S.C. S6981; and Safe Drinking Water Act at 42 U.S.C. S300j-l.

However, if a person or organization wishes to cooperate with the USEPA to perform some sort of voluntary activity, program officials should ask them to sign a "Visitor/Guest Worker" agreement, currently used at USEPA laboratories. This is because the Anti-Deficiency Act at 31 U.S.C. S1342 prohibits so-called "voluntary" services that could give rise to a claim for compensation. Moreover, accepting "volunteers" to fill USEPA employee "slots" might violate the personnel statutes and might give rise to a valid claim for compensation in violation of the Anti-Deficiency Act.

B. Contracts

The USEPA may contract for peer review services. The contract may be written solely for peer reviews or be included as part of an umbrella contract, which calls for performance of other tasks as well.

For assistance in preparing the necessary pre-award documents, program officials should consult The Cookbook: How to Get Contracts Awarded in EPA and Chapter 2 of the Contracts Management Manual (CMM). The following discussion identifies five key elements that should be considered in contracts for peer reviews: Statements of Work (SOWs), Advisory and Assistance Services (AAS) or Sensitive Activities, Management Controls, Identification of Peer Reviewers, and Federal Advisory Committees. Special considerations for small purchases are discussed in section I.C. Small Purchases.

1. Statement of work (SOW)

The SOW must clearly specify that the contractor is responsible for preparing peer review evaluations and set forth guidelines for the peer review of scientific or technical documents. The contractor may perform the peer review with in-house staff, subcontractors or consultants. Any guidelines for performing peer reviews to ensure soundness and defensibility must be developed by the program office and made part of the contract. The contractor would then ensure that the peer reviews adhered to the guidelines.

The SOW cannot simply define the role of the prime contractor as arranging for the services of others to perform peer reviews and logistics for meetings. Unless the prime contractor is clearly tasked with responsibility for performing peer reviews, individual peer reviewers' fees and associated travel expenses are not payable under the contract.

The USEPA may pay for the reviewer's comments or evaluation, and also for attendance at a meeting with the Agency and other reviewers to discuss the results of the peer review. If the SOW calls for the preparation of comments or an evaluation, and specifies a meeting with the Agency and other peer reviewers to discuss the results of the peer review, payment is appropriate. The peer reviewer's attendance at the meeting would then be part of contract performance.

2. Advisory and assistance services (AAS) or sensitive activities

Contracts that provide services that support or improve Agency decision-making or policy development are subject to special management controls. These services include: "...those services acquired from non-Governmental sources by contract or by personnel appointment to support or improve agency development, decision-making, management, and administration, or to support or improve the operation of management systems. Such services may take the form of information, advice, opinions, alternatives, conclusions, recommendations, training, and direct assistance." For additional information on advisory and assistance services and sensitive activities, program officials should review EPA Order 1900.2, Contracting at EPA and Chapter 2 of the CMM.

New contracts for these services require management approvals prior to issuance of the solicitation. The following thresholds have been established for approval of these justifications:

Contract Amount

Approved Concurrence By

$25,000,000 and over

Both the Program Office Assistant/ Associate Administrator (AA) or equivalent and the Deputy AA for Finance and Acquisition, Office of Administration and Resources Management

Program Office Senior Resource Official (SRO)

$5,000,000 to $24,999,999

Both the Program Office AA and Office Director Office of Acquisition Management

Program Office SRO Program AA and Office Director Office SRO

Under $5,000

Program Office AA

Program Office SRO

(See the June 30, 1993, Interim Procedures for Approval and Documentation of Procurements Involving AAS.) Note, these procedures and approval levels will be changed upon implementation of Office of Federal Procurement Policy (OFPP) Policy Letter 93-1, Management Oversight of Service Contracting in Chapter 2 of the CMM.

3. Management controls

Contracting for peer review services is permitted. However, because of the potential for improper use of these contracts, special management controls are required.

a. Inherently governmental functions (IGFs)

OFPP Policy Letter 92-1, dated September 23, 1992, describes (a) functions that are inherently governmental and must be performed only by Government employees and (b) functions that may be contracted, but so closely support Government employees in their performance of IGFs that the contract terms and performance require close scrutiny by Federal officials.

Peer reviews represent only a contractor's recommendations, advice or analysis of a document. Agency officials must make the official Agency decision regarding acceptability of the document. To ensure that Agency officials are not improperly influenced by recommendations in the peer review, management controls must be included in the contract. One possible control would be to require the peer reviewers to submit with their evaluations or comments a description of the procedures used to arrive at their recommendations; a summary of their findings; a list of sources relied upon; and make clear the methods and considerations upon which their recommendations are based. To the extent possible, the contract should set forth any guidelines or criteria for performance of the peer review. Agency officials should document their evaluations of the quality and validity of the peer review.

b. Conflict of interest (COI)

Another important factor is that the objectivity of the peer review should not be improperly influenced or undermined by the contractor performing the review. To identify and avoid or mitigate actual or potential COI, the contract should include controls. Such controls might require the contractor to report on prior and current work, and prior clients that might create COI. Other controls might include Agency review and placing limits or advance approval on future work. There should also be procedures implemented to assure that the contractor does not gain an unfair advantage in future requirements as a result of their performance of peer reviews. Program officials should consult the Contracting Officer (CO) for special contract clauses.

The EPA Acquisition Regulations (EPAAR) at 48 CFR Subpart 1509.5 generally mandates conflicts of interest solicitation provisions and contract clauses for contracts over $25,000, but makes them optional for small purchases of $25,000 or less.

See 48 CFR 1509.508(b) & (c)

48 CFR 1552.209-70, -71 & -72

Contract for peer review services: A USEPA contracting officer will include conflicts of interest solicitation provisions and contract clauses as a matter of course without involvement by the USEPA project officer, if the peer review services are obtained pursuant to a contract over $25,000. If the peer review services are subcontracted pursuant to a prime contract (over $25,000), then the prime contractor is ordinarily required to include a conflicts of interest clause substantially similar to the conflicts of interest clause in the primary contract in its subcontract to the peer reviewer.

Although the USEPA contracting officer and/or prime contractor has the primary responsibility to include the required conflicts of interest provisions/clauses, the USEPA project officer may nevertheless wish to:

1. Highlight the conflict of interest requirements in the Scope of Work for the procurement of the peer review services.

2. Develop a specific conflict of interest clause regarding the peer review at issue as a substitute to the standard conflicts of interest clause.

3. Review the solicitation/contract to make sure that the required conflicts of interest clause has been included.

Small purchase order for peer review services: Although conflict of interests requirements are optional for small purchases, they are nevertheless a good idea. Accordingly, an USEPA project officer obtaining peer review services with a small purchase order should request the purchasing agent/contracting officer to include a conflict of interest solicitation provision and contract clause in the purchase order.

    c. Confidential business information (CBI)/Privacy Act protected information and other sensitive information

When peer reviewers are not employees of the United States Government, it is unlikely that the USEPA will have authority to give reviewers access to confidential business information in the absence of consent for such disclosure by the CBI submitter. Therefore, all documents provided to non-Federal reviewers must be screened for information claimed as CBI. Even where business information has not been explicitly claimed as CBI, if it is of a kind where the submitter might be expected to object to its release, prior to release the submitter must be asked whether it wishes to assert a claim, unless the submitter has previously been informed that failure to assert a CBI claim may result in disclosure without notice. Language should be included in the contract to clearly identify any required procedures or processes prior to release of any protected information, including any requirements for confidentiality agreements, as well as limits on use and disclosure of the data by contractor personnel.

d. Personal services

Under contracts, the USEPA may not engage the peer reviewers in any improper personal services relationships, i.e., an arrangement under which contractor personnel are subject to relatively continuous supervision and direct control by an Agency official or employee. These relationships are characterized as one where the contractor employee interacts with the Agency in a manner similar to that of a federal employee.

To avoid these improper relationships, program officials should write well-defined SOWs. The SOWs should set forth the requirements in detail for work to be performed independently, including the manner in which it will be evaluated. The SOW must set forth what work is to be performed not how the work is to be performed. Technical direction may be used to clarify ambiguous technical requirements to ensure efficient and effective contractor performance, and is not considered supervision or assignment of tasks. For additional information, program officials should consult EPA Order 1901.1A, Use of Contractor Services to Avoid Improper Contracting Relationships.

4. Identification of peer reviewers

Program officials cannot interfere in a contractor's ability to perform work by "selecting" who will perform the peer review. The Federal Acquisition Regulation (FAR) governs the CO's and program officials' relationship with the contractor.

However, the CO does have the ability to review and consent to subcontractors and consultants. The contract can also specify which individuals are key personnel and include peer reviewers. The FAR and Agency implementing regulations set forth rules governing the use and replacement of key personnel. Further, the contract can require workplans for approval by the Agency, wherein the contractor will propose the peer reviewers it is considering for selection.

Program officials should identify the qualifications required to perform the review work and the criteria for technical acceptability. The USEPA may identify a pool of qualified subcontractors and consultants to the prime contractor, but cannot direct the use of any particular subcontractor or consultant.

5. Federal Advisory Committees

The Federal Advisory Committee Act (FACA) requires that the Agency develop a charter, maintain balanced membership, and hold open meetings when it establishes an "advisory committee." An advisory committee is any group established by the USEPA for the purpose of providing advice as a group to the Agency. It does not include advice coming from individual attendees at a meeting or groups established by a non-federal entity such as an EPA contractor. See, 41 C.F.R. S101-6.1004(i). If a contractor convened peer review panel will provide advice and recommendations to the USEPA as a group, then the group is subject to FACA chartering requirements.

C. Small Purchases

The acquisition of supplies or nonpersonal services from the open market and on a sole source basis when the aggregate amount involved in any one transaction does not exceed $25,000 constitutes a "small purchase". The USEPA has developed a guide entitled Small Purchases, A Guide for Program Offices, which provides basic information about small purchases and purchase orders. The guide should be used in preparing a procurement request (PR) for the purchase of peer review services under the small purchase limitation.

The same considerations in the preceding discussion on IGFs, COI, access to CBI, and personal services apply to small purchases. Normally, the Government issues a small purchase order directly to the individual peer reviewer, instead of to a prime contractor who may subcontract for performance of the peer review.

Approvals

All small purchases for peer reviews are considered AAS. The approval level for small purchase AAS is at least one organizational level above the initiating office. When award is made during the fourth quarter of the fiscal year, approval must be received from a program official at least two organizational levels above the initiating office.

Competition

The FAR requires competition for purchases in excess of $2,500. To accomplish competition, Purchasing Agents will solicit quotes from three vendors unless a requirement is justified on a sole source basis. A sole source justification must be detailed and fully describe the circumstances supporting the justification. Program officials should refer to EPAAR Subpart 1513.170-1 for more information on preparing justifications. Poor planning does not constitute a valid basis for a sole source justification. COs will make a small purchase award to the vendor with the lowest offered price.

Procurement Requests

Program Officers should include the following in all PRs for the purchase of peer reviews:

1. A fixed-price amount at or below the small purchase limitation.

2. A detailed description of the requested services, inclusive of:

    a. Total quantity per line item;

    b. Estimated unit price per line item;

    c. Total cost per line item;

    d. Specific deliverables for each line item; and

    e. Total cost of the purchase request.

3. The name, address, and phone number of three competitive sources if the value of the request is $2,500 or more.

a. Reference FAR Subpart 3.6 and Environmental Protection Agency Acquisition Regulation (EPAAR) Subpart 1503.601 regarding sources from Government employees or organizations owned and controlled by them.

b. Provide sources from small businesses if available.

4. If the request is a sole source purchase, justification must be provided in accordance with the EPAAR Subpart 1513.170-1.

II. TRAVEL

A. Contracts and Purchase Orders

Funds obligated on a contract or purchase order are available to pay for the costs of producing the peer review including the travel costs and fee of the peer reviewer.

The USEPA acquires peer reviews through small purchases issued directly to peer reviewers or through contracts with companies, which then acquire the services of peer reviewers. By issuing a purchase order or awarding a contract for a peer review, the USEPA may pay not only for the peer reviewer's comments, but also for his or her attendance at a meeting with

the Agency and other reviewers to discuss his or her comments. The scope of work of the contract must include the organization of peer reviews and indicate whether the contractor will be required to discuss a specific peer review work product with the Agency and/or with other peer reviewers. Attendance at a meeting to discuss a peer review work product would then be part of the contract's performance. Thus, the contract may serve as the mechanism to pay for a peer reviewer's fee and associated travel expenses to provide comments to the USEPA.

B. Special Government Employees

Travel and per diem expenses of experts hired as SGEs for peer review may only be paid through the issuance of invitational travel orders (5 U.S.C. '5703). These invitational travel and per diem expenses should be charged to an appropriate USEPA travel account.

Members of the SAB, SAP, and other FACA advisory committees are hired as SGEs. It is not appropriate to reimburse travel or per diem expenses of advisory committee members or other SGEs through a contract.

Jump to main content.