Jump to main content.

Quality Management Plan, March 2003

PDF Disclaimer

You will need Adobe Acrobat Reader to view PDF files. See EPA's PDF page for more information about getting and using the free Acrobat Reader.

Entering Intranet
Some links on this page are only available to EPA Intranet users only. If you are an outside contractor working for EPA, please contact your EPA representative for more information. If you are another federal agency or other party interested in EPA's web policies and procedures, please contact EPA through the "Contact Us" page on this site.

You will need Microsoft Power point view to view Power point documents. Click here to download Power Point Viewer.

Foreword

The Mine Waste Technology Program (MWTP) is housed in the Sustainable Technology Division (STD) of the National Risk Management Research Laboratory (NRMRL). As such, QA requirements specified in NRMRL's Quality Management Plan (QMP), revised March 2002, are applicable to this Program.

The Mine Waste Technology Program operates through an Inter-Agency Agreement (IAG) between EPA and the US Department of Energy (DOE). EPA is empowered to provide technical direction for the IAG and DOE provides administrative oversight to the program. As part of this agreement, EPA has assumed responsibility for QA oversight. MSE-Technology Applications (MSE-TA) is the DOE contractor who performs and documents the research activities in support of this Program. EPA MWTP personnel provide substantial oversight of MSE-TA activities through an active QA program as described in this document and provide technical direction to MSE-TA activities, through DOE.

EPA requires that MSE-TA prepare and implement a Quality Management Plan (QMP) which is consistent with the EPA QA requirements described in this document. The MSE-TA QMP is reviewed and approved by the EPA MWTP Program Manager and QA Manager and its implementation is assessed periodically.

The format of this MWTP QMP follows that of the NRMRL QMP. The purpose of this QMP is to document the specific procedures used by EPA in the implementation of the NRMRL QMP for the MWTP.

Approvals
Subhas Sikdar,
Director, STD
Roger Wilmoth,
Program Manager, MWTP
Lauren Drees,
QA Manager, STD
QA Manager, MWTP
Concurrences
Diana Bless,
EPA MWTP Project Manager
David Ferguson,
EPA MWTP Project Manager
George Huffman,
EPA MWTP Project Manager
Norma Lewis,
EPA MWTP Project Manager
Ivars Licis,
EPA MWTP Project Manager

Table of Contents

Foreword

Abbreviations and Acronyms


1.0 Management and Organization
1.1 MWTP Mission Statements
1.2 MWTP Organizational Policy on Quality Assurance
1.3 QA Organization for MWTP
1.4 MWTP QA Roles and Responsibilities
1.5 Activities, Programs, and Legislative Acts Supported by the MWTP Quality System

2.0 Quality System and Description
2.1 Graded Approach to Research Projects
2.2 Quality-Affecting Activities
2.3 Principal Components of the EPA MWTP Quality System
2.3.1 EPA MWTP Quality Management Plan
2.3.2 QAPPs
2.3.3 Assessments
2.3.4 QA Review of Research Products
2.3.5 Computer Tracking System
3.0 Personnel Qualifications and Training
4.0 Procurement of Services
4.1 Scope and Application
4.2 QA Responsibilities on IAGs
4.3 Evaluation of Quality Deliverables

5.0 Documentation and Records
5.1 EPA Records Management Policy
5.2 MWTP Program Documentation and Records
5.3 MWTP Project Documentation and Records
5.4 Research Laboratory-Specific Documentation and Records
5.5 Software Documentation and Records
5.6 EPA MWTP Quality Documentation and Records

6.0 Computer Hardware and Software

7.0 Planning

8.0 Implementation of Work Processes
8.1 QAPP Development, Review, and Approval
8.2 SOP Development
8.3 QAPP/SOP Implementation
8.4 QAPP/SOP Revisions

9.0 Assessment and Response
9.1 Types of Assessments
9.1.1 Quality Systems Audits
9.1.2 Technical Systems Audits
9.1.3 Audits of Data Quality
9.1.4 Performance Evaluations
9.1.5 Data Quality Assessments
9.2 Peer Review
9.3 Assessor Capabilities and Authority
9.4 MWTP Assessment and Response Procedures
9.5 "Stop Work" Order
9.6 Assessment Reports and Corrective Action

10.0 Quality Improvement

11.0 References

Attachment A: QAPP Requirements

Attachment B: Quality Assurance Planning Document Form

Abbreviations and Acronyms

ADQ Audit of Data Quality
ANSI American National Standards Institute
DOE Department of Energy
DQA Data Quality Assessment
DQI Data Quality Indicator
DQO Data Quality Objective
EPA US Environmental Protection Agency
IAG Interagency Agreement
NRMRL National Risk Management Research Laboratory
OEI Office of Environmental Information
OIRM Office of Information Resources Management
ORD Office of Research & Development
PE Performance Evaluation
QA Quality Assurance
QAM Quality Assurance Manager
QAPP Quality Assurance Project Plan
QC Quality Control
QMP Quality Management Plan
QSA Quality System Audit
RMO Records Management Office
SOP Standard Operating Procedure
STD Sustainable Technology Division
TSA Technical System Audit

Table of Contents

1.0 Management & Organization

The Mine Waste Technology Program (MWTP) resides in the Industrial Multimedia Branch (IMB) of the Sustainable Technology Division (STD) of the National Risk Management Research Laboratory (NRMRL). Subhas Sikdar, the STD Division Director, is responsible for all MWTP activities. Roger Wilmoth, the IMB Branch Chief, serves as the EPA MWTP Program Manager. Lauren Drees, the STD QA Manager, serves as the EPA MWTP QA Manager. Several members of the IMB technical staff serve as EPA MWTP Project Managers. EPA MWTP Project Managers include Diana Bless, David Ferguson, George Huffman, Norma Lewis, and Ivars Licis.

1.1 MWTP Mission Statement
The mission of MWTP is to provide engineering solutions to national environmental issues resulting from the past practices of mining and smelting metallic ores. In accomplishing this mission, MWTP develops and conducts a program that emphasizes treatment technology development, testing and evaluation at bench- and pilot-scale, and an education program that emphasizes training and technology transfer. Evaluation of the treatment technologies focuses on reducing the mobility, toxicity, and volume of waste; implementability; short- and long-term effectiveness; protection of human health and the environment; community acceptance; and cost reduction.

1.2 MWTP Organizational Policy on Quality Assurance
The MWTP policy on quality assurance (QA) is:

Mine Waste Technology Program Organization Chart Including QA Personnel

Figure 1. MWTP Organization Chart Including QA Personnel.

1.3 QA Organization for MWTP
The QA organization for the MWTP is reflected in Figure 1. The EPA MWTP QA Manager (QAM) reports to the STD Division Director and is independent of MWTP activities. The EPA MWTP QA Manager works closely with the EPA MWTP Program Manager and Project Managers to ensure quality requirements are met.

1.4 MWTP QA Roles and Responsibilities
MWTP Program Manager is responsible for:

STD Division Director is responsible for:

EPA MWTP Project Managers are responsible for:

EPA MWTP QA Manager is responsible for:

1.5 Activities, Programs and Legislative Acts Supported by the MWTP Quality System
The MWTP quality system supports a wide variety of technical activities and programs. These include:

Table of Contents

2.0 Quality System & Description

2.1 Graded Approach to Research Projects
Quality Assurance Project Plans (QAPPs) must be prepared to document QA/QC requirements for all MWTP research projects as for all NRMRL projects.

The NRMRL quality system "graded approach" is used for MWTP projects. Under the graded approach, the intended use of the data dictates the required level (or category) of quality. The graded approach utilizes four (4) QA categories for research projects. The four QA categories are:

For both Category I and Category II projects, EPA MWTP follows, in its entirety, the R-5 document for QAPP preparation developed by EPA's Quality Staff (QS) (http://www.epa.gov/quality/qs-docs/r5-final.pdf). For Category III and IV projects, a subset of applicable R-5 requirements is utilized. The requirements used for Category III (applied research) and Category IV projects (basic research) are presented in Attachment A. QAPP requirements are also presented in Attachment A for projects involving the use of secondary data, sampling and analysis, and modeling and software development. (In some cases, these types of projects may be designated Category I or II; additional requirements in R-5 would then be applicable.) As guidance, the requirements in Attachment A include short descriptions of the applicable types of research.

The required quality level (or category) and associated QAPP requirements are determined by the EPA MWTP Project Manager at the beginning of a project. The EPA MWTP QA Manager should be consulted, if needed.

2.2 Quality-Affecting Activities
NRMRL's data generating research and testing activities always require QA oversight. However, for the MWTP, NRMRL's responsibilities involve the oversight of the activities performed by or for MSE-TA. It is important that EPA ensure that quality requirements are met during the planning, implementation, and reporting of MWTP research. Additionally, MSE-TA activities include other functions that may not always be attributed to project-specific efforts, such as the use of in-house data and software systems and the procurement of quality items and services. EPA ensures that these quality-affecting activities are addressed in the MSE-TA MWTP Quality Management Plan (QMP).

2.3 Principal Components of the EPA MWTP Quality System
The following are the principal components of the EPA MWTP quality system.

2.3.1 EPA MWTP Quality Management Plan
This EPA MWTP Quality Management Plan documents the EPA MWTP quality system and encompasses the management and technical activities necessary to plan, implement, and assess the effectiveness of QA and QC operations applied to all MWTP research projects. The EPA MWTP Program Manager is responsible for the development of the EPA MWTP QMP, although its preparation is normally delegated to the EPA MWTP QA Manager. The QMP must be approved and signed by the STD Division Director, the EPA MWTP Program Manager, and the EPA MWTP QA Manager. In order to ensure that EPA MWTP Project Managers understand MWTP requirements, their concurrence signatures are required. The EPA MWTP QMP is reviewed annually by the EPA MWTP QA Manager and updated as needed. The approved QMP is valid for five years unless there is a significant change in the organization or in the procedures used. After five years or a significant change, the QMP is reviewed and revised as needed and subjected to the same approval processes described above.

2.3.2 QAPPs
QAPPs are prepared by MSE-TA/Montana Tech as described in Section 2.1. The QAPPs are reviewed and approved by the EPA MWTP Project Manager and QA Manager. Data collection activities should not begin until these approvals are obtained.

2.3.3 Assessments
Assessments are scheduled based on required periodic review, the importance of or public interest in the project or Program, problems in the measurement system (noted by the Project Manager or QA Manager), requests by the Project Manager or Program Manager, or selection by the QA Manager. The assessments planned by the Project Manager at the beginning of a project are specified in the QAPP. Specific requirements regarding assessments are presented in Section 9.0.

2.3.4 QA Review of Research Products
Final product reviews are conducted to evaluate the credibility of data, realization of project goals, comparability of data, validity of conclusions, and quality of data. MWTP reports should be subjected to a technical review by the EPA MWTP Project Manager and a quality review by the QA Manager prior to publication.

2.3.5 Computer Tracking System
A computer tracking system for the various elements of the quality system is in place in STD and is maintained by the QA Manager. This system tracks funding packages, QAPPs, QMPs, SOPs, audits, and project reports.

Table of Contents

3.0 Personnel Qualifications & Training

The training requirements specified in the NRMRL QMP are applicable to MWTP activities. It is the policy of NRMRL that all persons managing or directing NRMRL projects have appropriate training for their assigned work. Training needs may be identified by EPA regulation (e.g., requirements for project officer training), by those persons planning a technical activity (e.g., TLP), or by those managing persons performing a technical activity (e.g., branch chiefs or team leaders). Training programs are generally designed by the persons providing the training.

The process for specifying and documenting intramural training varies with the type of training:

Technical Training - Documentation of technical training may be an appropriate degree in an area of study. Other documentation of technical training may include professional certifications (e.g., Professional Engineer or PE) or certification of specialized training (e.g., classes in computer applications or instrument operation as provided by technology vendors). Other specialized training requirements established by the project officer/principal investigator/TLP for a given NRMRL project may be described in the QAPP document or facilities manual. General technical training needs for branch, facility, or team members may also be identified by NRMRL managers. Each employee is responsible for submitting his/her training records to the Division's Human Resources Management representative who keeps a record of each NRMRL employee's formal technical training.

Project Management Training - For extramural project management, NRMRL adheres to the Agency's requirements regarding initial and ongoing training, including requirements for training as Project Officer, Work Assignment Manager, and/or Contract Administrator. The Human Resources Management Division keeps records of project management training for NRMRL personnel. A certification that the designated EPA extramural project manager or work assignment manager has the appropriate management training is also required as part of each funding package.

Quality System Training - Training in NRMRL's Quality System policies and procedures is planned and performed by NRMRL's QA staff. NRMRL has a Policy requiring mandatory training in NRMRL's quality system for all NRMRL management and technical personnel. The QAMs will retain copies of the training certificates. Additional training needs are usually identified by the QA staff through interactions with NRMRL technical and managerial staff. Quality training is generally designed to: (1) inform NRMRL staff of new QA policies and procedures, (2) refresh and remind NRMRL staff of current QA policies and procedures, or (3) describe services provided by the NRMRL QA staff
(metrology, document review, technical assistance).

Safety Training - EPA specifies that personnel involved in laboratory or field activities receive appropriate safety training. The type and amount (hours) of training depend on the specific assignment and the nature of the potential hazards. Training requirements are specified by line managers and compliance with training requirements is monitored by Division/locality Safety Officers and other health and safety office personnel.

Records Management Training- Every EPA employee has responsibility for records management. Each employee is responsible for creating records necessary to document their activities and actions, filing records for safe and efficient retrieval, and disposing of records only in accordance with agency directives and Federal regulations.

To assist employees and contractors with their records management responsibilities, each location has a Records Management Officer (RMO) who will assist personnel with the implementation of the agency's National Records Management Program (NRMP). Further training and assistance are available online at http://intranet.epa.gov/records/training/index.htm. EPA Intranet Site

Table of Contents

4.0 Procurement of Services

NRMRL and EPA policies require that parties to assistance agreements shall have in place a quality system consistent with EPA requirements. Assistance agreements include grants, cooperative agreements, and interagency agreements (IAGs). Other extramural agreements covered by this policy include Cooperative Research and Development Agreements (CRADAs). Compliance with this policy is achieved by including appropriate written requirements into agreement documents.

4.1 Scope and Application
The EPA MWTP Program utilizes an interagency agreement (IAG) with the U.S. DOE. For IAGs, funding actions include the initial award funding and any modifications that add additional funds. For each funding action, the EPA MWTP Program Manager (or designee) must incorporate a completed Quality Assurance Planning Requirement (QAPR) form into the funding package (see Attachment B). The completed package is reviewed by the EPA MWTP QA Manager to ensure that appropriate requirements are included.

Typically, the QAPR form will include the requirement for a QMP, as well as QAPPs for each project. All QAPP types will be marked as possibly being required during the course of the IAG. The requirements for QAPPs (see Attachment A) are included in the IAG package.

4.2 QA Responsibilities on IAGs
If work on a project will be performed by EPA and one or more other organizations, the responsibility for QA must be negotiated within the agreement. The Program Manager, in consultation with the QA Managers in the various organizations must agree on and document (in the agreement and on a QAPR Form for Initial Extramural Actions) which organization will take the lead for QA, the specific person from the organization who will be responsible, and the QA requirements that will be adhered to during the agreement. For the MWTP, EPA assumes the lead for quality and establishes the quality requirements.

4.3 Evaluation of Quality Deliverables
In order to ensure that MSE-TA provides quality services that satisfy MWTP QA/QC requirements, the MWTP QA Manager and Project Managers review QAPPs, SOPs, and final products, and audit projects and programs. Review/audit comments generated by the QA Manager (or designee) are provided to the Project Manager who is ultimately responsible for the quality of services provided.

Table of Contents

5.0 Documentation & Records

The following is a discussion of MWTP procedures for documents and records, including computer-resident records. Records shall be maintained by the Project Manager so that all MWTP activities can be reconstructed. MWTP shall comply with EPA Records Management Policy.

5.1 EPA Records Management Policy
EPA records management policy is established by Office of Information and Resources Management and documented in the EPA Records Management Manual 2160. This procedure applies to all records, as defined under the Federal Records Act (44 U.S.C. 3101), regardless of media (including paper, microform, electronic, audiovisual, and record copies of Agency publications). All locations of NRMRL comply with EPA records management policy.

The management process that ensures that records accurately reflect completed work and/or fulfill statutory and contractual requirements, including any specific record keeping requirements is defined in EPA Order 2160 and EPA Directive 2100, Chapter 10. This document defines requirements and responsibilities for record transmittal, distribution, retention, protection, preservation, traceability, disposition, and retrievable and includes the roles and responsibilities for management and staff. Following this EPA Order ensures compliance with all statutory, contractual, and assistance agreement requirements for records from environmental programs and provides adequate preservation of key records necessary to support the mission of the organization. NRMRL has a designated Records Management Officer and each Division has a person designated as the Records Liaison.

5.2 MWTP Program Documentation and Records
The EPA MWTP Program Manager is responsible for maintaining the following records:

5.3 MWTP Project Documentation and Records
Documentation developed on a project-specific basis is the responsibility of the EPA MWTP Technical Project Monitors and is subject to EPA records management policy. The EPA MWTP Technical Project Monitors are responsible for maintaining the following records:

When a project is completed as determined by the Project Manager, the project file is archived. This is accomplished by the Project Manager in conjunction with the Division's Records Liaison.

5.4 Research Laboratory-Specific Documentation and Records
NRMRL requires that documentation and records that may not be directly applicable to individual projects, (e.g., refrigerator temperature logbooks, instrument logbooks, standard preparation logbooks, SOPs, balance calibration logbooks) also be maintained to support project results. EPA MWTP ensures that MSE-TA MWTP documentation (QMP, QAPPs) includes requirements for the generation and maintenance of this documentation.

5.5 Software Documentation and Records
NRMRL is required to maintain software quality documentation and records. EPA MWTP ensures that MSE-TA MWTP documentation (QMP, QAPPs) includes requirements for the generation and maintenance of this documentation.

5.6 NRMRL Quality Documentation and Records
The EPA MWTP QMP shall be controlled by the STD QA Office. Although the QA Manager may keep copies, records of reviews of QAPPs, SOPs, and final reports are the responsibility of the Project Manager.

Table of Contents

6.0 Computer Hardware & Software

NRMRL, including MWTP, follows guidance published by the Office of Information Resources Management (OIRM) to ensure that computer hardware and software used in environmental programs meets technical requirements and quality expectations. Specific guidance includes:

EPA Directive 2100, Information Resources Management Policy Manual
(http://www.epa.gov/irmpoli8/), and

EPA Directive 2182, System Design and Development Guidance
(http://www.epa.gov/irmpoli8/).

These comprehensive guidance documents address many issues regarding the use of computer systems, including development of software, design of databases, records management, security, and data standards.

EPA MWTP ensures that the MSE-TA QMP adequately addresses technical and quality requirements for computer hardware and software.

Project-specific requirements for hardware configurations and for configuration control are specified in specific project QAPPs. Project-specific requirements and plans for software testing, validation, and documentation are also described in the QAPPs for those projects.

Table of Contents

7.0 Planning

It is the policy of NRMRL that systematic planning shall occur for all NRMRL research projects, including MWTP projects. Planning documents (including QAPPs) appropriate to the scope are developed.

NRMRL has established minimum requirements for various categories of QAPPs (see Attachment A). These requirements are also applicable to MWTP research. For projects that require original environmental data collection and/or sampling and analysis or use of secondary data, a graded approach (See Section 2) is in place. The overall intent is to allow flexibility, while still meeting NRMRL's interpretation of Agency policy for QA.

Effective quality planning requires clear identification of project goals and intended use of data. The EPA MWTP Project Manager, in consultation with project participants, determines the appropriate QAPP requirements to be used. The EPA MWTP Program Manager and/or QA Manager can be consulted if necessary.

It is the EPA MWTP Project Manager's responsibility to ensure that MSE-TA identifies and involves any and all appropriate sponsoring organizations, responsible official(s), project personnel, stakeholders, scientific experts, etc. (e.g., all customers and suppliers) in the planning of the project. This includes research performed by Montana Tech under contract to MSE-TA. Once the planning is complete, project documentation should include (at a minimum) a complete description of the following:

Once all project planning is complete and documented, it is the EPA MWTP Project Manager's responsibility to submit the QAPP to the EPA MWTP QA Manager. The QA Manager performs a review against QAPP requirements and issues documentation to indicate whether planning requirements have been met and an assessment of whether project goals can be met. (If deemed necessary by the QA Manager, a statistical review of the experimental design may also be requested. The QA Manager typically arranges for this review.) The documentation from the QA Manager shall include either an approval or a non approval. In the case of non-approval, detailed review comments shall be provided to the Project Manager. Resolution of all findings shall be accomplished and documented before any research is started.

Table of Contents

8.0 Implementation of Work Processes

Once proper planning has been performed (as described in Section 7), it is important that the implementation of the required procedures be done. It is the responsibility of the EPA MWTP Project Manager to ensure that the required procedures are properly implemented by MSE-TA/Montana Tech and that the necessary reviews are performed.

8.1 QAPP Development, Review, and Approval
MSE-TA/Montana Tech develops a QAPP based on NRMRL guidance (see Section 2). The QAPP must be reviewed and approved by the EPA MWTP Project Manager and QA Manager before the project begins. QA comments are documented in a memo to the EPA MWTP Project Manager, with copies to the EPA MWTP Program Manager, and the MSE-TA/Montana Tech Project Manager and QA Manager, as applicable. The EPA MWTP Project Manager has final approval of the QAPP, once he/she determines that all technical and QA comments are adequately addressed.

8.2 SOP Development
In some instances, procedures for sampling, analysis, or other quality activity (e.g., standards preparation) are routine and are used for more than one project. Oftentimes, it is necessary that more than one person perform a procedure. In these cases, an SOP should be written. SOPs are also recommended for special or critical operations. SOPs can make the development of QAPPs more efficient, since they can be attached or referenced. The need for the development of an SOP is determined by the EPA MWTP Project Manager. Project-specific SOPs are reviewed by the EPA MWTP Project Manager and QA Manager during QAPP review. EPA MWTP ensures that SOP preparation procedures are adequately described in the MSE-TA QMP.

8.3 QAPP/SOP Implementation
It is the responsibility of the EPA MWTP Project Manager to ensure that the required procedures specified during the planning process for a specific project are implemented during data collection activities. This is done through regular discussions with MSE-TA/Montana Tech personnel, as applicable, by personally observing the procedures being performed, or by requesting that the EPA MWTP QA Manager perform an audit of the project activities outlined in a QAPP and/or SOP. In any case, deviations from project requirements must be documented, along with the corrective action performed. Corrective action must be performed as soon as possible to minimize any effect on data quality. If data quality is affected by any deviations, this must be discussed in any project report/paper.

8.4 QAPP/SOP Revisions
During the course of a project, it may be necessary that QAPPs and/or SOPs be revised. Revisions are required whenever a significant change in the plan or procedure occurs. Over time, it may also be necessary to revise QAPPs and/or SOPs to ensure they are still applicable for the work being performed. For long-term projects, QAPPs and project-related SOPs need to be reviewed on a yearly basis for long term projects. Revisions (as necessary) in the form of an addendum or fully revised document must be made as soon as a significant change is identified. The EPA MWTP Project Manager is responsible for ensuring that the required revisions are performed.

Significant revisions to a QAPP/SOP must be reviewed and approved by the EPA MWTP QA Manager. Copies of the most current document must be made available to the appropriate project/Program personnel. All previous document versions must be archived.

Table of Contents

9.0 Assessment & Response

NRMRL conducts a variety of assessments at the NRMRL-wide, Division, and project-specific level to provide an increased understanding of the program or system being examined, and to provide a basis for improving such programs or systems. The following discussion identifies the assessments which are performed by or for EPA in support of the MWTP. The required frequencies, the assessor capabilities, and the procedures for documenting and responding to assessments is also presented.

9.1 Types of Assessments
The following assessments are performed in support of the MWTP.

9.1.1 Quality Systems Audits
A quality systems audit (QSA) is an on-site review of the implementation of an organization's quality system as documented in the organization's approved QMP. This review is used to verify the existence of, and evaluate the adequacy of, the internal quality system.

The EPA MWTP QA Manager will perform an internal QSA of the MWTP Program every three years to assess the implementation of this QMP. The EPA MWTP QA Manager will perform an independent QSA of the MSE-TA quality system every three years to assess the implementation of the MSE-TA QMP.

9.1.2 Technical Systems Audits
A technical systems audit (TSA) is a qualitative on-site evaluation of sampling and/or measurement systems. The objective of the TSA is to assess and document acceptability of all facilities, maintenance, calibration procedures, reporting requirements, sampling and analytical activities, and quality control procedures. Normally, an approved QAPP provides the basis for the TSA. TSAs are most often scheduled by the Project Manager and conducted by the QA Manager (or designee). Assistance for TSAs is available from QA support contractors. TSAs are most useful when conducted early in the life cycle of a project when corrective actions (if necessary) can be performed that will minimize any loss of data. However, a TSA can be performed any time during a project's life cycle.

For MWTP, TSAs are required for all Category I & II projects. For Category III & IV projects, at a minimum, the EPA MWTP QA Manager will perform two TSAs each year. The identification of the projects which will be audited will be made by the EPA MWTP Program Manager.

9.1.3 Audits of Data Quality
An audit of data quality (ADQ) is an examination of data after they have been collected and verified by project personnel. Assessing whether the Data Quality Indicator (DQI) goals specified in the QAPP were met requires a detailed review of the recording, transferring, calculating, summarizing, and reporting of the data. ADQs will be performed by the EPA MWTP QA Manager, or designee, if requested by the EPA Project Manager or Program Manager.

9.1.4 Performance Evaluations
A performance evaluation (PE) is a quantitative evaluation of a measurement system. Although each measurement in a test program could be subjected to a performance evaluation, the critical measurements (designated in the QAPP) are more commonly evaluated. An evaluation of a measurement system usually involves the measurement or analysis of a reference material of known value or composition. The value or composition of reference materials must be certified or verified prior to use, and the certification or verification must be adequately documented. Ideally, the identity of the reference material is disguised so that the operator or analyst will treat the material no differently than a test program sample. PEs will be submitted to MSE-TA by the EPA MWTP QA Manager, or designee, if requested by the EPA Project Manager or Program Manager.

9.1.5 Data Quality Assessments
A Data Quality Assessment (DQA) is a scientific and statistical evaluation to determine if validated data obtained from environmental data operations are of the right type, quality, and quantity to support their intended use. The DQA process is described in Guidance for Data Quality Assessment, Practical Methods for Data Analysis, EPA QA/G-9, QA97 Update, EPA/600/R-96/084, July 2000, available at http://www.epa.gov/quality/qa_docs.html. For MWTP projects, these assessments are normally conducted by the MSE-TA Project Manager, and they should be performed routinely on all projects. The EPA MWTP Project Manager should verify that these data quality assessments are performed.

9.2 Peer Review
In NRMRL, peer review is a separate function from QA. All MWTP final reports must be subjected to peer review during the EPA clearance process. Two independent reviews are required. Reviewers should not be involved in the specific project being reviewed, and should be technically knowledgeable in the subject matter. The EPA MWTP Project Manager is responsible for ensuring that these reviews are performed and that peer review comments are adequately addressed.

9.3 Assessor Capabilities and Authority
Assessors should be thoroughly familiar with QA practices, policies, and procedures. The optimal assessment team consists of a QA professional, a person with expertise in the technical area being evaluated, and, if there are complex issues regarding experimental design or data analysis, a statistician. In many cases, a QA staff member can fill more than one role.

Assessors evaluate MWTP projects under the authority of and with the permission of the EPA MWTP Project Manager and Program Manager. Assessors do not have the direct authority to change project procedures or to alter project goals. Their role is to provide information to the Project Manager as to the best practices and to clearly point our errors observed in the implementation of the QAPP. These errors may be in technical procedures, in data archival or analysis, in project management or reporting, or in any aspect of a project that impacts the data quality.

9.4 MWTP Assessment and Response Procedures
The EPA MWTP QA Manager is responsible for planning, scheduling, and conducting assessments, as required by this QMP or requested by MWTP management. The EPA MWTP QA Manager may be assisted in the performance of assessments by QA support contractors. At the conclusion of each assessment, the assessment report shall include a description of the type of corrective actions required to resolve any findings of nonconformance to MWTP quality policy as specified in this QMP and project-specific documents.

9.5 "Stop Work" Order
QA staff members are often on-site and able to interact directly with project personnel. If a QA staff member observes work practices which could have serious adverse impacts on data quality, the QA staff member should promptly notify the EPA MWTP Project Manager.

9.6 Assessment Reports and Corrective Action
For QSAs and TSAs, on-site assessments are followed up by a debriefing to the auditee and EPA MWTP Project Manager (if available) at the end of the on-site portion of the evaluation. If the EPA MWTP Project Manager is not present at the debriefing, then assessments are also followed up by a verbal or brief written summary to the EPA MWTP Project Manager on the first working day after the completion of the on-site portion of the audit.

The general format for assessment reports is:

Assessment reports are generated by the assessor within three business days after completion of the assessment. Reports are sent as drafts to the EPA MWTP Project Manager and EPA MWTP QA Manager (if not the assessor) for review. After finalization of the report by the EPA MWTP QA Manager, the final assessment report is submitted to the EPA MWTP Project Manager, the EPA MWTP Program Manager, and the auditee.

It is the responsibility of the EPA MWTP Project Manager to ensure that MSE-TA and/or Montana Tech adequately resolve audit findings. If findings cannot be resolved by the EPA MWTP Project Manager, the EPA MWTP Program Manager's assistance is requested. The file on a given audit is maintained in active status until all findings are resolved and documented. Audit reports are then archived in the EPA MWTP Project Manager's project files.

Table of Contents

10.0 Quality Improvement

All of the policies and procedures described in this manual were developed to document QA planning for the MWTP. Two levels of quality improvement processes combine to evaluate the effectiveness of the NRMRL quality system and ensure continual quality improvement. At the QA Program level, QSAs discussed in Section 9.4 are performed by the EPA MWTP QA Manager. Any problems or areas where quality improvement can be made are identified and reported to the EPA MWTP Program Manager, who is responsible for implementing the necessary corrective action. At the Project QA level, the document reviews and audits discussed in Sections 8.0 and 9.0 are conducted to evaluate proposed procedures and their implementation. Any problems or areas where quality improvement can be made are identified and reported to the EPA MWTP Project Manager, who is responsible for implementing the necessary corrective action. In addition, all project personnel are responsible for quality improvement activities with respect to their particular roles.

Finally, it is the responsibility of the designated EPA MWTP QA Manager to monitor the quality procedures which are implemented on a day-to-day basis. This is done via audits, discussions with project managers, and involvement in the implementation of the quality system.

If possible, the cause of any identified problem will be determined before corrective action is determined. The corrective action shall be planned, documented, agreed upon, and implemented to minimize the effect on program/project quality.

Table of Contents

11.0 References

NRMRL Quality Management Plan, March 2002.

EPA Requirements for Quality Management Plans, EPA QA/R-2, EPA /240/B-01/002, March 2001,OEI, Washington, D.C.

Guidance for the Data Assessment, Practical Methods for Data Analysis, EPA QA/G-9, QA00 Update, EPA /600/R-96/084, July 2000, OEI, Washington, D.C.

Table of Contents

Attachment A: Planning Document Requirements and Guidance

For Category I (enforcement, litigation, or projects involving human subjects) and Category II (development of environmental regulations or standards), NRMRL's requirements are those listed in EPA Requirements for Quality Assurance Project Plans for Environmental Data Operations, EPA QA/R-5, EPA/240/B-01/003, March 2001. Guidance for Quality Assurance Project Plans, EPA QA/G-5, EPA/600/R-98/018, February 1998 may be used to help address these requirements. (These documents are available at http://www.epa.gov/quality/)

QAPP requirements for Category III (Applied Research) and Category IV (Basic Research) are presented in this appendix. Guidance for Quality Assurance Project Plans, EPA QA/G-5, EPA/600/R-98/018, February 1998 referenced above may be used to help address the requirements listed for Category III and IV QAPPs; however, only the requirements listed in this appendix need to be addressed.

QAPP requirements for Sampling and Analysis Projects, Secondary Data Research Projects, Methods Development Projects, and Software and Data Management Projects are also presented in this appendix. Requirements for modeling projects are currently under development. (See your QAM if you need these requirements.)

For projects involving design, construction, and/or operation of environmental technology, the requirements in Part C of "Specifications and Guidelines for Quality Systems for Environmental Data Collection and Environmental Technology," ANSI/ASQC E4-1994, American Society for Quality Control, Milwaukee, WI, January 1995 will be followed.

The requirements for QMPs submitted to NRMRL in response to solicitations are also presented in this appendix.

Table of Contents

QAPP Requirements for Sampling and Analysis Projects

A sampling and analysis activity or project is typically defined as a study performed to generate data to either monitor parameters on a routine basis or to characterize a particular population for later studies. The following requirements should be addressed as applicable.

Section 1.0, Project Description and Organization

1.1 The purpose of the study shall be clearly stated in the sampling and analysis plan (SAP).

1.2 Responsibilities and points of contact for each organization shall be identified in the SAP. This should include identification of key personnel and/or organization(s) responsible for sample collection and custody, analytical and/or process measurements, data reduction, report preparation, and quality assurance.

Section 2.0, Sampling

2.1 Sampling points for all measurements (i.e., analytical, physical, and process, including locations and access points) shall be identified in the SAP whenever possible. If the specific locations cannot be identified at the time of plan generation, discuss the documentation and/or communication mechanism(s) for ensuring adequate information is captured to later identify sampling points.

2.2 The anticipated sampling frequency (e.g., how many sampling events and how often events occur), the number of sample types (e.g., metals, VOCs, SVOCs, etc.), and the minimum number of samples of each type taken at each event shall be provided.

2.3 The expected measurements (i.e., specific analytes) planned for each sample type shall be summarized.

2.4 If applicable, known site-specific factors that may affect sampling procedures shall be described.

2.5 If applicable, any site preparation (e.g., sampling device installation, sampling port modifications) needed prior to sampling shall be described.

2.6 Each sampling procedure (including a list of equipment needed and the calibration of this equipment as appropriate) to be used shall be discussed or referenced. Maintenance requirements/procedures (as appropriate) must also be addressed in this section.

2.7 If compositing or splitting of samples is planned, the applicable procedures shall be described.

2.8 A list of sample quantities to be collected, and the sample amount required for each analysis, including QC sample analysis, shall be specified.

2.9 Containers used for sample collection, transport, and storage for each sample type shall be described.

2.10 Sample preservation methods (e.g., refrigeration, acidification, etc.) shall be described.

2.11 Requirements for shipping samples shall be described.

2.12 Holding times requirements shall be noted.

2.13 Procedures for tracking samples in the laboratory and for maintaining chain-of-custody when samples are shipped shall be described. COC procedures shall be described to ensure that sample integrity is maintained (labeling, seals, records).

2.14 Information to be recorded and maintained by field personnel shall be discussed.

Section 3.0, Testing and Measurement Protocols

3.1 Each analytical method to be used shall be referenced. This includes EPA-approved and other validated nonstandard methods.

3.2 If applicable, modifications to EPA-approved or other validated nonstandard methods shall also be described.

Section 4.0, QA/QC Check

4.1 The SAP shall list and define all calibrations and QC checks and/or procedures used for the project, both field and laboratory as needed.

4.2 For each specified calibration, QC check, or procedure, required frequencies and acceptance criteria shall be included.

Section 5.0, Data Reduction and Reporting

5.1 Data reduction procedures specific to the project, and also specific to each organization, shall be summarized.

5.2 The reporting requirements (e.g., units, reporting method [e.g., wet or dry]) for each measurement and matrix shall be identified.

Section 6.0, Reporting Requirements

The deliverables expected from each organization responsible for field and/or analytical activities shall be described.

Table of Contents

QAPP Requirements for Basic Research Projects

A basic research project is a study performed to generate data used to evaluate unproven theories, processes, or technologies.

Section 1.0, Project Objectives and Organization

State the project objectives.

1.2 Identify the responsibilities of all project participants (e.g., QAPP preparation, sample collection and analyses, data reduction/validation/analysis, report preparation, QA).

Section 2.0, Experimental Approach

2.1 Describe the process, site, facility, apparatus, and/or environmental system to be tested.

2.2 Describe all known or pre-established test conditions and variables, including replicate experimental runs.

2.3 Describe the planned approach (statistical and/or non-statistical) for evaluating project objectives (i.e., data analysis).

Section 3.0, Sampling and Measurement Approach and Procedures

3.1 Complete a table similar to the following to summarize the experimental sampling strategy to be used.

Sample/Measurement Location
Matrix
Measurement
Frequency
Experimental QC1
Total No. Samples
           
           

1QC samples generated during experiment, as applicable (e.g., blanks, replicate samples, spikes)

3.2 Complete a table similar to the following to summarize the experimental sampling and analytical procedures to be used.

Matrix
Measurement
Sampling / Measurement Method1
Analysis Method1
Sample Container / Quantity of Sample
Preservation / Storage
Holding Time(s)2
             
             

1Provide details in text, as necessary, if standard method or SOP cannot be referenced
2Both to extraction and analysis, if applicable

Section 4.0, QA/QC Checks

Complete a table similar to the following to summarize QA/QC checks.

Matrix
Measurement
QA / QC Check1
Frequency
Acceptance Criteria
Corrective Action
           
           

1 Include all QA/QC checks (experimental and analytical, as applicable) for accuracy, precision, detection limits, mass balance, etc. (e.g., matrix spikes, lab control samples, blanks, replicates, surrogates)

Section 5.0, Data Reporting

Describe data reduction procedures specific to the project.

Section 6.0, References

Provide references to methods and germane prior publications. In addition, when applicable:

QAPP Requirements for Applied Research Projects

An applied research project is a study to demonstrate the performance of technologies under defined conditions. These studies are often pilot- or field-scale. The following requirements should be addressed as applicable.

Section 0.0, Distribution List
A distribution list shall be provided to facilitate the distribution of the most recent current version of the QAPP to all the principal project participants.

Section 1.0, Project Description and Objectives

1.1 The purpose of study shall be clearly stated.

1.2 The process, site, facility, and/or environmental system to be tested shall be described.

1.3 Project objectives shall be clearly stated and identified as primary or non-primary.

Section 2.0, Project Organization

2.1 Key points of contact for each organization involved in the project shall be identified.

2.2 All QA Managers and their relationship in the organizations (i.e., location within each organization) shall be identified with evidence that the QA Manager is independent of project management.

2.3 Responsibilities of all other project participants and their relationship to other project participants shall be identified, meaning that organizations responsible for planning, coordination, sample collection, sample custody, measurements (i.e., analytical, physical, and process), data reduction, data validation, and report preparation shall be clearly identified.

SECTION 3.0, EXPERIMENTAL APPROACH

3.1 The general approach and the test conditions for each experimental phase shall be provided. The statistical methods that will be used to evaluate the data (i.e., ANOVA, or summary statistics) should be identified. (NOTE: As deemed appropriate to the project by the TLP, the information requested in Sections 3.2, 3.3, and 3.4 may be presented here or in Section 4; the information requested in Sections 3.5 may be presented here or in Section 5; and the information requested in Sections 3.6 may be presented here or in Section 7.)

3.2 The sampling strategy shall be included and evidence must be presented to demonstrate that the strategy is appropriate for meeting primary project objectives, i.e., a description of the statistical method or scientific rationale used to select sample sites and number of samples shall be provided.

3.3 Sampling/monitoring points for all measurements (i.e., including locations and access points) shall be identified.

3.4 The frequency of sampling/monitoring events, as well as the numbers for each sample type and/or location shall be provided, including QC and reserve samples.

3.5 All measurements (i.e., analytical [chemical, microbiological, assays], physical, and process) shall be identified for each sample type or process, and project-specific target analytes shall be listed and classified as critical or non critical in the QAPP.

3.6 The planned approach (statistical and/or non-statistical) for evaluating project objectives shall be included.

Section 4.0, Sampling Procedures

4.1 Whenever applicable, the method used to establish steady-state conditions shall be described.

4.2 Known site-specific factors that may affect sampling/monitoring procedures shall be described.

4.3 Any site preparation needed prior to sampling/monitoring shall be described.

4.4 Each sampling/monitoring procedure to be used shall be discussed or referenced. If compositing or splitting samples, those procedures shall be described.

4.5 For samples requiring a split sample for either QA/QC purposes or for shipment to a different laboratory, the QAPP shall identify who is responsible for splitting samples, and where the splitting is performed (e.g., field versus lab).

4.6 If sampling/monitoring equipment is used to collect critical measurement data (i.e., used to calculate the final concentration of a critical parameter), the QAPP shall describe how the sampling equipment is calibrated, the frequency at which it is calibrated, and the acceptance criteria for calibration or calibration verification, as appropriate.

4.7 If sampling/monitoring equipment is used to collect critical measurement data, the QAPP shall describe how cross-contamination between samples is avoided.

4.8 The QAPP shall include a discussion of the procedures to be used to assure that representative samples are collected.

4.9 A list of sample quantities to be collected, and the sample amount required for each analysis, including QC sample analysis, shall be specified.

4.10 Containers used for sample collection, transport, and storage for each sample type shall be described.

4.11 The method for uniquely identifying each samples shall be described.

4.12 Sample preservation methods (e.g., refrigeration, acidification, etc.), including specific reagents, equipment, and supplies required for sample preservation shall be described.

4.13 Holding time requirements shall be noted.

4.14 Procedures for packing and shipping samples shall be described.

4.15 Procedures to maintain chain-of-custody (e.g., custody seals, records) during transfer from the field to the laboratory, in the laboratory, and among contractors and subcontractors shall be described to ensure that sample integrity is maintained.

4.16 Sample archival requirements for each relevant organization shall be provided.

Section 5.0, Testing and Measurement Protocols

5.1 Each measurement method to be used shall be described in detail or referenced. Modifications to EPA-approved or similarly validated methods shall be specified.

5.2 For unproven methods, verification data applicable to expected matrices shall be included in the QAPP meaning the QAPP shall provide evidence that the proposed method is capable of achieving the desired performance.

5.3 For measurements which require a calibrated system, the QAPP shall include specific calibration procedures applicable to each project target analyte, and the procedures for verifying both initial and continuing calibrations (including frequency and acceptance criteria, and corrective actions to be performed if acceptance criteria are not met).

Section 6.0, QA/QC Checks

6.1 At a minimum, the QAPP shall include quantitative acceptance criteria for QA objectives associated with accuracy, precision, detection limits, and completeness for critical measurements (process, physical, and analytical, as applicable) for each matrix.

6.2 Any additional project-specific QA objectives shall be presented, including acceptance criteria. This includes items such as mass balance requirements.

6.3 The specific procedures used to assess all identified QA objectives shall be fully described.

6.4 The QAPP shall list and define all other QC checks and/or procedures (e.g., blanks, surrogates, controls, etc.) used for the project, both field and laboratory.

6.5 For each specified QC check or procedure, required frequencies, associated acceptance criteria, and corrective actions to be performed if acceptance criteria are not met shall be included.

Section 7.0, Data Reporting, Data Reduction, and Data Validation

7.1 The reporting requirements (e.g., units, reporting method [wet or dry]) for each measurement and matrix shall be identified.

7.2 The deliverables expected from each organization responsible for field and laboratory activities shall be listed.

7.3 Data reduction procedures specific to the project, and also specific to each organization, shall be summarized.

7.4 Data validation procedures specific to each organization used to ensure the reporting of accurate project data to internal and external clients shall be summarized.

7.5 Data storage requirements for each organization shall be provided.

7.6 The product document that will be prepared for the project shall be specified (e.g., journal article, final report, etc.). The contents of this document can be referenced to a NRMRL or program-specific QMP, if appropriate.

Section 8.0, Assessments

8.1 The QAPP shall identify all scheduled audits (i.e., both technical system audits [TSAs] and performance evaluations [PEs]) to be performed, who will perform these audits, and who will receive the audit reports.

8.2 The QAPP shall provide procedures that are to be followed that will ensure that necessary corrective actions will be performed.

8.3 The responsible party(-ies) for implementing corrective actions shall be identified.

Section 9.0, References

References shall be provided either in the body of the text as footnotes or in a separate section.

Table of Contents

QAPP Requirements for Projects Using Secondary Data

A secondary data project involves the gathering and/or use of existing environmental data for purposes other than those for which they were originally collected. These secondary data may be obtained from many sources, including literature, industry surveys, compilations from computerized databases and information systems, and computerized or mathematical models of environmental processes. For these projects, a QAPP shall be prepared to include the requirements identified below. If primary data will also be generated as part of the project, then the information below can be incorporated into the associated QAPP to address the secondary data. The following requirements should be addressed as applicable.

Section 1.0, Project Objectives, Organization, and Responsibilities

1.1 The purpose of study shall be clearly stated.

1.2 Project objectives shall be clearly stated.

1.3 The secondary data needed to satisfy the project objectives shall be identified. Requirements relating to the type of data, the age of data, geographical representation, temporal representation, and technological representation, as applicable, shall be specified.

1.4 The planned approach for evaluating project objectives, including formulas, units, definitions of terms, and statistical analysis, if applicable, shall be included.

1.5 Responsibilities of all project participants shall be identified, meaning that key personnel and their organizations shall be identified, along with the designation of responsibilities for planning, coordination, data gathering, data analysis, report preparation, and quality assurance, as applicable.

Section 2.0, Sources of Secondary Data

2.1 The source(s) of the secondary data must be specified.

2.2 The rationale for selecting the source(s) identified shall be discussed.

2.3 The sources of the secondary data will be identified in any project deliverable.

Section 3.0, Quality of Secondary Data

3.1 Quality requirements of the secondary data must be specified. These requirements must be appropriate for their intended use. Accuracy, precision, representativeness, completeness, and comparability need to be addressed, if applicable. (If appropriate, a related QAPP containing this information can be referenced.)

3.2 The procedures for determining the quality of the secondary data shall be described.

3.3 If no quality requirements exist, this shall be stated in the QAPP. If no quality requirements exist or if the quality of the secondary data will not be evaluated by EPA, the QAPP shall require that a disclaimer be added to any project deliverable to indicate that the quality of the secondary data has not been evaluated by EPA for this specific application. The wording for the disclaimer shall be defined.

Section 4.0, Data Reporting, Data Reduction, and Data Validation

4.1 Data reduction procedures specific to the project shall be described, including calculations and equations.

4.2 The data validation procedures used to ensure the reporting of accurate project data shall be described.

4.3 The expected product document that will be prepared shall be specified (e.g., journal article, final report, etc.).

Table of Contents

QAPP Requirements for Method Development Projects

A method development project is typically needed in situations for which there exists no standard or known method, or when an existing method needs to be modified to meet a project-specific need. The following requirements should be addressed as applicable.

Section 1.0, Background
A description of the situation that requires the generation of a new or modified method shall be clearly stated. Why are we doing this?

Section 2.0, Scope and Application
The scope and application of the method shall be clearly stated. Specifically, to what matrices, conditions, etc., will this method apply for this project? What detection limits and/or practical quantitation limits are needed? How is this method intended to be used in the future (e.g., research only, potential regulatory usage, etc.)?

Section 3.0, Project Organization
Responsibilities of all project participants shall be identified, meaning that key personnel and their organizations shall be identified, along with the designation of responsibilities for planning, coordination, sample collection, measurements (i.e., analytical, physical, and process), data reduction, data validation (independent of data generation), data analysis, report preparation, and quality assurance.

Section 4.0, Experimental Approach including Sampling and Analytical Specifications

4.1 A description of the test(s) to be conducted in order to support the development of the method shall be included. All known or pre-established test conditions and variables shall be provided.

4.2 All planned measurements (i.e., analytical [chemical, microbiological, assays, etc.], physical, and process) shall be identified, and project-specific target analytes shall be listed.

4.3 Any known restrictions/specifications for sampling (e.g., collecting soil samples from a site or water samples from a port, etc.) or sub sampling (e.g., mixing sample before taking sub sample for analysis, etc.) shall be documented. Include specifications for: type and size of sample containers; amount of sample needed for preparation and analysis; preservation; holding times; representativeness; compositing; QC samples; etc.

4.4 The type of instrumentation that will be used and any required instrument conditions shall be documented. Include a discussion of calibration and calibration verification including frequency, acceptance criteria, and corrective action to be taken if acceptance criteria are not met.

Section 5.0, QA/QC Checks
Any planned QC checks and criteria that must be met for the method to be considered successful shall be specified. QC checks may include spikes, replicates, blanks, controls, surrogates, etc. Note: For chemical methods, quality control procedures to determine the precision, accuracy, and method detection limit should be described. For microbiological methods, positive and negative control procedures should be described.

Section 6.0, Method Verification
The tests that will be used to verify the method's performance once it's been developed shall be specified.

Section 7.0, Report
The report for a successful method development project will be a method written in a format appropriate for the application e.g., SW-846 for RCRA applications, Standard Methods for bacteria in drinking water, a SOP for a specific application (with supporting method performance data appended), etc.

Section 8.0, References
References shall be provided either in the body of the text as footnotes or in a separate section.

Table of Contents

QAPP Requirements for Software and Data Management Projects

Types of projects to which this guidance applies include the following: software development, software/hardware systems development, data base design and maintenance, and data validation and verification systems. The QAPP requirements for software development in this appendix do not mandate a particular method for software development. Project managers should choose software development and QA methods best suited to their individual projects within the parameters set forth here. Table D-1 provides a set of alternative QAPP elements for situations in which the elements applicable to measurement projects are not appropriate. The applicability of different elements is based on (1) the QA category and (2) the size or complexity of the task. Projects that involve both measurement and software/systems development should have plans addressing all applicable QA elements. Main issues to consider for inclusion in a QAPP for software and data management are listed in the following sections. Additional guidance for software and data management projects is available from the QAMs.

Section 0.0, Approval by Project Participants
The EPA Technical Lead Person (TLP) shall be responsible for obtaining signatures of appropriate project participants on the signature page of the QA plan, documenting agreement to project objectives and the approach for evaluating these objectives.

Section 1.0, Project Description
This section should provide an overview of the project, its intended uses, quality objectives, schedules and appropriate milestones, information about the hardware and operating systems, and planning documents.

Section 2.0, Project Organization and Responsibilities
This section should discuss all important intramural and extramural project personnel and should show the relationship between the development team and the personnel responsible for QA and testing.

Section 3.0, Functional Requirements
This section should provide a list of the most important functions that the software system must address. This section can also state any quantitative or qualitative data quality objectives (DQOs) that might apply to the software.

Section 4.0, System Design Overview (High Level Design)
A brief description of the system design is all that is necessary in the QAPP, if additional design documentation is planned.

Section 5.0, Detailed Design
Complex projects and those with significant defensibility requirements should have a detailed design document.

Section 6.0, Implementation
Written standard operating procedures (SOPs) for software development should be provided for extremely large and complex software projects. The internal checks applied during development should also be described.

Section 7.0, Testing
The QAPP should outline the testing strategy to be used.

Section 8.0, Data Validation and Verification
The QAPP must describe the means for checking the correctness of outputs.

Section 9.0, Change Control and Configuration Management
This section should describe the procedures for controlling and documenting all significant changes to software and hardware.

Section 10.0, Audits and Reviews
This section should describe planned assessments, including performance evaluation audits (PEAs), technical systems audits (TSAs), quality systems audits (QSAs), and audits of data quality (ADQs). Additional types of reviews applicable to these projects include peer reviews and beta testing.

Section 11.0, Maintenance and User Support
Where software or data generated by the project will be distributed outside NRMRL, maintenance and user support must be addressed.

Section 12.0, System Documentation and Archiving
Documentation is required for software projects in all QA categories. Table D-2 gives documentation requirements by QA Category.

Section 13.0, QA Progress Reports to Management
System development QA and QC results and plans should be reported regularly, particularly in projects in Categories I and II and where contractually required.

Table D-1
QA Elements for Software and Data Management QAPPS
QAPP Element Category Applicability
- Title/Signature Page   I, II, III, IV
- Table of Contents   I, II, III
1. Project Description    
  a. Background   I, II, III, IV
  b. Intended Application for Software   I, II, III, IV
  c. Quality Objectives for Software   I, II, III, IV
  d. Scope of Work   I, II, III
  e. Schedule and Milestones   I, II, III, IV
  f. Facilities Description   I, II, III
  g. Experimental/Test Matrix Design   I, II*, III*
  h. Planning Documents   I, II*, III*
2. Project Organization and Responsibilities   I, II, III
3. Functional Requirements   I, II, III
4. System Design Overview   I, II, III
5. Detailed Design   I, II*
6. Implementation   I, II*
  a. Development of SOPs    
  b. QC for Implementation    
7. Testing   I, II, III, IV*
  a. Individual Module Tests    
  b. Integration Tests    
  c. System Tests    
  d. Retesting after Changes    
  e. Acceptance Testing (if applicable)    
  f. Beta Testing (if applicable)    
8. Data Validation and Verification   I, II, III, IV*
9. Change Control and Configuration Management   I, II, III, IV*
10. Audits and Reviews   I, II*, III*
11. Maintenance and User Support   I, II, III*
12. System Documentation and Archiving   I, II, III, IV
13. QA Progress Reports to Management   I

*These elements may not be applicable for all projects in the specific category.

Table D-2
Recommended Documentation for Archiving
by QA Category

Document
QA Category
QA Project Plan   I, II, III, IV
Requirements Document   I, II, III*
Design Document   I, II, III*
Coding Standards or SOPs   I, II, III*
Source Code with In-line Comments (archived)   I, II, III, IV
User's Manual   I, II, III
Command Summary or Instructions for Use
(in lieu of a formal user's manual) 		  IV
Maintenance Manual or Installation Instructions    
(if source code is distributed outside EPA)   I, II, III, IV
(if source code is not distributed)   I, II*, III*
Data Dictionary   I, II, III
Testing and Validation Procedures and Results   I, II, III*
Backup Source Code and Build Procedures on Computer-readable Media   I, II, III, IV
*Project Officer & QA Manager's option    

Table of Contents

Requirements For Quality Management Plans (QMPs)

This quality management plan (QMP) requirements list is applicable to multi-year, multi-project efforts and is based on EPA guidance (http://www.epa.gov/quality/qs-docs/r2-final.pdf). A QMP defines an organization's quality system and documents policies and procedures that will be used to meet customer's quality needs.

Note (1): If a requirement is not applicable, state "not applicable" and provide an explanation regarding why. Note (2): To eliminate redundancy, reference to other sections of the QMP is permissible. Note (3): Ensure that the QMP clearly designates the personnel responsible for performing each procedure described in the QMP.

Section 0.0, QMP Approval
An approval page for the signatures of the senior accountable manager, senior line management (as appropriate) and the QA manager of the organization(s) that are part of this program level quality system needs to be provided. Signatures must be obtained prior to submitting the QMP for Agency review.

Section 1.0, Quality System Management and Organization

1.1 Program Quality Policy.
State the "quality policy," established and implemented by program management, which ensures that this environmental program produces the type and quality of results needed and expected.

1.2 Program Organizational Structure and Communication.
Describe functional responsibilities (including QA personnel), levels of accountability, authority, and communication for each organization that is part of this program level quality system. Demonstrate that the QA Manager(s) is/are independent of groups generating, compiling, and evaluating environmental data. Include a discussion of how disputes regarding quality system requirements, QA/QC procedures, assessments, or corrective actions are resolved. Note: It may be helpful to include organizational charts in addition to a narrative discussion.

1.3 Technical Activities.
Briefly describe all technical activities (e.g., basic research, technology evaluation, modeling, technology construction, etc.) that are supported by the quality system.

1.4 Communicating the Program's Quality System.
Describe how program management ensures that the quality management plan is communicated to and understood by those who are required to follow it (e.g., training, meetings, etc.).

1.5 Resources.
Describe program management procedures for allocating resources (human and financial) to implement the quality system (including, but not limited to, personnel training, quality system audits).

1.6 Authority to Stop Work for Safety and Quality Considerations.
Define who has the authority to stop unsafe work, or work of inadequate quality.

1.7 Management Assessment of Quality System Adequacy.
Describe how program management assesses and documents quality system adequacy. Include frequency of assessment, description of assessment (purpose, types of activities reviewed), and possible response actions.

Section 2.0, Quality System and Description

2.1 Quality System Elements.
Generally describe the principal components (or "tools") comprising the quality system and how they are used to implement the quality system. This includes (but is not limited to): QMPs; QA project plans, standard operating procedures; and audits/assessments. Include how and when each component is applied to individual projects and tasks.

2.2 Quality Management Plan Reviews and Revisions.
Describe procedures for updating quality system documentation.

Section 3.0, Personnel Qualification and Training

3.1 Personnel Training and Qualification Procedures.
Describe how program management ensures that all personnel performing work (including subcontractors, if applicable) are trained and qualified to perform work prior to initiating work. In addition to formal education, include specific on-the-job training for technical and management personnel (e.g., lab, field, health and safety, management, QA).

3.2 Formal Qualifications and Certifications for Specialized Activities.
Identify when formal qualification or certification is required.

3.3 Training Documentation.
Describe how program management ensures that required training is performed and documented.

3.4 Evidence of Personnel Job Proficiency.
Describe how objective evidence of personnel job proficiency is documented and maintained.

3.5 Re-Training.
Describe how the need for re-training is evaluated.

Section 4.0, Procurement of Items and Services Related to Technical Activities

4.1 Procurement Planning and Control.
Describe procedures for planning and controlling the procurement of items and services (e.g., subcontractor who provides analytical support, subcontractor who provides drilling support, calibrated sampling equipment, etc.).

4.2 Procurement Technical and Quality Requirements.
Describe procedures for ensuring that procurement documents (e.g., formal contract with subcontractor, purchase order for equipment, etc.) clearly describe the item or service needed and the associated technical and quality requirements (including a quality system consistent with EPA requirements when applicable). Include a discussion of when procurement documents will require the supplier to furnish a demonstrated capability to furnish items and services that meet all requirements and specifications.

4.3 Procurement Document Specification of Verifying Supplier's Conformance.
Describe procedures for ensuring that procurement documents specify how the supplier's conformance to customer's requirements will be verified.

4.4 Procurement Document Review.
Describe procedures for the internal review of procurement documents to ensure accuracy and completeness.

4.5 Review of Changed Procurement Documents.
Describe procedures for ensuring that changed procurement documents receive the same level of internal review and approval as the original documents.

4.6 Review of Procured Items and Services.
Describe how procured items and services are reviewed to ensure compliance with requirements and specifications.

Section 5.0, Documents and Records

5.1 Records Management Procedures.
Describe records management procedures from preparation to disposal, including maintenance (protection from damage and deterioration), storage (including accessibility), and retention (including disposition in accordance with statutory or contractual requirements). Describe how these records management procedures are controlled and maintained. Include printed and electronic records. Identify which documents/electronic records are included under these record management procedures (e.g., QAPP, data package, electronic data, laboratory notebooks, chain-of-custody forms, etc.).

5.2 Document Control.
Identify which documents require control (e.g., technical manuals, operating procedures, QAPPs, SOPs, final reports, etc.). Describe procedures for document review and approval (internal to the organization and external, e.g., client), and revision (before and after submission to client). Include procedures for distribution, replacement of previous document versions, and for ensuring that obsolete documents are no longer used.

Section 6.0, Computer Hardware and Software
This section applies to computer hardware and software operations that directly impact the quality of the results of environmental programs (both developed and purchased) including: design, design analysis, data handling, data analysis, modeling of environmental processes and conditions, operations or process control, and data bases.

6.1Conformance to User and EPA Requirements. Describe procedures for ensuring computer software and computer hardware/software configurations meet user's requirements and conform to applicable EPA requirements (e.g., Y2K compliance, security [protection from physical loss of data], and privacy [protection from unauthorized use of data]).

6.2 Configuration Testing. Describe procedures for testing computer hardware/software configurations prior to use to ensure technical requirements and quality expectations are met; include how the results of configuration tests are documented and maintained.

6.3 Configuration Change Assessment.
Describe procedures for assessing changes to hardware/software configurations; include how changes are evaluated based on the impact on technical and quality objectives of the program.

6.4 Re-Testing and Re-Documentation.
Describe procedures for re-testing and re-documentation when components are changed (creating a new configuration) or when program requirements change (bringing capability of the configuration into question).

In addition to environmental data collection activities, Sections 7-10 must also be addressed for the design, construction, and operation of environmental technologies; use of secondary data (i.e., use of environmental data); and development/modification of mathematical models (i.e., use and/or generation of environmental data), when applicable.

Section 7.0, Project Planning

7.1 Planning and Documenting the Generation, Acquisition, and Use of Environmental Data.
Describe procedures for planning and documenting all work involving the generation, acquisition, and use of environmental data (and the design, construction, and operation of environmental technologies as applicable). Identify types of planning documents generated (e.g., work plans, QAPPs, etc.) and summarize their purpose.

7.2 Identifying and Documenting Type and Quality of Environmental Data Needed.
Describe how the type and quality of environmental data needed (including secondary data, mathematical models, and the quality of technology design) are identified. Describe how this information is documented (e.g., DQO Process, QAPP, etc.)

7.3 Including Key Users, Customers, and Technical Staff in Planning.
For each applicable technical activity, describe procedures for involving the key users and customers of the data (and technology or model), in addition to the technical staff, during project-specific planning.

7.4 Reviewing and Approving Planning Documents.
Describe procedures used for review and approval (internal to organization and external, e.g., client) of planning documents prior to initiation of work. Reference to Section 5.0 may be applicable.

Section 8.0, Implementation of Work Processes

8.1 Implementation of Work According to Planning Documents.
Describe procedures (e.g., meetings, documentation) for ensuring that all work is performed according to approved planning and technical documents.

8.2 Standard Operating Procedures Documentation.
Describe the process for documenting standard operating procedures. Describe how standard operating procedures should be written so that they are easily understood by the user and contain sufficient detail and clarity.

Section 9.0, Project Assessment and Response
For the purposes of this section, audits and assessments are synonymous.

9.1 Planning Project Assessments.
Describe how, in the planning stage, management determines the appropriate type of assessment activity (e.g., technical systems audit, performance evaluation audit, etc.) for a particular project.

9.2 Assessment Planning and Procedures.
Describe the process used to ensure assessments are performed according to approved written procedures.

9.3 Assessment Personnel Qualifications.
Describe procedures for ensuring that assessments are performed by qualified personnel. Assessors shall be capable of assessing technical requirements and other procedures specified in the planning document.

9.4 Assessor Responsibility and Authority to Stop Work.
Describe the assessor's responsibility and authority to stop work. Describe conditions under which a stop work order may be needed.

9.5 Assessment Documentation, Reporting, and Review.
Describe assessment documentation, reporting, and review procedures. Include procedures for documenting and reporting assessment results to management. Include a discussion of how the assessor reports the impact of a negative assessment result on planned operations. For each procedure, include a time line for completion.

9.6 Assessment Responses and Follow-up Action.
Describe procedures for documenting assessment responses and how corrective action occurs. Include how follow-up is performed to ensure action was taken. For each procedure, include a time line for completion.

Section 10.0, Assessment and Verification of Data Usability

10.1 Assessing, Verifying, and Qualifying Data.
Describe procedures for assessing, verifying/validating, and qualifying data obtained from environmental data operations according to their planned intended use, including secondary data and mathematical modeling. (Describe procedures for assessing and verifying the performance of environmental technology for its intended use.)

10.2 Expressing and Documenting Limitations on Data.
Describe procedures for expressing and documenting (in print, or electronically) any limitations on this intended data use (and/or technology performance).

10.3 Providing Independent Review of Data-Containing Project Reports.
Describe procedures for the independent review of project reports containing data, or reports containing the results of environmental data operations (and/or technology performance), to confirm that the data or results are presented correctly. Reference to Section 5.0 may be applicable.

10.4 Management Approval of Reports.
Describe procedures for obtaining management approval of these reports prior to release, publication, or distribution. Reference to Section 5.0 may be applicable.

Section 11.0, Quality System Improvement

11.1 Quality Improvement Process.
Describe the quality improvement process used to continuously develop and improve the quality system. Include procedures such as communication with and among customers and suppliers; staff identification of problems and solutions, etc.

11.2 Preventing, Detecting and Correcting Quality System Problems.
Describe procedures used to prevent, detect and correct quality system problems.

11.3 Response Actions.
Describe procedures for planning, documenting, and implementing response actions to quality system problems.

Table of Contents

Attachment B
QA Planning Requirements (QAPR) Form for Initial Extramural Actions
and NRMRL QA Requirements / Definitions List

Quality Assurance Planning Requirements (QAPR) Form for Initial Extramural Actions (PDF format)

NRMRL QA Requirements / Definitions List

EPA's Quality System Web site: http://www.epa.gov/quality/qs-docs/r5-final.pdf

Category Level Designations (determines the level of QA required):

Project Types:

Definitions:

Environmental Data - For EPA, environmental data include information collected directly from measurements, produced from software and models, and compiled from other sources such as data bases or the literature.

Incremental Funding - Incremental funding is partial funding, no new work.

NRMRL's Quality System Specifications for Extramural Actions - These requirements typically pertain to single project efforts. The five specifications are:

  1. a description of the organization's Quality System (QS) and information regarding how this QS is documented, communicated and implemented;
  3. an organizational chart showing the position of the QA function;
  5. delineation of the authority and responsibilities of the QA function;
  7. the background and experience of the QA personnel who will be assigned to the project; and
  9. the organization's general approach for accomplishing the QA specifications in the SOW.

Quality Assurance (QA) - Quality assurance is a system of management activities to ensure that a process, item, or service is of the type and quality needed by the customer. It deals with setting policy and running an administrative system of management controls that cover planning, implementation, and review of data collection activities and the use of data in decision making. Quality assurance is just one part of a quality system.

Quality Assurance Project Plan (QAPP) - A QAPP is a document that describes the necessary quality assurance, quality control, and other technical activities that must be implemented to ensure that the results of the work performed will satisfy the stated performance criteria. A QAPP documents project-specific information.

Quality Control (QC) - Quality control is a technical function that includes all the scientific precautions, such as calibrations and duplications, that are needed to acquire data of known and adequate quality.

Quality Management Plan (QMP) - A QMP is a document that describes an organization's/program's quality system in terms of the organizational structure, policy and procedures, functional responsibilities of management and staff, lines of authority, and required interfaces for those planning, implementing, documenting, and assessing all activities conducted. A QMP documents the overall organization/program, and is primarily applicable to multi-year, multi-project efforts. An organization's/program's QMP shall address all elements listed in the "Requirements for Quality Management Plans" in Appendix B of the NRMRL QMP.

Quality System - A quality system is the means by which an organization manages its quality aspects in a systematic, organized manner and provides a framework for planning, implementing, and assessing work performed by an organization and for carrying out required quality assurance and quality control activities.

Substantive Change - Substantive change is any change in an activity that may alter the quality of data being used, generated, or gathered.

Supplemental Funding - Supplemental funding is additional funding of new work.

Technical Lead Person (TLP) - This person is technically responsible for the project. For extramural contract work, the TLP is typically the contracting officer's representative (COR). For intramural work, the TLP is typically the Principal Investigator.

Abbreviations:

COR Contracting Officer's Representative
NRMRL National Risk Management Research Laboratory
QA ID Quality Assurance Identification
QAPP Quality Assurance Project Plan
QS Quality System
TLP Technical Lead Person
IAG Interagency Agreement
QA Quality Assurance
QAM Quality Assurance Manager
QMP Quality Management Plan
SOW Statement of Work
CRADA Cooperative Research & Development Agreement.

Table of Contents

Office of Research & Development | National Risk Management Research Laboratory

Local Navigation

Jump to main content.