Quality System and Description
Graded Approach to Research Projects
Quality Assurance Project Plans (QAPPs) must be prepared to document QA/QC
requirements for all research projects that involve the collection, generation,
use, or reporting of environmental data (e.g. laboratory and field measurement
projects, modeling projects, method development projects, etc.). A QAPP must
be prepared before a research project begins. For QA planning purposes, a
research project begins as soon as a TLP is prepared to collect data/information
to support Agency decisions or for inclusion in research products (e.g., reports
[both paper and electronic], journal articles, symposium/conference papers,
extended abstracts, computer products/software/models/databases, or scientific
data). Before proceeding with the collection of this data/information, a QAPP
applicable to the project's QA category must be prepared. As part of the
planning process for a research project, it may be necessary to perform preliminary
work to gather information used to define its scope. This preliminary work must
comply with the requirements specified in ORD Policy 13.4.
Data collected during this preliminary work cannot be published.
The NRMRL quality system utilizes a "graded approach" for establishing appropriate
requirements for QAPPs for various types of research activities. Under the graded
approach, the intended use of the data and the visibility of the research effort
dictate the required level of quality. For example, the quality controls of a
fundamental research program are different from that of a regulatory compliance
program because the purpose or intended use of the data is different. Once a
graded approach has been applied to the components of a quality system, a graded
approach is then applied to the documentation required by that system. The graded
approach utilizes four (4) QA categories for research projects.
The four QA
- Category I – establishes QAPP requirements for projects involving areas such as
enforcement activities, litigation, or research projects involving human subjects;
- Category II – establishes QAPP requirements for important, highly visible
Agency projects involving areas such as supporting the development of environmental
regulations or standards;
- Category III – establishes QAPP requirements for projects involving applied
research or technology evaluations; and
- Category IV – establishes QAPP requirements for projects involving basic
research or preliminary data gathering activities.
For both Category I and Category II projects, NRMRL follows, in its entirety,
the R-5 document for QAPP preparation developed by EPA's Quality Staff (QS) (40 pp, 120 KB).
Guidance for Quality Assurance Project Plans, EPA QA/G-5 (PDF) (111 pp, 401 KB) (EPA/240/R-02/009)
December 2002 may be used to help address these requirements. If a particular R-5
requirement does not apply to an individual Category I/II project, the TLP
provides a brief explanation for why the specific requirement does not apply.
The Quality Staff has developed other guidance documents that maybe used in
developing QAPPs. These documents are: Guidance for Geospatial Data Quality
Assurance Project Plans, EPA QA/G-5G (PDF) (106 pp, 1.42 MB) (EPA/240/R-03/003) March 2003; Guidance for Choosing a
Sampling Design for Environmental Data Collection for Use in Developing a Quality Assurance Project Plan, EPA QA/G-5S (PDF) (178 pp, 1.02 MB) (EPA/240/R-02/005) December 2002;
and Guidance for Quality Assurance Project Plans for Modeling, EPA QA/G-5M (PDF) (121 pp, 615 KB) (EPA/240/R-02/007)
For projects of QA Category I or II involving design, construction,
and/or operation of environmental technology, the requirements in
ANSI/ASQ E4-2004: Quality Systems for Environmental Data and Technology
Programs - Requirements with Guidance for Use, 2004, American Society for
Quality (ASQ), Milwaukee, WI, 2004, Section 7 (Part C) and Guidance on
Quality Assurance for Environmental Technology Design, Construction, and
Operation EPA QA/G-11 (PDF) (86 pp, 244 KB) (EPA/240/B-05/001) January 2005,
will be followed.
For Category III and IV projects, a subset of applicable R-5 requirements is
utilized (see Appendix B). The outlines of NRMRL QAPP Requirements for various
project types are condensed from typically applicable sections of R-5 and are
intended to serve as a starting point when preparing a QAPP. Requirements are
included for projects involving measurement research; secondary data; method development;
modeling; software development/data management; and design, construction, and operation
of environmental technologies. (In some cases, these types of projects may be designated Category I or II;
additional requirements in R-5 would then be applicable.) As guidance, the
requirements in Appendix B include short descriptions of the applicable types
of research. These lists and their format may not fit every research scenario,
and QAPPs must conform to applicable sections of R-5 in a way that fully describes
the research plan and appropriate QA and QC measures to ensure that the data are of
adequate quality and quantity to fit their intended purpose. Some research may not
neatly fall into just one of the project types listed above and may combine features
of more than one subset of requirements. For such a project, the EPA TLP should
consult with the division's QA personnel to determine the most appropriate QAPP
The required QA category and associated QAPP requirements are determined by the
EPA TLP at the beginning of a project, with the concurrence of the Division QAM.
This is documented on the forms discussed in Chapter 4 and Chapter 7 in this QMP.
The graded approach applies to other aspects of the quality system. The following
table summarizes these requirements for the different QA Categories.
| QA Requirement
|| Category I
|| Category II
|| Category III*
|| Category IV*
| QAPP Preparation
|| EPA R-5
|| EPA R-5
|| NRMRL QAPP Requirements
|| NRMRL QAPP Requirements
| Technical Systems Audits
|| Required for each project
|| Required for each project
|| Two Category III/IV projects/year
|| Two Category III/IV projects/year
| Audits of Data Quality**
|| Required (50% of the data sets)
|| Required (25% of the data sets)
|| Not Required
|| Not Required
| Records Retention
|| 20 years
|| 20 years
*Additional sets of requirements may also be required depending on the nature of the project.
**If problems are found, all data sets will be audited.
Top of page
NRMRL's data generating research and testing activities always require
QA oversight. However, NRMRL also performs other functions that may not always
be attributed to project-specific efforts; certain aspects of NRMRL's operations,
such as the use of in-house data and software systems, may be inherently part of
projects and the quality of these systems is often not addressed in a QAPP.
Establishing quality policy for these types of activities and monitoring these
activities is typically performed through the development of SOPs, and in other
cases, facility manuals. In addition, some generally accepted laboratory
practices need to be implemented to ensure that a minimum level of quality is
established in each laboratory. The list of minimum requirements for laboratory
research activities is presented in ORD Policy 13.4.
These requirements are a significant aspect of NRMRL's Laboratory Competency Implementation Plan.
Top of page
Principal Components of the Quality System
The following are the principal components of the NRMRL quality system.
NRMRL Quality Management Plan
The NRMRL Quality Management Plan documents the NRMRL quality system and
encompasses the management and technical activities necessary to plan,
implement, and assess the effectiveness of QA and QC operations applied to
all research and environmental programs at NRMRL. The NRMRL Laboratory
Director is responsible for the development of the NRMRL QMP, although its
preparation is normally delegated to the NRMRL DQA with assistance from the
Division QAMs. The QMP must be approved and signed by the NRMRL Laboratory
Director, Division Directors, and the DQA. The QMP is submitted to the OEI
QS for review. It must be approved and signed by the QS Director and the
Assistant Administrator of OEI (AA/OEI). It is reviewed annually by the NRMRL
DQA and QAMs and updated as needed. The QAARWP will be used to identify changes
in the QMP prior to the next scheduled revision of the document. The approved
QMP is valid for five years unless there is a significant change in the organization
or in the research areas emphasized. After five years or a significant change,
the QMP is reviewed and revised as needed and subjected to the same approval
processes described above.
Quality Assurance Annual Report and Work Plan (QAARWP)
NRMRL assesses the effectiveness of its quality system on an annual basis.
That assessment and planned work activities for the coming fiscal year related
to the quality system are documented in the NRMRL Quality Assurance Annual
Report and Work Plan (QAARWP). The NRMRL QAARWP is submitted to the QS at the
conclusion of each fiscal year. Division-specific input for the QAARWP is
prepared by the QAM for each Division. After review by the respective Division
Director, the inputs of the Divisions for the QAARWP are provided to the NRMRL DQA.
The DQA prepares the NRMRL QAARWP, which is reviewed by and discussed with the
Laboratory Director before being submitted to the QS under the Director's signature.
Extramural Quality Management Plans
EPA contractors, assistance agreement recipients, and some EPA programs may be
required to submit a Quality Management Plan. Again, a graded approach is used
by the EPA TLP in consultation with the QAM to determine if a QMP is necessary
(see Chapter 4). If a QMP is required, it is reviewed and approved by the QAM
and EPA TLP. The NRMRL requirements for QMPs are given in Appendix B.
NRMRL research projects use a systematic planning process that involves
defining project objectives and then designing and refining a plan that meets
these objectives. All project stakeholders are included in the planning process.
The graded approach is used to ensure that the level of detail is in accord with
the intended data use and the resources available.
The NRMRL TLP is responsible for using the systematic process when planning
their work effort. Although not mandatory, the Data Quality Objectives (DQO)
Process developed by the EPA is an example of a systematic planning process.
This process is described in Guidance on Systematic Planning using the Data
Quality Objectives Process (PDF) (121 pp 1.61 MB) (EPA/240/B-06/001) February 2006.
Quality Assurance Project Plans
As discussed in Section 2.1 and Section 2.3.3 of this QMP, NRMRL uses a "graded approach"
(i.e., project type) for the development of QAPPs. Determination of the
appropriate project type is performed by the NRMRL TLP in consultation with other
project participants and the Division QAM when needed. Different project types
have different requirements as specified in Appendix B. The QAPP must be reviewed
and approved by the Division QAM, with concurrence by the TLP's branch chief,
before a research project begins.
NRMRL has developed QAPP requirements for each category of its graded approach
(see Appendix B) and uses guidance documents developed by EPA's Quality Staff
(QS) to assist technical staff in addressing both NRMRL and Division-specific
quality specifications e.g., the R and G series of documents developed by QS.
NRMRL facility managers, lead researchers, and QA staff may develop facility
manuals to assist technical staff (both EPA and contractor) in documenting the
performance of routine operations. Facility manuals have been primarily used
by NRMRL's division in North Carolina. A facility is defined as a building
(or portion of a building) housing a set of equipment designed for a particular
area of research such as a furnace to study the impact of combustion parameters
on the pollutants emitted or a pilot plant to simulate a water distribution system.
Facility manuals may contain the following information on a specific NRMRL facility:
- Facility Charter (purpose)
- Quality Assurance
- Quality Control
- Data Handling
- Corrective Action
- Health and Safety
- QA/Test Plans
Appendices may contain:
- Current Facility Personnel
- Operating Procedures
- Standard Methods
- Technical Systems Audit Checklist
- Performance Evaluation Audit Ranges
Facility manuals are reviewed by the facility manager or lead researcher, QAM,
and health and safety (H&S) personnel if H&S requirements are included in the
manual; they are maintained and updated by facility managers. Formal revisions
and reviews are initiated when substantial changes are made to a facility manual.
A facility manual documents instrumentation and procedures, provides a historical
record of facility use, standardizes procedures, provides a training aid for new
employees, and facilitates preparation of QAPPs (which can reference standard
sections in the facility manual, with description of the specific tests and
schedules for testing).
Standard Operating Procedures and Methods
NRMRL develops SOPs for those operations that have become or will become routine
(e.g., laboratory operations such as balance use, analytical procedures,
sampling procedures, instrument calibration procedures) which are used for more
than one project or performed by more than one person. Before SOPs are implemented,
they are reviewed by a second person in the specific technical area and reviewed/approved
by the QAM. The QAM may serve as the second technical person if they have expertise
in the technical area. Otherwise, the QAM's review is to verify that the second
technical review was performed and that QA/QC is addressed in the SOP. Concurrence
by the branch chief (or designee) is required for branch SOPs, by the division
director (or designee) for division SOPs, and the Laboratory Director (or designee)
for NRMRL-wide SOPs.
Project-specific SOPs may also be written, and are particularly recommended to
facilitate updating the QAPP or when the procedure is performed by contract
personnel. Project-specific SOPs are reviewed and approved as part of the
QAPP approval process. The purpose of the SOPs is to facilitate the uniform
performance of routine procedures.
When appropriate, NRMRL uses approved standard methods (e.g., SW 846,
MCAWW, EPA 500 and 600 series, Standard Methods, CFR) to meet project-specific
Top of page
Audits are scheduled for specific systems, programs, and projects based on
required periodic review, the importance of or public interest in the program
or project, problems in the measurement system (noted by the TLP or QAM),
requests by the TLP or managers (branch chiefs, division directors), or
selection by the division QAM. The audits planned by the TLP at the beginning
of a project or required by the program (e.g., SITE, ETV) are specified in the
QAPP or QMP. For all assessments, the lead auditor/assessor (usually either
the DQA or a QAM) must ensure that the auditor(s)/assessor(s) has/have no real
or perceived conflicts of interest with the project/system/organization being
Quality Systems Audits
A quality systems audit (QSA) is an on-site review of the implementation of an
organization's quality system as documented in the organization's approved
QMP. This review is used to verify the existence of, and evaluate the adequacy of,
the internal quality system. A QSA may be a self-assessment or an independent
Independent QSAs of NRMRL are scheduled and conducted by personnel external to
NRMRL, such as QS personnel. At his/her discretion, the NRMRL DQA may schedule a
QSA of a Division to assess the implementation of the quality system in the
Division and may enlist the help of personnel from other Divisions. In any event,
the Division QAM is always invited as a member of the evaluation team to witness
first hand the implementation of the QMP in their Division observed during the
QSA and to respond to questions as needed.
Technical Systems Audits
A technical systems audit (TSA) is a qualitative on site evaluation of sampling
and/or measurement systems. The objective of the TSA is to assess and document
acceptability of all facilities, maintenance, calibration procedures, reporting
requirements, sampling and analytical activities, and quality control procedures.
Normally, an approved QAPP provides the basis for the TSA. Independent TSAs are
most often scheduled by the TLP and conducted by the Division QAM. Assistance for
the TSA is available to each QAM from other Divisions or from QA support contractors.
TSAs are most useful when conducted early in the life cycle of a project when
corrective actions (if necessary) can be performed that will minimize any loss
of data. However, a TSA can be performed any time during a project's life cycle.
The NRMRL DQA may schedule project-specific TSAs to assess the quality system
for individual projects in one or more Divisions and may enlist the help of
personnel in other Divisions. In any event, the Division QAM is always invited
as a member of the evaluation team.
TSAs are required for all Category I and II projects (both intramural and extramural).
For Category III and IV projects, each division will audit two Category III/IV
projects each fiscal year. At least one Category III/IV project must be an
Audits of Data Quality
An audit of data quality (ADQ) is an examination of data after they have been
collected and verified by project personnel. Assessing whether the Data Quality
Indicator (DQI) goals specified in the QAPP were met requires a detailed review of
the recording, transferring, calculating, summarizing, and reporting of the data.
ADQs are required for QA Category I and II projects. An ADQ may also be conducted
as required by specific programs or as requested by the TLP or other EPA stakeholder.
ADQs may be conducted by EPA or contractor personnel.
A performance evaluation (PE) is a quantitative evaluation of a measurement system.
Although each measurement in a test program could be subjected to a performance
evaluation, the critical measurements (designated in the QAPP) are more commonly
evaluated. An evaluation of a measurement system usually involves the measurement
or analysis of a reference material of known value or composition. The value or
composition of reference materials must be certified or verified prior to use, and
the certification or verification must be adequately documented. Ideally, the
identity of the reference material is disguised so that the operator or analyst
will treat the material no differently than a test program sample. PEs are conducted
as required by specific programs or projects or at the request of the TLP or other
Laboratory surveillances are conducted using the requirements specified in ORD QA Policy 13.4.
Each division will conduct surveillances of all laboratory areas at least once
every three years. Surveillances of all or a portion of the laboratory areas
will be conducted each year.
Top of page
Data Quality Assessments
A Data Quality Assessment (DQA) is a scientific and statistical evaluation to
determine if validated data obtained from environmental data operations are of
the right type, quality, and quantity to support their intended use. The DQA
process is described in Data Quality Assessment: A Reviewer's Guide, EPA QA/G-9R (PDF) (61 pp, 258 KB) (EPA/240/B-06/002) February 2006.
For research projects, these assessments are normally conducted by the TLP, and
they should be performed routinely on all projects.
QA Review of Research Products
Final product reviews are conducted to evaluate the credibility of data,
realization of project goals, comparability of data, validity of conclusions,
and quality of data. Research products must be subjected to QA review prior to
being distributed outside of NRMRL. A signed NRMRL Product QA/QC Verification
Report form (Appendix C) must accompany the NRMRL products (reports [both paper
and electronic], journal articles, symposium/conference papers, extended abstracts,
computer products/software/models/databases, or scientific data) submitted for QA
review if they contain environmental data. Environmental data are defined in EPA
Order 5360.1 A2 as any measurements or information that describe environmental
processes, location, or conditions; ecological or health effects and consequences;
or the performance of environmental technology. For EPA, environmental data
include information collected directly from measurements, produced from models,
and compiled from other sources such as databases or the literature.
Computer Tracking System
Different computer tracking systems for the various elements of the quality
system are in place in the various divisions. These systems track funding
packages, QAPPs, QMPs, SOPs, audits, and project products.
Mandatory quality system training for all NRMRL management and technical
personnel is described in Chapter 3. The NRMRL DQA and QAMs discuss
quality system training needs in the first quarter of the fiscal year during
the regular monthly NRMRL QA teleconference. The goal of the discussion is
to develop a consensus on the minimum quality system training to be required
for all of NRMRL for that year and how it will be accomplished. Possible
training methods are in-house by DQA or QAMs, contractor, agency presentations
(e.g., IQGs), or computer-based courses. Each QAM will use Division-specific
practices to determine the quality system training needs for the Division prior
to this discussion. Division-specific quality system training will be in addition
to NRMRL Laboratory-wide QA training.
Top of page
You will need Adobe Reader to view some of the files on this page.
See EPA's PDF page to learn more.