| |
Assessment and Response
NRMRL conducts a variety of assessments at the NRMRL-wide, Division, and
project-specific level to provide an increased understanding of the program
or system being examined, and to provide a basis for improving such programs
or systems. The following includes: a review of the various assessment
techniques, plans for appropriate response, and the process by which
management determines the assessment activities appropriate for each level.
9.1
Assessment Techniques and Practices
Basic assessment categories were described in Section 2.3.5 of this QMP.
In NRMRL, peer review is a separate function from QA. Program peer reviews
are not performed by QA staff unless directed by the NRMRL Director or a
Division Director. The NRMRL and Division QA offices respond to issues
regarding data quality identified during the peer review process and provided
to the QA staff.
9.2
Types of Assessments
A description of the four types of assessments is provided in this section.
9.2.1
Self-Assessment of a Quality System
A quality system self-assessment is a qualitative assessment of a quality system's
operations by those immediately responsible for overseeing and/or performing the
work to establish whether the prevailing quality structure, policies, practices,
and procedures are adequate for ensuring that the type and quality of results
needed are obtained.
9.2.2
Independent Assessment of a Quality System
An independent quality system assessment is a qualitative assessment of a
quality system's operations by someone other than the group performing the
work to establish whether the prevailing quality structure, policy, practices,
and procedures are adequate for ensuring that the type and quality of results
needed are obtained.
9.2.3
Technical Self-Assessment
A technical self-assessment is the evaluation process used by those immediately
responsible for overseeing and/or performing the work to measure the performance
or effectiveness of a technical system and its elements with respect to
documented specifications and objectives. Such assessments may include
qualitative and quantitative evaluations. These assessments are not required
but encouraged at the beginning of a project to ensure that all aspects or
processes are operational and ready to produce data of acceptable quality
(some disciplines call these assessments readiness reviews).
9.2.4
Independent Technical Assessment
An independent technical assessment is the evaluation process used by someone other
than the group performing the work to measure the performance or effectiveness of a
technical system and its elements with respect to documented specifications and
objectives. Such assessments may include qualitative and quantitative evaluations.
9.3
Assessor Capabilities and Authority
Assessors should be thoroughly familiar with QA practices, policies, and
procedures; have no real or perceived conflict of interest; and have no
direct involvement or responsibility for the work being assessed. The
optimal assessment team consists of a QA professional, a person with
expertise in the technical area being evaluated, and, if there are complex
issues regarding experimental design or data analysis, a statistician. In
many cases, a QA staff member can fill more than one role. The DQA (for
NRMRL assessments) or Division QA Manager (for Division assessments) will
be responsible for assembling an appropriate assessment team.
Assessors evaluate NRMRL projects under the authority of and with the
permission of the NRMRL TLP and NRMRL management. Assessors do not have
the direct authority to change project procedures or to alter project goals.
Their role is to provide information to the NRMRL TLP as to the best practices
and to clearly point out deviations observed in the implementation of the QAPP.
These deviations may be in technical procedures, in data archival or analysis,
in project management or reporting, or in any aspect of a project that impacts
the data quality.
Top of page
9.4
NRMRL-Wide Assessment and Response Procedures
The NRMRL DQA is responsible for planning, scheduling, and conducting the
following assessments. The NRMRL DQA may be assisted in the performance
of assessments by Division QA Office staff. At the conclusion of each
assessment, the assessment report shall include a description of the type
of corrective actions required to resolve any findings of nonconformance
to EPA and NRMRL policy as described in EPA and NRMRL quality documents
and Orders.
9.4.1
Self-Assessment of NRMRL Quality System
The DQA performs a review of the NRMRL QMP/quality system annually during
development of the NRMRL QAARWP. Any significant changes are identified in the QAARWP.
9.4.2
Independent Assessment of the Application of the NRMRL Quality System in a Division
QSAs of NRMRL Divisions will not routinely be performed by the DQA. However,
a QSA of a Division may be performed by the DQA at the discretion of the DQA,
e.g., if quality issues at the Division-level are known or suspected, or as
directed by senior management.
9.4.3
Independent Technical Assessment
Independent technical assessments performed by the DQA may include technical
system audits (TSAs), QSAs of Programs, and lab competency audits.
- TSAs - The conduct of TSAs will probably be performed very infrequently
by the DQA. TSAs conducted by the DQA will focus mainly on larger projects
involving more than one NRMRL location. The requirement for these TSAs
will be determined during project planning and will be specified in
project-specific QAPPs.
- QSA of Programs - QSAs of Programs will be performed based on specific
Program requirements or as directed by senior management.
- Lab Competency Audit - The DQA will oversee lab competency audits in
each Division every three years. The standard for the Lab Competency Audit is ORD Policy 13.4.
This audit may be conducted by a contractor or other Federal employees
including other ORD personnel (but not from NRMRL).
Top of page
9.5
Division Assessment and Response Procedures
The Division QAM is responsible for planning, scheduling, and conducting the
following assessments. The Division QAM may be assisted in the performance
of assessments by QA Office staff members from any Division. At the conclusion
of each assessment, the assessment shall include a description of the type of
corrective actions required to resolve any findings of nonconformance to EPA
and NRMRL policy as described in EPA and NRMRL quality documents and Orders.
9.5.1
Self-Assessment of the Application of the NRMRL Quality System to a Division
The Division QAM performs a review of the implementation of the NRMRL quality
system in the Division annually during development of the Division input to the
NRMRL QAARWP. Any significant changes are identified in the QAARWP.
9.5.2
Independent Technical Assessment
Independent technical assessments include review of Division SOPs, QAPPs, and
final research products; TSAs, PEs, and ADQs; and laboratory surveillances.
- Review of Division SOPs, QAPPs, and Final Research Products - All SOPs,
QAPPs, and final research products (all published reports (both paper and
electronic), journal articles, symposium/conference papers, extended
abstracts, and computer products/software/models/databases, or scientific
data) are reviewed and approved by the Division QAM.
- TSAs, PEs, ADQs - TSAs are conducted based on project-specific requirements
as described in Section 2.3.5.2. ADQs are conducted as described in Section 2.3.5.3. PEs are conducted as described in Section 2.3.5.4.
The applicable assessments tools for a specific project are determined
during project planning by the TLP, with input from the Division QAM if needed.
- Surveillances - Laboratory surveillances are conducted as described in Section 2.3.5.5.
Top of page
9.6
"Stop Work" Order
QA staff members are often on-site and able to interact directly with project
personnel. If a QA staff member observes work practices that could have serious
adverse impacts on data quality, the QA staff member should promptly notify the
NRMRL TLP or line manager for in-house projects who may issue a Stop Work Order.
For extramural projects, if the TLP concurs that work practices are unacceptable,
the TLP should notify the Contracting Officer, who may issue a Stop Work Order.
Top of page
9.7
Assessment Reports and Corrective Action
TSAs and QSAs are followed up by a debriefing to the auditee and TLP (if available)
at the end of the on-site portion of the evaluation. If the TLP is not present at
the debriefing, then assessments are also followed up by a verbal or brief written
summary to the NRMRL TLP on the first working day after the completion of the
on-site portion of the audit.
The general format for assessment reports is:
- Cover page (assessment identification)
- Summary (brief discussion of findings, those issues that require corrective action)
- Assessment procedures
- Assessment results (detailed accounting of findings and other
observations and corrective actions)
- Discussion (impact of findings)
- Appendices (checklists, protocols, other data)
Draft assessment reports are generated by the assessor within two weeks of the
receipt of requested audit materials. Reports are sent as drafts to the NRMRL
TLP and to the auditee, for correction of factual errors and addition of any
pertinent comments or responses to audit findings. Two weeks are generally
allowed for this response and review.
For extramural projects, after this review and correction, as necessary, the
final reports are distributed to the NRMRL TLP, the TLP's immediate supervisor
(usually a branch chief), and to the auditee. For intramural projects, the
final reports are distributed to the TLP and the TLP's immediate supervisor.
Typically, two weeks are allowed for issuance of the final report after receipt
of the corrected draft report. It is the responsibility of the NRMRL TLP to
resolve audit findings. If findings cannot be resolved by the NRMRL TLP, the
TLP's immediate supervisor's assistance is requested. The file on a given audit
is maintained in active status until all findings are resolved and documented.
Audit reports are then archived in the TLP's project files.
For surveillances, a checklist based on the requirements in ORD QA Policy 13.4 is
typically used to document the results; this checklist may also serve as the
report to the branch chief in charge of the laboratory area. A copy of the
checklist is retained by the QAM, and another copy or report summarizing the
results should be given to the TLP of the area.
Top of page |
![[logo] US EPA](http://www.epa.gov/epafiles/images/logo_epaseal.gif){kind=logo}