Jump to main content.


 Chapter 9

 

Assessment and Response

NRMRL conducts a variety of assessments at the NRMRL-wide, Division, and project-specific level to provide an increased understanding of the program or system being examined, and to provide a basis for improving such programs or systems. The following includes: a review of the various assessment techniques, plans for appropriate response, and the process by which management determines the assessment activities appropriate for each level.

9.1 Assessment Techniques and Practices

Basic assessment categories were described in Section 2.3.5 of this QMP.

In NRMRL, peer review is a separate function from QA. Program peer reviews are not performed by QA staff unless directed by the NRMRL Director or a Division Director. The NRMRL and Division QA offices respond to issues regarding data quality identified during the peer review process and provided to the QA staff.

9.2 Types of Assessments

A description of the four types of assessments is provided in this section.
9.2.1 Self-Assessment of a Quality System

A quality system self-assessment is a qualitative assessment of a quality system's operations by those immediately responsible for overseeing and/or performing the work to establish whether the prevailing quality structure, policies, practices, and procedures are adequate for ensuring that the type and quality of results needed are obtained.

9.2.2 Independent Assessment of a Quality System

An independent quality system assessment is a qualitative assessment of a quality system's operations by someone other than the group performing the work to establish whether the prevailing quality structure, policy, practices, and procedures are adequate for ensuring that the type and quality of results needed are obtained.

9.2.3 Technical Self-Assessment

A technical self-assessment is the evaluation process used by those immediately responsible for overseeing and/or performing the work to measure the performance or effectiveness of a technical system and its elements with respect to documented specifications and objectives. Such assessments may include qualitative and quantitative evaluations. These assessments are not required but encouraged at the beginning of a project to ensure that all aspects or processes are operational and ready to produce data of acceptable quality (some disciplines call these assessments readiness reviews).

9.2.4 Independent Technical Assessment

An independent technical assessment is the evaluation process used by someone other than the group performing the work to measure the performance or effectiveness of a technical system and its elements with respect to documented specifications and objectives. Such assessments may include qualitative and quantitative evaluations.

9.3 Assessor Capabilities and Authority

Assessors should be thoroughly familiar with QA practices, policies, and procedures; have no real or perceived conflict of interest; and have no direct involvement or responsibility for the work being assessed. The optimal assessment team consists of a QA professional, a person with expertise in the technical area being evaluated, and, if there are complex issues regarding experimental design or data analysis, a statistician. In many cases, a QA staff member can fill more than one role. The DQA (for NRMRL assessments) or Division QA Manager (for Division assessments) will be responsible for assembling an appropriate assessment team.

Assessors evaluate NRMRL projects under the authority of and with the permission of the NRMRL TLP and NRMRL management. Assessors do not have the direct authority to change project procedures or to alter project goals. Their role is to provide information to the NRMRL TLP as to the best practices and to clearly point out deviations observed in the implementation of the QAPP. These deviations may be in technical procedures, in data archival or analysis, in project management or reporting, or in any aspect of a project that impacts the data quality.

Top of page

9.4 NRMRL-Wide Assessment and Response Procedures

The NRMRL DQA is responsible for planning, scheduling, and conducting the following assessments. The NRMRL DQA may be assisted in the performance of assessments by Division QA Office staff. At the conclusion of each assessment, the assessment report shall include a description of the type of corrective actions required to resolve any findings of nonconformance to EPA and NRMRL policy as described in EPA and NRMRL quality documents and Orders.

9.4.1 Self-Assessment of NRMRL Quality System

The DQA performs a review of the NRMRL QMP/quality system annually during development of the NRMRL QAARWP. Any significant changes are identified in the QAARWP.

9.4.2 Independent Assessment of the Application of the NRMRL Quality System in a Division

QSAs of NRMRL Divisions will not routinely be performed by the DQA. However, a QSA of a Division may be performed by the DQA at the discretion of the DQA, e.g., if quality issues at the Division-level are known or suspected, or as directed by senior management.

9.4.3 Independent Technical Assessment

Independent technical assessments performed by the DQA may include technical system audits (TSAs), QSAs of Programs, and lab competency audits.

  • TSAs - The conduct of TSAs will probably be performed very infrequently by the DQA. TSAs conducted by the DQA will focus mainly on larger projects involving more than one NRMRL location. The requirement for these TSAs will be determined during project planning and will be specified in project-specific QAPPs.
  • QSA of Programs - QSAs of Programs will be performed based on specific Program requirements or as directed by senior management.
  • Lab Competency Audit - The DQA will oversee lab competency audits in each Division every three years. The standard for the Lab Competency Audit is ORD Policy 13.4. This audit may be conducted by a contractor or other Federal employees including other ORD personnel (but not from NRMRL).

Top of page

9.5 Division Assessment and Response Procedures

The Division QAM is responsible for planning, scheduling, and conducting the following assessments. The Division QAM may be assisted in the performance of assessments by QA Office staff members from any Division. At the conclusion of each assessment, the assessment shall include a description of the type of corrective actions required to resolve any findings of nonconformance to EPA and NRMRL policy as described in EPA and NRMRL quality documents and Orders.

9.5.1 Self-Assessment of the Application of the NRMRL Quality System to a Division

The Division QAM performs a review of the implementation of the NRMRL quality system in the Division annually during development of the Division input to the NRMRL QAARWP. Any significant changes are identified in the QAARWP.

9.5.2 Independent Technical Assessment

Independent technical assessments include review of Division SOPs, QAPPs, and final research products; TSAs, PEs, and ADQs; and laboratory surveillances.

  • Review of Division SOPs, QAPPs, and Final Research Products - All SOPs, QAPPs, and final research products (all published reports (both paper and electronic), journal articles, symposium/conference papers, extended abstracts, and computer products/software/models/databases, or scientific data) are reviewed and approved by the Division QAM.
  • TSAs, PEs, ADQs - TSAs are conducted based on project-specific requirements as described in Section 2.3.5.2. ADQs are conducted as described in Section 2.3.5.3. PEs are conducted as described in Section 2.3.5.4. The applicable assessments tools for a specific project are determined during project planning by the TLP, with input from the Division QAM if needed.
  • Surveillances - Laboratory surveillances are conducted as described in Section 2.3.5.5.

Top of page

9.6 "Stop Work" Order

QA staff members are often on-site and able to interact directly with project personnel. If a QA staff member observes work practices that could have serious adverse impacts on data quality, the QA staff member should promptly notify the NRMRL TLP or line manager for in-house projects who may issue a Stop Work Order. For extramural projects, if the TLP concurs that work practices are unacceptable, the TLP should notify the Contracting Officer, who may issue a Stop Work Order.

Top of page

9.7 Assessment Reports and Corrective Action

TSAs and QSAs are followed up by a debriefing to the auditee and TLP (if available) at the end of the on-site portion of the evaluation. If the TLP is not present at the debriefing, then assessments are also followed up by a verbal or brief written summary to the NRMRL TLP on the first working day after the completion of the on-site portion of the audit.

The general format for assessment reports is:
  • Cover page (assessment identification)
  • Summary (brief discussion of findings, those issues that require corrective action)
  • Assessment procedures
  • Assessment results (detailed accounting of findings and other observations and corrective actions)
  • Discussion (impact of findings)
  • Appendices (checklists, protocols, other data)
Draft assessment reports are generated by the assessor within two weeks of the receipt of requested audit materials. Reports are sent as drafts to the NRMRL TLP and to the auditee, for correction of factual errors and addition of any pertinent comments or responses to audit findings. Two weeks are generally allowed for this response and review.

For extramural projects, after this review and correction, as necessary, the final reports are distributed to the NRMRL TLP, the TLP's immediate supervisor (usually a branch chief), and to the auditee. For intramural projects, the final reports are distributed to the TLP and the TLP's immediate supervisor. Typically, two weeks are allowed for issuance of the final report after receipt of the corrected draft report. It is the responsibility of the NRMRL TLP to resolve audit findings. If findings cannot be resolved by the NRMRL TLP, the TLP's immediate supervisor's assistance is requested. The file on a given audit is maintained in active status until all findings are resolved and documented. Audit reports are then archived in the TLP's project files.

For surveillances, a checklist based on the requirements in ORD QA Policy 13.4 is typically used to document the results; this checklist may also serve as the report to the branch chief in charge of the laboratory area. A copy of the checklist is retained by the QAM, and another copy or report summarizing the results should be given to the TLP of the area.

Top of page


Local Navigation


Jump to main content.