Jump to main content.

Performance Characteristics and Measurement Quality Objectives (MQOs)

The basic performance characteristics of an assessment and its component methods include precision (repeatability of measurements), sensitivity (detection limit), and accuracy (proximity to the analytical truth). Measurement quality objectives (MQOs) are project-specific measurement goals of performance characteristics that are derived from the DQOs. MQOs include acceptance criteria for characteristics such as sensitivity (i.e., what detection or quantification limit is desired), selectivity (i.e., what components of the assemblage are to be targeted), precision (i.e., repeatability of the method among replicates, investigators, time, etc.). Although MQOs provide the criteria for how good the data must be, they do not specify exactly how the data must be produced, that is, the specific method or technology to be used (Crumbling 2001). For example, precision and sensitivity can be determined for benthic sampling methods using kick nets or Surber samplers.

Example MQOs:

Precision estimates are important to help interpret results from bioassessment efforts. An approach with low precision indicates lower confidence in the interpretation of data than one with high precision. Two fundamental requirements for a biological assessment are that samples of the assemblage of interest are consistently collected, and that the analytical data are reflective of the sample. Measurement of precision in these two requirements determines the level of confidence in the assessment. Precision is measured to identify errors and allow inferences to be made about the repeatability of an assessment. Once the precision of a method is known, the likelihood of replicating an assessment can be estimated and the level of confidence in an assessment can be characterized. Precision is estimated by evaluating (via ANOVA, CV, Signal/Noise) repeat samples from the same site(s) at different times with different crews (interteam) or with the same crew (intra-team) (Kaufmann et al. 1999; Hughes et al. 1998; McCormick et al. 2001). Ideally, precision is known for all of these components.

Biological assessments are most useful when the methods distinguish natural and index variability (i.e., “noise”) from a true environmental effect (i.e., “signal”). Therefore, the premise is that the site is representative of the population of sites, the sample is representative of the site examined and the assemblage measured, and the data are an accurate reflection of that sample. States typically establish a threshold based on this signal and then add other thresholds to distinguish among higher (e.g., outstanding natural resource waters, excellent warmwater habitat, or excellent/good habitat) and lower assessment categories (e.g., limited resource waters, fair/poor/very poor).

Regardless of approach, the primary purpose of an analytical threshold is to establish levels of biological quality that can be used in determining attainment or non-attainment of the designated use. The thresholds should allow for straightforward decisions including statements of uncertainty when biological data are compared against the thresholds to facilitate water quality management decisions. Decisions applying the threshold also need to be documented in the record.

Quality assurance programs encourage the continued documentation of variability to ensure the ability to detect long-term trends. An ongoing quality assurance program is also useful for periodically reevaluating the performance of the indicator and the calibration of reference conditions. Quality assurance procedures include examination of replicate field samples at some subset of the sample units (e.g., 10% of the sites) and reexamination of a proportion of samples by an independent taxonomist. For programs in which multiple field sampling crews are used, it is important to document variability in results caused by personnel. Side-by-side sampling by different field crews is done to document the magnitude of crew variability as a source of measurement error.

Overall variability (= total uncertainty, or error) of data from any measurement system results from accumulation of error from multiple sources (Taylor 1988; Clark and Whitfield 1994; Taylor and Kuyatt 1994; Diamond et al. 1996). Error can generally be divided into two types: systematic and random. Systematic error is the type of variability that results from a method and its application or misapplication; it is composed of bias that can, in part, be mediated by using an appropriate quality assurance program. Random error results from the sample itself or the population from which it is derived, and can only partly be controlled through a careful sampling design (see Figure). It is often not possible to separate the effects of the two types of error, and they can directly influence each other (Taylor 1988). The overall magnitude of error associated with a dataset is known as data quality; how statements of data quality are made and communicated is critical for data users and decision makers to properly evaluate the extent to which they should rely on scientific information (Peters 1988; Costanza et al. 1992). A stream assessment (in particular, a biological assessment) is a series of methods taken together as a protocol (Diamond et al. 1996; USEPA 1999) and, as such, each method can contribute to overall variability (see Figure). Thus, it is important to know something about the quality of the data produced at each step of the process.

Total error or variability associated with a biological assessment is a

For more information, please visit:

References:

Clark, M.J.R. and P.H. Whitfield. 1994. Conflicting perspectives about detection limits and about the censoring of environmental data. Water Resources Bulletin 30:1063-1079.

Diamond, J.M., M.T. Barbour, and J.B. Stribling. 1996. Characterizing and comparing bioassessment methods and their results: A perspective. Journal of the North American Benthological Society 15:713-727.

Hughes, R.M., P.R. Kaufmann, A.T. Herlihy, T.M. Kincaid, L. Reynolds, and D.P. Larsen. 1998. Development and application of an index of fish assemblage integrity for wadeable streams in the Willamette Valley Ecoregion, Oregon, USA. Canadian Journal of Fisheries and Aquatic Sciences 55:1618-1631.

Kaufmann, P.R., P. Levine, E.G. Robison, C. Seeliger, and D.V. Peck. 1999. Quantifying physical habitat in wadeable streams. USEPA. 620/R-99/003. Corvallis, OR.

Peters, J.A. 1988. Quality control infusion into stationary source sampling. Chapter 22, in, Lawrence H. Keith (editor), Principles of Environmental Sampling. Pp. 317-333. ACS Professional Reference Book. ISBN 0-8412-1173-6. American Chemical Society.

Taylor, J.K. 1988. Defining the Accuracy, Precision, and Confidence Limits of Sample Data. Chapter 6, pages 102-107, in Lawrence H. Keith (editor), Principles of Environmental Sampling. ACS Professional Reference Book. American Chemical Society. Columbus, OH.

Taylor, B.N. and C.E. Kuyatt. 1994. Guidelines for evaluating and expressing the uncertainty of NIST measurement results. NIST Technical Note 1297. National Institute of Standards and Technology, US Department of Commerce, Washington, DC.

USEPA. 1999. Rapid Bioassessment Protocols for Use in Streams and Wadeable Rivers: Periphyton, Benthic Macroinvertebrates and Fish. Second Edition. Office of Water, Washington, D.C. Authors: M.T. Barbour, J. Gerritsen, B.D. Snyder, and J.B. Stribling. EPA 841-B-99-002. Office of Water, Washington, D.C.

 

Biological Indicators | Aquatic Biodiversity | Statistical Primer

Local Navigation

Jump to main content.