An official website of the United States government.

We've made some changes to EPA.gov. If the information you are looking for is not here, you may be able to find it on the EPA Web Archive or the January 19, 2017 Web Snapshot.

Community Multiscale Air Quality Modeling System (CMAQ)

The Atmospheric Model Evaluation Tool

Introduction

The Atmospheric Model Evaluation Tool (AMET) was developed to aid in its evaluation of the meteorological and air quality models within the CMAQ modeling system. AMET organizes, provides consistency and speeds-up the evaluation process for operational meteorological and air quality model simulations. Although it was developed specifically to aid in the evaluation of the CMAQ modeling system, the AMET software can be adapted to work with other modeling systems. 

AMET is used to pair observations and gridded model output in space and time and to evaluate the model performance for a set of predicted or diagnosed atmospheric fields. There are separate modules in AMET for evaluating meteorological and air quality model output. This separation is necessary because both the observed and predicted meteorological and air quality data are quite different, utilizing different file formats for both the observed and model data. In addition, the observed meteorological and air quality data are often obtained from networks that use different sampling protocols, which can make pairing meteorological and air quality data together difficult. One advantage of separate meteorological and air quality modules in AMET is that the modules can be installed individually, allowing a user to reduce installation time and complexity if only meteorological or air quality analysis is required. 

Evaluation of model simulations, which usually includes comparing the model-predicted values against the available observed values, is critical to establishing the model’s credibility (Dennis et al., 2010). For the development of the Community Multiscale Air Quality model evaluation of model predicted concentrations for many different atmospheric pollutants is required since the model is used to make regulatory decisions by the federal as well as state governments. The evaluation is further complicated by the requirement of evaluating the meteorological predictions that are used to provide gridded meteorological data to the CMAQ model.  

Roughly a decade or so ago, CMAQ model simulations typically spanned only several weeks to a month (in some rare cases several months) and were limited spatially (for example, to the eastern half of the U.S.). The increase in computing power over time has allowed for larger model domains, longer simulation time periods and a greater number of model simulations (e.g. sensitivity and ensemble modeling studies).

Today, CMAQ simulations routinely span several months to a year and in some cases decades, and have expanded spatially to cover the continental U.S. and the Northern Hemisphere. The result is terabytes of data available for evaluation against observed data, which conventional methods of data analysis (e.g. spreadsheets) are not well suited. In addition to longer duration simulations and larger domains, the shorter wall-clock model run times have resulted in many more model simulations being performed. The increase in the duration, spatial size and number of model simulations all require greater organization and efficiency to systematically evaluate the large amounts of data being produced.

AMET Overview

AMET is built on several open-source software packages and uses the MySQL database software to store the paired observation and model output along with various metadata.  The pairing is performed by several scripts, which also insert the paired data into the MySQL database using Fortran and R programs.  Finally, the analysis of the data utilizes the R statistical software package.  Figure 1 presents a flow chart of the basic structure of the AMET software.

Flow chart showing the basic structure of the AMET meteorological and air quality modelsFig. 1.  Flow chart showing the basic structure of the AMET meteorological and air quality modules.

The first step required as part of the AMET software is to pair the observed and modeled values in space and time.  In the air quality module of AMET, this is accomplished through the Site Compare ("sitecmp") software. Site Compare performs matching using the nearest neighbor approach, and applies temporal averaging to the modeled data if required.  The formatted observation data files needed for running the Site Compare utility are available for 2000 through 2015 from the CMAS Center Data Clearinghouse under the heading "2000-2015 North American Air Quality Observation Data": https://www.cmascenter.org/download/data.cfmExit

In most cases, the Site Compare program is able to read the observation data files that are provided directly from the major air quality networks in the United States. 

The air quality networks that Site Compare/AMET currently support are the EPA's AQS, the Chemical Speciation Network (CSN), the Interagency Monitoring of PROtected Visual Environments (IMPROVE) network, the Clean Air Status and Trends Network (CASTNET), the National Atmospheric Deposition Program (NADP) networks (AIRMON, AMON and NTN), the SouthEastern Aerosol Research and CHaracterization Study experiment (SEARCH) and the Aerosol Robotic Network (AERONET).

The meteorological matching is performed using either bilinear interpolation or nearest neighbor. Additionally, AMET attempts to match coastal monitors with land based grid-cells as opposed to oceanic grid-cells, and both the model output and observations are considered to be on identical temporal resolutions (i.e. instantaneous values representing one hour). The meteorological module is designed to use observations from the Meteorological Assimilation Data Ingest System (MADIS). The MADIS is a widely used, comprehensive meteorological observational dataset, and requires a free subscription for access to the observation data. 

Currently, the meteorological module of the latest version of AMET is designed to work with meteorological output from either the Weather Research and Forecasting (WRF; Skamarock et al., 2008) model or the Model for Prediction Across Scales (MPAS; https://mpas-dev.github.io/). 

The AMET meteorological module supports evaluation of both surface and upper-air meteorological fields. The surface fields currently available in the AMET meteorological module include temperature, water vapor mixing ratio, wind speed, and wind direction. The upper-air fields available for analysis are wind speed and direction from vertical wind profilers; wind speed, direction, and potential temperature from aircraft profile (ACARS) data; and potential temperature, relative humidity, wind speed and direction from the National Weather Service balloon soundings and the model reanalysis data from the RAWINDS program.

The AMET Database

MySQL is an open-source relational database software that can used to store and quickly access large amounts of data. Aside from providing a convenient method for storing the observation-model pairs, the use of a relational database allows the entire set of data to be easily subset based on any criteria available in the database (e.g. by specific date, location or location metadata). The ability to subset the data quickly is extremely useful since model performance often varies spatially and temporally, and the spatial and temporal subsets are often different for each pollutant or chemical species. The MySQL software is free to individual users and is available for many different computing platforms.

R Statistical Package for AMET

The R project for Statistical Computing, “R”, is an open-source software for statistical computing and graphics based on the “S” language, and is available for many different computer platforms. The R software can be used through a command line interface or with the use of scripts, and contains a large number of functions for computing common statistical metrics and generating graphics (e.g. box plots and scatter plots). Additionally, the R software contains a library of modules, including a module to interface with the MySQL database software. All these features make R an attractive computational and plotting software package for AMET.

The information stored in the database is queried through for each R analysis script. Through the various commands included in the R MySQL library, the database can be queried based on any of the information stored in the database tables. For example, one could limit their data analysis to a single hour of the day, a particular day or week, or to an individual state or collection of states. The AMET scripts include instructions for performing common queries (e.g. queries based on a date range or location). However, anyone familiar with basic database queries could easily formulate their own query criteria to subset the data in any way they choose.

The AMET software already includes a collection of R scripts that perform various analyses such as computing standard statistical metrics such as bias, error and correlation; creating common statistical plots such as scatter plots and time series plots; and creating unique plots designed specifically for AMET. Table 1 lists the R scripts currently available for the AMET meteorological and air quality modules, along with a brief description of the script functions. These scripts are used to generate statistics and plot the results using the various plotting functions available in the R software. Many of the output products available in AMET are frequently used in the air quality modeling community. In addition to using the provided R scripts, the more advanced user could use these scripts as a template for creating new R scripts tailored to their particular application and analysis.

Table 1. List of R scripts currently available in the AMET meteorological and air quality modules.

Script Name

AMET Module

Description

MET AQ

Meteorological/Air Quality

Compares air quality and meteorological data. Creates a spatial plot of correlation and a time series plot of bias.

Spatial Surface Plot

Meteorological

Creates a series of maps comparing surface observations and model values. See Figure 2 for an example.

Summary Plot

Meteorological

Creates a summary plot of a series of meteorological variables. Includes time series, scatter and box plots. See Figure 3 for an example.

Met Time series

Meteorological

Creates a time series plot of temperature, mixing ratio, wind speed and direction for one or more sites. See Figure 4 for an example.

Daily Bar Plot

Meteorological

Creates a series of daily bar plots for all sites in the domain.

Profile Plot

Meteorological

Creates time-height plots of wind speed and direction profile statistics. See Figure 5 for an example.

Plot RAOB

Meteorological

Creates vertical profile plots of rawinsonde sites. 

Wind Prof

Meteorological

Creates wind arrow profiles for a set of specific monitoring sites.

Box plots

Air Quality

Creates a box plot of observed and modeled quartiles.

Bugle Plot

Air Quality

Creates a “bugle plot,” which plot bias and error as a function of concentration and includes curves representing performance goals. See Figure 6 for an example.

PAVE Overlay

Air Quality

Creates a PAVE overlay file of observed values.

Spatial Plots

Air Quality

Numerous spatial plots available. See Figure 7 for an example.

Scatter Plots

Air Quality

Various scatter plots available. Includes standard model/ob scatter plot, model to model scatter plot, model skill scatter plot, etc. See Figure 8 for an example.

Soccer Plot

Air Quality

Creates a “soccer goal” plot, which plots bias verses error with performance lines in the shape of a soccer goal. See Figure 6 for an example.

Stacked Bar Plots

Air Quality

Creates a stacked bar plot of total PM2.5 mass from networks with speciated PM2.5 data and color codes the contribution from individual PM2.5 species (e.g. sulfate, nitrate, ammonium, total carbon) to the total PM2.5 mass. See Figure 9 for an example.

Stats Plots

Air Quality

Creates spatial plots of NMB, NME, FB, FE and correlation, along with a comma delimited file with various statistics.

Time Series Plots

Air Quality

Various time series plots available (e.g. Model/Ob, Model/Model, time averaged). See Figure 10 for an example.

References

Dennis, R., Fox, T., Fuentes, M., Gilliland, A., Hanna, S., Hogrefe, C., Irwin, J., Rao, S.T., Scheffe, R., Schere, K., Steyn, D., Venkatram, A. (2010). A framework for evaluating regional-scale numerical photochemical modeling systems. Environ. Fluid Mech., 10, doi: 10.1007/s10652-009-9163-2Exit

Skamarock, W.C., Klemp, J.B., Dudhia, J., Gill, D.O., Barker, D.M., Duda, M.G., Huang, X-Y, Wang, W., and Powers, J.G. (2008). A description of the advanced research WRF version 3. NCAR Tech Note NCAR/TN 475 STR, 125 pp, [Available from UCAR Communications, P.O. Box 3000, Boulder, CO 80307.].

Related Publications/Presentations

Appel, K.W., Gilliam, R.C., Davis, N., Zubrow, A., and Howard, S.C. (2011). Overview of the Atmospheric Model Evaluation Tool (AMET) v1.1 for evaluating meteorological and air quality models, Environ. Modell. Softw.26(4), 434-443.

Galmarini, S., Bianconi, R., Appel, K.W., Solazzo, E., Mosca, S., Grossi, P., Moran, M., Schere, K., & Rao, S.T. (2016). ENSEMBLE and AMET: two systems and approaches to a harmonized, simplified and efficient assistance to air quality models development and evaluation. Atmos. Environ., 53, 51-59.

Appel, K.W., & Gilliam, R.C. (2008). Overview of the Atmospheric Model Evaluation Tool (AMET). 9th Conference on Air Quality Modeling, October 9-10, 2008, RTP, NC.

Appel, K.W., & Gilliam, R.C. (2008). Overview of the Atmospheric Model Evaluation Tool (AMET). 7th Annual CMAS Models-3 Users’ Conference, October 6-8, 2008, Chapel Hill, NC.

Additional Figures

Figure 2. Sample Spatial plot from the AMET meteorological module.  The plot is the root-mean-square error (RMSE) of 2-m temperature for June 2016.Fig. 2. Sample Spatial plot from the AMET meteorological module.  The plot is the root-mean-square error (RMSE) of 2-m temperature for June 2016.
 

Figure 3. Sample Summary plot from the AMET meteorological module.  The plot on the left is for temperature, while the plot on the right is for wind direction.Fig. 3. Sample Summary plot from the AMET meteorological module.  The plot on the left is for temperature, while the plot on the right is for wind direction.
 

Figure 4. Sample Time Series plot from the AMET meteorological module.  The top plot is for 2-meter mixing ratio, the middle for 2-meter temperature and the bottom for 10-meter wind speed.Fig. 4. Sample Time Series plot from the AMET meteorological module.  The top plot is for 2-meter mixing ratio, the middle for 2-meter temperature and the bottom for 10-meter wind speed.
 

Figure 5. Sample Profile Plot from the AMET meteorological module.  The color shading and white lines indicates model-predicted wind speed, while the black dots indicate the model-predicted planetary boundary layer height.Fig. 5. Sample Profile Plot from the AMET meteorological module.  The color shading and white lines indicates model-predicted wind speed, while the black dots indicate the model-predicted planetary boundary layer height.
 

Figure 6. Sample plots from the a) Soccer Plot script and b) the Bugle Plot script from the AMET air quality module.Fig. 6. Sample plots from the a) Soccer Plot script and b) the Bugle Plot script from the AMET air quality module. 
 

Figure 7. Sample Spatial Plot from the AMET air quality module. Shown is the difference in hourly ozone bias between two model simulations.Fig. 7. Sample Spatial Plot from the AMET air quality module. Shown is the difference in hourly ozone bias between two model simulations.
 

Figure 8. Example Scatter Plot for maximum daily 8-hr average ozone (shown is the Skill Scatter Plot) from the AMET air quality module.  Fig. 8. Example Scatter Plot for maximum daily 8-hr average ozone (shown is the Skill Scatter Plot) from the AMET air quality module. 
 

Figure 9. Sample plot from the Stacked Bar Plot script from the AMET air quality module.  Fig. 9. Sample plot from the Stacked Bar Plot script from the AMET air quality module. 
 

Figure 10. Example Time Series plot from the AMET air quality module. Shown on top is observed and two model simulations of ozone mixing ratio for July 2011 while the bottom plot shows the corresponding model bias.Fig. 10. Example Time Series plot from the AMET air quality module. Shown on top is observed and two model simulations of ozone mixing ratio for July 2011 while the bottom plot shows the corresponding model bias.