Jump to main content or area navigation.

Contact Us

Ecosystems Research Division

Model Evaluation

   

This page provides links to non-EPA web sites that provide additional information about models. You will leave the EPA.gov domain and enter another page with more information. EPA cannot attest to the accuracy of information on that non-EPA page. Providing links to a non-EPA Web site is not an endorsement of the other site or the information it contains by EPA or any of its employees. Also, be aware that the privacy protection provided on the EPA.gov domain (see Privacy and Security Notice) may not be available at the external link.  exit EPA

Modeling frameworks are important software tools that can be leveraged to conduct integrated environmental modeling, as in support of multimedia ecological exposure research. For such research to be relevant to policy, rigorous and quantitative model evaluation is necessary.

This tool survey was conducted to identify relevant model evaluation tools, nearly 70 were identified. The source of this table, the manuscript "Evaluating Integrated Environmental Models: A Survey of Concepts and Tools. L. Shawn Matott, Justin E. Babendreier, and S. Thomas Purucker (in prep) evaluates concepts for environmental modeling and surveys relevant tools." Currently, the prevalence of multiple uncertainty taxonomies and inconsistent model evaluation terminology is a source of considerable confusion. Two proposals in the manuscript help to remedy this problem: a minimal, unifying taxonomy for sources of uncertainty and a glossary of common terminology with "standard" definitions. Practical application of multiple independently-developed tools is hindered by software availability, differing input-output formats, and coding languages. The manuscript concludes with a review of proposed standards and application programming interfaces (APIs) that have been developed to facilitate tool unification.

In the table below, each identified tool is listed and a download link is provided. The table also summarizes the following information:

  • DA - Data Analysis (a: 1 - Population data, 2 - Geospatial data, 3 - Time-series data)
  • IA - Identifiability Analysis (b: 1 - Temporal, 2 - Behavioral, 3 - Spatial)
  • PE - Parameter Estimation (c: 1 - local search, 2 - global search, 3 - hybrid search, 4 - importance sampling, 5 - MCMC sampling)
  • UA - Uncertainty Analysis (d: 1 - Monte Carlo sampling, 2 - stratified sampling, 3 - importance sampling, 4 - Approximation methods)
  • SA - Sensitivity Analysis (e: 1 - screening methods, 2 - local methods, 3 - correlation-based, 4 - regression-based, 5 - variance-based)
  • MMA - Multi-model Analysis (f: 1 - qualitative, 2 - quantitative)
  • BN - Bayesian Networks (g: 1 - Hiearchical bayesian network, 2 - Bayesian maximum entropy, 3 - Bayesian decision network)
  • CIT - Number of citations determined by a search of SCOPUS database
  • AV - Available materials (h: 1 - method description only, 2 - source code, 3 - manual, 4 - executable)
  • DIS - Form of software distribution (i: 1 - web download, 2 - on request, 3 - software not available)
Tool Name DAa IAb PEc UAd SAe MMf BNg CIT AVh DISi
ACE (Alternating Conditional Expectation)   2           2 2 1
ACUARS (Automatic Calibration and Uncertainty Assessment using Response Surfaces)     4 3 1     2 1 3
AMALGAM (A Multi-ALgorithm Genetically Adaptive Multiobjective method)     3         5 2 2
BaRE (Bayesian Recursive Estimation)     4 3 1     75 1 3
BATEA (BAyesian Total Error Analysis)     5 3 1     34 2 1
BFL (Bayesian Filtering Library)   1           1 2-3 1
BFS (Bayesian Forecasting System)             1 68 1 3
BMC (Bayesian Monte Carlo)     4 3 1     39 1 3
BMElib (Bayesian Maximum Entropy - library)             2 54 2-4 1
BUGS (Bayesian inference Using Gibbs Sampling (plus extensions))             1 576 2-4 1
CANOPI (Confidence ANalysis Of Physical Inputs)       3       4 1 3
DAKOTA (Design Analysis Kit for Optimization and Terascale Applications)     1-3 1-2,4 2     72 2-4 1
DBM (Data-based Mechanistic modeling)   2           187 2-4 1
DDS, DDS-AU (Dynamically Dimensioned Search, DDS for Approximation of Uncertainty )     2,4 3 1     2 2,4 1
DUE (Data Uncertainty Engine) 1-3             7 2-4 1
DYNIA (DYNamic Identifiability Analysis)   1-2           51 2-4 1
EESA (Elementary Effects Sensitivity Analysis)         1     4 2 1
FANN (Fast Artificial Neural Network Library) n/a - tool for surrogate-based modeling 6 2-4 1
GEM (Gaussian Emulation Machine)     5 1 5     9 3-4 1
GLUE (Generalized Likelihood Uncertainty Engine)     4 3 1     539 3-4 1
HBC (Hierarchical Bayesian Compiler)             1 0 2-4 1
HDMR (High Dimensional Model Representation) n/a - tool for surrogate-based modeling 44 2 1
IBUNE (Integrated Bayesian UNcertainty Estimator)     5 3 1 2   3 1 3
JAGS (Just Another Gibbs Sampler)             1 3 2-4 1
JUPITER (Joint Universal Parameter IdenTification and Evaluation of Reliability)     1   2     4 2-4 1
LH-OAT (Latin Hypercube Sampling - One Factor At a Time)         1     10 2-4 1
MARS (Multivariate Adaptive Regression Splines) n/a - tool for surrogate-based modeling 814 3-4 1
MCAT (Monte Carlo Analysis Toolbox)   1-2     1,3     9 2-4 1
MCMC-SRSM (Markov Chain Monte Carlo - Stochastic Response Surface Method)     5 3 1     5 2-4 2
mGLUE (modified GLUE)     4 3 1     6 1 3
MICA (Model-Independent Markov Chain Monte Carlo Analysis)     5 3 1     5 2,4 2
MIPET (Model Independent Parameter Estimation Toolbox)     1,3         4 2,4 2
MLBMA, BMA (Maximum Likelihood Bayesian Model Averaging )           2   24, 300 2,3 1
MMA (Multi-Model Analysis)           2   11 2-4 1
MOCOM (Multi-Objective COMplex evolution)     2         159 2 2
MOGSA (Multi-Objective Generalized Sensitivity Analysis )         3     63 2 2
MOSCEM (Multi-Objective Shuffled Complex Evolution Metropolis)     5 3 1     69 2-4 1
NLFIT (Bayesian Non-Linear Regression Suite )     2,5 3 1     135 2,3 2
NSGA (Non-dominated Sorting Genetic Algorithm)     2         814 2 1
NUSAP (Numeral, Unit, Spread, Assessment, and Pedigree)           1   37 n/a
OSTRICH (Optimization Software Toolkit for Research In Computational Heuristics)     1-3   2 2   7 2-4 1
ParaSol (Parameter Solutions )     4 3 1     2 2-4 1
PEAS (Parameter Estimation Accuracy Software)     1   2     2 2-4 2
PEST (Parameter EStimation Toolkit)     1-3 3-4 2     197 2-4 1
PIMLI (Parameter Identification Method - Localization of Information)   1 5 3 1     24 1 3
PSO (Particle Swarm Optimization)     2         2473 2-4 1
PyMC (Python-based - Markov Chain Monte Carlo Library)     5 3 1   1 0 2-4 1
R (Package for statistical computing) 1-3             2439 2-4 1
ReBEL (Recursive Bayesian Estimation Library)   1           6 2-4 1
RIMME (Random-search Inverse Methodology for Model Evaluation)       3 3     4 2 1
SADA (Spatial Analysis and Decision Assistance ) 1-2             2 3-4 1
SAMPLING/ANALYSIS (Screening Level SAMPLING and Sensitivity ANALYSIS Tool)         1     5 2 2
SARS-RT (Sensitivity Analysis based on Regional Splits and Regression Trees)         3-4     11 1 3
SCE (Shuffled Complex Evolution)     2         533 2-4 1
SCEM (Shuffled Complex Evolution Metropolis)     5 3 1     74 2-4 1
SIMLAB (Simulation Laboratory for UA/SA)       1-2 1,3-5     101 2-4 1
SODA (Simultaneous Optimization and Data Assimilation method)   1 5 3 1     26 1 3
SOLO (Self-Organizing Linear Output map)   2           29 2 2
SRSM (Stochastic Response Surface Method) n/a - tool for surrogate-based modeling 32 2 1
SUFI,SUFI-2 (Sequential Uncertainty-FItting algorithm )     4         16 1 3
SUNGLASSES (Sources of UNcertainty GLobal Assessment using Split SamplES)       3       3 2-4 1
UCODE (Universal CODE for Inverse Modeling)     1 4 2     148 2-4 1
UNCERT (UNCERTainty analysis, geostatistics and visualization toolkit) 1-3             20 2-4 1
UNCSIM (UNCertainty SIMulator)   1 2,4-5 1-3 2     8 2-4 1
WebGUAC (Website Guidance for Uncertainty Assessment and Communication)           1   17 n/a
Legend
DA = Data Analysis (a: 1 - Population data, 2 - Geospatial data, 3 - Time-series data)
IA = Identifiability Analysis (b: 1 - Temporal, 2 - Behavioral, 3 - Spatial)
PE = Parameter Estimation (c: 1 - local search, 2 - global search, 3 - hybrid search, 4 - importance sampling, 5 - MCMC sampling)
UA = Uncertainty Analysis (d: 1 - Monte Carlo sampling, 2 - stratified sampling, 3 - importance sampling, 4 - Approximation methods)
SA = Sensitivity Analysis (e: 1 - screening methods, 2 - local methods, 3 - correlation-based, 4 - regression-based, 5 - variance-based)
MMA = Multi-model Analysis (f: 1 - qualitative, 2 - quantitative)
BN = Bayesian Networks (g: 1 - Hiearchical bayesian network, 2 - Bayesian maximum entropy, 3 - Bayesian decision network)
CIT = Number of citations determined by a search of SCOPUS database
AV = Available materials (h: 1 - method description only, 2 - source code, 3 - manual, 4 - executable)
DIS = Form of software distribution (i: 1 - web download, 2 - on request, 3 - software not available)

 

ERD Home | NERL Home | ORD Home