Tools for Automated Data Analysis (TADA)
What is TADA?
The U.S. Environmental Protection Agency (EPA) TADA (Tools for Automated Data Analysis) encompasses an R package and series of R Shiny applications currently under development – new features are added every month. These tools are designed to help Tribes, Tribal Nations, Pueblos, States and other stakeholders more efficiently compile and evaluate Water Quality Portal (WQP) monitoring data collected from surface water monitoring sites. Participating organizations submit their data to the WQP using the EPA's WQX, a framework designed to map their data holdings to a common data structure. WQP is the nation's largest source for water-quality monitoring data in the nation. TADA is designed specifically for use with WQP/WQX data, or any data formatted in the WQX/WQP schema. TADA also assists with pulling in relevant information from the EPA Assessment, Total Maximum Daily Load (TMDL) Tracking and Implementation System (ATTAINS). Once finished, TADA aims to meet the following user requirements: 1) data discovery and cleaning, 2) assessment unit and use integration, and 3) criteria and methodologies integration to support assessment unit-use-parameter level analyses. The TADA Team is using an agile development approach. User requirements are still being adjusted as needed during development using frequent feedback solicited from the user community.
How can I access TADA and get started using it?
Code repositories for the EPATADA R package, TADAShiny, TADAShinyJoinToAU, TADAShinyAnalyze, and TADACommunityHub are available on GitHub. The TADA Shiny applications provide easy to use web interfaces on top of the EPATADA R Package. Stakeholders are encouraged to test the functionality and provide feedback on TADA. Moreover, open-source software provides an avenue for water quality data originators and users to develop and share code, and the TADA Team welcomes your contributions! We encourage you to find more information on how to contribute and reach out as needed. A collaborative community dedicated to this effort is integral to the success of TADA and its users, where contributors can discover, share, and build the package functionality over time.
What are the capabilities of TADA?
EPATADA R Package: Visit the EPATADA GitHub repository for instructions on installation, contributing, viewing code, reporting issues or suggesting new features. The EPATADA Package Index provides a list of functions, their definitions and examples. The Articles page provides example workflows and past training materials to walk users through each TADA module using the R package. R users may choose to incorporate individual functions into their existing workflows (including their own Shiny apps) as desired allowing for maximum customization. The TADA R Shiny applications described below leverage the EPATADA R package so that the functionality can be made available to non-R users.
TADAShiny (Module 1: Data Discovery and Cleaning) retrieves data from the WQP by leveraging the USGS dataRetrieval R Package and runs it through a series of quality control screens and data wrangling steps. Features include flagging invalid results and metadata using validation reference tables, harmonization of synonyms, result and depth unit conversions, censored (detection limit) data substitutions, dataset filtering, and data visualizations. TADA leverages the EPA Water Quality eXchange (WQX) QAQCCharacterisiticValidation domain value service to flag results with invalid metadata. Users can review and download summary information about their dataset, along with a data file and that is ready for additional manual review and use in subsequent analyses. Within the application, users decide to flag data for removal or keep data depending on its quality and relevance for their specific use case.
Data in the WQP are not altered by TADA – if underlying data quality issues are found using TADA, users can contact the WQX helpdesk (WQX@epa.gov) for assistance fixing their organization’s data in the WQP. Only data submitting organizations are allowed to make changes to their data. If WQP data users find data quality issues for which they are not the data owner, they may also reach out to the WQX helpdesk who can let the data owner know about the issue. See the TADAShiny GitHub repository to view code, report issues, view installation instructions, etc.
TADAShinyJoinToAU (Module 2: Assessment Unit and Use Integration) integrates the output from Module 1 with data from ATTAINS to assist users in matching WQP monitoring locations with assessment units and uses. TADA leverages the ATTAINS geospatial web services to perform spatial joins between assessment units and WQP monitoring locations. Users review these assignments on a map before approving the monitoring location/assessment unit crosswalk. Then ATTAINS assessment data are accessed via Expert Query web services to identify designated uses assigned to each assessment unit in previous assessment cycle. An additional EPA R package, rExpertQuery, facilitate this use of ATTAINS Expert Query data. TADAUsers may make any required changes to the monitoring location/assessment unit crosswalk or the designated uses assigned to each assessment unit before moving to the next module. Module 2 is currently in development. You are welcome to visit the TADAShinyJoinToAU GitHub repository, test it out, and share feedback.
TADAShinyAnalyze (Module 3: Criteria and Methodologies Integration) incorporates the outputs from Module 1 and Module 2 to analyze and summarize WQP data. Users can customize a TADA-compatible table of criteria and methodologies and optionally submit the table to a shared repository in the TADACommunityHub to make the information findable, accessible and reusable. Users are guided through the process of assigning criteria from their organization or EPA 304(a) recommended criteria for each parameter/use combination and matching WQP characteristic names with ATTAINS parameter names. Users can choose to populate the criteria and methodologies table with magnitude values using a parameter and use alias table that maps the ATTAINS parameter names and use names with the Criteria Search Tool (CST) pollutant names and use names. Functionality is being developed to run analyses at the WQP monitoring location or ATTAINS assessment unit scale. Module 3 is currently in development. You are welcome to visit the TADAShinyAnalyze GitHub repository, test it out, and share feedback.
Background and Impetus for TADA
In 2015, the EPA began developing two tools to help organizations more easily discover, clean, visualize, and analyze data from the WQP. Both were developed using R and have an R Shiny interface intended to be user friendly. These tools, the Data Discovery Tool and Data Analysis Tool, are TADA’s predecessors and no longer maintained. The TADA project began with the intention to utilize and build on the existing functionality of these tools, and to review open-source tools developed by stakeholders for potential incorporation. In addition, the TADA team began conducting outreach to better understand tool requirements and potential barriers to tool use and adoption by stakeholders. For this purpose, the team initiated the TADA Working Group in 2020, which meets regularly and has continuously grown in membership. During the initial requirements gathering process in 2020-2021, the TADA Working Group developed and reviewed issue papers, answered assessment questions, found commonalities in assessment processes, and created a space for open-source development. The working group produced a Master List of Requirements for TADA and is maintaining an inventory of open-source water quality data tools. The team learned a tremendous amount through that process and looks forward to continued discourse throughout development of all TADA tools.
Join the TADA Working Group
Working Group Mission: To share and develop R code for evaluating and visualizing WQP data more efficiently though collaboration and open-source programming. This includes working together to find commonalities in assessment processes across the nation, creating flexible tools that can be easily customized to work within existing workflows, supporting each other in learning R, and ensuring products will be accessible to organizations most in need. New members are welcome! Contact the TADA Team (mywaterway@epa.gov) for more information.