| STORET| US EPA

What's new with the STORET Data Warehouse?

The STORET Data Warehouse now provides downloaded data in a compressed format using immediate or overnight batch processing.

This means that when downloading a file from the STORET Warehouse, you will no longer see a "pop-up window" filled with raw results in a text file format. Instead, a link to a zipped file containing your data is now delivered to you via email. When querying data from the Warehouse, the system will ask you to provide an email address to which the link is sent. The link then allows you download the zipped file containing the raw data you requested, metadata for this data, and other associated reference documents.

This page helps answer many of the questions that users have regarding these changes in the way data from the STORET Warehouse is being delivered. Please see How to Query and Download Data for detailed instructions on how to generate, download, export (to Microsoft Excel or Access), and analyze a data query from the STORET Data Warehouse

Why did the STORET Team do this?
What is a .tar.gz file and why isn't my download a textfile?
Why are there more files than just my query results in the downloaded file?
I can't make sense of my Results file when I open it.
Is there any useful information in the metadata file?
How do I make sense of downloaded file names?

Why did the STORET Team do this?

When the older "pop-up window" was used for immediate downloads, it was possible for users to lose data if the file was saved before the real-time download was completed. This is why the immediate "pop-up" option was discontinued.

A requirement of making the documented quality of the data more accessible meant adding the metadata file and other project-level reference documents to the compressed file. Having multiple files as part of the download mandated a compressed file format and is the reason why an email address is now needed for both Immediate and Overnight Downloads.

Smaller datasets can still be downloaded immediately. Processing smaller, immediate queries is still completed and downloadable from the URL within 1-30 minutes and they are not overnight batches unless selected as such.

As part of a future upgrade, using this compressed format will allow us to eliminate the need for three separate download queries (Regular, Biological, Habitat) which has confused a number of users in the past. Soon a single query will return all three result files in a compressed format.

What is a .tar.gz file and why isn't my download a textfile?

The downloaded .tar.gz file is a compressed (zipped) file. This means that you will need compression software like WINZIP® to open the file. It was necessary to move to this compressed format for both Immediate and Overnight downloads as all downloads now contain multiple files, including your results textfile.

Why are there more files than just my query results in the downloaded file?

In addition to the results data that you queried, you now automatically receive the metadata file and any (if any) reference documents that are associated with the organizations that own the data. This is so that you can better determine the quality of the data you downloaded. The result file(s) contain the raw data that you queried; regular, biological, and habitat result queries are found in individual files. The metadata file includes information about the organizations that own the data including contacts, methods, labs, and other info. The reference documents can be pictures, datalogger results, QAPPs, or any other project-level documents associated with the data owning organizations. The Warehouse currently does not serve up station or result level documents.

I can't make sense of my Results file when I open it.

All result (RegResult, BioResult, HabResult) files are in a tilde "~" delimited text format, which just means that we use the character "~" to separate the fields of the dataset. This is important when loading files into Excel, Access, or another database that can organize the file (make it readable) using the delimiter. Additionally, metadata files are in a tab delimited format and can be easily opened in Microsoft Word or Excel. Please see How to Query and Download Data for detailed instructions on how to generate, download, export (to Microsoft Excel or Access), and analyze a data query from the STORET Data Warehouse

Is there any useful information in the metadata file?

The metadata file contains the following summaries that can be used to contact the data owners, create a station list, describe the methods and procedures used, qualify the labs, correctly cite the data, and generally determine the quality of the data for yourself:

Organization summary
Cooperating Organization Summary
Project Summary
Sample Collection/Creation Procedure Summary
Sample Gear and Equipment Configuration Summary
Sample Preservation and Handling Profile Summary
Analytical Procedure and Equipment Detail Summary
Laboratory Summary
Lab Sample Preparation Procedure Summary
Bibliographic Citation Summary

How do I make sense of downloaded file names?

The files found in a download have four main components:

Type of Document: denotes whether the downloaded document file contains raw data ("Data") or is a reference document associated with the raw data ("RefDoc").

Data_XYZ_20070322_205714_RegResults.txt
RefDoc_XYZ_20070322_205714_Project_COPSBDC1_cua0001.pdf

Unique Identifier: user provided 3 character ID, followed by the date/time stamp to ensure user can uniquely identify a download.

Data_XYZ_20070322_205714_RegResults.txt
RefDoc_XYZ_20070322_205714_Project_COPSBDC1_cua0001.pdf

Type of Data: denotes what type of raw results data the downloaded document file contains ("RegResults" for regular results, "BioResults" for biological results, and "HabResults" for habitat results), which file is the metadata file ("Metadata"), or provides the Project ID and file name for the downloaded reference documents.

Data_XYZ_20070322_205714_BioResults.txt
Data_XYZ_20070322_205714_HabResults.txt
Data_XYZ_20070322_205714_RegResults.txt
Data_XYZ_20070322_205714_Metadata.txt
RefDoc_XYZ_20070322_205714_Project_COPSBDC1_cua0001.pdf

Type of File: denotes the extension for the format of the document file (.txt, .pdf, .bmp, .gif, .jpg).