An official website of the United States government.

We've made some changes to EPA.gov. If the information you are looking for is not here, you may be able to find it on the EPA Web Archive or the January 19, 2017 Web Snapshot.

Benchmark Dose Tools

BMDS Wizard Basics and Using BMDS Wizard to Evaluate Dichotomous Datasets

The previous exercise showed you how to evaluate a simple dichotomous dataset using BMDS. This exercise walks you through how to perform the same analysis using BMDS Wizard. This exercise also demonstrates various features and functions of the BMDS Wizard interface, including running the models and printing a summary report. 

Refer to the online training for deeper information on dichotomous BMD model concepts and the technical guidance document for specific guidelines on selecting and reporting dichotomous data.

The approach for selecting a "best" modeling result described on this page applies to noncancer analyses only. You should review the technical guidance for choosing the appropriate stage of a Multistage model for cancer modeling.

On this page:

Prerequisites

Refer to the setup instructions in the BMDS Wizard User's Reference Guide, Section 1. You should have, at a minimum, done the following:
  • Copied the templates to a directory that will hold all of your BMDS Wizard modeling spreadsheets. (Described in the BMDS Wizard User's Reference Guide, Section 1.2.1)
  • Renamed the spreadsheets and placed them in a new folder or path. (Section 2.1)
  • Ensured macros are enabled in the spreadsheet. (Section 1.2.3)

Top of Page

Exercise 1

  1. In your directory that holds a copy of all the BMDS Wizard template files, open the file BMDS Wizard v1.11-dichotomous-ExtraRisk.xlsm.
  2. When the file opens, select the Enable Content button (if visible) to enable the workbook macros.
  3. On the Startup tab, select the Main tab. The Main worksheet tab is displayed.
  4. On the Main tab, select the 1) Setup Instructions button for information on what you will do to create a session.
  5. On the Main tab, fill in the Study and Modeling Inputs fields as follows:
    1. BMDS Installation and Ouput File directories: Use the Select Folder buttons to navigate to the directories.
    2. BMD ID Number: 1
    3. Study & Year: Exercise_1
    4. Endpoint description: blank
    5. Dose Units: mg
    6. Text for Study Heading (optional): blank
    7. BMD or BMC Calculated?: Select the cell and use the down-pointing triangle to select BMD.
    8. Select Dataset Type: Select the cell and use the down-pointing triangle to select Dichotomous.
    9. Enter Study Data: Leave blank for now.
  6. Note that the box labeled “Add new models to BMDS Session,” which is below these cells, is empty.Add new models to BMDS Session box
  7. Click the cell in C21 (“Click here to enter data”) to move to the Data worksheet. You may need to click on the cell's text to activate the hyperlink.
  8. On the Data worksheet, enter the numbers from the following table. (This is the same data that you entered for the BMDS dataset exercise.) The quick way: copy the rows of numbers below and paste them into the Data worksheet starting at the Dose row.
    Dose N Effect
    0 50 0
    1000 50 2
    3000 50 6
    8000 50 25
  9. Click Return to Main at the top of the Data worksheet to display the Main tab. Pay attention to the following items:
    • Note that the box labeled “Add new models to BMDS Session” is now populated with BMDS models. Only those models that are compatible with the dose-response group you’re using will be modeled by BMDS. The most likely models to include in the BMDS session will be displayed in the BMDS Model Option Setups area of the worksheet.
    • In BMDS, you had to manually change parameters to generate 1st, 2nd, and 3rd degree multistage. But the BMDS Wizard includes all three.
    • Note the BMDS Model Option Setups section where you can tweak model parameters. These rows display parameters for the models that are compatible with the data you entered. Although we will not alter any parameters for this exercise, this would be where you could change values for Risk Type, BMRF, Confidence Level, etc. (In BMDS, you would set some of these parameters in the Model Options dialog.) The Wizard color-coding legend describes which cells can be edited and which should be left as-is. 
  10. Scroll back to the top of the Main tab. Select the AUTORUN button.
    In the background, the Wizard creates all of the input files, runs them in a BMDS session, and then imports the BMDS results into the Wizard’s Results tab. The Wizard reports its progress in the status bar at the bottom of the Excel window. For this exercise, the operations should take less than 30 seconds. (Refer to Section 2.6 of the BMDS Wizard User’s Guide for more information.)
  11. The BMDS Wizard automatically displays the Results tab when the session is done. Note the following tips about the Results display.
    • Each row is the output for one model option file specified on the Main Page. (Refer to Section 2.7 “Viewing Imported Results” of the BMDS Wizard User’s Guide for more information.)
    • Hover the mouse over each cell in the Model Type column to see a graph of the model result.
    • Select the View Output Images button to display the Output tab, which shows all the result graphs. Click the Back to Results link to return to the Results tab.
    • The column “BMDS Wizard Recommendation” highlights the cell for the recommended model.
    • For the recommended model, click its cell in the “Include in Summary Table?” column. From the picklist for that cell, choose “Select.” All models marked “Include” will be printed in the summary table report.Selecting the Wizard-recommended model
    • At the top of the sheet is a field labeled “Basis for Model Selection.” Enter a justification for choosing the model; for this exercise, the justification could be "Lowest AIC." The justification will be included in the Microsoft Word-based summary report.
    • For the justification, you can type in your own text or click the triangle to the right of “Basis for Model Selection” text field to select sample text you can modify.Entering a justification for the model selection
    • Refer to Section 2.8, “Viewing BMDS Wizard Recommendations and Selecting a Best-Fitting Model,” of the BMDS Wizard User’s Guide for more information and considerations.
  12. To generate summary tables and reports, click on the Report tab. The BMDS Wizard can generate a Microsoft Word report based on the EPA Integrated Risk Information System (IRIS) template for a benchmark dose appendix.
    • Select TRUE or FALSE to specify the items you want included in the report.
    • Select the Select MS Word Template button to specify the folder containing the report template (in a default BMDS installation, this will be the BMDS Wizard program directory).
    • Select the Select/Create MS Word Output File button to specify where the report will be created. You must enter a file name for the new report.
  13. Select the Print Word Report button. The Wizard will generate and display the report. Refer to Section 2.9, “Generating Summary Tables and Reports,” of the BMDS Wizard User’s Guide for more information on generating and modifying reports.

Top of Page

Questions

Using the results returned from the sample data you entered, answer the following questions on the Akaike Information CriteriaHelpAkaike Information CriteriaA statistical procedure that provides a measure of the goodness-of-fit of a dose-response model to a set of data. AIC = -2 x (LL - p), where LL is the log-likelihood at the maximum likelihood fit, and p is the degrees of freedom of the model (usually, the number of parameters estimated). (AIC) and BMDLHelpBMDLA lower one-sided confidence limit on the BMD. values.

Top of Page

Exercise 2

  1. Open a dichotomous worksheet file. Ensure macros are enabled in the spreadsheet (refer to the BMDS Wizard User’s Reference Guide for more information).
  2. On the Startup tab, click the Main Page button. The Main worksheet tab is displayed.
  3. On the Main Page, fill in the Study and Year as “Exercise_2.” You can specify info for the Study and Modeling Inputs boxes, but it’s not necessary for this exercise. Use the Select Folder buttons to navigate to the directories.
  4. Click the cell in C20 to move to the Data worksheet.
  5. At the Data worksheet, enter the numbers from the following table into the Data worksheet. The quick way: copy the rows of numbers below and paste them into the Data worksheet starting at the Dose Group 1 row.
    Dose N Response
    0 86 1
    0.55 81 19
    1.20 81 54
  6. Edit the labels that will appear in the output. Edit the “Column Name in BMDS” cells so they read “Dose,” “N,” and “Response.”
  7. Click Return to Main to display the Main Page.
  8. Scroll back to the top of the Main Page. Click the AUTORUN button. You will receive an error from the BMDS Wizard that looks like the following:Wizard error message
  9. Click OK. The Results tab will display, showing error messages for the models that could not optimize a solution.

    In the following edited screenshot of the Results tab, you can see the results displayed for each model.Edited screenshot of Wizard Results tab

    • Note the evaluations of each model in the BMDS Wizard Recommendation column.
    • Click in each cell of the BMDS Wizard Recommendation Notes column to view the application’s logic behind its recommendations.

Questions

  • Why do you think BMDS and BMDS Wizard evaluated so many models as “Questionable”? Why do you think the Dichotomous Hill and Multistage 3rd degree models failed?

    For this analysis, we are attempting to evaluate a dataset with only three dose groups. Typically, models with n dose groups should only use up to n-1 parameters in the modeling. This is why the Wizard declares some models to be “Questionable” with the note that the model was saturated. However, models that use 3 parameters (e.g., log-logistic) on data with only 3 dose groups may adequately describe the dose-response data, but estimate the value of each parameter, thus resulting in no available degrees of freedom with which to estimate the goodness-of-fit p-value.

    Careful consideration of other metrics of model fit (scaled residuals, visual fit) may indicate these models fit the data adequately and should be considered when determining the best fitting model. Models that use more than three parameters (i.e., Dichotomous Hill) cannot fit this data as the number of parameters is greater than the number of dose groups. These models may not converge properly under these circumstances; therefore, BMDS and the BMDS Wizard may display an error during the model run and/or report the model as “unusable."

Top of Page

Exercise 3

Model all endpoints from the following table:

Subjects Chamber Control 12.8 ppm 32 ppm 80 ppm
Incidences of Non-neoplastic Lesions of the Olfactory Epithelium in Male Rats in the 2-Year Inhalation Study of Chemical X
Number examined microscopically 50 50 49 49
Atrophy 3 12* 35** 48**
Inflammation, Chronic Active 0 5* 22** 49**
Metaplasia 6 5 26** 48**
Necrosis 0 7** 26** 39**
* represents .05

** represents .01

For each endpoint:
  1. Create and save a separate BMDS Wizard file.
  2. Run all appropriate models with BMR = 10% (on the Main Page tab, you may need to add models not included as part of the default model set). A 10% BMR should be the default; to make sure, Row 59 (“BMRF") on the Main tab should be set to 0.1.  Also, Row 58 (“Risk Type”) should be set to “Extra.”

    A note about “adding appropriate models”: The BMDS Wizard automatically detects n-1 dose groups from the Data worksheet and displays the appropriate Multistage models.  By default, the BMDS Wizard automatically includes Multistage 2 and 3 models in the included models, so for this dataset (four dose groups), no models need to be manually added. However, if the example included five dose groups, models 2 and 3 would be automatically included, but model 4 would not be. Therefore, you need to manually add model 4 to the list of models you want to run.

    In this case, you would go to the Main page, select model 4 from the “Add new models to BMDS Session” list and click Add Model & Load Model Defaults. Scroll down the page to the “BMDS Model Option Setups” section and scroll to the right; you will see that a Multistage 4 column has been added to the list of models.

     
  3. Analyze and interpret the results on the Results worksheet.
  4. Export the results (with the chosen model Selected) to the Word Report.

Top of Page

Question

  • Which model would you choose for each endpoint?

    The recommended models for each endpoint are:

    • Atrophy:  Log-probit, based on lowest AIC.
    • Inflammation: multistage 2nd degree, based on lowest AIC.
    • Metaplasia: Log-probit, based on lowest AIC.
    • Necrosis:  Log-probit, based on lowest AIC.

    You can follow the BMDS Wizard recommendations for the best model. For more information, you may find it helpful to refer to Section 2.5, “Decision Tree,” of the BMDS Technical Guidance; that section summarizes the general progression of steps in a BMD/BMDL calculation and considers the criteria needed for a model recommendation. On page 54, it states: “If the BMDL values from these remaining models are sufficiently close (given the needs of the assessment), the model with the lowest AIC may be selected to provide the BMDL.”

Top of Page

Next: Using BMDS Wizard to Evaluate Continuous Datasets