You will need Adobe Reader to view some of the files on this page. See EPA's PDF page to learn more.
Impact evaluation is a type of evaluation that determines what difference the program has made. This is achieved by comparing the observed outcomes with an estimate of what would have happened in the absence of a particular program.
Although typically more prominent in the medical, education and public health fields, impact evaluation is increasingly being applied to a wider variety of fields, including environmental management.
Examples of Impact Evaluations:
Working collaboratively to ensure that measurement and evaluation are integrated up front (before a project is implemented) is crucial for ensuring that the project will be able to attribute cause to a specific intervention. EPA's Region 4 is interested in determining the effectiveness of innovative enforcement tools and targeting strategies to improve the compliance of federal facilities with environmental regulations. The purpose of this evaluation is to assess the impact of Region 4's efforts. The Office of Policy and Region 4 are working with Georgia State University's Center for Evidence-based Environmental Policies and Programs to carry out the evaluation.
Randomized Control Trial
When complete, a link to the final evaluation report will be made available on this site.
In the United States each year, an estimated 10 percent of sold architectural paint (more than 750 million gallons) is left unused. Local household hazardous waste collection programs collect more paint than any other hazardous waste. Oregon has become the first state in the nation to enact a law requiring paint manufacturers to safely manage leftover latex and oil-based paint from consumer and contractor painting jobs. To ensure that paint is disposed of properly, this issue requires public awareness and a convenient and effective system for collecting paint. To support this legislation, a national partnership—the Paint Project Stewardship Initiative—was developed, and EPA's Office of Policy participates in this effort. Under the new law, the paint industry will set up a program to reduce paint waste, increase reuse and recycling, and safely dispose of remaining unusable paint.
A participatory evaluation has been a goal of the partnership since its inception. By building measurement and evaluation into the design of the pilot program, the Paint Project Stewardship Initiative has positioned itself to answer key evaluation questions important to all its stakeholders.
- To what degree was the pilot program, from planning to implementation, a collaborative process?
- Describe the Paint Stewardship Organization (PSO), including its funding mechanism and infrastructure.
- How did education materials and strategies affect consumer awareness and behavior?
- How has the program affected consumers' purchasing decisions and management of paint prior to drop-off at paint recycling facilities?
- How has the program affected the collection of post-consumer paint in terms of volume, cost, quality, environment, convenience, and infrastructure?
- How has the program affected used paint reprocessing, paint recycling, and paint-related energy recovery in terms of volume, infrastructure, and cost?
- What was the impact of the program on the household hazardous waste (HHW) facilities in terms of the types of paint collected, costs, and the way in which the facilities operate?
- How cost effective is the program?
- How was the program designed and implemented to move consumers up the waste hierarchy?
- How has the market for post-consumer paint been affected by the program?
- Based on the OR experience, what implementation and outcome-related information is required for other states to develop and implement leftover paint management systems?
- During the program and for each of its primary components, what were the primary external, unexpected and/or unintended influences and consequences?
Mixed-Methods including Direct Observation and Quasi-Experimental Design
EPA provides compliance assistance to the regulated community, including local governments and tribes, to help their regulatory obligations and to prevent violations. For example, EPA offers compliance assistance to help the auto body sector better understand their responsibilities in following established environmental rules. To achieve this, technical assistance ranging from newsletters, workshops, a help-line, and mailings is offered.
To assess the combined effectiveness of those activities, the validity of using a telephone survey to replace site visits, and the relative contributions of each assistance activity on compliance rates.
The primary goal of this evaluation is to understand the contribution of EPA's Compliance Assistance (CA) effort to improved environmental management and environmental benefits and to establish a statistically valid design methodology that will allow statistical estimation of this impact. This evaluation, managed by the Office of Policy's Evaluation Support Division, is focused on supporting EPA compliance assistance efforts to: (1) implement an outcome measurement pilot for compliance assistance activities conducted in FY09/10 that uses statistically valid methods and an OMB-approved ICR for data collection; (2) investigate through a comprehensive literature review analysis, the nature and extent of any correlation between compliance assistance (CA) activities and changes in recipient behavior (i.e., improved environmental management practices and reduction/elimination/treatment of pollution); and (3) develop and refine an evaluation methodology that has transferable elements so that future, routine CA activities may be measured and evaluated using statistically-valid methodologies but with less complexity and required resources than the pilot (e.g., transferable data collection methods and survey questions). This latter task will include assessing the accuracy of telephone survey information as a substitute for site visits.
- Do EPA's CA activities identified in this pilot contribute to improved environmental management practices, presumed to have direct environmental benefits, after a person has considered key contributing factors identified at the outset of the pilot design?
- Is the telephone survey a valid and reliable technique for performance measurement and program evaluation?
- Are the methods employed in this effort transferable to other assistance activities in EPA's CA programs? Moreover, what specific characteristics of the auto-body sector influence the transferability of the findings in this evaluation?
- What are the results of an analysis of the various methods currently used in compliance assistance activities?
Randomized Control Trial with Probability Sampling, Self-Report Survey, Criterion-Related Validity Study
- Measuring the Effects of EPA Compliance Assistance in the Auto Body Sector: A Statistically Valid Pilot Project (PDF) (80 pp, 1.1MB)
- Appendix A (PDF) (62 pp, 2.5MB)
- Appendix B (PDF) (64 pp, 221K)
- Appendix C (PDF) (33 pp, 374K)
- Appendix D (PDF) (22 pp, 296K)
- Appendix E (PDF) (41 pp, 316K)
- Appendix F (PDF) (17 pp, 561K)
- Appendix G (PDF) (11 pp, 93K)
- Fact Sheet (PDF) (2 pp, 41K)
The WasteWise program, run by the Office of Solid Waste and Emergency Response, uses information, technical assistance, public recognition, and awards to promote the prevention and recycling of municipal solid waste and select industrial materials. Municipal solid waste includes materials that could end up in an organization's (or its customers') trash.
Focusing on one member, the United States Postal Service (USPS), this evaluation helped to identify the most effective activities the program could use to reduce the waste of different types of program partners (e.g., size or sector of partners) and addressed the related question of attribution: how much of this waste reduction is fairly attributable to EPA activities? The evaluation also assessed whether the current information is sufficient (or if other information available but not already reported by partners is needed) to evaluate the program. This evaluation determined whether the WasteWise Program effectively reduced waste by comparing the waste reduction practices and results of its USPS partners and non-partners.
- WasteWise uses a variety of approaches to influence the behavior of partners. Which approaches—for example technical assistance, information, awards and recognition—are most effective for which types of partners?
- In addition to participation in WasteWise, what other factors may influence a partner organization's decisions to improve management of MSW (e.g., cost savings, consumer pressure, and other voluntary program opportunities)
- What can be determined about how WasteWise participation contributes to partner behavior regarding MSW management (e.g., by effecting waste management improvements sooner, better incorporating waste management as a permanent feature of corporate culture, facilitating non-participant changes by providing information)?
- What can EPA do to encourage WasteWise partners to submit sufficient environmental data for performance measurement and evaluation purposes?
Quasi-Experimental, Document Review, Survey, Best Practices Review