Jump to main content.

Innovation Analysis Modules

You will need Adobe Reader to view some of the files on this page. See EPA's PDF page to learn more.

The National Center for Environmental Innovation (NCEI) promotes the use of innovative approaches to environmental problem-solving efforts that deliver improved environmental results. As part of this process, NCEI promotes and conducts program evaluations to determine whether innovations deliver environmental results that surpass the traditional way of doing business and to identify lessons that can be applied broadly.

Environmental innovation comes in many shapes and sizes. It may be process or substance-oriented, innovation or program-based, on a facility or geographic scale, community-based or nationwide, or it may produce systemic organizational change. In all cases, it will be important to determine whether the innovation is achieving its intended outcomes, whether it provides greater process efficiencies or superior environmental performance compared to standard practice, and to identify ways in which to improve the innovation. Program managers must be able to measure and describe the impacts of their programs.

The process of evaluation will help innovation practitioners to answer these questions by identifying clear goals, developing performance indicators to track progress, establishing baseline data, setting targets for future performance, and measuring progress toward such targets. A well planned and thoughtfully conducted evaluation can help determine if program activities are providing the outcomes needed to achieve the stated goals. With evaluation, innovation practitioners can understand barriers to innovation and modify the program as needed to accomplish objectives, or modify program goals to set more appropriate or realistic expectations. Knowledge and insights obtained from an evaluation can serve as performance feedback. Evaluation can also play a pivotal role in mainstreaming innovative practices and policies into everyday work.

Companion User's Guide
(PDF, 38 pp., 630 KB)
Innovation Analysis Modules
(PDF, 16 pp., 176 KB)

Introduction (PDF, 7 pp., 116 KB)
Module 1 (PDF, 7 pp., 1,010 KB)
Module 2 (PDF, 7 pp., 1,340 KB)
Module 3 (PDF, 5 pp., 1,288 KB)
Module 4 (PDF, 4 pp., 680 KB)
Module 5 (PDF, 3 pp., 45 KB)
Module 6 (PDF, 5 pp., 560 KB)

 Module 1 (PDF, 2 pp., 37 KB)
Module 2 (PDF, 3 pp., 52 KB)
Module 3 (PDF, 2 pp., 46 KB)
Module 4 (PDF, 3 pp., 62 KB)
Module 5 (PDF, 2 pp., 31 KB)
Module 6 (PDF, 4 pp., 61 KB)
NCEI has developed a series of Innovation Analysis Modules and a Companion User's Guide to assist staff and program managers to answer these questions and others throughout the life cycle of an innovation—from pilot testing to broad-scale application of a successful innovation. The purpose of the companion user's guide is to assist innovation practitioners with the application and implementation of the modules. Each chapter of the user's guide corresponds to the actual module, each of which may be used independently or in combination with other modules, and is intended to serve three purposes:
  1. Inform the evaluative process
  2. Help an innovator plan for evaluation at the beginning or intermediate stages of an innovation
  3. Help serve as a innovation management/development tool
The modules and companion user's guide can be tailored to meet each of the purposes mentioned above and the needs of the innovation practitioner. Each module presents an assessment framework, which can be answered with varying degrees of rigor in order to answer the questions for his/her purposes. The main goal of these tools is to allow the practitioner to collect data in an organized manner, conduct an ongoing assessment of the innovation, and provide the information necessary to conduct a full evaluation.

[ return to Top ]

What are the innovation analysis modules?
The innovation analysis modules are a suite of evaluative questions that provide a framework to evaluate, understand, and share information on environmental innovations. Each module represents a compilation of research questions that EPA staff have used when evaluating innovations. The modules partially reflect the pioneering work of Everett Rogers, who analyzed and systematized the life cycle of innovations and wrote Diffusion of Innovations.

Because of the diversity of environmental innovations, NCEI designed the innovation analysis modules to provide guidance, direction, and flexibility. The questions are designed to encourage critical thinking and assessment of environmental data, successes, obstacles, and lessons learned in order to help the practitioner improve the innovation. Innovation practitioners are encouraged to consider the core questions contained in each module, determine their applicability to the innovation, make appropriate modifications to the questions, and gather the available data—quantitative, qualitative, or anecdotal—to assess progress. The quantity and quality of data that the practitioner has and the rigor with which the modules are applied will determine the quality of the analysis. Although all of the modules do not have to be completed, the information collected from all of the modules could offer a more complete picture of whether an innovation is working well or highlight areas that need improvement. Each module is described briefly below and is accompanied by detailed instructions from the user's guide and the questions for each module.

[ return to Top ]

Who should use the modules?
Program managers, designers, and staff participating in innovation should use the innovation analysis modules to focus their thinking about the innovative process, assess how well the innovation is working, and assist in innovation management and development. Different members of the innovation team may be responsible for different modules or different components of the modules. For example, one member of the innovation team may be in charge of data collection and management, whereas another member may be the coordinator of public participation. The modules have been designed so that EPA or other Federal government agencies, State agencies, local and Tribal governments, regulated entities and the public at large, can use them for analytical purposes. The questions in each module may be more or less relevant depending on who the innovation practitioner is and how the modules are applied. Another example may be a project manager who is designing an innovation and decides to use the modules to help build evaluation into the design of the innovation. For the purpose of clarity, we discuss the “innovation practitioner, innovator, or practitioner” as the main reader and user of the modules. However, an “evaluator” can use the modules as well.

[ return to Top ]

Who is the audience for the results of the evaluation modules?
Information gathered by using the innovation analysis modules will be a valuable resource for innovation team members, decision makers within an environmental agency, participants in the innovation, the public, and others interested in the progress of the innovation. The quality and results of the information obtained will likely influence decisions on the need for modifications to the innovation, or whether the innovation is ready for broad-scale application.

[ return to Top ]

How can the modules help you?
The modules are flexible tools intended to guide and shape decisions and discussions around important innovative environmental evaluative questions. The innovation analysis modules provide a systematic way to collect and analyze data, make adjustments to improve performance, and organize, track, and monitor the progress of the innovation. The innovation analysis modules can help determine whether an innovation is working as intended and whether it has the potential for broad-scale application. As a project management tool, they can be used to raise important questions and make methodological decisions explicit. Finally, they can (and should) be modified to adapt to the innovation and the needs of the innovation team.

The modules can also help the innovation practitioner conduct a formal evaluation study. The modules can help identify the evaluation questions about the innovation. Based on the data collected for each module, the innovation practitioner should have a clear picture of where the innovation is working well and where it is falling short of expectations. In addition, the modules will indicate areas of uncertainty or weakness in the innovation design. The evaluation can then be crafted to hone in on both the successful and problem areas through more data collection to understand why the innovation is performing a certain way. The modules will provide the quantitative data needed for the evaluation and some qualitative data. The evaluation should look for a more rigorous approach to collecting additional quantitative data if
needed, and the qualitative data that is often needed to understand an innovation. The user’s guide provides examples of how the evaluation can be crafted from the modules.

[ return to Top ]

At what phase of the innovation should the evaluation modules be used?
The modules are intended for use throughout the innovative process: 1) at the design and planning phase to help design the innovation for evaluation; 2) during implementation; 3) when the pilot experimentation is complete; and 4) when informing a formal evaluation. Exhibit 1 provides examples of how each module can be used during the three primary phases of an innovation – design and planning, implementation, and maturity of the innovative concept (or completion of a phase of the innovation).

Designing an Innovation—In order to design an innovation, it is necessary to clearly define the problem that the innovation is addressing and then outline how the innovation will address the stated goals. The modules provide a series of questions that will identify the following: 1) innovation goals and desired environmental and behavioral outcomes; 2) baseline data; 3) performance measures; 4) enforcement and compliance aspects; 5) public involvement requirements; and 6) how to plan for the transferability of the innovation. It is often difficult to assess how well the innovation is working without planning for data collection early on in the innovation design.

Implementation of an Innovation—An innovation can be assessed at different levels. As a basic step, every innovation should have a project tracking and monitoring component. The modules provide questions to help set up an adequate data collection system. Project tracking and monitoring means collecting information on how the innovation is working according to schedule or protocol and meeting stated objectives. Using the modules at this phase pinpoints and addresses successful elements of the innovation and any barriers to success. If the innovation is not meeting expectations in any one area of importance, the modules can be used to conduct a deeper level of analysis.

Assessing the Innovation at the Termination Point—The modules can be used to conduct more of an in-depth assessment to see how the innovation performed. Robust qualitative and quantitative information gained from each module can provide a complete picture of how well the innovation performed in meeting its stated goals. Each module asks the innovation practitioner to assess how well the innovation performed relative to the traditional approach. Determining the relative advantage of the innovation over the traditional approach is vital to innovation transferability and overall success.

Designing a Formal Evaluation—The modules can be used to help the practitioner design a formal evaluation study of the innovation. Evaluation looks at how well an innovation is working to achieve its stated outcomes and why it is working the way it is. The modules ask the practitioner to construct a logic model (Module 1), which is an integral first step to doing an evaluation and to formulating the right evaluation questions. In order to construct the rest of the evaluation study, the practitioner uses the completed modules to help address the following questions: 1) to what extent have the stated outcomes been achieved and why; 2) what aspects of the innovation lead to those outcomes; and 3) what is the context in which the outcomes were achieved.

[ return to Top ]

What kind of data is needed to complete the modules?
Depending on the innovator’s needs, the module questions may be answered with anecdotal, qualitative, and/or quantitative data. Generally, more robust data results in greater certainty in the analysis, or the ability to characterize the uncertainties or successes of an innovation in greater detail. For example, in some cases, an anecdotal reporting of the costs required to pursue an innovation may be sufficient, whereas a financial accounting of capital costs may be necessary for another type of innovation. Data collection will, however, depend largely on the needs of the innovation team and available resources. The modules together can be powerful evaluative tools if the innovation practitioner ensures that it is supported by thorough analyses and quality data.

[ return to Top ]

How to use the six evaluation modules
In the sections that follow, each module is described in greater detail, with guidance provided on its application, the organizational structure of the questions, useful sources of information, and a methodology for addressing each module’s content. Each module is attached in Appendix A. It is recommended that innovation practitioners answer the questions in each of the corresponding modules rather than within the companion user’s guide. In addition, the practitioner should collect information from a variety of sources and in an iterative fashion throughout the innovative process. By returning to the key questions contained in the modules at different phases of the innovative process to assess progress, the practitioner will be better informed and can make adjustments to the innovation, if necessary.

[ return to Top ]

Exhibit 1: Uses of the Modules
Print this table: Exhibit 1—Uses of the Modules (PDF, 2 pp., 43 KB)

Uses of the Modules

Module 1: Mapping the Innovation

Design and Planning

Implementation

End of the Innovation

Formal Evaluation

  • Identify goals
  • Identify partners and customers
  • Identify tools to assist project
  • Identify preliminary drivers and barriers
  • Create a logic model for the innovation, innovative project, or program
  • Develop project fact sheets and communication tools  
  • Modify to accommodate changes in project conception, tools, etc.
  • Review goals, partners, customers, drivers, barriers etc. identified in the planning stage
  • Review logic model for completeness and accuracy
     
  • Verify accuracy of original information
  • Adjust for unanticipated outcomes or changes in approach
  • Compare goals and items identified in the planning stage with what happened during implementation
  • Verify accuracy of logic model
  • Compare goals and items identified in the planning stage with what happened during implementation
  • Verify accuracy of logic model
  • Use logic model to look for gaps and unanswered questions
  • Use logic model to help identify key evaluation questions

Module 2: Assessing the Environmental Results of the Innovation

Design and Planning

Implementation

End of the Innovation

Formal Evaluation

  • Identify environmental goals
  • Identify feasible measurement approach
  • Characterize baseline
  • Identify anticipated medium and long-term behavioral and environmental outcomes
  • Identify data sources and collection/monitoring protocols to obtain outcome data
  • Setup schedule to update information
     
  • Review data collection and monitoring results to verify adherence to protocols
  • Normalize and compare mid-course data to baseline to determine need for midcourse corrections
  • Review data collection to ensure data will provide information on environmental and behavioral outcomes
  • Review monitoring and measuring approach, baseline data, and anticipated outcomes identified in the planning stage  
  • Normalize data to account for changes
  • Compare pre-innovation baseline to post-innovation results to determine net change
  • Identify areas of success and shortcomings
  • Assess environmental/public health relative advantage of the innovation
  • Determine if more in-depth evaluation is necessary
  • Determine why there is a difference between pre-innovation baseline and post-innovation results
  • Determine why there is or is not an environmental/public health relative advantage of the innovation
  • Describe environmental/public health results in terms of customer, partner and stakeholder satisfaction and discuss why the results have meaning

Module 3: Assessing the Costs and Cost Savings of the Innovation

Design and Planning

Implementation

End of the Innovation

Formal Evaluation

  • Identify types of savings and costs goals associated with project
  • Identify who is incurring the savings and costs (i.e., facility, government)
  • Characterize baseline
  • Identify data sources for savings and cost information
  • Review cost information for completeness and accuracy
  • Normalize and compare midcourse data to baseline to determine need for midcourse corrections
  • Review projections identified in the planning stage for comparison between perceived and actual results
  • Normalize cost/savings data to account for changes
  • Compare pre-innovation baseline to post-innovation costs/savings to determine net change
  • Identify areas of success and shortcomings
  • Is there a cost/cost savings relative advantage?
  • Determine if more in-depth evaluation or cost-benefit analysis is necessary
     
  • Determine why there is a difference between pre-innovation baseline to post-innovation costs/savings
  • Why or why not is there a cost/cost savings relative advantage?
  • Determine if more cost-benefit analysis is necessary—why or why not?
  • Describe costs/cost savings in terms of customer, partner and stakeholder satisfaction and discuss why the results have meaning

Module 4: Enforcement and Compliance Assurance

Design and Planning

Implementation

End of the Innovation

Formal Evaluation

  • Identify the monitoring, reporting, and recordkeeping requirements
  • Identify the method of determining compliance (i.e., record review, inspection)
  • Identify responsible parties for verifying data and information
  • Review and track information to monitor compliance and identify problems or trends that require midcourse corrections
  • Review requirements established during implementation
  • Verify and evaluate final record reviews, inspections, or other means of compliance assurance
  • Determine if innovation is practicably enforceable
  • Identify areas of success and shortcomings
  • Is there a relative advantage in compliance and enforcement to the innovation?
  • Determine if more in-depth evaluation is necessary
  • Determine if innovation is practicably enforceable and what it means
  • Determine why there is a relative advantage in compliance and enforcement to the innovation?
  • Describe enforcement and compliance assurance in terms of customer, partner and stakeholder satisfaction and discuss why the results have meaning

Module 5: Public Involvement and Stakeholder Feedback

Design and Planning

Implementation

End of the Innovation

Formal Evaluation

  • Identify key participants
  • Identify approach for engaging stakeholders
  • Determine resources available for addressing stakeholder issues
  • Identify potential stakeholder issues (e.g., Environmental Justice) upfront
  • Check in with stakeholders to assess whether there are stakeholder concerns and the level of participation
  • Assess availability of information to the public
  • Assess stakeholder participation and participation plans
  • Request that stakeholders provide feedback regarding the quality of their experience in the innovative project
  • Identify areas of success and shortcomings
  • Determine if more in-depth evaluation is necessary
  • Determine why there are areas of success and shortcomings
  • Why is or isn't there a relative advantage to the innovation in terms of public involvement (i.e., did the public have greater access to information or greater means to participate—why or why not?)
  • Analyze public involvement in terms of satisfaction and ask the question of why are they satisfied or dissatisfied
     

Module 6: Assessing the Potential Transferability of the Innovation

Design and Planning

Implementation

End of the Innovation

Formal Evaluation

  • Identify data necessary to determine relative advantage
  • Identify a path to disseminate information
  • Define and focus targets of diffusion efforts
  • Provide opportunities for potential early adopters of the innovation to participate in implementation
  • Communicate early positive results of innovation
  • Analyze innovation for its relative advantage
  • Develop and facilitate workshops and networking opportunities to promote learning
  • Develop users' guides and Web-based tools to facilitate scale-up
  • Identify areas of success and shortcomings
  • Determine if more in-depth evaluation is necessary
  • Determine how the innovation would fare if applied more broadly
  • Determine what aspects of the innovation are working well and those key aspects that need to be modified in order for the innovation to be more broadly applied

[ return to Top ]

Local Navigation

Jump to main content.