Outcomes Analysis for the Region 9 Partner Project

The table adopted by EPA Region 9 for their  Region 9 RFA for FY23-24 (see pages 11-12) was also presented to representatives of other EPA Regions. The other regions had the option to use ideas from the table when evaluating their applicants' submissions, or to consider the table in the next WPDG cycle.

After the FY23-24 awards cycle was complete, applications were evaluated for indications of whether the Organon table influenced how submitted proposals were framed. NVivo software was used to detect usage of words, phrases and concepts contained in the Organon table and compare the language patterns between the current and previous cycles of WPDG applications as qualitative evidence of Organon table influence. Limitations of the analysis stem from the existence of other factors that also could influence language usage, such as the appearance of similar language within other portions of the RFA or within the scientific literature. Indeed, as this table was developed as part of a widespread trend of increasing emphasis on resilience-based management, it is not possible to conclusively differentiate general awareness versus specific awareness due to our table.

That said, there are some noticeable trends in the NVivo analysis results (Tables 2-3) that point to a positive influence of the Organon table on incorporation of ecological resilience and public engagement language in WPDG proposals. Three terms associated with the concept of ecological resilience (resilience, storms, future conditions) increased by at least 20% in the FY 23-24 proposals ("post" addition of the table) compared to FY21-22 ("pre" addition of the table). We consider an increase in term usage of 20% or more to be a notable qualitative signal. In addition, two public engagement terms (frontline communities, local knowledge) also increased in usage by at least 20% compared to pre-table proposals. Several of these terms appear only in the Organon table (i.e., not elsewhere in the RFAs, see key in Table 2), implying a more reliable signal of Organon table influence. Some terms (e.g., resilience, frontline communities) appear in both the pre- and post-table RFAs as well as the Organon table. But these terms also are more frequent in resubmitted proposals (proposals by groups that had submitted a proposal in the previous cycle), suggesting an increase in focus on these concepts with increasing experience. We also recognize a pre- to post-table decrease in the use of some terms (e.g., vulnerability assessment, safety), which could reflect small sample sizes or proposals submitted in the previous cycle transitioning to work that emphasizes different concepts in the subsequent cycle. While there has been a general push for resilience-based management to be addressed in WPDG proposals, the Organon table emphasizes this more specifically, though there is, of course, no way to separate degree of causation.

Future benefits to the WPDG program and RFA process could be amplified if the concepts defined in the Organon table were explicitly incorporated in proposal evaluation criteria. In addition, the RFA could encourage applicants to use the Organon framework to assess and describe their existing wetland program status, as a basis for explaining how their proposed project would enhance their wetland program and make it more effective. Subsequently, winning applicants could use the framework to report on the progress and ultimately evaluate the success of their program development for achieving clearly articulated goals. 

Tables 2-3. For ecological resilience (top) and public engagement (bottom), percentage (%) differences in applicants' use of terms "Pre" (FY21-22) versus "Post" (FY23-24) addition of the Organon table in the RFA, with gray highlights indicating terms that increased in usage by at least 20%. "All Proposals" are compared to "Resubmitted Proposals" (proposals by organizations that had submitted a proposal in the previous cycle), with red/italics font indicating where an increase in term usage was enhanced in the more experienced group. Key: * Term appears in 23-24 RFA and the table. ** Terms appear in both FY RFAs and the table. All other terms are unique to the table. Numbers in parentheses are the sample sizes of different groups.

Ecological Resilience

Public Engagement