EPA Implementation of Government Performance and Results Act

EPA Office of Inspector General

GPRA

CHAPTER 1

INTRODUCTION

PURPOSE

We initiated this survey at the request of the Agency's former Chief Financial Officer who asked that we include in our audit plan, management and program audits that will help the Agency develop and implement a new management framework under the Government Performance and Results Act (GPRA or the Act). He also requested that we help the Agency ensure that the EPA programs currently participating in the GPRA pilot program serve as models as the Agency expands GPRA implementation to cover all of its programs.

The objectives of our survey were to answer the following questions:

Does EPA's five-year strategic plan clearly identify the Agency's mission?
Has the Agency identified measurable goals for tracking the Agency's progress towards accomplishing its mission?
Is the Agency taking the steps necessary to ensure the validity of its performance measurement data?
Is the Agency taking the necessary steps to link its strategic plan, annual budget submission and performance measurement information with information on the cost of its programs?

BACKGROUND

GPRA was enacted on August 3, 1993 to provide for the establishment of strategic planning and performance measurement in the Federal Government, in order to:

Improve public confidence in Federal agency performance by holding agencies accountable for achieving program results.
Initiate program performance reform by setting program goals, measuring program performance against those goals, and reporting on progress.
Improve Federal program effectiveness and public accountability by focusing on results, service quality, and customer satisfaction.
Improve Congressional decision making by clarifying and stating performance expectations up front.
Improve the internal management of the Federal government.

The intent of GPRA is to change the culture of federal agencies, from focusing on what agencies are doing to what they are accomplishing. The Act expands upon the firm foundation already built by the Chief Financial Officers (CFO) Act in gathering financial information to support decision making. It requires agencies to develop strategic plans, reach a reasonable degree of consensus on desired goals with key stakeholders, and measure and report progress toward achieving those goals. Specifically, GPRA requires all agencies to:

Develop a 5-year strategic plan and update the plan every three years, effective September 30, 1997.
Submit to OMB annual performance plans, consistent with the agency's strategic plan, covering each program activity included in the agency's budget, effective September 30, 1997.
Issue annual program performance reports, effective March 31, 2000.

GPRA reforms are a major undertaking and comprehensive program goal-setting and performance measurement and reporting will not be accomplished easily or quickly. In recognition of the length of time needed for implementation, the Act's approach to implementation is one of initiating multi-year pilot projects. The Act provides for three types of pilot projects to measure and report performance: (1) pilot projects for performance goals, (2) pilot projects for managerial accountability and flexibility, and (3) pilot projects for performance budgeting.

The pilot project for performance goals was designated for fiscal years 1994, 1995 and 1996. Agencies participating in this pilot must prepare performance plans and performance reports. The performance plans are to include objective, quantifiable, and measurable performance goals; resource requirements; performance indicators; a basis for comparing program results with the performance goals; and, a means to verify and validate performance data. The Agency's strategic plan is to be used in preparing the performance plans. Performance reports are due March 31 of the following year and must include an assessment of the Agency's success in achieving its performance goals; an evaluation of the Agency's performance as it relates to the current year's performance plan; explanations of why performance goals were not met, and plans and schedules for achieving the goals; and a summary of program evaluations conducted.

EPA nominated six programs for participation in the GPRA pilot project for performance goals: one for fiscal 1994, the Leaking Underground Storage Tank Program; two for fiscal 1995, the Acid Rain Program and the Drinking Water Program; and three for fiscal 1996, the Chesapeake Bay Program, the Environmental Technologies Initiative (ETI), and the Superfund Program. In fiscal year 1993, the President directed EPA to initiate the ETI program to develop more advanced environmental systems and treatment techniques. As a result of budget reductions in the ETI Program, program officials decided not to participate as a fiscal 1996 GPRA pilot.

Agencies participating in the pilot project for managerial accountability and flexibility designated for fiscal years 1995 and 1996, were to include as part of their performance plan, a proposal to waive administrative/procedural requirements and controls, in return for agreeing to achieve a specified performance goal. The Agency elected not to participate in this pilot project. Agencies participating in the pilot project for performance budgeting designated for fiscal years 1998 and 1999 are to prepare performance budgets. The budgets must present the varying levels of performance, including outcome-related performance, that would result from different budgeted amounts.

SCOPE AND METHODOLOGY

Our survey focused on the five EPA programs participating in the Government Performance and Results Act (GPRA) pilot project for performance goals for fiscal years 1994 through 1996. To satisfy the survey objectives, we reviewed the legislative requirements of GPRA and the CFO Act to ensure that the Agency's implementation efforts are consistent with the legislative requirements of both acts. We also reviewed applicable laws and regulations for the five programs participating as GPRA pilots to ensure that program officials have established performance indicators which will measure progress towards meeting their legislative objectives, as well as their individual program goals.

We met with Headquarters program officials in the Office of the Chief Financial Officer; Office of Policy, Planning, and Evaluation (OPPE); Office of Administration and Resources Management (OARM); Office of Water (OW); Office of Air and Radiation (OAR); and the Office of Solid Waste and Emergency Response (OSWER). We also met with program officials from the Region 3 Chesapeake Bay Program Office. We met with officials from these offices to discuss the following concerning the GPRA pilots: (1) how the performance measures were developed, (2) the source of their performance data, (3) how the performance data will be verified and validated, and (4) how they will track program costs.

To ensure the Agency has identified an adequate means to verify and validate its performance measurement data, we also reviewed the fiscal 1996 performance plans for all five GPRA pilots. However, as a part of this survey, we did not test the accuracy or reliability of the performance measurement data reported for the pilots. We also did not evaluate the effectiveness of the general and application controls over computer processed performance measurement data. In addition, we reviewed the performance plans to determine whether program officials have developed objective, quantifiable, and measurable performance goals which define the level of performance to be achieved. The Agency's fiscal year 1995 performance reports for the LUST Program, Acid Rain Program and the Surface Water Treatment Rule/Public Water System Supervision Program were reviewed to determine if any program evaluations were conducted during the fiscal year covered by the report.

In response to a Congressional request, the National Academy of Public Administration (NAPA) conducted a study of EPA's process to allocate resources to the most pressing environmental concerns. We reviewed the NAPA report "Setting Priorities and Getting Results - A New Direction For EPA" (April 1995) to identify recommendations that would also help the Agency meet the requirements of GPRA. In response to the NAPA report, the Agency convened the Planning, Budgeting, and Accountability (PBA) Task Force. We reviewed the Task Force's report to determine how its recommendations related to GPRA requirements.

In addition, we evaluated the Agency's strategic planning efforts. Specifically, we reviewed the following documents:

(2) Chief Financial Officer's (CFO) Strategic Plan for Resources Management (issued spring 1996). Many of the GPRA requirements will reside with the Agency's CFO. Therefore, we reviewed the CFO's strategic plan to determine if ongoing and planned activities will accommodate GPRA.

(3) EPA's Financial Management Status Report and Five Year Plan for 1994 - 1999 (issued August 1994) was reviewed because GPRA imposes an accountability structure that requires EPA to set performance goals and track costs related to achieving those goals. We reviewed this financial management plan and interviewed Financial Management Division personnel to determine if the Agency will be in a position to accommodate goals-based budgeting and accounting.

Regarding the Agency's environmental goals, which are to be included in the Agency-wide strategic plan, we reviewed drafts of the Agency's National Environmental Goals including the most recent version dated June 1996, to determine if the goals are outcome related and cover the Agency's major functions and operations. We also reviewed the goals to determine if they will measure progress towards accomplishing the Agency's mission.

At the conclusion of the Federal Government National Performance Review, the Office of Management and Budget (OMB) conducted the "OMB Spring Review on Program Performance" which focused on how to build more and better performance information into the fiscal 1997 budget decision-making process. We reviewed the Agency's response to the OMB Spring/Summer Review which identifies the Agency's key program areas for the fiscal 1997 budget.

We also reviewed the Agency's "Performance Partnership Grants for State and Tribal Environmental Programs: Interim Guidance" dated July 1996 to determine how the Agency will track the cost of these grants and relate the grants to specific program goals.

In accordance with the Federal Managers' Financial Integrity Act (FMFIA), program offices periodically evaluate their internal controls by conducting assessments. We obtained a copy of the fiscal 1995 FMFIA report to the President to determine if the Agency had identified any applicable weaknesses in its evaluation of internal controls. However, we did not perform any tests to verify the accuracy and integrity of the report data. We found that the Agency had reported its environmental data quality as a material weakness.

Finally, we visited two Federal government agencies (Departments of Agriculture and Interior) to discuss their ongoing and planned GPRA efforts and to identify any GPRA best practices.

We performed this survey in accordance with Government Auditing Standards, issued by the Comptroller General of the United States. We performed our fieldwork from July 1995 to March 1996.

PRIOR AUDIT COVERAGE

We reviewed prior audit reports to identify those that contained findings and recommendations related to our survey objectives. The results of those audits that relate to our survey are discussed in the applicable sections of our report. We did not assess the status of the audit recommendations contained in these reports as a part of this survey. Program officials have generally agreed with the recommendations contained in these reports and have agreed to initiate the recommended corrective actions.

CHAPTER 2

EPA's Planning, Budgeting, and Accountability

Task Force Established a Framework For

Implementing The Government Performance

and Results Act

To respond to the National Academy of Public Administration's (NAPA) report on EPA entitled "Setting Priorities, Getting Results" (issued April 1995), the Agency created the Planning, Budgeting and Accountability (PBA) Task Force to improve and better integrate these functions within the Agency. Specifically, the Task Force was charged with identifying ways the Agency could better use scientific information in setting priorities, and better link the Agency's planning, budgeting and accountability functions.

In its February 1996 report "Managing For Results," the Task Force recommended a major restructuring of the Agency's planning, budgeting and accountability system. It also recommended a goals-based budget and establishment of a budget process based on comparative risk and strategic planning. Further, the Task Force report stated that the system must be used by the entire Agency and must completely integrate the Agency's planning and budgeting functions. The recommended planning, budgeting and accountability system has four basic elements:

Analysis - Science, Data and Information: Pulls together all relevant sources of science and data for use by Agency decisionmakers. The task force recommended immediate initiation of a series of steps to develop and use the best available science in making planning and budgeting decisions.
Planning: A strategic planning process for the Agency based on Agency goals. The strategic planning process would become more extensive than today and serve as the basis for budget decisions.
Budgeting: Under the new system, each goal in the strategic plan would be considered during the Agency's annual planning process. With real resource decisions based on environmental, programmatic, and management goals being made during the planning phase, the budgeting component of the system becomes what it should be -- a process for finalizing internal resource allocations, presenting a comprehensive and defensible budget to the OMB and Congress, and executing the well documented decisions that have been made. Finally, the Task Force report stated that the Agency should redesign the current program elements to reflect an improved relationship between goals, performance plans and measures of success. A new goals-based program element structure is critical to helping the Agency successfully request, defend and account for the resources needed to carry out its strategic mission.
Accountability: Annual performance plans would serve as the basis for both fiscal and environmental accountability. Accountability should focus on the Agency's accomplishments relative to the commitments made in annual performance plans.

The Task Force reported that a multi-year implementation process will be necessary to successfully transition to the new system. It further stated that the recommended changes are extensive and development of such a system will take several years of active implementation. However, the Task Force encouraged the Agency to begin initiation efforts by: completing the National Environmental Goals project in as timely a manner as possible; beginning efforts to develop an Agency-wide strategic plan as soon as possible; developing performance plans during the FY 1998 budget cycle; revising the Agency's budget structure and mandating that certain minimum requirements be adopted by all EPA programs; and establishing an accountability team to facilitate the transition from the old system to the new Planning, Budgeting and Accountability System.

According to the PBA Task Force, to be effective, the new integrated system will need a strong organizational center which clearly identifies roles and responsibilities and encourages close cooperation. The Administrator has requested the Acting Chief Financial Officer lead the effort to establish the new office and manage the new Agency-wide system associated with implementing the Task Force recommendations.

We commend the Agency on its initial efforts to move the Agency in a results oriented management direction that will help the Agency meet the requirements of the Government Performance and Results Act (GPRA). The following chapters further discuss how the GPRA requirements can be implemented within EPA.

CHAPTER 3

EPA Can Better Articulate Its Direction By Finalizing

Its Environmental Goals, Updating Its Strategic Plan,

and Developing A Clearer And More Complete

Mission Statement

EPA has set in motion an effort to draft National Environmental Goals and a new strategic plan. The Goals Project is a noteworthy and innovative effort. Despite the efforts of Agency staff, the project has not moved as quickly as originally hoped. Continued delays in issuing these goals will impact the Agency's ability to effectively plan, budget for and assess the results of its environmental programs. GPRA requires the Agency to develop a five year strategic plan effective September 30, 1997, which includes (among other things): a comprehensive mission statement covering the major functions and operations of the Agency; and, general goals and objectives, including outcome-related goals and objectives, for the major functions and operations of the agency. The Agency's current strategic plan needs to be updated in some of these areas, such as focusing its mission statement more on core functions that can provide direction for developing its goals. According to the Agency's preliminary milestones, revisions to the strategic plan are scheduled to be completed by October 1997. Finalizing the Agency's goals and revising its mission statement are critical to the Agency's development of a strategic plan.

EPA Needs to Finalize Its National Environmental Goals

The National Academy of Public Administration (NAPA) reported that "national environmental goals express intended, long-range results of national efforts to protect the environment, reflecting best professional judgement about the measurable progress this nation can and should achieve." NAPA also stated that National Environmental Goals are critical to EPA's planning efforts.

National Environmental Goals are intended to encourage not only EPA, but national innovation in improving the effectiveness of and reducing the costs of environmental protection, and to allow for more accountable results. The public holds EPA and other federal agencies accountable for protecting the environment; they want to know what their tax dollars are being spent for. National Environmental Goals will lead to a clearer understanding of what improvements people can expect from the dollars invested in environmental protection. The goals should help stimulate collection and reporting of better environmental information.

EPA's National Environmental Goals were initially scheduled to be issued Earth Day (April 22) 1995. More than a year later the goals have not been finalized. The Agency's Planning, Budgeting and Accountability preliminary milestones indicated that the National Environmental Goals were scheduled to be issued for Government review on April 30, 1996 with plans to finalize them by December 31, 1996. As of June 1996, the National Environmental Goals had been issued to key Government agencies, but had not been issued for full Government review.

According to program officials, finalization of the National Environmental Goals has been delayed due to the extensive comments received on initial and subsequent drafts of the goals, and the need to obtain a reasonable degree of agreement from major stakeholders at each step of the process. Further, in responding to our draft report, they indicated that the review by key Government agencies is taking longer than expected. They stated that they are working with OMB to clear the goals document and move into full government review. However, it is unlikely that the Agency's National Environmental Goals will be finalized by year end.

We agree with Agency officials that the success of the goals depends largely on stakeholder's agreement and that the unplanned delays are unfortunate. We reviewed the Agency's draft Proposed Environmental Goals for America with Milestones for 2005 (National Environmental Goals - dated June 1996). We found that EPA is attempting to obtain a reasonable degree of consensus in developing its National Environmental Goals. In the early stages of developing its National Environmental Goals, the Agency met with business and agricultural communities, environmental groups, Government agencies and other citizens to discuss what topics the goals should cover and how the goals should be expressed. The Agency also has taken into consideration goals set by the States and other Federal agencies and countries. The Agency needs to get back to these groups with the results of their work and get their reactions.

EPA Needs To Develop Agency Goals

In addition to the development of the National Environmental Goals, the Agency plans to develop a series of interim Agency goals. The Agency goals will build on the National Environmental Goals. According to the PBA Task Force report (Managing For Results), the Agency goals will express the extent to which EPA can help the nation achieve its National Environmental Goals. The preliminary milestones included in the PBA Task Force's report (dated February 23, 1996) indicate that the interim Agency goals were to be developed by April 1996. Since the Agency's National Environmental Goals have been delayed the interim goals were not developed by April 1996. In the Acting Chief Financial Officer's response to the OMB Summer Review (dated June 26, 1996), the Agency goals are scheduled to be finalized in March 1997. The new Planning, Budgeting and Accountability organization, which is in the planning and development stage, has been charged with developing the interim Agency goals. The Agency should place a high priority on developing both the National Environmental Goals and interim Agency goals since these documents will provide a foundation for measuring environmental progress over the next decade.

EPA Needs to Update Its Strategic Plan - "The New Generation of Environmental Protection"

GPRA requires the Agency to submit a 5-year strategic plan to OMB and Congress by September 30, 1997. The strategic plan must include:

(1) a comprehensive mission statement covering the major functions and operations of the agency;

(2) general goals and objectives, including outcome related goals and objectives, for the major functions and operations of the agency;

(3) a description of how the goals and objectives are to be achieved, including a description of the operational processes, skills and technology, and the human, capital, information, and other resources required to meet those goals and objectives;

(4) a description of how the performance goals included in the performance plan relate to the general goals and objectives in the strategic plan;

(5) an identification of those key factors external to the agency and beyond its control that could significantly affect the achievement of the general goals and objectives; and

(6) a description of the program evaluations used in establishing or revising general goals and objectives, with a schedule for future program evaluations.

We reviewed the Agency's 5-Year Strategic Plan "The New Generation of Environmental Protection" (July 1994), to determine if it met the GPRA requirements recognizing the plan was not prepared to meet these requirements. We found that the Agency's strategic plan did not include many of the elements required under GPRA for a strategic plan. Specifically, the Agency's mission statement does not provide guidance and focus for the Agency (further discussed in the next section). As previously mentioned, the Agency has not finalized the National Environmental Goals or the Agency's goals. Additionally, the Agency will need to include in its revised strategic plan a description of key factors affecting achievement of general goals and objectives and a description of any program evaluations used in establishing and revising general goals and objectives (program evaluations are further discussed in Chapter 4).

Prior to GPRA, there was no federal requirement for the development of an Agency-wide strategic plan. NAPA reported that EPA's 1994 strategic plan improved upon past efforts in several ways, however, the plan was not successful in establishing direction for the Agency. NAPA stated that EPA's 1994 strategic plan does not establish explicit priorities or support Agency decision-making and, therefore, it does not establish sufficient direction for Agency activities. It further stated that a relatively small number of employees produced the plan in a relatively short period of time, limiting the opportunities for other employees and stakeholders to participate.

Strategic plans are tools for agencies to use in setting priorities and allocating resources consistent with these priorities. Although a strategic plan is not a budget request, it should provide an overall guide for the formulation of future Agency budget requests and the projected levels of goal achievement should be commensurate with anticipated resource levels.

The nature of EPA's work dictates that the Agency ensure its strategic and annual planning process is responsive to emerging external environmental trends and forces that are likely to affect the Agency's performance. For example, the plan should identify the extent to which the expected level of achievement is contingent upon the actions or other situations external to the Agency and beyond its control. The plan should also identify significant changes in environmental and technological trends and conditions.

Working together with their customers, key program managers, and other stakeholders in the planning process, the Agency can create ownership of the plan. By combining the views of its stakeholders in developing the National Environmental Goals, the Agency has taken the right steps for the development of a comprehensive strategic plan.

A Clearer And More Complete Mission Statement Will Provide Guidance And Focus For The Agency

As previously mentioned, GPRA requires the Agency to develop a five-year strategic plan which contains "a comprehensive mission statement covering the major functions and operations of the agency." We reviewed the Agency's five year strategic plan, which identifies the Agency's mission as:

"The people who work at the Environmental Protection Agency are dedicated to improving and preserving the environment---in this country and around the globe. Highly skilled and culturally diverse, we work with our partners to protect human health, ecosystems, and the beauty of our environment using the best available science. We value and promote innovative and effective solutions to environmental problems. We strive to achieve the productive and sustainable use of natural resources on which all life and human activity depend."

The Agency's mission statement is fundamentally sound in stressing the integrated nature of the environment and an interest in global environmental quality. However, as required by GPRA, a comprehensive mission statement for EPA should cover the major functions and operations of the Agency.

According to Part 2 of OMB Circular A-11, the mission statement should be brief, defining the basic purpose of the agency, with particular focus on its core programs and activities. In addition, the mission statement may include a concise discussion of enabling or authorizing legislation, as well as identification of issues that Congress specifically charged the agency to address.

The Agency's current mission statement does not clearly establish Agency-wide direction, guidance and focus. On March 6, 1996, GAO testified before the Senate Committee on Governmental Affairs and House Committee on Government Reform and Oversight that "EPA had not been able to target its resources as efficiently as possible to address the nation's highest environmental priorities because it did not have an overarching legislative mission and its environmental responsibilities had not been integrated." Since the Agency does not have a statutory mission, it is even more important that the Agency develop a mission statement that clearly establishes Agency-wide direction, guidance and focus.

RECOMMENDATIONS

We recommend that the Assistant Administrator For Policy, Planning, and Evaluation finalize and publish the Agency's National Environmental Goals.

We recommend that the Acting Chief Financial Officer:

1. Revise the Agency-wide strategic plan "A New Generation of Environmental Protection, EPA's Five Year Strategic Plan," to incorporate the following GPRA requirements:

b. a description of general goals and objectives and how they will be achieved;

c. a description of the relationship between performance goals in the annual performance plan and general goals and objectives in the strategic plan;

d. identification of key factors that could affect the achievement of the Agency goals and objectives; and

e. a description of program evaluations used and a schedule for future evaluations.

2. Finalize the Planning, Budgeting and Accountability System and Organization Development Draft Workplan (dated 6/18/96), and periodically monitor ongoing efforts to accomplish the scheduled activities by the targeted milestone dates.

AGENCY COMMENTS AND OIG EVALUATION

Generally, the Acting Chief Financial Officer (CFO) agreed with our findings and recommendations. The Agency agreed that the National Environmental Goals need to be completed in a timely manner, but indicated the Office of Policy, Planning, and Evaluation (OPPE) has the lead responsibility for the National Environmental Goals. Therefore, we have changed the action official for this recommendation.

The Acting CFO also stated that a schedule has been developed which will ensure timely revision and submission of the revised strategic plan to OMB and Congress. Preliminary milestones outlined in the Planning, Budgeting and Accountability System and Organization Development Draft Workplan (dated 6/18/96) will satisfy the GPRA requirements. Since the workplan is a draft document, we are recommending the Agency finalize the workplan and periodically monitor ongoing efforts to complete the activities contained in the workplan.

Overall, the Agency's planned actions are consistent with our recommendations and the intent of GPRA. Nevertheless, we feel it is necessary to reinforce the importance of finalizing and issuing the Agency's National Environmental Goals. We recognize that this project is a monumental effort and progress to date has been slower than planned. Finalizing the Agency's National Environmental Goals is critical to revising the Agency-wide strategic plan. We acknowledge that the strategic plan can and will be revised using the latest draft of the National Environmental Goals at the time. However, the ideal situation would have been for EPA to have finalized the National Environmental Goals prior to making the final revisions to the Agency-wide strategic plan. While it may take years to fully implement GPRA, the Agency's continued emphasis in this area will provide for a smooth transition into a results oriented management direction.

CHAPTER 4

Results Oriented Management Requires

Timely, Accurate, and Complete

Performance Data

GPRA is intended to improve the information necessary to strengthen program management, evaluate program performance and set goals for the future. Managers need timely, accurate and complete information on how their programs are performing to make decisions about the programs. To help ensure the quality of this data, GPRA requires EPA to describe in annual performance plans the means to be used to verify and validate its performance information. A program evaluation, audit or any other procedure that would evaluate the general accuracy and reliability of the information contained in the annual performance report can be used. The Act also requires the Agency to include in its annual performance report a summary discussion of the results of any program evaluations completed during the fiscal year. Program evaluations are directed at answering questions about how programs are working, why results vary, and which program alternatives have the most impact and are the most cost effective.

We are concerned about how the Agency will verify and validate its program performance data. We found that while the performance plans for EPA's five current GPRA pilots provide a data source, three do not address how the Agency will validate and verify the accuracy and reliability of the Agency's reported performance measurement data. During past audits, we identified problems with the quality of performance data for these three programs. In addition, the Agency terminated its Program Evaluation Division in fiscal year 1996, without establishing alternative requirements for conducting program evaluations throughout the Agency. Without a system for conducting periodic program evaluations, Agency decision-makers may have to make important policy and resource decisions without information that could improve those decisions.

Performance information should be used to assess how well the Agency is progressing towards achieving its stated goals. Program managers should analyze and use performance information to inform and influence their:

program implementation decisions: for example, determining which programs to adopt;
planning decisions: for example, using information on performance trends to set more challenging yet realistic program goals;
management decisions: for example, making decisions about aligning management systems and structures to support the organization's desired outcomes; and,
budget and resource allocation decisions: for example, using information about the connection between a given level of resources and a given level of results for a program to decide if additional budgetary expenditures are beneficial.

Some Performance Plans Did Not Discuss How Performance Data Will Be Verified And Validated

According to the Chief Financial Officer's Council, the validity and integrity of an organization's performance data is essential to the success of GPRA. The Council contends that performance information will become more and more important for making management decisions with results oriented management.

We met with program officials participating in the five GPRA performance pilots and reviewed the fiscal year 1996 pilot performance plans to determine how program officials plan to verify and validate their performance data. GPRA requires the Agency to include in its performance plans the means to be used to verify and validate measures. We found from our review of the GPRA pilot performance plans that three of the five performance plans (LUST, Superfund, and Water) did not contain sufficient means to verify and validate performance data. For example, the

Superfund Program - provided a discussion of program reporting requirements and the source (systems from which the performance data will be extracted) of the performance information. Program officials also mentioned that three of their measures are included in the Chief Financial Officer's Financial Statement Report.
LUST Program - provided a discussion of reporting procedures and consistency reviews of the performance data submitted by the states.
Public Water System Supervision Program - provided a discussion of state reporting requirements and the source of the performance data.

Although a data source is needed to ensure that data is consistently collected and compiled, providing a source for the performance data and the policies and procedures for reporting performance information does not ensure the accuracy and reliability of the information being reported.

Prior Audits Have Identified Problems With Data Quality

We are concerned about performance plans not providing specific details about how some data will be verified and validated because prior OIG audits have identified problems with the quality of performance data for these programs as discussed below.

LUST Program - Consolidated Report of the Leaking Underground Storage Tank Program (Report no. 6100264, dated August 6, 1996). This capping report summarizes issues relating to the LUST Program which have been identified by OIG audits. In it, we discuss the results of our evaluations of the accuracy of the reported number of confirmed releases, cleanups initiated, and cleanups completed for six states (California, Idaho, Kansas, New York, New Mexico, and Ohio). Program officials have reported these program activities as performance measures for the LUST Program GPRA performance pilot. In addition, these accomplishments are reported in the Agency's financial statements. We found that the states' activity reports were generally unreliable and potentially misleading. Performance data reported to EPA by states were inaccurate. Based on the audit work performed in these six states, it was concluded that the LUST Program's procedures to verify and validate its performance data will not ensure the data are accurate and complete.

Program officials concurred with the report recommendations and commented that management controls will be implemented to ensure that reliable and timely information is obtained, maintained, reported and used. However, they disagreed that the misstatement in the activity reports represents a significant reporting problem.

Surface Water Treatment Rule/Public Water System Supervision Program - At the request of the Office of Ground Water and Drinking Water, a nationwide audit of EPA's Procedures To Ensure Drinking Water Data Integrity (Report no. 51005516, dated September 29, 1995) was performed. One of the objectives of the audit was to determine the frequency with which operators of community, surface public water systems (PWS) throughout the nation reported invalid or potentially falsified drinking water test data.

The results of the nationwide audit showed that overall, operators of community, surface PWS's generally reported valid data. However, based on a statistical sample, it was projected that nationwide approximately 12 percent of PWS's reported erroneous data one or more times. EPA and States placed a low priority on reviewing records for invalid or falsified data and did not believe falsification was widespread. Without State or EPA officials reviewing the validity of data, serious health risks could go undetected and negate EPA's and the State's ability to prevent water contamination. The Agency generally agreed with these findings and agreed to encourage States to follow-up on the facilities that reported invalid or potentially falsified data.

In addition, in its fiscal year 1995 performance report the Public Water System Supervision Program/Surface Water Treatment Rule, program officials reported that the data base for the pilot needs to be updated. They stated that "The data base currently supporting the pilot effort is not accurate." However, according to the plan, EPA and the States are working to develop and implement a new Safe Drinking Water Information System (SDWIS), which will address serious data quality concerns. The Office of Water has placed a high priority on data quality and has redirected resources into implementing SDWIS.

Superfund Program - 40 CFR, Sections 300.415, 300.420, and 300.430 (guidance for the removal, pre-removal, and remedial programs), requires the preparation of sampling and analysis plans (SAPs) that provide a process for obtaining environmental data of sufficient quality and quantity to satisfy data needs for planning and documenting a removal action, site evaluation and hazard ranking system activities. A SAP is to include a quality assurance project plan (QAPP) which describes policy, organization and functional activities necessary to develop adequate data.

An OIG audit of Region 8's Superfund field sampling activities, "Implementing Controls Would Improve Region 8's Superfund Field Sampling Activities" (Report no. 5400034, dated January 27, 1995), identified problems with untimely submission and undocumented approvals of SAPs and QAPPs. The review also revealed that the Regional Quality Assurance Officer (RQAO) conducted a management system review prior to our audit, which identified these same problems. Our audit confirmed that these problems continued to exist, although the RQAO had previously reported them as problems. Adequate review and approval of these plans will ensure the reliability and completeness of the remedial program's data.

Some Agency Program Offices Have Developed Techniques To Verify and Validate Performance Data

Program officials for two GPRA pilot programs, the Chesapeake Bay and Acid Rain programs, have developed means for verifying and validating their performance measurement data. The techniques developed by these offices, which are described below, provide examples of the kinds of things that can be done to help ensure data quality.

The Chesapeake Bay Program provided some discussion of the means used to verify and validate the performance data in its performance plan. Although the performance plan did not provide a thorough discussion on verification and validation, we found from our discussions with program officials that their performance data undergoes, at a minimum, four rigorous quality assurance reviews. The Chesapeake Bay Program does not report accomplishments until data is reviewed by a Bay program sub-committee; a QA/QC officer; the Executive Council consisting of the EPA Administrator, State Governors and the Chair of the Chesapeake Bay Commission, EPA and States' scientists; and the program's Scientific and Technical Advisory Committee.
The Acid Rain Program provided four validation measures for ensuring the accuracy and reliability of the program's performance data. These validation measures included: the number of continuous emission monitoring systems (CEMS) certified; the percentage of CEMS that meet 10% relative accuracy; the percentage of CEMS that exceed 7.5% relative accuracy; and the number of quarterly reports processed. Emissions data is collected directly from CEMS which are maintained according to quality assurance standards of accuracy, and is then transferred directly into EPA's emission tracking system, electronically, eliminating a source of errors potentially caused by repeated data entry.

Overall, whenever possible, reporting systems with built-in quality control mechanisms, such as the Acid Rain Program's Continuous Emission Monitoring System should be used to collect performance measurement data. To ensure the reliability of the information from other sources, the Agency should review and document processes to assure that information is being collected in a consistent and reliable fashion. Finally, program evaluations (further discussed in the next section), as required by GPRA, should be performed and should include field verification of selected indicators to ensure the accuracy and reliability of reported data.

Policies And Procedures For Program Evaluations Need To Be Established

GPRA requires that annual performance reports be submitted to the President and Congress by March 31 of each year. We reviewed the performance reports for fiscal year 1995 for the LUST, Acid Rain, and Surface Water Treatment Rule/Public Water System Supervision programs to determine, among other things, if the program offices conducted any program evaluations during the fiscal year and if so, how the results of these evaluations influenced fiscal year 1996 management decisions. We found from our reviews of performance reports, that program officials did not provide a discussion of whether program evaluations were performed as required by GPRA. In addition, the GPRA FY95 Performance Report Outline developed by the Office of Policy, Planning and Evaluation (OPPE) did not require program officials to include a discussion on program evaluations conducted during the fiscal year covered by the report.

Full GPRA implementation will require the Agency to ensure that program officials conduct periodic program evaluations and report the results of such evaluations in their annual performance reports. Program evaluations as defined by GPRA are "an assessment, through objective measurement and systematic analysis, of the manner and extent to which Federal programs achieve intended objectives." Program evaluations are aimed at revealing aspects of the program itself or how it is administered which are either hindering or enabling the organization's achievement of intended goals. GPRA requires the Agency to include in its strategic plan "a description of the program evaluations used in establishing or revising general goals and objectives, with a schedule for future program evaluations." It also requires the Agency to include in the program's performance report, a summary discussion of the results of any program evaluation completed during the fiscal year. OMB guidance (Part 2 of OMB Circular A-11, dated September 1995) for the preparation and submission of strategic plans states:

"Program evaluations that were used in preparing a strategic plan should be briefly described. A schedule for future program evaluations should be included. The schedule should outline the methodology to be used (e.g., longitudinal study), timetable, and the general scope of the evaluation, as well as particular issues to be addressed. At a minimum, schedules for future evaluations should cover the fiscal years prior to the next revision of the strategic plan."

OPPE terminated its Program Evaluation Division in fiscal 1996 due to strategic redirection in the fiscal year 1996 budget. Program officials participating in the performance pilot have not taken on the responsibility of conducting their own evaluations. Without a system for conducting periodic program evaluations, Agency decision-makers may have to make important policy and resource decisions without information that could improve those decisions.

Program evaluations, if used effectively, can provide program managers with the tools needed to identify changes which would make programs work better or cost less, and suggest which policies, strategies, or operations should be retained. Program managers will also be in a better position to set sound priorities and make wise investments. Agency-wide policies and procedures for conducting program evaluations currently do not exist. Additionally, the Agency has no centralized means of identifying whether program offices conduct program evaluations, assessing the quality of any such evaluations, or reporting how evaluations are used.

In a study conducted by the National Academy of Public Administration (NAPA), it was reported that:

"the Agency is unable to assess centrally the success of its activities. Program evaluation at EPA is too critical to agency-wide decision-making to be dispersed throughout the agency with no core capacity to support the administrator. Further, there can be no creditable response to external demands for accountability without some form of central program evaluation."

NAPA recommended that the Agency establish a program evaluation function for the Agency as a whole. They further recommended ongoing evaluations be a routine responsibility of managers in every EPA program. In response to NAPA's recommendations, the Agency's Planning, Budgeting and Accountability (PBA) Task Force generated a "schematic" representation of the ideal accountability system. The PBA Task Force's accountability system incorporates an evaluation process which will evaluate expenditures against the achievement of environmental progress. However, the Task Force recognized that their ideal accountability system may take a number of years to implement. The Agency has approved and accepted the Task Force recommendation. In fact, under an existing contract, OPPE has requested the development of a program evaluation system for EPA's organization structure.

RECOMMENDATIONS

We recommend that the Acting Chief Financial Officer:

1. Work with the Agency's Chief Information Officer to develop and issue an Agency-wide policy requiring program managers to include in their GPRA performance plans a discussion of how their performance data will be verified and validated.

2. Develop and issue an Agency-wide policy requiring program managers to conduct program evaluations when goals are not being accomplished. If goals are being met, require periodic program evaluations be conducted to address whether goals are being met in the most efficient and effective manner. The policy should also identify a centralized reporting location.

3. Develop and issue Agency-wide interim performance report guidance for the GPRA pilots. Ensure that the guidance satisfies the requirements of GPRA that the report:

(2) evaluate the performance plan for the current fiscal year relative to the performance achieved toward the performance goals in the fiscal year covered by the report,

(3) explain and describe, where a performance goal has not been met, why the goal was not met, and provide plans and a schedule for achieving the established performance goal, and

(4) include the summary findings of those program evaluations completed during the fiscal year covered by the report.

AGENCY COMMENTS AND OIG EVALUATION

The Agency agreed that timely, accurate and valid data are important for the successful implementation of a results-based management system. While the Agency agreed with our findings, they stated that "Excessive demands on data quality may paralyze our efforts to move forward. The CFO Council's report on GPRA implementation recognizes the need to work with the data at hand, that useful data is more important than perfect data. Also, the GAO's guide on GPRA implementation recognizes that a balanced approach is needed; their report calls for agencies to collect sufficiently complete, accurate and consistent data." The Agency also commented that it would be premature at this point to attempt to establish data standards since the specific needs of the new accountability system have yet to be defined. They also mentioned that responsibility for data quality resides primarily with EPA's program offices and Chief Information Officer.

In the CFO Council's GPRA Subcommittee baseline report entitled "Integrating Performance Measures Into The Budget Process (dated January 1996)," they report that "The IG can play an important positive role as well by helping assess the relevance and reliability of performance measures, and/or the reliability of the processes used to verify and validate the performance measures. We agree and believe that it is important for us to stress the need for the Agency to identify means to verify and validate data early on since our past audit work has identified data quality problems. We also believe that it is important for the Agency to move forward with quality performance data that will withstand scrutiny by Congress and other EPA stakeholders.

The Acting CFO agreed that program evaluations are essential to a successful results oriented management system. The Acting CFO also stated that her office will work with other Agency offices to ensure that effective analyses are built into the new management approach. She did not agree, however, "that routinized evaluations at predetermined intervals in a predetermined format is the best approach." We have revised our recommendation to require that program evaluations be conducted when goals are not being accomplished and periodically thereafter.

In addition, the Acting CFO agreed that guidance on annual performance reporting is needed. However, the annual performance report is a report of accomplishments against the annual performance plan. Since OMB has not developed guidance on the content of annual performance plans, it would be premature to prepare guidance on annual performance reporting.

The Act's approach to implementation is one of initiating multi-year pilot projects. This approach provides agencies with the opportunity to identify where they are and what needs to be done to prepare for full GPRA implementation. EPA has been participating in the GPRA performance pilot since its inception in 1994 and has submitted performance reports since 1995. In recognition of the fact that the Agency's performance reports do not meet minimum GPRA requirements, we are recommending that the Agency proceed in developing guidance or revising OPPE's GPRA Performance Report guidance, and ensure that the guidance meets GPRA minimum requirements.

CHAPTER 5

Cost Information Is Critical

To The Implementation Of GPRA

GPRA requires each agency to set performance goals beginning in fiscal year 1997 (for fiscal year 1999) and track costs related to achieving those goals. In addition, the CFO Act of 1990 calls for complete, reliable and timely information for managing Agency programs. Program officials participating in EPA's GPRA performance pilots stated they plan to track program costs by program element. Based on the results of our survey work, we found that program elements in the Agency's accounting system will not provide program officials with accurate and complete cost data. Agency officials are aware of problems with the current program element structure and are planning to make changes to the structure. We also reviewed the Agency's Performance Partnership Grant (PPG) Guidance For State and Tribal Environmental Programs: Interim Guidance (dated July 1996), to determine how the costs for PPGs will be tracked. We found that it is not clear how the Agency will relate these costs to Agency goals.

The Managerial Cost Accounting Concepts and Standards for the Federal Government were issued in July 1995. These standards require each reporting entity to accumulate and report the full cost of its activities on a regular basis for management information purposes, effective October 1, 1996. Cost data is essential for budgeting and controlling cost, performance measurement, or setting priorities. Without accurate and complete cost data, Agency managers are at a serious disadvantage during the planning and budgeting processes. EPA still needs to develop and issue cost accounting policies and procedures requiring program officials to begin tracking program costs in support of the GPRA and CFO Act requirements.

Tracking The Cost Of GPRA Performance Pilots

Program officials participating in EPA's GPRA performance pilots told us that they track their program costs in IFMS by program element. However, we found that the Agency will not be able to extract accurate and complete program cost from IFMS using program elements. For example, one of the GPRA pilots, the Acid Rain program issues grants to states for oversight of the acid rain programs in their states. It is estimated that for fiscal 1995, the Acid Rain program issued approximately $5.2 million in state grants. These grants, earmarked for Acid Rain program activities, are incorporated with the Office of Air's state grants. Because the Acid Rain program grants are issued as part of the Office of Air state grants, the Acid Rain program element in IFMS does not contain the program's grant funding. As a result, the Acid Rain program element in IFMS will not provide program managers with complete program costs.

The fiscal 1995 budget for another GPRA pilot, the Chesapeake Bay Program, was supplemented by $452,700 from the Multi-Media program. However, these funds were not officially transferred to the Chesapeake Bay program element, thus, these costs are not captured in the Chesapeake Bay's program element. In addition, in Region 3, we found that six regional FTEs provided to the Chesapeake Bay Program were not properly transferred to the Chesapeake Bay program element. Therefore, the total cost of the Chesapeake Bay Program was not being captured.

During other OIG audits we have also noted problems with developing information on the costs of EPA's programs. For example, the "Review of Barriers to Superfund Site Cleanups" (Report no. 5400100, dated August 31, 1995), was conducted to review the costs of completed and ongoing cleanups and track the changes over time. Both the auditors and Superfund program officials encountered difficulties in gathering cost information and as a result, it was reported that cost data from IFMS could not be used to track program costs because Superfund program data had not been fully integrated with IFMS.

Further an OIG audit report, "Region 4's Budget Execution and Cost Allocation", (Report no. 6100249, dated July 12, 1996) reported that Region 4 used personnel funds and workyears intended for environmental programs for management and support activities; the region did not transfer workyears and personnel funds to reflect planned and actual use; and, the region also used available workyears (FTE vacancies) in environmental programs to offset overstaffing in management and support programs without transferring the associated workyears. As a result, in fiscal year 1995, 21.6 of Region 4's program workyears and $1 million in related costs were not accurately reflected in EPA's official accounting system.

In order for the Agency to fully accommodate a goals based framework, the current program element structure must be revised based on goals, milestones, strategies, performance plans, and activities. We found that the current program element structure has hampered the development of a goals based budgeting process. Agency officials are aware of problems with the current program element structure. In June 1995, the Agency considered changing the program element structure as part of its administrative reforms. Subsequently, the Administrator and Deputy Administrator decided to change the structure to reflect the new goals-based approach. This decision was announced to the EPA Senior Leadership Council in a memorandum dated March 5, 1996. However, a decision has not been made on the specific changes.

The PBA Task Force also reported in February 1996, "one of the clearly identified barriers to the new system is the current program element structure of the Agency's budget. The Agency should redesign the current program elements to reflect an improved relationship to goals, performance plans, and measures of success." The Agency must obtain approval from the Office of Management and Budget and Congress prior to changing the program element budget structure. The Task Force noted that redesigning the Agency's program element structure is a major undertaking that could take a number of years to complete.

Tracking The Cost of EPA's Performance Partnership Grants (PPGs)

The Agency is authorized to issue PPGs to States and Tribes under the fiscal year 1996 appropriations statute. The purpose of PPGs is to improve environmental protection by providing States and Tribes with greater flexibility in how they use Federal grant funds. PPGs are voluntary and will provide States and Tribes with the option to combine funds from two or more of their media specific grants into one or more PPGs.

PPG recipients are required to maintain accounting and financial records which adequately identify the source (i.e., Federal funds and match or in-kind services) of the funds. The Agency recommends that these records contain information such as obligations, unobligated balances, outlays, expenditures and program income. PPG recipients are not required to track costs by specific media program. Rather, the total effort or costs incurred for the PPG work will be tracked.

The PBA Task Force reported, "The Agency will need to be able to report accurately on where its resources are being spent ....." The PBA Task Force recognized this as a problem and reported that it is not clear how the Agency will account fully for resources committed to this area. According to program officials, grant recipients are held accountable for tracking costs. The Agency's Grants Administration Division is currently leading an effort to develop regulations for PPG's. Tracking PPG costs and linking those costs to the goals and strategies and recipient performance are major issues being addressed by the workgroup. The workgroup's ultimate goal is to merge the objectives of GPRA into the PPG regulation and to look for opportunities to link costs to cross-media goals.

Federal Cost Accounting Requirements

In 1993, the Vice President recommended that the Federal Accounting Standards Advisory Board (FASAB) issue cost accounting standards for all federal activities. In July 1995, the Statement of Federal Financial Accounting Standards Number 4 - Managerial Cost Accounting Concepts and Standards for the Federal Government was issued. These standards are aimed at achieving three general objectives:

Provide program managers with relevant and reliable information relating costs to outputs and activities;
Provide relevant and reliable cost information to assist the Congress and executives in making decisions about allocating federal resources, authorizing and modifying programs, and evaluating program performance; and
Ensure consistency between costs reported in general purpose financial reports and costs reported to program managers.

The standards require each reporting entity to accumulate and report the full cost of its activities on a regular basis for management information purposes, effective October 1, 1996. Cost information is needed to improve federal financial management and managerial decision making. In managing federal programs, cost information is essential for (1) budgeting and cost control, (2) performance measurement, (3) determining reimbursements and setting fees and prices, (4) performing program evaluations, and (5) making economic choice decisions. These costs may be accumulated either through the use of cost accounting systems or through cost finding techniques.

Implementing Cost Accounting Within EPA

According to the CFO's strategic plan, cost accounting policies and procedures which meet the FASAB requirements and that allow cost information to be collected down to the project level in order to support GPRA objectives will be implemented. However, the Agency has not developed and issued policies and procedures requiring program officials to begin tracking program costs in support of GPRA.

The Agency installed an IFMS module, "Project Cost Accounting Subsystem (PCAS)" in September 1994, which allows for Agency-wide project cost accounting. PCAS can track program costs, associate project costs with customers, distribute project costs to customers, and bill customers for reimbursable project costs. In addition, the account code structure in IFMS was expanded effective October 1, 1995, to accommodate project cost accounting. In particular, according to the Comptroller's June 16, 1995 memorandum, the expanded account code gives EPA the capability to (1) track program-specific projects in all appropriations; and (2) prepare for goal-based budgeting as well as for GPRA. The Chief Financial Officer's strategic plan states that PCAS was installed to support the working capital fund, edit Superfund site data and allow Agency cost accounting. The Comptroller and Chief, Financial Systems Branch issued memorandums advising Senior Resource Officials and Regional Comptrollers of the advantages of using the full IFMS structure. However, the Agency has not instructed the program offices to begin taking advantage of PCAS and the expanded account code structure to track program costs in support of the GPRA and CFO Acts.

In conclusion, the Agency's efforts to implement GPRA are in line with a results oriented management approach. However, additional consideration should be given to tracking program costs for GPRA. Revising the program element structure to accommodate goals based budgeting should help provide the Agency with accurate and complete program costs which meet the requirements of GPRA. In addition, guidance should be provided to program offices on how to use PCAS to collect data on program costs.

RECOMMENDATIONS

We recommend that the Acting Chief Financial Officer:

1. Develop and issue Agency-wide cost accounting policies and procedures which meet the FASAB requirements and allow for the collection of cost information at a level to support GPRA and CFO Act requirements.

2. In conjunction with the Acting Assistant Administrator for Administration and Resources Management, reevaluate the Agency's financial reporting requirements for the Performance Partnership Grants to ensure that States and Tribes are required to provide the financial information needed for the Agency to implement GPRA.

AGENCY COMMENTS AND OIG EVALUATION

In responding to our draft report, the Acting Chief Financial Officer agreed with both of our recommendations. She indicated that her office has formed a workgroup to address the implementation of the cost accounting standard. She also stated the ultimate goal of the Agency's PPG workgroup is to merge the objectives of GPRA into the PPG regulation and to look for opportunities to link PPG costs to cross-media goals. The actions planned are consistent with the intent of our recommendations.

APPENDIX II

REPORT DISTRIBUTION LIST

Acting Assistant Administrator for Administration and

Resources Management (3101)

Assistant Administrator for Air and Radiation (6101)

Assistant Administrator for Solid Waste and Emergency

Response (5101)

Assistant Administrator for Water (4101)

Deputy Assistant Administrator for Water (4101)

Associate Administrator for Regional Operations and State/Local Relations (1501)

Director, Office of Strategic Planning and Environmental

Data (2161)

Director, Office of Information Resources Management (3401)

Director, Office of Grants and Debarment (3901F)

Director, Office of Atmospheric Programs (6201J)

Director, Office of Emergency and Remedial Response (5201G)

Director, Office of Ground Water and Drinking Water (4601)

Director, Office of Program Management (5201G)

Director, Chesapeake Bay Program Office

Director, Financial Management Division (3303F)

Chief, Financial Compliance and Quality Assurance Staff (3303F)

Audit Liaison, Planning, Budgeting and Accountability (3301)

Audit Liaison, Office of Administration and Resources

Management (3102)

Audit Liaison, Office of Air and Radiation (6102)

Audit Liaison, Office of Solid Waste and Emergency

Response (5101)

Audit Liaison, Office of Water (4102)

Audit Liaison, Office of Policy, Planning, and Evaluation (2112)

Audit Liaison, Region 3

Assistant Regional Administrator, Region 8

Go To:

Table of Contents

Created February 12, 1997

Top of page

EPA Implementation of Government Performance and Results Act

EPA Office of Inspector General

GPRA

Local Navigation