Jump to main content.

PRESIDENT'S COUNCIL ON INTEGRITY AND EFFICIENCY REVIEW of APPLICATION SOFTWARE MAINTENANCE in FEDERAL AGENCIES

[PCIE Seal]

PRESIDENT'S COUNCIL ON INTEGRITY AND EFFICIENCY REVIEW of APPLICATION SOFTWARE MAINTENANCE in FEDERAL AGENCIES

INTRODUCTION

Background

Federal agencies continue to rely heavily on information technology. Accordingly, over the years, agencies have continued to invest significantly in information technology. In fiscal year 1996, executive agencies expect to obligate more than $26 billion for information technology (IT) investments and operations. This IT spending represents a critical investment of public tax dollars affecting virtually every government function.

Information technology continues to play a vital role in supporting the operations and missions of Federal agencies. This growing use of, and reliance on, information systems introduces opportunities for fraud, waste, and abuse in Federal programs. This, in turn, raises concern for maintaining an adequate level of computer system integrity and security. Thus, cost-effective internal controls are essential to properly manage and safeguard computer operations and sensitive agency data.

From this perspective, in September 1986, the President's Council on Integrity and Efficiency (PCIE) Computer Committee initiated the Computer Systems Integrity Project (CSIP). The Computer Committee's name was changed to the Information Technology Committee in 1990. The project is a multi-task effort focusing on controls, security, and other integrity issues related to the entire data processing system life cycle. The objectives of the overall project are to assess the integrity of Federal computer systems and develop recommendations for Governmentwide improvements in standards, procedures, documentation, and operations affecting computer systems integrity.

To date, four tasks have been completed. Task 1, Survey of Agency Implementation of Computer Systems Integrity Requirements, was completed with the June 1988 issuance of a summary PCIE report. Task 2A, Review of General Controls in Federal Computer Systems, was completed in October 1988 when a summary PCIE report was issued. Task 2B, Review of Application Controls in Federal Contract Tracking Systems was completed with the April 1991 issuance of a summary PCIE report. The lead agency for Task 3, Followup Audit on the Implementation of the PCIE CSIP Task 1 and Task 2A Audit Report Recommendations, concluded that the issuance of a consolidated summary PCIE report would produce few benefits, and therefore such a report was not prepared. For a detailed description of these tasks see Appendix A, page 33.

In May 1992, the Committee approved sponsorship of CSIP Task 4, Review of Application Software Maintenance in Federal Agencies. Software maintenance was selected as an area for review for two primary reasons. First and foremost, controls over application software modifications are vital to maintaining the reliability and integrity of sensitive and mission-critical application systems of Federal agencies. Failure to adequately control application software maintenance exposes an organization to corruption of system information--which in turn can lead to erroneous management decisions and/or the inability to meet organizational missions. The rapidly growing inventory of Federal application software systems is increasing the need for a strong, disciplined, clearly defined software maintenance approach which will guarantee the usefulness and integrity of the data maintained by those systems. Second, Federal managers have historically undervalued the economic importance of a sound, comprehensive software maintenance program.

The General Accounting Office's (GAO) February 1981 report entitled "Federal Agencies' Maintenance Of Computer Programs: Expensive And Undermanaged" (GAO/AMD-81-25) stated software maintenance in the Government was largely undefined, unquantified, and undermanaged. Specifically, Federal agencies spend millions of dollars annually on computer software (program) maintenance but little is done to manage it. GAO studied 15 Federal computer sites in detail, and received completed questionnaires from hundreds of others. All reported large maintenance efforts, but few had good maintenance records and very few managed software maintenance as a function. GAO concluded that improvements can and should be made both in reducing maintenance on existing software and in constructing new software to reduce its eventual maintenance costs. The report stated that published estimates of programmer time spent on the maintenance function range from 50 to 80 percent. Perhaps even more significant, some estimates say that up to 60 percent of all Automatic Data Processing (ADP) dollars will be spent on software maintenance. The report recommended that the National Bureau of Standards should issue a standard definition and specific technical guidelines for software maintenance. Heads of Federal agencies should require their automatic data processing managers to manage the software maintenance as a discrete function.

GAO's July 1995 report entitled "Information Technology Investment A Governmentwide Overview" (Report No. GAO/AIMD-95-208) stated the total amount of annual federal spending for IT is unknown because the Office of Management and Budget (OMB) does not collect comprehensive IT-related budget data on a Governmentwide basis. For example, agencies with annual IT-related obligations under $2 million prior to FY 1996 and under $50 million for FY 1996 and beyond, as well as the legislative and judicial branches of the federal government, are not required to report IT obligation data to OMB. In addition, computers that are embedded in weapon systems are not included in the reporting to OMB. The Department of Defense estimates the costs of these computers to be $24 billion to $32 billion annually. Agencies do not break out IT obligations as separate line items in their budget documents, but rather include this information within program or administrative costs. Software maintenance costs are included in the "Support Services" category along with maintenance costs for equipment, telecommunications, data entry, training, planning, studies, facilities management, custom software development, system analysis and design, and computer performance evaluation and capacity management. The

Overview of Governmentwide Requirements

Governmentwide policies and procedures for software maintenance have been prescribed in general terms by a variety of Federal sources, including Congress and OMB. For example, Public Law (P.L.) 96-511, Paperwork Reduction Act of 1980 requires Departments and Agencies to ensure (1) ADP and communications technologies are acquired and used in a manner which improves service delivery and program management; and (2) the collection, maintenance, use and dissemination of information by the Federal Government is consistent with applicable laws relating to confidentiality, including the Privacy Act. Furthermore, OMB Circular A-123, Internal Control Systems, requires agencies to establish and maintain a system of internal controls to provide reasonable assurance that Government resources, including information resources, are protected from fraud, waste, unauthorized use, and misappropriation.

The National Institute of Standards and Technology (NIST) also provides guidance on software maintenance through Federal Information Processing Standard Publication (FIPS Pub.) 106,

Guideline on Software Maintenance. This Guideline is intended for use by both managers and maintainers. It addresses the need for a strong, disciplined, clearly defined approach to software maintenance. It emphasizes that the maintainability of the software must be taken into consideration throughout the lifecycle of a software system. Software must be planned, developed, used, and maintained with future software maintenance in mind. The techniques discussed in this Guideline are recommended for use in all Federal ADP organizations. Specifically, this publication includes information on:

For more information on Federal criteria pertaining to software maintenance issues see Appendix B, page 37.

Scope

Seven Inspectors General (IG) Offices participated in Task 4: the Departments of Health and Human Services(1) (HHS), Housing and Urban Development (HUD), and State (DOS); Environmental Protection Agency (EPA); National Aeronautics and Space Administration (NASA); National Science Foundation (NSF); and the Railroad Retirement Board (RRB). The Inspectors General reviewed the management of the software maintenance process (e.g., policies and procedures, contract management, etc.) for major mission-support/administrative applications within their agencies. Total Government obligations for Support Services, which includes software maintenance, totals almost $8.4 billion for fiscal year 1995. A profile of the missions of each agency and the applications reviewed during this task is included in Appendix C, page 41.

Audit work at each participating agency was conducted in accordance with the U.S. GAO's "Government Auditing Standards." Based on the results of field work conducted between November 1992 through November 1995, six individual agency reports were prepared and issued (see Appendix D, page 49). The remaining participant, SSA, made no recommendations and issued a close-out memorandum in lieu of a report. The results of audit work at one agency indicated that the quality and quantity of software maintenance cost information was adequate, management of contractor performance was effective, and overall management of the software maintenance lifecycle process was effective. Therefore, this report consolidates the findings of the five individual agency assessments that reported weaknesses in the aforementioned areas, and presents recommendations addressing Governmentwide issues.

The primary objectives of Task 4 were to identify common software maintenance problems across Federal Agencies and to identify Governmentwide recommendations to oversight agencies. Specifically, agencies determined the (1) adequacy and completeness of agency software maintenance policies and procedures, (2) effectiveness of controls over software changes, (3) extent to which software maintenance is budgeted and tracked, and (4) effectiveness of agency management of software maintenance contractors. The EPA/OIG had overall responsibility for coordinating this task. As the task leader, EPA/OIG staff developed a Task 4 survey guide, audit guide, and data collection instrument which participants used in performing this audit work. See Appendix E, page 51, for a description of the audit methodology.

Definition of Software Maintenance

In accordance with FIPS Pub. 106, "Guideline on Software Maintenance," software maintenance is the set of activities which result in changes to the originally accepted (baseline) product set. These changes consist of corrections, insertions, deletions, extensions, and enhancements to the baseline system. Generally, these changes are made in order to keep the system functioning in an evolving, expanding user and operational environment.

Functionally, software maintenance activities can be divided into three categories: perfective, adaptive, and corrective.

Perfective maintenance includes all changes, insertions, deletions, modifications, extensions, and enhancements which are made to a system to meet the evolving and/or expanding needs of the user. Perfective maintenance refers to enhancements made to improve software performance, maintainability, or understandability. It is generally performed as a result of new or changing requirements, or in an attempt to augment or fine tune the software. Activities designed to make the code easier to understand and to work with, such as restructuring or documentation updates (often referred to as 'preventive' maintenance) and optimization of code to make it run faster or use storage more efficiently are also included in the perfective category. Perfective maintenance comprises approximately 60 percent of all software maintenance.

Adaptive maintenance consists of any effort which is initiated as a result of changes in the environment in which a software system must operate. These environmental changes are normally beyond the control of the software maintainer and consist primarily of changes to the: (1) rules, laws, and regulations that affect the system; (2) hardware configurations, e.g., new terminals, local printers, etc.; (3) data formats, file structures; and (4) system software, e.g., operating systems, compilers, utilities, etc. Approximately 20 percent of software maintenance falls into the adaptive category.

Corrective maintenance refers to changes necessitated by actual errors (induced or residual "bugs") in a system. Corrective maintenance consists of activities normally considered to be error correction required to keep the system operational. By its nature, corrective maintenance is usually a reactive process where an error must be fixed immediately. Not all corrective maintenance is performed in this immediate response mode; but all corrective maintenance is related to the system not performing as originally intended. The three main causes of corrective maintenance are: (1) design errors, (2) logic errors, and (3) coding errors. Corrective maintenance accounts for approximately 20 percent of all software maintenance.

SOFTWARE MAINTENANCE WEAKNESSES

SOFTWARE MAINTENANCE COSTS

Federal Departments and Agencies do not properly identify and account for software maintenance costs. For example, agencies do not consistently include the costs of administrative and clerical salaries, materials, computer usage, telecommunications, overhead costs, and Federal employee salaries in software maintenance costs. In addition, agencies reported cost-benefit analyses are not consistently prepared, updated, and/or maintained for application systems. As a result, agencies are not in a position to make informed budgeting and planning decisions regarding systems operations and maintenance. In addition, software maintenance expenses are being inaccurately reported to OMB. These weaknesses occurred primarily because agencies are not defining software maintenance consistently, and Federal accounting requirements are not being followed.

OMB Budget Criteria

OMB Circular A-11, dated June 6, 1995, entitled "Preparation and Submission of Budget Estimates" requires agencies that obligate more than $50 million annually for information technology activities to submit a report on obligations for the following categories:

Also, agencies are required to submit an annual report on obligations and full time equivalent (FTEs)(2) employment data for financial systems. FTE employment should include Federal employees identified as directly designing, operating, or maintaining financial systems.

OMB Circular A-109, dated April 5, 1976, entitled, "Major Systems Acquisitions" states each agency acquiring major systems should maintain a capability to: (1) predict, review, assess, negotiate, and monitor lifecycle cost(3); (2) assess acquisition cost, schedule and performance experience against predictions, and provide such assessments for consideration by the agency head at key decision points; (3) make new assessments where significant costs, schedule or performance variances occur; (4) estimate lifecycle costs during system design concept evaluation and selection, full-scale development, facility conversion, and production, to ensure appropriate trade-offs among investment costs, ownership costs, schedules, and performance; and (5) use independent cost estimates, where feasible, for comparison purposes.

OMB Circular A-130, dated February 8, 1996 and entitled "Management of Federal Information Resources" requires Federal agencies to account for the full costs of operating information technology facilities and recover these costs from the users. This Circular also requires Federal agencies to implement a system to distribute the full cost of providing services to the user. The term "full cost" is comprised of all direct, indirect, and general and administrative costs incurred in the operation of the facility. These costs include personnel, equipment, software, supplies, contracted services, space occupancy, intra-agency services, and other services. In addition, agencies are required to conduct a cost-benefit analysis for information systems to be used as budget justification material, as well as become part of the ongoing management oversight process to ensure prudent allocation of scarce resources. The cost-benefit analysis should be updated over the information system life cycle because of such factors as significant changes in projected costs and benefits, significant changes in information technology capabilities, and major changes in requirements (including legislative or regulatory changes).

Improper Identification of Software Maintenance Costs

Four of the five agencies are not properly identifying software maintenance costs. For example, agencies do not consistently include the costs of administrative and clerical salaries, materials, computer usage, telecommunications, overhead costs, and Federal employee salaries in software maintenance costs. One agency reported $4.7 million of system engineering software maintenance costs were categorized as development instead of maintenance. Another agency reported most system managers were only able to provide the contract dollars spent in support of their systems. In addition, agencies do not and in some cases cannot accurately distinguish software maintenance costs from development and operations. Agencies reported ADP support service contracts often cover many ADP-related activities, therefore making it difficult, if not impossible, to separate the cost of maintenance from other ADP development and operational activities. Without accurate software maintenance cost information, management cannot make informed decisions regarding system enhancements or replacements. According to National Bureau of Standards Special Publication 500-106 entitled "Guidance on Software Maintenance", federal managers responsible for software application systems estimate that 60 to 70 percent of the total application software resources are spent on software maintenance. Using that basis(4), it can be estimated that approximately $2.7 to $3.1 billion was spent on software maintenance efforts Governmentwide in fiscal year 1995. These figures are conservative because they do not include related software costs in the services, support services, and supplies categories. If we include 25 percent of the dollars in these categories, our Governmentwide software maintenance cost estimates would increase by $2 to $2.5 billion.

GAO reported in its July 1995 information technology report (see page 3) that the need to achieve high returns on information technology investments and reduce systems development risks has never been greater, given the public's demand for a government that works better and costs less. Increasingly, Federal agencies' ability to improve their performance and reduce costs depends on automated data processing information needed to make good decisions, hold down costs, and improve services to the public.

Managers cannot effectively evaluate the efficiency of software maintenance efforts without knowing the extent of resources devoted to maintenance. Attempts to reduce maintenance may fail because of lack of information. Costs associated with the software maintenance function should be monitored and recorded. Segregating the costs of the different work functions involved in data processing is a prerequisite for effective management of ADP costs. Some specific reasons for accumulating such data are:

System modification requests should be reviewed and evaluated before any actual work is performed on the system. The evaluation should consider, among other things, the costs and benefits of the change. In particular, perfective maintenance changes must be thoroughly analyzed, since they are optional in the sense that failure to implement them will not adversely affect system performance. Changes should be approved only if the benefits outweigh the costs. Because corrective and adaptive maintenance are not optional, cost-benefit analysis is most appropriately used to determine the best option for applying the required changes.

Cost-Benefit Analysis

Three agencies reported cost-benefit analyses are not consistently prepared, updated, and/or maintained for application systems. Two agencies reported that none of the system owners were able to provide a current updated cost-benefit analysis. System managers at another agency relied on contractor estimates to determine the level of effort and costs involved with a proposed modification. For many systems, these estimates, if performed at all, were solicited after the requested modification had been approved by the change control committee. These estimates did not weigh the benefits to be derived by implementation of the proposed software change as opposed to the costs involved in the modification process. Further, several system managers responded that decisions to implement a change were based on whether sufficient funds were available to do the job and whether system functionality was affected. Software maintenance was performed until available funds ran out; the benefits to be realized were a secondary consideration.

As a result of these weaknesses, system managers are not in a position to make informed and effective short- and long-term decisions, such as choosing the best enhancement, upgrade, or improvement based on accurate cost information or cost-benefit analysis. In particular, agencies cannot determine when systems should be evaluated for redesign or replacement because of excessive maintenance costs. A current cost-benefit analysis serves a valuable purpose, especially when allocating scarce budgetary resources. These analyses ensure the most cost-effective alternative which satisfies system requirements is chosen. Without it, scarce budgetary resources may be mis-allocated. Non-cost effective changes to the software could be approved by management and implemented into the production environment, despite the marginal benefits which would be realized from the modification. This could result in a lack of funds for those proposed software changes which are either mandatory in nature or necessary for proper application functioning.

Inaccurate Reporting

Software maintenance expenses are being inaccurately reported to OMB. For example, one agency reported it did not capitalize at least $38.1 million of software costs which distorted the accuracy of its current financial statements. This agency reported that all costs associated with the purchase, development, and enhancement of the ten systems they reviewed were treated as annual expenses. Another agency stated its annual information systems submission, "Report on Obligations for Major Information Technology Systems" is inaccurate because of improper maintenance cost classifications. Problems such as failing to include overhead charges, ignoring the cost of Federal personnel, commingling maintenance costs with other ADP-related expenses, and failing to capitalize and depreciate maintenance costs limit agencies' ability to report costs accurately. Additionally, agencies cannot prepare accurate financial statements and therefore are unable to meet the requirements of the Chief Financial Officers Act(5).

Reasons for Software Maintenance Cost Deficiencies

These weaknesses occurred primarily because agencies are not defining software maintenance consistently, and Federal accounting requirements are not being followed. Two of the participants stated their agencies do not have a standard definition of software maintenance. Three participants reported maintenance is defined by the types of software maintenance (i.e., perfective, adaptive, and corrective), while two others adopted the FIPS Pub. 106 definition. The remaining participant stated maintenance is distinguished from development by the length of time it takes to complete the task (e.g., a maintenance effort is generally performed within a shorter timeframe than a software development effort). This resulted in system managers not interpreting and applying a consistent definition of software maintenance. Specifically, the same types of costs (e.g., administrative and clerical salaries, materials, computer usage, telecommunications, overhead costs, and Federal employee salaries) are not consistently accounted for as maintenance costs. One participant reported the absence of a formal, consistently applied definition of software maintenance had created a significant understatement of maintenance costs. This participant requested various IT managers to review the project charges of seven systems in their tracking system over an 18 month period. The managers reclassified the charges based on the definition and functional classifications in FIPS Pub. 106. This review disclosed that maintenance costs were understated by $4.7 million in the tracking system. The primary reason for this understatement is that projects are categorized as either maintenance or development based on criteria and classifications that are not functionally segregated or consistent with FIPS Pub. 106. Also, it was noted the classifications used in both the project tracking and customer billing and allocation processes were inconsistent.

Further, information managers are not complying with Federal accounting requirements. These requirements dictate that information managers establish and implement mechanisms to track, differentiate, and capitalize maintenance costs. However, five agencies reported they lack a comprehensive process or system to accumulate costs. One agency stated they have not implemented a cost accumulation process for major information systems. The account code structure in its financial system does not provide for cost tracking by information systems and a cost accounting process has not been established to properly allocate cost by system. Another agency reported its cost allocation and billing process is complex, disjointed, requires much manual intervention, and does not interface with the central accounting system. In addition, the project cost chart of accounts lacks specific accounts of activities necessary for cost measurement purposes.

Recommendations:

We recommend that OMB:

SOFTWARE MAINTENANCE CONTRACTS AND CONTRACTOR OVERSIGHT

The contracts used by agencies for software maintenance do not adequately protect the Government's interests. Agencies are not consistently awarding contracts that motivate contractors to perform at optimal levels. In addition, the monitoring of these contracts present an unstructured, poorly controlled approach to the management of maintenance for critical Government applications. Consequently, agencies lack control over software maintenance activities and rely heavily on contractors. These weaknesses are primarily due to the use of non performance-based contracting methods and agencies not specifying performance measures in software maintenance contracts. Furthermore, Federal employees lacked the technical expertise required to adequately oversee maintenance contractors.

Office of Federal Procurement Policy Criteria

The Office of Federal Procurement Policy (OFPP) Letter 91-2 entitled "Service Contracting," dated April 9, 1991, states it is the policy of the Federal Government that agencies (1) use performance-based contracting methods to the maximum extent practicable when acquiring services, and (2) carefully select acquisition and contract administration strategies, methods, and techniques that best accommodate the requirements. Furthermore, contract types most likely to motivate contractors to perform at optimal levels shall be chosen. Fixed price contracts are appropriate for services that can be objectively defined and for which risk of performance is manageable. For such acquisitions, performance-based statements of work and measurable performance standards and surveillance plans shall be developed and fixed price contracts shall be preferred over cost reimbursement contracts. To the maximum extent practicable, contracts shall include incentive provisions to ensure that contractors are rewarded for good performance and quality assurance deduction schedules to discourage unsatisfactory performance. These provisions shall be based on measurement against predetermined performance standards and surveillance plans. In addition, agencies must document in the contract files the reasons for those instances where performance-based contracting methods are not used.

OFPP Pamphlet #4, entitled "A Guide for Writing and Administering Performance Statement of Work for Service Contracts," dated October 1980, provides guidelines for writing and administering performance statements of work for service contracts. It describes a systematic means to develop statements of work and quality assurance surveillance plans in order for agencies to define and measure the quality of contractors' performance.

Government's Interests Inadequately Protected

Three of the five agencies identified weaknesses with the type of contracts being used in the performance of software maintenance and/or the monitoring of these contractors. The contracts used by agencies for software maintenance do not adequately protect the Government's interests. Agencies reported they are not awarding performance-based contracts on a regular basis for software maintenance work. Instead, they are using labor hour contracts. One agency reported it almost exclusively awards labor hour (i.e., Cost Plus Fixed Fee (CPFF)) contracts for software development and maintenance efforts when other, cost-effective and performance-oriented contract types could be used. A key factor in determining if a CPFF contract should be used is the procuring agency's inability to sufficiently describe the work requirements. Consequently, CPFF contracts place the Government at a higher risk than other contract types because of this lack of identifiable work requirements. The use of CPFF contracts as the sole contracting vehicle minimizes the contractor's incentive to perform well and control costs. CPFF contracts should be used only after other performance-oriented contract types have been considered. While generally appropriate to accomplish the type of maintenance support an agency needs when maintaining its software, labor hour contracts require constant monitoring because the total cost of the project is not fixed. In a contract based on labor hours, the contractor is reimbursed strictly on the basis of hours worked at the fixed labor rates specified in the contract. Since a fixed price is not established by the contractor to deliver a final product, close monitoring of these contracts is necessary. Procurement officials stated it is acceptable, and quite common, to use labor hour contracts for software maintenance and enhancement projects, however they acknowledged that labor hour contracts provide no incentive for the contractor to control costs or improve labor efficiency. Contract types that motivate contractors to perform at optimal levels include:

In addition, the monitoring of labor hour contracts presents an unstructured, poorly controlled approach to the management of maintenance for critical Government applications. One agency reported Contracting Officer Representatives (COR) were generally not aware of the extent of their contract monitoring responsibilities(6). Some CORs had not received appropriate training and therefore were not aware of what their official COR duties entailed. Consequently, the CORs did not always fulfill the role that was intended, and some critical monitoring of contractor performance went undone. Some CORs failed to adequately monitor the technical performance of the contractor they were to oversee, and others did not specifically know how they could assure the Government received what it needed through its software maintenance contracts. A COR lacking the day-to-day knowledge of a contract, a schedule of the work required, and adequate technical knowledge may not have enough information to sufficiently monitor software maintenance contractors. Reviews of contract documents provided strong evidence that CORs routinely certify contract invoices for payment without supporting documentation. Even when contractors provided documentation in support of their bills, CORs frequently certified invoices for payment without evaluating/verifying the support. For example, one agency received two invoices that charged over $17,000 for 372 hours of work recorded by one contract employee during a two-week period. The COR approved payment for this total. However, agency auditors' work revealed that the billing was actually for a 2-month period. The COR stated that he relies on the Contracting Officer (CO) to question invoice charges that appear to be inappropriate or too high. However, the CO said he is not in a position to question the contractor's charges because he does not routinely receive copies of contractor invoices. One agency identified that COR duties were sometimes informally transferred to others, usually lower-grade employees. Transferring monitoring duties is problematic for two reasons. First, the officially designated COR may not have any assurance that the monitoring duties appropriate for the contract are fully and competently carried out. Second, the person to whom the COR duties are redelegated frequently does not have (1) day-to-day knowledge of the contract employees' activities, (2) overall knowledge of the terms of the contract, and (3) access to either the contract invoices or the contract terms and conditions.

Consequently, agencies lack control over software maintenance activities and rely heavily on contractor personnel to perform maintenance activities on critical Federal systems. One agency reported Federal management involvement focused on the initial review and approval of proposed changes, and chose to rely on contractor personnel to perform and oversee those reviews and controls which were built into the final stages of software modification. The assigned program office review board, or similarly responsible management personnel, had little or no interaction with the modification process once the software change had been sent to the contractor for work. Contractor personnel performed the actual design, coding, and testing of software changes for most application systems reviewed. Although independent peer reviews and unit tests were often part of the contractor's procedures, in many cases the originating program office did not participate in an oversight capacity. Auditors concluded the extent and frequency of program office interface during the final stages of change control processing was often minimal and, in some cases, limited to administratively routing the modified code to the data center for implementation. Another agency concluded the extent of software maintenance that is routinely performed by contractors increases the government's vulnerability to excessive contractor costs and limits its ability to control software maintenance activity. The heavy use of contractors has had negative effects on operations. For example, a help desk at one agency is staffed entirely with contract employees because of a shortage of knowledgeable Federal employees to help users with their problems. This situation raises management control issues because contractors may not report user complaints or may initiate unnecessary software changes thereby increasing the need for their services. In addition, if a contract fails to specify required contractor personnel qualifications, agencies' options are limited if the contractor does not supply qualified personnel.

Performance-Based Contracting Methods Should Be Utilized

These weaknesses are primarily due to the use of non performance-based contracting methods, and agencies not specifying performance measures in software maintenance contracts(7). One agency reported that none of the contract files reviewed contained the required documentation to justify why performance-based contracting methods were not used. They also noted that all of the contracts contained identical "boilerplate" Statements of Work (SOW) except for the staffing requirements section. Agency officials stated "boilerplate" SOWs are used because they allow flexibility in responding to time sensitive legislative and administrative requirements. However, SOWs for time sensitive tasks require specific, not "boilerplate," performance and quality requirements to ensure successful timely completion of tasks with minimal risk of rework. Another reason given for using "boilerplate" SOWs is that program officials cannot always specify software requirements. However, the SOW is the element of a contract that should be used to clearly communicate to the contractor the Government's expectations and serve as a basis for evaluating the contractor's performance. To avoid developing vague and imprecise SOWs, measurable performance standards, and acceptable quality levels must be identified within the SOW for the individual contracts. This gives agencies the necessary leverage to ensure that contractors' performance levels are of an acceptable quality and the Government only pays for products and services that meet contracting standards.

In addition, agencies are issuing contracts that do not contain appropriate measures to ensure adequate contractor performance. Agencies are issuing contracts that are vague, do not specify deliverables, lack product delivery dates, and lack a description of the government's responsibility for reviewing and approving contract deliverables. For example, a contract employee who helped maintain an application at one agency was never asked to provide performance and progress reports. The non-personal services contract under which he performed did not specify, other than in general terms, what was expected. While the contractor's work was supposed to be governed by the contract, he performed a variety of assignments not specified in any agreement. He was not required to submit invoices to document the hours worked. In fact, the contractor did not formally provide any products to management as an outcome of his contractual relationship, but was paid every two weeks based on an agreed upon annual salary. Officials at another agency indicated they rely on their personal and professional expertise, as well as user acceptance in determining the acceptability of deliverables and for measuring contractor performance. However, without establishing and incorporating measurable performance standards(8), acceptable quality levels(9), and quality assurance documents in the SOW, agencies cannot hold the contractor accountable for the quality of products or services provided. For example, in the ADP solicitation reviewed at one agency the performance measurement entitled, "Quality of Project Performance" indicated the contractor "will be evaluated to the extent the work is error free." However, a more appropriate measurable performance standard could be 100 percent of the software changes moved into the production environment do not require a correction release. The associated Acceptable Quality Level would indicate that deviation of more than 5 percent of releases will result in a deduction. The performance indicator would be operational in the production environment without error for two or more production cycles. Without performance measurements, agencies cannot determine whether the millions of dollars paid each year for contractors to develop and maintain application systems are spent wisely. Also, if performance data is not collected and analyzed, agencies cannot prevent serious operational problems caused by deficient contractor performance. Furthermore, Federal employees lacked the technical expertise required to adequately oversee maintenance contractors. One agency reported that most of their application systems relied on contractor personnel to perform software maintenance activities because of the absence of qualified full-time employees. The lack of technical expertise among Federal employees available to properly perform software maintenance and monitor technical contractors has hindered the Government's ability to reduce its reliance on contractors.

Recommendations:

We recommend that OMB:

CHANGE CONTROL MANAGEMENT

Federal Departments and Agencies are not adequately managing the software change control and configuration management(10) processes. Specifically, Federal Departments and Agencies cited weaknesses with the change request process, change review and approval, and testing. As a result, agencies lack assurance that (1) applications will perform as intended and (2) management controls will adequately safeguard the integrity of the applications. These weaknesses resulted from agencies not following software maintenance policies, standards, and procedures for requesting, approving, and testing changes.

NIST Guidance

FIPS Pub. 106, dated June 15, 1984 and entitled "Guideline on Software Maintenance," prescribes guidelines for achieving a strong, disciplined, and clearly defined approach to system software maintenance. As the publication explains, the primary purpose of change control is to assure smooth operational continuity and orderly evolution of the system. Effective change control is necessary to ensure that all system software installation and maintenance requirements are performed in a structured and controlled manner and provide management with a chronological history of all software modifications. An effective change control process includes the following control points: (1) a formal change request, (2) centralized review and approval of change requests, and (3) testing of changes. Compliance with such change control procedures further helps to ensure adherence to established standards and performance criteria for system software, and facilitates communication between software maintenance personnel and data center management.

Standard Change Request Forms Not Utilized

Three of the five agencies identified weaknesses with the method used to request application systems changes. Specifically, agencies were not using standardized change control request forms, or used no form at all. In fact, one agency report stated that even though numerous change control forms exist, the user community prefers to initiate changes by telephone call, or e-mail. All changes considered for a system should be formally requested in writing. These requests may be initiated by the user or maintainer in response to discovered errors, new requirements, or changing needs. Change requests should be submitted on forms which contain the following information: name of requestor, date of requestor, purpose for request, name of program(s) affected, name of document(s) affected, name of data file(s) affected, date request satisfactorily completed, date new version operational, name of maintainer, date of review, name of reviewer, and review decision. Procedures may vary regarding the format of a change request, but it is imperative that each request be fully documented, in writing, so it can be formally reviewed. Change requests should be carefully evaluated by the project manager or a change review board and decisions to proceed should be based on all the pertinent areas of consideration (probable effects on the system, actual need, resource requirements vs. resource availability, budgetary considerations, priority, etc.). The decision as to whether or not to accept the change, and reasons for that decision, should be recorded and included in the permanent documentation of the system. One agency reported that in several cases, the anticipated level of effort was the key factor in determining the exact submission channel and what data was required for submitting a change.

Without a standardized change request form, sufficient data may not be available to adequately evaluate the nature of a proposed software change or its resulting benefits, cost, or impact on the application system. In addition, approved changes could be misclassified or ranked inappropriately due to insufficient information. It is also possible that a review board may disapprove an important and necessary change due to a lack of information. Finally, valuable time and resources could be wasted by management in trying to discern the nature of the requested change or returning a request for more information.

Inadequate Review and Approval of Changes Prior To Implementation

Four of the five agencies identified weaknesses with the review and approval process for software changes. In fact, one agency report stated that 70 percent of the application systems it reviewed had minimal evidence to support that software changes were formally reviewed, approved, or "closed" by either the change control board or responsible originating management officials. Another agency report stated that none of the application systems reviewed had a centralized change review process. Change requests should be carefully reviewed and evaluated before any actual work is performed on the system.

The evaluation should take into consideration, among other things, the staff resources available versus the estimated workload of the request; the estimated additional computing resources which will be required for the design, test, debug and operation of the modified system; and the time and cost of updating the documentation. The approval point should be centralized for all software maintenance projects, so that one person, or group of persons, has the knowledge of all requested and actual work being performed on the system. In practice, the review process ranges from the review and sign-off by the project manager or user, to the convening of a change review board which formally approves or rejects changes. The purpose of this process is to ensure: (1) all of the requirements of the change request have been met; (2) the system will perform according to specifications; (3) the changes will not adversely impact the rest of the system or other users; (4) all procedures are followed and rules and guidelines adhered to; and (5) the change is indeed ready for installation in the production system.

Furthermore, offices are not adhering to the priority level assigned to each application change. For example, a report stated that in 66 percent of the applications reviewed, significant problems in applying the requested changes could be traced to not prioritizing all change requests. Specifically, the project listed as the top priority was frequently ignored, and technical staff and supervisors performed maintenance changes that were not prioritized. A priority level should be assigned to the requested work. All review actions and findings should be added to the system documentation folder. This process will reduce the amount of unnecessary and/or unjustified work which is often performed on a system.

Without a centralized review and approval point for application system changes, similar enhancements to the system may be processed individually, thereby wasting limited resources; or, software changes could be implemented despite the cost-effectiveness of the proposed modifications. In addition, proposed software changes may be implemented without sufficient impact analysis and, therefore, interact negatively with other application software components. For example, a major software interface problem was created at one agency which led to severe voucher payment backlogs, totaling over $1.25 million, and lasting from 1991 through 1993. Finally, proposed changes of lesser importance may be implemented ahead of more significant and necessary modifications. In fact, one agency reported that a project scheduled for implementation in six to eight months has taken more than two years, because it was never added to the priority listing schedule.

Inadequate Testing of Application Software Changes

Four of the five agencies reported weaknesses with the testing of application software changes. Specifically, test plans were not developed, test results were not analyzed, and proper test methodologies were not followed. For example, one agency reported that 93 percent of the maintenance projects reviewed did not have a written test plan. Another agency stated that over 70 percent of the project leaders were unable to provide a Test Plan and a Test Analysis and Results Report. Testing is a critical component of software maintenance. Testing standards and procedures should define the degree and depth of testing to be performed and the disposition of test materials upon successful completion of the testing. Whenever possible, the test procedures and test data should be developed by someone other than the person who performed the actual maintenance on the system. During the testing stage, the software and its related documentation should be evaluated in terms of readiness for implementation. The goal of testing is to find errors and, therefore, a test plan should (1) define the degree and depth of testing to be performed; (2) describe the expected output; and (3) test for valid, invalid, expected, and unexpected cases. The format and content of test plans and test analysis reports should emphasize the importance of (1) identifying and segregating the various functions of the program to be tested; (2) describing the strategy and limitations of the testing; and (3) describing the input data and expected output data for each planned test. A test analysis report should be prepared which summarizes and documents the test results and findings. The analysis summary should present the software capabilities, deficiencies, and recommendations.

Also, the functional user communities at two agencies were not given the opportunity to perform user acceptance testing. Acceptance testing is commonly performed by the functional users after system testing has been completed. Acceptance testing is considered the 'last line of defense' for the end user. The end users perform functional tests on the modified software using live data, test data, or a combination of data. The test should examine whether or not the program is doing what it is supposed to do. One agency report stated that functional and technical requirements for pending maintenance changes were poorly defined, not defined at all, or poorly communicated to technical personnel responsible for making the changes.

As a result of the above weaknesses, changes to software may not be sufficiently tested to account for all valid, invalid, expected, and unexpected outputs. Insufficient testing and analysis of test results could result in source code which fails when introduced to the production environment, due to unforeseen transaction conditions, interfaces, or user input. In addition, management cannot be certain the application will perform as intended. For example, during an eight month period at one agency, 26 releases(11) of software changes required 44 re-releases to correct one or more problems with the previous release. Furthermore, when an application lacks evidence that functional and technical requirements have been determined, reviewed, and formally approved in advance, it is difficult to ensure that all the necessary steps were completed. Without a functional acceptance test, program managers have no assurance that perfective or corrective changes made will satisfy the user needs when placed into production. This increases the risk that implementing the change will waste time and resources. In fact, at one agency 16 percent of all releases created new problems that were very costly in information technology and program office time. Without an effective testing process, there is no assurance that management controls exist to safeguard an application's integrity.

Agencies Do Not Follow Software Maintenance Policies, Standards, and Procedures

These change control weaknesses resulted from agencies not following software maintenance policies, standards, and procedures for requesting, approving, and testing changes. Departments and Agencies did not consistently apply standard procedures to ensure consistently good software maintenance practices. Furthermore, offices within the agencies determined their own maintenance procedures for the applications they develop. Due to this decentralized control, instances were found where maintenance changes were not centrally received, prioritized, and tracked. A software maintenance policy should address the (1) need and justification for changes; (2) responsibility for making changes; (3) change controls and testing procedures; and (4) use of modern programming practices, techniques and tools. Implementation of the policy has the effect of enforcing adherence to rules regarding the operating software and documentation from initiation through completion of the requested changes. The key to controlling changes to a system is the centralization of change approval and formal requesting of changes.

Recommendation:

We recommend that OMB:

Footnotes

  1. HHS committed to the assignment; however, the Social Security Administration (SSA) performed the work for this audit. On March 31, 1995, SSA became an independent agency under section 101 of the Social Security Independence and Program Improvements Act of 1994.
    2.

  2. FTE employment should include Federal employees identified as directly designing, developing, operating, or maintaining the system.

  3. The Circular defines lifecycle costs as the sum total of the direct, indirect, recurring, nonrecurring, and other related costs incurred, or estimated to be incurred, in the design, development, production, operation, maintenance and support of a major system over its anticipated useful life span.

  4. EPA Auditors included the costs of software, lease of software, and 50 percent of personnel costs in compiling total application software resources.

  5. This Act provides for the production of complete, reliable, timely, and consistent financial information for use by the executive branch of the Government and the Congress in the financing, management, and evaluation of Federal programs.

  6. Typical COR monitoring duties may include ensuring that hours billed by a contractor are reasonably accurate and appropriate, that the required work is performed, and other charges billed are in agreement with the terms of the contract.

  7. Both the OFPP Letter 91-2 and OFPP Pamphlet #4 identify the Quality Assurance Surveillance Plan (QASP) and the Performance Requirement Summary (PRS) as the two primary documents agencies can use for planning, measuring, and monitoring contractor performance. A QASP is a well-defined written document used to ensure that systematic quality assurance methods are used. The PRS summarizes the performance requirements of the contractor's evaluation.

  8. Performance standards are characterized as an acknowledged measure for comparison of quantitative and/or qualitative values. It is important to select performance standards that pertain to the deliverables, products, or services being acquired.

  9. Acceptable Quality Levels (AQLs) are the maximum percentage of allowable variance from the norm before a deliverable, product, or service is rejected. The AQLs recognize that defective performance sometimes happens unintentionally.

  10. Software configuration is defined as an arrangement of software parts, including all elements necessary for the software to work. Configuration management refers to the process of identifying and documenting the software configuration and then systematically controlling changes to it to maintain its integrity and to trace configuration changess.

  11. A release is a set of software changes.
Go To:
[Table of Contents Icon] Table of Contents [Appendices Icon] Appendices

Created February 2, 1997

Top of page

 

Local Navigation

Jump to main content.