Information Technology Risk Management

IJnited

States

General

Acrounling

Offic*r

GAO

Information Managernerd and Technology Division

Information Technology: An Audit Guide for Assessing Acquisition Risks

I;AO/lMTE(:-8.1.4

Preface

~~._

.–. –.~

The federal government depends heavily on a variety of information technology products and services to serve the public. Each year the government spends billions of dollars on computer equipment., services, software, and telecommunications. The success or failure of information system acquisitions affects executive agencies’credibility with the Congress and the public as well as their abilities to carry out their missions effectively and efficiently. The General Accounting Office (GAO) and offices of inspectors general have consistently identified problems with information technology acquisitions. Problems identified in numerous evaluations include information systems that do not meet users’needs, exceed cost estimates, or take significantly longer than expected to complete. This guide provides a logical framework for evaluating information technology acquisitions. It incorporates a risk assessment methodology intended to reduce audit planning time and ensure that significant issues are included. It is based on a model of the acquisition process developed by GAO in cooperation with a wide range of federal and private sector officials. ’The model outlines the process used to acquire information technologies and identifies elements of the process that are essential for planning and carrying out acquisitions. This guide is intended for use in planning and conducting risk assessments of computer hardware and software, telecommunications, and system development acquisitions. A risk assessment is the process of identifying potential risks in a system under development and then determining the significance of each risk in terms of its likelihood and impact on the acquisition’ cost, schedule, and ability s

The Essay on DIAD (Delivery Information Acquisition Device)

1. List the various ways that DIAD (Delivery Information Acquisition Device) improves customer service. DIAD has improved customer service by: Real time tracking and data tracking – The ability to track customer package on the spot using Technology (GPRS/CDMA) Fast delivery and on time Accountability and Reliable – Efficient tracking Efficient and Reduction of data entry error Availability ...

‘ information Technology: A Mtrdcl tu Help Managers Decrease Acquisition Risks (GACYIMTECX 1.G,August 1990)

Page 1 GAOIIMTEC-ELI.4 Assessing Acquisition Risks

Preface

to meet the agency’ needs. Such assessments may s have their greatest impact if carried out early, when an agency can more easily alter its acquisition plans and strategy to manage and control the identified

riSkS.

The audit guide consists of 10 chapters, with appendixes. Chapter 1 introduces the purpose of the guide, explains its essential concepts and techniques, and provides direction in tailoring the guide for use on specific assignments. Chapters 2 through 10 address specific activities in the acquisition process. The nine chapters present audit guidance on:

l l l l l l l l l

management and user support, project staffing, needs/requirements/specifications, alternatives, acquisition planning, solicitation document, source selection, contract management, and test and acceptance. Each chapter lists audit objectives, commonly expected documentation, detailed audit questions, and references to federal regulations and guidance. The chapters are intended to assist in the identification of specific risk areas and to contribute to an overall assessment of how well an agency is meeting its acquisition objectives. This audit guide is also available in a software format, accompanied by reference materials such as the GAO model of the acquisition process, relevant federal acquisition regulations, General Services Administration (GSA) guidance, and Office of Management and Budget (OMB) circulars. The software version utilizes a hypertext software package to help auditors quickly and flexibly review

The Research paper on The Impact of Internal Auditing on the Telecommunication Industry

Internal auditing is an independent appraisal functions established within an organization to examine, review and evaluate operations as a service to management. The internal audit department is an internal part of the organization and function under the policies established by management. An audit is not limited to the review of the management process in the accounting and financial areas alone. ...

Page 2

GAOAMTEC-8.1.1

Assessing

Acquisition

Risks

Preface

I

all the included documents. This software may be requested from GAO by returning the card included with this guide. This guide supplements and does not replace other GAO policies or procedures. It was prepared under the direction of Jack I,. Brock, Jr., Director, Government Information and Financial Management, who can be reached at (202>512-6406.Other major contributors to the guide are listed in appendix IV.

Ralph V. Carlone Assistant Comptroller General Information Management and Technology Division

Werner Grosshans Assistant Comptroller General for Policy

Page 3

GAOIIMTEC-8.1.4

Assessing

Acquisition

Risks

Contents

Preface Chapter 1 Introduction

Objectives Approach Assignment Planning Organization and Use of the Audit Guide Application of Audit Guide to Prototyping Methodology Presenting Conclusions and Recommendations Audit Objectives Documentation Required Audit Steps: Top Management Support Audit Steps: LJserInvolvement References Audit Objective Documentation Required Audit Steps: Project Management Audit Steps: Project Staff References Audit Objectives Documentation Reauired Audit Stew: Needs Determination Audit Steps: Requirements Analysis Audit Steps: Specifications Audit Steps: Test Plans References:

–

14 16 17 17 17 19 21 22 22 22 22 23 24 25 26 26 26 27 30 32 32 34 34 34

Chapter 2 Management and User Support

Chapter 3 Project Staffing

Chapter 4 Needs/ Reauirements/ Sp&ifkations

Chapter 5 Alternatives

Audit Objectives Documentation Required

Page 4

GAO/IMTEC-8.1.4

Asses&g

Acquisition

Risks

Contents

Audit Steps References

34 38 40 40 40 40 43

Chapter 6 Acquisition Planning Chapter 7 Solicitation Document Chapter 8

Audit Objective Documentation Required Audit Steps References Audit Objectives Documentation Required Audit Steps References

44

45 45 46 48

The Essay on Guide to Hardware Chapter 10 Answers

Place the following networking technologies in the order of their highest speed, from slowest to fastest: Fiber optic, dial-up networking, cable modem, Fast Ethernet Dial-up networking, cable modem, fiber-optic, Fast Ethernet 2. What is the difference between ADSL and SDSL? ADSL uses one upload speed from the consumer to an ISP and a faster download speed. SDSL uses equal bandwidths in both ...

50

Audit Steps References

50 51 51 54 55 56 56 56 59 60 61 61 61 63 64 68 80 86

Chapter 9 Contract Management Chapter 10 Test and Acceptance Appendixes

Audit Objectives Documentation Required Audit Steps References Audit Objectives Documentation Required Audit Steps References Appendix I: Acquisition Profile Appendix II: Management Metrics Appendix III: Reference Materials Appendix Iv: Major Contributors to This Audit Guide

Page 6

GAO/IMTEC-8.1.4

Assessing

Acquisition

Risks

contents

Glossary

87 11 73 74 75 76 78

Table Figures

Table I, 1: Map to the Audit Guide Figure 11.1: Problem Reports in Software Project Figure 11.2:Problems By Priority Levels and Number of Months Unresolved Figure II.3: Software Requirements Changes Figure 11.4:Development Progress Figure 11.5: Software Size Estimates Abbreviations

ANSI APR co COCOMO COTR DPA FAR FIPS FIRMR GAO GSA IEEE IFB IMTEC NIST QMB PM RFP SSA SSAC SSEB Page 6

American National Standards Institute agency procurement request Contracting Officer Constructive Cost Model Contracting Officer Technical Representative delegation of procurement authority Federal Acquisition Regulation Federal Information Processing Standards Federal Information Resource Management Regulation General Accounting Office General Services Administration Institute for Electrical and Electronic Engineers invitation for bid Information Management and Technology Division National Institute of Standards and Technology Office of Management and Budget program manager request for proposals source selection authority Source Selection Advisory Committee Source Selection Evaluation Board

GAOLIMTEC-8.1.4 Assessing Acquisition Risks

Page 7

GAO/IMTEC-B.Y.4

Assessing

Acquisition

Risks

Chapter 1

Introduction

This audit guide is based on and incorporates GAO’S information technology acquisition model. The model describes three phases in the acquisition process: presolicitation, solicitation and award, and postaward. It sets out essential activities in each phase along with critical factors related to those activities. The model is intended to give managers an overview of the acquisition process and to help them decrease acquisition risks. The model’ critical factors were drawn from the s judgment and expertise of a wide range of knowledgeable individuals from government and private industry. Compliance with these critical factors can increase the likelihood that an acquisition will meet an agency’ needs at a reasonable cost and s in a timely manner.

The Term Paper on United States Agency Nsa Information

FBI The Federal Bureau of Investigation (FBI) is the principal investigative arm of the United States Department of Justice. Title 28, United States Code, Section 533, which authorizes the Attorney General to "appoint officials to detect... crimes against the United States," and other federal statutes give the FBI the authority and responsibility to investigate specific crimes. At present, the FBI ...

Objectives

This guide is intended to help auditors conduct more focused reviews of information technology acquisitions by enabling them to quickly identify significant areas of risk. Using this guide will help auditors identify critical factors not addressed by management, make a general assessment of any procurement risks, and provide rapid feedback to agency officials so they can take corrective action in a timely and efficient manner. Use of the guide should be selectively tailored to the requirements of particular reviews and adapted to the status of the acquisition. Auditors will need to exercise professional judgment in assessing the significance of audit results or findings. For example, the guide assists auditors in determining how an agency identified and defined its requirements. Professional judgment is necessary to evaluate this information and determine if the agency conducted an adequate requirements analysis.

Page B

GAOIIMTEC-8.1.4

Assessing

Acquisition

Risks

Chapter 1 Introduction

Some areas of assessment will require the expertise of auditors with specific technical skills and experience. These specific areas include knowledge of solicitation procedures, benchmarking and other performance or capability validation techniques, and knowledge of technical areas such as database management and telecommunications networks. These areas are identified within the guide. The audit team should include experienced members with enough knowledge of information technologies to satisfy the Government Auditing Standards requirement that auditors have appropriate skills and knowledge.

Approach

The audit approach described in this guide is intended to result in a risk assessment of an acquisition project at any point in its development. The auditor will be trying to determine whether a project will result in a specified product or level of performance and will be delivered at a specified time and cost. An auditor should use this guide to identify areas that are most likely to result in technical failures, unmet user needs, cost overruns, or schedule delays. Those risks should then be brought to the attention of appropriate agency officials. The audit steps and questions provided are directed toward assessing whether an agency has sufficiently addressed critical factors, including support from managers and users, adequate project staff, and controls over the acquisition’ scope s before and after a contract is awarded.

The Term Paper on Audit Exam

The understanding, and the letter, should include the auditor's responsibilities, including the fact that an audit is designed to provide only reasonable assurance of detecting material misstatements, and therefore may not necessarily detect all misstatements that exist. 5. Management representation letter. Management's statement that there have been no communications from regulatory agencies ...

When planning a risk assessment, the auditor should first review the agency’ acquisition policies and s directives to identify the organizations and individuals responsible for approving procurements. Approval thresholds, for example, should show which officials have the authority to review and approve acquisitions. The agency’ directives should also show the specific s

Assignment Planning

Page 9

GAOIIMTEC-8.1.4

Assessing

L4rquisitior~

RXts

Chapter 1 Introduction

milestones and documentation required for a procurement. The auditor should also review previous studies or audits of the acquisition project and of the agency’ s information resources management functions. Reports by GAO or other auditing institutions can provide valuable background information. The auditor should also determine whether previous recommendations have been carried out.

Organization and Use of the Audit Guide

The chapters of this audit guide focus on logically distinct steps of the acquisition process, as described on page 10 of GAO’S acquisition model. ’The following table identifies the appropriate chapters for reviewing the various steps of an acquisition. In general, the auditor will want to concentrate on the steps that are relevant to the phase of the acquisition being reviewed. However, regardless of how far the acquisition has advanced, at a minimum the auditor should always ascertain whether senior managers and users were involved in the project’ initiation. The s auditor should also verify that the agency has defined its needs and requirements to support its mission, and that those requirements continue to be valid as the acquisition progresses through contract award and contract management.

‘ GAO/lMTEC-8.16,

August

The Essay on The Importance of A Project Management Office

Project management can be a tedious job especially if the personnel or department in charge is already loaded with tons of work. It may be hard to cope up with the schedule, time pressure, workload, and other factors. In line with this, the task of handling such tasks must be assigned to a specialized department known as Project Management Office. What is a Project Management Office? A Project ...

1990.

Page 10

GAOIIMTEC-8.1.4

Assessing

Acquisition

Risks

Chapter 1 Introduction

Table 1.1: Map to the Audit Guide

Acquisition

Phase

Steps in Each Phase Initiate Project Analyze Requirements Identify Alternatives Prepare Acquisition Plan Prepare Specifications

Chapter 2, 3 4 5 6 4 2, 3 7 0 0 8 8 9 9

I: Presolicitation

II. Solicitation Award

and

Maintain Project Structure Prepare Solicitation Release Solicitation Evaluate Proposals Negotiate with Vendors Select Contractor

III. Postaward

Establish Contract Management Monitor Contract Performance Test and Accept

Each of the following chapters provides references to regulations and other guidance relevant to material in the chapter. In addition, each chapter identifies specific audit objectives and documentation expected for the major activities at that point in the acquisition process. The documents listed may exist with different names than those used here. The auditor should refer to agency-specific requirements for more information. Finally, each chapter sets out audit steps to help plan and conduct the assessment of an acquisition. The questions may need to be tailored to

Page 11

GAO/IMTEC-8.1.4

Assessing

Acquisition

Risks

Chapter 1 Introduction

the particular circumstances of an audit, using the subject agency’ requirements. s The appendixes provide further information for use in planning and conducting an acquisition audit. Appendix I contains a worksheet, called an acquisition profile, to summarize essential information about the acquisition project. The worksheet should contain such information as the names and locations of units responsible for the acquisition, the project purpose, and the expected cost and time frames. The completed profile should be kept available for later reference. If a profile exists from an earlier assignment, the auditor should review and update it. Appendix II of this guide describes techniques to use in identifying software development risks of delays and cost overruns, known collectively as management metics or indicators. These techniques require information about the size of the system being developed and about progress after development has begun. Thus, the techniques are only appropriate for use after the agency has completed its design and progressed into developing the system. An auditor using this guide to review an acquisition that includes significant software development and has a contract in place should review appendix II for techniques to help identify variances from the agency’ cost and s schedule estimates. In some cases, the project team may be unable to complete a project on time due to an unrealistic schedule. The cost models described in appendix II can be used to make a rough estimate of how long a system development may require. Such an estimate can be compared to agency plans to see if the agency has committed itself to unrealistically short time frames. In other cases, delays in completing scheduled activities, such as system design, coding, and testing, can lead to project slippage later on. Auditors will need to tailor the use

Page 12

GAOIIMTEC-8.1.4

Assessing

Acquisition

Risks

Chapter 1

Introduction

of the tools described in appendix II to the circumstances of the audit. Appendix III provides a comprehensive list of references cited throughout the guide, as well as publications that may be useful for technical information. Finally, a glossary is attached that defines a wide range of technical and procurement terminology.

Application of Audit Guide to Prototyping Methodology

This acquisition guide is intended for use in reviewing any information technology acquisition, regardless of the system development methodology being used. The guide is structured around the federal acquisition process, and is independent of development methods for information systems. An auditor should recognize that there are different system development models that may be used when a system development effort is acquired. Such models may include the “waterfall” model, rapid prototyping, or evolutionary development. Some of the documents required under different development approaches may differ. Prototyping, for example, may act as part of the requirements definition process, helping the agency identify and control areas of high uncertainty and technical risk. In this situation the auditor should (1) focus on determining how one or more prototypes or incremental versions function to define the agency’ requirements and (2) determine how the s system development methodology used by the agency controls the prototyping process. One approach to using prototyping as part of the system development process has been described as a “spiral” model of system development. 2 The spiral model portrays a process in which an agency

“For a description of prototyping and the spiral model, see Roger S. Pressman, Software Engineering: A Practitioner’ Approach, 3rd ed. s (New York: McGraw-Hill, Inc., 1992), pp. 26-34.

Page 13 GAO/IMTEC-8.1.4 Assessing Acquisition Risks

,

Chapter 1 Introduction

iteratively (1) determines its objectives, alternatives, and constraints; (2) evaluates its alternatives and identifies and resolves risk issues; (3) develops and verifies its next-level products; and (4) plans its next phases. Prototypes are developed or modified as part of the second phase. As part of this model, a prototype is developed or revised whenever a risk analysis shows that significant areas of uncertainty remain that pose substantial risks to project success. When the system has been defined well enough to manage risks effectively, the agency develops and tests a full-scale system. In order to review an acquisition that is using prototypes, the auditor should determine what regulations or guidance the agency has to define a prototyping methodology. This methodology will establish the documentation and approval points that agency officials should meet. The auditor can then measure the progress of the prototypes against the agency’ criteria. s

Presenting Conclusions and

Recommendations

The auditor should conclude the review by identifying outstanding areas of risk and the agency’ actions to s address those risks. Note should be made of how well the agency has addressed the critical factors in GAO’ S 1990model of information technology acquisitions as well as agency compliance with acquisition regulations, standards, and other federal guidance. 3 Results of the review should be communicated to agency officials for their comment, in accordance with government auditing standards. The auditor should also recommend changes to ensure that the agency is properly addressing the critical factors in the GAO model. For example, the auditor may recommend a greater role for users if they are not involved in approving alternatives or

“GAO/iMTEC%. 1.6, August 1990.

Page 14 GAOffMTEC-8.1.4 Assessing Acquisition Risks

Chapter 1

Introduction

validating system requirements. Similarly, the auditor should recommend that the agency take appropriate actions where senior management involvement seems lacking, or where the project organization is unstable and subject to high turnover. The recommendations should be reviewed with agency officials and be appropriate to the probability and significance of the risks identified.

Page

16

EAO/IMTEC-8.1.4 Assessing

-kcquisition

Risks

Chapter 2

Management and User Support

This chapter focuses on levels of commitment and support for a planned acquisition by senior managers and users, two key stakeholders in an acquisition. Senior managers include those who have overall agency responsibility for strategic, including information-related, objectives. Users are those who operate or rely on agency information resources and include the managers and staff responsible for agency policies and programs supported by the acquisition. The involvement and support of senior management throughout an acquisition is essential for success, Senior management should envision the agency’ s acquisition goals, define strategic objectives, and oversee the projects that implement the overall vision. One or more senior managers should act as system sponsors, with sufficient authority to ensure that applicable resources are available for the project. Users should also be involved and provide support throughout the acquisition to ensure that their requirements are understood and that the resulting system is both accepted and used. User involvement should be sustained from the needs determination phase through final acceptance and implementation. User involvement in the acquisition process will help avoid the development of products that ultimately do not meet agency requirements. The audit steps in this section should be used to assess the potential risks posed by the lack of management or user support. An acquisition that lacks either or both of these elements is at risk of incurring unnecessary cost overruns, not meeting its planned delivery schedule, and not satisfying agency needs.

Page 16

GAO/IMTEC-8.1.4

Assessing

Acquisition

Risks

Chapter 2 Management

aud User Subport

Audit Objectives

1. To ensure that senior managers support and are actively involved throughout the development and implementation of an acquisition. 2. To ensure that users support the acquisition by actively participating in defining procurement requirements, developing the solicitation document, and verifying that the equipment and/or services contracted for meet the agency’ needs. s

Documentation Required

9 Decision papers, memoranda, or other records of senior management oversight and approval of acquisition objectives and plans. 0 Program management directives or other written directives from senior managers stating goals and objectives of the acquisition and delegating authority to carry out the acquisition. Budget exhibits and plans showing that sufficient funding is committed to the acquisition. . Project plans showing the role of users in planning and overseeing the acquisition. Any documentation of the users’role in validating acquisition requirements, alternatives, and the solicitation document. Users’ roles and responsibilities may be detailed in a memorandum of understanding between user organizations and the program office. + Agency policies or guidelines on the structure of steering committees or other oversight bodies, with responsibilities of project members and senior managers delineated.

l

Audit Steps: Top Management support

1. Identify senior management officials responsible for the acquisition. Include senior program officials heading the user organizations, information resource management officials, and members of senior oversight or steering committees. Determine the roles and responsibilities of any groups or committees of senior managers, and the relationships between them.

Page 17

GAOIIMTEC-8.1.4

Assessing

Acquisition

Risks

Chapter 2 Management

and User Support

2. Review the documentation related to the acquisition and determine if the senior managers identified in the acquisition profile: a. Approve the goals and objectives of the acquisition. b. Designate a program sponsor who is responsible and accountable for the acquisition. c. Establish a formal process to keep concerned parties appropriately informed. d. Participate in the specified reviews and decisions.

l

l

l

Determine how promptly management are conducted and approvals given and if the approvals are given at the appropriate levels. Review documentation of such reviews, such as decision memoranda and review-committee minutes. Determine the frequency of reviews and the quality of direction senior managers give to project personnel.

reviews

e. Provide initial funding for the project and establish neat- and long-term funding commitments, periodically informing Congress of acquisition objectives and status. f. Secure any necessary support from key external organizations, such as OMB, GSA, and relevant congressional committees. g. Assign independent officials to ensure that security and internal controls needs are met. 3. On the basis of the above audit steps and on contacts with other project officials, determine whether senior managers fostered good working relations among the sponsor, acquisition manager

Page 18

GAO/IMTEC-8.1.4

Assessing

Acquisition

Risks

Chapter 2 Management

and User Support

(program manager), other top managers, the technical offices, and the contracting community. Specifically, determine whether the managers: a. Coordinate agreement for developing the evaluation and source selection plans and gain acceptance of the evaluation process and criteria. b. Coordinate an agreement between the program staff and technical and contracting offices for managing the contract. c. Obtain the support of important acquisition offkials in the department or agency, such as the official responsible for source selection. 4. Obtain users’and project staffs evaluations of management’ support in the above steps. Find out s how long it took to get approvals from management and the direction management gave to project personnel.

Audit Steps: User Involvement

1. Identify the population of users from project documents and the agency’ organization charts. s Review agency criteria (regulations and procedures) to determine the roles the agency assigns to users. Determine from the program manager and selected users which user organizations are actively involved in the acquisition. Identify significant user groups who are not involved in the acquisition. 2. Determine if users: a. Are involved in periodic reviews, and if so, how frequently they are involved. b. Sign off on needs and requirements statements or otherwise validate the requirements and corresponding solutions when developed. (For

Page 19

GAO/IMTEC-8.1.4

Assessing

Acquisition

Risks

Chapter 2 Management

and User Support

more information see ch. 4, step 3 under Needs Determination, step 4 under Requirements Analysis, and step 1 under Specifications.) c. Validate alternatives against original requirements. (See step 1 of ch. 5 for related audit steps.) d. Approve the alternative selected (such as the choice between off-the-shelf technologies or custom development, centralized or distributed processing, etc).

(See step 1 of ch. 5 for related audit steps.) e. Validate the specifications against the requirements. (For more information see step 4a under Specifications, in ch. 4.) f. Provide acceptance criteria g. Assist in preparing the solicitation document and awarding the contract (for example, user representatives may assist in developing evaluation criteria and participate in the source selection team evaluating alternative proposals).

(See ch. 7, step 4, and ch. 8, step Id, for related information.) h. Participate in postaward activities that may include installation, test, and acceptance of equipment. i. Participate in reviews of contractor-prepared deliverable documents such as design specifications, system analyses, and user and training manuals. j. Participate in government/contractor working groups.

Page 20

GAO/IMTEC-S.l.4

Assessing

Acquisition

Risks

Chapter 2 Management

and User Support

k. Participate in the postaward audit for assessing the degree of success of the acquisition. 3. Determine if users allocate staff time and other resources to the project. 4. Determine if the users participating in the program have a high, moderate, or low turnover rate. 5. Assess the adequacy of users’funding support to the project. a. Obtain funding commitment from users. b. Identify appropriations to be used and their availability to support contract award.

References

. Federal Information Resource Management Regulation (FIRMR), Part 201-2:Designated Senior Officials. – GAO, Information Technology: A Model to Help Managers Decrease Acquisition Risks (G~of1~~~c-8.1.6, August 1990),Phase I, steps 2 and 3; Phase II, step 1; Phase III, step 1.

Page 21

GAO/IMTEC-8.1.4

Assessing

Acquisition

Risks

Chapter 3

Project Staffing

Project management for an acquisition is accomplished primarily by a program manager and staff responsible for carrying out project activities. The program manager should have sufficient authority and an appropriate mix of skills and experience to successfully manage the project. The acquisition project staff should be assigned clear roles and responsibjlities. The team should include members who are skilled in the information technology procurement process, understand the technology, and have experience in managing contracts. The team should also have members knowledgeable about the programs that the acquisition is to support.

Audit Objective

To determine if the acquisition team has the necessary skills and authority to effectively plan and execute the acquisition. A list of key project team members showing their responsibilities, job titles, and experience. The auditor may have to generate this list on the basis of interviews if one is not available. The agency procurement request for a delegation of procurement authority from GSA, showing names and experience of senior project officials (as required by GSA guidance detailed in its FIRMR Bulletin C-5).

Documentation Required

l

l

Audit Steps: Project Management

1. Review the agency’ criteria for acquisition to s determine the responsibility and accountability of the program manager and his/her required qualifications. 2. Review the documentation related to the acquisition to determine how clearly the program manager’ responsibility and accountability are s defined. Determine if the program manager has:

Page 22

GAOAMTEC-8.1.4

Assessing

Acquisition

Risks

Chapter 3 Project Staffing

a. A charter to establish authority, responsibility, and accountability. b. A clearly defined relationship with the program, technical, and procurement offices. c. A clearly defined relationship with the sponsor and users. d. Access to senior agency officials. e. The authority to manage acquisition funds. f. Input to the budgetary process. 3. Review the qualifications of the program manager to determine if the program manager has the appropriate mix of skills and experience. Has the program manager directed other projects of similar size and complexity? Is he or she trained to manage complex acquisitions (GSA’S Trail Boss program may be an example of such training).

4. Review management continuity on the acquisition as shown by the turnover rate of program managers.

Audit Steps: Project Staff

1. Review the agency’ acquisition criteria to s determine both the responsibility and accountability of project personnel as well as their required qualifications. 2. Review the make-up of the project team to ensure that a mix of appropriate acquisition skills are represented. (See ch. 9, steps 2 and 3, for related audit steps.) a. Determine the authority and experience level of the Contracting Officer’ Technical Representative. s

Page 23

GAOflMTEC-8.1.4

Assessing

Acquisition

Risks

Chapter 3 Project Staff&?

b. Identify key project staff and review their experience and qualifications. Determine whether the Contracting Officer is trained and experienced in information technology acquisitions. c. Determine if the project staff is experienced in managing contractors, d. Determine the extent of turnover within the project staff. 3. Determine if the agency trains project staff to maintain their skills and qualifications. 4. Review the reasonablenessof project milestones and schedule with project team members.

References

l

l

l

Circular A-109: Major System Acquisitions. Information Technology: A Model to Help Managers Decrease Acquisition Risks (GAODMTECGH.~, August 1990) Phase I, step 5…i GSA, Overview Guide, pp. 2-3 to 2-7. . GSA, Guide for Contracting Officers’ Technical Representatives, Chapter 2. . FTRMR Bulletin C-5: Delegation of Procurement Authority for a Specific Acquisition.

OMB GAO,

Page 24

GAO/IMTEC-8.1.4

Assessing

Acquisition

Risks

i

Y

Chapter 4

Needs/ Requirements/ Specifications

The purpose of this chapter is to guide the auditor in determining whether the agency has developed an accurate description of its information technology needs. The acquisition should be clearly linked to program needs, to the agency’ overall strategies, and s to governmentwide policies and standards. The agency should expand on its basic description of needs to define specific requirements so providers of information technology can respond with meaningful solutions. In some cases,an agency may use prototypes to help define or validate its requirements. The requirements then form the basis for even more deta.iIedspecifications. In identifying its requirements, an agency should plan for testing the information resources it needs. These plans should cover acceptance, security, and certification requirements. The test plans developed at this point form the basis for later evaluations of contract performance. Failure to clearly and accurately define information technology requirements poses high risks for any agency. For instance, improperly defined requirements may preclude alternatives, restrict competition, increase the risk of cost and schedule overruns, and lead to systems that are inconsistent with an agency’ overall architecture and s incompatibIe with other agency systems, The hardware or software purchased may also be inconsistent with government standards. Designing and implementing a system is also more difficult if input, output, and processing specifications are incomplete or inaccurate. In addition, if security and internal control requirements are not well defined, control over sensitive information or other assets may be lost.

E

Page 26

GAO/IMTEC-8.1.4

Assessing Acquisition

Risks

Y

Chapter 4 Needs! Requirements/ Specifications

Audit Objectives

understood needs or opportunities and that it is consistent with the overall strategy and architectures used by the agency. 2. To ensure that the agency defines its requirements, based on the needs identified earlier and validated by functional users, well enough to support the acquisition of hardware, software, telecommunications, and system development services. These requirements should primarily be expressed in functional terms in accordance with

FIRMRpOky.

3. To ensure that system specifications clearly and accurately summarize the agency’ requirements. s

Documentation Required

:

Needs statement. Requirements analysis or functional requirements document. . System specifications, if prepared. Also, draft specifications with industry comments if draft specifications were released. . Test plan and requirements prepared before contract award, Test requirements may be summarized in a

test

E

and

evaluation

master

plan.

Audit Steps: Needs Determination

1. Review the agency’ stated needs, which may be s documented in a Mission Element Needs Statement, Statement of Operational Need, or System Operational Concept. Determine whether the needs statement clearly and accurately reflects the users’ needs as indicated in the mission statement and strategic objectives of the users’organization, the strategic information plan, or the computer security plan. 2. Check the needs statement for:

Page 26

GAOLfMTEC-8.1.4

Assessing Acquisition

Risks

Chapter 4 Needsl Requirements/ Specifications

–

a. Existing system architecture and functions to be supported (i.e., description, cost, volume of work, projected growth).

b. Justification for changes, such as correcting deficiencies in existing capabilities, complying with new or changed program requirements, or taking advantage of opportunities for increased economy and efficiency. 3. Contact several users as well as project staff who are not users to determine if they generally agree that the needs anaIysis presented in the needs statement adequately addresses actual problems. (See ch. 2 for related questions.)

Audit Steps: Requirements Analysis

1. Review the requirements analysis to determine if it describes the current system. This description should include all the functions of the existing system that any new system will have to perform, The users, functions, work load, operating costs, and components of the current system should also be identified. 2. Confii that the agency has defined its information requirements for the new information resources. These requirements include: a. Information now being received or information that is needed but that is not being received. b. Information to be provided to or obtained from other agencies or the public. c. Sources available from which to obtain the needed information. d. Information relationships,

Page 27

GAO/IMTEC-8.1.4

Assessing

Acquisition

Risks

Chapter 4 Needs/ Requirements/ Specifications

e. The degree of information validation, integrity, accuracy, completeness, and reliability. f. The quantity of information to be processed and types of output expected. g. The timeliness and format of the information. h. The security, accessibility, and privacy requirements. 3. Determine if the agency has defined its functional and support requirements, including: a Present and projected work loads and capacity analysis, including peak load requirements and requirements for future capacity management. b. Privacy and security requirements. c. Contingency requirements for resources whose loss would either prevent or significantly impair the agency from performing its mission or would have an adverse impact on the nation. d. Records management factors relating to integration of electronic records with other agency records, records retention and disposition, and safeguards against unauthorized use or destruction of records. e. Space and environment factors, such as floor loading, heat dissipation, and power supply. f. Federal standards with which the new technology must comply. g. Organizational training needs. h. Interfaces with other systems.

Page 28

GAO/Wl’ EC-8.1.4

Assessing

Acquisition

Risks

Chapter 4 Needs/ Requirements/ SpecMcations

.-

i. User interface requirements. j. Compatibility limitation requirements. k. Capability or performance validation requirements. 4. Determine whether the requirements analysis provides for: a. A methodology for having users validate the requirements analysis for both capability and performance. (See ch. 2 for related questions).

b. Measurable requirements that may be used later to verify system effectiveness. 5. Determine if the requirements are presented in functional or performance terms in consideration of full and open competition. Functional requirements promote full and open competition, while performance requirements may not. 6. With regard to restrictive requirements (which do not lead to full and open competition), determine: a If brand-name-or-equalor specific make and model restrictions are appropriately justified. b. Whether all required justifications are completed and approved for other compatibility-limited requirements. (See ch. 6, step 3, for more information on procurements that do not promote full and open competition).

7. With regard to the process of revising requirements during the acquisition, determine:

Page 29

GAO/IMTEC-8.1.4

Assessing

Acquisition

Risks

Chapter 4 Needs/ Requirement.a/ Specifications

a. Whether a core of basic requirements has been identified in order to maintain project scope. b. If a formal change control process has been established to manage changes when necessitated by a changing environment, c. Who is responsible for reviewing and approving changes to requirements. d. How often requirements have been changed. e. Whether proposed new requirements are validated against mission needs. f. What process is used to analyze the impacts of changes on the other elements of the requirements.

Audit Steps: Specifications

1. Determine whether functional users and/or the program manager confirmed that the specifications accurately reflect the requirements and conform to the approved acquisition strategy discussed in the acquisition strategy module. Did users or the program manager sign off on the system specifications? (See ch. 2 for related questions.) 2. Examine the specifications document for: a. A summary of the functional requirements to be satisfied by the technology. b. Performance evaluation requirements. c. Performance requirements that address information accuracy; data integrity requirements; timing for response, update processing, information transfer, transmission, and throughput; and flexibility to changes in the requirements.

i

Page 30

GAO/IMTEC-8.1.4

Assessing

Acquisition

Risks

Chapter 4 Needs/ Requirements/ Specifications

d. An identification of new types of equipment required (e.g., processors, input/output devices, or information transmission devices).

e. An identification of support and test software. f. A description of interfaces. g. A description of overall security and privacy requirements. h. A description of the operational controls needed. i. A description of the operating characteristics of the user and computer centers where the software will be used. j. A description of the logic flow of the entire system. k. A specification of the functions to be satisfied by the software. 3. Determine how restrictions to full and open competition in the specifications (e.g., equipment characteristics and performance elements) are handled. Ensure that all required justifications are completed and properly approved for such restrictions. (See ch. 6, step 3, for more information on procurements that do not promote full and open competition.)

4.

Determine how changes to the original specifications are handled. a. Establish responsibility for reviewing and approving changes to specifications. Interview users to determine whether or not they actually reviewed and approved changes to specifications. (See ch. 2, step Ze, under User Involvement.)

Page 31.

GAOIIMTEC-8.1.4

Assessing

Acquisition

Risks

Chapter 4 Needsf Requirements/ Specifications

b. Review documentation on change requests. Determine how often specifications are changed, whether new specifications are validated against requirements, and what process is used to identify impacts of changes on other elements of specifications. 5. Determine if feedback (comments and questions) from users and industry is accounted for, considered, and incorporated as appropriate on a continual basis. (See ch. 7, step 6, for related information.)

—

Audit Steps: Test Plans

1. Determine if the agency has developed test plans based on acceptance criteria furnished or validated by the users. 2. Verify that test plans incorporate security and certification requirements. 3. Determine if test plans adequately measure system performance requirements to be specified in the request for proposals (RFP).

(Note: Refer to ch. 10 for more information on test plans.)

References

l l

l

l

l

l

201-20.I: Requirements Analysis. 201-20.303: Standards. Federal Acquisition Regulation (FAR) Part 6: Competition Requirements. FAR Part 10: Specifications, Standards, and Other Purchase Descriptions. GSA, Guide for Requirements Analysis and Analysis of Alternatives, Chapter 2: Requirements Analysis. American National Standards Institute/Institute for Electrical and Electronic Engineers (AN&IEEE) Standard 830: IEEE Guide to Software Requirements Specifications.

FIRMR F’IRMR

E

/

Page 32

GAO/IMTEC-8.1.4

Assessing

Acquisition

Risks

Chapter 4 Needs/ Requirements/ Specifications

4

l

l

Information Technology: A Model to Help Managers Decrease Acquisition Risks (GAO/IMTEC-S.M, August 1990), Phase I, steps 1, 6, 11, 12, 13, 14. Federal Information Processing Standards (FTPS) Publication 641Guidelines for Documentation of Computer Programs and Automated Data Systems for the Initiation Phase. FIPS Publication 101: Guideline for Lifecycle Validation, Verification, and Testing of Computer Software.

GAO,

r

Page 33

GAOAMTEC-8.1.4

Assessing

Acquisition

Risks

Y

Chapter 5

Alternatives

After identifying its requirements, the agency should assessalternatives for cost-effectively meeting those requirements. The approach selected should reflect an understanding of what is available in the commercial market as well as what is available within the government. Approaching the acquisition this way will lessen, but not eliminate, the risk that an agency may select an alternative that does not fully meet user requirements or that is unnecessarily complex and expensive.

Audit Objectives

reasonable alternatives for meeting its needs. 2. To determine if the agency identified the risks, costs, and benefits of each alternative. 3. To verify that the agency selected an alternative balancing expected benefits against costs, time, and risks of failure.

Documentation Required

. Record of alternatives analysis, such as a system decision paper. Economic and risk analyses should accompany or be a part of the decision paper. Market survey research conducted to identify alternatives for meeting user needs and to support cost estimates. Findings and approvals statements to support restrictions on specifications, such as compatibility-limited requirements. Cost/benefit analysis to justify the selection of the alternative selected over other alternatives, in dollar terms or in terms of some other criteria, such as effectiveness.

l l l

Audit Steps

1. Assess the involvement of responsible parties and verify whether:

Page 34

GAOIIMTEC-8.1.4

Assessing Acquisition

Risks

Chapter 6 Alternatives

a. Users agreed with the range of alternatives considered and were involved in validating those alternatives against the original requirements. (For related information on this and the next point see ch. 2, steps 2c and 2d, under User Involvement.) b. Users agreed with the alternative finally selected. c. Appropriate senior management approved the alternative selected. (See ch. 2, step 2, under Top-management Support, for more information.) d. The Contracting Officer or other contracting personnel participated in the alternatives analysis to ensure that a feasible acquisition approach was selected. e. Project staff conducted market surveys to determine how industry can best meet the agency’ s requirements. 2. Ensure that the agency considered, as appropriate, the alternatives included in GAO’S acquisition model and FIRMR 201-20.203-l. 3. Assess how the agency evaluated alternatives by determining whether: a The agency consistently analyzed alternatives using the same criteria for each alternative. b. The alternatives are described in sufficient detail to support time and cost. estimates and cost/benefit analyses. c. The alternatives considered fit within the agency’ information architecture. s

Page 35

GAO/MTEC-8.1.4

Assessing

Acquisition

Risks

Chapter 6 Alternatives

d. The range of alternatives considered was restricted by resource assumptions (staff or funding limitations).

4. Determine whether the agency consistently analyzed the costs and benefits of each alternative. Ensure that the economic analysis includes present values for costs and benefits and is updated periodically. Identify the system life used as a basis for evaluating alternatives and determine whether it appears realistic in light of user needs, expected changes in the technology, expected availability of maintenance and other support, and the time needed to prepare subsequent acquisitions. Verify that the economic analysis includes a sensitivity analysis to identify factors that affect the choice of one alternative over another. The costs and benefits considered for each alternative should include: a. Conversion costs. b. Personnel costs. c. Operation and maintenance costs. d. Nonrecurring but quantifiable benefits in terms of information processing, administration, and support (these may include cost reductions resulting from improved system operations or value enhancement through improved use of resources).

e. Recurring and quantifiable benefits on a monthly and/or quarterly basis over the system life from reductions in such items as salaries, fringe benefits, supplies, utilities, and space occupancy. f. Any nonquantifiable benefits, such as improved service and enhanced organizational image.

Page 36

GAO/MTEC-8.1.4

Assessing

Acquisition

Risks

Chapter 6 Alternatives

5. Determine whether the agency analyzed the following noncost factors for each feasible alternative: a. Obsolescence: strategies for avoiding outdated resources over the system life. A technology upgrade clause is one way to avoid obsolescence by allowing an agency to buy advanced versions of equipment or software when they become available. b. Availability: to what extent the system will be available to users. c. Reliability: how frequently the system requires corrective maintenance. d. Maintainability: the ease with which failed system components can be repaired, taking into account the level of service, personnel support, and supplies needed. e. Expandability: the ease with which the system can be enhanced to meet anticipated growth. f. Flexibility: the extent to which the alternative can accommodate changes in the nature of the work load. g. Security: the ability to prevent unauthorized access and tampering and consideration of national security and emergency preparations. h. Privacy: the extent to which the privacy of personnel-related data can be maintained. i. Affect on personnel: the impact on the level of support personnel needed, including the skills required.

Page 37

GAO/IMTEC-8.1.4

Assessing

Acquisition

Risks

Chapter 5 Alternatives

j. User acceptance: the overall impact on the user community, including the amount of change to user procedures. k. Accountability: the ability of the alternative to allow system activity to be tracked and measured. Review the risk analysis to see if it identifies sensitive data and vulnerabilities. Verify that the magnitude of each vulnerability has been stated. Determine whether or not the risk analysis conforms with the standards identified in FIPS Publications 65 and 73 and OMB Circular A-130. Verify that each alternative is evaluated for financial, technical, and schedule risks. Financial risk refers to the extent to which each alternative is subject to unexpected additional costs. Technical risk indicates the probability that each alternative’ s technical objectives will prove difficult to achieve in whole or in part. Schedule risk is the extent to which each alternative is subject to unexpected schedule delays and slippage in meeting the system’ technical s objectives, regardless of cost. Ensure that the agency has selected the most advantageous and realistic alternative with respect to benefits, costs, and risks (based on steps 4 and 7).

Verify that users and senior managers approve any changes to the planned scope of the project, r

References

Part 201-20.2:Analysis of Alternatives. Information Technology: A Model to Help Managers Decrease Acquisition Risks (GAO~~MTECS.~.~, August 1990), Phase I, steps 7,8,9. . GSA, Guide for Requirements Analysis and Analysis of Alternatives. ChaDter 3: Analvsis of Alternatives.

l

FIRMR GAO,

l

Page 38

GAOKMTEC-8.1.4

Assessing

Acquisition

Risks

Chapter

Alternatives

5

l

OMB Circular A-130: Management of Federal Information Resources. OMB Circular A-109: Major System Acquisitions. OMB Circular A-76: Performance of Commercial Activities. . FIPS Publication 64: Guidelines for Documentation of Computer Programs and Automated Data Systems for the Initiation Phase. FIPS Publication 65: Guidelines for Automatic Data Processing Risk Anaiyw ~____ ~~ FIPS Publication 73: Guidelines for Securitv of Computer Applications. _____.l l l l

Page 39

GAOAMTEC-8.1.4

Assessing

Acquisition

Risks

i

Chapter 6

Acquisition

Planning

Acquisition planning is the process of coordinating and integrating the efforts of personnel responsible for acquisitions. A major objective of acquisition planning is to promote and provide for full and open competition. To ensure that the phmning is accomplished in an effective, economical, and timely manner, the agency should prepare an acquisition plan containing an overall strategy for managing the preaward, acquisition, and postaward phases. An effective acquisition plan is critical to project success. The plan sets out what the agency will do to complete a procurement and how it will do it. The plan also specifies the type of contract that will be awarded, how the agency will select a contractor, cost and schedule goals, milestones, significant risk areas, and contract management controls. Auditors should assess the extent to which the agency’ acquisition planning is realistic and s comprehensive. One part of this assessment should be the review of the Agency Procurement Request (APR) to ascertain whether it is complete and accurately reflects the objectives and scope of the project.

Audit Objective

To verify that the agency has defined an effective strategy and plan for selecting a contractor and managing contract performance. 1 Acquisition plan and related documents as appropriate, such as a plan of action and milestones. . Agency procurement request and other correspondence with GSA. 1. Determine whether the acquisition plan was reviewed and approved in a timely manner by the

Documentation Required

Audit Steps

Page 40

GAOAMTEC-8.1.4

Assessing

Acquisition

Risks

Chapter 6 Acquisition

Planning

officials designated in the agency’ acquisition s regulations. Ensure that the program manager periodically reviews the acquisition plan and updates it when necessary. 2. Review the acquisition plan and determine if it contains the elements required by the FAR Section 7.1 and GAO’S Information Technology: A Model to Help Managers Decrease Acquisition Risks. These elements include acquisition objectives; cost goals; responsible decision-makers; capability or performance characteristics; risks associated with technical matters, scheduling, and costs; plan of action; competition; source selection procedures; contract type and special contract provisions; contract management procedures or organization; budget and funding; information needed to monitor contractor performance; test and evaluation; security and privacy; and acquisition milestones. The plan should also identify the acquisition method, key “go/no-go” points, a formal training plan, and a contingency plan to minimize losses. 3. Determine if the acquisition plan calls for full and open competition. If the plan calls for limiting the acquisition to resources compatible with existing equipment, verify that conversion cost studies have been completed to justify compatibility restrictions. If the plan calls for other than full and open competition, verify that restrictions on competition, such as make and model restrictions or sole source requirements, have been justified and approved by the designated authority. (See ch. 4 step 6, under Requirements and step 3, under Specifications.) 4. Determine whether the agency has planned a “grand design”project or organized the acquisition into modules. Incremental purchasing may limit risks by identifying problems earlier, which allows for easier change or correction.

Page 41

GAO/IMTEC-8.1.4

Assessing

Acquisition

Risks

Chapter 6 Acquisition

Planning

5. Evaluate the project management tools and techniques to satisfy management information requirements for monitoring contractor performance, tracking progress against the acquisition plan, and taking action on cost or schedule slippage. 6. Review the agency’ schedule for developing the s solicitation document and for source selection activities. Assess the reasonableness of the schedule through discussions with procurement officials and reviews of project progress. 7. Identify the dollar limit of the agency’ general s delegation of procurement authority from GSA. Confirm that the agency receives a specific delegation of authority from GSA if the value of the acquisition exceeds the agency’ authority level. Confirm that the s new delegation of procurement authority was received before a solicitation document is issued or a contract awarded. 8. Review the APR if the acquisition exceeds the agency’ delegated procurement authority level. s Determine if the APR identifies the officials responsible for managing the effort, in accordance with the GSA guidance provided in FIRMRBulletin C-5. The APR should include: a. Names and titles of senior project officials, with a description of their roles in the organization. If the acquisition exceeds $25 million, the APK should also describe the project manager’ experience in s previous acquisitions, responsibilities, and scope of authority, and the reporting structure for each official as well as whether each official is assigned full- or part-time to the acquisition. b. The project title and a brief description of the acquisition.

Page 42

GAOAMTEC-8.1.4

Assessing

Acquisition

Risks

i

Chapter 6 Acquisition

Planning

c. Information resources currently in use. d. Resources to be acquired, e. The contracting approach. The approach should include any limitations on competition, the planned dates for release of the solicitation and for contract award, and a strategy for a follow-on implementation if a prototype is to be used. f. Estimated contract life and contract cost. g. Completion dates for key project documents.

References

7: Acquisition Planning. 34: Major System Acquisition. GSA, Overview Guide, p. 4-3. OMB Circular A-109: Major System Acquisitions. . GAO, Information Technology: A Model to Help Managers Decrease Acquisition Risks (GAOITMTEC-8.1.6, August 1990) Phase I, step 10. FIRMR Bulletin C-5: Delegation of Procurement Authority for a Specific Acquisition.

l l l l l

FAR Part FAR Part

Page 43

GAO/IMTEC-8.1.4

Assessing

Acquisition

Risks

Chaoter 7

Solicitation Document

A solicitation document provides information necessary for vendors to propose equipment, software, and services to meet the agency’ s requirements. In most cases, informatlon resources will be purchased by issuing an RFP, which forms the basis for the resulting contract. Less commonly, an agency may acquire information resources using an invitation for bids. An RFP may be preceded by a request for information or request for quotation. be clear and comprehensive and include the elements described in GSA’ guidance on S standard solicitation documents. ’The areas of most interest to auditors include section C: Description/Specifications/Work Statement, section E: Inspection and Acceptance, and section M: Evaluation Factors for Award. Section C describes the tasks to be performed by the contractor and the products to be delivered. Section E sets out government and contractor responsibilities in ensuring that contract deliverables are acceptable to the agency. Section M explains how the agency intends to select a winning contractor by describing the importance of ah factors to be considered in evaluating proposals, In developing the RFP an agency may hold presolicitation or preproposal conferences in order to seek industry views on the planned acquisition and to encourage companies to offer proposals. Once the RFP is developed, it may be released in draft form in order to obtain industry questions and reactions. Auditors should determine what steps the agency has taken to get feedback on its requirements, how the agency has handled comments or questions on a proposed RFP, and whether the agency has acted to ensure that contractor proposals are competitive.

ISee, for example, U.S. General Services Administration, Information Resources Management Service, Overview Guide: Acquisition of Information Resources, (Jan. l@O).

Page 44 GAOAMTEC-8.1.4 Assessing Acquisition Risks

An RFP should

Chapter 7 Solicitation

Document

A proper RFP is a critical element of a successful acquisition because it becomes part of the binding contract once a proposal is made and accepted. If the RFP does not accurately and clearly describe the agency’ requirements, or if the evaluation factors do s not accurately reflect the agency’ priorities, then the s resulting acquisition may not meet user needs. If the RFP is unjustifiably restrictive, favoring one contractor over others, the agency may be unable to benefit from full and open competition. Auditors should become familiar with the standard format for an RFP. The audit team should include persons sufficiently knowledgeable about the information technology being purchased to judge how well the agency has defined its requirements in the RFP. Team members should include or have access to people who can identify areas where the RFP does not define the agency’ needs well enough to protect the s government’ interests. These persons should also s know enough about performance or capability validation techniques to determine whether or not the agency’ requirements are reasonable and effective. s

Audit Objectives

To determine whether or not the solicitation document is complete, clear, and consistent, verify that requirements continue to reflect user needs, and determine if the proposed evaluation process will result in an effective and economical acquisition. Record of presolicitation or preproposal conference. . Solicitation document: RFP or invitation for bid. Draft RFPS and Requests for Information, if any were issued, Report of a solicitation review panel or committee if appropriate. Source selection plan. . Benchmark materials or other capability and performance validation requirements.

l l l

Documentation Required

Page 46

GAOIIMTEC-8.1.4

Assessing

Acquisition

Risks

Chapter 7 Solicitation

Document

l

l

Vendor comments or questions on solicitation document. If a vendor protests the solicitation determine the basis of the protest and how it was resolved. Proposal evaluation guide.

Audit Steps

1. Determine whether the solicitation document

contains:

a. A statement of work or specifications statement that clearly and accurately describes the government’ requirements, including a clear s definition of all deliverables and the conditions of their acceptability. b. A clear definition of government and contractor responsibilities. c. The relative importance of evaluation factors. d. A proposal format requiring that cost and technical elements be separated. e. Reasonable provisions that protect the agency (such as liquidated damagesprovisions) or give incentives to the contractor (such as bonuses for good performance).

Identify option clauses that create uncertainty in work-load projections. 2. Examine the evaluation criteria to ensure: a. They are consistent with the requirements analysis, specifications, and proposal preparation instructions. b. They provide all the factors and significant subfactors to be considered in evaluating offers and the relative importance of different technical or cost factors, in accordance with FAR 15.605.

Page 46

GAWTMTEC-8.1.4

Assessing

Acquisition

Risks

Chapter 7 Solicitation

Document

3. Evaluate the performance evaluation package the agency will use. Determine whether the agency reimburses contractors for participating in benchmarking or other testing, and whether the cost of such performance evaluation efforts constitutes a barrier to competition.

a. If there is a benchmark, has the benchmark been

independently examined by some outside organization? Review the benchmark plan to confirm that demonstration criteria are clearly stated. Determine how the agency selected a representative mix of programs for the benchmark. Determine if the complexity of programs in the benchmark is representative of the projected work load. Determine how the agency validated the benchmark as representative of the agency’ work s load. b. If there is simulation or modeling, determine how the agency selected parameters for the model. Review any concerns or complaints raised by vendors. c. If a compute-off (demonstration of prototypes) is to be used, confirm that the agency has established plans for prototype and follow-on contracts. 4. Interview users and managers, if necessary, to determine whether they concur with the solicitation. Determine whether users or managers identify new or changed requirements not included in the RFP. Find out if there are any factors considered important for selecting an offer that are not included in the evaluation criteria or if other included factors unfairly restrict the competition. (For related information see ch. 2, step 2g, under User Involvement.) 5. Review the agency’ source selection plan to s ensure that it clearly describes the source selection

Page 47

GAOIIMTEC-8.1.4

Assessing

Acquisition

Risks

Chapter 7 Solicitation

Document

organization and activities. Ensure that the source selection plan has been approved by the source selection authority before presolicitation conferences are held or the solicitation document is issued, in accordance with FAR 15.612. 6. Review the industry feedback process and determine if: a. Comments received on the draft solicitation identify any need for clarification, restrictive specifications, or alternative ways of satisfying user needs. (See ch. 4, step 5 under Specifications, for a related question.) b. The agency responded promptly and thoroughly to the comments received. (Judgmental sampling techniques may be required in this section if the number of vendors and vendor comments are substantial.) c. The feedback provided adequate comments on product availability in the market. d. The use of an ombudsman facilitated the process of addressing vendor concerns, disputes, and grievances. 7. Determine who has performed legal reviews of the solicitation document. 8. Determine if any vendor submitted a protest, to whom (the agency, GAO, or GSA’S Board of Contract Appeals), and how it was resolved. Determine the basis of the protest and its resolution.

References

l l

GSA, FAR,

Overview Guide, Chapter 6. Part 15: Contracting by Negotiation, Subpart 15.4: Solicitation and Receipt of Proposals and Quotations;

Page 48

GAOIIMTEC-8.1.4

Assessing

Acquisition

Risks

Chapter 7 Solicitation

Document

Subpart 15.612:Formal Source Selection; Subpart 15.605:Evaluation Factors. . FAR Part 14: Sealed Bidding. . FAR Part 5: Publicizing Contract Actions. FAR Part 34: Major System Acquisition. Information Technology: A Model to Help Managers Decrease Acquisition Risks (GAOfiMTEG8.1.6, August 1990), Phase II, steps 2 through 7. FIPS Publications 75 and 42-l.

l l GAO, l

Page 49

GAO/IlblTEC-8.1.4

Assessing

Acquisition

Risks

Chapter 8

Source Selection

The source selection process is critical to securing the best value for the govemment. All proposals must be evaluated in accordance with the criteria published in the RIT. If the evaluation process does not conform to the RFP the agency runs a greater risk of a successful bid protest by losing vendors. The agency should receive proposals, evaluate the technical and cost merits of different proposals, negotiate with contractors, and award a contract in accordance with a source selection plan developed before release of the RF?. The auditor should be aware of the agency’ s organization and procedures for making a contract award. Agencies may use some or all of the following positions: Contracting Officer (co).

The Contracting Officer publicizes the procurement, amends the RFP if necessary, and conducts all negotiations with offerors. Source Selection Authority (SSA).

The SSA makes the final decision on contract award. The Contracting Officer may be the SSA for some procurements, while in other cases a more senior manager may serve as

SSA.

Source Selection Advisory Council (SSAC).

The SSAC reviews the evaluations of different proposals and makes a recommendation to the SSA on contract award. Source Selection Evaluation Board (SSEB).

The SSEB conducts technical and cost evaluations of vendor proposals.

Audit Objective

To ensure that the source selection process is planned and carried out in order to successfully reach a contract that gives the best value to the government.

Page 60

GAOAMTEC-8.1.4

Assessing

Acquisition

Risks

Chapter 8 Source Selection

Documentation Required

l

.

Source selection plan, including source selection organization. Lessons learned report or other report by the Contracting Officer describing negotiations and selection activities. Contracting Officer’ contract file. s Records of debriefings, if any. Results of benchmarks or other performance and capability validation techniques used. Correspondence between offerors and the agency regarding questions or clarifications and any amendments to the RFP. Proposal evaluation guide. Preaward survey reports.

Audit Steps

1. Examine the evaluation process by reviewing records of the source selection procedures and determine: a If evaluation personnel strictly adhered to and applied the publicized evaluation process and criteria in the solicitation. b. If evaluation factors were applied that were not listed in the solicitation. c. Whether cost and technical evaluations were done separately. d. The role users played in the evaluation process. (See ch. 2, step 2g, for related question.) e. If the process resulted in (1) the establishment of a competitive range and (2) removal of offerors from further consideration, in accordance with FAR 15.609. 2. Determine how many vendors received the solicitation and how many submitted proposals,

Page 61

GAO/IMTEC-8.1.4

Assessing

Acquisition

Risks

Chapter 8 Source Selection

3. Determine if any vendor submitted a protest and, if so, whether the protest was handled by the agency, GAO, or GSA’S Board of Contract Appeals. What was the basis of the protest? How was the protest resolved? 4. Determine whether the Contracting Officer established prenegotiation objectives for cost, profit and fee, and other issues in accordance with FAR 15.807.These objectives should help determine the overall reasonableness of proposed prices, and may be based on an independent government cost estimate and other information, such as a field pricing report on each contractor’ proposal. s 5. Determine if the agency obtained field pricing support in accordance with PAR 15.805-5.Was a preaward audit of the cost proposal obtained and used during negotiations? Were the offeror’ s proposed rates compared with the direct, indirect, overhead, and general and administrative rates recommended by the appropriate contract audit activity? 6. Examine the negotiation process. a. Confkn that discussions with all vendors in the competitive range were held and the proceedings documented. b.Determme from a review of documentation the controls that existed to protect the security of sensitive information. c. Contact responsible officials for their evaluations of security. d. Determine if technical leveling or technical transfusions occurred that would change vendors’ proposals. FAR 15.610describes these and other prohibited actions, such as indicating a cost or

Page 62

GAO/MTEC-8.1.4

Assessing

Acquisition

Risks

Chapter 8 Source Selection

price that offerors must meet to be considered for contract award or informing a vendor of its price standing relative to others seeking the contract. e. Determine if estimated life cycle costs were reconciled with each vendor’ proposal to ensure s that cost estimates appear realistic. 7. Examine how the agency handled best and final offers. a. Determine if the agency made multiple calls for best and final offers, justified in accordance with FAR 15.611. b. Determine if the best and final offers were solicited and evaluated in accordance with the source selection plan. 8. Determine if all debriefings: a. Were scheduled as soon as possible when requested by the vendor. b. Were based on a debriefing plan that addressed and resolved issues likely to cause concern and complaints among the losing vendors. c. Were documented. d. Adequately explained why the losing vendor(s) lost the contract. (Note: In the explanation, the agency cannot make point-by-point comparisons with other proposals, but can point out the government’ evaluation of significantly weak or s deficient elements in the proposal of the vendor being debriefed. Refer to FAR 15.1003for more information.)

Page 63

GAOflMTEC-8.1.4

Assessing

Acquisition

Risks

Chapter8

Source Selection

9. Examine the effort made to identify lessons learned. a. Determine what policies or procedures the agency users have to evaluate acquisition results and communicate “lessons learned” to staff conducting future assignments and to other agencies. Do these include a comparison of the preaward activities to the acquisition plan? b. Review the lessons learned report, if one was completed, to determine how agency officials assessed the contracting process.

References

l

l

l

l

FARPart 15: Contracting by Negotiation: Subpart 15.4: Proposals and Quotations. Subpart 15.6: Source Selection. Subpart 15.8: Price Negotiations. Subpart 15.10: Notifications, Protests, and Mistakes. FIRMR Part 201-39: Acquisition of Federal Information Processing Resources by Contracting. GSA, Overview Guide, Chapter 7: Source Selection. GSA, Guide for Acquiring Commercial Software. * GAO, Information Technology: A Model to Help Managers Decrease Acquisition Risks (GAO/IMTEW 1.6, August 1990), Phase II, steps 8 through 13.

Page 64

GAO/IMTEC-8.1.4

Asaessing

Acquisition

Risks

Chapter 9

Contract Management

Contract management includes the steps required to ensure that the agency receives products and services within established costs and time frames. An agency is required to monitir contractor performance, ensuring that work done conforms to the agency’ s requirements. The agency must also control changes to the contract and accept or reject contract deliverables. Finally, an agency should conduct postimplementation reviews to determine how well acquisition goals were met and whether the information resources acquired should be added to or replaced. The agency’ Contracting Officer and Contracting s Officer’ Representative/Contracting Officer’ s s Technical Representative hold primary responsibility for administering the contract. The Contracting Officer monitors costs as required by the contract type (fixed-price or cost-reimbursable) and makes contract modifications as needed. The program manager helps monitor contractor performance to ensure that user requirements are met by the products or services delivered and that senior officials provide support and oversight. A contract consists of the agency’ RFP, as amended, s and the successful vendor’ proposal. The contract s should specify all deliverables required from the vendor. The Contracting Officer and Contracting Officer’ Representative/Contracting Officer’ s s Technical Representative should ensure that deliverables are received as required. Any status and cost reports required from the contractor should be reviewed and action taken to correct probIems, if necessary. Training, documentation, and maintenance requirements should be fulfilled.

Page 66

GAOfiMTEC-8.1.4

Assessing

Acquisition

Risks

Chapter 9 Contract Management

Audit Objectives

To ensure that the agency: 1. Oversees contractor performance. 2. Ensures that contract requirements continue to accurately reflect user needs. 3. Verifies that products and services delivered meet user needs. 4. Implements configuration management. 5. Modifies the contract only as needed. 6. Enforces contract provisions intended to protect the agency, such as warranties or liquidated damages clauses.

Documentation Required

l

l l

l

l l

Agency regulations or directives specifying requirements for periodic reviews, management oversight, and configuration management. The contract as awarded and with modifications. The agency’ contract management organization and s structure. Current status reports and cost or schedule projections. Current budget reports. The configuration management plan for the project.

Audit Steps

1. Review agency directives to identify the agency’ s requirements for contract oversight. Department of Defense Standard 2167A, for example, requires periodic reviews of contract deliverables, with cost and status reports from the contractor. Defense also has directives governing configuration management activities to ensure that the contractor delivers the equipment or services called for and that no changes

Page 66

GAWlMTEC-6.1.4

Assessing

Acquisition

Risks

Chapter 9 Contract Management

to the contract are made without consideration of their overall impact. 2. Identify the roles of users and senior managers in monitoring the contract, verifying that both users and senior managers are involved in managing the contract and approving any changes. 3. Determine the authority and experience level of the Contracting Officer’ Representative or Contracting s Officer’ Technical Representative. (See ch. 3, step 2, s for a related step.) 4. Evaluate the project staff. a. Identify key project staff and review their experience and qualifications. Is the Contracting Officer trained and experienced in information technology procurement? Does the project include staff experienced in managing contractors? b. Determine how much turnover there has been within the project staff, including the project manager. (See ch. 3, step 2, for a related step.) 5. Assess changes to the agency requirements to ensure that the contract continues to reflect valid user needs. Review configuration management activities to verify that changes to requirements are recorded and controlled and that impacts of changes to contract requirements are identified. 6, kssess changes to the agency’ cost and schedule s estimates. Are variances in cost and schedule projections tracked by the project manager or Contracting Officer’ Technical Representative? Are s cost and schedule estimates changed appropriately? 7. Determine if agency monitoring performance includes: of the contractor’ s

Page 67

GAO/IMTEC-8.1.4

Assessing

Acquisition

Risks

Chapter 9 Contract Management

a. Periodically reviewing both scheduled and completed deliverables and effectively reacting to any delays. Determine how the agency compares contractor progress with the contract work

schedule.

r

b, Periodically reviewing contract reports. Review a sample of status and cost reports to verify that they are regularly submitted by the contractor as required by the contract. Discuss their usefulness with the project staff. c. Assessing the adequacy of the contractor’ s quality assurance process 8. Assess the effectiveness of the agency’ working s relationship with the contractor. a. Verify that the agency has controlled changes to the contract and integrated the change process into the acquisition management structure. Determine the impact of changes on contract cost and schedule. b. Determine if corrections are made, awards are implemented, and damagesassessed, as appropriate. 9. Determine if the agency controls contract modifications by: a. Requiring the contracting office to approve all contract modifications. b. Establishing a review process to ensure that proposed engineering changes are within the scope of the contract. c. Regularly comparing contract expenditures with the delegation of procurement authority to ensure

Page 58

GAO/IMTEC-8.1.4

Assessing

Acquisition

Risks

Chapter Contract

9 Management

that the agency does not exceed its authorized level of total expenditures.

GSA, Overview Guide, Chapter 8: Contract Administration. . GSA, Guide for Contracting Officers’ Technical Representatives, Chapter 2. . GAO, Information Technology: A Model to Help Managers Decrease Acauisition Risks CGAO/IMTE( Auguit 1990), Phase III.=

l

References

Z-8.1.6,

Page 59

GAOAMTEC-8.1.4

Assessing

Acquisition

Risks

Chapter 10

Test and Acceptance

Testing provides the basis for making decisions on whether to accept contract deliverables. For testing to be effective, it must be addressed relatively early in the acquisition so it can be properly included in planning. Test plans provide testing procedures and the evaluation criteria to assess results of the testing. An agency should establish its initial test plans in the presolicitation phase. These plans should show how the agency will verify that the acquired equipment, software, or services meet user needs and satisfy security requirements. After a contract is awarded the agency will need to carry out test and acceptance procedures. The auditor should ensure that test planning is conducted early enough so that test requirements are included in the contract. In assessing the postaward phase, the auditor should ensure that the agency has not accepted equipment or software that does not meet its requirements. The contract should specify conditions for acceptable performance. For example, the contract may require that a computer operate successfully for 30 consecutive days out of a go-day test period. Agency personnel should ensure that the contractor fully meets the conditions for acceptable performance. Internal auditors may be required to verify that the equipment or software pass the specified tests. Assessing the test and acceptance phase may require a high level of technical skill on the part of auditors, such as when an agency has contracted for software development services and must test the quality of delivered software. The auditor should be able to understand the system requirements, development methodologies, and test tools being used.

E

Page 60

GAO/IMTEC-8.1.4

Assessing

Acquisition

Risks

Chapter 10 Test and Acceptance

Audit Objectives

To confirm that the agency has: 1. Fully defined its requirements for testing the technology to be purchased. 2. Effectively carried out test and acceptance procedures to verify that the resources purchased meet the agency’ needs. s

Documentation Required

l

Agency directives giving requirements for cost and status reporting, configuration management, and management oversight. Records of configuration reviews or other progress reports. 9 Trouble reports or other records of deficiencies found by agency personnel. Records of product acceptances. 9 Test plans for inspection and acceptance.

l l

Audit Steps

1. Determine if test plans were developed to determine whether the mandatory: a. Functional requirements were satisfied. b. Security requirements arising from governmental policy, agency mission needs, and specific user needs were satisfied. 2. Determine if the test plans include: a. Types of testing. . Unit testing-e.g., in software, individual code modules are tested by the programmer who wrote them. * Integrated testing-e.g+, in software, aggregate functions formed by groups of modules and intermodule communication links are tested.

Page 61

GAOAMTEC-8.1.4

Assessing

Acquisition

Risks

Chapter 10 Test and Acceptance

l

System testing-examines the operation of the system as an entity in an actual or simulated operating environment. b. The locations for testing. c. A realistic testing schedule. d. The resource requirements.

l

l l

Test equipment needed, including the specific period of use, types, and quantities needed. Sofhvare needed to support the testing. Personnel from both user and acquisition groups with their needed numbers and skills specified. e. Testing materials to be used.

Documentation needed, such as source code and manuals. Software to be tested and its medium. Test inputs and sample outputs. + Test control software and worksheets.

l l l

f. Training in testing to be given, personnel to be trained, and the training staff. 3. Determine whether criteria have been established for certifying that security requirements are met. 4. Determine whether the appropriate user representative has formaily acknowledged the completion of testing and acceptance of the system. If not, determine the reasons and the potential impact. a. Determine if deficiencies discovered in contract deliverables are expeditiously resolved.

Page 62

GAOAMTEC-8.1.4

Assessing

Acquisition

Risks

Chapter 10 Test and Acceptance

b. Determine if any requirements that were not met by the delivered hardware, software, and telecommunications are still pending and why. 5. Interview system operators and users to determine if the system has been successfully integrated into the existing environment.

References

l

l

l

GSA, Overview Guide, chapter 9: Installation and Operation. GAO, Information Technology: A Model to Help Managers Decrease Acquisition Risks (GAOIIMTEC-8.1.6, August 1990), Phase I, step 14; Phase III, steps 4 to 6. FIPS Publication IOl: Guideline for Lifecycle Validation, Verification, and Testing of Computer Software.

Page 63

GAOAMTEC-8.1.4

Assessing

Acquisition

Risks

Appendix I

Acquisition Profile

The acquisition profile, which is a mechanism for documenting key information about an acquisition under review, is used to help auditors plan and conduct assessments of an acquisition. The profile, which is kept available for future reference, includes the

l

l

l

overall characteristics of the acquisition including its objectives shown within the context of the mission(s) or function(s) to be supported, management organization and staffing of the acquisition project, and acquisition schedule and cost estimates.

Overall Characteristics

1. What is the project’ name? s 2. What is the purpose of the information technology acquisition? What missions or functions is the acquisition to support? 3. What type of acquisition is it? 0 system integration, commercial off-the-shelf applications, software conversion, software development, hardware, or other (specify).

l l l

l

l

4. Are there any related in-house efforts? 5. With what systems will this acquisition interface? 6. Is the acquisition based on

l

full and open competition, . competition restricted by compatibility limitations,

Page 64

GAO/IMTEC-8.1.4

Aasessing

Acquisition

Risks

Appendix I Acquisition Profile

. limited competition (such as make or model limited),

or

. noncompetitive, sole source.

7. Is the contract based on

. fixed prices (specify type of fixed-price contract), or . cost reimbursement (specify type of

cost-reimbursement contract).

8. What project management tools or techniques are in use to oversee the project (such as Gantt charts or critical path method/Program Evaluation and Review Technique).

9. What, if any, development tools and techniques (such as Computer-Aided Software Engineering-CASE) are in use? 10. If a primary contractor has been selected for the acquisition, list the contractor name, address, telephone number, and point of contact. 11. Identify key subcontractors, with company names, addresses, telephone numbers, and points of contact.

Management Organization and Staffing

How is the management of E* organized and structured? the acquisition project 13. What are the title, name, and phone numbers of the acquisition sponsor, acquisition manager, contracting officer, user representatives, and senior officials who approve acquisitions. 14. What are the responsibilities and duties of the acquisition manager? What is his/her authority (for

Page 65

GAO/IMTEC-8.1.4

Assessing

Acquisition

Risks

Appendix I Acquisition Profile

example, planning, budget+staffing, or progress reporting)? 15. Has an acquisition steering committee been established? What are the committee’ responsibilities s and duties? 16. Who staffs the acquisition team and what are the team members’qualifications?

Acquisition Schedule and Cost Estimates

17. What was the most recent milestone or key decision point approved? 18. What are the actual or estimated dates for the following: original start/completion dates, current start/completion dates, requirements analysis completed and approved, solicitation document completed for release, * contract awarded, 0 initial operation, test and acceptance, and full operation.

l l l l l l

19. Is the project on schedule or has there been a slippage? 20. Has the agency completed an estimate of life cycle cost? If so, what are the original and current estimates? 21. Identify the budget authority and outlays by year for the project. 22. Identify the net benefits or cost savings projected by the costienefit analysis used to justify a chosen alternative, if the acquisition has progressed to this point.

Page 66

GAO/IMTECd.l.4

Assessing

Acquisition

Risks

Appendix I Acquisition Profile

23. Has the project received scheduled funds and resources or have there been shortfalls? (Explain any variances and their effects.)

Page 67

GAO/‘ lMTEC-8.1.4

Assessing

Acquisition

Risks

Appendix II

Management Metrics

Purpose

This appendixdescribes tools and techniques for measuring the status of acquisitions involving significant software development. It describes techniques, known as software metrics, for quantitatively measuring how closely a project conforms to development plans and assessing whether an acquisition is at risk of delay or cost increases. These techniques require considerable information about the system being developed and can only be used after system design, The metrics described here will generally be used after contract award, in order to assess how well the agency is managing the system development process. Software metrics, which use mathematical models to measure elements of the development process, are intended to help organizations better understand and manage the relationships between resource decisions, development schedules, and the cost of software projects. By using software metric tools an auditor can independently evaluate software development projects by analyzing project budgets, requirements, schedules, and resources, and then confirm or question cost, schedule, and resource estimates. Different metrics may be useful for an audit, depending on the objectives and status of an acquisition. Using cost models to estimate the cost and length of time necessary to develop a new software system, for example, is appropriate only after requirements have been defined, a system design has been developed, and the size of the new system has been estimated. The models described here require estimates of either the lines of code or number of function points that the new system will include. Cost models project the time and cost to develop a system on the basis of estimates of the system’ size and other pertinent factors. They may be s used before a solicitation for software development is issued in order to assess the reasonableness of the

Page 68

GAOfIMTEC-8.1.4

Assessing

Acquisition

Risks

Appendix II Management

Metrics

agency’ schedule (if a system design has been s prepared), or after a contract has been awarded and software development has begun. The other metrics described in this appendix require that system development work be underway. A comparison of the actual number of functions successfully tested to the number planned in the acquisition schedule, for example, requires that system testing be underway+

Cost Models

Cost models are tools that estimate the effort needed to develop software, based on assumed relationships between the size of a system and the effort needed to design, code, and test the software. These models can s help the auditor assess whether the acquisition’ estimated cost and schedule are reasonable. Each model uses cost drivers, which are parameters such as the level of experience of the programmers, the reliability requirements of the programs, and the complexity of the project, along with the estimated project size, to derive overall cost and schedule estimates for the acquisition. When a system involving software is being developed, one or more cost models may be useful, A model will be more reliable if it takes into account the agency’ historical experience s in developing systems. Cost models are available both commercially and from a few agencies within the Department of Defense. Most of the tools were developed initially for use with Defense department projects, but can also be used with non-Defense systems. Many are based on industry-recognized models such as the Constructive Cost Model (COCOMO), PRICE, Putnam, and Jensen. The commercial tools range in cost from about $500 to well over $20,000.Government-developed or government-modified tools are available free of charge with a nominal charge for upgraded copies of the software.

Page 69

GAO/IMTEC-8.1.4

Assessing

Acquisition

Risks

Appendix II Management

Metrics

Because many different packages are available, and because more than one can be used, auditors should determine what models, if any, are used in their agencies. Typically cost models are derived using data gathered from years of experience with a wide range of software projects. Therefore, accuracy of predictions may improve as further experience is accumulated in an agency. Auditors should use cost models to look for discrepancies with the cost and schedule estimates established for an acquisition. By varying the estimated values for cost drivers, the auditor may also be able to perform a sensitivity analysis illustrating where project estimates are most susceptible to change. However, cost models have significant limitations in accuracy unless their underlying assumptions of system size and cost drivers are carefully chosen and reflect the agency’ previous s experience in system development. It is a matter of auditor judgment to decide how discrepant project estimates and estimates provided by cost models should be to raise concerns about risks of cost and schedule overruns. In making this judgment, the auditor should take into account the uncertainty of estimates and assumptions made in using a cost model. Auditors should use a cost model to provide general estimates and not precise figures. Therefore, in applying software metrics to audit work, care must be taken to follow generally accepted government auditing standards when drawing conclusions based on the results of software metrics. Specifically, all findings should be qualified by the recognition that these tools are limited by the accuracy of the estimated system size and other project data provided for the model, the historical data from which the model was developed, and the fact that all estimates are projections of an inherently uncertain future.

Page 70

GAWWlTEC-8.1.4

Assessing

Acquisition

Risks

Appendix II Management

Metrics

Procedures for minimizing the effect of these limitations require, for example, proper technical advice, qualifying language in audit reports, and a review of data and assumptions by agency officials. Auditors should also ensure that the model used is consistent with the software development methodology of the project under consideration. A system developed through rapid prototyping, for example, should be evaluated with a model that takes prototyping into account.

Other Indicators

The following indicators are intended to help auditors assess how effectively an agency is managing a system development contract. These indicators measure differences between what an agency planned for its contractor to accomplish by certain points in the systems development life cycle and the actual results. In most cases this comparison is most easily done graphically. Using more than one indicator can give a broader picture of a project’ status. Auditors s should use indicators appropriate to the project under review and for which data are available. This indicator involves tracking the number of open and closed problems reported by a contractor as a system is developed. A problem could be any anomaly discovered during design, coding, testing, or implementation. Problems are distinguished from failures of code, which represent defects discovered during operation. Contracts should specify how problems are to be identified and reported, By reviewing these reports and noting how quickly problems are resolved, auditors can obtain an understanding of how well the contractor is performing.

Problem Reports

Page 71

GAO/IMTEC-8.1.4

Assessing

Acquisition

Risks

Appendix II Management

Metrics

The project manager or Contracting Officer’ s Technical Representative may be the best source for problem reports, Because an agency may choose to prioritize problems in order of their impact, the reports should distinguish between priority levels of problems. Figure II. 1 shows one way to present this kind of analysis. It shows the number of new problems reported and the number of problems closed in order to show a trend over time and to show any backlog that may be developing. The auditor may also choose to report total problems reported and resolved or break them out by level of priority. Figure 11.2 shows the length of time that problems have remained open in order to demonstrate how quickly software problems are resolved once found. In this example, three levels of priority are distinguished for reported problems.

Page 72

GAO/IMTEC-8.1.4

Assessing

Acquisition

Risks

Appendix II Management

Metrics

Figure 11.1: Problem

Number of Problems 55 50 45 40 35 30 25 20 15 10 .’ 5 0 1 Project reviews -.’

Reports

in Software

Project

Y

2

3

Problems reprlad Problems resolved

Page 73

GAOLIMTEC-8.1.4

Assessing

Acquisition

Risks

Appendix II Management

Metrics

Priority Levels and Number of Months Unresolved

220

Number of Unresolved Problems

200

oto12

13 to 24

25 lo 36

Over 36

Number of months problems have remained unresolved 0 ~ Level3 Level2 Level 1

Software Volatility

Software volatility is also measured graphically. Figure II.3 shows changes in the total number of approved system requirements. Cumulative changes are also tracked, including additions, deletions, and modifications to requirements. Steady increases in the number of requirements and changes to requirements may indicate that the project is at risk for delays and cost overruns.

Page 74

GAWIMTEC-8.1.4

Assessing

Acquisition

Risks

Appendix II Management

Metrics

Figure 11.3: Software

Requiremenls 2100 1950 1800 1650

Requirements

Changes

1200 1050 900 750 600 450 300 300 150 150 0 C.1 2 3 4 -.—.-.I-.-..-.-._-.-.-.—–_-..-.e. 4.—–

-+.cL-.+.cL–

.–

Requirements and changes at major decision points -Total requirements Cumulative changes

Development Progress

This indicator involves the comparison of actual and expected progress in the design, coding, and integrating of system units. Units can be measured in terms of computer software units or computer software configuration items. If the project team does not complete its design or programming and testing activities as planned, this indicator can show schedule delays before major milestones are reached. The progress indicator can show all elements of design, coding, testing, and integrating, or it may treat them as separate indicators. Figure 11.4shows how

Page 76

GAOIIMTEC-8.1.4

Assessing

Acquisition

Risks

Appendix II Management

Metrics

planned and actual progress in the design and coding of software units can be displayed.

b

Figure 11.4: Development

Number of Units 150 140

Progress

,’ 120 130 120 110 ..-.-.*’ ,.**

l’

.’

.-

I /’

/’ l,-’ /’ ,*’ ’ /’ ///’ // 90’

20 10 0 1 Major decision points during project

—— Units designed – planned Units designed – actual Units coded – planned Units coded – actual

1’

*/’

2

3

Software Size

This indicator records changes in the expected magnitude of the software development effort. The size of the system, measured in source lines of code as established by the system design, may change as the system is coded. Changes in size can be expressed

Page 76

GACNIMTEC-8.1.4

kssessing

Acquisition

Risks

b

Appendix II Management

Metrics

as total lines of code or can be broken out into new, modified, and reused lines of code. Changes in the estimated system size may indicate that the project was overestimated or underestimated in size and complexity. These changes may also indicate that requirements are changing and may be related to the software volatility issue described earlier. Increasing estimates of software size should alert the auditor that the project’ schedule and expected cost may be s underestimated. Changesin the expected system size may necessitate reestimates of the development cost, using the cost models described earlier in this appendix. Figure IL5 shows how the estimates of software size can be tracked over time.

Page 77

GACMMTEC-8.1.4

Assessing

Acquisition

Risks

Appendix II Management

Metrics

Figure 11.5: Software

120 110 1DO 90 a0

Size Estimates

Lines of Code In Thousands

.- ,.-. ..- .–

.

0 1 Major decision points during project . ..-I* –I 2 3 4

Total New Modified

Reused

Page 78

GAOAMTEC-8.1.4

Assessing

Acquisition

Risks

Appendix II Management

Metrics

Personnel Stability

Tracking the total number of people assigned to a system development effort compared to planned staffing levels provides another indicator of potential problems. The auditor can examine projected versus actual levels of total personnel or of key, experienced personnel. Understaffmg may result in schedule slippage. Adding personnel late in a project can actually cause further slippage. Other metrics may be chosen as needed. Auditors could, for example, compare expected to actuaI usages of hardware resources in order to identify emerging computer capacity problems. Another approach would involve tracking changes to the estimated completion date for a system. The metrics described here offer suggestions to be tailored for use as appropriate to particular projects.

Page 79

GAOAMTEC-8.1.4

Assessing

Acquisition

Risks

Appendix III

Reference Materials

Each chapter of this guide lists applicable reference materials including federal regulations and guidance published by GSA, OMB, and other agencies. When using these references, summarized below, auditors should ensure that the most current version available is used.

Federal Regulations

Applicable regulations include both general rules for regulations procurement, the FAR, and the FIRMR written by GSA specifically for acquiring federal information processing resources. GSA issues regulations under its Brooks Act authority. In addition to the FIRMR, GSA issues bulletins that provide guidance on a wide range of federal information processing acquisition issues: Federal Acquisition Regulation (FAR) Federal Information Resource Management Regulation (FTRMR)

Y

GSA

l

l

OMB Circulars

l

0 A-109: Major System Acquisitions A-130: Management of Federal Information Resources-under proposed revision A-76: Performance of Commercial Activities A-11: Preparation and Submission of Budget Estimates

l l

GSA Acquisition Guide Series

l

l

l

l

l

l

Overview Guide Guide for Requirements Analysis and Analysis of Alternatives Guide for Acquiring Maintenance Services Guide for Acquiring Commercial Software Guide for Contracting Officer’ Technical s Representatives Guide for Acquiring Systems Integration Services

Page 80

GAOKMTEC-8.1.4

Assessing

Acquisition

Risks

Appendix III Reference Materials

National Institute of Science and Technology (NIST)-Formerly the National Bureau of Standards

l

Federal Information Processing Standard (FIPS) Publications. FIPS Publications describe federal standards for hardware, software, and the management of information technologies. FIPS Publications relevant to the acquisition of information technology include the following: FIPS PUB 11-3-Guideline: American National Dictionary for Information Processing Systems, Feb. 1, 1991. FIPS PUB 3&Guidelines for Documentation of Computer Programs and Automated Data Systems, Feb. 15, 1976. FIPS PUB 42-l-Guidelines for Benchmarking ADP Systems in the Competitive Procurement Environment, May 15,1977. FIPS PUB 46-l-Data Encryption Standard, Jan. 22, 1988. FIPS PUB 64-Guidelines for Documentation of Computer Programs and Automated Data Systems for the Initiation Phase, Aug. 1, 1979. FIPS PUB 65-Guideline for Automatic Data Processing Risk Analysis, Aug. 1,1979. FIPS PUB 73-Guidelines for Security of Computer Applications, June 30, 1980. FIPS PUB 75-Guideline on Constructing Benchmarks for ADP System Acquisitions, Sept. 18,198O. FIPS PUB 76–Guideline for Planning and Using a Data Dictionary System, Aug. 20, 1980.

Page 81

GAO/IMTEC-8.1.4

Assessing

Acquisition

Risks

Appendix III Reference Materials

FIPS PUB 87-Guidelines for ADP Contingency Planning. Mar. 27.1981. FIPS PUB 88-Guideline on Integrity Assurance and Control in Database Administration, Aug. 14,1981. FIPS PUB 96–Guideline for Developing and Implementing A Charging System for Data Processing Services, Dec. 6,1982. FIPS PUB 99-Guideline: A Framework for the Evaluation and Comparison of Software Development Tools, Mar. 31,1983. FIPS PUB lOl-Guideline for Lifecycle Validation, Verification, and Testing of Computer Software, June 6,1983. FIPS PUB 102-Guidelines for Computer Security Certification and Accreditation. SeDt.27, 1983. FIPS PUB lo&—Guideline on So&are Maintenance, June 15, 1984. FIPS PUB 124-Guideline on Functional Specifications for Database Management Systems, Sept. 30, 1986. FIPS PUB 127-l-Database LanguageSQL, Feb. 2, 1990. FIPS PUB 132-Guideline for Software Verification and Validation Plans, Nov. 19, 1987. FIPS PUB 146-l-Government Open Systems Interconnection Profile, Apr. 3, 1991. FIPS PUB 151-1-POSIX: Portable Operating System Interface for Computer Environments, Mar. 28, 1990.

Page 82

GAO/IMTECd.P.I

Assesshg

Acquisition

Risks

Y

Appendix III Reference Materials

FIPS PUB 158-The User Interface Component of the Applications Portability Profile, May 29, 1990.

l

Special publications. Publications relevant to information technology acquisition include the following: 500-087-Management Guide for Software Documentation, January 1982. 500-OQO-Guideto Contracting for Software Conversion Services, May 1982. 500-105-Guide to Software Conversion Management. 500-106-Guide on Software Maintenance. 500-lOQ-Overview of Computer Security Certification and Accreditation. 500-120-Security of Personal Computer Systems: A Management Guide. 500-133-Technology Assessment: Methods for Measuring the Level of Computer Security. 500-134Guide on Selecting ADP Backup Processing Alternatives. 50@147–Guidance on Requirements Analysis for Office Automation Systems (Update).

500-148-Application Software Prototyping and Fourth Generation Languages. 500-153-Guide to Auditing for Controls and Security: A Svstem Develonment Life Cvcle Annroach. 500-154–Guide to Distributed Database Management,

Page 83

GAO/IMTEC-8.1.4

Assessing

Acquisition

Risks

Appendix III Reference Materials

500-155-Management Guide to Software Reuse. 500-161-Software Configuration Management: An Overview. 500-165-Software Verification and Validation: Its Roie in Computer Assurance and Its Relationship with Software Product Management Standards. 500-172~Computer Security Training Guidelines. 500-173-Guide to Data Administration. 500-174-Guide for Selecting Automated Risk Analysis Tools. 500-175-Management of Networks Based on Open Systems Interconnection (OSI)Standards: Functional Requirements and Analysis. 500-180-Guide to Software Acceptance. 500-183~Stable Implementation Agreements for Open System Interconnection Protocols. 500-l&l–Functional Benchmarks for Fourth Generation Languages. 500-187-Application Portability Profile, The U.S. Government’ Open System Environment Profile s OSE/l Version 1.0. 500-192-Government Open Systems Interconnection Profile Users’Guide, Version 2. 500-193-Software Reengineering: A Case Study and Lessons Learned. 8004-Computer Security Considerations in Federal Procurements: A Guide for Procurement Initiators,

Page 84

GAOiIMTEC-8.1.4

Assessing

Acquisition

Risks

Appendix III Reference Materials

Contracting Officers, and Computer Security Officials, March 1992.

GAO Audit Guidance

l

l

Information Technology: A Model to Help Managers Decrease Acquisition Risks (GAoLlM~~c4.1.6, August 1990).

Government Auditing Stantids, 1988Revision.

Page 86

GAOILMTEC-8.1.4

Assessing

Acquisition

Risks

Appendix IV

Major Contributors

to This Audit Guide

Information Management and Technology Division, Washington, D.C.

Mark E. Heatwole, Assistant Director David R. Turner, Evaluator-in-Charge Bernard R. Anderson, Senior Evaluator Peter C. Wade, Senior Evaluator Trinh N. Hoang, Computer Programmer David Chao, Technical Reviewer Shane D. Hartzler, Editor

Page 86

GAO/DMTEC-8.1.4

Assessing

Acquisition

Risks

Glossary

Acquisition

The obtaining, by contract with appropriated funds, of supplies or services (including construction) by and for the use of the federal government through purchase or lease, whether the supplies or services are already in existence or must be created, developed, demonstrated, and evaluated. Acquisition begins at the point when agency needs are established and includes the description of requirements to satisfy agency needs, solicitation and selection of sources, award of contracts, contract financing, contract performance, contract administration, and those technical and management functions directly related to the process of fulfilling agency needs by contract. The process by which the efforts of all personnel responsible for an acquisition are coordinated and integrated through a comprehensive plan for fulfiiling the agency need in a timely manner and at a reasonable cost. It includes developing the overall strategy for managing the acquisition. A request by a federal agency for GSA to acquire information processing resources or for GSA to delegate the authority to acquire these resources. The overall structure of a computer system including hardware and software. The degree to which a system or component is operational and accessible when required for use, often expressed as a probability. (1) A specification or product that has been formally reviewed and agreed upon that thereafter serves as

Acquisition Planning

Agency Procurement Request (APR) Architecture Availability

Baseline

Page 87

GAO/IMTEC-8.1.4

Assessing

Acquisition

Risks

Glossary

the basis for further development and that can be changed only through formal change control procedures. (2) A document or set of such documents formally designated and fixed at a specific time during the life cycle of a configuration item. Note: Baselines, plus approved changes from those baselines, constitute the current configuration identification. (3) Any agreement or result designated and fixed at a given time from which changes require justification and approval.

data designed to evaluate the performance of computer hardware and software in a given configuration.

Best and Final Offer Capability Validation

A final opportunity for offerors in the competitive range to revise proposals. The technical verification of the ability of a proposed system configuration, replacement component, or the features or functions of its software, to satisfy functional requirements. The intent is to ensure that the proposed resources can provide the required functions. Performance requirements are not implied or measured in the validation. Examples of capability validation include: a. operational capability demonstrations of the functions of the hardware, operating system, or support software; b. verification of conformance with information processing standards, c. expert examination of the technical literature supplied with the offer;

Page 88

GAO/MTEC-8.1.4

Assessing

Acquisition

Risks

Glossary

d. contacts with other users of the proposed information processing resource; and e. vendor certification of conformance with the functional requirements.

Certification

(1) A written guarantee that a system or component complies with its specified requirements and is acceptable for operational use. For example, a written authorization that a computer system is secure and is permitted to operate in a defined environment. (2) A formal demonstration that a system or component complies with its specified requirements and is acceptable for operational use. (3) The process of confirming that a system or component complies with its specified requirements and is acceptable for operational use. See configuration control. A daily publication that lists the govenment’ s procurement invitations, contract awards, subcontracting leads, sales, surplus property, and foreign business opportunities. (GWIRMS, A Guide for Acquiring Commercial Software, Jan. 1991,p. A-l.) (1) The ability of two or more systems or components to perform their required functions while sharing the same hardware or software environment. (2) The ability of two or more systems or components to exchange information. A statement of requirements expressed in terms that require items to be compatible with existing information processing resources.

Change Control Commerce Business Daily

Compatibility

CompatibilityLimited Requirement

Page 89

GAWIMTEC-8.1.4

Assessing Acquisition

Risks

Glossary

Competitive Range Configuration

The group of offerors selected, after technical and cost evaluation, to whom award of a contract is a reasonable possibility. (1) The arrangement of a computer system or network as defined by the nature, number, and chief characteristics of its functional units. (2) The physical and logical elements of an information processing system, the manner in which they are organized and connected, or both. The term may refer to a hardware configuration or a software configuration. An element of configuration management, consisting of the evaluation, coordination, approval or disapproval, and implementation of changes to configuration items after formal establishment of their configuration identification. An aggregation of hardware and/or software that is designated for configuration management and treated as a single entity in the configuration management process. The continuous control of changes made to a system’ s hardware, software, and documentation throughout the development and operational life of the system. A person with the authority to enter into, administer, and/or terminate contracts and make related determinations and findings.

Configuration Control

Configuration Item

Configuration Management Contracting Officer (CO)

Page 90

GAOIIMTEC8.1.4

Assessing

Acquisition

Risks

Glossary

–

Contracting Officer’ s Technical Representative (COTR) Conversion

An individual to whom the co delegates certain contract responsibilities, usually related to technical acceptance issues.

Modification of existing software to enable it to operate with similar functional capability in a different environment; for example, converting a program from FORTRAN to Ada or converting a program that runs on one computer to run on another. A contract in which the government reimburses the contractor for expenses so long as the contractor provides its “best effort” to complete the work called for. Authority to acquire information processing resources up to a specified limit, issued by GSA in response to an agency procurement request. The regulation that codifies uniform acquisition policies and procedures for executive agencies governmentwide. The regulation that sets forth uniform policies and procedures for acquiring information processing resources; used in conjunction with the FAR.

Cost-

Reimbursement Contract Delegation of Procurement Authority (DPA) Federal Acquisition Regulation (FAR) Federal Information Resources Management Regulation (FIRMR)

Page 91

GAODMTEC-8.1.4

Assessing

Acquisition

Risks

Glossary

Fixed-Price Contract Functional Requirement Independent Verification and Validation (IV&V) Invitation for Bid

(IFW

A contract that provides for a firm price or in appropriate cases, a firm price with fees or other adjustments. A requirement that specifies a function that a system or system component must be able to perform. Verification and validation performed by an organization that is technically, managerially, and financially independent of the development organization. (See verification and validation, defined separately as follows.) The solicitation document used when contracting by

Interoperability

The ability of information technology resources to provide services to and accept services from other resources and to use the services so exchanged to enable them to operate effectively together. Compensation to the government for a contractor’ s failure to perform in a timely manner. The ease with which maintenance of a functional unit can be performed in accordance with prescribed requirements. Attempts to ascertain whether other qualified sources capable of satisfying the government’ requirement s exist. This testing of the marketplace may range from written or telephone contacts with knowledgeable federal and non-federal experts regarding similar or duplicate requirements and the results of any market

Page 92 GAOAMTEC-8.1.4 Assessing Acquisition Risks

Liquidated Damages Maintainability

Market Survey

Glossary

test recently undertaken, to the more formal sources-sought announcements in pertinent publications (e.g., technical/scientific journals, the Commerce Business Daily), or solicitations for information or planning purposes.

Performance Validation

The technical verification of the ability of a proposed system configuration or replacement component to handle agency-specific work-load volumes (present and expected) within agency-determined performance time constraints.

A

Program Manager

The key management official who represents the program office in formulating resource requirements and-managing presolicitation activities. In some organizations the program manager or another management official is designated as the acquisition manager for a specific acquisition. A written objection by an interested party to (1) a solicitation for a proposed contract, (2) a proposed award, or (3) the award of a contract. A preliminary type, form, or instance of a system that serves as a model for later stages or for the final, complete version of the system. A hardware and software development technique in which a preliminary version of part or ah of the hardware or software is developed to permit user feedback, determine feasibility, or investigate timing or other issues in support of the development process.

Protest

Prototype

Prototyping

Page 93

GAOflMTEC-8.1.4

Assessing

Acquisition

Risks

Glosmry

Rapid Prototyping

A type of prototyping in which emphasis is placed on developing prototypes early in the development process to permit early feedback and analysis in support of the development process. The ability of a system or component to perform its required functions under stated conditions for a stated period of time. An announcement in the Commerce Business Daily or other publication requesting industry comment on draft specifications for resources. An announcement in the Commerce Business Daily or other publication requesting information from industry about a planned acquisition, and, in some cases, corporate capability information. The solicitation document used in negotiated procurements to communicate government requirements and to solicit proposals. An official government request for bids/proposals generally publicized in the Commerce Business Daily in accordance with federal regulations. The government official in charge of selecting the source for an acquisition. Most often the title is used when the selection process is formal and the official is other than the Contracting Officer.

Reliability

Request for Comment Request for Information

Request for Proposals (RFP) Solicitation

Source Selection Authority (SSA)

Page 94

GAO/MTEC-8.1.4

Assessing

Acquisition

Risks

Glossary

Source Selection Evaluation Board (SSEB) Source Selection Plan

A board composed of technical, contract, information resources managers, and other government personnel whose primary function is to evaluate proposals received in response to an RFP. A document that describes the entire process for awarding a contract-proposal evduation criteria, evaluation methodology, evaluator’ responsibilities, s and final selection procedures. A description of the government’ requirement for s resources which is so restrictive that only a particular manufacturer’ products will satisfy the government’ s s needs. A written description of the technical requirements for resources stated in an IFBor RFP. A technical description of resources, prepared for inclusion in a solicitation document. A projection of the time period that begins with the installation of the resource and ends when the agency’ need for that resource has terminated. s Helping an offeror to bring its proposal up to the level of other proposals through successive rounds of discussion, such as by pointing out weaknesses resulting from the offeror’ lack of diligence, s competence, or inventiveness in preparing the proposal. Government disclosure of technical information pertaining to a proposal that results in improvement of a competing proposal.

Specific Make and Model Specification Specification Statement of Work System Life

Y

Y

Technical Leveling

Technical Transfusions

Page 95

GAO/IMTEC-8.1.4

Assessing

Acquisition

Risks

Glossary

Test Plan Uniform Contract Format c User

A plan prepared by the government that details the specific tests and procedures to be followed. The format required by the solicitation.

FAR

for preparation of a

uses the services of an information processing system. (2) Any person or any thing that may issue or receive commands and messagesto or from the information system. The process of evaluating a system or component during or at the end of the development process to determine whether it satisfies specified requirements. (1) The process of evaluating a system or component to determine whether the products of a given development phase satisfy the conditions imposed at the start of that phase. (2) Formal proof of program correctness. The mix of tasks typically run on a given computer system. Major characteristics include input/output requirements, amount and kinds of computation, and computer resources required.

Validation

Verification

Work Load

(610767)

Page 96

GAO/IMTEC-8.1.4

Assessing

Acquisition

Risks

A software version of the audit guide is also available on a high density 3.5 inch disk suitable for use on MS-DOS computers. To receive your free software, print your name and mailing address below and mail this card. Please send me the software version of the audit guide for information technology acquisitions. Name: Title: Organization: Address:

Telephone:

GAOIIMTEC-8.1.4SW

b

Y

Y

U.S. Gcueti Accountixqt Once P.0, Box aa15 Gaithersburg, MD 20877 or visit: 700 4th St, NW (caner of 4th t G Sts. NW) U.S. General Accannting Ofllce Washington, DC Orders may aleo be placed by calling (202) 612-6000 or by using fax number

(201) 2684066.

Introduction Management and User Support Project Staffing Needs/Requirements/Specifications Alternatives Acquisition Planning Solicitation Document Source Selection Contract Management Test and Acceptance Appendixes