Software Risk Management Through Independent Verification and Validation

Validation

John R. Callahan1Tong C. ZhouRalph Wood

Department of Statistics & Computer ScienceConcurrent Engineering Research Center

West Virginia University

Abstract

Software project managers need tools to estimateand track project goals in a continuous fashionbefore, during, and after development of asystem. In addition, they need an ability tocompare the current project status with pastproject profiles to validate managementintuition, identify problems, and then directappropriate resources to the sources of problems.This paper describes a measurement-basedapproach to calculating the risk inherent inmeeting project goals that leverages past projectmetrics and existing estimation and trackingmodels. We introduce the IV&VGoal/Questions/Metrics model, explain its use inthe software development life cycle, anddescribe our attempts to validate the modelthrough the reverse engineering of existingprojects.

project, risk can be reduced if errors and otherdiscrepancies are found as early as possible inthe software development life cycle. Manystudies have shown that undetected errors in aproject will increase the likelihood of failures inlater life cycle phases when the cost to fix themincreases by orders of magnitude.

IV&V efforts are highly effective in early lifecycle phases [1] if they can successfully predictthe likelihood of problems based on an analysisof the current state of a project. It is difficult,however, to make such predictions withprovable accuracy and show correlation betweendevelopment activities and problems that arisein later life cycle phases. Formal softwaredevelopment models can provide some insightbased on quantified analysis of past softwaredevelopment efforts [2,3]. While such formalmodels are imperfect guides to future efforts,they are far more likely to predict problems thaninformal methods.

We have developed an approach called theIV&V Goal/Question/Metric method (IGQM)that allows IV&V managers to monitor the levelof risk in a software development project. UsingIGQM, managers can use past projects as\"yardsticks\" against which to measure presentprojects. They can also assess the potentialimpact of their decisions about resourceallocations, schedules, costs, and tradeoffs

1Introduction

Managers of large, complex software projectsoften rely on independent contractors to verifyand validate (V&V) the computer softwareproduced by a separate development contractor.An independent V&V (IV&V) contractor helpsidentify, manage, and reduce the potential riskof failures to meet intended requirements in asoftware project at all phases of development.While some level of risk will always remain in a work is supported by DARPA Grant MDA 972-91-J-102 under the DARPA Initiative forConcurrent Engineering (DICE) program, NASA Grant NAG 5-2129 and NASA Cooperative AgreementNCCW-0040 under the NASA Independent Software Verification and Validation (IV&V) Facility,Fairmont, WV.

1This

goal 1goal 2goal 3

question 1question 2question 3

metric 1metric 2metric 3

Figure 1: The Goal/Question/Metric (GQM) model

during execution of the development effort. TheIGQM method provides continuous reporting ofthe status of a project in terms of what areas areat risk of failure. The method represents aformal interface between the IV&V contractor,the software development contractor, and thecustomer. It summarizes the analysis workperformed by the IV&V contractor in terms ofwhat project goals are at risk of failure andallows managers to make informed decisionsabout why problems are occurring.

This paper discusses the IGQM model and itsuse in an automated support environment [4].Unlike existing metric-based models, ourapproach does not emphasize any specific set ofmetrics or functions for assessing risk. Themodel allows for use of other assessmentmodels. The IGQM model is used to collect andsummarize the metrics and relate them directlyto project goals. Although our approach toIV&V relies on metrics from past projects asbaselines, the model can be \"primed\" withinformal estimates or external project databases.Results from pilot projects are then used asfeedback to provide continuous improvement tothe model itself in order to improve ourpredictive accuracy.

The IGQM model can incorporate severalexisting software estimation and trackingmethods. These include the COCOMO method[2] and Software Equation [3] for estimatingcost, size, and effort. We describe our attemptsto validate our approach by using these methodsto reverse engineering past projects to determineif identifying risk sources early in the life cyclecould have helped prevent later problems.The IGQM model is embedded in an automatedsupport environment for software IV&V [4] thatallows continuous analysis of a project's status.IV&V is viewed as a complementary process to

the software development process [5] and it isresponsible for continuous assessment of thedevelopment process. As a softwaredevelopment process progresses, events aretriggered in the IV&V process. The IV&V teammust analyze changes in the developmentprocess and publish its findings to the customerin the form of an IV&V report. This report isgenerated using the IGQM method by a tool thatis integrated into a CASE environment. Thereporting tool collects and summarizes analysisresults (i.e., metrics) from other IV&V CASEtools in an incremental fashion. When a changeoccurs in the development process, the projectmeasurements and risk are updatedincrementally like values and formulas in aspreadsheet. The risk impact of each change isassessed immediately relative to the projectgoals. This paper does not discuss the details ofthe automated support environment, but focuseson the IGQM model around which theenvironment is organized.

2Approach

The IGQM approach to software IV&V focuseson the quantification, identification,management, and reduction of risk in softwaredevelopment projects based on objective metricstaken during the software development lifecycle. Metrics include process measures (i.e.,whether or not a particular procedure beenperformed at this phase) as well as artifactmeasures (i.e., quantitative measurements ofdocuments, code, tests, and other products).The IGQM tool formally defines the impact ofsuch measures on the failure or success inmeeting project goals.

Our approach is based on the Goal-Question-Metric (GQM) model [6] augmented with riskanalysis [7]. The GQM model depicted inFigure 1 allows managers to explicitly describe a

ProjectG1G2G3G4ConfidencesQ1Q21.000.361.000.361.000.361.000.36

Certainty

Q30.770.770.770.77

Q40.000.000.000.00

0.450.781.000.04

Uncertainty0.550.220.000.96

Importance0.800.300.900.10

Risk0.4400.0660.000.096

Table 1: Computing goal risks based on question confidence probabilities

QuestionsQ1Q2Q3Q4M134343434M211111111M388888888M499999999confidences1.000.360.770.00

Table 2: Computing question confidence probabilities based on project metrics

project in terms of a set of goals so that thedevelopment team has more precise knowledgeabout the intent of the customer. A project mustcompletely satisfy a set of goals to beimplemented successfully. Goals includerequirements but are much broader and caninclude ambiguous statements like \"the systemmust be highly reliable.'' Each goal is satisfiedby answering a set of related questions. Thequestions define the features needed to satisfy aparticular goal. Questions are answered true orfalse, but can be parameterized with limits, e.g.,\"does the system have a 10,000 hour mean timebetween failures?\" Each question is answeredbased on a set of quantifiable project metrics. Ametric might be \"lines of code\" or \"estimatedmean time between failures\" or any otherdiscrete value. The GQM approach is used as adialogue between customers and developmentorganizations for agreeing on the details of aproject. In this fashion, it should be clear to thedeveloper exactly what is expected of the finalproduct and the criteria for its acceptance.We have augmented the GQM model tocompute the risk of failure in a project to satisfythe intended goals. The risk of failing to satisfythe goal is defined as the uncertainty of reachingthat goal multiplied by the importance of thatgoal. Table 1 shows a list of goals, theirimportances, certainty, uncertainty, and risks foran example project. The goals G1,...,G4 mightbe

••••Low cost

Medium effortUse of prototypingHigh reliability

The questions related to each goal in the IGQMmodel will determine exactly what is meant byeach goal. The risk values associated with eachgoal should change during the softwaredevelopment life cycle. If we keep track of therisk at each step in the development process, wecan identify high-risk goals and ensure that theoverall risk is non-increasing over time, i.e.,while risk may increase at any step, the overallrisk trend is decreasing.

2.1 Risk associated with each goal

To calculate the risk associated with each goal,the importance of the goal is specified explicitlyby the manager, but its certainty is computedfrom answers to related questions in the GQMmodel. For each goal-questions group, weemploy a set of certainty functions G at eachstep of the development life cycle defined as

gi,tp=Gi,tp(Qx,tq,Qy,tr,Κ)

where gi,tp∈0Κ1 for the i goal at theprocess step tp and each Qx,tn is theprobabilistic confidence answering question xas true at process step tn. Thus, the certainty ofsatisfying each goal changes at each step in thesoftware development process. The certainty

th

Estimated probability of not exceeding size

250Q1200150SLOC x 1000

10050011025 (-1stddev)

50expval

84 (+1stddev)

9099 Q2Q3Q4Figure 2: Confidence functions for estimated SLOC in early life cycle phases

functions may be based on the baselines of pastprojects or on the results of simulated models.In either case, the results of certainty functionsare added to the baseline for use in futureprojects.

2.2 Confidence in answers to questionsHere is where existing estimation and trackingmethods fit into the IGQM model. Eachquestion can be answered true with acharacteristic probability called its confidence.A false answer has a confidence value of zero.The confidence of answering a question isdetermined by a unique function based oncollected project metrics. For each question-metrics group, we employ a set of confidencefunctions Q defined as

questions and in turn decreases the certainty ofsatisfying a goal.

3Predictive Functions

The characteristic certainty and confidencefunctions associated with goals and questionscan be based on many existing methods thathave evolved from experiences on large numbersof actual projects. The IGQM model simplytries to relate the calculation of risk to theanalysis these methods provide in order to helpidentify areas of a project that need attentionand allow managers to trace problems to theirsources.

For example, several methods exist forestimating the eventual number of source linesof code (SLOC) in a project [3]. Early estimatesof SLOC will be very inaccurate, but we canassess the probability of the correctness of ourestimate. Consider the goal of \"Small Program\"in which the related questions are:

1.2.3.4.

Are there less than 100 requirements?Are there less than 50 function points?Are there less than 50 modules?Are there less than 10,000 SLOC?

qx,tp=Qx,tp(Ma,tq,se,Mb,tr,sf,Κ)

where qx,tp∈0Κ1 for question x at theprocess step tp where each Ma,tz,se is a metrica at step tz provided by source se. Table 2shows a question and its related metrics fromwhich a confidence function is defined. Allmetric values are the same relative to eachquestion, but the confidence functions aredefined uniquely for each question and processstep. Metrics that are unknown at process stepscan still be used because the lack of knowledgecontributes to the risk calculation. Unknownmeasure decrease confidence in answering

In this example, question 4 might given themost weight in ultimately determining theacceptance criteria. However, in the early stagesof a project, we can only answer question 1 witha large degree of confidence, but the answer tothis question will not have a large impact ofincreasing the certainty of meeting the goal

according to our weighting. Figure 2 shows arisk profile for the different questions at thisstage of development. The relatively higherslopes of the other questions illustrates a greaterdegree of uncertainty.

The weighting of each question confidencemeasure in determining goal certainty willchange during the lifetime of the project, i.e.,the slopes will decrease and different measureswill play larger roles. Eventually, confidencefunctions may get better with more experienceand a broader database of actual projects. Thiswill also decrease the uncertainty.

Estimating functions are highly domaindependent. This is why it is important for eachorganization to institute measurement programsto improve the effectiveness of their predictions.The IGQM model can be primed with hand-picked estimate or those from external projects,but these initial estimates will be highlyinaccurate. Only with time can an organizationbuild confidence in their predictive models. Ofcourse, changes in personnel and the need totackle new projects can invalidate previousexperience, but

In the case of SLOC, we can determine theprobability of the eventual number of lines ofcode exceeding our estimate. Likewise, manymethods exist for cost, size, error, and effortestimation. Whereas many of these techniquesare only used early in a project to construct aproposal or plan, our approach allows managersto track actual measurements and compare themwith estimates. As a project evolves, a managercan gain greater confidence in the estimates asthey change dynamically based on actualperformance.

4Discussion

In Table 1, we can see that goal G1 is the onlygoal with significant associated risk. If theconfidence and certainty functions are based onmethods that leverage past project data, the riskassociated with G1 at this process step might saysomething like \"44% of the projects at this stepwith a similar goal-questions profile failed tosuccessfully satisfy this goal at time of delivery.''The interpretation is based on the characteristic

confidence and certainty functions related toeach goal and question respectively.

Creating the certainty and confidence functionsis not easy. They are based on profiles of pastprojects, contain coefficients that are specific toeach environment or project, and must beprimed initially with estimates or data fromexternal projects. By mapping our approach tocurrent software development and V&Vpractices, we \"reverse\" engineered theseestimates from informal measures on pastprojects. Even though some information wasnot available on these projects, they wereadequate enough to provide working estimates.In one case we wanted to verify the intuition ofV&V personnel who noted problems with thedelivery schedule of project milestones. In theirexpert opinion, the schedule was too short. Ourmodel, based on existing methods such asCOCOMO, confirmed that the intuition wascorrect.

In the next sections, we show how traditionalV&V activities can be mapped to our model.Specifically, we relate process management andtesting to see how they contribute to projectmeasurements and at what stages of the lifecycle. Based on this mapping, we can assess therelative effectiveness of these traditionalapproaches in controlling software projects.Process management, for example, ensures thatthe software development team follows allprocess steps (e.g., DOD 2167A) and follows upon all discrepancy reports and anomalies. Itmonitors that the proper artifacts (i.e.,documents and code) are produced on time andin their proper order. Testing, on the otherhand, is usually associated with code levelvalidation of the end-product system in asimulated environment. While it is widelybelieved that both of these approaches helpreduce project risk, they have serious limitationsin many projects, especially in large, complexsystems with volatile requirements. It ispossible that expensive and catastrophic errorsmay go undetected using traditional approaches.We show that according to our reverseengineered projects, late life cycle testing mayfind some errors but it is often too late to fixthem. This fact shows up not as an increase inrisk towards the end of the project, but as aninability of existing techniques to keep the risktrend non-increasing and within nominal limits.

Process management and testing alone areinadequate means to manage risk in large,complex projects.4.1 Process

First, an IV&V team can check to make sure thesoftware development process is followed at allsteps. The goal of this task is to reduce risk byensuring that a process is followed that increasesthe probability of success. The reasoning behindthis task is informal: if a past task wassuccessful using a process then each step mustbe repeated to guarantee success in otherprojects.

We can cast current software developmentpractices into the IGQM risk model by askingspecific questions about process stepsaccomplished. The metrics are Boolean valuesthat help answer questions at each stepregarding whether or not a procedure has beenperformed. In this fashion, the IGQM approachsubsumes these current \"checklist'' methods andprovides a metrics-based environment forformally validating whether or not genericassumptions about process effectiveness are true.Process tracking by the IV&V team is necessarybut insufficient to ensure risk reduction in thedevelopment project.4.2 Testing

Software testing has been a major focus ofIV&V efforts, but testing is expensive and hassevere limitations. Traditional testing cannotfind many problems or finds problems too late inthe software development life cycle where theyare too costly to fix. In the IGQM model, theresults of tests can be viewed as metrics (e.g.,pass-fail). From this metrics-based perspective,early analysis of requirements and design canalso be viewed as \"tests'' but the test results areviewed with less confidence than concrete testsat later stages of the development life cycle. Inaddition, the tests can be directly associated withrequirements or project goals in the IGQMmodel. In this case, the existence of a test isimportant for tracability. We used this approachto model traditional testing in the IGQM modeland showed that late testing reduces risk, butthat the risk trend is already too high at laterphases for testing to have any significant effect.

Traditional testing does not permit earlydetection of problems and it is often impossibleto exercise a system with a battery of tests thatcompletely characterize the operationalenvironment. If major problems occur, it isoften too late and expensive to fix them. As aresult, the software might experience traumaticfailure, the project is scrapped, must be redone,or the customer is left dissatisfied with apartially functional system. If the customer hadaccess to effective, predictive estimate earlier inthe development process, expectations might bemore realistic and the intentions better definedwith the development team.

5Implementation

We have implemented the IGQM model in atool for use by IV&V practitioners. The tool,called ADMIT (A Distributed MetricsIntegration Tool), is implemented in Tk/Tclunder the X Windows system and the UNIXoperating system. The tool primarily consists ofthree list boxes of goals, questions, and metricsand a multi-graph widget that shows thecumulative risk for the project, per goal,question confidences, and metric values.Metrics come from many sources in thedistributed environment. Some come fromshared files and databases (e.g., the NetworkFile System (NFS), Oracle). When the fileschange, the tool read the file, updates themeasure, and recalculates the associatedconfidence and certainty functions. Metric mayalso be source directly from CASE tools usingremote procedure call in which the ADMIT toolacts as the server. We are continuing to evolveour implementation as we integrate othersources of metrics and techniques for collectingthem.

6Summary

Our approach depends on an intense metricscollection and archival capability to providehigh levels of confidence in IV&V predictions.It also depends on the continuous evolution ofthe predictive certainty and confidencefunctions. While our approach does noteliminate risk from a project, it does formalizethe risk identification, management, andreduction. It makes risk management the

explicit objective of the IV&V process in orderto deliver effective results to the customer.Moreover, the confidence of predictions can beincreased as our baseline grows with eachproject. For well-defined application domains,we expect this approach will have most valuebased on extrapolating experiences with theIGQM model in practice.

While a statistical risk model of IV&V does notguarantee success, it represents a significantimprovement over existing practices that deliverdubious value to the IV&V customer and mayunknowingly harm software development effortswith needless paperwork. During the course ofour research, we continue to investigate (1)effective process models; (2) specific and usefulmetrics and their correlation within the process;and (3) continuous improvement of certaintyand confidence functions associated with theprocess.

7References

[1]The Cost-Effectiveness of IndependentSoftware Verification and Validation, NASA JetPropulsion Laboratory, 1985.

[2]Boehm, B., Software EngineeringEconomics, Prentice-Hall, Inc., EnglewoodCliffs, NJ, 1981.

[3]Putnam, L. and W. Myers, Measuresfor Excellence: Reliable Software on Time,within Budget, Prentice Hall, Inc., EnglewoodCliffs, NJ, 1992.

[4]Karinthi, R., S. Kankanahalli, S.Reddy, C. Cascaval, W. Jackson, S.Venkatraman, H. Zheng, CollaborativeEnvironment for Independent Verification andValidation of Software, In Proceedings of theThird IEEE Workshop on EnablingTechnologies: Infrastructure for CollaborativeEnterprises, April 17-19, 1994, Morgantown,WV.

[5]Lewis, R., Independent Verificationand Validation: A Life Cycle EngineeringProcess for Quality Software, John Wiley &Sons, New York, 1992.

[6]Basili, V., Applying theGoal/Question/Metric Paradigm in theExperience Factory, in Software QualityAssurance: A Worldwide Perspective, Chapman& Hall Publishers, 1994.

[7]Cardenas-Garcia, S. and M. Zelkowitz,A Management Tool for Evaluation of SoftwareDesigns, IEEE Transactions onSoftware Engineering, Volume 17, Number 9,September 1991.

8Biographies

John R. Callahan is an assistant professor ofcomputer science in the Department of Statisticsand Computer Science at West VirginiaUniversity and a research faculty member at theConcurrent Engineering Research Center(CERC) in Morgantown, West Virginia. Hereceived his Ph.D. from the University ofMaryland College Park in 1993 in softwareengineering and is currently working onresearch in independent verification andvalidation of computer software. He has workedfor Xerox Corporation of Tyson's Corner,Virginia and Palo Alto, California as well asNASA Goddard Space Flight Center and theDepartment of Defense. Dr. Callahan serves asthe NASA Research Associate at the NASAIV&V Facility in Fairmont, West Virginia. TheFairmont facility houses IV&V contract workfor the Mission To Planet Earth and SpaceStation projects.

Tong C. Zhou is a graduate student in theDepartment of Statistics and Computer Scienceat West Virginia University and a researcher atthe Concurrent Engineering Research Center(CERC) in Morgantown, West Virginia. Herinterests include formal methods, risk analysis,and automated test generation.

Ralph Wood is a visiting scientist at theConcurrent Engineering Research Center(CERC) in Morgantown, West Virginia. Dr.Wood is a former senior engineering scientistfor General Electric Research and Development.His interests include risk management andcost/schedule estimation.