Managing Organizational Risk Knowledge

Abstract: Risk planning requires an organization global view, as it is strongly centered in the experience and knowledge acquired in former projects. The larger the experience of the project manager the better will be his ability in identifying risks, estimating their occurrence likelihood and impact, and defining the mitigation and contingency plans. However, project manager risk knowledge cannot stay in an individual level, but it must be made available to the organization. This paper describes an approach to risk planning in software projects based on the organizational risk knowledge reuse. A risk management process focused on the capture and utilization of organizational knowledge together with a support case tool make part of this approach. An experimental study of the relations between risk-causing facts and risks of software projects was accomplished and its results used to define such a tool.

Keywords: Risks Management, Knowledge Management, Risks Planning.

Categories: D.2.0, D.2.9

1 Introduction

It is becoming more difficult to manage project risks due to the size and complexity of current software products [Garvey et al., 1997]. Project managers can inadvertently repeat past mistakes simply because they do not know the mitigation actions which have been successfully applied or even because they do not value risks caused by certain project restrictions and characteristics. Inefficient risk knowledge management contributes to maximize this problem. One of the reasons is the fact that project information concerning risk management is in individuals' minds or distributed among various documents, making its reuse difficult.

In a project, risks are those conditions or events whose occurrence is not certain, but whether they occur may adversely affect the project. Three aspects associated to a risk can be identified [Pfleeger et al., 2001]: (i) the loss associated with the event; (ii) the likelihood that the event will occur; and (iii) the degree to which event consequences may be changed. Risks can be generic or project-specific. Generic risks are those common to all software projects, such as requirements misunderstanding, key personnel losing, or insufficient time for testing. Project specific risks are threats that result from the particular vulnerabilities of the given project and organization.

For example, a vendor might promise to deliver some necessary network software at a particular date, but there is some risk that the software will not be available on time.

The lack of documentation on the success or failure of past experiences is one of the reasons for inefficient risk management utilization or non-utilization in software development organizations. Besides risk management knowledge, the past experiences analysis is fundamental to help project managers plan and control risks. Statz [Statz, 1999] discusses the importance of learning from the experience obtained in organization's former projects, and proposes the lessons learned documentation in software projects. Similarly, Markkula [Markkula, 1999] considers the project experiences the most important source of knowledge in software engineering, and describes the need of identifying and sharing the acquired experience.

Risk planning can be enriched by using knowledge and experience acquired by the various managers while working on the several organization projects. In order to do that, it is necessary that risk knowledge be captured and stored throughout projects development, so that it will make its future utilization possible. However, without an infrastructure that can make organization risk knowledge available it is very difficult to manage all the acquired risk knowledge and experience.

Garvey et al. [Garvey et al., 1997] define information architecture for risk management based on reusing experience acquired in previous organization projects. Williams et al. [Williams et al., 1997] show the results of experiences in using risk management by describing lessons learned in the SEI risk program. The program works with a database of software risks that supports the risk management activities. Kontio and Basili [Kontio and Basili, 1996] describe how data and experience acquired in measurements can be captured for risk management purposes. They also describe the Riskit Method and its integration with the Experience Factory framework.

The risk identification strategy proposed in this paper has some innovation when compared with related works. The approach explores the relationship between risks and particular software projects risk-causing facts, such as those related to technology, planning, personnel and external factors. A risk-causing fact is defined as any condition/restriction found or predicted by the project manager at the software project initial stage (the planning stage). Besides, a risk-causing fact is a potential cause of risks to the project. Examples of risk-causing facts are "development team inexperienced in software engineering", "project using innovative technology", "lack of software development process". The proposed tool uses knowledge acquired in similar projects and it is integrated into a software development environment. It supports the activity of risk planning by making available the organizational knowledge that might be useful to the project manager during the activities of Identifying risks, Analyzing risks, Prioritizing risks, Planning risk management and Monitoring risks.

This paper is organized as follow: Next section presents our proposed risk management process based on the ISO 10006 [ISO, 1997], the technical report 16326 of the ISO/IEC [ISO/IEC, 1999], and the IEEE standard for risk management [IEEE, 2001]. Such process emphasizes the use of organizational risk knowledge throughout several activities. Section 3 describes the experimental study of the relations between risk-causing facts and of software projects risks, describing their objective and exemplifying the results. Section 4 discusses the approach we propose to risk planning, presenting the Riskplan tool, which supports the several activities of the risk management process. Finally, Section 5 makes some final remarks and considerations.

2 Risk Management Process

Risk management is a continuous process that aims to systematically treat risks throughout the software project lifecycle. Its objective is to minimize the impact of potentially negative events, following and managing the risks that might threaten the success of a project. The ISO 10006 standard [ISO, 1997] recommends the use of experience and historical data from former projects throughout all process activities. Therefore, the process described in this document seeks the reuse of organizational knowledge and experience, one of the benefits emphasized by Knowledge Management.

Figure 1 shows the risk management process that is divided into the processes Evaluate Risks and Control Risks. The process Evaluate Risks, on its turn, is divided into the activities Identify risks, Analyze risks and Prioritize risks. The process Control Risks is subdivided into the activities Plan risk management, Integrate risk plan and Monitor risks. Each of the activities is then divided into sub-activities, where the utilization of organizational risk knowledge is recommended. The proposal is to support the execution of this process, making it possible to capture and utilize risk knowledge throughout the several activities performed.

The activity Identify risks is divided into the following sub activities:

• Identify risks originated from project decisions - During this sub activity, the project manager has to analyze the decisions taken throughout the project planning, trying to find risk-causing facts. All constraints, suppositions e decisions must be reviewed in order to identify potential problems caused by the decisions taken.
• Identify risks from similar projects - During this sub activity, the project manager must analyze the problems occurred and the lessons learned in previous similar projects, trying to find possible risks to the on-going project.

Page 672

• Identify other risks - Based on his/her previous experience with risk management, project management and software engineering, the project manager must analyze the risks identified up to the moment and insert any other he/she might judge pertinent.

The product generated in the risk identification activity is the project risk list. However, these risks still need to be analyzed and prioritized. In the Analyze Risks activity, the project manager conducts the sub activities ahead for each identified risk:

• Estimate risk occurrence probability - The qualitative estimate must be always performed and the risk can be categorized as low probability risk, medium probability risk or high probability risk. The quantitative estimate is performed when possible. The analysis of the risks that occurred in previous similar projects can help the execution of this sub activity. If a particular risk occurred in various similar projects this might mean that this is a risk with high probability of occurrence in the on-going project.
• Identify risks causes and consequences - In this sub activity it must be identified the risk occurrence causes and the consequences the risk might bring to the development process and to the software product. Again, the analysis of risk causes and consequences in previous similar projects can help the execution of this sub activity.
• Estimate the risk occurrence impact - The impact can be categorized as very high, high, medium or low. If necessary, a quantitative estimate must also be conducted. If the risk occurred in similar projects, it must be analyzed the caused impact in order to help the estimate.
  The product generated in the risk analysis activity is the project risk list updated with causes, consequences, occurrence probability and impact of each identified risk. After that, the Prioritize Risks activity is conducted and it is divided into the following sub activities:
• Compute the exposure to the risk value - For every identified risk, the exposure to the risk is calculated. In case it is not possible to perform quantitative estimates, the exposure to the risk must be computed as follows: High probability and high impact risks receive a greater value of exposure to the risk than the low probability and low impact risks. The exposure to the risk in this case receives the value of the probability given to the risk concerning the estimated probability and impact.
• List the risks in descendent order of exposure to the risk - The purpose of this sub activity is to separate the most important risks from the least important risks, making it possible the conduction of the next sub activity.
• Define the set of risks that will be managed throughout the project - Based on the exposure to the risk value associated to each risk, it is defined a cut line. Only the most important risks are managed throughout the project.

The product generated with the conduction of this activity is the project risk list prioritized. It can then be started the conduction of the Control Risks process, which first activity is the project risk management planning. The Plan risk management activity is divided into the following sub activities:

• Define risk treatment strategy to each risk - It must be defined the strategy that will be used to treat every project risk. It is helpful to review the strategies adopted in the previous similar projects, verify the strategies efficiency and then define how to treat the risk in the on-going project. For every risk, the project manager must choose one of the following strategies: avoid the risk, transfer the risk or assume the risk.
• Establish the mitigation and contingency plans to the assumed risks - A mitigation plan is developed to define a set of necessary actions to minimize the risk consequences. The mitigation actions must reduce the risk occurrence probability, the risk associated impact or both. It is also important to develop a contingency plan that specifies the actions to be taken in case some specific risk occurres. It is helpful to review the risk planning performed in previous similar projects. If the risk occurred in any similar project, it must be analyzed the efficiency of the defined mitigation and contingency plans. The reuse of mitigation plans can make it easier the conduction of this sub activity
• Analyze the cost-benefit relation of the mitigation actions - It is important to realize that the mitigation actions bring extra cost to the project. It is then necessary to evaluate if the benefits brought by the risk mitigation steps are greater than the cost associated to their implementation. The project manager must conduct a cost-benefit analysis for every assumed risk. If the exposure to the risk is lower than the risk mitigation cost, then it is better not to mitigate it but instead monitor it throughout the project.

The product generated in this activity is the Project Risks Plan that describes the identified risks, their priority, causes, consequences, occurrence probability, impact, mitigation plan and contingency plan. The next process activity is the integration of the risks plan into the project plan. In this activity, the following sub activities are performed:

• Update the development process plan - In this sub activity, the project manager must change the development process plan incorporating the steps related to the risks mitigation plan.
• Update the project plan - In this sub activity, the project manager changes the project plan, incorporating the costs associated to the risks mitigation plans, the eventual resources added to the project as well as all other changes that he/she considers to be caused by the performed risks planning.

With the completion of this activity, the project plan is updated and is completely integrated to the developed risks plan. The next process activity is monitoring the project risks and it is divided as follows:

• Review the project risks plan -This review must be conducted at the project established milestones, trying to observe if any risk has become a problem, if it is about to become a problem, if the mitigation plans are efficient, if any risk is no longer a problem or if new risks came up. Based upon the new perception of each risk, some previous performed activities can be reviewed. The following actions are related to monitoring a risk: Mitigation strategy change, in case it becomes inefficient; definition of a mitigation plan to any risk that becomes important; execution of a pre-planned contingency plan; change of the risk status to concluded when it no longer exists; inclusion of new risks in the project risks plan.

Page 674

• Perform the necessary changes in the project risks plan - After the review conducted at the project milestones, the risks plan must be updated to reflect the changes performed in the continuously monitored risks.

Risks monitoring make it possible to adequate the risks plan to the new perception of the project risks. Next, we will discuss how the organizational risk knowledge can support the execution of each one of the activities and how it is captured throughout the process.

2.1 Reusing Organizational Risk Knowledge

The information related to the predicted or occurred risks in former organization projects (such as their causes, consequences, their treatment and success of the mitigation and contingency actions) may help the project manager identify new project risks, estimate their probability and impact and plan risk management. Besides, lessons learned regarding risk management former projects might contribute to the enrichment of the project risk planning.

The risks list that have occurred in former similar projects, i.e., risks that became problems, may help new project risks identification, avoiding the situation in which potential problems could be forgotten or not valued by the manager. Suppose, for instance, that a manager is identifying risks in a project with certain characteristics and ignores the risks caused by some conditions or restrictions. The verification of the risks that occurred in similar projects may remind him of potential problems to be faced in the project he/she is currently working on.

In the same way, risk data from similar projects may help project manager during the estimates of risks occurrence likelihood and impact. Suppose, for instance, that the manager is estimating the likelihood of occurrence, causes and consequences and the impact associated to "high turnover" risk. It is useful to analyze how this risk has behaved in similar organization projects, verifying if it has become a problem, its consequences and impact it has caused.

Knowing how many similar projects in which a risk was predicted and how many projects in which a risk has occurred may also help the manager in the estimating of risk likelihood occurrence. For instance, if risk x occurred in 10 out of a total of 13 similar projects, the manager could conclude that the risk has a high probability of occurrence in the project. Inversely, if risk x occurred in zero out of a total of 13 similar projects, the manager can conclude that the risk has a low probability of occurrence.

During the planning of mitigation and contingency actions, similar projects risk data are also very useful. It is important to analyze the strategy of risk treatment adopted in similar projects and verify the efficiency of mitigation and contingency actions that were planned. This way, the manager learns from the facts of former projects, avoiding the recurrence of problems and reusing actions which were previously successful in the risk mitigation or contingency. The lessons learned in former projects concerning risk management also contribute a lot to the management of risks in new projects. Throughout the process of risk management, it is recommended to register the ideas and lessons learned by the project manager.

The risk data verification of similar projects requires the recovery of similar projects to a specific project followed by the recovery of the risks occurred in these projects. The approach of risk planning here described uses a search for similar projects based on the direct participation of the user.

After characterizing the software project, the project manager chooses which criteria will be used in the search, according to its specific objective. Examples of search objectives are to find similar projects aiming to identify risks pertaining to the Personnel category; identify risks pertaining to the Requirements category. Besides choosing the criteria to be used, the manager also decides if the similar projects are recovered having as basis all or at least one of the chosen criteria. Examples of criteria used in project characterization are: Industry in which the software is inserted, Type of software, Development paradigm, Nature of project, Experience level of project managers, Experience level of development team, Experience level of clients, Geographical distribution of team, Use of innovative technology and Possible restrictions of the project (Schedule, Performance, Security and Human Resources).

3 Experimental Study of the Relationships Between Risk-Causing Facts and Risks

The first risk management activity accomplished in a project is the identification of the project risks, where potential problems to be faced by the development team and project management are identified. Failures or items forgotten at this activity are propagated to the next risk management process activities. Thus, it is essential to conduct a careful analysis of all the facts that can potentially cause risks to the project.

Pfleeger et al. [Pfleeger et al., 2001] points out the importance of analyzing the suppositions and decisions regarding how a project will be carried out, who will take part on it and the resources that will be needed to identify risks involved in each supposition and decision taken by the project manager. Therefore, our risk planning approach proposes the use of a checklist as a risk identification technique. This checklist regards conditions and restrictions normally possible to be found or predicted at the project planning phase with the associated risks caused by them.

An experimental study was carried out to produce the checklist characterizing the relationships between risk-causing facts and risks commonly found in software projects. The process used for planning and executing the study was based on the Wohlin et al. proposal [Wohlin et al, 2000] for experimentation processes. Study results were used to feed checklist data making subjects' knowledge and experience regarding facts and risks shareable by the use of such a technique for risks planning.

The following research goals were identified for our study. Using the Goal-Question-Metric Paradigm [Basili et al., 1994], these goals were refined in questions characterizing the main study aspects. Metrics were also associated to the questions defined so that the data could be collected to support answering the questions. To collect data a questionnaire was prepared containing a set of 25 initial risk-causing facts and a set of 15 initial risks, both of them based on risk management technical literature. A characterization questionnaire was used to characterize the subjects.
G1: Analyze the set of risk-causing facts for the purpose of characterizing with respect to their use as risk factors in project planning from the point of view of software project managers in the context of software project risks planning.

G2: Analyze the set of risks for the purpose of characterizing with respect to their use as risks in project planning from the point of view of software project managers in the context of software project risks planning.
G3: Analyze the set of risk-causing facts and the set of risks for the purpose of characterizing with respect to the relationship between risk-causing facts and risks from the point of view of software project managers in the context of software project risks planning.

Subjects were chosen based on convenience. A total of 13 Brazilian software project managers with relevant experience in risk management were selected. Each subject was asked to exclude risk-causing facts and risks s/he considered unnecessary, include risk-causing facts and risks s/he considered important to be present at the sets and to explicitly mark down the relations between software projects risk-causing facts and risks, based on their risk management experience.

The characterization questionnaire information was used to group subjects by experience and to define criteria to evaluate data. Doing so, it was possible to collect project managers' opinions and experiences characterizing a set of risk-causing facts that they believe normally occur in software projects, a set of risks commonly found in software projects and the relations that do exist between these risk-causing facts and risks. Table 1 presents the results obtained with the study.

4 The Riskplan Tool

Riskplan supports the identification, analysis, prioritization, risk management planning and risk monitoring activities, defined in the proposed risk management process. The tool guides the user during the execution of the risk planning activities and it is integrated into an Enterprise-Oriented Software Development Environment (i.e., a software development environment that provides organizational knowledge required by software development and maintenance processes and by their management [Villela et al., 2001]).

Figure 2 shows the basic tool interface. On the left side we can identify the risk management process and on the right side the activity currently being performed by the user. The icons located under the title bar allow the knowledge search and registration concerned with process activities. The project manager can look for ideas and lessons learned registered by previous projects managers and can also register their own ideas and lessons. Throughout all risk management process, RiskPlan also makes it available the explicit knowledge contained in the tool repository concerning the activity that is being performed.

The checklist (resulted from the experimental study) is available to the project manager during the accomplishment of the activity "Identify risks from project facts". As the manager selects the risks-causing facts pertinent to the project, associated risks are displayed on the screen. In case the manager believes a risk is not pertinent, s/he can uncheck it and such risk will not be confirmed for the project risk list. Figure 2 illustrates the tool risk-causing fact checklist partially filled in.

During the execution of the activity "Identify risks from similar projects", the tool makes it available risk data that have been predicted or have occurred in similar projects once the manager has characterized the project. At this activity, RiskPlan presents to the manager the risks occurred in previous similar projects and that were not yet identified as pertinent to the current project, trying to prevent risk-causing facts from being forgotten. Based on the list of risks presented, project manager may use his/er personal experience to select among them those ones that are applicable to the project. Besides, executing the activity "Identify other risks" s/he can include new risks as that s/he judges necessary.

Once project risks have been identified, the manager queries the tool risk knowledge repository to facilitate risks likelihood estimates and impact and the definition of mitigation and contingency plans. Throughout the whole risk management process data obtained from similar projects can be consulted to support project manager analyzing risk behavior in the various similar projects found. Once risks are analyzed, prioritized and planned, the document Project Risks Plan may be generated and will serve as basis for the Risk Management Process.

5 Final Considerations

Software project risk planning is carried out by project managers and it requires an organization view, being strongly influenced by experiences acquired in previous projects. The larger the experience of the project manager the more s/he will be able to make risks identification, likelihood and impact estimates and to define mitigation and contingency plans. This paper has described an approach to support risk planning in software projects offering project managers risk planning knowledge and experience acquired throughout several organizational projects.

Among the main contributions of the approach here described, we point out: (i) description of the risk management process based on the risk knowledge management; (ii) definition of a risk identification strategy based on the reuse of the organizational risk knowledge; (iii) definition of an experimental study about the relations between software projects risk-causing facts and risks, which can be used in future study replications; (iv) list of the existent relationships between software project risk-causing facts and risks, resulted from the study and (v) definition and implementation of the RiskPlan tool, which supports the proposed approach.

Trying to improve and expand the risks planning approach that was proposed, some future work perspectives are pointed out. First, an interesting work would be repeat the experimental study carried out with project managers, now with a new initial set of facts and risks and with more participants. A new study would get the opinion and experience of others project managers and would improve the checklist used as risks identification technique.

The risks planning approach that was proposed uses a search for similar projects based on the direct participation of the user. After characterizing the software project, the project manager chooses which criteria will be used in the search, according to his specific objective. An interesting future work would be the definition and implementation of an automated search technique.

Acknowledgements

This study was partially supported by a grant from CAPES - Brazil that the authors would like to thank. The authors also acknowledge the support of CNPq -Brazilian Research Council to the Enterprise Oriented Software Development Environment Project.

References

[Basili et al., 1994] Basili, V. R. , Caldiera, G., Rombach, H. D. :"Goal Question Metric Paradigm". In John J. Marciniak, editor, Encyclopedia of Software Engineering, volume 1, pages 528532. John Wiley & Sons (1994).

[Garvey et al., 1997] Garvey, P.R., Phair, D.J., Wilson, J.A.: "An Information Architecture for Risk Assessment and Management", IEEE Software, 14, 3 (1997), 25-34.

[IEEE, 2001] IEEE Std 1540-2001: IEEE Standard for Software Life Cycle Processes - Risk Management (2001).

[ISO, 1997] ISO 10006 : Quality Management - Guidelines to Quality in Project Management (1997).

[ISO/IEC, 1999] ISO/IEC DTR 16326: "Software Engineering - Guide for the application of ISO/IEC 12207 to project management" (1999).

[Kontio and Basili, 1996] Kontio, J., Basili, V.R: "Risk Knowledge Capture in the Riskit Method", Proceedings of the 21st Software Engineering Workshop, NASA, Greenbelt, Maryland (1996).

[Markkula, 1999] Markkula, M.: "Knowledge Management in Software Engineering Projects", Software Engineering and Knowledge Engineering - SEKE 99; Kaiserlautern, Germany, June (1999).

[Pfleeger et al., 2001] Pfleeger, S. L., Haton, L., Howell, C.C.: "Solid Software"; Prentice Hall (2001).

[Statz, 1999] Statz, J.: "Leverage your Lessons", IEEE Software, 16, 2 (1999), 30-32

[Villela et al., 2001] Villela, K., Santos, G., Bonfim, C., et al.: "Knowledge Management in Software Development Environments", 14th International Conference Software & Systems Engineering and their Applications, Paris (2001).

[Wohlin et al., 2000] Wohlin, C., Runeson, P., Höst, M., Ohlsson, M., Regnell, B., Wesslén, A: "Experimentation in Software Engineering - An Introduction", Kluwer Academic Publishers (2000).

[Williams et al., 1997] Williams, C.R., Walker , J.A., Dorofee, A. J.: "Putting Risk Management into Practice", IEEE Software, 14, 3 (1997), 75-81.