A Review of Survey Research in Knowledge Management Performance Measurement: 1995-2004

Abstract: This paper surveys knowledge management (KM) development using a literature review and classification of articles from 1995 to 2004 with a keyword index and article abstract in order to explore how KM performance evaluation has developed during this period. Based on the scope of 76 articles from 78 academic journals of KM, this paper surveys and classifies KM measurements using the following eight categories: qualitative analysis, quantitative analysis, financial indicator analysis, non-financial indicator analysis, internal performance analysis, external performance analysis, project-oriented analysis, and organizational-oriented analysis together with their measurement matrices for different research and problem domains. Discussion is presented, indicating the followings future development directions for KM performance evaluation: (1) KM performance evaluation is getting more important. (2) The quantitative analysis is the primary methodology in KM performance evaluation. (3) Firms are now highlighting the KM performance of competitors, through benchmarking or best practices, rather than internally auditing KM performance via balanced scorecard. (4) Firms may begin to focus more on project management measurement, than on the entire organization.

Keywords: Knowledge Management, Performance Evaluation, Literature Survey

Categories: A, A.0 , A.1, I.2.4, K.6, SD K.6.2

1 Introduction

As a part of knowledge management (KM) research, this paper focuses on surveying KM development through a literature review and classification of articles from 1995 to 2004. The reason for choosing this period is that the knowledge spiral was proposed to corporations and organizations in 1995 and this model plays important roles, not only in fulfilling academic research studies, but also in creating, exploiting and recycling knowledge within the business environment. This literature survey started on January 2005 and it was based on a search in the keyword index and article abstract for 'knowledge management' on the Elsevier SDOS, IEEE Xplore, EBSCO, Ingenta, and Wiley InterScience online database, in which 3,667 articles were found. After topic filtering, there were 76 articles from 78 journals related to the keyword 'knowledge management performance evaluation'.

Based on the scope of 76 articles from 78 academic journals of KM, this paper surveys and classifies KM measurements using the following eight categories: qualitative analysis, quantitative analysis, financial indicator analysis, non-financial indicator analysis, internal performance analysis, external performance analysis, project-orientated analysis, and organizational-orientated analysis.

The rest of the paper is organized as follows. Sections 2 presents the survey results of KM performance evaluation based on the above categories, respectively. Section 3 presents some discussion of KM performance evaluation. Finally, Section 4 contains a brief conclusion.

2 KM Performance Evaluation Methodology

(1) Qualitative Analysis
A qualitative research approach was refined using the outcomes of a pilot study and reviews by researchers of organization learning. For example, the success of knowledge sharing in organizations culture, are not only technological but also related to behavior factors [Hertzum, 02] [Walsham, 02]. Besides, expert interviews, critical success factors method (CSFs), and questionnaires are used to implement qualitative methods for exploring specific human problem.

From the organizational perspective, attention to an organization's internal controls has increased significantly since the 1990s. Although management is ultimately responsible for ensuring that internal controls are adequate, managers often lack the knowledge of internal control concepts. Changchit et al. used a questionnaire in an experiment examining an expert system, which could facilitate the transfer of internal control knowledge to management [Changchit, 01]. The results indicated that expert systems are viable aids for transferring internal control knowledge to managers, whose work experience is outside of accounting and control systems. Longbottom and Chourides reported on interviews, with key staff within organizations, at various stages of approaching and deploying KM programs [Longbottom, 02]. In a follow-up paper, the research investigated issues concerning the CSFs and measurements of KM, establishing practical and key factors likely to enhance successful implementation. It accessed a range of critical factors and identified appropriate measures over five organizational perspectives: strategy; human resource management; information technology; quality; and marketing [Chourides, 03].

(2) Quantitative Analysis
The aim of quantitative analysis is to present the extent of the impact on both decision making and task performance, using historical data that is easily available, relevant, accurate and timely. This evaluation can avoid the drawbacks of qualitative analysis, especially in the subjective judgment of empirical results. Therefore, a quantitative research approach is designed to represent a tangible, visible and comparable 'ratio'. In other words, quantitative analysis can be used to measure the explicit knowledge of an organization or an individual, with both financial and non-financial indicators; this is discussed below.

(3) Financial Indicator Analysis
Traditional quantitative methods focus on well-known financial measures, such as analysis of financial statement, the payback period, the return on investment (ROI), the net present value (NPV), the return of knowledge (ROK), and the Tobin's q. These methods are best-suited to measure the value of daily transaction processing systems.

Laitamaki and Kordupleski used an ROI index to evaluate KM projects and performance in customer value added (CVA) [Laitamaki, 97]. From the managerial perspective, Stein et al. deployed a knowledge-based system, which was designed to automate tasks previously performed manually, train new staff members, and capture knowledge, to enable a university organization to improve services. Performance evaluation used NPV to diagnose the project outcome. Finally, the system could be viewed as an estimation tool, giving a competitive advantage to the organization [Stein, 01]. From an empirical point of view, it is well known that Tobin's q ignores replacement costs for intangible assets, because of the accounting treatment of intangibles [Lev, 01]. Tangible assets are capitalized and reported on firms' balance sheets. In contrast, intangibles are expensed, i.e. written off on the income statement, along with regular expenses such as wages, rents and interest. As a result, the book value of assets does not reflect the stock of intangibles, resulting from cumulative investments; market value does, however. In fact, it is a fairly common practice, in studies using Tobin's q as a measure of corporate performance, to "correct" the denominator of q for the presence of such intangibles. Examples include knowledge capital [Hall, 00], or customer assets.

(4) Non- Financial Indicator Analysis
In fact, non-financial measures method is different from traditional financial statement analysis. It uses non-financial indicators, such as the how many "frequencies" each employ logins knowledge bases, how many "times" each employ brings up proposals, how many "topic numbers" of discuss board, and What is the "amount" about communities of practice (CoP) in company? These indicators are all related to behavior factors and system usage situation.

CoP have begun to play an increasingly important role in modern, knowledge intensive organizations. Smits and Moor presented a Knowledge Governance Framework, which focused on how to define, measure, and use performance indicators for KM in a CoP. The results were successful and offer useful guidelines for KM procedures [Smits, 04]. To successfully manage knowledge, it must be measured. Holt et al. used four metrics to access organizational knowledge, including individual, context, content and process knowledge measures [Holt, 04]. These approaches enable us to relate knowledge to business performance more explicitly, and provide valuable insight into how knowledge may be strategically managed.

(5) Internal Performance Analysis
Internal performance measurement methods focus on process efficiency and goal achievement efficiency. These methods evaluate KM performance through the gap between target and current value. The well-known methods are including ROI, NPV, balanced scorecard (BSC), performance-based evaluation, activity-based evaluation, and other models.

Underlying Kaplan and Norton's concept of BSC was that all aspects of measurement have their drawbacks; however, if companies offset some of the drawbacks of one measure, with the advantages of another, the net effect can lead to decisions resulting in both short term profitability and long term success [Kaplan, 96]. As a result, they suggested that financial measures be supplemented with additional ones, reflecting customer satisfaction, internal business processes and the ability to learn and grow.

Many scholars have discussed the use of a Balanced Scorecard approach in determining a business-orientated relationship, between strategic KM usage and IT strategy and implementation [Martinsons, 99]. They applied an IT investment to KM, by creating a KM scorecard that focused on both the current financial impact of intellectual capital on core processes, as well as future earnings capabilities in structural or human capital.

As mentioned earlier, valuable knowledge resides within individual employees and is critical to an organization's ability to solve problems and create new knowledge. In a sense, KM can be viewed as an activity, which acts as a constituent of a community, performing one's task by using tools or technology [Hasan, 01].

(6) External Performance Analysis
External performance measurement methods always compare itself with benchmark companies, primary competitions, or whole industry average. With benchmarking or best practices methodologies, firms can understand its KM performance to compare competitions.

Benchmarking is also seen as a tool for identifying, understanding and adopting best practices, in order to increase the operational performance of intellectual capital (IC) [Marr, 04]. From an organizational learning perspective, benchmarking is concerned with enhancing organizational performance, by establishing standards against which processes, products and performance can be compared and consequently improved [Pemberton, 01].

The "Best Practice" approach is an essential component of KM. It provides an opportunity to retain and use knowledge, even when an expert has left the organization. Asoh et al. investigated how governments could deliver more innovative services to a demanding public [Asoh, 02]. They felt that governments must be involved in the deployment of new services, such as e-Government and e-Commerce.

(7) Project-orientated Analysis Recent studies of KM and organizational learning in project environments have emphasized instead the difficulties of learning from projects-not only within individual projects, but also across and between projects [DeFillippi, 01].

Bresnen et al. revealed that processes of the capture, transfer and learning of knowledge, in project settings, rely very heavily upon social patterns, practices and processes, in ways which emphasize the value and importance of adopting a community-based approach to managing knowledge [Bresnena, 03]. Bresnen et al.'s paper made a contribution to the development of knowledge management theory, within project environments.

Nevertheless, project organizations require particularly systematic and effective knowledge management, if they are to avoid knowledge fragmentation and loss of organizational learning. Kasvi et al. dealt with knowledge management and knowledge competences in project organizations, particularly from a programmers' perspective [Kasvi, 03]. Finally, they made a contribution by presenting the Learning Programme Model. In order to systematically manage the knowledge created within a project, the project, itself, must be systematically managed by the model.

(8) Organizational-orientated Analysis
The organization-oriented analysis is focus on whole organization, multi-dimension, and multi-layers in the firm. It can analyze KM performance evaluation from intellectual capital, BSC, technology, and process perspectives. The primary objective is estimated the level of KM performance in the whole organization.

Most organizations have only a vague understanding of how much they have invested in intellectual capital (IC) let alone what they may receive from those investments. Standard financial accounting systems do not allow for the easy estimation of intellectual capital investments. Among the most widely used approaches for IC management and reporting are the so-called Intangible Asset Monitor by Sveiby and the IC approach by Edvinsson and Van Buren, originally introduced by the insurance company Skandia [Sveiby, 98] [Edvinsson, 97]. These models are designed to measure human, innovation, process and customer capital, and represent a major step toward providing precisely the information that firms and their stakeholders need to foresee the future. Thus, these IC models can help visualize the knowledge-production process of research organizations.

This study reviewed previous KM literature at the start; these perspectives are summarized in Table 1.

3 Discussion

As shown in Table 2, we gather statistics which is the survey research in KM performance evaluation from 1995 to 2004. Besides, we aim at examining the research trend in KM performance evaluation change, then we use two phase to distinguish former five years (1995-1999) from latter five years (2000-2004). In the Figure 1, we can understand the change between former and latter five years. The main findings describe as follows:

1. KM performance evaluation is getting more important. The articles have published in letter five years is double amount for former five years. It shows the research topics have changed from KM creation, transformation, and implementation to evaluate KM performance.
2. The quantitative analysis is the primary methodology in KM performance evaluation. The results show the quantitative analysis has most research articles in latter five years. In traditional evaluation approach, most scholars suggest the financial indicators can distinct display the KM values. In opposition, scholars insist on evaluating KM performance by non-financial indicators in the social and behavior sciences approach
3. Firms will highlight the competitions' KM performance through benchmarking or best practices more than audit internal KM performance by BSC. The results explain the firms will compare KM performance with their foes. For this reason, firms use external performance approach to replace original BSC framework. Moreover, firms use benchmarking or best practices to integrate four perspectives in BSC activities.
4. Firms will focus on project management than whole organizational measures. The results explain the firms will care about the KM go live and control the achieved percentage of scheduled progress in KM project management. It is no doubt that firms want to measure the whole organization's KM performance is very difficult through process, leadership, culture, or technology perspectives. Therefore, firms will get better efficiency and effectiveness on KM performance by project-oriented approach.

4 Conclusion

This paper is based on a literature review on KM performance evaluation from 1995 to 2004 using a keyword index search. We conclude that KM performance measurements tend to develop towards expert-orientation and KM evaluation development is a problem-oriented domain. Different information technology methodologies, such as artificial intelligence methods, are suggested to implement in KM performance evaluation as another kind of technology. Finally, the ability to continually change and obtain new understanding is the power of KM performance evaluation and will be the core value of future works.

References

[Asoh, 02] Asoh, D., Belardo, S., Neilson, R.: "Knowledge Management: Issues, Challenges and Opportunities for Governments in the New Economy"; Proceedings of the 35th Hawaii International Conference on System Sciences, (2002).

[Bresnena, 03] Bresnena, M., Edelmanb, L., Newell, S., Scarbroug, H., Swan, J.: "Social practices and the management of knowledge in project environments"; International Journal of Project Management, 21, (2003), 157-166.

[Changchit, 01] Changchit, C., Holsapple, C.W., Viator, R.E.: "Transferring Auditors' Internal Control Evaluation Knowledge to Management"; Expert Systems with Applications 20, (2001), 275-291.

[Chourides, 03] Chourides, P., Longbottom, D., Murphy, W.: "Excellence in Knowledge Management: an Empirical study to identify critical factors and performance measures"; Measuring Business Excellence 7, 2 (2003), 29-45.

[DeFillippi, 01] DeFillippi, R.J.: "Project-based learning, reflective practices and learning outcomes"; Management Learning, 32, 1 (2001), 5-10.

[Edvinsson, 97] Edvinsson, L.: "Developing intellectual capital at Skandia"; Long Range Planning, 30, 3 (1997), 366-373.

[Hall, 00] Hall, B.H., JaffҒé, A., Trajtenberg, M.: "Market Value and Patent Citations: A First Look"; Working Paper No. 7741, National Bureau of Economic Research, Cambridge, MA, (2000).

[Hasan, 01] Hasan, H., Gould, E.: "Support for the Sense-Making Activity of Managers", Decision Support Systems, 3, 1 (2001), 71-86.

[Hertzum, 02] Hertzum, M.: "The importance of trust in software engineers assessment and choice if information sources"; Information and Organization, 12 (2002), 1-18.

[Holt, 04] Holt, D.T., Bartczak, S.E., Clark, S.W., Trent, M.R.: "The Development of an Instrument to Measure Readiness for Knowledge Management", Proceedings of the 37th Hawaii International Conference on System Sciences, (2004).

[Kaplan, 96] Kaplan, R., Norton, D.: "Using the Balanced Scorecard as a Strategic Management System"; Harvard Business Review 74, 1 (1996), 75-85.

[Kasvi, 03] Kasvi, J.J., Vartiainen, M., Hailikari, M.: "Managing knowledge and knowledge competences in projects and project organizations"; International Journal of Project Management, 21, (2003), 571-582.

[Laitamaki, 97] Laitamaki, J., Kordupleski, R.: "Building and Deploying Profitable Growth Strategies based on the Waterfall of Customer Value Added"; European Management Journal 15, 2 (1997), 158-166.

[Lev, 01] Lev, B.: "Intangibles: Management, Measurement, and Reporting"; Brookings Institution Press, Washington, DC (2001)

[Longbottom, 02] Longbottom, D., Chourides, P.: "Climbing New Heights: Conquering K2"; Knowledge Management Magazine, June, Bizmedia (2002).

[Martinsons, 99] Martinsons, M., Davison, R., Tse, D.: "The Balanced Scorecard: a Foundation for the Strategic Management of Information Systems"; Decision Support Systems 25, (1999), 71-88.

[Marr, 04]Marr, B.: "Measuring and benchmarking intellectual capital"; Benchmarking, 11, 6 (2004), 559-570.

[Pemberton, 01] Pemberton, J.D., Stonehouse, G.H., Yarrow,D.J.: "Benchmarking and the role of organizational learning in developing competitive advantage"; Knowledge and Process Management, 8, 2 (2001), 123-135.

Page 11

[Smits, 04] Smits, M., Moor, A.D.: "Measuring Knowledge Management Effectiveness in Communities of Practice"; Proceedings of the 37th Hawaii International Conference on System Sciences, (2004).

[Stein, 01] Stein, E.W., Manco, M.P., Manco, S.A.: "A Knowledge-based System to Assist University Administrators in Meeting Disability Act Requirements"; Expert Systems with Applications 21, 2 (2001), 65-74.

[Sveiby, 98] Sveiby, K.E.: "Intellectual Capital: Thinking Ahead"; Australian CPA, (1998)

[Walsham, 02] Walsham, G.: "Knowledge Management: the Benefits and Limitations of Computer Systems"; European Management Journal, 19, 6 (2002), 599-608.