Identifying Employee Competencies in Dynamic Work Domains: Methodological Considerations and a Case Study

Abstract: We present a formalisation for employee competencies which is based on a psychological framework separating the overt behavioural level from the underlying competence level. On the competence level, employees draw on action potentials (knowledge, skills and abilities) which in a given situation produce performance outcomes on the behavioural level. Our conception is based on the competence performance approach by [Korossy 1997] and [Korossy 1999] which uses mathematical structures to establish prerequisite relations on the competence and the performance level. From this framework, a methodology for assessing competencies in dynamic work domains is developed which utilises documents employees have created to assess the competencies they have been acquiring. By means of a case study, we show how the methodology and the resulting structures can be validated in an organisational setting. From the resulting structures, employee competency profiles can be derived and development planning can be supported. The structures also provide the means for making inferences within the competency assessment process which in turn facilitates continuous updating of competency profiles and maintenance of the structures.

Keywords: Skills Management, Competency Management, Knowledge Space Theory, Competence-Performance Approach, Human Resource Development, Repertory Grid Technique, Case Study

Categories: H.3.1, H.3.2, H.3.3, H.3.7, H.5.1

1 Introduction

1.1 Skills Management: two unresolved issues

For several years now, skills management or competency management has been suggested as a way to more effectively utilise employee skills in the workplace. The concept originated from Human Resource (HR) Management as a way to align HR processes (like selection, appraisal and development) to job requirements and organisational strategy [Green 1999]. Moreover, it has been suggested that in Knowledge Management (KM) approaches defining competencies can support knowledge management processes like goal setting and evaluation [Probst et al. 2000], or the assignment of teams in knowledge-based organisations (see [Deiters et al. 2000]). In both approaches "skills" or "competencies" are being defined in organisations in order to describe characteristics of individual employees to make better use of their expertise or to develop it further.

In this paper, we focus on two types of issues that have been the source of much debate within the field of Skills Management. The first one is concerned with the nature of competencies itself and the way they could be described or formalised as to provide better use within companies. The second issue relates to the way that - once defined - competencies can be identified or assessed, that is how competencies can be assigned to the people that possess them.

Within the present approach, we consider these issues in a certain setting, namely that of "dynamic work domains". We consider Skills Management in knowledge-based organisations in which job requirements change frequently and employees exert more control over their own work processes. These could typically be small or medium sized companies (like consultancies or research institutions), or subunits of larger organisations (like R&D departments) which are organised around projects.

1.2 Competency formalisation: what are competencies?

Although competencies have been considered increasingly important in HR and KM approaches, it is thus far an unresolved issue of what exactly competencies are. Traditionally, HR approaches have been most concerned with advancing theoretical understanding about the nature of competencies. [Boyatzis 1982] and [Spencer and Spencer 1993] define competencies as the capacities that exist within a person and which predict superior performance. They are usually seen to encompass a person's knowledge, skills, attitudes and behaviours which predict competent performance in a certain job. [Lucia and Lepsinger 1999] differentiate competencies as aptitudes, personal characteristics, skills and knowledge from the actual behaviours that people need to perform at the workplace. The distinction between competencies as underlying human qualities on the one hand and observable performance outcomes on the other has a long tradition in psychology. This distinction greatly influenced traditional approaches of skills or competency management. Especially when it was first introduced as a human resource management approach, for example by [Spencer and Spencer 1993], [Boyatzis 1982] and [Lawler and Ledford 1992], a certain combination of individual skills was seen as the determinant of competent job behaviour.

KM approaches, on the other hand, usually focus more on employees' technical or professional knowledge, for example when predicting whether an expert will successfully solve a given problem, or effectively complete a given project assignment. However, they share the same basic assumption with HR approaches, that is to predict future performance from some personal characteristics.

Within this paper, we will follow these views by defining competencies as personal characteristics (knowledge, skills, abilities) which are relatively stable across different situations. Competencies can be described in terms of distinguishable elements of underlying capacities or potentials which allow job incumbents to act competently in certain situations [Bergmann 2000]. Employees dynamically combine these elements according to the requirements of the situation in a self-organising process [Erpenbeck and Heyse 1999]. We distinguish competencies - which we understand as non-observable theoretical constructs - from overt behaviour that can be observed and judged according to certain standards of performance in a given situation. This basic distinction leads us to a framework that establishes a relationship between competencies and performance outcomes which will be described below in more detail.

1.3 Competency identification: how to assess competencies?

Another frequently debated issue is that of diagnosing competencies in organisational settings, or the assignment of which people possess which competencies. Several different kinds of approaches have been suggested. Subsequently, we will distinguish between manual and automatic profiling approaches.

A manual top-down driven approach is described in [Wöls et al. 2003]. It starts out on the organisational level where core competencies of an organisational unit are distinguished from competencies of individual employees. In order to ensure that employee competencies are managed in line with the future needs of the organisation, the Skills Management initiative starts by looking at future developments of the market and the needs of the customer and from these goes on to define the core competencies of the organisational unit. [Green 1999] gives an account of how to connect definitions of individual skills with a company's objectives and core competencies.

The next step is to define required individual competencies which is usually done for a single job or a family of jobs. The requirements are defined in terms of required technical skills and knowledge, management skills and social and personal skills [see Faix et al. 1991]. They are derived from job requirements and are influenced by the core competencies defined earlier. As a result, a number of job profiles, sometimes also called "competency models" [see Lucia and Lepsinger 1999], are obtained. The required skills are then taken as a basis for assessing existing skills of the workforce. This can be done by self-assessment or within the typical appraisal process.

Skill definitions and profiles are then being centrally stored in a data base, for which numerous software products are available. This ensures that the same definitions are used throughout the company and that profiles can be made transparent. Transparency of skills profiles is usually seen as a means for managing and developing employee capabilities in an active and goal directed manner.

While there has been a long tradition of manual profiling - mainly in HR Management - some shortcomings become immediately apparent: Most importantly, manual profiling requires substantial investments for the organisation, especially when requirements for the workforce change frequently and frequent updating of the profiles becomes necessary. Accordingly, ways to automate the process have been suggested by using capabilities of Information Technology (IT).

Most often, automatic extraction of keywords from textual documents is being used (see for example [Hulth et al. 2001]). If the author of the document is known, automatically derived profiles use a vector of these keywords to describe a person's expertise. Expertise profiling systems have been implemented in many KM software systems that incorporate a document management component, like Hyperwave (http://www.hyperwave.com) or Lotus (http://www.lotus.com). [McDonald and Ackerman 2000] suggest an expertise recommender system, in which a software component scans different kinds of documents to extract expert profiles.

Other ways to deal with the issue of identifying expertise include [Won and Pipek 2003] who take employee actions in a Groupware System (e.g. reading a newsgroup) as indicators for their expertise. [Stenmarck 2001] identified employees' fields of professional interests (taken as an indication of their tacit knowledge) through their usage of a recommender system.

1.4 The present approach

The purpose of looking at competencies in organisations is to support firms in the challenges they are facing in an increasingly knowledge-based economy. Human knowledge and expertise are becoming the foundation of many company's assets, and instruments that help manage these assets are being needed. In our research, we are considering primarily two instances of such instruments: The first one is one that dynamically adapts development paths for HR development to the current (and changing) competency requirements of the organisation. The second one is a "knowledge map" that provides better access to experts by visualising competencies within the organisation. In this paper, we will focus on the first instance, the second one will be briefly discussed in the concluding remarks.

For this purpose, we have developed a methodology for identifying employee competencies in organisations. As pointed out above, we look mainly at knowledge-based organisations in which job requirements change frequently and employees exert control over their own work processes. To reflect these kinds of environments, we have based our approach on documents employees are creating as part of their usual work assignments. While we see the advantage of automatic profiling in this case, the identification of competencies in organisations requires human input that provides interpretation of the results gained from automatic extraction. We therefore seek to establish a semi-automatic approach for competency identification, where human actors (e.g. HR manager, knowledge engineers or employees themselves) are supported in the process of competency identification. Consequently, the kind of competency formalisation we are suggesting allows for automatic inferences in the competency management process.

Meaningful, valid and useful competency assignments require a theoretical model that establishes relationships between competencies on the one hand and behaviour and performance on the other. We will be presenting such a theoretical framework in the next section [see Chapter 2]. An important part of our research is to validate the models we are using with empirical investigations. How this was achieved in a specific case will be presented in the case study section [see Chapter 4].

2 Competence Performance Theory

A theoretical framework that is consistent with the requirements we have established in the previous section is the Competence Performance Theory (CPT, see [Korossy 1997] and [Korossy 1999]). It belongs to a family of theories that originated from research into "knowledge spaces" (see [Doignon, Falmagne 1999] and [Albert, Lukas 1999]).

A knowledge space is a mathematical structure consisting of all the knowledge states within a certain domain that a person may be in. A knowledge state is formalised as the subset of tasks of the domain that a person is capable of accomplishing. Of great importance are the dependencies within the set of tasks which can be interpreted as meaning "if a person is capable of accomplishing task a, then he or she will also be able to accomplish task b".

These dependencies restrict the number knowledge states than can be expected to appear within a certain population of learners. These (and other) characteristics of the structures have been useful for creating adaptive tests (see for example [ALEKS Corp. 2003]). Adaptivity then means that an individual will be presented with those tasks that are maximally suited to his or her current state of knowledge and therefore are neither too demanding, nor too easy.

A possible limitation of knowledge space theory which has been put forward by [Korossy 1997] is its sole focus on the behavioural (or performance) level. The theory thereby neglects progress in cognitive psychology or educational sciences that have advanced theoretical understanding of the reasons for different levels of performance. Such cognitive theories of the underlying skills that shape performance offer better ways to suggest training and development measures.

Therefore, Korossy has suggested an extension of the theory of knowledge spaces, which takes into account two sets of concepts, namely the set of tasks (or more generally, the set of performance outcomes) and the set of competencies (knowledge, skills and abilities) that are necessary to accomplish the tasks. If the elements of the two sets are related to one another, structures can be derived which can be interpreted in terms of prerequisite relations or learning paths on the performance and on the competence level. Accordingly, the advantage of the competence-performance approach is that the competencies help to predict performance outcomes and provide an explanation for discrepancies in performance. For example, missing competencies can help to explain why an employee was not able to accomplish a certain task. Hence, development programs can be created that focus on theses underlying competencies.

Since its inception, knowledge space theory has been applied in many different contexts, e.g. in eLearning [Conlan et al. 2002], problem solving [Stefanutti and Albert 2003], workflow modeling [Stefanutti and Albert 2002], and in different kinds of knowledge assessment procedures [Dowling and Hockemeyer 2001]. For skills management purposes, [Alan and Zalewski 2003] have suggested knowledge space theory as a way to structure an ontology allowing to make inferences in skills identification.

The following example is taken from [Ley and Albert 2003] in which a competence-performance structure was created for the position of a Human Resource Manager. [Figure 1] shows part of this structure with the tasks for the position (1.1, 2.1, 4.2 and so forth) and the competencies required to accomplish these tasks (A, B, I, J, and so forth). The resulting competence structure (on the right) shows learning paths for the employees that proceed from the bottom to the top of the structure. Each step involves learning additional competencies so that new tasks can be mastered. For illustration purposes, it only shows part of the overall structure and provides a simplified view of Korossy's approach.

We have a set P of tasks, in the example those are the tasks that have to be carried out in a certain position. Subsets of P are called Performance States, if they contain the tasks (performances) a person is able to accomplish. A collection of Performance States closed under union is called a Performance Space . Of course the tasks are not independent of one another: If a person is able to accomplish a certain task, one can surmise that he or she will be able to accomplish a certain other tasks as well.

This relationship is formalised by the surmise function s: P -> which assigns to each task p P a set of Performance States s(p) in each of which p is included, and which are minimal with respect to , meaning that if a person is able to accomplish task p, that person is at least in one of the Performance States in s(p).

According to [Doignon and Falmagne 1999], there is a one-to-one correspondence between the surmise function s and the Performance Space , meaning that only consists of those states which can be expected on the basis of the surmise function and vice versa. In the above structure for instance, a Performance State that contains task 2.1, but doesn't contain task 1.1, cannot be expected, as 1.1 is a prerequisite for 2.1.

The second set is the set E of competencies which should be of smaller magnitude than the set of tasks, in order to provide some additional explanatory power: Performance in many different tasks should be determined by the combination of a few underlying competencies. E is structured in a similar way as P: A surmise function assigns to each competency e E a set of Competence States (e) in each of which e is included, and which are minimal with respect to , meaning that if a person is possesses competency e, that person is at least in one of the Competence States in (e). The surmise function establishes a Competence Space . The two spaces and are connected by an interpretation function k: P -> , which assigns to each task p P a set of Competence States k_x in which the tasks can be accomplished.

When the two sets E and P are related to one another in a matrix such as in [Figure 1, left part], a competence-performance structure can be derived. First of all, Competence states can be derived as subsets of competencies that are necessary for accomplishing the tasks (the columns of the matrix, e.g. {I;J}, {B;I;J;L;O;P} and so forth). By closing the collection of competence states under union, a Competence Space is derived. After that, all tasks that can be expected in a certain competence state are assigned to this state [see Figure 1, right part].

3 Competency elicitation techniques

While in the previous section, we have been concerned with the question of a suitable form of formalisation for human competence and performance, we will now consider how employee competencies can be elicited in an organisational setting.

It makes sense for several reasons to start out with individual views on competency models. First of all, we focus on knowledge-based work in which work requirements are frequently changing. It is mostly the employees themselves who control their own work processes. They develop their expertise in close interaction with the situational requirements. It is therefore essential to involve the individual employees directly in the elicitation process. No one else can usually give better account of the experts' competencies than the experts themselves. Secondly, our approach is in line with [Michellone and Zollo 2000] who have suggested to start a competency management process with individual views because eliciting competencies can be understood as an individual construction process.

Accordingly, a knowledge elicitation technique that seems to be well suited is the repertory grid technique (RGT). RGT has often been suggested for eliciting employee competencies (e.g. [Armstrong 1999], [Jankowicz 1990]), and it has been applied in the context of the competence-performance approach [Schweitzer 2001]. Having originated from George Kelly's Personal Construct Theory [see Kelly 1955], RGT was concerned with uncovering personal constructs that people use to construe events in order to make sense of their surroundings. RGT is very well suited for idiographic research designs where individual views are compared to one another. Also it puts the individual expert in the centre of attention and attempts to reduce the interviewer bias which is often a major source of inaccuracy in the process.

Simply put, RGT puts the expert in a situation where he or she is confronted with a set of objects (his/her documents in our case) and has to determine in what sense some of these objects are similar and differ from the others. For this purpose, the expert is presented with a subset of three documents and has to find an answer to the question "When you think of the skills and knowledge you used to create these documents, in what way are two of these documents alike and different from the third?" The answer to this question is a construct which the person uses to construe similarities and differences in the set of documents. In our special case it corresponds to a competency that was used in creating some of the documents, but not the others.

When one starts with individual views of competency models, there is an additional challenge with which one has to cope, that is transforming the individual views to a shared understanding within the organisation. This corresponds to transforming private memory into shared frames for the organisational memory [Michellone and Zollo 2000] or sharing mental models [Rollett et al. 2001].

The difficulty here is to establish a shared frame of reference that all employees in the organisation can relate to. Because this is a mutual construction process, there are substantial efforts involved for the organisation. At the present point in time, we have not focussed our efforts on this part of the methodology. The repertory grid offers techniques for group sessions or for constructing grids in an iterative process involving several people in the construction process. Also "knowledge maps" [see Eppler 2001] are a way for visualising competency models and thereby providing a tool for sharing individual views and arriving at a shared model.

4 Identifying competencies of projects managers: a methodology and a case study

In this section, we describe a case study in which we have applied the ideas stated so far. The case study was conducted at the Know-Center, a competence centre which carries out cooperative research projects with industry partners in the area of knowledge management. Topics of these projects are introducing Knowledge Management or eLearning systems in companies and providing necessary consulting services, composing technology studies or developing software applications for knowledge management purposes.

Table 1 gives an overview of the methodology we employed for deriving and validating competence-performance structures at the Know-Center. The remainder of the chapter goes through the seven steps of the methodology and illustrates them by case example.

4.1 Examining the purpose and setting of competency modelling

We would argue that the specific approach that is used to identify and structure competencies within a particular organisation should be developed on the basis of the actual modelling purpose and the setting within the organisation [Wöls et al. 2003]. For the Know-Center, purpose and setting derive from current challenges they were facing: After two years of operation and rapid growth, the Know-Center was confronted with the need to consider available competencies, design a coherent strategy and align projects and processes to this strategy. One purpose of the study was to provide an input for this strategy formulation process, namely to explicate what kind of knowledge and skills project managers had acquired in the projects conducted thus far.

Also, with the growing size and complexity of the projects, there was a need for more cooperation between the different business areas which made it necessary to provide a better transparency of the available expertise in the organisation. Thirdly, the study was supposed to provide insight into the job requirements of project managers in order to provide the basis for developing project management expertise in future employees.

4.2 Selecting a position and a group of employees

The position in the organisation that was selected was that of a project manager. At the Know-Center, project managers usually are recent graduates who have certain fields of expertise related to the work done at the Know-Center. In their first projects, they acquire skills for applying their expertise in a business setting, they develop knowledge in other areas and they learn about the project managing process.

For the pilot study, seven employees were involved. They came from all three business areas and had different levels of experience in conducting projects. They were then asked to participate in competency elicitation interviews to find out about the skills they had acquired in their recent projects.

4.3 Selecting the set of performance outcomes

In the present study, we chose to use documents that had been written by the project managers as the performance outcomes. In the company studied, as well as in many other similar companies, the documents produced by the workforce give good indication of past job requirements. Because work outcomes are mostly knowledge-based, the result is often a written document that gives account of the work performed.

In the case study, the respondents selected 5-8 documents they had recently created in projects they had been working in, and which provided a good sample of the work they had performed. 47 documents were named in total. The document types and the number of selections are given in [Table 2].

4.4 Performing competency elicitation

While documents were used as performance outcomes, the competencies were elicited by asking for "knowledge and skills used for creating the documents". These competencies went well beyond pure text production skills. Rather, it was always stressed that all knowledge and skills should be considered that were needed in the process of producing the document (e.g. research skills, communication skills or the skills necessary to analyse organisational processes if this happened to be the topic of the particular document).

An interviewer (the first author) performed a repertory-grid type interview in which a set of three documents was presented to the respondent and he/she was asked to name a competency that separated two of the documents from the third [see Section 3]. This process was repeated until no new competencies were named by the respondent. The interviews resulted in 11 to 20 competencies for each person, 113 competencies were named in total.

4.5 Obtaining individual competence-performance structures

Each respondent then considered all of his/her documents and all competencies he or she had named. To each document, the respondent assigned all competencies he or she had used when creating the document (document-competency assignment). Accordingly, a matrix was obtained for each respondent which consisted of n ( m assignments, where n was the number of documents ("performances") and m is the number of competencies named by that person. The cells of the matrix are filled with 0 and 1, where 1 indicates that a certain competency was used in creating the document, and 0 indicates that it was not.

These assignments establish the minimal interpretation of the documents in the set of competencies in the sense of the interpretation function k, [see Section 2]. From these matrices, one could derive individual competence-performance structures for each individual as described in [Section 2].

4.6 Obtaining organisational competence-performance structures

In order to arrive at useful results for the organisation as a whole, it is necessary to obtain an organisational competence-performance structure which combines the individual structures. Whereas [Doignon and Falmagne 1999] discuss some ways to automatically combine different structures, this was not possible in our case because the seven individual structures drew on different documents and on different competencies. We have therefore chosen to carry out a content analysis of the individual competencies. The respondents then rated the categorised competencies a second time.

4.6.1 Content analysis of individual competencies

By means of a content analysis, we looked for similarities within the set of competencies named by our respondents. The competencies can either be aggregated or categorised according to pre-defined competency categories (top-down analysis) or by finding similarities within the set of named competencies (bottom-up analysis). Categorisation is a necessary condition for drawing conclusions on an organisational level.

In our case study, both top-down and bottom-up categorisation was performed. Top-down categorisation on the set of all competencies named resulted in a distinction of the following two groups of competencies:

1. Skills and knowledge important for all employees belonging to the position under consideration were called general job competencies. These competencies are transferable between the specific projects in which employees work, and might often be included in a formal job description for the position. Examples include project planning or research skills. They are especially useful in identifying development needs that apply to the position of a project manager in general. 70 general job competencies were found.

Page 1509

2. Skills and knowledge applicable only in a specific domain (e.g. knowledge of eLearning tools or knowledge of the production of multimedia content) were considered to be domain specific competencies. While such competencies are often learned "on-the-job" and are therefore less focused on in planned development, they are very important when one is looking for expertise within the organisation (i.e. for knowledge mapping). They are usually not part of a formal definition of the position, but might be acquired "ad hoc" while working on a certain topic. 43 domain specific competencies were obtained.

The difference between domain specific competencies and general job competencies is similar to the differentiation of "technical knowledge and job skills" and "performance skills and competencies" that has been introduced by [Green 1999]. In practical applications, a distinction is made between "core competencies" or "business competencies", that is competencies everyone in a company needs, and "technical expertise" or "technical competencies" which are more specific to certain occupational groups (see [Bina and Newkirk 1998], [McDaniel 1998]).

After that, bottom-up classification of the general job competencies was performed and resulted in 14 categories of similar content. These categories resemble typical job competencies in knowledge-based companies, like 'research skills', 'information synthesis', 'communication' and so forth. The reason this was done was to make it easier for the interviewees to respond to the second part of questioning. Within the categories, competencies that were very similar were collapsed into one competency, leaving 59 of the original 70.

4.6.2 Obtaining organisational competence-performance matrices

A second round of document-competency assignments was then needed to obtain an organisational structure. Participants from the first part of the study were asked to make document-competency assignments for the documents they had created on all 102 categorised competencies (59 general, 43 domain specific) in a questionnaire type format. The names of the competencies were taken from the first part of the study. The general competencies were grouped into the categories, but no category name was provided. The seven assignments were then simply combined into an organisational matrix of 102 (competencies) x 47 (documents) entries.

4.6.3 Deriving organisational competence-performance structures

Using the organisational matrix, competency development structures were derived which aim to support development planning of employees. These structures establish the competencies that need to be developed and the ways this can be accomplished. To illustrate how this can be achieved, we have selected a subset of all competencies E (in this case all those having to do with communication to customers and team members) and a subset of all documents P (in this case all project related documents, types 6-8 from [Table 2]). [Table 3] gives the surmise function s for the corresponding Competence Space.

e E	Short Description	(e)
A	Communication about customer requirements	{A,D};{A,B,F}
B	Discussing ideas and concepts on an informal level	{A,B,D};{A,B,F}
C	Understanding goals of others	{C}
D	Discussing a common practice in a team	{A,D};{D,G}
E	Employing effective interview techniques	{A,B,C,D,E,F,G}
F	Presenting and selling own ideas	{F,G};{A,B,F}
G	Defining goals and persuading others	{F,G};{D,G}

Competencies listed in the table could be incorporated into general development plans for the organisation. The surmise function suggests that competency E ("Effective interview techniques") needs as a prerequisite all the others. Furthermore, it implies that competency C ("Understanding goals of others") is independent of the others.

When the complete Competence Space is obtained [see Section 2], development paths for individual employees can be derived. [Figure 2] gives the corresponding Competence Space for the surmise function s. The boxes represent the Competence States, characterised by a specific combination of competencies (A-G). States are connected by lines denoting a subset relation. Below the competencies, numbers of documents are given that can be created in the state (documents are only listed in the minimal state).

As can bee seen in the figure, the documents are grouped in the resulting space in a specific way. For example, documents 11, 17, 41 and 45 in the top two states are all Project Requirement Documents, which obviously produce the highest demands in terms of communication capabilities. When a performance weakness is assessed for a certain employee, required competencies can be derived quickly. For example, if an employee is good at creating documents 10, 12, 13 and 14, but lacks the skills for the others (competence state "ACDG"), one would concentrate on competencies B and F to strengthen capabilities for the project definition process.

4.7 Validating the results

As we have mentioned previously, establishing quality criteria for the resulting structures is an important part of our research. We apply criteria which have been established for psychological research: objectivity, reliability and validity (see for example [Rost 1996]). We are concentrating on measures on the individual level, organisational level analysis will not be considered here.

4.7.1 Objectivity of individual structures

Objectivity in our case requires that the results are independent of the researcher. This was not tested empirically, however, great care was taken within the process to establish objective results. After the interview for example, the results (competency descriptions and matrix) were fed back to the respondents to allow them to check whether the documentation corresponded to their own understanding.

4.7.2 Reliability of individual structures

As a measure of reliability, we measured test-retest reliability of the individual structures between the first and second part of the study. This was possible because for each respondent some of the document-competency assignments were the same in both parts. When classifying each document-competency assignment for one respondent into whether it was rated 1 or 0 in the first part of the study, and whether it was rated 1 or 0 in the second one, a 2x2 cross classifications of all ratings for each respondent was obtained. The cross-classification provides indication for reliability of the assignment, where the classes (1 and 1) and (0 and 0) provide the number of "reliable" assignment, the other two classes provide the number of "unreliable" assignment.

[Goodman and Kruskal 1954] have suggested as a measure of reliability which is based on the probability of correctly guessing the second assignment when the first one is known. thereby provides a measure for the decrease in error probability. can take values between -1 and +1, where 0 denotes that no information is gained from the first part for predicting the second.

For the 7 subjects in our case study, varied between 0.125 and 0.659. With an overall reliability of 0.494, the index is of satisfactory magnitude. One should bear in mind that 6 months lie between the first and the second assignment, that the second assignment was done in a different setting and that the competencies were put in a different order and grouped with other respondents' answers which could have considerably affected the assignment.

4.7.3 Validity of individual structures

There are different ways to measure validity. We have chosen to look at construct validity which looks at different operationalisations of the same construct and seeks to establish convergent and discriminant evidence (see [Brewer 2000]). Convergent validity shows a high correlation between different operationalisations of the same construct, discriminant validity shows low correlations between operationalisations of different constructs.

In the case study, we have looked at similarity of documents which we have operationalised in two ways. On the one hand, we have obtained similarity indices for pairs of documents which measure the similarity of two documents in terms of the competencies assigned to them. They were obtained by dividing the number of corresponding document-competency ratings (a competency was used for creating both documents or a competency was not used for both documents) by the number of all ratings. Accordingly, the index could vary between 0 (no correspondence in the competencies used) and 1 (full correspondence in competencies used). In our case, we derived two similarity indices s_g and s_d. s_g was based on similarity in general job competencies that were used in creating the two documents, s_d was based on similarity in domain specific competencies.

The second operationalisation of document similarity was attained by asking respondents to rate the similarities of pairs of their own documents. Respondents were asked to do this rating for two imagined situations: In both situations, participants were told that they had to use the documents for introducing a new employee to their work. In situation A ('general introduction'), the new employee was someone from a different professional background who they had to introduce to general knowledge and skills necessary for a project manager. In situation B ('professional introduction'), the new employee had the same professional background and the introduction was to convey the professional expertise participants had build up over the years. Respondents were told to picture themselves using the documents to introduce the new employee, and rate the similarity of pairs of documents on a five-point scale from 1 ('no similarity whatsoever') to 5 ('very similar') in the two situations.

These similarity ratings for situation A (s_A) and B (s_B) were then correlated to the similarity indices s_g and s_d. Pearson's correlation coefficient r was used. In order to establish convergent validity, high correlations were expected between similarity rating in situation A ('general introduction') and similarities in general job competencies (r(s_g, s_A)) and between similarity ratings in situation B ('professional introduction') and similarity in domain specific competencies (r(s_g, s_B)). The other pairs of intercorrelations were expected to be low (discriminant validity).

[Table 4] shows mean correlation coefficients for these four cases which are of moderate magnitude. Bold coefficients were expected to be higher than the other coefficients. The table shows that coefficients are in the expected direction, however, the differences are only marginal. In fact, of the 14 comparisons made (two for each of the 7 participants), 8 were in the expected direction, and 6 were in the opposite direction, which is not a significant result.

These results show that (1) the similarity ratings of the respondents are in fact moderately related to the competencies that have been assigned to the documents which establishes convergent validity for our methodology. However, discriminant validity could not be shown. This is either due to the classification of the competencies or to the situation manipulation. We will pursue this further in using other kinds of criteria, for example the pre established document categories.

5 Conclusion

We have presented a formalisation for employee competencies which is based on a set theoretic approach and a psychological conception of human competencies and performance and a methodology of how to assess employee competencies in dynamic work domains. We have shown how the resulting structures can be used for development planning and validated in an organisational setting.

Using documents that employees have created reflects their dynamic work assignments. Furthermore, documents are mostly structured into a certain workflow so that additional information about required competencies can be derived from the requirements of the workflow process. In our case, there was a specific project workflow (i.e. project offer, project definition, project close-out etc.), in which most of the documents were produced. This gives indication of the tasks involved and provides a further source of information for competence-performance analysis.

Moreover, because written documentation of work results is so important, most knowledge management systems include a document management component to which documents are being stored and from which they are retrieved, in order to make better use of the available information. By using documents in the present approach, it may be possible to offer a way by which automatic profiling of such systems can be improved.

The methodology we have described is useful for the case where no existing competence-performance structures are available, that is when a company is just engaging in a competency management effort for the first time. However, different methods would be required for updating the structures. It is an important feature of our approach that in the case of updating the structures the task would be much easier than the initial effort: Because the structures exploit prerequisite relations inherent in the tasks and in the competencies, it is possible to support many jobs that human actors would usually be required to do and which typically make competency management such tedious work. Once the prerequisite relations within the set of documents and the set of competencies have been found, adapting the space dynamically becomes much easier.

For example, if an employee has created a new document, a system using existing structures would have to ask a lot less document-competency assignments from the employee to update his or her competency profile than in the cases where no existing structures are available.

Consequently, it may be feasible to integrate such a system into a document management component. Combining the two would then offer ways to continuously update employee profiles based on the documents employees create and (few) additional questions they have to answer.

[Albert and Kaluscha 1997] have shown how known methods of adapting competence-performance structures can also be used in dynamic domains. It remains to be shown to what extent automation can reduce the efforts required for maintaining the structures. This certainly depends on the speed of change of the tasks. We would argue, that it would also depend on the nature of the competencies defined. Since we understand competencies as being relatively stable across time and situations, it is our expectation that underlying competencies would in fact change much slower and that rather different combinations of competencies would be able to determine performance for a person in quite diverse kinds of situations.

The competence-performance matrices can be the basis for other kinds of analyses as well. For example, Formal Concept Analysis (FCA, [Ganter and Wille 1999]) provides an alternative way of formalisation for documents and competencies. [Busch et al. 2001] have used FCA for mapping knowledge flows in an organisation. In general FCA seems especially well suited for knowledge mapping purposes, that is making available competencies visible within the company. This corresponds to the second kind of instrument we envision as an area of application for our work. Further developments are required and are currently being undertaken in an effort to extend applicability of the approach while at the same time taking into consideration different requirements of development planning and competency mapping [Wöls et al. 2003].

Acknowledgements

The Know-Center is a Competence Center funded within the Austrian Competence Center program K plus under the auspices of the Austrian Ministry of Transport, Innovation and Technology (www.kplus.at). Comments on a draft of this paper by the Computational Science Group at Graz University are kindly acknowledged.

References

[ALEKS Corp. 2003] ALEKS Corp.: "ALEKS - A Better State of Knowledge [WWW Document]"; http://www.aleks.com (2003).

[Alan and Zalewski 2003] Alan, Y., Zalewski, S.: "Ontologiebasierte Wissensräume [Ontology-based Knowledge Spaces]"; Proc. of WM 2003: Professionelles Wissensmanagement - Erfahrungen und Visionen, Gesellschaft für Informatik, Bonn (2003), 5-8.

[Albert and Kaluscha 1997] Albert, D., Kaluscha, R.: "Adapting knowledge structures in dynamic domains"; Beiträge zum Achten Arbeitstreffen der GI-Fachgruppe 'Intelligente Lehr-/Lernsysteme', Technical University of Munich, (1997), 89-100.

[Albert and Lukas 1999] Albert, D., Lukas, J.(Ed.) : "Knowledge Spaces: Theories, Empirical Research, and Applications"; Lawrence Erlbaum Associates, Mahwah, New Jersey (1999).

[Armstrong 1999] Armstrong, M.: "A Handbook of Human Resource Management Practice"; Kogan Page, London (1999).

[Bergmann 2000] Bergmann, B.: "Arbeitsimmanente Kompetenzentwicklung"; In Bergmann, B., Fritsch, A., Göpfert, P., Richter, F., Wardanjan, B., Wilczek, S. (Eds.): "Kompetenzentwicklung und Berufsarbeit", Waxmann, Münster (2000), 11-39.

[Bina and Newkirk 1998] Bina, M., Newkirk, J.: "Competencies at the Rock - Creating a Competency-Based Integrated Human Resource System for Prudential Health Care Group"; In Dubois, D. D. (Eds.): "The Competency Case Book", HRD Press, Amherst (1998), 93-120.

[Boyatzis 1982] Boyatzis, R. E.: "The Competent Manager: A Model for Effective Performance"; John Wiley & Sons, New York (1982).

[Brewer 2000] Brewer, M. B.: "Research Design and Issues of Validity"; In Reis, H. T., Judd, C. M. (Eds.): "Handbook of Research Methods in Social and Personality Psychology", Cambridge University Press, Cambridge (2000), 3-16.

[Busch et al. 2001] Busch, P., Richards D.; Dampney,, Dampney, C.: "Mapping Tacit Knowledge Flows in Organisation X"; Proc. of the 12th Australasian Conf. on Information Systems, December 5-7, Coffs Harbour/Australia, (2001), 85-94.

[Conlan et al. 2002] Conlan, O., Hockemeyer, C., Wade, V., Albert, D.: "Metadata driven approaches to facilitate adaptivity in personalized eLearning systems"; The Journal of Information and Systems in Education, 1 (2002), 38-44.

[Deiters et al. 2000] Deiters, W., Lucas, R., Weber, T.: "Skill-Management, ein Baustein für das Management flexibler Teams"; Information Management and Consulting, 15 (2000).

[Doignon and Falmagne 1999] Doignon, J., Falmagne, J.: "Knowledge Spaces"; Springer, Heidelberg (1999).

[Dowling and Hockemeyer 2001] Dowling, C., Hockemeyer, C.: " Automata for the Assessment of Knowledge"; IEEE Transactions on Knowledge and Data Engineering, 13, 3 (2001), 451-461.

[Eppler 2001] Eppler, M. J.: "Making Knowledge Visible Through Intranet Knowledge Maps: Concepts, Elements, Cases"; IEEE Computer Society, Los Alamitos (2001).

[Erpenbeck and Heyse 1999] Erpenbeck, J., Heyse, V.: "Die Kompetenzbiographie: Strategien der Kompetenzentwicklung durch selbstorganisiertes Lernen und multimediale Kommunikation"; Waxmann, Münster (1999).

[Faix et al. 1991] Faix, W. G., Buchwald, C., Wetzler, R.: "Skill-Management: Qualifikationsplanung für Unternehmen und Mitarbeiter"; Gabler, Wiesbaden (1991).

[Ganter and Wille 1999] Ganter, B., Wille, R.: "Formal Concept Analysis: Mathematical Foundations"; Springer, Heidelberg (1999).

[Goodman and Kruskal 1954] Goodman, L. A., Kruskal, W. H.: "Measures of Association for Cross Classifications"; Journal of the American Statistical Association, 49 (1954), 732-764.

[Green 1999] Green, P. C.: "Building Robust Competencies: Linking Human Resource Systems to Organizational Strategies"; Jossey-Bass, San Francisco (1999).

[Hulth et al. 2001] Hulth, A., Karlgren, J., Jonsson, A., Boström, H., Asker, L.: "Automatic Keyword Extraction Using Domain Knowledge"; Springer, Heidelberg (2001), 472-483.

[Jankowicz 1990] Jankowicz, A. D.: "Applications of Personal Construct Psychology in Business Practice"; In Neimeyer, G. J., Neimeyer, R. A. (Eds.): "Advances in Personal Construct Psychology", JAI Press, Greenwich (1990), 257-287.

[Kelly 1955] Kelly, G.: "The Psychology of Personal Constructs"; Norton, New York (1955).

[Korossy 1997] Korossy, K.: "Extending the theory of knowledge spaces: A competence-performance approach"; Zeitschrift für Psychologie, 205 (1997), 53-82.

[Korossy 1999] Korossy, K.: "Modeling Knowledge as Competence and Performance"; In Albert, D., Lukas, J. (Eds.): "Knowledge Spaces: Theories, Empirical Research, and Applications", Lawrence Erlbaum Associates, Mahwah, New Jersey (1999), 103-132.

[Lawler and Ledford 1992] Lawler, E., Ledford, G.: "A skill-based approach to human resource management"; European Management Journal, 10, 4 (1992), 383-391.

[Ley and Albert 2003] Ley, T., Albert, D.: "Kompetenzmanagement als formalisierbare Abbildung von Wissen und Handeln für das Personalwesen

[Competence management - formalizing knowledge and action for human resource management] "; Wirtschaftspsychologie, 5, 3 (2003), 86-93.

[Lucia and Lepsinger 1999] Lucia, A. D., Lepsinger, R.: "Competency Models: Pinpointing critical success factors in organizations"; Jossey-Bass, San Francisco (1999).

[McDaniel, D. L. 1998] McDaniel, D. L.: "A Competency Model for Human Resources"; In Dubois, D. D. (Eds.): "The Competency Case Book", HRD Press, Amherst (1998), 121-157.

[McDonald and Ackerman 2000] McDonald, D. W., Ackerman, M. S.: "Expertise Recommender: A Flexible Recommender System and Architecture"; ACM press, New York (2000), 231-240.

[Michellone and Zollo 2000] Michellone, G., Zollo, G.: "Competencies management in knowledge-based firms"; Internqational Journal of Technology Management, 20, 1/2 (2000), 134-155.

[Probst et al. 2000] Probst, G. J. B., Deussen, A., Eppler, M. J., Raub, S. P.: "Kompetenz-Management: Wie Individuen und Organisationen Kompetenz entwickeln"; Gabler, Wiesbaden (2000).

[Rollett et al. 2001] Rollett, H., Ley, T., Tochtermann, K.: "Supporting Knowledge Creation: Towards a Tool for Explicating and Sharing Mental Models"; Proc. of ECKM '01, IEDC Bled School of Management, Bled, Slovenia (2001).

[Rost 1996] Rost, J.: "Lehrbuch Testtheorie, Testkonstruktion

[Test Theory, Test Construction]"; Hans Huber, Bern (1996).

[Schweitzer 2001] Schweitzer, E.: "Expertenbefragung auf Basis der Kompetenz-Performanz-Theorie: Entwicklung einer Methode am Beispiel der Rechtschreibung

[Questioning experts on the basis of the competence-performance theory: development of a method for orthography]"; Unpublished Masters Thesis, Graz University, (2001).

[Spencer and Spencer 1993] Spencer, L. M., Spencer, S. M.: "Competence at Work: Models for Superior Performance"; John Wiley & Sons, New York (1993).

[Stefanutti and Albert 2002] Stefanutti, L., Albert, D.: "Efficient assessment of organizational action based on knowledge space theory"; Proc. of I-Know '02, Know-Center, Graz (2002), 183-190.

[Stefanutti and Albert 2003] Stefanutti, L., Albert, D.: "Skill Assessment in Problem Solving and Simulated Learning Environments"; Journal of Universal Computer Science, 9, 12 (2003), 1455-1468.

[Stenmarck 2001] Stenmarck, D.: "Leveraging Tacit Organisational Knowledge"; Journal of Management Information Systems, 17, 3 (2001), 9-24.

[Won and Pipek 2003] Won, M., Pipek, V.: "Sharing Knowledge on Knowledge - The eXact Peripheral Expertise Awareness System"; Journal of Universal Computer Science, 9, 12 (2003), 1388-1397.

[Wöls et al. 2003] Wöls, K., Kirchpal, S., Ley, T.: "Skills Management - An 'all-purpose' Tool?"; Proc. of I-Know '03, Know-Center, Graz (2003), 138-143.