OntoShare - An Ontology-based Knowledge Sharing System
for virtual Communities of Practice1
(BTexact, Orion 5-12, Adastral Park,
Ipswich IP5 3RE, UK
(BTexact, Orion 5-12, Adastral Park,
Ipswich IP5 3RE, UK
(Institute AIFB, University of Karlsruhe
D-76128 Karlsruhe, Germany
Abstract: An ontology-based knowledge sharing system OntoShare
and its evaluation as part of a case study is described. RDF(S) is are
used to specify and populate an ontology, based on information shared between
users in virtual communities. We begin by discussing the advantages that
use of Semantic Web technology afford in the area of knowledge management
tools. The way in which OntoShare supports WWW-based communities of practice
is described. Usage of OntoShare semi-automatically builds an RDF-annotated
information resource for the community (and potentially for others also).
Observing that in practice the meanings of and relationships between concepts
evolve over time, OntoShare supports a degree of ontology evolution based
on usage of the system - that is, based on the kinds of information users
are sharing and the concepts (ontological classes) to which they assign
this information. A case study involving OntoShare was carried out. The
evaluation exercise for this case study and its results are described.
We conclude by describing avenues of ongoing and future research.
Key Words: Knowledge Sharing Community, Semantic Web
Category: H.4.3 [Information Systems Application]: Communications
Applications - Information browsers
There are now several billion documents on the WWW, which are used by
more than 300 million users globally, and millions more pages on corporate
intranets. The continued rapid growth in information volume makes it increasingly
difficult to find, organize, access and maintain the information required
short version of this article was presented at the IKNOW '03 (Graz,
Austria, July 2-4, 2003).
A semantic web that provides enhanced information access based on the
exploitation of machine-processable metadata has been proposed [BLHL01].
We are particularly interested in the new possibilities afforded by semantic
web technology in the area of knowledge management and we discuss this
below before moving on in the rest of the paper to describe OntoShare,
a system for supporting Semantic Web-based communities of practice.
Central to the vision of the Semantic Web are ontologies. Ontologies
are seen as facilitating knowledge sharing and re-use between agents, be
they human or artificial [Fen01]. They offer this
capability by providing a consensual and formal conceptualization of a
given domain. As such, the use of ontologies and supporting tools offer
an opportunity to significantly improve knowledge management capabilities
in large organizations and it is their use in this particular area that
is the subject of this paper. In OntoShare, an ontology specifies a hierarchy
of concepts (ontological classes) to which users can assign information.
In this process, important metadata is extracted and associated with the
community information resource using RDF annotations.
1.2 Knowledge Management, Communities of Practice & the Semantic
The notion of communities of practice [SBD91] has
attracted much attention in the field of knowledge management. Communities
of practice are groups within (or sometimes across) organizations who share
a common set of information needs or problems. They are typically not a
formal organizational unit but an informal network, each sharing in part
a common agenda and shared interests or issues. In one example it was found
that a lot of knowledge sharing among copier engineers took place through
informal exchanges, often around a water cooler. As well as local, geographically
based communities, trends towards flexible working and globalization has
led to interest in supporting dispersed communities using Internet technology
[Dav00]. The challenge for organizations is to support
such communities and make them effective. Provided with an ontology meeting
the needs of a particular community of practice, knowledge management tools
can arrange knowledge assets into the predefined conceptual classes of
the ontology, allowing more natural and intuitive access to knowledge.
Knowledge management tools must give users the ability to organize information
into a controllable asset. Building an intranet-based store of information
is not sufficient for knowledge management; the relationships within the
stored information are vital. These relationships cover such diverse issues
as relative importance, context, sequence, significance, causality and
association. The potential for knowledge management tools is vast; not
only can they make better use of the raw information already available,
but they can sift, abstract and help to share new information, and present
it to users in new and compelling ways.
In this paper, we describe the OntoShare system which facilitates and
encourages the sharing of information between communities of practice within
(or perhaps across) organizations and which encourages people - who may
not previously have known of each other's existence in a large organization
- to make contact where there are mutual concerns or interests. As users
contribute information to the community, a knowledge resource annotated
with metadata is created. Ontologies are defined using RDF Schema (RDFS)
and populated using the Resource Description Framework (RDF). (RDF [LS99]
is a W3C recommendation for the formulation of metadata for WWW resources.
RDFS [LS99] extends this standard with the means to
specify domain vocabulary and object structures - that is, concepts and
the relationships that hold between them). In the next section, we describe
in detail the way in which OntoShare can be used to share and retrieve
knowledge and how that knowledge is represented in an RDF-based ontology.
We then proceed to discuss in Section 3 how the ontologies
in OntoShare evolve over time based on user interaction with the system
and motivate our approach to user-based creation of RDF-annotated information
resources. We describe in Section 4 the deployment
and evaluation of OntoShare in a particular community as part of a case
study in the project On-To-Knowledge (see http://www.ontoknowledge.org).
In Section 5 we first sketch the underlying methodology,
then we illustrate the application and evaluation of OntoShare according
to the methodology. We provide results of a user-focused evaluation of
OntoShare in Section 6. Before concluding, we present
further and related work.
2 Sharing and Retrieving in OntoShare
OntoShare is an ontology-based WWW knowledge sharing environment for
a community of practice that models the interests of each user in the form
of a user profile. In OntoShare, user profiles are a set of topics or ontological
concepts (classes declared in RDFS) in which the user has expressed an
interest. OntoShare has the capability to summarize and extract key words
from WWW pages and other sources of information shared by a user and it
then shares this information with other users in the community of practice
whose profiles predict interest in the information. OntoShare is used to
store, retrieve, summarize and inform other users about information considered
in some sense valuable by an OntoShare user. This information may be from
a number of sources: it can be a note typed by the user him/herself; it
can be an intra/Internet page; or it can be copied from another application
on the user's computer. As we will see be low, OntoShare also modifies
a user's profile based on their usage of the system, seeking to refine
the profile to better model the user's interests.
2.1 Sharing Knowledge in OntoShare
When a user finds information of sufficient interest to be shared with
their community of practice, a 'share' request is sent to OntoShare via
the Java client that forms the interface to the system. OntoShare then
invites the user to supply an annotation to be stored with the information.
Typically, this might be the reason the information was shared or a comment
on the information and can be very useful for other users in deciding which
information retrieved from the OntoShare store to access. At this point,
the system will also match the content being shared against the concepts
(ontological classes) in the community's ontology. Each ontological class
is characterized by a set of terms (keywords and phrases) and the shared
information is matched against each concept using the vector cosine ranking
algorithm [Har92]. The system then suggests to the
sharer a set of concepts to which the information could be assigned. The
user is then able to accept the system recommendation or to modify it by
suggesting alternative or additional concepts to which the document should
When information is shared in this way, OntoShare performs four tasks:
- An abridgement of the information is created, to be held on the user's
local OntoShare server. OntoShare's summarizer extracts key theme sentences
from the document. It is based on the frequency of words and phrases within
a document, using a technique based on lexical cohesion analysis [AB95].
Access to this locally held summary enables a user to quickly assess the
content of a page from a local store before deciding whether to retrieve
the (larger amount of) remote information.
- The content of the page is analyzed and matched against every user's
profile in the community of practice. As when recommending concepts to
the user, the vector cosine ranking model is used: here, however, the shared
information is matched against the set of terms (words and phrases) created
from the union of all terms associated with the concepts to which has user
has subscribed (i.e. the concepts which make up the user profile). If the
profile and document match strongly enough, OntoShare emails the user,
informing him or her of the page that has been shared, by whom and any
annotation added by the sharer.
- The information is also matched against the sharer's own profile in
the same way. If the profile does not match the information being shared,
the system will suggest one or more concepts which strongly match the shared
information that the user can then add to their profile. Thus OntoShare
has the capability to adaptively learn users' interests by observing user
- For each document shared, an instance of the class Document
is created, with properties holding metadata including keywords, an abridgement
of the document, document title, user annotation, universal resource locator
(URL), the sharer's name and date of storage.
(The ontological structure of the OntoShare store is shown in detail
in the next section.) In this way, a shared and enhanced information resource
is built up in OntoShare based on user contributions.
Given that users must make a conscious decision to store information,
the quality of the information in the OntoShare store is high - it is effectively
pre-filtered by OntoShare users. Thus each user leverages the assessment
of the information made by all the other users.
Figure 1: Ontological Structure in OntoShare
2.2 Ontological Representation
We said above that each piece of shared information leads to the creation
of a new entry in the OntoShare store and that this store is effectively
an ontology represented in RDFS and RDF. We now set this out in more detail.
RDFS is used to specify the classes in the ontology and their properties.
RDF is then used to populate this ontology with instances as information
is shared. Figure 1 shows a slightly simplified version
of the ontology for a community sharing information about the Semantic
Web, along with an example of a single shared document ("Document_1").
It is not our intention to describe each class and property and their
function here but we will mention a few key aspects. Firstly, notice Concept
and its subclasses: this is the set of concepts which the community of
practice at hand is interested in. Note that in the current version of
OntoShare, the concept structure is limited to a strict hierarchy. Another
key class is Document, which is the class used to represent shared
information: each document shared generates an instance of Document
with the set of properties shown. Document_1, for example, was stored
by John Smith into the concept RDF with the annotation "RDF tutorial
for beginners ..." with the summary and URI as shown in Figure
1. It also has a set of keywords associated with it. (For simplicity,
note that here we show only one keyword Kw_1, which is an instance
of the class Keyword, as is Kw_2 and furthermore it is not
shown that Keyword is a subclass of rdfs#Resource ). The
third central class is Profile, instances of which represent user
information, including the concepts in which they are interested, their
names and email addresses. Profile_1, for example, is the profile
of a user with name "John Smith". Finally, note that keyword
Kw_2 is one of (possibly many) terms (words and phrases) which characterize
the concept Language. Below we include excerpts from the RDFS and
RDF (in XML notation) used to represent the ontology depicted above. We
see the declarations of the classes Document, Profile and
Keyword in RDFS, followed by the descriptions of Document_1
and the user profile of John Smith in RDF.
<?xml version = " 1.0 " encoding="UTF-8" ?>
<!--#############RDFS SCHEMA ############# -->:
<rdfs:Class rdf:ID="Document " />
<rdfs:Class rdf:ID="Profile " />
<rdfs:Class rdf:ID="Keyword" />
<!-- Document properties -->
<rdf:Property rdf : ID="submitted_by ">
<rdfs:domain rdf:resource="#Document " />
<rdfs:range rdf:resource="#Profile " >
... ... ...
<!-- ############ RDF DATA ############ -->
<Document rdf: ID="Document_1">
<title>RDF Tutorial</title >
<summary>doc summary goes here</summary>
<isAbout rdf:resource="#RDF" ontoshare:ID="7" />
<annotation>RDF tutorial for beginners ... </annotation>
<user_name>John Smith</user name>
<interestedIn rdf:resource="#Sesame " ontoshare:ID="5" />
<interestedIn rdf:resource="#Tools " ontoshare: ID="2" />
... ... ...
2.3 Retrieving explicit knowledge in OntoShare
In this section, we discuss the ways in which OntoShare facilitates
access to and the automatic sharing of the information.
Email notification. As described above, when information is shared
in On-toShare, the system checks the profiles of other users in the community
of which the user is a member. If the information matches a user's profile
sufficiently strongly, an email message is automatically generated and
sent to the user concerned, informing the user of the discovery of the
information. Thus in cases where a user's profile indicates that they would
have a strong interest in information shared, they are immediately and
proactively informed about the appearance of the information.
Searching the community store - accessing information and people.
Via their OntoShare home page, a user can supply a query in the form
of a set of key words and phrases in the way familiar from WWW search engines.
Figure 2: Typical OntoShare Home Page
OntoShare then retrieves the most closely matching pages held in the
OntoShare store, using a vector space matching and scoring algorithm [Har92].
The system then displays a ranked list of links to the pages retrieved
and their abridgements, along with the scores of each retrieved page and
any annotation made by the original sharer is also shown. Importantly,
the user can elect to simultaneously search for other users by selecting
the appropriate check box. We will have more to say about this capability
to identify other users as well as information in Section
4 when we look at accessing tacit knowledge via other users using OntoShare.
Personalised Information. A user can also ask OntoShare to display
"Documents for me" as shown in the top right pane of Figure
2 below. The system then interrogates the OntoShare store and retrieves
the most recently stored information. It determines which of these pages
best match the user's profile. The user is then presented with a list of
links to the most recently shared information, along with a summary, annotations
where provided, date of storage, the sharer and an indication of how well
the information matches the user's profile (the thermometer-style icon
in Figure 2).
In addition, 2 buttons are provided (on the button bar at the bottom
of the screen in Figure 2) so that the user can indicate
interest or disinterest in a particular piece of information - this feedback
will be used to modify the user's profile. At this point, the system will
match the content of the current document against each concept (ontological
class) in the community's ontology. As described above, each ontological
class is characterized by a set of terms (keywords and phrases) and the
shared information is matched against he term set of each concept using
the vector cosine ranking algorithm [Har92]. The system
then identifies the set of zero or more concepts that match the information
above a given ranking threshold and suggests to the sharer that this set
of concepts be added to or removed from their profile in the cases of user
interest or disinterest respectively. The user is then free to accept the
system recommendation or to modify it by selecting from the set of suggested
Two further operations are possible on documents presented to the user.
These operations are selected from the "Documents" menu. Firstly,
a user can add their own annotation to information stored by another user.
Secondly, a user can request that OntoShare identifies other users with
an interest in the information under consideration.
This "Documents for me" information is in fact displayed on
the user's On-toShare home page, so that whenever they access the system,
they are shown the latest information. Figure 2 is
a typical OntoShare home page.
3 Creating Evolving Ontologies
So far we described how, when a user shares some information, the system
will match the content being shared against each concept (class) in the
community's ontology. Recall that each ontological class is characterized
by a set of terms (key-words and phrases) and that following the matching
process, the system suggests to the sharer a set of concepts to which the
information could be assigned. The user is then able to accept the system
recommendation or to modify it by suggesting alternative concept(s) to
which the document should be assigned. It is at this point that an opportunity
for ontology evolution arises.
Should the user indeed override the system's recommended classification
of the information being shared, the system will attempt to modify the
ontology to better reflect the user's conceptualization, as follows. The
system will extract the keywords and key phrases from the information using
the ViewSum system mentioned above. The set of such words and phrases are
then presented to the user as candidate terms to represent the class to
which the user has assigned the information. The user is free to select
zero or more terms from this list and/or type in words and phrases of his
own. The set of terms so identified is then added to the set of terms associated
with the given concept, thus modifying its characterization.
We call this approach usage-based ontology evolution and in this way
the characterization of a given concept evolves over time, this evolution
being based on input from the community of users. We believe that this
ability to change as users' own conceptualization of the given domain changes
is a powerful feature which allows the system to better model the consensual
ontology of the community. Clearly, this level of evolution is limited
to changing the semantic characterization of ontological classes and does
not support, for example, the automatic suggestion of new classes to be
added to the ontology. More advanced ontology evolution is the subject
of ongoing research and is described briefly in Section
5. It is also worth noting that we have not concerned ourselves with
ontology versioning (tracking and managing changes to an ontology) here.
This is an important issue with regard to ontology evolution and the reader
is referred to, for example, [SMMS02] for details
of work in this area.
As well as usage-based evolution, we have seen above how users also
indirectly annotate the information as a side-effect of sharing it with
the community and we discuss and motivate this approach below. Pragmatically
speaking, it is the case at the time of writing that only a very small
proportion of WWW- and intranet-based information resources are annotated
with RDF (meta)data. It is therefore beneficial to provide a system wherein
such annotation effectively occurs as a side-effect of normal usage. Another
important observation is that it is in the general case impossible to cover
the content of a document exhaustively by an RDF description. In practice,
RDF descriptions can never replace the original document's content: any
given RDF description of a set of resources will inevitably give one particular
perspective on the information described. Essentially, a metadata description
can never be complete since all possible uses for or perspectives on data
can never enumerated in advance. Our approach accommodates this observation
however in the sense that each community will create its own set of metadata
according to its own interest in and perception of information that is
added to its store. It is very possible that the same information could
be shared in two separate communities and emerge with different metadata
annotations in each.
4 Expertise Location and Tacit Knowledge
In Section 2, we focused on the technical aspects
of OntoShare and on the sharing and storing of explicit knowledge. Explicit
knowledge we take to be that knowledge which has been codified in some
way. This codification can take place in many different media (paper, WWW
page, audio, video, and so on). In the context of OntoShare, by explicit
knowledge, we mean the information shared in On-toShare, along with the
meta-information associated with it such as the sharer, the annotations
attached to it, and so forth. We now turn to the social aspects of the
system and tacit knowledge.
A large amount of the knowledge within an organization may of course
not be codified: it may be personal, context-specific and difficult to
write down, and may be better transmitted through a master-apprentice "learning
by watching and copying" arrangement. Such knowledge is referred to
as tacit knowledge [Pol58]. When tacit knowledge is
difficult to make explicit (codify), we need to find new ways of transmitting
the knowledge through an organization. Failure to do so can lead to loss
of expertise when people leave, failure to benefit from the experience
of others, needless duplication of a learning process, and so on.
One way in which a system such as OntoShare can encourage the sharing
of tacit knowledge is by using its knowledge of the users within a community
of practice to put people who would benefit from sharing their (tacit)
knowledge in touch with one another automatically.
One important way we gain new insights into problems is through 'weak
ties', or informal contacts with other people [Han97].
Everyone is connected to other people in social networks, made up of stronger
or weaker ties. Stronger ties occur between close friends or parts of an
organization where contact is maintained constantly. Weak ties are those
contacts typified by a 'friend of a friend' contact, where a relationship
is more casual. Studies have shown that valuable knowledge is gathered
through these weak ties, even over an anonymous medium such as electronic
mail and that weak ties are crucial to the flow of knowledge through large
organizations. People and projects connected to others through weak ties
are more likely to succeed than those not [Han97].
User profiles can be used by the OntoShare system to enable people to
find other users with similar interests. The user can request OntoShare
to show a list of people with similar interests to themselves. OntoShare
then compares their profile with that of every user in the store and a
list of names of users whose interests closely match their own. Each name
is represented as a hypertext link which when clicked initiates an email
message to the named user. Recall that profiles in OntoShare are a set
of phrases and thus the vector space model can be used to measure the similarity
between two users. A threshold can then be used to determine which users
are of sufficient similarity to be deemed to 'match'.
This notion is extended to allow a user to view a set of users who are
interested in a given document. OntoShare determines which members of the
community 'match' the relevant document above a predetermined threshold
figure and presents back to the user a list of user names. As before, these
names are presented as hypertext links, allowing the user to initiate an
email message to any or all of the users who match the document. In addition,
as already mentioned in Section 2.3, a user can carry
out a keyword search on other users and thus identify users with an interest
in a particular subject. In this way, OntoShare, while not claiming to
actually capture tacit knowledge, provides an environment which actively
encourages the sharing of tacit knowledge, perhaps by people who previously
would not otherwise have been aware of each other's existence.
Figure 3: Core Steps of the On-To-Knowledge Methodology
5 Case Study & Evaluation
OntoShare has recently been applied and evaluated using the On-To-Knowledge
methodology [SSS04]. Core to the process oriented
methodology are "Knowledge Meta Processes" and "Knowledge
Processes". While the former support the setting up of ontology based
knowledge management applications, the latter one support their usage.
A core element of the former that can not be shown in detail here is a
lifecycle for ontology engineering. In the following we will (i) briefly
sketch main steps of the "Knowledge Meta Process" (cf. Figure
3) and (ii) illustrate the way in which each of the steps were implemented
together with the results of the exercise.
5.1 Methodology Feasibility Study
Here the problem and opportunity area and potential solutions are identified.
These are then put into a wider organizational perspective. The intention
is to select the most promising focus area and target solution for the
Kickoff. In this phase, the actual development of an ontology
begins. A requirements specification of the ontology is developed which
includes identifying the resources that will contribute to the ontology
e.g. existing ontologies that may be re-used & valuable personnel that
could contribute to ontology building.
The domain and goal of the ontology should also be defined at this stage
e.g. the ontology will be used in a knowledge sharing application in the
domain of Knowledge Management. Design guidelines should be defined to
help users who are not familiar with modeling ontologies. These might include
an estimation of the number of concepts and the level of granularity required.
The outcome of the kickoff phase should be an ontology requirements specification.
Refinement. This phase involves developing a taxonomy (i.e. modeling
is-a relations) and then adding additional relations (other than is-a)
to form a more rich ontology. This is generally a cyclical process with
knowledge about the domain captured from the domain experts using brainstorming
Evaluation. There are three forms of evaluation in the methodology.
These are technology-focused evaluation, user-focused evaluation and ontology-focused
evaluation. (i) Technology-focused evaluation is concerned with evaluating
both the properties of the ontology developed and the tools and applications
used. Criteria such as language conformance, scalability, performance and
interoperability are used. (ii) User-focused evaluation should encompass
feedback from users of a prototype tool in the target application environment,
usage patterns of the ontology and most importantly whether the ontology
based technologies used are at least as good as already existing technologies.
(iii) Finally ontology-focused evaluation is concerned with formally evaluating
ontologies against a set of rules.
Application & Evolution. The final phase of the methodology
is concerned with applying the ontology to its intended domain and managing
its evolution. Both centralized and distributed strategies can be employed.
As shown in Figure 3, the final three phases of the
methodology are cyclical. It is to be expected that improvements to the
tools and changes to the ontology will be introduced in an iterative manner.
5.2 Feasibility Study
This involved identifying both an appropriate tool-set and a suitable
user group. The user group for the study consisted of approximately 30
researchers, devel opers and technical marketing professionals from
the research and development arm of a large telecommunications firm. The
interests of the users fell into 3 main groupings: conferencing, knowledge
and information management and personalization technologies. It was felt
that three separate yet overlapping topic areas would constitute an interesting
mix of interests for the purposes of the trial. The tool-set consisted
of OntoShare (which was the only tool to be used by the users), OntoEdit
[SAS03] and Sesame [BKv02]. OntoEdit
was used to capture the domain ontology produced by the experts.
Sesame was used to store this ontology and the OntoShare data store
(of added items, annotations, etc.) to provide access for other ontology-based
tools. OntoShare allowed the ontology to automatically evolve and extend
over the course of the study as documents were added. The effectiveness
of this evolutionary process will be considered in the evaluation exercise.
An interesting secondary outcome was to look at is the extent to which
the ontology built up by the community is useful to other users in other
contexts. Work on a searching and browsing facility over the community's
information for other users outside the community is described elsewhere
5.3 Kickoff and Refinement
The Kickoff and Refinement stages of the methodology were encompassed
by an ontology building process. This majority of this process was carried
out at a workshop held at the case study company's premises and run by
a knowledge engineer. A selection of 6 key people from the user group were
invited to attend the workshop. It was felt that as a whole, they would
be able to cover the domain of interest for the whole of the user group.
The workshop included a presentation that described in basic terms what
an ontology is and how they can be used. It also included a demonstration
of the OntoShare tool which introduced the tool to the users as well as
showing the use of an Ontology in a practical application. Following this,
the ontology building process took place. It was very much brainstorming
oriented during in the kickoff [SSS04, SAS03].
The following is a description of the steps that were carried out in the
- A brief discussion took place on the number of concepts that was felt
to be appropriate. Two suggestions of 20 and 50 were made. These were made
in the context of the OntoShare tool and how its interface dealt with the
- Each participant was asked to come up with 5+ topics that they felt
were important to them. These were then collected. Some early organisation
of the hierarchy took place at this stage .
- Further organisation into a hierarchy then took place. This included
adding and removing concepts and moving whole subbranches of the proposed
- The depth of the hierarchy was considered to be too deep. The UI aspects
were considered at this stage and a decision taken to restrict the depth
to a maximum of three levels where possible.
- It was remarked that organisational groupings (i.e. knowledge management,
conferencing, etc.) had been introduced at the top level of the hierarchy.
It was decided that this might introduce unwanted boundaries for the users
and that they might feel that they could only add documents to their particular
part of the hierarchy. Also, some of the subbranches straddled the
toplevel areas. The toplevel was then removed.
- The number of concepts across each level was considered in terms of
the UI. It was felt that 10-15 concepts at each level would be manageable
by the user. Concepts of minor importance were then removed or combined
at this stage in order to achieve this.
- The suitability of the overall ontology was then considered and a few
addi tional refinements were made.
The resulting ontology contained 10 top-level concepts and a total of
52 concepts with a maximum depth of three levels.
The group were able to produce this ontology at the workshop which meant
that most of the refinement stage (i.e. the organization of concepts and
relationships) had been carried out in tandem with the kickoff stage. Complex
relationships (i.e. those outside of the simple taxonomical ones) are not
included although these automatically develop as data items are added to
the OntoShare store and relationships between that data is inferred by
OntoShare itself. The over-riding message here is that user groups such
as the one in this trial can be expected to produce lightweight ontologies.
5.4 User-Focused Evaluation
Of the three forms of evaluation in the On-To-Knowledge methodology,
the most appropriate for use in the case study is user-focused. The tools
rely on a high degree of user interaction and as such the users are the
best resource for determining whether they meet their objectives. Various
user-focused evaluation methods were employed. This section will describe
these and the rationale for their use. The objectives of the evaluation
were to determine:
- What the users think of sharing knowledge in an environment such as
that used in the case study;
- whether the use of an ontology helps with the storing and sharing of
- whether the ontology evolution process is effective;
- whether the ontology developed as part of the case study was effective;
- the good and bad points of the knowledge sharing environment.
The principal means of evaluating the views of the users was with the
use of questionnaires. Questionnaires have the benefit of allowing the
views of a high proportion of the users to be canvassed without burdening
them a great deal.
The questionnaires consisted of a mixture of open questions that required
a qualitative response and a series of statements that required a quantitative
response indicating the level to which the respondent agreed or disagreed.
This mixed approach is endorsed by Eason [Eas88] who
states that "Structured questions have the virtue of easy analysis
and direct comparability. Their weakness is that they pre-define the answers
it is possible to give and may not therefore permit the user to report
the most important issues. We have always found it useful to use a structured
approach to reveal issues and, once an issue is located, to use an unstructured
method to explore the nature of the issue". A 'pre-trial' questionnaire
and a post-trial 'questionnaire' were developed.
The 'pre-trial' questionnaire was intended to determine the nature of
the users in the case study in terms of the way (and how often) they access,
receive and share information. The 'post-trial' questionnaire was intended
to extract the user's views of and experiences with the OntoShare system.
Particular focus was placed upon the users' views on the usage of an ontology
within the tool and the evolution of that ontology.
An additional form of evaluation involved the analysis of usage statistics
that were collected by an OntoShare module developed exactly for this purpose.
This was able to record every interaction that occurred on the OntoShare
server along with the user who performed it. This allows analysis to take
place on the use of different OntoShare functions by the group as a whole
as well as the behavior of individuals (which can then be cross-referenced
with the questionnaire responses). The combination of methods should allow
an evaluation of the individual OntoShare functions to be made. The usage
of the ontology can also be analyzed by recording the distribution of documents
added to the concepts in the ontology and the evolution of the ontology
over the course of the case study period.
The final form of evaluation was an expert usability analysis. This
involved an assessment of the OntoShare user interface and allowed a thorough
inspection to be made by an independent expert. It generally results in
a more objective and far reaching analysis than would be the case if it
was carried out by someone connected to the development of the tool.
5.5 Application & Evolution
This part of the methodology is concerned with use of the ontology in
its intended application and its evolution. It is cyclical with results
from the evaluation stage providing impetus for evolution. The delivery
of the application to the user group was carefully managed in order to
achieve the highest possible level of usage.
The actual delivery process is outside the scope of this paper. The
evolution of the ontology that was enabled by OntoShare was a topic of
the evaluation exercise.
6 Results of Case Study
This section will suggest and discuss a number of recommendations for
the evolution of the case study tools and deployment process. A set of
lessons learnt from the experience of carrying out the case study is presented.
Firstly, recommendations concerned with improving OntoShare are considered.
These have come to light as a direct result of the user-focused evaluation
Give careful consideration to the nature of the virtual community.
The evaluation showed that the majority of users were passive in their
use of OntoShare i.e. they were happy to receive e-mails and read documents
but did not add items to the system. As a result, the experience of the
majority of On-toShare users is determined by the actions of the few who
actually add items to the system. If those few users did not exist, the
knowledge sharing benefits would not be forthcoming. Depending on the local
organizational culture, dependence on a relatively small proportion of
the user community may or may not be appropriate. Many successful communities
are of this nature but in some cases alternative strategies may be required
to reduce the dependence upon active users. In their responses to questionnaires,
users made some useful suggestions in this regard. One user made the suggestion
that OntoShare needed to be "regularly seeded with potentially relevant
information or gain critical mass usage to offer a positive benefit and
justify the effort of maintaining profiles and entering articles or information".
This could be carried out manually by a knowledge engineer or automatically
by an information agent and could benefit the system by ensuring sufficient
data was added that was of interest to a wider cross section of the users.
Both methods have drawbacks in that the manual process is time-consuming
and the automatic method introduces the risk of downgrading the quality
of information that is shared. The recommendation is to experiment with
a combination of user, knowledge engineer and agent added data that can
be varied depending upon user input and the nature of the community.
Provide better interface access. A commonly occurring criticism
of the system was its use of a Java applet to provide the interface. This
proved to be slow to load and required a login step in order that the user
could be recognized. It was originally chosen over a HTML-based interface
because displaying the Ontology (with a collapsing folder structure) would
have proved difficult in that format. An alternative would be to provide
an application, however this needs to be installed and re-installed every
time a change is required which would have been disruptive for a system
in its infancy. Also, different versions are required for different operating
systems. The Java applet was so unpopular that using one of the alternatives
now seems more attractive.
to operate in a standard browser. If this is not appropriate then the use
of an installed application would probably be the best course of action.
Provide wider access to functions. Another drawback with the
system that was widely mentioned was the need to login to the system to
provide comments, add items, etc. Alternative methods of accessing individual
functions of OntoShare should be explored. These might include direct links
in notification e-mails to a comment adding facility and support in a web
browser for dropping a URL into the system. The intention should be to
reduce the effort that is required to use each function.
Richer ontological representation. The OntoShare ontology currently
contains concept relationships of the topic-subtopic type. Other relations
are inferred by OntoShare and the user can request to see these. A better
way of presenting these to the user is required. This might be a toggle
where the relationships are indicated on screen when it is turned on.
Provide better support to new users. When users first login they
are often daunted by the interface. Better support should be provided to
help them set up their profile and gain familiarity with the available
functions. This could be in the form of a 'splash screen' that is shown
on the first use of the tool or that can be disabled once users become
familiar. This would show tips on usage and a short description of each
Inform users about an ontology change. When changes to the ontology
are accepted or rejected by the users, they should be notified of the outcome
and invited to adjust their profiles accordingly. This will ensure that
users can gain access to items added to new areas without having to login
and browse the concepts. The following lessons in relation to the development
of ontologies have been learnt.
Physical presence is required. The approach used to produce a
domain ontology i.e. a group of experts in a focused workshop led by a
knowledge engineer who is physically present, proved to be fruitful. The
domain experts have limited time available, hence it is necessary to be
very focused. Had this capture been carried out over a period of time involving
a number of disparate people it would have probably been a drawn out process,
lacking in focus.
Domain experts can be expected to produce a taxonomy. A simple
ontology in the form of a taxonomy is the most likely outcome from a group
of domain experts asked to contribute in this way. More complex ontologies
require considerably more effort. The OntoShare system is suited to a simple
ontology with topic-subtopic relations.
The On-To-Knowledge methodology provides an effective frame-work
for the introduction of an ontology-based application. The application
of the methodology to the case study resulted in the rapid development
of an ontology that performed well in its intended application.
Users reported that they found it to be an appropriate ontology for
their domains despite the limited amount of time that was available for
In terms of the evaluation objectives introduced in Section
5.3, the following can be stated:
- Users are generally happy to receive shared knowledge from OntoShare
and will often read it if it appears interesting and they have enough time.
Only a minority of users actively share documents in OntoShare (mainly
due to time pressures) so steps should be taken to make it as easy as possible
to share. User shared data should be augmented with data added by a Knowledge
Engineer or an information agent.
- Users reported that they found it useful to have the ontology available
both when browsing for items and when adding items to the system.
- The ontology evolution process proved to be effective. Users were happy
to create new concepts and these were accepted by the other users. Users
reported that they found this facility useful although they were less inclined
to make use of the interesting - not interesting features to evolve individual
concept characterizations. Better access to facilities such as these has
been recommended in order to further their usage.
- The ontology that was developed proved to be effective in the case
study. It performed well in its intended application - measured by the
distribution of documents that were added to its concepts. Users reported
that they found it to be an appropriate ontology for their domains of interest.
- A number of good and bad points were identified which have been discussed.
These have led to a set of recommended improvements being made.
- The On-To-Knowledge methodology proved to be an effective framework
for the introduction of an ontology-based application. The application
of the methodology to the case study resulted in the rapid development
of an ontology that performed well in its intended application. Users reported
that they found it to be an appropriate ontology for their domains of interest
despite the limited amount of time that was available for ontology development.
The results of this evaluation will be used in the continued development
and exploitation of OntoShare and related tools.
7 Further and Related Work
Research and development of OntoShare is ongoing. A particular area
of focus currently is the ontological structure: a strict hierarchy (taxonomy)
of concepts about which the communities wants to represent and reason may
prove ultimately limiting and various possibilities for allowing a more
expressive concept map are under consideration.
One such is that OntoShare will be developed beyond the subtopic/supertopic
concept hierarchy with IsRelatedTo properties, allowing "horizontal"
links between concepts. The exploitation of this additional information
is again matter for further research. One proposal is that when seeking
to match users to other users, the system can use some notion of tree-matching,
taking into account the concepts in the users' profiles as well as not
only the subtopic links but also the IsRelatedTo links. These richer
ontologies may be better represented in a more expressive language such
as OWL, the W3C recommendation [SWM04]. A further
research area is the automatic identification and incorporation of new
concepts as they emerge in the community. Work on this is however at a
very early stage and is beyond the scope of this paper.
Turning to related work, Staab et al. [SAD + 00]
describe a system for building and maintaining community web portals. As
with OntoShare, a ontology-based is taken and an ontology is used to structure
and access information, using F-Logic as its underlying language for ontology
representation and querying. Relatively sophisticated querying is supported,
offering a degree of inferencing in the query engine not offered in OntoShare.
Semi-structured information provision is supported by the use of wrappers.
User profiling and automatic alerting are not supported, neither is the
ability to change the semantic characterization of a class as in OntoShare.
RiboWeb [ABC + 99] is another example of an ontology-based
community por tal RiboWeb holds information about ribosome data and computational
models for the processing thereof. Most data are scientific papers manually
linked to the appropriate ontological categories. Knowledge engineers maintain
the data and metadata, rather than the data being provided by the community
itself as in OntoShare.
8 Concluding Remarks
We have described OntoShare, an ontology-based system for sharing information
among users in a virtual community of practice. We motivated the use of
Semantic Web technology for KM tools and described how ontologies in OntoShare
are defined in RDFS. Communities are able to automatically share information
and create RDF-annotated information resources as a side-effect of this
activity. Furthermore, these information resources are then of course available
to other RDF-based tools for processing: the community semi-automatically
creates an ontology-based annotated information resource for use by itself
Importantly, the ontology used by a given community in OntoShare can
change over time based on the concepts represented and the information
that users choose to associate with particular concepts.
This is a significant advantage over a community attempting to reach
consensus on a set of concepts and how they relate to another at the outset
that is then difficult or impossible to change. Much remains to be done
in this area however, particularly with regard to the introduction of new
concepts. In addition, users have personal profiles according to the concepts
in which they have declared an interest and these profiles also evolve
automatically, seeking to match more closely a user's information needs
and interests based on the usage they make of the system.
The use of OntoShare in a case study and its subsequent evaluation has
been described. The results of the evaluation were presented which have
led to a number of recommendations for the continued development of OntoShare.
We indicated some further directions of research. OntoShare exemplifies
the much-improved knowledge management tools that the advent of the Semantic
Web and its support for ontologies makes possible.
The research described was partly funded by the EU IST-1999-10132 project
On-To-Knowledge (see http://www.ontoknowledge.org).
We thank all our partners for their fruitful cooperations, discussions
and the fun we had during the project.
[AB95] K. Ahmed and M. Benbrahim. Text summarisation:
the role of lexical cohesion analysis. New Review of Document and Text
Management, 1:321-335, 1995.
[ABC + 99] R. Altmann, M. Bada, X. Chai, M.W. Carillo,
R. Chen, and N. Abernethy. RiboWeb: an ontology-based system for collaborative
molecular biology. IEEE Intelligent Systems, 15(5), 1999.
[BKv02] J. Broekstra, A. Kampman, and F. van Harmelen.
Sesame: A generic architecture for storing and querying RDF and RDF Schema.
In Horrocks and Hendler [HH02], pages 54-68.
[BLHL01] T. Berners-Lee, J. Hendler, and O. Lassila.
The semantic web. Scientific American, 2001(5), 2001. available
[Dav00] N.J. Davies. Supporting Virtual Communities
of Practice, chapter 3, pages 199-212. Springer-Verlag, 2000.
[Dv02] A. Duke and J. van der Meer. User profile
construction. On-To-Knowledge deliverable 14, BT and AIdministrator Nederland
[Eas88] K. Eason. Information technology and
Organisational Change. Taylor & Francis, London, 1988.
[Fen01] D. Fensel. Ontologies: Silver bullet
for knowledge management and electronic commerce. Springer-Verlag,
[GPB02] A. Gómez-Pérez and V. R. Benjamins,
editors. Proceedings of the 13th International Conference on Knowledge
Engineering and Knowledge Management: Ontologies and the Semantic Web (EKAW
2002), volume 2473 of Lecture Notes in Artificial Intelligence (LNAI),
Siguenza, Spain, 2002. Springer.
[Han97] M.T. Hansen. The searchtransfer problem:
The role of weak ties in sharing knowledge across organisation subunits,
1997. Working Paper, Harvard Business School.
[Har92] D. Harman. Ranking Algorithms. Prentice-Hall,
NJ, USA, 1992.
[HH02] I. Horrocks and J. A. Hendler, editors. Proceedings
of the First International Semantic Web Conference: The Semantic Web (ISWC
2002), volume 2342 of Lecture Notes in Computer Science (LNCS),
Sardinia, Italy, 2002. Springer.
[LS99] O. Lassila and R. Swick. Resource Description
Framework (RDF). Model and Syntax Specification. W3C Recommendation 22
February 1999, 1999. available at http://www.w3.org/TR/REC-rdf-syntax.
[Pol58] M. Polanyi. The Tacit Dimension.
Doubleday & Co., Garden City, NY, 1958.
[SAD +00] S. Staab, J. Angele, S. Decker, M. Erdmann,
A. Hotho, A. Maedche, H.-P. Schnurr, R. Studer, and Y. Sure. Ontology-based
community web portals. In Proceedings of the AAAI-2000. MIT Press,
Menlo Park, USA, 2000.
[SAS03] Y. Sure, J. Angele, and S. Staab. OntoEdit:
Multifaceted inferencing for ontology engineering. Journal on Data Semantics,
[SBD91] J. Seely-Brown and P. Duguid. Organisational
learning and communities of practice. Organisational Science, 2(1),
[SMMS02] L. Stojanovic, A. Maedche, B. Motik, and
N. Stojanovic. User-driven ontology evolution management. In Gómez-Pérez
and Benjamins [GPB02], pages 285-300.
[SS04] S. Staab and R. Studer, editors. Handbook
on Ontologies in Information Systems. International Handbooks on Information
Systems. Springer, 2004.
[SSS04] Y. Sure, S. Staab, and R. Studer. On-To-Knowledge
Methodology. In Staab and Studer [SS04], 2004.
[SWM04] M. K. Smith, C. Welty, and D. McGuinness.
OWL Web Ontology Language Guide, 2004. W3C Recommendation 10 February 2004,
available at http://www.w3.org/TR/owl-guide/.