On Two Aspects of Improving Web-Based Training

Abstract: In this paper we discuss two novel ideas to improve teaching and information transfer between persons in general. The first aspect is centered around proposing to use the ``Tamagotchi craze'' for teaching purposes, the second deals with new ways of unobtrusively collecting data on the subjective satisfaction of persons with information and teaching material offered on the Web.
Key Words: Training, teaching, learning, courseware, web-based training, evaluation techniques
Category: J. 7, K 3, K.4

1 Harnessing the Tamagotchi Craze

The Tamagotchi craze that started in 1997 has driven many kids, teachers and parents to the brink of desperation. An estimated 20 million Tamagotchis or clones thereof were sold within one year, resulting in hundred millions of hours spent on keeping the virtual beings happy. It is the aim of this paper to propose to harness the appeal of caring for such virtual beings for teaching and training purposes: rather than taking care of Tamagotchis by ``feeding'', ``cleaning'' and ``entertaining'' them we propose to implement what we call VR-Friends (virtual friends) that differ in three crucial ways from traditional Tamagotchis: (i) they are kept happy if their owners answer questions correctly; (ii) they are implemented in software, not hardware and (iii) they live on the Web, in WWW servers. In a way, VR-Friends can be considered to be special types of Avatars (as often mentioned in connection with VR games), or intelligent agents ([7],[8]).

Psychologists have not been able yet to fathom the real reason why Tamagotchis have been successful in such an unprecedented way. But no matter what the explanation is, it seems worthwhile to exploit what are seen as major reasons for the success for the education of primary and secondary school children.

1.1 Introduction

VR-Friends are implemented as programs that run on WWW servers and can be accessed by ordinary Web browsers from e.g. Netscape or Microsoft. They come in various behavioral and knowledge domain classes in the sense that they act as instructional agents for different ages and subject matters. Their success as ``teachers'' is assumed to come from the fact that they are seen as live beings that thrive only if continually ``fed'' with correct answers to questions they keep asking. I.e. VR-Friends want to ``learn'' and in doing so - it is actually their owners that learn!

In what follows we present the basic ideas of how such VR-Friends will work. We do this in three subsections. In section 1.2 we describe a typical scenario. In

section 1.3 we explain the main implementational aspects. And in section 1.4 we argue why VR-Friends have indeed a chance to be successful yet why they may fail, and what practical use and further developments might look like.

1.2 A typical scenario

So far, twelve year old Mark has shown very little interest in learning basic geographical facts: Although the importance of factual knowledge and rote memorising has been much challenged by educationalists over the last few decades there is still a wide-spread feeling that some basic facts have to be stored inside our brain (rather than on easily accessible storage devices) as basis for ``associative'' and ``intuitive'' thinking.

To improve Mark's performance and following a counselor's recommendation, the geography teacher has activated an appropriate (i.e. certain level of geography) VR-Friend on the school's Web server. Mark has chosen the name VR-Nick and certain features (like a special T-shirt and trousers) for his VR-Friend when first logging in. Now, whenever Mark logs in on the school server for whatever activities the first thing Mark sees is VR-Nick. VR-Nick's appearance, his attitude, and what he says (yes, VR-Nick speaks) depends on Mark's past performance and on how much work Mark has ``cared for'' VR-Nick. Sometimes VR-Nick will be smiling, ``Hey, great you are looking me up again'', or sulky maybe even tearful (``You are really neglecting me''). Overall, if VR-Nick is treated well he stays healthy, becomes more and more friendly, will show Mark a cartoon, a joke, or tell him some titbits (e.g. pointing him to an interesting URL on the Web), otherwise, VR-Nick will look more and more sick, he will sit in a sad posture in a not so good-looking neighbourhood and, if seriously ``neglected'' will actually die. On the other hand, if Mark learns well (treats VR-Nick well), VR-Nick will eventually congratulate Mark and tell him that he has to move on, but will stay in contact with him: and he will once in a while (with a decreasing tendency) briefly show up when Mark logs in, tell him some titbits, and disappear again. Note that Mark may have a growing number of VR-Friends that way, that entertain Mark a bit here and there as he does other Web stuff.

The actual heart of the conversation between Mark and VR-Nick (who was after all created to teach some geographic facts) consists of questions asked by VR-Nick that have to be answered by Mark. (Mark can exit ``no time right now'' any time he wants). A typical ``geography'' VR-Nick may help the student (Mark) to learn about the location of cities, countries, etc. on the globe. Questions that VR-Nick might ask are ``show me the location of xxx'' (where xxx is a city, a country, a mountain, etc.) Mark's reaction is to click at the right place on the globe or map shown. The system is very patient: if Mark clicks on Antarctica when asked for the location of Vienna, the system (NOT VR-Nick) will explain that this is Antarctica, some facts about it, and will show the real location of Vienna and information on it and Austria. However, the system records the information (``Vienna, totally wrong'') in such case. A second totally wrong answer to Vienna's location would give a somewhat more stern reply to Mark, and the system would record again (``Vienna, totally wrong''). A third complete failure of Mark would result in the system storing (``just fooling around''), etc. Thus, the system basically records for each session between Mark and VR-Nick five numbers (a,b,c,d,t). Here a is the number of correct answers, b the number

of totally wrong answers, c the number of approximately correct answers, d the number of times Mark just seems to have been fooling around; t is the point of time when the session takes place. The ``development'' of VR-Nick depends on how often and lengthy sessions between Mark and VR-Nick take place; it is up to the teacher to set parameters like: ``at least one session with 10 questions every second school day and all questions answered correctly twice within a total of three months''.

Above is clearly just one typical scenario. Other questions (requiring e.g. text input or such) are of course conceivable. Finally, VR-Nick has always some surprises ``up his sleeve'': like instead of VR-Nick appearing, Mark finds a sign ``Out right now. Back in 2 minutes'', or finds some other VR-Friend, etc.

The main point is that although VR-Nick forces Mark to learn (unless Mark accepts that VR-Nick deteriorates rather than prospers) this learning is fun and full of surprises.

1.3 The implementation

The main idea of the implementation is to have two completely separate modules called development and learning.

The learning module is clearly domain dependent, i.e. differs from subject to subject and takes the age of the learner into account. However, learning modules are created readily with teaching wizards: the most simple form of a teaching wizard just helps preparing problems as follows: each problem consisting of a question and possibly a picture (image), and the answer is a click in a certain position, where a click is recorded as ``correct'', ``close'' or ``wrong'' dependent on how far from the current position the clock occurs. Note that the system will keep track of ``close'' or ``wrong'' answers to be able to decide if a student is seriously trying or just fooling around. Other teaching wizards that e.g. allow textual answers can also be used.

The main point is that the teaching module communicates to the development module only five numbers (a,b,c,d,t) as explained in section 1. The reaction of the VR-Friend as carried out by the development module only depends on the sequence of ``answer-quintuples'' (a,b,c,d,t) obtained. To keep the system simple there is no (or no substantive) data-flow from the development module to the teaching module.

Depending on the answer-quintuples the state of the VR-Friend (and hence its appearance and utterances) within the development module are changing. A random element makes sure that the VR-Friend behaves somewhat unpredictably. A number of VR-Friends (age, looks, clothing, ...) with certain developmental paths are built into the system, so new subject domains do not necessitate changes or additions of the development module, although tools to introduce new VR-Friends may well be provided in the future.

The development module and its functionality is clearly very much open ended. On a first and primitive level for each ``state'' of the VR-Friend one of a number of facial expressions, gestures, background images and utterances is combined in a random fashion. This leads to a fairly complex ``behaviour'' at moderate development cost.

It remains to be mentioned that both teachers and students can set certain environmental parameters as mentioned in passing: the teachers can define time

intervals that are ``acceptable'', students will choose sex, name and age of their VR-Friends.

1.4 Other issues

The most burning issue is certainly if VR-Friends will be interesting enough to indeed motivate students to learn. There are strong arguments both pro and con, and only experiments can answer this question.

On the one hand, Tamagotchis have been amazingly successful, and VR-Friends are similar in a number of important ways: once ``born'' they cannot be ``stopped'' but either develop well (if treated well) or else they visibly deteriorate, even to the extent that they die. On the other hand, while Tamagotchis are ``omnipresent'' and keep reminding users of their existence by beeping at appropriate or less appropriate times, VR-Friends only show up when a user logs into the correct server, typically the main server of a school or school-district.

There is another crucial issue. Kids who love to play and love to learn all kinds of irrelevant information seem to lose interest in games once they realise that they are actually learning useful information (!). It has been theorised that the association ``learning is work, work is unpleasant'' (a product it seems of our less than optimal school systems) is sufficiently strong to turn kids off. A typical case supporting above arguments is the huge success of the ``Space Quest'' adventure games vs. the meagre sales of the ``Goldrush'' adventures game, all by Sierra: all have the same interface and the same kind of story: users have to help the hero to discover certain facts: Those facts are ``useless'' in ``Space Quest'' but contain real historic information in ``Goldrush''. It is vexing to see that sales of Space Quest have been hundred-fold better than those of Goldrush!

However, there have been some educational games that have been successful (like the classic arithmetic learning game ``How the West was won'' or the geography/history games ``Where in the world (where in history) is Carmen San Diego''.) Thus, the final verdict whether playing cannot be combined with teaching is still out!

There is also another, rather opposite issue that can only be settled experimentally: can it happen that kids get attached to their VR-friends to an extent that their departure, let alone death, will cause serious emotional turmoil? Note that even Tamagotchis have turned out to be fairly addictive: What will a more human-like, reality-like VR-Friend do? Whatever the answer is it is clear that VR-Friends programmes have to make sure that users will not develop guilt feelings, e.g. when a VR-Friend gets sick or dies since its owner was sick, on vacations, or such.

The most interesting aspects of VR-Friends is the fact that they are on the Web. We leave a detailed discussion of the implications of this for a future paper. But observe that VR-Friends can potentially communicate, compete, or temporarily be taken care of by ``Baby-sitters'' via the Web. Note further that VR-Friends can use and point out relevant information (i.e. URL's) on the Web, lead to discussion forums, on-line chats between owners of similar VR-Friends, etc. Indeed the dialogue of users and their VR-Friends may lead to questions posed by other VR-Friends to other users. It should be clear that the complex interaction of VR-Friends will have to based on what has been learnt and developed in the area of intelligence agents as mentioned earlier.

The notion of Web-based VR-Friends used for educational applications opens a new and potentially very powerful way to communicate factual knowledge - and maybe even more. Experiments will have to demonstrate the viability of the approach. At the time of writing no polished version of VR-Friends software exists, but we expect to have one ready by mid 1998.

Note also that successful implementations of VR-Friends will be much eased by using powerful Web-servers such as Hyperwave [1], [2], [3].

2 Unobtrusively Collecting Data on Web-Based Material

When offering educational material on the Web it is of crucial importance to obtain feedback on effectiveness and user satisfaction. In this note we focus on one sub-aspect, obtaining data on subjective user satisfaction. We discuss ``traditional'' techniques, none of which has worked well in the past. We then present novel techniques that we believe could provide effective tools desperately needed to assure continuous quality improvement of Web based training material.

2.1 Classical ways to collect satisfaction data

There are two main classical ways to collect data on user satisfaction with educational material available on the Web: (a) the use of questionnaires and (b) evaluation of detailed log files.

The problem with questionnaires is that there is much user resistance to filling them out carefully, thus creating lots of statistical ``noise''. Even ``willing'' students are often at a loss of how to answer questions properly that refer to material that was worked through some time back.

The other approach, the evaluation of detailed log files is problematic for two reasons: the HTTP protocol used on the Web is ``stateless'', i.e. it is not possible to record the ``trail'' of users but only how often each page has been hit. Using more sophisticated techniques such as session oriented protocols, or simulating them by using ``cookies'' or ``session keys'' as is done e.g. in Hyperwave ([1], [2], [3]) it is possible to keep more detailed records of usage of educational material. However, such detailed records have already been kept in non-Web based systems as early as in Plato [4], but have rarely been terribly helpful. It seems that efforts to extract interesting information from a huge mass of log files have never been entirely successful.

Without going into further detail it should be clear that new techniques to judge subjective users satisfaction are needed for use on the Web which is becoming a major tool of teacher-student communication and course administration. We propose three such techniques in the next section.

2.2 Novel ways to judge subjective user satisfaction

The first alternative we suggest to use instead of questionnaires is the use of `questionnairelets'', Q-lets for short. Such a Q-let consists of a single question that can be answered within a few seconds. Q-lets are presented to users in a random fashion but never more than a few per session. As little bonus for answering a question an optional cartoon or joke is shown to encourage answering the Q-let.

Cartoons or jokes shown must be chosen so as not to disrupt the flow of learning. Note that an option ``skip question'' is presented, i.e. the answering of Q-lets is not enforced. However, clicking at the Q-let ``skip'' button will often be the same amount of work as clicking at one of the choices (e.g. radio buttons) offered in the Q-let.

Observe that users in a Q-let environment do not even need to fill out a complete questionnaire in total, yet may actually answer the same Q-let more than once. Different answers will thus be a good indication of the (un)``anonymously identified'' across session boundaries in the sense of [5], i.e. users have (self-chosen) names and passwords, so that detailed record can be kept for each ``anonymous'' user, yet the system does not know the real identity of any of the names chosen. Experimentation on the distribution and frequency of Q-lets used will have to be carried out.

An alternative to Q-lets (or an additional feature) is a so-called ``Feedback Button ''. When clicked at, a form appears where users can click at any of the presented checkboxes to voice their opinion about the current ``page'' presented, or even about a certain part or aspect of it.

We feel that collecting individual answers at points where they are relevant is both less bothersome for the user, and leads to more reliable results at the same time.

The second alternative to evaluate subjective user satisfaction comes from the approach used in GENTLE [6] to questions posed by students: at any point while working through some material students can ``ask a question'' by inserting a question mark anywhere on the screen. This action triggers the sending of an appropriate message to a certain user group (i.e. tutors), who will answer the question (synchronously or asynchronously) at which point the question mark turns into an exclamation mark; also, the answer is mailed to the person having asked the question, and other users seeing an exclamation mark can retrieve the ``question/answer'' dialogue that occurred earlier by just clicking at the exclamation mark.

By evaluating pages with such question marks and exclamation mark much valuable feedback can be collected. In first experiments it was observed that questions were not only asked when material was badly explained but actually even more frequently when the material interested the users to the extent that they wished further explanations!

The third alternative is called ``sensitive button''. After all, each ``page'' of courseware leads to one of a number of further pages clicking at some navigational button. Users are informed that the position where they click the button will be seen as the expression of an opinion concerning the current ``page'' at issue. There are many alternatives. However, to be specific, here is one scenario: clicking at a button on the left could mean ``I don't like this page'' (and upper left ``bad explanation'' middle left ``I don't like this page but I don't specify any reason'' and lower left ``The explanation is too terse''), while clicking on the middle of the button may mean ``my feelings are neutral'' and clicking on the right ``I like this page'' (where upper right could indicate ``I like the way things are presented'', middle right ``I like this page but I don't specify any reasons'' and lower right ``I like this page because of the contents provided''.) Clearly many alternatives are possible and need careful testing.

We feel that sensitive buttons (although they will generate some statistical noise) will work well after some initial period (during which the user may be

asked ``do you really mean xyz?''): some ``click'' to navigate is necessary anyway, so users ``may as well'' signal their level of satisfaction.

In this section we have argued that new ways of collecting feedback on user satisfaction with educational material is of great importance for optimising WBT (Web Based Teaching/Training). We have proposed a few novel techniques but believe that many other ways can be thought of and should become fields of serious experimentation.

References

1. Maurer, H.: HyperWave: The Next Generation Web Solution; (Ed.), Addison-Wesley Longman, London
2. Maurer, H., Mayrhofer, V.: Handling Large Web Sites on Internet and Intranets - The Official Guide to Hyperwave, dpunkt, Heidelberg (1998), to appear.
3. Maurer, H.: What We Want from WWW as Distributed Multimedia System; Proc. VSMM'97, Geneva. IEEE (1997), 148-155.
4. Bitzer, D.: The Wide World of Computer Based Education; Advances in Computers, Academic Press, 15 (1976), 239-283.
5. Flinn, B., Maurer, H.: Levels of Anonymity , JUCS 1, 1 (1995), 35-47.
6. Maurer, H., Dietinger, Th.: How Modern WWW Systems Support Teaching and Training; ICCE'97, Kuching, Malaysia (1997), to appear.
7. Riecken, D.: Intelligent Agents; Special Issue of the C.ACM 37,7 (1994).
8. Maes, P.: Agents That Reduce Work and Information Overload; C.ACM 37,7 (1994), 30-41.