Evaluating the Effectiveness of an ODL Hypermedia System and Courseware at the National Technical University of Athens: A Case Study

Abstract: The main purpose of this paper is to test a conceptual framework for the evaluation of the effectiveness of an open and distance learning (ODL) hypermedia system (EONT-ODL system) and courseware developed and trial used at the National Technical University of Athens (NTUA), Greece, within the EONT project. EONT is a partnership project between seven universities from seven European Union countries within the Socrates Framework Program. In this paper we deal with data elicited from the NTUA, since it was the only partner institution which provided adequate data for quantitative analysis. The evaluation framework is based on the assumption that ODL hypermedia systems are complex systems with a variety of organisational, administrative, instructional, and technological components. It has been hypothesized that the effectiveness of the EONT-ODL system would be influenced by a number of independent variables such as: design and presentation of the courseware; previous experience; time spent on working through the courseware; preference of mode of study; learning styles; interactions with peers, instructors and means of communication. In this evaluation research, two instruments integrated into one questionnaire for data collection were developed: the first was based on a number of standardized questions, reflecting the previously stated theoretical framework and the second on a number of open-ended questions, reflecting, likes and dislikes, added value, problems identified, suggestions etc. The regression analysis indicates that the `design and presentation of instructional material' alone explained almost 28% of the EONT- ODL system's effectiveness (R<sup>2</sup> adj.= .278). The preferred `mode of study' entered second by adding 11% (R² ch.=.113) of the effectiveness variance and finally students interactions with the instructor increased the effectiveness explained variance to 48%, a quite high percentage accounted for three significant predictors alone. All the other predictors, that is, previous experience with computers, time spent working with the EONT-ODL courseware, student learning styles, and interactions among students and communication means (e-mail and

computer conferencing) did not significantly contribute to the prediction of the effectiveness measure. These quantitative results are complemented by the qualitative conclusions.

Key Words: Evaluation, Open and Distance Learning systems, Hypermedia Courseware,

1. Background and Objectives

The Software Engineering Laboratory (Softlab) of the Electrical and Computer Engineering Department of the National Technical University of Athens (NTUA), Greece, offers an introductory course is Software Engineering targeted primarily to ninth semester students. Since 1996, the course components were: lectures, discussions, projects and text-based study material. It had been evidenced, however, that the percentage of students attending this course was less than 60 percent, mainly due to students' engagement in employment. A needs analysis for this course also revealed that the traditional mode of teaching faces major problems [Koutoumanos et al., 1996], such as:

• It was difficult for students to ask questions and receive answers outside of the instructor's appointed office hours.
• There was low interaction in classroom. Students often preferred not to ask questions in the classroom because they felt shy or that their questions might sound silly.
• The curriculum of the course was changing so rapidly that textbooks became quickly obsolete.

Having identified these problems, the Softlab research team started looking for remedies. It was discussed that the merger of the open distance learning (ODL) methods with new information technologies, such as the Internet and the World Wide Web (WWW), might contribute to the solution of the identified problems. A literature search revealed that providers of university education today are faced with the challenge of building an education system which could meet the current and future needs of society [Ford et al., 1996]. In this effort, open and distance learning (ODL), in particular based on hypermedia and computer networks, has witnessed an increased development, acceptance and recognition as an innovative and productive delivery mode of instruction and learning [Kaye, 1991]; [McConnell, 1991]; [MCConnell,1994]; [Riel & Harasim, 1994]; [Hiltz, 1995]. Indeed with the advent of the Internet, the WWW, and the accompanying WWW browsers, the provision of ODL courseware has taken on a whole new dimension [Maddux, 1996]; [Makrakis, 1996]; [Marshall & Hurley, 1996]. These technologies can be used in various ways for the implementation of learning systems in ODL.

One such learning system, the EONT-ODL system, was developed by the Softlab research team and is depicted in Fig. 1 [Koutoumanos et al., 1996]. In this system the course instructional material refers to courseware in hypermedia format on the introduction to software engineering which is stored in a server computer and accessed by the learners through multimedia client computers connected to the server via a computer network. The heart of the EONT-ODL system is the hypermedia system HyperWave [Maurer, 1996].

This learning system has been used in the EONT project [Papaspyrou et al. 1996]. EONT was a partnership project between seven universities from seven European Union countries within the Socrates Framework Program. Two of these universities were Distance Learning, whereas the rest were conventional. The partnership was formed on the basis of the partners' common interest in experimenting with ODL using new information and communication technologies with goals of investigating the effectiveness of these technologies. The project started on December 1st, 1995, and has been completed in two years (September 1997). The project partners, however, continue developments on a voluntarily basis. For the purpose of the experiment, each partner developed multimedia instructional material within the domain of Informatics as shown in Table 1, by adopting a common courseware engineering methodology [Retalis 1997]. The language of each course was both the native language of the associated partner (native version) and English for the international version. The national version had been offered once during the second year (1996/97) of the project and would have also been offered once during the third year. The international version would have also been offered during the third year, as a means of providing learners in one partner's country with access opportunities to the course instructional material of the other partners. However, the application for third year funding by the European Union was rejected.

Partner            Course content                      Language 

   P1   Introduction to Software Engineering           Greek (GR) 
   P2            Hypermedia Systems                    German (DE) 
   P3   User Interface Design and Development          English (EN) 
   P4            Multimedia                            Dutch (NL) 
   P5   Introduction to the Unix Operating System      Norwegian (NO) 
   P6       Elementary Course in Mathematics           French (FR) 
   P7   Software Engineering for Distributed Systems   German (DE)

To assess whether the goals set for the project were achieved, both formative and summative evaluation activities were conducted. These undertakings were considered as integral parts of the whole development and implementation process, since they provide valuable insights and feedback to the development team for necessary changes and additions.

In this paper, we deal with data from summative evaluation elicited from NTUA, since it was the only partner institution which provided adequate data for quantitative analysis. Two basic evaluation research questions guided the summative evaluation activities of the project:

1. Is the EONT-ODL system effective for delivering university courses either supplementary to the face-to-face, traditional university courses, and/or independently?
2. Which variables are associated with and explain or predict either good or poor outcomes in this new teaching and learning system?

2. Theoretical Framework

ODL systems have increasingly come to be viewed as complex systems with a variety of organisational, administrative, instructional, and technological components. As such, ODL systems are dynamic in nature and any change in one component of the system has effects on all others. While there is a consensus that ODL systems consist of multiple interrelated components, there are few evaluation instruments that document these components holistically or study their relationship to the effectiveness of the whole system of which they are a part. In most evaluation studies, the question in focus concerns the comparative effectiveness of various types of ODL systems as instructional

delivery modes to traditional modes [e.g. Hartley, 1994]; [Hiltz, 1995] rather than the innovation of the delivery mode itself and the factors which may contribute to its effectiveness. Effectiveness in this study was conceptualized as being related to a multiple measurement index consisting of cognitive and attitudinal outcomes.

Research shows that the effectiveness of hypermedia is constrained by two important factors: first, the design of the user interface, and second, the motivation and expertise possessed by the users [Reeves, 1992]. It has been also argued that the more valuable variables in evaluation research are those concerning the learner characteristics and numerous others, including instructional material design, pedagogy as well as content variables [Moore & Kearsley, 1996]. The evaluation of learning design in hypermedia courseware can involve a range of different dimensions of interest. Barker and King (1993) have developed a basic set of categories by which instructional software could be evaluated, such as mode and style of interaction, learning styles, adequacy of ancillary learning support tools, outstanding strengths, and attractive features. Marshall and Hurley (1996) have also included frequency of computer and ODL online courseware use as important variables in evaluating online ODL systems.

Moreover, for hypermedia-based learning, a dimension particular to the characteristics of hypermedia must be added to the evaluation design. This dimension includes variables related to the process of learning affected by access, learner control, and collaboration [Nielsen, 1990]. It would also be difficult and probably futile to evaluate ODL hypermedia courseware outside the context of its use. McDougall and Squires (1995) argue that the use of courseware can only be evaluated by considering the use of a package in particular learning situations. This requirement poses an inherent problem for predictive evaluation where, by definition, the evaluation is conducted out of context. McDougall and Squires propose a situated approach to predictive evaluation. This approach generates context specific evaluation issues by considering the interactions between the three principal actors associated with the development and use of educational software -- the teacher, the designer, and the students. All these actors have been considered in the formative evaluation undertakings.

In the summative part presented here, the parameters that were used to evaluate the EONT-ODL system and courseware were determined by a combination of the usability of the hypermedia system, the usability of the content and structure of the hypermedia courseware, the effects of the courseware and the learning processes involved on the user, and by how well these three components fit together. More specifically, it has been hypothesized that the effectiveness of the EONT-ODL system would be influenced by a number of independent variables such as:

• design and presentation of the instructional material;
• previous experience;
• time spent on working through the courseware;
• preference of mode of study
• learning styles
• interactions with peers, instructors and means of communication.

The analysis of the data provided by end-users will feed designers and instructors with

valuable interventions.

3. Research Methods

Contemporary perspectives of evaluation range from absolute ''measurement`` to a completely relativistic ''constructivistic`` perspective [Shadish, Cook and Leviton, 1991]. Effective evaluation should encompass both the empirical-analytic (measurement) and naturalistic (constructivistic) paradigms [Guba & Lincoln, 1981]; [House, 1991]; [Makrakis, 1997]. This view of evaluation enquiry is significant for the study of technology in ODL since many of the issues and questions which need to be addressed cannot be answered by the empirical-analytic paradigm alone, nor can they be answered by assessing individual ODL in isolation from the broader context [Dillon & Gunawardena, 1992], [Makrakis, 1996]. In the present evaluation research, two instruments integrated into one questionnaire for data collection were developed: the first was based on a number of standardized questions, reflecting the previously stated theoretical framework and the second on a number of open-ended questions, reflecting, likes and dislikes, added value, problems identified, suggestions etc.

3.1 Subjects

The total number of students from the NTUA which responded to the evaluation study reached 50 (15% women and 85% men) out of the 61 registered in the course. Of these students 4% indicated that they were computer novices, 30% had good experience, 44% had very good experience and 22% had professional experience. In terms of time spent working with the EONT-ODL system, 48% spent less than one hour , 38% from 1 - 2 hours, and 14% from 3 - 4 hours per week.

3.2 Research Instruments

Most of the main variables in this study are measured by multiple items, each measuring a slightly different aspect of the main variable. In building composite measurement scales, items included were first scrutinized for `face validity'. After the data were collected, the validated items in each composite variable were subjected to a Cronbach's Alpha reliability analysis for internal consistency of the instrument. In arriving at the final composite measurement indexes, every item which substantially lowered the Alpha coefficient was omitted and a new analysis was conducted in order to arrive at an index which had the highest possible reliability measure. The summary statistics of the item analysis for homogeneity and reliability indices, shown in Table 2 , indicate that the `effectiveness measurement' reached a very high alpha coefficient (a= .93), retaining all the 23 intended items. These items included dimensions of cognitive and attitudinal outcomes. The `design of instructional material' composite variable retained seven out of the ten intended items with a reliability a= .82. The items retained covered issues of content, coherence, presentation, objectives, and structure.

The preferred `mode of study' composite variable retained all the five intended items

with a reliability a=.81 and measured the comparative outcomes of the traditional mode of teaching and the EONT-ODL mode. All composite variables were measured by a five-point Likert-type scale, where 1 was coded as the lowest value and 5 the highest. Learning styles were measured by a number of variables which reflected three main dimensions: collaboration, innovation, and systematicity. The first refers to those who value more group-based course interactions, the second to those who value the challenge of something new and different and the third to those who value the formality of setting specific goals, plans, and detailed procedures. The interactions among peers, instructors, and means of communication included measurements of the frequency of using means such as e-mail, computer conferencing, and frequency of meetings among students and with instructors.

Table 2. Reliability of Items in the EONT-ODL Effectiveness Scale

ITEM-TOTAL STATISTICS

FOR EONT-ODL EFFECTIVENESS SCALE
                                                     ALPHA IF ITEM
                                                           DELETED

The EONT-ODL system increased my                               .92 
  knowledge and skill on the subject matter. 
The EONT-ODL system created a new and                          .92 
  innovative learning environment. 
The EONT-ODL system helped me to cope                          .92 
  with the demands of the course. 
The EONT-ODL system widen communication                        .93 
  with the instructor and other students. 
The EONT-ODL system relieved some of the                       .93 
  physical constraints at attending regularly 
  face to face lecturing. 
The EONT-ODL system allowed me to pursue                       .93 
  learning experiences in a self-directed way.          
The EONT-ODL system increased the potential                    .92 
  to pursue collaborative project work with other students. 
The EONT-ODL system increased my interest in                   .93 
  the subject matter. 
The EONT-ODL system increased the potential to                 .92 
  gather, send and receive information. 
The EONT-ODL courseware developed my problem                   .92 
  solving skills. 
A variety of learning experiences was provided                 .93
  through The EONT-ODL mode of teaching. 
The EONT-ODL system gave me sufficient                         .93 
  opportunity to utilize my personal experience. 
The design of The EONT-ODL courseware allowed me               .93 
  enough freedom to choose where, when and how to study. 
The EONT-ODL courseware links to activities in the             .92 
  classroom. 
The EONT-ODL courseware encouraged me to                       .93 

  
  participate actively in normal class activities 
  with other students.
I have sufficient opportunity to demonstrate what I have       .92 
  learned in the subject through The EONT-ODL system 
  of teaching. 
The EONT-ODL courseware required me to synthesize              .93 
  and put together ideas. 
The EONT-ODL system teaching stimulated me to                  .92 
  communicate more with other students in the class. 
The EONT-ODL courseware required me to evaluate                .93 
  using my judgment and intuition. 
The EONT-ODL courseware required me to apply what              .93 
  I have learned . 
The EONT-ODL courseware required me to understand              .92 
  concepts and ideas. 
The EONT-ODL system of teaching effectively                    .92 
  stimulated my interest in the subject matter. 
The EONT-ODL courseware motivated me for more                  .93 
  study on the subject matter. 

RELIABILITY COEFFICIENTS 23 ITEMS
ALPHA= .93 

ITEM-TOTAL STATISTICS 
FOR PREFERRED MODE OF STUDY SCALE
                                                     ALPHA IF ITEM 

                                                           DELETED 
The EONT-ODL mode proved to be more beneficial than            .77 
  conventional way of lecturing. 
The EONT-ODL mode has made me to prefer learning               .75 
  from this mode even when the same teaching is given
  in other ways. 
I found The EONT-ODL mode of teaching to offer better          .75
  experiences than conventional way of lecturing. 
The EONT-ODL mode was more pleasant than conventional          .78 
  way of lecturing. 
The EONT-ODL mode of teaching was more convenient than         .81 
  conventional way of lecturing. 

RELIABILITY COEFFICIENTS 5 ITEMS 
ALPHA= .81 

ITEM-TOTAL STATISTICS
 FOR COURSEWARE DESIGN SCALE 

The aims and objectives of the EONT-ODL system were clear.           .76 
The courseware included in the EONT-ODL system explained 
  things clearly.                                                    .76 
It was easy to find out hat was expected in the EONT-ODL system.     .79 
The assignments on the EONT-ODL courseware were appropriate to the 
  learning objectives.                                               .82 
The components of the EONT-ODL courseware linked well together.      .78 
The EONT-ODL courseware was presented in an attractive way.          .80 
The sequence of units/blocks in the EONT-ODL courseware is logical   .82 
  and well structured. 

RELIABILITY COEFFICIENTS 7 ITEMS 
ALPHA= .82 

3.3 Data Analysis

The analysis of the structured part of the questionnaire was based on univariate and multivariate statistical analysis and the open-ended part on qualitative content analysis. All appropriate tests for examining whether these variables fulfill the conditions for undertaking a regression analysis were conducted (e.g. normality, multicolinearity). These tests showed that all requirements were attained.

4. Evaluation Results

The regression analysis shown in Table 4, indicates that the ''design and presentation of instructional material`` alone explained almost 28% of the EONT-ODL system's effectiveness (R<sup>2</sup> adj.= .278). The preferred ''mode of study`` entered second by adding 11% (R² ch.=.113) of the effectiveness variance and finally students interactions with the instructor increased the effectiveness explained variance to 48%, a quite high percentage accounted for three significant predictors alone. All the other predictors, that is, previous experience with computers, time spent working with the EONT-ODL courseware, student learning styles, and interactions among students via communication means (e-mail and computer conferencing) did not significantly contribute to the prediction of the effectiveness measure.

The qualitative results complement the quantitative conclusion that the ''design of courseware`` and ''preferred mode of study`` are the most significant predicting variables for effectiveness. It is also important to note that the reason why ''previous computer experience`` had not accounted anything significant to the system's effectiveness, may be because most of the end-users had similar computer skills. Also

the time spent on the EONT-ODL system was very little to produce any significant impact on the EONT-ODL system's effectiveness. As revealed by the content analysis of the open-ended research questions, this might be caused by the difficulties in accessing the system outside the Softlab, mainly due to the lack of sufficient telecommunication lines available.

Although the quantitative analysis shows that learning styles did not contribute significantly to the effectiveness of the system, the content analysis of the responses provided by open-ended questions revealed a number of patterns, concerning the degree students' learning styles have been affected as a result of their experience with the EONT-ODL courseware delivery. It has been found that the great majority of the students revealed that the EONT-ODL system, despite of problems identified, had positively affected their study patterns, especially as it concerns the independence of learning and the deeep-level and fast level of information processing provided through this system. In the later way, students are required to take an active involvement in the acquisition and development of knowledge. In the former way, students are required to be more analytic and field independent. These ways seem to be affected by the flexibility provided by the EONT-ODL system in terms of time, place, and pace of instruction and learning, the high-tech design of lessons, and the searching facilities integrated in the EONT-ODL system.

In general, the average score of the effectiveness measurement was 3.25 on the five-point Likert scale, where 5 was coded as the highest subjective effectiveness perceptions and 1 the lowest. This is relatively high considering that the EONT-ODL system developed was at its first version. The `design of instructional material' factor reached an average score of 3.25 of the same scale, which shows a close correspondence with the effectiveness score. This implies that if design factors are improved, the effectiveness of the EONT-ODL system and of instructional material will show higher effectiveness score. The least score was found with respect to the preferred mode of study (Mean=2.80). Student responses indicate that there is a slight disagreement that the EONT-ODL system or mode of instruction is comparatively more preferred than the traditional face-to-face course instruction. This is explained by the attachment of students to more human-centred modes of instruction and secondarily on problems related to the administration, design, and development of EONT-ODL system.

5. Concluding Remarks and Recommendations

Generally, the results obtained from this evaluation research seemed to suggest that the quality of courseware design is of considerable importance in producing an effective Web-based ODL system. Similar results are reported by [Barker and King 1993]. Well-designed and dynamic (e.g. animation sequences, motion, and sound) interfaces must be provided to optimize the effectiveness of an ODL courseware, both in terms of content presented and the interactions supported by the system. The analysis of the qualitative data also suggests that in any web-based ODL courseware, designers and courseware developers should provide the maximum amount of user control, integrate group-oriented or collaborative learning assignments, use feedback

messages to reinforce performance, focus on the object-oriented programming, incorporate case tools and material related to new trends in the field, and provide diagnostic messages to correct errors. The integration of more authentic tasks in the form of interactive examples, case studies, and simulation was particularly stressed by respondents. Authentic tasks are those that have real-world relevance and utility, that have inter-disciplinary potential, that provide appropriate levels of complexity, and that allow students to select appropriate levels of difficulty or involvement [Jonassen, 1991].

This evaluation research also suggests that the ODL system design should be centred on the characteristics of the students, the courseware and the nature of the learning task rather than on the underlying technological platform. A number of suggestions were made for achieving this, such as: including more interactive examples, assignments, animation, and exercises, enriching the content with new material which should be presented in a more analytical way, providing summaries at the end of each chapter, highlighting the major aspects and key concepts, and providing better consistency of the topics, especially by merging small units. The world of tomorrow will be digital. It is certain that education will not remain unaffected by this evolution. New technologies seem promising to be used effectively. However, it is still difficult (and premature) to draw a firm conclusion rather than a tentative one.

Equation Number 1      Dependent Variable..       EONT-EFFECTIVENESS 

Variable(s) Entered on Step Number 
   1..    COURSEWARE DESIGN 

Multiple R        .52786 
R Square          .27864         R Square Change      .27864 
Adjusted R Square .26186         F Change           16.60967
Standard Error    .48883         Signif F Change      .0002 

Analysis of Variance 
                     DF      Sum of Squares       Mean Square 
Regression            1             3.96902           3.96902 
Residual             43            10.27520            .23896 

F =      16.60967     Signif F = .0002 

Variable(s) Entered on Step Number
    2..   MODE OF INSTRUCTION 

Multiple R        .62587 
R Square          .39171         R Square Change      .11307 
Adjusted R Square .36274         F Change            7.80676 
Standard Error    .45420         Signif F Change      .0078 

Analysis of Variance 
                     DF       Sum of Squares       Mean Square
Regression            2              5.57956           2.78978 
Residual             42              8.66465            .20630 

F =      13.52284     Signif F =     .0000 

Variable(s) Entered on Step Number 
   3..    MEETINGS WITH INSTRUCTOR

Multiple R        .69689 
R Square          .48565         R Square Change      .09394 
Adjusted R Square .44802         F Change            7.48857 
Standard Error    .42272         Signif F Change      .0091 

Analysis of Variance 
                     DF       Sum of Squares       Mean Square 
Regression            3              6.91773           2.30591
Residual             41              7.32649            .17869 

F =      12.90418     Signif F =     .0000

 ------------------ Variables in the Equation ------------------ 
Variable              B         SE B        Beta            T     Sig T 

DESIGN          .458597      .102515     .502194        4.473     .0001 
MODE            .253850      .089183     .320009        2.846     .0069 
MEET WITH INSTR .127335      .046532     .307003        2.737     .0091 
(Constant)      .581848      .434064                    1.340     .1875 

Acknowledgements

The project EONT is partially funded by the European Commission's SOCRATES Program. Its reference number is TM-OP-1995-1-GR-88(1/1). (URL address: http://hyperg.softlab.ntua.gr/EONT)

References

[Barker & King 1993] Barker, P. & King, T.: ''Evaluating interactive multimedia courseware- a methodology``; Computers and Education, 21, 4 (1993), pp.307-319.

[Dillon & Gunawardena 1992] Dillon, C. & C. Gunawardena, C.:. ''Evaluation Research in Distance Education``; British Journal of Educational Technology, 23, 4 (1992) pp.181-194).

[Ford et al. 1996]. Ford, P. et al.,:''Managing Change in Higher Education``; London: Open University Press (1996).

[Guba & Lincoln 1981] Guba, E.G. & Y.S. Lincoln, Y.S.: ''Effective Evaluation``; San Francisco: Jossey-Bass (1981).

[Hartley et al. 1994] Hartley, J. et al.,: ''The Comparative Evaluation of Computer Conferencing with other Methods of Teaching and Learning.`` Dept. of Organisational Psychology, University of London (1994).

[Hiltz, 1995] Hiltz, S.R.: ''The Virtual Classroom: Learning without Limits via Computer Networks``; Norwood, NJ: Ablex Publishing Corporation (1995).

[House 1981] House, E.R.: ''Realism in Research``; Educational Researcher, 20, 7 (1981) (pp. 2- 22).

[Jonassen 1991] Jonassen, D.H.: ''Evaluating constructivistic learning. Educational Technology``; 31, 9, (1991) pp. 28-33.

[Kaye 1991] Kaye, A.: ''Learning Together Apart``; In A. Kaye (Ed.), Collaborative Learning Through Computer Conferencing. Berlin: Springer Verlag (1991) (pp.1-24).

[Koutoumanos et al. 1996] Koutoumanos et al.:.''Towards a Novel Networked Learning Environment``; Proc. of the World Conference of The Web Society (WebNet'96) San Francisco-USA, November 1996.

[Maddux 1996] Maddux, C.D.: ''The State of the Art in Web-Based Learning. Computers in Schools``; 12, 4, (1991), pp.63-71.

[Makrakis 1996] Makrakis, V.: ''Assessing the Internet Connection on Future Educators: A Case Study from the University of Crete``; Proc. of CATE 96, Computers and Advanced Technologies in Education, CAIRO (1996) (pp.71-87).

[Makrakis 1997] Makrakis, V.: ''Data Analysis in Scientific Research: From Theory to Praxis``; Athens: Gutenberg (1997) (In Greek).

[Makrakis] Makrakis, V.: ''Issues of Designing, Developing and Evaluating Educational 13

Software``; (Forthcoming).

[Marshall & Hurley 1996] Marshall, A.D. & Hurley, S.: ''The Design, Development and Evaluation of Hypermedia Courseware for the World Wide Web``; Multimedia Tools and Applications: An International Journal, 3, 1 (1996) (pp.5-31).

[Maurer 1996] Maurer, H.: ''Hyperwave: The Next Generation Web Solution``; Addison Wesley (1996).

[McConnell 1991] McConnell, D.: ''Computer Mediated Communication for Management Learning``; In A. Kaye (Ed.), Collaborative Learning Through Computer Conferencing. Berlin: Springer Verlag (1991) (pp.51-68).

[McConnell 1994] McConnell, D.: ''Implementing Computer Supported Cooperative Learning``; London: Kogan Page (1994).

[McDougal. & Squires 1995] McDougal, A. & Squires, D.: ''An empirical study of a new paradigm for choosing educational software``; Computers and Education, 25, 3, (1995) pp. 93- 103.

[Nielsen 1990] Nielsen, J.: ''Evaluating Hypertext Usability``; In D.H. Jonassen & H. Mandl (Eds.). Designing Hypermedia for Learning. NATO ASI Series Vol. F67. Heidelber: Springer Verlag Berlin (1990).

[Riel. & Harasim, 1994] Riel, M. & Harasim, L.: ''Research Perspectives on Network Learning``; Machine-Mediated Learning, 4, .2&3, (1994) pp. 91-113.

[Papaspyrou et al. 1996]. Papaspyrou, N. et al.,: ''An Experiment in ODL using New Technologies``; Proc. of World Conference of Web Society (WebNet'96), San Francisco-USA, November 1996.

[Reeves 1992] Reeves, T.: ''Evaluating Interactive Multimedia. Educational Technology``, 32, 5, (1992) pp47-53.

[Retalis 1997] Retalis, S.: ''A Coureseware Development Methodology for ODL``; Proc. of CAISE' 97, Barcelona, Spain, May 1997.

[Shadish, Cook & Leviton 1991] Shadish, W. R., Cook, T.D. & Leviton, L.C.: ''Foundations of Program Evaluation: Theories of Practice``; Newbury Park, CA: Sage Publications (1991).