(V. Makrakis, S. Retalis, A. Koutoumanos, N. Papaspyrou, M. Skordalakis) Evaluating the Effectiveness of an ODL Hypermedia System and Courseware at the National Technical University of Athens: A Case Study

Abstract: The main purpose of this paper is to test a conceptual framework for the evaluation of the effectiveness of an open and distance learning (ODL) hypermedia system (EONT-ODL system) and courseware developed and trial used at the National Technical University of Athens (NTUA), Greece, within the EONT project. EONT is a partnership project between seven universities from seven European Union countries within the Socrates Framework Program. In this paper we deal with data elicited from the NTUA, since it was the only partner institution which provided adequate data for quantitative analysis. The evaluation framework is based on the assumption that ODL hypermedia systems are complex systems with a variety of organisational, administrative, instructional, and technological components. It has been hypothesized that the effectiveness of the EONT-ODL system would be influenced by a number of independent variables such as: design and presentation of the courseware, previous experience, time spent on working through the courseware, preference of mode of study, learning styles, interactions with peers, instructors and means of communication. In this evaluation research, two instruments integrated into one questionnaire for data collection were developed: the first was based on a number of standardized questions, reflecting the previously stated theoretical framework and the second on a number of open-ended questions, reflecting, likes and dislikes, added value, problems identified, suggestions etc. The regression analysis indicates that the `design and presentation of instructional material' alone explained almost 28% of the EONT- ODL system's effectiveness (R2 adj.= .278). The preferred `mode of study' entered second by adding 11% (R2 ch.=.113) of the effectiveness variance and finally students interactions with the instructor increased the effectiveness explained variance to 48%, a quite high percentage accounted for three significant predictors alone. All the other predictors, that is, previous experience with computers, time spent working with the EONT-ODL courseware, student learning styles, and interactions among students and communication means (e-mail and computer conferencing) did not significantly contribute to the prediction of the effectiveness measure. These quantitative results are complemented by the qualitative conclusions.

Keywords: Evaluation, Hypermedia Courseware, Open and Distance Learning systems