Assessment Issues for Educational Software

1 Introduction

The following eight articles are a continuation of a collection begun in the March issue of J.UCS. My introduction to Vol. 4, no 3 - "Advanced Educational Technologies - Promise and Puzzlement" - discusses the need for broadening the paradigm for assessment. This April issue continues with the theme of how we might expand our notions of assessment for advanced educational technologies in order to mediate change.

We are witnessing a global renewal of interest in improving teaching and learning. We are also seeing a related and concomitant focus on a re-definition of education that includes fundamental changes in the role of teacher, learner, and delivery mechanisms. Such large-scale re-evaluations mandate a broader perspective for assessment.

Figure 1 accounts for a spectrum of features in the "ecology" of education. This model suggests systematic and defensible design/evaluation of software/courseware in situ to promote "best teaching practices." Furthermore, the model implies consideration of a range of issues including integration of information technologies into existing curricula (easing the transition from traditional to innovative), establishing creative partnerships among students, technologies, and teachers, as well as selecting courseware /software for specific needs.

Suggested research questions that can be derived from this model include:

• Administration
• Developing infrastructure
• Policies and politics
• Incentives / inhibitors to change
• Support networks and intra-institutional communication
• Teachers
• Disciplines and intellectual cultures
• Teacher Training / In-service opportunities
• Methods of becoming "stakeholders" in technology transfer
• Support groups (intra and inter-organizational)
• Pedagogy and teaching styles
• Learners
• Computer Supported Collaborative Learning
• Individual characteristics
• Learning "cultures" and "sub-cultures"
• Cognitive, social, and affective dimensions of learning

2 Strategies for Change

The eight articles collected here present compelling examinations of a changing role for assessment in educational delivery and instructional enhancement. These articles provide understanding for a complex socio-

technical shift and serve as the basis for more informed decision-making among all constituencies concerned with education. I have grouped these eight papers into three thematic units.

2.1 Thinking Outside the Box

These three articles challenge us to move beyond the current philosophies, strategies, and methods. Mary E. Hopper ("Assessment in WWW-Based Learning Systems: Potentials and Opportunities") reminds us that as educators, our profession may be no better than its tools when it comes to automated forms of instructional assessment. She urges that manufacturers of authorware move beyond the idea of embedded "tests" as a method of measuring outcome. Specifically, Hopper suggests that educators lead the way toward enriched automated tools that reflect aspects of modern pedagogy - such as collaborative learning and constructivism.

Daniela Giordano ("Bridging Qualitative and Quantitative Approaches in Evaluating the Educational Effectiveness of a Shared Design Memory") suggests a new, more sensitive and sophisticated method of combining the strengths of several evaluation instruments through cross-referencing. This more robust approach recognizes that advanced educational technologies have both social/affective dimensions as well as cognitive development aspects. Giordano's approach is here applied to an evaluation of a shared design memory; however, it is appropriate for any multi-stages task using complex, highly creative acts of mentation.

Emilia Mendes, Wendy Hall, and Rachel Harrison ("Applying Metrics to the Evaluation of Educational Hypermedia Applications") suggest yet another approach for assessing advanced computer-mediated educational systems. These authors present a metric for using development time and system robustness as a reliable measure for the highly subjective activity of crafting educational software. As with the other articles in this cluster, this paper - by extension - suggests how we can avoid robbing the digitized world of its powers and strengths.

2.2 From Numbers to Knowledge

The two papers I have placed in this second cluster focus on methods for extracting meaningful interpretation from the data that can be automatically collected in a computer-mediated learning environment. Milton Campos ("Conditional Reasoning: A Key to Assessing Computer-based Knowledge-building Communication Processes") presents a method for collecting precise observations, reducing this rich - but minute representation - to analyzable patterns, and subsequently inferring the

occurrence of higher-level forms of cognition. This method of gleaning observations about the efficacy of an educational system is a bit like the more traditional method of protocol analysis. However, automation makes the procedure less resource intensive and - arguably - less subjective.

Peter Brusilovsky and John Eklund ("A Study of User-model Based Link Annotation in Educational Hypermedia") suggest that intelligent hypermedia systems (those whose links and nodes are annotated so that there is a knowledge-base within the web) can contain information robust enough to serve as a user-model to drive an adaptive pedagogy in an automate learning environment. The paper covers the evaluation of a hypermedia system that provides adaptive advice on navigating through a body of content using the embedded link annotation and student usage patterns as a source for evaluating individualized, optimal paths through the material.

2.3 Systems in Situ

Articles in the third cluster point to enriched ways of assessing educational technologies from a systemic or situational point of view. Ise Henin ("Evaluation of On-line Help") looks at expectations for on-line help in computer applications and identifies general problem-solving strategies necessary for timely, cogent assistance within a framework of solving problems.

Sabine Volbracht, Gitta Domik, Dorothea Backe-Neuwald, and Hans-Dieter Rinkens ("The 'City Game': An Example of a Virtual Environment for Teaching Spatial Orientation") ask the question of whether virtual reality (as a new, more advanced media) produces a better learning outcome than more traditional instructional materials. In short, the study ask the question of whether advanced media become a high-tech, high-cost equivalency for something educators can already do just as effectively with an alternative medium.

Douglas Williams, Susan Pedersen, and Min Liu ("An Evaluation of the Use of Problem-Based Learning Software by Middle School Students") present the case for using rich hypermedia systems to instantiate modern learning theories such as constructivism implemented in a problem-based learning scenario.