Aligned, Embedded and Scalable Assessment Technologies

J.UCS Special Issue

1 Introduction

Information and Communication Technologies are providing enhanced and new opportunities to assess learning. On the one hand, educational practice includes the use of off-the-shelf technologies (surveys, mind maps, etc.) since their affordances drive creative teachers to design activities with assessment purposes. On the other hand, a first generation of Assessment Technologies, particularly designed to this aim, has been focused on online tests or quizzes composed of traditional types of questions (e.g. multiple choice, multiple response) [Redecker, 2013]. These assessment technologies are typically integrated in Virtual Learning Environments (e.g., Moodle) and there are learning technologies specifications providing interoperable formats for the representation of tests and question items (e.g., IMS Question & Test Interoperability). This special issue solicited original research papers framing a new generation of Assessment Technologies facilitating aligned, embedded and scalable assessment. Section 2 describes we mean by aligned, embedded and scalable assessment.

The special issue has been possible thanks to the support of the steering and organization committee of CISTI2014, Conferencia Ibérica de Sistemas y Tecnologías de la Información, and of the colleagues acting as reviewers for this special issue. We received 13 submissions, including extended versions of CISTI2014 papers and additional papers submitted as a result of an open call for papers. Section 3 summarizes the five contributions that have been included in this special issue after a careful peer-review-process. The contributions have been written by authors from four different countries. Finally, section 4 lists the colleagues that have served as reviewers for this special issue.

2 Technologies for Aligned, Embedded, Scalable Assessment

Technologies for aligned assessment allow alignment of assessment with the intended learning outcomes by making possible scenarios in which students can demonstrate the competences they have developed in authentic contexts [Biggs, 03; Santos, 12]. This is facilitated by advanced learning technologies including digital games, mobile technologies, immersive learning scenarios, etc.

Technologies for embedded assessment enable the integration of assessment activities in learning flows, where the result of the assessment may condition the following learning activity to be presented to the students [Villasclaras, 13]. Technologies supporting personalization or adaptive learning, the use of interoperability specifications and standards can offer relevant approaches for embedded assessment.

Technologies for scalable assessment are especially critical in curses with no constraints in class size (e.g., Massive Open Online Courses). Assessment technologies supporting feasible assessment in massive online environments include quizzes embedded in videos, but also self-assessment and peer-assessment systems [Kay, 13].

3 Contributions of the special issue

The five contributions comprising this special issue allow aligned, embedded or scalable assessment, showing some of them support for a combination of the assessment aspects.

The first paper "Introducing a Formative E-Assessment System to Improve Online Learning Experience and Engagement" presents a general tool implementing a formative e-assessment model [Hettiarachchi, this issue]. The tool is designed for integration into the learning process so as to facilitate embedded assessmenthat is aligned with the expected learning outcomes. Evaluation results show that the tool is able of providing students with personalized feedback, also suitable for scalable skill assessment purposes, by tracking students' progress considering elements that go beyond related approaches mainly based on quizzes.

The article "Helping Teachers Align Learning Objectives and Evidence: Integration of ePortfolios in Distributed Learning Environments" [Lozano-Alvarez, this issue] proposes a scalable approach to tackle evidence collection in distributed learning environments. The approach builds on a strategy of alignment of the artefacts generated by students along the learning experience with the pedagogical intent expressed in a learning design. Based on this approach, the paper introduces, and illustrate with an evaluation study, a technological infrastructure for the automatic collection and organization of pieces of evidence in an ePortfolio.

The paper "Interoperability Framework for Competencies and Learning Outcomes" introduces a model for identifying, classifying and reusing definitions of competencies and learning outcomes (MICRA) and an ontology of the information management model (SICRA) [Angélico-Gonçalves, this issue]. The MICRA model has been applied to a particular case, where correspondence with Computer Science Knowledge Areas is shown. The verification of its defined functionalities led to ontology validation. This approach contributes to further formalization of competency descriptions that can serve purposes of evaluating the alignment between learning activities, assessment and competencies within a course and when contrasting courses in the frame of the European Higher Education Area.

In the paper, "A Context-aware Approach for Personalized Mobile Self- Assessment" [Harchay, this issue], the authors use semantic Web technologies to support embedded self-assessment in mobile learning situations. The proposed techniques enable personalized assessment activities and semantic access to contextualized semantic assessment resources. An evaluation of a system implementing the proposed techniques is also presented in the paper.

The paper "Virtual Learning Scenarios for Qualitative Assessment in Higher Education 3D Arts" describes gamification techniques and game engines for an immersive virtual reality environment supporting Scenario Centered Curriculum activities that enable the assessment of skills related to 3D arts [Vicent, this issue]. Students immersed in a game have to think, design, convey, validate, and build a civil project using technologies that help in the assessment process. The assessment is embedded in the learning process of creating of a 3D complex model focused on the construction of a building in a virtual space.

4 Reviewers

We would like to recognize all our colleagues who served as reviewers for this special issue. We express our gratitude for their valuable work on reviewing all contributions.

• August Climent (La Salle Open University, Andorra)
• Raquel M. Crespo (Universidad Carlos III de Madrid, Spain)
• Michael Derntl (RWTH Aachen University, Germany)
• Juan M. Dodero (University of Cádiz, Spain)
• David Fonseca (Universitat Ramon Llull, Spain)
• Paulo Fosatti (Unilasalle Canoas Brazil)
• Cynthia Ganote (Saint Mary's University of Minnesota, USA)
• Ana Guerrero (Universitat Oberta de Catalunya, Spain)
• Wolfgang Greller (Vienna University of Education, Austria)
• Enosha Hettiarachchi (Universitat Oberta de Catalunya, Spain)
• Rebecca Hopkins (Saint Mary's University of Minnesota, USA)
• Seiji Isotani (University of São Paulo, Brazil)
• Marco Kalz (Open University of the Netherlands)
• Alejandra Martínez-Monés (Universidad de Valladolid, Spain)
• Javier Melero (Université Pierre et Marie Curie, France)
• José Montero (Universitat Ramon Llull, Spain)
• Yishay Mor (Independent Consultant, UK)
• Mar Pérez-Sanagustín (Pontificia Universidad Católica de Chile, Chile)
• Francesca Pozzi (Instituto per a la Tecnologia Didattiche, Italy)
• Simos Retalis (University of Piraeus, Greece)
• Alvaro Rocha (University of Coimbra, Portugal)
• Adolfo Ruiz (Tallinn University, Estonia)
• Patricia Santos (University of West England, UK)

References

[Angélico-Gonçalves, this issue] Angélico Gonçalves, M.J., Rocha, A., Pérez-Cota, M., Interoperability Framework for Competencies and Learning Outcomes, Journal of Universal Computer Science, this issue.

[Biggs, 03] Biggs, J: Aligning Teaching and Assessment to Curriculum Objectives,Imaginative Curriculum Project, LTSN Generic Centre, 2003.

[Kay, 13] Kay, J., Reimann, P., Diebold, E., Kummerfeld, B.: MOOCs: So Many Learners, So Much Potential, Intelligent Systems, 28(3), 70-77, 2013.

[Harchay, this issue] Harchay, A., Cheniti-Belcadhi, L., Braham, R.: "A Context-aware Approach for Personalized Mobile Self- Assessment", Journal of Universal Computer Science, this issue.

[Hettiarachchi, this issue] Hettiarachchi, E., Mor, E., Huertas, M.A., Guerrero-Roldan, A-E.: Introducing a Formative E-Assessment System to Improve Online Learning Experience and Performance, Journal of Universal Computer Science, this issue.

[Lozano-Alvarez, this issue] Lozano-Alvarez, A., Asensio-Pérez, J. I., Vega-Gorgojo, G., Martínez-Monés, A.: Helping teachers align learning objectives and evidence: integration of ePortfolios in Distributed Learning Environments, Journal of Universal Computer Science, this issue.

[Redecker, 13] Redecker, C., The Use of ICT for the Assessment of Key Competences, JCR Scientific and Policy Report, European Commission, 2013.

[Santos, 12] Santos, P., Hernández-Leo, D., Pérez-Sanagustín, M., Blat, J.: Modeling the Computing Based Testing domain extending IMS QTI: a Framework, models and exemplary implementations, Computers in Human Behavior, 28(2012), 1648-1662, 2012.

[Vicent, this issue] Vicent, Ll., Villagrasa, S., Fonseca, D., Redondo, E.: Virtual learning scenarios for qualitative assessment in higher education 3D arts, Journal of Universal Computer Science, this issue.

[Villasclaras, 13] Villasclaras, E., Hernández-Leo, D., Asensio-Pérez, J.I., Dimitriadis, Y.: Web Collage: an implementation of support for assessment design in CSCL macro-scripts, Computers & Education, 67(Sep 2013), 79-97, 2013.