Development and Evaluation of Web-based In-Service Training System for Improving the ICT Leadership of Schoolteachers¹

Abstract: This paper describes an analysis of the effectiveness of an in-service training system developed by a project sponsored by the foundation of the Information-technology Promotion Agency, Japan (IPA). We developed and carried out a 10 days training course for 65 teachers in three different locations. The three main elements of this course were (1) training curriculum, (2) CD-ROM materials, and (3) Web-based support system. The participants of this course were hoping to become Information Technology (IT) leaders in their schools. An analysis was conducted to investigate the factors influencing the effectiveness of the training.

Based on our analysis, we were able to draw the following conclusions:

(1) Web-based training support system and CD-ROM materials were very effective to improve teachers' knowledge and skills, regardless of prior knowledge and skills

(2) Traditional instructional style (i.e. one-way instructor-centered style) was generally an ineffective training method.

(3) CD-ROM materials significantly enhanced the effectiveness of teachers' creating materials (such as a Web page), especially when the CD-ROM was used for self-study.

These findings will be useful for educators and educational designers who plan and conduct in-service training programs.

Keywords: Distance Education, Teacher Training System, Web-based Learning, Teacher Education, Training Program Assessment

Category: K.3

1 Introduction

The Japanese Ministry of Education decided to introduce the Internet to all Japanese schools (i.e. elementary to high schools) by 2001 to promote students' Information Communication Technology (ICT) literacy and the effective utilization of information tools to enhance subject-area learning. It has been reported that in order to realize information technology education, three major factors are crucial: Preparing sufficient hardware, providing excellent software, and promoting teacher's ICT literacy [Akahori 98a]. Several papers describe certain barriers for integrating ICT into curricula [Lawson, Comber 99b], Ertmer 99b]. The Ministry of Education in Japan introduced several projects aimed at promoting teacher training by removing these barriers. One of these projects was name "Project for development and evaluation of in-service training system for improving the ICT literacy of schoolteachers" sponsored by the Information-technology Promotion Agency foundation (IPA). This project was divided into two sub-projects, one to develop ICT leadership of schoolteachers, and another promote the training of ICT coordinators. This article examines the development of ICT leadership of schoolteachers, especially focusing on the evaluation of in-service training system.

The project consists of about forty members, joined by a variety of experts including engineers from computer software companies, ICT-experienced teachers, and university researchers. The project has conducted over one year starting in 1998, and included the following elements:

1. Development of training curriculum and support system
2. Implementation of the training according to the curriculum using the support system
3. Analysis of effectiveness of the training system

2 Outline of in-service training system

The training system consists of major three components: (1) training curriculum, (2) CD-ROM materials and (3) Web-based support system.

2.1 Training curriculum

The guiding concept of the training curriculum is based on a constructivist approach [Reeves, Reeves 97], and computer-mediated, problem-based training such as the learning from doing complex, challenging, and authentic problem through collaboration and communication [Laffey, Tupper, Musser, Wedman 98b]. The curriculum was designed to achieve the following goals:

1. To train ICT leaders to play the central role in promoting classroom teachers' ICT literacy in each school.
2. To attach greater importance to practical skills (rather than computer operation skills) by using ICT tools in actual classes [Ertmer 99b], [Willis, Thompson, Sadera 99b].

Page 212

3. To improve basic skills of networking such as connecting to the Internet and configuring the school's LAN.
4. To develop consultation skills for the leader to provide advise classroom teachers.
5. To provide a computer-mediated learning-support system for sharing knowledge dynamically.

2.2 Development of CD-ROM materials

CD-ROM materials were developed in cooperation with a computer software engineer, an expert teacher, and university researchers according to the training curriculum. Many resources related to the training course, including teachers' case studies and video of actual classes using ICT tools which were recorded onto CD-ROM and used for instructor's presentation and participant's self-learning [Jonassen 99b]. Figure 1 shows an example of an interview scene of expert teachers on CD-ROM materials, five disks in total.

2.3 Web-based support system

Figure 2 shows the configuration of a web-based support system for training. This support system consists of many modules such as database references, collaboration support, registration, and management of bulletin boards, checking items and participant's response in order to realize sharing discussion, collaborative

group-based works and participants' reflection on each other's work [Colllins 99b], [Sannomiya, Kawaguchi 99b]. Figure 3 shows an example of a bulletin board in the system where participants can submit and share their opinions.

3 Design of system evaluation

The training program was conducted in three different locations. Each program ran for a period of 10 days, and a total of 65 teachers participated. These participants were identified as candidates to become school ICT leaders. The program for the training course, which consists of 12 topics, is shown in table 1.

Evaluation plays a critical role in the improvement of a training system. Despite the importance of evaluation, little research exists to analyze the factors contributing to the effectiveness of the training system [Kumar 99b], [Ravitz 97], [Tessmer, McCann, Ludvigsen 99b]. In order to obtain findings helpful to the design of the training program, we carried out an evaluation according to the scheme shown in Figure 4. Data were collected from various sources in order to clarify the following three aspects: (1) How was the training carried out? (2) Was the training effective? (3) How was the training perceived by the participants?

Based on the overall curriculum and training strategies, instructors were asked to draw a lesson plan, which included process of training and a product evaluation plan. All training scenes (such as participants' talk, activities, collaborative work and instructors' presentations) were videotaped for measuring activities' time and conducting qualitative analysis, so we would know how the training was conducted.

As a measure of training effectiveness, we introduced a total of 106 items organized into 12 topic areas, for participants to evaluate themselves their knowledge and skills. Participants were required to respond to these 106 items before taking the course (pre-check), again just after taking each of the courses (post-check; about 10 items each), and one last time after all the training was finished (after check). We compared differences among scores of pre-, post-, and after check scores in the course. In order to analyze the factors influencing differences of pre- and post- scores, we carried out the regression analysis shown in a later section.

In order to obtain participants' reaction for their new training experience, we conducted questionnaires with pre-, post-, and after checking. Questionnaire included such aspects as contents, instructors, training style, CD-ROM and Web-based support system, and built-in self-evaluation processes. All of evaluation data were used to analyze the strengths and weakness of various aspects of the training system for confirmation of effectiveness and further revisions.

4 Results of differences among pre, post and after taking the course

The rating system consisted of four levels: 4 (good understanding), 3 (some understanding), 2 (little understanding), and 1 (no understanding at all). The maximum score is 424 (calculated by 4 times 106), and the minimum score is 106. Figure 5 shows the change of total scores with post- and after check sorted on pre-check scores. This figure shows that the scores with post- and after check are almost independent on pre-check scores, which means the training program was effective regardless of participants' prior knowledge and skills.

The figure also shows three regions which are the pre-achieved region before training, the achieved region by the training. and the unachieved region in spite of the training. Therefore, we can evaluate the area size of the achieved region by the training as the training effectiveness. In roughly estimation, it was as desirable as expected.

4.1. Analysis of factors affecting the training effectiveness

In order to pick up the factors affecting the training effectiveness, we made the following assumptions.

Assumption 1. The in-service training system will be effective to all participants with various prior ICT knowledge and skills, as anticipated from the previous section. .

Assumption 2. The Web-based training support system and CD-ROM materials will work well with comparison to the traditional training system such as lecture-based instructional style.

The model for testing these assumptions is as following.

Post-check score <- pre-check score + Web-based support system
                     + CD-ROM materials + instructional style

Here, pre-check score represents participants' prior knowledge and skills, post-check score represents participants' knowledge and skills achieved by the training, and instructional style shows instructor's presentation, participants' individual learning, group works and so on. This model is based on the relation that the post-check score, it is called as a dependent variable, is influenced by other independent variables such as pre-check score, Web-based support system, CD-ROM materials and instructional style. Here, we call the above independent variables as factors affecting the training effectiveness

The regression analysis should be suitable for evaluating the factor's size. The factor's size can be obtained by calculating the regression coefficients of each independent variable in the above regression equation. Here, we will adopt the time spent during the training course as a value of independent variables except pre-check score. This means that the factor with more spent time contributes to the training effectiveness more than the less spent time. We can estimate the size of regression coefficients as the affecting size of factors. Time spent in each factor was measured using videotapes recording all training course. A part of data is shown in table 2.

In the table, topics number shows topics in the training course, 12 in total, factors show lecture, presentation, group work Web-based support, CD-ROM for use and so on, and numerical data in the cell show the spent time in each factor and each topics with the unit minutes except scores.

Table 3 summarized the result, data of the regression coefficients' size, calculated by regression analysis and the result is shown in figure 5 as a visual graph. From table 3 and figure 6, we could find the following results.

1. The biggest influencing factor was the Web-based support system, which was statistically significant within 1 percent level.
2. More spent time using CD-ROM for self-learning contributed to higher scores in the post-check.  This means that the CD-ROM is more effective when used for self-study than other uses such as group work and instructor's presentation.
3. More spent time in instructor's presentation was less effective to the scores of post-check, which was statistically significant. This means that a traditional teaching style, such as one way directed lecture-based style, is ineffective to the training.
4. Differences were found between teachers of elementary and junior high schools.
   The elementary school teachers showed higher post-check scores than junior high school teachers, it is also deffrencees between training sites.
5. Scores of post-check were almost independent to scores of pre-check. This means that the training system will work effectively regardless of quantity or quality of a teacher's prior knowledge and skills as expected in the previous section.

From the above results, we can conclude that the first and second assumptions substantiated.

4.2. Analysis of factors affecting on the works produced by participants

In the training course, participants were assigned to produce works such as homepage, a lesson plan using CD-ROM, and so on. Instructors rate the produced works with four grades from the points of check items' view. The regression analysis was done using the similar method as the previous section. The result is shown in table 4.

We could find only the factor of CD-ROM materials as statistically significant, shown in table 4. From this result, we could find that only using CD-ROM materials for self-learning works effectively. In contract, the use of CD-ROM materials for instructor's presentation was relatively ineffective. It can be interpreted that one way directed instructional style was also ineffective to work-production, which was the same finding as shown in the previous section.

5 Conclusions

The conclusion is summarized as the following:

1. Web-based training support system and CR-ROM materials were effective to improvement of teacher's knowledge and skills, regardless of quantities and qualities of prior knowledge and skills.
2. Traditional training style, one-way directed and instructor-centered style, was ineffective to in-service training.
3. Using CD-ROM materials for self-learning enhanced the quality of teachers productions.

From the above conclusions, we can give some useful suggestions to educators and administrators who plan to conduct in-service training courses. The important points are:

1. Modify the training approach from a traditional teacher-centered to a participant-centered style [Collins 2000b],
2. To introduce rich materials for self-learning or group works [Dillon, Tearle, Hudson 99b],
3. To introduce Web-based support system,
4. To promote opportunities for exchanging teachers' knowledge and skills, especially rich-experienced practical know-how.

Suggestions (3) and (4) are very important because teachers can share their know-how with others, which works well when conducting real lessons in schools [Lawson, Comber 99b]. Web-based support system enables teachers to share their know-how and knowledge anytime, anywhere, and with anyone. Moreover, CD-ROM materials also can be used at anytime, anywhere, and by anyone. Therefore, it is important to provide rich self-learning resources that contain expert teachers' case studies and a support system that enables the sharing of dynamic, experience-based knowledge, and communication. Such resources can truly be called integrated training environments.

Acknowledgements

The authors would like to thank Ms. M. Suzuki, Fujitsu Learning Media Co., for her great assistance, and also Dr. D. J. Scott and Dr. M. Yanagisawa for their cooperation. We would also like to express our appreciation to the following people who assisted with this project: Mr. T. Maesako, Mr. K. Hatano, Mr. T. Horita, Mr. T. Nakagawa,

Mr. T. Sanada, Mr. K. Nagashima, Mr. M. Suzuki, Mr. J. Tamura, Mr. K. Nakagawa, Mr. K. Tanaka, Mr. Y. Mohri, Mr. M. Maeda, Mr. M. Ichikawa, and Mr. M. Itoh.

References

[Akahori 98a] Akahori K., "Effectiveness and some problems of the Internet: Utilization to education from overview of school practice research in Japan", proc. of SITE (Society for Information Technology & Teacher Education), 98 International Conf. in Washington, DC, 1157-1156,1998 

[Collins 2000b] Collins M., "Comparing Web, Correspondence and Lecture versions of a second-year non-major Biology course", British J. of Educ. Technol. Vol.31, No.1, 21-27, 2000

[Colllins 99b] Collis B., "Designing for differences: cultural issues in the design of WWW-based course-support sites", British J. of Educ. Technol. Vol.30, No.3, 201-215, 1999

[Dillon, Tearle, Hudson 99b] Dillon P., Tearle P., Hudson A., "The Images for Teaching Education Project: developing multimedia resources within an integrative framework", British J. of Educ. Technol. Vol.30, No.4, 359-367, 1999

[Ertmer 99b] Ertmer P. A., "Addressing First- and Second-Order Barriers to Change; Strategies for Technology Integration", Educational R & D, Vol.47, No.4, pp.47-61, 1999

[Jonassen 99b] Jonassen, D., Rohrer-Murphy, L., "Activity Theory as a Framework for Designing Constructivist Learning Environments", Educational R & D, Vol.47, No.1, pp.61-79, 1999

[Kumar 99b] Kumar K.L., "Assessment of the Effectiveness of a Short Course via Internet", Educational Technology Research (in Japan), Vol.22, pp.27-33, 1999

[Laffey, Tupper, Musser, Wedman 98b] Laffey J., Tupper T., Musser D., Wedman J., "A Computer-Mediated Support System for Project-Based Learning", Educational R & D, Vol.46, No.1, pp.73-86, 1998

[Lawson, Comber 99b] Lawson T., Comber C., "Superhighways Technology: personnel factors leading to successful integration of information and communications technology in schools and colleges", J. of Information Technol. for Teacher Educ. Vol.8, No.1, 41-53,1999

[Ravitz 97] Ravitz J., "Evaluating Learning Networks: A Special Challenge for Web-based Instruction", in "Web-based Instruction" (Editor Khan B.H.) Educational Technology Publications, New Jersey, 361-368, 1997

[Reeves, Reeves 97] Reeves, T.C., Reeves, P.M. "Effective Dimensions of Interactive Learning on the World Wide Web", in "Web-based Instruction" (Editor Khan B.H.) Educational Technology Publications, New Jersey, 59-66, 1997

[Sannomiya, Kawaguchi 99b] Sannomiya, M., Kawaguchi A., "Cognitive Characteristics of Face to Face and Computer Mediated Communication in group Discussion: An Examination from Three Dimensions", Educational Technology Research (in Japan), Vol.22, pp.19-25, 1999

[Tessmer, McCann, Ludvigsen 99b] Tessmer M., McCann D., Ludvigsen M., "Reassessing Training Programs: A Model for Identifying Training Excesses and Deficiencies", Educational R & D, Vol.47, No.2, pp.86-99, 1999

[Willis, Thompson, Sadera 99b] Willis J., Thompson A., Sadera W., "Research on Technology and Teacher Education: Current Status and Future Directions", Educational R & D, Vol.47, No.4, pp.29-45, 1999