An Evaluation of the Use of Problem-Based Learning Software By Middle School Students

Abstract: Research has shown the potential of a problem-based approach to enhance students' learning. The interactive nature of hypermedia technology and its ability to deliver information in different media formats can provide unique capabilities for implementing problem-based learning (PBL) environments. Yet, we know little about the types of tools that are effective in supporting students' learning in a hypermedia supported PBL environment. The purpose of this study was to investigate both the use of tools and design features in a piece of PBL software and their effectiveness on middle school students' learning. The findings of this study show that students who were exposed to the PBL environment increased their achievement scores from pre to posttest more than those students who learned the same content in the traditional classroom. Students' reading ability was found to be a better predictor for their achievement in PBL than their math ability. However, the brief treatment of the study had only limited impact on students' attitude toward learning science. The findings of the study and their implications are discussed in detail.

1 Research Framework

1.1 Problem-based Learning

There has been a growing body of research on authentic and situated learning environments utilizing the problem-based approach to learning [Cognition and Technology Group at Vanderbilt 1994], [Spiro, Feltovich, Jacobson, & Coulson 1992]. Problem-based learning (PBL) emphasizes solving authentic problems in authentic contexts. It is an approach where students are given a problem, replete with all the complexities typically found in real world situations, and work collaboratively to develop a solution. Problem-based learning provides students an opportunity to develop skills in problem definition and problem solving, to reflect on their own earning, and develop a deep understanding of the content domain learning [Cognition and Technology Group at Vanderbilt 1994], [Lajoie 1993], [Jacobson & Spiro 1995]. This approach was developed in the sixties for medical education, and

has since been used in various subject areas, such as business, law, education, architecture, and engineering. Howard Barrows, a pioneer in the use of PBL in

medical education, argues that though there are many variations of PBL, they share a number of common characteristics [Barrows 1996]: (1) Learning is student-centered; (2) learning occurs in small student groups; (3) teachers are facilitators or guides; and (4) problems form the organizing focus and stimulus for learning.

Literature has shown that problem-based learning can facilitate the improvement of student attitude toward the content area learning [Cognition and Technology Group at Vanderbilt 1992]. Explanations offered for this are that students perceive the relevance of the work [Barrows 1986] or compare the task of finding information and developing a solution to solving a mystery [Cognition and Technology Group at Vanderbilt 1992]. Medical students using a PBL curriculum have been shown to be able to pursue learning independently better than their peers receiving a traditional curriculum [Aspy, Aspy, & Quinby 1993], a finding which supports the claim that PBL prepares students to become independent, lifelong learners. High school students in a semester long class which used PBL exclusively showed a significant increase in their spontaneous use of one problem-solving step: problem finding, the ability to identify and formulate problem statements [Gallagher, Stepien, & Rosenthal 1992]. Similarly, students in a PBL curriculum have shown a greater ability than students in traditional classes to break complex problems into their component parts and identify subproblems which must be solved in order to solve the main problem [Cognition and Technology Group at Vanderbilt 1992], [Gallagher & Stepien 1996].

In recent years, there has been a growing interest among educators to use PBL in the K-12 setting. However, the experiences of college level programs and early efforts in K-12 schools have shown PBL to be a particularly demanding instructional approach for both students and teachers [Barrows & Tamblyn 1980], [Cognition and Technology Group at Vanderbilt 1992]. Students must perform a wide variety of tasks with which they may have had limited prior experience. These include problem finding, hypothesis generation, identification of learning needs, location of resources to meet these learning needs, data collection and organization, and development of a solution plan supported by evidence and reasoning. Teachers, in their role as facilitators, are responsible to provide support for this wide variety of activities to students who may vary greatly in their needs. Because the variability of the classroom teacher's ability to function in this role has a profound effect on the success of PBL [Walton & Matthews 1989], instructional materials which provide some of this support seem warranted.

1.2 Hypermedia and Problem-based Learning

The interactive nature of computers and the ability to deliver information in different media formats provides unique capabilities for the implementation of PBL environments. In particular, the capabilities of hypermedia have much to offer designers of advanced learning environments. Hypermedia is a rich environment containing information in many different forms, including text, graphics, audio, video, and animation. The student is not forced to access the resources in any predetermined order, but can navigate within the environment in a nonlinear fashion [Burton, Moore, & Holmes 1995]. The support of multiple media types and the possibility of

nonlinear navigation are particularly useful in the creation of computer-based PBL environments.

The nonlinear nature of hypermedia is consistent with the characteristics of PBL. As students are engaged in PBL, they need to collect data and access resources. This suggests a high degree of control on the part of the learner. Hypermedia can support this by allowing students to access needed resources at the time it is most appropriate. Hypermedia allows students to have more control over their learning. They become actively engaged in decision-making while traversing the environment. Research on learner control versus program control in hypermedia environments suggests that subjects under learner control score higher than those under program control on achievement posttest and have a more positive attitude toward learning [Hannafin & Sullivan 1995].

Much of the literature on PBL argues that learning cannot be separated from the context and the activities in which students engage. Yet it is unrealistic to think that schools can teach all subjects in real-world contexts. Because of safety concerns, cost issues, and time required, authentic environments are often impractical and difficult to create. A viable solution is to use the media capabilities of hypermedia to create environments that are consistent with real-world contexts. Hypermedia allows for the creation of authentic situations in which students can be immersed without the dangers or costs associated with real-world contexts.

Though PBL can be implemented with traditional media, hypermedia provides unique capabilities for its implementation. The nonlinear nature of hypermedia allows students to explore the PBL environment accessing resources as the need arises. Hypermedia can also facilitate the development of authenticity in the learning environment. Williams [Williams 1992] suggests that law and medical school curricula could be improved by the use of hypermedia environments to engage students in authentic activities within an authentic context.

1.3 Hypermedia and Cognitive Tools

When working in everyday situations, individuals use tools and resources such as computers, calculators, concepts, and formulas in order to solve problems. Therefore, tools must be considered when creating authentic environments for student learning. A tool does not necessarily have to be a tangible object. For example, an engineer may use a mathematical formula to calculate the area of a cylinder. Different disciplines and professions may use ideas and tools in very different ways.

Many researchers argue that cognitive tools can support and enhance student learning [Lajoie 1993], [Perkins 1991]. These tools come in many forms and can support students in a variety of tasks they must perform as they engage in problem solving. Hypermedia has the potential to make these tools readily available to students.

As discussed earlier, PBL occurs within a context where knowledge is naturally situated. Tools can be employed to create an authentic context in which students can work. The Adventures of Jasper Woodsbury series utilizes video-based scenarios in order to create a context for learning [Cognition and Technology Group at Vanderbilt 1992]. The video segment provides a focus for the learning activity and may also help students who are poor readers. Hypermedia can also present a scenario, but has the advantage of allowing students to explore the environment in which the scenario

is set. This exploration can mirror processes which people use to address problems in a real life settings.

Hypermedia can augment memory and support students in reflecting on their problem-solving process. Sherlock I, software that creates an environment in which Air Force technicians practice avionics troubleshooting, includes tools to support cognitive processing [Lajoie 1993]. Avionics troubleshooting is a complex task requiring the technician to entertain many variables and remember a series of completed tests. Sherlock I allows the student to view the steps he or she has taken in troubleshooting a problem. The ability to view the solution path supports students in reflection on the problem solving process without the need for them to rely on their recollection of every step. The software-generated problem solving steps also make explicit student thinking, an essential component in stimulating metacognitive awareness. Likewise, reflective questions can also be effective in promoting metacognitive thinking.

Hypermedia environments can offer a comprehensive set of resources to enable students to meet their learning needs. In order for students to engage in problem solving, they must have access to information. Information can be provided to students in a number of tools. In the Jasper series, the Cognition and Technology Group at Vanderbilt chose to make information available to students by embedding it in the video-based scenario [Cognition and Technology Group at Vanderbilt 1992]. Students revisit the scenario in order to sift through the information, finding items relevant to the problem at hand. Information can also be integrated in the learning environment as a searchable database. Bio-world followed this approach by providing an electronic library for students to access as they seek to diagnose patients in a simulated hospital [Lajoie 1993]. The information in the database can be in many forms. The Lab Design Project (LDP), which allows students to actively take part in sociological research concerning a fictitious biotechnology building, permits students to gather information from such diverse formats as interviews with employees, building plans, letters, and sketches [Honebein 1996].

Hypermedia can provide electronic notebooks, which, in addition to providing space for student note taking, can include advanced features to help support the student in constructing meaning. The notebook in Bio-world contains a section which displays the students' previous actions such as database searches and diagnoses for patients [Lajoie 1993]. These features help support the student's memory and metacognitive thinking. Other projects have augmented the traditional notebook with the ability to support multimedia, collaboration among students, and the ability to create links between concepts [Edelson, Pea, Gomez 1996].

Though literature supports the efficacy of problem-based learning, little research exists which investigates the types of tools or features that are effective in supporting students working in PBL environments. PBL software is beginning to find its way into schools. However, the relative effectiveness of the various tools incorporated in these programs has yet to be studied. In order to design an effective computer-supported PBL learning environment, it is important to understand the tools and design features included in the software, and their impact on learning. It is, therefore, the purpose of this study to examine and understand how middle school students use and interact with a piece of recently available computer-supported PBL software developed by a major publishing company.

2 Research Questions

This study investigates the use of tools and design features as employed in a piece of problem-based learning software and their effectiveness on middle school students' learning of science concepts. Specifically, these four research questions formed the focus of the study:

1. What is the effect of the computer-supported problem-based learning environment on the achievement of middle school science students' learning of science?
2. In what way do the students use the tools and design features built into the computer supported PBL environment while engaging in problem solving?
3. What is the effect of the computer supported problem-based learning environment on middle school students' attitudes toward learning science?
4. Is there a relationship between students' math or reading ability and their achievement in the problem-based learning environment?

3 Design Of Study

3.1 Sample

The participants of the study (N = 115) were students enrolled in seventh grade science classes at a middle school located in a medium-sized city in the southwestern United States. The school has a high percentage of minority students. The participants in the study consisted of 66% Hispanic Americans (N=76), 12% African Americans (N=14) and 22% white (N=25). The age of the students ranged from 12 - 14 years. Of the participants, 50 were male and 65 female.

3.2 Treatment

3.2.1 Treatment Conditions

This study consisted of three treatment conditions: computer-supported PBL, paper-based PBL, and a control. One class was chosen at random to be the control group. The remaining classes were assigned randomly to the treatment conditions resulting in three classes in the computer-supported PBL condition and two classes in the paper-based PBL condition. Because the treatments were included as part of the regular science class, complete random assignment was not possible in this case. There were 59 students in the computer-supported PBL condition, 38 in the paper-based PBL condition, and 18 in the control. To examine the effectiveness of the computer-supported PBL, it was necessary to compare students in the computer-supported PBL condition with students that did not use PBL (the control group), and with students in the paper-based PBL condition (problem solving using paper and pencil). Students in the computer and paper PBL groups formed small teams of four to five students in order to perform the problem-solving activity whereas the control group did not.

Each class in the computer-supported PBL condition consisted of four to six teams for a total of 15 teams, while there were 10 teams in the paper-based PBL condition.

The students in the computer-supported PBL condition used problem-based learning software recently developed by a major textbook publishing company. The CD-ROM program contains eight activities on different topics developed to support the middle school science curriculum. Upon starting one of the PBL activities, students find themselves in a virtual science laboratory. Immediately on entering the laboratory a short video segment plays in which a scientist provides important details about a scientific problem she is working on and solicits the students' help in developing a solution. The activity used for this study was concerned with the classification of a microorganism. It was selected because it was related to the topic being studied in the participating classes at the time of the study.

The students are supported in finding a solution to the problem by the availability of various tools in the virtual laboratory. An expert scientist built into the virtual laboratory provides information about the scenario and takes on the role of a mentor by providing hints and feedback during the activity. The expert scientist provides information essential in completing the activity. At the end of the video scenario the expert scientist tells the students that a fax will soon arrive reiterating the particulars of their task. As the video scenario concludes, a fax arrives in the inbox on the back table of the laboratory. The inbox provides a printed version of the problem scenario, thereby supporting students in gathering relevant information and defining the problem to be solved. The software also includes a lab manual which provides information on the use and importance of the tools found in the lab. The lab manual was intended to help orient students to the purpose and use of the lab equipment. The lab also furnishes a computer database containing information which is fundamental in solving the problem. The computer database consists of a series of hypertext documents regarding microorganisms. In addition to text, many of the documents in the database contain audio and video clips. A notepad offers students the option of taking notes during the activity. During the investigation, students have access to a microscope to view slides of microorganisms that the expert scientist has prepared on the lab table. On completion of the activity, students transmit an electronic fax with the results of their investigation. The fax ensures that students have completed the activity and serves as a tool by which the program can assess students' work and provide feedback.

The paper-based PBL group was engaged in a problem-solving activity equivalent in content and available resources to the computer-based software, except that the information given (including the information in the database and the pictures) was print-based. Though identical in the resources provided, the computer and the paper-based PBL are different in that (1) the computer-based version allows interactive access of information while the paper-based does not, and (2) the computer-based version provides information in multimedia format while the paper-based does not.

The control group learned the same content on microorganisms using the lecture-based traditional approach.

3.2.2 Treatment Phases

For the computer and paper PBL groups, the entire treatment consisted of two phases: the teacher modeling phase and the problem-solving phase. The teacher modeling phase was included because literature on PBL has shown that providing necessary scaffolding is a critical step in making PBL successful. Teacher modeling was done for both the computer-supported and paper-based conditions because we felt that the modeling should be part of problem solving regardless of the media involved. In addition, our primary interest in this study was to find out if the hypermedia environment could provide additional support to students. Therefore, the instruction of the two conditions were held to be the same. This modeling phase took about thirty minutes for each group. The control group worked through the regular lecture-based instruction on microorganisms with no intervention.

The modeling phase accomplished two purposes. First, it introduced students to the steps of problem solving and provided them with guided practice. For the computer-supported PBL group, students were given an opportunity to become familiar with the design features of the software and the tools available for their use. Second, it provided an opportunity for students to become familiar with the assigned roles they would perform during the group work. While the students had prior experience with cooperative learning, they were unfamiliar with some of the new roles assigned since the PBL environment was a new experience for them. The content used for this teacher modeling phase was different from that in the problem-solving phase.

In the modeling phase, students in the computer-support PBL treatment watched the opening video-based scenario on a large television screen. The scenario created the context for the activity by describing the problem to be solved. The teacher and one of the researchers modeled the group process for students. They discussed how to solve the problem, what steps to take, what computer-based tools to use, and what information to record. For example, the teacher modeled how to record responses to worksheet questions and how to log the tools they accessed on the tool sheet. Students then practiced recording such information on the provided print-based forms.

Students in the paper-based PBL group received similar modeling, with a few minor differences. The problem was presented through a written version of the video used in the computer-based treatment. One of the researchers read the problem aloud to the class. Similar discussions were held about how to proceed in solving the problem. The teacher and the researcher modeled the various roles students were expected to play. For example, the teacher modeled how to record their answers to the worksheet questions.

During the problem-solving phase, students in both groups were asked to work on the problem-solving activity based on the classification of microorganisms. The computer-supported group worked on the activity using the PBL software, while the paper group worked on the same activity using paper and pencil. It took both groups approximately 45 minutes to complete the activity.

3.3 Dependent Measures

3.3.1 Achievement

Students' knowledge about microorganisms was assessed through an achievement test on the content. This measure was created by the teacher and was used in previous years on the same unit. It consisted of eight short answer questions on viruses, the characteristics of microorganisms, and the role microorganisms play in the life of other organisms. All students were given the measure before and after the treatment.

3.3.2 Tool Use

Students in the computer-supported PBL group were asked to record the various tools they used in a given chart¹

3.3.3 Attitude

A fourteen-item questionnaire was used in order to examine students' attitude toward learning science. The Attitude Toward Science in School Assessment uses a 5-point Likert scale with 1 being "strongly agree" and 5 being "strongly disagree" [Germann 1988]. It addresses students' feelings about science as a subject and has a reported reliability of .95. This questionnaire was given to all students before and after the treatment.

3.3.4 Interviews and Observations

Interviews were conducted with students from 10 of the 15 teams in the computer-supported PBL condition. Of the interview questions, 70% were specific, asking students about their likes and dislikes of the program (e.g. "Did you like working with this software?"; "Which tool did you find the most useful while working on the problem?"; "Which tool did you find the least useful while working on the problem?" ). The remaining 30% of the questions were open-ended, allowing students to provide information they wanted. (e.g. "Do you have any suggestions about making this software better?"; "Is there anything you can tell us about your experience using this software?"). Observations were made by the researchers throughout the process. The researchers monitored the group problem-solving activity and answered questions from students. It was hoped that this triangulation of the quantitative and qualitative data would provide richer and more detailed information about the research questions.

¹Because the PBL software was already made, and there was no online data collection mechanism built in we created a matrix for students to record their use of tools.

3.4 Procedures

The entire experiment took place over a three-day period. On day one, the classroom teacher administered the pretest achievement measure and the pretest attitude questionnaire. On day two, the treatment took place, which consisted of the modeling phase followed by the problem-solving phase. After having completed the treatment, the students were asked to complete the posttest achievement measure and posttest attitude questionnaire. The researchers returned on day three to conduct interviews with selected students.

3.5 Analysis

To answer the first research question, "What is the effect of the computer-supported problem-based learning environment on the achievement of middle school science students?," a two-factor mixed ANOVA was conducted with the grouping (computer, paper, and control) as a between-subjects independent variable, and the data collection points (pre vs. post) as the repeated measure independent variable. The dependent variable was the pre and post achievement scores in the science content test.

To answer the second research question, "In what way do the students use the tools and design features built into the computer-supported PBL environment while engaging in problem solving?" students' use of the tools were tabulated and analyzed descriptively. Some interview questions were specifically targeted toward finding out if the tools were beneficial for problem-solving and in what ways.

To answer the third research question, "What is the effect of the computer supported problem-based learning environment on middle school students' attitudes toward science?," a two-factor mixed ANOVA was run with the grouping (computer, paper, and control) as a between-subjects independent variable, and the data collection points (pre vs. post) as the repeated measure independent variable. The dependent variable was the pre and post scores of the attitude questionnaire.

To answer the fourth research question, "Is there a relationship between students' math or reading ability and their achievement in the problem-based learning environment?," a multiple regression was performed with students' math and reading ability, measured by their most recent scores on the Texas Assessment of Academic Skills (TAAS), as the independent variables and their achievement test as the dependent variable. TAAS is a state-wide testing system that assesses the overall academic achievement of all students in Texas at different grade levels. It provides information on students' reading ability, math ability, and writing ability.

Students were selected for the post interviews after they completed the treatment. The purpose of the interviews was to find out (1) what the students liked and disliked about the environment; (2) what tools and design features they found most useful in the PBL environment; (3) their perception on using the hypermedia PBL software. The interview data were analyzed according to Miles and Huberman's framework of qualitative data analysis [Miles & Huberman 1994]. The data were first transcribed. Two researchers coded the data using the four research questions as a guide. The data were then categorized and grouped according to their common themes. In the data analysis process, the two researchers worked independently and then together to

ensure the interrater agreement was at least .95. Such qualitative data were used to provide more information and to substantiate the quantitative analyses.

4 Results And Discussion

4.1 Problem-Based Learning and Achievement

The results of the two-factor mixed ANOVA on achievement indicated that there was a significant two-way interaction between the grouping (computer, paper, and control) and the data collection points (pre vs. post) for the achievement scores: F (2, 96) =5.50, p <.01. All groups increased their achievement scores from pre to post. The gains from pre to post were significantly greater for the computer and paper groups than for the control group [see Tab. 1] and [Fig. 1]. The gain differences between the computer and the control groups (posttest -pretest =7.78), and the paper and the control groups (posttest - pretest=11.62) were significant at p < .05 level based on Fisher's PLSD post hoc tests. The gain difference between the paper and the control groups was also significant at p < .05 level according to the Scheffe post hoc test. This finding shows that both the computer and paper groups significantly improved their achievement scores after they participated in the study, while the increase for the control group was not significant. In other words, there was an effect of the problem-based learning environment on the achievement of middle school science students. Yet the difference between the computer-supported PBL and paper PBL was not significant.

* N indicates the number of students who turned in both the pre and post evaluation forms for each group.

** The pretest-posttest differences for the computer and paper groups were significantly different from that for the control group, p < .05

The results of this study are consistent with the PBL research in showing that PBL has a positive impact on students' acquisition of domain specific knowledge [Cognition and Technology Group at Vanderbilt 1992], [Gallagher & Stepien 1996]. When the students were exposed to the PBL environment, they increased their achievement scores more than those students who learned the same content in the traditional classroom. The findings also suggest that both the computer-supported and paper based PBL are equally effective in enhancing students' achievement. The lack of a significant difference in achievement between the computer and paper groups may be explained by several factors. First, in order to keep the two groups as equivalent as possible, both groups received the same instruction, same resource materials, and engaged in the same modeling and practice activity. The two groups differed only in the use of the delivery medium; one used paper and pencil and the other used

hypermedia. The use of hypermedia did not increase students' achievement more than the traditional paper medium when the problem-solving process was held constant. Second, the treatment was relatively brief. A longer treatment in which students in the computer group would be more familiar with the tools and understand the value of the tools may make a difference. Last and more importantly, it is necessary to examine the design of the hypermedia based PBL software, specifically, how the tools were designed and used by the students. It is our belief that the design of the PBL software used for this study did not encourage students to make best use of the tools.

4.2 The Use of Tools in the Problem-Based Learning Environment

Literature on hypermedia shows great potential of the technology to facilitate knowledge presentation, representation, and construction [Burton, Moore, & Holmes 1995], [Nelson & Palumbo 1992]. Studies have shown that the interactive nature of hypermedia with its nonlinear navigation can support student learning in authentic learning environments (Cognition and Technology Group at Vanderbilt 1992], [Lajoie 1993]. Yet, learning in such an environment is challenging for students. Cognitive

tools are therefore often built into the hypermedia environment to provide needed support and structure to students.

The hypermedia based PBL software under study included a number of cognitive tools for students to use. Table 2 [Tab. 2] details the use of tools by students. There were two kinds of tools in the software, those that can support cognitive processing (i.e. database, notepad) and those used for conducting science experiments (i.e. microscope, periodic table).

Of the three scientific tools available, the microscope was accessed the most by students. On average each group accessed the microscope seven times. This finding is not surprising since the problem-solving activity required students to view a

number of microorganisms under the microscope. Without accessing the microscope, the students would not be able to complete the activity. Students' comments in the interviews confirmed this. Students used the other two scientific tools less frequently.

Of most interest to this study was students' use and their perceived value of the cognitive tools built in the PBL environment. The data showed some interesting results. The expert scientist and the inbox tools were accessed more frequently by the students. The purpose of including the expert scientist is to provide the problem scenario to the students. Observation of the process indicated that students often returned to the expert scientist to clarify the problem at hand. That is, having this expert readily accessible is helpful for students.

In addition to the problem scenario provided by the expert scientist through video, it was presented to the students in the form of a fax accessible through the inbox. The results showed that the students accessed the video and text forms of the scenario equally (video [via the expert scientist] = 31 times, and text [via fax in inbox]=29). Some groups accessed more video information while other groups used text information more frequently. Students commented on the usefulness of the expert scientist and inbox tools when being interviewed. Some said it would be difficult to know what to do if they did not read the information provided by the fax

sent to the inbox. Others found the presentation by the expert scientist helpful for their understanding of the problem. This finding supports other research in showing that providing information in multiple media formats allows students to access the information more effectively and accommodates the preferences of students with different learning characteristics [Liu 1997/98], [Small & Ferreira 1994]. Pavio's dual coding theory states that the human brain processes information through multiple channels [Pavio 1991]. Information is received and processed in either the verbal or nonverbal channel. Information that is received through both the verbal and nonverbal channels allows for richer representations to be constructed by the individual. Providing the information both in video and textual forms can enable students to construct a richer representation of the problem scenario. It was especially helpful for some of the participating students in this study, who spoke English as a second language².

The computer database contained information which students needed in order to complete their investigation. Each group accessed it approximately two times during the activity. The observation data indicated that most groups did not access the database until the end of the investigation when it became clear they were missing information concerning the role microorganisms played in other organisms. In a sense, the problem required the students to use the information in the database. However, given that gathering, searching, and selecting information is an integral part of the problem-solving process, one would have expected students to use the database both at an earlier stage and more frequently. A hypermedia database should and can facilitate problem solving. One possible explanation for the limited use of the

database may be that most of the information there was in textual form. Because screens of scrolling text are not very appealing, students may not have been motivated to use it. For a generation that is familiar with video games and television programs, providing visual information is important and can encourage the use of such tools as a database.

When students felt they had found a solution to the problem, they accessed the fax form, completed it, and then clicked the "send" button to send their solution to the expert scientist. Accessing the fax form indicated that the students had worked out a solution. Most groups accessed the fax form at least one time. A few did not access this tool because they did not finish the activity. Several groups accessed the fax form a couple of times because they had a solution that was not viable. The groups with incorrect solutions were provided with feedback suggesting they return to the laboratory and do more research on the problem.

Two infrequently accessed cognitive tools were the lab manual and the notepad. The lab manual provided explanations of the use of the equipment found in the virtual laboratory. The lack of access to the lab manual is probably because of the simple and intuitive design of the interface. Though working in the hypermedia environment was new to the students, they quickly learned how to use the equipment in the lab without having to seek help in the lab manual.

² Studying how the learning environment affected the bilingual students was not the focus for this study. Rather, the finding of the study suggested that future research could be conducted to examine how hypermedia PBL could support students whose native tongue was not English.

The notepad was provided so that students could take notes during the investigation and record their reflections. Few students used it because, unlike some of the other tools, this tool was not an integral part of the problem scenario. The activity did not require the use of the notepad, and students did not find a need to use it in solving the problem. Some students said in the interview that "we clicked on the notepad but we didn't use it." Many said that they found the notepad was useless.

The findings on the access of the tools supports hypermedia and PBL research in showing that anchoring a problem scenario in an authentic setting can help students to acquire a better understanding of the problem, and provide an opportunity for them to develop skills in problem definition and problem solving (Cognition and Technology Group at Vanderbilt 1994], [Lajoie 1993], [Jacobson & Spiro 1995]. The virtual lab setting, the expert scientist, the inbox, the fax form, and the microscope helped to create the scenario and allowed students to engage in the scientific investigation like a scientist would. The intuitive design of the software allowed the hypermedia novices to use the program without getting lost in hyperspace.

The findings also showed that hypermedia can support students in working with domain specific tools. For example, the virtual laboratory included a microscope which allowed the student scientists to view microorganisms. In addition to such domain specific tools, some tools are task specific, such as the database, notepad, and fax form. Though such tools were used to some extent, they were not fully utilized. Literature on PBL points out the importance of reflection and often suggests the inclusion of a notebook or notepad as a tool for students to process information and reflect their thinking. Literature on hypermedia states that hypermedia can be an ideal tool to provide information resources in the form of a database to support the problem-solving process. Yet, this study found that students would not use a

cognitive tool if they did not see a need for it. They did not like to use the database when it was constructed in mostly textual form. In other words, just building in some cognitive tools in a hypermedia environment is not sufficient. Scaffolding and modeling on using such cognitive tools should be provided so that students can see the importance of using them. The challenge for hypermedia designers is to find ways to embed such tools in hypermedia environments to facilitate students' active learning.

4.3 Problem-Based Learning and Attitude

The results of the two-factor mixed ANOVA on attitude indicated that there was not a significant two-way interaction between the grouping (computer, paper, and control) and the data collection points (pre vs. post) for the achievement scores: F (2, 95)=.26, p= .77. Although there was a small increase in the attitude from pre to post for all groups, there were no differences among the three groups [see Tab. 1]. In other words, the treatment did not have an impact on the students' attitudes toward learning science as measured by the attitude questionnaire.

The interview data, on the other hand, showed that the participating students enjoyed using the hypermedia based PBL software and preferred learning in the computer-supported PBL environment. Some of the students' comments include:

I like it better than anything else.

[I am] willing to solve problems that will take two weeks.

[I] liked the activity better than regular science class.

[I] prefer this experiment..... like it because it is like doing a mystery.

Fun... more fun than an [regular] experiment.

While the qualitative data showed that students liked the computer PBL environment and found it to be a fun way to learn science, the analysis on the attitude questionnaire failed to produce any significant statistical differences. This statistically insignificant difference may be explained by the fact that the treatment only lasted a total of ninety minutes as constrained by the school curriculum and scheduling. PBL literature indicates that problem-solving is a complicated process. It requires application of various critical thinking skills. Students who are not used to problem based learning, as they were not in this study, need practice and time to develop their skills [Gallagher, Stepien, & Rosenthal 1992]. Instead of engaging students in one activity, multiple activities/problems should be used to allow students to acquire the necessary skills in solving problems and transfer those skills into new settings. The brevity of the treatment may account for the limited impact of PBL on students' attitude in this case. Future research should be conducted to see if a lengthened treatment with more extensive practice could help to enhance students' attitudes.

4.4 Problem-Based Learning and Math and Reading Abilities

We were also interested in finding out if there was a relationship between students' math and reading abilities and their achievement when working in PBL environments.

Students' scores in reading and math from the most recent TAAS test were used. The results of the multiple regression analysis showed that the significance of the relationship was moderately high among reading ability, math ability, and the achievement scores for students using PBL: r =.59, p < .01. This significant relationship was mainly attributed to students reading ability t(72) = 3.46, p <.01 [see Tab. 3]. That is, students' reading ability is a better predictor for students' achievement in a PBL environment than their math ability.

A PBL environment relies on problem identification, presentation, problem-solving, and student reflection. Though mathematical ability is obviously very

important in problem solving, being able to read and comprehend the problem is critical. This finding provides some evidence on this issue and suggests that in order for students to be successful in a PBL environment, teachers need to make greater efforts to increase students' reading levels.

One of the potentials of hypermedia technology is to present information in multiple forms and offer perspectives not easily conveyed through print [Ayersman 1996]. Is it possible to provide critical information in a multimedia format to accommodate the needs of both weaker and stronger readers? To what extent should the multimedia formats be used? More research is needed to determine if hypermedia environments can be designed to support the achievement of low level readers during problem-based learning.

5 Conclusion

The findings of this study suggest that problem-based learning can influence middle school students in their learning of science. When students were exposed to the PBL environment, they increased their achievement scores more than those students who learned the same content in the traditional classroom. Consistent to the literature on PBL, the results of the study provide some evidence in supporting the use of PBL. Students' reading ability was found to be a better predictor for their achievement in PBL than their math ability. However, the brief treatment of the study had only limited impact on students' attitude toward learning science.

The findings also suggest that the use of hypermedia did not increase students' achievement more than the traditional paper medium when the problem-solving process was held constant. Though students used some of the tools provided by the hypermedia PBL software, they obviously lacked an understanding of why and how to use the cognitive tools to facilitate their problem-solving process. No such support and modeling were built into the software. The findings suggest that simply presenting information using multimedia and including tools in the hypermedia software may motivate students, but are not sufficient. Hypermedia designers must consider how to integrate the cognitive tools into problem-based learning and provide necessary scaffolding for the students. That is, tools should be an integral part of the learning environment, not an addition. Using cognitive tools as part of the problem-solving process should facilitate the development of higher order thinking skills and enhance learning. The challenge for hypermedia designers is to find ways to accomplish this goal.

References

[Aspy, Aspy, & Quinby 1993] Aspy, D. N., Aspy, C. B., & Quinby, P. M. (1993). What doctors can teach teachers about problem-based learning. Educational Leadership, April, 22-24.

[Ayersman 1996] Ayersman, D. J. (1996). Reviewing the research on hypermedia-based learning. Journal of Research on Computing in Education, 28(4), 500-525.

[Barrows 1986] Barrows, H. S. (1986). A taxonomy of problem-based learning methods. Medical Education, 20, 481-486.

[Barrows 1996] Barrows, H. (1996). Problem-Based Learning in Medicine and Beyond: A Brief Overview. New Directions for Teaching and Learning, 68, 3-12.

[Barrows & Tamblyn 1980] Barrows, H. S., & Tamblyn, R. M. (1980). Problem-based learning: An approach to medical education. New York: Springer Publishing Company, Inc.

[Burton, Moore, & Holmes 1995] Burton, J. K., Moore, D. M., & Holmes, G. A. (1995). Hypermedia concepts and research: An overview. Computers in Human Behavior., 11(3-4), 345-69.

[Cognition and Technology Group at Vanderbilt 1992] Cognition and Technology Group at Vanderbilt. (1992). The Jasper Series as an Example of Anchored Instruction: Theory, Program Description, and Assessment Data. Educational Psychologist, 27(3), 291-315.

[Cognition and Technology Group at Vanderbilt 1994] Cognition and Technology Group at Vanderbilt (1994). Anchored instruction and situated cognition revisited. Educational Technology, 33(3), 52-70.

[Edelson, Pea, & Gomez 1996] Edelson, D. C., Pea, R. D., & Gomez, L. (1996). Constructivism in the collaboratory. In B. G. Wilson (Ed.), Constructivist learning environments: Case studies in instructional design, (pp. 151-164). Englewood Cliffs, NJ: Educational Technology Publications.

[Gallagher & Stepien 1996] Gallagher, S. A. & Stepien, W. J. (1996). Content acquisition in problem-based learning: Depth versus breadth in American studies. Journal for the Education of the Gifted, 19(3), 257-275.

[Gallagher, Stepien, & Rosenthal 1992] Gallagher, S. A., Stepien, W. J., & Rosenthal, H. (1992). The effects of problem-based learning on problem solving. Gifted Child Quarterly, 36(4), 195-200.

[Germann 1988] Germann, P. J. (1988). Development of the attitude toward science in school assessment and its use to investigate the relationship between science achievement and attitude toward science in school. Journal of Research in Science Teaching, 25(8), 689-703.

[Hannafin & Sullivan 1995] Hannafin, R. D., & Sullivan, H. (1995). Learner control in full and lean CAI programs. Educational Technology Research and Development, 43(1), 19-30.

[Honebein 1996] Honebein, P. C. (1996). Seven goals for the design of constructivist learning environments. In B. G. Wilson (Ed.), Constructivist learning environments: Case studies in instructional design, (pp. 11-24). Englewood Cliffs, NJ: Educational Technology Publications.

[Jacobson & Spiro 1995] Jacobson, M. & Spiro, R. J. (1995). Hypertext learning environments, cognitive flexibility, and the transfer of complex knowledge: An empirical investigation. Journal of Educational Computing Research, 12(4), 301-333.

[Lajoie 1993] Lajoie, S. P. (1993). Computer environments as cognitive tools for enhancing learning. In S. P. Lajoie and S. J. Derry (Eds.), Computers as Cognitive Tools, (pp. 261-288). Hillsdale, New Jersey: Lawrence Erlbaum Associates.

[Liu, M. 1997/98] Liu, M. (1997/98). Interactive multimedia in an informal setting: The use of a multimedia information kiosk by university students. Journal of Educational Technology Systems, 26(1), 35-53.

[Miles & Huberman 1994] Miles. M. B. & Huberman, A. M. (1994). Qualitative Data Analysis (2nd.), Thousand Oaks: CA: Sage Pub.

[Nelson & Palumbo 1992] Nelson & Palumbo (1992). Learning, instruction, and hypermedia, Journal of Educational Multimedia and Hypermedia, 1(3), 287-299.

[Paivio 1991] Paivio, A. (1991). Dual coding theory: Retrospect and current status. Canadian Journal of Psychology, 45, 255-287.

[Perkins 1991] Perkins, D. N. (1991). Technology meets constructivism: Do they make a marriage? Educational Technology, 31(5), 18-23.

[Small & Ferreira 1994] Small, R. V., & Ferreira, S. M. (1994). Information location and use, motivation, and learning patterns when using print or multimedia information resources, Journal of Educational Multimedia and Hypermedia, 3(3/4), 251-274.

[Spiro, Feltovich, Jacobson, & Coulson 1992] Spiro, R. J., Feltovich, P. L., Jacobson, M. J., & Coulson, R. L. (1992). Cognitive flexibility, constructivism, and hypertext: Random access instruction for advanced knowledge acquisition in ill-structured domains. In T. M. Duffy & D. H. Jonassen (Eds.), Constructivism and the technology of instruction: A conversation, (pp. 58-75). Hillsdale: Lawrence Erlbaum Associates.

[Spiro & Jehng 1990] Spiro, R. J., & Jehng, J. (1990). Cognitive flexibility theory: Theory and technology for the nonlinear and multidimensional traversal of complex subject matter. In D.

Nix & R. Spiro (Eds.), Cognition, Education, and Multimedia, (pp. 163-205). Hillsdale, NJ: Lawrence Erlbaum Associates.

[Walton & Matthews 1989] Walton, H. J., & Matthews, M. B. (1989). Essentials of problem-based learning. Medical Education, 23, 542-558.

[Williams 1992] Williams, S. M. (1992). Putting case-based instruction into context: Examples from legal and medical education. Journal of the Learning Sciences, 2(4), 367-427.