An Evaluation of the Use of Problem-Based Learning Software
By Middle School Students
Douglas C. Williams
(University of Texas at Austin, United States
(University of Texas at Austin, United States
(University of Texas at Austin, United States
Abstract: Research has shown the potential of a problem-based
approach to enhance students' learning. The interactive nature of hypermedia
technology and its ability to deliver information in different media formats
can provide unique capabilities for implementing problem-based learning
(PBL) environments. Yet, we know little about the types of tools that are
effective in supporting students' learning in a hypermedia supported PBL
environment. The purpose of this study was to investigate both the use
of tools and design features in a piece of PBL software and their effectiveness
on middle school students' learning. The findings of this study show that
students who were exposed to the PBL environment increased their achievement
scores from pre to posttest more than those students who learned the same
content in the traditional classroom. Students' reading ability was found
to be a better predictor for their achievement in PBL than their math ability.
However, the brief treatment of the study had only limited impact on students'
attitude toward learning science. The findings of the study and their implications
are discussed in detail.
1 Research Framework
1.1 Problem-based Learning
There has been a growing body of research on authentic and situated
learning environments utilizing the problem-based approach to learning
[Cognition and Technology Group at Vanderbilt
1994], [Spiro, Feltovich, Jacobson, & Coulson
1992]. Problem-based learning (PBL) emphasizes solving authentic
problems in authentic contexts. It is an approach where students are
given a problem, replete with all the complexities typically found in
real world situations, and work collaboratively to develop a
solution. Problem-based learning provides students an opportunity to
develop skills in problem definition and problem solving, to reflect
on their own earning, and develop a deep understanding of the content
domain learning [Cognition and Technology Group at
Vanderbilt 1994], [Lajoie 1993], [Jacobson & Spiro 1995]. This approach was
developed in the sixties for medical education, and
has since been
used in various subject areas, such as business, law, education,
architecture, and engineering. Howard Barrows, a pioneer in the use of
medical education, argues that though there are many variations of
PBL, they share a number of common characteristics [Barrows 1996]: (1) Learning is student-centered;
(2) learning occurs in small student groups; (3) teachers are
facilitators or guides; and (4) problems form the organizing focus and
stimulus for learning.
Literature has shown that problem-based learning can facilitate the
improvement of student attitude toward the content area learning [Cognition
and Technology Group at Vanderbilt 1992]. Explanations offered for
this are that students perceive the relevance of the work [Barrows 1986]
or compare the task of finding information and developing a solution to
solving a mystery [Cognition and Technology Group at
Vanderbilt 1992]. Medical students using a PBL curriculum have been
shown to be able to pursue learning independently better than their peers
receiving a traditional curriculum [Aspy, Aspy, &
Quinby 1993], a finding which supports the claim that PBL prepares
students to become independent, lifelong learners. High school students
in a semester long class which used PBL exclusively showed a significant
increase in their spontaneous use of one problem-solving step: problem
finding, the ability to identify and formulate problem statements [Gallagher,
Stepien, & Rosenthal 1992]. Similarly, students in a PBL curriculum
have shown a greater ability than students in traditional classes to break
complex problems into their component parts and identify subproblems which
must be solved in order to solve the main problem [Cognition
and Technology Group at Vanderbilt 1992], [Gallagher
& Stepien 1996].
In recent years, there has been a growing interest among educators to
use PBL in the K-12 setting. However, the experiences of college level
programs and early efforts in K-12 schools have shown PBL to be a particularly
demanding instructional approach for both students and teachers [Barrows
& Tamblyn 1980], [Cognition and Technology Group
at Vanderbilt 1992]. Students must perform a wide variety of tasks
with which they may have had limited prior experience. These include problem
finding, hypothesis generation, identification of learning needs, location
of resources to meet these learning needs, data collection and organization,
and development of a solution plan supported by evidence and reasoning.
Teachers, in their role as facilitators, are responsible to provide support
for this wide variety of activities to students who may vary greatly in
their needs. Because the variability of the classroom teacher's ability
to function in this role has a profound effect on the success of PBL [Walton
& Matthews 1989], instructional materials which provide some of
this support seem warranted.
1.2 Hypermedia and Problem-based Learning
The interactive nature of computers and the ability to deliver information
in different media formats provides unique capabilities for the implementation
of PBL environments. In particular, the capabilities of hypermedia have
much to offer designers of advanced learning environments. Hypermedia is
a rich environment containing information in many different forms, including
text, graphics, audio, video, and animation. The student is not forced
to access the resources in any predetermined order, but can navigate within
the environment in a nonlinear fashion [Burton, Moore,
& Holmes 1995]. The support of multiple media types and the possibility
nonlinear navigation are particularly useful in the creation of
computer-based PBL environments.
The nonlinear nature of hypermedia is consistent with the characteristics
of PBL. As students are engaged in PBL, they need to collect data and access
resources. This suggests a high degree of control on the part of the learner.
Hypermedia can support this by allowing students to access needed resources
at the time it is most appropriate. Hypermedia allows students to have
more control over their learning. They become actively engaged in decision-making
while traversing the environment. Research on learner control versus program
control in hypermedia environments suggests that subjects under learner
control score higher than those under program control on achievement posttest
and have a more positive attitude toward learning [Hannafin
& Sullivan 1995].
Much of the literature on PBL argues that learning cannot be separated
from the context and the activities in which students engage. Yet it is
unrealistic to think that schools can teach all subjects in real-world
contexts. Because of safety concerns, cost issues, and time required, authentic
environments are often impractical and difficult to create. A viable solution
is to use the media capabilities of hypermedia to create environments that
are consistent with real-world contexts. Hypermedia allows for the creation
of authentic situations in which students can be immersed without the dangers
or costs associated with real-world contexts.
Though PBL can be implemented with traditional media, hypermedia provides
unique capabilities for its implementation. The nonlinear nature of hypermedia
allows students to explore the PBL environment accessing resources as the
need arises. Hypermedia can also facilitate the development of authenticity
in the learning environment. Williams [Williams 1992]
suggests that law and medical school curricula could be improved by the
use of hypermedia environments to engage students in authentic activities
within an authentic context.
1.3 Hypermedia and Cognitive Tools
When working in everyday situations, individuals use tools and resources
such as computers, calculators, concepts, and formulas in order to solve
problems. Therefore, tools must be considered when creating authentic environments
for student learning. A tool does not necessarily have to be a tangible
object. For example, an engineer may use a mathematical formula to calculate
the area of a cylinder. Different disciplines and professions may use ideas
and tools in very different ways.
Many researchers argue that cognitive tools can support and enhance
student learning [Lajoie 1993], [Perkins 1991]. These tools come in many forms
and can support students in a variety of tasks they must perform as
they engage in problem solving. Hypermedia has the potential to make
these tools readily available to students.
As discussed earlier, PBL occurs within a context where knowledge is
naturally situated. Tools can be employed to create an authentic context
in which students can work. The Adventures of Jasper Woodsbury series
utilizes video-based scenarios in order to create a context for learning [Cognition and Technology Group at Vanderbilt
1992]. The video segment provides a focus for the learning
activity and may also help students who are poor readers. Hypermedia
can also present a scenario, but has the advantage of allowing
students to explore the environment in which the scenario
is set. This
exploration can mirror processes which people use to address problems
in a real life settings.
Hypermedia can augment memory and support students in reflecting on
their problem-solving process. Sherlock I, software that creates
an environment in which Air Force technicians practice avionics troubleshooting,
includes tools to support cognitive processing [Lajoie
1993]. Avionics troubleshooting is a complex task requiring the technician
to entertain many variables and remember a series of completed tests. Sherlock
I allows the student to view the steps he or she has taken in troubleshooting
a problem. The ability to view the solution path supports students in reflection
on the problem solving process without the need for them to rely on their
recollection of every step. The software-generated problem solving steps
also make explicit student thinking, an essential component in stimulating
metacognitive awareness. Likewise, reflective questions can also be effective
in promoting metacognitive thinking.
Hypermedia environments can offer a comprehensive set of resources
to enable students to meet their learning needs. In order for students
to engage in problem solving, they must have access to
information. Information can be provided to students in a number of
tools. In the Jasper series, the Cognition and Technology Group at
Vanderbilt chose to make information available to students by
embedding it in the video-based scenario [Cognition
and Technology Group at Vanderbilt 1992]. Students revisit the
scenario in order to sift through the information, finding items
relevant to the problem at hand. Information can also be integrated in
the learning environment as a searchable database. Bio-world
followed this approach by providing an electronic library for students
to access as they seek to diagnose patients in a simulated hospital
[Lajoie 1993]. The information in the database
can be in many forms. The Lab Design Project (LDP), which allows
students to actively take part in sociological research concerning a
fictitious biotechnology building, permits students to gather
information from such diverse formats as interviews with employees,
building plans, letters, and sketches [Honebein
Hypermedia can provide electronic notebooks, which, in addition to providing
space for student note taking, can include advanced features to help support
the student in constructing meaning. The notebook in Bio-world contains
a section which displays the students' previous actions such as database
searches and diagnoses for patients [Lajoie 1993].
These features help support the student's memory and metacognitive thinking.
Other projects have augmented the traditional notebook with the ability
to support multimedia, collaboration among students, and the ability to
create links between concepts [Edelson, Pea, Gomez 1996].
Though literature supports the efficacy of problem-based learning, little
research exists which investigates the types of tools or features that
are effective in supporting students working in PBL environments. PBL software
is beginning to find its way into schools. However, the relative effectiveness
of the various tools incorporated in these programs has yet to be studied. In order to design an
effective computer-supported PBL learning environment, it is important
to understand the tools and design features included in the software,
and their impact on learning. It is, therefore, the purpose of this
study to examine and understand how middle school students use and
interact with a piece of recently available computer-supported PBL
software developed by a major publishing company.
2 Research Questions
This study investigates the use of tools and design features as employed
in a piece of problem-based learning software and their effectiveness on
middle school students' learning of science concepts. Specifically, these
four research questions formed the focus of the study:
- What is the effect of the computer-supported problem-based learning
environment on the achievement of middle school science students' learning
- In what way do the students use the tools and design features built
into the computer supported PBL environment while engaging in problem solving?
- What is the effect of the computer supported problem-based learning
environment on middle school students' attitudes toward learning science?
- Is there a relationship between students' math or reading ability and
their achievement in the problem-based learning environment?
3 Design Of Study
The participants of the study (N = 115) were students enrolled in seventh
grade science classes at a middle school located in a medium-sized city
in the southwestern United States. The school has a high percentage of
minority students. The participants in the study consisted of 66% Hispanic
Americans (N=76), 12% African Americans (N=14) and 22% white (N=25). The
age of the students ranged from 12 - 14 years. Of the participants, 50
were male and 65 female.
3.2.1 Treatment Conditions
This study consisted of three treatment conditions:
computer-supported PBL, paper-based PBL, and a control. One class was
chosen at random to be the control group. The remaining classes were
assigned randomly to the treatment conditions resulting in three
classes in the computer-supported PBL condition and two classes in the
paper-based PBL condition. Because the treatments were included as
part of the regular science class, complete random assignment was not
possible in this case. There were 59 students in the
computer-supported PBL condition, 38 in the paper-based PBL condition, and 18 in the control. To examine the effectiveness of
the computer-supported PBL, it was necessary to compare students in
the computer-supported PBL condition with students that did not use
PBL (the control group), and with students in the paper-based PBL
condition (problem solving using paper and pencil). Students in the
computer and paper PBL groups formed small teams of four to five
students in order to perform the problem-solving activity whereas the
control group did not.
Each class in the computer-supported PBL
condition consisted of four to six teams for a total of 15 teams,
while there were 10 teams in the paper-based PBL condition.
The students in the computer-supported PBL condition used
problem-based learning software recently developed by a major textbook
publishing company. The CD-ROM program contains eight activities on
different topics developed to support the middle school science
curriculum. Upon starting one of the PBL activities, students find
themselves in a virtual science laboratory. Immediately on entering
the laboratory a short video segment plays in which a scientist
provides important details about a scientific problem she is working
on and solicits the students' help in developing a solution. The
activity used for this study was concerned with the classification of
a microorganism. It was selected because it was related to the topic
being studied in the participating classes at the time of the
The students are supported in finding a solution to the problem by
the availability of various tools in the virtual laboratory. An
expert scientist built into the virtual laboratory provides
information about the scenario and takes on the role of a mentor by
providing hints and feedback during the activity. The expert scientist
provides information essential in completing the activity. At the end
of the video scenario the expert scientist tells the students that a
fax will soon arrive reiterating the particulars of their task. As the
video scenario concludes, a fax arrives in the inbox on the
back table of the laboratory. The inbox provides a printed version of
the problem scenario, thereby supporting students in gathering
relevant information and defining the problem to be solved. The
software also includes a lab manual which provides information
on the use and importance of the tools found in the lab. The lab
manual was intended to help orient students to the purpose and use of
the lab equipment. The lab also furnishes a computer database
containing information which is fundamental in solving the
problem. The computer database consists of a series of hypertext
documents regarding microorganisms. In addition to text, many of the
documents in the database contain audio and video clips. A
notepad offers students the option of taking notes during the
activity. During the investigation, students have access to a
microscope to view slides of microorganisms that the expert
scientist has prepared on the lab table. On completion of the
activity, students transmit an electronic fax with the results
of their investigation. The fax ensures that students have completed
the activity and serves as a tool by which the program can assess
students' work and provide feedback.
The paper-based PBL group was engaged in a problem-solving activity
equivalent in content and available resources to the computer-based
software, except that the information given (including the information
in the database and the pictures) was print-based. Though identical in
the resources provided, the computer and the paper-based PBL are
different in that (1) the computer-based version allows interactive access of information while the paper-based does not, and (2) the
computer-based version provides information in multimedia format while
the paper-based does not.
The control group learned the same content on microorganisms using the
lecture-based traditional approach.
3.2.2 Treatment Phases
For the computer and paper PBL groups, the entire treatment
consisted of two phases: the teacher modeling phase and the
problem-solving phase. The teacher modeling phase was included
because literature on PBL has shown that providing necessary
scaffolding is a critical step in making PBL successful. Teacher
modeling was done for both the computer-supported and paper-based
conditions because we felt that the modeling should be part of problem
solving regardless of the media involved. In addition, our primary
interest in this study was to find out if the hypermedia environment
could provide additional support to students. Therefore, the
instruction of the two conditions were held to be the same. This
modeling phase took about thirty minutes for each group. The control
group worked through the regular lecture-based instruction on
microorganisms with no intervention.
The modeling phase accomplished two purposes. First, it introduced students
to the steps of problem solving and provided them with guided practice.
For the computer-supported PBL group, students were given an opportunity
to become familiar with the design features of the software and the tools
available for their use. Second, it provided an opportunity for students
to become familiar with the assigned roles they would perform during the
group work. While the students had prior experience with cooperative learning,
they were unfamiliar with some of the new roles assigned since the PBL
environment was a new experience for them. The content used for this teacher
modeling phase was different from that in the problem-solving phase.
In the modeling phase, students in the computer-support PBL treatment
watched the opening video-based scenario on a large television screen.
The scenario created the context for the activity by describing the problem
to be solved. The teacher and one of the researchers modeled the group
process for students. They discussed how to solve the problem, what steps
to take, what computer-based tools to use, and what information to record.
For example, the teacher modeled how to record responses to worksheet questions
and how to log the tools they accessed on the tool sheet. Students then
practiced recording such information on the provided print-based forms.
Students in the paper-based PBL group received similar modeling, with
a few minor differences. The problem was presented through a written version
of the video used in the computer-based treatment. One of the researchers
read the problem aloud to the class. Similar discussions were held about
how to proceed in solving the problem. The teacher and the researcher modeled
the various roles students were expected to play. For example, the teacher
modeled how to record their answers to the worksheet questions.
During the problem-solving phase, students in both groups were asked
to work on the problem-solving activity based on the classification of
microorganisms. The computer-supported group worked on the activity using
the PBL software, while the paper group worked on the same activity using
paper and pencil. It took both groups approximately 45 minutes to complete
3.3 Dependent Measures
Students' knowledge about microorganisms was assessed through an achievement
test on the content. This measure was created by the teacher and was used
in previous years on the same unit. It consisted of eight short answer
questions on viruses, the characteristics of microorganisms, and the role
microorganisms play in the life of other organisms. All students were given
the measure before and after the treatment.
3.3.2 Tool Use
Students in the computer-supported PBL group were asked to record the
various tools they used in a given chart1
A fourteen-item questionnaire was used in order to examine students'
attitude toward learning science. The Attitude Toward Science in School
Assessment uses a 5-point Likert scale with 1 being "strongly
agree" and 5 being "strongly disagree" [Germann
1988]. It addresses students' feelings about science as a subject and
has a reported reliability of .95. This questionnaire was given to all
students before and after the treatment.
3.3.4 Interviews and Observations
Interviews were conducted with students from 10 of the 15 teams in the
computer-supported PBL condition. Of the interview questions, 70% were
specific, asking students about their likes and dislikes of the program
(e.g. "Did you like working with this software?"; "Which
tool did you find the most useful while working on the problem?";
"Which tool did you find the least useful while working on the problem?"
). The remaining 30% of the questions were open-ended, allowing students
to provide information they wanted. (e.g. "Do you have any suggestions
about making this software better?"; "Is there anything you can
tell us about your experience using this software?"). Observations were made by the researchers
throughout the process. The researchers monitored the group
problem-solving activity and answered questions from students. It was
hoped that this triangulation of the quantitative and qualitative data
would provide richer and more detailed information about the research
1Because the PBL software was already made, and there was
no online data collection mechanism built in we created a matrix for students
to record their use of tools.
The entire experiment took place over a three-day period. On day one,
the classroom teacher administered the pretest achievement measure and
the pretest attitude questionnaire. On day two, the treatment took place,
which consisted of the modeling phase followed by the problem-solving phase.
After having completed the treatment, the students were asked to complete
the posttest achievement measure and posttest attitude questionnaire. The
researchers returned on day three to conduct interviews with selected students.
To answer the first research question, "What is the effect of
the computer-supported problem-based learning environment on the
achievement of middle school science students?," a two-factor
mixed ANOVA was conducted with the grouping (computer, paper, and
control) as a between-subjects independent variable, and the data
collection points (pre vs. post) as the repeated measure independent
variable. The dependent variable was the pre and post achievement
scores in the science content test.
To answer the second research question, "In what way do the
students use the tools and design features built into the
computer-supported PBL environment while engaging in problem
solving?" students' use of the tools were tabulated and analyzed
descriptively. Some interview questions were specifically targeted
toward finding out if the tools were beneficial for problem-solving
and in what ways.
To answer the third research question, "What is the effect of the
computer supported problem-based learning environment on middle school
students' attitudes toward science?," a two-factor mixed ANOVA was
run with the grouping (computer, paper, and control) as a between-subjects
independent variable, and the data collection points (pre vs. post) as
the repeated measure independent variable. The dependent variable was the
pre and post scores of the attitude questionnaire.
To answer the fourth research question, "Is there a
relationship between students' math or reading ability and their
achievement in the problem-based learning environment?," a
multiple regression was performed with students' math and reading
ability, measured by their most recent scores on the Texas Assessment
of Academic Skills (TAAS), as the independent variables and their
achievement test as the dependent variable. TAAS is a state-wide
testing system that assesses the overall academic achievement of all
students in Texas at different grade levels. It provides information
on students' reading ability, math ability, and writing ability.
Students were selected for the post interviews after they completed
the treatment. The purpose of the interviews was to find out (1) what the
students liked and disliked about the environment; (2) what tools and design
features they found most useful in the PBL environment; (3) their perception
on using the hypermedia PBL software. The interview data were analyzed
according to Miles and Huberman's framework of qualitative data analysis
[Miles & Huberman 1994]. The data were first transcribed.
Two researchers coded the data using the four research questions as a guide.
The data were then categorized and grouped according to their common themes.
In the data analysis process, the two researchers worked independently
and then together to
ensure the interrater agreement was at least .95.
Such qualitative data were used to provide more information and to substantiate
the quantitative analyses.
4 Results And Discussion
4.1 Problem-Based Learning and Achievement
The results of the two-factor mixed ANOVA on achievement indicated that
there was a significant two-way interaction between the grouping (computer,
paper, and control) and the data collection points (pre vs. post) for the
achievement scores: F (2, 96) =5.50, p <.01. All groups
increased their achievement scores from pre to post. The gains from pre
to post were significantly greater for the computer and paper groups than
for the control group [see Tab. 1] and [Fig. 1]. The gain differences between
the computer and the control groups (posttest -pretest =7.78), and the
paper and the control groups (posttest - pretest=11.62) were significant
at p < .05 level based on Fisher's PLSD post hoc tests. The gain
difference between the paper and the control groups was also significant
at p < .05 level according to the Scheffe post hoc test. This
finding shows that both the computer and paper groups significantly improved
their achievement scores after they participated in the study, while the
increase for the control group was not significant. In other words, there
was an effect of the problem-based learning environment on the achievement
of middle school science students. Yet the difference between the computer-supported
PBL and paper PBL was not significant.
* N indicates the number of students who turned in both the pre and
post evaluation forms for each group.
** The pretest-posttest differences for the computer and paper groups
were significantly different from that for the control group, p
Table 1: Mean and Standard Deviation (in Parenthesis) on
achievement and attitude for the Computer, the Paper and the Control Groups
Figure 1: Achievement scores for the groups from pre to post
The results of this study are consistent with the PBL research in
showing that PBL has a positive impact on students' acquisition of
domain specific knowledge [Cognition and Technology
Group at Vanderbilt 1992], [Gallagher &
Stepien 1996]. When the students were exposed to the PBL
environment, they increased their achievement scores more than those
students who learned the same content in the traditional
classroom. The findings also suggest that both the computer-supported
and paper based PBL are equally effective in enhancing students'
achievement. The lack of a significant difference in achievement
between the computer and paper groups may be explained by several
factors. First, in order to keep the two groups as equivalent as
possible, both groups received the same instruction, same resource
materials, and engaged in the same modeling and practice activity. The
two groups differed only in the use of the delivery medium; one used
paper and pencil and the other used
hypermedia. The use of hypermedia did not increase students'
achievement more than the traditional paper medium when the
problem-solving process was held constant. Second, the treatment was
relatively brief. A longer treatment in which students in the computer
group would be more familiar with the tools and understand the value
of the tools may make a difference. Last and more importantly, it is
necessary to examine the design of the hypermedia based PBL software,
specifically, how the tools were designed and used by the students. It
is our belief that the design of the PBL software used for this study
did not encourage students to make best use of the tools.
4.2 The Use of Tools in the Problem-Based Learning Environment
Literature on hypermedia shows great potential of the technology to
facilitate knowledge presentation, representation, and construction [Burton,
Moore, & Holmes 1995], [Nelson & Palumbo 1992].
Studies have shown that the interactive nature of hypermedia with its nonlinear
navigation can support student learning in authentic learning environments
(Cognition and Technology Group at Vanderbilt 1992],
[Lajoie 1993]. Yet, learning in such an environment
is challenging for students. Cognitive
tools are therefore often built
into the hypermedia environment to provide needed support and structure
The hypermedia based PBL software under study included a number of cognitive
tools for students to use. Table 2 [Tab. 2] details the use of tools by
students. There were two kinds of tools in the software, those that can
support cognitive processing (i.e. database, notepad) and those used for
conducting science experiments (i.e. microscope, periodic table).
|Name of the Tools
|Cognitive Tools (tools that can support information processing,
organizing, and presentation)
|Expert scientist (Presenting the problem scenario; provides support
|Inbox (Containing textual version of problem scenario in the form of
|Database (Containing information regarding microorganisms)
|Fax Form (Students fill at the conclusion of the activity; providing
|Lab Manual (Explaining the importance and use of tools in the virtual
|Notepad (Area for taking notes during the activity)
|Scientific Tools (tools that are needed for conducting science
|Microscope (Viewing slides of microorganisms)
|Easter Eggs (Small content related animation for entertainment purposes;
mouse click starts the animation)
|Periodic Table (Table with the symbols of each element)
Table 2: Frequency of the Use of Tools in the Hypermedia
Of the three scientific tools available, the microscope was
accessed the most by students. On average each group accessed the
microscope seven times. This finding is not surprising since the
problem-solving activity required students to view a
number of microorganisms under the microscope. Without accessing
the microscope, the students would not be able to complete the
activity. Students' comments in the interviews confirmed
this. Students used the other two scientific tools less
Of most interest to this study was students' use and their perceived
value of the cognitive tools built in the PBL environment. The data showed
some interesting results. The expert scientist and the inbox tools were
accessed more frequently by the students. The purpose of including the
expert scientist is to provide the problem scenario to the students. Observation
of the process indicated that students often returned to the expert scientist
to clarify the problem at hand. That is, having this expert readily accessible
is helpful for students.
In addition to the problem scenario provided by the expert
scientist through video, it was presented to the students in the form
of a fax accessible through the inbox. The results showed that the
students accessed the video and text forms of the scenario equally
(video [via the expert scientist] = 31 times, and text [via fax in
inbox]=29). Some groups accessed more video information while other
groups used text information more frequently. Students commented on
the usefulness of the expert scientist and inbox tools when
being interviewed. Some said it would be difficult to know what to do
if they did not read the information provided by the fax
sent to the
inbox. Others found the presentation by the expert scientist helpful
for their understanding of the problem. This finding supports other
research in showing that providing information in multiple media
formats allows students to access the information more effectively and
accommodates the preferences of students with different learning
characteristics [Liu 1997/98], [Small & Ferreira 1994]. Pavio's dual coding
theory states that the human brain processes information through
multiple channels [Pavio 1991]. Information is
received and processed in either the verbal or nonverbal
channel. Information that is received through both the verbal and
nonverbal channels allows for richer representations to be constructed
by the individual. Providing the information both in video and textual
forms can enable students to construct a richer representation of the
problem scenario. It was especially helpful for some of the
participating students in this study, who spoke English as a second
The computer database contained information which students needed
in order to complete their investigation. Each group accessed it
approximately two times during the activity. The observation data
indicated that most groups did not access the database until the end
of the investigation when it became clear they were missing
information concerning the role microorganisms played in other
organisms. In a sense, the problem required the students to use the
information in the database. However, given that gathering, searching,
and selecting information is an integral part of the problem-solving
process, one would have expected students to use the database both at
an earlier stage and more frequently. A hypermedia database should and
can facilitate problem solving. One possible explanation for the
limited use of the
database may be that most of the information there was in textual
form. Because screens of scrolling text are not very appealing,
students may not have been motivated to use it. For a generation that
is familiar with video games and television programs, providing visual
information is important and can encourage the use of such tools as a
When students felt they had found a solution to the problem, they
accessed the fax form, completed it, and then clicked the
"send" button to send their solution to the expert
scientist. Accessing the fax form indicated that the students had
worked out a solution. Most groups accessed the fax form at least one
time. A few did not access this tool because they did not finish the
activity. Several groups accessed the fax form a couple of times
because they had a solution that was not viable. The groups with
incorrect solutions were provided with feedback suggesting they return
to the laboratory and do more research on the problem.
Two infrequently accessed cognitive tools were the lab manual and
the notepad. The lab manual provided explanations of the use of the
equipment found in the virtual laboratory. The lack of access to the
lab manual is probably because of the simple and intuitive design of
the interface. Though working in the hypermedia environment was new to
the students, they quickly learned how to use the equipment in the lab
without having to seek help in the lab manual.
2 Studying how the learning environment affected the bilingual
students was not the focus for this study. Rather, the finding of the study
suggested that future research could be conducted to examine how hypermedia
PBL could support students whose native tongue was not English.
The notepad was provided so that students could take notes during the
investigation and record their reflections. Few students used it because,
unlike some of the other tools, this tool was not an integral part of the
problem scenario. The activity did not require the use of the notepad,
and students did not find a need to use it in solving the problem. Some
students said in the interview that "we clicked on the notepad but
we didn't use it." Many said that they found the notepad was useless.
The findings on the access of the tools supports hypermedia and PBL
research in showing that anchoring a problem scenario in an authentic setting
can help students to acquire a better understanding of the problem, and
provide an opportunity for them to develop skills in problem definition
and problem solving (Cognition and Technology Group
at Vanderbilt 1994], [Lajoie 1993], [Jacobson
& Spiro 1995]. The virtual lab setting, the expert scientist,
the inbox, the fax form, and the microscope helped to create the scenario
and allowed students to engage in the scientific investigation like a scientist
would. The intuitive design of the software allowed the hypermedia novices
to use the program without getting lost in hyperspace.
The findings also showed that hypermedia can support students in working
with domain specific tools. For example, the virtual laboratory included
a microscope which allowed the student scientists to view microorganisms.
In addition to such domain specific tools, some tools are task specific,
such as the database, notepad, and fax form. Though such tools were used
to some extent, they were not fully utilized. Literature on PBL points
out the importance of reflection and often suggests the inclusion of a
notebook or notepad as a tool for students to process information and reflect
their thinking. Literature on hypermedia states that hypermedia can be
an ideal tool to provide information resources in the form of a database
to support the problem-solving process. Yet, this study found that students
would not use a
cognitive tool if they did not see a need for it. They did not like
to use the database when it was constructed in mostly textual form. In
other words, just building in some cognitive tools in a hypermedia
environment is not sufficient. Scaffolding and modeling on using such
cognitive tools should be provided so that students can see the
importance of using them. The challenge for hypermedia designers is to
find ways to embed such tools in hypermedia environments to facilitate
students' active learning.
4.3 Problem-Based Learning and Attitude
The results of the two-factor mixed ANOVA on attitude indicated
that there was not a significant two-way interaction between the
grouping (computer, paper, and control) and the data collection points
(pre vs. post) for the achievement scores: F (2, 95)=.26, p=
.77. Although there was a small increase in the attitude from pre to
post for all groups, there were no differences among the three groups
[see Tab. 1]. In other words, the treatment did not have an impact on
the students' attitudes toward learning science as measured by the
The interview data, on the other hand, showed that the participating
students enjoyed using the hypermedia based PBL software and preferred
learning in the computer-supported PBL environment. Some of the students'
I like it better than anything else.
[I am] willing to solve problems that will take two weeks.
[I] liked the activity better than regular science class.
[I] prefer this experiment..... like it because it is like doing a mystery.
Fun... more fun than an [regular] experiment.
While the qualitative data showed that students liked the computer PBL
environment and found it to be a fun way to learn science, the analysis
on the attitude questionnaire failed to produce any significant statistical
differences. This statistically insignificant difference may be explained
by the fact that the treatment only lasted a total of ninety minutes as
constrained by the school curriculum and scheduling. PBL literature indicates
that problem-solving is a complicated process. It requires application
of various critical thinking skills. Students who are not used to problem
based learning, as they were not in this study, need practice and time
to develop their skills [Gallagher, Stepien, & Rosenthal
1992]. Instead of engaging students in one activity, multiple activities/problems
should be used to allow students to acquire the necessary skills in solving
problems and transfer those skills into new settings. The brevity of the
treatment may account for the limited impact of PBL on students' attitude
in this case. Future research should be conducted to see if a lengthened
treatment with more extensive practice could help to enhance students'
4.4 Problem-Based Learning and Math and Reading Abilities
We were also interested in finding out if there was a relationship
between students' math and reading abilities and their achievement
when working in PBL environments.
Students' scores in reading and math from the most recent TAAS test
were used. The results of the multiple regression analysis showed that
the significance of the relationship was moderately high among
reading ability, math ability, and the achievement scores for students
using PBL: r =.59, p < .01. This significant relationship
was mainly attributed to students reading ability t(72) =
3.46, p <.01 [see Tab. 3]. That is, students' reading
ability is a better predictor for students' achievement in a PBL
environment than their math ability.
||3.46, p < .1
||14.9, p < .01
||1.26, p =.21
Table 3: Results of the Multiple Regression Analysis
A PBL environment relies on problem identification, presentation, problem-solving,
and student reflection. Though mathematical ability is obviously very
in problem solving, being able to read and comprehend the problem is critical.
This finding provides some evidence on this issue and suggests that in
order for students to be successful in a PBL environment, teachers need
to make greater efforts to increase students' reading levels.
One of the potentials of hypermedia technology is to present information
in multiple forms and offer perspectives not easily conveyed through print
[Ayersman 1996]. Is it possible to provide critical
information in a multimedia format to accommodate the needs of both weaker
and stronger readers? To what extent should the multimedia formats be used?
More research is needed to determine if hypermedia environments can be
designed to support the achievement of low level readers during problem-based
The findings of this study suggest that problem-based learning can influence
middle school students in their learning of science. When students were
exposed to the PBL environment, they increased their achievement scores
more than those students who learned the same content in the traditional
classroom. Consistent to the literature on PBL, the results of the study
provide some evidence in supporting the use of PBL. Students' reading ability
was found to be a better predictor for their achievement in PBL than their
math ability. However, the brief treatment of the study had only limited
impact on students' attitude toward learning science.
The findings also suggest that the use of hypermedia did not
increase students' achievement more than the traditional paper medium
when the problem-solving process was held constant. Though students
used some of the tools provided by the hypermedia PBL software, they
obviously lacked an understanding of why and how to use the cognitive
tools to facilitate their problem-solving process. No such support and
modeling were built into the software. The findings suggest that
simply presenting information using multimedia and including tools in
the hypermedia software may motivate students, but are not
sufficient. Hypermedia designers must consider how to integrate the
cognitive tools into problem-based learning and provide necessary
scaffolding for the students. That is, tools should be an integral
part of the learning environment, not an addition. Using cognitive
tools as part of the problem-solving process should facilitate the
development of higher order thinking skills and enhance learning. The
challenge for hypermedia designers is to find ways to accomplish this
[Aspy, Aspy, & Quinby 1993] Aspy, D. N., Aspy,
C. B., & Quinby, P. M. (1993). What doctors can teach teachers about
problem-based learning. Educational Leadership, April, 22-24.
[Ayersman 1996] Ayersman, D. J. (1996). Reviewing
the research on hypermedia-based learning. Journal of Research on Computing
in Education, 28(4), 500-525.
[Barrows 1986] Barrows, H. S. (1986). A taxonomy
of problem-based learning methods. Medical Education, 20, 481-486.
[Barrows 1996] Barrows, H. (1996). Problem-Based
Learning in Medicine and Beyond: A Brief Overview. New Directions for
Teaching and Learning, 68, 3-12.
[Barrows & Tamblyn 1980] Barrows, H. S.,
& Tamblyn, R. M. (1980). Problem-based learning: An approach to
medical education. New York: Springer Publishing Company, Inc.
[Burton, Moore, & Holmes 1995] Burton, J.
K., Moore, D. M., & Holmes, G. A. (1995). Hypermedia concepts and research:
An overview. Computers in Human Behavior., 11(3-4), 345-69.
[Cognition and Technology Group at Vanderbilt
1992] Cognition and Technology Group at Vanderbilt. (1992). The Jasper
Series as an Example of Anchored Instruction: Theory, Program Description,
and Assessment Data. Educational Psychologist, 27(3), 291-315.
[Cognition and Technology Group at Vanderbilt
1994] Cognition and Technology Group at Vanderbilt (1994). Anchored instruction
and situated cognition revisited. Educational Technology, 33(3),
[Edelson, Pea, & Gomez 1996] Edelson, D. C.,
Pea, R. D., & Gomez, L. (1996). Constructivism in the collaboratory.
In B. G. Wilson (Ed.), Constructivist learning environments: Case studies
in instructional design, (pp. 151-164). Englewood Cliffs, NJ: Educational
[Gallagher & Stepien 1996] Gallagher, S. A.
& Stepien, W. J. (1996). Content acquisition in problem-based
learning: Depth versus breadth in American studies. Journal for the
Education of the Gifted, 19(3), 257-275.
[Gallagher, Stepien, & Rosenthal 1992] Gallagher,
S. A., Stepien, W. J., & Rosenthal, H. (1992). The effects of problem-based
learning on problem solving. Gifted Child Quarterly, 36(4), 195-200.
[Germann 1988] Germann, P. J. (1988). Development
of the attitude toward science in school assessment and its use to investigate
the relationship between science achievement and attitude toward science
in school. Journal of Research in Science Teaching, 25(8), 689-703.
[Hannafin & Sullivan 1995] Hannafin, R. D.,
& Sullivan, H. (1995). Learner control in full and lean CAI programs.
Educational Technology Research and Development, 43(1), 19-30.
[Honebein 1996] Honebein, P. C. (1996). Seven goals
for the design of constructivist learning environments. In B. G. Wilson
(Ed.), Constructivist learning environments: Case studies in instructional
design, (pp. 11-24). Englewood Cliffs, NJ: Educational Technology Publications.
[Jacobson & Spiro 1995] Jacobson, M. &
Spiro, R. J. (1995). Hypertext learning environments, cognitive flexibility,
and the transfer of complex knowledge: An empirical investigation. Journal
of Educational Computing Research, 12(4), 301-333.
[Lajoie 1993] Lajoie, S. P. (1993). Computer
environments as cognitive tools for enhancing learning. In S. P. Lajoie
and S. J. Derry (Eds.), Computers as Cognitive Tools, (pp. 261-288).
Hillsdale, New Jersey: Lawrence Erlbaum Associates.
[Liu, M. 1997/98] Liu, M. (1997/98). Interactive
multimedia in an informal setting: The use of a multimedia information
kiosk by university students. Journal of Educational Technology Systems,
[Miles & Huberman 1994] Miles. M. B. & Huberman,
A. M. (1994). Qualitative Data Analysis (2nd.), Thousand Oaks: CA:
[Nelson & Palumbo 1992] Nelson & Palumbo
(1992). Learning, instruction, and hypermedia, Journal of Educational
Multimedia and Hypermedia, 1(3), 287-299.
[Paivio 1991] Paivio, A. (1991). Dual coding
theory: Retrospect and current status. Canadian Journal of Psychology,
[Perkins 1991] Perkins, D. N. (1991). Technology
meets constructivism: Do they make a marriage? Educational Technology,
[Small & Ferreira 1994] Small, R. V., &
Ferreira, S. M. (1994). Information location and use, motivation, and learning
patterns when using print or multimedia information resources, Journal
of Educational Multimedia and Hypermedia, 3(3/4), 251-274.
[Spiro, Feltovich, Jacobson, & Coulson
1992] Spiro, R. J., Feltovich, P. L., Jacobson, M. J., & Coulson,
R. L. (1992). Cognitive flexibility, constructivism, and hypertext:
Random access instruction for advanced knowledge acquisition in
ill-structured domains. In T. M. Duffy & D. H. Jonassen (Eds.),
Constructivism and the technology of instruction: A
conversation, (pp. 58-75). Hillsdale: Lawrence Erlbaum
[Spiro & Jehng 1990] Spiro, R. J., & Jehng,
J. (1990). Cognitive flexibility theory: Theory and technology for the
nonlinear and multidimensional traversal of complex subject matter. In
Nix & R. Spiro (Eds.), Cognition, Education, and Multimedia,
(pp. 163-205). Hillsdale, NJ: Lawrence Erlbaum Associates.
[Walton & Matthews 1989] Walton, H. J., &
Matthews, M. B. (1989). Essentials of problem-based learning. Medical
Education, 23, 542-558.
[Williams 1992] Williams, S. M. (1992). Putting
case-based instruction into context: Examples from legal and medical education.
Journal of the Learning Sciences, 2(4), 367-427.