Bridging Qualitative and Quantitative Approaches in Evaluating the Educational Effectiveness of a Shared Design Memory

Abstract:A shared design memory emerging from the contributions of novice designers affords, theoretically, unique opportunities to support individual and organizational learning. Evaluation must take into account the "distributed" nature of the system that becomes realized. The proposed evaluation model is based on a cross-analysis of: the contents of the shared design memory, the quality of the design artifact produced be the designers teams, the characteristics of the student population, and their perceptions of the adequacy and usefulness of the representational formats adopted for the shared memory. Effects being sought are generational changes that indicate that design weaknesses typical of novices are being offset, and that good design practices are diffused and gradually incorporated as new quality standards. Preliminary results of the evaluation of a shared memory for Information Systems design show that shared memory underpinned an emergent quality in the new designs, characterized by increased structuredness, communicability, and attention to the dynamics of interactions in the system being designed. The shared memory was deemed useful and usable by the learners. Findings also clarify the relative merits of some representational formats (links among design cases and reviews attached to design cases) used for conveying design knowledge.

Keywords: shared design memory, organizational learning, information systems design, evaluation

Categories: K.3.1, K.3.2, H.5.1, H.5.3, H.2.1

1 Introduction

One emerging class of educational software comprises environments that are meant to facilitate learning by encouraging productive forms of collaboration among the learners and by making possible the creation of accessible, shared resources instrumental to the activities of a specific community of learners. Broadly, such environments may be seen as applications of Computer-Supported-Collaborative-Learning (CSCL) (e.g., [Silverman, 95]; [Koschmann, 96]).

The intermediate or final product of collaboration, e.g., the solution to a problem, or the argumentation towards building a shared understanding of a phenomenon, or a design artifact, can be recorded for later use by other groups or members of the community. In this case, the supporting software also aims at creating electronic "organizational memories", where the "organization" can be the classroom [Bruegge & Coyne, 94] or the virtual community of learners who have been enrolled in the

same course in different years [Giordano, 97a]. Peculiarly, such learning environments, typically based on distributed hypertexts architectures augmented by communication and repository technologies, are "content-free". They rather provide basic mechanisms affecting the structure of communication and the way resources for learning can be organized and accessed, thus encouraging (or discouraging) specific kinds of interactions and activities. For example, the learners' notes and discussion threads can be labeled according to types such as "the problem", "my theory", "evidence", "I need to understand" and so on [Scardamalia & Bereiter, 93]. On the other hand, such environments are "mode of discourse-dependent" because the nature of the learning goals, the domain pertaining the competencies to develop (e.g., scientific, technical, design, literary discourse and reasoning) and the constraints of the instructional situation may decree the appropriateness and efficacy of one set of communication and content organization mechanisms over another one.

Although the design of collaborative systems is increasingly grounded on principles of learning mostly stemming from the cognitive tradition to motivate the design features of the systems [Koschmann et al. 96], and deployment is informed by the accumulated wisdom on how social actors interact, these theoretical foci are not sufficient to ensure that learning is taking place. Evaluation of these environments is challenging both on methodological and theoretical counts [Brown, 92], because their deployment amounts to a complex intervention that largely affects the space of educational affordances and is often associated to emphasis on worthwhile but elusive learning goals such as fostering reflection, communication, articulation of ideas, collaboration, motivation, or enhancing metacognition.

Separability of the effects and their tracing to specific features of the technological environment and of its underlying pedagogical model is further complicated by the inherently social and distributed nature of the system that becomes realized, as it is accounted for in the paradigms of situated and distributed cognition [Brown et al., 89]; [Lave, 88]; [Salomon , 93]; [Pea, 93].

Most of the evaluation studies have focused on rich descriptions of how the system is being used, with much emphasis on its educational affordances, mostly following the case study method; the few quantitative studies that have been done have focused on the relationships between types of usage pattern and individual conceptual progress [e.g., Oshima et al., 96]. However, the relationship between the individual dimension of learning and the social, or "organizational" one, seems always to be left in the background as an unresolved issue, especially when it comes down to defining and evaluating the efficacy of the learning environment.

The article focuses on the approach that is being used to evaluate the instructional leverage afforded by a shared electronic design memory which is emerging from cycles of contributions of communities of learners engaged in Information Systems analysis and design [Faro & Giordano, 97a]; [Giordano, 97b]. The system allows the students to retrieve precedent design cases developed by other students, possibly for reuse and adaptation to their current design problem, and to examine (if any) the peer reviews, or any other comment, that have been made to such design cases.

The proposed evaluation method has at its core the relationship between individual learning and the learning of the community that hosts and sustains the process. The latter is referred to as "organizational learning" [Levitt & March, 88]; [Huber, 91]; [Brown & Duguid, 91], and accounts for the fact that although in such

learners' communities there is a complete turn-over of the participants (typically with a yearly frequency), they still manage to transmit myths, procedures and expectations, as well as content knowledge and misconceptions that both enable and limit the performance of the organization. The leading evaluation questions are to what extent designers are facilitated in carrying out tasks notoriously difficult for novices and in what respects the quality of their analysis and of their design artifacts is affected by using and contributing to the shared design memory. The units of analysis are both individuals or small design teams and the community of learners as a whole.

The article is organized as follows. [Section 2] briefly addresses the rationale behind the educational use of a shared design memory. [Section 3] illustrates how the general architectural principles of a shared design memory have been actualized in a system to learn Information Systems Analysis and Design. [Section 4] presents the evaluation model, which is based on a cross-analysis of data originating from the contents of the shared design memory, the actual design artifacts produced by the teams, and from questionnaires to gather individual attitudes. [Section 5] presents and discusses the results concerned with the students' attitudes and the contents of the shared memory. [Section 6], after illustrating the approach adopted to evaluate design quality, completes the preliminary assessment by characterizing the emergent properties of designs produced under shared memory. Some concluding remarks are in [Section 7].

2 Shared Design Memory as a Learning Tool

Key to the problem of supporting organizational learning is finding ways to make the knowledge embedded in artifacts, documents, and pointers to human experts more resilient and more widely available to the potentially interested members in an organization, i.e., enhancing the organization's "memory" [Walsh & Ungson, 91]; [Giordano, 97a]. The informal social networks that are established among the members of the organization are an essential component of the organizational memory, by providing (or blocking) the necessary pointers to relevant expertise sources, and by fueling, mostly through a process of narrative construction, the creation of a shared culture and knowledge base anchored to practice [Brown & Duguid, 91]. In a learning context, the processes above partly concretize in the informal talks with peers who have undergone the same experience, in the exchange of resources (notes, references, etc.) and originate the expectations and understandings that are brought by the students in approaching a new course. The relevance of such informal component of the organizational memory also underscores the limits of what can be represented and "recorded" to make the organization less vulnerable to turn-over of key members, and in general, perform better.

Electronic support of the organizational memory can take many forms, ranging from maintaining an updated system of answers typically asked of the experts [Ackermann & Malone, 90] to case-based reasoning systems [Kolodner, 93], which allow the retrieval of the cases (similar to the problem at hand) with the relevant solution. Experimentation on recording the rationale for decisions, especially those occurring in design meetings, has generated research into languages and graphical notations for expressing the argumentation underlying the design decisions, by structuring the discussion that surrounds any particular design decision, according to

categories such as "support", "object", "goal" [e.g., Conklin & Begeman, 88]. On the one side, the choice of one representational language amounts to a commitment to a certain mode of discourse or to the assumption that it fairly represents the one that is used in practice. On the other side, later intelligibility of the rich resulting picture is difficult even for those who participated, for the lack of context or contextual clues [Buckingam Shum, 97]. This makes even more difficult the transfer or "reuse" of that design experience across different projects or teams. The two points above highlight two requirements central to our discussion, i.e., that both what is recorded and the underlying representational format be as much targeted as possible in supporting the knowledge transfer to new generations of users (as the supervening students in a design course) to support continuity and enhance the process of acquiring and transforming design practices.

Part of the rationale for incorporating the process of using and building a shared design memory in the context of learning a design practice is that this process supports many modes of learning, such as learning by example, learning by collaboration, learning by critiquing, learning induced by the effort of clearly articulating ideas or formulating a query to search for a needed resource. The possibility of inspecting a variety of design artifacts is useful because such artifacts embed technical knowledge about how certain design cases are solved, the variety of cases can suggest more refined dimensions for analysis and help develop cognitive flexibility [Spiro et al., 91].

Most of the above modes of learning could be activated simply by using any resource that can provide enough richness in the examples, such as a ready-made, case-based system [Barber et al., 92], or a cognitive flexibility hypertext [Spiro et al., 91]. However, using and contributing to a shared organizational memory that is owned by the learning community in which it is used, because it gradually emerges from what has been done in the past and mirrors what is of current interest, provides some unique opportunities.

First, it is an experience that is "authentic", al least in two respects: the learners are introduced to a process of knowledge construction in which they have to take personal responsibility, rather than taking knowledge as a given, and they exercise the communication skills that facilitate transferring and reusing experience, in the effort to making their contribution as intelligible as possible (even to those that did not share the context of its creation) [Giordano, 1997a]. This latter aspect is especially important in domains in which competence requires being able to participate in the collaborative efforts of an interdisciplinary team.

Second, especially in the domain of design (in which there is a marked creative and personal component both in the way of framing problems and of finding solutions) the fact that the design cases are shared provides a reference model of the quality that can be achieved. This can be a motivational drive for those learners that are truly novices to the field, and also for those who are more experienced and confident. Thus one element of educational leverage is provided by the means of creating individual "identities" as a designer in the space of the shared design memory, which implies

being able to relate and differentiate one's own artifact from those that already exist [Giordano, 97a].

Because the contributions to the shared memory originate from novices, variation in their quality must be expected. This is an opportunity for the students to exercise

critical skills and avoid "blind reuse" of the design cases. It is also an opportunity to highlight design weaknesses in context and make them meaningful to a wider audience. But it is also entails the risk of promoting uniformity or the diffusion of incorrect or unproductive practices if the users do not generate enough variety and insights to make the system self-correcting. Thus, to fully gather the potential educational benefits, the adopted representation and structuring mechanisms must be carefully tailored to the design domain, ensuring consistency with the sought after modes of discourse, and must embed some features that counteract the risk of uniformity.

These issue s are central to the context problem, i.e., preserving and enhancing the meaningfulness of the information, and to deciding to what extent product and process must be represented in the memory, for example by summarizing the process that led to the solution of a problem together with the solution. How the above issues have been taken into account in the architecture of a shared design memory in the domain of Information Systems analysis and design, is illustrated in the next section.

3 StoryNet: a Shared Design Memory for Information Systems Analysis and Design

StoryNet [Faro & Giordano, 97a]; [Giordano, 97b] is a shared memory for the introductory course to Information Systems (IS) Analysis and Design at the undergraduate level in an information engineering program. Broadly, Information Systems analysis and design involve gaining knowledge about the business organization, building a conceptual model and "transforming" it into a formal representation detailing the requirements in a form suitable for design and implementation considerations. This transformation is very demanding because structured analysis and design methodologies, although providing procedural guidelines and diagramming notations, support only partially analysis, requirements acquisition and comprehension [Sutcliffe & Maiden, 92]. Capturing the requirements specifications in a conceptual model of the system requires solid analytical skills but is also affected by familiarity with the application domain, which underpins a sense of relevance in identifying the data and key processes in the organization. Mastering specific modeling techniques such as data modeling (e.g., the entity-relationship formalism), function modeling (e.g., data flows), process and event modeling (e.g., Petri networks) is a relevant component of IS analysis and design competence, but at its core are processes of continuous analysis and synthesis involved in the integration of different views, in tracing specifications to requirements, validating the semantic and dynamic aspect of the model under construction, and performing the ill-structured transition from the conceptual model to the architecture of the system to be implemented (e.g., in our case, a relational database).

StoryNet aims at supporting individual learning by enhancing the ability to carry out a deep, user-centered analysis of the business organization, soliciting uniqueness in the proposed design solution, and consolidating the ability to critique and verify the design, and organizational learning by supporting the circulation and acquisition of design ideas and practices. Its architecture stems from the idea of getting instructional leverage by jointly operating on an explicit model of the biases and difficulties of

novice analysts and designers, and on the methodological approach to design.

The extant literature on the cognition of novice data modelers [Batra & Antony, 94] and systems analysts [Maiden & Sutcliffe, 92] highlights some recurrent difficulties and biases in carrying out the analysis and design process. The difficulties involve scoping the problem and recognizing its boundaries; performing problem decomposition; reasoning on model completeness; and generating and testing hypothesis about the model by robust problem-solving strategies (such as heuristic-based reasoning or use of scenarios). The biases are: the tendency to concentrate immediately on implementation issues at the expense of high level analysis concerning the requirements; piecemeal modeling by "literal translation" of nouns to entities and verbs to relationships resulting in sub-optimization in the design solution; paying more attention to the syntax of the application than to its semantics; lack of specificity in the universe of discourse.

StoryNet embodies an analysis and design methodology akin to scenario-based design, i.e., the Story Telling Theory (STT) [Faro & Giordano, 97b]. Such methodology provides a template to formalize the use-cases of the Information System in a set of stories and episodes that inherently take into account the user point of view, and allows the designer to organize the requirements in such a way that propagation of effects among stories tend to be confined, thus allowing partial and incremental verification of the model. The method helps lessen the cognitive load involved in performing such task, and provides a contextualized framework that facilitates interpretation of the relevant data and process models.

Thus the prime organizing theme of StoryNet are "stories" and "episodes" that model the organization following the STT design methodology. The template ensures fine-grained indexicality of the materials to identify parts of any project that could be relevant to the domain selected by the student. Attached to these stories and episodes are multimedia documents illustrating the data models and snapshots of the user-interface of the implemented prototype. By examining cases organized in stories and episodes, their structure can be readily perceived, but not the motivations underlying that structuring. This might contribute to a stereotyping problem that has been observed when the students of the IS analysis and design course have access to previous projects but don't have ways to evaluate how they stand relatively to each other. Although methodologically correct, the approach to the analysis tends to re-propose surface level analogies with other organizations, rather than focusing on the differences, as required by truly user-centered design.

To counteract this tendency, cases in StoryNet are incrementally linked in a network of old and new related design cases to highlight how design solutions evolve across time and different groups of learners. Reference links point to units (stories or episodes) that have been used in the design of the retrieved unit and are typed to make apparent how and why that unit was taken into consideration. Available typed links are: `Correct', `Extend', `Detail', `Adapt', `Use as is', `Restructure', `Other'. Links are mediated by nodes that further qualify the typed conceptual link, by including an additional comment or "explanation". The annotated references of the project should make apparent the analytical and critical effort invested by the student in the project, his or her personal contribution in structuring and introducing new cases, and indirectly foster in the learners greater specificity in their analyses. The structure of

the case representation and the network is sketched in [Fig. 1]. StoryNet is currently implemented on Domino Lotus Notes.

An additional component of the shared design memory are the design critiques that are attached to the design representations. Such critiques are based on a guideline for reviewing a project that is an implicit model of how an expert would approach the evaluation of the design. The philosophy of the shared design memory is that there is some added value in sharing such documents beyond the process of producing them. Accordingly, the reason for sharing the critiques is that they have potential to provide a contextualized representation of the possible weaknesses in the reviewed precedents, and function as an intelligible and illustrative warning. The load of creating such resource is distributed among the students and, on average, if each team critique one precedent, as it is required, a representative sample of the biases and misconceptions can be generated.

The relationship between the design features of the shared memory and the hypothesized influence on the biases is sketched in [Fig. 2]

4 The Evaluation Model

As a consequence of the perspective that a shared design memory is just a part of a distributed system, evaluation of its effects on individual and organizational learning must take into account dimensions pertaining to the design artifacts, the community of learners, and the shared design memory, as being an artifact itself. The overall learning environment's effectiveness in supporting analysis and design skills can be evaluated by traditional measures of quality of the design output, although a better picture must include also aspects related to the process of using this environment, such as to what extent students are facilitated in the start-up of their projects, and how

the overall process of designing taking into account the shared design memory is perceived.

The underlying model is illustrated in [Fig. 3]. A community of learners comprises small groups and individuals, linked by networks of friendship, trust, recognized expertise Each of these "units" engages in the activity of producing a design artifact, which is eventually represented in the shared design memory. In the process, they resort also to the precedents available through the shared design memory and possibly contribute some useful knowledge, in the elementary form of a pointer to design resource, or of an articulated design insight.

Relevant dimensions to the community of learners are the group profile, the attitudes towards the shared memory, and the overall organizational learning of the community as a whole. Group profile must be considered when investigating the system because it can be a factor that affects the trends with which the overall system evolves, or that underlies possible failures of specific features of the shared memory to correspond their intended use. Level of initial competence in the design area of interest or in related fields is one aspect to normally include in the definition of the profile. As an heuristic criteria, any features that vary mostly in the overall community (such as schooling background, interest in the subject, to name a few)

Attitudes towards the shared design memory must be seen both globally (in terms of the community's acceptance of the system) and at the individual level (in terms of how each actor responds to the overall mode of dealing and thinking with representations enforced by the system). Thus one angle to investigate - the students' perception of the shared design memory - must regard the pedagogical rationale underlying the approach, i.e., the utility of examining precedents, the utility of critically reviewing some of them and the utility of sharing both the precedents and such reviews. Another angle of analysis must focus on the effort or difficulty involved in the conceptual and communicative operations that are requested of the students in order to sustain and develop the contents of the shared memory. Finally, another angle of analysis must directly address the assumptions made by the designers of the shared memory concerning the effectiveness of the formats chosen to represent the precedents, to link them, and the solutions adopted to browse and to contribute to the system. Related to this latter angle of analysis (and to the first one) are preferences that the users of the system might express on any kind of information that they would like to see in the shared memory.

Indicators of the organizational learning that is taking place due to the introduction of the shared memory are:

• the kind of design features or solutions that have become standard (i.e., tend to be present in the majority of the more recent projects, whereas they were rare in a former generation);
• the degree of innovation, i.e., the percentage of projects within a generation that exhibit new features;
• the number and type of design weaknesses, possibly related to the novice cognitive biases and difficulties, that tend to disappear or persist.

To highlight any trend of improvement, quality of designs can be compared to the quality attained in the former years, if the key elements in the used pedagogical approach have been kept constant (except for the introduction of shared memory) and the salient characteristics of the target population of students enrolled in the course for which the shared memory is implemented have not changed. Thus evaluation has an across-generations component and a within-generations component aimed at pointing out any differential effect that might be taking place. In this respect information about

the group profile is essential to understand the limits of comparison and generalization.

Concerning the shared design memory, case representation, architecture and usability must be taken into account explicitly because it is necessary to see whether (beyond the raw cases) the quality of the additional representations (e.g., comments, reviews, links) is good enough to be conducive to the hypothesized effect of highlighting from many perspectives design strengths and weaknesses. This amounts to evaluate whether the community is able to produce such statements and to use them productively.

Thus the evaluation method involves a cross-analysis of three types of data originating from: 1) design artifacts; 2) the shared design memory contents, i.e., cases, peer reviews, comments and links; and 3) an individual questionnaire, which collects personal data aimed at characterizing the student population, and asks questions to elicit the student's perception of the effectiveness and usability of the shared memory and a reflection on the kind of difficulties encountered in the design activities.

5 Applying the Evaluation Model: Some Preliminary Results

The first evaluation of StoryNet was conducted based on the data obtained from the period that goes from the initial deployment of the shared memory, March 1997 to the end of July 1997. At this point the shared memory contained 34 new design cases, relevant to the course activity of the then current academic year. At that stage, 56% of the students had taken the final exam (64 students out of 114 enrolled in the course). Data reported on the following originate from the questionnaires collected from the above 64 cases.

5.1 Characterizing the Students Population

The indicators chosen to characterize the student population are: a) the level of experience in database design prior to the course; b) an index of overall competence in skills potentially relevant to the course prior to enrolling at the university; referred to as p.r. (previous related) competence; c) grade point average (g.p.a.); and d) level of interest in the subject. Overall p.r. competence was computed by summing the levels of computer experience, programming experience and database design experience, after asking the students to rate their competence on a five level scale (none = 1, novice = 2, intermediate = 3, advanced = 4, expert = 5).

Individual level of interest in the subject

(a)

Individual level of database design experience

(b)

Overall p.r. competence of individual forming teams

(c)

Overall team p.r. competence by Team average g.p.a.

(d)

The population of students can be characterized as having a strong interest in the subject, with only 22% declaring a medium interest, 58% a high interest and 20% a very high interest [Fig. 4a]; specific prior competence in database design has a majority of cases with none experience (63%), and is skewed towards the none - novice end of the scale, which together account for approximately 80% of the total [Fig.4b].

Of the 34 design projects, 7 were carried out individually, 24 were carried out by teams of two members, and the remaining 3 were carried out by teams of three members. Those individuals who preferred to work alone (11%) did not differ with respect to the mean p.r. competence and mean g.p.a. from those individuals who formed teams.

An analysis of how teams were composed was performed to highlight whether there was any unbalanced distribution of the team characteristics that needed to be

considered when evaluating the achieved design quality. The distribution of the overall p.r. competence of individual who formed teams is shown in [Fig. 4c]. There was a tendency of students in the novice to intermediate range of forming groups with students in the intermediate to advanced range, whereas students with none or minimal experience mostly grouped among themselves. An indicator of overall team p.r. competence was obtained by summing the scores of the members and recoding them into three levels. The resulting distribution of the overall team p.r. competence (factored by team average g.p.a.) is shown in [Fig. 4d]. It indicates that the three types of teams had approximately equal proportions of students in the low and medium range of g.p.a., whereas the majority of the students with a high g.p.a. were in the none to novice category of overall team p.r. competence.

There was no correlation between the individual grade point average of the team members, and 58% of the teams were formed by students who had previously worked together.

5.2 Individual Attitudes Towards the Shared Design Memory

The responses concerning the individual perceptions of the shared memory were gathered by an individual questionnaire which ensured confidentiality of the results, and that was administered when the students turned in their projects, before undertaking the oral discussion, and returned at the end of the exam. A total of 64 questionnaires were collected. The first section of [Table 1] shows the ratings of the utility of examining precedents, critiquing them and sharing the critiques; the second section shows the rating of the ease of navigating and contributing to StoryNet.

Data in [Table 1] consistently report perceptions skewed towards the medium-high end of the scale, indicating a substantial agreement on the key tenets of the rationale underlying the use of the StoryNet. There are no particularly revealing differences among frequencies of the responses in the medium category and the sum of the responses in the high-very high categories, except for the utility of sharing the critique, which shows a clear orientation of the responses towards the high-very high categories of the scale, accounting for 54.8% of the cases, versus the 38.7% in the medium category and the 6.4% in the low-very low end of the scale. This result is particularly interesting because, regardless of the perceived utility of actually doing the critique exercise, the students seem to place a very high value on the possibility of looking at the reviews authored by their peers. Interestingly, there is a negative correlation between level of initial design competence and utility of doing the critique exercise (Kendall = -.281, p<.05)

(*) m.c.= missing cases

For the cases who rated low the utility of precedents (17.7%) the most frequent motivations for holding such view were: poor quality of the available precedents, incompleteness, diversity from the theme to be addressed in the new design case.

Although relevant only to a minority of cases, such responses highlight one risk inherent to using a shared memory constructed by the students themselves: that is, unless specific measures are taken, the weaknesses in the novice artifacts might not be for every student a helpful point to start exercising critiquing skills. However, a mitigating factor for the seriousness of theme diversity as a hindrance, is the start-up effect of the shared memory. In fact, to start the memory, the students were given by the instructor the precedents for review, based on criteria of availability, if relevance to the new design theme could not be satisfied. Therefore, at the time the responses were collected, there was not enough variety of cases formed in StoryNet to provide more relevant examples.

The average number of precedents examined individually is P = 2.36 (SD = 1.39, ranging from 0 to 6) and there is no interaction with level of design experience and level of overall p.r. competence. Concerning the source of the precedents that were examined, 42% of the students resorted to other design cases provided either by colleagues authors of such projects (96%) and/or by colleagues that were not the authors of the design case themselves (30%). These figures are indicative of the social phenomenon of exchanges naturally occurring among the students, and raise the questions of whether and in what respect a shared design memory adds to this process.

The first section of [Table 2] summarizes the agreement level, in a seven points scale, on some statements concerning the conceptual and communicative operations the students were required to perform. The set of questions concerning the links stem from the hypothesis that deploying links entails examining precedents with the aim of

finding proper elements for reuse, and that once the design has been carried out, one can still remember what specific aspects of the precedents have exerted influence. This distinction, that might seem unnatural, is due to the fact that in the phase of the start-up of the memory the actual design cases are examined outside StoryNet, and then represented. It is expected that at regimen this splitting will occur to a less extent and that the linking process will support the development and refinement of ontologies in stories and episodes within StoryNet.

The second section of [Table 2] summarizes the agreement level on statements aiming at capturing the possible roles that StoryNet, as a social system and as an instrumental tool, might perform.

The results in [Table 2] show substantial agreement with the statements that:

• to deploy links supports the process of articulating the reasoning that went on during design;
• such links can be useful to peers;
• StoryNet is useful as an index to cases;
• it is a tool that fosters reflection.

This shows acceptance of the system as implemented, and complements acceptance of its pedagogical tenets. On the other hand, the angle of analysis concerning the conceptual operations that are needed to sustain StoryNet reveals a difficulty in the retrospective reasoning that must be performed to articulate what aspects of the current design have been influenced by the precedents (40.6% neutral and 46.3% in the agree side of the scale), and that in the preliminary phase of scanning such precedents it is difficult to anticipate what specific aspects will prove useful in the development of the new project (31.5% neutral and 43.6% in the agree side of the scale). Less difficult is the process of formulating comments in a usable form to peers.

The above difficulties are interesting because whereas there is agreement on the utility of the linking mechanism (response n.1) there is also an indication that it requires an effort that not everybody can sustain, either because of an objective lack of experience that hinders the process of anticipating what might be important, or because there has not been sufficient training in the process of examining the precedents with an attitude towards comparing and contrasting. On the other hand, the difficulty of retrospectively tracing what features have been imported in the new design can also be symptomatic of a process of personal internalization and restructuring of the contents and representational formats that have been encountered. These considerations call for a more detailed analysis concerning whether the process of examining precedents is done more with the spirit of learning or with the spirit of reuse.

(*) m.c.= missing cases

There is also agreement on the idea that the StoryNet could benefit if authors respond to the peer reviews. Although StoryNet supports this mechanism, this is unlikely to happen because of the marked asynchronicity with which the groups carry out and complete the design. Indeed, most of the contributions occur just when the project has been completed, shortly before presenting the exam. So any after-course volunteering of information is completely discretional. The deployment of links (from the technical point of view) was rated as being somewhat difficult by approximately 30% of the respondents. This finding had a counterpart: a closer examination of the links actually deployed showed that in some cases the students created the typed links or the explanatory nodes attached to the design cases to be eventually connected but failed to complete the process by skipping the step of actually "closing" the connection among such nodes. Thus the next version of StoryNet must ameliorate this latent usability problem. On the other hand, the overall ratings of the ease of navigating and contributing to StoryNet are quite satisfactory, as shown in [Table1].

[Table 3] summarizes the opinions about examining in StoryNet different kinds of design representations and the various contributions coming from the participating social actors. Ratings vary on a three points scale: no utility, little utility, and much utility.

(*) m.c.= missing cases

A marked diversity of opinion characterizes the issue whether StoryNet can be useful if the whole design case cannot be accessed, which directly brings to bear on the issue of adequacy of the case representational format. Whereas [Table 3] indicates that the supported representations are rated mostly as having much utility, responses to an open-ended section asking whether some other kind on information was needed in the StoryNet indicated the following: a) entire analysis, 2) the software prototype, 3) the dimensioning of the system, 4) other diagrams, 5) best solutions, 6) the

instructor evaluation. Only 36% offered this information. [Table 3] shows that the rating of the links is markedly different form the other items, indicating a much more cautious attitude (56.7%) towards the utility of examining links. This result apparently clashes with the responses in Table 1, in which links are rated as a useful vehicle to express design reasoning. A plausible interpretation at this start-up stage, is that for the first generation of students who deployed the first links, the number of links to actually examine in the system was too low to generate the perception that the mechanism is worthwhile representing and has utility as an additional indexing and navigational mechanism.

5.3 Content Analysis of the Critique Exercise

The reviews produced by the teams were analyzed by classifying each critique statement as belonging to one of the following problem areas:

1. Dynamics: concerns the dynamical and simulation aspects of the domain analyzed, and the correct use of the appropriate modeling techniques;
2. Breakdowns: concerns the anticipation of possible breakdowns in the normal functioning of the system and devising exception handling procedures;
3. Problem scoping: concerns the definition of the problem boundaries;
4. Structuring: concerns issues of decomposition and organization, and includes the appropriate use of generalized structures;
5. Language: concerns the specificity or ambiguity of the language used;
6. Domain accuracy: concerns the accuracy of the description of the domain, including the omission of major defining features;
7. Design discourse: concerns all the aspects of data modeling and methodology application;
8. Communication: concerns all the aspects involved in the effective communication of the analysis and design representation, from the graphical layout to the explanatory comments to the design representations.

A total of 32 reviews were analyzed, yielding the distribution of detected weaknesses shown in [Fig. 5].

For the purposes of the present analysis, it can be remarked that the distribution shown in the graph must not be interpreted as representing the average quality of the reviewed precedents. Rather it is a representation of what was perceived by the students at their current level of maturation, given the "observation lens" of the critique guideline, the particular precedent examined, and their expressive ability. In fact, the most frequent counts in a problem area could be accounted for partly by a better sensibility of the reviewers towards the underlying issues. Thus the distribution is better interpreted as meaning that the overall system, socially, was able to find instances of issues to be aware of in the whole spectrum of problem areas, certainly with a varying degree of penetrating insights, but demonstrating that there was no "organizational blindness" to any if them.

Each review was also coded as to whether it was effective in highlighting the strengths and weaknesses of the precedent, by meeting the criteria of clarity of expression and localization, i.e., justification by direct reference to an easily identifiable part of the precedent itself. Additionally, a statement was considered as: a) conveying domain knowledge when the authors justified their evaluation making explicit reference to the application domain, and b) conveying design knowledge when the authors justified their evaluation by proposing design solutions, or by introducing a novel dimension other than those suggested in the review guidelines.

Approximately, half of the contributors were able to communicate effectively the strengths and weaknesses of the reviewed cases, and approximately 40% did so by referring to the domain application.

A similar pattern occurs for the links among the design cases. Of the overall contributing teams or individuals, 85% placed links to the precedents. The average

number of links placed was 2 (ranging from 1 to a maximum of 4). The majority of the links were of the type "Adapt" (24%), "Use partially" (18%), and "Other" (18%).

However, the links that cogently expressed design reasoning beyond the synthetic type were only 32% of the total. Relatively, precedents' reviews appeared to be a medium more easily tapped to augment the knowledge in the shared memory.

The findings that 40% of the reviews and approximately 30% of the links explicitly convey knowledge are important to estimate the actual potential of the shared memory for highlighting design weaknesses and strengths in context (when the number of cases increases) and the information gain with respect to the knowledge implicitly conveyed by the case representation itself.

6 Design Changes across Generations: Evaluating Design Quality

Although there are some "intrinsic" qualities of the conceptual model as a representation (such as its syntactic correctness or its readability) these are second order regard to the issue that the semantic of the model is correct and complete with respect to the domain and the activities that the information system is supposed to support. Syntactic quality, semantic quality, and pragmatic quality (i.e., that the model is understood by the interested parties) must be addressed in terms of feasibility goals [Lindland et al., 94]. Total validity and completeness, defined, respectively, as the property that all statements made by the model are correct and relevant to the problem, and as the property that the model contains all the statements about the domain that are correct and relevant, cannot be achieved except for very simple problems. The goals of semantic quality are feasible validity and feasible completeness of the model, and the goals of pragmatic quality are feasible comprehension of the model. Quality properties of the model such as correct, minimal, annotated and traceable, consistent, unambiguous are mostly subsumed by validity and completeness. Feasibility provides a stop rule to terminate the modeling activity, i.e., when the model has reached a state where further modeling is less beneficial than applying the model in its current state.

Quality of information is related to how data are used, thus quality of the data generated by the information system should not be couched in terms of "data centric" concepts such as entities, attributes and values, but in relation to intrinsic dimensions, assuming that the true intentions of the users have been captured in the conceptual model [Wand & Wang, 96]. Such intrinsic dimensions are: 1) Complete (there is no loss of information about the application domain, 2) Unambiguous (data generated by the information system cannot be interpreted in more than one way, 3) Meaningful (data can always be interpreted in a meaningful way, and 4) Correct (data derived from the information system conform to those used to create these data. This view complements the notion of feasible quality in the conceptual model by emphasizing the mapping process from requirements to data, in such a way to project the data model towards aspects of design more oriented the prototyped information system.

An important overlooked dimension in traditional approaches to quality is how the conceptual and data model that are being crafted handle issues of interactivity of the overall system, also considering that within a scenario-based approach to design (such as STT) the interface of the system is not a separate aspect of the design. A way to indirectly address this aspect of quality is to place emphasis also on the modeling of the dynamic interactions in the system, resorting to different representations to

carry through the design the user and his or her interactions with the data. This kind of modeling effort is also beneficial for the consistency of the data.

6.1 A Framework for Evaluating Design Quality

The framework adopted for evaluating the design quality is centered on the following points:

• Semantic quality of the overall conceptual model, especially focused on feasible validity;
• Emphasis on completeness meant as multidimensionality of analysis relative to the stories/episodes identified by the students, which set the internal standard for evaluating the design;
• Feasible completeness and validity of data modeling, relaxed with respect to the fact that small errors cease to assume relevance if they are corrected or disambiguated in further representations, often complemented by the prototype;
• Integration of a dimension that explicitly takes into account interactivity;
• Communication, to address feasible pragmatic quality of the design.

Accordingly, the design scoring template has been developed following a design features approach (i.e., grading 0 or 1 - either the absence or presence of a certain design feature, mostly concerning representational or stylistic solutions) and a graded quality evaluation for Domain modeling specificity and Data modeling (0.5 if a minor weakness is present, 0 if a major weakness is present and 1 for feasible validity).

The feature-based approach is particularly appropriate in our case because whereas in other studies the designs are comparable because they deal with the same problem, in this context it is necessary to find a common reference point across designs that tackle different problems. Also, a feature-based analysis is a more effective way to detect, from an organizational learning point of view, what tends to become consolidated, what is emerging or tends to disappear generation after generation.

The set of features and their values have been mapped to contribute to the definition of the values of the following quality indicators:

• Domain modeling specificity
• Structuring (decomposition)
• Structuring (composition)
• Data modeling
• Dynamics of interactions
• Breakdown
• Communication
• textual communication
• visual/hypertextual communication

The list of features that concur to form the indicators and the range of variation of the scores is reported in [Table 4].

Legend: DFD = Data Flow Diagram; ERM = Entity Relation Model; ELH = Entity Life History; PO = Process outline; PN = Petri Network

Legend: DFD = Data Flow Diagram; ERM = Entity Relation Model; ELH = Entity Life History; PO = Process outline; PN = Petri Network

The novice difficulties and biases mentioned in [Section 3] are mapped in the above dimensions as follows. "Domain modeling specificity" is an indicator the specificity of the language, derived from the ratios of domain specific entities to total entities, domain specific functions to total functions, and domain specific interactions to total interactions. "Breakdowns" addresses depth of analysis and indirectly indicates the ability to generate testing scenarios against which evaluating the quality of the data model. "Structuring (decomposition)" aggregates the design features that indicate an effort toward organizing the decomposition the static and dynamic data models by one or more of the following strategies: 1) decomposition of the static and dynamic data models following the story/episode approach, or the top-down approach, or both; 2) use of "consist of" structures in modeling the entities; 3) use of cross-reference matrices between stories and entities. "Structuring (composition)" aggregates the design features that indicate an effort towards maintaining in a global representation explicit information about the elements that are aggregated and composed, either by color-coding or with links, or by simulating through animation the incremental creation of the global representation.

6.2 Results

To evaluate the trend of design quality a random sample of 16 projects was drawn from the pool of 32 cases belonging to the new generation (henceforth, 2^nd generation) and a matching set was created by selecting the corresponding reviewed projects from the old generation (henceforth, 1^st generation). Because one of the reviewed precedents was missing and two cases from the sampled 2^nd generation had reviewed the same precedent, the matching set was completed by adding two other cases drawn from the pool of precedents available to the students. The projects were scored blindly by an independent rater, an expert in the field, following the template elaborated in the previous section. Data from the two generations were screened separately for outliers and plotted for a qualitative analysis of variations in their distributions. A t-test was performed to compare the mean score of each quality dimension for the two generations of design cases. A correlational analysis using the Spearman rank correlation coefficient was performed to investigate the relationships among the various dimensions of the design quality for the second generation, taking into account the characteristics of the teams. Also, a correlational analysis for elementary design features and generations was performed to highlight, at a micro level, what were the practices, or representational modes starting to become diffused in the new generation. The descriptive statistics for the dimensions of design quality are reported in [Table 5].

(n. of cases: 1^st Gen = 16, 2^nd Gen. =16)

Additional information about the trends in the two generations can be derived from the concise representation of the distribution of the scores for each variable, as shown in the boxplots in [Fig. 6]. The shaded area in the boxplot represents the range within which fall 50% of the scores (25^th to 75^th percentile) and the line in the box indicates the median. The vertical lines extending out of the box mark the largest and the smallest observed value that is not an outlier, i.e., more that 1.5 box-lenghts from the upper and lower edge of the box. The distribution of Domain modelling specificity in the 1^st generation (with 12 cases who scored 3, three cases who scored 0 or 1 and only one case who scored 4) reflects the bias of lack of specificity or "stereotyping" frequently observed in the precedents. The situation is significantly better in the 2^nd generation, in which the lowest value is 2.50 and 15 out of 16 cases fall between 3 and 4. Structuring (Decomposition) reveals the significant effect for the 2^nd generation regarding the increase in the structuredness of design. This indicates that some of the features, or different combination of features relative to modes of organizing and layering the design are becoming common. Structuring (composition) did not show any change across the generations. Data modeling reveals that for the 2^nd generation there was an increase in the spread of the values, and an increase in the number of cases in the higher range if the scale, thus indicating a tendency to improve. For the variable Breakdown there was no substantial variation, only the fluctuation of few cases in the lower end of the scale. The index of complexity remained practically unchanged across the two generations.

Statistically reliable differences among the two generations were found for: Domain modeling specificity (t= -2.68, p<.05); Structuring (decomposition) (t= -2.12, p<.05, df = 25 - adjusted for inequality of variance in the samples -Levene test), Dynamics of interactions (t= -3.92, p<.05), Communication (textual) (t= -3.95, p<.05), and Communication (visual-hypertextual) (t= -3.21, p<.05).

The above results indicate that the projects of the 2^nd generation showed more specificity in the language used to describe the domain, more effort aimed at structuring the analysis and an increased attention towards modeling the dynamics of interactions. Tapping more communication modalities, both textual and hypertextual, was also a distinctive mark of the 2^nd generation.