A COMPARATIVE EVALUATION OF PRINT AND ELECTRONIC REVIEWS OF MULTIMEDIA INFORMATION PRODUCTS

Abstract: Reviews of information CD-ROM products were examined in relation to those criteria which home users identify as important in the selection of information CD-ROMs. Specifically we were interested in identifying what relationship, if any, existed between the key overall elements which characterise the reviews and the medium (print or electronic) and type of publication in which they appeared. A coding scheme was devised based on work undertaken previously by the authors and with reference to appropriate literature. Content analysis was undertaken using NUD.IST qualitative data analysis software. Our findings indicate that the nature of the review is influenced by the publication medium and type.

Category: K3 - Computers and education

Keywords: reviews; multimedia; evaluation; applications, content analysis

1 Background

A previous study of home users of CD-ROM encyclopaedias by Gillham and Buckner [Gillham and Buckner 1996] showed that the users regarded two specific factors as being of crucial importance to their use of the products. These are the product's information content, and the ability to search for relevant information. Observations of actual use of such systems suggested that users were also interested in the multimedia aspects of the products, with considerable time being spent searching for and watching videos, listening to audio clips and viewing photographs. Many aspects of information products are of interest to home users and these would have to be considered when deciding which particular product to purchase. Where would potential users obtain relevant information to inform their decision making? Reviews of products (both print and electronic) are the most likely source of such information, and a study of the information content of reviews was undertaken [Buckner and Gillham 1997]. Our initial findings indicated that the reviewers commented most frequently on the multimedia content of the reviews (in 94% of documents and comprising 9.4% of the text). The study also confirmed the importance of searching and textual content with these being discussed in 52% and 65% of the documents

respectively (6% of the text related to searching and 5.5% related to textual content). Thus reviews do tend to provide information about those criteria which are of importance to home users.

2 Aims of this Study

Although our previous study found that the same criteria were important for both home users and reviewers, we did not at that time report in depth on the nature of the information provided by the reviewers.

The specific question addressed here is:

What are the relationships, if any, between the key overall elements which characterise the reviews? The factors of interest are:-

• the review distribution medium (print or electronic)
• the media's target audience (general non-specialist publications such as newspapers and lifestyle magazines verses specialist information/computing publications)
• the overall critical tone of the review (descriptive or evaluative)
• the product placing (new products, upgrades or not specific)
• the emphasis presented by the reviewer about the product (newness, technology, quantity of information, multimedia or not specific)
• the nature of the review in terms of qualitative or quantitative comment
• the space allocated in the review to different evaluation factors (the information contents, methods of access, interface and multimedia types).

3 Methodology

A range of reviews of text rich information CD-ROMs (eg encyclopaedias, reference books,) were selected. A detailed description of the identification of the categories and sub-categories of sources of reviews examined has been reported elsewhere [Buckner and Gillham 1997]. Two broad categories, print and electronic media were identified. Within the print media category we included newspapers, weekly and monthly popular computing magazines, lifestyle and other popular non-computing magazines, specialist magazines (aimed at professionals) and advertisements by CD-ROM publishers. Within the electronic media category we included newspapers, reviews by public bodies, ad hoc reviews, and reviews by CD-ROM publishers.

A total of 31 reviews were analysed, with a range of publications in each category and a variety of products being selected. The reviews were digitised and the data validated to ensure accuracy.

OVERALL EVALUATION FEATURES
CRITICAL TONE:                descriptive/ evaluative 
EMPHASIS OF REVIEW:           information/ multimedia/ newness/ 
                              quantity/ technology not specific
MEDIA TYPE: 
      Electronic:             ad hoc/ newspapers/ public bodies/ publishers 
      Print:                  adverts/ computer mags/ information mags/ 
                              lifestyle mags/ newspapers 
PRODUCT PLACING:              new product/ not specific/ upgrade 
SPECIFIC MARKET:              no/ yes 

GENENERAL EVALUATION 
DESCRIPTIVE/EVALUATIVE 
EVALUATIVE: 
      Comparative:            CD-ROMs/ Internet/ paper source 
      Rating given:           yes/no 
      Tone:                   negative/ positive /neutral 
DETAIL:
      Access:                 browsing/ filtering/ linking/ search/ timeline 
      Aesthetics 
      Information overall:    authority/ depth/ national focus/ quantity/ 
                              up to date 
      Interactivity:          games /presentations/quizzes/simulations 
      Interface:              buttons/ icons/ menus/ metaphor/ mouse/ 
                              windows 
      Internet links:         yes/ no 
      Multimedia:             animation/ atlas/ audio/ pictures/ 
                              tours/ video 
      Stated purpose:         children/ education/ fun / homework/ 
                              research 
      System factors:         performance/ pricing/ requirements
      Text 
      Usability 

A coding scheme was devised and the data was coded (using NUD.IST qualitative data analysis software [Q.S.R. NUD.IST 1997]). The choice of criteria for coding was influenced by work undertaken previously [Gillham, Kemp and Buckner 1995]. However, we also examined and took account of criteria used by others for evaluating user interfaces [Zink 1991; Bosch and Hancock-Beaulieu 1995; Rowley 1995], WWW sites [Rettig 1996], reference books [Stevens 1986], and electronic databases [Harry and Oppenheim 1993a, 1993b]. Indeed most of the literature on criteria for selection of CD-ROMs relates to the selection of CD-ROM databases for use in academic libraries. However, Clements and Nicholls offer a guide to reviews of CD- ROM reference works and produce a rating scheme for each product based on overall impression, text, multimedia aspects, and the user interface [Clements and Nicholls 1995]. Some guidance on the choice of criteria was taken from reviews of the process

of selecting printed reference works [Kobrin 1988; Stevens 1986]. Kobrin includes amongst her ''clues`` for evaluating children's books aesthetic elements such as attractiveness and book design, as well as more pragmatic aspects such as accuracy, appropriateness for audience, and authority. Based on theory and practice Stevens produced a set of 18 criteria against which to evaluate reference works for inclusion in academic libraries. These cover issues relating to content (including authority, accuracy, completeness, relevance and appropriateness of illustrations); organisation and arrangement (indexing etc); and, comparability with other reference works. Rettig uses these criteria to develop his own criteria for the review of WWW sites [Rettig 1996].

The aspects of the coding scheme which relate to this study are: coding of the entire document according to media type (electronic or print) and category (e.g. newspaper, PC magazine); critical tone of the review (whether evaluative or descriptive); and overall emphasis of the review (e.g. newness of information, quantity of information). Sections of the review text relating to evaluation were coded according to evaluative tone (positive, negative, or neutral). At the detailed level the information content was coded on both qualitative and quantitative aspects [see Tab. 1]. Other aspects included information about multimedia, access strategies, interactivity, and user interface.

4 Analysis and Discussion

Analysis was carried out on the reviews which had been coded as described above. Three sets of figure are available for each of the codings made:

• the percentage of documents that contain some text coded for a particular factor,
• the percentage of the text about the factor in those documents that have some text coded in the area of interest,
• the percentage of text about the factor in all the documents.

4.1 Effect of Distribution Medium and Target Audience on Content

Our analysis of reviews indicates that the distribution medium and the target audience of the publication are influential in determining the nature of the information included by the reviewers. This can be demonstrated by examining three aspects: product placing, emphasis of the review, and critical tone of the review.

4.1.1 Product Placing

Reviews were coded according to whether they were primarily about a new product or an upgrade to an existing product. In the print category we found that half were specifically about what the reviewer perceived of as a new product and half were about upgrades. In the electronic reviews only one review was about a ''new`` product while about half referred to upgrades from earlier versions and the remainder (6/15) were not specific about the focus of the review. Of these, four were reviews provided by the publishers themselves (where the purpose of the review is to attract purchases from both new and existing customers).

4.1.2 Emphasis of the Review

Each review was categorised as focusing primarily on one of the following: the newness of this type of product, technological aspects, the information content, the quantity of information or multimedia aspects. If there was no clear focus, the review was coded as having no specific emphasis. Most of the reviews (63% print and 47% electronic) had no specific emphasis, perhaps indicating a willingness by the majority of reviewers to present a balanced view of the product to the reader. Information content was the most frequently identified emphasis for print reviews (25%) with the reviews coming mostly from the lifestyle magazine category. The general interest nature of the audience for this type of publication is probably influential in determining the nature of the review. In the electronic category ''newness`` was the primary focus of 5 reviews and of these 4 were from electronic newspapers. The possibility that electronic newspapers are perhaps deemed to be a ''new`` way of viewing newspapers (and hence influencing content) does not really hold true as most of the articles for electronic newspapers also appear in the print medium.

4.1.3 Critical Tone by Media Type

The publication medium and type of publication in which the review appears is also likely to affect the nature of the review. Of the reviews in this study 94% contain some evaluation and of these, 61% of the text units are evaluative. However, 4 electronic reviews could be classified as being generally descriptive. These are the sites run by electronic publishers. It is not really surprising that these reviews are predominantly descriptive as the intended purpose is to encourage readers to purchase a particular product rather than to give a balanced view. Two of the print reviews can also be classified as being generally descriptive: an advert in a specialist journal and a review of a specialist CD-ROM on rugby in a lifestyle magazine. The advert is similar to the electronic publisher reviews and therefore likely to reflect the intended purpose. The review of the specialist CD-ROM was one of the smallest reviews in the sample and thus there was little space for the reviewer to develop any critical analysis.

The majority of reviews (73% of electronic reviews and 87% of print reviews) are evaluative rather than descriptive in overall tone. The critical tone (positive, negative or neutral) of the evaluative components of print and electronic reviews is shown in [Fig. 1] and [Fig. 2] respectively.

The lifestyle magazines are the most positive in approach, probably reflecting the overall tone of this type of magazine and matching the expectations of the intended audience. An example of the type of positive comment found in a lifestyle magazine is given below.

''Whether you need to beef up a school geography project, or want to get behind the news on Bosnia, Encarta 96 is a powerful information source. The world tour starts as you click and zoom in on any part of the globe. Behind the scenes there's a huge encyclopaedia of facts about the places and their peoples. The wildlife graphics, cultural anecdotes and clips of world music, in particular, really bring region alive. There's masses of material here on culture, natural resources, and industry.`` [LF_ENCWA]

In information magazines, a different approach is adopted. The commentators take a fairly balanced view giving 36% positive, 24 % negative and 17% neutral comment (amounts of text). Once again the intended audience is probably influential in determining the approach taken with information professionals preferring to be presented with both the positive and negative aspects of products which they might consider purchasing for their library or organisation. Taking a directly opposite view, multimedia publishers in both the print and electronic medium make relatively few evaluative comments, and the comments which are made are all positive. As indicated previously this is hardly surprising as it would not be beneficial to be critical of your own products. In general there is a tendency towards more positive comment in the reviews appearing in the print medium when compared with electronic reviews.

4.2 Evaluation Criteria in Relation to Distribution Medium

Multimedia information is the criterion most frequently referred to by both print and electronic reviewers. The amount of text retrieved about multimedia from these documents is also fairly substantial with 9.1% and 11% respectively for print and electronic documents [see Tab. 2].

Nearly all reviewers see multimedia as an important criterion to consider when examining information CD-ROMs, with 100% of print reviewers and all but 2 of the electronic reviewers making some comment about this aspect. Many of the mentions are very brief - often just quoting figures for the number of pictures, videos, or graphics included. However, some of the reviews are more qualitative, for example:

''One nice feature is the module which lets you build your multi-media presentations from the contents of the Encyclopaedia. So if you are interested in, say, space travel, you can piece together articles, pictures, movies and tables. The learning tool becomes a teaching tool but this is also a nice introduction to multimedia segmenting for those new to the idea.`` [PPCW_COM]

Access to information (frequently referred to as browsing or searching) is also significant in print reviews with just over 80% of print reviewers allocating 19% of the text written by them to this topic. However this is pushed into third place in terms of frequency of comment by reviewers in the electronic medium (17% of the text) where access to additional information via the Internet is given the largest space in the reviews (22%). The very fact that these reviews are appearing on the WWW means that the readers are going to be familiar with the Internet and indeed are likely to be seeking information on products which can enhance their use of the available information. All but 2 of the electronic reviews contained some information about Internet links. Some of the comments on access to information relate specifically to the links between the CD-ROM and the Internet. For example:

''Click on the more information button, while perusing any article, and web links related to that topic are displayed. Related topics from within Encarta 97 are also itemised, as are those that can be obtained from outside sources.`` [NGG_GROL]

The print reviewers are much less forthcoming in their commentary on the value of Internet access with only half of the reviewers making any reference to its significance (4.5% of text) and of these 62% were reviews within computer magazines. Once again the intended audience of the review appears to have an effect on the evaluation criteria discusses by the reviewer.

Interactivity has the second highest score, in terms of text allocation (16%), for print reviews, but it should be noted that a fairly limited number of reviewers consider it to be worthy of attention (5 or just over 31% of all print reviews). The electronic reviewers rank this down in 4^th place and once again this is limited to text from only 40% of all electronic reviews. Thus not all reviewers have the same perception of the importance of interacting with the information resource. Most of the comments on the interactivities are fairly brief, as in the examples:

''The Kaleidoscope is supposed to include a game where you guess what a particular picture is, but every picture is captioned, telling you the answer. This hasn't been properly thought out.`` [PPCP_COM]

''Used in an open-learning situation in a school library this could be the latest addictive game!`` [SSL_ENC]

Brevity does not, however, prevent them being critical!

At the more detailed level, we examined the nature and amount of discussion about different multimedia elements. All 4 of the main multimedia types (animation, audio, pictures, and video) are well represented in the amount of coverage offered by the reviewers. Pictures are discussed in 80% of the reviews, and within these reviews text relating to pictures accounted for 5% of the text (in both print and electronic reviews). Animation is discussed by 31% of the print reviewers (1.4% of text) and 53% of the electronic reviewers (3.0% of text). The larger proportion of text being allocated to this topic in the electronic distribution medium is mainly attributable to 2 of the publishers allocating larger proportions of the review to discussion of animation (7.1% and 10%). Regarding the nature of the discussion, most is fairly superficial and often contains generalised information about the quantity of information in the product:

''There are also hundreds of new photographs, video clips, maps, animations and other illustrations.`` [NG_ENC]

However some reviews do offer more critical evaluation such as this from the Electronic Telegraph:

''While some of the video is fascinating, such as a time-lapse sequence of a butterfly emerging from its chrysalis, some is present only to "nationalise" the disc, such as footage of the Coronation.`` [NT_ENC]

5 Conclusion

The difficulty of writing good, balanced reviews in 300-500 words is not a problem restricted to the domain of the software review. Weinrach discusses this issue in relation to the book reviewing process and suggests that reviewer bias is not unknown [Weinrach 1988]. Additionally, he suggests that in some cases the reviewer may only have limited experience in the review subject. Is this the case in reviews of CD-ROM reference products? Who selects the reviewers of information products? Are they self selected? Do reviews submitted to a journal, newspaper, or other publication ever get refused on the basis that they are of an inadequate quality? Further research in this area would be beneficial in taking our understanding of the value of reviews forward.

Stevens gives guidance to reviewers of printed reference books and suggests that, in addition to using his list of criteria, the reviewer should also examine the book within the context of the potential users of the reference work [Stevens 1986]. There is some evidence from our work in evaluating reviews of CD-ROMs that reviewers are aware of the context in which their readers might be using the product and that they do try to

provide a review which will assist the reader in choosing an appropriate product. There is also some evidence that the criteria used by reviewers does match the evaluation criteria used by home information users. However, it is also evident that the medium in which the review is published (electronic or print) can affect the content of reviews to such an extent that valuable information may be excluded (eg Internet access) or sidelined.

References

[Bosch & Hancock-Beaulieu 1995] Bosch, V.M. & Hancock-Beaulieu M.: CD-ROM user interface evaluation: The appropriateness of GUIs, Online and CD-ROM Review, 19, 5, (1995), 255-270.

[Buckner and Gillham 1997] Buckner K. and Gillham M.: What Use Are Reviews? An Analysis of Multimedia Information Software Reviews and Their Relevance to Home Information Consumers, in Proceedings of Ed-Media `97, AACE Publishing, Calgary, (1997),118-123.

[Clements and Nicholls 1995 ] Clements J. and Nicholls P.: A comparative survey of multimedia CD-ROM Encyclopaedias. Computers in Libraries, 15, 8 (1995) 53-59.

[Gillham and Buckner 1996] Gillham, M. and Buckner, K.: Interactive multimedia information: Evaluation by home users, Proceedings of Ed-Media `96, AACE Publishing, Boston, MA. (1996), 264-269.

[Gillham et al.1995] Gillham, M. Kemp, R. and Buckner, K.: Evaluating interactive multimedia products for the home. The New Review of Hypermedia and Multimedia, 1, (1995), 199-212.

[Harry & Oppenheim 1993a], Harry, V. & Oppenheim, C.: Evaluations of electronic databases, part I: Criteria for testing CD-ROM products. Online and CD-ROM Review, 17, 4, (1993) 211- 222.

[Harry & Oppenheim 1993b], Harry, V. & Oppenheim, C.: Evaluations of electronic databases, part II: Testing CD-ROM products. Online and CD-ROM Review, 17, 6, (1993), 339 --347.

[Kobrin 1988] Kobrin B.: How to judge a book by its cover: and nine other clues, School Library Journal, 35, 2 (1988) 42-3

[LF_ENCWA] LF_ENCWA Susan Aldridge, Encarta 96 World Atlas, Focus, April 1996

[NG_ENC] NG_ENC The Guardian (CD_ROM) Encarta, extracted from a longer article by Schofield J, Multimedia:Knowledge that knows no bounds 16 November 1995

[NGG_GROL] NGG_GROL Philip Clayton, Encarta 97, Geraldton Guardian, http://bilby.wn.com.au/gol/computers/software-scene/Enc97.htm

[NT_ENC] NT_ENC Partridge C. CD-ROM: The facts that are always up to date, The Electronic Telegraph: Innovations, Wednesday 8 November 1995 date accessed: 7 January 1998 http://www.telegraph.co.uk:80/et?ac=000154468633552&rtmo=32d50201&atmo=32 d50201&pg=/et/95/11/8/inoenc08.html,

[PPCP_COM] PPCP_COM Sue Taylor, Compton's Interactive Encyclopaedia 96, PC Plus, March 1996 p33

[PPCW_COM] PPCW_COM Compton's Interactive Encyclopaedia 1996 PCWorld, Vol 19 No 2, pp 274 February 1996

[Q.S.R. NUD.IST 1997] Q.S.R. NUD.IST, http://www.qsr.com.au/Nudist- Software/nudist-description.html (1997).

[Rettig 1996] Rettig J.: Beyond Cool: analog models for reviewing digital sources, Online, 20, 5, (1996), 52-64.

[Rowley 1995] Rowley J.:, Human/computer interface design in Windows-based CD- ROMs: an early review, Journal of Librarianship and information science, 27, 2, (1995), 77-87.

[SSL_ENC] SSL_ENC Microsoft's Encarta, School Library 2000, 1, September 1993, pp6-7

[Stevens 1986] Stevens N.: Evaluating reference books in theory and practice, Reference Librarian, 15, (1986) 9-19

[Weinrach 1986] Weinrach S. G. Reviewing the review process, Journal of College Student Development, 29, 2 (1988) 175-180.

[Zink 1991] Zink, S. D.: Toward more critical reviewing and analysis of CD-ROM user software interfaces. CD-ROM professional, 4, 1, (1991), 16.