Foundations of MIRACLE:
Multimedia Information Repository,
A Computer-supported Language Effort

Abstract: Research in neurosciences, cognitive psychology and media sciences indicates that "visual thinking" carries a potential of the human mind that is generally still neglected today but could heavily be fostered by novel types of communicating and archiving information. Computer technology (information systems, telecommunication and visual tools) in turn promises to provide a wide range of highly effective tools to support visual, dynamic communication. MIRACLE treads new paths to address a crucial issue: In what way and to what extent can and should current and future systems support new ways of communicating and archiving information using dynamic, visual information?

This paper gives a survey of the numerous attempts that have been made so far to overcome language barriers by introducing artificial languages (both on a spoken/text and on a visual basis). It also analyzes the general status of technology (computer hardware and software) to support such efforts as well as a number of specific projects. From this overview we draw the conclusion that computer-based systems designed to support communicating and archiving dynamic visual information should focus on the following features:

• Support dynamic language elements on a structural level in addition to traditional animated icons;
• Incorporate gestural language elements (inspired by natural sign languages) anticipating future ways of human-computer interaction;
• Allow evolutionary development of the language in a group-dynamic and interactive process involving large international groups of participants.

In a final section we give a brief outline of the cluster of specific projects carried out under the heading of MIRACLE.

Keywords: visual languages, visual communication, constructed languages, computer-supported communication, information archiving, information retrieval, language independent communication, converging technology, multimedia

Categories: A.1, H.3.7, H.4.3, H.5.1, J.4, J.5

1 Introduction

The search for the perfect language has been a companion of the cultural development of mankind for ages ([Eco97]). The question is, can our so-called "information age", with its omni-present video, computer and Internet infrastructure and the new ways of communication made possible by them, provide any new solution to this issue?

1.1 Two Streams merge: Humanities & Technology

Research in humanities and social sciences (psychology, linguistics, philosophy) indicates that people using visual information more than today would be more creative, exploit better the power of the human mind and probably could communicate across language-borders in a more intuitive way than is possible today with conventional text and natural language. Computer technology (information systems, telecommunication, visual tools) in turn promises to provide a wide range of highly effective tools to support visual, dynamic communication. MIRACLE combines these two streams of research and development to address a crucial issue: In what way and to what extent can and should current and future systems support new ways of communicating and archiving information using dynamic, visual information?

1.2 Our Multimedia World Requires and Produces New Ways of Thinking

Neglecting the steady stream of audio and video that surrounds people in everyday life, we can say that information today is still mostly stored using a combination of text and static images such as pictures, diagrams, maps, etc. One reason for this is that a large portion of the information is still tied to paper. The dream of the paperless office is still far from reality. In fact, more printed material is used today than ever before. The long tradition of paper-based writing and reading is not to be neglected thoughtlessly when trying to introduce new communication technologies (see [SH01], [TR00]). Robert E. Horn, author of the book "Global Communication for the 21st Century" ([Hor98]), argues that our "multimedia way of living" will show far-reaching consequences: changes in the very manner in which we will communicate in the future will occur. Therefore he introduces the basic concept of visual language and its global use, pointing out that such a language appears already to be emerging as an international auxiliary language - it can facilitate communication between people with different linguistic backgrounds and help to produce distance learning, teleconferencing etc.

1.3 Psychology, Neuroscience & Philosophy

But the issue is not just one of technology. Reaching much deeper it raises also the question of the nature of thought. Lev S. Vygotsky pointed out that language and thought are distinct functions. In [Vy34] he showed that thought and speech develop differently and function separately and that there is no constant correlation between them (although there is a close correspondence between them). Language is also a tool to organize thinking because it bears the concepts. The Sapir-Whorf hypothesis ([Sa29], [Wh40]) popular in the 1950s theorized that thoughts and perception are determined (or are at least influenced) by language. A dynamic visual language could bring more visual awareness so that people's thoughts could be to some degree determined by the categories made available to them by such a language.

If it is true that differences among languages cause differences in the thoughts of their speakers - this is what is claimed by "linguistic relativism" - then the use of dynamic visual languages would cause differences in thoughts by the people using them. Hence, a dynamic visual language could lead us to a different kind of 'more dynamic visual thinking', whatever that may be! However, the concept of "linguistic relativism" has neither been completely disputed nor definitely established to this day. Allowing to easily draw "politically" and factually incorrect conclusions, it continues to stir controversy.

Recent work in cognitive neuroscience has shown that thinking is a collation of sensory experiences combined through the exchange of neurological signals by the brain ([So94]). Thinking is a process that takes place without linguistic labels and involves exploration, abstraction, analysis and synthesis, completion, correction, and comparison. There is considerable evidence that both perceptual (externally based) and mental (internally based) images are essential for these processes, as well as for memory ([Pa96]). Thus, images are central to our thought processes ("and play a key role in normal perception", according to Stephan M. Kosslyn in [Kos94]) as well as for memory and visual thinking that is closely connected to verbal thought. A fundamental book arguing for the pictorial nature of thought is "Visual Thinking" by Rudolf Arnheim ([Ar70]).

Also Antonio Damasio, a leading neurophysiologist writes in his book "Descartes' Error" ([Da94]) that it is in the form of images that the factual knowledge required for reasoning and decision-making is present in our minds. Images are not stored like facsimile pictures; recalling an image involves some sort of re-interpretation, so new versions of the original evolve over time. According to Damasio images form the main content of our thoughts.

Throughout the twentieth century the view that visual images play a substantial role in rational thought, and that pictures are important as vehicle of information was a minority position in philosophy. However a few thinkers defended it, like Bertrand Russell in ([Ru19]) and Henry H. Price, who wrote in [Pr53]: "The whole of our higher education is directed to the encouragement of verbal thinking and the discouragement of image thinking. Let us hope that our successors will be wiser, and will encourage both". Also, Wittgenstein who had drawn more than 1400 sketches and diagrams himself, wrote in "Philosophical Grammar" ([Wi74]): "Thinking is quite comparable to the drawing of pictures" and "for the picture to tell me something it isn't essential that words should occur to me while I am looking at it; because the picture should be a more direct language."

1.4 New Types of Languages for New Types of Communication

Among other problems, structural difference has always been a particular obstacle when it comes to communicating across borders of natural languages. History has produced a number of natural "universal languages" such as the historic Lingua Franca (based on Latin) or various forms of Pidgin and Creole languages in reaction to this problem. They emerged out of multilingual communities and featured a simplified mixture of a number of linguistic components. Their main aim was to foster communication between different cultures in limited areas of interests (such as trade, diplomacy and seafaring).

Among the desired key features of such "universal languages" always are simplicity and regularity. In modern times there have been a large number of attempts to overcome language barriers by introducing artificial languages, but again, neither of them (particularly not Esperanto, their best known representative) can be called a global success.

In addition to such efforts made on entirely linguistic grounds, attempts to represent words or even complex ideas by graphic symbols go back at least to the ancient hieroglyphs. It is known since ancient times that pictures and images often have a greater potential than words. In the 13th century Saint Bonaventure argued that illiterate might learn from pictures as if from books. This fact was extensively used to communicate religious messages in the churches. Thus, there is much evidence that thinking in images, understanding in images and having feelings in connection with images and even communicating in images is more basic to human nature than thinking and communicating in words ([Ny02]).

In a modern context such ideas were reintroduced by Otto Neurath in the 1930s with his concept of Isotypes ([Neu91], [HB02]) which have been successful to some extent. After all, certain signs like the ones for "airport" or "filling station" or "departure area" (at an airport) are understood world-wide, independent of the knowledge of any particular natural language (but requiring a certain cultural affiliation). However, Isotypes alone have not been successful as universal, visual, international language. Since Neurath's time there have been a large number of ambitious attempts in that direction (one of the best known being Blissymbolics) but again, a large scale break-through of any of these is still not in sight.

In contrast to this, visual communication systems incorporating the human body (or some formal way to represent it) have been successfully developed for special application areas. Signs used by referees in sports, notations for dance and the divers' language are accepted internationally by the respective user groups. Finally, natural sign languages of the deaf are fully recognized linguistically and powerful enough to discuss by visual means anything that can be said in words, fully exploiting the potential of space and dynamics ([Sto], [Arm], [BC80], [BP95]).

1.5 Computer Support Unfolds the Full Potential

MIRACLE is going to use findings and results of all the approaches mentioned above, but will go beyond past attempts. For the first time, the power of computer systems will be fully exploited to ease information and knowledge transfer by making it more efficient and to gain a high degree of independence from natural languages.

One of the underlying assumptions of MIRACLE is that by the beginning of the next decade computers allowing the presentation of high resolution moving images will be as ubiquitous as today's mobile phones. This will for the first time allow to implement a novel concept of communication systems that employ modern multimedia concepts (animations, movies, interactive pictures, dynamic maps, etc.) in combination with dynamic visual languages.

Today's global telecommunication infrastructure including the Internet and mobile phones networks has produced a range of novel communication phenomena from extensively used abbreviations to simple text-based smileys to sophisticated graphic chat platforms that involve avatars acting in comic inspired scenarios. Social activities in such "virtual communities", including the development and use of specific forms of communication, are driven by group-dynamic processes ([Rh00]) are fascinating and can be studied from various perspectives.

In particular, the highly dynamic evolution of certain aspects of language (written as well as visual) is relevant in this context as it suggests that communities based on such de-centralized, wide-spread media may have a tendency to foster rapid, self-controlled development. MIRACLE will investigate such assumptions and benefit from corresponding findings.

Summarizing, MIRACLE will exploit the potentials of ubiquitous computers to develop new forms of dynamic, visual communication platforms which will allow the creation, exchange and archiving of information in a more complete (in terms of the number of cognitive channels addressed), intuitive, cross-cultural and language independent way.

2 Status of Research

In this section we present an overview of the state of the art in a number of areas relevant for MIRACLE. This includes artificial languages, visual languages (static and dynamic), and sign languages. Due to the wide range of relevant subject areas, we can only scratch the surface, but nevertheless hope to be able to give a broad overview.

2.1 Natural Language Understanding

Since the early 1970s comprehensive computer-supported efforts have been made to understand natural language and knowledge, usually under the heading of Artificial Intelligence. Formal frameworks have been set up attempting to map language elements to formal representations that can be handled by software. The Conceptual Dependency (CD) model developed by Roger Schank in the early 1970s ([Sch72], [Sch73]) is one of those theories that has been used in this respect by visual languages approaches, including CD-Icon, Minspeak, and VIL (see below). The core framework of CD (which has seen much development since its early incarnation) is a stratified linguistic system consisting of a network of language-free dependent concepts, where a concept may be considered to be an unambiguous "word-sense". There are two basic axioms in the CD theory:

1. For any two sentences that are identical in meaning, regardless of language, there should be only one representation.
2. Any information in a sentence that is implicit must be made explicit in the representation of the meaning of that sentence.

In CD, sentences are represented in a series of diagrams depicting actions using both abstract and real physical situations. Figure 1 gives an example of such a diagram for the sentence "Bill shot Bob". Note how the event of Bob's shooting is represented: his health parameter goes down to the minimum value -10, which means he died (the p stands for past tense). The physical event of the shooting is dissected into the components "bullet" and "gun" with the primitive action (ACT) "propel" as the connecting relation.

A disadvantage of CD is that knowledge must be decomposed down to a fairly low level. Despite the fact that with the mechanism of expandable macros low-level concepts could be grouped to form higher-level, more abstract entities, such excessive dissection seems to be rather counter-intuitive.

2.2 Neurolinguistics, Cognitive Sciences and Psycholinguistics

In the field of interaction between language and humans, psycholinguistics is the science dealing with the acquisition, comprehension, storage, and production of language, whereas neurolinguistics is the study of the neural and electrochemical bases of language development and use.

As MIRACLE addresses both the auditory and the visual channels and mechanisms of human perception it must also take into account the current scientific knowledge on how humans perceive, process and understand visual information, and, in particular, movement. Furthermore this relation of neuro- and psycholinguistic mechanisms of human language processing and the perception of images should be investigated. Sign languages could play a key role in this context, since they employ linguistic concepts but are physically based on visual perception as opposed to auditory perception. How does the brain extract linguistic meaning out of visual information? How do brain centers for aural and visual perception and processing cooperate in this task?

It is obvious that the vast range of all these issues cannot be handled in a single project such as MIRACLE. However, many if not all these issues have already been raised in other contexts and various fields of science could make valuable contributions to their development. Thus, at the current stage of MIRACLE we hope to attract the interest of experts willing to cooperate and contribute to the project in order to follow the right directions and evaluate properly the results the project will show. Being a multidisciplinary effort, the project has the potential to reveal a large number of interesting and stimulating findings in a variety of disciplines.

2.3 Artificial Languages

MIRACLE can learn a lot from the development of artificial languages. In developing an artificial language the most obvious idea is, first of all, to base the language on a simple grammar with no irregularities and a set of words easy to remember and easy to pronounce. In the context of MIRACLE the aspect of pronunciation is of subordinate importance because the project focuses on information stored visually and will fall back to spoken language only on a very low level, for instance to let the system speak isolated words.

A further potential step to reduce the amount one has to learn is to eliminate all kinds of inflection, i.e. to use an "isolating" language, where each word is used in one form only, with attributes replacing inflexion. The statement "They went" might end up grammatically as 'I (third person plural) go (past)'. One can also use attributes to reduce the number of words, thus ending up e.g. with 'person(female)' instead of 'woman', 'person(male)' instead of 'man', 'person(young+female)' instead of 'girl', etc. This principle can also be used to convert nouns into verbs by adding action attributes, i.e. 'leg(action)' means 'go' or 'walk', 'mouth(action)' means 'talking', etc.

Artificial languages or planned languages are consciously created according to definite criteria of an individual or group of individuals for some specific purpose (such as making international communication easier). Thus they are distinguished more or less sharply from natural languages that have evolved during long periods of time, and usually have accumulated a number of irregularities in the course of this development. But not all artificial languages have been constructed with the aim to find a tool for easier communication of the world's population. Some artificial languages have been invented to facilitate the communication between humans and computers (programming languages) or to represent the scope of some scientific field easily (e.g. the chemical formulae-language or the mathematical or logical languages as invented since the 16th century by philosophers like Francis Bacon, Jan A. Komensky (Comenius), René Descartes and Gottfried Wilhelm Leibnitz). Attempts were also made by Rudolf Carnap and his colleagues of the Vienna Circle to formalize the language of science according to the ideal of physics in order to eliminate all (so-called) metaphysical and (possibly) meaningless words from it ([Ca34]).

Other languages again have been invented as elements of fictional works of literature (e.g. Klingon, or the languages invented by J.R.R. Tolkien). Some languages have been constructed for plain scientific purposes to experiment with and to support certain (socio)linguistic theories (Lojban started that way!). And there are even invented "stealth" languages whose purpose is in fact a restricted use in a small, selected group of people. A fascinating example of such a secret language is Damin, a special language used by the Australian tribe of the Lardil for ceremonial purposes only open to the male population in a specific stage of initiation. According to Steven Pinker ([Pi94]) it has a unique 200-word vocabulary that is learnable in a day but can express the full range of concepts in everyday speech.

In the context of MIRACLE the most interesting constructed languages are those whose purpose is to ease communication across language borders. Such languages are called auxiliary languages, international languages, global languages, universal languages, interlanguages or interlingua.

Auxiliary languages appeared from the middle of the 19th century. Leibnitz pointed out the main necessary qualities for a global language: clear, simple and easy to learn. Another desired quality of a truly global language should be neutrality with respect to gender and linguistic, cultural and political background. Ideally no speaker or hearer of the international language should have the feeling of being culturally hegemonized. Only very few - if any - constructed languages have achieved this goal so far. Perhaps with a visually based language these goals could easier be achieved. However, at the moment this is merely speculative hope. In what follows we briefly present a few of the best known constructed languages that seem most interesting in the current context (for references see [ConLang]):

• Esperanto was introduced in 1887. Its vocabulary is formed by adding various affixes to individual roots and is clearly based on Indo-European languages. The grammar, too, is based on that of European languages but is greatly simplified, regular and flexible. Esperanto has a phonetic spelling and uses the Roman alphabet, each letter standing for only one sound. Sample: Mi revans pri lando kie Esperanto paroligas. (I dream about a land where Esperanto is spoken)
• Ido, a revision of Esperanto, was introduced in 1907 and quickly emerged as a significant improvement of Esperanto. Although it was simpler, yet more precise and powerful than Esperanto, it still failed to replace Esperanto. Sample: Lernar Lingui nacionala esas desfacila. (Learning national languages is difficult)
• Interlingua was created in 1951 by the International Auxiliary Language Association. It is derived from English and the Romance languages in both grammar and vocabulary, with a minimal grammar. It has been used at medical and scientific meetings. Sample: Le unitate del civilisation occidental corresponde in grande mesura a un unitate linguistic. (The unity of Western civilisation corresponds in great measure to a linguistic unity)
• Glosa: Continuing the idea of reducing inflexion as a main obstacle in learning languages, L. Hogben came up with Glosa in 1943 (then called Interglossa; reintroduced 1972 as Glosa by R. Clark and W.Ashby). As an isolating language (i.e. each word is used in one form only, see above) Glosa typologically resembles more an East Asian (especially Chinese) or African language (but English too), although its vocabulary roots stem from the classical languages Latin and Greek. Glosa might be a valuable source of inspiration for MIRACLE. Sample: Anti kali klima na pa resta intra. (In spite of the fine weather we stayed indoors)
• Lojban was founded in 1955 (then called Loglan) by C. Brown with the primary goal to test the Sapir-Whorf hypothesis. It is based on the principles of logic and the hope of removing a large portion of the ambiguity from human communication. Lojban has a root vocabulary of 1300 words, an unambiguous grammar (proven by computer analysis with the software tool YACC), is simple and supposed to be culturally neutral. In Lojban, sentences are formulated as sets of arguments tied together by predicates. Though Lojban was designed as a human language, due to its strictly logical structure, it is very suitable for being processed by computers. On top of the core concepts of predicate logic, Lojban adds non-logical constructs that do not obscure the logical structure, but allow communications that are not amenable to logical analysis. Thus, Lojban has a full set of emotional indicators. Sample: la fits.djerald. fanva zo'e le glico zo'e la .Odisis. (Fitzgerald translates something into English from some language as The Odyssey or The Odyssey is a translation into English by Fitzgerald)

In addition to artificial languages there have also been attempts to simplify existing languages: A special case in this connection is Basic English designed by Charles K. Ogden in 1930 [Og30]. It is different from the constructed languages mentioned here in that it is directly derived from a natural language, namely English. Basic English is correct English restricted to a vocabulary of 850 words (with only 18 verbs!) and a small subset of grammatical rules of the "full" English enhanced by a number of generalizations and with irregularities removed.

Interlinguistics is the study of international linguistic communication from all its aspects, including the roles, structures, and ways of development and application of ethnic and planned languages as international means of communication. This discipline deals with the comparison of languages, the natural process of the blending of languages in multicultural settings (resulting in such languages as Pidgin English or the historic Lingua Franca) and the construction of artificial languages. MIRACLE hopes to gain insight into a number of issues with the help of interlinguistic research, such as:

(a) Mechanisms of "successful" language simplification and trade-off between simplicity and efficiency of constructed languages.
(b) Relation between characteristics of constructed languages and parameters regarding their environment such as the community of speakers (users) and any specific application areas for which the languages might be designed.
(c) Aspects of cultural dependence and largest possible degree of cultural independence.

2.4 Static Visual Languages

The main idea in visual languages is to replace words by graphic symbols. The major aim is to have symbols that are easily understood, to use the combination of symbols to describe attributes, and to create compound symbols to represent new meanings. Symbols representing words can also be converted between functional forms, for instance by surrounding basic neutral word forms with shapes indicating their function (noun/verb/adjective etc) or by attaching specific graphic indicators to achieve a similar effect. Thus, in visual languages possibilities to shift between levels of abstraction should be provided. It is a formidable task to describe the some 40.000 words of a natural language using "few" symbols. However, since Basic English gets by with about 850 English words for most ordinary conversations the belief that even fewer symbols are sufficient is not unfounded (converting nouns into verbs by a single attribute, etc.). However, as Neurath observed, no graphic symbol can be clearly understood without assigning concrete meaning in a learning process.

2.4.1 Hieroglyphs, Pictorial Bibles, Comics, Web Pages, Computer Games ...

Of course visual, symbolic communication is not at all new. In fact in the development of recording media for human communication, drawing and painting came before scripting and writing, as impressively made obvious by all the wonderful cave paintings, hieroglyphs and other samples of pictorial communication found around the globe (see [Hon00] for an inspiring analysis of Maya glyphs in the light of computer icons).

Paintings were used in temples, churches and other public places to communicate political and religious messages to the illiterate who were able to decode the specific iconography. In the Middle Ages up to the 19th Century picture bibles were used for the same purpose and today, of course, the first books for children still are picture books. To represent the flow of time images were presented in a sequenced way from early on, an impressive example being emperor Trajan's column in Rome with its spiralled frieze of about 200 meters length.

In 1827 Swiss Rudolphe Töpffer created the first picture stories as sequences of pictures combined with text, and was soon followed by Wilhelm Busch. By the turn of the 19th century, true comics with balloons (which however have sporadically been used already 200 years earlier) emerged and have developed an amazing wealth of visual communication techniques since then (see [McCl93], [McCl00], [Ei85]). In figure 2 we see a cartoon which we can understand very easily. However, this is not a process as "intuitive" as we might think at first. There are a number of unspoken assumptions we have to make in order to understand the story. First, we must read it as a sequence, picture by picture (which fortunately are clearly separated in this case), from left to write. Then, we have to know how to interpret the gesture of a person extending an arm and the gesture of handshaking which are heavily culture dependent. And finally we have to know how balloons work in comics (a "thought balloon" in this case as opposed to a "speech balloon") and we have to read and interpret the word OK.

Thus, visual communication requires background knowledge as well! Its intuitive directness comes at the price of increased ambiguity. However, ambiguity is not always unwanted, but can be a desirable component of communication under various circumstances. In arts particularly, ambiguity is a key factor.

With the wide availability of interaction and animation on the Internet (and also computer games in general), comics have recently started to experiment with dynamics, interaction, and interlinked structures (hypertext). The Web is thus the playground where all types of currently available media merge: text, graphics, animation, video, audio, all seamlessly integrated and highly networked and interactive. This development reflects a general long-term shift from static paper-based media to dynamic computer-based ones.

2.4.2 Icons, Isotypes, and Pictograms

Viennese Otto Neurath (1882-1945) was, among other things, a pioneer in communication theory where he developed early steps in media literacy and a practical system of iconic communication called Isotypes. The aim of Isotypes was to directly communicate to the masses important facts about their environment and social circumstances. It was mainly about making statistical data visible in a direct way and showing the relations between various sets of data. Figure 3 shows an example depicting the percentage of occupied women, with colour indicating the industry areas.

Neurath's work ([Neu91], [HB02]) had far-reaching implications. It currently enjoys a revived interest which is in partly related to the importance of the type of iconic information that is so essential in modern user-computer interfaces. Today's software is mostly operated through so-called graphical user interfaces which employ such icons as control elements (plus the notion of "windows"). Also, the set of pictograms used at public places around the world (at airports in particular) to guide large international groups of people (see figure 4) are fully in line with Neurath's pioneering practical work and his more theoretical considerations (on issues of modern icons and visual communication see e.g. [Hor98], [YB00], and [CHT02]).

2.4.3 Blissymbolics

Developed by Charles Bliss in 1949 ([Bl49], [Bl84]), Blissymbolics (also called Semantography) contains primarily abstract visual symbols that serve as an alternative to traditional orthography. Figure 5 shows a set of basic Bliss symbols.

Blissymbols uses combinations of symbols to describe attributes, compose simple symbols to represent new meanings, etc. Based on a logical system, visual markers are added to symbols to change syntax and pragmatic functions. For instance, by attaching the simple "action indicator" ^ a symbol that otherwise denotes an object (noun) is turned into a verb. This concept of "visual attributes", when combined with orthogonality, is very interesting in the context of MIRACLE (orthogonality as defined by van Wijngaarden means that features carry across from one context to another; see [vW65]). While many Blissymbols are generally quite abstract, they are at the same time iconic and therefore often easy to understand. This applies even to attributes, for instance an arrow next to a heart symbol could indicate a positive or negative feeling, depending on whether it points up or down. Further indicators are used to denote direction (even in the realm of emotions: an arrow pointing upwards next to a heart means happy), quality, quantity, possession. Variants of the activity indicator are used to denote tense: ⁾ for past and ⁽ for future.

The addition of "strategic symbols" for not, opposite of, part of, similar, without adds further meaning. Figure 6 shows small numbers as "person indicators" whereas figure 7 shows the action (verb), adjective, thing, and plural indicators applied to the basic symbol for mind.

Simply by putting symbols next to each other (without overlapping which would result in derived symbols as shown in figure 8) new meaningful combinations of symbols can be constructed, called compound symbols (in the context of MIRACLE we would rather call them macros). This is shown in figure 9.

Blissymbolics is far from unambiguous or culturally independent. It fully relies on the intuitive power of visual symbolic information and requires a significant amount of intelligence and imagination to interpret the messages.

However, on the basis of its small and simple sets of symbols and rules, Blissymbols is amazingly flexible and can be used to express a wide range of information, from emotional to more technical content. The following samples should give a vague insight into the way Blissymbolics work.

	Did you hear that loud thunder last night?
	Please don't walk on the lawn.
	Weapon inspectors found 11 empty chemical warheads.

Blissymbolics is a simple yet powerful approach and probably is the most widely know of all temporary visual communication systems. For MIRACLE it serves as an important source of inspiration. In particular interesting for MIRACLE are the principles of combining symbols to form macros ("compound symbols") and the more or less orthogonal use of various indicator markers to add meaning.

2.4.4 CD-Icon

CD-Icon developed by Colin Beardon ([Bea92], [BDMY93]) was an early computer-mediated system for person-to-person communication. Its design was based on the theory of Conceptual Dependency (see section 2.1). In CD-Icon, messages were composed as four interconnected screens allowing to specify

• Message type (assertion, question, imperative, negation) and logical combination of components in case of compound message (AND, OR, implication, temporal, or spatial).
• Conceptualization by selecting the primitive ACT in the sense of CD. Icons representing ACT primitives can be animated to self-explain their meaning.
• Pictures as iconic representations of the items in the CD network, selected from a picture lexicon. The icons are accompanied by sets of modifiers (color, size etc.).
• Lexicon: a collection of pictures to chose from.

As one of the earliest projects in that field CD-Icon was very rudimentary and clumsy to use. However it has some significance as one of the first attempts to a formal approach to a computer-supported visual language as a tool for person-to-person communication, and as a forerunner of VIL (see section 2.4.9).

2.4.5 Minspeak

Minspeak ([BB82], [CPOT94], [Cha96], [ACP98]) is a system to support "semantic compaction" of language. It maps concepts on to multi-meaning icon sentences and uses these icon sentences to retrieve messages stored, based on a set of inference rules. The stored messages can be words or word sequences. Designed to support individuals with speech impairments in their communicative efforts, Minspeak has been the basis of a number of well-distributed applications in that field.

In another experimental application the framework was used as an interface to a virtual library (named Bookman). Extended to the multimedia field with so-called tele-action objects ([Cha96]) the system is also able to support dynamics, represented by Petri nets.

Minspeak is formally based on the concepts of Icon Algebra and - again - on Conceptual Dependency. In Icon Algebra, icons are combined according to a set of rules (relations) to derive new meanings for their combinations. In that way a small set of icons can be mapped onto a large number of sentences (meanings). Figure 10 shows the visual sentence "The children drive to school in the morning". If the house icon would be placed in the WHAT column (the context is changed) and the WHERE column would remain empty, the message would translate as "The children study in the morning".

Minspeak is interesting in two aspects. First, it does not try to avoid ambiguity, but considers ambiguity of icons a powerful feature. Second, the actual goal of Minspeak is to produce text out of visual representation, not to communicate through the pictures themselves. In that way, iconic representation of information is considered an intermediate stage. This is in contrast to the aims of MIRACLE.

2.4.6 The Elephant's Memory

The Elephant's Memory ([IH99]) is an artistic approach to a pictorial language consisting of about 150 combinable elements called logograms or signs. The goal of this language was to set up an experimental research environment to explore the field of invented languages, mainly with children. The project introduced experimental semiotics and visual grammars as "mindscapes" to be explored in playful ways, without trying to formally define language elements.

The language exploits the medium of a plain surface, as it can be written and read in all directions. It employs a "logographic compass" to assign meaning to orientation (mainly with respect to time and causality). With the wave form symbolizing time and the logographic compass providing proper orientation pointing from the past to the future, the logographic statements on the left in figure 11 mean the same: "Walking before sleeping".

Temporal aspects can thus be nested in a recursive way visually ("Yesterday I wrote you that next week I will visit you after my trip to Rome"). The same principles work with the causality symbol where the logographic compass provides the reason/consequence direction. The statement on the right in figure 11 can be read in either direction of the cycle, starting at an arbitrary point and its meaning will always be some variation of: "Running makes one tired, being tired makes one sleep which makes one happy, and being happy makes one run." Or, read counter-clockwise: "Sleep is the consequence of fatigue which follows from running which is done because of happiness which is a result of sleep".

Visual statements formulated in The Elephant's Memory are to be read along non-linear, recursive Hypertextual structures, with the grammatical structure based on two parameters: position and size. The language is egocentric in a recursive way as demonstrated by the example in figure 12: The first-person "speaker" is on the left (the head with the big eye) with the scene he or she is watching on the right and the resulting emotion below, both parts clearly connected to the speaker.

An interrogation sign is used in the sense of an attribute which can turn almost any part of a statement into a question about the aspect to which the sign is attached. The logogram on the right of figure 12 thus means: "Who is doing what?"

The Elephant's Memory does not have a written-down syntactic set of rules and semantic rules are only vaguely defined. So, it leaves much room to ambiguity which however, is not considered an obstacle by the designer but an intended feature. The Elephant's Memory was designed with the computer in mind from the beginning. However, no actual system has been developed beyond a rudimentary prototype and the project has been abandoned.

For MIRACLE the most interesting aspects of The Elephant's Memory are its concepts of multidirectional reading and writing combined with the logographic compass, and the general openness for the principle of nesting and recursion. The latter aspect would be an obvious candidate to be supported by a system like MIRACLE.

2.4.7 Rikchik

Rikchik is the language of a hypothetical population of aliens of the same name. Among the large number of amateur efforts in the area of constructed languages documented on the Internet, many of which are inspired by fantasy and science fiction scenarios, Rikchik is one of the most interesting ones ([Mo]). A Rikchik body consists of a one-eyed globe with 49 tentacles, 7 of which are used for communication purposes much in the sense of sign languages. As Rikchiks are entirely deaf and dumb, they have not developed acoustic communication and no language based on speech. While the semantics and contents of Rikchik statements of course are "alien" the structure of the language follows an extreme logic regime.

Four central tentacles are used to sign a basic morpheme, the remaining three tentacles are used in the sense of attributes to denote the properties aspect, relation and collector. The signs produced by the 7 tentacles can directly and unambiguously be denoted using a corresponding "scripting system". The aspect property determines the interpretation of the morpheme itself much in the way of the orthogonal Bliss indicators do. Among the 7 possible values of aspect are idea, action, name, animate and inanimate objects. Relation specifies the type of relation between morphemes (again there are 7 possible values) and the Collector attribute specifies to which of the previously signed morphemes this Relation applies. Formally, a Rikchik statement can thus be noted as a tree of triples, or a sequence of quadruples and is thus perfect input to be processed by computer programs. It thus appears to be a very strict and in the end conservative conception, very much rooted in principles of traditional linguistics and formal languages.

In the following example we see three samples of a Rikchik statement with only slight modifications of the attributes, resulting in other meanings. The morphemes from top down stand for insect, flower and mouth (which means "eating" when the activity aspect is attached). The aspect, relation and collector attributes are found on top of each morpheme and at its lower left and right corners.

One interesting aspect of Rikchik with respect to MIRACLE is that the hypothetic population using it communicates entirely and solely on a visual basis. However, there is one main point of critique connected to this: Rikchik does not involve any kind of dynamics and movement, although one would certainly expect such a feature from a sign language.

2.4.8 CAILS

In CAILS (Computer Assisted Iconic Language System, [Ch00]) graphical icons (representing the vocabulary of Basic English ([Og30]) are placed into simple geometric forms that express grammatical structure. These "grammar containers" are shown in figure 14 and stand for (from left to right): subject, object, origin (ablative), here (dativ), destination, adnominal (genitive), as well as verb past, present and future. The rightmost image is a grammar symbol filled with two icons, making up a "unit of expression" meaning "A book owned by a man".

Such "units of expression" are put into some relation by using special connectors (basically arrows of various types) expressing certain types of conjunction as shown in figure 15 (from left to right: that, if, because, but, and, with).

The small number of rules for building CAILS statements (about a dozen) quite strongly relies on conventional linguistic terms. While the project shows an interesting general approach, the ideas do not appear very well developed and the project apparently was closed in 2000 without resulting in any concrete follow-up activities.

2.4.9 VIL - A Visual Inter Lingua

VIL ([BL99], [Lem01]) is a well-founded effort towards a computer-mediated iconic language with a design based on conclusions drawn from simplified natural languages such as Pidgins and Creols as well as Basic English. Carrying on the approach of CD-Icon, VIL is again formally based on Conceptual Dependency (see section 2.1). Two key features are that it is verb centered, and order independent. The verb has to be selected first and it determines all further attributes (including nouns) that can be specified (some mandatory, some optional).

Figure 16 shows an icon with all its visual elements, as it is displayed by the VIL system. In this case it is a noun (apple) as defined by the type of the border (rectangle). The three small symbols to the right of the apple icon are control elements that visualize further attributes and grammatical parameters of the element, including the position of the element in the hierarchy of the icon lexicon ("iconicon").

Figure 17 shows the main screen of VIL after selecting the verb "put" as a starting point. The verb is in the center and shown on white background which means that the verb entity has already been instantiated. Verb icons can be animated be clicking on them to show the activity directly.

All other surrounding elements are in grey, meaning they have not been defined yet. The elements offered for specification are determined by the verb: different verbs will yield different situations. Clicking on the elements invokes specific screens to instantiate the parameters. Some of them are offered in hierarchical form, others can be selected by pointing on an image (map) or by using a slider (for selecting values from a range continuum such as "cold to hot") or other interactive controls (such as a clock for defining time).

Using a general, limited subject area as test-bed (cooking and nutrition), a thorough evaluation of VIL has been carried out using the Internet as a platform for online questionnaires. However, the system does not provide any facilities to support online communication other than sending VIL statements as email attachments.

VIL demonstrates that complex and accurate visual statements can be composed with an interactive computer tool. However, the system forces the users into quite strict structures of thinking and acting (when composing) that appear counter-intuitive even though they strongly follow linguistic lines (or maybe because they do so; maybe visual thinking follows different lines). The need to learn how to deal with this system seems to be significant. Leeman claims a number of requirements on a visual communication system in his well-researched analysis, but all in all, VIL fails to meet some of them.

2.5 Animated Icons

The concept of animated icons to support understanding of iconic information has been developed and used in a number of projects for more than 10 years ([MY90], [BSM91], [JGM93], [Dor94]). The basic idea is that icons representing activity can be "self-explaining" ([Dor94]) by "acting out" the activity itself in the form of a brief animation. There are various ways users can invoke such animations, for example by clicking on the icons, or by moving the mouse over them. One key difference between such simple forms of animated icons and more sophisticated forms of dynamic effects as described below is that the animation they show is clearly restricted to the icon itself and has no effect on or relationship with other visual components.

Of the projects described in the previous section, CD-Icon, CAILS and VIL incorporate animated icons in this sense described here. Figure 18 shows two rather abstract samples of animated icons in CAILS: Eating is shown as a slice from which step by step small pieces are "bitten off" whereas reading is symbolized by lines appearing from left to right, top down.

In VIL activities such as pour, mix, and put (from the cooking domain) are directly shown as rather concrete little animations. Minspeak allows the definition of dynamic structures that can be formalized as Petri nets. However the actual implication of this concept and to what extent it can be used in the sense of a structurally dynamic language as described in the following section is not quite clear.

2.6 Visual Languages with Structural Dynamics

When talking about dynamic visual languages we mean languages incorporating elements of change and/or movement on a structural level, not only as animated icons as described above. That means that dynamics carry meaning in the linguistic sense, expressing relationships between separate visual items or between different states of the same component over a period of time.

While static visual languages can in principle be used on the basis of paper and pencil, as soon as dynamics is involved, some sort of other medium is required supporting this new dimension. In our case (and opposed to the natural sign languages that use the human body as a medium) we assume some kind of computer supported platform to take on this role.

Efficient perception of movement is essential for any animal relying on vision in order to escape rivals and find prey or mates. Cognitive science is investigating the corresponding mechanisms and comes up with interesting results. For instance, movement of the human body can be very heavily abstracted and will still easily be recognized. If this also applies to parts of the body, in particular to the hands, this would explain the efficiency of sign languages. Such findings could have implications on the design of new dynamic visual languages.

2.6.1 MUSLI

Designed by Lennon and Maurer MUSLI (A Multi-Sensory Language Interface, [ML94], [Ho00], [ML01a], [ML01b])), with one of its important features being dynamics, can be considered a forerunner of MIRACLE. In some respects MUSLI is based on a functional grammar that classifies processes into material, mental, and relational types and consequently deals with doing, sensing, and being. The syntax of the visual language is simple and comprises actors, compound object, and relations. MUSLI statements consist of objects which appear on a "stage" and are subject to events. Also employing the concept of scenes (through which paths can be defined), to system follows a metaphor of theatre play to some degree.

The MUSLI project presented a rudimentary prototype implemented in Macromedia Director in which the basic concepts of orthogonal attributes, dynamics (including change of shape and colour) and abstraction (condensing complex information into "macros" for later expansion) of a computer supported visual language have been demonstrated. The story of Snow White and the Seven Dwarfs was used for demonstrational purposes. The three panels in figure 19 show how the bad queen kills Snow White. In MUSIL this is presented in a very intuitive way involving dynamic effects: a skull (symbolizing death) moves from the queen over Snow White who vanishes away.

Figure 20 shows the hierarchy of properties (symbolized visually) attributed to Snow White from left to right: she is a young, beautiful, royal woman. A woman is a female human, a human is an intelligent living being (the wave denoting intelligence, the circle denoting a living being), and a living being is a thing that lives (the square denoting a thing, the heart denoting that it is living).

Users can select the preferred level of abstraction of the presentation by simply clicking on an object to get a level deeper and to see the "hidden" attributes. This process is called expansion (as opposed to condensing). Also, a number of filters can be applied to scenes to omit the background, to switch between presentation styles (say, Disney to Van Gogh), or to omit living beings or inanimate objects (figure 21).

2.7 Sign languages

Sign languages are the natural languages of the deaf. They use the human body of the sender (mainly hands and face) as the "platform" or medium for communication. As entirely visual languages they employ space and movement as means of expression and thus belong in the same category as the main subject of MIRACLE: dynamic visual languages. There are two reasons why sign languages are to be considered in the framework of a project such as MIRACLE:

• Space, time, and structure: Because sign languages fully exploit time and space as means of visual communication, their overall structure and grammatical and semantic features are highly relevant to MIRACLE. It seems to be a valid and promising approach to analyse in detail the mechanisms of these features to find out to what extent they can be usefully applied in the MIRACLE setting.
• Naturalness: Because sign languages have developed naturally over a long period of time it can be assumed that they reflect certain basic cognitive principles regarding non-textual, dynamic visual communication. It seems wise to investigate these principles in order to derive appropriate guidelines regarding the user interface a platform such as MIRACLE could offer.

In passing we want to point out that signing has been and is also used outside the deaf communities in various contexts. To mention just a few, certain monastic brotherhoods developed simple sign languages to communicate while following their vow of silence ([US87]), divers use a set of standard gestures for communication and in mudras, the tantric dances of India, gestures and posture of fingers are charged with symbolic meaning according to strict rules. Going into more detail concerning the structure of sign languages of the deaf we find that a number of aspects are of particular interest in our context. These are discussed below.

• Iconicity: There are a large amount of signs in all natural sign languages that in some intuitive way directly show or hint at certain aspects of the communicated information. In order for iconic signs to trigger understanding, often rudimentary hints are sufficient.

• Shape: Shape is another key component of sign languages charged with meaning. On the human body there are mainly two parts that can be "shaped" intentionally: face and hands. Of the unlimited set of possible hand forms a small set of about 20 to 30 is used in all sign languages in an abstract way very much in the sense of an alphabet of a spoken language. This basic set in turns contains a subset of about 5 to 10 signs that are the kernel signs which typically can be related to a character/sound of the local spoken alphabet. These hand forms have particularly clear shape so they are easy to distinguish. Figure 22 shows the basic set of German sign languages: A, B, 5, G, C and O. In general, mimics ("face shape") often serve as an attribute strengthening the meaning of signs.

• Location: The location where a sign is carried out is an important component, of course always referring to the human body of the signer. Location can be seen as an attribute.
• Timeline: Generally, time is indicated by using space and movement. A number of virtual "time lines" are used to visually denote different temporal aspects (see figure 23: A for past to future; B for calendar units, sequence and duration; C for continuity and/or duration, and D as "human time" such as growth of a person etc).

• Attributes: Many verbs can be produced with different hand shapes representing persons, animals, or objects. In that way the specific hand shape serves as an attribute. Such "classificators" specify objects by referring to their typical visual characteristics such as size and shape but also to more abstract qualities or to concrete things such as tools. Figure 24, first row, shows "eating an apple" (round shape) whereas the second row means "eating a salt stick" (or something else very small and thin).

• Pointing: Pointing is a key component of sign languages used in a variety of situations. Some signs are used for specifying location (here, there, in/outside etc). Specific index signs allow allocating a person, animal or object to a certain "point in space" (called locus) for later reference. The signer does so by first naming or defining a person or object and then pointing to somewhere in the space in front of him. Later this object can be incorporated into complex signed statements by pointing to the locus again. In MIRACLE this concept corresponds in a natural way to the idea of macros, explained elsewhere. Finally, the role of possessive pronouns of spoken languages (my, your, his/her etc.) is taken by special index signs pointing to the possessing person, object or locus representing it.
• Movement: Sign languages heavily rely on movement and dynamic elements in general. All kinds of signs can include moving components but in particular signs representing activities. Usually, start and end point of the movement represent aspects of the activity such as subject/object, source/destination etc, depending of the nature of the statement.

• Efficiency through Simultaneity: In sign languages many features can occur simultaneously, resulting in "multidimensional" messages of high complexity. This is the reason why sign languages are about as efficient as spoken languages in terms of information transfer rate. Although within a statement signed features occur simultaneously, statements are separated by brief "halt segments" in which all features are "frozen" for a short moment.

• Formality: There are a number of approaches to formal notation of sign language, the best known being Sutton SignWriting (designed to support written communication based on signing, [Sutton]) and HamNoSys (designed for scientific purposes, [Ham]). They might serve as a model or at least as an inspiration on how human related movements can be structured and notated (and thus be stored) on a computer environment. Figure 25 compares signed statements with three ways of the corresponding formal notification.

• Simplicity vs. Context Sensitiveness: Sign languages are very simple compared to most spoken languages in terms of grammar: There is no inflection, gender, tense, case and no singular or plural. However, this comes at a price: Signed statements can be very context sensitive and hence ambiguous. Dependencies and implicit relationships may accumulate in a sequence of signs, so that a concluding sign may be "charged" with a lot of implicit information only to be understood in the context of the preceding signs.

While sign languages are restricted to the human body as communication medium, in MIRACLE the communication medium is computer-based. However, with the wide availability of wearable computers in the future this distinction might become irrelevant to a large extent. Future human-computer interaction facilities might involve new devices replacing current data gloves and data suits. And new tools for visual recognition of facial play and gesture recognition (input) and special glasses that project images directly onto the retina (output) might become available.

Computer systems to support translation between spoken language and sign language - in both directions - are already available in experimental or early commercial versions. See e.g. [Kad95] for an early experimental approach to sign recognition based on data-gloves, and e.g. [HO01] on an approach to visual sign recognition. Speech-to-sign translation is demonstrated in the EU project VISICAST (www.visicast.co.uk) and has been brought to the market by the products such as iCommunicator(tm) (www.myicommunicator.com) and SigningAvatar(tm) (www.vcom3d.com). Iconic gesture-based human-computer interfaces are also being developed ([MW98]).

However it is interesting to note that although all these systems exist, no attempts seem to have been made so far to exploit features of sign languages in others ways than for mere translation. Hardly any efforts have been made to adopt some of them to computer-based platforms to enhance visual communication. The concept of Worldsign/Symbolvision ([Worldsign]) is an exception that has to be mentioned here for thinking up a system of visual communication that can be signed dynamically or depicted as static icons. Although only a vague, incomplete concept, it also recognises the potential of the computer to dynamically visualize signed language elements. The notation system employs so-called "gestographs" which are strongly simplified representations of the human body parts involved in gesturing.

2.8 Conclusions of Background Survey

From the observations made in this section about the state of current research and corresponding projects we can draw a number of important conclusions that serve as the basis for further research in the framework of MIRACLE.

(a) Orthogonal and hierarchic concepts: Most of the well-known constructed written and graphical languages found today share a central set of features which includes principles of orthogonal attributes and recursive structures allowing the hierarchic definition of concepts (called macros or compound elements). However, implementation of such features in computer-based systems has been rather basic and often rudimentary so far.
(b) Limited dynamics: Dynamic elements in visual languages are common and well researched in form of animated icons only, i.e. simple animations that explain the meaning of an icon by "acting out" its meaning (which usually is a verb). However, dynamics in terms of structure are rare in current systems and concepts. The large field of issues opened by the use, meaning and control of movement of symbols (that themselves may be animated) has not been investigated so far.


Page 335

(c) Strong formality: Most serious projects try to define a complete visual language from scratch, with a language structure precisely described in a formal way (some based on Conceptual Dependency). Some of the projects also spend much effort in building large visual vocabularies.
(d) Neglecting social interaction in networked media: None of the projects investigated tries to understand and exploit the social-dynamic processes that determine the development of language on networked media such as the Internet.
(e) Limited multi-disciplinarity: There is currently no comprehensive, multidisciplinary effort towards a holistic approach regarding these issues. Such an approach would require a multidisciplinary team of experts and considerable resources and is expected to take a period of about five years to show profound results.
(f) Limited breadth: None of the projects that involve computer support to address the issues came up with more than quite rudimentary prototypes. Neither comprehensive nor representative experiments were made so far in multiple settings (regarding type of applications and user groups).

3 Technology Potential

MIRACLE is based on realistic assumptions about the further development of computer hardware and software. In this section we lay out some plausible scenarios of future computer systems with respect to the issues of MIRACLE.

3.1 Hardware: Revolutionary New Ways of Interaction

We assume that powerful networked computers not much larger than today's cell phones will be available before the end of this decade as ubiquitous companions. As a consequence, most information might be consumed not by reading from static material (such as paper) but from an interactive, dynamic, mobile device with very high display quality and a wealth of novel functionality. This will not only change the way we consume information, but also the type of information at issue. The exact form such a "portable personal computer" (PPC) will take on over the next five to seven years cannot be predicted now. However, this is irrelevant since what is predictable are the available functions of such a PPC. In what follows we claim 12 facts concerning such PPCs. We argue why they are likely and indicate a few of the consequences.

1. We will be able to do all with PPCs that we are doing today with powerful PCs: Advanced versions of cell phones are starting to offer more and more PC functionality. Conversely, a number of new handhelds run a full Windows operating system and can be used (usually with a small attachment) as powerful cell phone.

Page 336

2. It will act as cell phone: Because of the convergence of cell phone and computer technologies it is irrelevant to distinguish between them when talking about the future.
3. It will have continuous access to the then dominating version of today's Internet: That computers/cell phones will have permanent wireless access to networks is a consequence of emerging new wireless network standards such as UMTS which are not time but volume oriented. Paying for the volume of transmitted data instead of paying for connection time will clearly encourage the use of the Net even for trivial issues.
4. It will have a built in still and video-camera: Multimedia cell phones have already a built in camera today. In the future such cameras will not only be used to send pictures or movies, but for instance to use image processing of gestures as a new powerful input medium, to have the computer analyze the environment (from face recognition to object detection), or to allow the user to look (so to speak) through the camera using its zoom or macro facilities.
5. It will have a built in GPS (the satellite driven Global Positioning System).
6. It will have built in sensors to know the direction of the head of the user: The built-in GPS and direction sensor will allow guiding users in an unprecedented way: not just for routing, but as guide to buildings, scenery etc. Combining this with image processing provides a myriad of new applications. But, most important for MIRACLE, even gestures such as nodding, shaking the head etc. can be recognized by the PPC and can thus act as very simple ways to express commands.
7. It will have a microphone: PPC microphones may be hidden behind a tie, in a tooth, etc., allowing to pick up language even if it is spoken with closed mouth. Together with good speech recognition this provides an excellent way to generate input, quite likely superior to typing on a keyboard.
8. It will deliver stereo sound: PPC "loudspeakers" could be built into the sides of eyeglasses so that they deliver sound directly to the ear-bone, thus inaudible to other persons.
9. It will probably have no hard-disk or other movable storage device: Mechanical parts are error-prone. By the end of this decade memory-sticks with a capacity of 30 G Byte or more will be common and will be a small, simple and robust replacement of hard-disks.
10. It is likely to use new display technology rather than rigid display screens: A host of alternatives to current screen technologies is emerging. Using eyeglasses equipped with small mirrors to produce virtual images seems particularly attractive. But bendable or foldable screens, projections techniques (that work e.g. holographically without screen), digital ink, etc. may offer other still more interesting alternatives.
11. It will probably not have a traditional keyboard, but rather other novel interfaces: The keyboard as main input device will remain clumsy, given the size of our fingers. Alternative input devices can be based on speech recognition, on gesture recognition (following e.g. the movements of the fingers on a virtual keyboard!), on keyboards integrated into fabric, on recognizing a number of different brain states, on movements of the head (as mentioned earlier), on the recognition of some kind of stylus on some surface (today the screen of a tablet PC, tomorrow maybe the sleeve of our jacket with some sensors woven into it).

Page 337

12. It is likely to be powered not by batteries as we now know them: Industry is talking about a growing 'power gap' (see Wall Street Journal: "Gadget Makers Join the Scramble to Zap the 'Power Gap'", Wall Street Journal (01/16/03)). We can therefore assume that an alternative to the current technologies for power supply for mobile devices will be found.

For more information on the subject see also [Ge00] and MIT's Wearable Computing TTT (Things That Think) Lab (www.media.mit.edu/wearables), in particular [DV01] supporting the idea that explicit interaction will be reduced in favour of automatic detection of situation-dependent context. Other projects are going on at Columbia University under the headline of Augmented Reality, in particular the projects MARS and KARMA (see www.cs.columbia.edu/graphics and [Fei02]). For implications of such a device see [MO03].

3.2 Software: Towards an Understanding of the World

According to the saying "to understand language is to understand the world" there is a strong inherent relation between language and our conception of the world. Computer scientists have always dreamt of "intelligent" computer systems able to understand the world, in part through the ability to learn. In the past they came up with the Artificial Intelligence approach to address this issue. However, only limited success has been achieved. In the meantime computer scientists have become more modest. Nevertheless there is a new, more pragmatic approach which seems promising, feasible and solid: ontologies and semantic webs. A note of caution: The terms "ontology" and "semantics" as used here are often only vaguely defined and not always in line with the way they are used in pure philosophical context. They are nevertheless useful in a pragmatic sense.

Very simply put, ontologies are networks of related concepts which can usually be represented by words. In that way, hierarchical structures are built, which can formally be described as acyclic directed graphs (parts of which are trees). As an example, "living being" would be a high-level concept with a number of subordinated concepts, one of which would be "animal" which would in turn contain "human" as a sub-concept. Each concept can be part of many relations, depending on the context. Therefore, the context must be stored together with the relation to allow full interpretation. For instance, in our example the type of the relations seems to follow biological considerations (classification of living beings). So, how would the concepts of "pet" fit into the animal branch and how would "parents", "child" or "friend" fit into the human branch? Apparently, those concepts follow different lines (social ones?).

Ontologies are often formally defined in terms of XML (eXtended Markup Language) which makes them highly interchangeable among networked software systems. A number of major efforts are currently underway in these fields of research, some of which already provide practical solutions for specific areas of application ([Da03]). In their approach they combine computer power with enormous human efforts to maintain the knowledge networks. To name a few of the most relevant projects:

• OpenCyc (www.opencyc.org) is an open source spin-off of a commercial effort. The declared aim of Cyc is "to construct a foundation of basic 'common sense' knowledge - a semantic substratum of terms, rules, and relations - that will enable a variety of knowledge-intensive products and services". A core component is a "knowledge server" providing a very large, multi-contextual knowledge base.
• WordNet, at Princeton University (www.cogsci.princeton.edu/~wn) is a comprehensive network of concepts in English language. Variants for other languages are available (for instance GermaNet maintained by Tübingen University).
• SemanticWeb (www.semanticweb.org) is an initiative focussing a considerable number of individual efforts and providing a joint platform for the corresponding scientific community.
• OntoWeb (www.ontoweb.org) is a project funded by the European Commission, mainly directed towards industry applications of ontologies.

With respect to the specific aims of MIRACLE it can be observed that the hierarchies of concepts defined in ontologies nicely reflect the concepts of abstraction or of condensing/expanding of graphical macros found in many of the visual language approaches. In that way relations between word based concepts and their graphical representation could be established and levels of abstraction could introduced. MIRACLE will investigate ways to exploit ontologies along that line and try to find a practical way to combine a textual ontology with collections of graphical symbols.

4 The Goals of MIRACLE

Following from what has been said in the previous chapters we belief that we are nearing a turning point. We expect that within less than ten years computers providing high resolution colour images in a novel way will be as ubiquitous and small as today's mobile phones, and intelligent software systems will feature higher levels of "understanding" of the world than ever before. As a consequence the kind of information we will deal with can and will change dramatically according to the true, natural requirements of our minds as indicated by psychology and neurosciences. In the centre of future information systems will be a new type of visual information which can dynamically change and with which people can interact in an intuitive way to adapt it to personal circumstances (mother tongue, level of knowledge, preferred presentation styles etc). MIRACLE is one of the first projects to go in that direction and investigate its potential, basically by using interaction together with multimedia elements and the novel notion of dynamic visual language. The ultimate aim of the undertaking is to ease information and knowledge transfer between persons by making such transfer

(a) more efficient (which can mean a range of different things) and
(b) to gain a high degree of independence from natural languages (which may be contradictory to (a) in some respects).

The new kind of information archiving and exchange that is envisioned in MIRACLE will be more intuitive and natural than current approaches, will support a larger variety of different ways of thinking and understanding, and it will foster creativity and exploit additional "cognitive channels" of the human mind (visual, movement, symbolics etc ...). It will allow to make the abstract more concrete, and the concrete more abstract. This will be achieved through alternative views ([Len95]), an important didactic principle. In order to experiment with and validate the concepts of MIRACLE, prototype software systems will be developed focussing on following basic features:

• Visual, orthogonal attributes: Characteristics of objects are depicted as visual attributes. For instance a red cross could symbolize an abstract quality "medical". A house symbol combined with a red cross would consequently stand for a hospital, a person with a cross for a doctor, a plant with a cross for a medical herb etc. The way attributes are assigned to visual items follows the principle of orthogonality which means that attributes can always be combined to build more complex meaning. Unusual or even contradictory combinations of attributes would allow expressing creative thoughts in a poetic way similar to natural languages which are also not restricted to mirror reality.
• Dynamics: Activity and change will be naturally depicted as dynamic symbols moving and/or changing in shape, size or color. MIRACLE will go beyond traditional animated icons. Not only will it be possible to turn nouns into verbs, but to have a continuous range for expressing the "amount" of the movement, which is not possible in spoken language where we only have a discreet set of values (words) for describing. With the example of an animated icon standing for the concept of "walking", a very slow movement would represent very slow walking, fast movement would symbolize running, and tremendously fast movement could be a poetic expression of running "as fast as light". Thus, while in a spoken language we have only discrete states (slow walking, walking, fast walking, slow running, running, etc), in a suitable dynamic visual languages one can express almost poetically different shades in between.
• Property on analogous scales: Property attributes (ideally any item could be turned into an adjective) can be selected from a continuous range of values, for instance by using a slider. This should also be applicable to activity shown in a dynamic way as described above.
• Flexible levels of abstraction (macros): Complex combinations of symbols not understood by the user can be expanded (explained) by the system at will. Conversely, complex representations can be condensed by the user to simple symbols to speed up interpretation, yet can be expanded any time desired.

Page 340

• Language independency: As a visual language, MIRACLE will use words of conventional, spoken languages only as a last resort. However, with the help of proper dictionary programs individual words and phrases can be directly translated from one language to any other fairly well. This requires that disambiguation is done at input time because even on a word level the correlation of meanings in different languages is small. Just take a simple word like 'rich': the German word 'reich' is suitable in some cases ('rich person', 'rich in mineral resources') but not in others ('rich cake').
• Natural animation: Some situations are easily described using animation in a natural way. For instance, for showing a driver how to come from one place to another, the system would include a routing system already available for cars. Other examples of "natural animation" would be the description of how a bird flies, the course of the famous battle at Waterloo, or a weather forecast. MIRACLE will try to support such "natural animation" whenever possible.

An important issue in MIRACLE is to adapt the concept of dynamics to the actual computer environment typically found today, with a computer screen used for output. Here, already at this early stage number of issues can be identified, and more will turn up in the course of the project:

• Efficiency: Activity takes time. Thus, information that incorporates changing or moving visual symbols takes time to be "read" and "understood". Users should be allowed to control the speed of presentation as much as possible but the question remains under what conditions (and perhaps with what kind of content) can communication heavily based on dynamics be efficient in terms of the time/volume ratio.
• Time in terms of space: Aspects of time are very often expressed in terms of space. Think of a conventional analog clock: To interpret it one has to interpret the spatial arrangement of hands and face. A digital clock in turn works in an entirely different way. In our context we could say a digital clock is like conventional text whereas an analog clock corresponds to visualization. Another inspiration comes from the virtual time-lines of the sign languages, mentioned above. MIRACLE will explore ways to represent time in such analog ways of space, exploiting the means the computer is offering.

Page 341

• Activity vs. movement: Animation can be used to depict activity or a change of an object itself. But there is also real movement (in the sense of changing the location of an object) that can be visualized directly. In this case a reference space is necessary. It would also be possible not to move the object itself, but instead move the background.
• Action on a static medium: Movement can even be visualized on a static medium not only by adding an action symbol as described above. Comics have come up with a range of solutions addressing this issue. Although on computer driven environments it might seem obvious to exploit the potentials of dynamics, it may be wise to step back and analyse where it could make sense to incorporate proven mechanisms from conventional media. One reason is that dynamic elements can hardly be parallelized: if more than one changing item at a time is presented the user can hardly perceive the overall meaning. So in order to provide overviews, some mechanism of showing "frozen" dynamics (stills) will be required.

We see that there is much room and need to study the cognitive effort that is required at the users' side to interpret animation and dynamic elements. It will be one focus of MIRACLE to identify and address the basic questions related to these issues and to carry out concrete investigations based on the prototypes that will be developed.

5 The MIRACLE Approach

In this section we describe the main principles that will be followed in the MIRACLE project and outline the overall design of the project and some of its subprojects.

5.1 Multidisciplinarity

MIRACLE is a multi-disciplinary project with a broad spectrum of challenges. It is therefore essential to do a thorough investigation of the current state of the art of the relevant subject areas. As the basis for the definition of and experimentation with a new dynamic visual language MIRACLE will use the results of efforts made in the following areas:

• Interliguistics: Some of the better-known constructed languages will be investigated regarding their value to include parts of their vocabulary and/or structure. Candidates that seem particularly promising are Glosa and Lojban, as well as Basic English.
• Visual languages: The theoretical contributions of Otto Neurath and the concrete system of Blissymbolics will serve as particularly strong inspirations, but also concepts of some of the computer-supported visual languages presented here.
• Dynamic visual languages: The results of the few visual languages projects that incorporated dynamics (in particular MUSLI) will be investigated carefully. Also, dynamic mechanisms in natural sign languages will be analyzed with respects to their usability in the context of MIRACLE.
• Ontologies: The technology of ontology databases will be considered in order to find practical ways to structure hierarchies of graphical symbols.

• Virtual communities: MIRACLE will take into account available findings on social processes in virtual communities, in particular those who refer to communicational behaviour and the development of corresponding language phenomena. They are expected to provide valuable input regarding potential scenarios of a Web-based platform supporting the evolution of a visual, dynamic language.

To properly cover this multitude of issues MIRACLE will take a comprehensive approach, put into action by a multidisciplinary team of experts. Researchers in the appropriate fields are invited to join.

5.2 Requirements, General Conditions, and Principles

As a consequence of section 2.8 (Conclusions of Background Survey) in order to cover new ground, in MIRACLE we focus on the following principles:

(a) Dynamics on a structural level: As laid out before, MIRACLE will go beyond mere animated icons and investigate the potential of dynamics on a structural level.
(b) Gestural components: As part of structural dynamics, gestural components will be included to exploit the potential they show in natural sign languages.
(c) Internet: The Internet will be used as basis for potentially large and diverse user communities to unfold the full power of the new communication medium.
(d) Evolutionary Approach: Based on the Internet as platform, and in contrast to most efforts made in the past, a bottom-up approach will be followed. Only a small kernel of the visual language at the core of MIRACLE will be specified formally. The rest (grammar as well as vocabulary) will unfold in an evolutionary process, as the user communities use the language and form it by doing so. The MIRACLE system will provide strong support in that process of the development of the "self-adapting language".

From these principles a number of specific requirements can be derived regarding the design of the system and the overall set-up of the communication platform that will be the basis of the language evolution:

(a) No explicit learning: We want users to be able to use the system without explicit training, i.e. they should be able to use it in short time just by "learning by doing".
(b) Culture-independent: The system should also be truly culture-independent as much as possible. We are realistic enough to know that this can hardly be achieved fully but believe that a global medium like the Internet might be a good platform for first steps in that direction.
(c) Large user communities: Following from some of the principles and requirements stated above, it is clear that the platform will need to be used by large and diverse user communities. Otherwise no evolution will happen.
(d) Interesting forums: In order to draw and keep attention of large user communities it will be necessary to foster enjoyable environments and provide interesting thematic kernels for the communication. Fun and intuition will be keywords in that context.

6 Summary and Outlook

In this paper we give a survey of the vast background regarding visual (dynamic) languages with particular focus on their suitability regarding new ways of computer-supported visual communication. This investigation indicates that a number of key elements in natural and constructed languages (both spoken/written and signed/visual) can naturally be supported and significantly be enhanced by applying them in computer-supported environments. With respect to future technologies, we expect that this will eventually result in revolutionary new ways of communication that fully exploit the capacity of the human mind to deal with visual, dynamic information.

This paper also accompanies the kick-off to the MIRACLE project, which is a major effort to implement many of the concepts presented here in experimental prototypes. A number of specific subprojects dealing with applications of MIRACLE are underway. Details on these projects will a reported in subsequent publications and at the homepage of the MIRACLE project, http://www.MiracleProject.org.

Acknowledgements

A project called "Foundations of MIRACLE" has been submitted for funding to the FWF Austrian Science Fund (www.fwf.ac.at).

Sources of images are as follows: Fig.1: [Sch73]; fig 2: [Hor98]; fig.3: see text; fig.4: public domain; fig.5-9: [Ho00]; Fig. 10: [Cha96]; fig.11,12: [IH99]; fig.13: [Mo]; fig.14,15,18: [Ch00]; fig.16,17(adapted): [Lem01]; fig.19-21: [Ho00]; fig.22,23: [BP95]; fig.24: [Do02]; fig.25(adapted): [Sutton]; fig.26: [Len95];

References

[ACP98] Albacete P.L., Chang S.K., Polese G.: Iconic Language Design for People with Significant Speech and Multiple Impairments; in Mittel V.O. (ed.): Assistive Technology and AI, Springer LNAI 1458 (1998), pp. 12-32

[Ar70] Arnheim, Rudolf: Visual Thinking. Faber and Faber, London 1970.

[Arm] Armstrong, David F.: Original Signs - Gesture, Sign, and the Sources of Language; Gallaudet University Press

[BB82] Barrow H., Baker B.: Minspeak: A semantic compaction system that makes self expression easier for communicatively disabled individuals. Byte, Vol 7, No 9; 1982, pp. 186-202.

[BC80] Baker, C., Cokely, D.R.: American Sign Language - A teacher's resource text on grammar and culture; Silver Spring; T.J. Publishers

[BDMY93] Beardon, C., Dormann, C., Mealing, S. Yazdani, M.: Talking with pictures - Exploring the Possibilities of Iconic Communication"; Association for Learning Technology Journal, 1, 1, pp. 26-32, 1993.

[Bea92] Beardon, C.: CD-Icon - An Iconic Language Based on Conceptual Dependency, Intelligent Tutoring Media, Vol. 3, No. 4, November, 1992.

[Bl49] Bliss, Charles K.: International Semantography, Semantography Press, Sidney, 1949 reprinted in Semantography (Blissymbolics,. Semantography Press, Sidney, 1965

[Bl84] Bliss, Charles K: The Blissymbols Picture Book. Semantography Press: Sidney, 1984 see also http://home.istar.ca/~bci

[BL99] Becker Lee, Leemans Paul: VIL: A Visual Inter Lingua; Third International Conference on Iconic Communication, Bristol 1999; appeared in [YB00]; see also http://www.cs.wpi.edu/~nemleem/IC99Paper.htm

[BP95] Boyes Braem, Penny: Einführung in die Gebärdensprache und ihre Erforschung; Hamburg, Signum

[Br93] Brennan, Mary: Marking Time in British Sign Language; in Kyle, J, & Woll, B. (eds.) Language in Sign: An International Perspective on Sign Language, London 1983, Croom Helm

[BSM91] Baecker, Ronald ; Small, Ian; Mander, Richard: Bringing Icons to Life; Proceedings of SIGCHI conference on Human Factors in computing Systems, CHI '91, New Orleans 1991, ACM Press; pp. 1-6

[Ca34] Carnap, Rudolf: Logische Syntax der Sprache, Julius Springer-Verlag 1934 (English translation: The Logical Syntax of Language, New York, Humanities, 1937)

[Ch00] Champoux, Bernard; Fujisawa, Katsuki; Inoue, Tomoyoshi; Iwadate, Yuichi: Transmitting Visual Information: Icons Become Words; Proceedings of IEEE Symposium on Information Visualization 2000; pp 244; see also http://www.mic.atr.co.jp/organization/dept3/papers/Cails/cails_paper.html

[Cha96] Extending Visual Languages for Multimedia; IEEE Multimedia Fall 1996 (Vol. 3, No. 3), pp.18-26; see also http://www.cs.pitt.edu/~chang/365/vltaomm.html

[CHT02] Cavigliolo, Oliver; Harris, Ian; Tindall, Bill: Thinking Skills & Eye Q; Network Educational Press, 2002.

[ConLang] Links to Constructed Languages mentioned (the Internet is generally the medium of choice for the communities to foster ongoing development of constructed languages):
Esperanto: http://www.esperanto.net
Ido: http://members.aol.com/idolinguo
Interlingua: http://adam.cheshire.net/~jjbowks/auxiling/_igua.html
Glosa: http://www.glosa.org
Lojban: http://www.lojban.org

[CPOT94] Chang S.K., Polese G., Orefice S., Tucci M.: A Methodology and Interactive Environment for Iconic Language Design; Intl. Journal of Human Computer Interaction, Vol. 14, 1994, pp. 683-716

[Da94] Damasio, Antonio: Descartes' Error - Emotion, Reason, and the Human Brain; New York; Grosset/Putnam, 1994.

[Da03] Davies, John (ed.) et al.: Towards the Semantic Web: Ontology-Driven Knowledge Management; John Wiley 5 Sons 2003

[Do02] Dotter, Franz et al.: Grammatik der Österreichischen GebärdenSprache, Forschungszentrum für Gebärdensprache und Gehörgeschädigtenkommunikation, Klagenfurt 2002

[Dor94] Dormann, Claire: Self-Explaining Icons; Intelligent Tutoring Media, Vol 5, No 2 (1994), pp. 81-85

[DV01] DeVaul, Richard W.; Schwartz, Steven J.; Pentland, Alex "Sandy": MIThril: context-aware computing for daily life; The Media Laboratory, MIT; http://www.media.mit.edu/wearables/mithril/MIThril.pdf

[Eco97] Eco, Umberto: The Search for the Perfect Language; Blackwell Publishers 1997

[Ei85] Eisner, Will: Comics & Sequential Art, Poorhouse Press 1985

[Fei02] Feiner Steven K.: Augmented Reality A New Way of Seeing; Scientific Amercian, April 2002, issue 4/24/02; see also www.sciam.com/article.cfm?articleID=0006378C-CDE1-1CC6-B4A8809EC588EEDF

[Ge00] Gershenfeld, Neil A.: When Things Start to Think; Owl Books 2000

[Ham] HamNoSys (Hamburg Notation System) 4.0 (a draft working paper): http://www.sign-lang.uni-hamburg.de/Projekte/HamNoSys/HNS4.0/englisch/HNS4.pdf

[HB02] Hartmann, Frank; Bauer, Erwin K.: Bildersprache - Otto Neurath - Visualisierungen; Wiener Universitätsverlag 2002

[Ho00] Hollosi, Arno: Alternative Representations and Beyond: A New Proposal for a Multi-Sensory Language Interface (MUSLI); Master's Thesis in Telematics, Graz University of Technology; see also http://www.iicm.edu/thesis/ahollosi

[HO01] Holden Eun-Jung, Owens Robyn: Visual sign language recognition; in Multi-Image Search and Analysis. Klette, R., Huang T., and Gimel'farb, G. (Eds) Lecture Notes in Computer Science 2032, Springer 2001; pp. 270-287

[Hon99] Honeywill, Paul: Visual Language for the World Wide Web; Intellect Books, 1999

[Hon00] Honeywill, Paul: A Comparison Between Maya Hieroglyphs and Computer Icons; AI and Society 2000; see also http://www.w3icons.com/pdf/iconsglyphs.pdf

[Hor98] Horn, Robert E.: Visual Language - Global Communication for the 21st Century, Marco VUPpress, 1998

[IH99] Ingen-Housz, Timothee: The Elephant's Memory - In Search of a Pictorial Language; Learning Technology Review , Spring/Summer 1999, Apple Computer online publication: http://www.apple.com/education/LTReview/spring99/elephant

[JGM93] Jonassen, H., Goldman-Segal R., Maurer, H.: DynamIcons as Dynamic Graphic Interfaces: Interpreting the Meaning of a Visual Representation; Intelligent Tutoring Media, Vol. 6 (3/4); 1996, 149-158; see also http://www.iicm.edu/iicm_papers/dynamicons

[Kad95] Kadous, Waleed: GRASP - Recognition of Australian Sign Language using Instrumented Gloves; University of New South Wales , Australia, Oct 1995; see also: http://www.cse.unsw.edu.au/~waleed/thesis/thesis.html

[Kos94] Kosslyn, Stephan M.: Image and Brain - The resolution of the imagery debate. Cambridge, MA: MIT Press 1994.

[Lem01] Leemans Paul: VIL - A Visual Inter Lingua; dissertation, Worcester Polytechnic Institute; see also http://www.wpi.edu/Pubs/ETD/Available/etd-0424101-005711

[Len95] Lennon, Jennifer: Mental imagery and visualization: From the mind's machine to the computer screen; in H. Maurer, (ed.): Proc. ED-MEDIA'95, AACE, Virginia (1995), 27-32.

[McCl93] McCloud, Scott: Understanding Comics, Harper 1993

[McCl00] McCloud, Scott: Reinventing Comics, Harper 2000

[Me91] Mealing, Stuart: Talking pictures. Intelligent Tutoring Media, Vol 2, No 2, (1991), 63-69.

[ML94] Maurer, Hermann; Lennon, Jennifer: MUSLI -- A MUlti-Sensory Language Interface; Proc. ED-MEDIA' 94 (best paper award), AACE, Virginia (1994), 341-348.

[ML01a] Maurer, Hermann; Lennon, Jennifer: MUSLI: A hypermedia interface for dynamic, interactive and symbolic communication, JNCA Vol 24, 4 (2001), 273-292

[ML01b] Maurer, Hermann; Lennon, Jennifer: Augmenting text and voice conversations with dynamic, interactive abstractions using P2P networking; JNCA Vol 24, 4 (2001), 293-306;

[Mo] Denis Moskowitz' Rikchik homepage at http://suberic.net/~dmm/rikchik/intro.html.

[MO03] Maurer Hermann, Oliver Ron: The future of PCs and implications on society; J.UCS Vol 9, No 4 (2003), to appear

[MW98] Marsh Timothy, Watt Alan: Shape Your Imagination - Iconic Gestural-Based Interaction; IEEE Virtual Reality Annual International Symposium, Atlanta, GA, IEEE Computer Society

[MY90] Mealing, Stuart; Yazdani, Masoud: A Computer-based Iconic Language; Intelligent Tutoring Media, Vol 1, No 3 (1990), pp. 133-143

[Neu91] Neurath, Otto: Gesammelte bildpädagogische Schriften, herausgegeben von Rudolf Haller und Robin Kinross, Vienna; Verlag Hölder-Pichler-Tempsky 1991

[Ny02] Nyíri, Kristóf: Pictorial Meaning and Mobil Communication, Conference 2002, "Communications in the 21st Century"; a German version is available in Nyíri, Kristóf (Ed.): Allzeit zuhanden. Gemeinschaft und Erkenntnis im Mobilzeitalter, Passagen Verlag 2002

[Og30] Ogden, Charles Kay: Basic English - A General Introduction with Rules and Grammar, Treber, London (1930); see also http://www.basiceng.com/

[Pa96] Paivio, Allan: A Theoretical Analysis of the Role of Imagery in Learning and Memory, 1996

[Pi94] Pinker, Steven: The Language Instinct. The New Science of Language and Mind, Penguin, Harmondsworth 1994

[Pr53] Price, Henry H.: Thinking and Experience (1953)

[Rh00] Rheingold, Howard: The Virtual Community - Homesteading on the Electronic Frontier; MIT Press; revised edition 2000; see also http://www.rheingold.com/vc/book

[Ru19] Russell, Bertrand: On Propositions - What They Are and How They Mean (London, 1919)

[Sa29] Sapir, Edward: The Status of Linguistics as a Science"; Language 5:209, 1929

[Sch72] Schank Roger: Conceptual Dependency - A Theory of Natural Language Understanding, Cognitive Psychology, 1972, 3, p. 552-631

[Sch73] Schank Roger: Computer Models of Thought and Language; W. H. Freeman, San Francisco, 1973.

[Schu] Schulmeister, Rolf: ViSiCAST: Virtual Signing, Capture, Animation, Storage and Transmission; http://www.izhd.uni-hamburg.de/pdfs/ViCa.PDF

[SH01] Sellen, Abigail J.; Harper, Richard: The Myth of the Paperless Office; MIT Press

[So94] Solso, Robert L.: Cognition and Vision. London, England and Cambridge, Mass. 1994

[Sto] Stokoe, William C.: Language in Hand - Why Sign Came Before Speech; Gallaudet University Press; http://gupress.gallaudet.edu/bookpage/LIHbookpage.html

[Sutton] Sutton, Valerie: Sutton SignWriting; http://www.signwriting.org/forums/linguistics/ling004.html

[TR00] Tonfoni, Graziella; Richardson, James: Writing As A Visual Art; Scarecrow Press

[US87] Umiker-Sebeok, Jean; Sebeok, Thomas A.: Monastic Sign Languages, 2 Volumes, Berlin: Mouton de Gruyter, 1987.

[vW65] van Wijngaarden, Aad: Orthogonal design and description of a formal language. Technical report, MR 76, Mathematisch Centrum, Amsterdam, 1965

[Vy34] Vygotsky, Lev S.: Thought and Language (1934) reprinted 1962, M.I.T. Press

[Wh40] Whorf, Benjamin Lee: Science and Linguistics (1940), reprinted in: Language, Thought & Reality. Cambridge, MA: MIT Press (1956)

[Wi74] Wittgenstein, Ludwig: Philosophical Grammar (1974) Edited by Rush Rhees. Translated. by Anthony Kenny; Oxford, Blackwell, 1974

[Worldsign] Homepage of Worldsign / Symbolvision by David Orcutt: http://www.slocanlake.com/worldsign/

[Yaz93] Yazdani, Masoud (Ed.): Multilingual Multimedia - Bridging the language barrier with intelligent systems; Intellect Books, 1993

[YB00] Yazdani , Masoud (Ed.), Barker, Philip (Ed.): Iconic Communication; Intellect Books, 2000