Foundations of MIRACLE:
Multimedia Information Repository,
A Computer-supported Language Effort
(Graz University of Technology, Austria
JOANNEUM RESEARCH, Austria
(JOANNEUM RESEARCH, Austria
Daniela G. Camhy
(Graz University, Austria
Abstract: Research in neurosciences, cognitive psychology and
media sciences indicates that "visual thinking" carries a potential
of the human mind that is generally still neglected today but could heavily
be fostered by novel types of communicating and archiving information.
Computer technology (information systems, telecommunication and visual
tools) in turn promises to provide a wide range of highly effective tools
to support visual, dynamic communication. MIRACLE treads new paths to address
a crucial issue: In what way and to what extent can and should current
and future systems support new ways of communicating and archiving information
using dynamic, visual information?
This paper gives a survey of the numerous attempts that have been made
so far to overcome language barriers by introducing artificial languages
(both on a spoken/text and on a visual basis). It also analyzes the general
status of technology (computer hardware and software) to support such efforts
as well as a number of specific projects. From this overview we draw the
conclusion that computer-based systems designed to support communicating
and archiving dynamic visual information should focus on the following
- Support dynamic language elements on a structural level in addition
to traditional animated icons;
- Incorporate gestural language elements (inspired by natural sign languages)
anticipating future ways of human-computer interaction;
- Allow evolutionary development of the language in a group-dynamic and
interactive process involving large international groups of participants.
In a final section we give a brief outline of the cluster of specific
projects carried out under the heading of MIRACLE.
Keywords: visual languages, visual communication, constructed
languages, computer-supported communication, information archiving, information
retrieval, language independent communication, converging technology, multimedia
Categories: A.1, H.3.7, H.4.3, H.5.1, J.4, J.5
The search for the perfect language has been a companion of the cultural
development of mankind for ages ([Eco97]). The question
is, can our so-called "information age", with its omni-present
video, computer and Internet infrastructure and the new ways of communication
made possible by them, provide any new solution to this issue?
1.1 Two Streams merge: Humanities & Technology
Research in humanities and social sciences (psychology, linguistics,
philosophy) indicates that people using visual information more than today
would be more creative, exploit better the power of the human mind and
probably could communicate across language-borders in a more intuitive
way than is possible today with conventional text and natural language.
Computer technology (information systems, telecommunication, visual tools)
in turn promises to provide a wide range of highly effective tools to support
visual, dynamic communication. MIRACLE combines these two streams of research
and development to address a crucial issue: In what way and to what extent
can and should current and future systems support new ways of communicating
and archiving information using dynamic, visual information?
1.2 Our Multimedia World Requires and Produces New Ways of Thinking
Neglecting the steady stream of audio and video that surrounds people
in everyday life, we can say that information today is still mostly stored
using a combination of text and static images such as pictures, diagrams,
maps, etc. One reason for this is that a large portion of the information
is still tied to paper. The dream of the paperless office is still far
from reality. In fact, more printed material is used today than ever before.
The long tradition of paper-based writing and reading is not to be neglected
thoughtlessly when trying to introduce new communication technologies (see
[SH01], [TR00]). Robert E. Horn,
author of the book "Global Communication for the 21st Century"
([Hor98]), argues that our "multimedia way of
living" will show far-reaching consequences: changes in the very manner
in which we will communicate in the future will occur. Therefore he introduces
the basic concept of visual language and its global use, pointing out that
such a language appears already to be emerging as an international auxiliary
language - it can facilitate communication between people with different
linguistic backgrounds and help to produce distance learning, teleconferencing
1.3 Psychology, Neuroscience & Philosophy
But the issue is not just one of technology. Reaching much deeper it
raises also the question of the nature of thought. Lev S. Vygotsky pointed
out that language and thought are distinct functions. In [Vy34]
he showed that thought and speech develop differently and function separately
and that there is no constant correlation between them (although there
is a close correspondence between them). Language is also a tool to organize
thinking because it bears the concepts. The Sapir-Whorf hypothesis ([Sa29],
[Wh40]) popular in the 1950s theorized that thoughts
and perception are determined (or are at least influenced) by language.
A dynamic visual language could bring more visual awareness so that people's
thoughts could be to some degree determined by the categories made available
to them by such a language.
If it is true that differences among languages cause differences in
the thoughts of their speakers - this is what is claimed by "linguistic
relativism" - then the use of dynamic visual languages would cause
differences in thoughts by the people using them. Hence, a dynamic visual
language could lead us to a different kind of 'more dynamic visual thinking',
whatever that may be! However, the concept of "linguistic relativism"
has neither been completely disputed nor definitely established to this
day. Allowing to easily draw "politically" and factually incorrect
conclusions, it continues to stir controversy.
Recent work in cognitive neuroscience has shown that thinking is a collation
of sensory experiences combined through the exchange of neurological signals
by the brain ([So94]). Thinking is a process that takes
place without linguistic labels and involves exploration, abstraction,
analysis and synthesis, completion, correction, and comparison. There is
considerable evidence that both perceptual (externally based) and mental
(internally based) images are essential for these processes, as well as
for memory ([Pa96]). Thus, images are central to our
thought processes ("and play a key role in normal perception",
according to Stephan M. Kosslyn in [Kos94]) as well
as for memory and visual thinking that is closely connected to verbal thought.
A fundamental book arguing for the pictorial nature of thought is "Visual
Thinking" by Rudolf Arnheim ([Ar70]).
Also Antonio Damasio, a leading neurophysiologist writes in his book
"Descartes' Error" ([Da94]) that it is in
the form of images that the factual knowledge required for reasoning and
decision-making is present in our minds. Images are not stored like facsimile
pictures; recalling an image involves some sort of re-interpretation, so
new versions of the original evolve over time. According to Damasio images
form the main content of our thoughts.
Throughout the twentieth century the view that visual images play a
substantial role in rational thought, and that pictures are important as
vehicle of information was a minority position in philosophy. However a
few thinkers defended it, like Bertrand Russell in ([Ru19])
and Henry H. Price, who wrote in [Pr53]: "The
whole of our higher education is directed to the encouragement of verbal
thinking and the discouragement of image thinking. Let us hope that our
successors will be wiser, and will encourage both". Also, Wittgenstein
who had drawn more than 1400 sketches and diagrams himself, wrote in "Philosophical
Grammar" ([Wi74]): "Thinking is quite comparable
to the drawing of pictures" and "for the picture to tell me something
it isn't essential that words should occur to me while I am looking at
it; because the picture should be a more direct language."
1.4 New Types of Languages for New Types of Communication
Among other problems, structural difference has always been a particular
obstacle when it comes to communicating across borders of natural languages.
History has produced a number of natural "universal languages"
such as the historic Lingua Franca (based on Latin) or various forms of
Pidgin and Creole languages in reaction to this problem. They emerged out
of multilingual communities and featured a simplified mixture of a number
of linguistic components. Their main aim was to foster communication between
different cultures in limited areas of interests (such as trade, diplomacy
Among the desired key features of such "universal
languages" always are simplicity and regularity. In modern times there
have been a large number of attempts to overcome language barriers by introducing
artificial languages, but again, neither of them (particularly not Esperanto,
their best known representative) can be called a global success.
In addition to such efforts made on entirely linguistic grounds, attempts
to represent words or even complex ideas by graphic symbols go back at
least to the ancient hieroglyphs. It is known since ancient times that
pictures and images often have a greater potential than words. In the 13th
century Saint Bonaventure argued that illiterate might learn from pictures
as if from books. This fact was extensively used to communicate religious
messages in the churches. Thus, there is much evidence that thinking in
images, understanding in images and having feelings in connection with
images and even communicating in images is more basic to human nature than
thinking and communicating in words ([Ny02]).
In a modern context such ideas were reintroduced by Otto Neurath in
the 1930s with his concept of Isotypes ([Neu91], [HB02])
which have been successful to some extent. After all, certain signs like
the ones for "airport" or "filling station" or "departure
area" (at an airport) are understood world-wide, independent of the
knowledge of any particular natural language (but requiring a certain cultural
affiliation). However, Isotypes alone have not been successful as universal,
visual, international language. Since Neurath's time there have been a
large number of ambitious attempts in that direction (one of the best known
being Blissymbolics) but again, a large scale break-through of any of these
is still not in sight.
In contrast to this, visual communication systems incorporating the
human body (or some formal way to represent it) have been successfully
developed for special application areas. Signs used by referees in sports,
notations for dance and the divers' language are accepted internationally
by the respective user groups. Finally, natural sign languages of the deaf
are fully recognized linguistically and powerful enough to discuss by visual
means anything that can be said in words, fully exploiting the potential
of space and dynamics ([Sto], [Arm],
1.5 Computer Support Unfolds the Full Potential
MIRACLE is going to use findings and results of all the approaches mentioned
above, but will go beyond past attempts. For the first time, the power
of computer systems will be fully exploited to ease information and knowledge
transfer by making it more efficient and to gain a high degree of independence
from natural languages.
One of the underlying assumptions of MIRACLE is that by the beginning
of the next decade computers allowing the presentation of high resolution
moving images will be as ubiquitous as today's mobile phones. This will
for the first time allow to implement a novel concept of communication
systems that employ modern multimedia concepts (animations, movies, interactive
pictures, dynamic maps, etc.) in combination with dynamic visual languages.
Today's global telecommunication infrastructure including the
Internet and mobile phones networks has produced a range of novel
communication phenomena from extensively used abbreviations to simple
text-based smileys to sophisticated graphic chat platforms that
involve avatars acting in comic inspired scenarios. Social activities
in such "virtual communities", including the development and
use of specific forms of communication, are driven by group-dynamic
processes ([Rh00]) are fascinating and can be
studied from various perspectives.
In particular, the highly dynamic evolution of certain aspects of
language (written as well as visual) is relevant in this context as it
suggests that communities based on such de-centralized, wide-spread
media may have a tendency to foster rapid, self-controlled
development. MIRACLE will investigate such assumptions and benefit
from corresponding findings.
Summarizing, MIRACLE will exploit the potentials of ubiquitous computers
to develop new forms of dynamic, visual communication platforms which will
allow the creation, exchange and archiving of information in a more complete
(in terms of the number of cognitive channels addressed), intuitive, cross-cultural
and language independent way.
2 Status of Research
In this section we present an overview of the state of the art in a
number of areas relevant for MIRACLE. This includes artificial languages,
visual languages (static and dynamic), and sign languages. Due to the wide
range of relevant subject areas, we can only scratch the surface, but nevertheless
hope to be able to give a broad overview.
2.1 Natural Language Understanding
Since the early 1970s comprehensive computer-supported efforts have
been made to understand natural language and knowledge, usually under the
heading of Artificial Intelligence. Formal frameworks have been set up
attempting to map language elements to formal representations that can
be handled by software. The Conceptual Dependency (CD) model developed
by Roger Schank in the early 1970s ([Sch72], [Sch73])
is one of those theories that has been used in this respect by visual languages
approaches, including CD-Icon, Minspeak, and VIL (see below). The core
framework of CD (which has seen much development since its early incarnation)
is a stratified linguistic system consisting of a network of language-free
dependent concepts, where a concept may be considered to be an unambiguous
"word-sense". There are two basic axioms in the CD theory:
- For any two sentences that are identical in meaning, regardless of
language, there should be only one representation.
- Any information in a sentence that is implicit must be made explicit
in the representation of the meaning of that sentence.
In CD, sentences are represented in a series of diagrams depicting actions
using both abstract and real physical situations. Figure 1 gives an example
of such a diagram for the sentence "Bill shot Bob". Note how
the event of Bob's shooting is represented: his health parameter goes down
to the minimum value -10, which means he died (the p stands for past tense).
The physical event of the shooting is dissected into the components "bullet"
and "gun" with the primitive action (ACT) "propel"
as the connecting relation.
Figure 1: "Bill shot Bob" as a CD diagram
A disadvantage of CD is that knowledge must be decomposed down to a
fairly low level. Despite the fact that with the mechanism of expandable
macros low-level concepts could be grouped to form higher-level, more abstract
entities, such excessive dissection seems to be rather counter-intuitive.
2.2 Neurolinguistics, Cognitive Sciences and Psycholinguistics
In the field of interaction between language and humans, psycholinguistics
is the science dealing with the acquisition, comprehension, storage, and
production of language, whereas neurolinguistics is the study of the neural
and electrochemical bases of language development and use.
As MIRACLE addresses both the auditory and the visual channels and mechanisms
of human perception it must also take into account the current scientific
knowledge on how humans perceive, process and understand visual information,
and, in particular, movement. Furthermore this relation of neuro- and psycholinguistic
mechanisms of human language processing and the perception of images should
be investigated. Sign languages could play a key role in this context,
since they employ linguistic concepts but are physically based on visual
perception as opposed to auditory perception. How does the brain extract
linguistic meaning out of visual information? How do brain centers for
aural and visual perception and processing cooperate in this task?
It is obvious that the vast range of all these issues cannot be handled
in a single project such as MIRACLE. However, many if not all these issues
have already been raised in other contexts and various fields of science
could make valuable contributions to their development. Thus, at the current
stage of MIRACLE we hope to attract the interest of experts willing to
cooperate and contribute to the project in order to follow the right directions
and evaluate properly the results the project will show. Being a multidisciplinary
effort, the project has the potential to reveal a large number of interesting
and stimulating findings in a variety of disciplines.
2.3 Artificial Languages
MIRACLE can learn a lot from the development of artificial languages.
In developing an artificial language the most obvious idea is, first of
all, to base the language on a simple grammar with no irregularities and
a set of words easy to remember and easy to pronounce. In the context of
MIRACLE the aspect of pronunciation is of subordinate importance because
the project focuses on information stored visually and will fall back to
spoken language only on a very low level, for instance to let the system
speak isolated words.
A further potential step to reduce the amount one has to learn is
to eliminate all kinds of inflection, i.e. to use an
"isolating" language, where each word is used in one form
only, with attributes replacing inflexion. The statement "They
went" might end up grammatically as 'I (third person plural) go
(past)'. One can also use attributes to reduce the number of words,
thus ending up e.g. with 'person(female)' instead of 'woman',
'person(male)' instead of 'man', 'person(young+female)' instead of
'girl', etc. This principle can also be used to convert nouns into
verbs by adding action attributes, i.e. 'leg(action)' means 'go' or
'walk', 'mouth(action)' means 'talking', etc.
Artificial languages or planned languages are consciously created according
to definite criteria of an individual or group of individuals for some
specific purpose (such as making international communication easier). Thus
they are distinguished more or less sharply from natural languages that
have evolved during long periods of time, and usually have accumulated
a number of irregularities in the course of this development. But not all
artificial languages have been constructed with the aim to find a tool
for easier communication of the world's population. Some artificial languages
have been invented to facilitate the communication between humans and computers
(programming languages) or to represent the scope of some scientific field
easily (e.g. the chemical formulae-language or the mathematical or logical
languages as invented since the 16th century by philosophers like Francis
Bacon, Jan A. Komensky (Comenius), René Descartes and Gottfried
Wilhelm Leibnitz). Attempts were also made by Rudolf Carnap and his colleagues
of the Vienna Circle to formalize the language of science according to
the ideal of physics in order to eliminate all (so-called) metaphysical
and (possibly) meaningless words from it ([Ca34]).
Other languages again have been invented as elements of fictional works
of literature (e.g. Klingon, or the languages invented by J.R.R. Tolkien).
Some languages have been constructed for plain scientific purposes to experiment
with and to support certain (socio)linguistic theories (Lojban started
that way!). And there are even invented "stealth" languages whose
purpose is in fact a restricted use in a small, selected group of people.
A fascinating example of such a secret language is Damin, a special language
used by the Australian tribe of the Lardil for ceremonial purposes only
open to the male population in a specific stage of initiation. According
to Steven Pinker ([Pi94]) it has a unique 200-word
vocabulary that is learnable in a day but can express the full range of
concepts in everyday speech.
In the context of MIRACLE the most interesting constructed languages
are those whose purpose is to ease communication across language borders.
Such languages are called auxiliary languages, international languages,
global languages, universal languages, interlanguages or interlingua.
Auxiliary languages appeared from the middle of the 19th century. Leibnitz
pointed out the main necessary qualities for a global language: clear,
simple and easy to learn. Another desired quality of a truly global language
should be neutrality with respect to gender and linguistic, cultural and
political background. Ideally no speaker or hearer of the international
language should have the feeling of being culturally hegemonized. Only
very few - if any - constructed languages have achieved this goal so far.
Perhaps with a visually based language these goals could easier be achieved.
However, at the moment this is merely speculative hope. In what follows
we briefly present a few of the best known constructed languages that seem
most interesting in the current context (for references see [ConLang]):
- Esperanto was introduced in 1887. Its vocabulary is formed by adding
various affixes to individual roots and is clearly based on Indo-European
languages. The grammar, too, is based on that of European languages but
is greatly simplified, regular and flexible. Esperanto has a phonetic spelling
and uses the Roman alphabet, each letter standing for only one sound. Sample:
Mi revans pri lando kie Esperanto paroligas. (I dream about a land where
Esperanto is spoken)
- Ido, a revision of Esperanto, was introduced in 1907 and quickly emerged
as a significant improvement of Esperanto. Although it was simpler, yet
more precise and powerful than Esperanto, it still failed to replace Esperanto.
Sample: Lernar Lingui nacionala esas desfacila. (Learning national languages
- Interlingua was created in 1951 by the International Auxiliary Language
Association. It is derived from English and the Romance languages in both
grammar and vocabulary, with a minimal grammar. It has been used at medical
and scientific meetings. Sample: Le unitate del civilisation occidental
corresponde in grande mesura a un unitate linguistic. (The unity of Western
civilisation corresponds in great measure to a linguistic unity)
- Glosa: Continuing the idea of reducing inflexion as a main obstacle
in learning languages, L. Hogben came up with Glosa in 1943 (then called
Interglossa; reintroduced 1972 as Glosa by R. Clark and W.Ashby). As an
isolating language (i.e. each word is used in one form only, see above)
Glosa typologically resembles more an East Asian (especially Chinese) or
African language (but English too), although its vocabulary roots stem
from the classical languages Latin and Greek. Glosa might be a valuable
source of inspiration for MIRACLE. Sample: Anti kali klima na pa resta
intra. (In spite of the fine weather we stayed indoors)
- Lojban was founded in 1955 (then called Loglan) by C. Brown with the
primary goal to test the Sapir-Whorf hypothesis. It is based on the principles
of logic and the hope of removing a large portion of the ambiguity from
human communication. Lojban has a root vocabulary of 1300 words, an unambiguous
grammar (proven by computer analysis with the software tool YACC), is simple
and supposed to be culturally neutral. In Lojban, sentences are formulated
as sets of arguments tied together by predicates. Though Lojban was designed
as a human language, due to its strictly logical structure, it is very
suitable for being processed by computers. On top of the core concepts
of predicate logic, Lojban adds non-logical constructs that do not obscure
the logical structure, but allow communications that are not amenable to
logical analysis. Thus, Lojban has a full set of emotional indicators.
Sample: la fits.djerald. fanva zo'e le glico zo'e la .Odisis. (Fitzgerald
translates something into English from some language as The Odyssey or
The Odyssey is a translation into English by Fitzgerald)
In addition to artificial languages there have also been attempts to
simplify existing languages: A special case in this connection is Basic
English designed by Charles K. Ogden in 1930 [Og30].
It is different from the constructed languages mentioned here in that it
is directly derived from a natural language, namely English. Basic English
is correct English restricted to a vocabulary of 850 words (with only 18
verbs!) and a small subset of grammatical rules of the "full"
English enhanced by a number of generalizations and with irregularities
Interlinguistics is the study of international linguistic communication
from all its aspects, including the roles, structures, and ways of development
and application of ethnic and planned languages as international means
of communication. This discipline deals with the comparison of languages,
the natural process of the blending of languages in multicultural settings
(resulting in such languages as Pidgin English or the historic Lingua Franca)
and the construction of artificial languages. MIRACLE hopes to gain insight
into a number of issues with the help of interlinguistic research, such
(a) Mechanisms of "successful" language simplification and
trade-off between simplicity and efficiency of constructed languages.
(b) Relation between characteristics of constructed languages and parameters
regarding their environment such as the community of speakers (users) and
any specific application areas for which the languages might be designed.
(c) Aspects of cultural dependence and largest possible degree of cultural
2.4 Static Visual Languages
The main idea in visual languages is to replace words by graphic symbols.
The major aim is to have symbols that are easily understood, to use the
combination of symbols to describe attributes, and to create compound symbols
to represent new meanings. Symbols representing words can also be converted
between functional forms, for instance by surrounding basic neutral word
forms with shapes indicating their function (noun/verb/adjective etc) or
by attaching specific graphic indicators to achieve a similar effect. Thus,
in visual languages possibilities to shift between levels of abstraction
should be provided. It is a formidable task to describe the some 40.000
words of a natural language using "few" symbols. However, since
Basic English gets by with about 850 English words for most ordinary conversations
the belief that even fewer symbols are sufficient is not unfounded (converting
nouns into verbs by a single attribute, etc.). However, as Neurath observed,
no graphic symbol can be clearly understood without assigning concrete
meaning in a learning process.
2.4.1 Hieroglyphs, Pictorial Bibles, Comics, Web Pages, Computer Games
Of course visual, symbolic communication is not at all new. In fact
in the development of recording media for human communication, drawing
and painting came before scripting and writing, as impressively made obvious
by all the wonderful cave paintings, hieroglyphs and other samples of pictorial
communication found around the globe (see [Hon00] for
an inspiring analysis of Maya glyphs in the light of computer icons).
Paintings were used in temples, churches and other public places to
communicate political and religious messages to the illiterate who were
able to decode the specific iconography. In the Middle Ages up to the 19th
Century picture bibles were used for the same purpose and today, of course,
the first books for children still are picture books. To represent the
flow of time images were presented in a sequenced way from early on, an
impressive example being emperor Trajan's column in Rome with its spiralled
frieze of about 200 meters length.
In 1827 Swiss Rudolphe Töpffer created the first picture stories
as sequences of pictures combined with text, and was soon followed by Wilhelm
Busch. By the turn of the 19th century, true comics with balloons (which
however have sporadically been used already 200 years earlier) emerged
and have developed an amazing wealth of visual communication techniques
since then (see [McCl93], [McCl00],
[Ei85]). In figure 2 we see a cartoon which we can
understand very easily. However, this is not a process as "intuitive"
as we might think at first. There are a number of unspoken assumptions
we have to make in order to understand the story. First, we must read it
as a sequence, picture by picture (which fortunately are clearly separated
in this case), from left to write. Then, we have to know how to interpret
the gesture of a person extending an arm and the gesture of handshaking
which are heavily culture dependent. And finally we have to know how balloons
work in comics (a "thought balloon" in this case as opposed to
a "speech balloon") and we have to read and interpret the word
Figure 2: A comic sequence, intuitively comprehensible?
Thus, visual communication requires background knowledge as well! Its
intuitive directness comes at the price of increased ambiguity. However,
ambiguity is not always unwanted, but can be a desirable component of communication
under various circumstances. In arts particularly, ambiguity is a key factor.
With the wide availability of interaction and animation on the Internet
(and also computer games in general), comics have recently started to experiment
with dynamics, interaction, and interlinked structures (hypertext). The
Web is thus the playground where all types of currently available media
merge: text, graphics, animation, video, audio, all seamlessly integrated
and highly networked and interactive. This development reflects a general
long-term shift from static paper-based media to dynamic computer-based
2.4.2 Icons, Isotypes, and Pictograms
Viennese Otto Neurath (1882-1945) was, among other things, a pioneer
in communication theory where he developed early steps in media literacy
and a practical system of iconic communication called Isotypes. The aim
of Isotypes was to directly communicate to the masses important facts about
their environment and social circumstances. It was mainly about making
statistical data visible in a direct way and showing the relations between
various sets of data. Figure 3 shows an example depicting the percentage
of occupied women, with colour indicating the industry areas.
Figure 3: Sample of publication using Neurath's Isotypes:
Gertrude Williams "Women and War", Nicholson and Watson, London
Neurath's work ([Neu91], [HB02]) had far-reaching implications. It currently
enjoys a revived interest which is in partly related to the importance
of the type of iconic information that is so essential in modern
user-computer interfaces. Today's software is mostly operated through
so-called graphical user interfaces which employ such icons as control
elements (plus the notion of "windows"). Also, the set of
pictograms used at public places around the world (at airports in
particular) to guide large international groups of people (see figure
4) are fully in line with Neurath's pioneering practical work and his
more theoretical considerations (on issues of modern icons and visual
communication see e.g. [Hor98], [YB00], and [CHT02]).
Figure 4: Some samples of internationally recognized pictograms
Developed by Charles Bliss in 1949 ([Bl49], [Bl84]),
Blissymbolics (also called Semantography) contains primarily abstract visual
symbols that serve as an alternative to traditional orthography. Figure
5 shows a set of basic Bliss symbols.
Figure 5: Some basic Bliss symbols
Blissymbols uses combinations of symbols to describe attributes, compose
simple symbols to represent new meanings, etc. Based on a logical system,
visual markers are added to symbols to change syntax and pragmatic functions.
For instance, by attaching the simple "action indicator" ^ a
symbol that otherwise denotes an object (noun) is turned into a verb. This
concept of "visual attributes", when combined with orthogonality,
is very interesting in the context of MIRACLE (orthogonality as defined
by van Wijngaarden means that features carry across from one context to
another; see [vW65]). While many Blissymbols are generally
quite abstract, they are at the same time iconic and therefore often easy
to understand. This applies even to attributes, for instance an arrow next
to a heart symbol could indicate a positive or negative feeling, depending
on whether it points up or down. Further indicators are used to denote
direction (even in the realm of emotions: an arrow pointing upwards next
to a heart means happy), quality, quantity, possession. Variants of the
activity indicator are used to denote tense: ) for past and ( for future.
The addition of "strategic symbols" for not,
opposite of, part of, similar, without
adds further meaning. Figure 6 shows small numbers as "person
indicators" whereas figure 7 shows the action (verb), adjective,
thing, and plural indicators applied to the basic symbol for
Figure 6: Expressing person in Blissymbols
Figure 7: Bliss indicators for action, quality, thing and
Simply by putting symbols next to each other (without overlapping which
would result in derived symbols as shown in figure 8) new meaningful combinations
of symbols can be constructed, called compound symbols (in the context
of MIRACLE we would rather call them macros). This is shown in figure 9.
Figure 8: Derived Bliss symbols
Figure 9: A compound Bliss symbol
Blissymbolics is far from unambiguous or culturally independent. It
fully relies on the intuitive power of visual symbolic information and
requires a significant amount of intelligence and imagination to interpret
However, on the basis of its small and simple sets of symbols and rules,
Blissymbols is amazingly flexible and can be used to express a wide range
of information, from emotional to more technical content. The following
samples should give a vague insight into the way Blissymbolics work.
||Did you hear that loud thunder last night?
||Please don't walk on the lawn.
||Weapon inspectors found 11 empty chemical warheads.
Blissymbolics is a simple yet powerful approach and probably is the
most widely know of all temporary visual communication systems. For MIRACLE
it serves as an important source of inspiration. In particular interesting
for MIRACLE are the principles of combining symbols to form macros ("compound
symbols") and the more or less orthogonal use of various indicator
markers to add meaning.
CD-Icon developed by Colin Beardon ([Bea92],
[BDMY93]) was an early computer-mediated system
for person-to-person communication. Its design was based on the
theory of Conceptual Dependency (see section
2.1). In CD-Icon, messages were composed as four interconnected
screens allowing to specify
- Message type (assertion, question, imperative, negation) and logical
combination of components in case of compound message (AND, OR, implication,
temporal, or spatial).
- Conceptualization by selecting the primitive ACT in the sense of CD.
Icons representing ACT primitives can be animated to self-explain their
- Pictures as iconic representations of the items in the CD network,
selected from a picture lexicon. The icons are accompanied by sets of modifiers
(color, size etc.).
- Lexicon: a collection of pictures to chose from.
As one of the earliest projects in that field CD-Icon was very rudimentary
and clumsy to use. However it has some significance as one of the first
attempts to a formal approach to a computer-supported visual language as
a tool for person-to-person communication, and as a forerunner of VIL (see
Minspeak ([BB82], [CPOT94],
[Cha96], [ACP98]) is a system
to support "semantic compaction" of language. It maps concepts
on to multi-meaning icon sentences and uses these icon sentences to retrieve
messages stored, based on a set of inference rules. The stored messages
can be words or word sequences. Designed to support individuals with speech
impairments in their communicative efforts, Minspeak has been the basis
of a number of well-distributed applications in that field.
In another experimental application the framework was used as an interface
to a virtual library (named Bookman). Extended to the multimedia field
with so-called tele-action objects ([Cha96]) the system
is also able to support dynamics, represented by Petri nets.
Minspeak is formally based on the concepts of Icon Algebra and - again
- on Conceptual Dependency. In Icon Algebra, icons are combined according
to a set of rules (relations) to derive new meanings for their combinations.
In that way a small set of icons can be mapped onto a large number of sentences
(meanings). Figure 10 shows the visual sentence "The children drive
to school in the morning". If the house icon would be placed in the
WHAT column (the context is changed) and the WHERE column would remain
empty, the message would translate as "The children study in the morning".
Figure 10: Example of Minspeak environment
Minspeak is interesting in two aspects. First, it does not try to avoid
ambiguity, but considers ambiguity of icons a powerful feature. Second,
the actual goal of Minspeak is to produce text out of visual representation,
not to communicate through the pictures themselves. In that way, iconic
representation of information is considered an intermediate stage. This
is in contrast to the aims of MIRACLE.
2.4.6 The Elephant's Memory
The Elephant's Memory ([IH99]) is an artistic approach
to a pictorial language consisting of about 150 combinable elements called
logograms or signs. The goal of this language was to set up an experimental
research environment to explore the field of invented languages, mainly
with children. The project introduced experimental semiotics and visual
grammars as "mindscapes" to be explored in playful ways, without
trying to formally define language elements.
The language exploits the medium of a plain surface, as it can be written
and read in all directions. It employs a "logographic compass"
to assign meaning to orientation (mainly with respect to time and causality).
With the wave form symbolizing time and the logographic compass providing
proper orientation pointing from the past to the future, the logographic
statements on the left in figure 11 mean the same: "Walking before
Figure 11: Logographic compass with two sample of time logogram
(left) and one of cyclic causality (right)
Temporal aspects can thus be nested in a recursive way visually ("Yesterday
I wrote you that next week I will visit you after my trip to Rome").
The same principles work with the causality symbol where the logographic
compass provides the reason/consequence direction. The statement on the
right in figure 11 can be read in either direction of the cycle, starting
at an arbitrary point and its meaning will always be some variation of:
"Running makes one tired, being tired makes one sleep which makes
one happy, and being happy makes one run." Or, read counter-clockwise:
"Sleep is the consequence of fatigue which follows from running which
is done because of happiness which is a result of sleep".
Visual statements formulated in The Elephant's Memory are to be read
along non-linear, recursive Hypertextual structures, with the grammatical
structure based on two parameters: position and size. The language is egocentric
in a recursive way as demonstrated by the example in figure 12: The first-person
"speaker" is on the left (the head with the big eye) with the
scene he or she is watching on the right and the resulting emotion below,
both parts clearly connected to the speaker.
Figure 12: "Seeing elephants shot by men makes me cry"
(left) and interrogation (right)
An interrogation sign is used in the sense of an attribute which can
turn almost any part of a statement into a question about the aspect to
which the sign is attached. The logogram on the right of figure 12 thus
means: "Who is doing what?"
The Elephant's Memory does not have a written-down syntactic set of
rules and semantic rules are only vaguely defined. So, it leaves much room
to ambiguity which however, is not considered an obstacle by the designer
but an intended feature. The Elephant's Memory was designed with the computer
in mind from the beginning. However, no actual system has been developed
beyond a rudimentary prototype and the project has been abandoned.
For MIRACLE the most interesting aspects of The Elephant's Memory are
its concepts of multidirectional reading and writing combined with the
logographic compass, and the general openness for the principle of nesting
and recursion. The latter aspect would be an obvious candidate to be supported
by a system like MIRACLE.
Rikchik is the language of a hypothetical population of aliens of the
same name. Among the large number of amateur efforts in the area of constructed
languages documented on the Internet, many of which are inspired by fantasy
and science fiction scenarios, Rikchik is one of the most interesting ones
([Mo]). A Rikchik body consists of a one-eyed globe with
49 tentacles, 7 of which are used for communication purposes much in the
sense of sign languages. As Rikchiks are entirely deaf and dumb, they have
not developed acoustic communication and no language based on speech. While
the semantics and contents of Rikchik statements of course are "alien"
the structure of the language follows an extreme logic regime.
Four central tentacles are used to sign a basic morpheme, the remaining
three tentacles are used in the sense of attributes to denote the properties
aspect, relation and collector. The signs produced
by the 7 tentacles can directly and unambiguously be denoted using a corresponding
"scripting system". The aspect property determines the
interpretation of the morpheme itself much in the way of the orthogonal
Bliss indicators do. Among the 7 possible values of aspect are idea, action,
name, animate and inanimate objects. Relation specifies the type
of relation between morphemes (again there are 7 possible values) and the
Collector attribute specifies to which of the previously signed
morphemes this Relation applies. Formally, a Rikchik statement can
thus be noted as a tree of triples, or a sequence of quadruples and is
thus perfect input to be processed by computer programs. It thus appears
to be a very strict and in the end conservative conception, very much rooted
in principles of traditional linguistics and formal languages.
In the following example we see three samples of a Rikchik statement
with only slight modifications of the attributes, resulting in other meanings.
The morphemes from top down stand for insect, flower and mouth (which means
"eating" when the activity aspect is attached). The aspect, relation
and collector attributes are found on top of each morpheme and at its lower
left and right corners.
Figure 13: Slight variations of attributes in a Rikchik statement
One interesting aspect of Rikchik with respect to MIRACLE is that the
hypothetic population using it communicates entirely and solely on a visual
basis. However, there is one main point of critique connected to this:
Rikchik does not involve any kind of dynamics and movement, although one
would certainly expect such a feature from a sign language.
In CAILS (Computer Assisted Iconic Language System, [Ch00])
graphical icons (representing the vocabulary of Basic English ([Og30])
are placed into simple geometric forms that express grammatical structure.
These "grammar containers" are shown in figure 14 and stand for
(from left to right): subject, object, origin (ablative), here (dativ),
destination, adnominal (genitive), as well as verb past, present and future.
The rightmost image is a grammar symbol filled with two icons, making up
a "unit of expression" meaning "A book owned by a man".
Figure 14: Grammar symbols in CAILS
Such "units of expression" are put into some relation by using
special connectors (basically arrows of various types) expressing certain
types of conjunction as shown in figure 15 (from left to right: that, if,
because, but, and, with).
Figure 15: Conjunction connectors in CAILS
The small number of rules for building CAILS statements (about a dozen)
quite strongly relies on conventional linguistic terms. While the project
shows an interesting general approach, the ideas do not appear very well
developed and the project apparently was closed in 2000 without resulting
in any concrete follow-up activities.
2.4.9 VIL - A Visual Inter Lingua
VIL ([BL99], [Lem01]) is a
well-founded effort towards a computer-mediated iconic language with a
design based on conclusions drawn from simplified natural languages such
as Pidgins and Creols as well as Basic English. Carrying on the approach
of CD-Icon, VIL is again formally based on Conceptual Dependency (see section
2.1). Two key features are that it is verb centered, and order independent.
The verb has to be selected first and it determines all further attributes
(including nouns) that can be specified (some mandatory, some optional).
Figure 16 shows an icon with all its visual elements, as it is displayed
by the VIL system. In this case it is a noun (apple) as defined by the
type of the border (rectangle). The three small symbols to the right of
the apple icon are control elements that visualize further attributes and
grammatical parameters of the element, including the position of the element
in the hierarchy of the icon lexicon ("iconicon").
Figure 16: Components of a VIL icon
Figure 17 shows the main screen of VIL after selecting the verb "put"
as a starting point. The verb is in the center and shown on white background
which means that the verb entity has already been instantiated. Verb icons
can be animated be clicking on them to show the activity directly.
All other surrounding elements are in grey, meaning they have not been
defined yet. The elements offered for specification are determined by the
verb: different verbs will yield different situations. Clicking on the
elements invokes specific screens to instantiate the parameters. Some of
them are offered in hierarchical form, others can be selected by pointing
on an image (map) or by using a slider (for selecting values from a range
continuum such as "cold to hot") or other interactive controls
(such as a clock for defining time).
Using a general, limited subject area as test-bed (cooking and nutrition),
a thorough evaluation of VIL has been carried out using the Internet as
a platform for online questionnaires. However, the system does not provide
any facilities to support online communication other than sending VIL statements
as email attachments.
VIL demonstrates that complex and accurate visual statements can be
composed with an interactive computer tool. However, the system forces
the users into quite strict structures of thinking and acting (when composing)
that appear counter-intuitive even though they strongly follow linguistic
lines (or maybe because they do so; maybe visual thinking follows different
lines). The need to learn how to deal with this system seems to be significant.
Leeman claims a number of requirements on a visual communication system
in his well-researched analysis, but all in all, VIL fails to meet some
Figure 17: Main screen of VIL system
2.5 Animated Icons
The concept of animated icons to support understanding of iconic information
has been developed and used in a number of projects for more than 10 years
([MY90], [BSM91], [JGM93],
[Dor94]). The basic idea is that icons representing
activity can be "self-explaining" ([Dor94])
by "acting out" the activity itself in the form of a brief animation.
There are various ways users can invoke such animations, for example by
clicking on the icons, or by moving the mouse over them. One key difference
between such simple forms of animated icons and more sophisticated forms
of dynamic effects as described below is that the animation they show is
clearly restricted to the icon itself and has no effect on or relationship
with other visual components.
Of the projects described in the previous section, CD-Icon, CAILS and
VIL incorporate animated icons in this sense described here. Figure 18
shows two rather abstract samples of animated icons in CAILS: Eating is
shown as a slice from which step by step small pieces are "bitten
off" whereas reading is symbolized by lines appearing from left to
right, top down.
Figure 18: Animated icons in CAILS for "eating"
In VIL activities such as pour, mix, and put (from the cooking domain)
are directly shown as rather concrete little animations. Minspeak allows
the definition of dynamic structures that can be formalized as Petri nets.
However the actual implication of this concept and to what extent it can
be used in the sense of a structurally dynamic language as described in
the following section is not quite clear.
2.6 Visual Languages with Structural Dynamics
When talking about dynamic visual languages we mean languages incorporating
elements of change and/or movement on a structural level, not only as animated
icons as described above. That means that dynamics carry meaning in the
linguistic sense, expressing relationships between separate visual items
or between different states of the same component over a period of time.
While static visual languages can in principle be used on the basis
of paper and pencil, as soon as dynamics is involved, some sort of other
medium is required supporting this new dimension. In our case (and opposed
to the natural sign languages that use the human body as a medium) we assume
some kind of computer supported platform to take on this role.
Efficient perception of movement is essential for any animal relying
on vision in order to escape rivals and find prey or mates. Cognitive science
is investigating the corresponding mechanisms and comes up with interesting
results. For instance, movement of the human body can be very heavily abstracted
and will still easily be recognized. If this also applies to parts of the
body, in particular to the hands, this would explain the efficiency of
sign languages. Such findings could have implications on the design of
new dynamic visual languages.
Designed by Lennon and Maurer MUSLI (A Multi-Sensory Language Interface,
[ML94], [Ho00], [ML01a],
[ML01b])), with one of its important features being
dynamics, can be considered a forerunner of MIRACLE. In some respects MUSLI
is based on a functional grammar that classifies processes into material,
mental, and relational types and consequently deals with doing, sensing,
and being. The syntax of the visual language is simple and comprises actors,
compound object, and relations. MUSLI statements consist of objects which
appear on a "stage" and are subject to events. Also employing
the concept of scenes (through which paths can be defined), to system follows
a metaphor of theatre play to some degree.
The MUSLI project presented a rudimentary prototype implemented in Macromedia
Director in which the basic concepts of orthogonal attributes, dynamics
(including change of shape and colour) and abstraction (condensing complex
information into "macros" for later expansion) of a computer
supported visual language have been demonstrated. The story of Snow White
and the Seven Dwarfs was used for demonstrational purposes. The three panels
in figure 19 show how the bad queen kills Snow White. In MUSIL this is
presented in a very intuitive way involving dynamic effects: a skull (symbolizing
death) moves from the queen over Snow White who vanishes away.
Figure 19: Snow White dies by vanishing: example of dynamics
Figure 20 shows the hierarchy of properties (symbolized visually) attributed
to Snow White from left to right: she is a young, beautiful, royal woman.
A woman is a female human, a human is an intelligent living being (the
wave denoting intelligence, the circle denoting a living being), and a
living being is a thing that lives (the square denoting a thing, the heart
denoting that it is living).
Figure 20: Hierarchy of properties of Snow White
Users can select the preferred level of abstraction of the presentation
by simply clicking on an object to get a level deeper and to see the "hidden"
attributes. This process is called expansion (as opposed to condensing).
Also, a number of filters can be applied to scenes to omit the background,
to switch between presentation styles (say, Disney to Van Gogh), or to
omit living beings or inanimate objects (figure 21).
Figure 21: Filters in MUSLI
2.7 Sign languages
Sign languages are the natural languages of the deaf. They use the human
body of the sender (mainly hands and face) as the "platform"
or medium for communication. As entirely visual languages they employ space
and movement as means of expression and thus belong in the same category
as the main subject of MIRACLE: dynamic visual languages. There are two
reasons why sign languages are to be considered in the framework of a project
such as MIRACLE:
- Space, time, and structure: Because sign languages fully exploit
time and space as means of visual communication, their overall structure
and grammatical and semantic features are highly relevant to MIRACLE. It
seems to be a valid and promising approach to analyse in detail the mechanisms
of these features to find out to what extent they can be usefully applied
in the MIRACLE setting.
- Naturalness: Because sign languages have developed naturally
over a long period of time it can be assumed that they reflect certain
basic cognitive principles regarding non-textual, dynamic visual communication.
It seems wise to investigate these principles in order to derive appropriate
guidelines regarding the user interface a platform such as MIRACLE could
In passing we want to point out that signing has been and is also used
outside the deaf communities in various contexts. To mention just a few,
certain monastic brotherhoods developed simple sign languages to communicate
while following their vow of silence ([US87]), divers
use a set of standard gestures for communication and in mudras, the tantric
dances of India, gestures and posture of fingers are charged with symbolic
meaning according to strict rules. Going into more detail concerning the
structure of sign languages of the deaf we find that a number of aspects
are of particular interest in our context. These are discussed below.
- Iconicity: There are a large amount of signs in all natural
sign languages that in some intuitive way directly show or hint at certain
aspects of the communicated information. In order for iconic signs to trigger
understanding, often rudimentary hints are sufficient.
- Shape: Shape is another key component of sign languages charged
with meaning. On the human body there are mainly two parts that can be
"shaped" intentionally: face and hands. Of the unlimited set
of possible hand forms a small set of about 20 to 30 is used in all sign
languages in an abstract way very much in the sense of an alphabet of a
spoken language. This basic set in turns contains a subset of about 5 to
10 signs that are the kernel signs which typically can be related to a
character/sound of the local spoken alphabet. These hand forms have particularly
clear shape so they are easy to distinguish. Figure 22 shows the basic
set of German sign languages: A, B, 5, G, C and O. In general, mimics ("face
shape") often serve as an attribute strengthening the meaning of signs.
Figure 22: Basic hand shapes: A, B, 5, G, C, O
- Location: The location where a sign is carried out is an important
component, of course always referring to the human body of the signer.
Location can be seen as an attribute.
- Timeline: Generally, time is indicated by using space and movement.
A number of virtual "time lines" are used to visually denote
different temporal aspects (see figure 23: A for past to future; B for
calendar units, sequence and duration; C for continuity and/or duration,
and D as "human time" such as growth of a person etc).
Figure 23: Virtual time lines
- Attributes: Many verbs can be produced with different hand shapes
representing persons, animals, or objects. In that way the specific hand
shape serves as an attribute. Such "classificators" specify objects
by referring to their typical visual characteristics such as size and shape
but also to more abstract qualities or to concrete things such as tools.
Figure 24, first row, shows "eating an apple" (round shape) whereas
the second row means "eating a salt stick" (or something else
very small and thin).
Figure 24: "Eating an apple"(1st row) and "Eating
a salt stick"(2nd row)
- Pointing: Pointing is a key component of sign languages used
in a variety of situations. Some signs are used for specifying location
(here, there, in/outside etc). Specific index signs allow allocating a
person, animal or object to a certain "point in space" (called
locus) for later reference. The signer does so by first naming or defining
a person or object and then pointing to somewhere in the space in front
of him. Later this object can be incorporated into complex signed statements
by pointing to the locus again. In MIRACLE this concept corresponds in
a natural way to the idea of macros, explained elsewhere. Finally, the
role of possessive pronouns of spoken languages (my, your, his/her etc.)
is taken by special index signs pointing to the possessing person, object
or locus representing it.
- Movement: Sign languages heavily rely on movement and dynamic
elements in general. All kinds of signs can include moving components but
in particular signs representing activities. Usually, start and end point
of the movement represent aspects of the activity such as subject/object,
source/destination etc, depending of the nature of the statement.
- Efficiency through Simultaneity: In sign languages many features
can occur simultaneously, resulting in "multidimensional" messages
of high complexity. This is the reason why sign languages are about as
efficient as spoken languages in terms of information transfer rate. Although
within a statement signed features occur simultaneously, statements are
separated by brief "halt segments" in which all features are
"frozen" for a short moment.
- Formality: There are a number of approaches to formal notation of sign
language, the best known being Sutton SignWriting (designed to support
written communication based on signing, [Sutton])
and HamNoSys (designed for scientific purposes, [Ham]).
They might serve as a model or at least as an inspiration on how human
related movements can be structured and notated (and thus be stored) on
a computer environment. Figure 25 compares signed statements with three
ways of the corresponding formal notification.
Figure 25: Three variations of formal notation of sign language
- Simplicity vs. Context Sensitiveness: Sign languages are very
simple compared to most spoken languages in terms of grammar: There is
no inflection, gender, tense, case and no singular or plural. However,
this comes at a price: Signed statements can be very context sensitive
and hence ambiguous. Dependencies and implicit relationships may accumulate
in a sequence of signs, so that a concluding sign may be "charged"
with a lot of implicit information only to be understood in the context
of the preceding signs.
While sign languages are restricted to the human body as communication
medium, in MIRACLE the communication medium is computer-based. However,
with the wide availability of wearable computers in the future this distinction
might become irrelevant to a large extent. Future human-computer interaction
facilities might involve new devices replacing current data gloves and
data suits. And new tools for visual recognition of facial play and gesture
recognition (input) and special glasses that project images directly onto
the retina (output) might become available.
Computer systems to support translation between spoken language and
sign language - in both directions - are already available in experimental
or early commercial versions. See e.g. [Kad95] for
an early experimental approach to sign recognition based on data-gloves,
and e.g. [HO01] on an approach to visual sign recognition.
Speech-to-sign translation is demonstrated in the EU project VISICAST (www.visicast.co.uk)
and has been brought to the market by the products such as iCommunicator(tm)
(www.myicommunicator.com) and SigningAvatar(tm) (www.vcom3d.com).
Iconic gesture-based human-computer interfaces are also being developed
However it is interesting to note that although all these systems exist,
no attempts seem to have been made so far to exploit features of sign languages
in others ways than for mere translation. Hardly any efforts have been
made to adopt some of them to computer-based platforms to enhance visual
communication. The concept of Worldsign/Symbolvision ([Worldsign])
is an exception that has to be mentioned here for thinking up a system
of visual communication that can be signed dynamically or depicted as static
icons. Although only a vague, incomplete concept, it also recognises the
potential of the computer to dynamically visualize signed language elements.
The notation system employs so-called "gestographs" which are
strongly simplified representations of the human body parts involved in
2.8 Conclusions of Background Survey
From the observations made in this section about the state of current
research and corresponding projects we can draw a number of important conclusions
that serve as the basis for further research in the framework of MIRACLE.
(a) Orthogonal and hierarchic concepts: Most of the well-known
constructed written and graphical languages found today share a central
set of features which includes principles of orthogonal attributes and
recursive structures allowing the hierarchic definition of concepts (called
macros or compound elements). However, implementation of such features
in computer-based systems has been rather basic and often rudimentary so
(c) Strong formality: Most serious projects try to define a complete
visual language from scratch, with a language structure precisely described
in a formal way (some based on Conceptual Dependency). Some of the projects
also spend much effort in building large visual vocabularies.
(b) Limited dynamics: Dynamic elements in visual languages are common
and well researched in form of animated icons only, i.e. simple animations
that explain the meaning of an icon by "acting out" its meaning
(which usually is a verb). However, dynamics in terms of structure are
rare in current systems and concepts. The large field of issues opened
by the use, meaning and control of movement of symbols (that themselves
may be animated) has not been investigated so far.
(d) Neglecting social interaction in networked media: None of the
projects investigated tries to understand and exploit the social-dynamic
processes that determine the development of language on networked media
such as the Internet.
(e) Limited multi-disciplinarity: There is currently no comprehensive,
multidisciplinary effort towards a holistic approach regarding these issues.
Such an approach would require a multidisciplinary team of experts and
considerable resources and is expected to take a period of about five years
to show profound results.
(f) Limited breadth: None of the projects that involve computer
support to address the issues came up with more than quite rudimentary
prototypes. Neither comprehensive nor representative experiments were made
so far in multiple settings (regarding type of applications and user groups).
3 Technology Potential
MIRACLE is based on realistic assumptions about the further development
of computer hardware and software. In this section we lay out some plausible
scenarios of future computer systems with respect to the issues of MIRACLE.
3.1 Hardware: Revolutionary New Ways of Interaction
We assume that powerful networked computers not much larger than today's
cell phones will be available before the end of this decade as ubiquitous
companions. As a consequence, most information might be consumed not by
reading from static material (such as paper) but from an interactive, dynamic,
mobile device with very high display quality and a wealth of novel functionality.
This will not only change the way we consume information, but also the
type of information at issue. The exact form such a "portable personal
computer" (PPC) will take on over the next five to seven years cannot
be predicted now. However, this is irrelevant since what is predictable
are the available functions of such a PPC. In what follows we claim 12
facts concerning such PPCs. We argue why they are likely and indicate a
few of the consequences.
- We will be able to do all with PPCs that we are doing today with
powerful PCs: Advanced versions of cell phones are starting to offer
more and more PC functionality. Conversely, a number of new handhelds run
a full Windows operating system and can be used (usually with a small attachment)
as powerful cell phone.
- It will act as cell phone: Because of the convergence of cell
phone and computer technologies it is irrelevant to distinguish between
them when talking about the future.
- It will have continuous access to the then dominating version of
today's Internet: That computers/cell phones will have permanent wireless
access to networks is a consequence of emerging new wireless network standards
such as UMTS which are not time but volume oriented. Paying for the volume
of transmitted data instead of paying for connection time will clearly
encourage the use of the Net even for trivial issues.
- It will have a built in still and video-camera: Multimedia cell
phones have already a built in camera today. In the future such cameras
will not only be used to send pictures or movies, but for instance to use
image processing of gestures as a new powerful input medium, to have the
computer analyze the environment (from face recognition to object detection),
or to allow the user to look (so to speak) through the camera using its
zoom or macro facilities.
- It will have a built in GPS (the satellite driven Global Positioning
- It will have built in sensors to know the direction of the head
of the user: The built-in GPS and direction sensor will allow guiding
users in an unprecedented way: not just for routing, but as guide to buildings,
scenery etc. Combining this with image processing provides a myriad of
new applications. But, most important for MIRACLE, even gestures such as
nodding, shaking the head etc. can be recognized by the PPC and can thus
act as very simple ways to express commands.
- It will have a microphone: PPC microphones may be hidden behind
a tie, in a tooth, etc., allowing to pick up language even if it is spoken
with closed mouth. Together with good speech recognition this provides
an excellent way to generate input, quite likely superior to typing on
- It will deliver stereo sound: PPC "loudspeakers" could
be built into the sides of eyeglasses so that they deliver sound directly
to the ear-bone, thus inaudible to other persons.
- It will probably have no hard-disk or other movable storage device:
Mechanical parts are error-prone. By the end of this decade memory-sticks
with a capacity of 30 G Byte or more will be common and will be a small,
simple and robust replacement of hard-disks.
- It is likely to use new display technology rather than rigid display
screens: A host of alternatives to current screen technologies is emerging.
Using eyeglasses equipped with small mirrors to produce virtual images
seems particularly attractive. But bendable or foldable screens, projections
techniques (that work e.g. holographically without screen), digital ink,
etc. may offer other still more interesting alternatives.
- It will probably not have a traditional keyboard, but rather other
novel interfaces: The keyboard as main input device will remain clumsy,
given the size of our fingers. Alternative input devices can be based on
speech recognition, on gesture recognition (following e.g. the movements
of the fingers on a virtual keyboard!), on keyboards integrated into fabric,
on recognizing a number of different brain states, on movements of the
head (as mentioned earlier), on the recognition of some kind of stylus
on some surface (today the screen of a tablet PC, tomorrow maybe the sleeve
of our jacket with some sensors woven into it).
- It is likely to be powered not by batteries as we now know them:
Industry is talking about a growing 'power gap' (see Wall Street Journal:
"Gadget Makers Join the Scramble to Zap the 'Power Gap'", Wall
Street Journal (01/16/03)). We can therefore assume that an alternative
to the current technologies for power supply for mobile devices will be
For more information on the subject see also [Ge00]
and MIT's Wearable Computing TTT (Things That Think) Lab (www.media.mit.edu/wearables),
in particular [DV01] supporting the idea that explicit
interaction will be reduced in favour of automatic detection of situation-dependent
context. Other projects are going on at Columbia University under the headline
of Augmented Reality, in particular the projects MARS and KARMA (see www.cs.columbia.edu/graphics
and [Fei02]). For implications of such a device see
3.2 Software: Towards an Understanding of the World
According to the saying "to understand language is to understand
the world" there is a strong inherent relation between language and
our conception of the world. Computer scientists have always dreamt of
"intelligent" computer systems able to understand the world,
in part through the ability to learn. In the past they came up with the
Artificial Intelligence approach to address this issue. However, only limited
success has been achieved. In the meantime computer scientists have become
more modest. Nevertheless there is a new, more pragmatic approach which
seems promising, feasible and solid: ontologies and semantic webs. A note
of caution: The terms "ontology" and "semantics" as
used here are often only vaguely defined and not always in line with the
way they are used in pure philosophical context. They are nevertheless
useful in a pragmatic sense.
Very simply put, ontologies are networks of related concepts which can
usually be represented by words. In that way, hierarchical structures are
built, which can formally be described as acyclic directed graphs (parts
of which are trees). As an example, "living being" would be a
high-level concept with a number of subordinated concepts, one of which
would be "animal" which would in turn contain "human"
as a sub-concept. Each concept can be part of many relations, depending
on the context. Therefore, the context must be stored together with the
relation to allow full interpretation. For instance, in our example the
type of the relations seems to follow biological considerations (classification
of living beings). So, how would the concepts of "pet" fit into
the animal branch and how would "parents", "child"
or "friend" fit into the human branch? Apparently, those concepts
follow different lines (social ones?).
Ontologies are often formally defined in terms of XML (eXtended Markup
Language) which makes them highly interchangeable among networked software
systems. A number of major efforts are currently underway in these fields
of research, some of which already provide practical solutions for specific
areas of application ([Da03]). In their approach they
combine computer power with enormous human efforts to maintain the knowledge
networks. To name a few of the most relevant projects:
- OpenCyc (www.opencyc.org) is an
open source spin-off of a commercial effort. The declared aim of Cyc is
"to construct a foundation of basic 'common sense' knowledge - a semantic
substratum of terms, rules, and relations - that will enable a variety
of knowledge-intensive products and services". A core component is
a "knowledge server" providing a very large, multi-contextual
- WordNet, at Princeton University (www.cogsci.princeton.edu/~wn)
is a comprehensive network of concepts in English language. Variants for
other languages are available (for instance GermaNet maintained by Tübingen
- SemanticWeb (www.semanticweb.org)
is an initiative focussing a considerable number of individual efforts
and providing a joint platform for the corresponding scientific community.
- OntoWeb (www.ontoweb.org) is a
project funded by the European Commission, mainly directed towards industry
applications of ontologies.
With respect to the specific aims of MIRACLE it can be observed that
the hierarchies of concepts defined in ontologies nicely reflect the concepts
of abstraction or of condensing/expanding of graphical macros found in
many of the visual language approaches. In that way relations between word
based concepts and their graphical representation could be established
and levels of abstraction could introduced. MIRACLE will investigate ways
to exploit ontologies along that line and try to find a practical way to
combine a textual ontology with collections of graphical symbols.
4 The Goals of MIRACLE
Following from what has been said in the previous chapters we belief
that we are nearing a turning point. We expect that within less than ten
years computers providing high resolution colour images in a novel way
will be as ubiquitous and small as today's mobile phones, and intelligent
software systems will feature higher levels of "understanding"
of the world than ever before. As a consequence the kind of information
we will deal with can and will change dramatically according to the true,
natural requirements of our minds as indicated by psychology and neurosciences.
In the centre of future information systems will be a new type of visual
information which can dynamically change and with which people can interact
in an intuitive way to adapt it to personal circumstances (mother tongue,
level of knowledge, preferred presentation styles etc). MIRACLE is one
of the first projects to go in that direction and investigate its potential,
basically by using interaction together with multimedia elements and the
novel notion of dynamic visual language. The ultimate aim of the undertaking
is to ease information and knowledge transfer between persons by making
(a) more efficient (which can mean a range of different things) and
(b) to gain a high degree of independence from natural languages (which
may be contradictory to (a) in some respects).
The new kind of information archiving and exchange that is envisioned
in MIRACLE will be more intuitive and natural than current approaches,
will support a larger variety of different ways of thinking and understanding,
and it will foster creativity and exploit additional "cognitive channels"
of the human mind (visual, movement, symbolics etc ...). It will allow
to make the abstract more concrete, and the concrete more abstract. This
will be achieved through alternative views ([Len95]),
an important didactic principle. In order to experiment with and validate
the concepts of MIRACLE, prototype software systems will be developed focussing
on following basic features:
- Visual, orthogonal attributes: Characteristics of objects are
depicted as visual attributes. For instance a red cross could symbolize
an abstract quality "medical". A house symbol combined with a
red cross would consequently stand for a hospital, a person with a cross
for a doctor, a plant with a cross for a medical herb etc. The way attributes
are assigned to visual items follows the principle of orthogonality which
means that attributes can always be combined to build more complex meaning.
Unusual or even contradictory combinations of attributes would allow expressing
creative thoughts in a poetic way similar to natural languages which are
also not restricted to mirror reality.
- Dynamics: Activity and change will be naturally depicted as
dynamic symbols moving and/or changing in shape, size or color. MIRACLE
will go beyond traditional animated icons. Not only will it be possible
to turn nouns into verbs, but to have a continuous range for expressing
the "amount" of the movement, which is not possible in spoken
language where we only have a discreet set of values (words) for describing.
With the example of an animated icon standing for the concept of "walking",
a very slow movement would represent very slow walking, fast movement would
symbolize running, and tremendously fast movement could be a poetic expression
of running "as fast as light". Thus, while in a spoken language
we have only discrete states (slow walking, walking, fast walking, slow
running, running, etc), in a suitable dynamic visual languages one can
express almost poetically different shades in between.
- Property on analogous scales: Property attributes (ideally any
item could be turned into an adjective) can be selected from a continuous
range of values, for instance by using a slider. This should also be applicable
to activity shown in a dynamic way as described above.
- Flexible levels of abstraction (macros): Complex combinations
of symbols not understood by the user can be expanded (explained) by the
system at will. Conversely, complex representations can be condensed by
the user to simple symbols to speed up interpretation, yet can be expanded
any time desired.
- Language independency: As a visual language, MIRACLE will use
words of conventional, spoken languages only as a last resort. However,
with the help of proper dictionary programs individual words and phrases
can be directly translated from one language to any other fairly well.
This requires that disambiguation is done at input time because even on
a word level the correlation of meanings in different languages is small.
Just take a simple word like 'rich': the German word 'reich' is suitable
in some cases ('rich person', 'rich in mineral resources') but not in others
- Natural animation: Some situations are easily described using
animation in a natural way. For instance, for showing a driver how to come
from one place to another, the system would include a routing system already
available for cars. Other examples of "natural animation" would
be the description of how a bird flies, the course of the famous battle
at Waterloo, or a weather forecast. MIRACLE will try to support such "natural
animation" whenever possible.
Figure 26: Levels of abstraction in visual terms
An important issue in MIRACLE is to adapt the concept of dynamics to
the actual computer environment typically found today, with a computer
screen used for output. Here, already at this early stage number of issues
can be identified, and more will turn up in the course of the project:
- Efficiency: Activity takes time. Thus, information that incorporates
changing or moving visual symbols takes time to be "read" and
"understood". Users should be allowed to control the speed of
presentation as much as possible but the question remains under what conditions
(and perhaps with what kind of content) can communication heavily based
on dynamics be efficient in terms of the time/volume ratio.
- Time in terms of space: Aspects of time are very often expressed
in terms of space. Think of a conventional analog clock: To interpret it
one has to interpret the spatial arrangement of hands and face. A digital
clock in turn works in an entirely different way. In our context we could
say a digital clock is like conventional text whereas an analog clock corresponds
to visualization. Another inspiration comes from the virtual time-lines
of the sign languages, mentioned above. MIRACLE will explore ways to represent
time in such analog ways of space, exploiting the means the computer is
- Activity vs. movement: Animation can be used to depict activity
or a change of an object itself. But there is also real movement (in the
sense of changing the location of an object) that can be visualized directly.
In this case a reference space is necessary. It would also be possible
not to move the object itself, but instead move the background.
- Action on a static medium: Movement can even be visualized on
a static medium not only by adding an action symbol as described above.
Comics have come up with a range of solutions addressing this issue. Although
on computer driven environments it might seem obvious to exploit the potentials
of dynamics, it may be wise to step back and analyse where it could make
sense to incorporate proven mechanisms from conventional media. One reason
is that dynamic elements can hardly be parallelized: if more than one changing
item at a time is presented the user can hardly perceive the overall meaning.
So in order to provide overviews, some mechanism of showing "frozen"
dynamics (stills) will be required.
We see that there is much room and need to study the cognitive effort
that is required at the users' side to interpret animation and dynamic
elements. It will be one focus of MIRACLE to identify and address the basic
questions related to these issues and to carry out concrete investigations
based on the prototypes that will be developed.
5 The MIRACLE Approach
In this section we describe the main principles that will be followed
in the MIRACLE project and outline the overall design of the project and
some of its subprojects.
MIRACLE is a multi-disciplinary project with a broad spectrum of challenges.
It is therefore essential to do a thorough investigation of the current
state of the art of the relevant subject areas. As the basis for the definition
of and experimentation with a new dynamic visual language MIRACLE will
use the results of efforts made in the following areas:
- Interliguistics: Some of the better-known constructed languages will
be investigated regarding their value to include parts of their vocabulary
and/or structure. Candidates that seem particularly promising are Glosa
and Lojban, as well as Basic English.
- Visual languages: The theoretical contributions of Otto Neurath and
the concrete system of Blissymbolics will serve as particularly strong
inspirations, but also concepts of some of the computer-supported visual
languages presented here.
- Dynamic visual languages: The results of the few visual languages projects
that incorporated dynamics (in particular MUSLI) will be investigated carefully.
Also, dynamic mechanisms in natural sign languages will be analyzed with
respects to their usability in the context of MIRACLE.
- Ontologies: The technology of ontology databases will be considered
in order to find practical ways to structure hierarchies of graphical symbols.
- Virtual communities: MIRACLE will take into account available findings
on social processes in virtual communities, in particular those who refer
to communicational behaviour and the development of corresponding language
phenomena. They are expected to provide valuable input regarding potential
scenarios of a Web-based platform supporting the evolution of a visual,
To properly cover this multitude of issues MIRACLE will take a comprehensive
approach, put into action by a multidisciplinary team of experts. Researchers
in the appropriate fields are invited to join.
5.2 Requirements, General Conditions, and Principles
As a consequence of section 2.8 (Conclusions of Background Survey) in
order to cover new ground, in MIRACLE we focus on the following principles:
(a) Dynamics on a structural level: As laid out before, MIRACLE
will go beyond mere animated icons and investigate the potential of dynamics
on a structural level.
(b) Gestural components: As part of structural dynamics, gestural
components will be included to exploit the potential they show in natural
(c) Internet: The Internet will be used as basis for potentially
large and diverse user communities to unfold the full power of the new
(d) Evolutionary Approach: Based on the Internet as platform, and
in contrast to most efforts made in the past, a bottom-up approach will
be followed. Only a small kernel of the visual language at the core of
MIRACLE will be specified formally. The rest (grammar as well as vocabulary)
will unfold in an evolutionary process, as the user communities use the
language and form it by doing so. The MIRACLE system will provide strong
support in that process of the development of the "self-adapting language".
From these principles a number of specific requirements can be derived
regarding the design of the system and the overall set-up of the communication
platform that will be the basis of the language evolution:
(a) No explicit learning: We want users to be able to use the
system without explicit training, i.e. they should be able to use it in
short time just by "learning by doing".
(b) Culture-independent: The system should also be truly culture-independent
as much as possible. We are realistic enough to know that this can hardly
be achieved fully but believe that a global medium like the Internet might
be a good platform for first steps in that direction.
(c) Large user communities: Following from some of the principles
and requirements stated above, it is clear that the platform will need
to be used by large and diverse user communities. Otherwise no evolution
(d) Interesting forums: In order to draw and keep attention of large
user communities it will be necessary to foster enjoyable environments
and provide interesting thematic kernels for the communication. Fun and
intuition will be keywords in that context.
6 Summary and Outlook
In this paper we give a survey of the vast background regarding visual
(dynamic) languages with particular focus on their suitability regarding
new ways of computer-supported visual communication. This investigation
indicates that a number of key elements in natural and constructed languages
(both spoken/written and signed/visual) can naturally be supported and
significantly be enhanced by applying them in computer-supported environments.
With respect to future technologies, we expect that this will eventually
result in revolutionary new ways of communication that fully exploit the
capacity of the human mind to deal with visual, dynamic information.
This paper also accompanies the kick-off to the MIRACLE project, which
is a major effort to implement many of the concepts presented here in experimental
prototypes. A number of specific subprojects dealing with applications
of MIRACLE are underway. Details on these projects will a reported in subsequent
publications and at the homepage of the MIRACLE project, http://www.MiracleProject.org.
A project called "Foundations of MIRACLE" has been submitted
for funding to the FWF Austrian Science Fund (www.fwf.ac.at).
Sources of images are as follows: Fig.1: [Sch73];
fig 2: [Hor98]; fig.3: see text; fig.4: public domain;
fig.5-9: [Ho00]; Fig. 10: [Cha96];
fig.11,12: [IH99]; fig.13: [Mo];
fig.14,15,18: [Ch00]; fig.16,17(adapted): [Lem01];
fig.19-21: [Ho00]; fig.22,23: [BP95];
fig.24: [Do02]; fig.25(adapted): [Sutton];
[ACP98] Albacete P.L., Chang S.K., Polese G.: Iconic
Language Design for People with Significant Speech and Multiple Impairments;
in Mittel V.O. (ed.): Assistive Technology and AI, Springer LNAI 1458 (1998),
[Ar70] Arnheim, Rudolf: Visual Thinking. Faber and
Faber, London 1970.
[Arm] Armstrong, David F.: Original Signs - Gesture,
Sign, and the Sources of Language; Gallaudet University Press
[BB82] Barrow H., Baker B.: Minspeak: A semantic
compaction system that makes self expression easier for communicatively
disabled individuals. Byte, Vol 7, No 9; 1982, pp. 186-202.
[BC80] Baker, C., Cokely, D.R.: American Sign Language
- A teacher's resource text on grammar and culture; Silver Spring; T.J.
[BDMY93] Beardon, C., Dormann, C., Mealing, S.
Yazdani, M.: Talking with pictures - Exploring the Possibilities of Iconic
Communication"; Association for Learning Technology Journal, 1, 1,
pp. 26-32, 1993.
[Bea92] Beardon, C.: CD-Icon - An Iconic Language
Based on Conceptual Dependency, Intelligent Tutoring Media, Vol. 3, No.
4, November, 1992.
[Bl49] Bliss, Charles K.: International Semantography,
Semantography Press, Sidney, 1949 reprinted in Semantography (Blissymbolics,.
Semantography Press, Sidney, 1965
[Bl84] Bliss, Charles K: The Blissymbols Picture
Book. Semantography Press: Sidney, 1984 see also http://home.istar.ca/~bci
[BL99] Becker Lee, Leemans Paul: VIL: A Visual Inter
Lingua; Third International Conference on Iconic Communication, Bristol
1999; appeared in [YB00]; see also http://www.cs.wpi.edu/~nemleem/IC99Paper.htm
[BP95] Boyes Braem, Penny: Einführung in die
Gebärdensprache und ihre Erforschung; Hamburg, Signum
[Br93] Brennan, Mary: Marking Time in British Sign
Language; in Kyle, J, & Woll, B. (eds.) Language in Sign: An International
Perspective on Sign Language, London 1983, Croom Helm
[BSM91] Baecker, Ronald ; Small, Ian; Mander, Richard:
Bringing Icons to Life; Proceedings of SIGCHI conference on Human Factors
in computing Systems, CHI '91, New Orleans 1991, ACM Press; pp. 1-6
[Ca34] Carnap, Rudolf: Logische Syntax der Sprache,
Julius Springer-Verlag 1934 (English translation: The Logical Syntax of
Language, New York, Humanities, 1937)
[Ch00] Champoux, Bernard; Fujisawa, Katsuki; Inoue,
Tomoyoshi; Iwadate, Yuichi: Transmitting Visual Information: Icons Become
Words; Proceedings of IEEE Symposium on Information Visualization 2000;
pp 244; see also http://www.mic.atr.co.jp/organization/dept3/papers/Cails/cails_paper.html
[Cha96] Extending Visual Languages for
Multimedia; IEEE Multimedia Fall 1996 (Vol. 3, No. 3), pp.18-26; see
[CHT02] Cavigliolo, Oliver; Harris, Ian; Tindall,
Bill: Thinking Skills & Eye Q; Network Educational Press, 2002.
[ConLang] Links to Constructed Languages mentioned
(the Internet is generally the medium of choice for the communities to
foster ongoing development of constructed languages):
[CPOT94] Chang S.K., Polese G., Orefice S., Tucci
M.: A Methodology and Interactive Environment for Iconic Language Design;
Intl. Journal of Human Computer Interaction, Vol. 14, 1994, pp. 683-716
[Da94] Damasio, Antonio: Descartes' Error - Emotion,
Reason, and the Human Brain; New York; Grosset/Putnam, 1994.
[Da03] Davies, John (ed.) et al.: Towards the Semantic
Web: Ontology-Driven Knowledge Management; John Wiley 5 Sons 2003
[Do02] Dotter, Franz et al.: Grammatik der Österreichischen
GebärdenSprache, Forschungszentrum für Gebärdensprache und
Gehörgeschädigtenkommunikation, Klagenfurt 2002
[Dor94] Dormann, Claire: Self-Explaining Icons;
Intelligent Tutoring Media, Vol 5, No 2 (1994), pp. 81-85
[DV01] DeVaul, Richard W.; Schwartz, Steven J.; Pentland,
Alex "Sandy": MIThril: context-aware computing for daily life;
The Media Laboratory, MIT; http://www.media.mit.edu/wearables/mithril/MIThril.pdf
[Eco97] Eco, Umberto: The Search for the Perfect
Language; Blackwell Publishers 1997
[Ei85] Eisner, Will: Comics & Sequential Art,
Poorhouse Press 1985
[Fei02] Feiner Steven K.: Augmented Reality A New
Way of Seeing; Scientific Amercian, April 2002, issue 4/24/02; see also
[Ge00] Gershenfeld, Neil A.: When Things Start to
Think; Owl Books 2000
[Ham] HamNoSys (Hamburg Notation System) 4.0 (a draft
working paper): http://www.sign-lang.uni-hamburg.de/Projekte/HamNoSys/HNS4.0/englisch/HNS4.pdf
[HB02] Hartmann, Frank; Bauer, Erwin K.: Bildersprache
- Otto Neurath - Visualisierungen; Wiener Universitätsverlag 2002
[Ho00] Hollosi, Arno: Alternative Representations
and Beyond: A New Proposal for a Multi-Sensory Language Interface (MUSLI);
Master's Thesis in Telematics, Graz University of Technology; see also
[HO01] Holden Eun-Jung, Owens Robyn: Visual sign
language recognition; in Multi-Image Search and Analysis. Klette, R., Huang
T., and Gimel'farb, G. (Eds) Lecture Notes in Computer Science 2032, Springer
2001; pp. 270-287
[Hon99] Honeywill, Paul: Visual Language for the
World Wide Web; Intellect Books, 1999
[Hon00] Honeywill, Paul: A Comparison Between Maya
Hieroglyphs and Computer Icons; AI and Society 2000; see also http://www.w3icons.com/pdf/iconsglyphs.pdf
[Hor98] Horn, Robert E.: Visual Language - Global
Communication for the 21st Century, Marco VUPpress, 1998
[IH99] Ingen-Housz, Timothee: The Elephant's Memory
- In Search of a Pictorial Language; Learning Technology Review , Spring/Summer
1999, Apple Computer online publication: http://www.apple.com/education/LTReview/spring99/elephant
[JGM93] Jonassen, H., Goldman-Segal R., Maurer,
H.: DynamIcons as Dynamic Graphic Interfaces: Interpreting the Meaning
of a Visual Representation; Intelligent Tutoring Media, Vol. 6 (3/4); 1996,
149-158; see also http://www.iicm.edu/iicm_papers/dynamicons
[Kad95] Kadous, Waleed: GRASP - Recognition of Australian
Sign Language using Instrumented Gloves; University of New South Wales
, Australia, Oct 1995; see also: http://www.cse.unsw.edu.au/~waleed/thesis/thesis.html
[Kos94] Kosslyn, Stephan M.: Image and Brain - The
resolution of the imagery debate. Cambridge, MA: MIT Press 1994.
[Lem01] Leemans Paul: VIL - A Visual Inter Lingua;
dissertation, Worcester Polytechnic Institute; see also http://www.wpi.edu/Pubs/ETD/Available/etd-0424101-005711
[Len95] Lennon, Jennifer: Mental imagery and visualization:
From the mind's machine to the computer screen; in H. Maurer, (ed.): Proc.
ED-MEDIA'95, AACE, Virginia (1995), 27-32.
[McCl93] McCloud, Scott: Understanding Comics,
[McCl00] McCloud, Scott: Reinventing Comics, Harper
[Me91] Mealing, Stuart: Talking pictures. Intelligent
Tutoring Media, Vol 2, No 2, (1991), 63-69.
[ML94] Maurer, Hermann; Lennon, Jennifer: MUSLI --
A MUlti-Sensory Language Interface; Proc. ED-MEDIA' 94 (best paper award),
AACE, Virginia (1994), 341-348.
[ML01a] Maurer, Hermann; Lennon, Jennifer: MUSLI:
A hypermedia interface for dynamic, interactive and symbolic communication,
JNCA Vol 24, 4 (2001), 273-292
[ML01b] Maurer, Hermann; Lennon, Jennifer: Augmenting
text and voice conversations with dynamic, interactive abstractions using
P2P networking; JNCA Vol 24, 4 (2001), 293-306;
[Mo] Denis Moskowitz' Rikchik homepage at http://suberic.net/~dmm/rikchik/intro.html.
[MO03] Maurer Hermann, Oliver Ron: The future of PCs and implications on society; J.UCS Vol 9, No 4 (2003), to appear
[MW98] Marsh Timothy, Watt Alan: Shape Your Imagination
- Iconic Gestural-Based Interaction; IEEE Virtual Reality Annual International
Symposium, Atlanta, GA, IEEE Computer Society
[MY90] Mealing, Stuart; Yazdani, Masoud: A Computer-based
Iconic Language; Intelligent Tutoring Media, Vol 1, No 3 (1990), pp. 133-143
[Neu91] Neurath, Otto: Gesammelte bildpädagogische
Schriften, herausgegeben von Rudolf Haller und Robin Kinross, Vienna; Verlag
[Ny02] Nyíri, Kristóf:
Pictorial Meaning and Mobil Communication, Conference 2002,
"Communications in the 21st Century"; a German version is available in
Nyíri, Kristóf (Ed.): Allzeit zuhanden. Gemeinschaft und Erkenntnis im
Mobilzeitalter, Passagen Verlag 2002
[Og30] Ogden, Charles Kay: Basic English - A General
Introduction with Rules and Grammar, Treber, London (1930); see also http://www.basiceng.com/
[Pa96] Paivio, Allan: A Theoretical Analysis of the
Role of Imagery in Learning and Memory, 1996
[Pi94] Pinker, Steven: The Language Instinct. The
New Science of Language and Mind, Penguin, Harmondsworth 1994
[Pr53] Price, Henry H.: Thinking and Experience (1953)
[Rh00] Rheingold, Howard: The Virtual Community -
Homesteading on the Electronic Frontier; MIT Press; revised edition 2000;
see also http://www.rheingold.com/vc/book
[Ru19] Russell, Bertrand: On Propositions - What
They Are and How They Mean (London, 1919)
[Sa29] Sapir, Edward: The Status of Linguistics as
a Science"; Language 5:209, 1929
[Sch72] Schank Roger: Conceptual Dependency - A
Theory of Natural Language Understanding, Cognitive Psychology, 1972, 3,
[Sch73] Schank Roger: Computer Models of Thought
and Language; W. H. Freeman, San Francisco, 1973.
[Schu] Schulmeister, Rolf: ViSiCAST: Virtual Signing,
Capture, Animation, Storage and Transmission; http://www.izhd.uni-hamburg.de/pdfs/ViCa.PDF
[SH01] Sellen, Abigail J.; Harper, Richard: The Myth
of the Paperless Office; MIT Press
[So94] Solso, Robert L.: Cognition and Vision. London,
England and Cambridge, Mass. 1994
[Sto] Stokoe, William C.: Language in Hand - Why Sign
Came Before Speech; Gallaudet University Press; http://gupress.gallaudet.edu/bookpage/LIHbookpage.html
[Sutton] Sutton, Valerie: Sutton SignWriting; http://www.signwriting.org/forums/linguistics/ling004.html
[TR00] Tonfoni, Graziella; Richardson, James: Writing
As A Visual Art; Scarecrow Press
[US87] Umiker-Sebeok, Jean; Sebeok, Thomas A.: Monastic
Sign Languages, 2 Volumes, Berlin: Mouton de Gruyter, 1987.
[vW65] van Wijngaarden, Aad: Orthogonal design and
description of a formal language. Technical report, MR 76, Mathematisch
Centrum, Amsterdam, 1965
[Vy34] Vygotsky, Lev S.: Thought and Language (1934)
reprinted 1962, M.I.T. Press
[Wh40] Whorf, Benjamin Lee: Science and Linguistics
(1940), reprinted in: Language, Thought & Reality. Cambridge, MA: MIT
[Wi74] Wittgenstein, Ludwig: Philosophical Grammar
(1974) Edited by Rush Rhees. Translated. by Anthony Kenny; Oxford, Blackwell,
[Worldsign] Homepage of Worldsign / Symbolvision
by David Orcutt: http://www.slocanlake.com/worldsign/
[Yaz93] Yazdani, Masoud (Ed.): Multilingual
Multimedia - Bridging the language barrier with intelligent systems;
Intellect Books, 1993
[YB00] Yazdani , Masoud (Ed.), Barker, Philip (Ed.):
Iconic Communication; Intellect Books, 2000