Micro Visualisations

How can Micro Visualisations enhance text comprehension, memorability, and exploitation?

Jonas Parnow@zeto Potsdam University of Applied Sciences Originally submitted in June 2015. Updated in October 2015. Technical fixes in October 2023. Supervisors: Prof. Dr. Marian Dörk and Prof. Boris Müller This is the web version of the design master thesis. Download the PDF of the thesis. Download the IEEE-accepted paper and poster.

Abstract

In a novel approach, which is mainly based on the disciplines of (data) visualisation and typography, this thesis discusses visualisations as a means to enrich text in regard of its comprehensibility, memorability, and exploitation. A taxonomy is proposed that differentiates specific types of application visualisations may have in this context.

Drawing upon the taxonomy, the thesis elaborates two approaches to aligning a text’s visual appearance and its content. The first explores the addition of graphical elements right within or adjacent to a text while the other approach explores the optical modification of a text by means of visualisation. For this I evaluate how typographic techniques can be used as visual variables. Along with the proposed systematic I introduce the term »micro visualisation« describing the non-complex enhancements made to the »amorphous« text.

In this work I summarise the historical derivation of text and images from a media-theoretical perspective, discuss my approach in regard to the current state of the two disciplines of visualisation and typography, introduce a taxonomy in order to declaratorily categorise existing work and conclude with an analysis of opportunities and challenges for future work.

Zusammenfassung

Geprägt durch die Disziplinen der (Daten-)Visualisierung und der Typografie, diskutiert diese Arbeit einen neuartigen Ansatz, der mit den Mitteln der Visualisierung Texte in Bezug auf ihre Erschließung, Merkbarkeit und ihre Ausbeutung. Eine Taxonomie wird vorgetragen, die es erlaubt die unterschiedlichen Anwendungen von Visualisierungen in diesem Kontext zu kategorisieren.

Basierend auf der Taxonomie verfolgt die Thesis zwei Ansätze, um das textliche Erscheinungsbild und den Inhalt des Textes in Einklang zu bringen. Der erstere untersucht, wie das Hinzufügen grafischer Elemente direkt in den Zeilen oder dem Text beigestellt gestaltet werden kann. Der zweite Ansatz erforscht das optische Modifizieren des Textes mit den Mitteln der Visualisierung. Dafür werte ich die Zweckmäßigkeit von typografischen Techniken als Visuelle Variablen aus. Im Rahmen dieser Methodologie führe ich den Begriff »Micro Visualisierung« ein, der diese un-komplexen Verbesserungen am »gestaltlosen« Text vereint.

In dieser Arbeit fasse ich die historische Herleitung von Text und Bild aus einer medientheoretischen Perspektive zusammen, diskutiere meinen Ansatz in Bezug zum aktuellen Stand der Forschung in den zwei jeweiligen Disziplinen Visualisierung und Typografie. Desweiteren kategorisiere erläuternd bestehende Arbeiten anhand der eingeführten Taxonomie und schließe mit der Analyse von Möglichkeiten und Herausforderungen für zukünftige Arbeiten.

Acknowledgement

During the preparation of this thesis I have received support in many ways. Although the following list is by no means complete, I would like to express my gratitude to some people and institutions in particular.

  • I would like to thank both of my friends Paul Heinicker and Kim Albrecht for their constant feedback, advices particularly for the theoretical part, and rendering design more than a subject. Through the frequent, (mostly) critical discussions in and outside of the university I learned more than any course could have taught me.
  • I would like to thank my mother Ulrike Parnow, my sister Hanna Parnow, and Hans Matthes. Especially my sister helped me to revise the importance of financial success and ethical values.
  • I would like to thank Lara Junold for the long term friendship and especially for the proofreading of this thesis.
  • I would like to thank Hanna Weyer, Delia Kämmerer, and Yaiza Kuhlmann for showing me the life outside of university and work during my time in Berlin.
  • I would like to thank the University of Applied Sciences Potsdam for the integration of design and media studies. Also the Urban Complexity Lab, in particular Sebastian Meier and Jan-Erik Stange, who strongly influenced my studies. I also thank my colleagues at Golden Section Graphics, especially Klaas Neumann.
  • I would like to thank my interview partners Prof. Dr. Jan Distelmeyer, Sascha Venohr, and Jan Schwochow. I also thank Prof. Dr. Malte Zimmermann, Prof. Dr. Dr. h.c. Jan Assmann, Prof. Dr. Wolfgang Behr, Jonathan Corum, and Alberto Cairo for their help.
  • Finally, I would like to thank my first supervisor Prof. Dr. Marian Dörk. Not only during the completion of my thesis, but also during my entire time at the University, he strongly influenced my studies with his knowledge and engagement. I also thank my second supervisor Prof. Boris Müller, who granted me his time despite his many responsibilities at the University of Potsdam.

Table of Contents

Introduction

This chapter introduces into the general idea of my thesis. Correspondingly, I provide an outline of my motivation to discuss the topic of »visualisation«. The reader is introduced to the underlying structure of this work and the relevant terminology. Finally, the perspective from which this thesis’ arguments are developed is specified.

The story of communication is a story of changing media. And while the technology processes, the principles stay the same. From all the communication channels available to humans, this thesis focuses on the visual ones. More specifically on images, text, and the overlap of these two. Despite their common origin, these forms drifted apart in their application and thus truly integrated combinations are rarely used. One exception are information graphics, that combine all types of media to convey information: text, illustrations, photographs and visualisations.

Visualisations find application in all kinds of fields and environments nowadays. They work as an alternative or as an enrichment for notational explanations, where the amount of quantitative data or the underlaying qualitative structure exceed the linguistic or numeric cognitive capacity. As external visual cognition aids (Card et al., 1999) visualisations organise, equalise, and simplify the comprehension of information. But despite these – proven – benefits, visualisations are not used in a lot of media. In fact, many publications completely adhere to pure text, photographs or tabular representations. Newspapers for example stick to the traditional journalistic of tools text and photography but rarely explain situations graphically.

Even though the technology we use to exchange text in order to communicate with others has dramatically changed over the course of time, the basic techniques have not changed. In fact the book’s layout has not changed much since Gutenberg’s invention of the letterpress. It might be possible that we already found a perfect tool for knowledge transfer in the book, as Roland Reuß suggests in »the perfect reading-machine«. (Die perfekte Lesemaschine, Reuß, 2014) This belief may be due to conventions in the society and is probably advocated by a group of nostalgic book designers. But in the same way language changes through social, cultural, and technological evolution, the design of media that transport language needs to adapt as well. This thesis has no intentions of denying the book, but instead takes the book as starting point for a new design space, in which current communication, information, and education behaviour is reflected.

This work aims not at creating new types of visualisations or new typographical techniques; it rather elaborates

  • how the existing visualisation types can be applied in a different way to suit our reading conduct and
  • what typography can learn from visualisation.

The proposed approach creates a second layer that complements the first layer (the content in its »standard« formatting). This layer may act

  • as a meta-layer, revealing the inner structure,
  • as an amplification-layer, supporting, contextualising, or questioning the written statements visually,
  • or as a detail-layer that adds deeper or new information to the text.

Motivation

The visualisations produced in scientific research are often complex graphics with many data points, visualised in an abstract manner. I am concerned that these representations are too computer-driven and too complicated to understand for a layman. While such graphics are undoubtedly valuable and useful for experts, this thesis aims to bring visualisations to a broader audience. It seems that primarily textual and primarily graphical media are drifting apart. My approach brings this two worlds back together: by creating simpler, smaller, and easier understandable graphics that can be used in daily life. Even though definitions of the term »visualisation« are rather broad, the actual application is often strict in their tools. This thesis endorses such an understanding of visualisation in order to consider every graphical decision as a process of visualising information. Thereby, it contributes to a deliberated and coherent application, which is often missing in current visualisations.

Structure

The thesis is structured as follows: the first part describes a historical derivation from a media-theoretical perspective that leads from objects and symbols, to words and letters, as to images and graphics. The next part describes the disciplines most relevant to this thesis: typography and visualisation. Drawing upon existing theoretical and practical scientific research on related topics, a reasoning is developed that leads to the approach of micro visualisation in this top-down method. The third part introduces a taxonomy in which existing proposals are declaratorily categorised. This bottom-up perspective allows me in the next part to identify practical possibilities and challenges for the general concept of micro visualisations. In the fifth and last part I show the essence of interviews with experts from related disciplines, give a summary of the findings from previous parts and an outlook for future work.

Terminology

The term »visualisation« is used with a broad definition in this thesis. In fact, one concern of this work is to argue for this widening of what visualisations can be considered besides classical diagrams or graphs. The term is often used synonymical or in combination with »data (visualisation)«, which often implies or is accompanied by a great amount of quantitative values. Meanwhile, it should also apply to small amount and even qualitative values.

Stetter defines the diagram as follows: »It is […] an exhibitive representation of circumstances with graphical means« (»Es ist […] die exhibitive Darstellung eines Sachverhalts mit graphischen Mitteln.«) (Stetter, 2005) This broadens the concept to maps, schemes, sketches, and blueprints but also to any usage of graphical means, even when no optical »bounding box« defines it as figure or differentiates it from other elements. It is orientated towards the basic meaning of visualising: to make (something) visible to the eye. (Oxford Dictionary)

The term »micro« is chosen in front of a lack of a more encompassing and at the same time more specific term. The way it is used in this thesis could also be seen as »minimalistic« or »fragmentary«: it aims to describe visualisations that are not only small in physical space but can also be narrowed in terms of data dimension or points. Visualisations that have one or multiple of these characteristics are considered as micro. Hence »micro visualisations« are basic graphical modifications or additions, that enhance the comprehension of text.

Positioning

As designer my approach in this thesis is mainly driven by research issues related to the subject design. But since this discipline (and the one of information design in particular) not only touches, but also needs to consider other disciplines like journalism, media studies, computer science, linguistic, and statistics these prospects will hopefully be fulfilled with appropriate care and expertise.

This thesis aims to contribute to the current level of knowledge of (data) visualisation. The overarching endeavour is to expand the understanding of how micro visualizations are perceived and how they can help the reader to comprehend information. The second field, typography, is also expanded in how its techniques can be applied, but is not equipped with new ones.

Even though the results of this thesis can hopefully be applied all over the world, its foundation and especially its examination of the evolution of images, language, and writings mainly relate to a central European point of view and are therefore highly curated and partial.

Derivation: Four turns until presence

In this chapter I examine the evolution of communication within the media most relevant to this thesis: images and text. Their relation to each other is illustrated as well as their relative importance over time is considered. The chapter concludes with a revision of the current state of art and a proposal of how the newest form of images – visualisations – may enrich designers’ tools of communication.

Visual communication is essential in every aspect of our lives. Newborns first learn to connect basic visual cues such as shapes and colours to rudimental meanings. Accordingly a mother’s smile or a fire’s glowing appearance gain meanings that — among countless others — guide us from childhood on through a world dominated by visual information. These fundamental lessons determine our cognitive development more than the four other senses that we experience from our surroundings. Only after we internalise the interpretation of myriad visual impressions, we learn how to express these things in words.

An arbitrary form of communication

Words — first spoken, later written — have leveraged mankind’s evolution. With the ability to reference objects, express intentions, or describe relations we gained the ability to communicate about abstract systems and ideas with others. This development — starting with visual cues, the addition of spoken words and finally the expression of these words in (visual) writings — is something not only developed by humankind over the course of thousands of years but also something every child still runs through nowadays. Our long term memory enables us to store the information we gather in our lifetime to rapidly identify words and thousands of visual objects. (Ware, 2004)

Petroglyphs Giraffe, lion, and others in Namibia, Africa. Photograph by Greg Willis, 2006
Symbolism The Premier Leagues uses the symbol of the lion.

Observing this development, it is important to note that these forms of expression and impression are highly connected and never exclusive. The development of language and writing/reading are greatly influenced by each other and were initially inspired by the actual physical appearance. The Egyptologist Jan Assmann interprets writings by the Philosopher Moses Mendelssohn about this development as follows: The first step was an »iconic turn«. From the things itself to images of the things. Examples are cave paintings from the Paleolithic Age (18,000 BC) and the Petroglyphs (πέτρος petros »stone« and γλύφειν glýphein »to carve«) from the Neolithic Age (10,200 BC – 4,500 or 2,000 BC). (Borgo et al., 2013) These were pictograms, which were painted or carved in rock faces all over the world. Mendelssohn exemplifies this progress through the symbol of a majestic lion that gained its meaning from the actual lion.

Timeline From the birth of mankind until today

The second step is an »uniconic turn«, in which these symbols detach from their original reference objects and become words. (Assmann, 2002) René Magritte’s »The Treachery of Images« (« La Trahison des images ») illustrates this relationship between the signifier (signifiant) and the signified (signifié). The writing »This is not a pipe.« (« Ceci n’est pas une pipe. ») advises the beholder that the image he is seeing is not a pipe but instead the image of a pipe. The painting (signifier) is not capable of showing an actual object (signified), but rather — as Platon describes it — the idea. Furthermore the »lapidary truth of the sentence and the non less lapidary evidence of the illustration possess no common level. What we truly see, is not said in the linguistic statement in any way. What we truly read, is not confirmed by the eye.« (Boehm, 2014) The two signifiers — image and writing — are competing with each other. The once visually inspired language emancipated itself to abstract words.

semiotics The painting is only the idea of the actaul object

Today, two forms of »word-writings« can be generally distinguished: Pictorial and alphabetical systems. (Frutiger, 1998) The Chinese characters for example have a great amount of signs whose appearance is directly derived from the pictorial representation of an object. Their shape is motivated from their original form. In contrast alphabetical characters can no longer be assigned to semantic meanings on their own. Even though they have a »hieroglyphic memory«, their form became arbitrary. (Assmann, 2002) Instead, they represent tones, which gain meaning in their sequence.

Discursiveness as recipe for success

In Europe, the ability to write words with these »building blocks« pave the way to Gutenberg’s milestone invention of the letterpress. Instead of writing letters one by one to a single book, a composing galley is prepared once with movable types and can then be used to print multiple copies. While manuscripts have always leveraged the spreading of knowledge, with the development of the letterpress the book could finally break open this bottleneck of dissemination of knowledge where one friar could manufacture one book during his lifetime. It was a revolution on two levels: the rising availability of affordable, transportable, and uniform knowledge and the first serial production from the economical perspective. (Hausmann, 2009)

The book’s status influenced communication and the dissemination of knowledge so drastically that the Canadian philosopher of communication theory Marshall McLuhan declares the book as the leading medium of recent times. The fact that the Times magazine declared the German blacksmith as the most important person of the last millennium (Hausmann, 2009), supports McLuhan’s term of the »Gutenberg Galaxy«. It describes the great effects the book had not only on social interaction and conditions, knowledge transfer and research, alphabetisation and language standardisation, but also on the thinking itself. Thinking and designation were directly referring to each other before the domination of the book. With the optical fixation the language crowds between these two. (Assmann, 2002) The reading of sequenced letters, words and paragraphs demands and facilitates linearity in contrast to spatiality, discourse in favour of simultaneous recognition and capacity for abstraction instead of affectivity. (Assmann, 2002)

Linguistic Turn The language changes the way with think.

These effects were recognised as the »linguistic turn«, in which language as a medium of recognition and description of reality was critically studied. Ludwig Wittgenstein summarises these deliberations with the limits of his language as the limits of his world (Wittgenstein, 1922). This reveals how human’s devotion from images to language also implicates their comprehension and expression. This has been researched as »Linguistic Relativity« first by Wilhelm von Humboldt, later by Benjamin Lee Whorf and Edward Sapir, whose work is now summed up under the term Sapir–Whorf hypothesis. Research in this area has been done by Berlin and Kay (1969 in Ware, 2004) and Eva Heller (2009), who specified this analysis for colour designation and perception.

The world in images

Today the end of the Gutenberg Galaxy is discussed in various fields. The world shaped and controlled by electronic technology is »imploding« (McLuhan, 2011), the sender-receiver model is outdated and image generating media is becoming ubiquitous. Norbert Bolz exemplifies this with the rising use of photographs in newspapers: While many traditional newspapers rejected images for a long time not because of technical reasoning but rather conservatism, purely textual newspapers are unthinkable nowadays. (Bolz, 2012)

But mass media is not the only field affected. In opposition to the linguistic turn, the »iconic turn« was formulated in the 1990s. While the actual definition is still not finalised, it circulates around questions in a world more and more dominated by images and image technologies. (Bachmann-Medick, 2008) William Mitchell and Gottfried Boehm individually described and analysed the process with different terms and perspectives. Both the pictorial- (Mitchell) and iconic-turns (Boehm) emphasise the importance of images not only as references to objects but also as objects of analysis itself. (Bachmann-Medick, 2008) Even though the linguistic turn considered this impossible, it leads to not only thinking about images, but rather to thinking with images. (Bachmann-Medick, 2008)

But this newly formed visual culture (Bildwissenschaften) does not want to replace the previously proclaimed linguistic turn, but rather describe the »image as another way of thinking«. (Boehm, 2007) Prominent examples of analysis in this discipline is the use of images to »display« (Sichtbarmachung) enemies in so called »enemy images« (Feindbilder) (Kittsteiner, 2004) and use of photographs as »witnesses« for the Holocaust. (Didi-Huberman, 2006) In contrast in the book »Cosmigraphics: Picturing Space Through Time«, Michael Benson analyses the view of the world in images through history.

While these examples refer to the displayed content of images, another object of research is the figurative, graphical level. Even though this field has been known and used by painters and photographers for a long time, recent fundamental research especially in psychology opened up the field of perceptual psychology. Mostly evolutionary reasoned humans perceive visual information on different levels. From the beginning of humankind, survival depended on quick and profound decisions. Danger needed to be spotted quickly and food needed to be identified correctly. The human brain adapted to these factors with various visual pattern recognition techniques, which are based on basic visual forms. We search for useful possibilities for action and vision allows us to »perceive affordances of the environment directly, not indirectly by piecing together evidence from our senses.« (Ware, 2004)

These recognition skills can be still used today: The colour red still represents danger and creates attention (Heller, 2009), moving objects could become a threat and occupy our alertness (Ware, 2004), and bigger (closer) objects seem more important. (Lidwell et al., 2010) In fact, the Gestalt principles (Todorovic, 2008) are based on our evolutionary shaped perception.

Functional images

Completely based on this figurative and graphical level is the especially in recent time widely used discipline of visualisation. These abstract images utilise basic graphical forms to communicate information. Information that could not or only with high difficulties be communicated through language or even »concrete« images. While graphics, which are nowadays labelled as visualisations or information graphics, were already created in the last century, computers accelerated the creation of these »mathematical« images. Bar charts, pie charts or scatterplots became known to the general public through software like Microsoft Excel.

Apart from the general public a community of experts, consisting mainly of designers, mathematicians, statisticians and computer scientists, push the limits of the multidimensional data mapping. Computer-driven graphics and scientific environments move the discipline further away from the understanding of non-experts. While some great graphics make it into popular media, visualisation still remains far behind its potentials. Just like newspapers rejected photographs in the beginning in favour of Gutenberg’s legacy, knowledge mediators nowadays withhold from the usage of visualisations.

At this point my thesis starts. The avant-garde of newspapers like the New York Times and of magazines like National Geographic as well as early-adopters who take part for example in the Quantified Self movement bring visualisation to people outside of the community. In the same way I want to help to move visualisations away from its expert driven presence to everyday life. For this, my approach is to position graphics right at the point where currently still most knowledge is still imparted: in the written text. And while many realisations present a divided back and forth between graphic and text, my approach is quite literally by breaking up the boundaries between each medium.

My goal is to create Boehm’s »union of power« (Herrschaftsunion) between the linguistic and the iconic turn to create a fertile juxtaposition instead of opposite positions. (Bachmann-Medick, 2008)

Theoretical Foundation

This chapter draws upon the previously introduced tools of communication, i.e. text and image. These are evaluated in regard to the disciplines of typography and visualization, in which both concepts have developed rather separately. Against this background I argue that typography is in fact a variation of visualisation and that, accordingly, text and image should be perceived as complementary tools, which should be applied in unity. Serving this proposal, I characterize concerned disciplines and demonstrate my argument’s relevance from cultural, technological as well as design research perspectives.

What is communication?

When one tries to define what communication really is, he quickly comes to the conclusion that everything we do is communication. »Activity or inactivity, words or silence all have message value: they influence others […].« (Watzlawick et al., 1967) As designers, we shape many aspects of this wide field and help other people to communicate the way they wish to. Viewed with an even broader definition of design, humankind as designer has developed a wide variety of communication channels, ranging from body language and spoken language over to written and printed type, and finally to electronically processed communication. Each of these steps marked a milestone in mankind’s history, but none could completely replace previous mediums. Even though some of the utilised technology for each communication channel might have been replaced over time, the basic concept of communication stayed preserved.

There are two reasons for that: First, every form of communication has its own qualities, advantages, and applications at different settings, and second, each development rests upon the former one and therefore relies on the common understanding of the previous one.

To emphasise this relation even further, Paul Watzlawick’s famous quote »One cannot not communicate« (Watzlawick et al., 1967) expands the definition of what communication is. He argues that there is no opposite of to communicate and thus makes »communication synonymous, or nearly synonymous, with behaviour.« (Motley, 1990) This implicates that at least one of the communication channels available to human is always active. It also facilitates the theory of multi-layered communication and thus that a single form of expression is seldom and a single form of impression is impossible. The following example illustrates this: When one reads a text, one is not just reading. He is also seeing how the text is presented. The written, printed or displayed words are only one part of the information that is received. The material, the layout and the many aspects of typography all communicate on their own and thus influence how and what information is perceived from the actual text. Marketing knows that and defines CIs consisting of corporate design, language, sound, smell, and many more aspects that all strengthen the stringent branding of the company.

What can be taken from this is the broad understanding of what communication is. To use this methods not to make money but to impart knowledge to others through a wide variety of means of communication.

What is Typography?

The importance of the appearance of the text is well known and generations of type designers and typographers developed a variety of techniques to design and enhance the »text-seeing« and reading process. It is »the art, or skill, of designing communication by means of the printed word« (Childers et al., 2002) But even before books were printed, a highly elaborated discipline had formed around type design. (Drucker, 2014) Today typography is an inherent and essential part in design education. The variety of type design principles and typographic variables need to be thoughtfully applied to the text to comply with the content but also to be understood by the reader. Some of these principles are conventions, which evolved over time (italic and regular fonts), others are based on fundamental decisions such as reading from one side in one direction (justified, flush left, or centered text), while others base on perceptional reasoning (font size and weight).

Carried by this unique craft, written and later – with Gutenberg’s invention – printed books came to their prominent status they hold today. But the importance of typography is not justified by the decoration of the text, but rather the readability of the text and more fundamentally by the supportive character the design plays for the content of the text. Robert Bringhurst exemplifies this for the typeface: »When the type is poorly chosen, what the words say linguistically and what the letters imply visually are disharmonious, dishonest, out of tune.« (Bringhurst, 2012) To clarify, honour or disguise the content is seen as the most important role typography plays. But with the leverage its application has, it can also be misused: »In a world rife with unsolicited messages, typography must often draw attention to itself before it will be read.« (Bringhurst, 2012) This implies the consequences that typographic decisions have on the visual appearance and hence on the very first impression the reader has of the text. With other words: »Type is persuasive – and thus so is typography […]« (Santa Maria, 2014)

Readability-Model by Ralf Herrmann in a simplified version (2010)

Besides the used material, the selection of an appropriate typeface is the first and most fundamental decision when approaching a text. With the selection of one typeface the designer takes over many decisions that were previously made by the type designer. It may also demand a certain typographic application to suit its appearance. The explanation of the complete process of letter and type design would go beyond the scope of this study, but what is relevant for this thesis is the great impact the selection of a typeface has for the perception of the text. Stronger than any other typographic decision, the typeface affects many aspects of the reading experience. First the readability consisting of reader-friendliness (Lesefreundlichkeit), distinguishability (Unterscheidbarkeit), recognizability (Erkennbarkeit), and legibility (Leserlichkeit) (Herrmann, 2010) and secondly — even more prominent — the connotations that come with the typeface. From a consumer psychology view »typefaces convey meanings that have the potential to significantly influence important marketing constructs. These associations influence how consumers perceive brands, as well as, what they remember about brands.« (Childers & Jass, 2002) Taken the book as a product that represents the text, it becomes clear that we perceive and remember the content differently depending on the chosen typeface.

With this most obvious decision in typography as introduction, further »tools« that are available for typographers are for example margin, line-height, letter-spacing or letter case. A full exploration of these variables is discussed in relation to »visualisation« further down.

What is Visualisation?

The power of photographs has been used for a long time. The alleged objectivity induces publishers to show »the single truth«. The pretended evidence of the image reduces complex matters to single statements. (Bolz, 2012) As mentioned above, nearly every newspaper nowadays relies on the impact a photograph can have for a story. But with this practice problems arise concerning the photographability of certain subjects. (Bolz, 2012) Intangible and abstract problems cannot draw much attention to itself, because they cannot be captured in a photograph or an illustration. This becomes even more apparent in the context of social media, where posts with images reach higher click rates. Articles that do not show their content with attractive images cannot arise much attention. One answer to these undepictable subjects are visualisations, which encode abstract data into tangible graphics. Previously »invisible« and highly complex processes gain shape through abstract graphical elements. Thus visualisations are used for information explanation.

photographability How to visualise climate change? Photograph by Carla Lombardo Ehrlich for WWF

Visualisation is not a new field, but experienced an upturn in recent time. With abstract data becoming ubiquitous the challenge to make sense of this new resource became apparent in all industries. The term big data arised in the hope to find previously hidden insights from the analysis of great amounts of data. Besides classical statistical methods, visualisation is another promising and proven approach. For this purpose data visualisations are used for exploration. The statistician John Tukey already predicted 1965 »a greater and greater role for graphical techniques as aids to exploration and incisiveness« for his subject. (Tukey, 1965)

While the textual analysis of such an amount of data is not efficient, humans are very good at spotting pattern, abnormalities or trends on a visual level. (Few, 2014) Ware goes even further and says »There is no other way of presenting information so that structures, groups, and trends can be discovered among hundreds of data values«. (Ware, 2004)

Norbert Bolz speaks in this context about the concision (Prägnanz) of an image. This quality enables us to instantly see the content. (Bolz, 2012) In order to understand this process, the human sensory system needs to be examined. »The term sensory system includes all phenomena that manifest themselves in the consciousness as result of stimulation of any sensory organ«. (Haverkamp, 2009) This rather broad definition describes (all) the phenomena as something which is actively manifesting itself. This seems to be opposed to the widely established and accepted concept of constructivism, which describes how we perceive our surroundings on the basis of a theoretical model. In philosophy, anthropology and psychology — in contrast to constructivism in art — it describes the following process: When we observe objects, we reconstruct these as mental objects. These ideas are constantly matched to previous experiences and influences. These »images« of objects work as prototypes and can be recalled even with different appearance or after perspective transformation. (Ware, 2004) This implies that the human brain is no longer passive in its perception, but instead active in terms of selection, projection, interpretation and sense-giving: it constructs its world individually. (Maletzke, 1998)

This obviously raises questions regarding the »real reality«, which is a rather philosophical discussion that goes beyond the scope of this work. For designers, this model is interesting in regard of communication. Concepts of mass communications are no longer consistent with this theory. Every human needs to be viewed as an individual, from whom consistent reactions can no longer be expected. What is relevant for the purpose of this work and for the perception of visualisation is the active selection and interpretation of the presented image. Despite this, the passive perception of visual information is equally important and has been scientifically proven as well. The answer to this seeming contradiction lays in the way our visual sensory system works. Simplified it consists of three consecutive stages:

Parallel Processing In the first stage »billions of neurons work in parallel, extracting features from every part of the visual field simultaneously. This parallel processing proceeds whether we like it or not, and it is largely independent of what we choose to attend to […].« (Ware, 2004) This is one reason humans are good at comprehending large amounts of data points in short time. The parallel, rapid processing explains why visual information can be extracted quicker form graphics than from textual presentations, which have to be read serially.

Pattern Perception »At the second stage, rapid active processes divide the visual field into regions and simple patterns, such as continuous contours, regions of the same colour, and regions of the same texture«. (Ware, 2004) At this stage Gestalt principles play a decisive role. It is important to note that at this stage, the information is handled in serial, resulting in a much slower processing. At this point the long-term memory is already involved.

Sequential Goal-Directed Processing In the third stage »we construct a sequence of visual queries that are answered through visual search strategies«. (Ware, 2004) This means the active search for visual elements that fit the given task. For example, this could consist of connecting close located red points.

The last two stages involve active search, selection and interpretation of the given visual information where it is important to note that this »active« process is still processed mainly unconsciously. »Consider the eyeball as an information-gathering searchlight, sweeping the visual world under the guidance of the cognitive centers that control our attention. Information is acquired in bursts, a snapshot for each fixation. From an image buffer, the massively parallel machinery of early visual processing finds objects based on salient features of images.« (Ware, 2004)

In order to help spotting salient features, information designer should encode this information to »pop out«. Various studies analysed which encoding works best for which type of data. The first and most cited taxonomy of such kind was formulated by Jacques Bertin. In his book Semiology of Graphics (Bertin, Semiologie Graphique, 1983), the cartographer constrains his studies with some factors like printable, standard sized graphics on white paper. His work was later picked up by many scientists and expanded by unnoticed variables and the ones that were left out intentionally by Bertin, as he is rooted in cartography.

Bertin distinguishes seven forms of encoding. He describes these as »visual primitives« or as they are now commonly called »visual variables«, from which the designer can form the image to suggest perspective, reality, relationships or space. (Ware, 2004) These variables are: position, size, value (transition from light to dark), texture, colour hue, orientation and shape. I propose the following modifications for a better structure:

  • subordination of colour hue, value, and texture under appearance and
  • replacement of value with opacity, since the transition from light to dark can be treated as change of colour.

This results in the following taxonomy:

Visual Variables A modified version of Bertin’s visual variables

Bertin’s visual variables have often been extended, for example by MacEachren (2004) by crispness, resolution, and transparency. Chen and Floridi (2012) extended the list to 30 variables and subordinated them into four categories: Geometric, optical, topological and relational, and semantic channel. While this further definition is helpful it loses the idea of basic visual encodings.

Debatably all variables from the »Topological and Relational Channel« are rather results of the spacial positioning and the variables regarding the motion are also Bertin’s visual primitives combined with time. The concept of visual variables needs a second layer that covers such advanced cases and fill in the gap to the psychological Gestalt principles. While such an approach seems to be promising, it would go beyond the scope of thesis. Since Bertin’s approach still has a broader acceptance I will stick to the previously explained modified version of his taxonomy for this thesis.

Geometric Channels:

  • Size / length / width / depth
  • Orientation
  • Shape
  • Curvature
  • Smoothness

Optical Channels:

  • Intensity / brightness
  • Colour / hue / saturation
  • Opacity / transparency
  • Texture (partly geometric)
  • Line styles (partly geometric)
  • Shape / blur
  • Shading and lighting effects
  • Shadow
  • Depth (implicit / explicit cues)

Synthese:

  • Implicit motion / motion blur
  • Explicit motion / animation / flicker

Topological and Relational Channels:

  • Connection
  • Node / internal node / terminator
  • Intersection / overlap
  • Depth ordering / partial occlusion
  • Closure
  • Distance / density

Semantic Channels:

  • Number
  • Text
  • Symbol / ideogram
  • Sign / icon / logo / glyph / pictogram
  • Isotype

visual channels by Chen and Floridi (2012)

One helpful aspect of these visual variables is the ranking in regard to their effectiveness. This can be viewed from different angles. First is the ability to pop-out. Visualisation can help elements to stand out from the rest, making it easier to spot when searching for a certain one. Studies (Borgo et. al, 2013) reveal this order:

The second ability of visual variables can be measured by human’s perception to most easily decode the information »behind« the encoded data point. It is the variables’ ability to be definite. And also the effortlessness of comparison between multiple points varies between the channels. This aspect was also considered by Bertin: For instance, the position of an object along the x- and y-axis (the positioning in 3D spaces is being discussed in the visualisation community controversial) was described as the most powerful for extraction and comparison. This idea of effectiveness was then continued by Jock D. Mackinlay. He provided the sorting in relation to the given data type. According to S. S. Stevens, these data types are »nominal, ordinal, interval and ratio«. (Stevens, 1946) Even though his definition can be criticised, his work set a standard in statistics. For a clearer separation I will distinguish »interval« and »ratio« under »quantitative« data types.

Visual Variables A modified version of Bertin’s visual variables

Quantitative data can be measured by its amount. This category can be distinguished once more between interval and ratio. Interval data sits between two (arbitrary) points like 0° and 100° for Celsius temperature or values on a ph-scale. One needs to be careful, because even in case the gap between two values can be measured, 50° C is not twice as hot as 25° C. In contrast, quantitative data that is given in a ratio can be used for measuring and calculating for instance a number of people (1, 2, 13), or the weight of an object (1 pound, 14 kg). These values can be also measured as discrete (3 m) or continuous data (10.3 m).

Ordinal data cannot be measured quantitatively but instead in ordered categories. Given the example of a race, the gap between the first and the second place might not be the same as the gap between the third and fourth place, or half sufficient is not unsufficient. Quantitative data can be converted into ordinal data, but ordinal data cannot – without further information – be converted into quantitative data.

Nominal data is also categorised, but without any order and hence often referred to as qualitative. Numerical names »are used only as labels or type numbers, and words or letters would serve as well.« (Stevens, 1946) For example male is not better than female, Hindu, Muslim or Lutheran are equal, and also blood types like A, B, 0 or AB are not prioritised by default.

Typography as Visualisation?

»Letters were mainly regarded as a medium of communication; it is essential to recognise them as media of perception.«

 — Sybille Krämer, 2003

Writing as medium are fated in the same way every medium is considered: Aristotle already described that media of perception are indeed materialistic but at the same time need to be transparent. (Krämer, 2003) Fritz Heider continues this concept as he states that media make something visible which is not »of the ’nature‘ of the medium itself«. Media »aisthetisis« by »un-aisthetising« themselves (»Medien aisthetisieren, indem sie sich selbst an-aisthetisieren«). (Heider in Krämer, 2006) The media philosopher Sybille Krämer examines this relation between visibility and invisibility. In her ten theses about »notational iconicity« (Schriftbildlichkeit) she postulates the presentative instead of the representative perspective on letters. (Krämer, 2003)

The blurring of boundaries between presentation and means of presentation is also subject to the concrete poetry. Mostly through typographical arrangement of words »poems of figures« emerge. (Bauer et al., 2010) These figures raise questions on the similarity of the meaning and the optical appearance of the words. The distinction between figure and text can no longer be applied to this art form. In Eugen Gomringer’s poem the word »schweigen« (to be silent) has no similarity to the idea it represents, but illustrates its meaning through the arrangement.

Schweigen by Eugen Gomringer (Bauer et al., 2010)

In a similar way, my thesis highlights typography as a mean of communication for the text’s content and not only to serve as a tool for readability. To illustrate this problem in application: Journalists publish their articles in a newspaper. However, the typographic decisions were made long before the article was written by someone uninvolved in this particular or in every other article. The appearance of the text represents the newspaper, but not the text. Regulations on the font, the columns, use of emphases or layout may not be beneficial for the text.

In order to examine typography for visualising information, I discuss typography from a data visualisation’s perspective. The description of visual attributes like the ones by Bertin, Cleveland, Ware or Meirelles summarise typography as one visual variable. My hypothesis is that typography is not a subcategory like »size«, but rather a complete new »object« that can be modified with the visual variables described by Bertin and others. When comparing the visual primitives with basic techniques used in typography, one can see that these match in most cases concerning their implementation and effect. For example, a word can – just like a square – be scaled in size or changed in its position. But some major points need to be considered:

  • In typography there are even more visual variables available due to conventions like underlining or line-height.
  • It depends on whether it is applied to the whole paragraph, the sentence, the word, or only the letter.
  • Some of the »typographic variables« cannot be used without compromises on the legibility. In contrast, a rectangle in a regular visualisation may suffer in visibility but not in legibility.
  • In fact this is a complete new relationship between entity and label. Normally the entity is modified, while the label stays unaffected. In my approach the label is also the entity and thus requires careful modification.
  • Also, the initial situation is different since in regular visualisations each object represents one (or multiple aggregated) data point. There are no objects on the plane that look like data points, but hold no data. When applying visualisation to only some words or text elements, the other words remain in their original form and can act as »normal« for comparison.
  • The modification of any of the typographic variables influences the grey value (the perceived blackness from a distance) of the text. Sometimes, this is not intended. The change of the typeface for instance might hence not only have a semantic alteration but also a visual effect.

The table illustrates the typographic instruments for each form of encoding.

Shape

  • Typeface
  • Case / Style
  • Weight
  • Width

Size

  • Scale

Position

  • Kerning / Spacing
  • Indention / Margin
  • Alignment
  • Superior / Inferior
  • Line spacing

Orientation

  • Rotation

Appearance

  • Colour
  • Texture
  • Opacity


Additional variables

  • Underlining
  • Strike through

Typographic variables Typographic techniques sorted as visual variables
Shape

The shape of the letters cannot be mapped directly to a single typographical variable. There are multiple options to vary the font’s shape. The most radical would be the actual typeface. This categorical alternation should be treated with caution. Two major concerns need to be taken into account:

  • The distinction of different typefaces can be hard for the laymen. Serif, sans-serif, script, blackletter and decorative fonts might easily be distinguished, but within these categories non-specialists could hardly separate Renaissance, Baroque or Neoclassical.
  • Another thing to be concerned of is the inherent cultural association of that font. Blackletter fonts are commonly (and falsely) associated with the Third Reich, serif fonts with the antique and script fonts with a humanly touch. As these associations are mainly personal and individual impressions instead of facts, one should be careful when expecting the reader to link the contrasty, vertical orientation to an 18th century modern Romanism.

Another way to alternate the shape of a font is either its case (lowercase or uppercase) or its style: italic (and rarely left oblique), small caps and roman. This has been common practice since the sixteenth century (Bringhurst, 2012). This feature can only be used categorically since there is commonly no option of more or less italic, for example.

The third option is to use different font-weights. Depending on the amount of fonts the typeface consists of, the font weight is a feasible variable to visualise information. The weight of the font also influences the grey value of the text accordingly and is therefore – just like the colour value – able to communicate its value from first sight and from a greater distance by »popping-out«. In contrast to the shapes from regular visualisation, different font weight do have an order. The examples given by Bertin (rectangle, star, triangle, …) an not be ordered.

Another option is the width of the fonts. Some typefaces consist of extra fonts like condensed, narrow/compressed or extended ones. These fonts could be equally used as different weights, but since they are rather seldom (especially for antiqua typefaces), real world applications would be rare. Fonts should not be streched out programmatically in order to preserve their aesthetic. This is also why the width is not considered as size, but as shape.

One great possibility would be Multiple Master Fonts. These fonts can continuously interpolate between two »masters«. Possible axes can be weight, width, optical size and even style. With this technique individual interstages can be generated. Unfortunately, MM fonts could not establish itself and are hence not supported on current systems. Recent developments on this topic are JavaScript-based solutions such as Plumin.js, which allows to generate/manipulate fonts with JavaScript code. Prototypo is another approach that allows font manipulation with type designer variables like x-height, capital height, ascender and descender height, slant, overshoot, crossbar position, and width. Opentype.js even allows to directly manipulate anchor points. All these tools could potentially be used for visualisation with continuous interstates to represent values when they are further developed for this kind of application. The disadvantage of all of these tools is the purely mathematical generation of the fonts. The perception of letters and thus their design is not linear. This often leads to unpleasant results. Either the generation of fonts becomes more advanced and takes the non-linearity into account or the possible unaesthetic and illegible appearance needs to be accepted. Further development will show if these JavaScript-based tools can manage what the original Multiple Master tools could not.

Position

The position of a letter can be altered in various ways. On the x-axis the designer can change the kerning/spacing (the space between letters), the indention/margin (the space – usually – from the left-hand side) and the general alignment. On the y-axis the user can change the letters to superior/inferior or the general line-spacing.

In regard to single letters the spacing is a good option to communicate both qualitative as well as quantitative values. Despite the fact that exact decoding of the data is not possible, this technique allows for continuous data mapping. Another advantage of this method is also independency from the typeface: The spacing is applicable to every font. Obviously extreme values can cause illegibility or the drifting apart of letters. Shifting single letters in their y-position is already common typographic practise. For example, chemical, physical or mathematical notations use a shift for sub- and superscript to create a hierarchy between characters. What is important to note here is that because of the little possibilities of variation within the line height, the visualisation becomes categorical or discrete rather than continuous. In some cases the reader can only designate bottom, middle or top position instead of exact differences. The comparison is also difficult across multiple lines. As Cairo (2013) describes, objects with a common baseline are easier to collate. This also limits the modified amount of characters to keep the baseline within a line row. Another problem that could cause a lack of comparison on the y-axis is the different size of the letters. The ascenders and descenders suggest different starting points. It is therefore recommended to use capital letters to work with the same letter height.

In general, the repositioning can cause issues on the readability. Each letter, word, sentence and paragraph has an inherent position that is crucial for the content. The variance of the position is therefore limited and needs to be applied carefully. Letters and words are especially bound to their position.

The shift on the x-axis of sentences or paragraphs is already used. Lists in a table of contents for example use different positions to indicate its hierarchy. Again, a modest execution should be applied to display both categorical or quantitative data. One advantage of the continuous mapping is that people can instantly understand the technique.

An unique variable is the text alignment like flush left, justified or centered text. Spreadsheet software take advantage of this by aligning text on the left side, while numbers are aligned on the right side of the cell. Here, the cultural conventions need to be considered.

Appearance

As previously mentioned, colour has the strongest pop-out effect. In the commonly black and white surroundings of normal text, elements stressed with colour draw even more attention to themselves. On one side this can be used for advantage by using it to emphasise certain values, but on the other side it can also distract users from reading. Because colour can communicate information from a greater distance, it is often used in combination with text for way finding systems. While it allows to convey categorical information quickly, quantitative information is even more difficult to decode because of the small surface area. For this is advisable to use multi-hue diverging colour palettes since the size of the font might restrain the conception of diverse colours. When using the same colour on extensive objects and text, it should be applied to the latter with extra saturation or darkness to counteract the small area.

The use of opacity may cause problems for the legibility. Just like objects that do not stand out from the background are hard to be spotted, letters that are too similar to the background cannot be read anymore. Additionally, all changes of opacity (similar to colour) should be stressed especially on lightweight fonts to counteract the small surface area. A change of opacity is often more practicable when combined with a change of colour.

The application of a texture highly depends on the type of texture. Normally, the surface of the font is smaller than typical textured objects like for example countries on a map. This can result in an illegibility of the word and also an unidentification of the actual texture. Especially for small fonts this typographical variable is not recommended. It is worth to mention that from a distance the whole text appears as one image and hence gains a texture.

Size

The use of size to indicate value is an already common practise. Headlines, sub-headlines, body text and foot notes often use different font-sizes to indicate hierarchy. Cartography also makes great use of scaling in order to visualise importance of objects. (Imhof, 2007) But rather than scaling continuously for example the height of mountains or the size of cities, they are often categorised. Since the size of irregular forms like letters is more difficult to compare than the size of two regular squares, the discrete values are rather difficult to extract and should rather be treated as ordinal data.

Just like with the positioning of letters, words and sentences have an inherent value: Words with more letters are longer, capital letters are usually higher than lowercase letters. In the design of maps, inherent sizes often casuse biases. As countries with a greater landmass appear to be more important, »bigger« words also seem to be more important. This problem can be often seen with word clouds, which scale words for example in regard to their appearance. When encoding with size, these points should be taken care of:

  • Proportion As mentioned above, the width and the height of a font should not be individually scaled to preserver their legibility and aesthetic.
  • Word’s length Since the number of letters is the main influence on the word’s length and hence its initial size, it is advisable to use words with the same letter length whenever possible. For this, either representative abbreviations, shortcuts or codes can be used. For example, the ISO 3166 country codes can be used when dealing with countries, words can simply be shortened with an ellipsis or replaced with an acronym. Monospaced fonts also help to compensate varying letter width.
  • Letter’s height Capital letters can also help when encoding with size. Since capital letters often have the same height it also makes them easier to compare as they have an even initial size. They also have a common starting level as they have no ascender or descenders.
  • Minimum size Obviously, the scaling should have a minimum value in order to maintain the readability of words. But this might also be a problem: When the zero-point is at some minimum text size, the comparison can become difficult, as this minimum size needs to be subtracted from the total size.

Summarised, one should use the size when

  • comparison does not need to be exact and
  • when none of the underlying data falls below a value that would result in unreadable text size or
  • when the given words vary too much in their initial size.

To maintain legibility, it is also recommended to always scale width and height accordingly and not individually. When only the width is changed designated fonts like compressed, condensed or extended should be used.

Besides the scaling of the words, the size could be applied to the width or height of the »paragraph box«. When the text is long enough, the text boxes can imitate the usage of bar charts, where the line-length corresponds to the value.

Orientation

Different orientations can hardly be used without compromises on the text flow. While complete sentences or paragraphs can be read with varying slopes, letters or words should only be carefully rotated within sentences. Correct applications can often be found in cartography, where letters follow the path of a river or a word adheres to the coastline. In the same way angles are difficult to extract from objects, the slope of a word cannot be decoded without a doubt. Also ascenders and descenders can increase this difficulty.

Additional typographical variables

Over the course of hundreds of years, typographic conventions were established that are unique in graphical distinction. These are often additional graphical elements that can distinguish words. For example, the underlining of a word is clearly highlighting that word, while underlining a rectangle would not be directly understood as highlighted. With the Web’s growing size, significance and level of access the underlining of text became a semantic meaning: being a hyperlink. Especially in combination with a blue or purple text colour the association is widespread among Web users.

The highlighting of text passages also works with characters added at the beginning and end. This way markup languages *highlight* words or declare <div>boxes</div> of content. (But also in regular usage parentheses indicate a devaluation.)

Conclusion

Summarised, typographic variables contain parallels to visualisation variables and can be used as such within limits. Always in consideration of the legibility, typographic distinctions need to be applied more carefully than in regular visualisations. In the same way visual variables are also not equally suitable for every type of data, some typographic variables can used for qualitative values only.

While the position is widely regarded as the best way to encode in »normal« visualisations, it needs special treatment with text in order to keep the reading order and flow. »Weaker« visualisation variables like colour appear to be more useful because firstly, they do not displace the text corpus and secondly allow for comparison across multiple text lines, which is difficult with a change of x-position for example.

Typographic conventions can be beneficial, but need to be treated with respect for cultural inherent meanings. In general the proposed typographic visualisations are not »as strong« as regular visualisation. The extraction of exact quantitive data is never as good as with graphical means. This is mainly due to the lack of a coordinate system that would give the user an orientation and to the distribution across multiple lines that hinders comparison.

In conclusion, the visualisation with typography is mainly suitable for ordinal and nominal data. Quantitive visualisations should only be used in a supportive role.

Why adhere to text?

With visualisation as the seemingly solution to all modern problems, the question arises why one should stick to text. The answer to this is based on the cultural imprint by the book. As previously stated it had a tremendous effect on not only how knowledge is distributed but also in the way we think. Habits formed conventions and today most people are used to obtain information from written text with its discursiveness, linearity and abstraction. Mendelssohn bewails how this has made us literati:

»We teach and instruct one another only through writings; we learn to know nature and man only from writings.
We work and relax, edify and amuse ourselves through scribbling. The preacher doesn’t talk to his congregation; he reads or recites to it something he has written.
The teacher reads his written lectures from the podium. Everything is dead letter; the spirit of living conversation has vanished. […]
This has brought it about that man has almost lost his value for his fellow-man. We don’t try to meet with the wise man because we find his wisdom in his writings. […]
In short, we are literati, men of letters.«

 — Mendelssohn, 2011

»Wir lehren und unterrichten einander nur in Schriften; lernen die Natur und die Menschen kennen nur aus Schriften; arbeiten und erholen, erbauen und ergötzen uns durch Schreiberei; der Prediger unterhält sich nicht mit seiner Gemeinde, er liest oder deklamiert ihr eine aufgeschriebene Abhandlung vor.
Der Lehrer auf dem Katheder liest seine geschriebenen Hefte ab. Alles ist toter Buchstabe, nirgends Geist der lebendigen Unterhaltung. […]
Daher ist es gekommen, dass der Mensch für den Menschen fast seinen Wert verloren hat. Der Umgang des Weisen wird nicht gesucht; denn wir finden seine Weisheit in Schriften. […]
Mit einem Worte, wir sind litterati, Buchstabenmenschen.«

 — Mendelssohn, 1989

I do not to want people to turn away from books as Mendelssohn’s text might suggest, but rather stress how many fields are framed by the book. Even today where the production of images is effortless as never before, text does not disappear. Instead, new forms of text »types« evolve: Comic books, Instant Messaging and Twitter. In contrast to the pessimistic view of some writers, we should utilise these tools for they are still forms of expression based on textual communication. While some describe the printed book as »the perfect reading-machine« (Die perfekte Lesemaschine, Reuß, 2014) I personally see much potential for new forms of printed books, but even more for interactive, responsive, illustrative ebooks or any form of textual expression.

But the triumphant advance of text has not only to do with habituation. The linearity, the discourse and the abstraction might bring restriction but it also brings advantages over the visual communication. While images allow the viewer to freely explore the content, text guides the reader through the information. The linearity brings order, the discourse allows to follow argumentation. And with the ability to communicate abstractly, text often allows to communicate intangible ideas.

For this thesis I use the prominent status and the qualities of the text as starting point and enhance its comprehension through the combination with visualisations.

Why combine both?

These two disciplines – typography and visualisation – are too often treated as two separate things. Both work with visual variables that are applied according to the underlaying data. Where typography adds this data level to the text, visualisation works with graphical elements to communicate the data.

In the book »Understanding Comics«, Scott McCloud illustrates that words and images are two ends on a scale from concrete to abstract. (McCloud, 2001) He also shows how these two ends can interact with one another, each fulfilling its own purpose. He exemplifies this with »The Adventures of Tintin«: The simply, abstract drawn persons help the reader to identify with them, because their appearance is not so determining. On the other side, the detailed, concrete background shows the fantasy world the story takes place in.

But when typography is applied to the abstract text, it becomes more graphical and hence more concrete. On the other side, visualisations make use of graphical, iconic forms and thus make the images more abstract. Expanding McCloud’s model, the new order is: Images — visualisations — visualising typography — text. While most publications make only use of the extremes of the spectrum, my approach invites the designer to use all »tools« of this range to tell a story.

Range of Abstraction Images become more abstract and text becomes more concrete.

But not only results the combination in a greater variety of storytelling-elements for the sake of cross-mediality, but also helps the reader to understand the content. Albrecht Dürer wrote in his textbook for the adolescent youth already that he wants to combine image and letter to enhance memorability. (»Deshalb will ich das Wort und das Werk zammentan, auf daß mans dest baß merken müg.«) (Wenzel, 2006) Similarly, the brain researcher Wolf Singer states that when our perception is confirmed by two stimulus modalities, the perceived is evaluated as to be the truth. For example when the eye sees an object that is also recognised by another sensory perception, the assumption of what is »out there« is more reasonable. It passes the test of consensus. (Wenzel, 2006) Even though this does not directly justify the use of different encodings within one sensory medium, it paves the way to this research: And indeed the combination of images and words has been scientifically studied by a number of researchers. Most of the results support the theory of »dual coding« by Mayer et al. (1999 in Ware, 2004). »They suggest that if active processing or related material takes place in both visual and verbal cognitive subsystems, learning will be better.« (Ware, 2004) Summarised, this shows how an increased exploitation can be achieved through the combination of text and image.

One study that followed this thesis’ approach from the other side was conducted by Chandler and Sweller (1991 in Ware, 2004). It examined the integration of text directly inside graphics. Their test graphic — a diagram with operating instructions — had text snippets directly placed at the relevant point. Their successful experiment was justified by the »limited-capacity working memory«. »They argue that when the information is integrated, there is a reduced need to store information temporarily while switching back and forth between locations.« (Ware, 2004) This is often justified by the cognitive load theory. As the theory says, our working memory is limited. Hence object and key should be close together to reduce information that needs to be stored temporally in our memory. It may be expected that this concept performs in the same way with images directly integrated into the text.

Other psychological studies on reading comprehension can also be interpreted to support the thesis’ approach. Perrig and Kintsch (1985 in Müsseler et. al, 2002) described a fictitious city first from a bird’s perspective (»north of the highway and east of the river is a gas station«) and secondly from a driving perspective (»When crossing the river, you see a gas station to your left«). Candidates who received the bird’s perspective were less able to repeat the text, but instead could easily draw a map. (Müsseler et. al, 2002) It can be expected to support memorability when representing the information for example in driving perspective as text and from a bird’s view on a map. It needs to be evaluated in further research if this »dual coding« mentioned before also helps when text and image hold different perspectives.

Another reason, why the combination of these two provide additional insight, lays in the frequently given reasoning for visualisation. For this, the Anscombe’s quartet is often used. After presenting tables with numbers, visualizing the same numbers in diagrams seems obvious and convenient. And indeed it is much easier to compare and spot trends and patterns with the diagrams. But one thing is often neglected: The table »expresses […] values precisely and it provides an efficient means to look up values for a particular […]« dimension. (Few, 2014) For some graphics, the underlaying data might even not be extractable at all. For most applications that is absolutely valid: The more data points you have, the less important the single one become. Visualisations’ purpose is to serve for impression, rather than exact numbers. As John Tukey states, it is better to be »approximately right rather than exactly wrong«. (Tukey, 1965) What should be taken from this is that both »visualisations«  — table and graphic — have an unique quality, which are advisable for different settings. And the same holds true for text. Through the combination of these different forms the reader can decide which one suit his interest or ability to extract the data best.

Anscombe’s quartet the values in charts
Anscombe’s quartet the values in a table

These qualities can be exemplified by uncertainty and diversity. While visualising uncertainty is gaining more attention in recent time (Potter et al., 2012), text can often more easily express this uncertainty. And in other cases, where the verbal expressing generalises and labels information (Drucker, 2014), visualisation can demonstrate diversity.

Each form can compensate the difficulties of the other and can even help to overcome the lack of expression in text in general. As Marc Davis wrote in regard to emojis: »they are best used as an ‘adjunct’ to text — especially in social media — helping to make up for the the lack of gestures, facial expressions and intonation found in speech.« (Stern, 2015) And while this might seem only applicable for emojis, graphic novels show us how anger, fear or gentleness can be expressed through type. Hence, the mixture of multiple sources of information creates »a more complete image«, resulting in a greater exploitation for readers.

The impression an user is given by visualisations can also be used as benefit: Willett et al. describe the concept of Scented Widgets which act as »embedded visualizations that facilitate navigation in information spaces«. (Willett et al., 2007) The idea behind this approach is to give users a preview (a scent) of what they are about to see, when they click on the equipped widget. Because of the simplistic appearance users are able to compare multiple elements of information (or in this case choices) at once.

What needs to be considered is that not all text especially literature »wants« to be equipped with additional graphics. As these visualisations can add the appearance of an analysis and statistical effort, one needs to accept that especially lyrical texts are often not meant to be analysed by mathematical means.

Summarised, departments too often adhere to their medium: Journalists to text, financiers to tables, designers to graphics. Each discipline should more often use the adequate presentation for the message. Or as this thesis shows: A combination.

Why Micro Visualisation?

The combination of text and visualisation can be approached from many directions. Besides the aim to create a balance between the two, the focus could be either on the verbal or on the graphical side. For this thesis I will focus on text with visualisations embedded. There are six reasons for that:

  • Based on the previously discussed considerations on the history of communication I have come to believe that text is still human’s main medium of knowledge impartation. And even without the aspect of additional graphical visualisations that is discussed in this thesis, the research to enhance this standard text presentation through typographical visualisations is valuable for this field.
  • In regard to the first point I believe that an adduction to visualisation for the general public could be best achieved with simple graphics embedded right into the familiar medium. The simplification is the first step to make them accessible to a broader audience. With changing reading conventions and the acceptance of this novel sources of information, more complex visualisations can be used in mass media.
  • The use of both text and images is claimed to have a higher efficiency than single-channel encoding. One aspect that is crucial for this dual-coding and also for the concept of this thesis is the connection between these two levels. That supports the decision to directly integrate the visualisation inside the text to provide this kind of link.
  • Obviously this link needs to be explicit in order to connect corresponding elements. (Ware, 2004) Experiments by Zellweger et al. (2000 in Goffin et al. 2014) evaluated the positioning of meta-information related to hyperlinks in margins, footnotes or inline. »They found a large variability in participants’ preferences but observed that, in general, it was desirable to keep meta-information close to the original link. Where information was placed also had a significant impact on reading speed with information close to the links being read significantly faster.« (Goffin et al., 2014) Even though dealing with different information these results are a confirmation for the usage of inline visualisations for better and quicker comprehension.
  • Previous research in the field of visualisation has been mostly towards complex, data-rich graphics. I see a general need for smaller, simpler visualisations. As the media consumption tends towards mobile, content needs to be designed for the smaller display sizes. (Noirhomme-Fraiture et al., 2005) And with screens becoming cheaper and with OLED technology also more energy efficient so that they can be used for even more and also smaller devices like smartwatches, photography lenses or electronic toothbrushes, visualisation need to be specifically designed for this size. The »Internet of Things« is becoming inevitably a reality and all the devices hold opportunities for visualisations of the objects’ data. With their limited output capacities, the visualisation of their status is a crucial aspect. But not only for explicit small displays, but also for varying output devices small visualisation can be easier adopted to the available space.
  • But also in time crucial environments like newspapers Micro Visualisations can be helpful. Smaller graphics are often easier to produce. Especially with the proposed typographic visualisations, this type of visualisation can also be created using amateur software like Microsoft Word. Instead of aiming for complex and costly graphics, the options possible with the available means are often not taken into account.
  • As we can see from the three stages of perception: We see before we read. Using visual and typographic variables that convey the content visually allows the reader to comprehend the structure and possibly also parts of the content before reading. Especially colour that pops out from the rest can guide the reader to relevant parts.

Relevance

With the derivation of my approach in the last chapters, I will now further explain its relevance in a cultural and technological perspective and elaborate its relevance and implications for design. In the last part I will explain my own motivation for the approach.

Cultural and technological relevance

Technology and culture follow each other, facilitate and demand each other, depend on each other. The history of communication is a history of technology. Nowadays, as technology changes so rapidly, our behaviour highly depends on our technology usage. Our life is more than ever defined by devices and their inherent functioning. As these devices become more complicated to use we need to learn how to use them, learn to understand their output. The data generated by our computers and its amount is often beyond our understanding. But not only abstract data comes in an overwhelming amount. Also news, information of relevance or daily impressions inundate our life. The demand for sorting, contextualisation and filtering becomes stronger every day. In the following sections. I show how my thesis relates to these problems.

Rising quantities

McLuhan’s »implosion« (McLuhan, 2011) of the world brings people together in quantities that have never been possible before. Electronically processed information allows higher rates and greater amounts of data. People nowadays take a thousand pictures in their holidays, companies trade with goods in quantities of millions, countries have GDPs in billions of dollars. But besides higher dimensions we also measure smaller and smaller dimensions. Charles and Ray Eames’s short film »Powers of Ten« (Eames, 1977) illustrates this impressively: starting from above a couple having breakfast, the camera zooms out until it reaches the limit of our vision 100 million lightyears away and then zooms in to a single proton with a size of 0.0001 ångströms. And also Gavin Schmidt describes our capabilities to already measure 14 orders of magnitude of weather data: »from the small microscopic particles that seed clouds to the size of the planet itself«. (Schmidt, 2014) The recently populated term »big data« (which seems to have become a buzzword especially in business) is used for the rising quantities of data that are being captured through the electronical processing of scientific calculations and measurements, financial transactions, electronic communication and internet traffic in general. Companies like Google, Amazon, or Facebook, but also scientific institutions like the CERN or national agencies like the NSA use these data and the rising computing power to find previously undetectable patterns and insights. To emphasize this importance, the World Economic Forum rises the status of data to be the new oil. (World Economic Forum, 2011)

Computers allow us to process this amount of quantities, but it seems that our language lacks behind the technological progress: The difference between two pieces and a million pieces is tremendous and still the English and German languages for example have only two modes to describe quantities: singular and plural. Some other languages have more forms of expression quantities. Gravel G. Corbett summarises these in his book »Number« (2000): Some languages have dual, trial and (debatabally) quadrals. For example the Upper Sorbian language can differentiate between »you« (»ty«), »you two« (»wój«), and »you (all)« (»wy«). Accordingly, the trial and quadral expresses the amount of three respectively four. Other languages allow for a distinction of singular, paucal and plural. The paucal can be described as »small plural«, often used for quantities between two or to six. It can be found for example in Lihir, an Oceanic language spoken on a group of islands next to Papua New Guinea. They use five forms: Singular, dual, trial, paucal and plural. What can also be found is a split in the plural, to differentiate amounts of many. This serves for various functions: In the Arabic language for example it discerns between a general form (»debbānten« : flies) or the expression of abundance (»dababīn« : many flies). In Kaytetye (an Arandic language spoken in Central Australia) the two forms of plural discern between many and »all the x in the universe of discourse«. Two last examples come from the Celtic language Breton that describes »children« (»bugal-e«) and »groups of children« (»bugal-e-où«) and the Warekena language (an Arawakan language spoken in Brazil and Venezuela) that can express »pigs« (»abida-pe«), »very many pigs« (»abida-nawi«) and »very many pigs indeed, too many to count« (»abida-pe-nawi«) by composing different forms in one word. (Corbett, 2000)

What can be extracted from this excursus is that in fact very few languages have forms to differentiate between multiple quantities and it may be expected to cause translation and understanding problems in other languages that lack these forms. What visualisation can contribute here is the probably universally understandable form of expressing such quantities. Amounts of »two«, »tree«, »all«, »groups of multiple« and »too many to count« can be easily communicated in an image.

Quantities in Image and in written text

Images of visualised quantities gain importance for two reasons:

  • First, many languages do not provide methods to express amounts accentuated. Obviously missing grammatical forms can be compensated by further textual expressions, but these supplements often fail in conciseness and universality.
  • The second aspect is based on the rising amounts of data. Because nowadays the importance of an object relies so much on its underlaying quantities it needs to be directly integrated and connected.

In general, the data of an object becomes so important that this information should be placed together with the object.

Technological change

Processing in technology demands redesign. Every time a new medium is invented it is treated in the way the old technology was used. It takes time to adapt the usage to new input methods, faster transmission or greater range. I believe we live in a time where the application of technology is lacking behind the possibilities technology would allow.

Scott Klein showed in his talk »The forgotten History of News Graphics« (2014) that newspapers that published visualisations in the 1840s needed to build these graphics out of letters or other symbols. For rotated elements they needed to build special constructions in the composing galley. With the invention of offset print, this effort has been dramatically reduced. Elements can be freely composited digitally but still the design of the book has not changed much in the last 500 years. Now with the emergence of new digital technology the text layout is even less restricted to the classical design. But often the layout of books, posters or websites still keeps the form the letterpress required. Digital printing and displaying methods would allow absolute freedom over the book’s layout, but is »restrained by conventions of design and reading«. (Drucker, 2014) McLuhan spoke about this ever-repeating attitude in an interview in 1969:

»Theirs is the customary human reaction when confronted with innovation: to flounder about attempting to adapt old responses to new situations or to simply condemn or ignore the harbingers of change — a practice refined by the Chinese emperors, who used to execute messengers bringing bad news.
The new technological environments generate the most pain among those least prepared to alter their old value structures.The literati find the new electronic environment far more threatening than do those less committed to literacy as a way of life. When an individual or social group feels that its whole identity is jeopardized by social or psychic change, its natural reaction is to lash out in defensive fury. But for all their lamentations, the revolution has already taken place.«

— McLuhan, 1969

As he states, the »literati« are balking at the change, and I believe that non-specialists from other disciplines can help the future book to find its form But this needs to happen not only for the sake of change, but there is also a demand: Digitally published books cannot be designed in the same way analog letterpress books were designed, but need to consider varying reading environments, materials and different reading behaviours. Myers et al. state that we also need to »create user interfaces that work with vastly different sizes«. (Myers et al., 2000) Their article form March 2000 shows that difference of displays »are factors of over 1000 in number of pixels and 100 for size«, where it is noteworthy to mention that the highest resolution wall-sized display they describe was used by the Stanford University and had a similar pixel number to the laptop this thesis is written on 15 years later. In recent time, responsive web design evolved as an answer to this problem. Layouts from websites can now be specifically defined for every resolution. Since every display size leads to different reading distance as well, Oliver Reichenstein of iA discusses the concept of responsive typography. (Reichenstein, 2012) He distinguishes between the metric and the perceived size of the text. Since the metric size normally stays fixed on various devices the perceived size varies. He demands a fixed perceived size and hence an adaption of the metric size for each device’s reading distance.

But not only should the book’s design adapt to the user’s needs, but also explore newly formed options. Some avant-garde analog books already explore these possibilities. Jonathan Safran Foer’s book »Extremely loud and incredibly close« is an example of experimental typography. Converging and drifting apart text, marked and crossed out words, handwriting and photographs are used throughout the book. The book uses the possibilities of modern typography to support its content, the protagonist’s thoughts and activity and the overall mood of the book.

Drucker states that the current »new writing modes are shaped by social media, by email, blogs, Twitter, and wikis. In these changing conventions the surface of interface often conceals the back-end technical and conceptual processes by which they are produced.« (Drucker, 2014) As with all electronical devices, the appearance is separated from the function. Digitally published text for example is often designed for interaction. Facebook users can »like« everything published without needing to know how the servers store the »Like«. Only the metaphoric thumbs up icon brings the process back to the perceptible reality of the user. Other examples for these interactive texts are the platform Medium.com which allows for commenting single paragraphs, the browser plugin Dictionary of Numbers, which wants to »put numbers in human terms« by translating numbers like 86 million on websites into familiar terms, and Google Docs enables simultaneous work on a single document. Thus one might say that the display of the text becomes the interface of the text.

Andries van Dam demands interfaces to »enabl[e] and empower[…] people to be creative, productive, and less constrained by the limitations currently imposed by the technology.« (van Dam, 2001) My concern is that currently only few take advantage of what technology already enables us to do. The given examples are by leading companies or independent designers. The media consumed by the centre of the society is still dominated by classic text formats. We need more experiments and implementations of behaviour that would not be possible with letterpress technology and show how communication can be altered for good or also for bad. A change many industries who depended on the old technology seek for. Journalism for example is struggling with an identity crisis as its former distribution and speed is not compatible with people’s usage of the internet. A new role and a different form of communication could help overcome its problems. As companies search for information within their big data, newspapers as expounders of the world might seek for answers in the same data pools. As explained above, the use of visualisations helps to explore as well as to communicate these findings.

Summarised, digitally produced and published books or other essay forms allow for a form that combines multiple mediums within one. »Digital media have become the meta-medium par excellence, able to absorb and re-mediate all previous forms in a fluid environment in which remixing and culture jamming are the common currency.« (Burdick et al., 2012, p. 15)

Information overflow

We live in a time where »information gently but relentlessly drizzles down on us in an invisible, impalpable electric rain.« (von Baeyer, 2003) To find sense in these data is our daily challenge. Companies like Google specialise in this sense making by making available websites searchable. But even when we quickly find resources, the challenge to quickly comprehend and evaluate the given content remains.

One solution to this analysis of this content is the categorisation of objects. This has been a long practise for libraries or dictionaries in order to sort and later find information quicker. For this, objects are sorted into categories and subcategories, which can be easily navigated top-down. Cross-references also allow for links between objects. This way they stay in their place while still being connected. The whole internet is built upon the same structure: websites contain links that bring the users to the next website. When websites are organised in this way it can help users to quicker find relevant content. But not always is this structure visible. The UTS Library for example illustrates its catalogue with a stacked bar. Each colour represents one subject and its width represents the amount of books in this field. Each element also acts as navigation and splits out to more colours that represent subcategories. This Micro Visualisation illustrates the complete catalogue consisting of thousand of books in a memorisable, navigable, and compact way.

But also after the correct website is found on Google and on this website the relevant page is found, the presented document may be hard to comprehend. And again Micro Visualisation can help. Visualisations can for example act as an overview of the structure, they can help to find relevant points by »popping-out«, or they can show the information in its context.

When there is so much information from different sources the context becomes more important. To provide the user with this information but not overwhelm him is crucial. For this it is helpful to provide the reader with further information he can access if necessary. This concept matches Shneiderman’s »Overview first, […] then details-on-demand« mantra in its basic concept. While Shneiderman’s starting point is also a visualisation, my approach starts with the text in which the user can »zoom« to further information on demand through a visualisation.

To provide the user with orientation, navigation, and comprehension aids does not only help the reader, but also makes the content findable and accessible in the great amount of information given. Therefore the author should strongly support this process by not only presenting raw text or the »ten blue links«.

Relevance as a design research

This work has a high relevance for the discipline of design. Typography is one of the very bases of design that comes to application every day. Visualisation is just about to find its way into a broader public. For this mixed audience the requirements for such graphics change and need to shift from expert driven »tools« to casual readers. In the following I discuss three main points to this.

Visualisation towards micro visualisation

The term micro in this thesis does not only refer to small in size but also small in its number of data dimension or data points. Not in all cases a higher number of data points add value to the message of the graphic, but can sometimes even prevent the user from extracting relevant information. As most articles which can be found in newspaper or magazines today do not deal with a great amount of data, they should still be able to communicate these few points correctly and take advantage of visualisation techniques.

Broader understanding of visualisation

As I stated before in relation to typography, the term or discipline of visualisation should not be tied to bar charts, scatterplots or network diagrams. Every design object should be treated as a form of information visualisation. What I seek for is a functional design approach in which every aspect of design is tested towards its encoding possibilities. Most of my approaches could probably be described as applied typography, usability or interface design. And they certainly are in their methods, but despite the normal practise I propose execution with the data layer in mind. As Johanna Drucker describes, nearly all methods used for visualisation were developed in other disciplines. (Drucker, 2014) My wish is to bring these methods back to their subjects in their design-improved form. For example, geography, statistics, and – especially in regard of this thesis – typography can all profit from this exchange.

Dissolving the interface

Bill DeRouchey’s talk about the history of the button shows the transition from levers to buttons, metaphors of these buttons on surfaces and as he predicts to fluid buttons, where everything can act as a button. (DeRouchey, 2010) The process he describes is the deviation from the actual physical operation the button causes. This is only possible under two conditions:

The first is the technical development to electronically processed information. This allows »compressed time«, »compressed distance« and »abstract motion«. (DeRouchey, 2010) The electronic signal of the button being pushed can result in a much stronger result than the actual physical pushing. Physical cause and electronic effect drift apart.

And secondly, this non-conform behaviour needs to be accepted and learned by the user. We have become so familiar with this process that we drive our cars with our subconscience – they even become extensions of our body as McLuhan states. (McLuhan, 2011) We do not need to see the actual process anymore and so it comes that we have learned that links on websites transition from skeuomorphic button metaphors to underlined words.

Probably the same path of abstraction could be discussed for visualisation (stacked goods became histograms, landscapes became their two-dimensional shapes on maps, cities became nodes and streets their connecting edges in networks), but for this thesis I want to take the current state of visualisations and dissolve them. Their should be no border between images and text, just like there is no border between links and texts.

Personal experience with this topic

Having worked as an information designer for varying applications like journalism, public relations, travel guides, scientific research, and art, I encountered mainly two perspectives on the combination of text and information visualisation: One uses the text as an accessory part, the other considers visualisation as an accessory part. The interaction of these two is rarely equal. While visualisations often include text as labels or introduction, writers are restrained in the inclusion of graphics.

The company I work for had a big travel guide publisher as client, who reissued all of its guides to include infographics. The way they were included were on three to five double pages as specials in the book. We designed the graphics without any connection to the surrounding pages. Another example is a newspaper that includes a full page of visualisation in every issue. This is certainly a big step forward, but still the rest of the pages are mostly not equipped with any visualisation.

In reality the cooperation of the multiple necessary disciplines is unfortunately often exhausting, expensive and time consuming, but I still believe that each expert in his field can learn from the other subjects involved.

Conclusion

The combination of text and image has been practised for a long time. But still the idea to use simplified images for a more interwoven combination has not been elaborated in detail. Research from related areas shows evidence of potential in this approach. The analysis of typographic variables has shown how they could be utilised for visualisation within limits.

With the embedding and subordination of the graphics, requirements and challenges towards their appearance arise. They need to be physically small, reduced in data points or/and in the number of dimensions. This reduction will be referred to as »micro« in the course of this thesis. Hence »micro visualisations« are basic graphical modifications or additions, that enhance the comprehension of text.

Thus I propose a wider understanding of visualisation. Every form of informationally motivated visual alteration or addition is to be treated as visualisation, even if it visualises a single data point.

State of the Art

An examination of the current state of the art in visualisation and related disciplines is provided in this chapter, with a special focus on the concept of micro visualisation that is introduced previously.
In order to structure the evolving design space, a novel taxonomy is developed that differentiates between types of application and clusters them into four main categories.
The taxonomy’s validity is discussed against two existing propositions and more projects that support the notion of micro visualisation.

Main reference works

The term »micro visualisation« has not been used in this context before. Despite the fact that a number of projects and concepts have been developed which now fit into this scope, they have not been compiled under this term. The main inspiration and starting point for this considerations are Sparklines. They follow the exact idea this thesis is now pursuing: a small diagram that is embedded right within the text corpus.

Sparkline The last point in the visualisation and the entity should have the same emphasis

Sparklines have been elaborated by Edward Tufte in his book »The Visual Display of Quantitative Information«. Since Tufte is the inventor and strongest supporter of the data/ink-ration, he seeks for a maximum amount of data with the lowest possible »chart-junk«. Every unnecessary object, effect or point in a diagram is seen as junk and hence should be eliminated. Tufte preaches the highest possible data/ink-ratio: The greatest amount of data-points visualised through the lowest amount of (digital or analog) »ink«. (Tufte, 2001)

Tufte vs. Holmes With his view he is usually regarded as antagonists of Nigel Holmes, with his illustrative graphics. (Cairo, S. 63)

Another aspect of his research are high-resolution data graphics. He explores ways of mapping the highest amount of data in the smallest possible physical space. In connection to this efforts he introduces the concept of Sparklines as »datawords«. This idea brings two advantages:

  • Creating context Sparklines create context on two different levels. The idea of a great amount of data on a small surface is combined with the idea of directly placing that graphic into its relevant context. In contrast to many graphics in which the reader has to switch back and forth, Sparklines are directly at the relevant point. This brings the graphic into context. Tufte’s example of application is obvious: A medical patient’s current level of glucose gains much more value when it is viewed in its historical context; in this case the 80 most recent readings of glucose. The medical staff can instantly evaluate if the current figure is an exception, a repeating pattern or an ordinary value. Again the context visible through the dataword brings benefits. The text is put into context.
  • Amount of data The eye’s »resolving power […] enables it to differentiate to 0.1 mm«. (Tufte, 2001) This allows for 100 values per cm. Tables would need a much greater space to convey the same amount. Sparklines can hence reduce the necessary space dramatically.

A special form of Sparklines he describes are dual state charts. They display binary data with upward and downward pointing lines. Tufte’s examples for application are results of baseball games. The chronological sequence of win and loss for one team can be easily extracted from the chart. As Tufte’s example shows, this technique can visualise a complete season within little physical space. Gareth Watts developed a great jQuery plugin for Sparklines along with a number of other datawords. Besides line charts (regular and composition), he suggests bar charts (regular , stacked and tristate ), box plots , bullet charts , and pie charts . He also developed interactive and animated charts.

Dual State Charts Only two states can be displayed in chronological sequence.

The basis of these Sparklines is Tufte’s idea of being »approximately right rather than exactly wrong« (Tukey, 1965) By omitting all axes and the key, only the process that leads to the current value is extractable clearly. Exact values cannot be determined, especially since the extreme values are not known (in the standard execution of Sparklines). As this might not be suitable for every scenario, in the given case of glucose level it is absolutely valid and in most cases this additional information is better than just a single number which is normally presented. Further research on the concept of Sparklines suggested the markings of highest and lowest values through coloured dots.

Alternative Execution Exemplary alternatives by Edward Tufte

A second reference project is from New York Times’ »The Russia Left Behind« feature. The interactive article describes the 12-hour drive from St. Petersburg along several cities to Moscow. Next to the regular article text, a map of the path driven is given by the author. As the user scrolls down the article, the path is filled up in orange and the current city is highlighted. There are several reasons why this graphic is so beneficial for the article.

  • Undisturbed reading flow The graphic is not placed between two paragraphs as it is normally handled. The fixed position on the left side allows for constant accessibility yet without being affected by the scrolling and without disturbing the reading flow.
  • Instant understanding The map is reduced to the core elements relevant for this story: The cities, the path, and its physical course. This enables even people who are not familiar with maps to instantly understand its message. This is even more supported by the link between the scrolling and the path’s filling. This makes the lookup of the currently described city superfluous. What many graphics lack is the chance to find the relevant point for a text passage in the graphic.
  • Additional data level The text tells a story a graphic could not convey. The graphic on the other hand conveys information the story could hardly do: The distances of the cities and its geographical position. One thing that could be criticised is that the map is not oriented, which could create a false impression of the geographical positions of the cities. But since they provided an overview map with the path’s position and surroundings, this point can be neglected.
  • Intuitive navigation The cities on the map are also clickable. This allows for intuitive navigation in Ben Shneiderman’s sense: Overview (map) first, details-on-demand (article). (Shneiderman, 1996) This way the text is not only connected to the graphic through scrolling but also the other way round. Besides its information it can serve as a second navigation.
The Russia Left Behind Screenshot of the article in desktop view

Conceptual delimitation

The concept of the combination of text and graphic is not new. Even the combination of text and visualisation has already been elaborated. In recent times the term digital humanities emerged as the concept of digitally and computer supported analysis of resources of »literature, philosophy, classics, rhetoric, history, and studies of art, music, and design«. (Burdick et al., 2012) Visualisation is a common practise for text analysis. Especially complete text corpora, which are difficult to overlook in its entire, can be effectively visualised from a macro perspective. Previously hidden pattern that span over several hundred pages or books can be uncovered. A good example of the possible scope is Google’s Ngram Viewer. It is originally based on Google Books and the company’s ambition to scan every printed book from 1800 AC until today in various languages. Through the provided web interface users can search for arbitrary terms, whose frequency of occurrence is then plotted over time.

Google Ngram Viewer The frequency of the terms freedom and justice is plotted over time.

My work has many similarities with the one by digital humanists. By visualising a second layer – often on a meta-level – we seek for a greater understanding of the actual content. In terms of difference, digital humanities covers a greater area of media and more often utilises the macro perspective (especially in corpus linguistics), while my approach keeps the focus on the actual text and only enhances this view with visualisations from both a micro and macro perspective. In this regard my approach can be described as one part of the wide area of digital humanities.

Graphic novels and comics are surely an extensive combination of pictures and text. Onomatopoeia and speech bubble (which found their way into digital messaging application) are examples of inventions this form of art created. In fact these techniques connect the text with the images. Unfortunately, in most cases the text is rather uncaringly squeezed into the speech bubbles. The text is ever so often treated as a required appendage.

One noteworthy book is the previously mentioned »Understanding Comics« by Scott McCloud. This comic about comics discusses these »spatialised sequential narratives« in form of a nonfiction graphic novel itself. With this openminded definition of comic books, his findings can also act as a general guidance for visual storytelling. But while most comics focus on the visual storytelling through illustrations, I focus on textual storytelling.

Even though the application of the findings in this thesis can be applied to other types of text, the form of presentation I elaborate in this thesis is arranged in paragraphs instead of loose arrangements. One requirement I thus impose for the text is the retaining of its readability as text corpus.

Taxonomy

Since the term »Micro Visualisation« has not been used before for such graphics, I see one contribution of this thesis to depict existing examples and sort these into a taxonomy. While taxonomies for visualisations already exist, none of the previous fits my approach. Other taxonomies are for example these two:

  • Even though not initially intended to be a taxonomy (and also mainly developed for cartography) Jacques Bertin visual variables help to categorise visualisations. His system describes the deepest level of visualisation: the encoding of the data points.
  • Another approach to categorise graphics is the differentiation by type. In »A Tour through the visualisation Zoo« Jeffrey Heer, Michael Bostock, and Vadim Ogievetsky (2010) suggest five categories with several subcategories each. The types they propose are time-series data, statistical distributions, maps, hierarchies, and networks. In most cases the given data determines which of these five types can be used to visualise the structure. In the subcategories, sometimes the types are also determined by the data set (choropleth or graduated symbol map) and in other cases rather options to choose from (stacked graphs or small multiples) are given. Within these types the visual variables are then applied.

In contrast, the taxonomy I propose makes a further step by observing the application of the visualisation. These categories refer to different levels of modification and intervention of the original text, the source of data, and the actual purpose of the visualisation. Also instead of grouping types into categories the differentiation might not always be explicit for all graphics, but should rather constitute the extents of multiple aspects in which the visualisation can be located.

Taxonomy Positioning The taxonomy I propose acts on the highest level of three.
Level of change: Modification or addition

The first differentiation describes the level of change made to the original pure text by the additional data layer. The first and restrained approach is the modification through purely typographical instruments. The second direction is the enrichment of the original text with additional pure graphical elements. While additional elements are separated from the entity, modifications affect the entity itself. Visualisation and entity are the same object.

Positioning/scope: Integrated or adjacent

I use the positioning and scope to further describe the level of change. While one is smaller and directly integrated inside the text, the other is more spacious and happens adjacently – as for example in the margin.

The differentiation also affects the relation to the corresponding entity. While integrated visualisation (modification and additional) refers only to a single entity (and are in direct contact to this one), adjacent additional graphics can also relate to multiple entities they visualise. Adjacent modifications still refer to only one entity.

Relation between visualisations and entities Integrated visualisations have a different relation than adjacent ones.

This categorisation needs to be considered separately from the previously described distinction of modification and addition in more detail.

Modification of existing elements can be seen from the typographic perspective as the differentiation between micro and macro typography. Micro is referring to the typographic changes on a detailed level (single characters or words), while macro modifications describes rather extensive changes that involve the layout (complete sentences or paragraphs). The terms »integrated« and »adjacent« are admittedly not perfect in the case of modification of elements, but should rather be seen as detailed changes and spacious transformations that happen either to single elements and/or to whole text passages.

Additional elements can also be placed inside the text lines or outside when they are small in size. When elements are placed separately from the text-block, they are considered adjacent. Spacious graphical elements are also treated as separated even if their centre point is inside the text block, but their total scope excels the text-block. When they are placed in the physical context of the related entity, they are treated as integrated.

In their paper, Goffin et al. (2014) distinct three different contexts for the placement of datawords:

  • Strong context, where the »bounding boxes« of the visualisation and its reference word »touch«.
  • Weaker context, when the placement of the visualization is influenced by the word’s position.
  • Out-of-context describes the positioning unrelated to the entity’s position somewhere in the margin.

Although Goffin’s taxonomy is interesting, there is no need for such a detailed differentiation. Also, since no visualisation should be place completely »out-of-context« and for a general concept, the difference between stronger and weaker context is not helpful in regard to its design, two types should be sufficient. For their taxonomy they also only considered additional elements. Modifications on the word itself would only be in a strong context per definition.

This differentiation is also relevant for the goal and message of the whole document. If the relation between the entity and the corresponding visualisation is more important it should be integrated. If the comparison between the different data points is more relevant they should be placed together adjacent, because their distribution across multiple lines would impair the comparability. Hence, integrated visualisations are useful for proximity to have the graphic in context of the word, adjacent are good for comparison to have the data points in context of each other.

Data source: Innate, intrinsic or extrinsic

The next differentiation of visualisations is examining the source of information, which is mapped upon the given text. Three derivations can be distinguished.

  • In the first case the data is innate in the text. This can be further expanded to distinctions between the word’s or sentence’s type (e.g. noun, headline) or the content the words unfold (e.g. city name).
  • In the second case the status of the text acts as a source. It is hence intrinsic within the text but emerges through factors like changes, authorship or – and this might be seen as a narrow ridge between the third case – by further external influences.
  • In the third case, a complete new data layer is added from an external source. Single text elements act as symbolic reference to objects which are equipped by data-values through further links.

This second, visualisation layer can have different functions depending on this relation. By visualising innate or intrinsic data it can reveal, contextualise, and amplify the structure and content of the text. It acts as meta-layer. With extrinsic data it adds a detail-layery that adds deeper or new information to the text.

Point of relevance

The next classification describes the point during the reading process at which the visualisation becomes relevant for the reader and hence determines its purpose. For this, I differentiate between three stages: Before, during and after the reading process. Obviously, these situations are not always in linear order and one visualisation can be relevant at different stages.

Purpose

In these points of relevance, the visualisation has different purposes. Prior to the reading process, the reader needs an overview and orientation, navigation through the document and a sense of the meaning of the text. During the reading process, visualisations can enhance the memorability, serve for deepening and extension. A summary and aide memoire is relevant after the reading process.

Interactivity

While some techniques can be applied to traditional books, others demand a digital environment. Unchanging and static graphics can be printed; dynamic or interactive elements are generated every time the text is opened or the user interacts with the text.

Conclusion

The most relevant approach of this taxonomy is the differentiation between the type and the positioning/scope of change. These four characteristic divide the field into four segments the designer can work with. It can act as a starting point when one characteristic is fixed or already taken by other elements. In a fixed layout for example adjacent elements would not be possible, hence integrated visualisations would allow a second level. If one quality is more important then the appropriate type can also be used. The designer can decide between comparison and proximity.

Existing works

The following chapter highlights some of the existing works categorised in the presented taxonomy. For this, I distinguish between projects that modify the existing text presentation and the ones that add additional elements to the text. These two groups are further subdivided in integrated and adjacent placements/scopes of the visualisation. This creates four groups in which all projects can be sorted. The other criteria I suggested for the taxonomy are helpful for further describing a graphic but do not affect the initial design process drastically and are hence discussed for each project if they are of any significance. Often the four categories already define the point of relevance for the graphic: While the reader often stumbles over inline elements only while reading, external elements are noticed before and hence can act as an overview.

Star Structure The structure for the state of the art report is based upon the taxonomy
Modification
Integrated

The choice of typeface is often regarded as the starting point of setting a text. It is common practise to use different typefaces for the body and headline texts. These decisions are often lead by legibility and hence serif fonts are commonly used for body text and sans serif for headlines. (Santa Maria, 2014) But sometimes the decisions are also driven by the content: Quotes are commonly set in oblique or script fonts and source code is set in monospaced fonts. But Mike Hofmaier for example uses a monospaced font in his book »Verfassung Verstehen« (2013) to distinguish comments from the text. The symbolic convention of the typewriter font for notes and an antiqua font for »finished« text can also be seen in iA Writers’ different modes, that differentiate these writing-stages.

iA Writer Interface The font changes accordingly to the modes: notes, write, edit, and read

Varying the typeface for elements of the same »category« within the document is rather inconvenient and regarded as vexing for the reader. Typedesigner Luc(as) de Groot used this for his redesign of the German political weekly newspaper »Jungle World«. He designed every column to vary between Plantin and Minion. He used this »subversive solution, befitting the paper’s own editorial attitude.« (de Groot)

Besides this rather unconventional method, the mixing of typefaces is regarded as one the skills of typographers. In general, the mixed typefaces must not be too similar in character, but of a similar style on a detailed level. (Santa Maria, 2014)

Using different styles is also common practise: italic or bold fonts to emphasise, uppercase for people screaming in direct speech. Ken Kesey’s novel »Sometimes a Great Nation« from 1964 also »shifts repeatedly in mid-sentence between roman and italic to distinguish what characters say to each other from what they say in silence to themselves«. (Bringhurst, 2012)

Colours are also used to emphasise content. Traditional typographers tend towards a single colour – commonly red – while cheaper prints and digital media allow for several colours. Again colour is commonly used for distinction of hierarchy in the text and is rather seldom changing within one element. Besides links on websites the content is rarely reflected in the colour.

One positive example is »Silenc« by Manas Karambelkar, Momo Miyazaki, and Kenneth Robertsen from the Copenhagen Institute of Interaction Design. They visualised how much of the languages Danish, English and French is silent. »In one form of a book, silent letters are marked up in red yet remain in their original position. In another iteration, silent letters are separated from the pronounced text and exhibited on their own pages in the back of the book, the prevalence of silent letters is clearly evident.« (Miyazaki, 2013) The gestalt principle of proximity allows the reader to recognise the words as whole while reading even when in different colours and still being able to see the emphasised letters to »pop out«.

Project Silenc The silent letters in the text are coloured red

In computer science the colouring of source code is a common practise. Since »software code text is intended for the people writing it much more than for the computers running it« (Gendel, 2010), many conventions evolved over time to make the code more readable. Variable names, indention and whitespaces are visualisations that are completely ignored by the compiler, but help the programmer to clarify a code.

Another convention is the syntax highlighting. It colours the words according to their category: keywords (if, function, for, …), symbols (=, +, ;, ?, …), identifiers (variables and function names), comments, string literals, and numbers. (Wirth, 1976) While nowadays mainly colours are being used for this distinction, initially different font styles were used. In Niklaus Wirth’s book from 1976 »Algorithms + Data Structures = Programs«, the author prints out Pascal code with bold keywords, and identifies and string literals in italic. (Wirth, 1976) Besides this, it is noteworthy that he uses the same serif font used for the text and the source code. Today, monospaced fonts (rarely with italics and sometimes still with bold styles for keywords) are being used in all major code editors.

Source Code 1976 Pascal source code with syntax highlighting by Niklaus Wirth
Source Code Today Pascal source code with typical syntax highlighting
Semantic Hightlighting Instead of keywords and symbols, identifiers are coloured

Besides syntax highlighting, the idea of semantic highlighting emerged in recent time. In 2009 the creator of the code editor KDevelop for Linux’ KDE environment implemented »Local Variable Colorization«. »That colorization assigns a semi-unique color to each variable in a local context. This allows much easier distinguishing those variables, largely without reading their full name at all.« (Zwabel, 2009) In 2014, Evan Brooks suggested to not colour keywords and symbols at all, but instead use the full colour scope to individually colour each identifier. This allows quickly finding the usage for each one over the course of the program. This would help the reader to quicker identify the structure and the flow of the application instead of just »highlight[ing] the obvious«. (Brooks, 2014) But besides the ability to read the code afterwards, both highlighting techniques also help the programmer during the process of writing: As elements are coloured immediately after being typed, misspellings become instantly apparent.

One common approach in text corpus analysis is its change over time. Modifications, supplementations or removal of text passages are sometimes done by the a single author or by multiple people for example on Wikipedia articles. Ben Fry’s visualisation »On The Origin of Species: The Preservation of Favoured Traces« shows the changes over 13 years in six editions Charles Darwin made in his masterpiece. Text from each edition is differently coloured, revealing the amount and positions of every change. One of theses changes is for example »‘survival of the fittest’ – usually considered central to the theory and often attributed to Darwin – instead came from British philosopher Herbert Spencer, and didn’t appear until the fifth edition of the text.« (Fry, 2009) While the visualisation clearly shows the overall structure of changes (also through the time-lapse video), the text is only visible through mouseover. The project is still included in this micro visualisation discussion, since a smaller amount of text could also be shown in clear form.

Change over time The visualisation shows the appearance of each paragraph with text colours

One thing that is also invisible in Fry’s project is the removal of words. This is a common problem in visualisation: to show what is (no longer) there. This is especially relevant for changes on Wikipedia pages as discussions often revolve around what things are included and how these are written in the encyclopaedia. Behind the front page of each article there is a discussion and an article history page, where people debate over its content. The conflicts are often driven by larger societal debates. The project Contropedia makes these debates visible. It aims to »provide a better understanding of socio-technical phenomena that take place on the Internet and to equip citizens with tools to fully deploy the complexity of controversies.« (Contropedia, 2015)

They provide three different views: the layer, dashboard and network view. In the dashboard view »text is highlighted in green when it was inserted and in red if it was deleted in a specific edit.« Besides the use of red and green colour for added or removed elements, the text has the benefit to be able to be struck through. This is probably developed from analog paper and has also been common practise on typewrites. Other possibilities are transparency or the use of dotted instead of solid lines.

Unfortunately, they use red (in combination with blue) again for another purpose in the layer view. This time not the text but the background behind the text is coloured. »The hotter the color, the more controversial the corresponding elements.« (Borra et. al, 2015)



Layer and dashboard view Contropedia uses text-colour to indicate »hotness« of words.

If a typeface has serval weights, these different fonts can be used convey meaning. Again, this is common practise for different hierarchies or to emphasise single passages inside the text. This method is rather unpopular among typographers as it interrupts the reading flow by distorting the grey value of the text. Small capitals or italic fonts are often preferred. One project that takes advantage of this grey value for visualisation is FatFonts. While many artistic approaches use letter’s darkness for visual poetry or ASCII art (Nacenta et al., 2012 I), no actual meaning is conveyed in those projects. FatFont is designed »so that the amount of dark pixels in a numeral character is proportional to the number it represents. For example, ‘2’ has twice the ink than ‘1’«. (Nacenta et al., 2012) They also provide multi-level digits by arranging the numbers inside each other. Applications they suggest are histograms, inline numbers, and maps. The most useful utilisation are maps, where for example the elevation level is actually written on the corresponding position on the map. From far away the reader can see the patterns on the map; from a closer distance the reader can actually read the values. This is advantageous over traditional colour encoding usually used for the elevation level. Other representations need a legend placed next to the map, while here the information is double encoded right on the corresponding position. One disadvantage is the unaesthetic appearance of the numbers, which could be the reason why FatFonts are rather seldomly used.

Fat Fonts The only actual usage I could find was in a map in the German newspaper Die ZEIT Magazin about the average age in different regions in Germany.
Adjacent

As mentioned above, the differentiation between integrated and separated modifications is not perfectly clear. For this model I consider margins and orientations as changes on the layout and therefore as separated. Through the text’s appearance (especially from a distance), the general layout represents the content. In contrast to simple ordinal distinction of text elements which represent its status (headlines, body text, footnote, …) also quantitive and content related information is shown. In order to emphasise this approach I introduce the term »visualising layout« for this application.

One example that affects the complete layout is the Vilna edition of the Babylonian Talmud. The spirally arranged text »serves as a specific site of mediation«. (Drucker, 2014) The format represents the text message and its purpose through the way it is read.

Layout of the Talmud In the way the Talmud is layouted it serves the mediation.

The margins on a page are also one of the typographer’s essential tool for readability, whitespace and hierarchy. The indention from the left border is probably the most common technique to visualise different levels of hierarchy in text. Scientific books use it to order chapters, file browsers use it for the file structure and programmers use it to structure source code. Every subordinated level is spaced more afar from the left than its parent level. This tree structure is visualised in Jason Davies’ »Word Trees«. The visualisation tool allows the user to select a word from a text to be the root element. The text is then analysed on the sentences containing this word and are then subordinated.

Word Trees Discrete indention is used to create a tree-structure of texts.

While an extreme utilisation can lead to a destruction of an obvious reading order for the reader, changes on a single axis can indeed be used for the visualisation of the text content. One classic example is the book Far Tortuga by Peter Matthiessen, which was designed by Kenneth Miyamoto. He »uses two sizes of type, three different margins, free-floating blocks paragraphs and other typographic devices to separate thought, speech and action. (Bringhurst, 2012)

While all the above examples had discrete level of indention, the recent example »Stacking Up the Administration’s Drone Claims« by Lena Groeger and Cora Currier (2012) for ProPublica has continuous positions. Their visualisation shows four years of statements about drone strikes. Each statement is intended according to its date. Additionally, statements that refuse to even acknowledge the program are coloured in red. With this indention the chronological sequence becomes instantly obvious, even though there is no significant pattern that emerges through this. But what is good about this visualisation is that it does not interrupt the reading flow and still gives added value to the text by means of simple tools.

Drone Claims Continuous indention is used to create a timeline of events.

Owen Herterich’s project »To See & Hear« uses both margin and orientation to create fingerprints of books. »Each line in these visualisations is one line of dialogue in the book. These are mapped around an inner circle, the radius of which is determined by the length of the book. Accordingly, the more dense the visual representation, the higher the proportion of dialogue that exists in that specific work of literature.« (Herterich, 2013) Unfortunately, the change of orientation make the dialogues often unreadable and thus creating rather artworks than visualisations. In contrast, the project »Understanding Shakespeare« by Stephan Thiel (2010) uses smooth and slight change of orientation to create »bubbles« inside the text. The graphics »Shakespeare Googled« use this to »identify famous quotes from a play by comparing the number of results their text returns on Google. It then shows the impact each quote has on the entire text by a simulated force that is applied to each character around each quote. This force is in a direct relation to the number of Google results for the text.« (Thiel, 2010)

Understanding Shakespeare The text is slightly rotated and changed in weight and line-height.

Nancy Duarte’s book »resonate« explains techniques to structure and format information for presentations. While most of her explanations are surprisingly un-graphic, her analysis of existing seminal speeches are explained visually. She groups passages of the recited text into two states: »What is« and »What could be«. She thereby emphasises the mental and physical gap between these two situations. (Duarte, 2010) Typographically she does this with two columns, the first aligned right and second aligned left. As the speaker switches back and forth space is kept blank to maintain the sequence. Two famous examples she presents are Martin Luther King’s »I Have A Dream« and Steve Job’s iPhone presentation. She impressively illustrates how both speakers start with the current situation (»I have a dream …«) and a possible future (»… that my four little children will one day live in a nation where they will not be judged by the color of their skin, but by the content of their character.«). Steve Job’s talk is similar: Todays situation and the presentation of »life changing« possibilities given by the iPhone. (»This is a day I’ve been looking forward to for two and a half years. Every once in a while, a revolutionary product comes along that changes everything.«) What Duarte adds is the marking of different references, moments of applause during the speech, product demo time, and guest speaker time.

Steve Jobs’ iPhone Presentation The text (not visible here) is grouped in »what is« and »what can be«.
Conclusion modifying visualisations

Most of the inline modifications visualise information in a rather »classical typographic« way. Nearly all data was categorical or qualitative. Actual numeric values were rarely visualised. Viewed from the data-source perspective it can be described as mostly innate and intrinsic data. The reason for that might be the difficult en- and decoding of qualitative data from the visual variables provided with typography. For example the y-position within the line can vary only by a few pixel and is therefore harder to determine. It is also difficult to compare multiple values from various lines. This is also one characteristic I will discuss later. The only exception to this »categorical« visualisation is FatFont. But here the actual value is also »written« and not just visualised. Summarised, in current visualisations inline modifications are mostly used to make qualitative statements like »this is important«, »this is new«, or »this is type a, this is type b«. Further empirical tests should test how well people are able to extract quantitive data.

External modifications on the other hand sometimes use quantitative values. ProPublica’s »Stacking Up the Administration’s Drone Claims« for example uses a fluid x-position. Even though the information is not essential, it adds a second layer of information to the data. Here an extrinsic data source is used. The use of complete layout shifts seems to be more experimental and art driven. Summarised, all three types of data sources and qualitative as well as quantitive data is visualised in current graphics.

In general, the modification of text elements is mainly used with a supportive intention. It is utilised to reinforce the text’s content and not to overlay an additional layer. Further applications should test how this can change and how some approaches can lose their artistic usage and find their way into a general and scientific context.

Star Structure The structure for the state of the art report is based upon the taxonomy
Addition

Additional visualisations directly follow the approach of this thesis to mix visualisations with the text. Both inspirations for this thesis (Sparklines and »The Russia Left Behind«) use additional elements to enhance the text. The idea is to use a second, visual layer that displays the different data types described in the taxonomy.

Integrated

One group of additional inline elements are graphical supplements that are placed outside of the words’ or sentences’ scope for example directly following it as Tufte’s Sparklines do. The second group, which I will cover first as it is closer to modification, is right within (behind/before) the scope of one or multiple words. Whether these additions are an additional element or an integral part of the text is certainly debatable, but as it adds more elements graphically it should not be considered as a modification.

One project that was mentioned above makes use of the background colour. The layer view of Contropedia illustrates the »hotness« of a term through blue and red background-colours. »The hotter the color, the more controversial the corresponding elements.« (Borra et al., 2015) A project that uses the same technique is Serendip by Alexander et al. (2014). In the TextViewer, words are tagged by their importance. They found out that »tagging all words equally can sometimes be negatively informative« (Alexander et al., 2014) and hence they use importance binning that applies the colouring with thresholds. For the different rankings they use a decreasing transparency for more important words. These »ramped tags« make the text easier to read.

Ramped tags The colours are applied with thresholds

The idea of such a heat map is certainly one of the visualisations that require computerised creation and a digital output. Website analysis services for example offer heat maps that are overlayed over screenshots of webpages. Areas that have been clicked more often are coloured red, others are blue or transparent. Eye-Tracking software uses the same visualisation but displays the eye’s fixations on the screen.

Typo-Graph Each character represents one hour. The weight of the letter represents the air pollution. The background-color represents the tempearture.

The project »Typo-graph« by Michał Pawłowski and Jakub Wolak combines heat maps with the same idea as FatFonts. But instead of designing a new typeface, they use the grey value of different fonts from existing typefaces. Each text paragraph forms one month of the year: Each line is one day, the width of the paragraph is equally divided for every hour. The text describes the data they gathered about temperature in connection to pollution. Additionally, each letter is in the font-weight corresponding to the air pollution. The background colour of each letter is representing the temperature respectively. This enables the reader to compare the greyness to the colour from far away and read in detail about the data, when closer to the poster.

Since the change of the background colour is already a familiar tool for analog media with text marker, the metaphor is often used for digital media. The translation tool Linguee uses a yellow background colour to mark corresponding words within their sentence. Linguee focuses on context-related translations and since uses a »soft« marking. The aspect of uncertainty, which is often discussed in the visualisation scene, is here solved through this wider range of indication. Microsoft Office and equivalent software on the other hand uses fixed marked areas when the user searches an open document for a term. And since there is no uncertainty in the search term the user typed in this perfectly makes sense. The Sublime Text editor uses a combination of background colour and a border around the search term, as it offers the ability to select multiple words. The web browser Chrome also uses the yellow background for in-page search, but for search results through the search engine Google highlights the term in bold. (They also provide the search term’s context like Linguee does.) This demonstrates the differences these visualisations have: The yellow marker helps the user to spot the term in the text, the other lost its importance to the website’s title and is more relevant within its context.

Text hightlighting Linguee uses a soft »marker« to show uncertainty.

Instead of spotting single term the visualisation »How Senator John Walsh Plagiarized a Final Paper« by the New York Times uses different background colours to categorised text passage. A red »marker« means »Passages taken without attribution«, a yellow »marker« means »Passages with improper attribution, including using other authors’ language without quotations«.

Marker metaphor In order to highlight plagiarism the New York Times uses two background-colours

The usage of this marker metaphor perfectly makes sense in this context, since they are reviewing a paper and the image of somebody labelling every incorrect sentence is really conceivable. What is great about this graphic is that a real world phoneme is used as a visualisation and thus needs no need of explanation.

Another example for such a technique are further typographic additions like underlining. Microsoft Word and other writing applications use solid, dashed or curly lines underneath the word to indicate grammatical or orthography mistakes. Similar to crossing out words, this could be used to visualise the status of text passage. As mentioned above, methods to show something that does not exist any longer are rare. The line through a word is a learned convention that communicates this.

The first group mentioned above (placing additional elements after the entity) has already been discussed by Edward Tufte and seemed to be his inspiration for the Sparklines. One of the oldest examples he cites are by Galileo, who put simple outlines of Saturn’s appearance into the line. (Tufte, 2001) Other symbol-as-nouns have been pointed out by Phillip Troutman in »The Wonder Woman Precedent: Female (Super)Heroism on Trial«, Jon Gross in »Walking Bass Techniques«, and by Matt Reed in »Voltaire« (Tufte, without date). These books use the ♀-symbol, music notation right within the text and little graphics of a man to show things words cannot or only with great effort describe. A modern approach comes from ProPublica who introduced the StateFace font (2012). When using the font in documents or websites, every letter is replaced/shown as an US state. Everybody familiar with the states can quickly recognise each. The typeface has been used mainly in tables as addition to the actual name or on websites like isbarackobamathepresident.com, where the amount of states was more important than their recognition.

Stateface font Every letter in this font is one US state

All these techniques follow the idea of putting simple graphics either as replacement or as addition into the text, to quickly communicate something ineffable or support written content. But all these graphics are more like simple illustrations and not visualisations. Borgo et al. distinguish between these two forms: Pictograms (or pictograph) are symbolic representations of objects, places, or activities. In contrast Ideograms (or ideograph) represent »ideas« and thus are independent from physical objects. Examples are »way finding signage as well as technical notations such as arabic numerals, mathematical notations or binary systems, which maintain the same meaning despite the difference in language and environment.« (Borgo et al., 2013)

»In switching from vocal and face-to-face communications to primarily text, we have lost out on the feelings that can only be conveyed in inflection and facial expressions.«

 — Joanna Stern, 2015

In recent time a lot of pictograms and ideograms arose: Technology like SMS allowed people to quickly communicate during everyday life. Form and hence content was different from traditional letters. The desire to express oneself emotionally was compensated in letters through the length of the writing. Due to the technical limitations like capacity or bandwidth people invented smilies consisting of actual characters :). These representations of faces mostly consist of two characters and are often referred to as Emoticons as they mostly express emotions.

Pushed forward by the internet culture faces consisting of characters from the whole Unicode set emerged ಠ_ಠ. These were often not limited to the two-character-sideway appearance of the original ones. Instant messengers lead to the use of graphical smilies: Illustrations of mostly faces, like the »original« smiley invented in 1963 by Harvey Ball.

Instead of faces consisting of characters or symbols these smileys are small graphics sitting right within the text. While they automatically replaced the »written smileys« in the beginning, users are now able to insert them through a special keyboard layout on their mobile phones. In recent time a particular set called Emoji (絵 (e ≅ picture) 文 (mo ≅ writing) 字 (ji ≅ character)) was developed in Japan. In contrast to Emoticons Emojis show not only emotions (faces) but also objects or abstract symbols. (The Unicode Consortium, 2015) These ideograms quickly spread all over the world as lingua franca, since they are (nearly) universally understandable: »Cross-cultural studies by Paul Ekman and coworkers strongly suggest that certain human expressions are universal communication signals, correctly interpreted across cultures and social groups«. (Ware, 2004)

Mark Davis, the president and co-founder of Unicode describes Emojis as to have a »useful ambiguity«. Their interpretation varies from context, and hence making them versatile. (Stern, 2015)

Japan specific Emojis Some emojis are very Japanese specific and are not understandable outside of Japan. For example 🏮 means Izakaya lantern and 🔰 means beginner. (Stern, 2015)

The human brain is not only capable of recognising these highly abstract images – sometimes only consisting of two dots and a line – as faces and correctly interpret »their« emotions, but in fact we cannot not see faces in various everyday objects like cars, British plug sockets or even in a mountain on Mars. (McCloud, 2001) The phenomenon called Pareidolia has been scientifically studied but also found applications in art and design. In an experiment with apes who were exposed to images of an ape face, a human’s face with and without eyes, a smiley and random lines, a neuron in the ape’s inferior temporal cortex answered the strongest to an image of his conspecific, but showed similar even though weaker reactions to human faces or even smilies. (Goldstein, 2002)

Face on Mars This picture of a mountain was taken by the NASA

Stepping quickly back to the representation of information in a symbol, in 1992 the psychologist Christopher Wickens suggested the use of »object display as a graphical device employing ‘a single contoured object’ to integrate a large number of separate variables.« (Ware, 2004) These accumulation of multiple information in an object are nowadays used in visualisation under the term glyph. Instead of showing multiple data points in a single visualisation, these »small independent visual objects« (Borgo et al., 2013) show single data points but with multiple data dimensions through a variety of encodings: »The design of glyphs can make use of many different visual channels such as shape, colour, texture, size, orientation, aspect ratio or curvature, enabling the depiction of multi-dimensional data attributes.« (Borgo et al., 2013) Combining this idea with the human’s ability to connect multiple visual information from the face to recognise a person and his expression appears obvious.

Chernoff faces The faces for lawyers’ ratings of twelve judges by Avenue

One example are the Chernoff faces, named after the inventor Herman Chernoff. They were one of the first implementations in 1973. »Data variables are mapped to different facial features, such as the length of the nose, the curvature of the mouth, the size of the eye, the shape of the head, etc..« (Ware, 2004) Even though the usage seems reasonable, real world utilisation is seldom. An example by Steve C. Wang (2008) illustrates a possible application. He generated faces for multiple baseball managers, by applying variables like »number of different lineups used«, »pinch-runners used« and »sacrifice-bunt attempts« to ears height, face width and mouth form. Unfortunately it becomes apparent that this visualisation is at least similarly complicated to decode as a graphic with an equal amount of data variables visualised »traditionally«. The ability to recognise faces is proven, but the ability to map single visual information in a face to arbitrary variables seems questionable. In his »A Critique of Chernoff Faces« Robert Kosara also notes the following to be aware of: »There are features that are clearly much more important (eyes, lips) than others (overall shape). Thus, representing data through these visual features means that some data will be much more visible than others.« (Kosara, 2007) Despite the ability to generate Chernoff Faces with Wolfram|Alpha or with the language R, another reason for the unfrequent use might be the rather time-consuming generation of these faces.

Glyphs in general can be seen as one form of micro visualisation, but are mostly driven by the ambition to encode as many dimensions as possible in small space. They have been discussed mostly for three dimensional data representations, but have been treated sceptically »in the community about the encoding capability […] primarily due to its size, limited capacity of individual visual channels and cognitive demand for learning and memorization.« (Borgo et al., 2013) As the design guidelines of glyphs have some similarities to the proposed micro visualisations, these should be discussed later in the context of creating such graphics.

The idea to place more conventional visualisations inside the text and hence getting closer to Tufte’s Sparklines was behind the creation of the typeface FF Chartwell by Travis Kochel. (Kochel, 2012) Through the OpenType technology it is possible to create charts consisting of pies, lines, horizontal and vertical bars as well as rose, rings, and radar diagrams. For example, by simply writing the given values for each sector in a pie chart and selecting a specific stylistic style, the written values are replaced by a visualisation. This enables even amateurs to quickly create and integrate these graphics into the text.

Fontastic Generated »fonts« Equalizer-Font.pde and PieChartFont.pde

More examples have been created by the Processing Library Fontastic. Some of the rather artistic approaches can be actually used as visualisation: Equalizer-Font.pde represents each letter with equalizer-like bars, PieChartFont.pde adds a sector for each letter to a pie chart, and FrequencyFont.pde uses the frequency of the letters in the English language to create corresponding sized bars for every typed letter. (Koller, 2014)

The idea to represent each letter with a graphic has mostly found application in art. Lauren DiCioccio (without date) for example encodes each letter with a colour dot, Catalina Viejo Lopez de Roda (without date) writes letters with paper shavings in content-related colour schemes, and the colouring of piano keyboards and music sheets can help students to connect musical notations with the keys. Besides the accentuation of the actual letters, the first two projects unfortunately do not help the text content, but in fact make it nearly impossible to read.

Sol Lewitt This piece of art shows a network of the word »art« on two pages

One seldom approach is the analog network visualisation in a book by the artist Sol LeWitt. He used printed pages in a book to draw networks of the word »art«. He connected each appearance with red lines, and the corners and the midpoints with blue and green lines. (Castro, without date) While network visualisations often use force directed layouts that place objects in regard to their »connectivity«, this layout is similar to a map that uses the geo-spacial placement of the nodes to draw the edges. In this case the words are located all over the text, the connections can help to comprehend the structure of the text. Unfortunately his artistic approach does not support this and is probably not meant for this.

Adjacent

This is certainly how most regular visualisations work: A graphic as an addition to a describing text. To differentiate relevant graphics for this thesis, they need to support the text’s content rather than visualising something completely new. The above mentioned »The Russia Left Behind« micro visualisation is rather useless on its own without any context, but is essential in combination with the text.

The first (unscientific) project is the children’s book »Where The Wild Things Are« written and illustrated by Maurice Sendak. While this example is not a visualisation in a strict sense it literally illustrates how the image-text relation can evolve over the course of a book and hence its examination can also help to create data-driven visualisations. The double pages mostly consist of text and illustration areas. As in most picture books, the images visualise the content of the text. What is special in this book is the area each part takes of the page. As the protagonist drifts in his fantasy world, the story becomes more surreal and the pictures take over the page. Michael McAghon describes the illustrations to »indicate where we are in Max’s concocted world and to reinforce the imagination experience.« (McAghon, 2009) When he comes back into the real world the rational text retrieves importance and the pictures become smaller.

Book layout Sketch by McAghon of the layout of »Where The Wild Things Are«

The practise to show where the reader is currently situated within the book, is commonly achieved in books through running heads and footers, page numbers, headings or colours. In scientific contexts headings are usually supplemented with counting numbers and correspondingly indented. In digital process it is often shown through a progress bar. A visualisation technique that aims to support this as well are dendrograms. Rolf Fredheim’s approach (2013) even shows how the different topics relate to each other. In his implementation he uses only the headlines for each chapter, but it could be (to some extent) applied to the complete text as well. He connects the tree structure from dendrograms with the headlines to visualise the text’s inner structure. Since this visualisation only shows the organisation of the text that could also be extracted from the indention or the running numbers of the headings, he proposes »DendroArcs« that reveal the relations within the text. This network has each chapter as fixed nodes in the list and the relations as edges represented with arcs. The wider the arc, the more connected are the two topics . These (positive and negative) correlations are additionally visualised for each node with a pie chart. While this might help to analyse a vast amount of text in context of digital humanities, this can also help the reader to jump from topic to topic, following his interest.

Dendoarcs The arcs show the relation between paragraphs. The pie charts indicate correlations

Instead of providing overview through graphical elements directly connected to the text, other approaches create symbolic representations of the content. This can span from miniature, wire-frame or even more abstract representations of the content. The above mentioned Sublime Text editor offers a down-scaled version of the written source code in a separate »Minimap«. This allows for navigation in the document but also provides context and overview for the reader/writer. Since the current visible area is also highlighted in the Minimap, it acts like the scrollbar known from various applications. The idea to include a miniature preview in the scrollbar has already been described by Scott McCrickard and Richard Catrambone in »Beyond the Scrollbar: An Evolution and Evaluation of Alternative Navigation Techniques« (1999) with the concern that »considerable information about the content of the space is not communicated in the scrollbar.« (McCrickard et al., 1999) But instead of simply scaling down everything, their »Mural Bars« provide the same space for every item: »Since an equal amount of space is given to each item, any item can be seen with equal clarity no matter where they are in the list.« (McCrickard et al., 1999) A current application of these navigable miniature representation is also the »Navigator« found in Adobe Photoshop and other applications by Adobe. It provides a minimised version of the current image, with an overlaid rectangle for the current viewpoint.

Where you are The screenshot shows »previews« of each article

The website for the book »Where you are« (2013) on the other hand provides wire-frames for each chapter. This visualises not only the length of the article, but also its structure and the picture-text ratio. As discussed in the beginning: We see before we read. The simplified representation of the website mimics this impression. This provides two things: First, we can estimate what we are about to see, and second, when we revisit the website, we can optically search for the specific article we read some time ago. Just like we tend to remember »the red book with the white font» or »the website with the big image at top«, this wireframe presentation can help us recognise from their optical appearance.

An even more abstract representation of a website can be found in the Google Chrome Browser. When the user searches for a term, the locations of the terms in relation to the website’s height are marked in the scrollbar. This concept requires more learning from the user, but is quickly understood since the idea of the scrollbar is used since the 1980s.

Reactive documents When the user changes one value, all others change accordingly.

The last example is mainly shaped by Bret Victor and his »Explorable Explanations« and »Reactive Documents«. (2011) His interactive documents can be best described as textbooks with connected text and diagrams. Most projects explain scientific topics with text, formulas, images, and diagrams. Interactive elements within these elements can be dragged, changed or simple hovered. What makes them so interesting it that as the user changes an element in the text, the corresponding elements in the other elements change accordingly. As Victor states, this interactive dual-coding brings transparency, explanation, and encouragement. (Victor, 2011)

Explorables examples Text, formulas and graphics are connected. The interactive graphs allow interactivity
Conclusion Additional Elements

The two main groups I was able to find as examples for integrated additional visualisations are background-colours so that the colour conveys additional meaning and graphics that are placed behind the corresponding word. The use of the background-colour is mostly inspired by the physical marker. They emphasise words or label to a certain category. The additional graphics can be divided into three groups: Sparkline-like visualisations that are inspired by common charts, symbols like smilies or ideograms and glyphs. Despite the fact that glyphs have been elaborated for some time now, they have not been placed inside the text, but rather as standalone graphics or within a spatial projection. It is up to further research to elaborate this missing application. In general, the symbols were mostly used for qualitative data, while the actual minimised visualisations convey quantitative data. The symbols often substitute a word or emphasise its content; their data source is often innate or intrinsic. In contrast, the data words as described by Tufte add additional extrinsic data to the corresponding word.

Three things that can be noticed in the usage of additional elements are: the size, the placement and the static content of the graphics. All the graphics are placed in the same height and at the same baseline as the text. Moreover, the source of the data is mostly static. The inclusion of dynamic data from the user or live updates is seldom. I see potential for this missing features and will describe it further down.

The external additional category is the most diverse one. The techniques used can be divided into four groups: Image/text-ratio, alternative representation of the content, text-lines/paragraphs as nodes and abstract representation of the actual text. Also all types of data sources are being used for these visualisations. The New York Times graphic adds an additional level of spatial information to the story, the abstract representation shows innate attributes of the text and the edges between the text node represent the intrinsic relations between text passages.

While the modification of the text mostly acted as a supporting element, the additional graphical elements often overlay the existing text with an extra information layer that adds further extrinsic data or show the innate or intrinsic data from another perspective. What needs to be examined separately is the interaction between the text and the additional elements and how they can be recognised as connected. The work of Bret Victor is certainly pioneering for future work.

Conclusion

It can be seen that the existing work already covers a wide area of subjects and also fills out the taxonomy. It has been proven useful to separate these four fields to systematically discuss each project in a systematic way. Some projects used multiple techniques and covered more than one field. This can be seen as an inspirational approach for future visualisations to combine several fields in order to use encodings that are far enough separated in their application to not distract each other. The potential of this combination of multiple types is discussed further down.

So far only the difference between »additional elements« and »modification« has been examined. But when we review integrated and adjacent visualisations, one quality and hence application becomes apparent for each. Inline visualisations allow for an unmatched closeness of the objects in its context and the visual representation of its data points. External visualisations on the other hand allow for the comparison of these visual representations, which is harder when the objects are distributed over the text. In the following chapter I will discuss techniques to overcome these limitations through linking.

Another point that needs to be considered when comparing integrated and adjacent visualisation is the different size. Obviously, graphics that need to be placed inline need to be smaller. »Heer et al. compared small line charts against two types of Horizon Graphs (1-band and 2-band) and investigated the impact of chart height on both designs. They found that small chart heights negatively impacted how accurately and quickly participants estimated the difference between two data points.« (Goffin et al., 2014)

The third aspect that needs to be considered is the reading flow. Graphical elements that pop-out too much may not be welcome and can hinder the readability of the text. Pearson et al. (2009, in Goffin et al., 2014) found out that users preferred the placement of larger annotations to be in the margin. Further empirical research is needed to balance between readability, reading speed and preferences of the reader.

In general, micro visualisations need a different approach than regular visualisations, especially due to their size. Their applications are always trade-offs. One needs to make compromises on the shown data dimensions, the number of data points, the underlaying coordinate system or the legend. Even more than common visualisations they should be utilised for impressions, not exact value extraction. As stated before, the quality of tables is to represent data in an accurate way, text on the other hand guides the reader through the information in a discursive way and visualisations help to get an overall image and an impression of single or multiple data points.

What can be summarised from the existing work is that micro visualisations can help the text in several ways:

  • The modification of the text can optically support the text’s content while reading. It can function as visual impression of the text without reading the whole text and it can help to gain an overview of the text before or after reading it.
  • Additional elements can also support the text’s content, function as anchor point when the user scans the document, and add additional data that is not as important as the text’s content. All this acts as support and hence should focus on the possibility to quickly offer the user an impression of data instead of deviating him from the actual text. Given this, the abandonment of an coordinate system that would normally help the user to get an idea of the data is justifiable.

Evaluation

This chapter builds upon the previous chapter – hence the theoretical argumentation and the analyses of states of the art – and provides a review of the approach.
Blank spots – revealed through the taxonomy – are identified and suggestions made on how this could be accomplished in future work. As a final demonstration of the micro visualisation concept’s value, I provide essential insights from three expert interviews.
In its very last part, my thesis makes some concluding remarks and closes with suggesting future directions in research on micro visualization and its application.

Future approaches

From the gathering, analysis and classification of the existing work several approaches emerge. With the taxonomy blank spots in the micro visualisation sphere can be discovered and also novel types can be deviated through the combination of multiple fields. A further goal is to turn the limitations into advantages and features.

Combination of visualisations

Since micro visualisations are not so dominant in their appearance, they can be combined with other visualisations. In fact this can be seen as one strength of micro visualisations: The combinations of these four types allow for additional data levels that would not be possible with a single visualisation or text.

Combination Two visualisations can coexist, work together as reference or one can act as thumbnail

This can be done in two different ways: Completely independent visualisations showing different data or multiple visualisations that deal with the same data set and support each other. The latter can be for example used with an additional adjacent and an inline approach. The inline visualisation can either work with modifications to highlight their connectedness to the adjacent graphic or with additional elements. These graphics could serve as thumbnails or previews (or scented widgets) for the adjacent visualisation.

Breaking the borders

The fact that inline modifications or additional elements are limited to the line height can not only be seen as limitation but also as stylistic device. When visualisations show data that has outliers or an extreme amplitude, the designer sometimes shows this by literally breaking the layout with these parts. The famous »hockey stick graphs« used in the context of climate change have a steep rise towards the end. In the »Atlas der Globalisierung – Spezial – Klima« (Atlas of Globalisation – Special – Climate, Bovet et al., 2008) by Le Monde diplomatique multiple graphics use this technique, where bars or line charts go beyond the borders of the graphic or even the page. Al Gore used the same technique in his »An Inconvenient Truth« presentation. In this rhetorically impressive gesture he uses a lift to follow the course of the CO2 concentration in the earth’s atmosphere.

Hockey stick graph The CO2-value breaks out

For micro visualisations the same can be used. Topics that break the limit can exceed the borders of the line height. This could be realised with graphs or circles that lay behind the actual text to ensure readability. Visually this would create two layers that need to be optically distinct. When used in a digital medium this could even be enhanced by making each layer responsive in its visibility.

Breaking out The limitations can be used for the message
Responsive data

Regular visualisations often deal with quite a great amount of data that does not change frequently. Micro visualisations on the other hand deal with only very little data points and hence can be updated easily. The jQuery implementation of Sparklines already does this by showing live finance data or the current mouse speed. (Watts, 2012) This can easily be applied for other applications: Current temperatures of cities in a travel guide, next departure times of trains and current traffic jam situations, popularity on Twitter, number of Google search results for that term, or up-to-date user ratings. But instead of showing every user the same information, the visualisations and hence the text could respond to the reader. This is a great opportunity to involve the reader more into the story. Since many articles are written for the general public, the given examples or numbers could possibly not relate to the reader. He might be unfamiliar with the currency, the local time or weather at the described city or even with the location of the place. Similar to the previously described browser plugin »Dictionary of Numbers«, this can help the reader to understand the given information better. To realise this conversion to familiar values, the user can either put in the required data or it could be received through the browser’s information.

Contextual information became quite a buzzword in the recent time. Google’s advertising is so successful because of their knowledge about the user. They can individually change the advert to the user’s profile. Furthermore, their search results depend on previous search queries and the current location of the user. Similar approaches for visualisations seem promising and should be elaborated in detail in future works.

Varying inline placement

In »Exploring the Placement and Design of Word-Scale Visualizations« Goffin et al. (2014) empirically tested the various possible placements additional visualisations could be in. Besides interesting interactive and dynamic placements, they evaluated seven static positions:

  • Traditional In-Line Placement
  • Overlay Word-Scale Visualizations on Entities
  • Using Existing Inter-Line Space
  • Using Inter-Line & Increased Inter-Word Space
  • Increasing Inter-Line & Inter-Word Space
  • Allowing Visualizations To Overlap
  • Allowing Visualizations to Overlap Surrounding Text

Besides these fixed positions that are required for printed versions, digital media would allow dynamic placements of the graphics. (Goffin et al., 2014) The most basic form would allow the user to hover over the reference word to display the graphic either by replacing it or placing the graphics next to the element. The other form they describe in the paper is a toggle for all visualisations in the document. This would allow the text to be less cluttered, but on the other hand this would require the input and knowledge of the user. Besides these techniques others which involve numerous interactions would be possible: Visualisation could be become apparent through scrolling, through text input, or depending on the visiting time. Besides the visibility, interaction could also change the position of the graphic. This would allow for individual placement to enhance comparison, but also open up a complete new research.

Interactive micro visualisations

Besides the previously described interactive approaches for the placement, interaction could also be used to enhance comparison between multiple inline elements or the link between adjacent elements with their corresponding words.

Since the comparison is hindered when the visualisations are distributed all over the document, interactions between these can counteract this. Either the complete visualisation or — in the case of modified text — passages could be highlighted while other elements fade out. Another possibility is to highlight corresponding categories that occur in each visualisation. For Sparklines this could be the data points for a certain time.

The second possibility deals with the problem to connect adjacent visualisations to corresponding entities. Steinberger et al. (2011) conducted research on how visual links can connect related information across multiple visualisations. As they describe, their invasive but also more expressive method aims at minimising the occlusion but also make the links visually stand out from surrounding information. Even though they found evidence in their approach, their scenario is not completely comparable. Since my approach focuses on the readability, methods that use an »invisible« line need to be researched. In visualisation this is commonly realised through markers that can be found in the legend and in the graphic. But this requires memorisation and search. As the example »The Russia Left Behind« shows, the link can also be created through interactivity: As the user scrolls down, the relevant part is highlighted.

As previously shown, Bret Victor does this in a similar way in his sophisticated work »Explorable Explanations« and »Reactive Documents«. (2011) The user can hover over the entity and the corresponding part in the visualisation is highlighted. Bas Broekhuizen also describes how to »integrate text and images interactively« (Broekhuizen, 2014) in a blog post. He shows examples of graphics in which areas are highlighted when the user hovers over a link. While this seems simple, it can help the storyteller to guide the reader through the graphic. For this kind of interactivity, I see a great potential, especially in combination of micro visualisation integrated and adjacent as described above.

Minimising regular visualisations

The way Sparklines work is by taking a regular line chart, splitting it up into the different data rows, removing the coordinate system and minimise it to line height. The second step is basically the idea of Small Multiples. These types of graphics are a well elaborated concept that is often used to declutter graphics when too many data rows are involved or data is shown over time, which can result in too many data points in one graphic. The idea is to show the same coordinate system multiple times, each with different data points shown. Small multiples are a field on their own that is in the same way popular as it is criticised. Further research should elaborate how previous research can also help micro visualisations.

Line chart to Sparkline A line chart becomes a Small Multiple, becomes a Sparkline.

But the previously described concept of taking apart regular visualisation brings further general approaches: exploring this technique for regular visualisation types. Applying this to bubble charts, treemaps or stacked bar charts can result in useful applications. In fact, the shown work by Sol LeWitt is a common network diagram, in which the spatial position of the nodes is given by the position within the text. In a similar way the other striped and minimised graphics would have trade-offs to their original form, but besides being useable as micro visualisations, they could result in less cluttered graphics similar to small multiples.

One visualisation type that has not yet been used for micro visualisations are single stacked bar charts. In contrast to Sparklines which use time series data, these bar charts work with percentage values and can hence adapt to the word’s width. When working with multiple visualisation of such type it needs to be elaborated if they should adapt to the size of their corresponding entity, to the size of the smallest entity in the document or extent over the entity’s width to enable comparison.

Interviews

In order to evaluate my approach on a theoretical level I interviewed experts from different fields about their experience from daily life when dealing with text and visualisation, how they assess my approach and what this could mean for the future of visualisation.

Jan Distelmeyer

Prof. Dr. Jan Distelmeyer lectures at the University of Applied Sciences Potsdam for Europäische Medienwissenschaft. In the interview I asked him about his assessment of my approach. In regard to McLuhan’s »Gutenberg Galaxis« he points out that the book does not only influence the linearity of text, but also the uniformity. Handwriting, which still stands as a symbol of the one’s own. But the unique has given way to the always equally looking letterpress. Every book suddenly looks the same. But McLuhan does not focus on the medium itself. For him the medium is always equate to technology. What he is actually interested in is the connection between the technology and the cultural and social impact it has. That is why he stresses not on the disappearance of handwriting in terms of cultural loss, but rather the effects. The letterpress allowed for a greater production of books, allowing all social strata access knowledge. During the French Revolution the term « égalité » does not only reflect the equality of people but also the equality in writing: A symbol for accessibility to books and the liberation from dependency. The notational appearance represents the people.

An example in which this relationship between typeface and handwriting is addressed is one project named »Schönschrift — Notizen zur Hochkultur« (»Best Writing — Notes on high culture«). Together with reviews on various subjects, the authors attached their hand written notes as PDF. The genesis of the text from individuality to uniformity can be understood.

The next question was, how he assesses the relationship between the typography and the text. Distelmeyer states that we need to be aware of the far-reaching consequences the figure has for plain text we read. The fact that newspapers define what the articles should look like illustrates this importance. But he emphasises that not only the typography, but also factors like the brand-typical style of writing have influences on how we perceive the content. Many accompanying symptoms come with usage and form of the book, many are being described by Gérard Genette.

Distelmeyer also explains that the combination of text and graphic in fact does not cancel the linearity of the text, the same prescribed structure of the author. Even when these graphics contextualise, interrupt, or actually question the text, they are part of this format. In contrast to this, the general structure of the web allows readers to individually follow links, scroll or quit the website. This results in a new form of perception, with an structure that is individual to each reader.

Answering the question on how he evaluates Nobert Bolz’ »Prägnanz« of images, Distelmeyer states that Bolz’ media ontological approach cannot be applied so generally. He explains that the progress the photo has made can now be seen with the diagram. Probably no one would nowadays agree that a photograph shows the perfect representation of the reality it was praised for in the beginning. The obviousness and factuality that tables or diagrams have today is about to face the same fate. The quality a medium holds are more a cultural tradition of how we treat it. The practical value does not correlate with the quality of the table or graphic itself.

The changing usage and treatment of language through technology is an exciting progress. Even though the principles stay the same the dissemination and application differs. In recent time, the evolvement of the PC brings in another interesting factor: The computer does not care if the data represents a text or a graphic. Luckily all the information are being processed for us to read them. But in fact, this results in an even greater uniformity. Producing, viewing and distribution all meet in one device. The idea of « égalité » is even stronger when the printing house and broadcaster are in the hand of the producer. The acceptance of the people to mix different media is thus becoming even stronger. Only in recent time, this empowerment was discussed as a deprivation of power through the surveillance enabled through the electronic processing.

Sascha Venohr

The »Head of Data Journalism« for ZEIT Online Sascha Venohr has worked at the interface between journalism and technology for many years. In my interview, I asked him about the current state of online journalism and visualisations. For him, the two most important aspects about typography is the recognition value and the legibility. People recognise the brand Die ZEIT Online/Die ZEIT in the fonts, the layout and all the typographic decisions. The aspect of type/content matching has to recede in favour of constant legibility. In this tightly synchronised environment practicability is a crucial factor. The author is also the designer when working with the CMS that offers limited typographic options like bold, italic, underline and bullet points. Extended possibilities come with »raw modules« that can be placed between paragraphs and offer complete freedom over the source code. These modules do not interact with the surrounding elements and also cannot demolish the layout.

Complex stories that have been designed in the past and break out of this structure gather a lot of attention but are rather boon and bane. Every technical deviation entails efforts on the technical side. Venohr stresses the fact that online journalism has to supply content for multiple output devices: The website provides views on different screens, the app uses some of these layouts, while other content is pulled out in a plain text-format. This requires every exception from the common format to be defined for every type. Basically, the article needs to be medium-neutral, otherwise it would require more work by the author. The diversity of devices forces the layout back to a form print layouts have had many years ago. Venohr states that from this perspective, print media have greater flexibility. In response to the question on whether the reader might not be able to understand complex , »visualising« layouts, he admits that they are not experimenting enough. They rather under-challenge the reader.

A question I was asking myself concerning the combination of data visualisations and text is whether this could convey more »truth« than a (persuasive) text. Venohr states that as a data-journalist, he is also a story teller. Presenting unedited, unfiltered amounts of data is not the journalist’s job. By picking interesting parts and not overwhelming the reader the story becomes interesting. He explains that the data used in visualisations often lacks reliability. In classical journalism, stories are required to use at least two sources for information. Data-journalists are depending on data and yet the information used in many graphics is based on only one – often in transparent – source. For example, the unemployment rate is based on numbers by the »Bundesagentur für Arbeit«. The veracity cannot be verifiable since no other source can provide this data. Mistakes during the processing can also occur. The more data the author handles, the more likely it is to make mistakes. Unfortunately, these faults cannot always be noticed immediately.

Sascha Venohr sees a great opportunity to use visualisation to bring the story closer to the reader. In recent works they used the user’s input (for example the zip code) to customise the appearance of the visualisation. Venohr stresses that interaction is not only customisation but can also be used to immerse oneself in the story. It increases the »engagement«. The ability to share a custom view or test results from the graphic (for example the hours of average sleep) with friends is more and more desired by the user and also results in a higher number of visitors through their sharing. But automatically presenting different data based on information received from the browser does not find a ready market. A recent example is a visualisation about the Eurovision Song Contest. Based on the user’s language settings the graphic was shown from a different perspective. People did not expect that they influenced the »state of aggregation« of the graphic and thus had difficulties when sharing it or exchanging opinions about it. This required the newspaper to built in information messages with options to change the view and thus making it more transparent for the reader. The same applies for mobile and desktop versions of graphics that sometimes show information in different amounts of detail. Venohr states that we are not ready for this yet. Journalists still need to figure out how to deal with this paternalism.

Jan Schwochow

The infographic artist Jan Schwochow was head of department infographic for both Stern and KircherBurkhardt before he founded his own agency Golden Section Graphics. For him, the chosen medium (may it be text, image, graphic, film or sound) has to suit the information. The technique which, allows the reader to decode the content the quickest and the most efficiently, is chosen. This is of course a question of what actually can be transported through an image. Text has dominated the knowledge transfer for a long time, but not all information are equally easy to communicate linguistically. Lengthly descriptions of voting results for example are much easier to describe in a graphic. Schwochow wishes for as little text as necessary. On the other hand, he admits that text is sometimes irreplaceable. As labels, legends or explanation text makes the graphic understandable.

In general, he wishes for more images in all kinds of media. One example is the popular science book »Darm mit Charme« by Giulia Enders. The text is undoubtedly well written but Schwochow wishes for better graphics. The illustrations in the book may be charming and funny, but do not add any value to the content. They simply represent the content in an illustrative way. Schwochow sees a great opportunity to offer visualisations that would allow the reader for a deeper immerse. This could help to transport more detailed information about topics in the book.

One reason why especially daily newspapers do not use more graphics is a lack of time. Content needs to be produced quickly, but many graphics still need more time to be produced. Moreover, the environment in which the content is consumed is important. Many people read newspapers on the go on the their mobile phone. Large, explanatory graphics do not suit the small display and the reading behaviour. Schwochow sees a great opportunity for Sparklines to especially work in this environment. The Apple Watch could potentially even increase this. These small graphics are quicker to produce and thus would allow application even in time critical projects. They also lead to a different storytelling. While bigger graphics allow for exploration, micro visualisations are rather explanatory. He says that these two types could prospectively be the difference between mobile and desktop usage. The context and hence the communication goal defines the appearance of the graphic in the future. This can already be seen in the interactive graphic »Lobbying in Europe« for Transparency International (I was involved in the production of this graphic). The countries can be explored on a map on the desktop version and in a simple list on mobile phones. The map would not allow the user to satisfactorily navigate the graphic on a small screen and hence he can use a simplification of the graphic.

Typography being used for visualisation is one thing we definitely see more in the future, says Schwochow. He illustrates the intelligent usage of typography in cartography: Every font-size, style and positioning has a meaning. While typography was a crucial part of the corporate design until now, these boundaries soften more and more in the future. We can already see that there are only »news«, which were suggested by our friends. The author becomes more and more irrelevant. For the brand this means a shift in their positioning. Tesla is a good example of a company that is already doing this. The idea (the electric car) is more important than the appearance. They rather focus on the emotion: They describe it as »Tesla moment« when one sits in a car that accelerates quicker than most Porsche’s. What they communicate is authenticity and this strategy has become the brand. Schwochow says that we should take away the typography from the CD to communicate actual content. Based on this development he predicts a big opportunity for »visualisation« typography.

Conclusion

In summary, this thesis finds convincing support for a complementarity of text and visualisations. Arguments for the integration of text and visualisations in terms of micro visualisations have been made on theoretical grounds by showing shortcuts in existing theories as well as by pointing out compelling benefits of applying such an integrated approach. Also existing research and expert opinions back up this potency as well as they point to an encouraging wide range of possible applications and purposes. In summary, the question how Micro Visualisations can enhance text comprehension, memorability, and exploitation could be answered with multiple examples.

The proposed taxonomy has proven as an useful methodological framework for assessing the state of the art in Micro Visualisations. Approaches of both, modification and addition, have been shown to be distinguished by individual qualities, limitations, and purposes. Therewith, the taxonomy allows for a novel perspective on the application of visualisation which may flourish even beyond the outlined future approaches.

In the following the contributions of each part are listed.

Contributions

The overarching goal of this work was to substantiate the concept of micro visualisations. Instead of inventing a new type of visualisation this thesis focused on a new type of application of existing work. Especially in regard to typography the question was »What can we do with existing tools?«

Since it is a new approach in the field of visualisation this work mainly laid the theoretical foundation with a media theoretical deviation, a structuring taxonomy, and an elaboration of the current state, and thus aids as a starting point for future practical work. The thesis led to four main types of contributions: evidence, concept, reflection, and outlook. One part that is certainly missing in this work are concrete design studies that use empirical tests to show evidence.

Derivation as evidence

The derivation that leads to the concept of micro visualisation is both directed from a media theoretical side and a design/visualisation perspective. This multi-view aspect is also present in the interviews in which experts from different disciplines were asked for their assessment and experience with the topic.

The analysis of these resources lead to a confidence in the actual functioning is based on scientific studies from both psychological and design research. Especially the models of dual coding and cognitive load support the proposed approach.

Taxonomy as concept

The taxonomy proposed in this thesis aims on the type of application a visualisation has rather than its actual appearance. This allows for a wider currency among the data visualisation community as it can be stretched to other fields. In regard to existing taxonomies that focus on the type or visual variable, the suggested system acts as meta-taxonomy, in which current systems can also fit in. The distinction between modifying and adding visualisations, between integrated and adjacent placement allows to find new combinations, new utilisation, and new interactions of known graphics. In the new field of micro visualisations this functional taxonomy helps to structure existing projects and reveal white spots that have not been elaborated yet.

State of the art report as reflection

Since many concepts, projects, and graphics that can be regarded as micro visualisations already exist, one main contribution of this work is a state of the art report (STAR). This concept of review is used in the visualisation community to capture the great amount of work that is happening in the different areas of the subject. These reports are especially useful since they provide an overview of existing view for both extension and improvement. To prove the concept of the taxonomy, the existing work has been reviewed in the same structure. But not only practical projects have been elaborated but also the theoretical research that is related to this approach is covered in the derivation and allows continuation.

Future work as outlook

Based on these three parts I provide an outlook of future work that needs to be elaborated on the basis of this thesis. Through the analysis of recent and current work this thesis revealed multiple starting points for future work. Especially the taxonomy can act as tool to identify not yet realised projects. The most important aspects are listed in the following part.

Future Research

Empirical research

The scientific research from related topics suggests evidence for the use of micro visualisations. Regular visualisations have been proven to be effective for communicating information, the text’s status is undisputedly important for knowledge transfer and even the combination of images and text has been evaluated. But very little empirical research has been conducted that actually supports this thesis’ approach. Personally, I see the most urgent questions in the comprehensibility of these text/graphics-documents, the change of the reading flow and behaviour and in the long-term effects, especially on how memorable the text’s content is.

But also the attitude of the common reader towards such questionable enhancements needs to be tested. While some visualisations can inconspicuously be integrated and adapted, other (especially integrated) changes could initially distract the reader and not be recognised as enhancement or as visualisation at all. The challenge is to find compromises between the accepted and learned form of text and images and beneficial, subliminal and learnable combinations.

Text-Bild-Wissenschaften

In his pleading »Narrative images and pictorial literature — Pleading for a text-image-culture« (Erzählende Bilder und bildhafte Literatur — Plädoyer für eine Text-Bildwissenschaft, 2006) Horst Wenzel discusses the history of the combination of images and text with a focus on medieval illustrations. While his idea of images certainly differs from the visualisations discussed in this thesis, his pleading is nonetheless applicable and useful for the debated images here. He criticises the systematic separation of the science of art and of literature. As he states this detachment has never been dogmatically cultivated by the artists like Leonardo da Vinci, Albrecht Dürer, Alfred Kubin or Günther Grass themselves but instead was held up by the academics. Not only do the described projects with a strong amalgamation of images and text verify this statement, but also the concepts like concrete poetry or FatFonts that cannot be clearly separated as text or image anymore dispute the old classification. Despite this being a niche application compared to purely textual or pictorial forms, the research of the combination holds potential for the rising merging in the future.

But not only the general visual culture should be discussed in media studies, but the only recently emerged field of »diagrammatic« should be paid particular attention. This grammar of diagrams (in the broader definition of representation with graphical means) discusses mental and materialistic diagrams, theories of knowledge, and also cultural and media theoretical aspects of visualisation. (Bauer et al., 2010) With the recently proclaimed era of big data (Economist, 2010) the »tools« which make the superabundant amount of information manageable in a visual way need to be further developed.

Technological change

As technology evolves, both text and visualisations need to adapt to new environments and behavioural patterns. Different display sizes for example are still challenging designers when creating graphics. Micro visualisations appear to be one solution when dealing with varying layouts. Since integrated graphics can be mostly treated as text, the responsive design applied to text could be used in a similar way. But also applications that are placed in a small size per se – as in smartwatches or as a single graphical interface element – can benefit from this knowledge.

Alternatively, micro visualisation could be seen as the representation of regular graphics for smaller screens. As the display becomes smaller, regular visualisations are taken apart and positioned along their reference objects. For this, not only the question of responsive design but also of responsive content could become fertile.

Technological change also brings new possibilities of realisation. Frameworks like jQuery, D3, and Skrollr already changed the way websites and visualisations are programmed. The jQuery plugin by Gareth Watts already allows numerous dataword-graphics like Sparklines. Many of the discussed techniques can already be realised through current technology. When the field of micro visualisations gains more attention, frameworks specifically created for this type of graphics are debatable.

Closing Remarks

Through the course of the work I have gained strong confidence in the expediency of the outlined approach. Also, the expert interviews have confirmed the thesis’ practical relevance. This thesis has aimed to provide a starting point in this novel field. In all modesty, the outlined arguments and evidence promise to constitute a substantial basis on which future research can build.

As a designer, the products one creates are in most cases also relevant to one’s own life. Accomplishing this work, I wished the changes I propose herein would already have been realised to facilitate my research. Especially with today’s technological possibilities there are unseen opportunities to create novel media of knowledge. Books have come a long way; it is time to take them to the next level — in any form possible.

Appendix

Bibliography

  • Alexander, Eric; Kohlmann, Joe; Valenza, Robin; Witmore, Michael; Gleicher, Michael (2014) Serendip: Topic Model-Driven Visual Exploration of Text Corpora. IEEE Symposium on Visual Analytics Science and Technology 2014 November 9-14, Paris, France
  • Assmann, Prof. Dr. Jan (2002): Frühzeit des Bildes: Der Iconic Turn im Alten Ägypten, Iconic Turn - Felix Burda Memorial Lectures: Hubert Burda Stiftung. https://www.youtube.com/watch?v=-TGTN4NFYWw (Retrieved 2015-06-08)
  • Bachmann-Medick, Doris (2008): Gegen Worte – Was heißt ›Iconic/Visual Turn‹? In Gegenworte - Visualisierung oder Vision? Bilder (in) der Wissenschaft. 20. Heft Herbst 2008 http://bachmann-medick.de/wp-content/uploads/2012/10/Bachmann-Medick%20Iconic%20Turn:Gegenworte.pdf (Retrieved 2015-05-24)
  • Bauer, Matthias; Ernst, Christoph (2010): Diagrammatik - Einführung in ein kultur- und medienwissenschaftliches Forschungsfeld. 1. Auflage. ranscript Verlag, Bielefeld
  • Bertin, Jacques (1983): Semiology of Graphics: Diagrams, Networks, Maps. University of Wisconsin Press, Wisconsin
  • Boehm, Gottfried (2007): Iconic Turn. Ein Brief. In Belting, Hans (Publisher): Bilderfragen: Die Bildwissenschaften im Aufbruch, pages 27–36. Wilheml Fink Verlag, München
  • Boehm, Gottfried (2014): Bildbeschreibung. Über die Grenzen von Bild und Sprache. In Müller, M. R. et al. (Publisher): Grenzen der Bildinterpretation, Wissen, Kommunikation und Gesellschaft, pages 15-24. Springer Fachmedien Wiesbaden 2014
  • Bolz, Norbert (2012): Nachhaltigkeit im Informationsraum. See-Conference #7. https://vimeo.com/41987401 (Retrieved 2015-05-23)
  • Borgo, R.; Kehrer, J.; Chung, D. H. S.; Maguire, E.; Laramee, R. S.; Hauser, H.; Ward, M.; and Chen, M. (2013): Glyph-based Visualization: Foundations, Design Guidelines, Techniques and Applications - STAR – State of The Art Report: Eurographics 2013
  • Bovet, Philippe; Rekacewicz, Philippe; Sinai, Agnès; Vidal, Dominique (2008): Le Monde diplomatique: Atlas der Globalisierung Spezial - Klima. »Le Monde diplomatique«/taz Verlags- und Vertriebs GmbH, Berlin
  • Bret Victor (2011): Explorable Explanations. http://worrydream.com/ExplorableExplanations/ (Retrieved 2015-06-08)
  • Bringhurst, Robert (2012): The Elements of Typographic Style. Fourth Edition (version 4.0). Hartley & Marks, Seattle, Vancouver
  • Broekhuizen, Bas (2014): How to integrate text and images interactively. http://www.basbroekhuizen.nl/2014/03/12/how-to-integrate-text-and-images-with-interactivity/ (Retrieved 2015-06-08)
  • Brooks, Evan (2014): Coding in color. https://medium.com/@evnbr/coding-in-color-3a6db2743a1e (Retrieved 2015-05-09)
  • Burdick, Anne; Drucker, Johanna; Lunenfeld, Peter; Presner, Todd; Schnapp, Jeffrey (2012): Digital_Humanities. Massachusetts Institute of Technology, Massachusetts
  • Cairo, Alberto (2013): The Functional Art - An Introduction To Information Graphics. New Riders, Berkeley
  • Card, S. K.; Mackinlay, J. D.; Shneiderman, B. (1999): Readings in Information Visualization: Using Vision to Think. Academic Press, San Diego
  • Castro, Ricky (without date): Sol Le Witt Lecture. http://www.visible.org/site/cornish/slide_lectures/sol_le_witt/ (Retrieved 2015-06-08)
  • Chen, Min; Floridi, Luciano (2012): An analysis of information visualisation. Springer Science+Business Media Dordrecht
  • Childers, Terry L.; Jass, Jeffrey (2002): All Dressed Up With Something to Say: Effects of Typeface Semantic Associations on Brand Perceptions and Consumer Memory. In Journal Of Consumer Psychology, 12(2). Pages 93–106. Lawrence Erlbaum Associates
  • Cobett, Greville G. (2000): Number. Cambridge University Press, Cambridge
  • Contropedia (2015) http://contropedia.net/ (Retrieved 2015-05-08)
  • Davies, Jason (2007): Word Trees. https://www.jasondavies.com/wordtree/ (Retrieved 2015-06-08)
  • de Groot, Luc(as): Case Study: Jungle World. http://www.lucasfonts.com/case-studies/jungle-world/ (Retrieved 2015-08-08)
  • de Roda, Catalina Viejo Lopez (without date): Letter Collages - Cartas Collage http://www.catalinaviejo.com/collagemenuletters.htm (Retrieved 2015-06-08)
  • DeRouchey, Bill (2010): History of the Button. SXSW Talk on March 12, 2010
  • DiCioccio, Lauren (without date): http://laurendicioccio.com/paintings- (Retrieved 2015-06-08)
  • Didi-Huberman, Georges (2007): Bilder trotz allem. Translated by Peter Geimer. Wilhelm Fink Verlag, München
  • Drucker, Johanna (2014): Graphesis - Visual Forms of Knowledge Production. Harvard University Press, Harvard
  • Duarte, Nancy (2010): Resonate - Present Visual Stories That Transform Audience, John Wiley & Sons, New York
  • E. Borra, D. Laniado, E. Weltevrede, M. Mauri, G. Magni, T. Venturini, P. Ciuccarelli, R. Rogers, and A. Kaltenbrunner (2015): A Platform for Visually Exploring the Development of Wikipedia Articles, ICWSM ‘15 - Proceedings of the 9th International AAAI Conference on Web and Social Media
  • Eames, Charles and Ray (1977): Powers of Ten https://www.youtube.com/watch?v=0fKBhvDjuy0 (Retrieved 2015-05-07)
  • Economist Newspaper Limited, The (2010): Data, data everywhere – A special report on managing information. February 27th 2010
  • Few, Stephen (2014): Data Visualization for Human Perception. In Soegaard, Mads and Dam, Rikke Friis (Publisher). The Encyclopedia of Human-Computer Interaction, 2nd Edition. Aarhus, Denmark: The Interaction Design Foundation. https://www.interaction-design.org/encyclopedia/data_visualization_for_human_perception.html (Retrieved 2015-06-08)
  • Fredheim, Rolf (2013): Visualising Structure in Topic Models. http://quantifyingmemory.blogspot.de/2013/11/visualising-structure-in-topic-models.html (Retrieved 2015-06-08)
  • Frutiger, Adrian (1998): Der Mensch und seine Zeichen - Schriften, Symbole, Signete, Signale. 6. Auflage: Fourier Verlag HmbH, Wiesbaden
  • Fry, Ben (2009) On the Origin of Species: The Preservation of Favoured Traces. http://benfry.com/traces/ (Retrieved 2015-05-08)
  • Gendel, Ido; Baumann, Benny (2010): A brief overview on Syntax Highlighting. http://blog.benny-baumann.de/?p=766 (Retrieved 2015-05-09)
  • Goffin, Pascal; Willett, Wesley; Fekete, Jean-Daniel; Isenberg, Petra (2014): Exploring the Placement and Design of Word-Scale Visualizations. IEEE Transactions on Visualization and Computer Graphics, Institute of Electrical and Electronics Engineers (IEEE)
  • Goldstein, E. Bruce (2002): Wahrnehmungspsychologie. Spektrum Akademischer Verlag, Heidelberg, Berlin
  • Groeger, Lena; Currier, Cora (2012): Stacking Up the Administration’s Drone Claims. ProPublica Sept. 13, 2012. http://projects.propublica.org/graphics/cia-drones-strikes (Retrieved 2015-06-08)
  • Hausmann, Albrecht (2009): Zukunft der Gutenberg-Galaxis. In Aus Politik und Zeitgeschichte (APuZ 42-43/2009): Zukunft des Buches, pages 32-39. Bundeszentrale für politische Bildung, Bonn
  • Haverkamp, Michael (2009): Synästhetisches Design: kreative Produktentwicklung für alle Sinne. Carl Hanser Verlag GmbH, München
  • Heer, Jeffrey; Bostock, Michael; Ogievetsky, Vadim (2010): A tour through the visualization zoo. Communications of the ACM, 53(6):59–67
  • Heller, Eva (2009): Wie Farben wirken – Farbpsychologie, Farbsymbolik, Kreative Farbgestaltung. 5th Edition. Rowohlt Verlag GmbH, Hamburg
  • Herrmann, Ralf (2010): Das Zwiebelschichtenmodell der Lesbarkeit. In TypoJournal 2: Wayfinding & Lesbarkeit. Pages 2–8. Seite7 Designagentur, Jena
  • Herterich, Owen (2014) To See & Hear. http://owenherterich.com/projects/dialogue/ (Retrieved 2015-06-08)
  • Hofmaier, Mike (2013): Verfassung verstehen: Das Grundgesetz in Infografiken Eine visuelle Analyse der deutschen Verfassung. Hermann Schmidt Verlag, Mainz
  • Imhof, Eduard (2007): Cartographic relief presentation. Esri Press, Redlands
  • Kittsteiner, Heinz D. (2004): »Iconic turn« und »innere Bilder« in der Kulturgeschichte. In: Kittsteiner, Heinz D.: Was sind Kulturwissenschaften? 13 Antworten. Pages 153–182 Paderborn 2004: Wilhelm Fink Verlag, München
  • Klein, Scott (2014): The forgotten History of News Graphics. At Malofiej 22, Pamplona
  • Kochel, Travis (2012): How to Use FF Chartwell. https://www.fontfont.com/how-to-use-ff-chartwell (Retrieved 2015-06-08)
  • Koller, Andreas (2014): Fontastic. http://code.andreaskoller.com/libraries/fontastic/ (Retrieved 2015-06-08)
  • Kosara, Robert (2007): A Critique of Chernoff Faces. https://eagereyes.org/criticism/chernoff-faces (Retrieved 2015-06-08)
  • Krämer, Sybille (2003): »Schriftbildlichkeit« oder: Über eine (fast) vergessene Dimension der Schrift. In Krämer, Sybille; Bredekamp, Horst: Bild Schrift Zahl. Pages 157–176. Wilhelm Fink Verlag, München
  • Lidwell, William; Holden, Kritina; Butler, Jill (2010): Universal Principles of Design. Rockport Publishers, Massachusetts
  • Maletzke, Gerhard (1998): Kommunikationswissenschaft im Überblick: Grundlagen, Probleme, Perspektiven. Westdeutscher Verlag GmbH, Opladen/Wiesbaden
  • McAghon, Michael (2009): An Imagination Experience. http://blog.mcaghon.com/post/98471242041/an-imagination-experience (Retrieved 2015-05-25)
  • McCloud, Scott (2001): Comics richtig lesen - Die unsichtbare Kunst. Carlsen Comics, Hamburg
  • McCrickard, D. Scott; Catrambone, Richard (1999): Beyond the Scrollbar: An Evolution and Evaluation of Alternative Navigation Techniques, Georgia Institute of Technology, Georgia
  • McLuhan, Marschall (2011): Das Medium is die Massage - Ein Inventar medialer Effekte. With Quentin Fiore. 3rd Edition. Compiled by Agel, Jerome (Publisher). Tropen Sachbuch, Leipzig
  • McLuhan, Marshall (1969). Interview in Hefner, Hugh (Publisher): Playboy Vol. 16, no. 3 – March 1969
  • Mendelssohn, Moses (1989): Jerusalem oder über religiöse Macht und Judentum. In Mendelssohn, Moses: Schriften über Religion und Aufklärung. Pages 421f. Published by Thom, M., Darmstadt
  • Mendelssohn, Moses (2011): Jerusalem – Religious Power and Judaism. Published by Bennett, Jonathan. Early Modern Texts. http://www.earlymoderntexts.com/pdfs/mendelssohn1782.pdf (Retrieved 2015-06-08)
  • Miyazaki, Momo; Karambelkar, Manas; Robertsen, Kenneth Aleksander (2012): Silenc. http://ciid.dk/education/portfolio/idp12/courses/data-visualisation/projects/silenc/ (Retrieved 2015-05-08)
  • Motley, Michael T. (1990): On whether one can(not) not communicate: An examination via traditional communication postulates. In Western Journal of Speech Communication, 54 (Winter 1990). Pages 1–20
  • Müsseler, Jochen; Prinz, Wolfgang (Publisher) (2002): Allgemeine Psychologie. 1. Auflage. Spektrum Akademischer Verlag, Heidelberg, Berlin
  • Myers, Brad; Hudson, Scott E., Pausch, Randy (2000): Past, Present, and Future of User Interface Software Tools. Human Computer Interaction Institute, School of Computer Science, Carnegie Mellon University, Pittsburgh
  • Nacenta, Miguel; Hinrichs, Uta; Carpendale, Sheelagh (2012): FatFonts: Combining the Symbolic and Visual Aspects of Numbers. Conference: Proceedings of the International Working Conference on Advanced Visual Interfaces
  • Noirhomme-Fraiture, Monique; Randolet, Frédéric; Chittaro, Luca; Custinne Grégory (2005): Data Visualizations on small and very small screens. In ASMDA: Proceedings of Applied Stochastic Models and Data Analysis
  • Potter, Kristin; Rosen, Paul; Johnson, Chris R. (2012): From Quantification to Visualization: A Taxonomy of Uncertainty Visualization Approaches. Scientific Computing and Imaging Institute, University of Utah
  • Propublica (2012): StateFace. http://propublica.github.io/stateface/ (Retrieved 2015-06-08)
  • Reichenstein, Oliver (2012): Responsive Typography: The Basics http://ia.net/know-how/responsive-typography-the-basics (Retrieved 2015-05-08)
  • Reuß, Roland (2014): Die perfekte Lesemaschine - Zur Ergonomie des Buches. Ästhetik des Buches. Wallstein Verlag, Göttingen
  • Santa Maria, Jason (2014): On Web Typography. A Book Apart, New York
  • Schmidt, Gavin (2014): The emergent patterns of climate change. TED Talk. http://www.ted.com/talks/gavin_schmidt_the_emergent_patterns_of_climate_change/transcript?language=en#t-62460 (Retrieved 2015-05-07)
  • Shneiderman, Ben (1996): The Eyes Have It: A Task by Data Type Taxonomy for Information Visualizations. Department of Computer Science Human-Computer Interaction Laboratory, and Institute for Systems Research University of Maryland
  • Steinberger, Markus; Waldner, Manuela; Streit, Marc; Lex, Alexander; Schmalstieg, Dieter (2011): Context-Preserving Visual Links. IEEE Transactions On Visualization And Computer Graphics, Vol. 17, No. 12, December 2011
  • Stephan Thiel (2010): Understanding Shakespeare. http://www.understanding-shakespeare.com/ (Retrieved 2015-06-08)
  • Stern, Joanna (2015): How I Learned to Love Writing With Emojis. The Wall Street Journal. 19.05.2015 http://graphics.wsj.com/how-i-learned-to-love-writing-with-emojis/ (Retrieved 2015-05-23)
  • Stetter, Christian (2005): Bild, Diagramm, Schrift. In Grube, Gernot; Kogge, Werner; Krämer, Sybille (Editors): Schrift. Kulturtechnik zwischen Auge, Hand und Maschine. Pages 115–135. Wilhelm Fink Verlag, München
  • Stevens, S. S. (1946): On the Theory of Scales of Measurement. In Science Vol. 103, No. 2684
  • The Unicode Consortium (2015): Emoji and Dingbats http://www.unicode.org/faq/emoji_dingbats.html#1 (Retrieved 2015-06-08)
  • Todorovic, Dejan (2008): Gestalt principles. Scholarpedia, 3(12):5345. http://www.scholarpedia.org/article/Gestalt_principles (Retrieved 2015-06-08)
  • Tufte, Edward (without date): Saturn embedded in Galileo’s text, best work of analytical design ever? http://www.edwardtufte.com/bboard/q-and-a-fetch-msg?msg_id=00004p (Retrieved 2015-06-08)
  • Tufte, Edward R. (2001): The Visual Display of Qunatitative Information. Second Edition. Graphics Press, Cheshire, Connecticut
  • Tukey, John W. (1965): The Technical Tools of Statistics. In The American Statistician, Vol. 19, No. 2. (Apr., 1965). Pages 23-28
  • Van Dam, Andries (2001): User Interfaces: Disappearing, Dissolving, and Evolving. Communications Of The Acm. March 2001/Vol. 44, No. 3
  • von Baeyer, Hans Christian (2003): The New Language of Science. Cited in Lima, Manuel (2009): Information Visualization Manifesto. http://www.visualcomplexity.com/vc/blog/?p=644 (Retrieved 2015-06-08)
  • Wang, Steve C. (2008): Professor Puts a Face on the Performance of Baseball Managers. Interview with Alan Schwarz. http://www.nytimes.com/2008/04/01/science/01prof.html (Retrieved 2015-06-08)
  • Ware, Colin (2004): Information Visualization – Perception for Design. Second Edition. Morgan Kaufmann Publishers, San Francisco, CA..
  • Watts, Gareth (2012): jQuery Sparklines. http://omnipotent.net/jquery.sparkline/ (Retrieved 2015-06-08)
  • Watzlawick, Paul; Beavin, Janet Helmick; Jackson, Don D. (1967): Some Tentative Axioms of Communication. In Pragmatics of human communication: A study of interactional patterns, pathologies, and paradoxes Pages 48–71. W. W. Norton & Company, New York
  • Watzlawick, Paul; Jackson, Don D.; Lederer, William J. (1967): Pragmatics of Human Communication: A Study of Interactional Patterns, Pathologies, and Paradoxes. W. W. Norton & Company, New York
  • Wenzel, Horst (2006): Erzählende Bilder und bildhafte Literatur. Plädoyer für eine Text-Bildwissenschaft. In Maar, Christa; Burda, Hubert (Publisher): Iconic Worlds. Neue Bilderwelten und Wissensräume. Pages 232–250. DuMont Buchverlag, Köln
  • Where You Are (2013): A Collection of Maps That Will Leave You Feeling Completely Lost. http://where-you-are.com/ (Retrieved 2015-06-08)
  • Willett, Wesley; Heer, Jeffrey; Agrawala, Maneesh (2007): Scented Widgets: Improving Navigation Cues with Embedded Visualizations. Computer Science Division at the University of California, Berkeley
  • Wirth, Niklaus (1976): Algorithms + Data Structures = Programs. Eidgenössische Technische Hochschule Zürich. Prentice-Hall, New Jersey
  • Wittgenstein, Ludwig (1922): Tractatus Logico-Philosophicus. Ogden, K. (Publisher): International Library of Psychology Philosophy and Scientific Method. London Kegan Paul, Trench, Trubner & Co., Ltd. New York: Harcourt, Brace & Company, Inc.. http://www.gutenberg.org/ebooks/5740 (Retrieved 2015-06-08)
  • World Economic Forum (2011): Personal Data: The Emergence of a New Asset Class. http://www3.weforum.org/docs/WEF_ITTC_PersonalDataNewAsset_Report_2011.pdf (Retrieved 2015-06-08)
  • Zwabel (2009): C++ IDE Evolution: From Syntax Highlighting to Semantic Highlighting. https://zwabel.wordpress.com/2009/01/08/c-ide-evolution-from-syntax-highlighting-to-semantic-highlighting/ (Retrieved 2015-05-09)