Micro Vi­su­al­i­sa­tions

How can Micro Vi­su­al­i­sa­tions en­hance text com­pre­hen­sion, mem­o­ra­bil­ity, and ex­ploita­tion?

Jonas Parnow@zeto Pots­dam Uni­ver­sity of Ap­plied Sci­ences Orig­i­nally sub­mit­ted in June 2015. Up­dated in Oc­to­ber 2015. Tech­ni­cal fixes in Oc­to­ber 2023. Su­per­vi­sors: Prof. Dr. Mar­ian Dörk and Prof. Boris Müller This is the web ver­sion of the de­sign mas­ter the­sis. Down­load the PDF of the the­sis. Down­load the IEEE-ac­cepted paper and poster.

Ab­stract

In a novel ap­proach, which is mainly based on the dis­ci­plines of (data) vi­su­al­i­sa­tion and ty­pog­ra­phy, this the­sis dis­cusses vi­su­al­i­sa­tions as a means to en­rich text in re­gard of its com­pre­hen­si­bil­ity, mem­o­ra­bil­ity, and ex­ploita­tion. A tax­on­omy is pro­posed that dif­fer­en­ti­ates spe­cific types of ap­pli­ca­tion vi­su­al­i­sa­tions may have in this con­text.

Draw­ing upon the tax­on­omy, the the­sis elab­o­rates two ap­proaches to align­ing a text’s vi­sual ap­pear­ance and its con­tent. The first ex­plores the ad­di­tion of graph­i­cal el­e­ments right within or ad­ja­cent to a text while the other ap­proach ex­plores the op­ti­cal mod­i­fi­ca­tion of a text by means of vi­su­al­i­sa­tion. For this I eval­u­ate how ty­po­graphic tech­niques can be used as vi­sual vari­ables. Along with the pro­posed sys­tem­atic I in­tro­duce the term »micro vi­su­al­i­sa­tion« de­scrib­ing the non-com­plex en­hance­ments made to the »amor­phous« text.

In this work I sum­marise the his­tor­i­cal de­riva­tion of text and im­ages from a me­dia-the­o­ret­i­cal per­spec­tive, dis­cuss my ap­proach in re­gard to the cur­rent state of the two dis­ci­plines of vi­su­al­i­sa­tion and ty­pog­ra­phy, in­tro­duce a tax­on­omy in order to de­clara­to­rily cat­e­gorise ex­ist­ing work and con­clude with an analy­sis of op­por­tu­ni­ties and chal­lenges for fu­ture work.

Zu­sam­men­fas­sung

Ge­prägt durch die Dis­zi­pli­nen der (Da­ten-)Vi­sua­li­sie­rung und der Ty­po­gra­fie, dis­ku­tiert diese Ar­beit einen neu­ar­ti­gen An­satz, der mit den Mit­teln der Vi­sua­li­sie­rung Texte in Bezug auf ihre Er­schlie­ßung, Merk­bar­keit und ihre Aus­beu­tung. Eine Ta­xo­no­mie wird vor­ge­tra­gen, die es er­laubt die un­ter­schied­li­chen An­wen­dun­gen von Vi­sua­li­sie­run­gen in die­sem Kon­text zu ka­te­go­ri­sie­ren.

Ba­sie­rend auf der Ta­xo­no­mie ver­folgt die The­sis zwei An­sät­ze, um das text­li­che Er­schei­nungs­bild und den In­halt des Tex­tes in Ein­klang zu brin­gen. Der ers­te­re un­ter­sucht, wie das Hin­zu­fü­gen gra­fi­scher Ele­men­te di­rekt in den Zei­len oder dem Text bei­ge­stellt ge­stal­tet wer­den kann. Der zwei­te An­satz er­forscht das op­ti­sche Mo­di­fi­zie­ren des Tex­tes mit den Mit­teln der Vi­sua­li­sie­rung. Dafür werte ich die Zweck­mä­ßig­keit von ty­po­gra­fi­schen Tech­ni­ken als Vi­su­el­le Va­ria­blen aus. Im Rah­men die­ser Me­tho­do­lo­gie führe ich den Be­griff »Micro Vi­sua­li­sie­rung« ein, der diese un-kom­ple­xen Ver­bes­se­run­gen am »ge­stalt­lo­sen« Text ver­eint.

In die­ser Ar­beit fasse ich die his­to­ri­sche Her­lei­tung von Text und Bild aus einer me­di­en­theo­re­ti­schen Per­spek­ti­ve zu­sam­men, dis­ku­tie­re mei­nen An­satz in Bezug zum ak­tu­el­len Stand der For­schung in den zwei je­wei­li­gen Dis­zi­pli­nen Vi­sua­li­sie­rung und Ty­po­gra­fie. Des­wei­te­ren ka­te­go­ri­sie­re er­läu­ternd be­stehen­de Ar­bei­ten an­hand der ein­ge­führ­ten Ta­xo­no­mie und schlie­ße mit der Ana­ly­se von Mög­lich­kei­ten und Her­aus­for­de­run­gen für zu­künf­ti­ge Ar­bei­ten.

Ac­knowl­edge­ment

Dur­ing the prepa­ra­tion of this the­sis I have re­ceived sup­port in many ways. Al­though the fol­low­ing list is by no means com­plete, I would like to ex­press my grat­i­tude to some peo­ple and in­sti­tu­tions in par­tic­u­lar.

  • I would like to thank both of my friends Paul Heinicker and Kim Al­brecht for their con­stant feed­back, ad­vices par­tic­u­larly for the the­o­ret­i­cal part, and ren­der­ing de­sign more than a sub­ject. Through the fre­quent, (mostly) crit­i­cal dis­cus­sions in and out­side of the uni­ver­sity I learned more than any course could have taught me.
  • I would like to thank my mother Ul­rike Parnow, my sis­ter Hanna Parnow, and Hans Matthes. Es­pe­cially my sis­ter helped me to re­vise the im­por­tance of fi­nan­cial suc­cess and eth­i­cal val­ues.
  • I would like to thank Lara Junold for the long term friend­ship and es­pe­cially for the proof­read­ing of this the­sis.
  • I would like to thank Hanna Weyer, Delia Kämmerer, and Yaiza Kuhlmann for show­ing me the life out­side of uni­ver­sity and work dur­ing my time in Berlin.
  • I would like to thank the Uni­ver­sity of Ap­plied Sci­ences Pots­dam for the in­te­gra­tion of de­sign and media stud­ies. Also the Urban Com­plex­ity Lab, in par­tic­u­lar Se­bas­t­ian Meier and Jan-Erik Stange, who strongly in­flu­enced my stud­ies. I also thank my col­leagues at Golden Sec­tion Graph­ics, es­pe­cially Klaas Neu­mann.
  • I would like to thank my in­ter­view part­ners Prof. Dr. Jan Dis­telmeyer, Sascha Venohr, and Jan Schwo­chow. I also thank Prof. Dr. Malte Zim­mer­mann, Prof. Dr. Dr. h.c. Jan Ass­mann, Prof. Dr. Wolf­gang Behr, Jonathan Corum, and Al­berto Cairo for their help.
  • Fi­nally, I would like to thank my first su­per­vi­sor Prof. Dr. Mar­ian Dörk. Not only dur­ing the com­ple­tion of my the­sis, but also dur­ing my en­tire time at the Uni­ver­sity, he strongly in­flu­enced my stud­ies with his knowl­edge and en­gage­ment. I also thank my sec­ond su­per­vi­sor Prof. Boris Müller, who granted me his time de­spite his many re­spon­si­bil­i­ties at the Uni­ver­sity of Pots­dam.

Table of Con­tents

In­tro­duc­tion

This chap­ter in­tro­duces into the gen­eral idea of my the­sis. Cor­re­spond­ingly, I pro­vide an out­line of my mo­ti­va­tion to dis­cuss the topic of »vi­su­al­i­sa­tion«. The reader is in­tro­duced to the un­der­ly­ing struc­ture of this work and the rel­e­vant ter­mi­nol­ogy. Fi­nally, the per­spec­tive from which this the­sis’ ar­gu­ments are de­vel­oped is spec­i­fied.

The story of com­mu­ni­ca­tion is a story of chang­ing media. And while the tech­nol­ogy processes, the prin­ci­ples stay the same. From all the com­mu­ni­ca­tion chan­nels avail­able to hu­mans, this the­sis fo­cuses on the vi­sual ones. More specif­i­cally on im­ages, text, and the over­lap of these two. De­spite their com­mon ori­gin, these forms drifted apart in their ap­pli­ca­tion and thus truly in­te­grated com­bi­na­tions are rarely used. One ex­cep­tion are in­for­ma­tion graph­ics, that com­bine all types of media to con­vey in­for­ma­tion: text, il­lus­tra­tions, pho­tographs and vi­su­al­i­sa­tions.

Vi­su­al­i­sa­tions find ap­pli­ca­tion in all kinds of fields and en­vi­ron­ments nowa­days. They work as an al­ter­na­tive or as an en­rich­ment for no­ta­tional ex­pla­na­tions, where the amount of quan­ti­ta­tive data or the un­der­lay­ing qual­i­ta­tive struc­ture ex­ceed the lin­guis­tic or nu­meric cog­ni­tive ca­pac­ity. As ex­ter­nal vi­sual cog­ni­tion aids (Card et al., 1999) vi­su­al­i­sa­tions or­gan­ise, equalise, and sim­plify the com­pre­hen­sion of in­for­ma­tion. But de­spite these – proven – ben­e­fits, vi­su­al­i­sa­tions are not used in a lot of media. In fact, many pub­li­ca­tions com­pletely ad­here to pure text, pho­tographs or tab­u­lar rep­re­sen­ta­tions. News­pa­pers for ex­am­ple stick to the tra­di­tional jour­nal­is­tic of tools text and pho­tog­ra­phy but rarely ex­plain sit­u­a­tions graph­i­cally.

Even though the tech­nol­ogy we use to ex­change text in order to com­mu­ni­cate with oth­ers has dra­mat­i­cally changed over the course of time, the basic tech­niques have not changed. In fact the book’s lay­out has not changed much since Guten­berg’s in­ven­tion of the let­ter­press. It might be pos­si­ble that we al­ready found a per­fect tool for knowl­edge trans­fer in the book, as Roland Reuß sug­gests in »the per­fect read­ing-ma­chine«. (Die per­fekte Lese­mas­chine, Reuß, 2014) This be­lief may be due to con­ven­tions in the so­ci­ety and is prob­a­bly ad­vo­cated by a group of nos­tal­gic book de­sign­ers. But in the same way lan­guage changes through so­cial, cul­tural, and tech­no­log­i­cal evo­lu­tion, the de­sign of media that trans­port lan­guage needs to adapt as well. This the­sis has no in­ten­tions of deny­ing the book, but in­stead takes the book as start­ing point for a new de­sign space, in which cur­rent com­mu­ni­ca­tion, in­for­ma­tion, and ed­u­ca­tion be­hav­iour is re­flected.

This work aims not at cre­at­ing new types of vi­su­al­i­sa­tions or new ty­po­graph­i­cal tech­niques; it rather elab­o­rates

  • how the ex­ist­ing vi­su­al­i­sa­tion types can be ap­plied in a dif­fer­ent way to suit our read­ing con­duct and
  • what ty­pog­ra­phy can learn from vi­su­al­i­sa­tion.

The pro­posed ap­proach cre­ates a sec­ond layer that com­ple­ments the first layer (the con­tent in its »stan­dard« for­mat­ting). This layer may act

  • as a meta-layer, re­veal­ing the inner struc­ture,
  • as an am­pli­fi­ca­tion-layer, sup­port­ing, con­tex­tu­al­is­ing, or ques­tion­ing the writ­ten state­ments vi­su­ally,
  • or as a de­tail-layer that adds deeper or new in­for­ma­tion to the text.

Mo­ti­va­tion

The vi­su­al­i­sa­tions pro­duced in sci­en­tific re­search are often com­plex graph­ics with many data points, vi­su­alised in an ab­stract man­ner. I am con­cerned that these rep­re­sen­ta­tions are too com­puter-dri­ven and too com­pli­cated to un­der­stand for a lay­man. While such graph­ics are un­doubt­edly valu­able and use­ful for ex­perts, this the­sis aims to bring vi­su­al­i­sa­tions to a broader au­di­ence. It seems that pri­mar­ily tex­tual and pri­mar­ily graph­i­cal media are drift­ing apart. My ap­proach brings this two worlds back to­gether: by cre­at­ing sim­pler, smaller, and eas­ier un­der­stand­able graph­ics that can be used in daily life. Even though de­f­i­n­i­tions of the term »vi­su­al­i­sa­tion« are rather broad, the ac­tual ap­pli­ca­tion is often strict in their tools. This the­sis en­dorses such an un­der­stand­ing of vi­su­al­i­sa­tion in order to con­sider every graph­i­cal de­ci­sion as a process of vi­su­al­is­ing in­for­ma­tion. Thereby, it con­tributes to a de­lib­er­ated and co­her­ent ap­pli­ca­tion, which is often miss­ing in cur­rent vi­su­al­i­sa­tions.

Struc­ture

The the­sis is struc­tured as fol­lows: the first part de­scribes a his­tor­i­cal de­riva­tion from a me­dia-the­o­ret­i­cal per­spec­tive that leads from ob­jects and sym­bols, to words and let­ters, as to im­ages and graph­ics. The next part de­scribes the dis­ci­plines most rel­e­vant to this the­sis: ty­pog­ra­phy and vi­su­al­i­sa­tion. Draw­ing upon ex­ist­ing the­o­ret­i­cal and prac­ti­cal sci­en­tific re­search on re­lated top­ics, a rea­son­ing is de­vel­oped that leads to the ap­proach of micro vi­su­al­i­sa­tion in this top-down method. The third part in­tro­duces a tax­on­omy in which ex­ist­ing pro­pos­als are de­clara­to­rily cat­e­gorised. This bot­tom-up per­spec­tive al­lows me in the next part to iden­tify prac­ti­cal pos­si­bil­i­ties and chal­lenges for the gen­eral con­cept of micro vi­su­al­i­sa­tions. In the fifth and last part I show the essence of in­ter­views with ex­perts from re­lated dis­ci­plines, give a sum­mary of the find­ings from pre­vi­ous parts and an out­look for fu­ture work.

Ter­mi­nol­ogy

The term »vi­su­al­i­sa­tion« is used with a broad de­f­i­n­i­tion in this the­sis. In fact, one con­cern of this work is to argue for this widen­ing of what vi­su­al­i­sa­tions can be con­sid­ered be­sides clas­si­cal di­a­grams or graphs. The term is often used syn­onymi­cal or in com­bi­na­tion with »data (vi­su­al­i­sa­tion)«, which often im­plies or is ac­com­pa­nied by a great amount of quan­ti­ta­tive val­ues. Mean­while, it should also apply to small amount and even qual­i­ta­tive val­ues.

Stet­ter de­fines the di­a­gram as fol­lows: »It is […] an ex­hib­i­tive rep­re­sen­ta­tion of cir­cum­stances with graph­i­cal means« (»Es ist […] die ex­hi­bi­ti­ve Dar­stel­lung eines Sach­ver­halts mit gra­phi­schen Mit­teln.«) (Stet­ter, 2005) This broad­ens the con­cept to maps, schemes, sketches, and blue­prints but also to any usage of graph­i­cal means, even when no op­ti­cal »bound­ing box« de­fines it as fig­ure or dif­fer­en­ti­ates it from other el­e­ments. It is ori­en­tated to­wards the basic mean­ing of vi­su­al­is­ing: to make (some­thing) vis­i­ble to the eye. (Ox­ford Dic­tio­nary)

The term »micro« is cho­sen in front of a lack of a more en­com­pass­ing and at the same time more spe­cific term. The way it is used in this the­sis could also be seen as »min­i­mal­is­tic« or »frag­men­tary«: it aims to de­scribe vi­su­al­i­sa­tions that are not only small in phys­i­cal space but can also be nar­rowed in terms of data di­men­sion or points. Vi­su­al­i­sa­tions that have one or mul­ti­ple of these char­ac­ter­is­tics are con­sid­ered as micro. Hence »micro vi­su­al­i­sa­tions« are basic graph­i­cal mod­i­fi­ca­tions or ad­di­tions, that en­hance the com­pre­hen­sion of text.

Po­si­tion­ing

As de­signer my ap­proach in this the­sis is mainly dri­ven by re­search is­sues re­lated to the sub­ject de­sign. But since this dis­ci­pline (and the one of in­for­ma­tion de­sign in par­tic­u­lar) not only touches, but also needs to con­sider other dis­ci­plines like jour­nal­ism, media stud­ies, com­puter sci­ence, lin­guis­tic, and sta­tis­tics these prospects will hope­fully be ful­filled with ap­pro­pri­ate care and ex­per­tise.

This the­sis aims to con­tribute to the cur­rent level of knowl­edge of (data) vi­su­al­i­sa­tion. The over­ar­ch­ing en­deav­our is to ex­pand the un­der­stand­ing of how micro vi­su­al­iza­tions are per­ceived and how they can help the reader to com­pre­hend in­for­ma­tion. The sec­ond field, ty­pog­ra­phy, is also ex­panded in how its tech­niques can be ap­plied, but is not equipped with new ones.

Even though the re­sults of this the­sis can hope­fully be ap­plied all over the world, its foun­da­tion and es­pe­cially its ex­am­i­na­tion of the evo­lu­tion of im­ages, lan­guage, and writ­ings mainly re­late to a cen­tral Eu­ro­pean point of view and are there­fore highly cu­rated and par­tial.

De­riva­tion: Four turns until pres­ence

In this chap­ter I ex­am­ine the evo­lu­tion of com­mu­ni­ca­tion within the media most rel­e­vant to this the­sis: im­ages and text. Their re­la­tion to each other is il­lus­trated as well as their rel­a­tive im­por­tance over time is con­sid­ered. The chap­ter con­cludes with a re­vi­sion of the cur­rent state of art and a pro­posal of how the newest form of im­ages – vi­su­al­i­sa­tions – may en­rich de­sign­ers’ tools of com­mu­ni­ca­tion.

Vi­sual com­mu­ni­ca­tion is es­sen­tial in every as­pect of our lives. New­borns first learn to con­nect basic vi­sual cues such as shapes and colours to rudi­men­tal mean­ings. Ac­cord­ingly a mother’s smile or a fire’s glow­ing ap­pear­ance gain mean­ings that — among count­less oth­ers — guide us from child­hood on through a world dom­i­nated by vi­sual in­for­ma­tion. These fun­da­men­tal lessons de­ter­mine our cog­ni­tive de­vel­op­ment more than the four other senses that we ex­pe­ri­ence from our sur­round­ings. Only after we in­ter­nalise the in­ter­pre­ta­tion of myr­iad vi­sual im­pres­sions, we learn how to ex­press these things in words.

An ar­bi­trary form of com­mu­ni­ca­tion

Words — first spo­ken, later writ­ten — have lever­aged mankind’s evo­lu­tion. With the abil­ity to ref­er­ence ob­jects, ex­press in­ten­tions, or de­scribe re­la­tions we gained the abil­ity to com­mu­ni­cate about ab­stract sys­tems and ideas with oth­ers. This de­vel­op­ment — start­ing with vi­sual cues, the ad­di­tion of spo­ken words and fi­nally the ex­pres­sion of these words in (vi­sual) writ­ings — is some­thing not only de­vel­oped by hu­mankind over the course of thou­sands of years but also some­thing every child still runs through nowa­days. Our long term mem­ory en­ables us to store the in­for­ma­tion we gather in our life­time to rapidly iden­tify words and thou­sands of vi­sual ob­jects. (Ware, 2004)

Pet­ro­glyphs Gi­raffe, lion, and oth­ers in Namibia, Africa. Pho­to­graph by Greg Willis, 2006
Sym­bol­ism The Pre­mier Leagues uses the sym­bol of the lion.

Ob­serv­ing this de­vel­op­ment, it is im­por­tant to note that these forms of ex­pres­sion and im­pres­sion are highly con­nected and never ex­clu­sive. The de­vel­op­ment of lan­guage and writ­ing/read­ing are greatly in­flu­enced by each other and were ini­tially in­spired by the ac­tual phys­i­cal ap­pear­ance. The Egyp­tol­o­gist Jan Ass­mann in­ter­prets writ­ings by the Philoso­pher Moses Men­dels­sohn about this de­vel­op­ment as fol­lows: The first step was an »iconic turn«. From the things it­self to im­ages of the things. Ex­am­ples are cave paint­ings from the Pa­le­olithic Age (18,000 BC) and the Pet­ro­glyphs (πέτρος pet­ros »stone« and γλύφειν glýphein »to carve«) from the Ne­olithic Age (10,200 BC – 4,500 or 2,000 BC). (Borgo et al., 2013) These were pic­tograms, which were painted or carved in rock faces all over the world. Mendelssohn ex­em­pli­fies this progress through the sym­bol of a ma­jes­tic lion that gained its mean­ing from the ac­tual lion.

Time­line From the birth of mankind until today

The sec­ond step is an »uni­conic turn«, in which these sym­bols de­tach from their orig­i­nal ref­er­ence ob­jects and be­come words. (Ass­mann, 2002) René Magrit­te’s »The Treach­ery of Im­ages« (« La Tra­hi­son des images ») il­lus­trates this re­la­tion­ship be­tween the sig­ni­fier (sig­nifi­ant) and the sig­ni­fied (signi­fié). The writ­ing »This is not a pipe.« (« Ceci n’est pas une pipe. ») ad­vises the be­holder that the image he is see­ing is not a pipe but in­stead the image of a pipe. The paint­ing (signi­fier) is not ca­pa­ble of show­ing an ac­tual ob­ject (sig­ni­fied), but rather — as Pla­ton de­scribes it — the idea. Fur­ther­more the »lap­idary truth of the sen­tence and the non less lap­idary ev­i­dence of the il­lus­tra­tion pos­sess no com­mon level. What we truly see, is not said in the lin­guis­tic state­ment in any way. What we truly read, is not con­firmed by the eye.« (Boehm, 2014) The two sig­ni­fiers — image and writ­ing — are com­pet­ing with each other. The once vi­su­ally in­spired lan­guage eman­ci­pated it­self to ab­stract words.

semi­otics The paint­ing is only the idea of the ac­taul ob­ject

Today, two forms of »word-writ­ings« can be gen­er­ally dis­tin­guished: Pic­to­r­ial and al­pha­bet­i­cal sys­tems. (Frutiger, 1998) The Chi­nese char­ac­ters for ex­am­ple have a great amount of signs whose ap­pear­ance is di­rectly de­rived from the pic­to­r­ial rep­re­sen­ta­tion of an ob­ject. Their shape is mo­ti­vated from their orig­i­nal form. In con­trast al­pha­bet­i­cal char­ac­ters can no longer be as­signed to se­man­tic mean­ings on their own. Even though they have a »hi­ero­glyphic mem­ory«, their form be­came ar­bi­trary. (Ass­mann, 2002) In­stead, they rep­re­sent tones, which gain mean­ing in their se­quence.

Dis­cur­sive­ness as recipe for suc­cess

In Eu­rope, the abil­ity to write words with these »build­ing blocks« pave the way to Guten­berg’s mile­stone in­ven­tion of the let­ter­press. In­stead of writ­ing let­ters one by one to a sin­gle book, a com­pos­ing gal­ley is pre­pared once with mov­able types and can then be used to print mul­ti­ple copies. While man­u­scripts have al­ways lever­aged the spread­ing of knowl­edge, with the de­vel­op­ment of the let­ter­press the book could fi­nally break open this bot­tle­neck of dis­sem­i­na­tion of knowl­edge where one friar could man­u­fac­ture one book dur­ing his life­time. It was a rev­o­lu­tion on two lev­els: the ris­ing avail­abil­ity of af­ford­able, trans­portable, and uni­form knowl­edge and the first se­r­ial pro­duc­tion from the eco­nom­i­cal per­spec­tive. (Haus­mann, 2009)

The book’s sta­tus in­flu­enced com­mu­ni­ca­tion and the dis­sem­i­na­tion of knowl­edge so dras­ti­cally that the Cana­dian philoso­pher of com­mu­ni­ca­tion the­ory Mar­shall McLuhan de­clares the book as the lead­ing medium of re­cent times. The fact that the Times mag­a­zine de­clared the Ger­man black­smith as the most im­por­tant per­son of the last mil­len­nium (Haus­mann, 2009), sup­ports McLuhan’s term of the »Guten­berg Galaxy«. It de­scribes the great ef­fects the book had not only on so­cial in­ter­ac­tion and con­di­tions, knowl­edge trans­fer and re­search, al­pha­beti­sa­tion and lan­guage stan­dard­i­s­a­tion, but also on the think­ing it­self. Think­ing and des­ig­na­tion were di­rectly re­fer­ring to each other be­fore the dom­i­na­tion of the book. With the op­ti­cal fix­a­tion the lan­guage crowds be­tween these two. (Ass­mann, 2002) The read­ing of se­quenced let­ters, words and para­graphs de­mands and fa­cil­i­tates lin­ear­ity in con­trast to spa­tial­ity, dis­course in favour of si­mul­ta­ne­ous recog­ni­tion and ca­pac­ity for ab­strac­tion in­stead of af­fec­tiv­ity. (Ass­mann, 2002)

Lin­guis­tic Turn The lan­guage changes the way with think.

These ef­fects were recog­nised as the »lin­guis­tic turn«, in which lan­guage as a medium of recog­ni­tion and de­scrip­tion of re­al­ity was crit­i­cally stud­ied. Lud­wig Wittgen­stein sum­marises these de­lib­er­a­tions with the lim­its of his lan­guage as the lim­its of his world (Wittgen­stein, 1922). This re­veals how human’s de­vo­tion from im­ages to lan­guage also im­pli­cates their com­pre­hen­sion and ex­pres­sion. This has been re­searched as »Lin­guis­tic Rel­a­tiv­ity« first by Wil­helm von Hum­boldt, later by Ben­jamin Lee Whorf and Ed­ward Sapir, whose work is now summed up under the term Sapir–Whorf hy­poth­e­sis. Re­search in this area has been done by Berlin and Kay (1969 in Ware, 2004) and Eva Heller (2009), who spec­i­fied this analy­sis for colour des­ig­na­tion and per­cep­tion.

The world in im­ages

Today the end of the Guten­berg Galaxy is dis­cussed in var­i­ous fields. The world shaped and con­trolled by elec­tronic tech­nol­ogy is »im­plod­ing« (McLuhan, 2011), the sender-re­ceiver model is out­dated and image gen­er­at­ing media is be­com­ing ubiq­ui­tous. Nor­bert Bolz ex­em­pli­fies this with the ris­ing use of pho­tographs in news­pa­pers: While many tra­di­tional news­pa­pers re­jected im­ages for a long time not be­cause of tech­ni­cal rea­son­ing but rather con­ser­vatism, purely tex­tual news­pa­pers are un­think­able nowa­days. (Bolz, 2012)

But mass media is not the only field af­fected. In op­po­si­tion to the lin­guis­tic turn, the »iconic turn« was for­mu­lated in the 1990s. While the ac­tual de­f­i­n­i­tion is still not fi­nalised, it cir­cu­lates around ques­tions in a world more and more dom­i­nated by im­ages and image tech­nolo­gies. (Bach­mann-Medick, 2008) William Mitchell and Got­tfried Boehm in­di­vid­u­ally de­scribed and analysed the process with dif­fer­ent terms and per­spec­tives. Both the pic­to­r­ial- (Mitchell) and iconic-turns (Boehm) em­pha­sise the im­por­tance of im­ages not only as ref­er­ences to ob­jects but also as ob­jects of analy­sis it­self. (Bach­mann-Medick, 2008) Even though the lin­guis­tic turn con­sid­ered this im­pos­si­ble, it leads to not only think­ing about im­ages, but rather to think­ing with im­ages. (Bach­mann-Medick, 2008)

But this newly formed vi­sual cul­ture (Bild­wis­sen­schaf­ten) does not want to re­place the pre­vi­ously pro­claimed lin­guis­tic turn, but rather de­scribe the »image as an­other way of think­ing«. (Boehm, 2007) Promi­nent ex­am­ples of analy­sis in this dis­ci­pline is the use of im­ages to »dis­play« (Sicht­bar­ma­chung) en­e­mies in so called »enemy im­ages« (Feind­bil­der) (Kittsteiner, 2004) and use of pho­tographs as »wit­nesses« for the Holo­caust. (Didi-Hu­ber­man, 2006) In con­trast in the book »Cos­mi­graph­ics: Pic­tur­ing Space Through Time«, Michael Ben­son analy­ses the view of the world in im­ages through his­tory.

While these ex­am­ples refer to the dis­played con­tent of im­ages, an­other ob­ject of re­search is the fig­u­ra­tive, graph­i­cal level. Even though this field has been known and used by painters and pho­tog­ra­phers for a long time, re­cent fun­da­men­tal re­search es­pe­cially in psy­chol­ogy opened up the field of per­cep­tual psy­chol­ogy. Mostly evo­lu­tion­ary rea­soned hu­mans per­ceive vi­sual in­for­ma­tion on dif­fer­ent lev­els. From the be­gin­ning of hu­mankind, sur­vival de­pended on quick and pro­found de­ci­sions. Dan­ger needed to be spot­ted quickly and food needed to be iden­ti­fied cor­rectly. The human brain adapted to these fac­tors with var­i­ous vi­sual pat­tern recog­ni­tion tech­niques, which are based on basic vi­sual forms. We search for use­ful pos­si­bil­i­ties for ac­tion and vi­sion al­lows us to »per­ceive af­for­dances of the en­vi­ron­ment di­rectly, not in­di­rectly by piec­ing to­gether ev­i­dence from our senses.« (Ware, 2004)

These recog­ni­tion skills can be still used today: The colour red still rep­re­sents dan­ger and cre­ates at­ten­tion (Heller, 2009), mov­ing ob­jects could be­come a threat and oc­cupy our alert­ness (Ware, 2004), and big­ger (closer) ob­jects seem more im­por­tant. (Lid­well et al., 2010) In fact, the Gestalt prin­ci­ples (Todor­ovic, 2008) are based on our evo­lu­tion­ary shaped per­cep­tion.

Func­tional im­ages

Com­pletely based on this fig­u­ra­tive and graph­i­cal level is the es­pe­cially in re­cent time widely used dis­ci­pline of vi­su­al­i­sa­tion. These ab­stract im­ages utilise basic graph­i­cal forms to com­mu­ni­cate in­for­ma­tion. In­for­ma­tion that could not or only with high dif­fi­cul­ties be com­mu­ni­cated through lan­guage or even »con­crete« im­ages. While graph­ics, which are nowa­days la­belled as vi­su­al­i­sa­tions or in­for­ma­tion graph­ics, were al­ready cre­ated in the last cen­tury, com­put­ers ac­cel­er­ated the cre­ation of these »math­e­mat­i­cal« im­ages. Bar charts, pie charts or scat­ter­plots be­came known to the gen­eral pub­lic through soft­ware like Mi­crosoft Excel.

Apart from the gen­eral pub­lic a com­mu­nity of ex­perts, con­sist­ing mainly of de­sign­ers, math­e­mati­cians, sta­tis­ti­cians and com­puter sci­en­tists, push the lim­its of the mul­ti­di­men­sional data map­ping. Com­puter-dri­ven graph­ics and sci­en­tific en­vi­ron­ments move the dis­ci­pline fur­ther away from the un­der­stand­ing of non-ex­perts. While some great graph­ics make it into pop­u­lar media, vi­su­al­i­sa­tion still re­mains far be­hind its po­ten­tials. Just like news­pa­pers re­jected pho­tographs in the be­gin­ning in favour of Guten­berg’s legacy, knowl­edge me­di­a­tors nowa­days with­hold from the usage of vi­su­al­i­sa­tions.

At this point my the­sis starts. The avant-garde of news­pa­pers like the New York Times and of mag­a­zines like Na­tional Ge­o­graphic as well as early-adopters who take part for ex­am­ple in the Quan­ti­fied Self move­ment bring vi­su­al­i­sa­tion to peo­ple out­side of the com­mu­nity. In the same way I want to help to move vi­su­al­i­sa­tions away from its ex­pert dri­ven pres­ence to every­day life. For this, my ap­proach is to po­si­tion graph­ics right at the point where cur­rently still most knowl­edge is still im­parted: in the writ­ten text. And while many re­al­i­sa­tions pre­sent a di­vided back and forth be­tween graphic and text, my ap­proach is quite lit­er­ally by break­ing up the bound­aries be­tween each medium.

My goal is to cre­ate Boehm’s »union of power« (Herrschaft­sunion) be­tween the lin­guis­tic and the iconic turn to cre­ate a fer­tile jux­ta­po­si­tion in­stead of op­po­site po­si­tions. (Bach­mann-Medick, 2008)

The­o­ret­i­cal Foun­da­tion

This chap­ter draws upon the pre­vi­ously in­tro­duced tools of com­mu­ni­ca­tion, i.e. text and image. These are eval­u­ated in re­gard to the dis­ci­plines of ty­pog­ra­phy and vi­su­al­iza­tion, in which both con­cepts have de­vel­oped rather sep­a­rately. Against this back­ground I argue that ty­pog­ra­phy is in fact a vari­a­tion of vi­su­al­i­sa­tion and that, ac­cord­ingly, text and image should be per­ceived as com­ple­men­tary tools, which should be ap­plied in unity. Serv­ing this pro­posal, I char­ac­ter­ize con­cerned dis­ci­plines and demon­strate my ar­gu­ment’s rel­e­vance from cul­tural, tech­no­log­i­cal as well as de­sign re­search per­spec­tives.

What is com­mu­ni­ca­tion?

When one tries to de­fine what com­mu­ni­ca­tion re­ally is, he quickly comes to the con­clu­sion that every­thing we do is com­mu­ni­ca­tion. »Ac­tiv­ity or in­ac­tiv­ity, words or si­lence all have mes­sage value: they in­flu­ence oth­ers […].« (Wat­zlaw­ick et al., 1967) As de­sign­ers, we shape many as­pects of this wide field and help other peo­ple to com­mu­ni­cate the way they wish to. Viewed with an even broader de­f­i­n­i­tion of de­sign, hu­mankind as de­signer has de­vel­oped a wide va­ri­ety of com­mu­ni­ca­tion chan­nels, rang­ing from body lan­guage and spo­ken lan­guage over to writ­ten and printed type, and fi­nally to elec­tron­i­cally processed com­mu­ni­ca­tion. Each of these steps marked a mile­stone in mankind’s his­tory, but none could com­pletely re­place pre­vi­ous medi­ums. Even though some of the utilised tech­nol­ogy for each com­mu­ni­ca­tion chan­nel might have been re­placed over time, the basic con­cept of com­mu­ni­ca­tion stayed pre­served.

There are two rea­sons for that: First, every form of com­mu­ni­ca­tion has its own qual­i­ties, ad­van­tages, and ap­pli­ca­tions at dif­fer­ent set­tings, and sec­ond, each de­vel­op­ment rests upon the for­mer one and there­fore re­lies on the com­mon un­der­stand­ing of the pre­vi­ous one.

To em­pha­sise this re­la­tion even fur­ther, Paul Wat­zlaw­ick’s fa­mous quote »One can­not not com­mu­ni­cate« (Wat­zlaw­ick et al., 1967) ex­pands the de­f­i­n­i­tion of what com­mu­ni­ca­tion is. He ar­gues that there is no op­po­site of to com­mu­ni­cate and thus makes »com­mu­ni­ca­tion syn­ony­mous, or nearly syn­ony­mous, with be­hav­iour.« (Mot­ley, 1990) This im­pli­cates that at least one of the com­mu­ni­ca­tion chan­nels avail­able to human is al­ways ac­tive. It also fa­cil­i­tates the the­ory of multi-lay­ered com­mu­ni­ca­tion and thus that a sin­gle form of ex­pres­sion is sel­dom and a sin­gle form of im­pres­sion is im­pos­si­ble. The fol­low­ing ex­am­ple il­lus­trates this: When one reads a text, one is not just read­ing. He is also see­ing how the text is pre­sented. The writ­ten, printed or dis­played words are only one part of the in­for­ma­tion that is re­ceived. The ma­te­r­ial, the lay­out and the many as­pects of ty­pog­ra­phy all com­mu­ni­cate on their own and thus in­flu­ence how and what in­for­ma­tion is per­ceived from the ac­tual text. Mar­ket­ing knows that and de­fines CIs con­sist­ing of cor­po­rate de­sign, lan­guage, sound, smell, and many more as­pects that all strengthen the strin­gent brand­ing of the com­pany.

What can be taken from this is the broad un­der­stand­ing of what com­mu­ni­ca­tion is. To use this meth­ods not to make money but to im­part knowl­edge to oth­ers through a wide va­ri­ety of means of com­mu­ni­ca­tion.

What is Ty­pog­ra­phy?

The im­por­tance of the ap­pear­ance of the text is well known and gen­er­a­tions of type de­sign­ers and ty­pog­ra­phers de­vel­oped a va­ri­ety of tech­niques to de­sign and en­hance the »text-see­ing« and read­ing process. It is »the art, or skill, of de­sign­ing com­mu­ni­ca­tion by means of the printed word« (Childers et al., 2002) But even be­fore books were printed, a highly elab­o­rated dis­ci­pline had formed around type de­sign. (Drucker, 2014) Today ty­pog­ra­phy is an in­her­ent and es­sen­tial part in de­sign ed­u­ca­tion. The va­ri­ety of type de­sign prin­ci­ples and ty­po­graphic vari­ables need to be thought­fully ap­plied to the text to com­ply with the con­tent but also to be un­der­stood by the reader. Some of these prin­ci­ples are con­ven­tions, which evolved over time (italic and reg­u­lar fonts), oth­ers are based on fun­da­men­tal de­ci­sions such as read­ing from one side in one di­rec­tion (jus­ti­fied, flush left, or cen­tered text), while oth­ers base on per­cep­tional rea­son­ing (font size and weight).

Car­ried by this unique craft, writ­ten and later – with Guten­berg’s in­ven­tion – printed books came to their promi­nent sta­tus they hold today. But the im­por­tance of ty­pog­ra­phy is not jus­ti­fied by the dec­o­ra­tion of the text, but rather the read­abil­ity of the text and more fun­da­men­tally by the sup­port­ive char­ac­ter the de­sign plays for the con­tent of the text. Robert Bringhurst ex­em­pli­fies this for the type­face: »When the type is poorly cho­sen, what the words say lin­guis­ti­cally and what the let­ters imply vi­su­ally are dishar­mo­nious, dis­hon­est, out of tune.« (Bringhurst, 2012) To clar­ify, ho­n­our or dis­guise the con­tent is seen as the most im­por­tant role ty­pog­ra­phy plays. But with the lever­age its ap­pli­ca­tion has, it can also be mis­used: »In a world rife with un­so­licited mes­sages, ty­pog­ra­phy must often draw at­ten­tion to it­self be­fore it will be read.« (Bringhurst, 2012) This im­plies the con­se­quences that ty­po­graphic de­ci­sions have on the vi­sual ap­pear­ance and hence on the very first im­pres­sion the reader has of the text. With other words: »Type is per­sua­sive – and thus so is ty­pog­ra­phy […]« (Santa Maria, 2014)

Read­abil­ity-Model by Ralf Her­rmann in a sim­pli­fied ver­sion (2010)

Be­sides the used ma­te­r­ial, the se­lec­tion of an ap­pro­pri­ate type­face is the first and most fun­da­men­tal de­ci­sion when ap­proach­ing a text. With the se­lec­tion of one type­face the de­signer takes over many de­ci­sions that were pre­vi­ously made by the type de­signer. It may also de­mand a cer­tain ty­po­graphic ap­pli­ca­tion to suit its ap­pear­ance. The ex­pla­na­tion of the com­plete process of let­ter and type de­sign would go be­yond the scope of this study, but what is rel­e­vant for this the­sis is the great im­pact the se­lec­tion of a type­face has for the per­cep­tion of the text. Stronger than any other ty­po­graphic de­ci­sion, the type­face af­fects many as­pects of the read­ing ex­pe­ri­ence. First the read­abil­ity con­sist­ing of reader-friend­li­ness (Le­se­freund­lich­keit), dis­tin­guisha­bil­ity (Un­ter­scheid­bar­keit), rec­og­niz­abil­ity (Er­kenn­bar­keit), and leg­i­bil­ity (Le­ser­lich­keit) (Her­rmann, 2010) and sec­ondly — even more promi­nent — the con­no­ta­tions that come with the type­face. From a con­sumer psy­chol­ogy view »type­faces con­vey mean­ings that have the po­ten­tial to sig­nif­i­cantly in­flu­ence im­por­tant mar­ket­ing con­structs. These as­so­ci­a­tions in­flu­ence how con­sumers per­ceive brands, as well as, what they re­mem­ber about brands.« (Childers & Jass, 2002) Taken the book as a prod­uct that rep­re­sents the text, it be­comes clear that we per­ceive and re­mem­ber the con­tent dif­fer­ently de­pend­ing on the cho­sen type­face.

With this most ob­vi­ous de­ci­sion in ty­pog­ra­phy as in­tro­duc­tion, fur­ther »tools« that are avail­able for ty­pog­ra­phers are for ex­am­ple mar­gin, line-height, let­ter-spac­ing or let­ter case. A full ex­plo­ration of these vari­ables is dis­cussed in re­la­tion to »vi­su­al­i­sa­tion« fur­ther down.

What is Vi­su­al­i­sa­tion?

The power of pho­tographs has been used for a long time. The al­leged ob­jec­tiv­ity in­duces pub­lish­ers to show »the sin­gle truth«. The pre­tended ev­i­dence of the image re­duces com­plex mat­ters to sin­gle state­ments. (Bolz, 2012) As men­tioned above, nearly every news­pa­per nowa­days re­lies on the im­pact a pho­to­graph can have for a story. But with this prac­tice prob­lems arise con­cern­ing the pho­tographa­bil­ity of cer­tain sub­jects. (Bolz, 2012) In­tan­gi­ble and ab­stract prob­lems can­not draw much at­ten­tion to it­self, be­cause they can­not be cap­tured in a pho­to­graph or an il­lus­tra­tion. This be­comes even more ap­par­ent in the con­text of so­cial media, where posts with im­ages reach higher click rates. Ar­ti­cles that do not show their con­tent with at­trac­tive im­ages can­not arise much at­ten­tion. One an­swer to these un­de­pictable sub­jects are vi­su­al­i­sa­tions, which en­code ab­stract data into tan­gi­ble graph­ics. Pre­vi­ously »in­vis­i­ble« and highly com­plex processes gain shape through ab­stract graph­i­cal el­e­ments. Thus vi­su­al­i­sa­tions are used for in­for­ma­tion ex­pla­na­tion.

pho­tographa­bil­ity How to vi­su­alise cli­mate change? Pho­to­graph by Carla Lom­bardo Ehrlich for WWF

Vi­su­al­i­sa­tion is not a new field, but ex­pe­ri­enced an up­turn in re­cent time. With ab­stract data be­com­ing ubiq­ui­tous the chal­lenge to make sense of this new re­source be­came ap­par­ent in all in­dus­tries. The term big data arised in the hope to find pre­vi­ously hid­den in­sights from the analy­sis of great amounts of data. Be­sides clas­si­cal sta­tis­ti­cal meth­ods, vi­su­al­i­sa­tion is an­other promis­ing and proven ap­proach. For this pur­pose data vi­su­al­i­sa­tions are used for ex­plo­ration. The sta­tis­ti­cian John Tukey al­ready pre­dicted 1965 »a greater and greater role for graph­i­cal tech­niques as aids to ex­plo­ration and in­ci­sive­ness« for his sub­ject. (Tukey, 1965)

While the tex­tual analy­sis of such an amount of data is not ef­fi­cient, hu­mans are very good at spot­ting pat­tern, ab­nor­mal­i­ties or trends on a vi­sual level. (Few, 2014) Ware goes even fur­ther and says »There is no other way of pre­sent­ing in­for­ma­tion so that struc­tures, groups, and trends can be dis­cov­ered among hun­dreds of data val­ues«. (Ware, 2004)

Nor­bert Bolz speaks in this con­text about the con­ci­sion (Prä­gnanz) of an image. This qual­ity en­ables us to in­stantly see the con­tent. (Bolz, 2012) In order to un­der­stand this process, the human sen­sory sys­tem needs to be ex­am­ined. »The term sen­sory sys­tem in­cludes all phe­nom­ena that man­i­fest them­selves in the con­scious­ness as re­sult of stim­u­la­tion of any sen­sory organ«. (Haverkamp, 2009) This rather broad de­f­i­n­i­tion de­scribes (all) the phe­nom­ena as some­thing which is ac­tively man­i­fest­ing it­self. This seems to be op­posed to the widely es­tab­lished and ac­cepted con­cept of con­struc­tivism, which de­scribes how we per­ceive our sur­round­ings on the basis of a the­o­ret­i­cal model. In phi­los­o­phy, an­thro­pol­ogy and psy­chol­ogy — in con­trast to con­struc­tivism in art — it de­scribes the fol­low­ing process: When we ob­serve ob­jects, we re­con­struct these as men­tal ob­jects. These ideas are con­stantly matched to pre­vi­ous ex­pe­ri­ences and in­flu­ences. These »im­ages« of ob­jects work as pro­to­types and can be re­called even with dif­fer­ent ap­pear­ance or after per­spec­tive trans­for­ma­tion. (Ware, 2004) This im­plies that the human brain is no longer pas­sive in its per­cep­tion, but in­stead ac­tive in terms of se­lec­tion, pro­jec­tion, in­ter­pre­ta­tion and sense-giv­ing: it con­structs its world in­di­vid­u­ally. (Malet­zke, 1998)

This ob­vi­ously raises ques­tions re­gard­ing the »real re­al­ity«, which is a rather philo­soph­i­cal dis­cus­sion that goes be­yond the scope of this work. For de­sign­ers, this model is in­ter­est­ing in re­gard of com­mu­ni­ca­tion. Con­cepts of mass com­mu­ni­ca­tions are no longer con­sis­tent with this the­ory. Every human needs to be viewed as an in­di­vid­ual, from whom con­sis­tent re­ac­tions can no longer be ex­pected. What is rel­e­vant for the pur­pose of this work and for the per­cep­tion of vi­su­al­i­sa­tion is the ac­tive se­lec­tion and in­ter­pre­ta­tion of the pre­sented image. De­spite this, the pas­sive per­cep­tion of vi­sual in­for­ma­tion is equally im­por­tant and has been sci­en­tif­i­cally proven as well. The an­swer to this seem­ing con­tra­dic­tion lays in the way our vi­sual sen­sory sys­tem works. Sim­pli­fied it con­sists of three con­sec­u­tive stages:

Par­al­lel Pro­cess­ing In the first stage »bil­lions of neu­rons work in par­al­lel, ex­tract­ing fea­tures from every part of the vi­sual field si­mul­ta­ne­ously. This par­al­lel pro­cess­ing pro­ceeds whether we like it or not, and it is largely in­de­pen­dent of what we choose to at­tend to […].« (Ware, 2004) This is one rea­son hu­mans are good at com­pre­hend­ing large amounts of data points in short time. The par­al­lel, rapid pro­cess­ing ex­plains why vi­sual in­for­ma­tion can be ex­tracted quicker form graph­ics than from tex­tual pre­sen­ta­tions, which have to be read se­ri­ally.

Pat­tern Per­cep­tion »At the sec­ond stage, rapid ac­tive processes di­vide the vi­sual field into re­gions and sim­ple pat­terns, such as con­tin­u­ous con­tours, re­gions of the same colour, and re­gions of the same tex­ture«. (Ware, 2004) At this stage Gestalt prin­ci­ples play a de­ci­sive role. It is im­por­tant to note that at this stage, the in­for­ma­tion is han­dled in se­r­ial, re­sult­ing in a much slower pro­cess­ing. At this point the long-term mem­ory is al­ready in­volved.

Se­quen­tial Goal-Di­rected Pro­cess­ing In the third stage »we con­struct a se­quence of vi­sual queries that are an­swered through vi­sual search strate­gies«. (Ware, 2004) This means the ac­tive search for vi­sual el­e­ments that fit the given task. For ex­am­ple, this could con­sist of con­nect­ing close lo­cated red points.

The last two stages in­volve ac­tive search, se­lec­tion and in­ter­pre­ta­tion of the given vi­sual in­for­ma­tion where it is im­por­tant to note that this »ac­tive« process is still processed mainly un­con­sciously. »Con­sider the eye­ball as an in­for­ma­tion-gath­er­ing search­light, sweep­ing the vi­sual world under the guid­ance of the cog­ni­tive cen­ters that con­trol our at­ten­tion. In­for­ma­tion is ac­quired in bursts, a snap­shot for each fix­a­tion. From an image buffer, the mas­sively par­al­lel ma­chin­ery of early vi­sual pro­cess­ing finds ob­jects based on salient fea­tures of im­ages.« (Ware, 2004)

In order to help spot­ting salient fea­tures, in­for­ma­tion de­signer should en­code this in­for­ma­tion to »pop out«. Var­i­ous stud­ies analysed which en­cod­ing works best for which type of data. The first and most cited tax­on­omy of such kind was for­mu­lated by Jacques Bertin. In his book Semi­ol­ogy of Graph­ics (Bertin, Semi­olo­gie Graphique, 1983), the car­tog­ra­pher con­strains his stud­ies with some fac­tors like print­able, stan­dard sized graph­ics on white paper. His work was later picked up by many sci­en­tists and ex­panded by un­no­ticed vari­ables and the ones that were left out in­ten­tion­ally by Bertin, as he is rooted in car­tog­ra­phy.

Bertin dis­tin­guishes seven forms of en­cod­ing. He de­scribes these as »vi­sual prim­i­tives« or as they are now com­monly called »vi­sual vari­ables«, from which the de­signer can form the image to sug­gest per­spec­tive, re­al­ity, re­la­tion­ships or space. (Ware, 2004) These vari­ables are: po­si­tion, size, value (tran­si­tion from light to dark), tex­ture, colour hue, ori­en­ta­tion and shape. I pro­pose the fol­low­ing mod­i­fi­ca­tions for a bet­ter struc­ture:

  • sub­or­di­na­tion of colour hue, value, and tex­ture under ap­pear­ance and
  • re­place­ment of value with opac­ity, since the tran­si­tion from light to dark can be treated as change of colour.

This re­sults in the fol­low­ing tax­on­omy:

Vi­sual Vari­ables A mod­i­fied ver­sion of Bertin’s vi­sual vari­ables

Bertin’s vi­sual vari­ables have often been ex­tended, for ex­am­ple by MacEachren (2004) by crisp­ness, res­o­lu­tion, and trans­parency. Chen and Floridi (2012) ex­tended the list to 30 vari­ables and sub­or­di­nated them into four cat­e­gories: Geo­met­ric, op­ti­cal, topo­log­i­cal and re­la­tional, and se­man­tic chan­nel. While this fur­ther de­f­i­n­i­tion is help­ful it loses the idea of basic vi­sual en­cod­ings.

De­bat­ably all vari­ables from the »Topo­log­i­cal and Re­la­tional Chan­nel« are rather re­sults of the spa­cial po­si­tion­ing and the vari­ables re­gard­ing the mo­tion are also Bertin’s vi­sual prim­i­tives com­bined with time. The con­cept of vi­sual vari­ables needs a sec­ond layer that cov­ers such ad­vanced cases and fill in the gap to the psy­cho­log­i­cal Gestalt prin­ci­ples. While such an ap­proach seems to be promis­ing, it would go be­yond the scope of the­sis. Since Bertin’s ap­proach still has a broader ac­cep­tance I will stick to the pre­vi­ously ex­plained mod­i­fied ver­sion of his tax­on­omy for this the­sis.

Geo­met­ric Chan­nels:

  • Size / length / width / depth
  • Ori­en­ta­tion
  • Shape
  • Cur­va­ture
  • Smooth­ness

Op­ti­cal Chan­nels:

  • In­ten­sity / bright­ness
  • Colour / hue / sat­u­ra­tion
  • Opac­ity / trans­parency
  • Tex­ture (partly geo­met­ric)
  • Line styles (partly geo­met­ric)
  • Shape / blur
  • Shad­ing and light­ing ef­fects
  • Shadow
  • Depth (im­plicit / ex­plicit cues)

Syn­these:

  • Im­plicit mo­tion / mo­tion blur
  • Ex­plicit mo­tion / an­i­ma­tion / flicker

Topo­log­i­cal and Re­la­tional Chan­nels:

  • Con­nec­tion
  • Node / in­ter­nal node / ter­mi­na­tor
  • In­ter­sec­tion / over­lap
  • Depth or­der­ing / par­tial oc­clu­sion
  • Clo­sure
  • Dis­tance / den­sity

Se­man­tic Chan­nels:

  • Num­ber
  • Text
  • Sym­bol / ideogram
  • Sign / icon / logo / glyph / pic­togram
  • Iso­type

vi­sual chan­nels by Chen and Floridi (2012)

One help­ful as­pect of these vi­sual vari­ables is the rank­ing in re­gard to their ef­fec­tive­ness. This can be viewed from dif­fer­ent an­gles. First is the abil­ity to pop-out. Vi­su­al­i­sa­tion can help el­e­ments to stand out from the rest, mak­ing it eas­ier to spot when search­ing for a cer­tain one. Stud­ies (Borgo et. al, 2013) re­veal this order:

The sec­ond abil­ity of vi­sual vari­ables can be mea­sured by human’s per­cep­tion to most eas­ily de­code the in­for­ma­tion »be­hind« the en­coded data point. It is the vari­ables’ abil­ity to be def­i­nite. And also the ef­fort­less­ness of com­par­i­son be­tween mul­ti­ple points varies be­tween the chan­nels. This as­pect was also con­sid­ered by Bertin: For in­stance, the po­si­tion of an ob­ject along the x- and y-axis (the po­si­tion­ing in 3D spaces is being dis­cussed in the vi­su­al­i­sa­tion com­mu­nity con­tro­ver­sial) was de­scribed as the most pow­er­ful for ex­trac­tion and com­par­i­son. This idea of ef­fec­tive­ness was then con­tin­ued by Jock D. Mackin­lay. He pro­vided the sort­ing in re­la­tion to the given data type. Ac­cord­ing to S. S. Stevens, these data types are »nom­i­nal, or­di­nal, in­ter­val and ratio«. (Stevens, 1946) Even though his de­f­i­n­i­tion can be crit­i­cised, his work set a stan­dard in sta­tis­tics. For a clearer sep­a­ra­tion I will dis­tin­guish »in­ter­val« and »ratio« under »quan­ti­ta­tive« data types.

Vi­sual Vari­ables A mod­i­fied ver­sion of Bertin’s vi­sual vari­ables

Quan­ti­ta­tive data can be mea­sured by its amount. This cat­e­gory can be dis­tin­guished once more be­tween in­ter­val and ratio. In­ter­val data sits be­tween two (ar­bi­trary) points like 0° and 100° for Cel­sius tem­per­a­ture or val­ues on a ph-scale. One needs to be care­ful, be­cause even in case the gap be­tween two val­ues can be mea­sured, 50° C is not twice as hot as 25° C. In con­trast, quan­ti­ta­tive data that is given in a ratio can be used for mea­sur­ing and cal­cu­lat­ing for in­stance a num­ber of peo­ple (1, 2, 13), or the weight of an ob­ject (1 pound, 14 kg). These val­ues can be also mea­sured as dis­crete (3 m) or con­tin­u­ous data (10.3 m).

Or­di­nal data can­not be mea­sured quan­ti­ta­tively but in­stead in or­dered cat­e­gories. Given the ex­am­ple of a race, the gap be­tween the first and the sec­ond place might not be the same as the gap be­tween the third and fourth place, or half suf­fi­cient is not un­suf­fi­cient. Quan­ti­ta­tive data can be con­verted into or­di­nal data, but or­di­nal data can­not – with­out fur­ther in­for­ma­tion – be con­verted into quan­ti­ta­tive data.

Nom­i­nal data is also cat­e­gorised, but with­out any order and hence often re­ferred to as qual­i­ta­tive. Nu­mer­i­cal names »are used only as la­bels or type num­bers, and words or let­ters would serve as well.« (Stevens, 1946) For ex­am­ple male is not bet­ter than fe­male, Hindu, Mus­lim or Lutheran are equal, and also blood types like A, B, 0 or AB are not pri­ori­tised by de­fault.

Ty­pog­ra­phy as Vi­su­al­i­sa­tion?

»Let­ters were mainly re­garded as a medium of com­mu­ni­ca­tion; it is es­sen­tial to recog­nise them as media of per­cep­tion.«

 — Sybille Krämer, 2003

Writ­ing as medium are fated in the same way every medium is con­sid­ered: Aris­to­tle al­ready de­scribed that media of per­cep­tion are in­deed ma­te­ri­al­is­tic but at the same time need to be trans­par­ent. (Krämer, 2003) Fritz Hei­der con­tin­ues this con­cept as he states that media make some­thing vis­i­ble which is not »of the ’na­ture‘ of the medium it­self«. Media »ais­theti­sis« by »un-ais­thetis­ing« them­selves (»Me­di­en aisthe­ti­sie­ren, indem sie sich selbst an-aisthe­ti­sie­ren«). (Hei­der in Krämer, 2006) The media philoso­pher Sybille Krämer ex­am­ines this re­la­tion be­tween vis­i­bil­ity and in­vis­i­bil­ity. In her ten the­ses about »no­ta­tional iconic­ity« (Schrift­bild­lich­keit) she pos­tu­lates the pre­sen­ta­tive in­stead of the rep­re­sen­ta­tive per­spec­tive on let­ters. (Krämer, 2003)

The blur­ring of bound­aries be­tween pre­sen­ta­tion and means of pre­sen­ta­tion is also sub­ject to the con­crete po­etry. Mostly through ty­po­graph­i­cal arrange­ment of words »poems of fig­ures« emerge. (Bauer et al., 2010) These fig­ures raise ques­tions on the sim­i­lar­ity of the mean­ing and the op­ti­cal ap­pear­ance of the words. The dis­tinc­tion be­tween fig­ure and text can no longer be ap­plied to this art form. In Eugen Gom­ringer’s poem the word »schwei­gen« (to be silent) has no sim­i­lar­ity to the idea it rep­re­sents, but il­lus­trates its mean­ing through the arrange­ment.

Schweigen by Eugen Gom­ringer (Bauer et al., 2010)

In a sim­i­lar way, my the­sis high­lights ty­pog­ra­phy as a mean of com­mu­ni­ca­tion for the text’s con­tent and not only to serve as a tool for read­abil­ity. To il­lus­trate this prob­lem in ap­pli­ca­tion: Jour­nal­ists pub­lish their ar­ti­cles in a news­pa­per. How­ever, the ty­po­graphic de­ci­sions were made long be­fore the ar­ti­cle was writ­ten by some­one un­in­volved in this par­tic­u­lar or in every other ar­ti­cle. The ap­pear­ance of the text rep­re­sents the news­pa­per, but not the text. Reg­u­la­tions on the font, the columns, use of em­phases or lay­out may not be ben­e­fi­cial for the text.

In order to ex­am­ine ty­pog­ra­phy for vi­su­al­is­ing in­for­ma­tion, I dis­cuss ty­pog­ra­phy from a data vi­su­al­i­sa­tion’s per­spec­tive. The de­scrip­tion of vi­sual at­trib­utes like the ones by Bertin, Cleve­land, Ware or Meirelles sum­marise ty­pog­ra­phy as one vi­sual vari­able. My hy­poth­e­sis is that ty­pog­ra­phy is not a sub­cat­e­gory like »size«, but rather a com­plete new »ob­ject« that can be mod­i­fied with the vi­sual vari­ables de­scribed by Bertin and oth­ers. When com­par­ing the vi­sual prim­i­tives with basic tech­niques used in ty­pog­ra­phy, one can see that these match in most cases con­cern­ing their im­ple­men­ta­tion and ef­fect. For ex­am­ple, a word can – just like a square – be scaled in size or changed in its po­si­tion. But some major points need to be con­sid­ered:

  • In ty­pog­ra­phy there are even more vi­sual vari­ables avail­able due to con­ven­tions like un­der­lin­ing or line-height.
  • It de­pends on whether it is ap­plied to the whole para­graph, the sen­tence, the word, or only the let­ter.
  • Some of the »ty­po­graphic vari­ables« can­not be used with­out com­pro­mises on the leg­i­bil­ity. In con­trast, a rec­tan­gle in a reg­u­lar vi­su­al­i­sa­tion may suf­fer in vis­i­bil­ity but not in leg­i­bil­ity.
  • In fact this is a com­plete new re­la­tion­ship be­tween en­tity and label. Nor­mally the en­tity is mod­i­fied, while the label stays un­af­fected. In my ap­proach the label is also the en­tity and thus re­quires care­ful mod­i­fi­ca­tion.
  • Also, the ini­tial sit­u­a­tion is dif­fer­ent since in reg­u­lar vi­su­al­i­sa­tions each ob­ject rep­re­sents one (or mul­ti­ple ag­gre­gated) data point. There are no ob­jects on the plane that look like data points, but hold no data. When ap­ply­ing vi­su­al­i­sa­tion to only some words or text el­e­ments, the other words re­main in their orig­i­nal form and can act as »nor­mal« for com­par­i­son.
  • The mod­i­fi­ca­tion of any of the ty­po­graphic vari­ables in­flu­ences the grey value (the per­ceived black­ness from a dis­tance) of the text. Some­times, this is not in­tended. The change of the type­face for in­stance might hence not only have a se­man­tic al­ter­ation but also a vi­sual ef­fect.

The table il­lus­trates the ty­po­graphic in­stru­ments for each form of en­cod­ing.

Shape

  • Type­face
  • Case / Style
  • Weight
  • Width

Size

  • Scale

Po­si­tion

  • Kern­ing / Spac­ing
  • In­den­tion / Mar­gin
  • Align­ment
  • Su­pe­rior / In­fe­rior
  • Line spac­ing

Ori­en­ta­tion

  • Ro­ta­tion

Ap­pear­ance

  • Colour
  • Tex­ture
  • Opac­ity


Ad­di­tional vari­ables

  • Un­der­lin­ing
  • Strike through

Ty­po­graphic vari­ables Ty­po­graphic tech­niques sorted as vi­sual vari­ables
Shape

The shape of the let­ters can­not be mapped di­rectly to a sin­gle ty­po­graph­i­cal vari­able. There are mul­ti­ple op­tions to vary the font’s shape. The most rad­i­cal would be the ac­tual type­face. This cat­e­gor­i­cal al­ter­na­tion should be treated with cau­tion. Two major con­cerns need to be taken into ac­count:

  • The dis­tinc­tion of dif­fer­ent type­faces can be hard for the lay­men. Serif, sans-serif, script, black­let­ter and dec­o­ra­tive fonts might eas­ily be dis­tin­guished, but within these cat­e­gories non-spe­cial­ists could hardly sep­a­rate Re­nais­sance, Baroque or Neo­clas­si­cal.
  • An­other thing to be con­cerned of is the in­her­ent cul­tural as­so­ci­a­tion of that font. Black­let­ter fonts are com­monly (and falsely) as­so­ci­ated with the Third Reich, serif fonts with the an­tique and script fonts with a hu­manly touch. As these as­so­ci­a­tions are mainly per­sonal and in­di­vid­ual im­pres­sions in­stead of facts, one should be care­ful when ex­pect­ing the reader to link the con­trasty, ver­ti­cal ori­en­ta­tion to an 18th cen­tury mod­ern Ro­man­ism.

An­other way to al­ter­nate the shape of a font is ei­ther its case (low­er­case or up­per­case) or its style: italic (and rarely left oblique), small caps and roman. This has been com­mon prac­tice since the six­teenth cen­tury (Bringhurst, 2012). This fea­ture can only be used cat­e­gor­i­cally since there is com­monly no op­tion of more or less italic, for ex­am­ple.

The third op­tion is to use dif­fer­ent font-weights. De­pend­ing on the amount of fonts the type­face con­sists of, the font weight is a fea­si­ble vari­able to vi­su­alise in­for­ma­tion. The weight of the font also in­flu­ences the grey value of the text ac­cord­ingly and is there­fore – just like the colour value – able to com­mu­ni­cate its value from first sight and from a greater dis­tance by »pop­ping-out«. In con­trast to the shapes from reg­u­lar vi­su­al­i­sa­tion, dif­fer­ent font weight do have an order. The ex­am­ples given by Bertin (rec­tan­gle, star, tri­an­gle, …) an not be or­dered.

An­other op­tion is the width of the fonts. Some type­faces con­sist of extra fonts like con­densed, nar­row/com­pressed or ex­tended ones. These fonts could be equally used as dif­fer­ent weights, but since they are rather sel­dom (es­pe­cially for an­ti­qua type­faces), real world ap­pli­ca­tions would be rare. Fonts should not be streched out pro­gram­mat­i­cally in order to pre­serve their aes­thetic. This is also why the width is not con­sid­ered as size, but as shape.

One great pos­si­bil­ity would be Mul­ti­ple Mas­ter Fonts. These fonts can con­tin­u­ously in­ter­po­late be­tween two »mas­ters«. Pos­si­ble axes can be weight, width, op­ti­cal size and even style. With this tech­nique in­di­vid­ual in­ter­stages can be gen­er­ated. Un­for­tu­nately, MM fonts could not es­tab­lish it­self and are hence not sup­ported on cur­rent sys­tems. Re­cent de­vel­op­ments on this topic are JavaScript-based so­lu­tions such as Plumin.js, which al­lows to gen­er­ate/ma­nip­u­late fonts with JavaScript code. Pro­to­typo is an­other ap­proach that al­lows font ma­nip­u­la­tion with type de­signer vari­ables like x-height, cap­i­tal height, as­cen­der and de­scen­der height, slant, over­shoot, cross­bar po­si­tion, and width. Open­type.js even al­lows to di­rectly ma­nip­u­late an­chor points. All these tools could po­ten­tially be used for vi­su­al­i­sa­tion with con­tin­u­ous in­ter­states to rep­re­sent val­ues when they are fur­ther de­vel­oped for this kind of ap­pli­ca­tion. The dis­ad­van­tage of all of these tools is the purely math­e­mat­i­cal gen­er­a­tion of the fonts. The per­cep­tion of let­ters and thus their de­sign is not lin­ear. This often leads to un­pleas­ant re­sults. Ei­ther the gen­er­a­tion of fonts be­comes more ad­vanced and takes the non-lin­ear­ity into ac­count or the pos­si­ble un­aes­thetic and il­leg­i­ble ap­pear­ance needs to be ac­cepted. Fur­ther de­vel­op­ment will show if these JavaScript-based tools can man­age what the orig­i­nal Mul­ti­ple Mas­ter tools could not.

Po­si­tion

The po­si­tion of a let­ter can be al­tered in var­i­ous ways. On the x-axis the de­signer can change the kern­ing/spac­ing (the space be­tween let­ters), the in­den­tion/mar­gin (the space – usu­ally – from the left-hand side) and the gen­eral align­ment. On the y-axis the user can change the let­ters to su­pe­rior/in­fe­rior or the gen­eral line-spac­ing.

In re­gard to sin­gle let­ters the spac­ing is a good op­tion to com­mu­ni­cate both qual­i­ta­tive as well as quan­ti­ta­tive val­ues. De­spite the fact that exact de­cod­ing of the data is not pos­si­ble, this tech­nique al­lows for con­tin­u­ous data map­ping. An­other ad­van­tage of this method is also in­de­pen­dency from the type­face: The spac­ing is ap­plic­a­ble to every font. Ob­vi­ously ex­treme val­ues can cause il­leg­i­bil­ity or the drift­ing apart of let­ters. Shift­ing sin­gle let­ters in their y-po­si­tion is al­ready com­mon ty­po­graphic prac­tise. For ex­am­ple, chem­i­cal, phys­i­cal or math­e­mat­i­cal no­ta­tions use a shift for sub- and superscript to cre­ate a hi­er­ar­chy be­tween char­ac­ters. What is im­por­tant to note here is that be­cause of the lit­tle pos­si­bil­i­ties of vari­a­tion within the line height, the vi­su­al­i­sa­tion be­comes cat­e­gor­i­cal or dis­crete rather than con­tin­u­ous. In some cases the reader can only des­ig­nate bot­tom, mid­dle or top po­si­tion in­stead of exact dif­fer­ences. The com­par­i­son is also dif­fi­cult across mul­ti­ple lines. As Cairo (2013) de­scribes, ob­jects with a com­mon base­line are eas­ier to col­late. This also lim­its the mod­i­fied amount of char­ac­ters to keep the base­line within a line row. An­other prob­lem that could cause a lack of com­par­i­son on the y-axis is the dif­fer­ent size of the let­ters. The as­cen­ders and de­scen­ders sug­gest dif­fer­ent start­ing points. It is there­fore rec­om­mended to use cap­i­tal let­ters to work with the same let­ter height.

In gen­eral, the repo­si­tion­ing can cause is­sues on the read­abil­ity. Each let­ter, word, sen­tence and para­graph has an in­her­ent po­si­tion that is cru­cial for the con­tent. The vari­ance of the po­si­tion is there­fore lim­ited and needs to be ap­plied care­fully. Let­ters and words are es­pe­cially bound to their po­si­tion.

The shift on the x-axis of sen­tences or para­graphs is al­ready used. Lists in a table of con­tents for ex­am­ple use dif­fer­ent po­si­tions to in­di­cate its hi­er­ar­chy. Again, a mod­est ex­e­cu­tion should be ap­plied to dis­play both cat­e­gor­i­cal or quan­ti­ta­tive data. One ad­van­tage of the con­tin­u­ous map­ping is that peo­ple can in­stantly un­der­stand the tech­nique.

An unique vari­able is the text align­ment like flush left, jus­ti­fied or cen­tered text. Spread­sheet soft­ware take ad­van­tage of this by align­ing text on the left side, while num­bers are aligned on the right side of the cell. Here, the cul­tural con­ven­tions need to be con­sid­ered.

Ap­pear­ance

As pre­vi­ously men­tioned, colour has the strongest pop-out ef­fect. In the com­monly black and white sur­round­ings of nor­mal text, el­e­ments stressed with colour draw even more at­ten­tion to them­selves. On one side this can be used for ad­van­tage by using it to em­pha­sise cer­tain val­ues, but on the other side it can also dis­tract users from read­ing. Be­cause colour can com­mu­ni­cate in­for­ma­tion from a greater dis­tance, it is often used in com­bi­na­tion with text for way find­ing sys­tems. While it al­lows to con­vey cat­e­gor­i­cal in­for­ma­tion quickly, quan­ti­ta­tive in­for­ma­tion is even more dif­fi­cult to de­code be­cause of the small sur­face area. For this is ad­vis­able to use multi-hue di­verg­ing colour palettes since the size of the font might re­strain the con­cep­tion of di­verse colours. When using the same colour on ex­ten­sive ob­jects and text, it should be ap­plied to the lat­ter with extra sat­u­ra­tion or dark­ness to coun­ter­act the small area.

The use of opac­ity may cause prob­lems for the leg­i­bil­ity. Just like ob­jects that do not stand out from the back­ground are hard to be spot­ted, let­ters that are too sim­i­lar to the back­ground can­not be read any­more. Ad­di­tion­ally, all changes of opac­ity (sim­i­lar to colour) should be stressed es­pe­cially on light­weight fonts to coun­ter­act the small sur­face area. A change of opac­ity is often more prac­ti­ca­ble when com­bined with a change of colour.

The ap­pli­ca­tion of a tex­ture highly de­pends on the type of tex­ture. Nor­mally, the sur­face of the font is smaller than typ­i­cal tex­tured ob­jects like for ex­am­ple coun­tries on a map. This can re­sult in an il­leg­i­bil­ity of the word and also an uniden­ti­fi­ca­tion of the ac­tual tex­ture. Es­pe­cially for small fonts this ty­po­graph­i­cal vari­able is not rec­om­mended. It is worth to men­tion that from a dis­tance the whole text ap­pears as one image and hence gains a tex­ture.

Size

The use of size to in­di­cate value is an al­ready com­mon prac­tise. Head­lines, sub-head­lines, body text and foot notes often use dif­fer­ent font-sizes to in­di­cate hi­er­ar­chy. Car­tog­ra­phy also makes great use of scal­ing in order to vi­su­alise im­por­tance of ob­jects. (Imhof, 2007) But rather than scal­ing con­tin­u­ously for ex­am­ple the height of moun­tains or the size of cities, they are often cat­e­gorised. Since the size of ir­reg­u­lar forms like let­ters is more dif­fi­cult to com­pare than the size of two reg­u­lar squares, the dis­crete val­ues are rather dif­fi­cult to ex­tract and should rather be treated as or­di­nal data.

Just like with the po­si­tion­ing of let­ters, words and sen­tences have an in­her­ent value: Words with more let­ters are longer, cap­i­tal let­ters are usu­ally higher than low­er­case let­ters. In the de­sign of maps, in­her­ent sizes often ca­suse bi­ases. As coun­tries with a greater land­mass ap­pear to be more im­por­tant, »big­ger« words also seem to be more im­por­tant. This prob­lem can be often seen with word clouds, which scale words for ex­am­ple in re­gard to their ap­pear­ance. When en­cod­ing with size, these points should be taken care of:

  • Pro­por­tion As men­tioned above, the width and the height of a font should not be in­di­vid­u­ally scaled to pre­server their leg­i­bil­ity and aes­thetic.
  • Word’s length Since the num­ber of let­ters is the main in­flu­ence on the word’s length and hence its ini­tial size, it is ad­vis­able to use words with the same let­ter length when­ever pos­si­ble. For this, ei­ther rep­re­sen­ta­tive ab­bre­vi­a­tions, short­cuts or codes can be used. For ex­am­ple, the ISO 3166 coun­try codes can be used when deal­ing with coun­tries, words can sim­ply be short­ened with an el­lip­sis or re­placed with an acronym. Mono­spaced fonts also help to com­pen­sate vary­ing let­ter width.
  • Let­ter’s height Cap­i­tal let­ters can also help when en­cod­ing with size. Since cap­i­tal let­ters often have the same height it also makes them eas­ier to com­pare as they have an even ini­tial size. They also have a com­mon start­ing level as they have no as­cen­der or de­scen­ders.
  • Min­i­mum size Ob­vi­ously, the scal­ing should have a min­i­mum value in order to main­tain the read­abil­ity of words. But this might also be a prob­lem: When the zero-point is at some min­i­mum text size, the com­par­i­son can be­come dif­fi­cult, as this min­i­mum size needs to be sub­tracted from the total size.

Sum­marised, one should use the size when

  • com­par­i­son does not need to be exact and
  • when none of the un­der­ly­ing data falls below a value that would re­sult in un­read­able text size or
  • when the given words vary too much in their ini­tial size.

To main­tain leg­i­bil­ity, it is also rec­om­mended to al­ways scale width and height ac­cord­ingly and not in­di­vid­u­ally. When only the width is changed des­ig­nated fonts like com­pressed, con­densed or ex­tended should be used.

Be­sides the scal­ing of the words, the size could be ap­plied to the width or height of the »para­graph box«. When the text is long enough, the text boxes can im­i­tate the usage of bar charts, where the line-length cor­re­sponds to the value.

Ori­en­ta­tion

Dif­fer­ent ori­en­ta­tions can hardly be used with­out com­pro­mises on the text flow. While com­plete sen­tences or para­graphs can be read with vary­ing slopes, let­ters or words should only be care­fully ro­tated within sen­tences. Cor­rect ap­pli­ca­tions can often be found in car­tog­ra­phy, where let­ters fol­low the path of a river or a word ad­heres to the coast­line. In the same way an­gles are dif­fi­cult to ex­tract from ob­jects, the slope of a word can­not be de­coded with­out a doubt. Also as­cen­ders and de­scen­ders can in­crease this dif­fi­culty.

Ad­di­tional ty­po­graph­i­cal vari­ables

Over the course of hun­dreds of years, ty­po­graphic con­ven­tions were es­tab­lished that are unique in graph­i­cal dis­tinc­tion. These are often ad­di­tional graph­i­cal el­e­ments that can dis­tin­guish words. For ex­am­ple, the un­der­lin­ing of a word is clearly high­light­ing that word, while un­der­lin­ing a rec­tan­gle would not be di­rectly un­der­stood as high­lighted. With the Web’s grow­ing size, sig­nif­i­cance and level of ac­cess the un­der­lin­ing of text be­came a se­man­tic mean­ing: being a hy­per­link. Es­pe­cially in com­bi­na­tion with a blue or pur­ple text colour the as­so­ci­a­tion is wide­spread among Web users.

The high­light­ing of text pas­sages also works with char­ac­ters added at the be­gin­ning and end. This way markup lan­guages *highlight* words or de­clare <div>boxes</div> of con­tent. (But also in reg­u­lar usage paren­the­ses in­di­cate a de­val­u­a­tion.)

Con­clu­sion

Sum­marised, ty­po­graphic vari­ables con­tain par­al­lels to vi­su­al­i­sa­tion vari­ables and can be used as such within lim­its. Al­ways in con­sid­er­a­tion of the leg­i­bil­ity, ty­po­graphic dis­tinc­tions need to be ap­plied more care­fully than in reg­u­lar vi­su­al­i­sa­tions. In the same way vi­sual vari­ables are also not equally suit­able for every type of data, some ty­po­graphic vari­ables can used for qual­i­ta­tive val­ues only.

While the po­si­tion is widely re­garded as the best way to en­code in »nor­mal« vi­su­al­i­sa­tions, it needs spe­cial treat­ment with text in order to keep the read­ing order and flow. »Weaker« vi­su­al­i­sa­tion vari­ables like colour ap­pear to be more use­ful be­cause firstly, they do not dis­place the text cor­pus and sec­ondly allow for com­par­i­son across mul­ti­ple text lines, which is dif­fi­cult with a change of x-po­si­tion for ex­am­ple.

Ty­po­graphic con­ven­tions can be ben­e­fi­cial, but need to be treated with re­spect for cul­tural in­her­ent mean­ings. In gen­eral the pro­posed ty­po­graphic vi­su­al­i­sa­tions are not »as strong« as reg­u­lar vi­su­al­i­sa­tion. The ex­trac­tion of exact quan­ti­tive data is never as good as with graph­i­cal means. This is mainly due to the lack of a co­or­di­nate sys­tem that would give the user an ori­en­ta­tion and to the dis­tri­b­u­tion across mul­ti­ple lines that hin­ders com­par­i­son.

In con­clu­sion, the vi­su­al­i­sa­tion with ty­pog­ra­phy is mainly suit­able for or­di­nal and nom­i­nal data. Quan­ti­tive vi­su­al­i­sa­tions should only be used in a sup­port­ive role.

Why ad­here to text?

With vi­su­al­i­sa­tion as the seem­ingly so­lu­tion to all mod­ern prob­lems, the ques­tion arises why one should stick to text. The an­swer to this is based on the cul­tural im­print by the book. As pre­vi­ously stated it had a tremen­dous ef­fect on not only how knowl­edge is dis­trib­uted but also in the way we think. Habits formed con­ven­tions and today most peo­ple are used to ob­tain in­for­ma­tion from writ­ten text with its dis­cur­sive­ness, lin­ear­ity and ab­strac­tion. Mendelssohn be­wails how this has made us literati:

»We teach and in­struct one an­other only through writ­ings; we learn to know na­ture and man only from writ­ings.
We work and relax, edify and amuse our­selves through scrib­bling. The preacher doesn’t talk to his con­gre­ga­tion; he reads or re­cites to it some­thing he has writ­ten.
The teacher reads his writ­ten lec­tures from the podium. Every­thing is dead let­ter; the spirit of liv­ing con­ver­sa­tion has van­ished. […]
This has brought it about that man has al­most lost his value for his fel­low-man. We don’t try to meet with the wise man be­cause we find his wis­dom in his writ­ings. […]
In short, we are literati, men of let­ters.«

 — Mendelssohn, 2011

»Wir leh­ren und un­ter­rich­ten ein­an­der nur in Schrif­ten; ler­nen die Natur und die Men­schen ken­nen nur aus Schrif­ten; ar­bei­ten und er­ho­len, er­bau­en und er­göt­zen uns durch Schrei­be­rei; der Pre­di­ger un­ter­hält sich nicht mit sei­ner Ge­mein­de, er liest oder de­kla­miert ihr eine auf­ge­schrie­be­ne Ab­hand­lung vor.
Der Leh­rer auf dem Ka­the­der liest seine ge­schrie­be­nen Hefte ab. Alles ist toter Buch­sta­be, nir­gends Geist der le­ben­di­gen Un­ter­hal­tung. […]
Daher ist es ge­kom­men, dass der Mensch für den Men­schen fast sei­nen Wert ver­lo­ren hat. Der Um­gang des Wei­sen wird nicht ge­sucht; denn wir fin­den seine Weis­heit in Schrif­ten. […]
Mit einem Worte, wir sind lit­te­ra­ti, Buch­sta­ben­men­schen.«

 — Men­dels­sohn, 1989

I do not to want peo­ple to turn away from books as Mendelssohn’s text might sug­gest, but rather stress how many fields are framed by the book. Even today where the pro­duc­tion of im­ages is ef­fort­less as never be­fore, text does not dis­ap­pear. In­stead, new forms of text »types« evolve: Comic books, In­stant Mes­sag­ing and Twit­ter. In con­trast to the pes­simistic view of some writ­ers, we should utilise these tools for they are still forms of ex­pres­sion based on tex­tual com­mu­ni­ca­tion. While some de­scribe the printed book as »the per­fect read­ing-ma­chine« (Die per­fekte Lese­mas­chine, Reuß, 2014) I per­son­ally see much po­ten­tial for new forms of printed books, but even more for in­ter­ac­tive, re­spon­sive, il­lus­tra­tive ebooks or any form of tex­tual ex­pres­sion.

But the tri­umphant ad­vance of text has not only to do with ha­bit­u­a­tion. The lin­ear­ity, the dis­course and the ab­strac­tion might bring re­stric­tion but it also brings ad­van­tages over the vi­sual com­mu­ni­ca­tion. While im­ages allow the viewer to freely ex­plore the con­tent, text guides the reader through the in­for­ma­tion. The lin­ear­ity brings order, the dis­course al­lows to fol­low ar­gu­men­ta­tion. And with the abil­ity to com­mu­ni­cate ab­stractly, text often al­lows to com­mu­ni­cate in­tan­gi­ble ideas.

For this the­sis I use the promi­nent sta­tus and the qual­i­ties of the text as start­ing point and en­hance its com­pre­hen­sion through the com­bi­na­tion with vi­su­al­i­sa­tions.

Why com­bine both?

These two dis­ci­plines – ty­pog­ra­phy and vi­su­al­i­sa­tion – are too often treated as two sep­a­rate things. Both work with vi­sual vari­ables that are ap­plied ac­cord­ing to the un­der­lay­ing data. Where ty­pog­ra­phy adds this data level to the text, vi­su­al­i­sa­tion works with graph­i­cal el­e­ments to com­mu­ni­cate the data.

In the book »Un­der­stand­ing Comics«, Scott Mc­Cloud il­lus­trates that words and im­ages are two ends on a scale from con­crete to ab­stract. (Mc­Cloud, 2001) He also shows how these two ends can in­ter­act with one an­other, each ful­fill­ing its own pur­pose. He ex­em­pli­fies this with »The Ad­ven­tures of Tintin«: The sim­ply, ab­stract drawn per­sons help the reader to iden­tify with them, be­cause their ap­pear­ance is not so de­ter­min­ing. On the other side, the de­tailed, con­crete back­ground shows the fan­tasy world the story takes place in.

But when ty­pog­ra­phy is ap­plied to the ab­stract text, it be­comes more graph­i­cal and hence more con­crete. On the other side, vi­su­al­i­sa­tions make use of graph­i­cal, iconic forms and thus make the im­ages more ab­stract. Ex­pand­ing Mc­Cloud’s model, the new order is: Im­ages — vi­su­al­i­sa­tions — vi­su­al­is­ing ty­pog­ra­phy — text. While most pub­li­ca­tions make only use of the ex­tremes of the spec­trum, my ap­proach in­vites the de­signer to use all »tools« of this range to tell a story.

Range of Ab­strac­tion Im­ages be­come more ab­stract and text be­comes more con­crete.

But not only re­sults the com­bi­na­tion in a greater va­ri­ety of sto­ry­telling-el­e­ments for the sake of cross-me­di­al­ity, but also helps the reader to un­der­stand the con­tent. Al­brecht Dürer wrote in his text­book for the ado­les­cent youth al­ready that he wants to com­bine image and let­ter to en­hance mem­o­ra­bil­ity. (»Des­halb will ich das Wort und das Werk zam­men­tan, auf daß mans dest baß mer­ken müg.«) (Wen­zel, 2006) Sim­i­larly, the brain re­searcher Wolf Singer states that when our per­cep­tion is con­firmed by two stim­u­lus modal­i­ties, the per­ceived is eval­u­ated as to be the truth. For ex­am­ple when the eye sees an ob­ject that is also recog­nised by an­other sen­sory per­cep­tion, the as­sump­tion of what is »out there« is more rea­son­able. It passes the test of con­sen­sus. (Wen­zel, 2006) Even though this does not di­rectly jus­tify the use of dif­fer­ent en­cod­ings within one sen­sory medium, it paves the way to this re­search: And in­deed the com­bi­na­tion of im­ages and words has been sci­en­tif­i­cally stud­ied by a num­ber of re­searchers. Most of the re­sults sup­port the the­ory of »dual cod­ing« by Mayer et al. (1999 in Ware, 2004). »They sug­gest that if ac­tive pro­cess­ing or re­lated ma­te­r­ial takes place in both vi­sual and ver­bal cog­ni­tive sub­sys­tems, learn­ing will be bet­ter.« (Ware, 2004) Sum­marised, this shows how an in­creased ex­ploita­tion can be achieved through the com­bi­na­tion of text and image.

One study that fol­lowed this the­sis’ ap­proach from the other side was con­ducted by Chan­dler and Sweller (1991 in Ware, 2004). It ex­am­ined the in­te­gra­tion of text di­rectly in­side graph­ics. Their test graphic — a di­a­gram with op­er­at­ing in­struc­tions — had text snip­pets di­rectly placed at the rel­e­vant point. Their suc­cess­ful ex­per­i­ment was jus­ti­fied by the »lim­ited-ca­pac­ity work­ing mem­ory«. »They argue that when the in­for­ma­tion is in­te­grated, there is a re­duced need to store in­for­ma­tion tem­porar­ily while switch­ing back and forth be­tween lo­ca­tions.« (Ware, 2004) This is often jus­ti­fied by the cog­ni­tive load the­ory. As the the­ory says, our work­ing mem­ory is lim­ited. Hence ob­ject and key should be close to­gether to re­duce in­for­ma­tion that needs to be stored tem­po­rally in our mem­ory. It may be ex­pected that this con­cept per­forms in the same way with im­ages di­rectly in­te­grated into the text.

Other psy­cho­log­i­cal stud­ies on read­ing com­pre­hen­sion can also be in­ter­preted to sup­port the the­sis’ ap­proach. Per­rig and Kintsch (1985 in Müsseler et. al, 2002) de­scribed a fic­ti­tious city first from a bird’s per­spec­tive (»north of the high­way and east of the river is a gas sta­tion«) and sec­ondly from a dri­ving per­spec­tive (»When cross­ing the river, you see a gas sta­tion to your left«). Can­di­dates who re­ceived the bird’s per­spec­tive were less able to re­peat the text, but in­stead could eas­ily draw a map. (Müsseler et. al, 2002) It can be ex­pected to sup­port mem­o­ra­bil­ity when rep­re­sent­ing the in­for­ma­tion for ex­am­ple in dri­ving per­spec­tive as text and from a bird’s view on a map. It needs to be eval­u­ated in fur­ther re­search if this »dual cod­ing« men­tioned be­fore also helps when text and image hold dif­fer­ent per­spec­tives.

An­other rea­son, why the com­bi­na­tion of these two pro­vide ad­di­tional in­sight, lays in the fre­quently given rea­son­ing for vi­su­al­i­sa­tion. For this, the Anscombe’s quar­tet is often used. After pre­sent­ing ta­bles with num­bers, vi­su­al­iz­ing the same num­bers in di­a­grams seems ob­vi­ous and con­ve­nient. And in­deed it is much eas­ier to com­pare and spot trends and pat­terns with the di­a­grams. But one thing is often ne­glected: The table »ex­presses […] val­ues pre­cisely and it pro­vides an ef­fi­cient means to look up val­ues for a par­tic­u­lar […]« di­men­sion. (Few, 2014) For some graph­ics, the un­der­lay­ing data might even not be ex­tractable at all. For most ap­pli­ca­tions that is ab­solutely valid: The more data points you have, the less im­por­tant the sin­gle one be­come. Vi­su­al­i­sa­tions’ pur­pose is to serve for im­pres­sion, rather than exact num­bers. As John Tukey states, it is bet­ter to be »ap­prox­i­mately right rather than ex­actly wrong«. (Tukey, 1965) What should be taken from this is that both »vi­su­al­i­sa­tions«  — table and graphic — have an unique qual­ity, which are ad­vis­able for dif­fer­ent set­tings. And the same holds true for text. Through the com­bi­na­tion of these dif­fer­ent forms the reader can de­cide which one suit his in­ter­est or abil­ity to ex­tract the data best.

Anscombe’s quar­tet the val­ues in charts
Anscombe’s quar­tet the val­ues in a table

These qual­i­ties can be ex­em­pli­fied by un­cer­tainty and di­ver­sity. While vi­su­al­is­ing un­cer­tainty is gain­ing more at­ten­tion in re­cent time (Pot­ter et al., 2012), text can often more eas­ily ex­press this un­cer­tainty. And in other cases, where the ver­bal ex­press­ing gen­er­alises and la­bels in­for­ma­tion (Drucker, 2014), vi­su­al­i­sa­tion can demon­strate di­ver­sity.

Each form can com­pen­sate the dif­fi­cul­ties of the other and can even help to over­come the lack of ex­pres­sion in text in gen­eral. As Marc Davis wrote in re­gard to emo­jis: »they are best used as an ‘ad­junct’ to text — es­pe­cially in so­cial media — help­ing to make up for the the lack of ges­tures, fa­cial ex­pres­sions and in­to­na­tion found in speech.« (Stern, 2015) And while this might seem only ap­plic­a­ble for emo­jis, graphic nov­els show us how anger, fear or gen­tle­ness can be ex­pressed through type. Hence, the mix­ture of mul­ti­ple sources of in­for­ma­tion cre­ates »a more com­plete image«, re­sult­ing in a greater ex­ploita­tion for read­ers.

The im­pres­sion an user is given by vi­su­al­i­sa­tions can also be used as ben­e­fit: Wil­lett et al. de­scribe the con­cept of Scented Wid­gets which act as »em­bed­ded vi­su­al­iza­tions that fa­cil­i­tate nav­i­ga­tion in in­for­ma­tion spaces«. (Wil­lett et al., 2007) The idea be­hind this ap­proach is to give users a pre­view (a scent) of what they are about to see, when they click on the equipped wid­get. Be­cause of the sim­plis­tic ap­pear­ance users are able to com­pare mul­ti­ple el­e­ments of in­for­ma­tion (or in this case choices) at once.

What needs to be con­sid­ered is that not all text es­pe­cially lit­er­a­ture »wants« to be equipped with ad­di­tional graph­ics. As these vi­su­al­i­sa­tions can add the ap­pear­ance of an analy­sis and sta­tis­ti­cal ef­fort, one needs to ac­cept that es­pe­cially lyri­cal texts are often not meant to be analysed by math­e­mat­i­cal means.

Sum­marised, de­part­ments too often ad­here to their medium: Jour­nal­ists to text, fi­nanciers to ta­bles, de­sign­ers to graph­ics. Each dis­ci­pline should more often use the ad­e­quate pre­sen­ta­tion for the mes­sage. Or as this the­sis shows: A com­bi­na­tion.

Why Micro Vi­su­al­i­sa­tion?

The com­bi­na­tion of text and vi­su­al­i­sa­tion can be ap­proached from many di­rec­tions. Be­sides the aim to cre­ate a bal­ance be­tween the two, the focus could be ei­ther on the ver­bal or on the graph­i­cal side. For this the­sis I will focus on text with vi­su­al­i­sa­tions em­bed­ded. There are six rea­sons for that:

  • Based on the pre­vi­ously dis­cussed con­sid­er­a­tions on the his­tory of com­mu­ni­ca­tion I have come to be­lieve that text is still human’s main medium of knowl­edge im­par­ta­tion. And even with­out the as­pect of ad­di­tional graph­i­cal vi­su­al­i­sa­tions that is dis­cussed in this the­sis, the re­search to en­hance this stan­dard text pre­sen­ta­tion through ty­po­graph­i­cal vi­su­al­i­sa­tions is valu­able for this field.
  • In re­gard to the first point I be­lieve that an ad­duc­tion to vi­su­al­i­sa­tion for the gen­eral pub­lic could be best achieved with sim­ple graph­ics em­bed­ded right into the fa­mil­iar medium. The sim­pli­fi­ca­tion is the first step to make them ac­ces­si­ble to a broader au­di­ence. With chang­ing read­ing con­ven­tions and the ac­cep­tance of this novel sources of in­for­ma­tion, more com­plex vi­su­al­i­sa­tions can be used in mass media.
  • The use of both text and im­ages is claimed to have a higher ef­fi­ciency than sin­gle-chan­nel en­cod­ing. One as­pect that is cru­cial for this dual-cod­ing and also for the con­cept of this the­sis is the con­nec­tion be­tween these two lev­els. That sup­ports the de­ci­sion to di­rectly in­te­grate the vi­su­al­i­sa­tion in­side the text to pro­vide this kind of link.
  • Ob­vi­ously this link needs to be ex­plicit in order to con­nect cor­re­spond­ing el­e­ments. (Ware, 2004) Ex­per­i­ments by Zell­weger et al. (2000 in Gof­fin et al. 2014) eval­u­ated the po­si­tion­ing of meta-in­for­ma­tion re­lated to hy­per­links in mar­gins, foot­notes or in­line. »They found a large vari­abil­ity in par­tic­i­pants’ pref­er­ences but ob­served that, in gen­eral, it was de­sir­able to keep meta-in­for­ma­tion close to the orig­i­nal link. Where in­for­ma­tion was placed also had a sig­nif­i­cant im­pact on read­ing speed with in­for­ma­tion close to the links being read sig­nif­i­cantly faster.« (Gof­fin et al., 2014) Even though deal­ing with dif­fer­ent in­for­ma­tion these re­sults are a con­fir­ma­tion for the usage of in­line vi­su­al­i­sa­tions for bet­ter and quicker com­pre­hen­sion.
  • Pre­vi­ous re­search in the field of vi­su­al­i­sa­tion has been mostly to­wards com­plex, data-rich graph­ics. I see a gen­eral need for smaller, sim­pler vi­su­al­i­sa­tions. As the media con­sump­tion tends to­wards mo­bile, con­tent needs to be de­signed for the smaller dis­play sizes. (Noirhomme-Frai­ture et al., 2005) And with screens be­com­ing cheaper and with OLED tech­nol­ogy also more en­ergy ef­fi­cient so that they can be used for even more and also smaller de­vices like smart­watches, pho­tog­ra­phy lenses or elec­tronic tooth­brushes, vi­su­al­i­sa­tion need to be specif­i­cally de­signed for this size. The »In­ter­net of Things« is be­com­ing in­evitably a re­al­ity and all the de­vices hold op­por­tu­ni­ties for vi­su­al­i­sa­tions of the ob­jects’ data. With their lim­ited out­put ca­pac­i­ties, the vi­su­al­i­sa­tion of their sta­tus is a cru­cial as­pect. But not only for ex­plicit small dis­plays, but also for vary­ing out­put de­vices small vi­su­al­i­sa­tion can be eas­ier adopted to the avail­able space.
  • But also in time cru­cial en­vi­ron­ments like news­pa­pers Micro Vi­su­al­i­sa­tions can be help­ful. Smaller graph­ics are often eas­ier to pro­duce. Es­pe­cially with the pro­posed ty­po­graphic vi­su­al­i­sa­tions, this type of vi­su­al­i­sa­tion can also be cre­ated using am­a­teur soft­ware like Mi­crosoft Word. In­stead of aim­ing for com­plex and costly graph­ics, the op­tions pos­si­ble with the avail­able means are often not taken into ac­count.
  • As we can see from the three stages of per­cep­tion: We see be­fore we read. Using vi­sual and ty­po­graphic vari­ables that con­vey the con­tent vi­su­ally al­lows the reader to com­pre­hend the struc­ture and pos­si­bly also parts of the con­tent be­fore read­ing. Es­pe­cially colour that pops out from the rest can guide the reader to rel­e­vant parts.

Rel­e­vance

With the de­riva­tion of my ap­proach in the last chap­ters, I will now fur­ther ex­plain its rel­e­vance in a cul­tural and tech­no­log­i­cal per­spec­tive and elab­o­rate its rel­e­vance and im­pli­ca­tions for de­sign. In the last part I will ex­plain my own mo­ti­va­tion for the ap­proach.

Cul­tural and tech­no­log­i­cal rel­e­vance

Tech­nol­ogy and cul­ture fol­low each other, fa­cil­i­tate and de­mand each other, de­pend on each other. The his­tory of com­mu­ni­ca­tion is a his­tory of tech­nol­ogy. Nowa­days, as tech­nol­ogy changes so rapidly, our be­hav­iour highly de­pends on our tech­nol­ogy usage. Our life is more than ever de­fined by de­vices and their in­her­ent func­tion­ing. As these de­vices be­come more com­pli­cated to use we need to learn how to use them, learn to un­der­stand their out­put. The data gen­er­ated by our com­put­ers and its amount is often be­yond our un­der­stand­ing. But not only ab­stract data comes in an over­whelm­ing amount. Also news, in­for­ma­tion of rel­e­vance or daily im­pres­sions in­un­date our life. The de­mand for sort­ing, con­tex­tu­al­i­sa­tion and fil­ter­ing be­comes stronger every day. In the fol­low­ing sec­tions. I show how my the­sis re­lates to these prob­lems.

Ris­ing quan­ti­ties

McLuhan’s »im­plo­sion« (McLuhan, 2011) of the world brings peo­ple to­gether in quan­ti­ties that have never been pos­si­ble be­fore. Elec­tron­i­cally processed in­for­ma­tion al­lows higher rates and greater amounts of data. Peo­ple nowa­days take a thou­sand pic­tures in their hol­i­days, com­pa­nies trade with goods in quan­ti­ties of mil­lions, coun­tries have GDPs in bil­lions of dol­lars. But be­sides higher di­men­sions we also mea­sure smaller and smaller di­men­sions. Charles and Ray Eames’s short film »Pow­ers of Ten« (Eames, 1977) il­lus­trates this im­pres­sively: start­ing from above a cou­ple hav­ing break­fast, the cam­era zooms out until it reaches the limit of our vi­sion 100 mil­lion lightyears away and then zooms in to a sin­gle pro­ton with a size of 0.0001 ångströms. And also Gavin Schmidt de­scribes our ca­pa­bil­i­ties to al­ready mea­sure 14 or­ders of mag­ni­tude of weather data: »from the small mi­cro­scopic par­ti­cles that seed clouds to the size of the planet it­self«. (Schmidt, 2014) The re­cently pop­u­lated term »big data« (which seems to have be­come a buzz­word es­pe­cially in busi­ness) is used for the ris­ing quan­ti­ties of data that are being cap­tured through the elec­tron­i­cal pro­cess­ing of sci­en­tific cal­cu­la­tions and mea­sure­ments, fi­nan­cial trans­ac­tions, elec­tronic com­mu­ni­ca­tion and in­ter­net traf­fic in gen­eral. Com­pa­nies like Google, Ama­zon, or Face­book, but also sci­en­tific in­sti­tu­tions like the CERN or na­tional agen­cies like the NSA use these data and the ris­ing com­put­ing power to find pre­vi­ously un­de­tectable pat­terns and in­sights. To em­pha­size this im­por­tance, the World Eco­nomic Forum rises the sta­tus of data to be the new oil. (World Eco­nomic Forum, 2011)

Com­put­ers allow us to process this amount of quan­ti­ties, but it seems that our lan­guage lacks be­hind the tech­no­log­i­cal progress: The dif­fer­ence be­tween two pieces and a mil­lion pieces is tremen­dous and still the Eng­lish and Ger­man lan­guages for ex­am­ple have only two modes to de­scribe quan­ti­ties: sin­gu­lar and plural. Some other lan­guages have more forms of ex­pres­sion quan­ti­ties. Gravel G. Cor­bett sum­marises these in his book »Num­ber« (2000): Some lan­guages have dual, trial and (de­bata­bally) quadrals. For ex­am­ple the Upper Sor­bian lan­guage can dif­fer­en­ti­ate be­tween »you« (»ty«), »you two« (»wój«), and »you (all)« (»wy«). Ac­cord­ingly, the trial and quadral ex­presses the amount of three re­spec­tively four. Other lan­guages allow for a dis­tinc­tion of sin­gu­lar, pau­cal and plural. The pau­cal can be de­scribed as »small plural«, often used for quan­ti­ties be­tween two or to six. It can be found for ex­am­ple in Lihir, an Oceanic lan­guage spo­ken on a group of is­lands next to Papua New Guinea. They use five forms: Sin­gu­lar, dual, trial, pau­cal and plural. What can also be found is a split in the plural, to dif­fer­en­ti­ate amounts of many. This serves for var­i­ous func­tions: In the Ara­bic lan­guage for ex­am­ple it dis­cerns be­tween a gen­eral form (»debbānten« : flies) or the ex­pres­sion of abun­dance (»dababīn« : many flies). In Kayte­tye (an Arandic lan­guage spo­ken in Cen­tral Aus­tralia) the two forms of plural dis­cern be­tween many and »all the x in the uni­verse of dis­course«. Two last ex­am­ples come from the Celtic lan­guage Bre­ton that de­scribes »chil­dren« (»bu­gal-e«) and »groups of chil­dren« (»bu­gal-e-où«) and the Warekena lan­guage (an Arawakan lan­guage spo­ken in Brazil and Venezuela) that can ex­press »pigs« (»abida-pe«), »very many pigs« (»abida-nawi«) and »very many pigs in­deed, too many to count« (»abida-pe-nawi«) by com­pos­ing dif­fer­ent forms in one word. (Cor­bett, 2000)

What can be ex­tracted from this ex­cur­sus is that in fact very few lan­guages have forms to dif­fer­en­ti­ate be­tween mul­ti­ple quan­ti­ties and it may be ex­pected to cause trans­la­tion and un­der­stand­ing prob­lems in other lan­guages that lack these forms. What vi­su­al­i­sa­tion can con­tribute here is the prob­a­bly uni­ver­sally un­der­stand­able form of ex­press­ing such quan­ti­ties. Amounts of »two«, »tree«, »all«, »groups of mul­ti­ple« and »too many to count« can be eas­ily com­mu­ni­cated in an image.

Quan­ti­ties in Image and in writ­ten text

Im­ages of vi­su­alised quan­ti­ties gain im­por­tance for two rea­sons:

  • First, many lan­guages do not pro­vide meth­ods to ex­press amounts ac­cen­tu­ated. Ob­vi­ously miss­ing gram­mat­i­cal forms can be com­pen­sated by fur­ther tex­tual ex­pres­sions, but these sup­ple­ments often fail in con­cise­ness and uni­ver­sal­ity.
  • The sec­ond as­pect is based on the ris­ing amounts of data. Be­cause nowa­days the im­por­tance of an ob­ject re­lies so much on its un­der­lay­ing quan­ti­ties it needs to be di­rectly in­te­grated and con­nected.

In gen­eral, the data of an ob­ject be­comes so im­por­tant that this in­for­ma­tion should be placed to­gether with the ob­ject.

Tech­no­log­i­cal change

Pro­cess­ing in tech­nol­ogy de­mands re­design. Every time a new medium is in­vented it is treated in the way the old tech­nol­ogy was used. It takes time to adapt the usage to new input meth­ods, faster trans­mis­sion or greater range. I be­lieve we live in a time where the ap­pli­ca­tion of tech­nol­ogy is lack­ing be­hind the pos­si­bil­i­ties tech­nol­ogy would allow.

Scott Klein showed in his talk »The for­got­ten His­tory of News Graph­ics« (2014) that news­pa­pers that pub­lished vi­su­al­i­sa­tions in the 1840s needed to build these graph­ics out of let­ters or other sym­bols. For ro­tated el­e­ments they needed to build spe­cial con­struc­tions in the com­pos­ing gal­ley. With the in­ven­tion of off­set print, this ef­fort has been dra­mat­i­cally re­duced. El­e­ments can be freely com­pos­ited dig­i­tally but still the de­sign of the book has not changed much in the last 500 years. Now with the emer­gence of new dig­i­tal tech­nol­ogy the text lay­out is even less re­stricted to the clas­si­cal de­sign. But often the lay­out of books, posters or web­sites still keeps the form the let­ter­press re­quired. Dig­i­tal print­ing and dis­play­ing meth­ods would allow ab­solute free­dom over the book’s lay­out, but is »re­strained by con­ven­tions of de­sign and read­ing«. (Drucker, 2014) McLuhan spoke about this ever-re­peat­ing at­ti­tude in an in­ter­view in 1969:

»Theirs is the cus­tom­ary human re­ac­tion when con­fronted with in­no­va­tion: to floun­der about at­tempt­ing to adapt old re­sponses to new sit­u­a­tions or to sim­ply con­demn or ig­nore the har­bin­gers of change — a prac­tice re­fined by the Chi­nese em­per­ors, who used to ex­e­cute mes­sen­gers bring­ing bad news.
The new tech­no­log­i­cal en­vi­ron­ments gen­er­ate the most pain among those least pre­pared to alter their old value struc­tures.The literati find the new elec­tronic en­vi­ron­ment far more threat­en­ing than do those less com­mit­ted to lit­er­acy as a way of life. When an in­di­vid­ual or so­cial group feels that its whole iden­tity is jeop­ar­dized by so­cial or psy­chic change, its nat­ural re­ac­tion is to lash out in de­fen­sive fury. But for all their lamen­ta­tions, the rev­o­lu­tion has al­ready taken place.«

— McLuhan, 1969

As he states, the »literati« are balk­ing at the change, and I be­lieve that non-spe­cial­ists from other dis­ci­plines can help the fu­ture book to find its form But this needs to hap­pen not only for the sake of change, but there is also a de­mand: Dig­i­tally pub­lished books can­not be de­signed in the same way ana­log let­ter­press books were de­signed, but need to con­sider vary­ing read­ing en­vi­ron­ments, ma­te­ri­als and dif­fer­ent read­ing be­hav­iours. Myers et al. state that we also need to »cre­ate user in­ter­faces that work with vastly dif­fer­ent sizes«. (Myers et al., 2000) Their ar­ti­cle form March 2000 shows that dif­fer­ence of dis­plays »are fac­tors of over 1000 in num­ber of pix­els and 100 for size«, where it is note­wor­thy to men­tion that the high­est res­o­lu­tion wall-sized dis­play they de­scribe was used by the Stan­ford Uni­ver­sity and had a sim­i­lar pixel num­ber to the lap­top this the­sis is writ­ten on 15 years later. In re­cent time, re­spon­sive web de­sign evolved as an an­swer to this prob­lem. Lay­outs from web­sites can now be specif­i­cally de­fined for every res­o­lu­tion. Since every dis­play size leads to dif­fer­ent read­ing dis­tance as well, Oliver Re­ichen­stein of iA dis­cusses the con­cept of re­spon­sive ty­pog­ra­phy. (Re­ichen­stein, 2012) He dis­tin­guishes be­tween the met­ric and the per­ceived size of the text. Since the met­ric size nor­mally stays fixed on var­i­ous de­vices the per­ceived size varies. He de­mands a fixed per­ceived size and hence an adap­tion of the met­ric size for each de­vice’s read­ing dis­tance.

But not only should the book’s de­sign adapt to the user’s needs, but also ex­plore newly formed op­tions. Some avant-garde ana­log books al­ready ex­plore these pos­si­bil­i­ties. Jonathan Safran Foer’s book »Ex­tremely loud and in­cred­i­bly close« is an ex­am­ple of ex­per­i­men­tal ty­pog­ra­phy. Con­verg­ing and drift­ing apart text, marked and crossed out words, hand­writ­ing and pho­tographs are used through­out the book. The book uses the pos­si­bil­i­ties of mod­ern ty­pog­ra­phy to sup­port its con­tent, the pro­tag­o­nist’s thoughts and ac­tiv­ity and the over­all mood of the book.

Drucker states that the cur­rent »new writ­ing modes are shaped by so­cial media, by email, blogs, Twit­ter, and wikis. In these chang­ing con­ven­tions the sur­face of in­ter­face often con­ceals the back-end tech­ni­cal and con­cep­tual processes by which they are pro­duced.« (Drucker, 2014) As with all elec­tron­i­cal de­vices, the ap­pear­ance is sep­a­rated from the func­tion. Dig­i­tally pub­lished text for ex­am­ple is often de­signed for in­ter­ac­tion. Face­book users can »like« every­thing pub­lished with­out need­ing to know how the servers store the »Like«. Only the metaphoric thumbs up icon brings the process back to the per­cep­ti­ble re­al­ity of the user. Other ex­am­ples for these in­ter­ac­tive texts are the plat­form Medium.com which al­lows for com­ment­ing sin­gle para­graphs, the browser plu­gin Dic­tio­nary of Num­bers, which wants to »put num­bers in human terms« by trans­lat­ing num­bers like 86 mil­lion on web­sites into fa­mil­iar terms, and Google Docs en­ables si­mul­ta­ne­ous work on a sin­gle doc­u­ment. Thus one might say that the dis­play of the text be­comes the in­ter­face of the text.

An­dries van Dam de­mands in­ter­faces to »enabl[e] and em­power[…] peo­ple to be cre­ative, pro­duc­tive, and less con­strained by the lim­i­ta­tions cur­rently im­posed by the tech­nol­ogy.« (van Dam, 2001) My con­cern is that cur­rently only few take ad­van­tage of what tech­nol­ogy al­ready en­ables us to do. The given ex­am­ples are by lead­ing com­pa­nies or in­de­pen­dent de­sign­ers. The media con­sumed by the cen­tre of the so­ci­ety is still dom­i­nated by clas­sic text for­mats. We need more ex­per­i­ments and im­ple­men­ta­tions of be­hav­iour that would not be pos­si­ble with let­ter­press tech­nol­ogy and show how com­mu­ni­ca­tion can be al­tered for good or also for bad. A change many in­dus­tries who de­pended on the old tech­nol­ogy seek for. Jour­nal­ism for ex­am­ple is strug­gling with an iden­tity cri­sis as its for­mer dis­tri­b­u­tion and speed is not com­pat­i­ble with peo­ple’s usage of the in­ter­net. A new role and a dif­fer­ent form of com­mu­ni­ca­tion could help over­come its prob­lems. As com­pa­nies search for in­for­ma­tion within their big data, news­pa­pers as ex­pounders of the world might seek for an­swers in the same data pools. As ex­plained above, the use of vi­su­al­i­sa­tions helps to ex­plore as well as to com­mu­ni­cate these find­ings.

Sum­marised, dig­i­tally pro­duced and pub­lished books or other essay forms allow for a form that com­bines mul­ti­ple medi­ums within one. »Dig­i­tal media have be­come the meta-medium par ex­cel­lence, able to ab­sorb and re-me­di­ate all pre­vi­ous forms in a fluid en­vi­ron­ment in which remix­ing and cul­ture jam­ming are the com­mon cur­rency.« (Bur­dick et al., 2012, p. 15)

In­for­ma­tion over­flow

We live in a time where »in­for­ma­tion gen­tly but re­lent­lessly driz­zles down on us in an in­vis­i­ble, im­pal­pa­ble elec­tric rain.« (von Baeyer, 2003) To find sense in these data is our daily chal­lenge. Com­pa­nies like Google spe­cialise in this sense mak­ing by mak­ing avail­able web­sites search­able. But even when we quickly find re­sources, the chal­lenge to quickly com­pre­hend and eval­u­ate the given con­tent re­mains.

One so­lu­tion to this analy­sis of this con­tent is the cat­e­gori­sa­tion of ob­jects. This has been a long prac­tise for li­braries or dic­tio­nar­ies in order to sort and later find in­for­ma­tion quicker. For this, ob­jects are sorted into cat­e­gories and sub­cat­e­gories, which can be eas­ily nav­i­gated top-down. Cross-ref­er­ences also allow for links be­tween ob­jects. This way they stay in their place while still being con­nected. The whole in­ter­net is built upon the same struc­ture: web­sites con­tain links that bring the users to the next web­site. When web­sites are or­gan­ised in this way it can help users to quicker find rel­e­vant con­tent. But not al­ways is this struc­ture vis­i­ble. The UTS Li­brary for ex­am­ple il­lus­trates its cat­a­logue with a stacked bar. Each colour rep­re­sents one sub­ject and its width rep­re­sents the amount of books in this field. Each el­e­ment also acts as nav­i­ga­tion and splits out to more colours that rep­re­sent sub­cat­e­gories. This Micro Vi­su­al­i­sa­tion il­lus­trates the com­plete cat­a­logue con­sist­ing of thou­sand of books in a mem­o­ris­able, nav­i­ga­ble, and com­pact way.

But also after the cor­rect web­site is found on Google and on this web­site the rel­e­vant page is found, the pre­sented doc­u­ment may be hard to com­pre­hend. And again Micro Vi­su­al­i­sa­tion can help. Vi­su­al­i­sa­tions can for ex­am­ple act as an overview of the struc­ture, they can help to find rel­e­vant points by »pop­ping-out«, or they can show the in­for­ma­tion in its con­text.

When there is so much in­for­ma­tion from dif­fer­ent sources the con­text be­comes more im­por­tant. To pro­vide the user with this in­for­ma­tion but not over­whelm him is cru­cial. For this it is help­ful to pro­vide the reader with fur­ther in­for­ma­tion he can ac­cess if nec­es­sary. This con­cept matches Shnei­der­man’s »Overview first, […] then de­tails-on-de­mand« mantra in its basic con­cept. While Shnei­der­man’s start­ing point is also a vi­su­al­i­sa­tion, my ap­proach starts with the text in which the user can »zoom« to fur­ther in­for­ma­tion on de­mand through a vi­su­al­i­sa­tion.

To pro­vide the user with ori­en­ta­tion, nav­i­ga­tion, and com­pre­hen­sion aids does not only help the reader, but also makes the con­tent find­able and ac­ces­si­ble in the great amount of in­for­ma­tion given. There­fore the au­thor should strongly sup­port this process by not only pre­sent­ing raw text or the »ten blue links«.

Rel­e­vance as a de­sign re­search

This work has a high rel­e­vance for the dis­ci­pline of de­sign. Ty­pog­ra­phy is one of the very bases of de­sign that comes to ap­pli­ca­tion every day. Vi­su­al­i­sa­tion is just about to find its way into a broader pub­lic. For this mixed au­di­ence the re­quire­ments for such graph­ics change and need to shift from ex­pert dri­ven »tools« to ca­sual read­ers. In the fol­low­ing I dis­cuss three main points to this.

Vi­su­al­i­sa­tion to­wards micro vi­su­al­i­sa­tion

The term micro in this the­sis does not only refer to small in size but also small in its num­ber of data di­men­sion or data points. Not in all cases a higher num­ber of data points add value to the mes­sage of the graphic, but can some­times even pre­vent the user from ex­tract­ing rel­e­vant in­for­ma­tion. As most ar­ti­cles which can be found in news­pa­per or mag­a­zines today do not deal with a great amount of data, they should still be able to com­mu­ni­cate these few points cor­rectly and take ad­van­tage of vi­su­al­i­sa­tion tech­niques.

Broader un­der­stand­ing of vi­su­al­i­sa­tion

As I stated be­fore in re­la­tion to ty­pog­ra­phy, the term or dis­ci­pline of vi­su­al­i­sa­tion should not be tied to bar charts, scat­ter­plots or net­work di­a­grams. Every de­sign ob­ject should be treated as a form of in­for­ma­tion vi­su­al­i­sa­tion. What I seek for is a func­tional de­sign ap­proach in which every as­pect of de­sign is tested to­wards its en­cod­ing pos­si­bil­i­ties. Most of my ap­proaches could prob­a­bly be de­scribed as ap­plied ty­pog­ra­phy, us­abil­ity or in­ter­face de­sign. And they cer­tainly are in their meth­ods, but de­spite the nor­mal prac­tise I pro­pose ex­e­cu­tion with the data layer in mind. As Jo­hanna Drucker de­scribes, nearly all meth­ods used for vi­su­al­i­sa­tion were de­vel­oped in other dis­ci­plines. (Drucker, 2014) My wish is to bring these meth­ods back to their sub­jects in their de­sign-im­proved form. For ex­am­ple, ge­og­ra­phy, sta­tis­tics, and – es­pe­cially in re­gard of this the­sis – ty­pog­ra­phy can all profit from this ex­change.

Dis­solv­ing the in­ter­face

Bill DeR­ouchey’s talk about the his­tory of the but­ton shows the tran­si­tion from levers to but­tons, metaphors of these but­tons on sur­faces and as he pre­dicts to fluid but­tons, where every­thing can act as a but­ton. (DeR­ouchey, 2010) The process he de­scribes is the de­vi­a­tion from the ac­tual phys­i­cal op­er­a­tion the but­ton causes. This is only pos­si­ble under two con­di­tions:

The first is the tech­ni­cal de­vel­op­ment to elec­tron­i­cally processed in­for­ma­tion. This al­lows »com­pressed time«, »com­pressed dis­tance« and »ab­stract mo­tion«. (DeR­ouchey, 2010) The elec­tronic sig­nal of the but­ton being pushed can re­sult in a much stronger re­sult than the ac­tual phys­i­cal push­ing. Phys­i­cal cause and elec­tronic ef­fect drift apart.

And sec­ondly, this non-con­form be­hav­iour needs to be ac­cepted and learned by the user. We have be­come so fa­mil­iar with this process that we drive our cars with our sub­con­science – they even be­come ex­ten­sions of our body as McLuhan states. (McLuhan, 2011) We do not need to see the ac­tual process any­more and so it comes that we have learned that links on web­sites tran­si­tion from skeuo­mor­phic but­ton metaphors to un­der­lined words.

Prob­a­bly the same path of ab­strac­tion could be dis­cussed for vi­su­al­i­sa­tion (stacked goods be­came his­tograms, land­scapes be­came their two-di­men­sional shapes on maps, cities be­came nodes and streets their con­nect­ing edges in net­works), but for this the­sis I want to take the cur­rent state of vi­su­al­i­sa­tions and dis­solve them. Their should be no bor­der be­tween im­ages and text, just like there is no bor­der be­tween links and texts.

Per­sonal ex­pe­ri­ence with this topic

Hav­ing worked as an in­for­ma­tion de­signer for vary­ing ap­pli­ca­tions like jour­nal­ism, pub­lic re­la­tions, travel guides, sci­en­tific re­search, and art, I en­coun­tered mainly two per­spec­tives on the com­bi­na­tion of text and in­for­ma­tion vi­su­al­i­sa­tion: One uses the text as an ac­ces­sory part, the other con­sid­ers vi­su­al­i­sa­tion as an ac­ces­sory part. The in­ter­ac­tion of these two is rarely equal. While vi­su­al­i­sa­tions often in­clude text as la­bels or in­tro­duc­tion, writ­ers are re­strained in the in­clu­sion of graph­ics.

The com­pany I work for had a big travel guide pub­lisher as client, who reis­sued all of its guides to in­clude in­fo­graph­ics. The way they were in­cluded were on three to five dou­ble pages as spe­cials in the book. We de­signed the graph­ics with­out any con­nec­tion to the sur­round­ing pages. An­other ex­am­ple is a news­pa­per that in­cludes a full page of vi­su­al­i­sa­tion in every issue. This is cer­tainly a big step for­ward, but still the rest of the pages are mostly not equipped with any vi­su­al­i­sa­tion.

In re­al­ity the co­op­er­a­tion of the mul­ti­ple nec­es­sary dis­ci­plines is un­for­tu­nately often ex­haust­ing, ex­pen­sive and time con­sum­ing, but I still be­lieve that each ex­pert in his field can learn from the other sub­jects in­volved.

Con­clu­sion

The com­bi­na­tion of text and image has been prac­tised for a long time. But still the idea to use sim­pli­fied im­ages for a more in­ter­wo­ven com­bi­na­tion has not been elab­o­rated in de­tail. Re­search from re­lated areas shows ev­i­dence of po­ten­tial in this ap­proach. The analy­sis of ty­po­graphic vari­ables has shown how they could be utilised for vi­su­al­i­sa­tion within lim­its.

With the em­bed­ding and sub­or­di­na­tion of the graph­ics, re­quire­ments and chal­lenges to­wards their ap­pear­ance arise. They need to be phys­i­cally small, re­duced in data points or/and in the num­ber of di­men­sions. This re­duc­tion will be re­ferred to as »micro« in the course of this the­sis. Hence »micro vi­su­al­i­sa­tions« are basic graph­i­cal mod­i­fi­ca­tions or ad­di­tions, that en­hance the com­pre­hen­sion of text.

Thus I pro­pose a wider un­der­stand­ing of vi­su­al­i­sa­tion. Every form of in­for­ma­tion­ally mo­ti­vated vi­sual al­ter­ation or ad­di­tion is to be treated as vi­su­al­i­sa­tion, even if it vi­su­alises a sin­gle data point.

State of the Art

An ex­am­i­na­tion of the cur­rent state of the art in vi­su­al­i­sa­tion and re­lated dis­ci­plines is pro­vided in this chap­ter, with a spe­cial focus on the con­cept of micro vi­su­al­i­sa­tion that is in­tro­duced pre­vi­ously.
In order to struc­ture the evolv­ing de­sign space, a novel tax­on­omy is de­vel­oped that dif­fer­en­ti­ates be­tween types of ap­pli­ca­tion and clus­ters them into four main cat­e­gories.
The tax­on­omy’s va­lid­ity is dis­cussed against two ex­ist­ing propo­si­tions and more pro­jects that sup­port the no­tion of micro vi­su­al­i­sa­tion.

Main ref­er­ence works

The term »micro vi­su­al­i­sa­tion« has not been used in this con­text be­fore. De­spite the fact that a num­ber of pro­jects and con­cepts have been de­vel­oped which now fit into this scope, they have not been com­piled under this term. The main in­spi­ra­tion and start­ing point for this con­sid­er­a­tions are Sparklines. They fol­low the exact idea this the­sis is now pur­su­ing: a small di­a­gram that is em­bed­ded right within the text cor­pus.

Sparkline The last point in the vi­su­al­i­sa­tion and the en­tity should have the same em­pha­sis

Sparklines have been elab­o­rated by Ed­ward Tufte in his book »The Vi­sual Dis­play of Quan­ti­ta­tive In­for­ma­tion«. Since Tufte is the in­ven­tor and strongest sup­porter of the data/ink-ra­tion, he seeks for a max­i­mum amount of data with the low­est pos­si­ble »chart-junk«. Every un­nec­es­sary ob­ject, ef­fect or point in a di­a­gram is seen as junk and hence should be elim­i­nated. Tufte preaches the high­est pos­si­ble data/ink-ra­tio: The great­est amount of data-points vi­su­alised through the low­est amount of (dig­i­tal or ana­log) »ink«. (Tufte, 2001)

Tufte vs. Holmes With his view he is usu­ally re­garded as an­tag­o­nists of Nigel Holmes, with his il­lus­tra­tive graph­ics. (Cairo, S. 63)

An­other as­pect of his re­search are high-res­o­lu­tion data graph­ics. He ex­plores ways of map­ping the high­est amount of data in the small­est pos­si­ble phys­i­cal space. In con­nec­tion to this ef­forts he in­tro­duces the con­cept of Sparklines as »data­words«. This idea brings two ad­van­tages:

  • Cre­at­ing con­text Sparklines cre­ate con­text on two dif­fer­ent lev­els. The idea of a great amount of data on a small sur­face is com­bined with the idea of di­rectly plac­ing that graphic into its rel­e­vant con­text. In con­trast to many graph­ics in which the reader has to switch back and forth, Sparklines are di­rectly at the rel­e­vant point. This brings the graphic into con­text. Tufte’s ex­am­ple of ap­pli­ca­tion is ob­vi­ous: A med­ical pa­tient’s cur­rent level of glu­cose gains much more value when it is viewed in its his­tor­i­cal con­text; in this case the 80 most re­cent read­ings of glu­cose. The med­ical staff can in­stantly eval­u­ate if the cur­rent fig­ure is an ex­cep­tion, a re­peat­ing pat­tern or an or­di­nary value. Again the con­text vis­i­ble through the data­word brings ben­e­fits. The text is put into con­text.
  • Amount of data The eye’s »re­solv­ing power […] en­ables it to dif­fer­en­ti­ate to 0.1 mm«. (Tufte, 2001) This al­lows for 100 val­ues per cm. Ta­bles would need a much greater space to con­vey the same amount. Sparklines can hence re­duce the nec­es­sary space dra­mat­i­cally.

A spe­cial form of Sparklines he de­scribes are dual state charts. They dis­play bi­nary data with up­ward and down­ward point­ing lines. Tufte’s ex­am­ples for ap­pli­ca­tion are re­sults of base­ball games. The chrono­log­i­cal se­quence of win and loss for one team can be eas­ily ex­tracted from the chart. As Tufte’s ex­am­ple shows, this tech­nique can vi­su­alise a com­plete sea­son within lit­tle phys­i­cal space. Gareth Watts de­vel­oped a great jQuery plu­gin for Sparklines along with a num­ber of other data­words. Be­sides line charts (reg­u­lar and com­po­si­tion), he sug­gests bar charts (reg­u­lar , stacked and tris­tate ), box plots , bul­let charts , and pie charts . He also de­vel­oped in­ter­ac­tive and an­i­mated charts.

Dual State Charts Only two states can be dis­played in chrono­log­i­cal se­quence.

The basis of these Sparklines is Tufte’s idea of being »ap­prox­i­mately right rather than ex­actly wrong« (Tukey, 1965) By omit­ting all axes and the key, only the process that leads to the cur­rent value is ex­tractable clearly. Exact val­ues can­not be de­ter­mined, es­pe­cially since the ex­treme val­ues are not known (in the stan­dard ex­e­cu­tion of Sparklines). As this might not be suit­able for every sce­nario, in the given case of glu­cose level it is ab­solutely valid and in most cases this ad­di­tional in­for­ma­tion is bet­ter than just a sin­gle num­ber which is nor­mally pre­sented. Fur­ther re­search on the con­cept of Sparklines sug­gested the mark­ings of high­est and low­est val­ues through coloured dots.

Al­ter­na­tive Ex­e­cu­tion Ex­em­plary al­ter­na­tives by Ed­ward Tufte

A sec­ond ref­er­ence pro­ject is from New York Times’ »The Rus­sia Left Be­hind« fea­ture. The in­ter­ac­tive ar­ti­cle de­scribes the 12-hour drive from St. Pe­ters­burg along sev­eral cities to Moscow. Next to the reg­u­lar ar­ti­cle text, a map of the path dri­ven is given by the au­thor. As the user scrolls down the ar­ti­cle, the path is filled up in or­ange and the cur­rent city is high­lighted. There are sev­eral rea­sons why this graphic is so ben­e­fi­cial for the ar­ti­cle.

  • Undis­turbed read­ing flow The graphic is not placed be­tween two para­graphs as it is nor­mally han­dled. The fixed po­si­tion on the left side al­lows for con­stant ac­ces­si­bil­ity yet with­out being af­fected by the scrolling and with­out dis­turb­ing the read­ing flow.
  • In­stant un­der­stand­ing The map is re­duced to the core el­e­ments rel­e­vant for this story: The cities, the path, and its phys­i­cal course. This en­ables even peo­ple who are not fa­mil­iar with maps to in­stantly un­der­stand its mes­sage. This is even more sup­ported by the link be­tween the scrolling and the path’s fill­ing. This makes the lookup of the cur­rently de­scribed city su­per­flu­ous. What many graph­ics lack is the chance to find the rel­e­vant point for a text pas­sage in the graphic.
  • Ad­di­tional data level The text tells a story a graphic could not con­vey. The graphic on the other hand con­veys in­for­ma­tion the story could hardly do: The dis­tances of the cities and its ge­o­graph­i­cal po­si­tion. One thing that could be crit­i­cised is that the map is not ori­ented, which could cre­ate a false im­pres­sion of the ge­o­graph­i­cal po­si­tions of the cities. But since they pro­vided an overview map with the path’s po­si­tion and sur­round­ings, this point can be ne­glected.
  • In­tu­itive nav­i­ga­tion The cities on the map are also click­able. This al­lows for in­tu­itive nav­i­ga­tion in Ben Shnei­der­man’s sense: Overview (map) first, de­tails-on-de­mand (ar­ti­cle). (Shnei­der­man, 1996) This way the text is not only con­nected to the graphic through scrolling but also the other way round. Be­sides its in­for­ma­tion it can serve as a sec­ond nav­i­ga­tion.
The Rus­sia Left Be­hind Screen­shot of the ar­ti­cle in desk­top view

Con­cep­tual de­lim­i­ta­tion

The con­cept of the com­bi­na­tion of text and graphic is not new. Even the com­bi­na­tion of text and vi­su­al­i­sa­tion has al­ready been elab­o­rated. In re­cent times the term dig­i­tal hu­man­i­ties emerged as the con­cept of dig­i­tally and com­puter sup­ported analy­sis of re­sources of »lit­er­a­ture, phi­los­o­phy, clas­sics, rhetoric, his­tory, and stud­ies of art, music, and de­sign«. (Bur­dick et al., 2012) Vi­su­al­i­sa­tion is a com­mon prac­tise for text analy­sis. Es­pe­cially com­plete text cor­pora, which are dif­fi­cult to over­look in its en­tire, can be ef­fec­tively vi­su­alised from a macro per­spec­tive. Pre­vi­ously hid­den pat­tern that span over sev­eral hun­dred pages or books can be un­cov­ered. A good ex­am­ple of the pos­si­ble scope is Google’s Ngram Viewer. It is orig­i­nally based on Google Books and the com­pany’s am­bi­tion to scan every printed book from 1800 AC until today in var­i­ous lan­guages. Through the pro­vided web in­ter­face users can search for ar­bi­trary terms, whose fre­quency of oc­cur­rence is then plot­ted over time.

Google Ngram Viewer The fre­quency of the terms free­dom and jus­tice is plot­ted over time.

My work has many sim­i­lar­i­ties with the one by dig­i­tal hu­man­ists. By vi­su­al­is­ing a sec­ond layer – often on a meta-level – we seek for a greater un­der­stand­ing of the ac­tual con­tent. In terms of dif­fer­ence, dig­i­tal hu­man­i­ties cov­ers a greater area of media and more often utilises the macro per­spec­tive (es­pe­cially in cor­pus lin­guis­tics), while my ap­proach keeps the focus on the ac­tual text and only en­hances this view with vi­su­al­i­sa­tions from both a micro and macro per­spec­tive. In this re­gard my ap­proach can be de­scribed as one part of the wide area of dig­i­tal hu­man­i­ties.

Graphic nov­els and comics are surely an ex­ten­sive com­bi­na­tion of pic­tures and text. Ono­matopoeia and speech bub­ble (which found their way into dig­i­tal mes­sag­ing ap­pli­ca­tion) are ex­am­ples of in­ven­tions this form of art cre­ated. In fact these tech­niques con­nect the text with the im­ages. Un­for­tu­nately, in most cases the text is rather un­car­ingly squeezed into the speech bub­bles. The text is ever so often treated as a re­quired ap­pendage.

One note­wor­thy book is the pre­vi­ously men­tioned »Un­der­stand­ing Comics« by Scott Mc­Cloud. This comic about comics dis­cusses these »spa­tialised se­quen­tial nar­ra­tives« in form of a non­fic­tion graphic novel it­self. With this open­minded de­f­i­n­i­tion of comic books, his find­ings can also act as a gen­eral guid­ance for vi­sual sto­ry­telling. But while most comics focus on the vi­sual sto­ry­telling through il­lus­tra­tions, I focus on tex­tual sto­ry­telling.

Even though the ap­pli­ca­tion of the find­ings in this the­sis can be ap­plied to other types of text, the form of pre­sen­ta­tion I elab­o­rate in this the­sis is arranged in para­graphs in­stead of loose arrange­ments. One re­quire­ment I thus im­pose for the text is the re­tain­ing of its read­abil­ity as text cor­pus.

Tax­on­omy

Since the term »Micro Vi­su­al­i­sa­tion« has not been used be­fore for such graph­ics, I see one con­tri­bu­tion of this the­sis to de­pict ex­ist­ing ex­am­ples and sort these into a tax­on­omy. While tax­onomies for vi­su­al­i­sa­tions al­ready exist, none of the pre­vi­ous fits my ap­proach. Other tax­onomies are for ex­am­ple these two:

  • Even though not ini­tially in­tended to be a tax­on­omy (and also mainly de­vel­oped for car­tog­ra­phy) Jacques Bertin vi­sual vari­ables help to cat­e­gorise vi­su­al­i­sa­tions. His sys­tem de­scribes the deep­est level of vi­su­al­i­sa­tion: the en­cod­ing of the data points.
  • An­other ap­proach to cat­e­gorise graph­ics is the dif­fer­en­ti­a­tion by type. In »A Tour through the vi­su­al­i­sa­tion Zoo« Jef­frey Heer, Michael Bo­stock, and Vadim Ogievet­sky (2010) sug­gest five cat­e­gories with sev­eral sub­cat­e­gories each. The types they pro­pose are time-se­ries data, sta­tis­ti­cal dis­tri­b­u­tions, maps, hi­er­ar­chies, and net­works. In most cases the given data de­ter­mines which of these five types can be used to vi­su­alise the struc­ture. In the sub­cat­e­gories, some­times the types are also de­ter­mined by the data set (choro­pleth or grad­u­ated sym­bol map) and in other cases rather op­tions to choose from (stacked graphs or small mul­ti­ples) are given. Within these types the vi­sual vari­ables are then ap­plied.

In con­trast, the tax­on­omy I pro­pose makes a fur­ther step by ob­serv­ing the ap­pli­ca­tion of the vi­su­al­i­sa­tion. These cat­e­gories refer to dif­fer­ent lev­els of mod­i­fi­ca­tion and in­ter­ven­tion of the orig­i­nal text, the source of data, and the ac­tual pur­pose of the vi­su­al­i­sa­tion. Also in­stead of group­ing types into cat­e­gories the dif­fer­en­ti­a­tion might not al­ways be ex­plicit for all graph­ics, but should rather con­sti­tute the ex­tents of mul­ti­ple as­pects in which the vi­su­al­i­sa­tion can be lo­cated.

Tax­on­omy Po­si­tion­ing The tax­on­omy I pro­pose acts on the high­est level of three.
Level of change: Mod­i­fi­ca­tion or ad­di­tion

The first dif­fer­en­ti­a­tion de­scribes the level of change made to the orig­i­nal pure text by the ad­di­tional data layer. The first and re­strained ap­proach is the mod­i­fi­ca­tion through purely ty­po­graph­i­cal in­stru­ments. The sec­ond di­rec­tion is the en­rich­ment of the orig­i­nal text with ad­di­tional pure graph­i­cal el­e­ments. While ad­di­tional el­e­ments are sep­a­rated from the en­tity, mod­i­fi­ca­tions af­fect the en­tity it­self. Vi­su­al­i­sa­tion and en­tity are the same ob­ject.

Po­si­tion­ing/scope: In­te­grated or ad­ja­cent

I use the po­si­tion­ing and scope to fur­ther de­scribe the level of change. While one is smaller and di­rectly in­te­grated in­side the text, the other is more spa­cious and hap­pens ad­ja­cently – as for ex­am­ple in the mar­gin.

The dif­fer­en­ti­a­tion also af­fects the re­la­tion to the cor­re­spond­ing en­tity. While in­te­grated vi­su­al­i­sa­tion (mod­i­fi­ca­tion and ad­di­tional) refers only to a sin­gle en­tity (and are in di­rect con­tact to this one), ad­ja­cent ad­di­tional graph­ics can also re­late to mul­ti­ple en­ti­ties they vi­su­alise. Ad­ja­cent mod­i­fi­ca­tions still refer to only one en­tity.

Re­la­tion be­tween vi­su­al­i­sa­tions and en­ti­ties In­te­grated vi­su­al­i­sa­tions have a dif­fer­ent re­la­tion than ad­ja­cent ones.

This cat­e­gori­sa­tion needs to be con­sid­ered sep­a­rately from the pre­vi­ously de­scribed dis­tinc­tion of mod­i­fi­ca­tion and ad­di­tion in more de­tail.

Mod­i­fi­ca­tion of ex­ist­ing el­e­ments can be seen from the ty­po­graphic per­spec­tive as the dif­fer­en­ti­a­tion be­tween micro and macro ty­pog­ra­phy. Micro is re­fer­ring to the ty­po­graphic changes on a de­tailed level (sin­gle char­ac­ters or words), while macro mod­i­fi­ca­tions de­scribes rather ex­ten­sive changes that in­volve the lay­out (com­plete sen­tences or para­graphs). The terms »in­te­grated« and »ad­ja­cent« are ad­mit­tedly not per­fect in the case of mod­i­fi­ca­tion of el­e­ments, but should rather be seen as de­tailed changes and spa­cious trans­for­ma­tions that hap­pen ei­ther to sin­gle el­e­ments and/or to whole text pas­sages.

Ad­di­tional el­e­ments can also be placed in­side the text lines or out­side when they are small in size. When el­e­ments are placed sep­a­rately from the text-block, they are con­sid­ered ad­ja­cent. Spa­cious graph­i­cal el­e­ments are also treated as sep­a­rated even if their cen­tre point is in­side the text block, but their total scope ex­cels the text-block. When they are placed in the phys­i­cal con­text of the re­lated en­tity, they are treated as in­te­grated.

In their paper, Gof­fin et al. (2014) dis­tinct three dif­fer­ent con­texts for the place­ment of data­words:

  • Strong con­text, where the »bound­ing boxes« of the vi­su­al­i­sa­tion and its ref­er­ence word »touch«.
  • Weaker con­text, when the place­ment of the vi­su­al­iza­tion is in­flu­enced by the word’s po­si­tion.
  • Out-of-con­text de­scribes the po­si­tion­ing un­re­lated to the en­tity’s po­si­tion some­where in the mar­gin.

Al­though Gof­fin’s tax­on­omy is in­ter­est­ing, there is no need for such a de­tailed dif­fer­en­ti­a­tion. Also, since no vi­su­al­i­sa­tion should be place com­pletely »out-of-con­text« and for a gen­eral con­cept, the dif­fer­ence be­tween stronger and weaker con­text is not help­ful in re­gard to its de­sign, two types should be suf­fi­cient. For their tax­on­omy they also only con­sid­ered ad­di­tional el­e­ments. Mod­i­fi­ca­tions on the word it­self would only be in a strong con­text per de­f­i­n­i­tion.

This dif­fer­en­ti­a­tion is also rel­e­vant for the goal and mes­sage of the whole doc­u­ment. If the re­la­tion be­tween the en­tity and the cor­re­spond­ing vi­su­al­i­sa­tion is more im­por­tant it should be in­te­grated. If the com­par­i­son be­tween the dif­fer­ent data points is more rel­e­vant they should be placed to­gether ad­ja­cent, be­cause their dis­tri­b­u­tion across mul­ti­ple lines would im­pair the com­pa­ra­bil­ity. Hence, in­te­grated vi­su­al­i­sa­tions are use­ful for prox­im­ity to have the graphic in con­text of the word, ad­ja­cent are good for com­par­i­son to have the data points in con­text of each other.

Data source: In­nate, in­trin­sic or ex­trin­sic

The next dif­fer­en­ti­a­tion of vi­su­al­i­sa­tions is ex­am­in­ing the source of in­for­ma­tion, which is mapped upon the given text. Three de­riva­tions can be dis­tin­guished.

  • In the first case the data is in­nate in the text. This can be fur­ther ex­panded to dis­tinc­tions be­tween the word’s or sen­tence’s type (e.g. noun, head­line) or the con­tent the words un­fold (e.g. city name).
  • In the sec­ond case the sta­tus of the text acts as a source. It is hence in­trin­sic within the text but emerges through fac­tors like changes, au­thor­ship or – and this might be seen as a nar­row ridge be­tween the third case – by fur­ther ex­ter­nal in­flu­ences.
  • In the third case, a com­plete new data layer is added from an ex­ter­nal source. Sin­gle text el­e­ments act as sym­bolic ref­er­ence to ob­jects which are equipped by data-val­ues through fur­ther links.

This sec­ond, vi­su­al­i­sa­tion layer can have dif­fer­ent func­tions de­pend­ing on this re­la­tion. By vi­su­al­is­ing in­nate or in­trin­sic data it can re­veal, con­tex­tu­alise, and am­plify the struc­ture and con­tent of the text. It acts as meta-layer. With ex­trin­sic data it adds a de­tail-lay­ery that adds deeper or new in­for­ma­tion to the text.

Point of rel­e­vance

The next clas­si­fi­ca­tion de­scribes the point dur­ing the read­ing process at which the vi­su­al­i­sa­tion be­comes rel­e­vant for the reader and hence de­ter­mines its pur­pose. For this, I dif­fer­en­ti­ate be­tween three stages: Be­fore, dur­ing and after the read­ing process. Ob­vi­ously, these sit­u­a­tions are not al­ways in lin­ear order and one vi­su­al­i­sa­tion can be rel­e­vant at dif­fer­ent stages.

Pur­pose

In these points of rel­e­vance, the vi­su­al­i­sa­tion has dif­fer­ent pur­poses. Prior to the read­ing process, the reader needs an overview and ori­en­ta­tion, nav­i­ga­tion through the doc­u­ment and a sense of the mean­ing of the text. Dur­ing the read­ing process, vi­su­al­i­sa­tions can en­hance the mem­o­ra­bil­ity, serve for deep­en­ing and ex­ten­sion. A sum­mary and aide mem­oire is rel­e­vant after the read­ing process.

In­ter­ac­tiv­ity

While some tech­niques can be ap­plied to tra­di­tional books, oth­ers de­mand a dig­i­tal en­vi­ron­ment. Un­chang­ing and sta­tic graph­ics can be printed; dy­namic or in­ter­ac­tive el­e­ments are gen­er­ated every time the text is opened or the user in­ter­acts with the text.

Con­clu­sion

The most rel­e­vant ap­proach of this tax­on­omy is the dif­fer­en­ti­a­tion be­tween the type and the po­si­tion­ing/scope of change. These four char­ac­ter­is­tic di­vide the field into four seg­ments the de­signer can work with. It can act as a start­ing point when one char­ac­ter­is­tic is fixed or al­ready taken by other el­e­ments. In a fixed lay­out for ex­am­ple ad­ja­cent el­e­ments would not be pos­si­ble, hence in­te­grated vi­su­al­i­sa­tions would allow a sec­ond level. If one qual­ity is more im­por­tant then the ap­pro­pri­ate type can also be used. The de­signer can de­cide be­tween com­par­i­son and prox­im­ity.

Ex­ist­ing works

The fol­low­ing chap­ter high­lights some of the ex­ist­ing works cat­e­gorised in the pre­sented tax­on­omy. For this, I dis­tin­guish be­tween pro­jects that mod­ify the ex­ist­ing text pre­sen­ta­tion and the ones that add ad­di­tional el­e­ments to the text. These two groups are fur­ther sub­di­vided in in­te­grated and ad­ja­cent place­ments/scopes of the vi­su­al­i­sa­tion. This cre­ates four groups in which all pro­jects can be sorted. The other cri­te­ria I sug­gested for the tax­on­omy are help­ful for fur­ther de­scrib­ing a graphic but do not af­fect the ini­tial de­sign process dras­ti­cally and are hence dis­cussed for each pro­ject if they are of any sig­nif­i­cance. Often the four cat­e­gories al­ready de­fine the point of rel­e­vance for the graphic: While the reader often stum­bles over in­line el­e­ments only while read­ing, ex­ter­nal el­e­ments are no­ticed be­fore and hence can act as an overview.

Star Struc­ture The struc­ture for the state of the art re­port is based upon the tax­on­omy
Mod­i­fi­ca­tion
In­te­grated

The choice of type­face is often re­garded as the start­ing point of set­ting a text. It is com­mon prac­tise to use dif­fer­ent type­faces for the body and head­line texts. These de­ci­sions are often lead by leg­i­bil­ity and hence serif fonts are com­monly used for body text and sans serif for head­lines. (Santa Maria, 2014) But some­times the de­ci­sions are also dri­ven by the con­tent: Quotes are com­monly set in oblique or script fonts and source code is set in mono­spaced fonts. But Mike Hof­maier for ex­am­ple uses a mono­spaced font in his book »Ver­fas­sung Ver­ste­hen« (2013) to dis­tin­guish com­ments from the text. The sym­bolic con­ven­tion of the type­writer font for notes and an an­ti­qua font for »fin­ished« text can also be seen in iA Writ­ers’ dif­fer­ent modes, that dif­fer­en­ti­ate these writ­ing-stages.

iA Writer In­ter­face The font changes ac­cord­ingly to the modes: notes, write, edit, and read

Vary­ing the type­face for el­e­ments of the same »cat­e­gory« within the doc­u­ment is rather in­con­ve­nient and re­garded as vex­ing for the reader. Type­designer Luc(as) de Groot used this for his re­design of the Ger­man po­lit­i­cal weekly news­pa­per »Jun­gle World«. He de­signed every col­umn to vary be­tween Plan­tin and Min­ion. He used this »sub­ver­sive so­lu­tion, be­fit­ting the paper’s own ed­i­to­r­ial at­ti­tude.« (de Groot)

Be­sides this rather un­con­ven­tional method, the mix­ing of type­faces is re­garded as one the skills of ty­pog­ra­phers. In gen­eral, the mixed type­faces must not be too sim­i­lar in char­ac­ter, but of a sim­i­lar style on a de­tailed level. (Santa Maria, 2014)

Using dif­fer­ent styles is also com­mon prac­tise: italic or bold fonts to em­pha­sise, up­per­case for peo­ple scream­ing in di­rect speech. Ken Kesey’s novel »Some­times a Great Na­tion« from 1964 also »shifts re­peat­edly in mid-sen­tence be­tween roman and italic to dis­tin­guish what char­ac­ters say to each other from what they say in si­lence to them­selves«. (Bringhurst, 2012)

Colours are also used to em­pha­sise con­tent. Tra­di­tional ty­pog­ra­phers tend to­wards a sin­gle colour – com­monly red – while cheaper prints and dig­i­tal media allow for sev­eral colours. Again colour is com­monly used for dis­tinc­tion of hi­er­ar­chy in the text and is rather sel­dom chang­ing within one el­e­ment. Be­sides links on web­sites the con­tent is rarely re­flected in the colour.

One pos­i­tive ex­am­ple is »Si­lenc« by Manas Karam­belkar, Momo Miyazaki, and Ken­neth Robert­sen from the Copen­hagen In­sti­tute of In­ter­ac­tion De­sign. They vi­su­alised how much of the lan­guages Dan­ish, Eng­lish and French is silent. »In one form of a book, silent let­ters are marked up in red yet re­main in their orig­i­nal po­si­tion. In an­other it­er­a­tion, silent let­ters are sep­a­rated from the pro­nounced text and ex­hib­ited on their own pages in the back of the book, the preva­lence of silent let­ters is clearly ev­i­dent.« (Miyazaki, 2013) The gestalt prin­ci­ple of prox­im­ity al­lows the reader to recog­nise the words as whole while read­ing even when in dif­fer­ent colours and still being able to see the em­pha­sised let­ters to »pop out«.

Pro­ject Si­lenc The silent let­ters in the text are coloured red

In com­puter sci­ence the colour­ing of source code is a com­mon prac­tise. Since »soft­ware code text is in­tended for the peo­ple writ­ing it much more than for the com­put­ers run­ning it« (Gen­del, 2010), many con­ven­tions evolved over time to make the code more read­able. Vari­able names, in­den­tion and white­spaces are vi­su­al­i­sa­tions that are com­pletely ig­nored by the com­piler, but help the pro­gram­mer to clar­ify a code.

An­other con­ven­tion is the syn­tax high­light­ing. It colours the words ac­cord­ing to their cat­e­gory: key­words (if, func­tion, for, …), sym­bols (=, +, ;, ?, …), iden­ti­fiers (vari­ables and func­tion names), com­ments, string lit­er­als, and num­bers. (Wirth, 1976) While nowa­days mainly colours are being used for this dis­tinc­tion, ini­tially dif­fer­ent font styles were used. In Niklaus Wirth’s book from 1976 »Al­go­rithms + Data Struc­tures = Pro­grams«, the au­thor prints out Pas­cal code with bold key­words, and iden­ti­fies and string lit­er­als in italic. (Wirth, 1976) Be­sides this, it is note­wor­thy that he uses the same serif font used for the text and the source code. Today, mono­spaced fonts (rarely with ital­ics and some­times still with bold styles for key­words) are being used in all major code ed­i­tors.

Source Code 1976 Pas­cal source code with syn­tax high­light­ing by Niklaus Wirth
Source Code Today Pas­cal source code with typ­i­cal syn­tax high­light­ing
Se­man­tic High­t­light­ing In­stead of key­words and sym­bols, iden­ti­fiers are coloured

Be­sides syn­tax high­light­ing, the idea of se­man­tic high­light­ing emerged in re­cent time. In 2009 the cre­ator of the code ed­i­tor KDe­velop for Linux’ KDE en­vi­ron­ment im­ple­mented »Local Vari­able Col­oriza­tion«. »That col­oriza­tion as­signs a semi-unique color to each vari­able in a local con­text. This al­lows much eas­ier dis­tin­guish­ing those vari­ables, largely with­out read­ing their full name at all.« (Zwa­bel, 2009) In 2014, Evan Brooks sug­gested to not colour key­words and sym­bols at all, but in­stead use the full colour scope to in­di­vid­u­ally colour each iden­ti­fier. This al­lows quickly find­ing the usage for each one over the course of the pro­gram. This would help the reader to quicker iden­tify the struc­ture and the flow of the ap­pli­ca­tion in­stead of just »high­light[ing] the ob­vi­ous«. (Brooks, 2014) But be­sides the abil­ity to read the code af­ter­wards, both high­light­ing tech­niques also help the pro­gram­mer dur­ing the process of writ­ing: As el­e­ments are coloured im­me­di­ately after being typed, mis­spellings be­come in­stantly ap­par­ent.

One com­mon ap­proach in text cor­pus analy­sis is its change over time. Mod­i­fi­ca­tions, sup­ple­men­ta­tions or re­moval of text pas­sages are some­times done by the a sin­gle au­thor or by mul­ti­ple peo­ple for ex­am­ple on Wikipedia ar­ti­cles. Ben Fry’s vi­su­al­i­sa­tion »On The Ori­gin of Species: The Preser­va­tion of Favoured Traces« shows the changes over 13 years in six edi­tions Charles Dar­win made in his mas­ter­piece. Text from each edi­tion is dif­fer­ently coloured, re­veal­ing the amount and po­si­tions of every change. One of the­ses changes is for ex­am­ple »‘sur­vival of the fittest’ – usu­ally con­sid­ered cen­tral to the the­ory and often at­trib­uted to Dar­win – in­stead came from British philoso­pher Her­bert Spencer, and didn’t ap­pear until the fifth edi­tion of the text.« (Fry, 2009) While the vi­su­al­i­sa­tion clearly shows the over­all struc­ture of changes (also through the time-lapse video), the text is only vis­i­ble through mouseover. The pro­ject is still in­cluded in this micro vi­su­al­i­sa­tion dis­cus­sion, since a smaller amount of text could also be shown in clear form.

Change over time The vi­su­al­i­sa­tion shows the ap­pear­ance of each para­graph with text colours

One thing that is also in­vis­i­ble in Fry’s pro­ject is the re­moval of words. This is a com­mon prob­lem in vi­su­al­i­sa­tion: to show what is (no longer) there. This is es­pe­cially rel­e­vant for changes on Wikipedia pages as dis­cus­sions often re­volve around what things are in­cluded and how these are writ­ten in the en­cy­clopae­dia. Be­hind the front page of each ar­ti­cle there is a dis­cus­sion and an ar­ti­cle his­tory page, where peo­ple de­bate over its con­tent. The con­flicts are often dri­ven by larger so­ci­etal de­bates. The pro­ject Con­tro­pe­dia makes these de­bates vis­i­ble. It aims to »pro­vide a bet­ter un­der­stand­ing of so­cio-tech­ni­cal phe­nom­ena that take place on the In­ter­net and to equip cit­i­zens with tools to fully de­ploy the com­plex­ity of con­tro­ver­sies.« (Con­tro­pe­dia, 2015)

They pro­vide three dif­fer­ent views: the layer, dash­board and net­work view. In the dash­board view »text is high­lighted in green when it was in­serted and in red if it was deleted in a spe­cific edit.« Be­sides the use of red and green colour for added or re­moved el­e­ments, the text has the ben­e­fit to be able to be struck through. This is prob­a­bly de­vel­oped from ana­log paper and has also been com­mon prac­tise on type­writes. Other pos­si­bil­i­ties are trans­parency or the use of dot­ted in­stead of solid lines.

Un­for­tu­nately, they use red (in com­bi­na­tion with blue) again for an­other pur­pose in the layer view. This time not the text but the back­ground be­hind the text is coloured. »The hot­ter the color, the more con­tro­ver­sial the cor­re­spond­ing el­e­ments.« (Borra et. al, 2015)



Layer and dash­board view Con­tro­pe­dia uses text-colour to in­di­cate »hot­ness« of words.

If a type­face has ser­val weights, these dif­fer­ent fonts can be used con­vey mean­ing. Again, this is com­mon prac­tise for dif­fer­ent hi­er­ar­chies or to em­pha­sise sin­gle pas­sages in­side the text. This method is rather un­pop­u­lar among ty­pog­ra­phers as it in­ter­rupts the read­ing flow by dis­tort­ing the grey value of the text. Small cap­i­tals or italic fonts are often pre­ferred. One pro­ject that takes ad­van­tage of this grey value for vi­su­al­i­sa­tion is Fat­Fonts. While many artis­tic ap­proaches use let­ter’s dark­ness for vi­sual po­etry or ASCII art (Na­centa et al., 2012 I), no ac­tual mean­ing is con­veyed in those pro­jects. Fat­Font is de­signed »so that the amount of dark pix­els in a nu­meral char­ac­ter is pro­por­tional to the num­ber it rep­re­sents. For ex­am­ple, ‘2’ has twice the ink than ‘1’«. (Na­centa et al., 2012) They also pro­vide multi-level dig­its by ar­rang­ing the num­bers in­side each other. Ap­pli­ca­tions they sug­gest are his­tograms, in­line num­bers, and maps. The most use­ful util­i­sa­tion are maps, where for ex­am­ple the el­e­va­tion level is ac­tu­ally writ­ten on the cor­re­spond­ing po­si­tion on the map. From far away the reader can see the pat­terns on the map; from a closer dis­tance the reader can ac­tu­ally read the val­ues. This is ad­van­ta­geous over tra­di­tional colour en­cod­ing usu­ally used for the el­e­va­tion level. Other rep­re­sen­ta­tions need a leg­end placed next to the map, while here the in­for­ma­tion is dou­ble en­coded right on the cor­re­spond­ing po­si­tion. One dis­ad­van­tage is the un­aes­thetic ap­pear­ance of the num­bers, which could be the rea­son why Fat­Fonts are rather sel­domly used.

Fat Fonts The only ac­tual usage I could find was in a map in the Ger­man news­pa­per Die ZEIT Mag­a­zin about the av­er­age age in dif­fer­ent re­gions in Ger­many.
Ad­ja­cent

As men­tioned above, the dif­fer­en­ti­a­tion be­tween in­te­grated and sep­a­rated mod­i­fi­ca­tions is not per­fectly clear. For this model I con­sider mar­gins and ori­en­ta­tions as changes on the lay­out and there­fore as sep­a­rated. Through the text’s ap­pear­ance (es­pe­cially from a dis­tance), the gen­eral lay­out rep­re­sents the con­tent. In con­trast to sim­ple or­di­nal dis­tinc­tion of text el­e­ments which rep­re­sent its sta­tus (head­lines, body text, foot­note, …) also quan­ti­tive and con­tent re­lated in­for­ma­tion is shown. In order to em­pha­sise this ap­proach I in­tro­duce the term »vi­su­al­is­ing lay­out« for this ap­pli­ca­tion.

One ex­am­ple that af­fects the com­plete lay­out is the Vilna edi­tion of the Baby­lon­ian Tal­mud. The spi­rally arranged text »serves as a spe­cific site of me­di­a­tion«. (Drucker, 2014) The for­mat rep­re­sents the text mes­sage and its pur­pose through the way it is read.

Lay­out of the Tal­mud In the way the Tal­mud is lay­outed it serves the me­di­a­tion.

The mar­gins on a page are also one of the ty­pog­ra­pher’s es­sen­tial tool for read­abil­ity, white­space and hi­er­ar­chy. The in­den­tion from the left bor­der is prob­a­bly the most com­mon tech­nique to vi­su­alise dif­fer­ent lev­els of hi­er­ar­chy in text. Sci­en­tific books use it to order chap­ters, file browsers use it for the file struc­ture and pro­gram­mers use it to struc­ture source code. Every sub­or­di­nated level is spaced more afar from the left than its par­ent level. This tree struc­ture is vi­su­alised in Jason Davies’ »Word Trees«. The vi­su­al­i­sa­tion tool al­lows the user to se­lect a word from a text to be the root el­e­ment. The text is then analysed on the sen­tences con­tain­ing this word and are then sub­or­di­nated.

Word Trees Dis­crete in­den­tion is used to cre­ate a tree-struc­ture of texts.

While an ex­treme util­i­sa­tion can lead to a de­struc­tion of an ob­vi­ous read­ing order for the reader, changes on a sin­gle axis can in­deed be used for the vi­su­al­i­sa­tion of the text con­tent. One clas­sic ex­am­ple is the book Far Tor­tuga by Peter Matthiessen, which was de­signed by Ken­neth Miyamoto. He »uses two sizes of type, three dif­fer­ent mar­gins, free-float­ing blocks para­graphs and other ty­po­graphic de­vices to sep­a­rate thought, speech and ac­tion. (Bringhurst, 2012)

While all the above ex­am­ples had dis­crete level of in­den­tion, the re­cent ex­am­ple »Stack­ing Up the Ad­min­is­tra­tion’s Drone Claims« by Lena Groeger and Cora Cur­rier (2012) for ProP­ub­lica has con­tin­u­ous po­si­tions. Their vi­su­al­i­sa­tion shows four years of state­ments about drone strikes. Each state­ment is in­tended ac­cord­ing to its date. Ad­di­tion­ally, state­ments that refuse to even ac­knowl­edge the pro­gram are coloured in red. With this in­den­tion the chrono­log­i­cal se­quence be­comes in­stantly ob­vi­ous, even though there is no sig­nif­i­cant pat­tern that emerges through this. But what is good about this vi­su­al­i­sa­tion is that it does not in­ter­rupt the read­ing flow and still gives added value to the text by means of sim­ple tools.

Drone Claims Con­tin­u­ous in­den­tion is used to cre­ate a time­line of events.

Owen Her­t­erich’s pro­ject »To See & Hear« uses both mar­gin and ori­en­ta­tion to cre­ate fin­ger­prints of books. »Each line in these vi­su­al­i­sa­tions is one line of di­a­logue in the book. These are mapped around an inner cir­cle, the ra­dius of which is de­ter­mined by the length of the book. Ac­cord­ingly, the more dense the vi­sual rep­re­sen­ta­tion, the higher the pro­por­tion of di­a­logue that ex­ists in that spe­cific work of lit­er­a­ture.« (Her­t­erich, 2013) Un­for­tu­nately, the change of ori­en­ta­tion make the di­a­logues often un­read­able and thus cre­at­ing rather art­works than vi­su­al­i­sa­tions. In con­trast, the pro­ject »Un­der­stand­ing Shake­speare« by Stephan Thiel (2010) uses smooth and slight change of ori­en­ta­tion to cre­ate »bub­bles« in­side the text. The graph­ics »Shake­speare Googled« use this to »iden­tify fa­mous quotes from a play by com­par­ing the num­ber of re­sults their text re­turns on Google. It then shows the im­pact each quote has on the en­tire text by a sim­u­lated force that is ap­plied to each char­ac­ter around each quote. This force is in a di­rect re­la­tion to the num­ber of Google re­sults for the text.« (Thiel, 2010)

Un­der­stand­ing Shake­speare The text is slightly ro­tated and changed in weight and line-height.

Nancy Duarte’s book »res­onate« ex­plains tech­niques to struc­ture and for­mat in­for­ma­tion for pre­sen­ta­tions. While most of her ex­pla­na­tions are sur­pris­ingly un-graphic, her analy­sis of ex­ist­ing sem­i­nal speeches are ex­plained vi­su­ally. She groups pas­sages of the re­cited text into two states: »What is« and »What could be«. She thereby em­pha­sises the men­tal and phys­i­cal gap be­tween these two sit­u­a­tions. (Duarte, 2010) Ty­po­graph­i­cally she does this with two columns, the first aligned right and sec­ond aligned left. As the speaker switches back and forth space is kept blank to main­tain the se­quence. Two fa­mous ex­am­ples she pre­sents are Mar­tin Luther King’s »I Have A Dream« and Steve Job’s iPhone pre­sen­ta­tion. She im­pres­sively il­lus­trates how both speak­ers start with the cur­rent sit­u­a­tion (»I have a dream …«) and a pos­si­ble fu­ture (»… that my four lit­tle chil­dren will one day live in a na­tion where they will not be judged by the color of their skin, but by the con­tent of their char­ac­ter.«). Steve Job’s talk is sim­i­lar: To­days sit­u­a­tion and the pre­sen­ta­tion of »life chang­ing« pos­si­bil­i­ties given by the iPhone. (»This is a day I’ve been look­ing for­ward to for two and a half years. Every once in a while, a rev­o­lu­tion­ary prod­uct comes along that changes every­thing.«) What Duarte adds is the mark­ing of dif­fer­ent ref­er­ences, mo­ments of ap­plause dur­ing the speech, prod­uct demo time, and guest speaker time.

Steve Jobs’ iPhone Pre­sen­ta­tion The text (not vis­i­ble here) is grouped in »what is« and »what can be«.
Con­clu­sion mod­i­fy­ing vi­su­al­i­sa­tions

Most of the in­line mod­i­fi­ca­tions vi­su­alise in­for­ma­tion in a rather »clas­si­cal ty­po­graphic« way. Nearly all data was cat­e­gor­i­cal or qual­i­ta­tive. Ac­tual nu­meric val­ues were rarely vi­su­alised. Viewed from the data-source per­spec­tive it can be de­scribed as mostly in­nate and in­trin­sic data. The rea­son for that might be the dif­fi­cult en- and de­cod­ing of qual­i­ta­tive data from the vi­sual vari­ables pro­vided with ty­pog­ra­phy. For ex­am­ple the y-po­si­tion within the line can vary only by a few pixel and is there­fore harder to de­ter­mine. It is also dif­fi­cult to com­pare mul­ti­ple val­ues from var­i­ous lines. This is also one char­ac­ter­is­tic I will dis­cuss later. The only ex­cep­tion to this »cat­e­gor­i­cal« vi­su­al­i­sa­tion is Fat­Font. But here the ac­tual value is also »writ­ten« and not just vi­su­alised. Sum­marised, in cur­rent vi­su­al­i­sa­tions in­line mod­i­fi­ca­tions are mostly used to make qual­i­ta­tive state­ments like »this is im­por­tant«, »this is new«, or »this is type a, this is type b«. Fur­ther em­pir­i­cal tests should test how well peo­ple are able to ex­tract quan­ti­tive data.

Ex­ter­nal mod­i­fi­ca­tions on the other hand some­times use quan­ti­ta­tive val­ues. ProP­ub­lica’s »Stack­ing Up the Ad­min­is­tra­tion’s Drone Claims« for ex­am­ple uses a fluid x-po­si­tion. Even though the in­for­ma­tion is not es­sen­tial, it adds a sec­ond layer of in­for­ma­tion to the data. Here an ex­trin­sic data source is used. The use of com­plete lay­out shifts seems to be more ex­per­i­men­tal and art dri­ven. Sum­marised, all three types of data sources and qual­i­ta­tive as well as quan­ti­tive data is vi­su­alised in cur­rent graph­ics.

In gen­eral, the mod­i­fi­ca­tion of text el­e­ments is mainly used with a sup­port­ive in­ten­tion. It is utilised to re­in­force the text’s con­tent and not to over­lay an ad­di­tional layer. Fur­ther ap­pli­ca­tions should test how this can change and how some ap­proaches can lose their artis­tic usage and find their way into a gen­eral and sci­en­tific con­text.

Star Struc­ture The struc­ture for the state of the art re­port is based upon the tax­on­omy
Ad­di­tion

Ad­di­tional vi­su­al­i­sa­tions di­rectly fol­low the ap­proach of this the­sis to mix vi­su­al­i­sa­tions with the text. Both in­spi­ra­tions for this the­sis (Sparklines and »The Rus­sia Left Be­hind«) use ad­di­tional el­e­ments to en­hance the text. The idea is to use a sec­ond, vi­sual layer that dis­plays the dif­fer­ent data types de­scribed in the tax­on­omy.

In­te­grated

One group of ad­di­tional in­line el­e­ments are graph­i­cal sup­ple­ments that are placed out­side of the words’ or sen­tences’ scope for ex­am­ple di­rectly fol­low­ing it as Tufte’s Sparklines do. The sec­ond group, which I will cover first as it is closer to mod­i­fi­ca­tion, is right within (be­hind/be­fore) the scope of one or mul­ti­ple words. Whether these ad­di­tions are an ad­di­tional el­e­ment or an in­te­gral part of the text is cer­tainly de­bat­able, but as it adds more el­e­ments graph­i­cally it should not be con­sid­ered as a mod­i­fi­ca­tion.

One pro­ject that was men­tioned above makes use of the back­ground colour. The layer view of Con­tro­pe­dia il­lus­trates the »hot­ness« of a term through blue and red back­ground-colours. »The hot­ter the color, the more con­tro­ver­sial the cor­re­spond­ing el­e­ments.« (Borra et al., 2015) A pro­ject that uses the same tech­nique is Serendip by Alexan­der et al. (2014). In the TextViewer, words are tagged by their im­por­tance. They found out that »tag­ging all words equally can some­times be neg­a­tively in­for­ma­tive« (Alexan­der et al., 2014) and hence they use im­por­tance bin­ning that ap­plies the colour­ing with thresh­olds. For the dif­fer­ent rank­ings they use a de­creas­ing trans­parency for more im­por­tant words. These »ramped tags« make the text eas­ier to read.

Ramped tags The colours are ap­plied with thresh­olds

The idea of such a heat map is cer­tainly one of the vi­su­al­i­sa­tions that re­quire com­put­erised cre­ation and a dig­i­tal out­put. Web­site analy­sis ser­vices for ex­am­ple offer heat maps that are over­layed over screen­shots of web­pages. Areas that have been clicked more often are coloured red, oth­ers are blue or trans­par­ent. Eye-Track­ing soft­ware uses the same vi­su­al­i­sa­tion but dis­plays the eye’s fix­a­tions on the screen.

Typo-Graph Each char­ac­ter rep­re­sents one hour. The weight of the let­ter rep­re­sents the air pol­lu­tion. The back­ground-color rep­re­sents the tem­pear­ture.

The pro­ject »Typo-graph« by Michał Pawłowski and Jakub Wolak com­bines heat maps with the same idea as Fat­Fonts. But in­stead of de­sign­ing a new type­face, they use the grey value of dif­fer­ent fonts from ex­ist­ing type­faces. Each text para­graph forms one month of the year: Each line is one day, the width of the para­graph is equally di­vided for every hour. The text de­scribes the data they gath­ered about tem­per­a­ture in con­nec­tion to pol­lu­tion. Ad­di­tion­ally, each let­ter is in the font-weight cor­re­spond­ing to the air pol­lu­tion. The back­ground colour of each let­ter is rep­re­sent­ing the tem­per­a­ture re­spec­tively. This en­ables the reader to com­pare the grey­ness to the colour from far away and read in de­tail about the data, when closer to the poster.

Since the change of the back­ground colour is al­ready a fa­mil­iar tool for ana­log media with text marker, the metaphor is often used for dig­i­tal media. The trans­la­tion tool Linguee uses a yel­low back­ground colour to mark cor­re­spond­ing words within their sen­tence. Linguee fo­cuses on con­text-re­lated trans­la­tions and since uses a »soft« mark­ing. The as­pect of un­cer­tainty, which is often dis­cussed in the vi­su­al­i­sa­tion scene, is here solved through this wider range of in­di­ca­tion. Mi­crosoft Of­fice and equiv­a­lent soft­ware on the other hand uses fixed marked areas when the user searches an open doc­u­ment for a term. And since there is no un­cer­tainty in the search term the user typed in this per­fectly makes sense. The Sub­lime Text ed­i­tor uses a com­bi­na­tion of back­ground colour and a bor­der around the search term, as it of­fers the abil­ity to se­lect mul­ti­ple words. The web browser Chrome also uses the yel­low back­ground for in-page search, but for search re­sults through the search en­gine Google high­lights the term in bold. (They also pro­vide the search term’s con­text like Linguee does.) This demon­strates the dif­fer­ences these vi­su­al­i­sa­tions have: The yel­low marker helps the user to spot the term in the text, the other lost its im­por­tance to the web­site’s title and is more rel­e­vant within its con­text.

Text high­t­light­ing Linguee uses a soft »marker« to show un­cer­tainty.

In­stead of spot­ting sin­gle term the vi­su­al­i­sa­tion »How Sen­a­tor John Walsh Pla­gia­rized a Final Paper« by the New York Times uses dif­fer­ent back­ground colours to cat­e­gorised text pas­sage. A red »marker« means »Pas­sages taken with­out at­tri­bu­tion«, a yel­low »marker« means »Pas­sages with im­proper at­tri­bu­tion, in­clud­ing using other au­thors’ lan­guage with­out quo­ta­tions«.

Marker metaphor In order to high­light pla­gia­rism the New York Times uses two back­ground-colours

The usage of this marker metaphor per­fectly makes sense in this con­text, since they are re­view­ing a paper and the image of some­body la­belling every in­cor­rect sen­tence is re­ally con­ceiv­able. What is great about this graphic is that a real world phoneme is used as a vi­su­al­i­sa­tion and thus needs no need of ex­pla­na­tion.

An­other ex­am­ple for such a tech­nique are fur­ther ty­po­graphic ad­di­tions like un­der­lin­ing. Mi­crosoft Word and other writ­ing ap­pli­ca­tions use solid, dashed or curly lines un­der­neath the word to in­di­cate gram­mat­i­cal or or­thog­ra­phy mis­takes. Sim­i­lar to cross­ing out words, this could be used to vi­su­alise the sta­tus of text pas­sage. As men­tioned above, meth­ods to show some­thing that does not exist any longer are rare. The line through a word is a learned con­ven­tion that com­mu­ni­cates this.

The first group men­tioned above (plac­ing ad­di­tional el­e­ments after the en­tity) has al­ready been dis­cussed by Ed­ward Tufte and seemed to be his in­spi­ra­tion for the Sparklines. One of the old­est ex­am­ples he cites are by Galileo, who put sim­ple out­lines of Sat­urn’s ap­pear­ance into the line. (Tufte, 2001) Other sym­bol-as-nouns have been pointed out by Phillip Trout­man in »The Won­der Woman Prece­dent: Fe­male (Super)Hero­ism on Trial«, Jon Gross in »Walk­ing Bass Tech­niques«, and by Matt Reed in »Voltaire« (Tufte, with­out date). These books use the ♀-sym­bol, music no­ta­tion right within the text and lit­tle graph­ics of a man to show things words can­not or only with great ef­fort de­scribe. A mod­ern ap­proach comes from ProP­ub­lica who in­tro­duced the State­Face font (2012). When using the font in doc­u­ments or web­sites, every let­ter is re­placed/shown as an US state. Every­body fa­mil­iar with the states can quickly recog­nise each. The type­face has been used mainly in ta­bles as ad­di­tion to the ac­tual name or on web­sites like is­barack­oba­math­ep­res­i­dent.com, where the amount of states was more im­por­tant than their recog­ni­tion.

State­face font Every let­ter in this font is one US state

All these tech­niques fol­low the idea of putting sim­ple graph­ics ei­ther as re­place­ment or as ad­di­tion into the text, to quickly com­mu­ni­cate some­thing in­ef­fa­ble or sup­port writ­ten con­tent. But all these graph­ics are more like sim­ple il­lus­tra­tions and not vi­su­al­i­sa­tions. Borgo et al. dis­tin­guish be­tween these two forms: Pic­tograms (or pic­to­graph) are sym­bolic rep­re­sen­ta­tions of ob­jects, places, or ac­tiv­i­ties. In con­trast Ideograms (or ideo­graph) rep­re­sent »ideas« and thus are in­de­pen­dent from phys­i­cal ob­jects. Ex­am­ples are »way find­ing sig­nage as well as tech­ni­cal no­ta­tions such as ara­bic nu­mer­als, math­e­mat­i­cal no­ta­tions or bi­nary sys­tems, which main­tain the same mean­ing de­spite the dif­fer­ence in lan­guage and en­vi­ron­ment.« (Borgo et al., 2013)

»In switch­ing from vocal and face-to-face com­mu­ni­ca­tions to pri­mar­ily text, we have lost out on the feel­ings that can only be con­veyed in in­flec­tion and fa­cial ex­pres­sions.«

 — Joanna Stern, 2015

In re­cent time a lot of pic­tograms and ideograms arose: Tech­nol­ogy like SMS al­lowed peo­ple to quickly com­mu­ni­cate dur­ing every­day life. Form and hence con­tent was dif­fer­ent from tra­di­tional let­ters. The de­sire to ex­press one­self emo­tion­ally was com­pen­sated in let­ters through the length of the writ­ing. Due to the tech­ni­cal lim­i­ta­tions like ca­pac­ity or band­width peo­ple in­vented sm­i­lies con­sist­ing of ac­tual char­ac­ters :). These rep­re­sen­ta­tions of faces mostly con­sist of two char­ac­ters and are often re­ferred to as Emoti­cons as they mostly ex­press emo­tions.

Pushed for­ward by the in­ter­net cul­ture faces con­sist­ing of char­ac­ters from the whole Uni­code set emerged ಠ_ಠ. These were often not lim­ited to the two-char­ac­ter-side­way ap­pear­ance of the orig­i­nal ones. In­stant mes­sen­gers lead to the use of graph­i­cal sm­i­lies: Il­lus­tra­tions of mostly faces, like the »orig­i­nal« smi­ley in­vented in 1963 by Har­vey Ball.

In­stead of faces con­sist­ing of char­ac­ters or sym­bols these smi­leys are small graph­ics sit­ting right within the text. While they au­to­mat­i­cally re­placed the »writ­ten smi­leys« in the be­gin­ning, users are now able to in­sert them through a spe­cial key­board lay­out on their mo­bile phones. In re­cent time a par­tic­u­lar set called Emoji (絵 (e ≅ pic­ture) 文 (mo ≅ writ­ing) 字 (ji ≅ char­ac­ter)) was de­vel­oped in Japan. In con­trast to Emoti­cons Emo­jis show not only emo­tions (faces) but also ob­jects or ab­stract sym­bols. (The Uni­code Con­sor­tium, 2015) These ideograms quickly spread all over the world as lin­gua franca, since they are (nearly) uni­ver­sally un­der­stand­able: »Cross-cul­tural stud­ies by Paul Ekman and cowork­ers strongly sug­gest that cer­tain human ex­pres­sions are uni­ver­sal com­mu­ni­ca­tion sig­nals, cor­rectly in­ter­preted across cul­tures and so­cial groups«. (Ware, 2004)

Mark Davis, the pres­i­dent and co-founder of Uni­code de­scribes Emo­jis as to have a »use­ful am­bi­gu­ity«. Their in­ter­pre­ta­tion varies from con­text, and hence mak­ing them ver­sa­tile. (Stern, 2015)

Japan spe­cific Emo­jis Some emo­jis are very Japan­ese spe­cific and are not un­der­stand­able out­side of Japan. For ex­am­ple 🏮 means Iza­kaya lantern and 🔰 means be­gin­ner. (Stern, 2015)

The human brain is not only ca­pa­ble of recog­nis­ing these highly ab­stract im­ages – some­times only con­sist­ing of two dots and a line – as faces and cor­rectly in­ter­pret »their« emo­tions, but in fact we can­not not see faces in var­i­ous every­day ob­jects like cars, British plug sock­ets or even in a moun­tain on Mars. (Mc­Cloud, 2001) The phe­nom­e­non called Parei­do­lia has been sci­en­tif­i­cally stud­ied but also found ap­pli­ca­tions in art and de­sign. In an ex­per­i­ment with apes who were ex­posed to im­ages of an ape face, a human’s face with and with­out eyes, a smi­ley and ran­dom lines, a neu­ron in the ape’s in­fe­rior tem­po­ral cor­tex an­swered the strongest to an image of his con­spe­cific, but showed sim­i­lar even though weaker re­ac­tions to human faces or even sm­i­lies. (Gold­stein, 2002)

Face on Mars This pic­ture of a moun­tain was taken by the NASA

Step­ping quickly back to the rep­re­sen­ta­tion of in­for­ma­tion in a sym­bol, in 1992 the psy­chol­o­gist Christo­pher Wick­ens sug­gested the use of »ob­ject dis­play as a graph­i­cal de­vice em­ploy­ing ‘a sin­gle con­toured ob­ject’ to in­te­grate a large num­ber of sep­a­rate vari­ables.« (Ware, 2004) These ac­cu­mu­la­tion of mul­ti­ple in­for­ma­tion in an ob­ject are nowa­days used in vi­su­al­i­sa­tion under the term glyph. In­stead of show­ing mul­ti­ple data points in a sin­gle vi­su­al­i­sa­tion, these »small in­de­pen­dent vi­sual ob­jects« (Borgo et al., 2013) show sin­gle data points but with mul­ti­ple data di­men­sions through a va­ri­ety of en­cod­ings: »The de­sign of glyphs can make use of many dif­fer­ent vi­sual chan­nels such as shape, colour, tex­ture, size, ori­en­ta­tion, as­pect ratio or cur­va­ture, en­abling the de­pic­tion of multi-di­men­sional data at­trib­utes.« (Borgo et al., 2013) Com­bin­ing this idea with the human’s abil­ity to con­nect mul­ti­ple vi­sual in­for­ma­tion from the face to recog­nise a per­son and his ex­pres­sion ap­pears ob­vi­ous.

Cher­noff faces The faces for lawyers’ rat­ings of twelve judges by Av­enue

One ex­am­ple are the Cher­noff faces, named after the in­ven­tor Her­man Cher­noff. They were one of the first im­ple­men­ta­tions in 1973. »Data vari­ables are mapped to dif­fer­ent fa­cial fea­tures, such as the length of the nose, the cur­va­ture of the mouth, the size of the eye, the shape of the head, etc..« (Ware, 2004) Even though the usage seems rea­son­able, real world util­i­sa­tion is sel­dom. An ex­am­ple by Steve C. Wang (2008) il­lus­trates a pos­si­ble ap­pli­ca­tion. He gen­er­ated faces for mul­ti­ple base­ball man­agers, by ap­ply­ing vari­ables like »num­ber of dif­fer­ent line­ups used«, »pinch-run­ners used« and »sac­ri­fice-bunt at­tempts« to ears height, face width and mouth form. Un­for­tu­nately it be­comes ap­par­ent that this vi­su­al­i­sa­tion is at least sim­i­larly com­pli­cated to de­code as a graphic with an equal amount of data vari­ables vi­su­alised »tra­di­tion­ally«. The abil­ity to recog­nise faces is proven, but the abil­ity to map sin­gle vi­sual in­for­ma­tion in a face to ar­bi­trary vari­ables seems ques­tion­able. In his »A Cri­tique of Cher­noff Faces« Robert Kosara also notes the fol­low­ing to be aware of: »There are fea­tures that are clearly much more im­por­tant (eyes, lips) than oth­ers (over­all shape). Thus, rep­re­sent­ing data through these vi­sual fea­tures means that some data will be much more vis­i­ble than oth­ers.« (Kosara, 2007) De­spite the abil­ity to gen­er­ate Cher­noff Faces with Wol­fram|Alpha or with the lan­guage R, an­other rea­son for the un­fre­quent use might be the rather time-con­sum­ing gen­er­a­tion of these faces.

Glyphs in gen­eral can be seen as one form of micro vi­su­al­i­sa­tion, but are mostly dri­ven by the am­bi­tion to en­code as many di­men­sions as pos­si­ble in small space. They have been dis­cussed mostly for three di­men­sional data rep­re­sen­ta­tions, but have been treated scep­ti­cally »in the com­mu­nity about the en­cod­ing ca­pa­bil­ity […] pri­mar­ily due to its size, lim­ited ca­pac­ity of in­di­vid­ual vi­sual chan­nels and cog­ni­tive de­mand for learn­ing and mem­o­riza­tion.« (Borgo et al., 2013) As the de­sign guide­lines of glyphs have some sim­i­lar­i­ties to the pro­posed micro vi­su­al­i­sa­tions, these should be dis­cussed later in the con­text of cre­at­ing such graph­ics.

The idea to place more con­ven­tional vi­su­al­i­sa­tions in­side the text and hence get­ting closer to Tufte’s Sparklines was be­hind the cre­ation of the type­face FF Chartwell by Travis Kochel. (Kochel, 2012) Through the Open­Type tech­nol­ogy it is pos­si­ble to cre­ate charts con­sist­ing of pies, lines, hor­i­zon­tal and ver­ti­cal bars as well as rose, rings, and radar di­a­grams. For ex­am­ple, by sim­ply writ­ing the given val­ues for each sec­tor in a pie chart and se­lect­ing a spe­cific styl­is­tic style, the writ­ten val­ues are re­placed by a vi­su­al­i­sa­tion. This en­ables even am­a­teurs to quickly cre­ate and in­te­grate these graph­ics into the text.

Fontas­tic Gen­er­ated »fonts« Equal­izer-Font.pde and PieChart­Font.pde

More ex­am­ples have been cre­ated by the Pro­cess­ing Li­brary Fontas­tic. Some of the rather artis­tic ap­proaches can be ac­tu­ally used as vi­su­al­i­sa­tion: Equal­izer-Font.pde rep­re­sents each let­ter with equal­izer-like bars, PieChart­Font.pde adds a sec­tor for each let­ter to a pie chart, and Fre­quen­cy­Font.pde uses the fre­quency of the let­ters in the Eng­lish lan­guage to cre­ate cor­re­spond­ing sized bars for every typed let­ter. (Koller, 2014)

The idea to rep­re­sent each let­ter with a graphic has mostly found ap­pli­ca­tion in art. Lau­ren Di­Cioc­cio (with­out date) for ex­am­ple en­codes each let­ter with a colour dot, Catalina Viejo Lopez de Roda (with­out date) writes let­ters with paper shav­ings in con­tent-re­lated colour schemes, and the colour­ing of piano key­boards and music sheets can help stu­dents to con­nect mu­si­cal no­ta­tions with the keys. Be­sides the ac­cen­tu­a­tion of the ac­tual let­ters, the first two pro­jects un­for­tu­nately do not help the text con­tent, but in fact make it nearly im­pos­si­ble to read.

Sol Le­witt This piece of art shows a net­work of the word »art« on two pages

One sel­dom ap­proach is the ana­log net­work vi­su­al­i­sa­tion in a book by the artist Sol Le­Witt. He used printed pages in a book to draw net­works of the word »art«. He con­nected each ap­pear­ance with red lines, and the cor­ners and the mid­points with blue and green lines. (Cas­tro, with­out date) While net­work vi­su­al­i­sa­tions often use force di­rected lay­outs that place ob­jects in re­gard to their »con­nec­tiv­ity«, this lay­out is sim­i­lar to a map that uses the geo-spa­cial place­ment of the nodes to draw the edges. In this case the words are lo­cated all over the text, the con­nec­tions can help to com­pre­hend the struc­ture of the text. Un­for­tu­nately his artis­tic ap­proach does not sup­port this and is prob­a­bly not meant for this.

Ad­ja­cent

This is cer­tainly how most reg­u­lar vi­su­al­i­sa­tions work: A graphic as an ad­di­tion to a de­scrib­ing text. To dif­fer­en­ti­ate rel­e­vant graph­ics for this the­sis, they need to sup­port the text’s con­tent rather than vi­su­al­is­ing some­thing com­pletely new. The above men­tioned »The Rus­sia Left Be­hind« micro vi­su­al­i­sa­tion is rather use­less on its own with­out any con­text, but is es­sen­tial in com­bi­na­tion with the text.

The first (un­sci­en­tific) pro­ject is the chil­dren’s book »Where The Wild Things Are« writ­ten and il­lus­trated by Mau­rice Sendak. While this ex­am­ple is not a vi­su­al­i­sa­tion in a strict sense it lit­er­ally il­lus­trates how the im­age-text re­la­tion can evolve over the course of a book and hence its ex­am­i­na­tion can also help to cre­ate data-dri­ven vi­su­al­i­sa­tions. The dou­ble pages mostly con­sist of text and il­lus­tra­tion areas. As in most pic­ture books, the im­ages vi­su­alise the con­tent of the text. What is spe­cial in this book is the area each part takes of the page. As the pro­tag­o­nist drifts in his fan­tasy world, the story be­comes more sur­real and the pic­tures take over the page. Michael McAghon de­scribes the il­lus­tra­tions to »in­di­cate where we are in Max’s con­cocted world and to re­in­force the imag­i­na­tion ex­pe­ri­ence.« (McAghon, 2009) When he comes back into the real world the ra­tio­nal text re­trieves im­por­tance and the pic­tures be­come smaller.

Book lay­out Sketch by McAghon of the lay­out of »Where The Wild Things Are«

The prac­tise to show where the reader is cur­rently sit­u­ated within the book, is com­monly achieved in books through run­ning heads and foot­ers, page num­bers, head­ings or colours. In sci­en­tific con­texts head­ings are usu­ally sup­ple­mented with count­ing num­bers and cor­re­spond­ingly in­dented. In dig­i­tal process it is often shown through a progress bar. A vi­su­al­i­sa­tion tech­nique that aims to sup­port this as well are den­dro­grams. Rolf Fred­heim’s ap­proach (2013) even shows how the dif­fer­ent top­ics re­late to each other. In his im­ple­men­ta­tion he uses only the head­lines for each chap­ter, but it could be (to some ex­tent) ap­plied to the com­plete text as well. He con­nects the tree struc­ture from den­dro­grams with the head­lines to vi­su­alise the text’s inner struc­ture. Since this vi­su­al­i­sa­tion only shows the or­gan­i­sa­tion of the text that could also be ex­tracted from the in­den­tion or the run­ning num­bers of the head­ings, he pro­poses »Den­droArcs« that re­veal the re­la­tions within the text. This net­work has each chap­ter as fixed nodes in the list and the re­la­tions as edges rep­re­sented with arcs. The wider the arc, the more con­nected are the two top­ics . These (pos­i­tive and neg­a­tive) cor­re­la­tions are ad­di­tion­ally vi­su­alised for each node with a pie chart. While this might help to analyse a vast amount of text in con­text of dig­i­tal hu­man­i­ties, this can also help the reader to jump from topic to topic, fol­low­ing his in­ter­est.

Den­doarcs The arcs show the re­la­tion be­tween para­graphs. The pie charts in­di­cate cor­re­la­tions

In­stead of pro­vid­ing overview through graph­i­cal el­e­ments di­rectly con­nected to the text, other ap­proaches cre­ate sym­bolic rep­re­sen­ta­tions of the con­tent. This can span from minia­ture, wire-frame or even more ab­stract rep­re­sen­ta­tions of the con­tent. The above men­tioned Sub­lime Text ed­i­tor of­fers a down-scaled ver­sion of the writ­ten source code in a sep­a­rate »Min­imap«. This al­lows for nav­i­ga­tion in the doc­u­ment but also pro­vides con­text and overview for the reader/writer. Since the cur­rent vis­i­ble area is also high­lighted in the Min­imap, it acts like the scroll­bar known from var­i­ous ap­pli­ca­tions. The idea to in­clude a minia­ture pre­view in the scroll­bar has al­ready been de­scribed by Scott Mc­Crickard and Richard Catram­bone in »Be­yond the Scroll­bar: An Evo­lu­tion and Eval­u­a­tion of Al­ter­na­tive Nav­i­ga­tion Tech­niques« (1999) with the con­cern that »con­sid­er­able in­for­ma­tion about the con­tent of the space is not com­mu­ni­cated in the scroll­bar.« (Mc­Crickard et al., 1999) But in­stead of sim­ply scal­ing down every­thing, their »Mural Bars« pro­vide the same space for every item: »Since an equal amount of space is given to each item, any item can be seen with equal clar­ity no mat­ter where they are in the list.« (Mc­Crickard et al., 1999) A cur­rent ap­pli­ca­tion of these nav­i­ga­ble minia­ture rep­re­sen­ta­tion is also the »Nav­i­ga­tor« found in Adobe Pho­to­shop and other ap­pli­ca­tions by Adobe. It pro­vides a min­imised ver­sion of the cur­rent image, with an over­laid rec­tan­gle for the cur­rent view­point.

Where you are The screen­shot shows »pre­views« of each ar­ti­cle

The web­site for the book »Where you are« (2013) on the other hand pro­vides wire-frames for each chap­ter. This vi­su­alises not only the length of the ar­ti­cle, but also its struc­ture and the pic­ture-text ratio. As dis­cussed in the be­gin­ning: We see be­fore we read. The sim­pli­fied rep­re­sen­ta­tion of the web­site mim­ics this im­pres­sion. This pro­vides two things: First, we can es­ti­mate what we are about to see, and sec­ond, when we re­visit the web­site, we can op­ti­cally search for the spe­cific ar­ti­cle we read some time ago. Just like we tend to re­mem­ber »the red book with the white font» or »the web­site with the big image at top«, this wire­frame pre­sen­ta­tion can help us recog­nise from their op­ti­cal ap­pear­ance.

An even more ab­stract rep­re­sen­ta­tion of a web­site can be found in the Google Chrome Browser. When the user searches for a term, the lo­ca­tions of the terms in re­la­tion to the web­site’s height are marked in the scroll­bar. This con­cept re­quires more learn­ing from the user, but is quickly un­der­stood since the idea of the scroll­bar is used since the 1980s.

Re­ac­tive doc­u­ments When the user changes one value, all oth­ers change ac­cord­ingly.

The last ex­am­ple is mainly shaped by Bret Vic­tor and his »Ex­plorable Ex­pla­na­tions« and »Re­ac­tive Doc­u­ments«. (2011) His in­ter­ac­tive doc­u­ments can be best de­scribed as text­books with con­nected text and di­a­grams. Most pro­jects ex­plain sci­en­tific top­ics with text, for­mu­las, im­ages, and di­a­grams. In­ter­ac­tive el­e­ments within these el­e­ments can be dragged, changed or sim­ple hov­ered. What makes them so in­ter­est­ing it that as the user changes an el­e­ment in the text, the cor­re­spond­ing el­e­ments in the other el­e­ments change ac­cord­ingly. As Vic­tor states, this in­ter­ac­tive dual-cod­ing brings trans­parency, ex­pla­na­tion, and en­cour­age­ment. (Vic­tor, 2011)

Ex­plorables ex­am­ples Text, for­mu­las and graph­ics are con­nected. The in­ter­ac­tive graphs allow in­ter­ac­tiv­ity
Con­clu­sion Ad­di­tional El­e­ments

The two main groups I was able to find as ex­am­ples for in­te­grated ad­di­tional vi­su­al­i­sa­tions are back­ground-colours so that the colour con­veys ad­di­tional mean­ing and graph­ics that are placed be­hind the cor­re­spond­ing word. The use of the back­ground-colour is mostly in­spired by the phys­i­cal marker. They em­pha­sise words or label to a cer­tain cat­e­gory. The ad­di­tional graph­ics can be di­vided into three groups: Sparkline-like vi­su­al­i­sa­tions that are in­spired by com­mon charts, sym­bols like sm­i­lies or ideograms and glyphs. De­spite the fact that glyphs have been elab­o­rated for some time now, they have not been placed in­side the text, but rather as stand­alone graph­ics or within a spa­tial pro­jec­tion. It is up to fur­ther re­search to elab­o­rate this miss­ing ap­pli­ca­tion. In gen­eral, the sym­bols were mostly used for qual­i­ta­tive data, while the ac­tual min­imised vi­su­al­i­sa­tions con­vey quan­ti­ta­tive data. The sym­bols often sub­sti­tute a word or em­pha­sise its con­tent; their data source is often in­nate or in­trin­sic. In con­trast, the data words as de­scribed by Tufte add ad­di­tional ex­trin­sic data to the cor­re­spond­ing word.

Three things that can be no­ticed in the usage of ad­di­tional el­e­ments are: the size, the place­ment and the sta­tic con­tent of the graph­ics. All the graph­ics are placed in the same height and at the same base­line as the text. More­over, the source of the data is mostly sta­tic. The in­clu­sion of dy­namic data from the user or live up­dates is sel­dom. I see po­ten­tial for this miss­ing fea­tures and will de­scribe it fur­ther down.

The ex­ter­nal ad­di­tional cat­e­gory is the most di­verse one. The tech­niques used can be di­vided into four groups: Image/text-ra­tio, al­ter­na­tive rep­re­sen­ta­tion of the con­tent, text-lines/para­graphs as nodes and ab­stract rep­re­sen­ta­tion of the ac­tual text. Also all types of data sources are being used for these vi­su­al­i­sa­tions. The New York Times graphic adds an ad­di­tional level of spa­tial in­for­ma­tion to the story, the ab­stract rep­re­sen­ta­tion shows in­nate at­trib­utes of the text and the edges be­tween the text node rep­re­sent the in­trin­sic re­la­tions be­tween text pas­sages.

While the mod­i­fi­ca­tion of the text mostly acted as a sup­port­ing el­e­ment, the ad­di­tional graph­i­cal el­e­ments often over­lay the ex­ist­ing text with an extra in­for­ma­tion layer that adds fur­ther ex­trin­sic data or show the in­nate or in­trin­sic data from an­other per­spec­tive. What needs to be ex­am­ined sep­a­rately is the in­ter­ac­tion be­tween the text and the ad­di­tional el­e­ments and how they can be recog­nised as con­nected. The work of Bret Vic­tor is cer­tainly pi­o­neer­ing for fu­ture work.

Con­clu­sion

It can be seen that the ex­ist­ing work al­ready cov­ers a wide area of sub­jects and also fills out the tax­on­omy. It has been proven use­ful to sep­a­rate these four fields to sys­tem­at­i­cally dis­cuss each pro­ject in a sys­tem­atic way. Some pro­jects used mul­ti­ple tech­niques and cov­ered more than one field. This can be seen as an in­spi­ra­tional ap­proach for fu­ture vi­su­al­i­sa­tions to com­bine sev­eral fields in order to use en­cod­ings that are far enough sep­a­rated in their ap­pli­ca­tion to not dis­tract each other. The po­ten­tial of this com­bi­na­tion of mul­ti­ple types is dis­cussed fur­ther down.

So far only the dif­fer­ence be­tween »ad­di­tional el­e­ments« and »mod­i­fi­ca­tion« has been ex­am­ined. But when we re­view in­te­grated and ad­ja­cent vi­su­al­i­sa­tions, one qual­ity and hence ap­pli­ca­tion be­comes ap­par­ent for each. In­line vi­su­al­i­sa­tions allow for an un­matched close­ness of the ob­jects in its con­text and the vi­sual rep­re­sen­ta­tion of its data points. Ex­ter­nal vi­su­al­i­sa­tions on the other hand allow for the com­par­i­son of these vi­sual rep­re­sen­ta­tions, which is harder when the ob­jects are dis­trib­uted over the text. In the fol­low­ing chap­ter I will dis­cuss tech­niques to over­come these lim­i­ta­tions through link­ing.

An­other point that needs to be con­sid­ered when com­par­ing in­te­grated and ad­ja­cent vi­su­al­i­sa­tion is the dif­fer­ent size. Ob­vi­ously, graph­ics that need to be placed in­line need to be smaller. »Heer et al. com­pared small line charts against two types of Hori­zon Graphs (1-band and 2-band) and in­ves­ti­gated the im­pact of chart height on both de­signs. They found that small chart heights neg­a­tively im­pacted how ac­cu­rately and quickly par­tic­i­pants es­ti­mated the dif­fer­ence be­tween two data points.« (Gof­fin et al., 2014)

The third as­pect that needs to be con­sid­ered is the read­ing flow. Graph­i­cal el­e­ments that pop-out too much may not be wel­come and can hin­der the read­abil­ity of the text. Pear­son et al. (2009, in Gof­fin et al., 2014) found out that users pre­ferred the place­ment of larger an­no­ta­tions to be in the mar­gin. Fur­ther em­pir­i­cal re­search is needed to bal­ance be­tween read­abil­ity, read­ing speed and pref­er­ences of the reader.

In gen­eral, micro vi­su­al­i­sa­tions need a dif­fer­ent ap­proach than reg­u­lar vi­su­al­i­sa­tions, es­pe­cially due to their size. Their ap­pli­ca­tions are al­ways trade-offs. One needs to make com­pro­mises on the shown data di­men­sions, the num­ber of data points, the un­der­lay­ing co­or­di­nate sys­tem or the leg­end. Even more than com­mon vi­su­al­i­sa­tions they should be utilised for im­pres­sions, not exact value ex­trac­tion. As stated be­fore, the qual­ity of ta­bles is to rep­re­sent data in an ac­cu­rate way, text on the other hand guides the reader through the in­for­ma­tion in a dis­cur­sive way and vi­su­al­i­sa­tions help to get an over­all image and an im­pres­sion of sin­gle or mul­ti­ple data points.

What can be sum­marised from the ex­ist­ing work is that micro vi­su­al­i­sa­tions can help the text in sev­eral ways:

  • The mod­i­fi­ca­tion of the text can op­ti­cally sup­port the text’s con­tent while read­ing. It can func­tion as vi­sual im­pres­sion of the text with­out read­ing the whole text and it can help to gain an overview of the text be­fore or after read­ing it.
  • Ad­di­tional el­e­ments can also sup­port the text’s con­tent, func­tion as an­chor point when the user scans the doc­u­ment, and add ad­di­tional data that is not as im­por­tant as the text’s con­tent. All this acts as sup­port and hence should focus on the pos­si­bil­ity to quickly offer the user an im­pres­sion of data in­stead of de­vi­at­ing him from the ac­tual text. Given this, the aban­don­ment of an co­or­di­nate sys­tem that would nor­mally help the user to get an idea of the data is jus­ti­fi­able.

Eval­u­a­tion

This chap­ter builds upon the pre­vi­ous chap­ter – hence the the­o­ret­i­cal ar­gu­men­ta­tion and the analy­ses of states of the art – and pro­vides a re­view of the ap­proach.
Blank spots – re­vealed through the tax­on­omy – are iden­ti­fied and sug­ges­tions made on how this could be ac­com­plished in fu­ture work. As a final demon­stra­tion of the micro vi­su­al­i­sa­tion con­cept’s value, I pro­vide es­sen­tial in­sights from three ex­pert in­ter­views.
In its very last part, my the­sis makes some con­clud­ing re­marks and closes with sug­gest­ing fu­ture di­rec­tions in re­search on micro vi­su­al­iza­tion and its ap­pli­ca­tion.

Fu­ture ap­proaches

From the gath­er­ing, analy­sis and clas­si­fi­ca­tion of the ex­ist­ing work sev­eral ap­proaches emerge. With the tax­on­omy blank spots in the micro vi­su­al­i­sa­tion sphere can be dis­cov­ered and also novel types can be de­vi­ated through the com­bi­na­tion of mul­ti­ple fields. A fur­ther goal is to turn the lim­i­ta­tions into ad­van­tages and fea­tures.

Com­bi­na­tion of vi­su­al­i­sa­tions

Since micro vi­su­al­i­sa­tions are not so dom­i­nant in their ap­pear­ance, they can be com­bined with other vi­su­al­i­sa­tions. In fact this can be seen as one strength of micro vi­su­al­i­sa­tions: The com­bi­na­tions of these four types allow for ad­di­tional data lev­els that would not be pos­si­ble with a sin­gle vi­su­al­i­sa­tion or text.

Com­bi­na­tion Two vi­su­al­i­sa­tions can co­ex­ist, work to­gether as ref­er­ence or one can act as thumb­nail

This can be done in two dif­fer­ent ways: Com­pletely in­de­pen­dent vi­su­al­i­sa­tions show­ing dif­fer­ent data or mul­ti­ple vi­su­al­i­sa­tions that deal with the same data set and sup­port each other. The lat­ter can be for ex­am­ple used with an ad­di­tional ad­ja­cent and an in­line ap­proach. The in­line vi­su­al­i­sa­tion can ei­ther work with mod­i­fi­ca­tions to high­light their con­nect­ed­ness to the ad­ja­cent graphic or with ad­di­tional el­e­ments. These graph­ics could serve as thumb­nails or pre­views (or scented wid­gets) for the ad­ja­cent vi­su­al­i­sa­tion.

Break­ing the bor­ders

The fact that in­line mod­i­fi­ca­tions or ad­di­tional el­e­ments are lim­ited to the line height can not only be seen as lim­i­ta­tion but also as styl­is­tic de­vice. When vi­su­al­i­sa­tions show data that has out­liers or an ex­treme am­pli­tude, the de­signer some­times shows this by lit­er­ally break­ing the lay­out with these parts. The fa­mous »hockey stick graphs« used in the con­text of cli­mate change have a steep rise to­wards the end. In the »Atlas der Glob­al­isierung – Spezial – Klima« (Atlas of Glob­al­i­sa­tion – Spe­cial – Cli­mate, Bovet et al., 2008) by Le Monde diplo­ma­tique mul­ti­ple graph­ics use this tech­nique, where bars or line charts go be­yond the bor­ders of the graphic or even the page. Al Gore used the same tech­nique in his »An In­con­ve­nient Truth« pre­sen­ta­tion. In this rhetor­i­cally im­pres­sive ges­ture he uses a lift to fol­low the course of the CO2 con­cen­tra­tion in the earth’s at­mos­phere.

Hockey stick graph The CO2-value breaks out

For micro vi­su­al­i­sa­tions the same can be used. Top­ics that break the limit can ex­ceed the bor­ders of the line height. This could be re­alised with graphs or cir­cles that lay be­hind the ac­tual text to en­sure read­abil­ity. Vi­su­ally this would cre­ate two lay­ers that need to be op­ti­cally dis­tinct. When used in a dig­i­tal medium this could even be en­hanced by mak­ing each layer re­spon­sive in its vis­i­bil­ity.

Break­ing out The lim­i­ta­tions can be used for the mes­sage
Re­spon­sive data

Reg­u­lar vi­su­al­i­sa­tions often deal with quite a great amount of data that does not change fre­quently. Micro vi­su­al­i­sa­tions on the other hand deal with only very lit­tle data points and hence can be up­dated eas­ily. The jQuery im­ple­men­ta­tion of Sparklines al­ready does this by show­ing live fi­nance data or the cur­rent mouse speed. (Watts, 2012) This can eas­ily be ap­plied for other ap­pli­ca­tions: Cur­rent tem­per­a­tures of cities in a travel guide, next de­par­ture times of trains and cur­rent traf­fic jam sit­u­a­tions, pop­u­lar­ity on Twit­ter, num­ber of Google search re­sults for that term, or up-to-date user rat­ings. But in­stead of show­ing every user the same in­for­ma­tion, the vi­su­al­i­sa­tions and hence the text could re­spond to the reader. This is a great op­por­tu­nity to in­volve the reader more into the story. Since many ar­ti­cles are writ­ten for the gen­eral pub­lic, the given ex­am­ples or num­bers could pos­si­bly not re­late to the reader. He might be un­fa­mil­iar with the cur­rency, the local time or weather at the de­scribed city or even with the lo­ca­tion of the place. Sim­i­lar to the pre­vi­ously de­scribed browser plu­gin »Dic­tio­nary of Num­bers«, this can help the reader to un­der­stand the given in­for­ma­tion bet­ter. To re­alise this con­ver­sion to fa­mil­iar val­ues, the user can ei­ther put in the re­quired data or it could be re­ceived through the browser’s in­for­ma­tion.

Con­tex­tual in­for­ma­tion be­came quite a buzz­word in the re­cent time. Google’s ad­ver­tis­ing is so suc­cess­ful be­cause of their knowl­edge about the user. They can in­di­vid­u­ally change the ad­vert to the user’s pro­file. Fur­ther­more, their search re­sults de­pend on pre­vi­ous search queries and the cur­rent lo­ca­tion of the user. Sim­i­lar ap­proaches for vi­su­al­i­sa­tions seem promis­ing and should be elab­o­rated in de­tail in fu­ture works.

Vary­ing in­line place­ment

In »Ex­plor­ing the Place­ment and De­sign of Word-Scale Vi­su­al­iza­tions« Gof­fin et al. (2014) em­pir­i­cally tested the var­i­ous pos­si­ble place­ments ad­di­tional vi­su­al­i­sa­tions could be in. Be­sides in­ter­est­ing in­ter­ac­tive and dy­namic place­ments, they eval­u­ated seven sta­tic po­si­tions:

  • Tra­di­tional In-Line Place­ment
  • Over­lay Word-Scale Vi­su­al­iza­tions on En­ti­ties
  • Using Ex­ist­ing In­ter-Line Space
  • Using In­ter-Line & In­creased In­ter-Word Space
  • In­creas­ing In­ter-Line & In­ter-Word Space
  • Al­low­ing Vi­su­al­iza­tions To Over­lap
  • Al­low­ing Vi­su­al­iza­tions to Over­lap Sur­round­ing Text

Be­sides these fixed po­si­tions that are re­quired for printed ver­sions, dig­i­tal media would allow dy­namic place­ments of the graph­ics. (Gof­fin et al., 2014) The most basic form would allow the user to hover over the ref­er­ence word to dis­play the graphic ei­ther by re­plac­ing it or plac­ing the graph­ics next to the el­e­ment. The other form they de­scribe in the paper is a tog­gle for all vi­su­al­i­sa­tions in the doc­u­ment. This would allow the text to be less clut­tered, but on the other hand this would re­quire the input and knowl­edge of the user. Be­sides these tech­niques oth­ers which in­volve nu­mer­ous in­ter­ac­tions would be pos­si­ble: Vi­su­al­i­sa­tion could be be­come ap­par­ent through scrolling, through text input, or de­pend­ing on the vis­it­ing time. Be­sides the vis­i­bil­ity, in­ter­ac­tion could also change the po­si­tion of the graphic. This would allow for in­di­vid­ual place­ment to en­hance com­par­i­son, but also open up a com­plete new re­search.

In­ter­ac­tive micro vi­su­al­i­sa­tions

Be­sides the pre­vi­ously de­scribed in­ter­ac­tive ap­proaches for the place­ment, in­ter­ac­tion could also be used to en­hance com­par­i­son be­tween mul­ti­ple in­line el­e­ments or the link be­tween ad­ja­cent el­e­ments with their cor­re­spond­ing words.

Since the com­par­i­son is hin­dered when the vi­su­al­i­sa­tions are dis­trib­uted all over the doc­u­ment, in­ter­ac­tions be­tween these can coun­ter­act this. Ei­ther the com­plete vi­su­al­i­sa­tion or — in the case of mod­i­fied text — pas­sages could be high­lighted while other el­e­ments fade out. An­other pos­si­bil­ity is to high­light cor­re­spond­ing cat­e­gories that occur in each vi­su­al­i­sa­tion. For Sparklines this could be the data points for a cer­tain time.

The sec­ond pos­si­bil­ity deals with the prob­lem to con­nect ad­ja­cent vi­su­al­i­sa­tions to cor­re­spond­ing en­ti­ties. Stein­berger et al. (2011) con­ducted re­search on how vi­sual links can con­nect re­lated in­for­ma­tion across mul­ti­ple vi­su­al­i­sa­tions. As they de­scribe, their in­va­sive but also more ex­pres­sive method aims at min­imis­ing the oc­clu­sion but also make the links vi­su­ally stand out from sur­round­ing in­for­ma­tion. Even though they found ev­i­dence in their ap­proach, their sce­nario is not com­pletely com­pa­ra­ble. Since my ap­proach fo­cuses on the read­abil­ity, meth­ods that use an »in­vis­i­ble« line need to be re­searched. In vi­su­al­i­sa­tion this is com­monly re­alised through mark­ers that can be found in the leg­end and in the graphic. But this re­quires mem­o­ri­sa­tion and search. As the ex­am­ple »The Rus­sia Left Be­hind« shows, the link can also be cre­ated through in­ter­ac­tiv­ity: As the user scrolls down, the rel­e­vant part is high­lighted.

As pre­vi­ously shown, Bret Vic­tor does this in a sim­i­lar way in his so­phis­ti­cated work »Ex­plorable Ex­pla­na­tions« and »Re­ac­tive Doc­u­ments«. (2011) The user can hover over the en­tity and the cor­re­spond­ing part in the vi­su­al­i­sa­tion is high­lighted. Bas Broekhuizen also de­scribes how to »in­te­grate text and im­ages in­ter­ac­tively« (Broekhuizen, 2014) in a blog post. He shows ex­am­ples of graph­ics in which areas are high­lighted when the user hov­ers over a link. While this seems sim­ple, it can help the sto­ry­teller to guide the reader through the graphic. For this kind of in­ter­ac­tiv­ity, I see a great po­ten­tial, es­pe­cially in com­bi­na­tion of micro vi­su­al­i­sa­tion in­te­grated and ad­ja­cent as de­scribed above.

Min­imis­ing reg­u­lar vi­su­al­i­sa­tions

The way Sparklines work is by tak­ing a reg­u­lar line chart, split­ting it up into the dif­fer­ent data rows, re­mov­ing the co­or­di­nate sys­tem and min­imise it to line height. The sec­ond step is ba­si­cally the idea of Small Mul­ti­ples. These types of graph­ics are a well elab­o­rated con­cept that is often used to de­clut­ter graph­ics when too many data rows are in­volved or data is shown over time, which can re­sult in too many data points in one graphic. The idea is to show the same co­or­di­nate sys­tem mul­ti­ple times, each with dif­fer­ent data points shown. Small mul­ti­ples are a field on their own that is in the same way pop­u­lar as it is crit­i­cised. Fur­ther re­search should elab­o­rate how pre­vi­ous re­search can also help micro vi­su­al­i­sa­tions.

Line chart to Sparkline A line chart be­comes a Small Mul­ti­ple, be­comes a Sparkline.

But the pre­vi­ously de­scribed con­cept of tak­ing apart reg­u­lar vi­su­al­i­sa­tion brings fur­ther gen­eral ap­proaches: ex­plor­ing this tech­nique for reg­u­lar vi­su­al­i­sa­tion types. Ap­ply­ing this to bub­ble charts, treemaps or stacked bar charts can re­sult in use­ful ap­pli­ca­tions. In fact, the shown work by Sol Le­Witt is a com­mon net­work di­a­gram, in which the spa­tial po­si­tion of the nodes is given by the po­si­tion within the text. In a sim­i­lar way the other striped and min­imised graph­ics would have trade-offs to their orig­i­nal form, but be­sides being use­able as micro vi­su­al­i­sa­tions, they could re­sult in less clut­tered graph­ics sim­i­lar to small mul­ti­ples.

One vi­su­al­i­sa­tion type that has not yet been used for micro vi­su­al­i­sa­tions are sin­gle stacked bar charts. In con­trast to Sparklines which use time se­ries data, these bar charts work with per­cent­age val­ues and can hence adapt to the word’s width. When work­ing with mul­ti­ple vi­su­al­i­sa­tion of such type it needs to be elab­o­rated if they should adapt to the size of their cor­re­spond­ing en­tity, to the size of the small­est en­tity in the doc­u­ment or ex­tent over the en­tity’s width to en­able com­par­i­son.

In­ter­views

In order to eval­u­ate my ap­proach on a the­o­ret­i­cal level I in­ter­viewed ex­perts from dif­fer­ent fields about their ex­pe­ri­ence from daily life when deal­ing with text and vi­su­al­i­sa­tion, how they as­sess my ap­proach and what this could mean for the fu­ture of vi­su­al­i­sa­tion.

Jan Dis­telmeyer

Prof. Dr. Jan Dis­telmeyer lec­tures at the Uni­ver­sity of Ap­plied Sci­ences Pots­dam for Europäische Me­di­en­wis­senschaft. In the in­ter­view I asked him about his as­sess­ment of my ap­proach. In re­gard to McLuhan’s »Guten­berg Galaxis« he points out that the book does not only in­flu­ence the lin­ear­ity of text, but also the uni­for­mity. Hand­writ­ing, which still stands as a sym­bol of the one’s own. But the unique has given way to the al­ways equally look­ing let­ter­press. Every book sud­denly looks the same. But McLuhan does not focus on the medium it­self. For him the medium is al­ways equate to tech­nol­ogy. What he is ac­tu­ally in­ter­ested in is the con­nec­tion be­tween the tech­nol­ogy and the cul­tural and so­cial im­pact it has. That is why he stresses not on the dis­ap­pear­ance of hand­writ­ing in terms of cul­tural loss, but rather the ef­fects. The let­ter­press al­lowed for a greater pro­duc­tion of books, al­low­ing all so­cial strata ac­cess knowl­edge. Dur­ing the French Rev­o­lu­tion the term « éga­lité » does not only re­flect the equal­ity of peo­ple but also the equal­ity in writ­ing: A sym­bol for ac­ces­si­bil­ity to books and the lib­er­a­tion from de­pen­dency. The no­ta­tional ap­pear­ance rep­re­sents the peo­ple.

An ex­am­ple in which this re­la­tion­ship be­tween type­face and hand­writ­ing is ad­dressed is one pro­ject named »Schön­schrift — No­ti­zen zur Hochkul­tur« (»Best Writ­ing — Notes on high cul­ture«). To­gether with re­views on var­i­ous sub­jects, the au­thors at­tached their hand writ­ten notes as PDF. The gen­e­sis of the text from in­di­vid­u­al­ity to uni­for­mity can be un­der­stood.

The next ques­tion was, how he as­sesses the re­la­tion­ship be­tween the ty­pog­ra­phy and the text. Dis­telmeyer states that we need to be aware of the far-reach­ing con­se­quences the fig­ure has for plain text we read. The fact that news­pa­pers de­fine what the ar­ti­cles should look like il­lus­trates this im­por­tance. But he em­pha­sises that not only the ty­pog­ra­phy, but also fac­tors like the brand-typ­i­cal style of writ­ing have in­flu­ences on how we per­ceive the con­tent. Many ac­com­pa­ny­ing symp­toms come with usage and form of the book, many are being de­scribed by Gérard Genette.

Dis­telmeyer also ex­plains that the com­bi­na­tion of text and graphic in fact does not can­cel the lin­ear­ity of the text, the same pre­scribed struc­ture of the au­thor. Even when these graph­ics con­tex­tu­alise, in­ter­rupt, or ac­tu­ally ques­tion the text, they are part of this for­mat. In con­trast to this, the gen­eral struc­ture of the web al­lows read­ers to in­di­vid­u­ally fol­low links, scroll or quit the web­site. This re­sults in a new form of per­cep­tion, with an struc­ture that is in­di­vid­ual to each reader.

An­swer­ing the ques­tion on how he eval­u­ates Nobert Bolz’ »Prägnanz« of im­ages, Dis­telmeyer states that Bolz’ media on­to­log­i­cal ap­proach can­not be ap­plied so gen­er­ally. He ex­plains that the progress the photo has made can now be seen with the di­a­gram. Prob­a­bly no one would nowa­days agree that a pho­to­graph shows the per­fect rep­re­sen­ta­tion of the re­al­ity it was praised for in the be­gin­ning. The ob­vi­ous­ness and fac­tu­al­ity that ta­bles or di­a­grams have today is about to face the same fate. The qual­ity a medium holds are more a cul­tural tra­di­tion of how we treat it. The prac­ti­cal value does not cor­re­late with the qual­ity of the table or graphic it­self.

The chang­ing usage and treat­ment of lan­guage through tech­nol­ogy is an ex­cit­ing progress. Even though the prin­ci­ples stay the same the dis­sem­i­na­tion and ap­pli­ca­tion dif­fers. In re­cent time, the evolve­ment of the PC brings in an­other in­ter­est­ing fac­tor: The com­puter does not care if the data rep­re­sents a text or a graphic. Luck­ily all the in­for­ma­tion are being processed for us to read them. But in fact, this re­sults in an even greater uni­for­mity. Pro­duc­ing, view­ing and dis­tri­b­u­tion all meet in one de­vice. The idea of « égalité » is even stronger when the print­ing house and broad­caster are in the hand of the pro­ducer. The ac­cep­tance of the peo­ple to mix dif­fer­ent media is thus be­com­ing even stronger. Only in re­cent time, this em­pow­er­ment was dis­cussed as a de­pri­va­tion of power through the sur­veil­lance en­abled through the elec­tronic pro­cess­ing.

Sascha Venohr

The »Head of Data Jour­nal­ism« for ZEIT On­line Sascha Venohr has worked at the in­ter­face be­tween jour­nal­ism and tech­nol­ogy for many years. In my in­ter­view, I asked him about the cur­rent state of on­line jour­nal­ism and vi­su­al­i­sa­tions. For him, the two most im­por­tant as­pects about ty­pog­ra­phy is the recog­ni­tion value and the leg­i­bil­ity. Peo­ple recog­nise the brand Die ZEIT On­line/Die ZEIT in the fonts, the lay­out and all the ty­po­graphic de­ci­sions. The as­pect of type/con­tent match­ing has to re­cede in favour of con­stant leg­i­bil­ity. In this tightly syn­chro­nised en­vi­ron­ment prac­ti­ca­bil­ity is a cru­cial fac­tor. The au­thor is also the de­signer when work­ing with the CMS that of­fers lim­ited ty­po­graphic op­tions like bold, italic, un­der­line and bul­let points. Ex­tended pos­si­bil­i­ties come with »raw mod­ules« that can be placed be­tween para­graphs and offer com­plete free­dom over the source code. These mod­ules do not in­ter­act with the sur­round­ing el­e­ments and also can­not de­mol­ish the lay­out.

Com­plex sto­ries that have been de­signed in the past and break out of this struc­ture gather a lot of at­ten­tion but are rather boon and bane. Every tech­ni­cal de­vi­a­tion en­tails ef­forts on the tech­ni­cal side. Venohr stresses the fact that on­line jour­nal­ism has to sup­ply con­tent for mul­ti­ple out­put de­vices: The web­site pro­vides views on dif­fer­ent screens, the app uses some of these lay­outs, while other con­tent is pulled out in a plain text-for­mat. This re­quires every ex­cep­tion from the com­mon for­mat to be de­fined for every type. Ba­si­cally, the ar­ti­cle needs to be medium-neu­tral, oth­er­wise it would re­quire more work by the au­thor. The di­ver­sity of de­vices forces the lay­out back to a form print lay­outs have had many years ago. Venohr states that from this per­spec­tive, print media have greater flex­i­bil­ity. In re­sponse to the ques­tion on whether the reader might not be able to un­der­stand com­plex , »vi­su­al­is­ing« lay­outs, he ad­mits that they are not ex­per­i­ment­ing enough. They rather un­der-chal­lenge the reader.

A ques­tion I was ask­ing my­self con­cern­ing the com­bi­na­tion of data vi­su­al­i­sa­tions and text is whether this could con­vey more »truth« than a (per­sua­sive) text. Venohr states that as a data-jour­nal­ist, he is also a story teller. Pre­sent­ing unedited, un­fil­tered amounts of data is not the jour­nal­ist’s job. By pick­ing in­ter­est­ing parts and not over­whelm­ing the reader the story be­comes in­ter­est­ing. He ex­plains that the data used in vi­su­al­i­sa­tions often lacks re­li­a­bil­ity. In clas­si­cal jour­nal­ism, sto­ries are re­quired to use at least two sources for in­for­ma­tion. Data-jour­nal­ists are de­pend­ing on data and yet the in­for­ma­tion used in many graph­ics is based on only one – often in trans­par­ent – source. For ex­am­ple, the un­em­ploy­ment rate is based on num­bers by the »Bun­de­sagen­tur für Ar­beit«. The ve­rac­ity can­not be ver­i­fi­able since no other source can pro­vide this data. Mis­takes dur­ing the pro­cess­ing can also occur. The more data the au­thor han­dles, the more likely it is to make mis­takes. Un­for­tu­nately, these faults can­not al­ways be no­ticed im­me­di­ately.

Sascha Venohr sees a great op­por­tu­nity to use vi­su­al­i­sa­tion to bring the story closer to the reader. In re­cent works they used the user’s input (for ex­am­ple the zip code) to cus­tomise the ap­pear­ance of the vi­su­al­i­sa­tion. Venohr stresses that in­ter­ac­tion is not only cus­tomi­sa­tion but can also be used to im­merse one­self in the story. It in­creases the »en­gage­ment«. The abil­ity to share a cus­tom view or test re­sults from the graphic (for ex­am­ple the hours of av­er­age sleep) with friends is more and more de­sired by the user and also re­sults in a higher num­ber of vis­i­tors through their shar­ing. But au­to­mat­i­cally pre­sent­ing dif­fer­ent data based on in­for­ma­tion re­ceived from the browser does not find a ready mar­ket. A re­cent ex­am­ple is a vi­su­al­i­sa­tion about the Eu­ro­vi­sion Song Con­test. Based on the user’s lan­guage set­tings the graphic was shown from a dif­fer­ent per­spec­tive. Peo­ple did not ex­pect that they in­flu­enced the »state of ag­gre­ga­tion« of the graphic and thus had dif­fi­cul­ties when shar­ing it or ex­chang­ing opin­ions about it. This re­quired the news­pa­per to built in in­for­ma­tion mes­sages with op­tions to change the view and thus mak­ing it more trans­par­ent for the reader. The same ap­plies for mo­bile and desk­top ver­sions of graph­ics that some­times show in­for­ma­tion in dif­fer­ent amounts of de­tail. Venohr states that we are not ready for this yet. Jour­nal­ists still need to fig­ure out how to deal with this pa­ter­nal­ism.

Jan Schwo­chow

The in­fo­graphic artist Jan Schwo­chow was head of de­part­ment in­fo­graphic for both Stern and Kircher­Burkhardt be­fore he founded his own agency Golden Sec­tion Graph­ics. For him, the cho­sen medium (may it be text, image, graphic, film or sound) has to suit the in­for­ma­tion. The tech­nique which, al­lows the reader to de­code the con­tent the quick­est and the most ef­fi­ciently, is cho­sen. This is of course a ques­tion of what ac­tu­ally can be trans­ported through an image. Text has dom­i­nated the knowl­edge trans­fer for a long time, but not all in­for­ma­tion are equally easy to com­mu­ni­cate lin­guis­ti­cally. Lengthly de­scrip­tions of vot­ing re­sults for ex­am­ple are much eas­ier to de­scribe in a graphic. Schwo­chow wishes for as lit­tle text as nec­es­sary. On the other hand, he ad­mits that text is some­times ir­re­place­able. As la­bels, leg­ends or ex­pla­na­tion text makes the graphic un­der­stand­able.

In gen­eral, he wishes for more im­ages in all kinds of media. One ex­am­ple is the pop­u­lar sci­ence book »Darm mit Charme« by Giu­lia En­ders. The text is un­doubt­edly well writ­ten but Schwo­chow wishes for bet­ter graph­ics. The il­lus­tra­tions in the book may be charm­ing and funny, but do not add any value to the con­tent. They sim­ply rep­re­sent the con­tent in an il­lus­tra­tive way. Schwo­chow sees a great op­por­tu­nity to offer vi­su­al­i­sa­tions that would allow the reader for a deeper im­merse. This could help to trans­port more de­tailed in­for­ma­tion about top­ics in the book.

One rea­son why es­pe­cially daily news­pa­pers do not use more graph­ics is a lack of time. Con­tent needs to be pro­duced quickly, but many graph­ics still need more time to be pro­duced. More­over, the en­vi­ron­ment in which the con­tent is con­sumed is im­por­tant. Many peo­ple read news­pa­pers on the go on the their mo­bile phone. Large, ex­plana­tory graph­ics do not suit the small dis­play and the read­ing be­hav­iour. Schwo­chow sees a great op­por­tu­nity for Sparklines to es­pe­cially work in this en­vi­ron­ment. The Apple Watch could po­ten­tially even in­crease this. These small graph­ics are quicker to pro­duce and thus would allow ap­pli­ca­tion even in time crit­i­cal pro­jects. They also lead to a dif­fer­ent sto­ry­telling. While big­ger graph­ics allow for ex­plo­ration, micro vi­su­al­i­sa­tions are rather ex­plana­tory. He says that these two types could prospec­tively be the dif­fer­ence be­tween mo­bile and desk­top usage. The con­text and hence the com­mu­ni­ca­tion goal de­fines the ap­pear­ance of the graphic in the fu­ture. This can al­ready be seen in the in­ter­ac­tive graphic »Lob­by­ing in Eu­rope« for Trans­parency In­ter­na­tional (I was in­volved in the pro­duc­tion of this graphic). The coun­tries can be ex­plored on a map on the desk­top ver­sion and in a sim­ple list on mo­bile phones. The map would not allow the user to sat­is­fac­to­rily nav­i­gate the graphic on a small screen and hence he can use a sim­pli­fi­ca­tion of the graphic.

Ty­pog­ra­phy being used for vi­su­al­i­sa­tion is one thing we def­i­nitely see more in the fu­ture, says Schwo­chow. He il­lus­trates the in­tel­li­gent usage of ty­pog­ra­phy in car­tog­ra­phy: Every font-size, style and po­si­tion­ing has a mean­ing. While ty­pog­ra­phy was a cru­cial part of the cor­po­rate de­sign until now, these bound­aries soften more and more in the fu­ture. We can al­ready see that there are only »news«, which were sug­gested by our friends. The au­thor be­comes more and more ir­rel­e­vant. For the brand this means a shift in their po­si­tion­ing. Tesla is a good ex­am­ple of a com­pany that is al­ready doing this. The idea (the elec­tric car) is more im­por­tant than the ap­pear­ance. They rather focus on the emo­tion: They de­scribe it as »Tesla mo­ment« when one sits in a car that ac­cel­er­ates quicker than most Porsche’s. What they com­mu­ni­cate is au­then­tic­ity and this strat­egy has be­come the brand. Schwo­chow says that we should take away the ty­pog­ra­phy from the CD to com­mu­ni­cate ac­tual con­tent. Based on this de­vel­op­ment he pre­dicts a big op­por­tu­nity for »vi­su­al­i­sa­tion« ty­pog­ra­phy.

Con­clu­sion

In sum­mary, this the­sis finds con­vinc­ing sup­port for a com­ple­men­tar­ity of text and vi­su­al­i­sa­tions. Ar­gu­ments for the in­te­gra­tion of text and vi­su­al­i­sa­tions in terms of micro vi­su­al­i­sa­tions have been made on the­o­ret­i­cal grounds by show­ing short­cuts in ex­ist­ing the­o­ries as well as by point­ing out com­pelling ben­e­fits of ap­ply­ing such an in­te­grated ap­proach. Also ex­ist­ing re­search and ex­pert opin­ions back up this po­tency as well as they point to an en­cour­ag­ing wide range of pos­si­ble ap­pli­ca­tions and pur­poses. In sum­mary, the ques­tion how Micro Vi­su­al­i­sa­tions can en­hance text com­pre­hen­sion, mem­o­ra­bil­ity, and ex­ploita­tion could be an­swered with mul­ti­ple ex­am­ples.

The pro­posed tax­on­omy has proven as an use­ful method­olog­i­cal frame­work for as­sess­ing the state of the art in Micro Vi­su­al­i­sa­tions. Ap­proaches of both, mod­i­fi­ca­tion and ad­di­tion, have been shown to be dis­tin­guished by in­di­vid­ual qual­i­ties, lim­i­ta­tions, and pur­poses. There­with, the tax­on­omy al­lows for a novel per­spec­tive on the ap­pli­ca­tion of vi­su­al­i­sa­tion which may flour­ish even be­yond the out­lined fu­ture ap­proaches.

In the fol­low­ing the con­tri­bu­tions of each part are listed.

Con­tri­bu­tions

The over­ar­ch­ing goal of this work was to sub­stan­ti­ate the con­cept of micro vi­su­al­i­sa­tions. In­stead of in­vent­ing a new type of vi­su­al­i­sa­tion this the­sis fo­cused on a new type of ap­pli­ca­tion of ex­ist­ing work. Es­pe­cially in re­gard to ty­pog­ra­phy the ques­tion was »What can we do with ex­ist­ing tools?«

Since it is a new ap­proach in the field of vi­su­al­i­sa­tion this work mainly laid the the­o­ret­i­cal foun­da­tion with a media the­o­ret­i­cal de­vi­a­tion, a struc­tur­ing tax­on­omy, and an elab­o­ra­tion of the cur­rent state, and thus aids as a start­ing point for fu­ture prac­ti­cal work. The the­sis led to four main types of con­tri­bu­tions: ev­i­dence, con­cept, re­flec­tion, and out­look. One part that is cer­tainly miss­ing in this work are con­crete de­sign stud­ies that use em­pir­i­cal tests to show ev­i­dence.

De­riva­tion as ev­i­dence

The de­riva­tion that leads to the con­cept of micro vi­su­al­i­sa­tion is both di­rected from a media the­o­ret­i­cal side and a de­sign/vi­su­al­i­sa­tion per­spec­tive. This multi-view as­pect is also pre­sent in the in­ter­views in which ex­perts from dif­fer­ent dis­ci­plines were asked for their as­sess­ment and ex­pe­ri­ence with the topic.

The analy­sis of these re­sources lead to a con­fi­dence in the ac­tual func­tion­ing is based on sci­en­tific stud­ies from both psy­cho­log­i­cal and de­sign re­search. Es­pe­cially the mod­els of dual cod­ing and cog­ni­tive load sup­port the pro­posed ap­proach.

Tax­on­omy as con­cept

The tax­on­omy pro­posed in this the­sis aims on the type of ap­pli­ca­tion a vi­su­al­i­sa­tion has rather than its ac­tual ap­pear­ance. This al­lows for a wider cur­rency among the data vi­su­al­i­sa­tion com­mu­nity as it can be stretched to other fields. In re­gard to ex­ist­ing tax­onomies that focus on the type or vi­sual vari­able, the sug­gested sys­tem acts as meta-tax­on­omy, in which cur­rent sys­tems can also fit in. The dis­tinc­tion be­tween mod­i­fy­ing and adding vi­su­al­i­sa­tions, be­tween in­te­grated and ad­ja­cent place­ment al­lows to find new com­bi­na­tions, new util­i­sa­tion, and new in­ter­ac­tions of known graph­ics. In the new field of micro vi­su­al­i­sa­tions this func­tional tax­on­omy helps to struc­ture ex­ist­ing pro­jects and re­veal white spots that have not been elab­o­rated yet.

State of the art re­port as re­flec­tion

Since many con­cepts, pro­jects, and graph­ics that can be re­garded as micro vi­su­al­i­sa­tions al­ready exist, one main con­tri­bu­tion of this work is a state of the art re­port (STAR). This con­cept of re­view is used in the vi­su­al­i­sa­tion com­mu­nity to cap­ture the great amount of work that is hap­pen­ing in the dif­fer­ent areas of the sub­ject. These re­ports are es­pe­cially use­ful since they pro­vide an overview of ex­ist­ing view for both ex­ten­sion and im­prove­ment. To prove the con­cept of the tax­on­omy, the ex­ist­ing work has been re­viewed in the same struc­ture. But not only prac­ti­cal pro­jects have been elab­o­rated but also the the­o­ret­i­cal re­search that is re­lated to this ap­proach is cov­ered in the de­riva­tion and al­lows con­tin­u­a­tion.

Fu­ture work as out­look

Based on these three parts I pro­vide an out­look of fu­ture work that needs to be elab­o­rated on the basis of this the­sis. Through the analy­sis of re­cent and cur­rent work this the­sis re­vealed mul­ti­ple start­ing points for fu­ture work. Es­pe­cially the tax­on­omy can act as tool to iden­tify not yet re­alised pro­jects. The most im­por­tant as­pects are listed in the fol­low­ing part.

Fu­ture Re­search

Em­pir­i­cal re­search

The sci­en­tific re­search from re­lated top­ics sug­gests ev­i­dence for the use of micro vi­su­al­i­sa­tions. Reg­u­lar vi­su­al­i­sa­tions have been proven to be ef­fec­tive for com­mu­ni­cat­ing in­for­ma­tion, the text’s sta­tus is undis­put­edly im­por­tant for knowl­edge trans­fer and even the com­bi­na­tion of im­ages and text has been eval­u­ated. But very lit­tle em­pir­i­cal re­search has been con­ducted that ac­tu­ally sup­ports this the­sis’ ap­proach. Per­son­ally, I see the most ur­gent ques­tions in the com­pre­hen­si­bil­ity of these text/graph­ics-doc­u­ments, the change of the read­ing flow and be­hav­iour and in the long-term ef­fects, es­pe­cially on how mem­o­rable the text’s con­tent is.

But also the at­ti­tude of the com­mon reader to­wards such ques­tion­able en­hance­ments needs to be tested. While some vi­su­al­i­sa­tions can in­con­spic­u­ously be in­te­grated and adapted, other (es­pe­cially in­te­grated) changes could ini­tially dis­tract the reader and not be recog­nised as en­hance­ment or as vi­su­al­i­sa­tion at all. The chal­lenge is to find com­pro­mises be­tween the ac­cepted and learned form of text and im­ages and ben­e­fi­cial, sub­lim­i­nal and learn­able com­bi­na­tions.

Text-Bild-Wis­sen­schaf­ten

In his plead­ing »Nar­ra­tive im­ages and pic­to­r­ial lit­er­a­ture — Plead­ing for a text-im­age-cul­ture« (Er­zäh­len­de Bil­der und bild­haf­te Li­te­ra­tur — Plä­doy­er für eine Text-Bild­wis­sen­schaft, 2006) Horst Wen­zel dis­cusses the his­tory of the com­bi­na­tion of im­ages and text with a focus on me­dieval il­lus­tra­tions. While his idea of im­ages cer­tainly dif­fers from the vi­su­al­i­sa­tions dis­cussed in this the­sis, his plead­ing is nonethe­less ap­plic­a­ble and use­ful for the de­bated im­ages here. He crit­i­cises the sys­tem­atic sep­a­ra­tion of the sci­ence of art and of lit­er­a­ture. As he states this de­tach­ment has never been dog­mat­i­cally cul­ti­vated by the artists like Leonardo da Vinci, Al­brecht Dürer, Al­fred Kubin or Günther Grass them­selves but in­stead was held up by the aca­d­e­mics. Not only do the de­scribed pro­jects with a strong amal­ga­ma­tion of im­ages and text ver­ify this state­ment, but also the con­cepts like con­crete po­etry or Fat­Fonts that can­not be clearly sep­a­rated as text or image any­more dis­pute the old clas­si­fi­ca­tion. De­spite this being a niche ap­pli­ca­tion com­pared to purely tex­tual or pic­to­r­ial forms, the re­search of the com­bi­na­tion holds po­ten­tial for the ris­ing merg­ing in the fu­ture.

But not only the gen­eral vi­sual cul­ture should be dis­cussed in media stud­ies, but the only re­cently emerged field of »di­a­gram­matic« should be paid par­tic­u­lar at­ten­tion. This gram­mar of di­a­grams (in the broader de­f­i­n­i­tion of rep­re­sen­ta­tion with graph­i­cal means) dis­cusses men­tal and ma­te­ri­al­is­tic di­a­grams, the­o­ries of knowl­edge, and also cul­tural and media the­o­ret­i­cal as­pects of vi­su­al­i­sa­tion. (Bauer et al., 2010) With the re­cently pro­claimed era of big data (Econ­o­mist, 2010) the »tools« which make the su­per­abun­dant amount of in­for­ma­tion man­age­able in a vi­sual way need to be fur­ther de­vel­oped.

Tech­no­log­i­cal change

As tech­nol­ogy evolves, both text and vi­su­al­i­sa­tions need to adapt to new en­vi­ron­ments and be­hav­ioural pat­terns. Dif­fer­ent dis­play sizes for ex­am­ple are still chal­leng­ing de­sign­ers when cre­at­ing graph­ics. Micro vi­su­al­i­sa­tions ap­pear to be one so­lu­tion when deal­ing with vary­ing lay­outs. Since in­te­grated graph­ics can be mostly treated as text, the re­spon­sive de­sign ap­plied to text could be used in a sim­i­lar way. But also ap­pli­ca­tions that are placed in a small size per se – as in smart­watches or as a sin­gle graph­i­cal in­ter­face el­e­ment – can ben­e­fit from this knowl­edge.

Al­ter­na­tively, micro vi­su­al­i­sa­tion could be seen as the rep­re­sen­ta­tion of reg­u­lar graph­ics for smaller screens. As the dis­play be­comes smaller, reg­u­lar vi­su­al­i­sa­tions are taken apart and po­si­tioned along their ref­er­ence ob­jects. For this, not only the ques­tion of re­spon­sive de­sign but also of re­spon­sive con­tent could be­come fer­tile.

Tech­no­log­i­cal change also brings new pos­si­bil­i­ties of re­al­i­sa­tion. Frame­works like jQuery, D3, and Skrollr al­ready changed the way web­sites and vi­su­al­i­sa­tions are pro­grammed. The jQuery plu­gin by Gareth Watts al­ready al­lows nu­mer­ous data­word-graph­ics like Sparklines. Many of the dis­cussed tech­niques can al­ready be re­alised through cur­rent tech­nol­ogy. When the field of micro vi­su­al­i­sa­tions gains more at­ten­tion, frame­works specif­i­cally cre­ated for this type of graph­ics are de­bat­able.

Clos­ing Re­marks

Through the course of the work I have gained strong con­fi­dence in the ex­pe­di­ency of the out­lined ap­proach. Also, the ex­pert in­ter­views have con­firmed the the­sis’ prac­ti­cal rel­e­vance. This the­sis has aimed to pro­vide a start­ing point in this novel field. In all mod­esty, the out­lined ar­gu­ments and ev­i­dence promise to con­sti­tute a sub­stan­tial basis on which fu­ture re­search can build.

As a de­signer, the prod­ucts one cre­ates are in most cases also rel­e­vant to one’s own life. Ac­com­plish­ing this work, I wished the changes I pro­pose herein would al­ready have been re­alised to fa­cil­i­tate my re­search. Es­pe­cially with today’s tech­no­log­i­cal pos­si­bil­i­ties there are un­seen op­por­tu­ni­ties to cre­ate novel media of knowl­edge. Books have come a long way; it is time to take them to the next level — in any form pos­si­ble.

Ap­pen­dix

Bib­li­og­ra­phy

  • Alexan­der, Eric; Kohlmann, Joe; Valenza, Robin; Wit­more, Michael; Gle­icher, Michael (2014) Serendip: Topic Model-Dri­ven Vi­sual Ex­plo­ration of Text Cor­pora. IEEE Sym­po­sium on Vi­sual An­a­lyt­ics Sci­ence and Tech­nol­ogy 2014 No­vem­ber 9-14, Paris, France
  • Ass­mann, Prof. Dr. Jan (2002): Frühzeit des Bildes: Der Iconic Turn im Alten Ägypten, Iconic Turn - Felix Burda Memo­r­ial Lec­tures: Hu­bert Burda Stiftung. https://​www.​youtube.​com/​watch?​v=-​TGTN4NFYWw (Re­trieved 2015-06-08)
  • Bach­mann-Medick, Doris (2008): Gegen Worte – Was heißt ›Iconic/Vi­sual Turn‹? In Gegen­worte - Vi­su­al­isierung oder Vi­sion? Bilder (in) der Wis­senschaft. 20. Heft Herbst 2008 http://​bachmann-​medick.​de/​wp-​content/​uploads/​2012/​10/​Bachmann-​Med​ick%20I​coni​c%20T​urn:​Gegenworte.​pdf (Re­trieved 2015-05-24)
  • Bauer, Matthias; Ernst, Christoph (2010): Di­a­gram­matik - Einführung in ein kul­tur- und me­di­en­wis­senschaftliches Forschungs­feld. 1. Au­flage. ran­script Ver­lag, Biele­feld
  • Bertin, Jacques (1983): Semi­ol­ogy of Graph­ics: Di­a­grams, Net­works, Maps. Uni­ver­sity of Wis­con­sin Press, Wis­con­sin
  • Boehm, Got­tfried (2007): Iconic Turn. Ein Brief. In Belt­ing, Hans (Pub­lisher): Bilder­fra­gen: Die Bild­wis­senschaften im Auf­bruch, pages 27–36. Wil­heml Fink Ver­lag, München
  • Boehm, Got­tfried (2014): Bildbeschrei­bung. Über die Gren­zen von Bild und Sprache. In Müller, M. R. et al. (Pub­lisher): Gren­zen der Bild­in­ter­pre­ta­tion, Wis­sen, Kom­mu­nika­tion und Gesellschaft, pages 15-24. Springer Fachme­dien Wies­baden 2014
  • Bolz, Nor­bert (2012): Nach­haltigkeit im In­for­ma­tion­sraum. See-Con­fer­ence #7. https://​vimeo.​com/​41987401 (Re­trieved 2015-05-23)
  • Borgo, R.; Kehrer, J.; Chung, D. H. S.; Maguire, E.; Laramee, R. S.; Hauser, H.; Ward, M.; and Chen, M. (2013): Glyph-based Vi­su­al­iza­tion: Foun­da­tions, De­sign Guide­lines, Tech­niques and Ap­pli­ca­tions - STAR – State of The Art Re­port: Eu­ro­graph­ics 2013
  • Bovet, Philippe; Rekacewicz, Philippe; Sinai, Agnès; Vidal, Do­minique (2008): Le Monde diplo­ma­tique: Atlas der Glob­al­isierung Spezial - Klima. »Le Monde diplo­ma­tique«/taz Ver­lags- und Ver­triebs GmbH, Berlin
  • Bret Vic­tor (2011): Ex­plorable Ex­pla­na­tions. http://​worrydream.​com/​Exp​lora​bleE​xpla​nati​ons/ (Re­trieved 2015-06-08)
  • Bringhurst, Robert (2012): The El­e­ments of Ty­po­graphic Style. Fourth Edi­tion (ver­sion 4.0). Hart­ley & Marks, Seat­tle, Van­cou­ver
  • Broekhuizen, Bas (2014): How to in­te­grate text and im­ages in­ter­ac­tively. http://​www.​bas​broe​khui​zen.​nl/​2014/​03/​12/​how-​to-​integrate-​text-​and-​images-​with-​int​erac​tivi​ty/ (Re­trieved 2015-06-08)
  • Brooks, Evan (2014): Cod­ing in color. https://​medium.​com/@​evnbr/​coding-​in-​color-​3a6db2743a1e (Re­trieved 2015-05-09)
  • Bur­dick, Anne; Drucker, Jo­hanna; Lunen­feld, Peter; Pres­ner, Todd; Schnapp, Jef­frey (2012): Dig­i­tal_Hu­man­i­ties. Mass­a­chu­setts In­sti­tute of Tech­nol­ogy, Mass­a­chu­setts
  • Cairo, Al­berto (2013): The Func­tional Art - An In­tro­duc­tion To In­for­ma­tion Graph­ics. New Rid­ers, Berke­ley
  • Card, S. K.; Mackin­lay, J. D.; Shnei­der­man, B. (1999): Read­ings in In­for­ma­tion Vi­su­al­iza­tion: Using Vi­sion to Think. Aca­d­e­mic Press, San Diego
  • Cas­tro, Ricky (with­out date): Sol Le Witt Lec­ture. http://​www.​visible.​org/​site/​cornish/​slide_​lectures/​sol_​le_​witt/ (Re­trieved 2015-06-08)
  • Chen, Min; Floridi, Lu­ciano (2012): An analy­sis of in­for­ma­tion vi­su­al­i­sa­tion. Springer Sci­ence+Busi­ness Media Dor­drecht
  • Childers, Terry L.; Jass, Jef­frey (2002): All Dressed Up With Some­thing to Say: Ef­fects of Type­face Se­man­tic As­so­ci­a­tions on Brand Per­cep­tions and Con­sumer Mem­ory. In Jour­nal Of Con­sumer Psy­chol­ogy, 12(2). Pages 93–106. Lawrence Erl­baum As­so­ci­ates
  • Co­bett, Gre­ville G. (2000): Num­ber. Cam­bridge Uni­ver­sity Press, Cam­bridge
  • Con­tro­pe­dia (2015) http://​contropedia.​net/ (Re­trieved 2015-05-08)
  • Davies, Jason (2007): Word Trees. https://​www.​jasondavies.​com/​wordtree/ (Re­trieved 2015-06-08)
  • de Groot, Luc(as): Case Study: Jun­gle World. http://​www.​lucasfonts.​com/​case-​studies/​jungle-​world/ (Re­trieved 2015-08-08)
  • de Roda, Catalina Viejo Lopez (with­out date): Let­ter Col­lages - Car­tas Col­lage http://​www.​cat​alin​avie​jo.​com/​col​lage​menu​lett​ers.​htm (Re­trieved 2015-06-08)
  • DeR­ouchey, Bill (2010): His­tory of the But­ton. SXSW Talk on March 12, 2010
  • Di­Cioc­cio, Lau­ren (with­out date): http://​lau​rend​icio​ccio.​com/​paintings- (Re­trieved 2015-06-08)
  • Didi-Hu­ber­man, Georges (2007): Bilder trotz allem. Trans­lated by Peter Geimer. Wil­helm Fink Ver­lag, München
  • Drucker, Jo­hanna (2014): Graph­e­sis - Vi­sual Forms of Knowl­edge Pro­duc­tion. Har­vard Uni­ver­sity Press, Har­vard
  • Duarte, Nancy (2010): Res­onate - Pre­sent Vi­sual Sto­ries That Trans­form Au­di­ence, John Wiley & Sons, New York
  • E. Borra, D. La­ni­ado, E. Wel­tevrede, M. Mauri, G. Magni, T. Ven­turini, P. Ciuc­carelli, R. Rogers, and A. Kaltenbrun­ner (2015): A Plat­form for Vi­su­ally Ex­plor­ing the De­vel­op­ment of Wikipedia Ar­ti­cles, ICWSM ‘15 - Pro­ceed­ings of the 9th In­ter­na­tional AAAI Con­fer­ence on Web and So­cial Media
  • Eames, Charles and Ray (1977): Pow­ers of Ten https://​www.​youtube.​com/​watch?​v=0fK​BhvD​juy0 (Re­trieved 2015-05-07)
  • Econ­o­mist News­pa­per Lim­ited, The (2010): Data, data every­where – A spe­cial re­port on man­ag­ing in­for­ma­tion. Feb­ru­ary 27th 2010
  • Few, Stephen (2014): Data Vi­su­al­iza­tion for Human Per­cep­tion. In Soe­gaard, Mads and Dam, Rikke Friis (Pub­lisher). The En­cy­clo­pe­dia of Hu­man-Com­puter In­ter­ac­tion, 2nd Edi­tion. Aarhus, Den­mark: The In­ter­ac­tion De­sign Foun­da­tion. https://​www.​interaction-​design.​org/​enc​yclo​pedi​a/​data_​vis​uali​zati​on_​for_​human_​perception.​html (Re­trieved 2015-06-08)
  • Fred­heim, Rolf (2013): Vi­su­al­is­ing Struc­ture in Topic Mod­els. http://​qua​ntif​ying​memo​ry.​blogspot.​de/​2013/​11/​visualising-​structure-​in-​topic-​models.​html (Re­trieved 2015-06-08)
  • Frutiger, Adrian (1998): Der Men­sch und seine Ze­ichen - Schriften, Sym­bole, Signete, Sig­nale. 6. Au­flage: Fourier Ver­lag HmbH, Wies­baden
  • Fry, Ben (2009) On the Ori­gin of Species: The Preser­va­tion of Favoured Traces. http://​benfry.​com/​traces/ (Re­trieved 2015-05-08)
  • Gen­del, Ido; Bau­mann, Benny (2010): A brief overview on Syn­tax High­light­ing. http://​blog.​benny-​baumann.​de/?​p=766 (Re­trieved 2015-05-09)
  • Gof­fin, Pas­cal; Wil­lett, Wes­ley; Fekete, Jean-Daniel; Isen­berg, Petra (2014): Ex­plor­ing the Place­ment and De­sign of Word-Scale Vi­su­al­iza­tions. IEEE Trans­ac­tions on Vi­su­al­iza­tion and Com­puter Graph­ics, In­sti­tute of Elec­tri­cal and Elec­tron­ics En­gi­neers (IEEE)
  • Gold­stein, E. Bruce (2002): Wahrnehmungspsy­cholo­gie. Spek­trum Akademis­cher Ver­lag, Hei­del­berg, Berlin
  • Groeger, Lena; Cur­rier, Cora (2012): Stack­ing Up the Ad­min­is­tra­tion’s Drone Claims. ProP­ub­lica Sept. 13, 2012. http://​projects.​propublica.​org/​graphics/​cia-​drones-​strikes (Re­trieved 2015-06-08)
  • Haus­mann, Al­brecht (2009): Zukunft der Guten­berg-Galaxis. In Aus Poli­tik und Zeit­geschichte (APuZ 42-43/2009): Zukunft des Buches, pages 32-39. Bun­deszen­trale für poli­tis­che Bil­dung, Bonn
  • Haverkamp, Michael (2009): Synästhetis­ches De­sign: kreative Pro­duk­ten­twick­lung für alle Sinne. Carl Hanser Ver­lag GmbH, München
  • Heer, Jef­frey; Bo­stock, Michael; Ogievet­sky, Vadim (2010): A tour through the vi­su­al­iza­tion zoo. Com­mu­ni­ca­tions of the ACM, 53(6):59–67
  • Heller, Eva (2009): Wie Far­ben wirken – Farbpsy­cholo­gie, Farb­sym­bo­lik, Kreative Far­bgestal­tung. 5th Edi­tion. Rowohlt Ver­lag GmbH, Ham­burg
  • Her­rmann, Ralf (2010): Das Zwiebelschicht­en­mod­ell der Les­barkeit. In Ty­po­Jour­nal 2: Wayfind­ing & Les­barkeit. Pages 2–8. Seite7 Des­ig­na­gen­tur, Jena
  • Her­t­erich, Owen (2014) To See & Hear. http://​owe​nher​teri​ch.​com/​projects/​dialogue/ (Re­trieved 2015-06-08)
  • Hof­maier, Mike (2013): Ver­fas­sung ver­ste­hen: Das Grundge­setz in In­fo­grafiken Eine vi­suelle Analyse der deutschen Ver­fas­sung. Her­mann Schmidt Ver­lag, Mainz
  • Imhof, Ed­uard (2007): Car­to­graphic re­lief pre­sen­ta­tion. Esri Press, Red­lands
  • Kittsteiner, Heinz D. (2004): »Iconic turn« und »in­nere Bilder« in der Kul­turgeschichte. In: Kittsteiner, Heinz D.: Was sind Kul­tur­wis­senschaften? 13 Antworten. Pages 153–182 Pader­born 2004: Wil­helm Fink Ver­lag, München
  • Klein, Scott (2014): The for­got­ten His­tory of News Graph­ics. At Mal­ofiej 22, Pam­plona
  • Kochel, Travis (2012): How to Use FF Chartwell. https://​www.​fontfont.​com/​how-​to-​use-​ff-​chartwell (Re­trieved 2015-06-08)
  • Koller, An­dreas (2014): Fontas­tic. http://​code.​and​reas​koll​er.​com/​libraries/​fontastic/ (Re­trieved 2015-06-08)
  • Kosara, Robert (2007): A Cri­tique of Cher­noff Faces. https://​eagereyes.​org/​criticism/​chernoff-​faces (Re­trieved 2015-06-08)
  • Krämer, Sybille (2003): »Schrift­bildlichkeit« oder: Über eine (fast) vergessene Di­men­sion der Schrift. In Krämer, Sybille; Bre­dekamp, Horst: Bild Schrift Zahl. Pages 157–176. Wil­helm Fink Ver­lag, München
  • Lid­well, William; Holden, Kritina; But­ler, Jill (2010): Uni­ver­sal Prin­ci­ples of De­sign. Rock­port Pub­lish­ers, Mass­a­chu­setts
  • Malet­zke, Ger­hard (1998): Kom­mu­nika­tion­swis­senschaft im Überblick: Grund­la­gen, Prob­leme, Per­spek­tiven. West­deutscher Ver­lag GmbH, Opladen/Wies­baden
  • McAghon, Michael (2009): An Imag­i­na­tion Ex­pe­ri­ence. http://​blog.​mcaghon.​com/​post/​98471242041/​an-​imagination-​experience (Re­trieved 2015-05-25)
  • Mc­Cloud, Scott (2001): Comics richtig lesen - Die un­sicht­bare Kunst. Carlsen Comics, Ham­burg
  • Mc­Crickard, D. Scott; Catram­bone, Richard (1999): Be­yond the Scroll­bar: An Evo­lu­tion and Eval­u­a­tion of Al­ter­na­tive Nav­i­ga­tion Tech­niques, Geor­gia In­sti­tute of Tech­nol­ogy, Geor­gia
  • McLuhan, Marschall (2011): Das Medium is die Mas­sage - Ein In­ven­tar me­di­aler Ef­fekte. With Quentin Fiore. 3rd Edi­tion. Com­piled by Agel, Jerome (Pub­lisher). Tropen Sach­buch, Leipzig
  • McLuhan, Mar­shall (1969). In­ter­view in Hefner, Hugh (Pub­lisher): Play­boy Vol. 16, no. 3 – March 1969
  • Mendelssohn, Moses (1989): Jerusalem oder über re­ligiöse Macht und Ju­den­tum. In Mendelssohn, Moses: Schriften über Re­li­gion und Aufklärung. Pages 421f. Pub­lished by Thom, M., Darm­stadt
  • Mendelssohn, Moses (2011): Jerusalem – Re­li­gious Power and Ju­daism. Pub­lished by Ben­nett, Jonathan. Early Mod­ern Texts. http://​www.​ear​lymo​dern​text​s.​com/​pdfs/​men​dels​sohn​1782.​pdf (Re­trieved 2015-06-08)
  • Miyazaki, Momo; Karam­belkar, Manas; Robert­sen, Ken­neth Alek­sander (2012): Si­lenc. http://​ciid.​dk/​education/​portfolio/​idp12/​courses/​data-​vis​uali​sati​on/​projects/​silenc/ (Re­trieved 2015-05-08)
  • Mot­ley, Michael T. (1990): On whether one can(not) not com­mu­ni­cate: An ex­am­i­na­tion via tra­di­tional com­mu­ni­ca­tion pos­tu­lates. In West­ern Jour­nal of Speech Com­mu­ni­ca­tion, 54 (Win­ter 1990). Pages 1–20
  • Müsseler, Jochen; Prinz, Wolf­gang (Pub­lisher) (2002): All­ge­meine Psy­cholo­gie. 1. Au­flage. Spek­trum Akademis­cher Ver­lag, Hei­del­berg, Berlin
  • Myers, Brad; Hud­son, Scott E., Pausch, Randy (2000): Past, Pre­sent, and Fu­ture of User In­ter­face Soft­ware Tools. Human Com­puter In­ter­ac­tion In­sti­tute, School of Com­puter Sci­ence, Carnegie Mel­lon Uni­ver­sity, Pitts­burgh
  • Na­centa, Miguel; Hin­richs, Uta; Carpen­dale, Shee­lagh (2012): Fat­Fonts: Com­bin­ing the Sym­bolic and Vi­sual As­pects of Num­bers. Con­fer­ence: Pro­ceed­ings of the In­ter­na­tional Work­ing Con­fer­ence on Ad­vanced Vi­sual In­ter­faces
  • Noirhomme-Frai­ture, Monique; Ran­do­let, Frédéric; Chit­taro, Luca; Custinne Grégory (2005): Data Vi­su­al­iza­tions on small and very small screens. In ASMDA: Pro­ceed­ings of Ap­plied Sto­chas­tic Mod­els and Data Analy­sis
  • Pot­ter, Kristin; Rosen, Paul; John­son, Chris R. (2012): From Quan­tifi­ca­tion to Vi­su­al­iza­tion: A Tax­on­omy of Un­cer­tainty Vi­su­al­iza­tion Ap­proaches. Sci­en­tific Com­put­ing and Imag­ing In­sti­tute, Uni­ver­sity of Utah
  • Prop­ub­lica (2012): State­Face. http://​propublica.​github.​io/​stateface/ (Re­trieved 2015-06-08)
  • Re­ichen­stein, Oliver (2012): Re­spon­sive Ty­pog­ra­phy: The Ba­sics http://​ia.​net/​know-​how/​responsive-​typography-​the-​basics (Re­trieved 2015-05-08)
  • Reuß, Roland (2014): Die per­fekte Lese­mas­chine - Zur Er­gonomie des Buches. Ästhetik des Buches. Wall­stein Ver­lag, Göttin­gen
  • Santa Maria, Jason (2014): On Web Ty­pog­ra­phy. A Book Apart, New York
  • Schmidt, Gavin (2014): The emer­gent pat­terns of cli­mate change. TED Talk. http://​www.​ted.​com/​talks/​gavin_​schmidt_​the_​emergent_​patterns_​of_​climate_​change/​transcript?​language=en#​t-​62460 (Re­trieved 2015-05-07)
  • Shnei­der­man, Ben (1996): The Eyes Have It: A Task by Data Type Tax­on­omy for In­for­ma­tion Vi­su­al­iza­tions. De­part­ment of Com­puter Sci­ence Hu­man-Com­puter In­ter­ac­tion Lab­o­ra­tory, and In­sti­tute for Sys­tems Re­search Uni­ver­sity of Mary­land
  • Stein­berger, Markus; Wald­ner, Manuela; Streit, Marc; Lex, Alexan­der; Schmal­stieg, Di­eter (2011): Con­text-Pre­serv­ing Vi­sual Links. IEEE Trans­ac­tions On Vi­su­al­iza­tion And Com­puter Graph­ics, Vol. 17, No. 12, De­cem­ber 2011
  • Stephan Thiel (2010): Un­der­stand­ing Shake­speare. http://​www.​und​erst​andi​ng-​shakespeare.​com/ (Re­trieved 2015-06-08)
  • Stern, Joanna (2015): How I Learned to Love Writ­ing With Emo­jis. The Wall Street Jour­nal. 19.05.2015 http://​graphics.​wsj.​com/​how-​i-​learned-​to-​love-​writing-​with-​emojis/ (Re­trieved 2015-05-23)
  • Stet­ter, Chris­t­ian (2005): Bild, Di­a­gramm, Schrift. In Grube, Ger­not; Kogge, Werner; Krämer, Sybille (Ed­i­tors): Schrift. Kul­turtech­nik zwis­chen Auge, Hand und Mas­chine. Pages 115–135. Wil­helm Fink Ver­lag, München
  • Stevens, S. S. (1946): On the The­ory of Scales of Mea­sure­ment. In Sci­ence Vol. 103, No. 2684
  • The Uni­code Con­sor­tium (2015): Emoji and Ding­bats http://​www.​unicode.​org/​faq/​emoji_​dingbats.​html#​1 (Re­trieved 2015-06-08)
  • Todor­ovic, Dejan (2008): Gestalt prin­ci­ples. Schol­ar­pe­dia, 3(12):5345. http://​www.​sch​olar​pedi​a.​org/​article/​Gestalt_​principles (Re­trieved 2015-06-08)
  • Tufte, Ed­ward (with­out date): Sat­urn em­bed­ded in Galileo’s text, best work of an­a­lyt­i­cal de­sign ever? http://​www.​edwardtufte.​com/​bboard/​q-​and-​a-​fetch-​msg?​msg_​id=00004p (Re­trieved 2015-06-08)
  • Tufte, Ed­ward R. (2001): The Vi­sual Dis­play of Qunati­ta­tive In­for­ma­tion. Sec­ond Edi­tion. Graph­ics Press, Cheshire, Con­necti­cut
  • Tukey, John W. (1965): The Tech­ni­cal Tools of Sta­tis­tics. In The Amer­i­can Sta­tis­ti­cian, Vol. 19, No. 2. (Apr., 1965). Pages 23-28
  • Van Dam, An­dries (2001): User In­ter­faces: Dis­ap­pear­ing, Dis­solv­ing, and Evolv­ing. Com­mu­ni­ca­tions Of The Acm. March 2001/Vol. 44, No. 3
  • von Baeyer, Hans Chris­t­ian (2003): The New Lan­guage of Sci­ence. Cited in Lima, Manuel (2009): In­for­ma­tion Vi­su­al­iza­tion Man­i­festo. http://​www.​vis​ualc​ompl​exit​y.​com/​vc/​blog/?​p=644 (Re­trieved 2015-06-08)
  • Wang, Steve C. (2008): Pro­fes­sor Puts a Face on the Per­for­mance of Base­ball Man­agers. In­ter­view with Alan Schwarz. http://​www.​nytimes.​com/​2008/​04/​01/​science/​01prof.​html (Re­trieved 2015-06-08)
  • Ware, Colin (2004): In­for­ma­tion Vi­su­al­iza­tion – Per­cep­tion for De­sign. Sec­ond Edi­tion. Mor­gan Kauf­mann Pub­lish­ers, San Fran­cisco, CA..
  • Watts, Gareth (2012): jQuery Sparklines. http://​omnipotent.​net/​jquery.​sparkline/ (Re­trieved 2015-06-08)
  • Wat­zlaw­ick, Paul; Beavin, Janet Helmick; Jack­son, Don D. (1967): Some Ten­ta­tive Ax­ioms of Com­mu­ni­ca­tion. In Prag­mat­ics of human com­mu­ni­ca­tion: A study of in­ter­ac­tional pat­terns, patholo­gies, and para­doxes Pages 48–71. W. W. Nor­ton & Com­pany, New York
  • Wat­zlaw­ick, Paul; Jack­son, Don D.; Led­erer, William J. (1967): Prag­mat­ics of Human Com­mu­ni­ca­tion: A Study of In­ter­ac­tional Pat­terns, Patholo­gies, and Para­doxes. W. W. Nor­ton & Com­pany, New York
  • Wen­zel, Horst (2006): Erzählende Bilder und bild­hafte Lit­er­atur. Plädoyer für eine Text-Bild­wis­senschaft. In Maar, Christa; Burda, Hu­bert (Pub­lisher): Iconic Worlds. Neue Bilder­wel­ten und Wis­sensräume. Pages 232–250. Du­Mont Buchver­lag, Köln
  • Where You Are (2013): A Col­lec­tion of Maps That Will Leave You Feel­ing Com­pletely Lost. http://​where-​you-​are.​com/ (Re­trieved 2015-06-08)
  • Wil­lett, Wes­ley; Heer, Jef­frey; Agrawala, Ma­neesh (2007): Scented Wid­gets: Im­prov­ing Nav­i­ga­tion Cues with Em­bed­ded Vi­su­al­iza­tions. Com­puter Sci­ence Di­vi­sion at the Uni­ver­sity of Cal­i­for­nia, Berke­ley
  • Wirth, Niklaus (1976): Al­go­rithms + Data Struc­tures = Pro­grams. Ei­d­genössis­che Tech­nis­che Hochschule Zürich. Pren­tice-Hall, New Jer­sey
  • Wittgen­stein, Lud­wig (1922): Trac­ta­tus Logico-Philo­soph­i­cus. Ogden, K. (Pub­lisher): In­ter­na­tional Li­brary of Psy­chol­ogy Phi­los­o­phy and Sci­en­tific Method. Lon­don Kegan Paul, Trench, Trub­ner & Co., Ltd. New York: Har­court, Brace & Com­pany, Inc.. http://​www.​gutenberg.​org/​ebooks/​5740 (Re­trieved 2015-06-08)
  • World Eco­nomic Forum (2011): Per­sonal Data: The Emer­gence of a New Asset Class. http://​www3.​weforum.​org/​docs/​WEF_​ITTC_​Per​sona​lDat​aNew​Asse​t_​Report_​2011.​pdf (Re­trieved 2015-06-08)
  • Zwa­bel (2009): C++ IDE Evo­lu­tion: From Syn­tax High­light­ing to Se­man­tic High­light­ing. https://​zwabel.​wordpress.​com/​2009/​01/​08/​c-​ide-​evolution-​from-​syntax-​hig​hlig​htin​g-​to-​semantic-​hig​hlig​htin​g/ (Re­trieved 2015-05-09)