M

Mixed Reality, GamifiedPresence, and Storytelling forVirtual Museums

George Papagiannakis1,2,Efstratios Geronikolakis1,2, Maria Pateraki2,Victor M. López-Menchero3, Michael Tsioumas4,Stella Sylaiou5, Fotis Liarokapis6,Athina Grammatikopoulou7,Kosmas Dimitropoulos7, Nikolaos Grammalidis7,Nikolaos Partarakis2, George Margetis2,Giannis Drossis1,2, Martha Vassiliadi8,Alan Chalmers9, Constantine Stephanidis1,2 andNadia Magnenat-Thalmann101Computer Science Department, University ofCrete, Heraklion, Greece2Foundation for Research and Technology Hellas,Heraklion, Greece3Spanish Society of Virtual Archaeology, Seville,Spain4Hellenic Ministry of Culture and Sports, Serviceof Modern Monuments and Technical Works ofCentral Macedonia, Thessaloniki, Greece5Hellenic Open University, Patras, Greece6Masaryk University, Ponava, Brno, CzechRepublic7Information Technologies Institute, CERTH,Thessaloniki, Greece8School of Philology, Aristotle University ofThessaloniki, Thessaloniki, Greece9WMG, University of Warwick, Coventry, UK10MIRALab, University of Geneva, Geneva 4,Switzerland

Synonyms

Augmented reality; Cultural heritage;Gamification; Holographic augmented reality;Mixed reality; Presence; Storytelling; Virtualreality

Definition

Mixed reality as display technology, gamificationas motivational element, and storytelling as inter-action metaphor while maintaining the feeling ofpresence are identified as key elements for defin-ing the next generation of virtual museums.

Introduction

Storytelling, presence, and gamification are threebasic fields that need to be taken into accountwhen developing novel mixed reality applicationsfor cultural heritage, based on the recent renais-sance of commercial VR and AR hardware. Thissurvey aims to cover a gap in the bibliography andthe last relevant surveys of Papagiannakis et al.(2008), Jung et al. (2011), Foni et al. (2010), andAnderson et al. (2009) which are more than7 years old, whereas some of these topics arementioned in several different chapters withinIoannides et al. (2017) but not in single, compar-ative study. Section “The Role of Storytelling inCultural Heritage” of this survey covers the state

# Springer International Publishing AG 2018N. Lee (ed.), Encyclopedia of Computer Graphics and Games,https://doi.org/10.1007/978-3-319-08234-9_249-1

of the art in storytelling for cultural heritage andvirtual museums, section “The Role of Presence inCultural Heritage” the role of presence, and sec-tion “The Role of Gamification in Cultural Heri-tage” the usage of gamification principles. In thelast section “Survey of Recent MR Methods forVirtual Museums,” a comparison among latestmethods in the above areas is presented.

The Role of Storytelling in CulturalHeritage

Storytelling in MuseumsMuseums have realized the value of storytellingthe second half of the twentieth century. Nowa-days museums find themselves competing withthe large offer of cultural products coming notonly from the cultural sector but also from theentertainment industry. Therefore, museumsneed to differentiate and make their productsmore appealing and attractive to a variety of audi-ences. Influenced by the “new museology” con-cept and the transformation of the museologicalpractice, which refers to a shift in the social rolesof museums encouraging new styles of commu-nication and expression, in contrast to the “cul-tural authority” of museums based on classic,collections-centered models, museums havechanged.

Museums can act as primus inter pares (firstamong equals) and create a direct communicationwith the visitors, in which museum is the commu-nicator and the visitor is the receiver and viceversa, providing the visitor the opportunity toactively participate in the story. Museums canhave a conversation with the visitor, which canshare her/his experiences and personal views. Tothis end, recent AR/VR commercial h/w techno-logical advances enable the use of sophisticatedtools to deliver virtual museum stories and infor-mation in a number of ways for experienceenhancement, knowledge construction, andmeaning making (Sylaiou et al. 2009).

Storytelling Using AR and MR (Merging AMIInstallations with Mobile Devices and PhysicalArtifacts Through Stories)

Storytelling in Mixed RealityStatic visualizations have been traditionallyemployed to support storytelling in the form oftext, diagrams, and images. The adoption ofdynamic approaches utilizing state-of-the-art 2Dand 3D graphics is emerging in an effort toexplore the full potential of interactive narration.Rather than simply constituting an additionallayer to exhibitions, the enhancement of CH insti-tutions through interactive MR exhibits addedvalue to the overall user experience (Marshallet al. 2016), especially if combined with person-alization to each user’s interests (Partarakis et al.2016). Interactive storytelling in MR environ-ments bridges digital and physical information,augmenting the real world and offering interactionwhich corresponds to the user’s actions in thephysical space. Storytelling can be employed forassociating tangible and intangible information;such an example is Huang and Huang (2013),where the authors compound information for thepromotion of indigenous cultural heritage.

Storytelling AuthoringAuthoring refers to the process of creating narra-tions that form a digital story. In terms ofauthoring, the most common story types arecharacter-based stories (Cavazza et al. 2002), lin-ear timeline-based stories (Bimber et al. 2003;Drossis et al. 2013b), and ontology-based stories(Casillo et al. 2016). In terms of authoring virtualworlds, Lu et al. (2008) present an editing envi-ronment for facilitating the construction of 3Dmuseums. Additionally, narratives descriptionand structuring is also performed using authoringtools. Ardito et al. (2017) create a similar tool forstorytelling creation and customization, focusingon the aspects of smart objects integrated in CHinstallations. Modern cameras are equipped withdepth sensing capabilities allowing the real-timescanning of surroundings with sufficient detail.Such cameras are increasingly being adopted byconsumer smartphones, such as the Google Tangocompatible devices (Klingensmith et al. 2015).

2 Mixed Reality, Gamified Presence, and Storytelling for Virtual Museums

Therefore, such approaches can facilitate aug-mented reality environment rapid prototypingand authoring in a straightforward manner with-out requiring user expertise on computer science.

The Role of Presence in Cultural Heritage

“Presence” refers to the phenomenon of peoplebehaving and feeling as if they “are there” in thevirtual world created by computer displays(Ioannides et al. 2017). It is an incredibly power-ful sensation, which is unique to MR, as it is notpossible to recreate it in any other medium.

The Role of Illumination in MR Presence forCultural Heritage

IntroductionThere are two key components that are necessaryin order to achieve the right illumination andthereby a high sense of presence in a MR culturalheritage environment: authentic appearance ofthe light and the correct dynamic range. Prior tothe introduction of electricity, past societies reliedentirely on daylight and lighting from flames forillumination. Any MR cultural heritage environ-ment which is lit by modern lighting would thusnot be authentic. In addition, the real world con-tains a wide range of lighting conditions, fromdark shadows to bright sunshine. The MR envi-ronment needs to recreate this dynamic range of ascene. Where this is not possible using existingtechnology, techniques such as tone mappingneed to be used.

High-Dynamic-Range (HDR) EnvironmentsThe dynamic range in a scene is defined as theratio between the darkest part of the scene and thebrightest. While the human visual system is ableto adapt to the full range of light in a scene,traditional (also known as low-dynamic-range(LDR)) imaging is not able to capture or displaya dynamic range of more than 256 to 1 (8 stops).High-dynamic-range (HDR) imaging, on theother hand, by using 32 bit IEEE floating pointvalues to represent each color channel, can cap-ture and display all the visual data in a scene.

Failure to capture the full range of visual datacan lead to important information being misseddue to under- or overexposed regions in theimage. Even if a scene has been captured, orcreated in HDR, detail can still be lost when theimage is subsequently displayed on a devicewhich is not HDR. In such a case, the HDRimages need to be tone mapped in order to attemptto preserve the perception of the real scene on theLDR device. Many tone mapping operators(TMOs) have been presented over the years.More recently new TMOs have needed to bedeveloped specifically for displaying 360o HDRimages on head-mounted displays (HMDs) whichare not yet HDR display devices. This is becausethe lighting in a full environment can vary signif-icantly depending where the person is looking.HDR imaging is especially important in MR envi-ronments to ensure the virtual objects are relit withthe same levels of lighting as in the real scene. Assuch, HDR is increasingly being been used withinCH applications to improve their authenticity.

The Impact of Virtual Narrators to Presence inVirtual MuseumsOne of the important aspects which enhance thefeeling of presence in CH (Papaefthymiou andPapagiannakis 2017) is the interaction with andbehavior of the virtual narrators. These narratorsshould have humanlike behaviors, so that the vis-itors will feel like the virtual narrator that is speak-ing to them is real and that he/she is in the sameroom with them. An issue that rises at this point ishow a virtual narrator can be as realistic as possi-ble, with humanlike behavior and characteristics.For a virtual character to look as real as possible, itis not enough just to create him/her programmat-ically with code. It would be more realistic if thatcharacter could be reconstructed out of a realhuman. That way, the appearance of the characterwould resemble the appearance of the real personbased on whom this character was reconstructed.This can be realized by scanning a real personwith the Occipital™ depth sensor, speciallydesigned for this task. The photogrammetricmethod is one of the best methods yet to recon-struct the texture of a real person. By scanning theperson with the special sensor mentioned before

Mixed Reality, Gamified Presence, and Storytelling for Virtual Museums 3

and applying the texture, a 3D model of thatperson can be created. That way, a 3D model ofa real person will be able to be used in computergraphics and mixed reality applications, in orderto tell a story and communicate with the peoplethat will use the specific mixed reality application,as shown in Figs. 1 and 2 (Papaefthymiou andPapagiannakis 2017).

Mobile AR Interactive Applications for VirtualMuseumsMobile augmented reality systems (MARs) areincreasingly currently being tested in rich contentheritage environments by both creative profes-sionals and laymen. Recent mobile hardwaresuch as GPU-enhanced, multicore smartphonesand novel untethered AR headsets (e.g., HoloLens

Mixed Reality, Gamified Presence, and Storytellingfor Virtual Museums, Fig. 1 The hologram of the priestof the Asinou Church and the viewer interacting via

gestures through Microsoft HoloLens, in the ITN-DCHproject (Papaefthymiou and Papagiannakis 2017)

Mixed Reality, Gamified Presence, and Storytellingfor Virtual Museums, Fig. 2 Digitization of the priest ofAsinou Church (to reconstruct his 3D model as a virtual

narrator) using the Structure Sensor (Papaefthymiou andPapagiannakis 2017)

4 Mixed Reality, Gamified Presence, and Storytelling for Virtual Museums

byMicrosoft) pave the way for a new breed of ARservices and applications; however there are anumber of issues to consider which regard theaspects of information presentation and physicalinteraction (Papaefthymiou and Papagiannakis2017; Papagiannakis 2017; Ioannides et al.2017; Li and Duh 2013). With respect to directhand interaction, there are limitations in handhelddevices where the free hand is used for interactionwith the 3D objects (e.g., small screen size,unsuitability of point and click gestures formanipulation, finger occlusion), whereas inHMDs hands are an intuitive input channelalthough bimanual interaction cannot ensure bet-ter performance than single-hand interaction, andit is important to optimally assign functions to twohands. For example, in recently introduced ARKitby Apple, which represents a commercial tippingpoint in terms of mass-market adoption of ARtechnology, interactions are kept simple withobject’s movements restricted on a plane, andthe possibility of recognition of conflicting ges-tures. On the HMD side Microsoft HoloLens,although it has promised a hands-free experience,interaction can be frustrating given the limitedfield of view and the misreading of gestures.Although the promise of MARs is turning intoreality, the technical challenges with respect tocomputational efficiency, information retrievalfrom different data sources, markerless detection,and hand gesture recognition performance stillaffect the overall user experience when interactingwith MARs.

Immersive Experiences for Interaction withCultural Heritage

Mixed Reality InstallationsMixed reality has the potential not only to increasemotivation to learn but also to raise interest onCH. Thus, CH institutions can increase theirappeal and enhance visitor engagement throughinteractive installations that include public dis-plays (Partarakis et al. 2017). In addition toimproving the aesthetic experience, mixed realityenvironments have influence on visitor experi-ence, thus favoring the probability of revisiting aspecific attraction (Jung et al. 2016). Apart from

providing an enjoyable experience, mixed realityinstallations can facilitate cultural awareness, his-torical reconstruction, and heritage awareness.State-of-the-art approaches are not limited toinstallations in indoor spaces (Grammenos et al.2012), but can also involve vehicles that act asportable kiosks (Zidianakis et al. 2016).

Setups and InteractionInteractive installations in public spaces such as inCH institutions have certain requirements in termsof interaction and setup. The installations need toadapt to fit to the space available, provide contentwhich interests both domain experts and non-expert users, and also present thorough informa-tion on demand (Mortara et al. 2014). At the sametime, the system design should provide informa-tion immediately and support straightforwardinteraction techniques. Multiuser interaction withpublic displays is an open issue and constitutes anactive area of research. Once people approach theinteractive display, they decide their actions withregard to the system. Especially in the context ofMR applications, the establishment of interactionwith a public display involves transitioning fromimplicit to explicit interaction (Vogel andBalakrishnan 2004) as the users become engagedto the pervasive display.

Tangible interaction is a form of interactionwith mixed reality installations in which physicalitems act as mediators between the users and theenvironment. One aspect of tangible interactionrefers to physical objects, which are simply themedium of communicating information. Forinstance, physical paper is employed as a portabledisplay, augmenting maps or glass-protectedmodels (Grammenos et al. 2011). Interactivemaps (Margetis et al. 2017) are a mixed realityapplication, where printed maps are augmentedwith additional multimedia information. Further-more, tangible interaction can involve objectswhich have meaningful substance with a semanticmeaning, such as smart objects (i.e., physicalitems equipped with sensors such as RFID tags).Smart objects can facilitate related information inits context of use (Petrelli et al. 2016).

Mixed Reality, Gamified Presence, and Storytelling for Virtual Museums 5

Bridging Worlds: Combining AR, MR, and VRDigital cultural heritage content is combined withphysical 3D replicas (Bugalia et al. 2016) in orderto provide a virtual tour in architectural sites usingMR. This approach has the advantage of movingin a wall-projected 3D virtual environment whileplacing a physical model in front of the users toact as a minimap assisting navigation. Head-mounted displays are employed in the context ofCH foundations for holographic AR and addition-ally support VR visualizations (Pedersen et al.2017). VR can act as a medium to visualize CHexhibits otherwise unavailable to users as well asreenactments of historical events offering hands-on experience regarding unattainable content.Such indicative examples of VR usage are ele-ments which are under conservation or even sto-len (The Museum of Stolen Art 2017) and such asthe battle of Thermopylae (Christopoulos et al.2011), respectively. A different approach formixing realities involves using tangible objectsas souvenirs which summarize narratives as theywere presented to users (Petrelli et al. 2017), act-ing as personalized keepsakes of their visit in CHinstitutions. Finally, an interesting approach atGeevor Tin Mine Museum (Jung et al. 2017)combines AR, VR, and 3D printed objects withthe common goal of adding value to the overalluser experience. The technologies are employedfor enriching visits (AR), providing access toinaccessible areas (VR), and souvenirs which theusers can keep, respectively (3D printing).

The Role of Gamification in CulturalHeritage

Gamification has been applied to a number ofapplication domains including CH, and it is apopular approach to increase the entertainmentand thus the motivation factor of users(Vassileva 2012). It is different from computergames even if it shares a lot of common theoriesand practices of development. It is considered tobe one of the more recent developments of com-puter game design principles that have beenresearched since their appearance (Liarokapiset al. 2017a). A major objective of gamification

is the improvement of users’ engagement that canprovide positive effects, but the effects are heavilydependent on personalization (Hamari et al.2014). Recently, an interactive tool called Photo-Trip was presented that can autonomously recom-mend CH locations along travel itineraries (Bujariet al. 2017). The application can identify thesepoints of interest by gathering pictures and relatedinformation from social media and then providesuggestions and recommendations.

Gamification elements also exist in a numberof simulations ranging from serious games tovirtual and augmented reality. Similarly, edutain-ment and gamification are argued to be commonrecurring themes in the fields of education andcultural communication (Mancuso et al. 2017).In terms of virtual reality, gamification elementswere recently shown in a kinesthetic applicationof sculpturing Cycladic figurines (Koutsabasisand Vosinakis 2017). The user takes the role ofan ancient craftsman who creates a figurine withbare hand in a virtual reality environment. Inanother study, an immersive virtual underwaterenvironment was developed to explore Mazotosshipwreck site and get relevant information aboutit and its contents (Liarokapis et al. 2017b).

In the following sections, key recent MR pro-jects with rich gamification elements arepresented.

Playful Interaction with Cultural HeritageExhibits (Improved UX)Immersion is based on covering physically a per-son’s stimuli, namely, vision, spatialized sound,and haptic feedback (Bowman and McMahan2007). Haptic feedback (Ott et al. 2007) is exam-ined in literature as an additional means ofenhancing immersion. Kosmalla et al. (2017)combine tactile feedback from physical worldswith a virtual rock climbing environment in com-bination with full body movement and exertion.Immersion is also strongly related to the interac-tion process: in addition to perceiving a mixedreality (MR) application with human senses, theinteraction modality employed constitutes a deci-sive factor in feeling of immersion and the overalluser experience. Contrary to mission-criticaldomains such as a working environment, CH

6 Mixed Reality, Gamified Presence, and Storytelling for Virtual Museums

applications belong to a field in which users aremore open to novel alternate interaction modali-ties. Requirements such as precision and effi-ciency are not fundamental in such a context, asusers may be willing to sacrifice precision andoverrule tiredness for entertainment and playfuluser experience (Drossis et al. 2013a). Playfulnessencourages exploration, fosters creativity, andstimulates social interaction by entertaining usersand allowing them to escape from the reality(Tsekleves and Darby 2017). Playful interactionis employed for attracting users toward publicinstallations and therefore facilitating userengagement (Williamson and Sundén 2015).

Re-Play: A Cultural Heritage Project thatAllows to Replay Ancient GamesIn Europe only, there are over 3000 TraditionalSports and Games (TSG). The EU project Re-Playhas focused on two families of TSG (Gaelic andBasque) that are integral to the fabric of theircommunities of origin and have successfullystaved off this trend of convergence. The groupat MIRALab, University of Geneva, has focusedon the development of a realistic real-time anima-tion and rendering platform that enabled the visu-alization of the virtual national sports heroes aswell as the visualization of avatars representingthe local heroes (Tisserand et al. 2017). This state-of-the-art animation-rendering system includesdynamic muscle effects, which are modeled overthe skin using a novel GPU approach. Based onexisting MRI datasets and accurate reconstructed3D models, the Re-Play platform computes thedisplacement map and assigns weights to eachcoordinate in the displacement map based onEMG measurements. To increase the feeling ofimmersion, photo-realistic 3D models acquiredusing a 3D scanner have been employed to ani-mate and render the virtual sports heroes.

Gamification Applications for Folklore DanceIntangible cultural heritage (ICH) includes fragileexpressions mainly transmitted orally or by ges-tures from one generation to the next. In order tosupport the learning and transmission of ICHexpressions, gamification can also be used.Based on the theory of “experiential learning”

(Kolb, D.A., 1984, Experiential learning: Experi-ence as the source of learning and development,Englewood.), one of the main pillars of game-based learning, the acquisition of learning isachieved by observing, reflecting, mentallyrepresenting, and enacting movements. In accor-dance with this theory, within the i-Treasures(Dimitropoulos et al. 2018) project, a number ofgame-like applications for sensorimotor learningof specific dance types and other activities involv-ing full body gestures were designed and devel-oped to support the learning and transmission of anumber of ICH expressions. Specifically, a seri-ous game application for transmitting ICH knowl-edge concerning the Greek traditional dance“Tsamiko” was developed (Kitsikidis et al.2015) and shown in the figure below. The gamewas structured as a set of activities, aiming toteach different variations of the dance allowingusers to learn by either observing experts’ pre-recorded movements in the Observe mode or bystarting practicing the dance step sequencesrespectively in the Practice mode (Fig. 3).

During the Practice mode the learner isexpected to imitate the moves of the expert avatardisplayed on the screen. The addition of a virtualtutor to correct/manipulate/guide the user by pro-viding visual and audio feedback was alsosupported in order to encourage the learners’ “par-ticipation” and engagement.

Survey of Recent MRMethods for VirtualMuseums

In Table 1, a summary of key papers in the last7 years, after the last relevant survey paper fromAnderson et al. (2009), is presented. Althoughthere is no specific MR method that featuresgamified storytelling with heightened interactionthat still maintains full immersion and the feelingof presence, several conclusions and recommen-dations for next lines of research can be drawn andsummarized.

The MR technologies that are used in the keypapers (located in Table 1) contribute to the pres-ervation of cultural heritage, each one with its ownlevel of storytelling, presence, gamification,

Mixed Reality, Gamified Presence, and Storytelling for Virtual Museums 7

interaction, and tracking methods. Since all of theinstallations below are MR applications, many ofthem take into account the gamification field, inorder for those applications to be interesting andfun for the viewers. Some of them include story-telling elements (e.g., Papaefthymiou andPapagiannakis 2017; Pedersen et al. 2017),which are used to inform the viewers about thestory of a monument, for instance, thus contribut-ing in cultural heritage curation. Furthermore, itcan be noted that most of the MR methods belowsupport partial immersion and few of them sup-port full immersion. The term “immersion” isincluded in purpose as it can be easily quantifiedbased on the display, whereas “presence” is elu-sive and depends on many parameters and thusdifficult to provide that holds true throughout asimulation. VR HMDs thus support full immer-sion, whereas AR and holographic AR supportpartial one. Moreover, some applications that runtethered with a computer and do not support VRor AR provided no immersion at all. The majorityof these key papers use mobile AR or holographicAR, and that explains the majority of the partialimmersion entries on Table 1. A recommendationat this point would be that there should be moreMR applications that support desktop or mobileVR with full immersion, as it creates a morerealistic experience for the viewers because theydo not have access to the real world and thefeeling of presence is respectably higher than itis with partial immersion. These technologies canalso be categorized in two additional categories,tethered and untethered, according to if an instal-lation needs to be connected to a PC or not

(standalone device), respectively. An example ofan untetheredMR technology, which can be foundin the table below, is the Papaefthymiou andPapagiannakis (2017), which uses the AppleiPad Pro device in order to run. It uses Apple’sARKit for camera tracking. The viewers’ move-ments are not limited by cables (since no cablesare used), which enhances the feeling of presenceas their movements would not be limited if theywere exploring the real monument. Also,gamification and storytelling elements aresupported, and along with the feeling of presenceand freedom of movements, all those elementscreate the perfect experience for the viewers.Another version of this work runs on the Micro-soft HoloLens holographic AR HMD, which isalso an example of untethered AR.

On the other hand, there are some great MRapplications (e.g., Pedersen et al. 2017; Drossiset al. 2015; Tisserand et al. 2017), which supportgamification and/or storytelling elements, butthey limit the movements of the viewers due tothe existence of cables or need of a connectionwith a desktop PC. Although these applicationsare impressive and greatly contribute to culturalheritage preservation, they restrict the freedom ofmovement to the users, which in some cases maydisrupt the feeling of presence. Based on thisdiscussion, it is recommended for MR applica-tions to be able to run in MR installations that donot restrain the movements of the viewers in anyway. Without restriction of movements and withfull immersion, the experience of a viewer willreach very high levels. Finally, SpatialStories(Foni et al. 2010), a very recent commercial effort,

Mixed Reality, Gamified Presence, and Storytelling for Virtual Museums, Fig. 3 Observe and Practice screen ofTsamiko’s gamified application within the i-Treasures EU project (Kitsikidis et al. 2015)

8 Mixed Reality, Gamified Presence, and Storytelling for Virtual Museums

Mixed

Rea

lity,

Gam

ifiedPresen

ce,a

ndStorytellin

gforVirtual

Museu

ms,Ta

ble

1Com

parisonof

recent

MRmetho

dsforvirtualm

useums

Mixed

reality

metho

dMRinstallatio

nGam

ificatio

nStorytelling

Interaction

Tracking

Immersion

Intang

ible

heritage

And

ersonetal.

(200

9)Rom

eRebornseriou

sgame

xx

–No

Ancient

Pom

peiiapplication

xx

–No

Parthenon

Project

xx

–No

VirtualEgy

ptianTem

ple

xx

Walking

/mov

ement

CaveAutom

atic

Virtual

Env

iron

ment

(CAVE)

Full

No

The

Ancient

OlympicGam

esx

xNavigationwand

oftheVRsystem

–No

VirtualPrioryUnd

ercroft

xx

–No

Com

mercialhistoricalgames

xx

–No

Bug

aliaetal.

(201

6)3D

printedmod

els,projector,camera,laser

pointer

xx

Laser

pointer

IRcamera,

projectcam

era,

view

camera

Partial

No

Don

getal.

(201

7)VRapplicationusingFocus

3DX33

0Faro

scanner,Sph

eron

PanoC

am,twoGoP

roHero

4cameras

forscanning

,and

utility

for

rend

ering

xDepending

onVR

platform

Depending

onVRplatform

Full

No

Drossisetal.

(201

5)3D

mod

elprototyp

e(depth

sensor,tou

chscreen,interactiv

ecube,p

rojection)

xTou

chscreen,

interactivecube,

walking

/mov

ement

RGB-D

sensor,

Kinecto

rAsus

Xtio

ncamera

Partial

No

Gim

enoetal.

(2011)

Regionof

Valenciamap,seven

pointers,and

Hostess

xWalking

/mov

ement

Map,p

ointers,

Kinectcam

era

Partial

No

Grammenos

etal.(20

11)

Tabletopaugm

entedreality

system

xFinger-basedinpu

tProjector

and

pieces

ofwhite

paper

Partial

Yes/no

Javo

rnik

etal.

(201

7)App

leiPad

Pro

xTou

chscreen

InneriPad

camera

Partial

Yes

Kitsikidisetal.

(201

5)MicrosoftKinectS

ensor

xMotioncapture

Skeletontracking

with

Kinect

Sensor

Partial

Yes

Kosmallaetal.

(201

7)HTCVivewith

LeapMotion

xCon

trollers

Position

aland

hand

tracking

Full

No

LeapMotion

xHandgestures

Partial

No

(con

tinued)

Mixed Reality, Gamified Presence, and Storytelling for Virtual Museums 9

Mixed

Rea

lity,

Gam

ifiedPresen

ce,a

ndStorytellin

gforVirtual

Museu

ms,Ta

ble

1(con

tinue

d)

Mixed

reality

metho

dMRinstallatio

nGam

ificatio

nStorytelling

Interaction

Tracking

Immersion

Intang

ible

heritage

Kou

tsabasis

andVosinakis

2017

Handandfing

ertracking

Liarokapisetal.

(201

7b)

HTCVive

xCon

trollers

Motionandlaser

based

Full

No

Margetis

etal.

(201

7)Interactivemap

tabletop

system

xHandw

ritin

g,gestures,tou

ch/

click

Projector,h

igh-

resolutio

ncamera,depth

sensor

Partial

Yes/no

Nakevskaetal.

(201

7)CAVE

xx

Backprojectio

non

walls

Pressuresensors

Full

Yes/no

Papaefthy

miou

and

Papagiann

akis

2017

OculusRift

xx

Con

trollers

Rotationaland

positio

nal

(sensors)

Full

No

Papaefthy

miou

and

Papagiann

akis

2017

App

leiPad

Pro

xx

Tou

chscreen

App

le’sARKit

forcamera

tracking

Partial

No

Papaefthy

miou

and

Papagiann

akis

2017

MicrosoftHoloL

ens

xx

Voice

commands,

gestures

Bod

ymotion

Partial

No

Papagiann

akis

etal.(20

04)

DELLP4M50

Mob

ileWorkstatio

nx

Real-tim

emarkerless

cameratracking

Full

No

Pedersenetal.

(201

7)MetadeveloperKit

xx

Gestures

Markerless

surfacetracking

,head

mov

ement

Partial

No

SpatialStories

(201

8)Too

lkitforVR/ARplatform

xx

Depending

onVR/ARplatform

Depending

onVR/ARplatform

Full/p

artial

depend

ingon

theplatform

No

Tisserand

etal.

(201

7)Com

puterwith

KinectS

ensorfortradition

alsportspreservatio

nx

Mov

ement

KinectS

ensor

Partial

Yes

10 Mixed Reality, Gamified Presence, and Storytelling for Virtual Museums

is a toolset for real-time interactive VR/AR expe-riences featuring storytelling for non-programmers. Although it has not yet been fullytested, from information gathered from theirwebsite, its manual, and its videos, it poses apromising commercial solution in contributing toMR cultural heritage applications.

Conclusions

In conclusion, the ability to innovate in the field ofMR gamified storytelling while maintaining pres-ence in the virtual museums of the future will bedirectly related to the ability to find synergiesbetween the above fields of knowledge, withoutforgetting that new technologies can play a tran-scendental role in this equation. And as the quin-tessential reductive machine, MR technologiesare employed as the device that will restore itsunity through multiplicity and fragmentation.Hence this work needs to be considered as afurther step toward reconciliation and a renewedmutual beneficial relationship between humani-ties, social sciences, and computer science.

Cross-References

▶Gamification and Serious Games▶Gamification in Crowdsourcing Applications

References

Anderson, E.F., McLoughlin, L., Liarokapis, F., Peters, C.,Petridis, P., de Freitas, S.: Serious games in culturalheritage. In: Ashley, M., Liarokapis, F. (eds.) The 10thInternational Symposium on Virtual Reality, Archaeol-ogy and Cultural Heritage VAST – State of the ArtReports, St Julians, Malta, 22–25 Sept 2009, pp. 29–48

Ardito, C., Buono, P., Desolda, G., Matera, M.:Empowering CH experts to produce IoT-enhancedvisits. In: UMAP ‘17 25th Conference on User Model-ing, Adaptation and Personalization, Bratislava, 9–12July 2017, pp. 327–328. ACM, New York (2017)

Bimber, O., Encarnação, L.M., Schmalstieg, D.: The vir-tual showcase as a new platform for augmented realitydigital storytelling. In: IPT/EGVE03 Imersive Pro-gection Technologies/Eurographics Virtual

Environments, Zurich, 22–23 May 2003, pp. 87–95.ACM, New York (2003)

Bowman, D.A., McMahan, R.P.: Virtual reality: howmuchimmersion is enough? IEEE Comput. 40(7), 36–43(2007)

Bugalia, N., Kumar, S., Kalra, P., Choudhary, S.: Mixedreality based interaction system for digital heritage. In:VRCAI ‘16 The 15th International Conference on Vir-tual-Reality Continuum and its Applications in Indus-try, Zhuhai, 3–4 December 2016, pp. 31–37. ACM,New York (2016)

Bujari, A., Ciman, M., Gaggi, O., Palazzi, C.E.: Usinggamification to discover cultural heritage locationsfrom geo-tagged photos. Pers. Ubiquit. Comput.21(2), 235–252 (2017)

Casillo, M., Colace, F., De Santo, M., Lemma, S.,Lombardi, M., Pietrosanto, A.: An ontologicalapproach to digital storytelling. In: MISNC, SI, DS2016 The 3rd Multidisciplinary International SocialNetworks Conference, SocialInformatics 2016, DataScience 2016, Union, 15–17 August 2016, p. 27.ACM, New York (2016)

Cavazza, M., Charles, F., Mead, S.J.: Character-basedinteractive storytelling. IEEE Intell. Syst. 17(4),17–24 (2002)

Christopoulos, D., Mavridis, P., Andreadis, A.,Karigiannis, J. N.: Using virtual environments to tellthe story: the battle of Thermopylae. In: 2011 ThirdInternational Conference on Games and Virtual Worldsfor Serious Applications (VS-GAMES), pp. 84–91.National Technical University Great Amphitheater,Athens (2011)

Dimitropoulos, K., Manitsaris, S., Tsalakanidou, F.,Denby, B., Buchman, L., Dupont, S., Nikolopoulos,S., Kompatsiaris, Y., Charisis, V., Hadjileontiadis, L.,Pozzi, F., Cotescu, M., Ciftci, S., Katos, A., Manitsaris,A. and Grammalidis, N.: A Multimodal Approach forthe Safeguarding and Transmission of Intangible Cul-tural Heritage: The Case of i-Treasures. IEEE IntellSyst, pp. 1–1 (2018)

Dong, Y., Webb, M., Harvey, C., Debattista, K., Chalmers,A.: Multisensory virtual experience of tanning in medi-eval Coventry. In: EUROGRAPHICS Workshop onGraphics and Cultural Heritage, Graz, 27–29 Sept 2017

Drossis, G., Grammenos, D., Bouhli, M., Adami, I.,Stephanidis, C.: Comparative evaluation amongdiverse interaction techniques in three dimensionalenvironments. In: International Conference on Distrib-uted, Ambient, and Pervasive Interactions, pp. 3–12.Springer, Berlin/Heidelberg (2013a)

Drossis, G., Grammenos, D., Adami, I., Stephanidis, C.:3D visualization and multimodal interaction with tem-poral information using timelines. In: IFIP Conferenceon Human–Computer Interaction, pp. 214–231.Springer, Berlin/Heidelberg (2013b)

Drossis, G., Ntelidakis, A., Grammenos, D., Zabulis, X.,Stephanidis, C.: Immersing users in landscapes usinglarge scale displays in public spaces. In: International

Mixed Reality, Gamified Presence, and Storytelling for Virtual Museums 11

Conference on Distributed, Ambient, and PervasiveInteractions, pp. 152–162. Springer, Cham (2015)

Foni, A., Papagiannakis, G., Magnenat-Thalmann, N.:A taxonomy of visualization technologies for culturalheritage applications. ACM J. Comput. Cult. Herit.3(1), 1–21 (2010)

Gimeno, J., Olanda, R., Martinez, B., Sanchez, F.M.: Mul-tiuser augmented reality system for indoor exhibitions.In: IFIP Conference on Human–Computer Interaction,pp. 576–579. Springer, Berlin/Heidelberg (2011)

Grammenos, D., Michel, D., Zabulis, X., Argyros, A.A.:PaperView: augmenting physical surfaces withlocation-aware digital information. In: TEI ‘11 FifthInternational Conference on Tangible, Embedded, andEmbodied Interaction, Funchal, 22–26 January 2011,pp. 57–60. ACM, New York (2011)

Grammenos, D., Zabulis, X., Michel, D., Padeleris, P.,Sarmis, T., Georgalis, G., . . ., Adam-Veleni, P.: Mace-donia from fragments to pixels: a permanent exhibitionof interactive systems at the archaeological museum ofThessaloniki. In: Euro-Mediterranean Conference,pp. 602–609. Springer, Berlin/Heidelberg (2012)

Hamari, J., Koivisto, J., Sarsa, H.: Does gamificationwork? – a literature review of empirical studies ongamification. In: Proceedings of the 47th Hawaii Inter-national Conference on System Sciences (HICSS),IEEE Computer Society, Waikoloa, pp. 3025–3034(2014)

Huang, C.H., Huang, Y.T.: An annales school-based seri-ous game creation framework for Taiwanese indige-nous cultural heritage. J. Comput. Cult. Herit. 6(2),9 (2013)

Ioannides, M., Magnenat-Thalmann, N., Papagiannakis,G. (eds.): Mixed Reality and Gamification for CulturalHeritage. Springer, Heidelberg (2017). https://doi.org/10.1007/978-3-319-49607-8

Javornik, A., Rogers, Y., Gander, D., Moutinho, A.:MagicFace: stepping into character through an aug-mented reality mirror. In: CHI ‘17 CHI Conference onHuman Factors in Computing Systems, Denver, 6–11May 2017, pp. 4838–4849. ACM, New York (2017)

Jung, Y., Kuijper, A., Fellner, D.W., Kipp, M., Miksatko,J., Gratch, J., Thalmann, D.: Believable virtual charac-ters in human–computer dialogs. Eurographics(STARs). 2011, 75–100 (2011)

Jung, T., tom Dieck, M.C., Lee, H., Chung, N.: Effects ofvirtual reality and augmented reality on visitor experi-ences in museum. In: Information and CommunicationTechnologies in Tourism 2016, pp. 621–635. Springer,Cham (2016)

Jung, T.H., Jung, T.H., tom Dieck, M.C., tom Dieck, M.C.:Augmented reality, virtual reality and 3D printing forthe co-creation of value for the visitor experience atcultural heritage places. J. Place Manag. Dev. 10(2),140–151 (2017)

Kitsikidis, A., Kitsikidis, A., Dimitropoulos, K., Uğurca,D., Bayçay, C., Yilmaz, E., Tsalakanidou, F., . . .,Grammalidis, N.: A game-like application for dancelearning using a natural human computer interface. In:

International Conference on Universal Access inHuman–Computer Interaction, pp. 472–482. Springer,Cham (2015)

Klingensmith, M., Dryanovski, I., Srinivasa, S., Xiao, J.:Chisel: real time large scale 3D reconstruction onboarda mobile device using spatially hashed signed distancefields. In: Robotics: Science and Systems, vol. 4(2015). http://dx.doi.org/10.15607/rss.2015.xi.040

Kosmalla, F., Zenner, A., Speicher, M., Daiber, F., Herbig,N., Krüger, A.: Exploring rock climbing in mixed real-ity environments. In: CHI ‘17 CHI Conference onHuman Factors in Computing Systems, Denver, 6–11May 2017, pp. 1787–1793. ACM, New York (2017)

Koutsabasis, P., Vosinakis, S.: Kinesthetic interactions inmuseums: conveying cultural heritage by making useof ancient tools and (re-) constructing artworks. VirtualReality. pp 1–16. (2017). https://doi.org/10.1007/s10055-017-0325-0

Li, N., Duh, H.: Cognitive issues in mobile augmentedreality: an embodied perspective. In: Huang, W., et al.(eds.) Human Factors in Augmented Reality Environ-ments. Springer, New York (2013). https://doi.org/10.1007/978-1-4614-4205-9_5

Liarokapis, F., Petridis, P., Andrews, D., de Freitas, S.:Multimodal Serious Games Technologies for CulturalHeritage, Mixed Reality and Gamification for CulturalHeritage, Part V, pp. 371–392. Springer InternationalPublishing (2017a). http://dx.doi.org/10.1007/978-3-319-49607-8_15

Liarokapis, F., Kouřil, P., Agrafiotis, P., Demesticha, S.,Chmelík, J., Skarlatos, D.: 3D modelling and mappingfor virtual exploration of underwater archaeologyassets. In: Proceedings of the 7th International Work-shop on 3DVirtual Reconstruction and Visualization ofComplex Architectures and Scenarios, ISPRS, Napflio,pp. 425–431 (2017b)

Lu,W., Zeng, D., Pan, J.: AnXML-based scene descriptionlanguage for 3D virtual museum. In: 30th InternationalConference on Information Technology Interfaces,2008, ITI 2008, Croatia Hotel, Cavtat/Dubrovnik,pp. 445–450. IEEE (2008)

Mancuso, S., Muzzupappa, M., Bruno, F.: Edutainmentand gamification: a novel communication strategy forcultural heritage. J Pub Archaeology. 1, 79–89 (2017)

Margetis, G., Ntoa, S., Antona, M., Stephanidis, C.:Interacting with augmented paper maps: a user experi-ence study. In: Proceedings of the 12th Biannual Con-ference of the Italian SIGCHI Chapter (CHITALY2017), Cagliari, 18–20 Sept 2017

Marshall, M.T., Dulake, N., Ciolfi, L., Duranti, D.,Kockelkorn, H., Petrelli, D.: Using tangible smart rep-licas as controls for an interactive museum exhibition.In: TEI ‘16 Tenth International Conference on Tangi-ble, Embedded, and Embodied Interaction, Eindhoven,14–17 February 2016, pp. 159–167. ACM, New York(2016)

Mortara, M., Catalano, C.E., Bellotti, F., Fiucci, G., Houry-Panchetti, M., Petridis, P.: Learning cultural heritage byserious games. J. Cult. Herit. 15(3), 318–325 (2014)

12 Mixed Reality, Gamified Presence, and Storytelling for Virtual Museums

Nakevska, M., van der Sanden, A., Funk, M., Hu, J.,Rauterberg, M.: Interactive storytelling in a mixed real-ity environment: the effects of interactivity on userexperiences. Entertain. Comput. 21, 97–104 (2017)

Ott, R., Thalmann, D., Vexo, F.: Haptic feedback in mixed-reality environment. Vis. Comput. 23(9), 843–849(2007)

Papaefthymiou, M., Papagiannakis, G.: Gamified aug-mented and virtual reality character rendering and ani-mation enabling technologies. In: Ioannides, M.,Magnenat-Thalmann, N., Papagiannakis, G. (eds.)Mixed Reality and Gamification for Cultural Heritage,pp. 333–357. Springer (2017). https://doi.org/10.1007/978-3-319-49607-8

Papagiannakis, G.: Gamification and serious games. In:Lee, N. (ed.) Encyclopedia of Computer Graphics andGames. Springer, Cham., https://doi.org/10.1007/978-3-319-08234-9_90-1, ISBN 978-3-319-08234-9(2017)

Papagiannakis, G., Schertenleib, S., Ponder, M., Arevalo-Poizat, M., Magnenat-Thalmann, N., Thalmann, D.:Real-time virtual humans in AR sites. In: IEE VisualMedia Production (CVMP04), London, pp 273–276(2004)

Papagiannakis, G., Singh, G., Magnenat-Thalmann, N.:A survey of mobile and wireless technologies for aug-mented reality systems. J Comput. Anim. VirtualWorlds. 19(1), 3–22 (2008)

Partarakis, N., Antona, M., Zidianakis, E., & Stephanidis,C.: Adaptation and content personalization in the con-text of multi user museum exhibits. In B.N. De Carolis,C. Gena, T. Kuflik, & F. Nunnari (Eds.), Proceedings ofthe 1st Workshop on Advanced Visual Interfaces forCultural Heritage (AVI*CH 2016), Bari, Italy, 7-10June. Published by: CEUR-WS (Vol. 1621) (2016)

Partarakis, N., Grammenos, D., Margetis, G., Zidianakis,E., Drossis, G., Leonidis, A., Stephanidis, C.: Digitalcultural heritage experience in ambient intelligence. In:Mixed Reality and Gamification for Cultural Heritage,pp. 473–505. Springer, Cham (2017)

Pedersen, I., Gale, N., Mirza-Babaei, P., Reid, S.: Morethan meets the eye: the benefits of augmented realityand holographic displays for digital cultural heritage.J Comput. Cult. Herit. 10(2), 11 (2017)

Petrelli, D., Dulake, N., Marshall, M.T., Pisetti, A., Not, E.:Voices from the war: design as a means of understand-ing the experience of visiting heritage. In: CHI ‘16 CHI

Conference on Human Factors in Computing Systems,San Jose, 7–12 May 2016, pp. 1033–1044. ACM,New York (2016)

Petrelli, D., Marshall, M.T., O’Brien, S., McEntaggart, P.,Gwilt, I.: Tangible data souvenirs as a bridge between aphysical museum visit and online digital experience.Pers. Ubiquit. Comput. 21(2), 281–295 (2017)

SpatialStories: A toolset for real-time interactive VR/ARexperiences. http://apelab.ch/spatialstories (2018).Accessed 6 Jan 2018

Sylaiou, S., Liarokapis, F., Kotsakis, K., Patias, P.: Virtualmuseums, a survey and some issues for consideration.J. Cult. Herit. 10(4), 520–528 (2009)

The Museum of Stolen Art.: http://mosa.ziv.bz/ (2017).Accessed 22 Dec 2017

Tisserand, Y., Magnenat-Thalmann, N., Unzueta, L.,Linaza, M.T., Ahmadi, A., O’Connor, N.E., Zioulis,N., Zarpalas, D., Daras, P.: Preservation andgamification of traditional sports. In: Ioannides, M.,Magnenat-Thalmann, N., Papagiannakis, G. (eds.)Mixed Reality and Gamification for Cultural Heritage,pp. 421–446. Springer, Cham (2017)

Tsekleves, E., Darby, A.: The role of playfulness andsensory experiences in design for public health andfor ageing well. In: Sensory Arts and Design. Blooms-bury p. 49–66. 18 p (2017)

Vassileva, J.: Motivating participation in social computingapplications: a user modeling perspective. User Model.User-Adap. Inter. 22(1–2), 177–201 (2012)

Vogel, D., Balakrishnan, R.: Interactive public ambientdisplays: transitioning from implicit to explicit, publicto personal, interaction with multiple users. In: UIST‘04 17th Annual ACM Symposium on User InterfaceSoftware and Technology, Santa Fe, 24–27 October2004, pp. 137–146. ACM, New York (2004)

Williamson, J.R., Sundén, D.: Enter the circle: blendingspherical displays and playful embedded interaction inpublic spaces. In: Proceedings of the 4th InternationalSymposium on Pervasive Displays (PerDis ‘15),Saarbruecken, 10–12 June 2015, pp. 195–200. ACM,New York (2015)

Zidianakis, E., Margetis, G., Paparoulis, S., Toutountzis,T., Stratigi, K., Paparoulis, G., Stephanidis, C.: Turningan electric cargo vehicle into a portable interactiveinformation kiosk. In: International Conference onHuman–Computer Interaction, pp. 463–469. Springer,Cham (2016)

Mixed Reality, Gamified Presence, and Storytelling for Virtual Museums 13