Evaluating tangible and multisensory museum visiting experiences: Lessons learned from the meSch project

Areti Damala, University of Strathclyde, Glasgow, UK, Merel van der Vaart, University of Amsterdam / Allard Pierson Museum, the Netherlands, Loraine Clarke, University of Strathclyde, UK, Eva Hornecker, Bauhaus-Universität Weimar, Germany, Gabriela Avram, University of Limerick, Ireland, Hub Kockelkorn, Museon, The Netherlands, Ian Ruthven, University of Strathclyde, UK

Abstract

Keywords: tangible interaction, embodied interaction, multisensoriality, evaluation, audience research

1. Defining tangible and multisensory museum visiting experiences

The use of digital interactive technologies is becoming increasingly common and widespread in museums and other cultural heritage contexts. However, digital technologies, particularly screen-based museum interactives, are also accused of isolating visitors and distracting their attention from the real, physical objects on display. The European meSch project (http://mesch-project.eu/) sets as an aim to explore the potential of tangible interaction as an alternative way of using technology in museums so that the bonds with the museum objects on display are enforced. This is achieved by embedding digital properties and information into Cultural Heritage sites, museum exhibitions or objects: visitors are encouraged to interact with Cultural Heritage artefacts physically, cognitively, and emotionally using their senses (Petrelli et al., 2014; Zancanaro et al., 2015).

Measuring the impact of new technologies has been identified as one of the most important challenges for museums and cultural heritage institutions alike within the NMC Museum Horizon Report, and one that we understand but for which solutions are elusive (Johnson et al., 2015). This is particularly true in the case of digitally enabled, tangible, embodied, and multisensory museum visiting experiences. Can this approach encourage a multisensory, multifaceted visitor engagement in cultural heritage settings (Dudley, 2010)? For example, would the use of three-dimensional printed replicas of real museum objects that can be touched and manipulated instil a sense of wonder for the museum visitor? What kind of impact this might have for learning and meaning-making? And how can all of the above questions be assessed and evaluated?

Taking as an example two temporary exhibitions held in two museums in the Netherlands in which the meSch technology was integrated, Museon and Allard Pierson Museum, we focus on the subtleties of devising and planning evaluation studies for tangible, embodied, and multisensory museum visiting experiences. We discuss the enunciation of key evaluation questions and the qualitative and quantitative methods we used to gather visitors’ feedback. We identify strengths and weaknesses for each method used (observations, interviews, video data, questionnaires, meaning maps, and post-visit interviews), and we share lessons learned in terms of data gathering, analysis, sharing, and interpretation. Finally, we conclude with a set of best practices and guidelines that can serve as a springboard and starting point for discussion with researchers and practitioners in the Visitor Studies and Human Computer Interaction communities alike.

2. Atlantic Wall: War in the City of Peace, Museon, The Hague

2.1 Exhibition theme and message

From April until November 2015, Museon in The Hague organized the exhibition The Hague and the Atlantic Wall: War in the City of Peace, in which meSch technology was fully integrated. The Atlantic Wall was erected during World War II along the European west coast from Norway to the Spanish border so as to block an Allied invasion. Within this context, a broad strip throughout the city of The Hague was demolished to make room for the Atlantic Wall. Museon itself is built in the former anti-tank ditch with important international organisations in peace, justice, and safety established nearby, like OPCW and Europol. This intertwining in between the times of war and peace was made explicit in the exhibition design: the exhibition plan was based on the city map, with exhibits arranged in visiting stations, representing different city neighbourhoods and public spaces. Locals, but also tourists, had the chance to recognise street names, iconic buildings, and special places from The Hague.

At each visiting station, the objects on display served as entry points for different narratives such as “How was it to be forced to leave your home? To live in the almost empty fortress? To enter or leave the fortress? To experience the launch of a V2 rocket?” (http://mesch-project.eu/connecting-opposites-mesch-technology-integrated-in-an-exhibition/). Each story could be revealed from three different perspectives: this of the Dutch civilian, the German soldier, or the Dutch civil servant, involved in the evacuation of the city. In order to “unlock” and activate these narratives, the visitors had to pick up one or more of the six replicas that were available at the beginning of the exhibition or “check-in” station. These replicas were exact copies of original World War II everyday-life objects that were exhibited at the “check-in” station, behind the box with the replicas (Figure 1a; Figure 3, right). Thus, a scenting bag with surrogate tea was used for the Dutch version of the civilian perspective and a box of surrogate sugar for the English version of the same perspective. A Delft blue mug, once given as a Christmas gift to German soldiers, was recreated using 3D printing to represent the German perspective in Dutch (Figure 1a), while a phrasebook given to German soldiers was used for the English version. The armband worn by a civil servant or the ID issued by the German authorities for access to restricted areas reflected the perspective of an official (Dutch version and English version respectively). RFID tags able to read using NFC (Near Field Communication Technology) were integrated within each object (http://mesch-project.eu/why-we-are-using-smart-replicas/). The replicas would be placed on a round indent that could be found at each visiting station; this would activate the audio narrative the visitors could hear using an earpiece (Figure 1). The audio narratives were based on sources such as interviews, documents from archives, radio broadcasts and newspapers of the era. Once the audio started, relevant photographs and video clips from various archives were projected on the showcases, which also featured real artefacts, via a pico projector.

Figures 1a and 1b: choosing a replica and activating the narratives

2.2 Evaluation questions and evaluation methodology

A summative evaluation study was devised for the exhibition. Our main research questions were:

• How do visitors use and share the smart replicas, and what is their impact on the museum visiting experience?
• Are the different perspectives appreciated by visitors?
• How are the different meSch and non-meSch exhibition components (real museum objects, replicas, audio narratives, slideshows, text labels and signs, multimedia stations, polling stations) used by the visitors?
• What is the impact of emotions and memories for learning and meaning-making?

Given these questions, we opted for a mixed-method evaluation approach that would combine both quantitative and qualitative data (Gunn & Harknett, 2010; Diamond et al., 1999). We decided to use questionnaires, semi-structured interviews, observations, video, and audio recordings. We were also aware that the meSch technology would allow us to log data on visitors’ interactions with the replicas.

A month before the actual study, our team visited the exhibition to map the evaluation protocol in the exhibition space and carry out a pilot study. We made short video and audio recordings from different angles within the exhibition, tried out different observation strategies, and composed a first draft of the interview guide, which we then tried out. Experiencing the exhibition first as visitors (and only later as researchers) also allowed us to identify potential tweaks that could be made in the exhibition to improve visitor experience.

The study was carried out during nine days, from June 12 to June 21, 2015. Six researchers worked on site, while another four joined the team during data analysis. The following sections provide an overview of lessons learned during this field study.

2.3 Gathering video data

Video recordings are underutilized in visitor studies, though they can provide an extremely powerful tool for audience research (Heath et al., 2010). They can be precise and unobtrusive, as well as detailed, allowing repeated viewing and coding using different schemes while providing a wealth of information about visitor interactions (vom Lehn et al., 2002). One challenge is related with ethics concerns and national legislation that may vary from country to country and from one institution to another; planning for the study and receiving approval from all implicated authorities required a great investment in time and resources.

Signage notified the visitors of both video and audio recording taking place on gallery, its purpose, and information on how to opt out. Four video cameras were placed on the ceiling, capturing a bird’s eye view of visitors in the exhibition space (Figure 2). They were out of visitors’ lines of sight and thus relatively unobtrusive. Two cameras captured the start and end stations. Two other cameras were positioned to capture the two meSch stations closest to the check-in. In this way, approximately 50 percent of the exhibition space was covered. The cameras could be plugged into the main power so the batteries wouldn’t needed to be replaced during the exhibition opening hours. SD cards of high capacity were used to ensure each camera had enough memory to store a full day’s recording. At the end of each day, the collected data were moved to an external computer and the SD cards were cleared.

In order to capture the intended area, the cameras needed to be placed on the ceiling, upside down. In two occasions, we had to stop recording because visitors requested to opt out. Over nine days, these four cameras recorded a total of 37 hours and 50 minutes. For the analysis phase, sharing “big,” heavy video files proved to be challenging.

Figure 2: video recordings setup

2.4 Audio recordings

Audio data was also gathered at the beginning of the exhibition, at the check-in station, where visitors would collect and choose their replica. A lapel microphone was positioned among the replicas (Figure 3, left). Despite fear of this being more intrusive for visitors, and of poor-quality audio recordings due to background sound from moving and choosing the replicas, the audio recordings proved to be of very good quality.

At this station, a signpost announced capturing of both video and audio. Audio recordings complemented the video data, allowing us to both see and hear visitors’ first encounters with the meSch replicas. We hoped to find out how intuitive and straightforward the use of the replicas was, as well as whether visitors made the connection with the real objects imitated. An initial analysis showed that visitors’ conversations were orientated towards working out together what to do with the replicas with both members from their own groups and strangers. However, we have so far discovered only one conversation with a reference to the real artefacts at the “check-in” station. This verifies our concern that the relation between real objects and replicas could have been made clearer.

This finding will be further examined using the video recordings. The main challenges faced regarding the audio data was the language barrier as well as the synchronising of the audio data with all other video recordings.

Figure 3: replicas and setup of the lapel microphone (left); replicas and real objects (right)

2.5 Observations

During the pilot study, we experimented with several observation strategies. We concluded that observations would allow us to observe trajectories and visiting patterns through the exhibition, modalities of use of smart objects, and interactions between visiting companions and strangers (Goulding, 2000; Ciolfi, 2003). An observation sheet and a map of the exhibition were designed to allow us to note visitors’ trajectories and time spent at different stations. However, once on site, we found out that the traditional method of a notebook worked best.

Several visiting patterns and trends were identified through observations. For example, the use of the replicas was more straightforward if other present visitors would already use them, particularly among senior visitors. Another recurring behaviour was for visitors to explore the modalities of using the replicas together with their fellow visitors, then separating and following different trajectories. Interesting behaviours linked to specific visiting groups, would result in observation notes, prompting us to later link and compare this data with the video recordings. The contained size and layout of the exhibition could mean that in times where few visitors were present in the exhibition space, making observations while remaining unnoticed proved to be challenging. Observations could also be time consuming. Visitors who were observed while visiting the exhibition were asked at the end of their visit to fill in a questionnaire or participate in the interview. Unfortunately, not all visitors were available or willing to do so. However, in some cases observations and video data could be linked to questionnaire data as well, providing insights into visitors’ interactions and behaviours.

2.6 Visitor Experience Questionnaire (VEQ)

In addition to a short survey Museon uses a as part of its standard way of gathering visitor feedback, we decided that for the Atlantic Wall exhibition a longer questionnaire would be needed. Our questionnaire consisted of four A4 pages and sought to gather information on:

• Demographics (museum visiting habits, use of digital technologies, age, gender, place of residence)
• Expectations from the visit
• Time dedicated and exhibition components used
• Visitor experience from using the replicas
• Learning impact and emotional impact

The “Inspiring Learning” question bank, an Initiative of the UK Museums, Libraries and Archives Council, provided us with a lot of inspiration, particularly regarding ways of finding evidence of engagement and learning (http://www.artscouncil.org.uk/media/uploads/ilfa_downloads/S3D3_Researchers_Question_Bank.doc).

Due to the theme and the public this exhibition targeted, we decided to break the “65 and older” category into three new distinct age groups: the “65 to 74”, “75-84” and the “over 84 years ” old groups. The questionnaire was designed in such a way so as to be easy to answer, and both visitors who used the replicas and who didn’t were randomly invited to participate. Eighty-eight questionnaires were filled in. An initial analysis shows that approximately three-fourths of the visitors visited in a group, with male and female visitors equally represented. 97% live in the Netherlands, with approximately half of the visitors living in The Hague. One-third of visitors were older than 64 years, mainly in the 65 to 74 age group (26%) and 75 to 84 group (8%). Thus, our inkling of adding further age categories to the questionnaire for this specific exhibition was right. Another 33% of visitors were 55 to 64 years old, while all other groups accounted for the remaining 33%.

The data gathered by Museon via the standard short survey will allow us to triangulate whether our findings are representative of this exhibition’s visitor profiles. Participants turned out to be not only frequent museum visitors but also very aware of their expectations: Out of the eighty persons who filled the relevant open-ended question, forty-three explicitly mentioned as their main motivation wanting to find out more about World War II and the story of The Hague. Most respondents had used the replicas (77%), yet a good number hadn’t (23%). As for replica choice, the results obtained, mirror those provided by the log files: The top three preferences were the Dutch civilian (Tea bag, Dutch version), German soldier (beer mug – Dutch version) and civil servant (ID card-Dutch version). Complementing the logs, the questionnaires revealed that the main criterion for choosing a replica was firstly because of the perspective revealed (34%) and then only for the object itself (26%). However, as also revealed during an earlier meSch study (Van der Vaart & Damala, 2015), ease of use does not always rhyme with intuitiveness. Finally, a novel method tried out in the Atlantic Wall exhibition consisted of an Affective Impact questionnaire, which encouraged the visitors to rate, using semantic differential scales, the extent to which they experienced different moods and emotions (McManus, 1993) during their visit. This questionnaire was integrated in the Visitor Experience Questionnaire. Employing this well-known and widely used quantitative method allowed us to see and analyse schemes and patterns that would be very difficult to capture using any other tool.

2.7 Post-visit interviews

During the pilot study, an interview guide was devised for semi-structured interviews. We approached visitors toward the end of their visit, offering a coffee in the museum cafeteria and a small reward -Museon’s permanent exhibition catalogue- to thank them for their time. When no Dutch-speaking members of the team were available, some visitors were concerned that their English might not be good enough. However, the large majority of visitors approached were open to the idea of being interviewed in English.

In total, forty-nine visitors were interviewed. The interviews were recorded and later transcribed. The interviews gave us the possibility not only to discuss different types of interactions observed but also ask the participants directly on some of our main research questions. In short, interviews allowed us to explore the “why” as in contrast with the “what” that had been observed, documented, or recorded. Each interview lasted approximately twenty minutes. As with the observations and due to the number of researchers on-site, coordination was needed, particularly when the interviewee was willing to participate in an interview but preferred to do so in Dutch. Despite the lively environment of the cafeteria, sitting and chatting at a table over a cup of coffee worked as an icebreaker, as did starting with introductory questions such as the context or circumstances around the visitor’s visit on that occasion (Figure 4). For those participants who also filled in questionnaires, we were both pleased and surprised to see that a large number was willing to be recontacted for a longitudinal, follow-up interview. This study is currently in progress and will allow us to investigate the long-term impact of their museum visit in terms of sense-making and learning (Hooper-Greenhill, 2004).

Figure 4: interview set-up

2.8 Data gathered versus resources committed

The Atlantic Wall evaluation occurred for a period of nine days, with six researchers on site from three collaborating institutions both in Dutch and English. The amount of video and audio data captured, sums up to 37 hours and 50 minutes of video data (from four different camera angles) and 10 hours of audio recordings. Forty-nine participants were interviewed, and approximately the same number of visitors was observed. Eighty-eight visitors filled in a questionnaire, while more than half of them declared they would be happy to provide a follow-up interview. This longitudinal study will provide us with additional insights about the long-term impact the exhibition had on the museum visitors.

3. Feint: Illusion in Ancient Greek Art, Allard Pierson Museum, Amsterdam

3.1 Exhibition theme and message

The second exhibition that integrated meSch technology was hosted at the Allard Pierson Museum (APM) in Amsterdam, where Feint: Illusion in Ancient Greek Art evolved around the question of how Ancient Greek artists depicted movement in art. The exhibition ran from June 26 to September 6, 2015 presenting top pieces from both the APM and the Dutch National Museum of Antiquities (http://mesch-project.eu/apm-feint-video/).

The exhibition consisted of a series of consecutive rooms. In each room, the subject of movement in Ancient Greek art was explored through one of eight themes: movement in abstract motives, the human body, depicting gods, depicting dance, depicting women, depicting men, depicting well known stories, and tondos—the play of light and water. The objects on display consisted mostly of Greek earthenware, reliefs, plaster casts, and sculptures. In contrast with the Atlantic Wall exhibition at Museon, the interpretation materials used in this exhibition could be called “traditional”: a silent slide show in the introductory space, large introductory text panels, and shorter texts accompanying the objects.

The goal of the exhibition was to explain how Ancient Greek artists were the first to truly depict movement. However, a second goal was to let visitors experience the same sense of wonder that the Ancient Greeks would have felt when they first saw the lifelike depiction of movement. This sense of wonder played an important role in the final room of the exhibition, which was dedicated to a particular kind of drinking vessel, a shallow, wide-rimmed bowl called a kylix (plural: kylikes), which was used at social gatherings dedicated to drinking wine. A circular, often figurative, decoration can often be found on the bottom of these bowls. This decoration, called the tondo, was the main subject of this final room: tondos—the play of light and water. Tondos were painted in such a way that, when covered by a liquid, the refraction of light will create the illusion of movement in the image. Besides a display of original kylikes and accompanying text labels, this room contained two digital exhibits that were developed and evaluated as part of the meSch project.

The Tondo room was approximately 20 square meters, and visitors would approach the space through a corridor, which led to the exhibition’s exit. When approaching the tondo room, the first thing visitors would see were an introduction text on the tondos on the left-hand wall and a large projection at the back wall of the space. The projection on the back wall showed a virtual space in which a symposion was taking place, providing the social context in which the kylix was traditionally used; a gathering of men drinking wine, enjoying entertainment, and conversing. In front of the wall projection stood a plinth carrying a 3D printed replica of a kylix accompanied by the text “Pick me up.” Beside the plinth, a replica Greek daybed was installed, identical to the daybeds that were present in the wall projection (Figure 5). Actions by visitors could trigger activities on screen, displaying various interactions that would have been common practice during a symposion. When a visitor would pick up the replica kylix from the pedestal, one of the virtual characters would also lift his kylix, toast, and take a sip of wine. When the replica was placed back, the musician would start playing her instrument, receiving applause from one of the other characters. When a visitor would sit down on the daybed, one of the virtual characters would flick a drop of wine from his kylix, aiming at a stand in the middle of the room, also receiving applause (representing the famous drinking game of kottabos). The wall projection, replica kylix on the pedestal, and day bed formed the first digital exhibit, called the Symposion installation.

Figure 5: “Tondos — the play of light and water” installations

The second installation in this room was the Tondo installation and was placed at the right-hand wall (Figure 5 right, Figure 6). A built-in display case would present the original museum artefacts, the kylikes. In front of this display case stood a rectangular table, holding a second replica kylix and four shallow, round indentations in which the replica could be placed. When not in use, the kylix would be placed in the first round. Above each of the other three indentations, a pico projector was mounted at the top front of the showcase, hidden from view. This part of the exhibition focused on the illusion of movement in the tondos. A text on the table read, “Place the cup on one of the images and be moved.” When the replica was placed in one of these indentations, a reproduction of the tondo of one of the original object was projected inside the replica kylix (Figure 6). This projection showed the tondo image as if it was covered by a liquid, showing what the Ancient Greeks would have seen when using the kylikes centuries ago. Then, to trigger a sense of surprise, the image would actually start moving. For example, the fish on the bottom of the bowl would start swimming around. After this, the projection would end. From a technological perspective, the installations in Museon and APM were very similar. Using NFC tags embedded in replica objects, visitors could trigger images and sound. However, the resulting visitor experiences in both cases were very different.

Figure 6: the Tondo installation

3.2 Feeling moved: Research questions and evaluation methodology

The main aim of the audience research related to this exhibition was to get a better understanding of how the two digital exhibits could encourage and facilitate visitors’ engagement with objects. As a result, the following research questions were formulated:

• How do visitors relate objects to information, in particular the information provided through the digital installations, in this exhibition?
• How do visitors use and interact with the two digital installations?
• How do visitors value the use of smart objects as a way to convey information?
• How do the two digital installations influence the way visitors relate objects to information?

In order to answer these questions, we used observations, visitor experience questionnaires, personal meaning maps, and interviews. All visitors engaging in either of the questionnaires, the personal meaning map, or interview were also asked to fill out a visitor profile questionnaire. In the instance of the two questionnaires, the questions of the profile questionnaire were integrated with the other questions. Most of these methods were also used for the Atlantic Wall exhibition evaluation, except for the Personal Meaning Maps (PMM), which were unique to the evaluation of the Feint exhibition. The evaluation took place between August 13 and September 6, 2015, conducted by two researchers.

3.3 Videos recordings: Why didn’t we use them?

As with the Atlantic Wall exhibition, video data might have provided valuable information about the way visitors handled the replicas, as well as social interactions that might occur around the exhibits. However, constraints posed by the design of the exhibition meant it was not possible to collect video data. First, the small size and layout of the room meant it was not possible to position cameras in the Tondo room without them being obtrusive. Second, no power outlets were available, meaning all cameras would have run on battery power, which would have resulted in a change of batteries every 120 minutes in the best-case scenario. This would be an additional distraction and disturbance for the visitors. Therefore, it was decided not to collect video data.

3.4 Observations

An important element of this study was to better understand how visitors interacted with the objects on display in combination with the information available to them, either on text labels or through a digital exhibit. Because of the valuable insights observation can provide regarding visitor behaviour, it was decided to track visitors through that technique. Observations, in contrast to questionnaires, would also highlight behaviours that visitors would not necessarily be aware of, offering a more holistic view of the museum visit, allowing for new routes of enquiry to emerge through data analysis.

However, tracking visitors throughout their entire visit would be too labour intensive, considering the available resources. Also, it would provide the team with a disproportionate amount of information on visitor behaviour in rooms that did not contain digital technology. Tracking visitors through part of their visit was thought to be the best option. In order to be able to make a comparison between the two types of content, textual and digital/visual, visitors were tracked while visiting the last three rooms of the exhibition; two rooms without technology and one with technology. Since these spaces were relatively small and visitor numbers were modest, the presence of an observer would be obvious to most visitors. By making observations in two rooms without technology, it was hoped that visitors could get accustomed to the presence of the observer. Also, personal preferences with regards to themes or objects could be mitigated somewhat, by comparing visitor activity in two rooms dedicated to different themes. Choosing rooms that were placed in consecutive order had practical reasons: it was easier and took less time to track visitors without interruptions. Also, with all rooms located towards the end of the exhibition, it was expected that any form of “museum fatigue” would not have a bigger impact on behaviour in one room compared to the next.

Visitors were notified of the observations taking place through a sign at the entrance of the exhibition. Observations sheets were made, tested, and adjusted before the study took place (Figure 7). These were designed in a way that would allow for quick and succinct data collection, including demographic data, number of visiting companions, visitor movement, social interaction, time spent in each room, and any other relevant notes. Visitor movements were tracked on maps of each of the three rooms where objects, texts and technology were included. The observer could mark entrance and exit points and mark further movements with arrows by numbering stops and dwell time. Eleven additional observations were carried out, only focusing on visitors’ interaction with and response to the two digital installations in the final room. In total, thirty-two observations were carried out.

Figure 7: observation sheet example

As expected, observations did indeed provide valuable information on visiting paths and behaviours both for the more traditional exhibition displays and the two meSch installations. One of the most interesting outcomes was that the sequence of action and reaction used in the Tondo installation seemed to be much easier to comprehend by the visitors than the more complex interaction options used in the Symposion installation. Within the latter, visitors actions suggested that the three actions triggering a reaction in the projection—picking up the replica, putting it back on the plinth, and sitting down on the daybed—were not recognised as such. Others seemed to think they had far more influence on the projection than was actually the case, resulting in them using complicated gestures. For example, visitors were observed toasting with the replica, waving it in front of the screen, and drinking from it. Some visitors took the replica kylix and sat on the daybed with it, attentively watching the projection. It was surprising to see how long some visitors would look at the wall projection, without interacting with the replica or the daybed, and as a result watching what was effectively a still image. The music played by the flute, which started when the kylix was placed back, on the contrary was much more noticed and triggered visitors to watch the projection.

The most important challenge faced was that it was impossible to foresee how much time an observation would take. In some instances, visitors would spend 45 minutes visiting the three last rooms of the exhibition. As said before, due to the small spaces and low visitor traffic, it was often inevitable that the presence of an observer would be noticed, potentially affecting the visitors’ behaviour. In some instances it would also happen that a visitor who seemed to have finished his or her visit would reappear on gallery some time later while a new observation had started. This goes to show that even with very linear exhibition design, visitors might not engage with content and objects in the intended order.

3.5 Visitor Profile Questionnaire (VPQ)

A VPQ was developed in order to collect the same baseline information of all visitors who participated in the study, except for the observed visitors. This questionnaire was developed to create profiles of the museum’s visitors. The large number of VPQs that were being collected meant this data could be quantified. The VPQ consisted of two sheets of A4 paper with questions divided in various sections.

The VPQ assisted us in exploring visitors’ level of interest in archaeology, Ancient Greek culture, and art; museum-visiting habits and behaviours in general; and demographic questions. Visitors were also asked how familiar they were with the theme of the exhibition Feint as well as how they used the different available interpretation tools for this visit. The VPQ questionnaire was integrated in the Visitor Experience Questionnaire. A total of 101 VPQs were filled out. Upon writing, the data for 54 VPQs have been inputted. Most visitors who contributed to the study are frequent museum visitors, with a majority (44) claiming they visit museums four times a year or more. Of the visitors, 22 were part of a visiting group, while 16 visited the museum alone; the percentage of single visitors seems exceptionally high but is not alien for visitors of the APM. 32 visitors were male, 20 female, and 1 transgender. The age group of 55- to 64-year-olds is best represented, followed by the 18-to-34 age group, which might be caused by the presence of some student groups on the days the evaluation took place. Participants gave high scores for all three interest areas; archaeology, Ancient Greek culture, and art, with the scores for art clearly surpassing the other two.

As in the Atlantic Wall evaluation, though the questionnaires are easier to process and analyse (as in comparison with interviews or PMMs), inputting of the raw data is time consuming, with strategic decisions that need to be taken on the use of software and statistical analysis tools so that the data is accessible to non-collocated team members.

3.6 Interviews

It was decided to conduct a number of interviews in order to gather more in-depth information about visitors’ motivations, use, and interpretation of both the textual information that was provided and the two digital installations, so as to further investigate possible patterns that would arise from the observations and user experience questionnaire.

Within the research period, certain times were dedicated to recruiting visitors for the interview. Both single visitors and visiting groups who were eighteen years or older were asked to participate. In the case of a visiting group, the interview would be carried out with all members of the group. Visitors were offered a cup of tea or coffee during the interviews, which took place in a quiet area near the museum café. A list of set questions was used for a semi-structured questionnaire. Based on the responses visitors gave, prompts were at times used to encourage visitors to elaborate. After the interview, visitors were asked to fill out the VPQ. In total, seven interviews were carried out. Just like for the Atlantic Wall exhibition, 20 minutes proved to be adequate. However, 20 minutes also seem long after a couple of hours spent in the gallery. For this reason, more visitors declined participation in comparison to the other methods.

3.7 Visitor Experience Questionnaire (VEQ)

The VEQ was designed to provide quantifiable data about the user experience of the exhibition as a whole and the two digital exhibits in particular. The questionnaire consisted of four A4 paper sheets in form a booklet. The first four sections of the VEQ were the same as those of the VPQ. The section about the exhibition Feint was extended with several questions on the objects on display and the interplay of text and the real objects. One of the sections focused on the use of the two digital installations in particular. For the Symposion installation, the first questions were designed to get information about visitors’ understanding of the exhibit, the ease of use, intuitiveness, the relationship with the original objects, and potential added value of the installation. For the Tondo installation, the same series of questions was used, using five-point Likert scales. Additional questions focused on the amount of information in the exhibit (too much or too little), the relationship between the exhibit and the theme of the exhibition, and the degree in which the exhibit could be shared with fellow visitors.

During the period in which the evaluation took place, certain times were dedicated to collecting data through the VEQ. After a brief explanation of the study, the consent form was signed and the questionnaire was handed to the visitor. A researcher was at hand to answer questions, if needed. A maximum of five visitors would be asked to fill out the questionnaire using as a “work” area a long bench situated at the central corridor of the museum, just outside the exhibition. In total, sixty VEQs were filled out.

One of the challenges we were faced with in this case is that a number of open-ended questions resulted in questions from the part of the visitor about what to answer. Also, as with the VPQ, all data had to be processed, and sharing the data in a meaningful format was a challenge, as partners had various preferences with regards to what statistical software to use. Of the fifty-eight VEQs that have so far been analysed, close to 20 percent had not used either of the two digital installations. This percentage is close to the one observed for the questionnaires of the Atlantic Wall exhibition. Finally, the Affective Impact questionnaire that had been integrated in the VEQ at the Atlantic Wall exhibition was used in the Feint exhibition, too, but as a separate questionnaire that was filled in by 18 visitors.

3.8 Personal Meaning Maps (PMM)

The PMM method was only trialled at the APM, with the exhibition thought as a very good case-study for trying out this evaluation methodology. PMMs have been used by various researchers in the United States, United Kingdom, and Australia, mostly to gauge a visitor’s knowledge of a certain, broad exhibition topic (Falk et al., 1998; Adelman et al., 2000). A visitor is asked to write words or phrases s/he associates with the topic on a sheet of paper. Often, the same visitor is asked to fill out a PMM both before and after visiting the exhibition, in order to compare his or her answers. Sometimes, filling out the PMM is accompanied by a semi-structured interview, during which prompts are used to get a better understanding of a visitor’s responses (Fritsch & Moussouri, 2004).

Figure 8: example of a Personal Meaning Map

In our case, the PMM was used to gather information about people’s understanding and meaning-making around objects that were accompanied by text labels, in comparison to objects that were accompanied by digital exhibits. Therefore, each visitor was given two PMMs. One was a blank sheet of A4 paper showing a small image of one of the objects on display (Figure 8); the second sheet contained a similar picture, but of one of the kylikes in the final room of the exhibition, both with their names and dates. Before and after the visit, visitors were asked to write any words, phrases, or questions they associated with the objects on the sheets of paper. Different colour pens were used each time, to make it easy to differentiate the earlier answers from the post-visit answers. In total, sixteen PMMs were completed. It was striking to see the range of different ways in which the forms were filled out. Some wrote associative texts, some summed up their knowledge, while others listed questions at first and added answers after the visit. The responses also show that visitors engage with objects in many different ways, ranging from aesthetic appreciation to wondering about use, decorative style, and associated feelings. While most visitors were happy to participate, analysing the data of the PMM proves to be a challenge. Despite its depth, this type of data is difficult to quantify and is not homogeneous.

3.9 Data gathered versus resources committed

The data gathered during the Feint exhibition audience research study consists of 60 VEQs, 31 observations, 16 PMMs, 18 Affective Impact questionnaires, and 101 VPQs. Though the overall surface the exhibition covered was approximately four times larger in comparison with the Atlantic Wall exhibition, the theme of the Feint exhibition, the very nature of Allard Pierson Museum as a museum for art lovers, the linear (in comparison with the Atlantic Wall) layout of the exhibition, and the small size of the Tondo room where the two digital installations were placed, made the visitor flow continuous yet easier to monitor and follow. Thus, with hindsight, the optimal number of researchers on site given this setting was—depending on the visitors’ flow—from one to three, in comparison with the resources committed for the Atlantic Wall exhibition. Figure 9 provides a comparative overview of the methods used in both studies in which only adult visitors participated.

Figure 9: overview of methods used

4. Best practices, lessons learned, and guidelines

This section presents lessons learned as well as practical guidelines for devising and carrying out audience research studies, with a focus on the specific challenges of evaluating tangible and embodied multisensory museum visiting experiences.

1. Plan and articulate carefully your research questions, and choose your methods wisely. Different methods have different strengths and weaknesses, with this being particularly true when evaluating digital learning offers with which visitors have no prior experiences. Be open in identifying unanticipated findings and outcomes.
2. Trial your evaluation protocol, and adapt if needed. In both studies presented, the evaluation protocol had to be tweaked and adjusted so as to be more efficient. Even after tweaking, what we thought would be a useful and easy-to-use coded observation template at Museon underperformed in comparison with a traditional notebook. In APM, on the other hand, due to the more linear exhibition design, the devised observation template and coding scheme worked well.
3. Know your study and exhibition space, and explore the possibilities the space gives you that might allow, enable, or prevent you from deploying a specific evaluation methodology. The use of video in the Atlantic Wall exhibition assisted us in gathering invaluable data but could have been detrimental to the visitor experience if used in the Feint exhibition.
4. Log, structure, and model your data if possible on an everyday basis, particularly if the decision to combine both qualitative and quantitative methodologies comes from a need to triangulate, but also study in depth your evaluation participants (i.e., some of our participants have been observed and/or interviewed, filled-in a questionnaire, or contacted for a follow-up interview). If you are working within a large, multidisciplinary, and non-collocated team, data structuring becomes even more fundamental, and so do the strategies for sharing and the statistical analysis software that will help you analyse your data.
5. Protect and take care of your study participants. Be aware that participating in your study is not part of their agenda. Thank them for their time and participation, and ask whether they would like to be updated on how they helped you; this was the case with a large number of our study participants.
6. The use of informed consent forms is almost universally a prerequisite. Always prepare an information sheet and make this part of the study (presentation, goals, use and protection of personal data, consent form) as concise and easy to comprehend as possible. If your participants need to sign a consent form or provide other personal data, explain why. If working within an international team, study the relative legislation (i.e., use of video, use of data, data act protection, etc.) and file the appropriate ethics applications well in advance, as legislation and regulations might differ considerably from country to country and institution to institution, not just for data gathering but also for storage and sharing.
7. If you are working with questionnaires, be sure that the questionnaire is well designed and not excessively long. Our experience has shown that the maximum a museum visitor can take is four, single-sided, A4 pages. Make sure to provide space for visitors’ thoughts and comments that might not fit in other questions. Approaching visitors at the start of their visit to ask them whether they would volunteer to fill in a questionnaire after their visit can work miracles.
8. Be sufficiently prepared to handle visitors of different nationalities and cultural backgrounds, and be aware of different cultural codes that different groups of visitors or minorities might be using. Translate all supporting documents in advance, as sometimes a language might not have an exact term for the concept you need to reveal (e.g., there is not an exact parallel for the English word “engaged” in Dutch).

5. Conclusion

While introducing this work, as well as devising our evaluation strategies, we focused on the particular challenges of evaluating tangible and embodied multisensory museum visiting experiences. But is it possible to untangle and evaluate specifically the digital, embodied, tangible, or multisensory components of an exhibition in comparison with evaluating the more “traditional” ones? Our experience showed that this is only possible up to a point: a truly multisensory, embodied, and tangible museum visiting experience will seamlessly merge the digital with the physical, the “traditional” with the less well-known or untraditional, allowing the visitors to get the best of both worlds and to smoothly make use of all exhibition components they are provided with. In both of these large-scale studies, we used more or less well-known and tried-out evaluation methodologies from the Visitor Studies and Human Computer Interaction communities, so as to gain insights and comprehend the potential of the -still under-explored- impact of the intertwining of the physical with the digitally mediated tangible, embodied, and multisensory. At all times, we tried to be aware that, no matter how thoroughly an evaluation study has been prepared, “you can’t evaluate the stars” (Diamond et al., 1999): and each and every single museum visitor should be rightfully considered as one.

Acknowledgements

The research described in this paper is part of the Material Encounters with Digital Cultural Heritage (meSch) project, ICT Call 9: FP7-ICT-2011-9, GA 600851. The authors would like to thank the museum staff of Museon and Allard Pierson Museum for all their support and assistance, as well as the museum visitors for participating in our study.

References

Adelman, L.M., J.H. Falk, & S. James. (2000). “Impact of National Aquarium in Baltimore on Visitors’ Conservation Attitudes, Behavior, and Knowledge.” Curator: The Museum Journal 43(1): 33–61.

Ciolfi, L. (2003). “Understanding spaces as places: extending interaction design paradigms.” Cognition, Technology & Work 6: 37–40.

Diamond, J., J.J. Luke, & D.H. Uttal. (1999). Practical evaluation guide: Tools for museums and other informal educational settings. Rowman Altamira.

Dudley, S. (2010). “Museum materialities: Objects, sense and feeling.” In S. Dudley (ed.), Museum Materialities: Objects, Engagements, Interpretations. Routledge. 1–18.

Falk, J.H., T. Moussouri, & R. Coulson. (1998). “The Effect of Visitor’s Agenda on Museum Learning.” Curator: The Museum Journal 41(2). doi:10.1111/j.2151-6952.1998.tb00822.

Fritsch, J., & T. Moussouri. (2004). Jameel Gallery of Islamic Art Front-end Evaluation Report. London. Consulted January 29, 2016. Available www.vam.ac.uk/files/file_upload/17174_file.doc

Goulding, C. (2000). “The museum environment and the visitor experience.” European Journal of Marketing 34: 261–278.

Gunn, I., & S.-J. Harknett. (2010). “Wonderland or Wander-Land? An Integrated Approach to Evaluation.” Proceedings of the NAI International Conference, Queensland Australia, April 13–17, 106–112.

Heath, C., J. Hindmarsh, & P. Luff. (2010). Video in Qualitative Research: Analysing Social Interaction in Everyday Life. London: Sage Publications Ltd.

Hooper‐Greenhill, E. (2004). “Measuring learning outcomes in museums, archives and libraries: The Learning Impact Research Project (LIRP).” International Journal of Heritage Studies 10(2): 151–174.

Johnson, L., S. Adams Becker, V. Estrada, & A. Freeman. (2015). NMC Horizon Report: 2015 Museum Edition. Austin, Texas: The New Media Consortium. Consulted January 29, 2016. Available http://cdn.nmc.org/media/2015-nmc-horizon-report-museum-EN.pdf

McManus, P.M. (1993). “Memories as indicators of the impact of museum visits.” Museum management and curatorship 12(4), 367–380.

Petrelli, D., E. Not, A. Damala, D. van Dijk, & M. Lechner. (2014). “meSch–Material Encounters with Digital Cultural Heritage.” In Marinos Ioannides et al. (eds.), Digital Heritage. Progress in Cultural Heritage: Documentation, Preservation, and Protection. Springer International Publishing. 536–545.

Van der Vaart, M., & A. Damala. (2015). “Through the Loupe: Visitors’ Engagement with a Primarily Text-Based Handheld AR Application.” Proceedings of the Digital Heritage International Congress, IEEE, 553–560.

vom Lehn, D., C. Heath, & J. Hindmarsh. (2002). “Video-based Field studies in Museums and Galleries,” Visitor Studies Today 5, 15–23.

Zancanaro, Massimo, Elena Not, Daniela Petrelli, Mark Marshall, Taco van Dijk, Martin Risseeuw, Dick van Dijk, Adriano Venturini, Dario Cavada, and Thomas Kubitza. (2015). “Recipes for tangible and embodied visit experiences.” MW2015: Museums and the Web 2015. Published January 30, 2015. Consulted January 29, 2016. Available
http://mw2015.museumsandtheweb.com/paper/recipes-for-tangible-and-embodied-visit-experiences/