Sensing context: Reflexive design principles for intersensory museum interactions

Daniel Harley, Ryerson University, Canada, Melanie McBride, York University, Canada, Jean Ho Chu, Georgia Institute of Technology, USA, Jamie Kwan, Ryerson University, Canada, Jason Nolan, Ryerson University, Canada, Ali Mazalek, Ryerson University, Canada

Abstract

Keywords: museum experience, design, sensory interaction, narrative, multisensory, intersensory

1. Introduction

Museums have traditionally presented historical artifacts behind glass displays, decontextualized from their social, cultural, or environmental locations, which are typically communicated through visual and text media. This “look, but don’t touch” model not only produces a physical distance between visitor and artifact, but also further abstracts an object’s context by limiting the visitor’s participation in sensory learning and meaning-making. The glass case tradition is the most implicit endorsement of an ocularcentric model of knowledge and knowing, which “reduces vision to the interpretation of images” (Ingold, 2011, p. 223). We suggest that directly experienced sensory properties, behaviours, and qualities of historically situated objects are especially consequential for museums. Our use of the term “intersensory” stresses the reciprocal relationship of our senses as a total ecology of perception, while also differentiating our approaches and conceptual foundations from those of the multisensory paradigm.

Despite the acknowledgement that museum visitors seek experiential and participatory interactions with history, “multisensory museum experiences are still few and far between” (Levent & Pascual-Leone, 2014, p. xviii). As well, the multisensory museum paradigm has yet to meaningfully integrate the chemical senses of smell and taste. At best, emerging conceptualizations of the multisensory museums offer novel routes to sensory engagement; at worst, they appeal to one-size-fits-all notions of immersion. We question the uncritical endorsement of certain multisensory perspectives, particularly those that are said to have “swept” the academy (Levent & Pascual-Leone, 2014) without reference to counter-perspectives. It seems premature to suggest that these claims enjoy uncontroversial adoption across disciplines.

A critical assessment of multisensory perspectives is needed if we are to design for meaningful and contextual museum experiences, particularly when claims of immersion, engagement, or learning are advanced. We argue that design involving sensory phenomena should be situated relative to the ever-changing conditions and contingencies of spaces, atmospheres, and bodies. Beginning with a brief review of the literature of the multisensory museum, this paper presents a case study of our design and prototype of an intersensory encounter with early sixteenth-century boxwood prayer-nuts. Initiated as part of author Kwan’s applied master’s thesis, our interactive prototypes grew into a larger collaborative project (Kwan et al., 2016) among researchers in human-computer interaction, education, and media at the Responsive Ecologies Lab and the Synesthetic Media Lab at Ryerson University and Georgia Tech. While we did not have the opportunity to formally implement our prototypes within a museum, our transdisciplinary lab environment allowed us to experiment with interactions traditionally unavailable in museums. Our prototypes makes use of smell, touch, sight, and sound to elicit the historical, social, and cultural context of the artifact. We conclude with five considerations for “intersensory” design (McBride & Nolan, in press) founded on inclusive, accessible, and relatively Do-It-Yourself (DIY) approaches.

2. Reconceptualizing the multisensory museum: A critical review

The recognition that the senses are integral to human learning, culture, and communications is said to be reflected in the so-called multimodal turn (Jewitt, 2009; Levent & Pascual-Leone, 2014) in education, technology, digital communications, and museology. Proponents of multimodality, such as Gunther Kress, Theo Van Leeuwen, and Carey Jewitt, argue that there is a need for “substantially” new forms of literacy and communication that move “beyond language” (Kress & Selander, 2012; Jewitt, 2009). This logic also informs trends in multisensory exhibit design that look to digital media technologies as a means of creating dynamic mediations of traditional exhibits. However, much of what is said to be technologically innovative in the multisensory museum literature still predominantly emphasizes the visual, paying little attention to the physicality of the body or marginalized senses, such as proprioception, tactility, taste, and smell. Furthermore, these technologies have critical challenges of their own.

For example, while augmented reality can supplement text descriptions with infographics or imagery to provide missing context or detail, actual artifacts can become secondary to their digital counterparts, either obscured by the technology or in need of virtual reconstruction. These technologies do not necessarily require fundamental changes to existing structures, and the trend towards the virtual continues to neglect the nondominant senses such as smell and taste. There is some question as to whether these innovations serve to establish sensory connection or further isolate objects behind a virtual veneer.

Similarly, the notion that tactile experiences may be more “scientifically” engaged through emerging neuroscientific or cognitive models (Levent & Pascual-Leone, 2014) intimates that we are already “touching” exhibits and there is no need to change. However, the prospect of actually touching exhibits has been enthusiastically received, if not widely implemented, using replicas in exhibits for the visually impaired, in children’s museums, and more recently with tangible interaction technologies (Chu et al., 2015). These tangible technologies seek to encourage interactions that include physical movement and tangible manipulation in an effort to move beyond decontextualized interaction models in which visitors are limited to pressing buttons.

Given our use of smell for the following case study, we pay special attention to the treatment of smell in the museum literature and its overwhelming appeal to psychological or neuroscientific claims (Levent & Pascual-Leone, 2014). By and large, these accounts deploy smell’s commonly cited use-value as a trigger or channel for recalling past experiences or simulating immersion (McBride et al., 2016). This return to interiority fails to recognize the role of smell as a form of “invisible information” (Agapakis & Tolaas, 2012) that is engaged through the embodied and physiological practice of detecting, identifying, and discriminating between odours in our immediate environment. As architect and urban planner Victoria Henshaw (2013) notes, built environment professionals do not adequately consider the role of the body or “bodily states” (p. 28), such as a visitor’s relative hunger, illness, fatigue, excitement, or hormonal activities, in shaping or influencing the perception, intensity, or quality of sensory ambiances.

These perspectives highlight a critical tension between universalist claims of multisensory design and the often unstable variables of spaces, atmospheres, and bodies. A concern with such contingencies reflects James Gibson’s theory of “ecological perception” (Gibson, 2013), which advances an orientation towards the empirical discrimination of sensory affordances in the environment. As Ingold (2011) elaborates, an ecological concern with perceiving the immediate environment is in opposition to perspectives that reduce the body and the senses to “instruments of playback, capturing moments of experience and relaying them back to a reflexive consciousness” (p. 137). This latter position appears to underwrite the conceptual foundations of multisensory claims that are preoccupied with interiority over physicality. The intersensory museum, then, is one that understands the inseparable and responsive relationship of bodies and senses in a given environment.

In the following discussion of our own intersensory case study, we seek to demonstrate the relative simplicity and effectiveness of interactions focused on sensing in the “present tense” (McBride & Nolan, in press). Our three prototypes attempt to speak to missing modalities, such as smell and touch, while simultaneously recontextualizing approaches to sound and vision. We then propose a series of considerations that can be used at the earliest stages of intersensory design.

3. Contemplating the senses: A reflexive case study for intersensory design

Figure 1: the interior (left) and the exterior (right) of a sixteenth-century prayer-nut. The prayer-nut is approximately 6 centimeters in diameter. (Image courtesy of the Metropolitan Museum of Art)

Crafted in the Northern European low countries in the sixteenth century, prayer-nuts are small, intricately carved devotional objects (Figure 1) that were created for a rising merchant class as symbols of wealth and piety. The outer carvings show Gothic patterns and architectural motifs; the inner carvings depict biblical scenes, such as the Passion, accompanied with descriptive Latin text. Prayer-nuts were also used as fragrant pomanders containing aromatic materials thought to have apotropaic (protective) qualities (Falkenburg, 1999; Scholten, 2011). In museums, prayer-nuts are presented in glass displays, leaving visitors to imagine their sensory qualities without the chance to encounter them directly.

In an effort to convey and contextualize the historical, sensory, and embodied information of the prayer-nuts, we designed three interactive prototypes. Using archival images from available museum resources, the prototypes are three-dimensionally printed tangible objects, borrowing the form of the artifact to convey a tactile sense of scale and texture, and embedded with sensors to detect the interactions and deliver appropriate feedback. As our goal was to design embodied ways of experiencing the artifact without “replacing” it, we did not create exact copies that might discourage an examination of the intricacies of the original. Future iterations of our prototype would require more robust technological solutions as well as greater attention to look-and-feel issues, such as the quality of the materials.

Many of these early design decisions are also an indication of some of the inherent challenges of remediating or conceptualizing historical content. Our choices are interpretations of scholarly work, which are themselves interpretations of historical truths. The interactions are not a means of recreating history; instead, they are a means of engaging with some of the available historical information. Each prototype focuses on a specific historical quality of the artifact, and the digital and sensory materials that we use are our attempt to provide visitors with more than one way to experience these qualities. As the interactions are experienced separately and in no requisite order, each prototype is an alternate entry point into the history and use of the prayer-nuts.

Figure 2: “Visual Voyage” interactions (top) and the highlighted scenes (bottom). The visitor’s touch highlights the corresponding section in an adjacent display and provides text describing its significance.

“Visual Voyage” is inspired by the visual and tactile qualities of the prayer-nuts. Scholten (1999) writes: “Opening [the prayer-nut] allows the worshipper, as it were, to enter into the episode depicted. The manual act of opening and closing that this meditative technique entails is reminiscent of the opening of a prayerbook, or even of the panels of a large altar” (p. 19). The visitor opens the tangible object to reveal a representation of the biblical scene, triggering a projection of an enlarged and more detailed version. By touching specific parts of the image represented in the tangible object, corresponding sections in the projection are highlighted with additional text that explains its origin and importance (figure 2).

Designed as a simple cause-and-effect interaction, text that would traditionally be presented as “wall-text” is generated by touch. As our interactions are based on the original use of the prayer-nuts, we invite visitors to develop an embodied understanding of how and why the original owners may have used these artifacts. We recognize that these interpretations about the original owner are speculative, and there is no true substitute for handling the actual artifact. However, the prototype is a tool to experience the artifact’s empirical qualities, like its weight, tactility, or smell. The enlarged image of the interior of the prayer-nut acts as a reminder of the intricate detail of the carvings, inviting visitors to look again at the expert craftsmanship of the original.

Figure 3: “Experiencing Spirituality” interaction (top) and projected animations (bottom). The visitor is invited to adopt a meditative breathing pace while listening to conceptual soundscapes of the era.

“Experiencing Spirituality” is inspired by the personal prayer practices that, in the sixteenth century, began to transition outside the church. Falkenburg (1999) suggests that the prayer-nuts “aided and directed the soul during prayer and meditation” (p. 32), and represented a “complete meditative world encompassing in itself the entire rosary prayer” (p. 41). The visitor begins by listening to the ambient sounds of a sixteenth-century marketplace. When the visitor picks up the tangible object, text and animations encourage the visitor to adopt a relaxed and meditative breathing pace. A display showing the historical marketplace begins to transition into a scene depicting a cathedral; the sounds of the marketplace transition to sixteenth-century sacred music (figure 3).

Designed as a contrast to the brief, goal-oriented interactions commonly found in museums, this interaction includes opportunities for contemplation. The time that the visitor takes while listening to the sounds of the marketplace is not defined. The conceptual soundscape includes calls from merchants, bleating livestock, birds, wind, and a multitude of voices, providing identifiable information about a time and place. The sensory affordances invite the visitor to contrast the acoustic ambiances of contemporary spaces, such as museums, with those of a traditional marketplace. Similarly, these soundscapes are an opportunity for the listener to contrast the secular and sacred music of the time. Although conveying a sense of the meditative practice is the outward goal of this prototype, our design includes these additional layers of engagement to provide visitors with alternate paths of inquiry.

Figure 4: “Scents of Power” interaction (top) and animations (bottom). The visitor is invited to smell aromas inspired by historical ingredient lists. We draw attention to the raw materials and their significance with images and text.

“Scents of Power” communicates the olfactory ambiances of sixteenth-century rosaries, pomanders, and prayer-nuts. Scholten (2011) writes:

A fragrant substance—perhaps a mixture of dough, herbs and dried flowers—would have been put in the empty space between the removable inner reliefs and the outer shell and, used as a rosary pendant, the prayer-nut would have released a sweet scent through the quatrefoil openings (p. 451).

This prototype features two replica prayer-nuts, affixed to a table with animations projected directly onto the replicas and the tabletop. Opening the replicas activates a dynamic and colourful animation of the fragrant materials contained within, accompanied with descriptions of the nature and meaning of the aromatic contents. Further text prompts and visual cues invite the visitor to compare and identify the scents. When both objects are opened, an animation connects the objects, denoting the blending of aromas into a fragrant composition (figure 4).

Figure 5: scents created by author McBride for “Scents of Power,” inspired by historical texts.

The marginalization of smell from Western historical accounts (Classen, Howes, & Synnott, 1994) is also evident in the lack of recipes or compositional details of the “sweet-smelling ingredients” used in prayer-nuts (Falkenburg, 1999). Given the experimental nature of our prototype (i.e., a student project that was not formally implemented in a museum), we took a DIY approach to smell composition (McBride & Nolan, in press) using ingredients identified in scholarly texts (figure 5) that we obtained from local sources. Seemingly obscure materials listed in scholarly articles, such as “styrax,” were accessible in their common forms (i.e., benzoin resin) found in specialty and health food stores. In keeping with the historical scent compositions identified in the literature, we created a pair of scents intended to provide contrast and complexity. The first and more complex of the two is composed of nutmeg, cinnamon, clove, frankincense, and rose. The second is a simple combination of lavender and cinnamon. In order to compensate for the missing aromas of the original prayer-nuts’ fragrant wood housing, each scent included a small amount of sandalwood and cedarwood essential oils.

By contrasting relatively accessible aromas, such as lavender and cinnamon, with a more complex and dimensional scent, we invite visitors to compare two different scent compositions that might have been common to prayer-nuts. Images of the raw materials are accompanied with text describing their historical significance. In addition to the introspective qualities of smelling, the specificity of these ingredients encourages the development of embodied literacies of aroma discrimination that can be applied outside the museum. Conceptualizing smell as a distinct source of information rather than as a trigger for affect, we are situating smell as a mode of physical knowledge (McBride & Nolan, in press).

Throughout the development of our prototypes, we noted the need for guiding principles and practices, several of which are necessary at the earliest stages of design. Beginning with the intersensory design principles by authors McBride and Nolan (in press), the following considerations extend the insights gained from developing our prototype. These principles can inform a more robust iteration of our project and also, by extension, a critical foundation for intersensory museum experiences:

1. Perspectives and paradigms: What’s your sensory standpoint?

Inspired by methodological interventions from feminist and critical social research, the “sensory standpoint” (McBride & Nolan, in press) is an important initial stage of sensory design. Adopting a sensory standpoint involves an acknowledgement of beliefs, assumptions, and values that influence our chosen sensory paradigms or perspectives. Inspired in part by our own initial discussions about what constitutes a sensory or multisensory experience, this principle asks: To what degree do our chosen sensory paradigms (i.e., multisensory, neuroscientific, psychological) acknowledge counter-perspectives, debates, or controversies associated with our chosen view? Does our standpoint account for implicit appeals to authority or exceptionalism in an effort to justify or qualify our approaches?

2. Reflexive design practice: Sensory autoethnography

Designing sensory encounters for museum visitors should begin with an inquiry of our own “sensory order” (Howes, 2005) and its influence on our sensory standpoint. With our varied sensory experiences and differing cultural and material reference points, we capitalized on our relative investments in sound, smell, and touch. Before designing an encounter with smell, curators or museum professionals might first critically acknowledge their own beliefs, assumptions, or experiences and their influence on chosen strategies or approaches.

3. Critical considerations of equity, diversity, and accessibility

Inseparable from a consideration of space is how the visitor will interact within the space, particularly in the case of those with limited mobilities or impairments. Approaches to accessibility should be initiated by, or in collaboration with, persons with disabilities, not simply conducted on their behalf (Nolan & McBride, 2015). Discussing questions of bodily state (Henshaw, 2013), we recognized a need for transparency in communicative scaffolding of the nature and safety of our aromatic materials, as well as the physical requirements of our interactions. This consideration asks: To what degree do our concepts or practices engage visitors in a “whole body” experience that is inclusive of different kinds of bodies and sensory integrations?

4. Situating the senses within a responsive ecology

To take a more ecological approach to sensory design involves the consideration of the interrelationship between sociocultural, spatial, temporal, and ambient contingencies of experience and perception. While we conceived of a possible layout for our interactions within a museum space, future work would require a deeper consideration of the sensory constraints or affordances of a chosen environment. This is particularly crucial for multi-use facilities in which the ambiances of sound or smell are varied and inconsistent. This principle asks: How might sensory encounters be responsive to the total social, environmental, and spatial ecologies in which they are situated?

5. Design for inquiry: Sensory literacy and learning

While our prototypes represent only an initial stage of development, our interactions suggest opportunities for sensory experiences that would not be available through text alone. For example, encouraging new forms of literacy, such as smelling as a mode of knowing through the empirical comparison of odours, might establish further historical context than the elicitation of purely hedonic responses of like or dislike. Applying McBride and Nolan’s (in press) critical pedagogical perspectives on “intersensory learning inquiry,” this principle encourages an orientation towards “unknowable” rather than prescriptive learning outcomes in order to permit a variety of possible experiences. This is also, finally, an intervention of multisensory educational claims that are advanced in the absence of any pedagogical criteria to assess what is said to be learned.

4. Conclusion and implications

Through our review of perspectives and counter-perspectives of the multisensory museum, and our analysis of the challenges and opportunities that emerged from our own process of designing sensory interactions, this paper aims to reconceptualize current approaches to interaction and exhibit design involving the senses. We created three prototypes based on sixteenth-century prayer-nuts to examine how exhibits can provide visitors with a more embodied and sensory understanding of an artifact. The process of creating the prototypes underscored the importance of an early consideration of the interrelationship of the visitor in the environment. Our proposed considerations function as a call to sensory context that seek to account for sensory orientations that are otherwise neglected by universalist views of cognition, experience, and perception. These considerations also encourage critical reflection into the influence and consequences of our underlying beliefs and perspectives in order to disrupt the inadvertent reproduction of dominant sensory hegemonies.

Future work could apply these principles and practices to other artifacts, beginning with our considerations and experimenting with a variety of sensory interactions. By testing and implementing these interactions in a museum setting, we might have the opportunity to further investigate and assess what is said to be “sensory learning.” We contend that responsive, inclusive, and ecologically situated considerations for intersensory design can encourage a more directly embodied and meaningful experience with history and culture.

Acknowledgements

This research was undertaken, in part, thanks to funding from the Canada Research Chairs program, Canadian Foundation for Innovation, and Ontario Ministry of Research and Innovation.

References

Agapakis, C.M., & S. Tolaas. (2012). “Smelling in multiple dimensions.” Current Opinion in Chemical Biology 16(5): 569–575.

Chu, J.H., P. Clifton, D. Harley, J. Pavao, & A. Mazalek. (2015). “Mapping Place: Supporting Cultural Learning through a Lukasa-inspired Tangible Tabletop Museum Exhibit.” In Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction (pp. 261–268). ACM.

Classen, C., D. Howes, & A. Synnott. (1994). Aroma: The cultural history of smell. Routledge: London.

Falkenburg, R., (1999). “Toys for the Soul Prayer-Nuts and Pomanders in Late Medieval Devotion.” In F. Russell (ed.). A Sense of Heaven: 16th Century Boxwood Carvings for Private Devotion. Leeds: The Henry Moore Institute.

Gibson, J.J. (2013). The ecological approach to visual perception. New York: Psychology Press.

Henshaw, V. (2013). Urban smellscapes: understanding and designing city smell environments. London: Routledge.

Howes, D. (2005). Empire of the senses: the sensual culture reader. Oxford: Berg.

Ingold, T. (2011). Being alive: essays on movement, knowledge and description. London: Routledge.

Jewitt, C. (2009). The Routledge handbook of multimodal analysis. London: Routledge.

Kress, G., & S. Selander. (2012). “Multimodal design, learning and cultures of recognition.” The Internet and Higher Education 15(4): 265–268.

Kwan, J., J.H. Chu, D. Harley, M. McBride, & A. Mazalek. (2016). “Grasping Cultural Context with Multisensory Interactions.” Extended abstract in the Tenth International Conference on Tangible, Embedded, and Embodied Interaction, ACM.

Levent, N., & A. Pascual-Leone (eds.). (2014). The multisensory museum: cross-disciplinary perspectives on touch, sound, smell, memory, and space. Maryland: Rowman & Littlefield.

McBride, M., & J. Nolan. (in press; expected 2016). “Including smell: An intersensory curriculum, by design.” In V. Henshaw, K. MacLean, D. Medway, C. Perkins, & G. Warnaby (eds.). Designing with smell: practices, techniques and challenges. New York: Routledge.

McBride, M., D. Harley, A. Mazalek, & J. Nolan. (2016). “Beyond vaporware: considerations for meaningful design with smell.” In CHI’16 Extended Abstracts on Human Factors in Computing Systems, ACM.

Nolan, J., & M. McBride. (2014). “Beyond gamification: reconceptualizing game-based learning in early childhood environments.” Information, Communication & Society, 17(5): 594–608.

Nolan, J., & M. McBride. (2015). “Embodied semiosis: Autistic ‘stimming’ as sensory praxis.” In In P. Trifonas (ed.). The international handbook of semiotics (pp. 1069–1078). Berlin: Springer.

Scholten, F. (1999). “Prayer-Nuts and Other Boxwood Micro-Carvings.” In F. Russell (ed.). A Sense of Heaven: 16th Century Boxwood Carvings for Private Devotion. Leeds: The Henry Moore Institute.

Scholten, F. (2011). “A prayer-nut for Francois Du Puy.” Burlington Magazine 153(1300): 447–451.