Finding value beyond the dashboardPaper
Alex Green, Wellcome Library, UK, Annelise Andersen, Wellcome Library, UK
Published paper: Finding value beyond the dashboard
In evaluating digital projects, the focus for many cultural institutions has been on quantitative methods, using defined metrics to identify and measure impact, value, and success. Continued efforts to develop standardised frameworks and tools, such as aggregate influence scores and dashboard templates, has meant that this is increasingly viewed as quick, broadly applicable, and easily understood. However, there remains dispute over whether this form of measurement, which places numbers and statistics above more subjective, qualitative response, is too reductive. This, in turn, prompts questions for us when evaluating our digital projects:
• What are we actually measuring, and what are the numbers really telling us?
• Why might these results be significant and privileged over other forms of insight?
• Are we at risk of oversimplifying and narrowing evaluation processes for digital projects because of the abundance and relative ease of digital evaluation tools?
While recognising the value of Web analytics, we have sought to engage with the above questions when designing evaluation of digital projects at Wellcome Library. Drawing from Simon Tanner’s Balanced Value Impact model, we explore iterative approaches to evaluation through agile frameworks, which can be adapted to allow for changing priorities and iterated for the needs of different digital projects. Through a multimodal approach, we aim to combine rigorous data analysis with qualitative methods to reveal emergent patterns and behaviour, in order to better understand the often complex relationships our users have with our digital offer.
This approach has generated rich, nuanced data, giving greater understanding of our audiences and clear pathways to improve our digital services. Though it remains in tension with many institutions’ reporting requirements, it is an approach that is lively with possibility and accommodates the continual evolution and range of response in users’ experiences.
Anderson, M. L. (2004). Metrics of Success in Art Museums (Essay). Getty Leadership Institute. Available at http://www.cgu.edu/pages/7265.asp#metrics
Bamberger, M., Rau, V., & Woolcock, M. (2010). Using Mixed Methods in Monitoring and Evaluation: Experiences from International Development (Working Paper No. 5245). The World Bank Development Research Group. Available at http://evaluation.zunia.org/post/using-mixed-methods-in-monitoring-and-evaluation-experiences-from-international-development/
Charmaz, K. (2006). Constructing Grounded Theory: a Practical Guide Through Qualitative Analysis. London; Thousand Oaks, Calif.: Sage Publications.
Coleman, E. G. (2010). Ethnographic Approaches to Digital Media.
Annual Review of Anthropology, 39, 487–505. doi:10.1146/annurev.anthro.012809.104945
Dobreva, M., O'Dwyer, A., & Feliciati, P. (2012). User studies for digital library development. Facet Publishing.
Malde, S., Finnis, J., Kennedy, A., Ridge, M., Villaespesa, E., and Chan, S. (2013) ‘Let’s get real: A journey towards understanding and measuring digital engagement’, WeAreCulture24. Brighton: Culture24. Consulted September 30, 2014. http://weareculture24.org.uk/projects/action-research/
Meyer, E. T. (n.d.). What are analytics? Toolkit for the Impact of Digitised Scholarly Resources, Oxford Internet Institute. Available at http://microsites.oii.ox.ac.uk/tidsr/kb/42/what-are-analytics
Petras, V., Stiller, J., & Gäde, M. (2013). Building for Success?: Evaluating Digital Libraries in the Cultural Heritage Domain. In C. Cool, & K. Ng (Eds.) Recent Developments in the Design, Construction, and Evaluation of Digital Libraries: Case Studies (pp. 141-163). Hershey, PA: Information Science Reference.
Stein, Robert and Bruce Wyman. "Seeing the Forest and the Trees: How Engagement Analytics Can Help Museums Connect to Audiences at Scale." MW2014: Museums and the Web 2014. Published February 1, 2014. Consulted September 30, 2015.
Tanner, S. (2012) Measuring the Impact of Digital Resources: The Balanced Value Impact Model. King’s College London, October 2012. Available at: http://www.kdcs.kcl.ac.uk/innovation/impact.html
Villaespesa, E., & T. Tasich. (2012). “Making Sense of Numbers: A Journey of Spreading the Analytics Culture at Tate.” In N. Proctor & R. Cherry (eds.). Museums and the Web 2012: Proceedings. San Diego: Archives & Museum Informatics. Consulted January 22, 2015. Available http://www.museumsandtheweb.com/mw2012/papers/making_sense_of_numbers_a_journey_of_spreading
Villaespesa, Elena and John Stack. "Finding the motivation behind a click: Definition and implementation of a website audience segmentation." MW2015: Museums and the Web 2015. Published January 30, 2015. Consulted September 30, 2015.