Early on in the artcasting project I conducted interviews with a range of stakeholders involved in evaluation within museum and gallery settings. In order to inform the development of artcasting it was important to understand how evaluation was experienced and perceived by those involved in the process.
Nine interviews were conducted representing delivery (4), operational management (2) and strategic (3) perspectives. Interview questions focused on exploring respondent experiences of evaluation, identifying challenges and eliciting examples of creative, innovative and digital evaluation practice. Interviewees were also asked to characterise their current evaluation practice through metaphor.
The interviews were useful in highlighting some tensions around contemporary arts evaluation practice, which I shared at the Connected Communities Heritage Network Symposium held in Lincoln in January (and can be viewed here). These tensions include:
- dominant (numerical) indicators of cultural value focusing on what is easily measurable
- purpose (audience development versus audience engagement; accountability rather than organisational learning)
- differing organisational priorities (delivery takes priority over evaluation)
- method (self-completion surveys / summative evaluation)
- digital approaches provide efficiency (saves manually collating evaluation data)
- little room for invention or imagination in evaluation
With the recent publication of the AHRC Cultural Value Project report it is interesting to find some of these tensions recognised more broadly within the cultural sector, particularly the focus on summative evaluation and accountability agendas.
Asking interviewees to frame their evaluation practice metaphorically proved a generative way to explore the pitfalls and possibilities that thinking about value in the cultural sector seems to evoke.
Those involved in the evaluation process described their practice as transactional (handing over a report to acquire a final tranche of project funding), a chore (routine, burdensome) and, amongst other metaphors, an indeterminate length of string (confused, lacking clarity of purpose). At the same time, evaluation was viewed aspirationally where it was associated with dialogue (between organisations and funders, within organisations where used for organisational learning, where space for reflection on the evaluation material gathered was possible). Lastly, the Holy Grail of evaluation in the cultural sector was described as longitudinal evaluation (unattainable by nature of their reporting requirements but desirable nonetheless, perhaps also holding insight into beliefs about the transformative qualities of the arts).
In May, Jen and I visited Harris Museum & Art Gallery in Preston; and Jeremy and Jen visited Ferens Art Gallery in Hull as part of our programme of dissemination activities. Our aim was to share and reflect on our project findings and to generate discussion around the issues artcasting raises. As part of these workshops we asked attendees to create and share their own metaphors of evaluation. These included:
- A magician pulling a rabbit from a hat where the magician (evaluator) knows what will emerge from the hat in advance of the trick
- A conveyor belt, chore-like, predictable, routine
- A spaghetti junction, multiple strands of activity representing different directions/objectives of different people involved in the process
- Bookends holding together a series of books that can be shuffled and re-ordered and reused multiple purposes
- A finish line – a place from which to celebrate, look back, and reflect on what could have been different
- A headless chicken, for those working directly with the public – there is often a lot going on, and evaluation creates additional tensions and pressures, especially when visitors are not keen to participate
- Oxygen – it’s essential for survival, and we breathe subconsciously, but when we start thinking about it, it becomes much harder and less natural.
- Shark finning – we catch the shark, cut off the fin, throw the rest away, and it can feel wasteful, for example, using postcode data from comment cards and throwing the comments out.
These additional metaphors support the analysis of our interview data and the conceptualisation of evaluation as a collage activity, and show just how varied and complex the picture of evaluation is within cultural heritage organisations.
Our discussions with museum and gallery staff in interviews and workshops highlighted the role of metaphor as a useful device to ‘problematize the meaning and values that people attribute to phenomena’ (Karlsson, 2001:217).