Evaluation as Governance:
The Practical Politics of Reviewing, Rating and Ranking on the Web

There is hardly anything these days that is not being evaluated on the web. Books, dishwashers, lawyers, ex-boyfriends, doctors, haircuts and websites are just some examples targeted by novel review, rating and ranking schemes. Used in an increasing number of areas, these schemes tend to be conceptualised as techno-scientific solutions to public problems. By soliciting and aggregating feedback and distributing it as comments, lists, ratings and stories, they are thought to make hidden qualities transparent, hold people to account and foster participation. At the same time, the rapid proliferation of evaluative practice and its far-ranging implications have raised a number of concerns.

In this project, I take a closer look at the mundane, everyday practices that go into establishing, maintaining, and (at times) disrupting these schemes and the ways in which they reconfigure networks of governance and accountability. Building on recent arguments in Science and Technology Studies (STS), governance theory, and ethnomethodology, I develop a relational view of governance and accountability as a contingent and situated accomplishment. The project therefore aims to recast the current debate about online reviews, ratings and rankings and offer an alternative outlook by "respecifying" (Garfinkel, 1991) evaluation as governance. Who, which, or what is governing what, which, or whom—and what are the implications? How can a better understanding of evaluative practice contribute to theoretical and policy debates about transparency, accountability and "democratic" participation? And how to study all this in view of vastly distributed information systems?

Empirically, I explore these questions ethnographically by following reviews and ratings in two different settings: web-based patient feedback, a "small data" setting, in which evaluations are thought to be grounded in individual experience of care; and search engine optimization (SEO), a "big data" setting, which is characterized by automated algorithmic ordering. The project shows that attending to evaluative practice—whether through counting or accounting—allows us to appreciate the importance and mechanics of non-obvious modes of governance, including the moderation of “experience”, strategies of dealing with moments of wonder (when you realize it could be otherwise), the un-knowability of relevance accreditation, as well as different ways of "ethicising" participation in evaluative practice.

The project is generously supported by a DAAD Doctoral Scholarship and a PGP Corporation Scholarship.

Download: email me for a copy!

Follow-up projects

One outcome of this research was the ESRC-funded How's My Feedback? project, a collaborative design initiative to rethink and evaluate web-based review and rating schemes.

Related publications

• Ziewitz, M. (2012). Evaluation as governance. The practical politics of reviewing, rating, and ranking on the web. DPhil Thesis. University of Oxford.
• Ziewitz, M. (2011). How to attend to screens? Technology, ontology and precarious enactments. Encounters 4(2): 203-228.

Related talks and presentations

• “Can stories change the world? Notes from an ethnography of web-based patient feedback,” Health Experiences Institute/Management in Healthcare speaker series, Green Templeton College, Oxford, UK, November 12, 2012
• “In search of the algorithm: A not quite random walk,” Society for the Social Studies of Science (4S)/EASST Annual Meeting, Copenhagen, Denmark, October 18, 2012
• “Ambiguous analytics: Achieving relevance in search engine optimization,” Society for the Social Studies of Science (4S) Annual Meeting, Cleveland, OH, USA, November 3, 2011
• “Evaluation as Governance?”, InSIS and ValueS Workshop, Saïd Business School, Oxford, UK, October 22, 2011
• “How’s my feedback?,” STS Summer Workshop and How’s My Feedback? Conference, Saïd Business School, University of Oxford, Oxford, UK, June 28, 2011 (with Steve Woolgar)
• “How’s my feedback? Six puzzles and some thoughts on web-based review and rating schemes,” Harvard-Yale-MIT Cyberscholar Working Group, MIT Media Lab, Cambridge, MA, May 11, 2011
• “What architecture for multi-stakeholder participation and policy-making?,” Internet Freedom Conference, Panel Speaker, Council of Europe, Strasbourg, France, April 19, 2011
• “What is it with experience? Postings, governance and multiplicity in web-based patient feedback,” Knowledge, Innovation and E-health Research Group Seminar, Warwick Medical School, United Kingdom, March 2, 2011
• “Researching search: Ethnographic stories from the search marketing industry,” Business School, University of Exeter, United Kingdom, February 2, 2011
• “How to attend to screens? Mundane encounters at the int(er/ra)face,” PhD Workshop “Framing Screens: Knowledge, Interaction, Practice”, IT University of Copenhagen, Denmark, September 29, 2010
• “Accountability multiple? The practical politics of web-based patient feedback,” 4S Annual Meeting 2010, Tokyo, Japan, August 26, 2010
• “Innovation in governance as a practical accomplishment,” First Berlin Forum Innovation in Governance, Technical University Berlin, Germany, May 20, 2010
• “Governance and accountability in/of/through internet-based rating systems,” ESRC Workshop “Imagining and Demystifying Empirical Socio-Legal Research,” Loughborough University, United Kingdom, April 6, 2009

Updated: January 25, 2013