Voluntary Response Bias and Social Media

In books like Here Comes Everybody and What Would Google Do?, social media are presumed to empower and engage all consumers equally. Because these new technologies lower (or virtually remove) the cost of creating and sharing content, it has been assumed that anyone and everyone is involved in the process.

Clay Shirky describes an episode wherein irate travellers, stuck on a plane for an excruciatingly long delay, organized a mass protest over the internet. Soon, the world knew of their ordeal, and the airline looked horrible as a result.

Jeff Jarvis tells the story about PC manufacturer Dell ignoring customers and losing market share as disappointed customers began sharing horror stories of faulty laptops via blogs and comments. Eventually, the company realized what was happening, and began to engage disgruntled ex- and current customers, going so far as to have top executives write their own blogs and respond directly to questions and complaints.

The observation that both authors (and many others) make is that social media will force institutions to provide better service, because the information infrastructure acts as a mechanism for constant feedback and monitoring. While I do not disagree with this idea, the reality of the situation has some nuance that is only now becoming apparent.

What the authors are missing is that the information gathered and shared by social media–the sum total of the communication–will always suffer from a bias similar to that of self-selection in social science.

Very little research is conducted wherein the pool of respondents themselves choose to take part in a study.  As any researcher knows, self-selected groups are not random samples of a larger society, and findings from such groups are not generalizable to larger groups.

A good example of why this is problematic would be the non-scientific polls that TV stations run all the time. A news anchor might introduce ‘tonight’s question’, and invite viewers to respond online or via text message. At the end of the show, the station will reveal the results.

These results are absolutely worthless for anyone trying to learn anything about the world. If FOX News and msnbc were to pose the exact same question they would find wildly divergent results. That much is obvious, but perhaps less obvious is the fact that neither could validly state that their results were representative of even their own viewership. This is because even among their biased pool of viewers, only a select (non-random) sample will be motivated to make the extra effort to respond to the poll.

What does this all have to do with social media? While Facebook, Twitter, et al. may not be attempting to undertake valid academic studies, the same issues come into play when observers try to claim that they will create anything approaching consumer democracy or force companies to behave better.

From the perspective of institutions, what is important is to isolate those groups and individuals most active in content creation and sharing and catering to their tastes. Of course, this is a very, very complicated task. There are two main complications in doing so: (1) the inherent difficulty of predicting (or tracking) who most likely to be active and influential in this sphere, and (2) the fact that different people will be motivated to create and share under different circumstances.

The former is difficult, although not impossible, to overcome. The best place to start looking for the answers would be the academic work of Eszter Hargittai and Richard Florida. Hargittai, through surveys conducted at the University of Illinois-Chicago, has demonstrated that while most college students are consumers of social media, only a non-random sample are producers. Demographically, we can tell who these people are: the better-educated, privileged members of society. Of course, we need to understand that only a small sample of this group actually produces content, so we can look to Florida’s work on the ‘creative class’ to figure out what motivates certain people to create and share content and ideas more than others.

The point is, we can figure out who is more likely to create and share content online.

I opened this post with references from Shirky and Jarvis to illustrate the latter point. People who would normally be less likely to be active in discussing companies or experiences will be far more likely to do so when those experiences fall to an extreme end of the spectrum. If you want to see an example, check out reviews on Yelp, Amazon, or any similar site. On five-point scales, you will see an exorbitant amount of fives and ones, and relatively few threes or fours.

In part, this may be due to a tendency to exaggerate one’s opinion on the internet. But there is also certainly the dynamic that people just are not interested in taking the effort to share their opinions of relatively adequate of average experiences. And it’s hard to imagine that they ever will be.

Really, we should recognize that the story we’re seeing unfold isn’t exactly ‘Here Comes Everybody’, but rather ‘Here Come Some People All the Time, and Some Others Some of the Time, Based on the Quality of their Individual Experiences’. It isn’t catchy, and it isn’t simple. But neither is the world. And the lesson for companies and institutions isn’t ‘if you don’t provide good service, people will talk’, but ‘if you provide terrible service, people will talk. On the other hand, if you provide adequate service, you don’t really have to worry’.

9 Responses to “Voluntary Response Bias and Social Media”


  1. 1 Charles July 29, 2009 at 1:20 pm

    I don’t know if I’ve ever taken the time to leave a review when I purchase a product from someone on eBay or Amazon. I’ve always treated the experience much as I would have treated buying the product in a store.

    When I enter a store, my intent is to find the thing that I went there to buy. I obtain it from the shelf and store it on my person. I take it to the register and pay for it. I then go back to my car and leave. I don’t stop people in the parking lot and say, “The employees in this store sold this to me.” That is what a store is SUPPOSED to do. I suppose that had my attempt to purchase been refused, I would have been upset. At that point, it is likely that I would have cussed the store and the employees out to anyone that was within earshot. This has rarely happened (although it did happen a few weeks ago. There was a used 2nd Edition D&D Player’s Handbook on the shelf at The Fortress Comics and Games on Albert in East Lansing. I expected there to be a discount sticker on it somewhere since it was used and two editions old. Failing to find one, I asked the clerk what the price was. He said that he didn’t know what was being asked for it. He then got up from the counter and watched TV in a different part of the store. I’ve told many people about it and have no intention of ever going back unless I’m simply planning to [redacted]).

    I think this holds true with your idea that you will only get reported or commented on should you choose to provide terrible service. Although, I think that many of the 1- and 5- star treatments that many items get is simply the result of people not caring about the quality of their review. It’s not necessarily the case that only the people who legitimately believe that the item or the service actually warrant that kind of treatment. You’ve pretty much said this already.

    Also the Burger King in Frankenmuth is really slow. Don’t eat there.

  2. 2 Charles July 29, 2009 at 1:20 pm

    by the way, I like the new format.

  3. 5 Tim Marco July 29, 2009 at 2:41 pm

    Burger King Line -> +1 xp

    You’re right about getting back to the front page. I’m just fooling around with different ideas for the design now. I’ll see what I can do about it.

    • 6 Mike July 30, 2009 at 9:00 am

      It’s not the perfect solution, but you could simply change the title of the main wordpress page from “blog” to “home”. Then when you’re viewing an individual entry, clicking “home” will always take you back to the front page.

      I like when the title links back to the homepage, but since the header is a single large .gif, that’s a bit more complicated. (The new header looks great, BTW.)

      Great post, by the way. Definitely one of the more troublesome aspects of online research. On your last point, I do think that in the long-run, adequate service doesn’t cut it. Eventually the companies that provide exceptional service get noticed for it. Feedback is a lot more responsive and reliable when it comes to terrible experiences, however.

      • 7 Tim Marco July 30, 2009 at 9:13 am

        Thanks for the input. The only issue is that my WordPress is running on a single .php file that doesn’t differentiate the header between the default (home) page and blog entries, so it seemed a little redundant to have a ‘home’ link on the home page. But it’s a better solution. Thanks for the input.

        To get more into the nitty-gritty meta, I feel like I ought to be editing the CSS for these here comment boxes. But do you think that the comment box should be in the same font as the posts will appear? I like having some differentiation, but I also like consistency. Decisions!

        ****
        There’s definitely something to be said about the ‘good’ end of the spectrum, and excellent service will get noticed. But I still wonder if providing adequate service 90% of the time and great service 10% of the time would serve firms better than great service 20% AND horrid service 15%.

        I think I’m going to write a whole series on this topic, so I hope you’ll stay tuned for more.


  1. 1 Update on Voluntary Response Bias and Social Media « As We Now Think Trackback on August 2, 2009 at 10:36 pm
  2. 2 What If Your CEO Is Right To Be Afraid Of Social Media? (Part One) « i C P G Trackback on August 17, 2009 at 9:46 am

Leave a Reply