Building Web Reputation Systems [Paperback]

By Eric Goldman

For the past couple of years, I have been researching how we regulate reputation systems. As part of researching other disciplines' approaches to reputation systems, I was pleasantly surprised to find this book, which discusses web reputation systems from a technical/product development standpoint. I'm not aware of other books directly on point, so that alone makes the book noteworthy.

The word "reputation" is a complex and nuanced word. This book defines reputation as "information used to make a value judgment about an object or a person." Notice how this definition treats reputation as actionable information (i.e., making a "judgment"). I favor that approach; my work also uses an actionable definition of reputation.

Their definition equally treats both objects and people as having "reputation," and this does not work. In general, people are dynamic, i.e., they can change behavior; while content is static, i.e., an item of content does not change its character unless subsequently edited. This single definition of "reputation" created significant tension throughout the book. Recognizing this, the authors often bifurcated the discussion to separately address the process of establishing a person's "reputation" (which they confusingly called "karma"). However, the book primarily focuses on grading and sorting content items, especially user-generated content, and I personally would not describe content items as having a "reputation." As a result, I think the book is mistitled. It principally addresses content filtering, not "reputation" as I use the term.

Although this analytical tension pervades the book, the book nevertheless contained a lot of useful insights about both content filtering and establishing user trustworthiness. The authors have a lot of experience building filtering systems for different websites, so the book is packed with the kind of first-hand observations that only an insider can offer. There's no substitute for the voice of experience when designing Web 2.0 UGC systems, and this book provides an easy and accessible way to learn some of these tips and tricks.

The book emphasizes the authors' contributions to the reputation system at Yahoo Answers, and rightly so. Yahoo Answers has emerged into a bona fide success story and recently trumpeted its billionth answer. In my opinion, the book's high point is Chapter 10, a case study of how Yahoo Answers developed a new filtering and reputation system that helped turbocharge the Yahoo Answers community.

Although the book doesn't say this directly, two key lessons from Yahoo Answers' evolution are:

1) UGC websites should let users vote on content, but not all user votes should be weighted equally.

2) UGC websites do not need to publish all user-supplied content items in an equally prominent manner. Perhaps some content should be obscure/hard-to-find until other users validate it.

The book pitches these conclusions as novel, but they seemed fairly intuitive to me. We implemented a very similar system embodying these two points back in 2000-01 at Epinions. Epinions allowed users to grade each others' content; we weighted votes differentially based on users' credibility; and we displayed ungraded and poorly graded content only to registered users (a small fraction of our readers). The fact that the authors "discovered" these conclusions at Yahoo Answers shows the dire need for books like this to help websites implement best UGC management practices without reinventing the wheel.

The fact that the authors didn't acknowledge the Epinions precedent (and other systems like it) highlights another weakness of the book. There is a deep academic literature addressing the book's topics (especially on content filtering and user incentive systems), but the book barely acknowledges this literature. For example, several times the authors cite Dan Ariely's Predictably Irrational for descriptions of human psychology and foibles. That's a perfectly credible citation, but it should be one of many literature citations, not the only citation. Instead of dipping into the rich academic literature, the book almost exclusively relies on the authors' experience-based impressions. These impressions are a valuable information source that makes the book worth reading. However, because those impressions aren't tempered with more rigorous academic findings, it's not clear to me at all that the authors' conclusions represent true best practices...or even state-of-the-art.

Because of its many structural flaws, this edition will not become a classic. Nevertheless, I have enthusiastically recommended the book to several UGC start-ups because the book provides a good repository of high-value experience-based perspectives that are not readily available elsewhere. Even if the book's recommendations are debatable, it's a debate worth having.

I'm researching different models to use for our corporate collaboration platform and I'm looking to implement a social recognition/reputation framework. Our framework needs to make sense for users and content in many different businesses and support functions. It's one thing to rate things on Amazon and Netflix and another to deploy a system inside an enterprise. (I'd also researched peter Reiser's "Community Equity" - [...])

This excellent book proved *extremely* useful. It dissects the different popular models - Amazon, eBay, Yahoo, Digg - as well as many others I was not familiar with. It analyzed each and clearly highlighted the pros and cons of each; how different models can be used (and abused) to meet different business objectives. Priceless.

More than a dry analysis, the book is filled with interesting (to me!) insights on the potential pitfalls of each model. I was often surprised at how things I took for granted had some hidden biases or other traps. The sections on the issues with leaderboards, for example, were clear and practical.

My only advice when reading is to skim the initial chapter on the "graphical language" for reputation models. It seemed unnecessary and too abstract. It almost put me off from reading the rest of the book but I needed help. And I'm really glad I continued. The remaining chapters got more and more engaging. The final case study on preventing abuse in Yahoo Answers reads like a page-turner. (Really!)

If you are building any kind of application with ratings or some kind of recognition/social capital/reputation for users, then read this book. It will help you avoid the mistakes made by others before you (you'll never forget the story of The Dollhouse Mafia). And it will help you meet your business objectives for including ratings and reputation in the first place.

(Note: The authors are also on twitter: @frandallfarmer, @soldierant, and @buildingrep.)

This is a really interesting book about allowing users to provide feedback/ratings on your content or services, including common algorithms for calculating those ratings, the value of each user's ratings, and how to manage this information. I found this very enlightening, as many of the ideas discussed in this book hadn't initially occurred to me.

Perhaps the most useful material, in my opinion, was the discussion around different reputation models. Included in the discussion were the models used by eBay, Yahoo!, Slashdot, and others. Also very valuable were the algorithms for tabulating ratings and reputation so that performance is not impacted.

I found the information in this book invaluable in building online rating systems. If you're working on such a project, I'd highly recommend checking out this book.

› Go to Amazon.com to see all 7 reviews  4.6 out of 5 stars