If I can't understand it, why would I trust it?

Ever hear of the online small business review site known as Yelp? I hadn't until recently when it became the subject of some pretty harsh public scrutiny and a few class action law suits, to boot.

You see, Yelp's value proposition is that reviews are based on a very intelligent "algorithm" that crunches hundreds (or thousands) of opinions from people who have directly experienced specific small businesses everywhere. These businesses are everyday neighborhood restaurants, retail shops, and a host of service companies. The more customers enter their opinions on such topics as price, service, quality and brand appeal, the more relevant Yelp becomes to any consumer who wants valuable information before deciding to buy from a particular small business.

Here's the catch. Yelp makes money by selling advertising on its site to small business owners. Many of these small business owners are up in arms (and have been posting their angry comments on Internet chat boards) because they claim that Yelp sales representatives are pressing them to buy advertising with a clear threat that if they don't...the reviews will be manipulated in a not so flattering way. Talk about a conflict of interest and a serious legal dilemma, if in fact, these accusations are true.

Last Sunday, this Q&A with Yelp's CEO appeared in The New York Times (link to Talking to the Chief of Yelp, the site that businesses, love to hate). You can see that Jeremy Stoppelman tries to diffuse the crisis by answering a number of very pointed questions about this small business owner claim (which he flatly denies ever took place) and a variety of other misunderstood issues regarding Yelp and its model. The company has also recently posted direct answers to these heated questions on its blog and this video tries to simplify what Yelp is all about by proactively answering a variety of questions in a very friendly tone with explanations that are also easy to understand.

With the exception of one critical flaw, I commend Yelp for moving quickly to try and diffuse this situation through open, transparent communications to all stakeholders. Unfortunately, this flaw is something I can't overlook, though and it focuses on Yelp's nebulous algorithm. Yelp's entire positioning is to provide the consumer with trustworthy reviews that can be counted on. Yet, the algorithm used to calculate this information is murky. And, when Jeremy is asked (in The New York Times article) to explain how this algorithm (or filter) is used to determine whether a review is trustworthy, he refuses to reveal anything. In fact, his answer is downright patronizing ("I really can't be very specific. The more that we explain the algorithm, the less effective it becomes"), as if we (the readers and his consumers) are too small minded to understand it.

I'm a firm believer in measurement. But, whether one uses formulas, algorithms are anything else to get at an end outcome, the methodology has to be something that can be implicitly understood. If not, why should anyone believe that the actual score (or in this case, review) is verifiable. Jeremy's absurd answer stopped me in my tracks and turned me off to Yelp. That's kind of ironic since it's the lesser of the issues that this company is actually dealing with right now.

I think that Yelp still needs a little help...

Comments

[Steve]

I think the basic flaw is in this entire category of service. It is true that when a rating algorithm is understood, it can be easily manipulated. To take a ridiculous example, if part of the scoring is to count the number of exclamation marks in a review, knowing that, people who want to game the system will include! more! exclamation! marks! Anything that can be counted by computer can by manipulated by people, so to the extent the mechanics of the site are understood, the site becomes less reliable...

...which then gets to your very good point, Ed: why trust something you can't understand? It's one thing for temple or church to ask people to take something on faith; another for a business--especially one whose business is measurement--to do so.

Inherently then, for any business like this, there may literally be no way to tell a legitimate model or system from an illegitimate one, because (shades of Heisenberg's Uncertainty Principal), to analyze how the ratings are produced destroys the reliability of the ratings.

[Kinda grumpy]

In the past week I discovered Yelp and enthusiastically posted 14 reviews. I was so pleased that as a consumer, I finally had a place to give proper credit or criticism to businesses. Of those reviews over 50% of them were filtered and suppressed within hours of posting them. Little did I know my independent reviews posted in earnest with no ties to any business whatsoever were considered suspect by the mysterious Yelp algorithms. When questioned, Yelp offers no answer other than the typical "we appreciated your input" crap.

Additionally I noticed a new Yelper and perused her account out of curiosity. Out her 17 reviews, a whopping TEN had been suppressed & filtered from view.

As a consumer, a user, & a contributor, this type of excessive suppression of honest reviews really makes me wonder what the flying Fark is Yelp doing?

So, what really is Yelp doing? A dopey patronizing video aimed at their perceived knucklehead users is insulting and leaves more questions than it answers.

Additionally Yelp gives it's users the illusion that one's reviews are all visible as they always remain from the users vantage point. I wonder how many contributors would continue to help Yelp by writing free reviews knowing that a large percentage of their contributions are suppressed.