An honest opinion?

x Sponsored content on Research Live and in Impact magazine is editorially independent.
Find out more about advertising and sponsorship.

Ratings have become increasingly vital currency for helping us choose businesses or services and deciding how to spend our money. Review sites such as TripAdvisor have irrevocably altered the hotel and hospitality industries, for instance, while 84% of consumers trust online reviews as much as they would a personal recommendation.   

The reliance on ratings is even more keenly felt in the gig economy, where they play a fundamental role in how peer-to-peer platforms such as Uber and Airbnb operate. Customers turn to ratings to decide which services they use and, in many cases, feedback is directly linked to a provider’s overall success – for instance, Airbnb listings with ‘superhost’ status receive greater exposure in the site’s search results, while a bad Uber rating could cost the driver their job. 

Perhaps this is why positive reviews seem to have become the norm on gig- economy platforms, with little in the way of poor feedback. Indeed, if the ratings are to be taken as genuine, the overall standard is seemingly increasing over time. Intrigued about whether this increase was the result of a genuine improvement in service, or something else, researchers from New York University undertook a study on the rise of positive ratings within a reputation system. 

The paper, Reputation Inflation, argues that the effectiveness of public ratings between trading partners deteriorates over time because reviewers feel under pressure to leave ‘better’ feedback, resulting in inflated ratings. Over time, the researchers argue, these “individually rational choices” about what feedback to give can create a less informative equilibrium, “and, in the extreme, put the reputation system on an inexorable path towards uninformativeness”.

Analysis of an unnamed gig-economy online marketplace, with data spanning more than 10 years, found the majority of workers had overwhelmingly positive recent feedback, but this was not always the case. The share of workers receiving the highest possible rating (five stars) increased from 33% to 85% in six years. The researchers also looked at longitudinal data for four other online marketplaces that had shown a similar increase in positive feedback ratings over time. 

Private ratings

To determine whether the improved ratings were because of standards being raised or reviewers lowering their criterion, the researchers took an alternative measure of user satisfaction, using data from an experimental private ratings system introduced by the company. This feedback was not shared with either the worker involved or with the public, and the system operated alongside existing public feedback, both written and numerical. 

The study found that the average scores for private feedback fell at the same time as average public feedback scores for the same transactions were increasing – users who were actually dissatisfied with the service were still leaving good public ratings. More than a quarter of users ( 28.4%) who privately reported that they would definitely not hire the same worker in future then publicly rated them four or more stars out of five. This ruled out the hypothesis that ratings were increasing because of improved standards; users were simply lowering their standards when it came to feedback, rather than becoming more satisfied with the service.

The role of private feedback was the most interesting aspect of the paper for John Horton, assistant professor in the Information Systems Group at New York University’s Stern School of Business, and one of the paper’s authors. “We actually wrote this paper over several years, writing most of it before the platform began the experiment,” he says. “That we witnessed rapid inflation more or less in accordance with our theory was surprising and gratifying, as it suggested we had hit on the root cause.”

After the platform had been collecting private feedback for 10 months, it began to release aggregated anonymous private scores publicly – and, although the workers would not know which user had given what feedback, the move meant that the private feedback scores suddenly became consequential. As a result, when these private scores were made public, private feedback immediately began to increase. 

Regardless of whether workers could retaliate to a bad rating, users were aware that their scores had consequences (that is, the ratings were now public and could influence future users). According to the paper, this suggests “an inherent tension between ratings being consequential and ratings being informative”.

The researchers argue that reviewers lower their standards for public feedback because of the perceived ‘cost’ associated with giving a negative rating. Some reviewers feared retaliation for leaving a bad review, while others expressed concern that a bad rating could harm the individual involved.

The paper also looked at the sentiment in comments accompanying numerical ratings and used a model to predict numerical feedback from written observation. 

This meant researchers could identify what proportion of average feedback growth was because of changes in market ‘fundamentals’ – such as improved features, better cohorts of workers, lower prices, and so on – and the residual component of growth that could not be explained by these improvements. 

Inherent tension

While predicted feedback scores (based on written comments) rose over time, these figures had not increased as much as the real numerical feedback scores. Using this model, the authors suggest that more than 50% of the boost to rating levels recorded over the six-year period could be attributed to reputation inflation – reviewers lowering their standards.

When it comes to businesses applying some of the paper’s principles, or developing alternative feedback models, Horton says firms could try to incentivise truthful reviews, focus on those that can lead to service improvement, and create less subjective questions – for instance, asking ‘Was the pick-up on time?’ instead of ‘Is this a five-star driver?’

He adds: “Trying to shield raters from retaliation is good but, as our study shows, this doesn’t seem to fully ‘work’. I think there’s an inherent tension in these systems in that bad ratings need to be harmful for the system to work, but it’s that harm that causes the inflation.” 

Reference:

Filippas, Apostolos and Horton, John and Golden, Joseph, Reputation Inflation ( 8 March, 2018 ).