Company Ranking in Bayesian Perspective

Published in

Analytics Vidhya

5 min readJul 31, 2021

Background

Recently I came across one article: Revou Happiness Index. The report assesses 38 technology companies in Indonesia and ranks them using company reviews in Jobstreet and Glassdoor. Below is a cropped image of their result. Do you notice that there are companies with a lower number of reviews but have higher rank while having a pretty small difference in their average review? Example: Payfazz (23 reviews, avg 4.7) and Happyfresh (62 reviews, avg 4.65) with 0.05 difference in average review.

Having more data means that we have more evidence about something right? In this context, how could we decide what company should be in 1st rank? Isn’t that 0.05 difference pretty small? On the other hand, we should give more weight to trust reviews for Happyfresh compare to Payfazz? (Since they have more reviews). Here we face a difficult situation where how far should we trust the data that we have? Then the next possible question is how could we incorporate uncertainty in the rank?

I create the analysis to share how could we approach the problem using the Bayesian perspective. By Bayesian means that we will take into account how far we should adjust our belief (of something) given data/evidence that we have, hence uncertainty will be considered in the rank calculation.

The scope of the analysis is a bit different compare to Revou’s report. We will only take reviews of the 38 companies from Jobstreet. Because Glassdoor only shares the average review of a company, not in review level of granularity.

Problem Statement

Given company reviews from JobStreet, how should we better rank the companies?

Data

We could get reviews for 38 companies from JobStreet using this script. This analysis uses JobStreet data per 20 July 2021. Below is the top 10 companies based on the mean review. Notice that the fewer the number of reviews, the higher the standard deviation. It represents our uncertainty about the data we have. Less data, less confidence. Later, we will incorporate the uncertainty for ranking.

Alternative Methods

Rank by Weighted Average Review
Rank by Statistics from Posterior distribution (Bayesian Analysis)

1. Weighted Average Review

It’s inspired by how IMDb previously rank their Top 250 movies. Following is the formula.

weighted rank (WR) = (v ÷ (v+m)) × R + (m ÷ (v+m)) × C where:
  R = average review for the company = (mean_review)
  v = number of review for the company = (sum_review)
  m = minimum num of review required to be listed in the analysis
  C = the average review across the whole reportnote for this analysis
m = 5
C = 4.318

Notice that if a movie has only a few reviews then the WR will be closer to the overall mean review (C). Contrary, if a movie has a far higher number of reviews than m, the WR will be close to the mean review given to the movie. It emphasizes the notion of believing the average review if only it has a high number of reviews. Below is the top 10 companies based on weighted mean review.

If we look at two companies: DANA and Mekari. Their difference in mean review is only 0.006 (left table), but their number of reviews differs a lot. Mekari has 69 reviews while DANA only has 5 reviews. As a result, if we rank them based on the weighted mean review, Mekari will be placed higher compare to DANA (right table).

2. Bayesian Analysis

In Bayesian Analysis, we have a prior belief about a parameter and we will update the belief given data about the parameter. In this analysis we have one parameter: company review with values between 1 and 5. We will assume the parameter have Gaussian (re: Normal) prior distribution. We will have Normal posterior distribution because Normal-Normal conjugate prior.

We will use the same prior for all of the companies. It’s considered weak prior. The prior (normal distribution) has parameter mean = 0 and std = 1. The observation in reviews for a company will change our prior, then we get our posterior. The tighter the posterior distribution means that we are certain about our belief. Contrary, the wider the posterior distribution means we are rather uncertain about our belief.

The advantage of using Bayesian Analysis is that we have posterior distribution instead of point estimation. From the posterior distribution, we could derive any statistics about the parameter. For sorting/ranking purposes we could use percentile 25 of the distribution instead of mean. The below image shows why the mean of the posterior distribution is not a good metric for ranking. Contrary, p25 will tells us how uncertain we are about the posterior distribution.

Below is the top 10 companies based on the 25th percentile of the posterior distribution. As mentioned in the above image, DANA is placed lower because we are rather still uncertain about its review. Let’s look at Blibli and GoJek, they have quite identical mean reviews of 4.455 and 4.453, respectively. But Gojek has a higher number of reviews (245) compare to Blibli (134). Gojek also has denser reviews with 5 stars compare to Blibli. That’s how Gojek is placed higher than Blibli in the right table.

Conclusion

We now see that’s there are alternative methods for ranking companies using the Bayesian perspective. These methods suit you if you wanted to incorporate “uncertainty” in your analysis. With that, you won’t blindly trust the data you have. Rather, you will update your prior belief with the data accordingly.

All of the source code for this article is in my github/ajisamudra.

Reference

Probabilistic Programming & Bayesian Methods for Hacker, Cameron Davidson-Pilon.
Google Answer, Alternative to IMDb formula.
Revou
JobStreet