Community Focus: BGG Ratings Part 2: Hate Rating – Don’t Mess with #1

Setup

If you are coming to this article first, you may want to start at Part 1 of this series to be sure you have the full context.

The impetus for this article (which became this series) was the rise of “hate rating” (my term) reported to be occurring on BoardGameGeek as Pandemic Legacy: Season 1 rocketed up the rankings and in particular, as it approached the coveted #1 position long held by Twilight Struggle. As Pandemic Legacy approached the top spot on the BGG rankings, some users rated Pandemic Legacy a “1”, apparently attempting to keep it from rising higher in the ranking. Some raters flat out stated in their comments that this was their intent, so we know this was happening. Some stated other reasons (which we will discuss in Part 3) and some remained silent, so we don’t know their intent.

In this part of the article we discuss the practice of “hate rating,” review some of the stats on the ratings and discuss the information gleaned from those ratings and potential impact on the BGG rankings. Although this event was well-covered in the tabletop media, what remains lacking are specifics about the ratings – everything I heard or read was anecdotal.

Note: Since I have collected the stats this month and users can change their ratings and other information at any time, this will not be exactly reflective of the case in January-February 2016, but parallels can be drawn.

Round 1: Pandemic Legacy Ratings

Since this article is looking into the past and commenting both on the current state as well as the state a year ago, let’s start by looking at the ratings for Pandemic Legacy for background and historical perspective.

Let’s first look at the ratings breakdown as reported on the Ratings and Comments page on BGG. (Note: On the BGG Ratings and Comments page, the number of ratings in the “Ratings Breakdown” and the actual number of ratings displayed when the filter is applied usually don’t agree, but are close. I have used the 2 significant figure numbers in the “Breakdown” here.)

Naturally, it is not surprising for a top rated game to have a rise in high ratings, but compare this to Twilight Struggle below.  The continuous rise to “10” for Pandemic Legacy is unusual. Also unusual is the bump of “1” ratings, but that’s why we are here.

Since these two games have been in a struggle for the top spot and we suspect that the “1” and “10” ratings may have received unusually high volume (there is a smaller bump in the “1” ratings for Twilight Struggle), let’s also compare that to the #10 and #20 games, Puerto Rico and Robinson Crusoe respectively. These both show a more typical statistical pattern – a Gaussian distribution around their average rating.

With Pandemic Legacy being on the market for over a year now, we can look at the timing of the ratings of “1” and “10”.

Although both  ratings trends show a high volume in ratings for the first few months after release, the “1” ratings are more pronounced for that period and have a dramatic drop from January to February 2016 (when Pandemic Legacy had soundly taken the top spot on BGG). Unfortunately, I don’t know what was happening around Oct 2016 when the “1” ratings spiked again, but that may be the subject of a future inquiry. Given that is the time of Gencon releases actually hitting the market (as was the case for Pandemic Legacy in 2015) I suspect there is a relation to that event.

The gradual rise of “10” ratings over the last 6 months or so is interesting. I would speculate it is related to the time it has taken for many purchasers to get their group together and have a significant number of plays to feel ready to rate it.

Round 2: Nothing to See Here

I was taking snapshots of the Pandemic Legacy ratings and the BGG rankings in December 2015 as Pandemic Legacy made its move to the top. Unfortunately, life got in the way and I missed January when it topped the chart and didn’t catch up until Feb 2 after it had already claimed a rating that was well above any jeopardy of falling back to #2. However, looking at the data before and after the chart-topping event and data still available today, I can extrapolate some round numbers. When Pandemic Legacy took position #1, it had about 5000 ratings, of which about 150 were ratings of “1”. Twilight Struggle had about 21000 ratings at that time. When I pulled the data for this article in Feb 2017, Pandemic Legacy had 15094 ratings, of which 332 were ratings of “1”.

One of the first questions that comes to my mind is, “Is hate rating a problem?” Before diving into the data any deeper, let’s consider this question at a few different levels.

Should BGG users be allowed to do this?

In reviewing comments on ratings and in BGG forums, this question has been kicked around quite a bit. Some users suggest that there should be some criteria to be met for a user to be able to rate a game. I read some interesting and some convoluted suggestions on what those criteria should be and how BGG should administer them. Ultimately, this is a public rating system and anything required beyond having a user login, would likely be costly and difficult to implement and maintain. Additionally, based on the data associated with the user logins I reviewed, a small percentage of users maintain anything consistently, so the ratings would only come from a small percentage of the users, making them less representative of the community sentiment. Humorously, some recommended criteria were not met by their proponents.

Are enough BGG users doing this to have an impact?

Using the Pandemic Legacy data above as the example, the ratings of “1” represented a pretty small percentage (~3%) of the total ratings at the time of its climb and a smaller percentage today (~2%). Even if all ratings of “1” were “hate ratings”, there does not appear to be a significant problem posed to the ranking system by this behavior.

Is hate rating ethical?

As we saw in Part 1, the BGG ranking guidance is based on the presumption that the rater has played the game – that they are speaking from experience. Personally, I don’t rate games that I haven’t played and think it is contrary to the intent of the ratings. However, the “1” rating guidance, “Awful – defies game description,” taken alone doesn’t suggest that the rater has played the game. It could be the rationale for not wasting the time on playing the game or for not accepting it as a game at all.

Personally, I equally disdain using the ratings (1 or 10) to counter other user ratings – as was the case with Pandemic Legacy fans rating Twilight Struggle a “1”. In either case, the practice is inconsistent with the intent of the system – to provide an honest opinion about how much I enjoyed playing the game.

However, if the implications in some of the comments associated with ratings of “1” are ominous as they sound, then rating Pandemic Legacy a “1” was the only ethical thing to do.

• The wastefulness of “Legacy” games is going to destroy the planet’s resources. (really)
• Legacy or cooperative or simple or whatever games rising in popularity on BGG portends the end of BGG as a reliable source of information.
• Legacy or cooperative or simple or whatever games rising in popularity on BGG portends the end of the hobby.

As will become evident from the ratings data, there are almost as many ratings systems as there are users, so who is to say rating any particular game a “1” for any reason is wrong? Also, not all ratings of “1” are what I have termed hate ratings, so let’s try not to confuse the two.

Is the BGG Ranking method acceptable?

Some complain that the BGG ranking method doesn’t do enough to prevent the ratings of a few zealots from blowing up the whole ranking. Let’s take a look at some background. The BGG ranking page shows two columns of ratings:

• Avg Rating: This is the average of all user contributed ratings.
• Geek Rating: This combines the user ratings with a mathematical adjustment that only the BGG administrators know.

The purpose of the Geek Rating adjustment is to temper ratings for games when there are “few” total ratings – if they are all high, the game would shoot up the charts, if they are all low, the game rating would tank. The Geek Rating is intended to prevent something like the Pandemic Legacy rocket up the chart. Some complained that it doesn’t go far enough, tough – proven by the rocketing of Pandemic Legacy. I have heard reports that the math had been adjusted at some point. When the site had fewer users, dramatic shifts were far less likely. So as the site community grew, a revision to the adjustment method was required.

BGG provides both sets of numbers – they are open about the data. They are just not open about the calculation method for the Geek Rating, but understandably so. Revealing that opens them to more criticism and potential attempts to thwart its intent. Pandemic Legacy (and a few other games since: Through the Ages: A New Story of Civilization, Scythe, Terraforming Mars, etc.) may have risen quickly, but based on the stats I collected, no amount of tweaking would have made much of a difference – perhaps the rise would have been slowed, but topping the chart was inevitable.

BGG has continued to grow and maybe additional adjustment is advisable, but who is to say? Maybe there is perpetual tweaking going on now or a self-adjusting system has been implemented (e.g., add 1 ranking of x for every y number of users). We will look into this a little deeper in an upcoming article, “The Real BGG Accessibility Question: Careful What You Wish For”.

Round 3: Single Ratings

While it is frustrating to see how many people have only 1 rating on BGG and that rating is a “1” for Pandemic Legacy, it is hardly an outbreak. There are 16 users who fit in this category. Based on the creation time for the account, the rating date for Pandemic Legacy, and the last login date for some of these users, they obviously created an account or visited an old account specifically to rate Pandemic Legacy and have had no further action since.

Round 4: The Pandemic Struggle

Without reporting on specific individuals, here are some stats on those who ranked Pandemic Legacy a “1”.

A significant percentage (59%) of those who have rated Pandemic Legacy a “1” also rated Twilight Struggle (194 of 328 users). Of those who only rated 2 games (10 users), 60% (6 users) rated Twilight Struggle a 9 (1 user) or a 10 (5 users). Of all users who rated both games, the breakdown of ratings for Twilight Struggle looks like this: (rounding to the nearest whole number rating)

Since it was known that some users were rating Pandemic Legacy to keep it off the top rank, it is no surprise that most of those who rated Pandemic Legacy a “1” and also rated Twilight Struggle rated Twilight Struggle high (8, 9, or 10). I was a bit surprised to see how many also rated Twilight Struggle low, but when looking at what those users rated highly, the surprise was explained. Other top 10 games were potentially also getting this “hate rating” protection from users; primarily Terra Mystica and Through the Ages. Note: I captured other games rated a 10 in my notes, but did not track specific numbers.

Now, these other top ranked games are there because they get high ratings, so there is a danger here in assuming that the two ratings are correlated without having done a comparison of low total ratings as I did specifically for Twilight Struggle. However, here are two lists of highly rated games by users who rated Pandemic Legacy a “1”. These were generated using the BGG advanced stats function for two scenarios. Note:  This function does not distinguish between base games and expansions and use the minimum “Average Rating”, not the “Geek Rating”. I have added the current ranking of each game to the list generated on BGG.

Round 5: Maybe They Just Didn’t Like the Game

Well, maybe these users just didn’t like the game. But did they play it? This is where the data gets a little harder to handle – primarily because we are working with a small set of data and the subset of those who actually track plays is even smaller (the minority of a minority). Also, for those who have tracked plays, there is no way of knowing their method or consistency in tracking plays. For example: Some people try it for a while and then peter out. There is no way of know that even someone who has tracked 1000 plays was tracking them while they were rating Pandemic Legacy. The presumption here is that most who tracked plays, did so fairly consistently and that the average behavior can be compared between the two games.

For the users who rated both games (we will look at the total population of Pandemic Legacy “1” raters in Part 3), we will look at the number of users having recorded a play of either game based on the total number of tracked plays for that user.

So for those who have rated both Twilight Struggle and Pandemic Legacy a “1”, only 13% have recorded a play of Pandemic Legacy. The percentage of those who similarly track plays and have rated Twilight Struggle is 61% for comparison. Note, though, that Twilight Struggle has been rated on BGG for a very long time, so may naturally have had more people over time track at least 1 play.

Round 6: Pandemic Legacy Hasn’t Been Played Much

One could make the argument that Pandemic Legacy simply hasn’t been played much and that is the reason for the difference in percentage of raters who have tracked a play.

Realizing that Pandemic Legacy made its rise a year ago, let’s take a look at the BGG stats for the top 4 games for comparison. Is there something in the stats that indicates that Pandemic Legacy has been played less or has attained and held its position through some mathematical anomaly?

Number of Ratings

At this point Pandemic Legacy (after 15 months) has roughly 59% of the number of ratings as Twilight Struggle (after 12 years). Similar to the rise of Pandemic Legacy, Through the Ages: A New Story of Civilization has risen to the second spot with about the same number of reviews (6085) that Pandemic Legacy had at the time of its rise. It just moved up a little slower.

Well, maybe all the owners of Pandemic Legacy rate it high and the Twilight Struggle owners don’t rate as much. Maybe there needs to be a “get out the vote” campaign. The percentage of ratings per owner for Pandemic Legacy is 60%, which is the lowest of the four games. Twilight Struggle and Through the Ages are at 74% and Terra Mystica a whopping 86%. (Note: the ratings are not necessarily made by the owners, but the number of ratings compared to number of owners, provides a valid measure.)

Number of Plays

We will get into the argument that Legacy Games are worthless because they are disposable in Part 3, but will look at the comparison of plays here to see if people are rating Pandemic Legacy highly without having played it on average. (With ~6000 ratings of “10”, I am not going to do the homework of evaluating every user who rated it thus). Pandemic Legacy exceeds Twilight Struggle in number of recorded plays by almost 33%. It also exceeds Terra Mystica by 41% and Through the Ages (ANSoC) by almost 7x. Can you imagine what that difference would be if Pandemic Legacy wasn’t a disposable game? Compared to other top games, Pandemic Legacy is not only highly rated, but highly played.

Round 7: What’s in a Name?

I can think of several different names for the practice of rating a game specifically to keep it from rising in the rankings and for the people who practice it – descriptive, but not offensive. I can think of a few offensive names as well, but will relegate those to my mutterings at my computer screen when I see it happen. Here are a few that come to mind based on my love of language and gamer jargon: (in no particular order)

End Game

From the data, it can be surmised that there were users who rated Pandemic Legacy a “1” to keep it from overtaking Twilight Struggle (and other top rated games) on the BGG rankings. However, the practice was not widespread and had little to no effect on the ultimate position of Pandemic Legacy on the rankings.

The number of owners and raters of Pandemic Legacy does not seem to be out of line with other top games, so there is nothing indicating that its rise was somehow inappropriate (mathematically). If anything, it has fewer zealots per owner than other top games. Given hindsight, now 12 months of holding its position, and continuing high ratings, the proof is in that Pandemic Legacy is going to be in the top games, if not the top, for a long while to come. The rise to the top spot with only about 5000 ratings has also been validated by the rise of Through the Ages: A New Story of Civilization to #2 in similar fashion. (I did not hear any outcry about this event, though.)

You may not feel that Pandemic Legacy belongs in its current position on the BGG rankings or even in a list of “games,” but it rose to the spot and has held it and the numbers show that it is rightfully there.

Note: This is one of the articles that I started a while back, but abandoned when life got in the way, but is still relevant. So I have revised it for current posting, using newly acquired data.

Your Turn

Do you think that rating a game you have never played is OK? Do you think BGG should do anything more than what they have in buffering the rankings of games? Do you think BGG should put criteria or restrictions on the community regarding providing ratings? Do you have a name for people who rank games in attempts to manipulate the BGG ranking?

If you find this article interesting, you may want to check out all the articles in the Analysis Paralysis category where we analyze and discuss issues in the tabletop industry and community.

Continue with this series:

• Part 1: Know Thyself
• Part 2: Hate Rating – Don’t Mess with #1
• Part 3: Hate Rating – That’s Just Stupid
• Part 4: Disposable Games
• Part 5: Rating and Tracking Data and Methods
• Part 6: Conclusions and the Future