Kickstarter: A Source for Quality Games?

Background

Previously I reported on some statistics for excellent games that have been published through Kickstarter and how Kickstarter has provided the opportunity for some new designers and publishers to launch successful game development companies. (Game Designers: Impressive First Impressions). One of the comments/criticisms that the article received was the common refrain, “Sure there are some Kickstarter successes, but just not many of them.”

Not one to stand by while anecdotes and opinion are used to substantiate claims, I dug into the Board Game Geek ratings for Kickstarted games compared to all games published in the years 2010-2014.

Objective

Discover and illustrate the facts regarding the average quality of Kickstarter backed games compared to the general population of games recorded on Board Game Geek as defined by the BGG Rank.

Note: I have no dog in this hunt. I have no projects on Kickstarter. I have backed my share of projects on Kickstarter and have also had my share of duds. However, I get frustrated when those reporting on the hobby game industry fail to do their research and perpetuate anecdotal claims. When I set out to gather these statistics (and those previously for Game Designers: Impressive First Impressions), I had no idea what the results would be.

Setup

All data for this research was collected from Board Game Geek; primarily using the Advanced Search function. However, identifying board games as Kickstarted is more difficult since the only way to get this collection is to use the “Crowdfunding: Kickstarter” Family which is not available through Advanced Search. In my series of articles about Hobby Game Trends 2000-2014 I discuss some of the challenges with acquiring and classifying data from Board Game Geek, so if you want a greater understanding of those challenges jump into that series at Round 3: Game Categories.

Also in that series I created a definition for the “Hobby Game” as it relates to the Board Game Geek Advanced Search criteria. You can read about the criteria I applied for “Hobby Games” in Round 2: Game Releases. I will use that definition in this article for comparison to Kickstarted games.

I will compare Kickstarted Games with “Hobby Games” and “All Games”, but before jumping in there are some details related to the data that should be understood:

  • The data is presented for three collections:
    • Kickstarted Games: Games that are designated as being in the “Crowdfunding: Kickstarter” Family on BGG. (I have done no additional research to validate this data).
    • Hobby Games: As defined in my earlier posts.
    • All Games: All “games” recorded in the BGG database.
  • Kickstarted Games is a subset of All Games.
  • Hobby Games is a subset of All Games.
  • Kickstarted Games and Hobby Games overlap, but I have no (reasonable) way to determine the extent of the overlap.
  • In all cases, I have eliminated “Expansions” either by using the BGG Advanced Search criteria or, in the case of Kickstarted Games, personally identifying the records. Yes, reviewing 2100+ records (with a little help from some scripting.)
  • The Hobby Games classification attempts to get closer to a representation of what hobby gamers consider to be good games.
  • Since, presumably, Kickstarted Games are at least as likely to not fit the Hobby Game category as the general population of All Games, the most reasonable comparison is Kickstarted Games to All Games. However, I include the comparison to Hobby Games as well since, that is what we in the hobby are really talking about when we say “good games.”
  • The data population for some categories is too small for good statistics, but you’ll get the idea. (E.G., When looking at the Top 100 games, the population of games for any particular year or collection is necessarily very small.)

Round 1: Comparing BGG Rank of Kickstarter to All Games

Naturally, the first thing that comes to mind as a means of comparing quality is to compare number of games (as a percentage of all games within a publishing collection) that are in the Top X% of BGG Ranks. It is difficult to say where one person might consider the quality games end and the crap games begin in the BGG rankings, so I selected several thresholds. The Top: 100, 250, 1000, 5000, and 10000. You can decide for yourself where to draw the quality line.

The question Round 1 attempts to answer is: “If I look at a collection of games; those on Kickstarter, all those on BGG, those likely to be described as hobby games, what is the likelihood that I will find a good game?”

The following graph and chart show the percent of games within a publishing collection that fit within a quality category for games published within the 5-year span 2010-2014.

The following is the same chart for 2014 only.

The following two graphs use the same data, focusing on the top 2 categories so these small populations are not lost in the larger numbers.

The data suggests that if I randomly selected a game that had funded on Kickstarter, I would more likely get a “good game” than if I randomly selected one from the collection of Hobby Games or All Games. However, I am less likely to get a Top 100 game.

Round 2: Comparing BGG Rank of Kickstarter Games Based on Publication Year

Arguments could be made to support the quality of games on Kickstarter going up or down over the years. For example:

  • Large publishers getting more involved might drive the quality up.
  • New Kickstarter Generation designer/publishers maturing might drive the quality up.
  • More amateurs getting involved might drive the quality down.
  • Success of crap games on Kickstarter will breed more crap games and drive the quality down.

The following graph and chart shows the percent of Kickstarter games in each quality category for each publication year.

The data suggests that the quality as a percentage of games published is dropping.

The following graph shows the same data for raw numbers rather than percentages.

The raw number of quality games has been very consistent over 2012, 2013, and 2014. It appears that there is a certain capacity within games production that uses Kickstarter that is fairly consistent, but the total number of games produced continues to rise, which is diluting the average quality.

Round 3: Pulling it all together

Now let’s look at the percentage of All Games that are Kickstarter Games in a quality category for each year. The “Total” bars on the far left indicate the percent of total games that were published using Kickstarter. So a bar higher than this bar for any given year illustrates a higher average quality of Kickstarter Games and a lower bar illustrates a lower average quality.

And now the same graph for Hobby Games. (This graph makes the assumption that all Kickstarter Games are a subset of Hobby Games, which is unlikely, but probably relatively close).

End Game

Kickstarter Quality Compared to Other Publication Options

Despite the anecdotal evidence to the contrary, the real data seems to suggest that the average quality of Kickstarter games is higher than through other publication methods (as a whole). This holds true historically (looking back to 2010) and currently.

Kickstarter Quality Over Time

While arguments can be made predicting that the quality of games on Kickstarter should go up or down, the data seems to suggest that there is a certain capacity to produce quality games through Kickstarter and that the number of projects is diluting the quality of games. 

What Next?

I wouldn’t use the data presented here as a reason to go back any game on Kickstarter, but it does temper the anecdotes and blind opinion that a game published through Kickstarter is more likely crap than a good game.

Next I will use this analysis as a basis to continue the discussion regarding potential reasons why the anecdotal evidence is contrary to the raw data in an upcoming article called “Kickstarter: Backer Perspectives.”

Also, as a result of gathering the data for this article, I also have some interesting data that may end the argument over which year had the best games published.

Your Turn

Are you surprised by the data? What did you expect to be the average rank of games published via Kickstarter? Is the data consistent with your experience with the games you have backed? Although, much of the data I have collected is provided here, if you want the full data, just ask.

Subject: