14 September, 2016
If you don’t follow game industry news, you might have missed that Valve changed how they use reviews. The short version is that only if you’ve bought a game directly from steam does your review count by default.
This has some pretty tremendous consequences for game developers, and probably for game players. Let’s take a look at what this means in a bit more detail, shall we?
“Something Must Be Done”
If you want to know more of the details, this post has a pretty good overview of the problems and the issues that Valve is trying to address. Go read that, but take any additional commentary with a grain of salt.
Like the author of that article, I an certain understand that this is a problem. Unlike the author, I don’t quite agree that “Something Must Be Done”. Something should be done, sure, but rushing headlong into a change without considering the consequences is asking for trouble. Worse, it could end up hurting the people that Steam should be helping: independent developers who play fair.
Hurting the little guy
So, some independent people aren’t happy. I can sympathize with Dave here. He’s an honest guy who has played by the rules. He’s on the cusp of bigger things, but he struggles to get his work recognized. Part of what he relies on is for the Steam platform to help people find his game.
But now it’s harder. The keys he gave as a reward for people who left good feedback for his previous games? They don’t count in reviews. Press keys given out to YouTube reviewers who legitimately fall in love with the game enough to go review the game? They don’t count, either. People who might Kickstart a game and review if it lives up to its promise? Yep, they don’t count. All these people have legitimate points of view but are silenced by this change.
Of course, this move simply reinforces that the little guys are at the mercy of Valve and their whims. This policy is unlikely to change an already successful game from a large company with a sizable marketing budget into a struggling failure. And games like No Man’s Sky can get all sorts of negative reviews (see the image at the bottom of Dave’s post above) and the developers are still probably making more money than 99% of developers will ever see. Worse still, it seems like Valve had done little to talk about these changes and prepare developers for them.
But, let’s look at the bad actors.
A few bad apples spoiling it for everyone
So the problem is, as stated in the first post I linked, that some developers were giving out keys to get favorable reviews. Some made it explicit: promise a good review and you get a free game. And, yeah, that’s kinda sleazy. But, did it really affect things?
Steam makes their data publicly available, so there’s a great site called Steam Spy which scrapes that information and figures out what it can. It has to do some statistical analysis on the data to get some estimates for some of the stats. So, the site did an analysis of the games most affected by this change and posted a list of those games:
— Steam Spy (@Steam_Spy) September 14, 2016
As you can see, not exactly a huge racket going on here. Most of these games aren’t very big, and the reviews gained from any sort of potential trickery was pretty limited. I don’t know about you, but was probably unlikely to base my buying decision on a game called Worst Case Z simply on the reviews.
Do reviews even matter?
But, this ultimately forces us to consider: how much do reviews matter? That’s perhaps an interesting question.
Back when I worked at 3DO, the upper management held the opinion that reviews don’t matter. The important thing was to get good coverage for previews, because most sales happened near launch. Previews were basically marketing letting people know the game existed. By the time actual reviews were posted, most people were either enjoying the game or suffering from buyer’s regret. Of course, this was in an era when magazine reviews were the norm. With long print leads and publishing cycles, a review might not come out for a while after launch.
Of course, things are a bit different now. Steam reviews are immediately visible. So, the person who hates the font on the title screen can turn around and post a hateful screed calling the developer an idiot. But, still, does this matter?
I’m not sure. One writer I talked to said that when it comes to books, the review score doesn’t influence sales very much. The most important thing? The number of reviews. The more reviews a book has, the better the sales are likely to be. Of course, it might be that having a lot of sales generates a lot of reviews, not vice-versa. But, it does make sense; people are influenced by social proof, so something that appears popular will be of more interest than something that seems ignored.
Still, it can rankle a developer to see negative reviews being given a prominent place on the page for their game. If nothing else, it can hurt the ego even if it doesn’t necessarily hurt sales.
Solving the problem
So, let’s accept that review inflation is a bad thing and that Valve wanted to take steps to fix it. What could they do?
Honestly, I don’t think this is a place where an automated solution is the right answer. As much as Valve might seem to loathe human intervention (even though that was an answer to another problem), this is probably one area where it might be worthwhile. Take the data build a too like what SteamSpy did, and then flag accounts with suspicious activity. Look for games where an overwhelming number of free keys resulted in positive reviews. Or where the scores for people holding free keys were much more positive than the rest of the reviews. Flagging potential bad actors and then investigating is probably the right answer here. But, given Valve’s non-hierarchical structure, who is going to volunteer to do that?
So, what do you think? Do you rely on steam reviews? Do you think this was a major problem? Do you think the solution will improve your buying experience?