Psychochild's Blog

A developer's musings on game development and writing.

14 September, 2016

“Pray I don’t alter it any further”
Filed under: — Psychochild @ 11:18 PM

If you don’t follow game industry news, you might have missed that Valve changed how they use reviews. The short version is that only if you’ve bought a game directly from steam does your review count by default.

This has some pretty tremendous consequences for game developers, and probably for game players. Let’s take a look at what this means in a bit more detail, shall we?

“Something Must Be Done”

If you want to know more of the details, this post has a pretty good overview of the problems and the issues that Valve is trying to address. Go read that, but take any additional commentary with a grain of salt.

Like the author of that article, I an certain understand that this is a problem. Unlike the author, I don’t quite agree that “Something Must Be Done”. Something should be done, sure, but rushing headlong into a change without considering the consequences is asking for trouble. Worse, it could end up hurting the people that Steam should be helping: independent developers who play fair.

Hurting the little guy

So, some independent people aren’t happy. I can sympathize with Dave here. He’s an honest guy who has played by the rules. He’s on the cusp of bigger things, but he struggles to get his work recognized. Part of what he relies on is for the Steam platform to help people find his game.

But now it’s harder. The keys he gave as a reward for people who left good feedback for his previous games? They don’t count in reviews. Press keys given out to YouTube reviewers who legitimately fall in love with the game enough to go review the game? They don’t count, either. People who might Kickstart a game and review if it lives up to its promise? Yep, they don’t count. All these people have legitimate points of view but are silenced by this change.

Of course, this move simply reinforces that the little guys are at the mercy of Valve and their whims. This policy is unlikely to change an already successful game from a large company with a sizable marketing budget into a struggling failure. And games like No Man’s Sky can get all sorts of negative reviews (see the image at the bottom of Dave’s post above) and the developers are still probably making more money than 99% of developers will ever see. Worse still, it seems like Valve had done little to talk about these changes and prepare developers for them.

But, let’s look at the bad actors.

A few bad apples spoiling it for everyone

So the problem is, as stated in the first post I linked, that some developers were giving out keys to get favorable reviews. Some made it explicit: promise a good review and you get a free game. And, yeah, that’s kinda sleazy. But, did it really affect things?

Steam makes their data publicly available, so there’s a great site called Steam Spy which scrapes that information and figures out what it can. It has to do some statistical analysis on the data to get some estimates for some of the stats. So, the site did an analysis of the games most affected by this change and posted a list of those games:

As you can see, not exactly a huge racket going on here. Most of these games aren’t very big, and the reviews gained from any sort of potential trickery was pretty limited. I don’t know about you, but was probably unlikely to base my buying decision on a game called Worst Case Z simply on the reviews.

Do reviews even matter?

But, this ultimately forces us to consider: how much do reviews matter? That’s perhaps an interesting question.

Back when I worked at 3DO, the upper management held the opinion that reviews don’t matter. The important thing was to get good coverage for previews, because most sales happened near launch. Previews were basically marketing letting people know the game existed. By the time actual reviews were posted, most people were either enjoying the game or suffering from buyer’s regret. Of course, this was in an era when magazine reviews were the norm. With long print leads and publishing cycles, a review might not come out for a while after launch.

Of course, things are a bit different now. Steam reviews are immediately visible. So, the person who hates the font on the title screen can turn around and post a hateful screed calling the developer an idiot. But, still, does this matter?

I’m not sure. One writer I talked to said that when it comes to books, the review score doesn’t influence sales very much. The most important thing? The number of reviews. The more reviews a book has, the better the sales are likely to be. Of course, it might be that having a lot of sales generates a lot of reviews, not vice-versa. But, it does make sense; people are influenced by social proof, so something that appears popular will be of more interest than something that seems ignored.

Still, it can rankle a developer to see negative reviews being given a prominent place on the page for their game. If nothing else, it can hurt the ego even if it doesn’t necessarily hurt sales.

Solving the problem

So, let’s accept that review inflation is a bad thing and that Valve wanted to take steps to fix it. What could they do?

Honestly, I don’t think this is a place where an automated solution is the right answer. As much as Valve might seem to loathe human intervention (even though that was an answer to another problem), this is probably one area where it might be worthwhile. Take the data build a too like what SteamSpy did, and then flag accounts with suspicious activity. Look for games where an overwhelming number of free keys resulted in positive reviews. Or where the scores for people holding free keys were much more positive than the rest of the reviews. Flagging potential bad actors and then investigating is probably the right answer here. But, given Valve’s non-hierarchical structure, who is going to volunteer to do that?

So, what do you think? Do you rely on steam reviews? Do you think this was a major problem? Do you think the solution will improve your buying experience?


Next Post: »





9 Comments »

  1. Well…

    First of all in response to your last comment there: yes, taking the data that feeds into a tool like SteamSpy used would be great for flagging suspicious activity and handling that manually. I agree with that so much I don’t know how to express it.

    But also I can immediately think of how better to handle this automatically. Just look at Rotten Tomatoes.

    First off, clearly Steam knows which reviews come from what kind of purchase. That means, they can just generate three scores from that: a steam purchase score, a CD key score, and a combined score (hello RT system).

    Second, if they really feel it, they can default to showing the first score instead of the combined score. If the CD key score is still there to look at, users can make up their own mind. Steam has done what they needed, and it’s caveat emptor for the rest.

    Next, you could calculate a more complex combined score: use CD keys for new projects, but make sure the user understands these might be preliminary or biased scores. Use steam purchases later on. I’m sure there are better algorithms than that, but you get the idea.

    Last, users could be allowed to configure which score they prefer to look at.

    Comment by unwesen — 15 September, 2016 @ 3:10 AM

  2. Just display at the top how the user came to receive their key. Add sorting options too. No different from a games journalist disclosing press releases.

    Comment by C T Murphy — 15 September, 2016 @ 4:34 AM

  3. Everyone is talking about scores following that change but it doesn’t stop there and the change still doesn’t address actual concerns devs have. Honestly, if “Simple Ball: Extended Edition” wasn’t at the top of this list nobody would know this game even exists so as a dev I don’t care if it had a score 88 before “maybe” coming from free keys given away only to get positive reviews.

    My game March of the Living is rated “very positive” and “only” lost 26 legit positive reviews from this change. Not too bad? Well, now the first 4 reviews showing up by default for my game are 4 negative reviews. For a game rated “very positive” it’s sending a weird message.

    This is because of the “helpful” mark you can leave on reviews which is essentially an echo chamber. You didn’t like my game or hated it and asked for a refund? Of course you’ll probably mark other negative reviews as useful. If you bought my game and are enjoying it then what are the odds you’ll scan positive reviews to mark them as useful?…

    The other issue about the review system that should be addressed but was left alone to “fix” something that wasn’t so much a big problem is the fact that if you happen to be unlucky during some week you can end up with a negative recent review score even if the global score is positive.

    Shouldn’t matter much you say? In fact it does. When the last Summer Sale begun MotL had a “very negative” recent score based on only 13 reviews (!!!) (despite having a very positive global score based on 363 reviews) and sales were terrible. 3 days later the recent score disappeared (because “recent” reviews were now not recent enough) and sales went up.

    I can see how the recent score might be relevant to early access games as they constantly change but in my case the game wasn’t updated past the first week after release to fix bugs. What value does the recent score have in this context? It’s the same exact game as before so it’s just a score based on some arbitrary random time period. My game is a modest success so it doesn’t sell that many copies each week so it’s quite possible that at times 7-8 persons buying my game don’t like it and it creates such negative recent score. Those reviews are given an enormous power.

    I could keep going on about how the review system could be improved to not penalize small legit devs but here’s a simple way to at least start balancing it:

    Why now show side by side the most helpful (even if I don’t like this notion) positive and negative reviews. Show both sides on equal foot instead of showing a bunch of negative reviews first that might not give an accurate picture of the score Steam likes so much. That’d be a start at least as right now if you have a game well received but is a modest success, the slightest change to reviews can make a huge difference.

    And yes I care because this is my only source of income so that’s why I’m so concerned about this. I’m doing fine but this new change just made it a little harder for me for no good reason.

    Comment by Dave Toulouse — 15 September, 2016 @ 5:29 AM

  4. When I first read about this change, my thoughts were of all the games I’ve purchased through Humble, GOG, Amazon…any source other than Steam that can provide a Steam key for the product. Reviews I leave for any of those games will not count toward the overall “score” under this new system and none of those purchases have anything to do with keys given out for free by a developer.

    Valve’s own post on the matter references only a tiny percentage of games that might have had this problem with reviews, so maybe I am missing something, but the (potential) negative impact of this change seems disproportionately large compared to the problem they are trying to solve.

    (And yes, I have had it pointed out to me that a person can still go to the bottom of the Steam store page for a game and see ALL the reviews, but I would be surprised if most customers do this. I’m not even sure most get far enough down the page to read system requirements. :P)

    Comment by Tanek — 15 September, 2016 @ 8:41 AM

  5. Casting Valve as Darth Vader in the title in a world where EA is still a thing? Seems a bit harsh. How do you get green lit on Origin?

    Superficially, this is Valve protecting the people who pay the bills at the expense of a group of developers who probably aren’t adding a ton to the company’s bottom line nor necessarily helping the image of Steam in the first place. The problem with a low barrier to entry is that it doesn’t keep crap out. In ten years Steam has gone from host Valve games to hosting select third party games to becoming an unwieldy storehouse for crap. Which is, of course, why reviews matter so much now and which leads to abuse of the system.

    I would be happier if, rather than tinkering with reviews, Valve was simply more discerning in what they put up on their store front. But that low barrier is what makes the indies happy… or as happy as an indie dev can get, which isn’t all that happy most days of the week so far as I can tell.

    Finally, in my experience, if this really hurts some devs, Valve will revisit the situation. Small comfort today, I know, but they have been reasonable in the past.

    Comment by Wilhelm Arcturus — 15 September, 2016 @ 3:25 PM

  6. The problem with a low barrier to entry is that it doesn’t keep crap out

    Thing is, I never see the “crap”. I really don’t see this as an issue. First we’d need to define “crap” but let’s pick “Simple Ball: Extended Edition” that was the game that saw its score went down the most after the recent change.

    I never heard of that game, never saw it pop in any suggestion box, never appeared in my discovery queue, etc. Steam is already doing a good job keeping the “crap” out of sight and any indie dev with a modest success will tell you they are also good to keep their game out of sight as well even if they might not qualify as “crap”.

    So the “problem” of letting anything get on Steam really isn’t one.

    Superficially, this is Valve protecting the people who pay the bills

    When refund is as easy as it is on Steam this is also not an issue. Everyone can get its money back quite easily. All games are effectively demos now and it’s actually not serving players and here’s an example.

    My last game didn’t have that much negative reviews after it was released so I replied to many negative review to try to see how I could improve the game based on the often vague reviews and here’s an example of what I got as a reply: “sorry, already asked for a refund”

    Instead of offering the kind of support indie devs are supposed to be great at (because we care and can handle the volume) I was shut with the refund argument. This means that I still don’t know what the player didn’t like as the review was so vague and am unable to make any kind of adjustment to the game to make him feel like he didn’t waste his money because… well, he he got his money back anyway.

    Comment by Dave Toulouse — 16 September, 2016 @ 5:05 AM

  7. Fun fact, you can leave a cookie recipe as review and it’s apparently fine. Here’s the 3rd “most helpful” review for Skyrim…

    http://steamcommunity.com/id/zdrmamo/recommended/72850/

    Comment by Dave Toulouse — 16 September, 2016 @ 5:46 AM

  8. It has to be said that even following the rules results in skewing reviews in a way that could be viewed as unethical. Giving out keys to people who are already known to like the developer and his/her style of games, or rewarding people for giving good YouTube coverage with Steam keys, is just another way of getting more positive reviews without adding any negative reviews. I’m not saying that these reviewers would a review they don’t truly stand by, but there’s a definite disincentive to post anything negative because the developer won’t give you a key next time. So it biases the statistics in a way that encourages people to game the system even further. And because the rating does seem to have some tangible effects, including search engine rankings, the people who are slightly skewing things encourage others to deliberately skew things, and it’s a race to the bottom.

    As such I can totally agree with a simple approach of “no payment, no review”. It’s not perfect but it’s better – nobody can review unless they were invested enough to have paid for the product, and that this can be verified by the company hosting the review.

    Still, a better approach would be for these ratings to come from an outside source, one that was paid to study and criticise games (meaning they’re hard to bribe and incentivised to be honest) and give some sort of professional assessment that isn’t dependent on the game’s popularity. We could call them ‘journalists’! And imagine if we could go one better, and aggregate these professional opinions, which still contain some degree of subjectivity, and average out the results from each critic to get something more representative – we could call that ‘metacritic’! It’ll never catch on…

    (In theory I am totally in favour of users being able to review and comment on things; but in practice we now know that they are often capricious and vindictive, and bad actors tend to be better coordinated than good ones, so there need to be checks on this.)

    So, setting aside lamentations on how we threw game journalism under the bus in favour of mob rule and anarchy the ‘wisdom’ of crowds, the main lesson from this is that entire industries shouldn’t be built around depending on the whims of one company, because whether we agree with their rules or not, they can and will change them when it suits them. But we ignored it with Google and we ignored it with Facebook and gamedevs and gamers ignore it with Valve as well.

    (P.S. Hi Brian! Just belatedly catching up on blogs…)

    Comment by Kylotan — 23 September, 2016 @ 2:49 AM

  9. So, last week Valve reversed course a bit and made ratings based on all reviews by default. You can choose to switch to only reviews from Steam customers, though.

    http://store.steampowered.com/news/24331/

    Which, honestly, should have probably been the first step here. But, better late than never.

    Wilhelm Arcturus wrote:
    Casting Valve as Darth Vader in the title in a world where EA is still a thing? Seems a bit harsh.

    It’s more a comment on the power relationship than on morality. Darth Vader had Lando in a position where he couldn’t really negotiate. Luckily the developers are more numerous than Lando, so they can let their voice be heard and change can happen.

    Kylotan wrote:
    …the main lesson from this is that entire industries shouldn’t be built around depending on the whims of one company….

    Yes, that’s the ideal. But, the reality is that economies of scale will tend to favor one group or another. I know I’m hesitant to install yet another platform just to buy another game, and I’m sure lots of other people feel the same way. So, the reality is one we just have to deal with.

    Comment by Psychochild — 26 September, 2016 @ 9:46 PM

Leave a comment

I value your comment and think the discussions are the best part of this blog. However, there's this scourge called comment spam, so I choose to moderate comments rather than giving filthy spammers any advantage.

If this is your first comment, it will be held for moderation and therefore will not show up immediately. I will approve your comment when I can, usually within a day. Comments should eventually be approved if not spam. If your comment doesn't show up and it wasn't spam, send me an email as the spam catchers might have caught it by accident.

Line and paragraph breaks automatic, HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Email Subscription

Get posts by email:


Recent Comments

Categories

Search the Blog

Calendar

October 2017
S M T W T F S
« Jun    
1234567
891011121314
15161718192021
22232425262728
293031  

Meta

Archives

Standard Disclaimer

I speak only for myself, not for any company.

My Book





Information

Around the Internet

Game and Online Developers

Game News Sites

Game Ranters and Discussion

Help for Businesses

Other Fun Stuff

Quiet (aka Dead) Sites

Posts Copyright Brian Green, aka Psychochild. Comments belong to their authors.

Support me and my work on