21 July, 2007
This weekend’s challenge is once again reader-submitted. Mike Rozak wrote in with an idea about getting feedback from users.
Mike has been working on a game of his own called CircumReality. He has opened it up to the public in the past and tried to get user feedback. However, it has been difficult for him to get really good feedback.
So, think about that as a challenge: How can you get useful user feedback?
Mike’s thoughts and specific challenge after the jump.
From my experience at Microsoft and working on my game, I’ve found that asking users for feedback is an incredibly painful task, not because the feedback is negative (I love negative feedback so long as it’s actionable), but because getting useful information out of users is extremely difficult.
For example, I posted on a few forums, saying that my game was still in early stages, and asked for any comments:
- Only 10%-20% of the people that tried my game actually provided feedback… which for a large beta test would be pretty good. At Microsoft, only 1%-2% of beta testers actually provided any feedback.
- Some people posted replies even though they obviously hadn’t tried the game. See “trolls”, below. (Some of these replies were reasonable: Requests for Mac, Linux, or lower-minimum systems. These people couldn’t try the game because it wouldn’t work on their system.)
- Some people just replied, “It sucks!” This requires follow-up with, “Why did it suck?” “Because the graphics sucked; They should look like Crysis.” Etc.
- Trolls had a field day; but does quickly shutting up a troll cause other players to suddenly shut up too?
- There was a lot of noise. Some people complained about a specific and very minor feature. I had to read between the lines to figure out the larger issue that was bothering them, but which they didn’t mention. Non-game example: When people say, “I don’t like him because he laughs like a hyena”, they probably mean that he’s obnoxious, and his laughter is only a symptom.
- Some people pointed to their favorite game (such as WoW or Zork) and requested features from that game, whether or not those features would work in my game.
- Only a small handful actually provided meaningful feedback… That feedback ended up being helpful though. Even the non-meaninful feedback (from trolls, people that wrote “It sucks”, etc.) was helpful, but only in vague metrics, like “People think the system requirements are too high”.
The design challenge is:
- How do you solicit feedback in a way that produces better-quality feedback?
- How do you read between the lines?
- How do you stay sane while sifting through the rantings of trolls and idiots to see if there’s a kernel of truth?
- How do you balance your own vision and the players’ feedback? (One episode of the Simpsons had Homer’s long-lost half brother come and visit. He ran an auto manufacturer and had the “every man”, Homer, design a new car. The new car ended up being, ugly, expensive, and drove the auto manufacturer broke.)
Mike mentioned that CircumReality will have another round of testing currently scheduled for November, for those interested in providing good feedback.
Mike’s issues are not unique, unfortunately. Although many games rely on “beta tests” with lots of users for feedback. Getting good feedback is often very hard, because users are not often motivated to provide feedback. In some cases, such as gameplay-wrecking bugs that give players an advantage over others, there is motivation not to give information.
I think one of the main issues is user ability to give meaningful feedback. Mike points this out above with the “laughs like a hyena” example; the user knows something is wrong, but can’t quite pinpoint it exactly. This is a reason why we (should) have highly trained QA people to find and report our errors. For smaller projects, this can be an impossible cost, though, I know.
I think Mike’s last challenge up there could be a challenge all its own. Balancing a strong vision while still using customer feedback in a meaningful way is hard to do. The example of the Simpson’s episode shows the extremes of what happens when you listen too closely to a (particularly inept) source.
What do you think? How can you get quality feedback from your users?