Psychochild's Blog

A developer's musings on game development and writing.

21 July, 2007

Weekend Design Challenge: Collecting feedback
Filed under: — Psychochild @ 1:38 PM

This weekend’s challenge is once again reader-submitted. Mike Rozak wrote in with an idea about getting feedback from users.

Mike has been working on a game of his own called CircumReality. He has opened it up to the public in the past and tried to get user feedback. However, it has been difficult for him to get really good feedback.

So, think about that as a challenge: How can you get useful user feedback?

Mike’s thoughts and specific challenge after the jump.

Mike wrote:

From my experience at Microsoft and working on my game, I’ve found that asking users for feedback is an incredibly painful task, not because the feedback is negative (I love negative feedback so long as it’s actionable), but because getting useful information out of users is extremely difficult.

For example, I posted on a few forums, saying that my game was still in early stages, and asked for any comments:


  • Only 10%-20% of the people that tried my game actually provided feedback… which for a large beta test would be pretty good. At Microsoft, only 1%-2% of beta testers actually provided any feedback.

  • Some people posted replies even though they obviously hadn’t tried the game. See “trolls”, below. (Some of these replies were reasonable: Requests for Mac, Linux, or lower-minimum systems. These people couldn’t try the game because it wouldn’t work on their system.)

  • Some people just replied, “It sucks!” This requires follow-up with, “Why did it suck?” “Because the graphics sucked; They should look like Crysis.” Etc.

  • Trolls had a field day; but does quickly shutting up a troll cause other players to suddenly shut up too?

  • There was a lot of noise. Some people complained about a specific and very minor feature. I had to read between the lines to figure out the larger issue that was bothering them, but which they didn’t mention. Non-game example: When people say, “I don’t like him because he laughs like a hyena”, they probably mean that he’s obnoxious, and his laughter is only a symptom.

  • Some people pointed to their favorite game (such as WoW or Zork) and requested features from that game, whether or not those features would work in my game.

  • Only a small handful actually provided meaningful feedback… That feedback ended up being helpful though. Even the non-meaninful feedback (from trolls, people that wrote “It sucks”, etc.) was helpful, but only in vague metrics, like “People think the system requirements are too high”.

The design challenge is:

  1. How do you solicit feedback in a way that produces better-quality feedback?

  2. How do you read between the lines?

  3. How do you stay sane while sifting through the rantings of trolls and idiots to see if there’s a kernel of truth?

  4. How do you balance your own vision and the players’ feedback? (One episode of the Simpsons had Homer’s long-lost half brother come and visit. He ran an auto manufacturer and had the “every man”, Homer, design a new car. The new car ended up being, ugly, expensive, and drove the auto manufacturer broke.)

Mike mentioned that CircumReality will have another round of testing currently scheduled for November, for those interested in providing good feedback.

Mike’s issues are not unique, unfortunately. Although many games rely on “beta tests” with lots of users for feedback. Getting good feedback is often very hard, because users are not often motivated to provide feedback. In some cases, such as gameplay-wrecking bugs that give players an advantage over others, there is motivation not to give information.

I think one of the main issues is user ability to give meaningful feedback. Mike points this out above with the “laughs like a hyena” example; the user knows something is wrong, but can’t quite pinpoint it exactly. This is a reason why we (should) have highly trained QA people to find and report our errors. For smaller projects, this can be an impossible cost, though, I know.

I think Mike’s last challenge up there could be a challenge all its own. Balancing a strong vision while still using customer feedback in a meaningful way is hard to do. The example of the Simpson’s episode shows the extremes of what happens when you listen too closely to a (particularly inept) source.

What do you think? How can you get quality feedback from your users?







18 Comments »

  1. Guilty as charged… I was one of the 80%-90% of non-respondents. I mentioned the release in a blog post, but that was about it.

    My reasons: at the time, I felt I had nothing of value to offer. The parts that frustrated me, I wasn’t sure I understood; the parts that didn’t, I had nothing to add. Add to the mix that I intended to go back and take another stab at it, and as usual, other demands on my time popped up, and it just never happened.

    My suggestions as to obtaining feedback:

    Directed questioning:
    Ask specific questions. Not “What did you think?”, but “Did you interact with the vendor? How did that go? What worked? What didn’t?” That not only helps make the feedback more meaningful, the wording may actually help the user clarify what was intended

    Convenience:
    Convenience is key. Making the user go to separate forums instead of having in-game feedback probably reduces the response rate by 50% or more… and I suspect I’m being conservative with that estimate. Worst case, have the app open a web browser directly to the forums as it exits, and remind people as they download, enter, play, _and_ exit the game that you’d really appreciate some feedback, so that the more conscientious will plan and set aside some time to do so.

    Trolls:
    Not just a fact of life… an indication of (some small measure of) success. If you haven’t had a troll, you probably haven’t been noticed at all. That said, all you can really do is ignore the ranting and drive on. If they choose to be persistent, make a personal determination as to when they are doing more harm than good, and ban them at that point. What else can you really do?

    Balancing feedback with vision:
    The tough part, to be sure, especially since the vast majority of feedback will be negative to begin with. (Positive feedback is like good news… considered less important, and usually saved for “later, if there’s time”.)

    For my part, I’m going with “know what you want to achieve, and be willing to have it crash and burn so you can pick through the wreckage.” Not really an option when you have a budget spending someone else’s money and all that, but as a personal side project, it is vaguely viable. Criticism that falls within my parameters is incorporated, criticism that falls outside is politely responded to with some variation of the following.

    “Thanks, but that’s not what this version is about. Perhaps next time around… also, if you head that direction with your own project, please let me know, I’d love to try it out.”

    Doubles as both a troll-detector and a veiled invitation to “put up or shut up”, at least in the particular circumstance I’m usually in. YMMV.

    My two cents…

    Comment by Craig Huber — 22 July, 2007 @ 6:51 AM

  2. Thanks for the feedback on feedback.

    Some specific comments:

    “The parts that frustrated me, I wasn’t sure I understood” – This is a mild form of a problem we had at Microsoft with usability tests. People would be placed in front of the software, wouldn’t understand it, and think they had “failed the test”. I heard that some would leave crying. I suppose all those years of school tests have a lasting effect. Barring the occasional idiot, a piece of software should be easy/accessible to everyone.

    Directed questioning – Good point. I wonder if I should put the questions in the game…

    Convenience is key. Making the user go to separate forums instead of having in-game feedback probably reduces the response rate by 50% or more – Interesting. I have in-game feedback, a bug icon, but only one person seemed to find/use it. There’s also an in-game forum, but that didn’t seem to get used. It would be easy for me to bring up a forum website. I’ll think about this.

    “know what you want to achieve, and be willing to have it crash and burn so you can pick through the wreckage.” – Yeah, basically what I’m doing, and expect to be doing for the next year (or more).

    Comment by Mike Rozak — 22 July, 2007 @ 3:19 PM

  3. I don’t recall seeing the bug icon, but as I recall, I found the entire interface a bit overwhelming so I could well have missed it (understand, this all may have changed drastically… this was back when you first announced on mud-dev). Even if I had, I probably would not have reported some of the things that I ran into via that mechanism, simply because I wasn’t sure they were “bugs”, per se.

    For example, I started with a Roo, and never did get much out of the gentlemen in the hut beyond “I don’t understand”, but I wasn’t sure if it was intentional due to different languages, if I had missed a setting allowing me to speak the common tongue, as well as understand it, for example.

    Another example: combat with the Rats in the tunnels, I didn’t really feel I had a good grasp of how it was supposed to work, what I was supposed to be trying to do. The feedback from clicking around, choosing different areas to attack, just wasn’t obvious enough to me, or something, I still don’t really know.

    I beat up quite a few of them trying to get my arms around it, and never really felt comfortable that I knew what I was doing (plus I didn’t really want to be attacking them in the first place, but that’s a different quirk of mine entirely)… definitely a place where a tutorial would have been helpful, but I wasn’t going to post that either, because, duh, it’s a first pass beta/test of concept, of course it doesn’t have tutorials yet.

    Anyway, you can probably see the various ways I talked myself out of commenting… hopefully it helps in terms of eliciting greater feedback on the next iteration.

    Comment by Craig Huber — 22 July, 2007 @ 6:37 PM

  4. Thanks. Some of the issues have been fixed (or are on the schedule), while others have just been added to my list.

    I suppose I have the same problems as any business: How many times have you tried a restaurant, decided you didn’t like the food, and never went back? And the owner was none the wiser as to why.

    Sticking with the restaurant metaphor: Another problem is that I’m not opening a burger joint, or pizza place. It’s niche, like Moroccan, nuveau cuisine, japanese, etc. When people do provide feedback and say that they don’t like my Moroccan dishes, is it that they don’t like Moroccan in general, or my cooking/recipies? In the case of nuveau cusine, maybe mixing ice cream and smoked salmon is just a bad idea, period. I have no idea how to reliably differentiate the three sorts of feedback.

    Comment by Mike Rozak — 22 July, 2007 @ 7:18 PM

  5. This is probably blindingly obvious, and somewhat off-topic, but as a first-time designer with an inexperienced team and virtually no production support, I ran headlong into it about a year or so ago. I’m pretty sure I’m not the only dumb designer in that situation, so here goes:

    Get the elephant gun.

    That is to say, don’t give your game to ANY testers, other than very very committed internal testers, until you have hunted down every last elephant in your living room. I’ve read several people observing that you should grab testers to look at your game and provide feedback ASAP, but the vast majority of testers will not be able to see past the elephants to provide valuable feedback.

    Typical elephants are:

    - extensive use of placeholder art, even if it’s elegantly abstract (this killed us)
    - blindingly obvious but non-fatal bugs
    - missing user feedback (“it’s planned for later on, honest!”)
    - easily-accessible debug keys or codes which have visible effects (actual quote from me to one of the coders: “For our next test, PLEASE disable the ‘Start a Riot’ key.”)

    This is part of the reason why I’ve begun to believe in Agile: it allows you to test early while still shooting all the relevant elephants.

    Comment by n.n — 23 July, 2007 @ 2:44 AM

  6. Here are some issues that come to mind:

    - There is a perception (correct or otherwise) that player feedback is tossed into the bit bucket upon receipt. This is a tough one because you obviously can’t go back to each and every player to work on their issue personally. But I wonder if something like this would help: there is a restaurant near where I live that posts their customer feedback cards on a clapboard outside their restaurant. You can literally read what the customers are saying. The restaurant claims they post all feedback cards they receive, and to their credit I have seen some less than impressed feedback. I wonder if a system like this could be used to encourage feedback. If they can actually see their feedback (and perhaps some feedback to their feedback from the team), then it might give players the notion that their feedback means something.

    - I personally have never used a feedback system that was easy to use. They usually boil down to a bunch of pulldowns that zero in and try to anticipate my problem, but don’t actually ever succeed in truly nailing it. Eventually it takes me to a text entry box, where I feel like my problem won’t be understood because I couldn’t find quite the tree to drill down to. For example, if my issue is that I can’t trade odd numbers of gold to another player, but the best explanation I can drill down to is “Unable to trade items with another player,” then I am left with this niggling feeling that my problem might not reach the right person.

    Finally, I think developers need to actively engage the community more. I know this is tough when you’re asking your devs to put in insane hours just to get the product shipped, but if you want people to talk to you then you need to listen. If you’re only getting a 1% feedback rate, take that feedback and hit the forums. Post saying “Hey, we’ve gotten some feedback that a few of you are having problems trading odd numbers of gold. We’re having a hard time duplicating it, is anyone else having this problem?” Hopefully discussion will ensue, and you’ll eventually discover that it’s only happening to players who dragged gold icons instead of typing numbers into the text box. Anything that will counter the perception that the game is being worked on in a vacuum will help players want to give you better feedback.

    Sorry about the length, I should have just articulated this better and just made it a post. :)

    Comment by Amber — 23 July, 2007 @ 11:31 AM

  7. Bah, and just to be clear in my restaurant analogy that I’m not re-inventing the feedback forum, what I’m really talking about is a kind of “scorecard” where bug reports are actually viewable to your beta testers at a level deeper than just a forum. Sort of a window into part of the dev process to show that feedback they submit is actually being taken seriously.

    Comment by Amber — 23 July, 2007 @ 11:44 AM

  8. It might be possible to have a publically-viewable part of your bug database, so users can see the feedback that they and others have submitted.

    You might allow users to attach “me too” comments to the bugs.

    A lot of open-source projects work this way, I wonder if it would be useful for closed-source projects too?

    Comment by moo — 23 July, 2007 @ 12:03 PM

  9. To expand on Amber’s comment about perception, I think it would help to change or augment the beta forums system in some way. In forums, a thread will often get pushed down (often off the first page) very quickly, giving the tester the impression that his feedback got drowned out in the noise.

    This is particularly a problem with straightforward feedback. If the tester is talking about a balance issue, that’s something other testers will almost definitely comment on and keep pushing the thread to the first page. But if the feedback is something that others don’t feel they can add to, like “there’s a problem with the clipping plane between Area-A and Area-B”, then it falls out of sight quickly.

    Another type of feedback that drops out of sight quickly is feedback about uncommon areas of interest or expertise. For example, most gamers don’t pay much attention to sounds and music (elements which are usually intended to be peripheral and complimentary to the visual elements of the game).

    I suppose one solution is to have more sub-forums. In each category (graphics, audio, etc.), provide separate sub-forums for aesthetic issues and bugs.

    Comment by Aaron — 23 July, 2007 @ 1:13 PM

  10. Random thought: If the bug database were accessible from in-world, would players be able to comprehend it? (I suspect the UI would be vaguely similar to foum UI.) Would they add bugs to it? Would they add too many duplicates?

    Comment by Mike Rozak — 23 July, 2007 @ 1:56 PM

  11. Actually, giving the player access to an entire secondary UI system devoted to reporting bugs could be help avoid duplicate reports. What if the player could hold down the Alt key to access a UI similar to the Windows Start Menu? Common bugs (clipping plane issues, pathing problems, texture problems, etc.) would each be listed separately. When the player runs across one of these bugs that is limited to a specific point or object, he simply finds the bug listing (ex: Textures), selects it, and then clicks on the object or location with the bug. The bug is then marked on the developer’s server map and any repetition of the same report is discounted or quantified (as in, an “8″ signifying that the bug has been reported 8 times).

    Of course, most testers would probably need to be educated with a description of each bug listing. Many wouldn’t go through the trouble of familiarizing themselves with the system. But if it’s true that only a very small percentage of testers provide useful feedback anyway, then perhaps those same testers are likely to care enough to use the system. If for nothing else, it might prove useful for internal testing.

    Comment by Aaron — 23 July, 2007 @ 7:37 PM

  12. More random thoughts about in-game bug reporting:

    - In-game bug reporting does sound very web 2.0-ish. I’d be able to be part of a buzz-word.

    - If a player reports in-game, you can take a snapshot of their locations, objects nearby, etc., stuff that may be relevent to the bug and which a player wouldn’t normally report. Example: Players are likely to report “The rat quest is broken!”… which leads to questions about “Which rat quest?”, “Where/how is it broken?”, etc. Logging the player’s current state, etc. could help with this. A date/time stamp could also point back to the system log.

    Comment by Mike Rozak — 23 July, 2007 @ 8:37 PM

  13. There’s the various sorts of usability tests that come to mind, which usually offer far better feedback than users will give (though not really about bugs). One type of usability test involves an observer watching the evaluator (player) and recording observations of what they do.

    There may be ways of incorporating that in games. In instanced content, the observer could be warned with a popup “raid entering instance xxxyxyy” and they could warp in with an invisible avatar and observe. Of course, there’s so much going on some automated gathering of the character’s activities would be necessary. The other problem is usability testing often works best if you can hear and talk to the person, that would only work in a game with built-in voice chat, and the players would have to agree to let a tester listen in or even ask questions “why are you doing this or that” etc.

    Though obviously this type of testing and observation is more geared towards usability and are players playing the way we think they are? It’s not for bugs, or finding out what is “fun”.

    Another way of gathering information on what players do would be to record player actions, at certain boss fights, pvp fights. Those recordings could be analyzed for trends and compared to expected behavior.

    From this information you could find out if a feature or skill is not being used, or being used much greater than other skills, or over/underpowered (you’d have to record effects of the skills too) and then adjust the balance. It would certainly be more objective than user feedback on forums.

    The server load is the big problem though, you certainly wouldn’t want to be recording all the time. It probably works best on instanced content as well.

    And I like the snapshot idea for bug reports, it is so hard to get testers to give the full context, and usually that just puts them off if you try to enforce it. Of course, again, taking big snapshots adds server load.

    Comment by yunk — 24 July, 2007 @ 9:18 AM

  14. - Get professional testers in-house. Even if it’s just one, the difference that having a dedicated tester makes is big. Obviously more are better, as you get a wider variety of views and playstyles. These are the people that will pick up the problems that your beta tester crowd won’t, things that normal gamers don’t look for, or unconsciously block out. Don’t use an agency though, it costs more, and their testers are often not trained, working out as paying for beta testers.

    - Interact with your playerbase. Hold occasional “chat with the developer” Q&A sessions, where you can get lots of feedback fast, and direct the topics to an extent. Chat with players at random in-game.

    - Offer rewards for good feedback. Best provider of feedback gets a lifetime free pass, or a cash prize. Best dozen providers of feedback get a mention in the credits. Best being a balance of both quality and quantity, of course.

    - One that every developer should do: Watch the game over people´s shoulders. Get friends, family to play it, and just watch them without interfering. Invite a random sampling of your beta testers to a day with the devs, and watch them play. Just watching how other people interact with the game is often the best way to highlight problems.

    Comment by Lobosolitario — 25 July, 2007 @ 4:21 AM

  15. Linkage 20070725

    [...] a lull at his gig: 3 posts in one week? Unheard of! I commented in this Weekend Challenge about gathering feedback, thought I’d link to it as well, maybe prompt a few additional [...]

    Pingback by Voyages in Eternity — 25 July, 2007 @ 4:43 AM

  16. Dont forget to play your own game :)

    Comment by paul — 26 July, 2007 @ 3:36 AM

  17. Bug-reporting UIs take a lot of designing. It would have to be simple enough that beta testers could use it without training, yet comprehensive enough to cover most, if not all problems that could crop up (or else you just end up with a thousand bug reports listed under misleading headings). Added to the fact that one (untrained) person’s controller issue is another person’s gameplay issue, and the man-hours needed to implement and maintain/improve the UI, it’s debatable whether such a system actually brings any benefits on a small scale.

    I’ve worked with bug-reporting software used by one of the largest companies in the industry, and it took a fair amount of training to get new testers to be able to use it effectively, and then another dose of training on top of that to get them to actually write decent bug reports. Since you can’t really train beta testers, the design would have to be top-notch to start with, which is easier said than done.

    Of course, you can certainly post up bug-reporting guidelines, in the hope that some of them will read them.

    Comment by Lobosolitario — 26 July, 2007 @ 8:41 AM

  18. Participation inequality

    [...] can present problems to the developer, and is related to the recent challenge on collecting feedback. A friend forwarded me an interesting article recently called, “Participation Inequality: [...]

    Pingback by Psychochild’s Blog — 27 July, 2007 @ 3:08 PM

Leave a comment

I value your comment and think the discussions are the best part of this blog. However, there's this scourge called comment spam, so I choose to moderate comments rather than giving filthy spammers any advantage.

If this is your first comment, it will be held for moderation and therefore will not show up immediately. I will approve your comment when I can, usually within a day. Comments should eventually be approved if not spam. If your comment doesn't show up and it wasn't spam, send me an email as the spam catchers might have caught it by accident.

Line and paragraph breaks automatic, HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Email Subscription

Get posts by email:


Recent Comments

Categories

Search the Blog

Calendar

September 2019
S M T W T F S
« Aug    
1234567
891011121314
15161718192021
22232425262728
2930  

Meta

Archives

Standard Disclaimer

I speak only for myself, not for any company.

My Book





Information

Around the Internet

Game and Online Developers

Game News Sites

Game Ranters and Discussion

Help for Businesses

Other Fun Stuff

Quiet (aka Dead) Sites

Posts Copyright Brian Green, aka Psychochild. Comments belong to their authors.

Support me and my work on