Psychochild's Blog

A developer's musings on game development and writing.

19 August, 2006

Design tests fail
Filed under: — Psychochild @ 1:20 AM

I sometimes apply for jobs in the industry to see what I can find out. Hey, you never know, I might find a really cool position that scratches the itch I have. So far, I really haven’t found much that has gotten me interested. Of course, there’s also the places that have rejected me. Most of the time it’s been a good thing that I didn’t get the job, since things changed quite dramatically right after the position was filled. Like the SOE R&D group getting absorbed into a typical development team and Raph leaving the company shortly afterward. ;)

But, I once ran into something that boggled my mind. A design test for a lead designer position. Now, I wasn’t exactly thrown at first, since it’s not uncommon for for programmers to have to take a test during the interview process. I mean, if you’re going to hire a graphics programmer it would be nice if he or she knew how to do matrix math, right?

But, the more I think about the design test the more I think what an awful waste of time it was.

And, no, I won’t name the company. It’s a small industry, and I’m probably burning bridges just mentioning this as a generic example. People get touchy. But, this is a blog and I must rant, right?

The format of the test was a bit odd. The test was completely automated online and you are given a limited amount of time to answer a series of questions. Again, the test-taking programmer in me didn’t balk at first and dove right in. But, there are two major problems here. First, the test is one-sided. The best design comes from collaboration, but all I was able to do was throw my answers into the void and hope they resonated with the person scanning the test for an excuse to reject the likely thousands of candidates they have trying to take the test. I have always found that riffing off people, even when I am the lead, helps to make more interesting designs. Second, design isn’t something that can be measured in a few hours. Real design treatments for a single game take weeks upon weeks of work to complete and are the work of teams. Answering a half dozen questions in a much more restricted time doesn’t really give a good feel for what kind of job a person can do as a lead designer. It’s quite different being able to write about a cool idea for a little bit and having the dedication to stick with a project and really start considering the consequences of choices and changes made to the design.

I’m becoming particularly sensitive to this second issue as I enter the home stretch of editing the chapters for the book I’m working on. People have submitted chapters in all sorts of states, from the highly polished final manuscript to the really, really, really rough draft. But, just because someone writes one of those very rough drafts doesn’t mean the person is a terrible writer; it’s actually my job as editor to take that writing and guide it to being informative, entertaining, and fit within the publisher’s guidelines.

There’s also the issue of what the test reviewer is looking for, too. As I mentioned above, all I can hope is that the answers I provide resonate. But, there’s also meta issues to think about. For example, if a question asks about how to achieve game balance, is it acceptable to say something to the effect of, “you cannot”? Is it a trick question intended to get you to try to say you can achieve the unachievable: perfect balance that will stop people from complaining? Or, are they looking for someone to state this fact. (And, if they think that perfect balance is achievable, I’m pretty sure I don’t want to work at that place, anyway. Talk about unrealistic expectations from a lead designer!)

Most of these complaints are the same ones I brought when Tadhg Kelly wrote about a design test on his blog. I felt the test really didn’t test people fairly because it didn’t offer the opportunity for feedback and iteration. It only really gives a general idea of what the person can generate off-the-cuff, which is rarely used in the final version of real game projects. Someone who can listen to feedback and incorporate it into the design is much more highly prized. But, this is incredibly hard to measure even with a non-automated system.

In the end, I think the biggest sin here is that the test was automated, especially since this is a lead role. I admit, I’m not exactly a huge celebrity in this industry, although I do have my minor bits of fame. But, if you want to see some of my design philosophy in action, I have four years of patch notes that I can offer up for discussion. On the other hand, if I had bothered to use my contacts to get contact information inside the company, I probably could have offered the patch notes as an alternative. Or discussed things with a real person.

What do you think? If you’re a company owner, would you use an automated test? If you’re a designer, would you want to be subject to one of these tests? Comments are, as always, welcomed.


« Previous Post:





8 Comments »

  1. Hmmm…. well, I’m the first to admit that I’m not a great designer. I once thought I was, before I got actual game design experience. After all, I know games, and I know what’s fun, right? It’s kinda like everyone thinking they could write the Great American Novel. Or insert nationality of choice. I mean, almost everyone is literate and can write these days – and everyone’s got a story or two to tell.

    Then you get into the thick of it and discover how much more is really needed.

    Moving back to the topic at hand though – I think a test might be reasonable if you assume there are no right or wrong answers, or even a pass / fail sort of thing, but it’s just some kind of an indication to the reviewer to see how a candidate thinks.

    It could also help weed out the grossly inept or inexperienced folks. You know, the kind that might answer a question by saying, “I think you shouldn’t do skill based systems because Ultima Online had skill based systems and I hated that gaem because it sucked and it was because it uses skills and it was so stupid.”

    Though I would hope that would be easily determined within 5 minutes of the job interview. Ah, well, basic competency gets tested in programmer tests, too.

    It may not be a strong case, and I’d certainly not not weight such a test heavily, but I can at least see those reasons for it.

    Comment by Jay Barnson — 19 August, 2006 @ 11:38 AM

  2. Really, this is what a cover letter and resume are for. If you have someone with a poorly written cover letter filled with insults and no game experience on their resume, you can pretty much assume they’re not the right person for the job. On the other hand, someone with nearly a decade of experience might be someone you would want to at least talk to on the phone. At least, that’s my opinion.

    I guess it still amazes me that someone thinks an automated test is a good assessment of a lead designer of interactive games. I might be able to see it if this were an entry level position, since you can weed out a lot more people looking for the cream of the crop. But, for a lead position? You’re not really measuring the skill set necessary for leading other designers.

    Some clarification on my thoughts,

    Comment by Psychochild — 19 August, 2006 @ 3:39 PM

  3. A face-to-face interview is key even for a junior position. We all need to get over the idea that someone who is in HR can possibly provide anything useful in the hiring process except mechanically bring the person on board.

    They can’t read resumes properly, can’t do interviews, so the only improvement an automated test provides is that it replaces some of the non-productive HR folks.

    Comment by PlayNoEvil — 19 August, 2006 @ 5:13 PM

  4. I’ve had to take a written test before and I felt it was done pretty well. The position had a lot to do with writing and having previous experience playing MMOs, and the questions were crafted in such a way that it would reveal a lot about what I knew (and how well I could write).

    I believe one of the questions was something along the lines of, “If everyone in your game accidentally duped coin for a full week before you caught it, what would you do?” Tough question, but it was a good one. Another question was essentially, “Write journal text for a delivery quest.” That allowed for some creativity and exposed my ability to write (and showed that I did in fact know what a delivery quest was).

    For a lead position, however, I’m not sure it’s necessary. You shouldn’t be interviewing anyone for a lead position unless they’ve been part of a development team in the game’s genre (such as “MMO”), so tests like that don’t really do much good.

    Comment by Ryan Shwayder — 19 August, 2006 @ 7:46 PM

  5. I’m not in the game industry, but I’m in a technical field and my company does use automated testing for screening applicants, but as a manager I really don’t like it and I have “overridden” the tests on occasions because I don’t trust them to NOT screen out potentially great candidates who just F up on a test.

    I think the tests (in my company’s case) are a good way to weed out the nutcases and the inexperienced folks who are (for now anyway) aiming just a bit too high for themselves. However, I have also interviewed a few people who have not done well on the test but really, really impressed me in the interview. And, if I’m going to take the time to interview them anyway why bother with the test, right? Good question.

    All I know is that the best guy on my staff (and he is truly great) would not have made it to the interview if I had relied on his test screening. The opposite is true also: I’ve had guys do well on the test who interview so poorly that I wonder if they actually had someone else take the test (being facetious – our tests are administered onsite).

    In the end, for my industry anyway, I don’t think there’s a substitute for face-to-face interviewing.

    Comment by chabuhi — 21 August, 2006 @ 6:40 AM

  6. I just recently took a test which was not a design test. But since we’re all ranting….

    I got “give us an interface that looks like this and uses these data. Here’s a laptop. You have one hour.”

    VS2005, no problem. I sit down, write a few generic classes, get dx9 working properly, decide to integrate with the db via connection string stored in config and can’t remember the changed syntax for retrieving AppSettings. So i hit F1 and find that MSDN is not installed.

    Sorry. I walked.

    If you want to hire somebody, you do not ask them to do a job without the basic tools. My job is to understand systems and how they work. It is no longer to remember the arcaneries of syntax and it hasn’t been for almost ten years.

    Go hire an Indian college grad instead.

    Comment by Cael — 21 August, 2006 @ 8:45 AM

  7. Hey watch out for the personality “inventories”, which is another name for profiling tests. They don’t test for your skills or the way you think, but your attitude and other unique personality traits.

    These are far worst!

    And if the company doesn’t have a strong privacy policy, the info may ciculate or remain on file forever!!!

    Comment by magicback (frank) — 21 August, 2006 @ 1:39 PM

  8. Programming tests don’t always pass…

    Recently I went to a job interview for a programming position. One part of the interview was a written programming test….

    Trackback by Psychochild's Blog — 30 November, 2006 @ 4:02 PM

Leave a comment

I value your comment and think the discussions are the best part of this blog. However, there's this scourge called comment spam, so I choose to moderate comments rather than giving filthy spammers any advantage.

If this is your first comment, it will be held for moderation and therefore will not show up immediately. I will approve your comment when I can, usually within a day. Comments should eventually be approved if not spam. If your comment doesn't show up and it wasn't spam, send me an email as the spam catchers might have caught it by accident.

Line and paragraph breaks automatic, HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Email Subscription

Get posts by email:


Recent Comments

Categories

Search the Blog

Calendar

March 2019
S M T W T F S
« Nov    
 12
3456789
10111213141516
17181920212223
24252627282930
31  

Meta

Archives

Standard Disclaimer

I speak only for myself, not for any company.

My Book





Information

Around the Internet

Game and Online Developers

Game News Sites

Game Ranters and Discussion

Help for Businesses

Other Fun Stuff

Quiet (aka Dead) Sites

Posts Copyright Brian Green, aka Psychochild. Comments belong to their authors.

Support me and my work on