19 August, 2006
I sometimes apply for jobs in the industry to see what I can find out. Hey, you never know, I might find a really cool position that scratches the itch I have. So far, I really haven’t found much that has gotten me interested. Of course, there’s also the places that have rejected me. Most of the time it’s been a good thing that I didn’t get the job, since things changed quite dramatically right after the position was filled. Like the SOE R&D group getting absorbed into a typical development team and Raph leaving the company shortly afterward. ;)
But, I once ran into something that boggled my mind. A design test for a lead designer position. Now, I wasn’t exactly thrown at first, since it’s not uncommon for for programmers to have to take a test during the interview process. I mean, if you’re going to hire a graphics programmer it would be nice if he or she knew how to do matrix math, right?
But, the more I think about the design test the more I think what an awful waste of time it was.
And, no, I won’t name the company. It’s a small industry, and I’m probably burning bridges just mentioning this as a generic example. People get touchy. But, this is a blog and I must rant, right?
The format of the test was a bit odd. The test was completely automated online and you are given a limited amount of time to answer a series of questions. Again, the test-taking programmer in me didn’t balk at first and dove right in. But, there are two major problems here. First, the test is one-sided. The best design comes from collaboration, but all I was able to do was throw my answers into the void and hope they resonated with the person scanning the test for an excuse to reject the likely thousands of candidates they have trying to take the test. I have always found that riffing off people, even when I am the lead, helps to make more interesting designs. Second, design isn’t something that can be measured in a few hours. Real design treatments for a single game take weeks upon weeks of work to complete and are the work of teams. Answering a half dozen questions in a much more restricted time doesn’t really give a good feel for what kind of job a person can do as a lead designer. It’s quite different being able to write about a cool idea for a little bit and having the dedication to stick with a project and really start considering the consequences of choices and changes made to the design.
I’m becoming particularly sensitive to this second issue as I enter the home stretch of editing the chapters for the book I’m working on. People have submitted chapters in all sorts of states, from the highly polished final manuscript to the really, really, really rough draft. But, just because someone writes one of those very rough drafts doesn’t mean the person is a terrible writer; it’s actually my job as editor to take that writing and guide it to being informative, entertaining, and fit within the publisher’s guidelines.
There’s also the issue of what the test reviewer is looking for, too. As I mentioned above, all I can hope is that the answers I provide resonate. But, there’s also meta issues to think about. For example, if a question asks about how to achieve game balance, is it acceptable to say something to the effect of, “you cannot”? Is it a trick question intended to get you to try to say you can achieve the unachievable: perfect balance that will stop people from complaining? Or, are they looking for someone to state this fact. (And, if they think that perfect balance is achievable, I’m pretty sure I don’t want to work at that place, anyway. Talk about unrealistic expectations from a lead designer!)
Most of these complaints are the same ones I brought when Tadhg Kelly wrote about a design test on his blog. I felt the test really didn’t test people fairly because it didn’t offer the opportunity for feedback and iteration. It only really gives a general idea of what the person can generate off-the-cuff, which is rarely used in the final version of real game projects. Someone who can listen to feedback and incorporate it into the design is much more highly prized. But, this is incredibly hard to measure even with a non-automated system.
In the end, I think the biggest sin here is that the test was automated, especially since this is a lead role. I admit, I’m not exactly a huge celebrity in this industry, although I do have my minor bits of fame. But, if you want to see some of my design philosophy in action, I have four years of patch notes that I can offer up for discussion. On the other hand, if I had bothered to use my contacts to get contact information inside the company, I probably could have offered the patch notes as an alternative. Or discussed things with a real person.
What do you think? If you’re a company owner, would you use an automated test? If you’re a designer, would you want to be subject to one of these tests? Comments are, as always, welcomed.