Psychochild's Blog

A developer's musings on game development and writing.

30 November, 2006

Programming tests don’t always pass
Filed under: — Psychochild @ 4:02 PM

Sorry for the interruption in posting lately. I fell sick with something vile very early Tuesday morning and have been recovering. I’m on the mend, though, and probably not going to die soon. Probably….

Recently I went to a job interview for a position at a game company with a heavy emphasis on programming, and one part of the interview was a written programming test. I’ve ranted about game design tests before, so I figured it was only fair that I write a bit about programming tests.

If you read my rant on game design tests, you might already know part of my feeling on this type of test. I think that programming tests are generally more acceptable than a designer test, since you can test things for correctness easier. There’s a bit of subjectiveness in “is this clean code”? If the test is for C++, for example, the issue of where to place the opening brace could introduce some potentially unfair bias.

There are actually two areas that I think need improvement. First, this test was a written test. Now, most programmers I know have terrible handwriting. Actually, I can write fairly nice if I take my time. But, worrying about handwriting on a timed programming test isn’t exactly at the top of my list of things to consider. I see no reason not to give the applicant a computer with a text editing program to type in responses. Let the applicant print off copies if you need something physical to pass around. In addition to handwriting issues, the backspace key is much cleaner than scratching out incorrect bits of writing. (Or smudging the sheet because it was written in pencil instead.)

“But, Brian,” you interject, “if they have access to a computer, they could cheat and look up the answers online!”

Well, this gets my second criticism: doing these tests in isolation. Am I really a better programmer for remembering the rules for virtual functions in C++ without a reference? I don’t think so. Especially for a test taken on site with a time limit, I see little harm in allowing a person a reference. After all, if someone can teach themselves the language well enough in an hour to pass such a test, then they can probably do a pretty good job despite it all.

And, really, it’s silly to think that a test-taker is always going to be in isolation. I was left alone in a cubicle with the test and my cell phone; my cell phone is internet-enabled and I could have used it to look up the answers, anyway, if that were possible. (I didn’t, though.)

Further, I think not having a reference is also unfair to otherwise good programmers that know a lot of languages. I know half a dozen programming languages well at this point. I’ve also been working mostly in Python and a proprietary scripting language over the past few years, so my C++ skills aren’t razor-honed. So, it took me a bit to remember the specific syntax for a while loop in C++ as opposed to another language. Something you’d catch pretty easily during compiling anyway.

Finally, you’re not necessarily going to be able to test the applicant very well with a simple test. Even if I remember C++ syntax, that doesn’t mean that I can understand a particular piece of dense code. Most game code isn’t for the faint of heart, so it’s one thing to write your own implementation of atoi() without calling C runtime functions, but it’s another to trace through dozens of function calls through half a dozen classes to figure out how a spell is actually cast. (This is a about how complex it is to cast a spell in M59.)

In the end, programming tests are necessary, though. Unfortunately, a few people overstate their competence in coding, so you need to weed out the good from the bad. But, I think there are some things that can be done to improve the system and help really separate the pretenders from the people that have a clue.

What are your thoughts?







14 Comments »

  1. I agree 100%.

    I’ve walked out of isolated environment tests (write this program in 20 minutes!) if the appropriate man pages or MSDN are not available. Programmers are paid to understand their platform and know how best to manipulate it, not to memorize syntax.

    Design tests i would define as slightly different – it’s more than possible to draw out object libraries, tiers and architecture without any reference and that’s probably a better guage of understanding anyway.

    Comment by Cael — 30 November, 2006 @ 4:43 PM

  2. Design tests i would define as slightly different….

    I should specify that I’m talking about game design tests here, not software design. I edited the post to make it clearer.

    Comment by Psychochild — 30 November, 2006 @ 5:33 PM

  3. Depends entirely on how they’re used.

    A lot of companies (like ours) use them to filter out the kinds of people who (extreme example) gladly put “C++, 10 years” on their resume, and then can’t tell you a single C or C++ way to count the chars in a string. Great way to cut the interview short.

    Personally, I love companies who do tests like this and then take the results too seriously. I interviewed with one before ending up at SOE, was told that I “didn’t do well enough,” and ended up with a far better job at a much better company (read: here). If they’re going to rely on a written test that much, no sane coder wants to work for idiots like that anyway.

    The filter effect goes both ways. :)

    Comment by Scott Hartsman — 30 November, 2006 @ 8:35 PM

  4. Heh. My resume states “C/C++ (rusty, but competent)”; I don’t bother with the minor detail that I got first exposure back in middle school.

    Comment by Michael Chui — 30 November, 2006 @ 10:22 PM

  5. Looking at it from the opposite point of view: hiring a programmer who can’t do what his resume says he can do, or hiring a designer who talks a good game but doesn’t deliver usable work product – these things can single-handedly set your project back months. Interviews are fine, but paint only a partial picture of a candidate. Actually getting a sense for how things work is better. I’m definitely pro-testing of candidates.

    That being said, I’ve seen a lot of extremely lousy tests in my time in the industry. One guy tried to test a game scripter on C++ (a language he’d never used before). Another asked every applicant about the inner workings of SGI servers (a skill that no other company in the industry would be likely to require). I’ve taken a test where I was asked to design a chess AI on the fly (a particularly annoying design test, as there was clearly a right answer that the guy had read in Science Magazine that month or something).

    A good test focuses on what the team actually needs. If your team is weak in C++ and you desperately need someone to fill a slot, filling a test with C++ is fine. But most of the time, what you want to test for is how that person thinks, how that person interacts with team members, and whether or not that player dots his ‘i’s and crosses his ‘t’s. My last company did programming tests in person – the programmer was given a problem, but the lead programmer stayed in the room to answer questions and bounce ideas off of him. This process seemed to yield us good results.

    And on the flip side, I’ve seen a design test submitted by someone that was a 150 page game design, which was full of typos, incomplete sentences, and template text (“Write zone description here”). It only takes receiving two or three tests like that from guys who gave good interview to be thankful that you have some sort of filter in place.

    Comment by Damion Schubert — 30 November, 2006 @ 10:53 PM

  6. I gather that there is no ‘industry standardization’ when it comes to programming skills? Seems that certain core competencies would be common across the field of game design, and well designed tests could filter for those skills without being wasteful. Creates an opportunity for a wise recruiting company, or does it?

    Comment by Grimwell — 30 November, 2006 @ 10:57 PM

  7. @Grimwell:

    Object models are object models and inheritance is a concept that’s easy to express in any OO language or ever pseudocode, so those kinds of tests are not only fine, i’d say they were a great measure of an applicant’s understanding.

    Beyond that, yu really need to start asking deep questions about the platform you’re interested in. Delegates and resource issues in DirectX, for example.

    Comment by Cael — 30 November, 2006 @ 11:57 PM

  8. Scott Hartsman wrote:
    Depends entirely on how they’re used.

    Fair enough. But, it can be hard to tell how people will use the test when taking it. And, my major complaints were how unrealistic the test is. I will always have have a reference at my fingertips because I’m not always going to remember some details of the language. And, most game programmers worth their salt will be able to pick up new concepts easily. For example, I haven’t used virtual functions in much of my C++ programming work, but it’s something I could learn in a few minutes with a good reference. If I couldn’t remember the details of virtual functions on a test, should that count against me? Would the test giver know the difference between me not having used virtual functions much before compared to me being a lying n00b? Probably not.

    Damion Schubert wrote:
    …hiring a programmer who can’t do what his resume says he can do, or hiring a designer who talks a good game but doesn’t deliver usable work product – these things can single-handedly set your project back months. Interviews are fine, but paint only a partial picture of a candidate. Actually getting a sense for how things work is better.

    Yes, but how often am I going to have to write a program using only a pen and paper and no reference material? Or, referring back to the design test, how often is a designer going to sit at home and have a only a few hours to come up with design ideas out of whole cloth? Interviewing only paints a partial picture, but testing gives you even less in most cases.

    My last company did programming tests in person – the programmer was given a problem, but the lead programmer stayed in the room to answer questions and bounce ideas off of him. This process seemed to yield us good results.

    Of course, this can be problematic, too. In one interview I sat in on (not the one being interviewed), the programmer asking questions seemed more interested in establishing himself as alpha programmer rather than actually testing the candidate. He asked the obscure version of a question (“How would you iterate 257 times with only an 8-bit variable to keep count of iterations?”) rather than the useful version (“Name some differences between while and for loops.”). Of course, that programmer probably did that applicant a favor…. ;)

    In another instance, a female friend of mine went to interview for a technical position at another company. She told me this horror story about the guys interviewing her seemly only being interested in showing geek dominance over her. Again, as Scott said above, I guess the test went both ways here. But, it kinda sucks to have to deal with this type of stress while already in the uncomfortable waters of interviewing.

    You also have the issue something you mentioned before (I think it was at Austin): where the applicant may be qualified, just not in the way you expect. You mentioned that most designers have level layout or world building ability whereas your focus is more on documentation. A company giving a design test that has a lot of level layout issues might be missing a qualified applicant that just happens to be superb at writing documentation as you are. The company will just have to rely on other designers for level layout ability. :)

    Cael wrote:
    Object models are object models and inheritance is a concept that’s easy to express in any OO language or ever pseudocode, so those kinds of tests are not only fine, i’d say they were a great measure of an applicant’s understanding.

    Sometimes. The problem is that sometimes the terminology doesn’t match up. The applicant may understand what “polymorphism” or “cohesion” is, even if he or she doesn’t use (or remember) those specific terms.

    Some interesting points of view, though. Good discussion. :)

    Comment by Psychochild — 1 December, 2006 @ 2:28 AM

  9. Not to mention, I don’t care if the applicant can quote “polymorphism” or “cohesion”.

    When interviewing I usually use a short written test. The programming component of it is rather interesting. On one hand, it is important to include to see if the person can actually write an open brace successfully. On the other hand, I don’t think failing to recall the nuances of virtual destructors on a test should be a job-denying failure. The trick, I think, has less to do with the quality of the test as the quality of the interpretation of the test. First, the test needs to be abstract enough to not require online research. Ie, any test like: “Draw a cube on the screen with Direct X” is rather pointless. The right answer is to cut and paste from the Microsoft sample directory. (OTOH, I might actually be able to code that blind for GL…) I tend to like simple straight forward algorithms, such as “Do a binary search on a sorted array”. As for interpreting the results? Then, in interpretting the results, it usually is very easy to see various things. If the person is just rusty in C, one can take that into account.

    This is where a computer might be a disservice – if you provide a computer, I’d also provide source code to work with and a compiler and ask them to make a specific change. This would be harder for the person rusty in C than letting them drop the semicolons and let the test-interpretor deal with it.

    Grimwell: I’m not sure what I’d think about a SAT style programming test. I would have hoped that the ability to graduate with a CS degree would imply the applicant could readily pick up any language we threw at them, but we’d be quite wrong. My fear is such a testing scheme would just reward those who study for the test, almost ensuring the high marks on the test are the worst programmers.

    Comment by Brask Mumei — 1 December, 2006 @ 7:16 AM

  10. The big point in my mind is the additional headache tests can cause for a programmer that works with more than 4 languages on a daily basis. For your standard financial business type setting you work with dozens of systems. Mainframes, windows machines, hpux you name it. For myself in particular it was very easy to take coding tests when I was in school, focused on one language every single day. If I were to interview for a job using both c# and java I would no doubt mix both languages together :/
    Mix in oracle, db2, mysql and MS Sql and youve got yourself quite the fun time.

    Comment by aaron — 1 December, 2006 @ 7:43 AM

  11. My fear is such a testing scheme would just reward those who study for the test, almost ensuring the high marks on the test are the worst programmers.

    A very valid fear from what I’ve seen in my past. When I worked at a major finance company in support, I’d watch people get hired on the networking team that were rock stars on paper. MCSE, Novell credentials, all that paper jazz. They interviewed well and knew the answers to all the questions. Then you’d put them on a real network and they would utterly fail. They were rock stars in the land of theory, but in the land of reality they couldn’t adjust to the little things that actually happen on a live network.

    Comment by Grimwell — 1 December, 2006 @ 8:20 AM

  12. I sometimes gave people a programming task they DIDN’T know how to solve, such as how to write a program like Spy++ and get the names of all the buttons in another application’s dialog box.

    I stuck them in front of a computer with visual C++ (and all the help). The point was to see how well they figured out how to solve the problem (with all the docs at hand), and then execute on the program.

    Comment by Mike Rozak — 1 December, 2006 @ 6:07 PM

  13. Q: What does multiple inheritance mean in C++?

    A: You are in for a world of hurt.

    Comment by Brask Mumei — 6 December, 2006 @ 7:37 AM

  14. In my experience, programming tests are just as much an anti-pattern of technical management as game design tests. What usually happens is that a designer or programmer is promoted to technical or creative director and given responsibilities that they have never had before. They are given no training in interviewing. They have no interest in either the human resource requirements of their company or the career development of potential hires. So they blag it.

    In that situation, tests are extremely appealing: they present an appearance of competence to the uninitiated, they’re cheap (for the test setter), they give us the illusion of security, and they provide us with a fall guy when it all goes wrong.

    However, on the other side of the interview table, they are an important warning sign for those of us with enough experience in the industry to know about it. I avoid companies that test their applicants; assuming they’ve made it clear up front. If they hide it, and spring a test on me during an interview anyway, then I walk. As a contractor this has been my own company policy for a couple of years now and (according to reports) has allowed me to avoid a number of game development horror shows. I recommend this approach to anyone working in games (or anywhere else for that matter.)

    To those who set tests in the firm belief that they are necessary, I suggest a rethink. You are implying that without the tests you may accidentally employ somebody incompetent. Is this really the image you want to present?

    Comment by Paul Sinnett — 10 June, 2007 @ 3:48 AM

Leave a comment

I value your comment and think the discussions are the best part of this blog. However, there's this scourge called comment spam, so I choose to moderate comments rather than giving filthy spammers any advantage.

If this is your first comment, it will be held for moderation and therefore will not show up immediately. I will approve your comment when I can, usually within a day. Comments should eventually be approved if not spam. If your comment doesn't show up and it wasn't spam, send me an email as the spam catchers might have caught it by accident.

Line and paragraph breaks automatic, HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Email Subscription

Get posts by email:


Recent Comments

Categories

Search the Blog

Calendar

December 2018
S M T W T F S
« Nov    
 1
2345678
9101112131415
16171819202122
23242526272829
3031  

Meta

Archives

Standard Disclaimer

I speak only for myself, not for any company.

My Book





Information

Around the Internet

Game and Online Developers

Game News Sites

Game Ranters and Discussion

Help for Businesses

Other Fun Stuff

Quiet (aka Dead) Sites

Posts Copyright Brian Green, aka Psychochild. Comments belong to their authors.

Support me and my work on