Psychochild's Blog

A developer's musings on game development and writing.

27 August, 2016

Testing game programming
Filed under: — Psychochild @ 9:23 PM

Hiring programmers is a complex process. You need to find someone with the competence to perform the work and the temperament to work well with a team. In game programming, you also have to find people who will do highly technical work while making less than they could in just about any other industry.

But, the industry does itself no favors here. The way we test and judge programmers demonstrates a mindset stuck in the past. Let’s take a look, shall we?

Testing the wrong way

I wrote previously about problems with programming tests. This particular test dealt with a test in person, but a lot of the same sentiment applies.

Since that time, it seems more fashionable to give people tests remotely. That means that a typical game programmer needs to get past the HR screening, the tech screening call, and a programming test before they get to get flown in for a technical interview, probably one where you’re going to be writing programs on a whiteboard.

And this assumes that each step of the way is flawless. I spoke with a game designer at a large MMO company, mentioning how I had a terrible interview experience previously; the person doing the tech screening call was aggressively rude to me. The designer asked a name, saying they wanted to “confirm a suspicion”. After giving the name, the designer said that, yes, that person was indeed the person they were thinking of. So, the possibility of me working at one company was scuttled by someone who is known for being rude to others.

The other problem with many tests is that they don’t mimic working conditions. Not only are they usually adversarial in nature, they often involve programming alone. Most game companies are using some variation of Agile, which means that you don’t have programmers programming in isolation. Many places have code reviews, were your code is evaluated by another programmer before it gets committed to the codebase. So, mistakes are often caught by someone else; the expectation is that nobody is perfect and “given enough eyeballs, all bugs are shallow.” But, this isn’t reflected in the testing. In fact, you get very little feedback short of a generic HR response.

So, the gauntlet of steps to go through the hiring process is not easy. And, keep in mind, someone might be applying to many different positions, meaning they might be juggling many different programming tests.

Testing the wrong things

What do you hire a programmer for? Particularly a Senior or Lead programmer? Often in these positions, raw programming ability is only one part of their responsibilities. Often you’ll want someone who can plan out and architect a system, someone who can mentor other engineers, and someone who can discuss difficult problems with others. A lot of companies could benefit from better leadership, but requiring practical work from your leaders may mean they don’t do the best job of leading. But, how do you test for leadership capability?

Now, I know I’m a “snowflake” with my unique background. Yes, I have a degree in Computer Science, but I’ve been active in design and even lead designer on some projects. So, I’ve been more interested in the technical designer positions more than pure engineering, although technical designers are still pretty rare in the mainstream industry. But, hiring me purely for my technical ability largely misses the point of why you would want me in your company. A friend of mine said that I was the type of person who would solve any problem that gets thrown at me, maybe with a bit of learning in the middle that I would also enjoy. This is accurate as it’s something I’ve had to do with the startups I’ve worked at. But, how do you test for that?

So, giving a programmer a test to see how well he or she can solve a CS 201 problem in C++ can miss the point. The test isn’t going to reveal this aspect or how eager someone is to learn new things even if not given the chance to become a domain master. It doesn’t reflect their experience beyond perhaps seeing and remembering a solution to that problem.

In addition, some tests also assume certain knowledge. One programming test required that you solve a problem heavy in vector math. Not every programmer is an expert in vector math, so you’re selecting pretty heavily for a certain type of programmer, and ignores the linguistic approach to programming which is as valid as a mathematical approach.

One last problem is that there are a ton of little things that the assessor might judge someone on. Does a preference for camelCase instead of underscored_variable_names count against a prospective programming hire? Programmers are known for their holy wars, and you never know when someone is going to take exception to your default style for not being “right”.

Ignoring experience

The interesting thing here is that most companies ignore experience. Someone might have held senior engineer positions at many companies, but they must still take the programming test that “everyone must take”. As if no other company tests for engineers.

The usual complaint about this is that some programmers might have experience listed on their resume that they don’t actually have ability with. But, assuming that someone held a senior position at another company, it probably means that the person had some qualities that made them fit for that position at that company. But, again, the process seems adversarial. Incompetence is assumed until proven otherwise by a test that may not actually test for competence.

This type of testing also tends to favor younger applicants, those who have more recently CS 201 tests and are comfortable with that type of testing. The game industry has a pretty notorious ageism problem where the assumption is that older people can’t possibly endure the rigors of making a game. And, it stands to reason that this type of programmer testing contributes to that problem.

Ignoring the internet

I read an article lamenting “digital amnesia”, where people think it’s easier to look up some fact rather than commit it to memory. Which makes sense to me; the less time I spend memorizing trivia is more brainpower I can commit to more interesting work. Of course, some people think that memorizing trivia is somehow more virtuous than knowing how to look up that trivia. And, really, the game industry is like that.

One programming test I took admonished against talking with other people or doing “too much” research on the internet. If you have to look up how to solve the problem, it warned, you won’t “survive” the in-person interviews. But, what is the most efficient way to solve a problem you don’t understand? Ask someone or look it up on the internet, the two things explicitly prohibited by the test.

Modern programming has had a strong focus on not reinventing the wheel. Object orientated programming was supposed to make it easier to use work that others have already done. Software libraries and well-defined APIs let you build your work on top of the work of others. Expecting someone to code something from scratch may show you their programming approach, but it’s not an indicator of how well they will do in day-to-day programming that is well beyond CS 201 problems.

Improving the process

The main problem is that these type of tests tend to select for a very narrow type of person. It favors younger people since it uses tests reminiscent of college exams. It favors mathematical approaches to programming, which tends to favor male applicants as stereotypes have shown to discourage women from mathematics. This is on top of notorious industry “crunch” that prefers people without families.

The result is that we get a lot of the same type people working at companies. And the industry wonders why it has a diversity problem. And other companies wonder why they have a hard time hiring programmers when the process comes up with a lot of false negatives for people who would be able to do the work but not do a test.

So, how can we improve the process? We’re game developers, you think we’d be able to do better while accomplishing a lot of other goals such as having better project management and more diversity of employees. But as I said, this is a hard problem to solve, therefore easy answers aren’t abundant.

Trying to do certifications would fail, as game technology tends to move quickly. And, certifications would be even more homogenous than current testing systems.

Getting prospective programmers to do actual work would be tough, as many have non-compete agreements. Any work done would be owned by their current employer, and a programmer would need to quit before doing the trial work. Although contract work might be a good way to try out unemployed applicants, particularly experienced ones. But, again, the game industry is slow to realize the power of the internet and a lot of places are wary of remote contractors.

So, what is the answer? Perhaps there isn’t a good one. Or perhaps the industry is simply too set in its way to want to figure it out.







3 Comments »

  1. Long before we ever heard of “digital amnesia” I used to claim that the only skill I learned at University (where I read English Literature) was how to look things up. Indeed, I felt then, and feel now, that most arts degrees provide precisely that benefit for employers – potential employees who, when faced with a problem, know where and how to do research to find a solution.

    The advent of the world-wide web and always-on access to a global data store has made doing that research a lot faster and easier (sometimes) but I don’t believe it’s been any kind of paradigm change.

    Comment by bhagpuss — 28 August, 2016 @ 12:32 AM

  2. I feel your pain. I’m a systems person rather than a programmer, but I still have to write code on whiteboards sometimes. My writing is terrible, to add to the fun. Your point about the Internet is well-made. I interviewed with a certain Silicon Valley professional social network once and they had Internet access on the laptop used for the test. It was what we’d call a ‘lab’ test, with VMs with different problems I had to troubleshoot and fix in turn. There was a time limit. If I could solve it faster by looking stuff up, that was fine. They also had slightly more original problems than I usually run into in these things. But that’s the only time I can think of where that was allowed.

    Comment by The Alien — 28 August, 2016 @ 9:35 AM

  3. Your experience seems representative of software engineering interviews at most companies, video game studios and otherwise. I can empathize with the feeling that these interviews, and the process in general, feels adversarial. I think the primary motivation for the structure of these interviews is to reduce false positives under the guise that a bad hire is far worse than losing potentially good hires due to the methodology. There is also a concern for expediency: how efficiently can a company determine whether a candidate has the skills to contribute to the team? When you are scaling up a team and fielding hundreds of resumes for a handful of positions, methods that purport to have a high signal-to-noise ratio are desirable even if they lack a certain level of humaneness. The questions often asked in interviews are “academic” in nature and could feel like trivia. However, I think the rationale is less knowledge of a particular problem’s solution and more that the candidate can demonstrate a level of competence to solve equally complex problems with core computer science/mathematical principles.

    Recently I participated in many interviews as we ramped up our engineering team at my current studio. Our process evolved over time and at the moment consists of (for most candidates at least) a technical phone screen, a work test using a small toybox game we built that we ask candidates to add small features to, and an onsite interview, part of which involves live coding. It was an eye opening experience and reinforced to me that hiring software engineers is an unsolved problem. A remarkable number of people at every claimed level of experience failed at basic programming tasks or could not answer basic conceptual questions. It is easy to criticize the software engineering process, but it is hard for me to think of one that is substantially better. An ideal process would respect a candidate’s time – I don’t think it is fair to expect many hours or even days of unpaid work on take home assignments; would lead to a diverse team (in gender/age/life experience/etc); would ensure candidates could meaningfully contribute to the roles they are being hired for; and would evaluate/recognize the totality of what someone does/could bring to a role.

    Don’t get discouraged and I hope you find a great opportunity that utilizes the breadth of your experience soon.

    Comment by Sean Boocock — 29 August, 2016 @ 6:40 PM

Leave a comment

I value your comment and think the discussions are the best part of this blog. However, there's this scourge called comment spam, so I choose to moderate comments rather than giving filthy spammers any advantage.

If this is your first comment, it will be held for moderation and therefore will not show up immediately. I will approve your comment when I can, usually within a day. Comments should eventually be approved if not spam. If your comment doesn't show up and it wasn't spam, send me an email as the spam catchers might have caught it by accident.

Line and paragraph breaks automatic, HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Email Subscription

Get posts by email:


Recent Comments

Categories

Search the Blog

Calendar

October 2017
S M T W T F S
« Jun    
1234567
891011121314
15161718192021
22232425262728
293031  

Meta

Archives

Standard Disclaimer

I speak only for myself, not for any company.

My Book





Information

Around the Internet

Game and Online Developers

Game News Sites

Game Ranters and Discussion

Help for Businesses

Other Fun Stuff

Quiet (aka Dead) Sites

Posts Copyright Brian Green, aka Psychochild. Comments belong to their authors.

Support me and my work on