27 August, 2016
Hiring programmers is a complex process. You need to find someone with the competence to perform the work and the temperament to work well with a team. In game programming, you also have to find people who will do highly technical work while making less than they could in just about any other industry.
But, the industry does itself no favors here. The way we test and judge programmers demonstrates a mindset stuck in the past. Let’s take a look, shall we?
Testing the wrong way
I wrote previously about problems with programming tests. This particular test dealt with a test in person, but a lot of the same sentiment applies.
Since that time, it seems more fashionable to give people tests remotely. That means that a typical game programmer needs to get past the HR screening, the tech screening call, and a programming test before they get to get flown in for a technical interview, probably one where you’re going to be writing programs on a whiteboard.
And this assumes that each step of the way is flawless. I spoke with a game designer at a large MMO company, mentioning how I had a terrible interview experience previously; the person doing the tech screening call was aggressively rude to me. The designer asked a name, saying they wanted to “confirm a suspicion”. After giving the name, the designer said that, yes, that person was indeed the person they were thinking of. So, the possibility of me working at one company was scuttled by someone who is known for being rude to others.
The other problem with many tests is that they don’t mimic working conditions. Not only are they usually adversarial in nature, they often involve programming alone. Most game companies are using some variation of Agile, which means that you don’t have programmers programming in isolation. Many places have code reviews, were your code is evaluated by another programmer before it gets committed to the codebase. So, mistakes are often caught by someone else; the expectation is that nobody is perfect and “given enough eyeballs, all bugs are shallow.” But, this isn’t reflected in the testing. In fact, you get very little feedback short of a generic HR response.
So, the gauntlet of steps to go through the hiring process is not easy. And, keep in mind, someone might be applying to many different positions, meaning they might be juggling many different programming tests.
Testing the wrong things
What do you hire a programmer for? Particularly a Senior or Lead programmer? Often in these positions, raw programming ability is only one part of their responsibilities. Often you’ll want someone who can plan out and architect a system, someone who can mentor other engineers, and someone who can discuss difficult problems with others. A lot of companies could benefit from better leadership, but requiring practical work from your leaders may mean they don’t do the best job of leading. But, how do you test for leadership capability?
Now, I know I’m a “snowflake” with my unique background. Yes, I have a degree in Computer Science, but I’ve been active in design and even lead designer on some projects. So, I’ve been more interested in the technical designer positions more than pure engineering, although technical designers are still pretty rare in the mainstream industry. But, hiring me purely for my technical ability largely misses the point of why you would want me in your company. A friend of mine said that I was the type of person who would solve any problem that gets thrown at me, maybe with a bit of learning in the middle that I would also enjoy. This is accurate as it’s something I’ve had to do with the startups I’ve worked at. But, how do you test for that?
So, giving a programmer a test to see how well he or she can solve a CS 201 problem in C++ can miss the point. The test isn’t going to reveal this aspect or how eager someone is to learn new things even if not given the chance to become a domain master. It doesn’t reflect their experience beyond perhaps seeing and remembering a solution to that problem.
In addition, some tests also assume certain knowledge. One programming test required that you solve a problem heavy in vector math. Not every programmer is an expert in vector math, so you’re selecting pretty heavily for a certain type of programmer, and ignores the linguistic approach to programming which is as valid as a mathematical approach.
One last problem is that there are a ton of little things that the assessor might judge someone on. Does a preference for camelCase instead of underscored_variable_names count against a prospective programming hire? Programmers are known for their holy wars, and you never know when someone is going to take exception to your default style for not being “right”.
The interesting thing here is that most companies ignore experience. Someone might have held senior engineer positions at many companies, but they must still take the programming test that “everyone must take”. As if no other company tests for engineers.
The usual complaint about this is that some programmers might have experience listed on their resume that they don’t actually have ability with. But, assuming that someone held a senior position at another company, it probably means that the person had some qualities that made them fit for that position at that company. But, again, the process seems adversarial. Incompetence is assumed until proven otherwise by a test that may not actually test for competence.
This type of testing also tends to favor younger applicants, those who have more recently CS 201 tests and are comfortable with that type of testing. The game industry has a pretty notorious ageism problem where the assumption is that older people can’t possibly endure the rigors of making a game. And, it stands to reason that this type of programmer testing contributes to that problem.
Ignoring the internet
I read an article lamenting “digital amnesia”, where people think it’s easier to look up some fact rather than commit it to memory. Which makes sense to me; the less time I spend memorizing trivia is more brainpower I can commit to more interesting work. Of course, some people think that memorizing trivia is somehow more virtuous than knowing how to look up that trivia. And, really, the game industry is like that.
One programming test I took admonished against talking with other people or doing “too much” research on the internet. If you have to look up how to solve the problem, it warned, you won’t “survive” the in-person interviews. But, what is the most efficient way to solve a problem you don’t understand? Ask someone or look it up on the internet, the two things explicitly prohibited by the test.
Modern programming has had a strong focus on not reinventing the wheel. Object orientated programming was supposed to make it easier to use work that others have already done. Software libraries and well-defined APIs let you build your work on top of the work of others. Expecting someone to code something from scratch may show you their programming approach, but it’s not an indicator of how well they will do in day-to-day programming that is well beyond CS 201 problems.
Improving the process
The main problem is that these type of tests tend to select for a very narrow type of person. It favors younger people since it uses tests reminiscent of college exams. It favors mathematical approaches to programming, which tends to favor male applicants as stereotypes have shown to discourage women from mathematics. This is on top of notorious industry “crunch” that prefers people without families.
The result is that we get a lot of the same type people working at companies. And the industry wonders why it has a diversity problem. And other companies wonder why they have a hard time hiring programmers when the process comes up with a lot of false negatives for people who would be able to do the work but not do a test.
So, how can we improve the process? We’re game developers, you think we’d be able to do better while accomplishing a lot of other goals such as having better project management and more diversity of employees. But as I said, this is a hard problem to solve, therefore easy answers aren’t abundant.
Trying to do certifications would fail, as game technology tends to move quickly. And, certifications would be even more homogenous than current testing systems.
Getting prospective programmers to do actual work would be tough, as many have non-compete agreements. Any work done would be owned by their current employer, and a programmer would need to quit before doing the trial work. Although contract work might be a good way to try out unemployed applicants, particularly experienced ones. But, again, the game industry is slow to realize the power of the internet and a lot of places are wary of remote contractors.
So, what is the answer? Perhaps there isn’t a good one. Or perhaps the industry is simply too set in its way to want to figure it out.