There Is No Perfect Interview

I’m guessing that everyone reading this has been through at least a few tech interviews. Many people say tech interviews are awful. I’m not going to question that. I think, for the most part, all interviews are awful. It is both easy to decide in 30 - 60 minutes if you feel that someone can do the job, and hard to be sure that you are making that decision for the right reasons. Too many people interview one way just because “that’s how it is done.” The whole process is further complicated by unconscious biases and the fact that people are wonderfully diverse in their beliefs and skill sets. Because of all that, even in the best cases interviews will be an imperfect instrument. That said, all of our current processes are terrible.

“Trivia” Questions and Brain Teasers Are Bad

When I was first looking for a job, I was told to prep for what I call “trivia” questions. Things like what is the big-o runtime of Radix Sort? Or what is the range of a 32 bit signed integer? Also in this category is the famous question, “Why are Manhole Covers Round?” which I was asked at an interview at Microsoft.

These questions don’t give you much information about whether you should hire people. You either are straight out of school, or you spend time memorizing the runtimes of a bunch of common algorithms and read through a book of “puzzle” questions, so you are familiar with the right answers. If we believe that development is mostly problem-solving, working in groups, and applying our tools to novel situations than any question with a single correct answer is not a good screening tool. That means trivia, brain teasers, and most knowledge questions shouldn’t be used in interviews.

Whiteboard Coding Is Bad

Whiteboard coding is probably the biggest dread of new developers looking for jobs. I get it. Many folks are horrible at giving whiteboard interviews. They spend the entire time looking for syntax errors and misspellings instead of looking at the overall architecture and approach you took to the problem. The worst interviewers compare your answer to a canonical answer and use some sort of gut feel hamming distance to figure out how well you did. If you miss a semicolon or reverse the arguments to a function, you are out. Obviously, this type of interview doesn’t resemble the day job of most programmers at all.

Some places do a variant on whiteboard coding where they let you bring your own development environment. This approach is somewhat better than the whiteboard. Letting the candidate use their own tools should make them more successful. But it still isn’t a great screening tool. I don’t spend my days building simple data structures from primitives, and I don’t want to work with folks who reinvent everything. If there’s a suitable data structure or library for the problem domain, I want to work with folks who will use that. In most whiteboard interviews we reward the folks who are good at reinventing the wheel. So I agree that whiteboard coding is pretty flawed.

Soft Questions Aren’t Sufficient

I’ve done a couple of interviews of relatively senior folks. One of the oddities of tech interviews is that the more senior you are, the more soft questions you get. Personally, I don’t think soft questions give a strong signal either. If someone flubs a question like “Tell me about a time that you had to resolve a technical disagreement?” or “What is your biggest weakness?” The only thing I learn is that they weren’t prepared properly. These are very standard interview questions across many industries, and the internet is full of the “correct” way to answer them. If someone gets a soft question correct, all I know is that they studied. I still don’t have any idea how they’ll handle tricky workplace situations.

Also if we are hiring for technical positions most of us want to verify that the candidate can get up to speed in a reasonable about of time and then do the job successfully. To do that we need to have some way of assessing technical skills. Soft skills are important but technical skills are also important. An interview of entirely behavior questions does not help me assess technical skill.

Projects/Homework/Pairing Are Unfair

About eight years ago I started to see an increase in the number of companies doing some sort of “real” programming task as part of their interview as a push back against whiteboard coding. Often the task was given as homework. Candidates were supposed to finish it before their phone screen or on-site interview. People pointed out that interview homework gives preference to folks without significant family or job commitments (so younger workers and often men over women).

To address this some companies have candidates do the homework assignment as part of the on-site interview. I did this in an interview in 2004. While I enjoyed it, I’ll be the first to tell you it wasn’t fair. It was evident during the evaluation phase that the company preferred iterative solutions with unit tests. This seems reasonable, but at the time no university training program taught these skills. I had learned them on internships and from family friends who worked in software. Someone who didn’t have these outside experiences would not have known that frequent check-ins and unit tests were desirable to many employers.

Other companies have candidates come and work on-site for the day, either by themselves or by pairing with a current employee. The idea is that this most closely simulates the actual environment that the candidate will work in if they are hired. This interview method favors folks who are familiar with pairing, and those that can take an entire day to do unpaid labor. Also, the success of this type of interview depends as much on the current employee the candidate pairs with as it does on the candidate. Many folks who will make fine employees struggle with being dropped into the deep end like this.

Portfolio Review Is Unfair

At a recent CoffeeOps, someone brought up the importance of having an online presence for your career. Usually, this is contributions to open source or projects on GitHub, but it could also be blog posts, community involvement, or speaking. The idea is that your entire corpus of work is more representative of your skill than how you do on four problems on a whiteboard. Some of the folks who are champions of portfolio review are open source contributors who’ve struggled with whiteboard coding interviews.

Yet, portfolio review can be unfair too. Many developers cannot share the code from their day jobs. Others have commitments (or employment contracts) that make open source coding in their off hours hard. Because of this, their portfolios are slim. There are some fields, like user interface design, that do use portfolio review as part of the hiring process, but it feels unfair to base developer interviews on portfolio review when so many of us cannot share what we do at our day jobs.

Conclusion

Technical interviews are broken. None of the types of interviews I see in use today are universally fair and reasonable. None of them a suitable way to ensure you are getting the best candidates. Every method favors some folks and disadvantages others. So what do we do instead?

If we want to find the best people we need to adopt a combination approach. We need to use several of the methods above, either as mandatory parts of the process, or as options for the candidate, and then make our hiring decisions based on the parts where the candidate does well. In college, my discrete mathematics professor used this grading system:

Your grade will be calculated as follows:

• 25% Homework
• 25% 1st midterm
• 25% 2nd midterm
• 50% Final

Drop the lowest 25%

My interview process at Google (the time I got hired) used a variety of interview methods, and it felt fair. There were some trivia and whiteboard coding questions. My portfolio or previous talks and open source was reviewed. Someone attended one of my talks in person, and I had to prepare a talk to give to the interviewers as homework. I had several conversations about a wide range of topics with the hiring manager as well. I know I flubbed at least one of the whiteboard coding questions, and I said “I don’t know” to a few of the knowledge questions. But because there were multiple components to the interview it felt like messing up one bit was okay. I knew they were getting a good picture of who I was and I felt like I had several opportunities to show my strengths to the team. If you have influence on the hiring process, your company uses I recommend giving folks several different ways to show their strengths. You’ll have happier candidates, and I believe you’ll find better employees.