Why a work sample test will identify the software developers you need to hire

Before it was easy to give a work sample test to anyone who fell into your tech recruitment funnel, the technical hiring process was a minefield. Even before I got into the industry, I remember hearing stories of the frustrations my friends and colleagues faced trying to navigate the technical interview process both as candidates and hiring managers.

Candidates were hit with barrages of irrelevant, time consuming, and dispiriting algorithmic tests and whiteboard interviews which heavily favored recent college grads and gave them no insight into the job. It was equally difficult for recruiters and hiring managers. They had very few effective methods to determine which of their dev candidates would go on to do great work for them. So they used what was available and had to put up with the variability of the results. But there is a better way.

The work sample test is a well-established idea in other fields, revolutionizing the hiring process but is now gaining traction in tech.

Find out more about technical interviews and skill testing in The ultimate guide to the technical interview.

What is a work sample test?

Simply defined, a work sample test is a test that, according to Google’s former Senior Vice President of People Operations Laszlo Bock,

That sounds pretty simple, doesn’t it? It almost sounds like a trial period but work sample tests differ in that they meet all of these criteria from Sockpuppet.org:

• They mirror as closely as possible the actual work a candidate will be called on to perform in their job
• They’re standardized so that every candidate faces the same test
• They generate data and a grade, not a simple pass/fail result
• They are much shorter, usually no more than an hour or two

So if it isn’t a trial, what is it? Ryan Daigle, director of Engineering at Spreedly puts it best:

---

A work sample is an objectively and blindly graded domain-specific development task that the candidate completes on their own time. Image source: LinkedIn

---

In other words, a software development task or a custom software development task. So why put in the extra effort to test your candidates this way?

A work sample test is the best predictor of a developer’s future performance

There has been an extensive amount of research done into the efficacy of different methods of testing candidates. The findings of these studies generally point to work sample tests as being the best predictor of future performance.

Frank L. Schmidt of the University of Iowa and John E. Hunter of Michigan State University did a review of the previous 85 years worth of research on the subject. They found that work sample tests were pretty much neck and neck with general mental aptitude (GMA) tests as the best predictors of future performance with the bonus that they don’t bias against minorities like GMA tests do.

Source: “The Validity and Utility of Selection Methods in Personnel Psychology:Practical and Theoretical Implications of 85 Years of Research Findings”

Not only does bias hurt your recruitment efforts by narrowing your candidate pool but according to Tikhon Jelvis, lead data scientist at Target, it can open you up to lawsuits under the principle of disparate impact. This leaves works sample tests as the clear winner, a finding backed up by Laszlo Bock, who in this BBC interview reveals that they were the best indicators of future performance of any of the test they ran at Google.

But these findings are general so how can they be applied to screening bespoke software developers?

What makes a good work sample test for software developers

For a work sample test to be effective, it needs to include a task which is important and representative of the work that will be done. What this means for a software developer is best summed up by Ryan Daigle from Spreedly,

Work samples should be a reflection of what the company does and how they do it. They are as much about the candidate measuring the company as the company measuring the candidate.

A sample programming task should reflect the kinds of problems that the company solves and the environment that they do it in. This means seeing how comfortable your candidate is in using the libraries and frameworks your team already uses. As Ryan points out, the test isn’t just about assessing the skills of the candidate. It also supports a good candidate experience by giving the candidate a taste of the technologies the company uses and the types of tasks they use them for.

The task should be done in a real-world setting

The U.S. Office of Personnel Management has done extensive research into the efficacy of different methods of determining hiring skills. Their conclusion is that,

Because work samples require applicants to perform tasks identical or highly similar to tasks from the job, great care is taken in trying to mimic the work environment to the greatest extent possible.

In an interview for a software developer, there are two areas where you need to consider this effect. The first is what we mentioned above about the task being the same kind of task the candidate will do if they get hired.

The second is to make the work environment the same. Give the candidate access to all of the resources they would normally use at work. These include libraries and frameworks, as well as outside resources like Stack Overflow, GitHub, and Google, just like your current developers use.

This even extends to the IDE they use. Let them use the one they are most comfortable with. It makes no sense to waste their time on getting to know a tool which they will only use in the interview when they could be doing the task.

You should enforce a time limit

Setting a time limit for the test is important for two reasons. The first is that no developer has an infinite amount of time to perfect their code. At a certain point, they will need to finish their project for it to have any value.

The second is that a time limit helps you define the scope of what you are looking for from the candidate. If you simply give a task and ask the candidate to hand it in when they are done, does that mean they should take a week to make it perfect or should it be their best effort within an hour? Adding a time limit helps remove the bias that some programmers have against work sample tests like jasode on Hacker News. They don’t have time to take a week to do a project so will drop out of your recruitment funnel if the programming test looks too open-ended.

You should score the test according to a rubric

We have spoken elsewhere on this blog about the importance of having an objective rubric to use in interviews. They help eliminate bias by making the evaluation process as objective as possible. The same is true for your work sample test. Decide the criteria you want to have for the test and then stick to them. It will help you get the best people by filtering out the noise that can cloud your judgement.

Just ask Elena Grewal, Airbnb’s head of Data Science. In an interview in the LA Times, she explains the importance of using a rubric when grading a work sample test:

---

We took a close look at this and realized that the people who were grading the exercise had no clear rubric, so we changed this and made it clear what we were looking for, we made the grading consistent, and if a person was successful they were moved to the next round. Image source: LinkedIn

The change led to doubling the number of women hired. Note that they didn’t change their standards. Instead, they introduced a greater degree of objectivity that gave them access to more qualified candidates, the whole point of the test in the first place.

The candidate should be allowed to prep and get detailed feedback soon afterwards

The whole point of this test is for the candidate to do their best at a normal work task. Normally at work, you understand the kinds of tasks the company does and are rarely surprised by new tasks in a completely new environment. You should explain exactly what the test and the task will be about so the candidate knows exactly what to expect.

After the test, you should give the candidate detailed feedback on their performance. Not only does this make for a good candidate experience, it will also improve your employer reputation within in the community of developers.

How different companies build work sample tests for software developers

A number of different companies have taken different approaches to creating their own work sample tests. Here are a few examples you can use.

PolicyStat (since acquired by iContracts) has used this tactic for over 300 candidates for 6 different positions on their tech team. For all of them, they give a specific project which mirrors the work as closely as possible. This means getting more granular than language and actually testing the specific technologies used by the team, in their case Django.

Candidates are asked to submit a pull request that ads a feature and fixes a bug on a Django based project.

Spreedly is a payments tool so their work sample is to actually create an ActiveMerchant gateway adapter to a mock payment gateway. This is exactly the kind of project their devs do all the time so it fits the role perfectly.

You can actually have a look at their sample on GitHub. If you go there, you can see how they lay out exactly what the task is, how to submit it, what they are looking for, and the scope of the project.

Sockpuppet.org/Latacora is a Rails shop. What they do is to take a Rails application they have deployed and then carve out some functional areas of the application. This could be a search feature or a customer order updater. They then ask the candidate to add the feature back.

At DevSkiller, we’ve taken the concept of a work sample test for software developers and built a platform to automate the process for the tester and the candidate. Tests can be conducted on the platform from anywhere in the world at the candidate’s leisure. The test is then automatically evaluated saving the time of a technical interviewer. Recruiters can either choose from one of their many premade tests or build their own with the actual code that the company uses.

A good example of a DevSkiller test is this test for Java developers who write RESTful blog applications. As you can see the task description lays out the technologies that will be helpful (Spring, Spring Data JPA, Hibernate among others) and the task (add a commenting feature).

At the top of the page, there is a helpful countdown clock so that the candidate knows how much time they have.

And there is a build console so that the candidate can run tests during the work sample test.

The idea is to create clear parameters for the project but give the candidate the opportunity to use all of the resources available to them to come up with a solution. Unlike the other solutions, DevSkiller speeds up the process by automatically sending the test to the candidate, receiving the finished test, cloning the task, building it, and reviewing it without the quality of the tests being compromised. As Arthur Brooks of CodeSpaghetti puts it, “[Devskiller] test are very thorough and give a meaningful overview of candidate’s performance.” At the end, the recruiter gets a detailed account of the results so that the candidate can be objectively compared to the other candidates who take the test.

Your next move

Work sample tests are an incredibly accurate way of assessing whether your software development candidate can do the job or not. Whether you build your own test from scratch or use an automated platform, here are some things to remember when building the test.

Do

• Objectively measure the submission
• Make the scope of the task manageable and well defined
• Test the specific programming environment you use with those technologies

Don’t

• Use this as an opportunity to get free work from the candidates*

*There have been some cases in the past (see this question from Quora) where companies, often cash-strapped startups, will use a hiring task as a cheap way to add a feature to their product. Just don’t do it. Your candidates will figure it out and it will put them off your company. Not only that, they will tell their friends, ruining your employer reputation.

How do you assess the coding skills of your candidates? I’m looking forward to your hearing your thoughts in the comments section below!