Skip to content
bits&pieces by pekala

A Coding Challenge That Sucks Less

Improving every candidate's and reviewer's least favorite interview part.

Published on December 4th, 2020
~17 min read

In my career, I have found myself on both sides of the hiring process many times. Every company's approach is different but when it comes to a software developer role, there was almost always a form of a take-home coding challenge involved.

For both the candidate and the company, they are a necessary evil. It would be hard to find someone claiming they like solving or reviewing coding assignments. However, there are different levels of evil, and making that part of the process suck less should be in everyone's best interest.

Note #1

This piece ended up much longer than I expected. If you would prefer to skip the background starter and go straight to the main course, here is the link to the rules of a good coding challenge.

Note #2

Some companies choose to skip this step in their hiring process, sometimes conditionally (e.g. for folks with substantial open source contributions or for referrals) and sometimes completely. While there are alternatives that allow to do that and still hire confidently, and while I think some of them might have advantages, a take-home assignment remains an industry standard.

In this piece, I will focus on how to improve it, and not on questioning its merit or value. I can recommend this post by Michelle Barker if you want to explore that topic.

Note #3

Most of what I describe in this piece likely applies to any software engineer role. However my experience is heavily skewed towards frontend development. If you plan to apply these learnings to other roles, your mileage may vary.

First impressions

A coding assignment usually comes pretty early in the process, right after the first screening. From the perspective of the candidate, it can set a tone for how they perceive the company they are trying to get hired by. A bad experience, which is not rare, can make it hard to remain excited about the role later on. Come to think of it, so far I've only accepted positions from companies where I at least didn't hate the coding assignment ‒ it was never the deciding factor, but an early indicator for which places to avoid.

For the company, a malformed coding challenge can distort the first impression of the candidate. It can accidentally reward things like an abundance of free time while failing to evaluate traits that actually matter for the job at hand.

On the flip side

Ironically, when I joined these companies and got the chance to be at the other end of the same process that I found enjoyable as a candidate, things looked much different. I've often found reviewing those coding challenges to be a painfully long, demanding, and confusing task.

When I recently got a chance to rethink the hiring process for frontenders at my current company I've decided to take a better look at where that mixed experience comes from. I embarked on a mission to design an assignment that works for both the candidate and the reviewer while still doing a good job at evaluating the match. I started by considering what the incentives are for everyone involved.

The deadlock of incentives

Candidate ‒ outside, looking in

As a candidate you want to get hired ‒ that's the main reason people apply for jobs. If the company asks you to solve a take-home coding challenge, you will probably first roll your eyes, but your best chance to get the job is to solve it as well as you can.

A few observations follow from this. First, knowing what criteria for "well" are is very helpful and welcomed by the candidate. Second, you want the assignment to allow you to showcase what you consider your strongest skills. Depending on your personal situation, optimizing for the least time "wasted" on it can be very important too.

Additionally, from my experience, the challenges I've hated the least were the ones that didn't feel dull, or as if I was asked to do unpaid work for the company, solving a random ticket from their backlog.

I also enjoyed when, as an added benefit, a hiring task was an opportunity to try a library or technique I was planning on learning anyhow, so I could fill two needs with one deed.

And after all is said and done, learning how well you've done and getting feedback about it is much appreciated. There are few things as annoying as submitting an assignment and then never hearing about it again.

Reviewer ‒ inside, looking out

Since the coding assignments are technical, the task of reviewing the submission will usually end up on the desk of a developer. As a reviewer, you want to fairly and accurately evaluate how well the candidate solved the challenge and how their solutions compared to other people's applying for the role. You also want to do so efficiently, you have a whole backlog of things to work on after all. How difficult of a task all that is depends to a large extent on how the assignment is designed.

If every hiring submission that lands in your inbox is a unique snowflake that doesn't follow any predictable structure, you're in for bad times. Unconstrained solution space forces the reviewer to either spend a long time getting familiar with that particular solution or to make the evaluation based on a rough skim of it. In both cases, the company loses, either in the time of the reviewer or the quality of the review and thus the confidence level it can have from the review.

Reviewer's best friends are clear guidelines for what a good and bad submission looks like, as well as a familiar review process. Switching processes, tools, and criteria just to evaluate someone's take-home challenge code can be a frustrating ordeal.

Company ‒ looking inside out

The person or people that represent the interest of the company in the hiring process will vary, but usually it will be an internal hiring manager, engineering manager, CTO, or all of these together. The reviewer can be seen in this role too. "The company" in this piece refers to that group.

What the company wants is to hire the best candidate, the one that is a match for the role and will be the optimal contributor. A daunting task, really.

A coding challenge is a tool that can help with shedding some light on a narrow portion of the full picture needed to make the decision ‒ and an imperfect tool at that. It will help to answer some questions about the candidate, but which questions will depend on how it's designed. For example, it can answer a question: "How well does the candidate know the quirks of JavaScript?". Or "What technologies would the candidate choose to start a new project today?". Answers to these questions do tell something about the candidate, but it's unlikely they would be very helpful in hiring the best candidate for the role.

The question we really want to finetune our coding challenge to answer is "How good of a technical contributor to our projects would the candidate be?". Note that "technical contributor" can mean more than just coding ‒ a good take-home coding assignment can teach the company about more aspects of a candidate's profile than just their ability to write code.

Ideally, the challenge is calibrated and flexible enough that it allows detecting outstanding candidates. Identifying highly desirable people early in the process can help in ultimately hiring them, e.g. by shortening the process later on.

Finally, all this should be accomplished without damaging the perception of the company that the candidates go into the process with. A poorly designed or poorly executed assignment can easily have a net negative value for the company.

Balancing act

As it's clear now, there is a lot at stake here. It's also easy to see where the potentially vastly different experiences for candidates and reviewers come from. Their incentives can seem contradictory, and without careful consideration, it's easy to end up with a test that caters to the needs of one of these audiences, while ignoring those of the others.

Can we optimize for everyone's needs, or do we need to sacrifice the experience or best interest of the candidate, reviewer, or the company? It's not an easy task and might require significant investment to execute well, but I believe it's possible to find a balance that satisfies everyone involved.

The rules of a good challenge

We've finally arrived at the bullet point list of Certified Good Advice, based on my experiences reviewing and solving coding assignments as well as my recent experience designing one myself. Here is how to make your coding test suck less:

1. Replicate the way you work on code day-to-day as close as possible.

2. Don't make it too open...

3. ...but leave space for exploration and creativity

4. Don't make it feel like work

5. Provide explicit and clear instructions and success criteria

6. Always follow up on the challenge

How it worked for me

Trying to stay true to these rules as much as I could, I've designed the new frontend coding challenge for my company.

Using the SpaceX API I've built a simple web app that shows some rocket launches and landing sites. I've used React and SWR, two libraries we're using in our production app. I've opted for a simple setup using CRA, no additional tooling or technologies. The interface is built using Chakra UI to replicate having a design system, which we are in the process of adopting in our codebase.

The challenge has three tasks, all asking the candidate to make changes to that app. The first task is to solve a simple, but not trivial bug involving timezones. The second task is about implementing a new feature, based on a rough sketch and a brief written spec. The last task is open - it asks the candidate to improve the app in any way they'd like, giving a few ideas to let their imagination going. The assignment includes detailed guidelines about each task and practicalities around the submission process. We ask candidates to submit solutions to each task as a PR against a cloned repo, and to keep the quality as if it was production code.

After using it as part of the process for a few weeks, the results have been positive. Here is a sample of comments from candidates and reviewers:

Loved the “bug + feature + anything-cool” approach, it’s way more appropriate than making a small toy app from scratch. It’s quite like what you’d be actually doing and shows how you’d blend your code into an existing codebase.

— Feedback from a colleague who solved the new assignment to get hired

I definitely enjoy reviewing now more than I used to. I get to learn how others approach similar problems in vastly different ways. For grading, I focus on how they deliver, the commits, the description in the PR, and notes they leave in the PR description as well as in code to highlight areas for future improvements.

— Feedback from a colleague who reviews the assignments

Just wanted to say that I really appreciate how you set up the challenge - with existing codebase and bug/feature tasks.

— Candidate submitting the solution for the new challenge

I've mostly sent out open assignments in the past (start from scratch and build something). What works even better is having a take-home assignment where you see how the candidate actually works in an existing codebase. This takes a lot of time to build... But the best assignment I've personally done was the [My Current Company]'s assignment.

— A message shared by a friend who went through the process but eventually opted for a role at a different company

The constructive feedback I've received so far was mostly focused on the implementation of the base project - e.g. library choice or coding style. Good thing these details are the easiest to adjust!

An imperfect tool

All my learnings can be summarized into one key takeaway ‒ when designing the coding challenge guide yourself by empathy first. Taking the time to look at it from the perspective of all the people involved in the process will lead you towards the best results.

And remember, no matter how much you tweak and optimize the coding test it will stay just that ‒ a coding test. There are many inherent limitations to what it can tell you about the candidate. But if used responsibly and designed deliberately, it can be a valuable ‒ and bearable ‒ tool in the developer hiring process.

Brought to you by Maciek . Who dat?

You might also enjoy

A vintage photo of a sinking vessel
/pieces/ Finding Time for Tech Debt Installments
A monograph of Stonehenge columns from 1806
/bits/ Theme-Aware Syntax Highlighting