What I Look for in Interviews

This post is definitely incomplete, highly controversial, and wrong. It comes under the banner of “stuff I keep rough notes on, that might as well be public.”

This post started off SRE-specific, but I waffled so much I’ve split the more abstract stuff out. It still uses SRE as the example role, and platform stuff as the example skills.

What is the Role?

Sit back and think about what you need now to keep the lights on, and what you want in the future to make your company better. Then write a Role Spec. Ideally, write a Career Framework, ie define what SRE I, SRE II, SRE III, and Senior SRE look like to you. Each of those levels essentially contains a Role Spec, against which you ask questions.

Levelling candidates is a big issue in itself, but simply put, either:

• you’re growing a bit, and have a specific need for an SRE III - publish that Role Spec, tell everyone up front that’s what they’ll be tested against, and tell them the damn salary range for it.
• you’re growing a lot, and are hiring at all levels. Publish all the Role Specs, and let candidates choose which one they wanna test at. Or, more likely, they’ll come to you via a recruiter with a salary expectation attached. See which of your bands this lands in, and send them that spec. Don’t feel bad about this; be open and honest - “hey, if you want $50k, we’re gonna need to see you test out at SRE II, here’s the definition of that”

What to Ask

Breadth

Ask questions across the whole range of skills they’re meant to have. If they’re going for SRE, or Sales Engineer, don’t assume that just because they can use Kubernetes, or make a pitch deck, that they can code well. If they want to be a tech lead, check they can chair a meeting and give difficult feedback. Your career framework / role spec should spell them out, and you should make sure you touch on each one.

Seniority aka Depth

Your career framework come in here again. When you’re writing it, consider Bloom’s Taxonomy of Learning. A junior should know stuff (or have the framework put in place by their education to learn effectively), but may not think of it automatically. A junior person might PR an n^2 algorithm. In your PR review, you can comment “hey, this is called a lot, its performance characteristics don’t seem appropriate,” and they’ll know what you mean and go fix it (Knowledge). A Senior won’t PR it like that, because they’ve seen it all before and a little bell rings in their head when they see code like that, even if it comes from their own fingers (Application). A Principal can Evaluate - choose the least-bad tech in ambiguous circumstances - and Synthesise - invent new stuff and use things in ways that weren’t envisaged.

A great engineer once told me that the definition of Senior is working under ambiguity. Mid-career people should be able to talk about what wasn’t perfect about a project, and Seniors should be able to tell you why that was still the best possible timeline - what tradeoffs (functional, technical, and political) were made and why was that the best choice given the circumstances?

If they’re academically-minded, can they talk about Cynefin and Satisficing?

I’d add to that - to be able to really grab a system by the neck and not be afraid of it - to be able to debug it, which means both observing and controlling it - you need to understand one layer of abstraction below what you normally interface with. Java programmer? Understand memory management and page tables. C programmer? Understand assembler and CPU pipelines.

Layers

I should have drawn a picture.

• Breadth says they know a wide range of stuff
• Depth says they know relevant things very well

I would add a third dimension, which I’m calling layers.

The more senior someone is (claiming to be), the more depth you should check for - it helps them find nastier bugs, it helps them better predict the behaviour of systems and therefore design them, and so on. But when you’re interviewing for a Senior position, don’t just test the “hard” topics ; those in the Senior SRE box. Check they (still) know the basics too.

People can come from all kinds of backgrounds. Maybe five years in management kept them involved in high-level discussions about data pipeline design, but they haven’t written a line of code for ages. Maybe they’re trying to move sideways in their career and have done a lot of book learning. Maybe they just crammed for the interview, only reading about the topics they thought you’d ask about.

Depth means they can answer questions on a topic in the Senior SRE box for 15 mins rather than 5. Layers means they can answer questions on topics in the SRE I box as well.

The more senior stuff is always going to be more fluffy and subjective, so your tests will be more noisy. Also, you expect a Senior SRE to be able to go away for three months and build a CI/CD system solo. But you can’t practically test that in an interview, so you talk, and they talk, and maybe they talk very well. But you can check if they know how to see the CPU usage on a linux box from the CLI (seriously). Check if they know how to find what process is bound to particular port. What’s disk sleep? What’s a zombie process? What does git rebase do to the graph, and when should we use it?

So take that career framework. Of course test at the level they’re applying at. But then test skills from every level below that. They need these bushcraft skills, and you can’t infer them based on higher-level performance. Don’t be embarrassed to ask this stuff, and reject them if they take offence.

Hitting the Ground Running

We’d ideally hire very smart, engaged, talented people, and they’d just work everything out. I saw that done once - everyone was from elite colleges and had massive IQs - it turned into intellectual Lord of the Flies. You need to test those soft skills, and you need to hire people who can deliver value for the business now, not just in some theoretical future when you re-write everything in prolog.

Don’t get me wrong; I’m definitely in the camp of “a programmer is a programmer”. If I talked to a painter who said he could only use brown; green wasn’t his thing, I’d question if he had a clue, or if he only worked well in rooms that are already brown. On the other hand, I’m in my 30s with two decades of experience in imperative languages. Can I pick Python or Ruby up quickly? I’d say so. But will I be productive on day 1, month 1, or even year 1, in a mind-bending paradigm shift like Haskell? Probably not, not at the level (and hence salary) of my Go and Rust anyway.

Why talk about all this? You need to work out how much this person needs to be able to do it now. This is outside of the role spec. A role spec is the ideal candidate. Most people are coming from a place adjacent to it (some of the skills but not all), or below it (essentially this would be a promotion for them, and there will be some element of growing into the role and level on your time).

Are you in growth mode, with more people than urgent work, the senior people and time to mentor? Or are your backs against the wall, needing relief right now?

Consider this: you had two candidates, both meeting the bar. The less good one was made redundant and can start tomorrow, the better one wants to take a career break and can start in six months. Which do you chose? Taking on someone who’ll “grow into it” it like choosing the six-month option, expect you’re paying them in the meantime. It might be the right choice, but be mindful.

So, why all the stuff about “a programmer is a programmer”? Don’t be afraid to test very specific things. Some people get offended by this - their ego is insulted, or they think you have them pegged at a low level. But if your k8s cluster is on fire, you need someone who’s in practice with hand-on k8s skills and knowledge. No amount of waffle about eventual consistency will get them typing the right kubectl commands at 3am. K8s is learnable, but it takes 6 months. Check they can do it already, and that they still remember how.

How to Ask it

Hypotheticals are bunk. For soft-skills this is really obvious - don’t ask “what would you do if a co-worker was rude”, ask about a time it actually happened. What they did, that effect that had on them and others, if it even worked. Better to fail, learn, and be honest about it, than to paint a picture of the perfect employee.

Same for technical stuff. “Tell me about a time you designed a system that had to be highly-available. Sketch what you came up with. What were the tradeoffs? How well did it work? How did you evaluate that? What would you change given what you know now?”

This kind of question is also great for checking they actually did it - that they weren’t just standing on the pavement looking down the manhole. Ask “what was your direct contribution to this project? How did that add value?” Push on the details - if they really did it, they should know the why, not just the what.

That said, it’s ok to ask them to design a new system, or write a new piece of code. I guess that’s not so hypothetical - you’re asking them to design, not talk about how they’d design. But the devil is in the detail - a basic sharded design is easy to draw, but what about rebalancing the shards? What if that saturates the network? My point is that evaluating a whiteboard design is hard.

Either way, I much prefer practical, hands-on, realistic problems. Coding on a whiteboard is better than nothing, coding on a laptop is better. Checking they get complexity by asking them to emit logs in a format that can be searched and filtered later is great. Opening with “so you have an infinite chess board” isn’t.

I’m not saying Google doesn’t know what it’s doing, or that they haven’t quantified and optimised their interview process to the nth degree. But do you need those skills? Do you get a pipeline of candidates who do well on questions like that, or do you just have to drop your bar so far that the test is meaningless?

Nerdsniping

aka asking questions about minutae. This gets a bad rep, but I’m in favour of it. I wouldn’t rely soley on it, and don’t let the interview become a pissing contest between your engineers and the candidate. However, if they really are using tool X every day, and they’re paying attention, they should know it backwards. This is a good way to weed out lifestylers and people who’ve watched-but-not-done.

• What’s the iteration order or Maps in Go?
• What are the four namespaces in C?

You’re testing if they read and grok errors, or if they just mash the keyboard until it builds.

Open-Ended Questions

These also get a bad rep, but again I like them in small doses. A good answer will show breadth and depth. They show that they understand a level below, which means they’ve either done some real debugging sessions, or they’re inquisitive.

I asked one candidate “tell me as much as you can about what happens when you type ls and press return.” They said “the files in the directory are listed.” Contrast that with another person who talked about the shell as a REPL, lexing, parsing, builtins, fork and exec, the loader, the ELF format, syscalls and libc, dirents and inodes, and so on.

Diversity and Inclusion

I said this blog post is wrong, and I stand by that. I’ve advocated for some stuff that can have a negative impact on D&I if you’re not careful. This isn’t my subject, but briefly:

• Role specs have been shown to be bad. Studies have shown that men will apply if meet one bullet; they’ll just try to blag the rest. Women won’t apply for a job unless they meet all the criteria, putting them at a big disadvantage.
• Beware asking about their hobby projects, or setting take-home tests. Free time and good health are a privilege. If someone’s a carer, or a single parent, or working two jobs to get out of a bad situation, they can’t put the same time and energy into this kind of thing. It’s a poor signal for their ability to succeed in full-time, paid employment.
• Don’t be ageist, in either direction. Especially, don’t think that if someone hasn’t hit X milestone (title, salary) by Y age that they’ve “failed” - that they’re inherently less able. Not everyone is trying, or able, to optimise their career at all times. Maybe they followed their partner to a place without a good tech scene. Maybe they got unlucky with a couple of roles that didn’t mentor them. Maybe they spent their 20s partying, doing the minimum at work, but now have a family and are doubling down.