Adversarial Interviews
In my last post I talked about my thoughts on how effective information security education is preparing the junior people entering the security industry.
Pratik Amin
In this post I want to continue talking about the industry but pivot a bit into talking about interviews. My thoughts here are pretty heavily inspired by the content in this blog post, which talks about the challenges in hiring developers in the era of AI agents and interview support tools.
Gergely Orosz
Developer interviews are a pretty well documented process - the whole concept of things like coding challenges, take home assessments and LeetCode is well understood by anyone in the tech scene. Interviews for security positions are in many ways quite similar.
I think that interviewing for security candidates is a actually a pretty fascinating proposition. Interviews in general are a bit of an adversarial situation. Candidates want to convince you they should be hired and have all the incentive in the world to gain any advantage they can.
If you are hiring for an offensive security position you likely want candidates that are good at finding and exploiting gaps in systems. Being able to look at a set of rules and constraints to identify how they can be abused to gain an advantage is often a direct job responsibility.
At the same time from a business point of view you want to be able to truthfully evaluate the skills of the candidate in front of you. You want someone who could manipulate a process for their own good, but also someone who you trust is not doing that to you right now.
At the end of the day interviews are about trust. Do you trust that the candidate in front of you has the skills they say?
Looking for truths and lies in a Resume
The first entry point when evaluating a given candidate is reviewing their CV. If you have posted any job posting in recent years there are a lot of common patterns you will pick up, some of them are:
- Lots of resumes - Any job posting for an online role, especially for a remote work role will quickly generate hundreds or thousands of applications. A lot of these applicants will be completely unqualified or invalid for various reasons. Of the applicants who could work out, it is a tedious task to filter it down to a handful of candidates to actually go through the process. This process also will end up cutting candidates who might have been great.
- A higher floor for resume quality - An old adage to reviewing resumes was to look for ones that had poor writing, typos or seemed poorly put together. Given how much templated resumes and AI is used to write content these days you likely are going to see few really bad resumes, this again makes it harder to really parse the good from the bad.
- A lot of the same - Close your eyes and picture a resume for a tech job, thats probably what 90% of your applicants resumes will look like. For the same reasons as mentioned above you will see a lot of duplication and sameness across your applicants.
Once you actually cut through the forest to get to a list of resumes that you like you have to actually go through evaluate each one. The truth is that every resume is a carefully constructed lie, it is a window into someones skills that they are trying to sell you on. You need to evaluate it in the same way you would a marketing pitch for a product you see online.
Technical Skills
I'll start with a common section in resumes that is a pet peeve of mines. You have probably seen in a resume a list of skills that an individual has. This section is of course very important, it provides anyone looking at the resume with an overview of what the candidate is claiming to be good at.
In practice, these sections have honestly become a bit of a parody. People put so many things in here that it becomes both completely unrealistic and at the same time pedantic in an effort to bulk up a resume. The following are a handful of generalized examples I've seen on resumes for junior resources:
Skills: Kali Linux, Linux, Ubuntu, Ubuntu Server, Fedora, Red Hat
You see this all the time and I hate to break it to you but these are all the same operating system. Distro heads, please spare my blasphemy but from a skillset point of view these flavours of Linux are all basically the same. Documenting that you can use both apt and dnf is not worthy of top-line billing on your resume. Kali Linux is also tossed out so often you would think its the secret codeword that would get you into the cool kids club. Using Kali does not make you a better security professional anymore than buying a box set of power drill bits make you a better handyman.
Being comfortable with Linux is a great skill to have but listing out a bunch of distros and thinking it makes you look more competent discloses more about your lack of understanding than it helps.
Skills: TCP/IP, UDP, 802.11x, <Insert other protocol here>
Understanding protocols is a tough game. I've seen people reference their expertise with TCP/IP on resumes hundreds of times but I would guess that a vast majority would be unable to describe a TCP packet on even a high level (hell, I can't even do that without looking it up).
Whenever I see people reference some protocol that they maybe used once, or more likely read about once as a skill they posses it really brings me to question if they have any idea of what they are committing to. If you asked someone to write a client that communicated to a server using a protocol they are an "expert" in would they be able to do that in a day, a week even? I'm not saying you need to be able to deeply understand how niche protocols work but lets be real here.
Skills: MITRE ATT&CK Framework, Red Teaming, Assumed Breach, Phising
You would be shocked to see how many people who have just finished a short school program and have no practical work experience will proudly state the fact that they are deeply comfortable with performing a full on red-team or being able to implement something like the MITRE ATT&CK framework on their resume.
If you know anything at all about these topics you would realize how laughably that is. In fact if you talk to people in the industry they would lament on how difficult these things are to be good at and stay good at.
Look, I get it - its a tough world out there and you gotta sell yourself as best as you can. It is a pretty common situation to see people in interviews struggle to talk about stuff they have written, verbatim in their own resume. It does not come across well. These lists have gone from "here are the topics I am good at" to "here is a list of words I am aware of".
A candidate who has a shorter list of skills but speaks clearly and with confidence on those topics comes across to frankly a indescribably degree better.
Certification
A common practice I have started to see is people just plopping certifications on their resume regardless of if they have done it or not. Its actually quite hard to even validate most of them - especially if its a random online certificate.
The OSCP is probably the most common certification in our industry now. In order to pass it you need to perform a proctored exam which requires you to exploit a bunch of systems. Someone who passed this certificate should be able to speak to at least some of the types of techniques that they used or their general approach.
I have personally been in many, many interviews where a candidate who claims to have an OSCP struggles through basic infrastructure questions. When prodded about their certification many people will switch their story to saying that the OSCP is "in-progress".
The OSCP now contains a QR code which can be used to verify the validity of a given certification. Other certifications like CEH also have a verification process. The reality is that most companies don't verify to this level and while its tough to cheat a proctored exam its a lot easier to just straight up lie about it.
Past Work Experience
If you can't really rely on what skills a individual self-reports on, or what certifications they may posses one of the main thing left is their previous work experience.
This is often the most critical part. Even more than what they have learned at school someones previous work experience will have molded them into the person they are now. It's hard to just straight up lie about past work history because we have a system to verify this that hiring professionals are good at.
People do tend to exaggerate what they did in a previous role. It is pretty common advice to try and fluff up what your role responsibilities were but this can end up back firing to someone who knows what to look for.
For example, take a look at this advice from a website named "Resume Worded", this advice is for a cyber security professional resume.
At first glance you would probably say that the text in the green box is better - and it does help that there is a bit more detail. As a professional who has worked in this industry for a while a lot of the stuff in the second box is an eye-rolling claim.
- As a pentester, outlining how many tests you have performed is actually a very valuable metric to share. That is a very countable stat that shows your experience.
- Counting vulnerabilities on the other hand is a pretty precarious thing. It is not about getting a high-score, its about evaluating the posture of a target. If that ends up being 5 vulnerabilities, 10 or even zero it does not really matter as long as coverage was reached. People focused on the total amount of issues they find tend to focus on reporting garbage.
- Suggesting that vulnerabilities you found led to a reduction in security events is complete nonsense. There is no clear correlation between those two stats and to be blunt the candidate probably has no idea what the actual number of security incidents that occurred even are.
- Similarly, it is impossible for an individual to say that their findings or controls saved the company a specific amount of money. When I read things like this I know the person just made it up.
Recently I have seen a huge bump in the amount of numbers that are included into resumes. People will list out explicit numbers of how many findings they find, how much money is saved and all sorts of other data. I would actually argue that for the vast majority of organizations out there this is not easily identifiable data so an individual making specific numerical claims about what they did at multiple jobs is almost certainly pulling those numbers out of thin air.
What do I tend to look for?
In the modern world I feel like listing out things that you can look for authenticity in a resume just leads to that being unethically optimized next.
That being said what I tend to gravitate towards are candidates that seem passionate about -something-. I don't really care what that is but if someone shows that cared enough about something to really learn about it, to understand it as much as possible than that is probably a more of a positive sign than anything else.
Technical Interviews - A modern day Voight-Kampff Test
Like I mentioned earlier a inspiration for this post came from an article talking about the challenges of doing interviews in the area of AI assistants and interview helpers. To be honest, its a pretty frustrating problem to deal with when hiring remotely.
There are many tools out there, maybe the most popular at the moment is https://www.interviewcoder.co/
This tool both directly helps you cheat during interviews by passing text directly to an LLM but it also works around screen-share detection. As a security professional, I gotta say its some cool tech - as someone who hires and interviews people I am about to tell noah to get the boats.
Doing a remote interview now becomes this paranoid exercise where you are tracking a candidates eyes, listening to how they respond or just generally trying to see if you can detect anything untoward during their responses. It sucks.
How can you prevent this or catch it if its happening to you? Honestly I don't know. At least right now, you can kind of tell when someone is out of their depth. Security, thankfully has a lot of situational questions that allow you dive deeper into how something works. A candidate who is literally just parroting an LLM would tend to struggle to provide consistent responses.
Huge swings in someones ability to respond to a question are also red flags. If you ask someone to describe a vulnerability and they provide a clear, crisp response that explains it perfectly but then have no idea how to apply that information its clear they don't actually understand.
The easiest answer, of course is to do in person interviews. All we would need to do then is check secret ear implants ;)
Take-Home Assessments
Its common for tech interviews to incorporate some element of a take home test. For developer interviews I think the results are a bit easier to push through an AI agent since there has been a lot of work put into "programmer" style AI products.
A practical assessment which requires a candidate to perform a penetration test, using external tools like Burp Suite is at a state where it can't be fully automated out (depending on the target). Realistically even with this there is a lot of open, fuzziness in a take-home test.
- Did someone else perform the test?
- Did someone share the details of a test with the candidate before hand?
One specific annoyance from the increase in AI usage is that its hard to evaluate anyones writing anymore. In the past when reviewing the words that a candidate wrote you can get an idea of how good their writing is, how clearly they understand the issues they are reporting on and you can also check to see if they plagiarized the work.
These days? Well if you just ask a model to write it out for you thats pretty much the end of it.
Evaluating the person, at the end of the day.
So yeah, its a tough moment right now for everyone. As a candidate you likely need to do things like exaggerate to get your resume in front of someone. Most (but not all!) people are not acting maliciously. A lot of the people I interview are genuinely very smart and very eager people, they want the opportunity to grow and learn in the industry.
The sad part of all of this is that the people who really try to cheat the system (lie on their resume, use AI cheat software during interviews and tests) end up hurting the entire process. The lie can only last for so long, eventually it comes out (several examples of this are in the pragmatic engineer blog post). When new hires end up being unusably bad the impacts last for a long time.
The optimistic view is that after moving on from a hire like this a company would just go back to the job market maybe with some stricter controls in place. In my opinion, many companies are likely to swing the other way and stop hiring. Hiring (and firing) people is an expensive process both in terms of money and time.
As an interviewer, what can you really do? I find that personally, sometimes you gotta just make a call based on the vibes you get from a candidate. I often finish an interview thinking that the person I just spent an hour with is someone I actually DO want to work with. That often can make all the difference.