And somewhere in the back of his mind was a small, small note of confusion, a sense of something wrong about that story; and it should have been a part of Harry’s art to notice that tiny note, but he was distracted. For it is a sad rule that whenever you are most in need of your art as a rationalist, that is when you are most likely to forget it. –Harry Potter and the Methods of Rationality, Chapter 3
Noticing You’re Confused is a technique from the rationalist community where you train yourself to recognize even a slight feeling of confusion as a valuable signal that something about your model of the world is wrong. It’s similar to what people sometimes mean by ‘trust your gut’, but where ‘trust your gut’ is usually an end to the conversation, noticing confusion treats the feeling as the starting point for understanding what’s happening.
I want to tell you a short story about the value of Noticing You’re Confused.
When I was running my company, we searched for a VP of Engineering and it was taking longer than expected, the demand for qualified engineering managers being far, far, greater than the supply.
One day I had a call with a very promising candidate. Let’s call him Sam. In his last two roles, Sam had managed seventy and thirty person engineering teams at well known, respected companies. Our conversation went well and I invited him in to interview.
In my nine years at ZeroCater I probably interviewed around three thousand people. To this day, Sam stands out as one of the best interviews I ever had. Every question was answered thoroughly and thoughtfully, with no unnecessary digressions. I distinctly remember feeling like there were bullet points forming in the air as he talked.
I had the rest of the executive team and several engineers interview Sam. I remember asking one of the engineers how it’d gone after the interview. “Yeah, really impressive guy. You almost expect people to stumble at least a little, but he just knocked down my questions one by one.” The rest of the team unanimously loved him. The next stage was reference checks.
There are two kinds of reference checks: front door (those provided by the candidate), and back channel (those you find yourself). I started with the front door references.
They were good. All of them emphasized his reliability and how little management he needed. The most notable was a VP he’d reported to whose LinkedIn profile showed that she’d gone on to executive roles at multiple Fortune 500 companies. She told me he was one of the best people she’d ever managed and would certainly hire him again given the chance.
Unfortunately, the only backchannel references I’d been able to find were extremely weak. One was a friend of a coworker’s husband who’d never worked with Sam and only knew him casually. The other was the founder of a company I knew where Sam had mentioned having several friends. I reached out to him asking if he knew Sam. Apparently Sam had applied for a Director of Engineering job there a year ago. He hadn’t interviewed him personally and the only note in their system was ‘not a culture fit’, which he admitted was pretty specific to them.
My team was getting impatient. One of our VPs forwarded me a ‘strong yes’ email an engineer had written with the following memes attached.
People wanted to get this done, but I felt a small note of wrongness. It was at this point that I actually said out loud to myself “I notice I’m confused.”
Confusion feels like unresolved surprise. It’s a violation of your expectations that happened somewhere along the way that was never explained. It’s a feeling of things not quite fitting together.
I sat down and tried to trace the source of my confusion. Sam didn’t have a LinkedIn profile, and when I asked him why he said he turned it off because he got too much recruiter spam, which was entirely believable. (1) He said he’d turn it back on and send me the link, but hadn’t followed up. It wasn’t much, but it was certainly a bit odd given how reliable he seemed in general. It was enough to reaffirm my desire to find a good back channel reference.
I racked my brain trying to think of people I knew at one of Sam’s previous employers. Suddenly I remembered a company happy hour from two years back. A friend had brought the CTO of the same company where Sam had managed seventy engineers, and we’d had a couple drinks together. I sent him a message on Facebook reintroducing myself, mentioned we were considering their former Director of Engineering, Sam ******, and asked if he had time for a quick call. He called me immediately.
“I’m so sorry, this isn’t the first time this has happened. He wasn’t a Director and he didn’t manage an engineering team of seventy; he managed a quality assurance team of three.”
“What do you mean? I talked to your VP who said he was amazing.”
“Who?”
“Here’s her LinkedIn profile.”
“I’ve never heard of this person and they never worked for us.”
“Then who did I talk to?”
“Probably his wife.”
I was stunned. Not only had he lied about his experience, he’d set up fake identities complete with LinkedIn profiles with hundreds of connections, then gotten people who were complicit in his lie to pretend to be those people on the phone. (2)
Sam was a con man, and in all likelihood, a sociopath. I don’t even want to think about the kind of damage someone like him could have caused in a position of power. The only thing that saved me was noticing that small, small note of confusion.
(You can learn more about practical rationality on the LessWrong Sequences, or if you want the ebook version, Rationality: From AI to Zombies. If that seems boring, Harry Potter and the Methods of Rationality is a fun read and has a lot of the same concepts in fiction form. I also recommend the four day workshops hosted by The Center for Applied Rationality.)
Notes
(1) Some people have asked why Sam would bother setting up multiple fake LinkedIn profiles and not have one himself. My guess is it’s because that LinkedIn would have suggested him as a contact to his former coworkers who would have seen that he was claiming to have done a job he never did. Also, if he happened to keep it up, I would have personally been tempted to contact whoever hired him next.
(2) People have asked me if this means that interviews are worthless. To quote legendary intel CEO Andy Grove:
We know how hard it is to assess the actual past performance of our own subordinates even though we spent much time working closely with them. Here we sit somebody down and try to find out in an hour how well he is likely to perform in an entirely new environment. If performance appraisal is difficult, interviewing is just about impossible. The fact is, we managers have no choice but to perform the interview, no matter how hard it is. But we must realize that the risks of failure are high.
A test that gives false positives 20% of the time is still a useful test. You just need to combine it with other tests, like references. I will admit that since this experience I’ve often thought that if I had to choose I’d rather have a really solid back channel reference from someone I knew was good who’d worked closely with the person than an interview.