How Rick Holland's Diverse Experience Helps Him Find Security Talent in Unique Places

Tod Beardsley:

Mm-hmm (affirmative).

Jen Ellis:

Our guest today has been a friend of ours for a long time. I think as long as I've been in security, I've known Rick. Rick Holland, he'll be joining us, who is the CISO of Digital Shadows, and has had a long and illustrious career in cybersecurity. Hi, Rick.

Rick Holland:

Hi. What's up, Jen, Tod; it's good to be with you guys today.

Tod Beardsley:

Hello.

Jen Ellis:

One of the things I love about you Rick, you've got this like nice...I'm British, so I love the drawl in your accent, and the fact that you're always very calm in how you talk, whereas I'm so manic.

Rick Holland:

It's hard to think that I have a drawl, though I am from Texas. But when I was younger, growing up, had a "how y'all doing" kind of drawl. Somehow, maybe in the military and living abroad got rid of that. I don't know, but yeah, it used to be really, really thick.

Tod Beardsley:

I'll tell you what, I've been...so first off I say phrases, like, "I'll tell you what." I've been living in Texas for 20 years. I have caught myself using, "I'll tell you what," and "y'all," not ironically, probably over the last-

Rick Holland:

I do "y'all" all the time. What about "might could"? You ever say "might could"? That's one I'm embarrassed about. I'm like, "Yeah, I might could do that."

Jen Ellis:

I don't think I've heard that one before, but I have a soft spot for the "y'all." I do enjoy it.

Rick Holland:

Anyway, that's local idioms cast.

Jen Ellis:

I do feel like I'm on with the Texas massive today, which I really enjoy. So you mentioned you've lived abroad, where did you-

Rick Holland:

I was in the Army. The first time I left the country, I lived in Kuwait for six months. I was 19 at the time. That was quite an experience. I was also stationed near Cambridge, so, close to you Jen, for two years when I was in the Army as well. Spent two years in the UK, never came home once, spent a lot of my time traveling across the UK and Ireland, and also in mainland Europe. Then, at Digital Shadows, our company started in London and pre-COVID, over the past four years, I've probably done 16, 18 trips to the UK. So, I spent a lot of time abroad professionally, in the military and cybersecurity, and then international travel. My wife and I did our honeymoon in Australia; we went to South America. Then we had kids and then that all just ended. No more travel.

Tod Beardsley:

No more fun for anybody.

Rick Holland:

Not at all. But yeah, I think travel is actually a great thing for people to do, because it exposes you to other cultures. Lets you think about things, not in a typical American frame of reference, or whatever country you're from. I just think it's a great way to open your eyes to other ways of thinking and other ways of living.

Jen Ellis:

I could not agree more. Plus one on everything you said. And I am obviously a little biased on hearing you talk about your experience in the UK, and I'm hoping that it means that post-COVID you'll come and visit.

Rick Holland:

I miss it. I haven't been to the UK since February, and I really want to get a good curry.

Jen Ellis:

Okay. So yes, we'll book that in. Before we get into talking about your role at Digital Shadows, which I really want to do, I do want to ask you a very important question, very serious, important question. Last episode, Tod was telling me that there is a "Hackers 2" movie in discussion. So my question to you-

Tod Beardsley:

Pre-pre-pre-production.

Jen Ellis:

Right. Tod gave me his rundown of what he would like to see happen in this movie. I would like to know: What would your perfect "Hackers" sequel involve?

Rick Holland:

Oh wow. First of all, I got excited this year; I didn't realize that this was in talks for being made. Because I think "Hackers" is one of the classic ones. I also like "Sneakers" quite a bit.

Jen Ellis:

Right? Yes. Who doesn't? So good.

Rick Holland:

Oh man, I don't know. Then you have the "Die Hard" movies that were so—"Die Hard" whatever it was, "Die Hard 18"—so over the top.

Jen Ellis:

Come on, the one where he is somehow magically able to withstand radiation poisoning, it's amazing. Why would people not think that's-

Rick Holland:

I don't know, it's interesting. That's a really good question. I think definitely, it's got to be a cyber Pearl Harbor. We've been hearing about cyber Pearl Harbor since Leon Panetta said it, was it the late '90s? I can't remember the timeline on that. Whatever it is, it's definitely got to be a cyber Pearl Harbor, because you got to up the stakes. You can't just have a tiny little individual, regional outage or something like that; it's got to be a global type of event on the cyber's level though.

Jen Ellis:

I like this. Okay. So, cyber Pearl Harbor, that's your-

Tod Beardsley:

I think we have the subtitle now: "Hackers 2: Cyber Pearl Harbor."

Jen Ellis:

I feel like, Tod, we could ask this question; get people to add one idea each time and just build up a total vision.

Rick Holland:

Then you have the screenplay done, just like that.

Tod Beardsley:

Yeah, exactly.

Jen Ellis:

Crowd-sourced. Would you have it be the same characters, or would you have whole new characters?

Rick Holland:

Probably a mix. What about children of characters? Let's take the Cobra Kai model. You have those that we know and love; I don't know that I really liked Daniel's son very much, but have a mix. So, you have the classic characters that we all want to see, and then you have the new, younger children that are more up to date on all the cool young, whatever the kids are hacking these days at that age. So yeah, I think you have a mix. Because also, if you're thinking about you want to make money, you got to speak to both audiences.

Jen Ellis:

I like this. I like where this is going. Okay. I hope that you made notes on all of that.

Tod Beardsley:

Absolutely, yep.

Jen Ellis:

We can take this forward. All right. Thank you. That was the most important question done with. Back to the other stuff. So, tell us about your role at Digital Shadows.

Rick Holland:

My role at Digital Shadows has evolved over the years. I joined from Forrester, where Digital Shadows was a client, like Rapid7 as well.

Jen Ellis:

For people who don't know what Forrester is, just a quick-

Rick Holland:

Yeah. So, Forrester Research and Gartner IDC are industry analyst firms. As a recovering analyst, I'm trying to think: How can I make fun of industry analysts? They write lots of charts that have vendor placement, like consumer reports that you're...you want to be up and to the right. Also, a book by a former Gartner analyst, Richard, but really, the industry analyst role is probably twofold. One, you have a certain coverage area. For me, I had incident response, vulnerability management, threat intelligence; also had email and web security, and you're writing about the space. So, a CISO like me and their team, they could get a report and say, "Okay, here's the landscape. Here's some vendors to look at." It's like the consumer reports or wire cutter type of angle.

Rick Holland:

Then you're also working with the vendors that are out there and giving guidance to them. Then you're also writing just research in general about the space. My colleague, John Kindervag, who you may not know his name, but if I say Zero Trust, everyone will have heard of Zero Trust. That was something that Forrester wrote about. So, I did that for four and a half years. It was a really good experience. Now I have to deal with analysts. That's not always a good experience. I have a lot more empathy towards the vendors out there, now that I've actually been on the vendor side.

Jen Ellis:

Yeah. It's a shift. Right?

Rick Holland:

Is shift what you said, or did you say something else? Sorry.

Jen Ellis:

I don't know what you mean. So, you were at Forrester, then you moved on to Digital Shadows, where you were apparently having to now be on the other side of the relationship.

Rick Holland:

In Digital Shadows, we look at the outside, outside in. What are the external threats to a company? What are the adversaries that are selling access to your environment? We're looking at extortionists, we're looking at people that are leaking data on GitHub and GitLab and Pastebin, that sort of stuff. When I started, I was an individual contributor, and I think it's pretty germane to where the conversation's going to go. I had no direct reports, and I had not had direct reports for two years at Digital Shadows. I had not had direct reports for four and a half years at Forrester, and before that, I was an SE without direct reports. Then I became CISO, and suddenly I had a team, and I had to start building a team and it started growing from there. So yeah, I've always been, and you know my style, very self-deprecating. It's how I try to disarm audiences and even make it a little bit more comfortable for me. I've got imposter syndrome as well.

Jen Ellis:

Which is the disease du jour in security.

Rick Holland:

Yeah. I try to joke around about it and try to make myself feel a little bit better through humor on it. But yeah, suddenly I was a security leader and I had not managed people for 10 years; and even at that scale, it was a smaller scale. Now, I have about 33, 34 people across my multiple teams, so it's been an interesting journey on: How do you manage people? How do you manage yourself when you're trying to enable people? I've enjoyed it. And at times it's maybe been less enjoyable, because I've struggled with it.

Jen Ellis:

As you have approached managing, we talked about this. You said that thinking about how to create really great career experiences and think about how to develop your people; this is an initiative that you've taken really seriously, and it's a really big component of your security program, and also being open to people who come from non-traditional paths into security and embracing that diversity of thinking that you get. So, can you tell us a little bit about that?

Rick Holland:

I think a good way...I did not get accepted to this talk at RSA; which I'm sure many people that have submitted to RSA have had that experience. But I did a submission several years ago, it involved the rock, and it was about becoming a self-fulfilling prophecy on the cybersecurity job shortage. You see these headlines every single day and I find it annoying, because I think at a macro level we say this and at a micro level, we can change it. But, I think a lot of organizations or leaders, they buy into the hype of what they read on whatever the journalist outlet is. So, I think the first thing is: Let's not be a self-fulfilling prophecy with this shortage of people that's out there, and how do you do that?

Rick Holland:

I've tried to put it in a framework for recruiting, for retention, for career development, to try to minimize that. One very specific example on the non-traditional route of recruiting, and if my colleague Charles hears this, he'll know where I'm going, but we have a guy on our security team, his name is Charles. He was an EMT, paramedic background. So, he was 10 years in basically, healthcare and emergency medicine. My joke with him, he and I presented a piece at ISCW last year, almost a year ago, and when I introduced him, I said, "Hey, this is someone that did a different type of incident response. By the way, the type of incident response Charles did was quite different than ours. If we make a mistake, unless it started a cyber Pearl Harbor, nobody dies."

Rick Holland:

If Charles made a mistake when he was treating a patient and stuff like that, it could be quite catastrophic. So, if you think of the mindset that he has for doing incident response, talk about being able to work under pressure. That is a great field to recruit from. So, that's just one example of ways that you can find people that can be well-suited. And Charles was a very good self-starter, he was building out labs in AWS when he was interviewing and he had some really nice experience that he had done all on his own at night. So, that was just a really cool way to bring someone into the org.

Tod Beardsley:

That's really cool. I'm with you, there is not a shortage of people; the hiring funnel is just too narrow. Because we throw up our own roadblocks when we're hiring people. Like I say, when you hire people, you want to hire for retention; because hiring new people is crazy expensive compared to hiring junior people and training them up and just having that talent pipeline in-house. So, I love hiring people who are not in a straight shot. I graduated from Purdue with a degree in cybersecurity, and now I'm ready for my new cybersecurity job, sir. Like, good for you, Purdue kids, but-

Jen Ellis:

There's nothing wrong with people who have that background; that is also a totally bonafide path.

Tod Beardsley:

But, it's one of several. It's not the path.

Jen Ellis:

I completely agree with you. As we look at our own experience at Rapid7, a couple of things: One, your story about Charles made me chuckle, because you remember Josh Feinblum, our friend, former head of security at Rapid7? So, when he was younger, he'd worked as a volunteer firefighter and he'd gone through all the training that you have to do to be a volunteer firefighter. So, when it came to incident response and building an incident-response plan, he thought of it very much in those terms. Like, you train, that's what you do. You recognize that the incident or the fire is agile and it will change and it will be dynamic, but you train and you prepare. That's what you do so that you can get really good at it, and you can then be more reactive and more flexible and able to react when there are unexpected things that happen.

Jen Ellis:

So, the lessons that he brought from such a completely different space, this idea of being a firefighter, to incident response, I think it was actually extraordinarily valuable. And it had nothing to do with a security background. Then, one of the people that he hired, who still is one of our absolute bright young leaders at Rapid7, Katie LeDoux, who is head of governance and trust, she doesn't come from a typical traditional security background, either. She came from comms, and she had worked in comms in a security company and she had got really interested in security, and she had immersed herself in the sphere of security so that she could be good at what she did in comms and understand who she was talking to. But, she didn't come from a technical background at all. It hasn't held her back in the slightest. In fact, what she's done is she's brought a completely fresh way of thinking into governance, compliance, and trust, and has built new models of how you engage customers, how you engage employees to build awareness and communicate needs and what we're doing in security. I think it's fantastic to leverage that expertise and that different point of view, so I love hearing that people are open to that and that you've been embracing that at Digital Shadows. So, tell us what else is in your program? What else are you looking at?

Rick Holland:

One of the things I like to talk to people about: it's not a conversation unless you bust out the Venn diagrams and when I-

Tod Beardsley:

You know how I know you're an analyst?

Rick Holland:

Here's the Forrester coming out. But I was like, "Here's a circle of what Digital Shadows needs for the role, and here's what you need for your own career and your career development. Let's get these circles as close as possible." And I also tell people, "I'm not ignorant. You're not going to be at Digital Shadows forever. I'm not going to be at Digital Shadows forever. How can I maximize the time that we have together to help you do the things that you want to do to line up your next job and to help us get what we need in this maybe 18-month, two-year run, depending on the role?" It could be less than a year with some of the turnover that you can see in some of the types of roles in the organization.

Rick Holland:

So, I think first is just to be honest with people and say, "Look, I know you have goals beyond Digital Shadows. How can I help you achieve those goals?" And, one of the things that has been beneficial for me with my Forrester background—and anyone that's probably been in their careers as long as I have—we've built up big networks, so I can introduce people. For example, we had an intern, an Italian intern in London who was doing some stuff on data science and machine learning, and I was able to set him up with a call with a gentleman named Alex Pinto. He's at Verizon now, he's a great guy. He's an awesome dude. Brazilian-

Jen Ellis:

We should get Alex on. This is a good idea, I like this.

Rick Holland:

I love Alex; great guy, but they were able to have a conversation and give him some guidance. Because I was like, "Oh, this guy wants some real world data science." Well, there's probably two people that came to mind. One of them happens to be at Digital Shadows—at Rapid7, rather—and Bob, that are like the people that I trust in a space. So, finding out where people are interested and exposing my network to them, that is a very useful thing. It helps them build their network out. Just trying to find out what they want to do, and how can you help enable that? You'll get some benefit as well for mentoring people like that. So, that's been one way. Let's just be honest, people aren't going to stay at your company forever, and just try to maximize the time that you have with them.

Jen Ellis:

Yeah. I think that's smart. I think recognizing that that's not personal, it's not a thing to be affronted by. And that actually, as a manager, part of your job is to help them create a great career path. It's not just to rinse them of what you can get out of them.

Rick Holland:

Right. We're not a sweatshop. We're not trying to get 18 months, 12 months. I think it's really important if, in organizations where you think about a SOC analyst, you want to have the career path. They can move from SOC analyst and maybe they're going to be vulnerability threat team, maybe they want to go to the red team route and they start doing pen testing, and they go red team. Maybe it's a good segue right there is to talk about the HR relationship. From the time that I took over teams, I've basically had standing calls with the various HR stakeholders in my organization at least every other week, if not weekly, for: How can we work together to build resiliency into the program?

Rick Holland:

If the only time you talk to HR is when there's a hiring or a firing, you're missing out on things. A good example of this would be salary bands. People that are in large companies will definitely get this. You need to promote someone because they've been with you a year, and now they could go and get a 20, 30% raise, maybe more, because they've done SOC work for a year, they didn't have that experience, now they can double it up. Do you want to have to recruit someone new? So, talking to HR and say, "Hey, look, our recruiting fees, outside recruiter, could be 20%. Plus we have our internal fees. Can we make exceptions to these salary bands for staff? Because, I don't want to have to start from scratch and recruit again and all of the opportunity lost."

Rick Holland:

By having a regular cadence with HR and building a relationship—I always say this to the teams, too—in a very positive way, it's all social engineering. I'm not meaning that in a negative way, but I'm trying to have a relationship with HR, understand what they're trying to accomplish, understand what the business is trying to accomplish, and then say, "Hey, look, I think we can accomplish our business goals if we can find ways to make exceptions for staff and give them out-of-cycle raises; higher raises, that sort of stuff." So, I think HR really should be a partner. I know there's different types of HR folks out there, some may be very transactional, but if you can find someone that wants to be more transactional and more strategic, they can be a really strong ally in trying to recruit and critically retain stuff.

Jen Ellis:

Yeah, I agree. Absolutely. I agree. So how has this gone? Generally, when we ask people to come on and talk about initiatives or projects that they've spearheaded, we asked them to tell us a little bit about the results. How has it gone? Has it been effective taking this approach?

Rick Holland:

I don't know. I've struggled with it. I've had success, because it's great to have this plan and then reality hits or COVID hits and stuff like that. I've never done this before at the scale that I have, I've been running teams now-

Jen Ellis:

Anyone who thinks that managing people is easy, I think, is not putting enough effort in, frankly.

Rick Holland:

No, no. I also think when you hear people talk about the different types of security leaders that are out there, I am still connected to the technical thing, for the teams that run on technical on the security side or technical on the intel offering that we have at Digital Shadows. But, you really should have a lot of your time on development and enablement of your staff. I think one of the things I struggled with is letting things go, letting responsibilities go. I have found—and I've improved here—but I become a choke point. There's something I could delegate down to another manager or to an individual contributor that I will keep at my level, and then things don't get done and I become a choke point.

Rick Holland:

Part of it is just being comfortable with other people doing work. Like, do you really need to do this work, Rick? Okay, you may like this work, but someone else might like it more. Do you really need to do this? Is this going to impact the change that you need for your organization the most? So, I've really struggled with letting things go and then becoming a choke point. In fact, one of the things that I've done that's been pretty valuable for us is I spent a thousand dollars on SurveyMonkey and that thousand-dollar annual subscription to SurveyMonkey has been great. We've done surveys of me to my direct reports and then to their direct reports on their management styles; soliciting feedback and I was getting, "Hey, Rick, you're becoming a chokepoint on some of this stuff," last year when I did one. It's been eye-opening to get that feedback, but you also have to be open to feedback. I've struggled with it, and I'm constantly learning. I think the people on my team will say, "Hey, Rick really does care. Sometimes we don't nail what we're trying to accomplish, but I know that Rick does care about me as a person and where I want to go in my career." And I feel pretty good about that.

Jen Ellis:

Yeah, I think you should. To your point, doing those kinds of surveys, that takes courage actually, and it takes a willingness to confront your own shortcomings that I think is hard; really, really, really hard. Bravo to you for doing it. I think more people should look for ways to facilitate honest feedback and then think about how to internalize it. But it's a hard thing to do, particularly if you're busy and it feels exhausting.

Rick Holland:

It takes time. Time, I think for management right now, is tough too. Because, in the past, when we were in offices, you could go into an office, you could get your touch with your staff quite easily. Now, every single one is a 15- to 30-minute call, which is really important, but you need to take the time for it. So, time is even more challenging now in a Zoom and WebEx type of world.

Jen Ellis:

Yeah, absolutely. I totally agree. I think it's great that you're challenging yourself this way, and I do think it's a lot. It's challenging, and it's good to be honest about that, because you're never going to achieve perfection; it's good to know where to go. So, what is next? Do you have plans for how to develop and continue to grow on this? Or even just maintain? Because it's not a one-and-done.

Rick Holland:

I think there's a maintain of: some of the things that I've put into place is we're doing six-month SurveyMonkey. We will compare the results of our previous to our latest assessment to see where we have improved or not improved. So, on that constant feedback and improvement train there. One of the other areas that I started doing, and what we started after COVID, was doing skip-levels. Skip-level was easy when I could see people in person, but a skip level is when you go directly to your reports' reports and have a conversation with them. I haven't hit the whole team up on skip-levels, but just saying, "Hey, how are things going? How's your boss doing?" That sort of stuff. So, keep that in place.

Rick Holland:

I was just talking with HR yesterday. In 2021, trying to see like, is there other training that we could bring in for the directors and managers on the team? And for me, myself, what can we do more? I have found that Google has a lot of good data out there. They kind of have, open source might, would be my analogy. But, the Google HR stuff, like for my SurveyMonkey, Google has a, I think it's a 10- or 12-question survey that they use for all of their leadership; and I took that and I tweaked it. I'll send it to you all, so you can put it in the show notes. But, that's basically what I've built my SurveyMonkey off of, was 80% of that Google model. Then it was just a simple "1 to 5" rating on these questions about your leader.

Rick Holland:

I think for me, I generally feel that I have a good framework in process and we've received feedback on the framework; framework being me as a leader and the leaders on the team. And now it's like: how do you improve that? A big one was, it's not just for me, as I said, but also, for some of my directs, it was also being a choke point. One of the things that we did coming out of that, was ask the team, "Give us 2 or 3 examples of things that you think you can do, that your manager is currently doing, to try to enable people to take on more, and then reduce that choke point. I think it's just constantly adapting to feedback is really what the 2021 plan will be.

Jen Ellis:

Oh, I love that.

Rick Holland:

It is eye-opening on how much stuff you either are hugging on purpose, that you want to keep, because maybe you have such high standards and you have a difficult time delegating, or you just don't know it, maybe you're just not aware of it. So, I found it to be a pretty helpful exercise.

Jen Ellis:

Yeah. Or you don't know that somebody else would actually be interested in taking it on. Yeah. That's great, love it. The program sounds fantastic, and it sounds very effective. I think there's good lessons that lots of managers in every sector could take on from this. So, thank you for coming on and telling us all about it, really appreciate it. Of course, we will be in touch about the script for "Hackers 2," let you know where this is going.

Tod Beardsley:

We'll give you a story credit.

Rick Holland:

Yeah.

Jen Ellis:

It was so great to get a chance to chat with you again, Rick. I hope I'll get to see you in Cambridge sometime soon.

Rick Holland:

Yeah. I really enjoyed it. Thanks for having me.

Jen Ellis:

All right, cheers. Bye.

Jen Ellis:

So thank you again to Rick for coming on and talking to us about his perspective on building your team up and creating great points. So, Tod...

Tod Beardsley:

Yay! We did it. We got through an election.

Jen Ellis:

Amazing, amazing. It feels like it's been forever. This was not a quick election. This was a slow election.

Tod Beardsley:

And still technically ongoing.

Jen Ellis:

But from what I hear it was a very secure election.

Tod Beardsley:

I do hear that as well and I'm super grateful to everybody who's worked on this for at least the last four years. From Krebs on down has really made a real impact on how we conduct elections. Pretty much everywhere as far as I know are using a paper-backed electronic counting mechanism, which is great. There are audits of bound. It is quite easily the most cyber-secure election we've ever had for sure, and I'm delighted. I'm delighted with how things turned out and we predicted that certain things would happen afterwards and they are happening. So-

Jen Ellis:

Do you get to take a break now that the election has happened? Is that how it works now? You don't have to be on your advocacy boat?

Tod Beardsley:

Well, there's still stuff to do. Just as a poll worker, an election judge, there's a runoff election coming up in December that I should probably work, if that's okay boss.

Jen Ellis:

Yeah, absolutely.

Tod Beardsley:

My cadre of election cyber people, we still are talking and we'll probably put something together for whatever the next conferences that we all are involved in but I mean there's still tons of stuff to do. I do think that we've gotten the message through that hacking voting machines is kind of a sideshow and the real risk is in all the IT systems that back that up. Things like the poll book, county websites, things like that. So I think that's where we're going.

Jen Ellis:

That's good. That seems like an important message and definitely making a maturing of the thinking around this. So, that's principled.

Tod Beardsley:

I think so as well. Yes, but that's enough about the election.

Jen Ellis:

Yes. Because something else happened recently. Right?

Tod Beardsley:

Yeah.

Jen Ellis:

Was it like...

Tod Beardsley:

So while I was being all crazy, doing lecture stuff on Halloween, oh! Hallows Eve. Our good friend, Samy Kamkar for some reason, releases on Halloween when he knew no one would be looking. You'll recall that Halloween was on a Saturday.

Jen Ellis:

Maybe he feel that it was very frightening. Maybe that's what it is.

Tod Beardsley:

It's pretty good. So he releases this thing that's been like sitting basically at the top of my inbox for the last two weeks of, man, I really should look at that, really should look at that, and I got a chance to look at it this morning, as I got back from vacation and it's called NAT slipstreaming. NAT stands for Network Address Translation, and this is... The short story is, it's a mechanism by which an attacker on the internet can trick your router into opening straight-shot ports to any listening service on your machine. So, and it involves a whole bunch of like IP tomfoolery and TCP tomfoolery and some UDP tomfoolery and basically it tickles all my network, funny questions when I read it.

Jen Ellis:

And you just used the word tomfoolery, which is always a plus for you.

Tod Beardsley:

Several times, even. And so if you were like me and we're busy on Halloween and after go back into the way back machine and look up Samy Kamkar's write up slipstream, he's released on Github and that is his technology that we all rely on at home at least. Initially it was to like help conserve IP addresses out in the world. So when you're at home, you have several devices, they all are probably running something local, right? They're on what's called a non ratable IP address space. So something like 10.1.1.1 or 192.168.0 dot whatever.

Tod Beardsley:

These are not "real IP addresses" and that they can't route beyond your house, but what your router does, which is the thing that talks to the cable company or your DSL provider whatever. Throughout the router says, okay, all you guys, you all share one IP address and I'll just keep track of who's talking to whom and the router keeps the table and everything seems to work.

Tod Beardsley:

It feels like you're directly on the internet basically,

Jen Ellis:

But...

Tod Beardsley:

This is such a normal technology that we all kind of taken for granted. It has like the knock on effect of being like one of the best security things that like we've ever produced as a species when it comes to working on the internet, because it means I don't have to run complicated firewalls. I kind of be kind of anarchy inside my house or inside my office because I know none of that can get reached by the internet because we're all running on routable IP, which means I can't talk to it over the internet would that be. Well, what Sammy has done is... Hey, not as pretty cool and it's also kind of featureful in that there are some protocols that require you to negotiate several connections for it to work.

Tod Beardsley:

One of them, the classic one is FTP, where with FTP, you talk to an FTP server and say, "Hey, I want to download a file," and the FTP serves as "Cool, where should I send it?" And then you say, "Over here," and I'm going to open a port. Well, your NAT has to know about all that stuff, but NAT says, "Oh, I see that there's like an FTP request. I'm going to listen for that. I'm going to see which port the internal machine says is going to be receiving it. I'll open it for that external server," And then he can just shove the file over it. That's kind of how, if he works, you'll notice that FTP is bogus. You should not do FTP.

Tod Beardsley:

This is a bad way to transfer files event hough file transfer is done the same. There are other protocols that do things like this namely, like STUN is one that's all caps stun, stunned, and turn. They have to do with like voiceover IP, HD23 is another voiceover IP protocol that handles this and web RTC, which is the thing that lets you do seamless video and audio in your browser. Again, because all of these things require several connections, all to be negotiated kind of all up front. So with all that said...

Jen Ellis:

Yeah.

Tod Beardsley:

If you kind of seeing where this is going, where if I can trick the Nat device into believing that I'm trying to say, do an FTP command or I'm responding to an FTP service like request from this internet, then that device may say, "Oh, okay, well I'll go ahead and open that port."

Tod Beardsley:

Even though that port might be doing something else, that port might be listening for a shared media thing. Where you have windows media server, for example, where you stream things from your computer to your TV. It might be doing things say you're in kind of more of a enterprisey situation. You're running an electronic medical record system that has a database that listens on a port, but it's never exposed to the internet. So it's no big deal. Well, now it's explicitly that if you're browsing the same thing and then you go to the evil website. A lot of caveats to this. So for one, it's all very interesting because he does it in a way where he's shoving through HTP responses from a website, but then cutting off the packet at a very specific point.

Tod Beardsley:

So the next fragment, because there's fragmentation in internet protocol, the next fragment starts off with something that really looks like a STUN request or an FTP response or something like that. So it tricks the NAT into thinking, "Oh, something's going on? I'm going to open a port." That's essentially the trick. But in order to exploit this, you have to go to the evil website for starters. Does a whole bunch of rig and roll, which is all very interesting, to find out what your internal IP address scheme is, which is kind of in the write-up the kind of throws us as... Oh, by the way, you have to discover what internal IP's are and here's how you do that. Which itself is kind of neat, really neat. That's like a long time thing of how do I trick a browser into revealing that?

Tod Beardsley:

Well, there's a couple of different ways to do it in the paper and then, once you know that, then you can start massaging, like the message sizes and say, "No. You can only send packets that are this size and no bigger," and now I know exactly where to cut off packets so I can have the fragments start right at the border where I need them to start and it all feels very like buffer overflowly, except that it's with a network Zack and it's super cool. So-

Jen Ellis:

Oh, he's thinking too, I wasn't getting that at all. Oh...

Tod Beardsley:

Very excited about it. I'm excited and not terribly worried for a couple reasons. One, like I said, you have to go to an evil website or website that has a malvertisment. An evil bit of JavaScript that's being served up by an ad network. Not impossible.

Jen Ellis:

That seems like a thing people do. People do go to those websites.

Tod Beardsley:

People go to websites that turns out.

Jen Ellis:

It does turn out. Yeah.

Tod Beardsley:

It only works on the machine you're on. So it's not like I can use this trick to expose your TV and then hack your TV. I can only expose the laptop you're on or the iPad you're on or something like that. Right. It's only that one endpoint. I don't get to branch out to other things and network. And I would say that like in many cases kind of big whoop, unless you're running... It already requires you to be running something locally, that's advertising to your local network that you don't expect to be on the internet. So like I said, media servers are a big one. Industrial control systems often have like a thick client. That's written in Java that has like a bunch of listening ports. So that's kind of getting sketch.

Jen Ellis:

Okay. It's like what could possibly go wrong.

Tod Beardsley:

Security, camera control, basically anything that has a thick client, that's not like running in a browser, and that is expecting connections in. So there are classes of software like this. So I guess the takeaway is don't browse to evil websites on those computers.

Jen Ellis:

Yeah. I mean it’s fair, probably stick to very, very specific things if you're using your ICS system. Yep.

Tod Beardsley:

Yeah. That's the thing, right? You shouldn't... Don't do this on a domain controller. Right. Don't be logged into their main controller and browser at it. That's, jobs zero here.

Jen Ellis:

That'd be just read it. That was better than where I was always going to go.

Tod Beardsley:

Sure.

Jen Ellis:

So have you just have a set of interest have you written something like a fanboy letter or is this... Are we taking it out of this podcast? Is that like, do you need to-

Tod Beardsley:

I assume that that Samy Kamkar is like a lawnmower man who just like is tuned into the internet at all times and will just be able to pick out his name.

Jen Ellis:

He'll know that the internet is talking about him now, and so he, when you answered this is to imagine that he just listens all the time to this, that you wouldn't like, he's not bad. It's just that he knows when the Internet's talking about him.

Tod Beardsley:

I mean, we can try to get them on to try to talk about it. So this is a evolution of the work he did 10 years ago because he's done things with NAT pending before that don't work anymore. But I guess the other thing is that, that doesn't make me too worried about it is that the POC is really just a proof of concept. It is not weaponized in the slightest. It takes a lot of work to go and you got to do a lot of weird things like run a web server that has like a packet sniffer running and then parse the packets in real time, and then do other things. You have to do a bunch of stuff. Once you automate it all up, it gets easy but like none of it is that. Right. So I don't expect to see this come up in real attacks. This feels more like a CIA kind of attack. It feels like something they would be interested in, but not the NSA. You know?

Jen Ellis:

I'm not going to delve too much into that distinction. I want us to be able to keep making the podcast

Tod Beardsley:

There's a lot of Weirdness with browsers of which browsers you can do this way and which protocols you can do it with and there's like a mix and match on that. So anyway, there's a lot of moving parts and it's still in the realm of, "Oh, that's a weird way that internet working works," and not so much, "Oh, you got to go patch all your stuff," but I will say one thing while his research is very, NAT focused to the point of like getting firmware for links, routers and things, there is no reason to think that this wouldn't work in grown-up firewalls that work in enterprises. So that, is left open as open research. So I do expect to see some work on this probably over the next year of how can I make like a real enterprise firewall act this way to given the way that IP fragmentation works. So...

Jen Ellis:

So basically if you have a grownup firewall, as opposed to one of those kiddie firewalls,

Tod Beardsley:

Uh-huh (affirmative) yeah.

Jen Ellis:

Pay attention. Fortunately, I didn't have a Mickey mouse firewall. All right. So I think that's all we have time for. I think...

Tod Beardsley:

If you want to go look it up it's Samy Kamkar's slipstream. Google that the first hit will be the thing.

Jen Ellis:

And then the second one will be Tod's open love letter.

Tod Beardsley:

And by the way, the website that has the writeup does not in fact explain this vulnerability. All right.

Jen Ellis:

Are you sure?

Tod Beardsley:

I have to.

Jen Ellis:

It seems like a missed opportunity, but okay. All right. So that just leaves me to say thank you to you for educating me. You just have me for inspiring you and thank you to our amazing guests in this episode, Rick Holland, who I'm sure we will speak to me in the future and thank you as ever to our incredible producer, the goddess of patience, Bri. Thank you very much.