Joe FitzPatrick on the Future of Hardware Security Training Sessions

Tod Beardsley:

Hi, Jen. I'm great. Hopefully, we'll reduce some of the mystery, but keep the enchantedness.

Jen Ellis:

This week, we're picking up the thread that we've had. We've had a little bit of discussion around this on and off through the... copocalypse? No. That doesn't work very well. I don't like it. Whatever.

Tod Beardsley:

Coronageddon.

Jen Ellis:

Coronageddon? Fine. We'll just go with that one. So, we've talked a little bit about the new normal, which is ironic, me talking about normal, and we talked a little bit about the move to adopt remote working. Obviously, you've talked about Zoom 1,001 times, and we've talked a little bit about virtual conferences. Had that great conversation with John Strand. If people haven't listened to it, they should go and totes check it out because it was great. So, today we're picking up that thread a little bit more and we're talking about security trainings being taken virtual.

Jen Ellis:

This is in light of news that Hacker Summer Camp is definitely going to be virtual, which I think is a really smart decision. We love it. So now, all of those people who do amazing trainings at Hacker Summer Camp, which is crazy big business, are now feverishly taking those trainings and making them work online, which is very exciting. So, with us today as special guest is Joe FitzPatrick, who is a trainer and researcher at securinghardware.com, and a veteran of providing excellent training. I imagine, Joe, you are now taking time out of your incredibly busy schedule of trying to turn those trainings virtual to come talk to us. So, thank you very much for doing that. We really appreciate it. Welcome.

Joe FitzPatrick:

Well, thank you. When my business is traveling all over the place to go to large groups of people and teach them, I don't actually have a lot of that happening right now. So, I've got the time to go and make this meeting remote.

Tod Beardsley:

Oh, no. Yeah.

Jen Ellis:

Right. I'm sure you've just been sitting around twiddling your thumbs, wondering what to do with yourself until we picked up the phone and then you were like, "Yes. Now, I have a mission."

Joe FitzPatrick:

With training, January and February is the low slow part of the year. So, it's nice to have that time to relax. Everybody spent their last year training budget in December, so now January, February, I can take some time off, get some stuff done. Then, March comes around and everything's canceled.

Jen Ellis:

You're like, "This wasn't in the plan." I think lots of people are saying this wasn't in the plan at the moment.

Joe FitzPatrick:

Yeah.

Jen Ellis:

So, now you are presumably preparing for not Vegas, as I will call it. So, what does preparing for not Vegas entail?

Joe FitzPatrick:

So, I do hardware training in particular, which is a little bit different from software training. I've always been jealous of the people who show up with a USB drive and have virtual machine images in the cloud and then start teaching class because last year I believe I shipped about 800 pounds of equipment for all the combined classes and stuff that I was working on. I show up with equipment to hack on, the tools to hack with, and it's hard to do hardware hacking without the hardware. So, figuring out how to do that remotely is an interesting challenge.

Jen Ellis:

So, let's take a step back because I did fly us straight into the belly of the beast. So, you do hardware hacking. What kind of hardware hacking? What kind of hardware do you focus on?

Joe FitzPatrick:

So, the classes I've been doing focus a lot on the embedded Linux type systems. So, you've got a Wi-Fi router, you've got an IP camera, you've got a Raspberry Pi class embedded device. Typically, what happens with IOT manufacturers is they get an off the shelf SOC, get the off the shelf software stack on top of it, and they throw their fancy web app and-

Jen Ellis:

What's a SOC?

Joe FitzPatrick:

SOC is system on chip, not security operation center. Very different thing. I get confused sometimes.

Jen Ellis:

Yes, right.

Joe FitzPatrick:

But most IoT defenders, they don't know what the hardware is underneath. So, in my class we basically go and we look at the hardware and from observation, figure out what it is and then figure out how we can use a little bit of hardware, a few wires here and there, to communicate with it and basically use physical access to give software privilege. 

Jen Ellis:

You've been running trainings now for a while, right? Your trainings are very successful.

Joe FitzPatrick:

So far.

Jen Ellis:

So, how frequently do you run a training and how many people would typically be in a class? Help us understand what it looks like.

Joe FitzPatrick:

I end up doing, typically in a normal year, roughly a training a month. The training's two to five days long and it's anywhere from 10 to 30 people. Some of them are private trainings onsite for customers, some of them are public trainings at conferences like Black Hat. I also run another event, hardware security art training, which is me and several other hardware security trainers who get together and put our trainings together into a separate event.

Tod Beardsley:

We've used this term a lot, training. I think one of the complications of this year is that training isn't teaching. It's a different experience, and maybe you can elaborate on that a little bit, because like you say, hardware training remotely, hardware's there in your hand, but there are other things that go on in an in-person training.

Joe FitzPatrick:

Yeah. So, it's funny to go and look at a slate. You have two days to teach a bunch of people a bunch of stuff, and what do you choose to cover, what do you want them to come away with, and what do you eliminate? Because, I could sit there and I could talk every day for eight hours a day about hardware for a month, but no one wants to hear that. Maybe some people do, but I don't know. I don't know why anybody would pay for that. So, I got to figure out in two days, what are the most important points that people need to come away with? So, I want to figure out the skills they want, figure out how to give hands-on in-class practice to do those things, so that when they go home after two days, they can repeat that.

Joe FitzPatrick:

That's my objective when I do an in-person training and I have two days to fill that. So, I have a combination of lectures and hands-on labs and lots of Q&A. One thing that I find really is helpful is there's a lot of very lecture-focused, lecture heavy training, but in my experience, when someone is trying to learn something, you can lecture and give them 10 important points to remember, and they'll remember two, three, five, eight of them, depending on the person. But if you let them go and try and do it themselves and they have a problem and they can't figure it out and they have to ask a question like, "Hey, where do I put this wire? Hey, what's this command line?" Since they've asked the question, then they get the answer, then it's far more ingrained than if I had just lectured that answer. Does that make sense?

Tod Beardsley:

For sure.

Jen Ellis:

So, how did you get into this and how long have you been doing it?

Joe FitzPatrick:

My summer job through high school was teaching at a summer camp. I got through college and grad school as a teaching assistant. So, I've done a little bit of teaching.

Jen Ellis:

So, it was kind of ingrained?

Joe FitzPatrick:

Yeah. Actually, it's funny. My mom does technical training as well. She did application training in the late eighties and nineties when suddenly everybody had a computer on the desk and didn't know how to do it. So, she's been doing training for her whole career. I was working at Intel in a product security group and we needed to expand the knowledge of hardware security, but we couldn't hire 500 people for a hardware security group. So, we needed to go and give the intro of hardware security basics to all the people who are doing validation in the company. So, I didn't develop that material, but I picked up and did a lot of the teaching of it and really enjoyed it. Wanted to do hands-on instead of lecture-based stuff, so when it was time for me to be done at a big corporate job, I knew exactly what I wanted to do.

Jen Ellis:

Very cool. So, now you've been doing it since then and that sounds like it's been a while. So, typically in a Black Hat, for example, do you train every year at Black Hat?

Joe FitzPatrick:

Yeah. For the past several years, every year.

Jen Ellis:

Your class is presumably I'm guessing from Tod's crazy class list that you pretty much sell out every year. Is that-

 Joe FitzPatrick:

Yeah, most of the time.

Jen Ellis:

Do you do just one class or multiples?

Joe FitzPatrick:

So, last year I decided to go all out. So, I've done four-day classes, I've done two two-day classes over the four days. Last year, I did four unique two-day classes over the four days. I had eight assistants who were helping me, so I was hopping between two classrooms with the beginner pair of classes and the advanced pair of classes. So, it turned into a big deal.

Jen Ellis:

Yeah. I was going to say, I'm exhausted just listening to you talk about it.

Tod Beardsley:

Right.

Jen Ellis:

I don't know. I mean, by the time they get around to the trainers' party, you must've just been like drooling on yourself.

Joe FitzPatrick:

Yeah. Last year, I definitely overdid it. I remember by the time Black Hat was done, I was like, "Oh yeah, DEF CON." I basically stayed in my room and at the pool and vegetated for two days, which was great.

Jen Ellis:

Well, yeah. That actually sounds like a very civilized way of doing DEF CON. Talking of sitting around a pool, you're actually on vacation as you speak to us right now, aren't you? You're actually very nicely taking time out of your crazy vacation to talk to us while you're in the wilderness. Is that fair?

Joe FitzPatrick:

Yeah. We had to wake up at 6:30 this morning because there was a bear on the back deck.

Tod Beardsley:

Oh, wow.

Jen Ellis:

Wow. I mean, did you wake up so that you could be like, "Oh, there's a bear on the back deck." Or was it more like, "Run!"

Joe FitzPatrick:

My brother-in-law, who's been here since, March, he was going fishing this morning. So, 6:30, he's going to go fishing and notices there's a bear trying to climb up on the back deck, so he starts yelling at it. So, of course, everybody wakes up.

Jen Ellis:

Apart from the bear who's like, "Whatever dude."

Joe FitzPatrick:

Yeah.

Jen Ellis:

Very cool. All right. So, this year, before everything went nuts, what was the plan for Vegas? I'm guessing you were not going to do four classes again.

Joe FitzPatrick:

Yeah, actually, I was dumb enough to try that. So, I had four classes in the schedule and actually, have some people who have been assisting me for several years who were going to lead those classes. So, I would be the support person and they would be the leader. So, trying to get more people in the ability to go and teach these classes. Then, this all came down and I realized pretty quickly that even if Black Hat was going to happen, I wasn't sure I wanted to go to it. So, I'd been brainstorming long before the announcement of the decision about how to make it happen. I realized I could get one class done in time for Black Hat and one class made sense, the introductory class, because that's the one that has the biggest demand. So basically, there's a lot of logistics to figure out, and then once you figure out the logistics, then there's all the details.

Jen Ellis:

Right. So, how's that going?

Tod Beardsley:

Yeah. I'm super curious how this is all going to work.

Joe FitzPatrick:

So, two sides of it. There's logistics of equipment, and Black Hat is usually people coming from all over the place to take a class and, from a privacy perspective, I just teach the class to them. So, we got to get equipment to these people to take the class. So, I have to go and get that equipment to Black Hat, black has to distribute it to people which sometimes includes international shipping. Typically, I reuse equipment, which is great because I have equipment that I've fine-tuned and honed and everything, but the reality is I can't just get the equipment back. So, now it's you get the equipment and you keep the equipment, which means instead of needing 30 of everything, now I need 30 of everything every time I teach a class of 30.

Tod Beardsley:

Oh, wow.

Joe FitzPatrick:

So, now it's frantically ordering stuff from China, which has some limitations on shipping and throughput right now to get all the equipment in hand so I can go and distribute it. There's a few custom boards in the kit and there's other stuff that needs to be modified, so it's not yet a this off the shelf product, this off the shelf, this off the shelf product, but it's getting there. So, the other side of it though is actually delivering the content. So, there's a lot of web-based training stuff available, which is great. There's lots of resources. There's lots of how-tos, but it turns out that InfoSec training is actually really good compared to most of what I see for training. So, when you get into this realm of what is online training, why do people do online training?

Joe FitzPatrick:

Well, until last year, number one was for cost. They didn't want to send people to training, so they did an online training. The other was that training is not to actually learn stuff. Training is all about compliance. So, "Oh, we need to give our employees a 30-minute course on compliance with this or a 30-minute coverage of this in order to meet this requirement." That's where all this money comes into huge high-volume training, whereas the InfoSec training, we are more about delivering skills that people are actually going to use. Skills that people want so that they can go and use, as opposed to skills that people don't really care about that they're required to receive, that their management has to come up with the way to tease them into clicking through enough links to get the check box, to get the compliance.

Tod Beardsley:

Right. So, typically your online training experience is a grudge buy where everybody taking it doesn't want it. It's all anti-corruption stuff or sexual harassment in the workplace, the kinds of like boring HR-mandated trainings. But your stuff, which is like stuff that people actually want to do and will be paying attention and will get mad if it sucks.

Joe FitzPatrick:

I mean, seeing this reminds me of how lucky I am that I get to teach something that people want to learn. Not every trainer gets that opportunity.

Tod Beardsley:

Yeah. 

Joe FitzPatrick:

So, the other thing is, you go on the internet and there's all these resources about, "Oh, you can use our system to set up your training" and "You can use our systems to set up your training." There's very interesting price tiers. There's the hundred-dollar tier, then there's the $3,500 tier. It's like, "Why would I pay $3,500 for you to share some videos?" They all cater to metrics of your compliance, of your training. The other thing they cater to is sales funnels. I don't know if you ever heard this phrase before, but it's this whole deal where you have a training where you're going to learn something, but they don't actually teach you anything until you give them your email address and then you start a conversation and then it turns out it's just a sales pitch.

Tod Beardsley:

You teach them how to buy your product.

Joe FitzPatrick:

Exactly.

Jen Ellis:

Yeah. So, you're painting a depressing view of this whole training shenanigan, so where does that leave you for your preparations for this year and moving forward on teaching people how to phish with a P-H?

Joe FitzPatrick:

Some things that I've discovered both in trial runs as well as in reading a lot about this and in figuring out what I'm going to do for my class is, when I was doing my in-person class, I often tried to keep my lectures to about half an hour, which is a good amount of time for engagement, get enough information, and get some interaction. Then, we'd have about 90 minutes of lab time where people would work on a challenge, a problem, an activity of some sort. When you get to an online training, it doesn't work the same way because people are not as engaged. The lectures are not interactive. They're more dictating. So, what I've done is I basically sliced my class up and I have five to 10-minute lectures and 20 to 30-minute labs.

Joe FitzPatrick:

So, every lecture-lab unit in my in-person class is sliced into three smaller ones for this. So, that also caters to the fact that there are some constraints of training. So, training intact in a bunch of people in the same place at the same time, there's a lot of synchronous stuff that has to happen, but if you're going to online training, people are in different time zones, people have different work schedules and family commitments, and some of them are still working from home or always work from home. So, taking advantage of that asynchronous nature is really helpful. Something I had always considered doing was flipping the classroom. This is the concept where instead of giving the lectures in class and then having homework be at home, you give the lectures as videos to watch at home, and then you have the classroom time to work.

Joe FitzPatrick:

So, this is something more like grade school and upper education have been trying, and I've always thought about it, but people have seemed very reluctant, but this is my opportunity to take advantage of it. So, I've got those five to 10-minute lectures recorded so you can watch them. You can watch them before the class. You can watch them after class. You can watch them three times, which for some people is really helpful. It's disappointing. Sometimes people will sit through the lecture and they get 90% of it, but there's those one little pieces they didn't know were important and so they have to go and re-grab those. By having these recorded lectures available for people to re-watch, suddenly you're taking advantage of this asynchronous nature and giving people more opportunity to learn more things better.

Tod Beardsley:

It sounds like a very Khan Academy style of better learning where you can pause, you can rewind, you can redo the thing. If you were spaced out or you responded to a Slack notification or something, you can just stop and back up five minutes or whatever.

Joe FitzPatrick:

Or kids knock on the door or a bear is on the deck.

Tod Beardsley:

Yep. Or if you get a bear.

Jen Ellis:

Or if the bear wants to take the training.

Tod Beardsley:

Don't teach bears how to hardware hack. That would be a bad idea. I don't want to be speciest or anything, but that's not a good skill for bears.

Jen Ellis:

Okay. So, I mean, honestly, that sounds really fascinating. I think it's going to be interesting to see. Will you share your results afterwards? Will you blog about it or do a presentation on it? I think it's going to be interesting. I think you're going to learn some pretty cool stuff through this process.

Joe FitzPatrick:

I always plan to, but I don't always get around to some of that stuff, but since I don't have any in-person trainings, maybe I'll have time for it.

Tod Beardsley:

Yeah. You'll have loads of time.

Jen Ellis:

Right, because it's not like you're doing stuff right now, other than making some people in China super happy with all the hardware you're buying from them. I've been seeing that you are now buying in bulk quite a lot, which has to be making some vendors pretty happy, even though it's probably not making you super happy. I mean, it must have an impact on your economics having to do it this way now, compared to where you were before. It must be a very different undertaking now.

Joe FitzPatrick:

It is. From a business operations perspective, I've been doing training for almost eight years now independently. It's interesting to run a business for eight years. It's been profitable for eight years. It's been a successful business for eight years, but when you look at the business side of things, cashflow is very different from profitability.

Jen Ellis:

Yeah.

Joe FitzPatrick:

So, running a business with positive cashflow, I didn't realize it. The first five years I had a profitable business, but I had very bad cashflow, and now I actually have a profitable business that can sustain it. So, "Oh, I can actually afford to go and buy all the equipment for a few hundred people to take this class over the next year and get all that equipment." And it's not a big deal because I know that I'm going to have the interest for this. So, confidence, cashflow, it all helps.

Jen Ellis:

It's actually interesting because it's a little ironic because a lot of tech companies, a lot of companies generally, I guess, I just know more about tech companies, have the opposite, where they're not profitable, but they have cashflow. So, it's interesting to hear about you the other way around. So, how are you feeling about the upcoming conferences? I mean, Black Hat's the big one you're working towards, right?

Joe FitzPatrick:

Well, I'm feeling confident that this training is going to come together. My biggest worry is about equipment. So, when I'm teaching a class, equipment breaks and I normally have spares of everything, but when I'm teaching a remote class, it's not necessarily that easy. So, I've put a lot of additional checks into the course like, "Oh, before you do this, check this. Before you do this, check this." But that's going to be the big one that comes up is when I run this class at Black Hat, is anybody's equipment going to break? Are they going to be unable to finish the class? Again, that falls back on the asynchronous part of thing. I'll send them replacement equipment, they'll get the chance to go and fill in the blanks, but when you're hoping to get this class done in two days and then your equipment breaks, it's going to be pretty tragic. So, that's the one thing I'm worried about.

Jen Ellis:

Yeah, I mean, I can understand why. So, look, I have one more question. Tod might have others, but one of the things that we really focused on as you probably heard in the intro for this podcast is, when people have taken on projects, try and do something cool and different and that exploratory quality, that going out and forging your way, and obviously, everything you're talking about is that. You've come across some pretty major roadblocks and you've adapted to get over them, which I think is very laudable. I think it'll be really fascinating to see what the outcome of that is. One thing that we do ask people is, for other people who are looking at climbing their own mountains and taking on their own huge challenges, wrestling their own bears, whichever way you want to put it, what advice would you offer?

Joe FitzPatrick:

Well, there's two levels to that. There's the "What advice do I have someone who's looking at training as a possibility?" And then, "What for anybody in any generic task?" For training in particular, it's interesting to see what we have for InfoSec training. We have a lot of experts on topics teaching classes on those topics, and that's great. We have some people who are dedicated trainers who are also training as well. If you think that, "Oh, I'm an expert on X. I can teach a class on X," that is true, and you can teach a class and there will be an audience for that, but if you want it to be a profitable business, you have to consider, "Well, how many people want to learn X and how many people a year can I teach X and how can I make sure that I am sustaining a market?"

Joe FitzPatrick:

So, don't just jump into it and say, "I know X. I can teach X." That is great. That is an important part of this, but don't think that throwing it and turning it into a profitable primary business is as easy as it looks.

Jen Ellis:

Yeah. That makes perfect sense.

Joe FitzPatrick:

On the other side of things, the more general, you've got a challenge in front of you, what do you do? Iterative approaches just work. So, when I got into the training, the first training I did is my second DEF CON. I went into the Hardware Hacking Village with a bunch of kits, teaching people how to do side channel attacks on an Arduino. Then, I refined that. I went back and I had a workshop that I did at a couple of BSides Conferences while at the Hardware Hacking Village at DEF CON, doing basically basics of FPGAs and how to use them. By doing that, by showing up with equipment, by handing it out, by teaching people, even though it was an hour to a three-hour workshop, I got through the steps. That's how I got the knowledge of how I should be doing it for a two-day class or a four-day class.

Joe FitzPatrick:

So, the more practice you get, the better you get at what you do. So, take advantage of that. The other thing is, if you keep doing it and you're not getting better, then you're getting worse.

Tod Beardsley:

Oh, that's true.

Jen Ellis:

I also really like the idea that an experiment is worth a thousand theories. You can get into analysis paralysis, and at some point, you have to try and see what happens. I think that is good advice. Just really quickly, those technical terms you just mentioned, I always try and make sure that people explain the technical terms. I'm not sure that everybody that listens is as technically capable or knowledgeable because I'm pretty sure that my mum makes up for at least half of our listeners.

Tod Beardsley:

An Arduino, which is a system on chip thing? Well, an Arduino is a whole board, obviously. Yeah. But your side channel attacks are ... go ahead. What is a side channel attack, Joe?

Joe FitzPatrick:

A side channel attack is when we look at how long it takes a processed attack to happen in order to figure out more about what's going on. Basically, it takes longer for it to calculate an almost right password or key than it does to just say, "Oh, no. Your password is the wrong length, so it's totally wrong."

Tod Beardsley:

Got it.

Jen Ellis:

And there was an acronym you used.

Tod Beardsley:

FPGA. FPGA is field programmable gate arrays. If you can imagine a CPU or a microcontroller, this runs code. It runs a bunch of instructions, a bunch of programs. So, you can make your own code and programs and plug them together. You can use libraries. That's the level of building with gears and pulleys and stuff, but if you go the next level down with an FPGA, you're using gates. So, it's like you're using Legos to build the gears, to put together, to make the machine that does the work. So, FPGA is just like a layer underneath what you might expect in a microcontroller. They're pretty useful for when you want to get very precise timing or fast protocols.

Jen Ellis:

No, that was super helpful. Thank you very much. Joe, thank you so, so much for taking time out of your vacation and your bear wrestling to talk to us about this stuff. Good luck with your training program. I am absolutely super interested to hear how it goes, so I hope that you'll come back at some point and tell us how it went and give us a little update. We would love to hear more about it.

Joe FitzPatrick:

Sounds great. I'd love to give you an update. I'm really looking forward to seeing how this goes at Black Hat. Hopefully, I will have some success to report on. I'm sure that some parts of it will fail and that's setting us up for the next round of success.

Jen Ellis:

Right. I love that philosophy. I completely agree. It's all good learning opportunity which, for somebody who does training, is what you're all about.

Joe FitzPatrick:

Exactly.

Jen Ellis:

Awesome. Thanks so much, Joe. We really appreciate it. Good luck.

Joe FitzPatrick:

Thanks for having me.

Jen Ellis:

Well, thanks, Joe. That was awesome. I hope you don't get eaten by a bear. It would be devastating.

Tod Beardsley:

Or beaten by a bear.

Jen Ellis:

Don't get beaten by a bear, either. That'd be really bad. But I'm just going to leave it at that and not start trying to find other words that rhyme with eaten and beaten.

Tod Beardsley:

Alright.

Jen Ellis:

I'm still thinking of them. Are you thinking of them? So back to stuff and security. Tell us, save me from myself, what's happening in security at the moment?

Tod Beardsley:

Well, tons of stuff. But the thing that I'd love to talk about is a little report that Rapid7 put out, yes?

Jen Ellis:

What? No, never heard of them.

Tod Beardsley:

Yep. A little crass self-promotion here. We put out a fairly hefty tome on the state of the internet. It is an internet atlas, basically. What's going on on the internet today. And it is called the NICER Report and you can go get it at rapid7.com/NICER. So yeah. So while you're waiting for that thing to load, here's what it's about. The NICER Report is a study of the internet as a whole, in terms of exposure along matrice of national exposure, industrial exposure, and cloud exposure. So that's where the "n", the "i", and the "c" come from.

Jen Ellis:

Wow.

Tod Beardsley:

And we take a look at ... Yeah. It's kind of re-acronym, back-cronym. I mean, it went kind of back and forth there. At this point, I'm not sure what came first, the name or the subjects.

Jen Ellis:

Oh, it's the classic chicken-or-egg problem.

Tod Beardsley:

It was a super fun report to write because we get to nerd out a lot on all of the protocols that make up the internet, not just the web. When you talk about the internet with most people, they probably just think of the web, but hey, it turns out there's a ton of other stuff on there. And a lot of stuff that probably shouldn't be on there. So we talk a lot about the prevalence of SMB servers, of which there are about a half a million. The prevalence of Telnet servers, of naked database servers, so scandalous, and all kinds of other stuff, and basically break it down into really kind of three or four core takeaways. One, there is still a whole bunch of cleartext, not encrypted stuff on the internet. Ask me why that's bad.

Jen Ellis:

Why is that bad, Tod?

Tod Beardsley:

Well, I'll tell you. It's not ... Thank you for asking.

Jen Ellis:

Such a surprising question. This one seems a little self-explanatory. Like some of the others, I think I really will be like, why is that bad, Tod. But this one, that's obviously terrible.

Tod Beardsley:

Well, crypto gives you not just secrets, but it gives you authenticity too. Cryptography controls give you a way to say the machine you think you're talking to is actually the machine you think you're talking to. There are cryptographically provable identities. And it also gives you an assurance that the data you're getting from those sites on the internet has not been tampered with in transit. If someone's going to tamper with them, they're going to tamper with them way earlier and that is much harder to do. So it really gives you kind of an assurance, not just that you're keeping secrets, but that the secrets you're keeping are in fact the secrets you think they are. So that's good.

Tod Beardsley:

We get bent out of shape when we see things like Telnet consoles hanging out on the internet, because Telnet does not give you any kind of crypto, just in general. But speaking of consoles, that's another thing that we kind of talk about a lot in this report, is just the prevalence of console access across the internet. The internet was designed with this in mind. The whole idea of the internet was like, I could type on my keyboard here and it would run commands over there. And then the data, the results of those commands, would come back here. And that's great and that is super innovative when you're in 1969, when all of this stuff is getting invented. It's a different world today.

Jen Ellis:

It is? Wait, what? But everyone says nothing ever changes. Are you saying things do change?

Tod Beardsley:

Things change over time, it turns out, and things like threat models happen. So the internet is not just a collection of academic computers anymore. It is all computers. And so when every computer can talk to every other computer, it turns out they will. And so when you have these consoles exposed on the internet and they might be Telnet, which is super bad and that's clear text. They might be SSH, which is better, but still a console in many cases. Or things like RDP, which is remote desktop protocol, or VNC, which is virtual network computing. Any of these kind of mechanisms for typing commands on a computer and getting results, they should be kind of safely tucked away behind a corporate VPN or maybe your school's VPN or something like that, or at the very least have some kind of second factor of authentication.

Tod Beardsley:

SSH works very well with duo stuff, for example. Or 2FA, 2FA exists for all of these mechanisms. And that second factor, may be just log into a VPN first, or it may be your more traditional 2FA where you get a text or something like that. That gets you into a place where not everyone can just walk up to your keyboard on the internet. And that's kind of what we're protecting against. And the thing is, is that, and we talk about this a bunch in the report, go read it, wee see a bunch of these Telnet consoles hanging off of kind of core router infrastructure. So things like the routers that run ISBs for whole countries may have Telnet exposed on them, which is just waiting for someone to guess the password and sometimes they're using defaults. So that's a big ... That is at the level of a national security issue where we find these things.

Tod Beardsley:

And speaking of national security, we are talking to governments around the world, mostly Five Eyes, maybe a couple other friends, of what they can do to help with this. We've written this report with them kind of top of mind. We're also writing to cloud providers, the folks that are giving platform as a service, internet as a service offerings. We had thought that maybe cloud providers would be making things better because they're new and they don't have a bunch of legacy just built in. But it turns out people are shifting their legacy from their data center to the cloud's data center. And that's also a problem, and we talk about that a lot in the report.

Jen Ellis:

So, I mean, obviously it sounds like the findings are interesting and there's lots of findings. Perhaps like-

Tod Beardsley:

There's like 80 findings in here. It's a lot.

Jen Ellis:

It sounds like they're maybe not going to shock security professionals, but on the other hand, it may help them to understand where to focus attention. And having taken a little shifty through the report, one of the things that I really like about it is that for each section, you've sort of given a TLDR to make it easy to consume. You've given attacker's perspective as well as the defendant's perspective. And then you've given actual guidance and you've broken that out by the security and IT people, the cloud providers, and policymakers, which I think is an awesome way of doing it.

Jen Ellis:

So I feel like even though it's a hefty report, it's very easy to dip in and out of it and pull out the most relevant parts to whoever you are as a reader. One of the things that we do whenever we're interviewing somebody for the podcast is we always ask them if they've taken on a project, why did they take it on? What was the thing that drove them to do it? So I will ask you that same question. Why did you do this report?

Tod Beardsley:

It's a little mini interview in the Rapid Rundown. So yeah, so I ... We took this on, we were going to do this anyway at the beginning of the year, because we like to keep track of what's going on on the internet and answer questions like, how many SMB servers are there actually? Just kind of normal questions like that. But then as we started ramping things up, COVID happened. We ended up redoing all of our data collection, targeting specifically the last two months of ... Or, last two months of March, it feels like it. The last two weeks of March and about the first week, week and a half of April. Because that period really captured, I think, what the new normal of the internet was going to be.

Tod Beardsley:

Because Asia was just starting to exit lockdown, Europe and the Middle East were in the throes of lockdown, and U.S. had just started lockdown. U.S. is already in lockdown for maybe a week or so. And so we expected to see a ton of change there and we expected to see everyone opening up all their firewalls and saying forget that security stuff, we need to get work done. Turns out that didn't happen. And that was itself kind of shocking for security people, for the folks on this.

Jen Ellis:

Yeah. But also kind of cool.

Tod Beardsley:

It tells me that the internet, as an invention, one of its design purposes was to withstand physical attack, withstand some kind of physical disaster on the planet. Well, we have one and it's a biological virus followed by a recession, so an economic disaster, and the internet seems pretty resilient against that, it turns out. I was surprised that not only did things ... not only did we not see a whole bunch of new insecure services show up, we actually saw a reduction in some of our favorite bug bears, and that's SMB and Telnet. SMB was leading the pack. We lost 16% of the total SMB population.

Tod Beardsley:

So while I say there's still half a million SMB servers, that's lower than the almost 700,000 SMB servers that were there in 2019. So that was pretty great. And that was a real interesting finding. So anyway, the long way around to that is we need to measure the internet fairly routinely to see what's going on and where the trends are and what people can do to make their own corner of the internet more secure compared to their neighbors. And also more tactically, I guess, and more immediately, what are the effects of COVID and recession on the internet? And it turns out the internet is chugging along.

Jen Ellis:

Which is great, actually. Chugged on.

Tod Beardsley:

It is great. But it's exactly that. But one of the themes that we kind of came into on this report was, things are getting better, but ...

Jen Ellis:

I sensed a "but" here.

Tod Beardsley:

Yeah. It's not getting better fast enough. We still see a long tail on patching, on whole version upgrades for certain services. We see SSH servers that are 14 years old.

Jen Ellis:

Let's just take a moment here. Let's just take a moment here to say, I think if we've learned anything over the past few months, and hopefully we've learned lots of things over the past few months], not patching is a terrible idea, people. I get that it's hard. It is really hard. We understand there are genuine reasons why people don't do it, but there have been a lot of category 10 severity vulnerabilities during the rounds.

Tod Beardsley:

There are cat-37 vulnerabilities.

Jen Ellis:

Right, right. I've lost count.

Tod Beardsley:

As we record this, virtually every remote access technology out there has had some kind of pretty major bug and fix.

Jen Ellis:

It's so weird that people are scrutinizing remote access at the moment. So strange.

Tod Beardsley:

Right. Exactly. Your intuition, I think, is correct. People are looking at this because we rely on it 10 times more today than we did a year ago. And so things like firewalls, VPNs, routers, all of these things are very much under scrutiny right now. And we're seeing all these ... we see all these patches come out and it's great that we have patches and it's great that we have coordinated vulnerability disclosure, but it is that last step. We will see. But in our report we talk a lot about the lack of patches and the lack of even just basic vulnerability management and version management and asset management. Check back in October because that will be right around the 90 day mark from this last crop of really kind of terrible vulnerabilities. One of them is going to get exploited in a big way, for sure.

Tod Beardsley:

Because if I saw a patch uptake rate of 50% of the vulnerable population got a patch within 30 days, I would be ecstatic. That would be so much better than where we're at right now, for anything. For anything. And that is kind of the mind boggling thing. It's like, I guess people have ... well, people? Management? I don't know, let's blame the bosses. Why not? That the things that we put on the internet are not perpetual motion machines. We do need to kind of keep them going and keep putting work into them to make sure that they're doing the things they're supposed to do. And I think that people are reluctant to patch things that don't seem broken. Because it's counterintuitive. It's one of the problems with security. It's like, you don't know it's broken until it's really broken.

Jen Ellis:

Yeah, well, WannaCry being the problem that it was, right? Is that-

Tod Beardsley:

Well, the patches for WannaCry were released in February and WannaCry happened in May. So you had, big air quotes, "plenty of time" to get to it and a lot of folks didn't get to it. So I think that we need to just be kind of constantly kind of reassessing how fast can we patch and how reliable can we patch. It's on the vendors top. Vendors should be releasing patches for security and security only. No feature updates or no feature removals or things like that. Just security only patches would be great. Some vendors are great at that, some vendors are not so great at that, and some vendors do both. So it is sometimes a grab bag. But I would say that if you're on the hook for maintaining any of these remote access technologies, and if you haven't ... if you don't remember the last time you patched, you're definitely out a date. And if you patched maybe two weeks ago, like June-ish, you still have a lot of work to do. So, sorry, but that's kind of where we are.

Jen Ellis:

So, you know how I'm always telling you not to speculate wildly. I'm now going to ask you if you want to place a bet on which of them it is that's going to get massively exploited.

Tod Beardsley:

Oh, man, there's so many to pick from. It's like playing roulette, but with no colors. I mean, I have at least 30 to pick from. I mean, if I wanted to cause maximal chaos, that Cisco Telnet vulnerability is pretty bad, and it's bad for a couple of reasons. One, if you're running Cisco gear, which is fine, Cisco is great and everything, you should not be exposing Telnet to the internet. Full stop. And so you've already kind of failed before you've even got to thinking about patching. You've already exposed Telnet, which tends to mean you have kind of lax border security controls. And it's a good vulnerability, it's a remote code execution, so if I get control of your router, I can get control of your whole network. I can flip it on and off. I can do all kinds of weird rowdy things with it.

Jen Ellis:

And we have exposure data on that that sort of indicates that the exposure to that one is pretty high, right? And also I feel like I've heard anecdotally in the company that we've had conversations with people who've been like, well, we had no idea that was hanging out there. It shouldn't have been hanging out there. So I would say to organizations, even if you think you definitely don't have it, double check.

Tod Beardsley:

Yeah. And my call to action there is like, hey, ISPs, just turn off Telnet. See who complains. You know? Because there's really no good reason to have Telnet on the internet anymore. As an aside, I play some dopey ASCII games and they've all converted from Telnet to SSH. So if my stupid gaming community can do it, so can you.

Tod Beardsley:

There is another vulnerability I like a lot as an attacker, and that is the Windows DNS vulnerability. It's a little bit harder to mitigate because you need that Windows DNS to run Active Directory and so it's always going to be running inside. So you can't say like, Oh, just turn off that DNS and you'll be fine. That's not going to work. But that thing seems pretty bad. There are way more Windows machines running DNS on the internet than I was expecting, I think we clocked in at right around like 40,000ish, not quite 50,000. Which is, in the space of millions and millions of machines, that doesn't seem like a ton, but I own that one Windows DNS machine and that is a straight shot inside your network to almost all of your ... if you're running Active Directory, then I get all that. And so I get everything. That is a good kind of slam dunk. So those are my wild speculations. Cisco Telnet, Windows DNS.

Jen Ellis:

I appreciate the wild speculation. We will see what happens. Hopefully you'll be wrong and there'll be no exploitation of any of these things.

Tod Beardsley:

I'm sure it'll be wrong. It's summertime. Hackers take vacations too.

Jen Ellis:

Right. Well, right. Sure. Yeah. All right. Well, thank you so much. Thank you for talking me through all these things and thank you for the magnificent research report. If you are interested in reading NICER, and you really should be, even though it is a biggie, but it's super easy to dip in and out of I think you can get to it by, I'm going to say rapid7.com/NICER.

Tod Beardsley:

Rapid7.com/NICER. And it is well chapterized. You can put it down and pick back up again.

Jen Ellis:

I had no idea that chapterized was a word, but okay.

Tod Beardsley:

I just made it up.

Jen Ellis:

All right. So thank you so much and check out the report. And thank you to Bri, not just for taking on the amazing job of editing this podcast, but also because she also edited the beast of the report.

Tod Beardsley:

Holy macaroni. Could not do it without her. I couldn't do any of this without her.

Jen Ellis:

And thank you again to our special guest Joe, good luck in virtual Vegas, we hope that all your virtual dreams will come true. Your virtual training will go very, very well. Until the next episode. Thanks.