Advancements in Vulnerability Reporting in the Post-PGP Era: A Conversation with Art Manion

Tod Beardsley:

Sometimes interesting people with cool things.

Jen Ellis:

It is. That's it. It's like, ooh, which way round will those ones be?

Tod Beardsley:

It's called A/B testing. Jen, you used to be in marketing, you know what this is.

Jen Ellis:

Oh, hush your mouth. I feel like I need to get a thesaurus and try some other words than cool and interesting. So, I'm your host, you're very limited-vocabulary host, Jen Ellis, Rapid7's VP of Community and Public Affairs. And with me as you already heard, is my amazing cohost, Tod Beardsley. I think amazing is probably how I always describe you as well.

Tod Beardsley:

Pretty much always, yes.

Jen Ellis:

How about debonair? My debonair cohost.

Tod Beardsley:

I love it. I'm feeling pretty debonair.

Jen Ellis:

I feel like today's guest, what adjective would I use for him?

Art Manion:

Oh boy.

Jen Ellis:

I mean, firstly his mutton chops deserve their very own superlative.

Art Manion:

Yes.

Jen Ellis:

Which now people are going to be like, is it Dan Geer? It's not Dan Geer. I'm sorry, guys. It's the other guy in security with amazing mutton chops. It's Art Manion. We're so happy to have you.

Art Manion:

Hello, hello.

Jen Ellis:

So for those who don't know, Art is the Vulnerability Analysis Technical Manager at CERT Coordination Center, or CERT/CC, as we in the biz say. Which is part of the Software Engineering Institute at Carnegie Mellon University. That,  my friends, may be the longest title string we've ever had to do for a guest. Maybe we could make this into a thing, we could try and recruit more people with absurdly long title strings. Anyway, Art, it is a true pleasure to have you on despite your absurdly long title string. How are you doing?

Art Manion:

No, I'm doing great. It is a pleasure to be here. It's great to hear your voices. And yeah, looking forward to it. Yeah.

Jen Ellis:

So a lot of people, Art, have been growing COVID beards.

Art Manion:

Yes, I am one of them. Yes.

Jen Ellis:

Really? So the mutton chops are currently filled in? They've joined?

Art Manion:

Yeah. They are hard to see on a webcam anymore. Probably in person as well. But the next important customer meeting or wedding or funeral, they will be coming back. So-

Jen Ellis:

Yeah, like dude, this is your personal brand. You can't mess with this.

Art Manion:

It'll be back. It's just a matter... It's honestly mostly just laziness and lack of need. But there'll be back, don't worry. Yeah.

Jen Ellis:

Okay. Well, I hope so.

Art Manion:

Okay.

Jen Ellis:

And for those of you not familiar, there should be a photo of Art that accompanies the podcast, so it'll be his pre-COVID photo where you'll be able to see the magnificence that is his mutton chops. All right. So, Art, what are you going to talk to us about today? I think you want to talk about VINCE. Is this right? Before we get into it, actually, we should probably talk a little bit about what CERT/CC is, what you guys do.

Art Manion:

Sure. No. CERT/CC goes back 30 years, pretty much. Which in internet time, is a long time.

Tod Beardsley:

It's practically all of the time.

Art Manion:

It's almost all of the time. I mean, really it's getting back to the days. My rough understanding of the original cast of CERT/CC was one person possibly part-time who may have had the ability to call or email all the people who ran the internet at the time, almost. So it goes back pretty far.

Jen Ellis:

Let's get that person on the phone sometime.

Art Manion:

He is probably happily retired and spending his time fishing. Yes. Like proper fishing.

Jen Ellis:

And totally unwilling to answer the phone to anyone who wants to talk about CERT.

Art Manion:

He's a wise man. I suspect he is out. Yeah. I can tell stories about him being able to reach everybody personally on the internet. And this goes back to, and here's the fun part of the story. There was the relatively at the time famous Morris worm, which ran around and exploited, guess what? Surprise. Vulnerabilities in internet-facing software. That were not patched. And the CERT Coordination Center's first public posting advisory was about, hey, go and fix your internet-facing things, which were like Sendmail and FTP at the time. And 30 years later, we are still doing the same work.

Tod Beardsley:

Still sending that email, huh?

 

Art Manion:

Yes.

Jen Ellis:

Have you at least changed the font of the email?

Art Manion:

Well we've over the years progressed from a fixed-width plain text to web pages. These things called web pages with dynamic fonts, I think you download from a third-party site. And graphics, there's some colors involved, I understand. So we're getting better with the presentation. And, in fact, it's a great lead-in, because VINCE, this thing we're calling VINCE, is the next phase in the evolution of how we talk to the rest of the world about vulnerabilities. And there's-

Jen Ellis:

Look at how you smoothly transition that and set it up.

Art Manion:

You set me up so nicely, thank you.

Jen Ellis:

You are such a pro. Amazing.

Art Manion:

Yeah, I set it up. Yes.

Tod Beardsley:

So it's named after Vince Vaughn, right?

Art Manion:

It is not named after Vince Vaughn. And in fact, and I had to remember it. Sorry. It is the, we're backronym happy. So this is the Vulnerability INformation and Coordination Environment. And I owe a colleague who has moved on something, a beer or a smack for that backronym. But it works well, we get to call it VINCE. There's something short, great for DNS and websites and things. But this is really, for decades and decades as we discussed with the PGP signed email and the fixed width text, we are moving to a web-based platform to do not only the advisory publication but all of what leads up to the advisory publication. And some folks see this and some folks don't. Is often a lot of private, non-public back-and-forth secret activity, private vulnerability coordination that goes on before the advisories and the announcements and the CVEs come out. So VINCE is a platform to do that work, and we are trying to move away from a number of things including email as the transport mechanism for all this communication.

Jen Ellis:

Oh my god, that's going to make Tod happy, because he is not a fan of email.

Tod Beardsley:

Well, I'm not a fan of email. And I'm especially not a fan of PGP.

Art Manion:

Yes.

Tod Beardsley:

I am a fan of getting receipts for things. Not like read receipts, like the Outlook version of read receipts. But I do like having timestamps on things. And I don't want to steal your thunder here, Art, but I am a beta user of VINCE. That was one of my items of feedback is like, I need dates sprinkled everywhere. I want to know when things happened. Mainly so I could prove it to other people, and because my memory is horrible and I don't want to keep paper notes anymore. But I still do.

Jen Ellis:

Plus one all of that.

Art Manion:

Well, and actually those are great points. We've-

Tod Beardsley:

And it does now.

Art Manion:

Yeah. Well, and we struggled with, there are tradeoffs, right? So yes, Tod, PGP email. I occasionally feel like getting into the fray when someone says on Twitter that PGP is hard. And I want to say, look, I can use it successfully, but that doesn't scale at all. And it is hard.

Tod Beardsley:

Yeah.

Jen Ellis:

... such a bulls**t response. PGP's hard. Well, I can use it fine.

Art Manion:

Yeah, yeah. After decades of effing it up. I mean, it is hard. And it's hard at scale.

Tod Beardsley:

After you've made every mistake possible, which PGP is more than happy to let you do.

Art Manion:

It is. It is. So, okay, so I'm going to agree that PGP email as a transport at scale is hard and almost ridiculous. So I get it. I get it. Email is nice because it's a transaction log though. We have decades of cert@cert.org emails stored. My colleague can mine it for data. It's got timestamps. It's a great transaction log. Tod, I expect that we have your feedback noted somewhere in an issue. And I suspect we're doing something about it, but I don't have the answer directly at my fingertips. So I will check on it.

Tod Beardsley:

This is not the Tod Beardsley VINCE feedback session.

Art Manion:

We got a bunch of feedback in February, March before quarantine happened. And we are taking it seriously. And we've made a bunch of changes. And part of the release of VINCE will be noting some of those changes we made. But anyway, understood point. And this is actually Tod, again, a great setup. Moving from email to web platforms has some drawbacks. And a common communications channel is one of those issues. VINCE is its own custom web portal. What if we all had our own custom web portals for vulnerability coordination?

Tod Beardsley:

Oh that kind of sucks, huh?

Art Manion:

Yeah. We'd all have to have our own human interface with every different ... in different coordination platforms. I'm not sure how that's going to scale.

Tod Beardsley:

I have a feature request.

Art Manion:

Yes, please.

Tod Beardsley:

Because this is... No, it is actually the VINCE beta program podcast. Here's the feature request. Is that, why not dump all communications in the backend, like back to cert@cert.org? And that way you can continue on with your mining.

Art Manion:

We are planning to do some of that, and maybe more. That's a great point. Thank you. But we're trying to look at this, so we have our own operational service to take care of, right? Our own coordination service to take care of. But we are finding, we talk a lot with the vendors and developers who are going to end up being responsible for creating the bugs and for fixing them. Right? So we talk to product security teams, or PSIRTs, a lot. And we're seeing motion there to move to their own platforms and ticket tracking platforms as well. Some off-the-shelf stuff, some custom stuff. So we're noticing that this is, how are we going to scale in a post-email world? We are hoping that perhaps the dream of a common API might be a useful approach here. So along with email tracking as a transaction log, VINCE will support an API, and it is our hope to work that out by testing and production. And figure out if there is interest and viability in a common API, because we really don't want to scale to dozens or even low hundreds of vendors and coordinators and researchers all with their own custom web platform. It won't work well.

Tod Beardsley:

And I know that you're a standards API kind of guy. I mean, I assume you've looked at... Yeah, yeah. I assume you've looked at things like Veris, right? For something like that maybe, for having this common API language. Because the moment you said that, I just had visions of AttackerKB, which is a Rapid7 project. Talking to VINCE, and VINCE talking to AttackerKB and having their own machine conversations. And wouldn't that be a wonderful world?

Art Manion:

Again, the vision is, yes machines talking to machines. It is excellent standards practice to first look around and see what already exists before making a new one. Did you say Veris?

Tod Beardsley:

I did.

Art Manion:

Okay. Noted. Thank you. We will do that. But again, it's an issue for directly ourselves for VINCE. But as you know, CERT/CC, we are self-mandated and directed by our primary sponsor CISA, part of DHS, to make the community better. Right? Make the vulnerability coordination and vulnerability management ecosystem work better. So it's not just about us having our platform be awesome, it is about helping everybody else work better together. Reduce friction. So this idea of, how do we all speak together? A common API is part of the bigger picture, it's not just to make VINCE work well.

Tod Beardsley:

And the things that you cover is your mandate. Your primary goal here, right, is coordination. It's in the name. Whereas, I can imagine an API that is common between NVD, CVE, AttackerKB and VINCE, right? But they all have different goals.

Art Manion:

Yeah.

Tod Beardsley:

So you'd have different argots and dialects of kind of the same thing.

Art Manion:

Yeah.

Tod Beardsley:

I don't know. Let's design it right now, Art. This is all I want to do today.

Jen Ellis:

Yeah. This is the most active I've heard Tod be on a podcast for ages.

Art Manion:

It's a great visionary idea, but the problem, it's a vision. So you have to actually try to do it. And then it gets to be a slog. So, yeah. But-

Tod Beardsley:

Well so far I've been delighted with the VINCE experience.

Art Manion:

Great to hear it.

Tod Beardsley:

It was just going through the beta program was great. It seems like you have really good people working on it. So I'm excited to start giving you bad news over it.

Art Manion:

I am also. Yeah. I am also excited and a little nervous. We are trying to pin down a May go-live date.

Tod Beardsley:

Oh wow.

Art Manion:

So as I type dates and email to people, I'm feeling both excited and nervous. It's a big rollout and it turns out we're not a regular software development shop, so we're proceeding very cautiously.

Jen Ellis:

Well and there's some weird stuff going on at the moment in the world. I don't know if you've heard.

Art Manion:

There is. There is, yeah. But I think that's going to not block us, but it is making things more interesting certainly.

Tod Beardsley:

And you are an unusual software development shop, right? You don't super care about having a minimum viable product out first like security does. I would hope that security comes first over in CERT/CC land.

Art Manion:

Yeah, we've been doing as much security dogfooding as possible. We have some highly skilled researchers on our team who we have thrown at our software. And we actually have a peer team that does assessments and penetration tests that we're going to also try to throw at VINCE. And we will be very mature about the fact that the odds are in favor of us releasing software with, get this, a security bug somewhere in it.

Art Manion:

And if and when that happens, we will be a good vendor and fix it quickly and be transparent about what happened. And do our best to thank and give credit to the person who found it and told us nicely, hopefully.

Jen Ellis:

It's so meta.

Art Manion:

It is meta, but it's a thing you got to... If we're going to go out saying stuff, we have to live the example. Otherwise, it would be supreme hypocrisy otherwise.

Jen Ellis:

And that's not the thing that people do. Yeah. Right. That's so flexible.

Art Manion:

I think our mandate is to not, yeah, not hypocrisy as part of our mandate maybe. Anyway.

Jen Ellis:

Is that how it's worded? Not hypocrisy.

Art Manion:

That's in the contract. Yes.

Jen Ellis:

Do you have it on stickers? Is that-

Art Manion:

Not hypocrisy. We don't, but that's a good idea.

Art Manion:

Yes. So, yeah, yeah. We're very excited. I appreciate the odd Tod. So yeah. Yep.

Jen Ellis:

So what's the process? So one, what was it that made you guys all be like, this is a thing we must do and we must do it now?

Art Manion:

The idea has been floating around for years. I think a culmination of things, and not to get too deep into our internal processes, but having highly qualified development resources available and having some internal things line up priority-wise was a big part of it. We've taken a couple of starting runs with this problem over the years, but it's a big project. And I have a, as Tod already mentioned, I've got a great, great, great team, and great development team here. So that's really been key to getting this working now. Also, we got a lot of mileage out of our old platform, but it was 20 years old. And great investment, great return on investment, but getting dated. Really old Perl code, some other stuff that was getting fragile, long in the tooth, hard to support. So normal software development things, the life cycle had ended on that sucker. And time to move on.

Art Manion:

And again, right, if we're going to be a leader in this space by any means, it was also time, right? We're seeing the bug bounty platforms, people using normal commodity ticketing systems, adapting them for coordinated disclosure, other custom builds going out. It's time to get with what's going on and try to figure out, again, what are the interoperability at scale problems that this is going to cause? Like right, how do we all interface with our respective services that it's not human sending email anymore? It's machines trying to read APIs or something.

Jen Ellis:

So once you made the decision, what has your process been? Obviously you have some sort of beta running right now.

Art Manion:

We do. Yeah. So we've got, I'd say, a very small beta running.

Jen Ellis:

Do you hear that Tod, that you are special?

Tod Beardsley:

I know.

Art Manion:

Tod is special.

Tod Beardsley:

This is not news.

Jen Ellis:

I just wanted to make sure you were aware. I'm sorry.

Art Manion:

Tod is one of our special, special beta users.

Jen Ellis:

He is a very special beta user. Yeah.

Tod Beardsley:

I want a button for my denim jacket.

Art Manion:

I'm going to have to look.

Jen Ellis:

I feel like we can make this happen.

Tod Beardsley:

How about a patch?

Art Manion:

A patch?

Tod Beardsley:

Perfect. I love it.

Art Manion:

A button? Okay. The first beta was February and we got a bunch of feedback. Tod was mentioning, we did some usability testing and UX testing. We got in front of our... Probably one of the biggest user groups for us is the vendors and the PSIRT teams. So we got in front of them a couple times in February and late March, collected a lot of feedback. Have made some changes based on that feedback. Basically been working through those things since then. And again, looking at May here.

Jen Ellis:

And have there been any things that were big flags or real big learnings that came out of this?

Art Manion:

There is a big flag. There's at least one big flag. And we pretty much expected this. There's the VINCE platform and the technology basically implements a shift in the coordination and dynamics. So with email, the CERT Coordination Center was, if you're a network topology person you can imagine a hub and spoke. So a researcher sends email to CERT and we decrypt that and process it, and then we read it. And there's something that needs to be done, we send that out onto the vendor, or multiple vendors. Or we do a back and forth with the researcher, refine the message, and then send that to the vendor. So CERT/CC is the moderator of all of the email comms going between usually one researcher, sometimes more, and one or more vendors. And this is nice because we can moderate and tune messages and extract, delete superfluous things and let the objective tech stuff go through when needed.

Art Manion:

On the other hand, we're the blocker, right? If the analyst is busy and has 10 cases, or is out for a week and we transfer it and someone's behind, we slow things down. So VINCE implements a bit more of a bus topology where all of the participants in a case have, among other ways to communicate, they have a common case chat, or a case room where everybody in the case, all the vendors, all the researchers, can see what's going on in the common room. So in this case, the moderation is basically removed because the CERT Coordination Center is not going to be in the middle of a message allowing it to go through or not. On the other hand, as soon as the message is sent, everybody in the case sees it, and we're not slowing things down. So that's a major shift. And there understandably with some feedback from the vendor community about the researchers seeing all of the otherwise private comms that's going on.

Art Manion:

It's our pretty strong position these days that we are going to assume good behavior and professionalism and benevolence on all parts of all parties. And let's open this up and have this out. And if someone's misbehaving, there will be some discussions about that. But we want to remove ourselves from the blocking position and encourage a norm where everyone's talking, getting along, getting the things fixed and going about their business. And we're not in the middle of it and closing the sometimes real and sometimes perhaps imagined researcher and vendor gap.

Jen Ellis:

And I think that sounds like a pretty reasonable reflection of the change, the maturing of the culture around this. And the maturing of the understanding, on both sides, right? That there should be a greater expectation of open collaboration on both sides. And both parties coming to the table prepared and expecting that to be part of the process.

Art Manion:

Yeah. And that's where it's not just technology, we're trying to advance the norms and the practice. And I know you guys at Rapid7 have been behind this for many years, and great supporters of the community beyond just what Rapid7 does for business.

Jen Ellis:

Oh, shucks.

Art Manion:

No, really.

Jen Ellis:

Say it again.

Art Manion:

It's entirely true. And I am saying it out loud. Thank you both. And thank you Rapid7 for the community aspects of your work.

Jen Ellis:

This is great. You can come on anytime.

Art Manion:

Yeah, yeah. I can. Hey, I call it like I see it, it's for real. But right, again, mandate is not just do a good job of our own hamster wheel, but make things better. And we want vulnerability coordination and reporting to be kind of mundane and boring. And it's going to stay exciting, don't worry. But we want it to be normal.

Jen Ellis:

No, but it needs to get to a business as usual point of view. I agree. That doesn't mean not prioritized.

Art Manion:

Correct.

Jen Ellis:

But it means not panicked and hysterical. I totally get that. Yeah.

Art Manion:

Yes. So this is sometimes perhaps not obvious to people, but all software has bugs. And some of those are security bugs.

Jen Ellis:

What?

Art Manion:

It's weird. The rain will fall, the sun will rise, software has bugs. I'm sorry. Don't sweat it. What are you doing about your bugs? That's what we're going to focus on, not that you have them. You are forgiven for developing software with bugs. Please pay attention to your response to the security bugs. That's it. Anyway.

Tod Beardsley:

So the downside here though is, I'm sure you've seen this a mile away of, if you're no longer moderating you are going to run into a case where a hacker is going to be threatening a vendor and a vendor is going to be threatening a hacker. And then things escalate crazy quick, especially if they're both in weird time zones.

Art Manion:

Yeah. Yeah.

Tod Beardsley:

So I assume you have some kind of backstop for this, like you can swoop in and moderate or something, or what?

Art Manion:

So the plan is, one of the changes we made post that the first big round of feedback is, there is the ability to have a... We can set up a private channel within a case. And of course there are always outside events, back channels, right? Where despite the fact that I'm saying, PGP email is dead, we just made bug fixes to our PGP email system because it's not really going to go completely away. And I'm interested also in, right, if there's a common discussion and someone is behaving questionably, I plan to go in there and we will use the tools at our disposal but there's a private chat feature. We can go out of band. We can say something in the group about someone's behavior and professionalism, or we can try it privately. And worst case, this is a social construct, right? If I misbehave in the group, the group doesn't want me there next time. That's kind of where this leads to if it comes to that level of action.

Tod Beardsley:

So it's in the interest of vendors of software to behave themselves. Right? Or else-

Jen Ellis:

Likewise, for researchers.

Tod Beardsley:

Right, exactly. Researchers too.

Art Manion:

Both, yes.

Tod Beardsley:

Yeah, I mean there is a power imbalance there.

Art Manion:

There is.

Tod Beardsley:

It seems like the power does rest with badly... Like badly behaving researchers cost more, I'd say, than badly behaving vendors. Like badly behaving vendors don't get bug reports anymore. Right? That's kind of the end state. If you're just not responsive and you're threatening and all that, it's much better than to just never report bugs. Because why would you bring that heartache upon yourself? Whereas a badly behaving researcher who shows up and says, if you don't act in a day, then I'm going to drop this on Twitter. It's like, well, I mean that is kind of the ultimate... That seems to be the end state. Right? Not considering the fact that you're going to have researchers come in that don't have actual... Like they're going to report a thing that's a bug that's not really a bug. Maybe it's a feature. Maybe it's something you can configure around, or whatever. But yeah, I feel like badly behaving vendors have more to lose.

Jen Ellis:

I think, yeah. I think that's right. I think that unless you have broken the law in some way, which don't do that, don't-

Tod Beardsley:

Yeah, don't do that.

Jen Ellis:

But unless you've broken the law in some way, then the researcher really just has their patience to lose, probably. Although actually we have stories of people who've lost jobs and that kind of stuff.

Art Manion:

Yeah.

Jen Ellis:

But I think that the bottom line is, if as a researcher what you care about is seeing things fixed and seeing people protected... Like if you care about security, right, that's your end state that you want to get to is improved security. Then it behooves you as a researcher to also behave appropriately. Because if you don't, then the vendor is much more likely not to.

Art Manion:

Yeah. Yeah.

Jen Ellis:

And so you need to... Like building trust is a two-way street.

Art Manion:

It is.

Jen Ellis:

It's a two-way process. You have to not just meet people in the middle, but almost overcompensate to say to them, because you have to recognize that what you're doing is a situation that creates defensiveness. It just is. And to Art's point, hopefully one day it won't be. But for today it definitely is. And so you have to kind of have a little empathy there.

Tod Beardsley:

And it's adversarial, but adversarial systems are sometimes okay, right? Like we have lots of adversarial systems that we need. And it's like the US court system is an adversarial system and we need it. And so I imagine that there will be a future where bugs are boring, like Art said. And it's just kind of matter of course. There will, I think, always be an element of adversary in that. And it's just up to everybody to just grow up and be mature. 

Art Manion:

You're both of course spot on. And it's okay for it to be inherently an adversarial system. But everyone, the mature players, understand that despite that there's an etiquette, there's a protocol, there's a norm. And here's what I've looked at over the years. Even for a vendor, if you want to imagine the most sort of venal bottom line focused commercial vendor you could possibly imagine with the most lawyers possible, it is less... Just an imaginary hypothetical vendor, not any specific one. Imagine a hypothetical situation like this, and here's my message to that imaginary hypothetical organization. You will save money/make money. It is cheaper for you to be cooperative in this process than it is to be adversarial.

Tod Beardsley:

Boy, I sure want science on that too. Do you have that? Does anyone have that? This is an MIT research paper.

Art Manion:

I don't have science, but I will say it as an opinion as loudly as I can. Yeah.

Jen Ellis:

Has Katie ever published anything on that? I feel like she has.

Art Manion:

Oh, you know-

Jen Ellis:

And actually we should give a shout out. Because I feel like any conversation around the sort of stuff without referencing your partner in crime, your co-conspirator Katie Moussouris who isn't part of the aforementioned CERT/CC software engineering issue of Carnegie Mellon University. But instead is the founder and CEO of Luta Security. Hi Katie. We're hoping you'll come on in a couple of weeks. But I think she in the past may have published some data on this stuff.

Art Manion:

Yeah, she may have. And basically I'm very aware of her work but it's a matter of you have to read it and get into the details. Or have an update chat with Katie about it. Yeah. Yeah

Jen Ellis:

And it may not be up to date anymore but-

Tod Beardsley:

I know she has done stuff on crossover of discovered bugs versus bugs that were in the wild, and all of that. And it's good research. What I'm after is dollar cost. I want an actuary table kind of thing of like-

Art Manion:

We do not have it.

Tod Beardsley:

... patching a bug versus fixing a bug. What are the cost differences?

Art Manion:

I will claim expert opinion. But right, someone like Katie who of course had spent time at a large company's PSIRT team, and probably knows what the internal costs look like. And I have seen large commercial companies' PSIRT teams move to more cooperative behavior. I highly suspect that those for profit companies figured out the bottom line on this.

Tod Beardsley:

I can't imagine they do it just like-

Art Manion:

No, they do because it's saving them money. It has to be.

Tod Beardsley:

Right. It's got to be.

Art Manion:

So remember, even if you don't like the situation you're in, it's cheaper. So if nothing else, please do it for that reason. We'd like you to be nice. Also, please be nice people. But, right?

Jen Ellis:

Just feel free to just be cheap, though. I like that as a message.

Art Manion:

Sure.

Jen Ellis:

That's great.

Art Manion:

Oh, so actually this touches on another item that, if you don't mind, very mild topic switch. But this goes back to if we are all going to have our own websites and web platforms, and we all have our own interfaces to these things. And how do we all communicate at scale, right? Beyond one researcher and one vendor exchanging button clicks on each other's websites. Email, at the bottom of email there's sometimes a long paragraph about you have to delete it and your lawyer said something and whatever.

Art Manion:

But email's sort of email. And I'm sure there's some legal stuff around email that I probably don't know what it is and don't listen to it. But emails are different. Email's a thing. But then when I go to a website, and maybe I sign up for a free account, I have probably clicked through some terms of service. And when I click submit on the vulnerability report form, there's probably some terms there. And these can be, it's one thing if I'm signing up for a bug bounty, right? And I am agreeing to, for example, cede to the vendor the authority to decide when to go public in exchange for money. Right? That's an exchange. And I read the terms upfront as a researcher and I agree to those terms, and I've got a contract. And an actual contract with the vendor, and in exchange for something I'm going to give them the bug report and behave a certain way.

Art Manion:

All well and good. When you step away from that a little bit and take away the compensation, those terms are a little bit less enticing. For instance, what if the terms were, the vendor decides when to go public and in exchange for that nothing? Or, we might put your name in the advisory. So there's a specific term here about what things a researcher might sign away in exchange for the other half of the contract. But even beyond that specific type of term, how does a researcher, or a research organization, or coordinator, parse and track down and decide in advance if they're going to agree to all of the terms on all of the web platforms where they're going to be trying to talk to people? And just collecting the terms is a mess because the terms point to other terms. And they're all written. And how do I parse them all, right?

Art Manion:

So there's a scale problem like the interface to the website. There's the interface with the policy and the terms problem at scale. And then there are the actual terms themselves. And we're seeing a little bit, we're hoping it's edge case stuff so far. But we're seeing this example where, hey, someone built a website, and while we're at it, why don't we ask the submitter to agree to some terms that we like. For instance, don't disclose unless we say so. And we're seeing a little bit of pushback in a couple of cases on, why would I agree to that in advance when I could just put it on Twitter or post it, or not report at all? And this gets back to your point, Tod.

Art Manion:

If the vendor is going to be a little firm about some of their terms, you very quickly push researchers to, your options are at a high level, tell the vendor in advance, tell no one, drop it publicly, maybe sell it to someone, right? One of those options is good for the vendor, the others are all not as good. So don't push researchers to not disclose or to go public. Anyway, terms, legalese, disclosure process, web submission form terms at scale are a problem, and slippery terms about giving too much control to one of the parties are a problem also.

Tod Beardsley:

If only we had some kind of common disclosure set of terms of their-

Art Manion:

Once again.

Tod Beardsley:

But we do, it's called Disclosures.io. And I think that's a bug crowd thing or a KCLS thing, or both of those things.

Art Manion:

Disclose.io?

Tod Beardsley:

Disclose.io. I am going to register disclosure.io right now.

Jen Ellis:

If you want to know more about it, we chatted with Casey just a few weeks ago, so there is...

Tod Beardsley:

If you scroll back I think four episodes I think it is. Yeah. Maybe five.

 

Jen Ellis:

And so yeah, so definitely check it out. So what is the key takeaway that you want people to have on VINCE? Other than that Tod is special, obviously.

Art Manion:

Tod is one of the special people. But yes, I think a lot of us know that, and appreciate it. So right. If you do coordinated disclosure business with CERT, the times are a changing. There's a new platform coming and you can set aside your PGP email and you can get out your two factor auth and your web interfaces and we'd be happy to talk, talk and test API with you.

Art Manion:

There's a bit of a shift in the coordination communications paradigm where everyone's going to see each other in the same common room. And those are the big shifts. And I guess that's going to change your interfacing with us, but we are very interested in, again, doing this at scale. As everyone else moves to a platform, how do we all manage to talk to each other and interpret each other's terms at scale so that we're not hung up with having to click on 50 different websites for 50 different vendors for a vulnerability report?

Jen Ellis:

Awesome.

Tod Beardsley:

Good luck.

Jen Ellis:

Yeah. Firstly, as he said, good luck. And thank you. Thank you not just for coming on and talking to us about it, but thank you for everything you do on this. You've been at CERT for a very long time and working tirelessly, or maybe you're like, no. I'm exhausted. On vulnerability disclosure. And you are very active in the security research community, helping people understand the process and advocating for better processes and better understanding and better compassion. So we're super grateful to you for what you do on that, and your efforts on the ISOs and yeah. And of course your mutton chops.

Art Manion:

I do what I can, including the appearances. So, yeah. Thank you. And again, thanks for having me on. It's great to have a chance to chat about VINCE. And we'll be doing some with the SEI and with CERT as well. But great to have a chance to reach your audience as well. So thanks, thanks.

Jen Ellis:

Art, thank you so much again. It was so good to have you on, and speaking of vulnerabilities-

Tod Beardsley:

Oh boy.

Jen Ellis:

Mr. Beardsley, yes. What's happening in vulnerability land?

Tod Beardsley:

Speaking of vulnerabilities, that CERT has been involved in, even.

Jen Ellis:

Ooh.

Tod Beardsley:

So all the InfoSec Twitter, is all a-twitter about the Ripple 20 vulnerabiliities.

Jen Ellis:

It sounds like an ice cream flavor.

Tod Beardsley:

It is almost as good as ice cream.

Jen Ellis:

No, I don't believe you.

Tod Beardsley:

Oh, they're good. So first off, confusingly, there are 19 vulnerabilities described in Ripple 20.

Jen Ellis:

Does the 20 refer to the year, though?

Tod Beardsley:

It does.

Jen Ellis:

Oh, look at that.

Tod Beardsley:

Which, took me a second to figure that out. But this is a set of vulnerabilities discovered, validated, and disclosed by a group called the JSOF research lab, or J-S-O-F. I'm not super sure how they pronounce it. I've never heard anyone say their name aloud, but they're legit. And they found two heaping handfuls of vulnerabilities in some pretty core networking libraries.

Jen Ellis:

Just to be clear, is that how we measure vulnerabilities now, is in handfuls?

Tod Beardsley:

Yes, that's right. Heaping handfuls. So a handful is maybe seven, a heaping handful is 10.

Jen Ellis:

Nice.

Tod Beardsley:

This is two heaping handfuls, so.

Jen Ellis:

This is good for people to know.

Tod Beardsley:

So these bugs, they all revolve around a set of network library code produced by a company called Trek. They're a company out of Ohio, as far as I know, no one has ever heard of Trek before, except for, I guess the people who work there and the people that work with because they work apparently with everyone, they provide a lot of, basically the TCPIP nuts and bolts to a bunch of IoT gear, industrial gear, OT gear. Anything that's not really a computer. So they do real-time TCPIP networking. They were founded in 1997 and their code has been, shipped and reshipped and repackaged. And it was like this whole-

Jen Ellis:

It sounds like Heartbleed all over again, no one was paying attention, but everybody was using it.

Tod Beardsley:

You know what, it kind of is. Because you have little tiny boutique companies that make like their one infusion pump, and that's it, and they use this thing. And then you have other companies like Cisco and HP that also use this stuff. So it's the entire range here. And so, these guys are real InfoSec heroes because they started this research back in September 2019, and they found they were just pulling apart IoT devices. We do the same thing here at Rapid7. And we're like, Oh, let's look at this, this like class of IoT device and pull them apart and see what happens. They found a couple of vulnerabilities and then they've noticed, they could have just stopped there. Find a couple of vulnerabilities, publish it on whatever the gear there was, move on with their lives.

Tod Beardsley:

But they didn't, they took that next step. That next, very fraught step of like, well, huh, this stack is written by some third party, not by this manufacturer. I wonder what else they're in. And it turns out everything. It turns out it's a ton of time of stuff. So they contact Trek. Trek has never been mentioned in any of the security literature. They're not like a common thing. It's not like VxWorks or open WRT, it's not anything like that. It's just like this kind of custom library that's good for low-resource devices. So they track this down all over the place and then, oh yeah, so they talk to Trek and Trek's like, what are you talking about? Because they've never had a security vulnerability discoverer come to them before they've never had a disclosure, anything like that.

Tod Beardsley:

So they say, okay. And then they work it out and it's the normal thing. They gave him 90 days and a couple extensions. Trek came up with some patches about 45 days in, which is pretty normal, right?

Jen Ellis:

Yes.

Tod Beardsley:

But then it came to notifying everyone and actually getting these patches out. And that was why we're talking about this here in June of 2020, and not way back in like December of 2019. So Trek and JSOF partnered up and ended up coordinating with basically everyone, CERT, CISA, ICS CERT, Intel, Cisco, HP. So everyone who has a vulnerability disclosure program or is involved in vulnerability disclosure, was involved in this. So it ends up being a pretty great story for at least as far as disclosure goes, because now if you go to Trek's website, they have a whole thing, like report a vulnerability here-

Jen Ellis:

Yay. I love it.

Tod Beardsley:

-and here's the current ones, here's all the stuff. And it looks like they've been doing it forever. It feels very professional, way they have it now. And they've only been at this for like a couple of months.

Jen Ellis:

So bravo to them for learning through the process.

Tod Beardsley:

Right? And like, and not suing anyone and not being jerks about it and not issuing statements after the fact saying like, oh, this is all irresponsible. Which some people do, still. Or just not ignoring it, and leaving it all on JSOF, to do all this coordination. They appear to have been involved like all along the way. So it's a really great vulnerability disclosure story. The vulnerabilities themselves are bad. I've mentioned this before and I might climb back up on the soapbox again. If you want to find vulnerabilities in core, networking protocols, read the RFC, read where they're defined and then implement that.

Tod Beardsley:

And you will find a bug in somebody. Because there's lots of goofy things that happen in the course of real networking. There's this whole mechanism of doing IP fragmentation, which doesn't really happen in the real world. But the spec was written at a time where we didn't really know, what networks were going to look like in the future. So maybe we'd have different networks with different, fatness of their cables. And so, you can't put through it like a normal IP packet. So you have to break it up into multiple packets. Well, this doesn't really normally happen in the real world, but this is a great place where it's the kind of code where, it anticipates a problem, it tries to solve the problem, you have one unit test to say, yep, we did it.

Tod Beardsley:

And then you move on and it's never exercised ever again, in the real world or anywhere else. So, that's the kind of place where you want to find vulnerabilities. And those are the kinds of vulnerabilities these are. They're like IP re-fragmentation vulnerabilities. They're like IP tunneling vulnerabilities where you have one kind of packet, stuffed into another packet. And then it's on the networking gear to extract it. You can imagine, this is the kind of place also where you're going to find bugs and that's exactly these kinds of bugs. And then the other thing is, a lot of this gear has been out there for a long time. It just kind of quietly runs and never breaks. And so it never gets fixed. You never see a patch.

Tod Beardsley:

One side note to this whole adventure with Trek is that, when they started on the path of contacting their downstream customers saying like, hey, we have these vulnerabilities, you want this patch? Or like, oh yeah, I forgot that we dealt with you 15 years ago. Maybe we need a support contract now? I think Trek ended up making some money out of this deal or re-upping support contracts and maintenance contracts.

Jen Ellis:

Good for them.

Tod Beardsley:

So good for them, I guess. So, note to other smaller networking companies, keep up with your customers and give them patches and they will remember to pay you.

Jen Ellis:

Yes. I mean, that's actually a pretty cool success story. I like that one.

Tod Beardsley:

It's a nice side story. But the downside is that some of these downstream companies went out of business. Or got subsumed by somebody else.

Jen Ellis:

Yes.

Tod Beardsley:

And then the product's got end-of-lifed or whatever. For whatever reason, these things are never going to see a patch ever. These are forever a kind of style bugs. So most of the time, as usual, you don't want to see these things on the open internet where anybody can just walk up and start throwing UDP packets at them.

Jen Ellis:

I like it. You make it sound as if it's like the packets of ketchup you get McDonald's you're just going to start lobbing them at people.

Tod Beardsley:

Yes. I remember throwing UDP packets at passing cars as a youth.

Jen Ellis:

Those were the days, simpler times.

Tod Beardsley:

Yes. But yes, that, generally you don't want to have happen, but of course it does by accident and there's never going to be a patch. And so the first warning, a lot of these like installation bases are going to get that they have a problem is that their thing just up and died and they don't know why. Which is probably the best case that you just lost them. Many of these have remote code execution vulnerabilities. So you can do things like install, crummy little crypto miners on these things, or use them as like a beachhead. If you've gotten into the network some other way, then you can stash your command and control on these goofy OT devices and things like that. So, that's it. That's the story.

Tod Beardsley:

So basically if you know that you're running somewhat specialized IoT, OT, SCADA, ICS gear, then take a look. Hopefully you know about it. If you don't know, it's fine, you can go scan your network. And it really shouldn't be on the internet. But if it is, don't worry, we found it for you. Come to OpenData.Rapid7.com and maybe it's maybe it's lurking in there too. So that's the story of that bug. Do we have time for the other thing for the Zoom update?

Jen Ellis:

Oh, there's more on Zoom? You do love an opportunity to talk about your Zoom blog.

Tod Beardsley:

I do. So you should go see my Zoom blog. Just Google "Zoom bugbears." You'll find it.

Jen Ellis:

Yes. The bugbear part is the important part.

Tod Beardsley:

Just real quick. Zoom announced, I believe it, yes, it was the 17th of June. Zoom announced that they will be rolling out end-to-end encryption for everyone, not just for paid customers. So this is a reversal from their initial plan that they weren't super talking about until-

Jen Ellis:

Weird.

Tod Beardsley:

Until a couple of people noticed.

Tod Beardsley:

Yes. There were some unfortunate things said about working with law enforcement, that seemed a little tone deaf, especially at the beginning of June.

Jen Ellis:

Yes, I remember that. Yes, where they basically made it sound as if warrants just don't apply to them.

Tod Beardsley:

Right? Yes. It's like, Oh, don't worry, we are incorporated in Gibraltar or something. And we have, yes...

Jen Ellis:

We have a get-out-of-jail-free card.

Tod Beardsley:

They're not, they're incorporated in the U.S. Zoom does tend to do the right thing, eventually. Which is derivative, but they do respond to public pushback. Right?

Jen Ellis:

Yes.

Tod Beardsley:

So making a stink about things on Twitter, for whatever reason, seems to work.

Jen Ellis:

Yay, power to the people.

Tod Beardsley:

Yes. And I'm sure there was a lot of side conversations, not on Twitter.

Jen Ellis:

Yes.