Tod Beardsley: Hi, everybody.
Jen Ellis: There we go! Tod and I are also super excited to introduce our very first guest, and a huge thank-you to Mr. Zate Berg for being our very first guest and taking this maiden voyage with us. We are delighted and honored. Zate heads up the information security team at Indeed.com. He's worked in technology for more than 20 years, 15 of which have been spent working in infosecurity specifically, including a brief stint at Rapid7 a million years ago, which I'm sure we might touch on later. His relationship with technology goes back even further, though, to his childhood in Northwestern Australia, where he lived on a million-acre farm. I don't think I made that up, right? It's a million-acre farm? That's a real thing?
Zate Berg: That's a real thing. It's just over a million acres. They call it a station, which is a like ranch.
Jen Ellis: Do they not just call it a city because it's a million freaking acres?
Zate Berg: No. Because there's only about 15 people on the entire thing.
Jen Ellis: Wow. So you went to school via school of the air over a radio, is that right?
Zate Berg: Yeah, so where we lived was so remote, we didn't have electricity and all of that kind of stuff. And they would fly our mail and everything in every couple of weeks. And so there was no school for me to go to. The nearest other human population to us was 125 miles away. So, every morning, Monday to Friday, I get on the radio for 30 minutes and talk to my classmates spread out over a very large geographical area. And then I’d do some kind of homeschooling stuff with my parents. And then the rest of the time, they would just let me roam the Outback, pretty much.
Jen Ellis: I'm so completely fascinated by the idea of this upbringing and particularly about the idea of somebody who grew up in an area with no electricity, who has then worked for 20 years in technology. How did you fall in love in technology? How did you get involved with it?
Zate Berg: Basically, I had a late start. We always lived in places where we were living in the middle of nowhere for a lot of it. And so after we lived there, I went straight from living in the middle of nowhere to living in the middle of the big city. And it was the first big city I lived in, and I was like 12 or 13. And I loved technology right from the get-go. And so I kind of made up for it. I dived straight into it and made up for it and out and immersed myself in it.
Jen Ellis: Fantastic. What a great, amazing journey. And that leads us naturally into talking about the specific project you have been undertaking at Indeed.com.
Zate Berg: Sure! So, the mission at Indeed is that we help people get jobs. That’s what we do. And as part of that, we basically aggregate and collect data and we build software to make that data available and usable to people to help them find jobs. We collect data and build software, and therefore, you know, software engineering and cybersecurity and stuff is a large focus. But we also have a software engineering organization who is really good, and like everyone else, they want to go fast. They want to be able to iterate and try new things and deliver value to our customers and pivot and change as fast as possible. And so that leads me to an environment where it’s very difficult for me to put any blocks in the way of the engineering team to say, you must pass this gate to continue going or you have to get this far and then come see my team and we’ll do a review.
Jen Ellis: I don’t know what you’re talking about, engineering teams love being answerable to security teams. It's super popular.
Zate Berg: Yeah, it certainly is. But that leads me to looking for ways to basically build security into their pipeline, into their process. We have quite a lot of scale to deal with, thousands of sub-products inside the whole website, lots and lots of different pieces of code, thousands of developers doing hundreds of production deploys a week. So how do I build software assurance? How do I get a good feeling that we’re catching as much as we can and reducing the risk as much as we can for that pipeline without slowing it down or putting in arbitrary blocks? And also without having to run a 500-person security team to manually look at all of those things. So that led us to build what we internally call our “AppSec Pipeline,” which is an automated DAST or dynamic security tool that integrates with how we build things to basically measure and look at every one of my external-facing applications within a 24-hour period and to take less than 24 hours to automatically detect a new piece of code or a new thing that someone has done, and scan it.
Jen: Where are you at in the project? It sounds fascinating.
Zate Berg: We've been working on it honestly for two years off and on. And we're at the point now where it's gone from a two little pieces of code that work together to 16 pieces of code or applications or APIs that work together. And like I said, we can scan everything that is in our GitLab even when someone creates a new thing within 24 hours. And we’re actually at the point now where we’re starting to say, okay, how do I take pieces of that and bundle it into smaller pieces that an engineer can run on their laptop while they’re building things, or a QA person can attach to the QA thing they’re doing and automatically tell them about vulnerabilities? It’s also been heavily integrated with our bug bounty so that I can take a finding from our bug bounty on one piece of software and say, okay, let me look for that same thing across everywhere, and oh look, I have another 60 instances of it that would not have gotten found without the bug bounty.
Tod Beardsley: So, Zate this, automated desktop scanner thingy. When you run this thing, can you give me an example of the kind of finding that a developer might see? Is this something that says, oh, hey dawg, you just wrote yourself a SQL injection, let's not ship that. Is that where you're at with it? Or is that a goal?
Zate Berg: That's the goal is to get that on the developer endpoint view. Right now, it sits on the build pipelines. When people pushing code and that kind of stuff, we will pull a copy of it to the side, detect the changes, and run our dynamic analysis. We started off trying to use some open source, off-the-shelf tools for this thing. And we had a requirement that it not add more than 15% overhead in terms of time to build. So a lot of the open source products out there were not very good at their coverage on understanding everything that an application is and were taking a very long time to go through and do the dynamic part of it.
Zate Berg: We wrote a couple of things to build a very accurate picture of the application. And also then we wrote basically our own custom vulnerability scanner. And we started off by supporting what are our most common findings from our bugcrowd and then we added a few other different pieces. So it does cross-site scripting, open redirects, SQL injections, the whole gamut of the traditional stuff. It doesn’t do well at some things that require really good knowledge of the application for like IDOR and those types of things, but we have other little pieces we’ve built for them.
Tod Beardsley: Neat, and I assume you have some authenticated API checker as well running at the same time? Cool.
Zate Berg: Yeah. So, it’s made up of a bunch of different pieces. It started off with a thing we call WES, which would go and build a dynamic syntax tree, basically, of an application source code. It would go through and crawl all of that and identify methods and URL endpoints and parameters and all that kind of stuff to say, hey, these are all the things on the outside that someone could touch and that we should absolutely run a scan on and check. That also allows, as a side benefit, actually, it builds a very cool way for our developers to come and take a look or our software engineers to take a look and say, hey, this endpoint with this parameter and they can click on and drill on down to that and it will actually show them in what code repository and in what line of code that is.
So, they are actually using that entire product. And we discovered this by accident because we thought this was just for us, but they found it and started using it, and when we took it down for some maintenance, we got a bunch of, “Hey, where is that thing? I like that thing.” But it allows them to literally go down and see which piece of code is responsible for which endpoint if we get a finding or a bug—even just a QA bug. We don’t really do this, we could go in and say, like, who checked that code and that stuff. So it really gives us a good piece there. And then we added to that another little piece we wrote called NOMAD that is like our own version of a spider using Selenium and Headless Chrome and a bunch of other things to help build that outside view of the application and then we combined both of those things into a product called ERGO that lets us look at those and combine them into one object that can then be fed through a scheduling app into AVA, which is the vulnerability scanner. So it takes an input of here is everything you need to know about the application and then AVA just runs through all of those things and does all of our scanning on it.
Tod Beardsley: That's pretty cool. I mean, that sounds like you've got a pretty great process over there at Indeed of always raising the low-hanging fruit. You get the lowest-hanging fruit and then you get the next lowest-hanging fruit so you don't have to wait around for your bug bounty to get a random hit. It sounds like you're doing a lot of real forward-looking auditing and just like always automating up like the next thing.
Zate Berg: Yeah. And they’ve had a bug bounty here since before I joined. It was one of the first major undertakings they took in the whole software assurance space, and it’s transitioned from it’s finding a bunch of things of things and we rely on that to look or stuff to it’s a health check, a sanity check—it’s an oh, you missed this thing kind of a deal. When we find a new thing out of our bug bounty, we write the check for it and put it into AVA so we can literally go through within an hour or two and find that across just about any other code base. From the necessity of, you can’t stop people pushing code to production, people can literally create things and go all the way through and push it out to production. We needed to understand when that was happening. We would crawl our DNS, we looked for new repos. The good thing is that engineering helps us by saying that all the repos, when created, have to have some metadata in there that talks about is this an external service, who owns it, all that kind of stuff so we can quickly and easily track down which teams own something when we find something and get in there and get it fixed quickly.
Tod Beardsley: Yeah. And it sounds like your bug bountying right, too, right. You're not relying on a bug bounty as your primary source because that way leads to madness. It’s more of a safety net. If you don’t happen to see a thing or they find something novel over there and I’m like oh, okay, cool, I can re-implement this as like an automated check. Never have to worry about it again.
Zate Berg: Yeah. And when I joined Indeed, they had about eight generalists and I have almost 40 people spread across the entire org now.
Jen Ellis: When was this?
Zate Berg: About three years ago.
Jen Ellis: Okay, so from eight to 40 in about three years.
Zate Berg: Yeah, a lot of hiring and that kind of stuff. But it’s not just appsec, you know, I’m responsible for the security of pretty much anything cybersecurity-related at Indeed. So I build out multiple different teams.
Tod Beardsley: And what does Indeed use to find qualified candidates?
Zate Berg: We use all kinds of things. I’ve actually got some blog posts coming at some stage on how I went about building that team because we did some really interesting, cool things around not only finding candidates, but making sure we got the right candidates.
Jen Ellis: Oh my god, we’re totally having you on in the future to talk about that. But back to your appsec stuff—so it sounds like you’ve made some pretty amazing progress with it. Are you happy with where you are? Are you happy with how things are going?
Zate Berg: I think we're happy that we've gotten to base camp, like if you’re climbing a mountain. I’m happy we’ve gotten to a good piece where the work that we’ve put in—and like I said, this took two years off and on and a couple of very savvy software developers on my team to build this. And the fantastic thing about this, and I guess this goes into how my team operates, I didn’t sit down and say, “I want this thing.” The team went, well, if we can’t go in there and put gates in places and stop stuff, how are we going to solve this problem? And a couple of members—there’s four people involved in this—they sat down and thought about the whole thing and they built the first little bit, the WES and the AVA, which we call WASABA, which we’ve open-sourced out there, the basic components of that. And they built that first but then over the next year and a half, they added a way to do authenticated scans, a controller in the middle that schedules things that the developers and software engineers can go and take a look at when their application is going to be scanned or was scanned. We built a GUI frontend to reporting and integration with Jenkins and a bunch of other things in there.
So I feel like that piece is at a good place, but the next phase for us is, how do we take the core of that and keep the speed but shrink it down and make it available to a developer on their own work station and available to QA. So the next piece is, I think we call SID PROXY, which is taking that AVA engine and saying, I’m going to build a little piece that hooks into how QA runs their checks. So, they automate a whole bunch of their QA checks with Headless Chrome and Selenium and all that stuff, and we were like hey, we’re doing a bit of that on our side. What if we wrote some stuff that just plugged into how they do their normal QA checks that are very curated and fully maintained and everything, and those QA checks ran through a proxy that helped us build that view of what the app is. And then we ran it through AVA in parallel and put that as actually part of the build process, and what if the build failed if it found significant AVA findings, which was kind of that next step for us. And that’s where we literally are right at this moment, is testing that out on a few different applications. And it’s opt-in right now. If an application owner or a product owner wants to opt in to this stuff, and what we find is many of them do, because they fully grok that doing this and failing it out in that way and treating these security bugs as just another form of QA is leading to less rework and problems to deal with down the road.
Jen Ellis: So you say that they’re finding that. I’m imagining, though, that there’s been some level of education or persuasion that’s gone on, at least at the beginning of the project to get people to give it a go and see the benefit of it. What was that process like for you?
Zate Berg: So, initially the process was, let’s pretend nobody wants this at all, ever. How do we build it, and how do we make it work where we don’t need anything from them? And so that was the initial piece. But since they’ve discovered, hey, look, it as a whole UI I can go to and look exactly at all the endpoints, it gives me a few of my applications we’ve never had before. And then they just naturally find, hey look, I can see the findings from my applications, these are interesting, do we have these in our backlog? And like I said, it’s been a natural evolution in there and now we’re at the point where we’re definitely educating and seeking out people and talking about and say, “Hey, we have this thing. Let us work with it.” It was very important to understand the potential things that are going to make them kind of go, eh, that’s weird or that’s going to add too much time to my build times. If it was taking an hour to build something and we add another hour on top of that, that’s just not going to work. And I think it’s easy to say, but oh my god, security, security is important. And it is, but building software is what we do as a company, not securing that kind of stuff, so want to build that security into the existing process and not have ruined too much.
Jen Ellis: You said that you started off by assuming no one wanted it. As you were going through ideation and development of the project, did you include representatives of your constituents?
Zate Berg: There were some teams we worked really close with early on that the people building this just work with those teams who were very on board. Some of those teams are teams internally who do experiments, anyhow, who were very experimental in what they build and that kind of stuff. And they tend to build, for instance, a lot of stuff on a very standard platform. One of the pieces with this that is harder is that you have to build some of these elements to be reasonably close to what it is you’re scanning and to get the automation, you need the consistency. If your developers are just randomly picking whatever they want off the shelf to build applications, building something like this is much harder. We were lucky in that we had standardization in, for instance, the Java frameworks that they’re using for these things or the Python frameworks that they’re using. And so we were able to pick the two or three that just about everybody uses that we’ve already worked very hard to build security things into and say, we’re going to make this work with them. We want to open source most of this, eventually. But part of the work we’re trying to do on this now is say, how do we make this more modular so that if we do open source it, other people can just write up a plugin to say, WES now supports Golang or Node.js or some other thing, and AVA now supports this other kind of check in different ways. So, we’re trying to build it to be more modular for our own use as well because as software engineering advances and evolves, they pick up new things. For example, we’ve had to go back and engineer and put a bunch of more stuff in there because a lot of our teams are doing far more JavaScript- and React-flavored things and initially it didn’t support some of that stuff. We don’t want to leave it static because our coverage as we go along in the engineering org evolves, our coverage will shrink until we’re only covering the legacy things and not the new things. And so that was important as well.
Jen Ellis: So, it all sounds as if it’s gone pretty damn smoothly, but I’m going to guess that nothing really ever goes that smoothly. What challenges have you encountered as you’ve gone through the process?
Zate Berg: I think a lot of the really early challenges were, it took exceptionally longer than anticipated. I said, hey, look, here’s this problem, you’ll go away and work at how to fix it, and they came back a little later and said, hey we think we want to build this thing. Okay, well how long do we think it’s going to take? They’re like oh, a quarter, three, four months. No worries. Okay, sweet. It took about two years, but the reason was because they were going to use a bunch of off-the-shelf things and just kind of tie them together and as we got into it and realized, it’s not good enough for it to just find vulnerabilities, it has to be modular. It has to detect new things really quickly. It has to be able to run across thousands of potential endpoints within every 24 hours. So scale and speed was one of the biggest things to come across to say, how do we build this in such a way that it’s going to be very modularized? It’s going to run a lot of things in parallel. This whole thing basically exists on a Docker swarm that is pretty much just for this thing, with bunches of nodes and stuff. And when migrating it, as the business does stuff with migrating and looking for places to keep it, right next to where our software engineers are working.
But yeah, scale was probably one of the biggest things, and we had to do a rewrite of AVA at one stage where it was just taking too long to do certain kind of checks. The other thing that was really apparent is that this is not comprehensive and 100% of all the things, like when you go out there and buy an off-the-shelf scanner and that kind of stuff, you’re trying to get, say, I want it to cover all the different bits and pieces. There’s some low things that we were just like, we’re not going to spend time looking for and worrying about that. We’ll just let the bug bounty people find those particular problems. We want to build to solve the harder things.
An example of that, is, we actually got partway through and said, I think we need a reflected blind XSS checker for this. So we then went and wrote a bunch of pieces to put a listener out in AWS and it injects back some JavaScript code to tell us where that thing detonated and all that kind of stuff. That was pretty tricky to get done. It was very specific kind of XSS that we felt was very dangerous, whereas we didn’t put as much effort into just some of the simple reflected XSS. That’s actually pretty easy to find in many other ways.
Jen Ellis: When you were dealing with things taking longer than you anticipated, given that your team is very busy, you’ve got a lot to do, you’ve got day jobs, how did you keep momentum going? How did you keep everybody in it?
Zate Berg: One of the things I’ve learned at Indeed, and especially with plenty of other places, is that if it’s not what somebody focuses on every day, then it’s going to fall to the bottom of the pile. After we worked out it wasn’t going to be done in that quarter and that stuff, I sat down and went, alright, I’m building this team and I’m hiring people to solve all different kinds of things. I think I want to keep at least one of these people who seems to be really passionate about this thing, but this is what he’s doing right now, and it may take him a year, and I’m prepared to invest a person for that amount of time to do this thing. Indeed is very big on building vs. buying for a lot of things, so it wasn’t hard for me to convince people that I’m going to take this resource and dedicate them to do this thing and then they’ll just get help off and on from one or two other people and people will rotate through. But having that one architect on there, and that one person who was really doing that, and that’s kind of how we went from having WES and AVA, which was kind of that first little iteration, to now I have 15, 16 parts of this big thing was dedicating one person to it.
Jen Ellis: What was the biggest thing you learned from the process?
Zate Berg: For me, as the manager/leader who didn’t write a single piece of code and honestly didn’t architect or do any specific stuff and I get to talk about it, and that’s great, but I didn’t build it and that kind of stuff. For me, it was that it was really great to see that left alone with a problem, a kind of core problem to solve, my people just got together and worked it out and solved it in some pretty interesting ways. And now I feel very confident in taking some of the same people who worked on this or some of the people from other places and saying, hey I want to do that thing there, but I want to do that for this other problem over here. How do we sit down and fundamentally rethink how we’ve done this thing and build something that solves this problem at scale for us? I learned that I can absolutely rely on my people to get stuff done but we also learned that if you build stuff that helps solve ancillary problems for your engineering team, that might just solve some security problems and get some adoption, that maybe it’s important to spend some time and help them solve their other problems and then just sneak in a bit of security stuff in there, too.
Jen Ellis: I’m having to resist making a “Field of Dreams” reference here. This is my last question. If you had to give yourself a piece of advice on taking on a project like this, what would it be? Or if you were giving other people advice, what would your No. 1 thing be?
Zate Berg: I think my No. 1 piece of advice here is software is magic. We kind of live in this world where outside of our reality, you go into the world of lord of the computers and stuff, you can kind of control a lot of things. You can step down a level, when you’re running up against the laws of physics inside your program, you can step down a level and change those things. And we had to do that a little here for some of the speed stuff. We had to say, okay, we’re running into this problem at a higher level, what if we stepped down a level and rewrote some pieces there? And so understanding that, you know, you can build this big thing, and when you identify your bottlenecks, that as long as you’re prepared to dig in a little bit, you can really get down in there and solve those particular bottlenecks, which is what was really key to making this something that our software team would adopt.
So understanding the limitations of the things they said were out of bounds, or hey, this would make it too difficult for me to use this for my project, let us put some really good priorities and borders that were not purely certificate security related and let us dig down in there. So initially, I thought that was not useful when I was talking with the team. I’m like, okay, I get the speed thing, but I want it to cover all the vulnerabilities and that kind of stuff. It’s far more effective in where we ended up than where maybe my thought pattern may have taken it. Initially I was looking for accuracy over speed, and what we got at the end was we cover 50% to 75% of all the really important things, or we cover all the important things and another 25% of mostly important but we cover it really well and we cover it at speed, that has led to way more adoption in the enterprise than if we had stuck the other way. I’m glad that I stepped aside and let my people build a thing and did my job, which was not building this thing.
Jen Ellis: I think it’s incredibly challenging when you hit a roadblock to take a step back and look at it at a different level from how you’ve been looking at it, and to seek new approaches that are not within the same paradigms as the original approaches. So the fact that you guys were doing that and coming at it from a different level was pretty amazing and a good example for others to try to emulate, even though it’s pretty challenging. I’m not sure how great I am at it. Tod, did you have anything else that you wanted to jump on before we wrap up?
Tod Beardsley: No. I think it’s great, and I really like the analogy to software as magic because you get to change the laws of physics when you run into problems. I’m reminded very much of Scottie’s approach to engineering on “Star Trek.” It’s, well, you’ve got 20 minutes to fix this problem. It’s going to take two days. Well, okay, I can give you 25, then you go for it. I agree 100%. It’s different, right? Like Zate said early on, I want to build a thing. Well, Zate, I feel like you built a thing. You’re living the dream. It’s hard for security people to switch over to making vs. breaking, but it sounds like you’re doing it dude.
Jen Ellis: Yeah! And successfully, as well.
Zate Berg: I think you can use a bit more polish, but I think we're going to over the course of the rest of the year, try to put more of this out there for other people to get hold of and use. Like I said, WASABA is available on our Indeed security GitHub right now with some other tools we build, and it’s an important part of the things our team builds that we should be able to put it out there for others to use. We have a few other unique solutions that we’ve come up with that we’re going to put out there.
Jen Ellis: Well, this gives me a perfect opportunity to say, will you come back on when you have open sourced everything, and you can also talk to us about how you go about hiring at the same time.
Zate Berg: Sure, I’m happy to come back and talk about some of those other topics. I have a number of things I’m going to write some blog posts on around managing teams and building teams and looking to deal with burnout with teams that I’m looking to talk about anytime.
Jen Ellis: That sounds great. Thank you so much, Zate. We really appreciate it. Thank you for being our very first guest and we look forward to speaking to you again.
Zate Berg: I really appreciate it. And it can only go up from here!
Jen Ellis: What are you saying about me? Take it easy mate, bye.
Tod Beardsley: Well, hey, everybody! Welcome to this week’s edition of the Rapid Rundown, with me, Tod Beardsley, and Matthew Ryder. This week, we’re going to be hitting three headlines from infosecurity.
First up, we’re going to talk about BlueKeep, everyone’s favorite named vulnerability this week. BlueKeep, aka the only CVE number I know by heart (CVE-2019-0708). This was a bug in Microsoft Remote Desktop Protocol. It was announced a few weeks ago, so you might be thinking, “Why are you talking about it this week? This is not news.” Well, actually it is. This week, CISA, the Cybersecurity and Infrastructure Security Agency, which is part of the U.S. Department of Homeland Security, issued a new advisory on this. And basically they say a few things on this that are kind of new. It does, in fact, affect Windows 2000, and they seem to be implying rather strongly that they have a hold of a remote code execution exploit. So this is just your weekly reminder to please, please, please, use some continuous vulnerability scanning in your enterprise and see what kind of RDP is floating around your network. Turn it off, if you can, patch if you can’t, and best of luck to you.
Moving on, I also want to talk about fresh as of this recording, we have an out-of-band patch for Oracle WebLogic. Oracle WebLogic is built on Java. It’s a piece of middleware, usually, sometimes used on the front-end, though. And it has a bit of a patchy history (no pun intended) when it comes to Java deserialization. If you’re an Oracle shop, you’ll want to check out CVE-2019-2729 (unlikely to remember that after this podcast), but it is a pretty good vulnerability. No exploit, yet, but I wouldn’t be surprised if an exploit did not surface in the next two or three weeks.
Matt Ryder: I think what’s scary to me about that is if you dive into the CVE score, you don’t need to be on the network, you don’t need any privileges, there’s no user interaction, it’s a low complexity to exploit. So that's a scary one.
Tod Beardsley: Yeah, Java deserialization bugs are building-block exploits at this point. They’re pretty well-understood by the black-hat attacker, criminal community and penetration testers. So that’s why I say an exploit is pretty imminent with this.
Finally, rounding out our top three is a story we saw pop up on the Register about some research from Avast, Stanford University and the University of Illinois at Urbana-Champaign, home of the HAL 9000. Turns out, IoT is still awful! Avast, along with these other university researchers, surveyed their customer base with their consent, which is interesting, for their home IoT infrastructure and found a boatload of default usernames and passwords being used in local, consumer IoT. We always know this is a problem, and we know attackers know this is a problem, this is the reason why Mirai was a thing. That was a worm from about three years ago that ran around to look for DBRs that were hanging out on the internet with open telnet and the very short password list. The interesting thing to me is this is one of the few cases where we have a pretty decent sample size of the Avast consumer base that also looked at password auditing. As you know, trying usernames and passwords over the internet on other people’s stuff is no bueno. That is illegal in anywhere there are laws. But Avast is able to look at this with user consent for a local scan so that’s pretty cool. That’s a good bit of research from them. Hat tip!
So that’s it, for this edition of Rapid Rundown! I’ve been Tod Beardsley.
Matt Ryder: And I’ve been Tod Beardsley’s British friend whose off to patch his router.
Tod Beardsley: We’ll talk to you next time.
Jen Ellis: Thanks to the amazing Tod for giving us an update on everything we should know in security this week, and thank you for tuning in. In the next episode, we’re going to have my good friend Lee Brotherston talking about how to build a security program from scratch. Don’t miss it!
