Last updated at Tue, 05 Dec 2017 21:41:59 GMT

Welcome to Defender Spotlight! In this weekly blog series, we interview cybersecurity defenders of all varieties about their experience working in security operations. We’ll inquire about their favorite tools, and ask advice on security topics, trends, and other know-how.

Today, we're talking with Ben Hughes. Ben has an extensive background in network engineering, IT operations, cyber security, and all the in between! Currently doing the security thing at Etsy, he's previously held a variety of positions at companies like Puppet Labs, Canon (Europe), Empora Group, and more. Let's hear what he has to say. :)

Tell us about yourself, and your history working in security operations.


Hi! I'm Ben, originally from never all that sunny England, now living in San Francisco, by way of Amsterdam, Melbourne, and Portland. Not bad for someone who only got a passport to go snowboarding one time. I fell in to security the way many of us did in the late 90s, mostly by enjoying the fact that we suddenly had Internet access at home and there we even fewer security professionals then than there is now, so... "exploring" other people's networks was a very available option. This is pretty much how I learned Linux as I was an impressive academic dropout. I don't recommend this path for all, but like Hunter S. Thompson said, it worked for me.

Since those heady 14.4k modem days, I've flipped between being a systems engineer (sysadmin, operations engineer, whatever it's called this week), a network engineer (I will break BGP for free!) and security work. Which means I've had a good understanding of how it all works together and some of the challenges that security teams often ignore, and means I'm an absolute master of nothing!

Can you tell us about a moment in your career where you were proud to be working as a defender?


Pride as a defender is a hard topic, because of the asymmetry in attack/defense. If the Target breach was stopped at the very first point of entry, it wouldn't have stopped a however many billions of dollars attack that it turned into, it would just have stopped a small attack. So you often only get the magnitude when it goes horribly wrong. But enough of my hard done by heroics byline.

Way back in the early 2000s, I had the pleasure of working at a lovely small local ISP in Brighton, England. Like a lot of ISPs, we hosted machines in a co-location facility so they could take advantage of our impressive 10mbit Internet connection (early early 2000s...). On that network I put a tarpit (LaBrea), which just listened for any connection and then slowly eked out enough of a protocol to keep the connection alive, all the while alerting us of the connection. With this, due to the exploit du jour, we were able to notify many many customers of worms like Slammer and other such unpleasantries long before they were aware of them themselves. They were always very grateful and pleased, which was gratifying.

What are some of your favorite products, software, or tools that you use on a daily basis? How do they make your job easier?


Like so many of the jaded persona's I have come to call colleagues in the industry, I'm rarely a fan of vendors and the good work they do. That being said, Duo have consistently made my world better. Having two factor authentication that doesn't suck and isn't such a pain to users is really powerful. They focused on the hard bit of security that everyone seems to think isn't important, the UX. If people find it more annoying to use your measures than not to, they'll do their best to avoid it, which defeats the point of having it.

I'm excited by what my friend Jesse is doing over at TechMate. He used to be Dropbox's corporpate security person, and dealing with managing all their 100s Mac laptops. Upon finding how painful this is, he decided to co-found a company to fix that itch. Puppet + Simian/Munki have served both Dropbox and Google well, but it's not for everyone and managing Macs is still far too hard due to not having a good package manager, and how it can't decide if it wants to be a for everyone GUI OS or a UNIX based vision of a future.

What are some of the trends in the security industry that you find encouraging?

I think the rise of bug bounties has been good. The debate on "responsible" versus "full" versus pastebin disclosure will probably never die. But companies accepting that there may be bugs in their stuff, and if you happen to find them, tell them and they'll give you some cash is a great step compared to just denying everything and taking them to court.

Now that Katie Moussouris moved on from Hacker One, I'm really excited to see what her new company, Luta Security, does with leading people on disclosure and bounties. Plus she's a fantastic singer. My only unanswered question with a lot of the bug bounty models that no one is really talking about is how fine everyone seems to be with effectively offshoring security work. Normally there's huge uprisings when offshoring occurs, but for once there hasn't been. Perhaps we've found the right economic model for it with bug bounties and everyone is a winner, which would be fantastic.

The, how can I say this courteously, rise of ThreatButt and the fall of "Threat Intelligence" and IoC sharing ventures is in my eyes, a good realisation of what's actually valuable in security and what's RSAC snake oil. If changing a single bit in a file means it evades your MD5 based IoCs, what's the point? If I see another "attack map" on a dashboard, with someone thinking that this increases security in any way, I will probably take up crocheting and sell my wares on Etsy to make rent.

What are the top 3 things defenders should be worrying about today? What worries you the most personally?


Everyone's environment is different, but I can speak on three of the things that help keep me awake at night.

  • Endpoints, Macs have taken over, and they're just a flimsy NextStep clone with a nice window manager, but woah, look at all those processes running, what do they do. The Windows 10 kernel seems to have a lot of cool security features, but OSX is seemingly a little lacking, and that's ignoring management of OSX fleets, which is years behind Group Policy. OSX scares me, because it's like handing everyone a Linux laptop (but that can play MP3s and connect to wifi), and expecting them to manage it and keep it mostly safe. It's how I would definitely compromise an organisation, were I on the offensive side, and much better.
  • Threat modelling. A lot of press is given to CHINA and the Snowden/NSA woes, which has everyone worrying about encrypting all their comms so that government agencies can't spy on them, blissfully unaware that they mostly don't care about you. You're much better working out who is most likely to attack you, who would profit from it, and put your defences there, rather than trying to boil the oceans against some attacker with a larger budget and remit, and zero interest in compromising you. Moreover, there's lots of media scare stories over 0-day and that whole Wassenaar thing, when 99.99999999% of organisations have machines that are months or years behind patching. On top of that, whilst patching is still important, and 0-day does exist, I can't help but feel that a lot of compromises don't require exploits or reversing or anything. They're just logic bugs, or trust issues. Things given too much access, etc etc. So a focus on what the reality of an infrastructure compromise would be, rather than a spreadsheet or Nessus PDF of version numbers.
  • Once your house is all in order and you have logs galore, etc, how do you know what a positive signal looks like? Running game day/"red team"/pen test, whatever you want to call them, to as much as possible, recreate a real attack, so you get experience in either knowing what a worrying silence or a stampede of alerts feels like. Being able to get to a place in between complete alert fatigue and convinced that you've never been attacked is the ideal end goal, though security rarely turns out that way.

 

What advice would you give to someone getting started in security?


I'm bad at this part. The industry has changed so much since I were a lad. It's now a lot harder to get a job without a university education. There are even places focussing on it such as the excellent programmes offered by NYU Poly. Security wasn't even a thing when I was in education, which is how I learned a lot about it. (:

A while back, I was attempting to give such sage-like advise at NYU's women in security event, and I was touting the oft heard maxim of the moment, that security has a skills shortage, and there's so many jobs. Wonderfully, the woman I was speaking to smacked some reality into me. There are a lot of those roles, for people who've been in the industry for 5-10 years, but it's much harder when you're just starting out and don't have a full picture of what you want to focus on.

I do love being wrong; it means I have an opportunity to learn. So with this in mind, get that foot in a door, get some experience, find what you enjoy, find what you don't enjoy. Due to being quite niche in the technology industry, security is a very hard thing for recruiting, so the majority of hires come via referrals. So get your name out there and use and build whatever network you can.

What are some of the best industry events to attend and why?


Best is so subjective in this industry... Personally I've been trying to speak at events that aren't directly security related, as everyone in the security echo chamber has heard it all a thousand times and it hasn't magically made us more secure. So I've been to things like the numerous DevOpsDays events and Velocity conference to try to preach to the not yet converted and get other people talking to and about security.

I've always wanted to get myself to Kiwicon down in New Zelia, as that seems a fantastic conference both in terms of the effort the organisers put in, and the quality of the talks presented. That and it's tiny cousin Wrong Island Con, wherever it may be this time around.

A ton of great and honest insight from Ben! If you have any other questions or just want to get in touch, you can find him on Twitter.