Last updated at Fri, 01 Dec 2017 20:46:38 GMT
Welcome to Defender Spotlight! In this weekly blog series, we interview cybersecurity defenders of all varieties about their experience working in security operations. We’ll inquire about their favorite tools, and ask advice on security topics, trends, and other know-how._
In this edition, we're featuring Paul Halliday, a Senior Software Engineer at Critical Stack, recently acquired by Capital One. Paul is an avid open source author and staunch OSS supporter (author of the SQUERT project) with a history entrenched in infosec.
Let's see what he has to say!
Tell us about yourself, and your history working in security operations.
When I finished high school I landed a job as a carpenter and industrial electrician for a contracting company in Ontario, Canada. While I loved this job, a rare opportunity arose for my wife in a small town on the east coast that would really help bootstrap her career. Having a really solid skillset and over 10 years experience in commercial construction I naively assumed there was very little risk in a relocation for me. I would just pick up where I left off. Unfortunately, this was a different place with a dramatically different economy.
I can say this: It was good thing I had a hobby, but we’ll get to that shortly. There were very few jobs and a tremendous amount of competition. I had no post secondary education and all of the formal qualifications and certifications I did have were in a field that didn’t exist (at a sustainable scale anyway). After numerous failed attempts of not even making it on the short list for interviews, I decided to enroll in the IT program at the local college so that I could at least get in front of someone to plead my case. They offered me a job midway through my second year.
I went on to design, build, and manage that College's first Network Security Monitoring (NSM) infrastructure using commodity hardware and open source tools. The system watched over 21 locations (13 campuses) throughout the entire province.
Once everything was up and running, I quickly ran into my first real problem: I had no incident handlers. One of the core tools that this monitoring system was built on was a project called Sguil which was chosen because its distributed architecture was a perfect fit for my use case. It also fit perfectly into my budget which was $0.
There was a pretty big drawback though. The client for this tool was, well, a little esoteric. It was made for hard core incident handlers, and unfortunately the only resource I had at my disposal were campus technicians.
This is when I started conceptualizing and writing Squert, a simpler incident handling UI that could access Sguil’s backend and display the data in format that would be easier for the techs (who were generalists) to digest. I had no prior web programming experience up until this point (and it showed) but within 3 months I managed to throw together a working prototype that was easy to understand.
It was a tough go and took a while to get everyone onboard. But once I could easily articulate and show the techs what was wrong at each of their sites, buy-in came quickly. Within 6 months we managed to clean up a very messy network.
Whenever I track my trajectory I often laugh to myself. How on earth did I end up here?
What you are working on these days?
In May of last year, I joined a startup called Critical Stack. I work with a small team on some interesting problems. One of our core projects is an "Easy-Bake oven" that really helps lower the bar of entry for defenders. We have created a custom RHEL-based operating system with turn-key clustering support that maximizes resources and minimizes downtime (thanks lilTone).
This system will help you deploy complex applications and make sure they are setup and running correctly, operating optimally, scale accordingly, and of course securely all with the click of a button.
Can you tell us about a moment in your career where you were proud to be working as a defender?
This would definitely be when I was working for the NSCC. Coming into an organization so young that has experienced rapid growth but hasn’t yet tipped; that is to acknowledge that security is important, is very challenging. In the beginning, there was very little support. I think I threatened to walk out at least 4 times. Thankfully, the more I picked away at the label the more I exposed what was truly going on inside. Once enough people got a good look, we reached critical mass.
Being the catalyst to transform this same organization into one that is not only forward thinking, but embraces the relevance of information security and “gets it” gives me an enormous sense of pride and accomplishment.
Every time we can help an organization embrace this stuff, the stronger and safer we all are.
In your opinion, what are the most important elements of implementing a successful security operations center capability? What do companies struggle with the most?
If starting from scratch:
- Enough people with the relevant skill sets. Start small with the caveat that growth is an option if necessary. Focus on team cohesion and fit, especially in the beginning.
- Enough money to sufficiently fund the endeavor for at least 5 years before you start trying to calculate ROI
- Enough buy-in to let it evolve organically in those early years
Anecdotally, I think medium to smaller organizations can struggle a lot with this due to the lack of a proper supporting organizational structure or just poor leadership.
There are a lot of orgs that don’t have the necessary leadership roles (CIO, CISO) in the right places, or even at all. This can make starting and maintaining a SOC next to impossible.
What are the top things defenders should be worrying about today? What worries you the most personally?
Beware snake oil, pew pew chart junk, and MSSPs.
Recently, I was doing a talk on OSINT and early in the talk I asked the audience if they were familiar with Bro, one of the most popular Network Security Monitors. A single hand was raised. Slightly confused, I followed up with: How many of you have heard of Bro? Again, a single hand.I then followed up with: Are there any security professionals in this room? I was in MSSP territory.
Now, outside of making quite a few people very rich, this model of providing security services is inherently flawed as it is impossible for the level of service to scale proportionally with an ever increasing customer base. There has to be a ceiling, or sacrifices.
If you are involved in security and use an MSSP to help you check an audit box this is fine. However if you are a security professional and have no idea what Bro is and you signed off on an MSSP then it’s impossible that you would have been able to objectively do so.
As someone living in a rural area, I am very familiar with the “last mile” rule. More specifically that there are nuances in the service that I will receive because the conditions are outside the “norm” and the amount of effort/resources to provide “that norm” are cost prohibitive.
Our weakest links in our organizations are very much like that last mile. They are too arcane, nuanced and require real effort and resources to protect. An MSSP can’t touch it.
What are some of the best industry events to attend and why?
The small ones. I have yet to attend a BSides that I didn’t like. I think the smaller venues have the tendency to be more laid back, people more relaxed, the speakers more approachable. Random break out sessions are common and if there is an after party it is usually equally informative. Defcon was pretty great circa 4 and 5 but I hear they have gone downhill.
What hobbies do you enjoy?
Almost forgot about the hobby! Growing up my father worked for a company that was owned by IBM. We always had some sort of computer in the house, the first was probably an 8088. I had an incredible fascination with these things, especially taking them apart. Once Ebay showed up I would buy up anything exotic I could find. I picked up numerous Suns, DEC Alphas, SGIs, HP(PA-RISC) even Cisco Routers. Most of these boxes would come pre-loaded with their respective operating systems which I would then play around with until I knew how to do something with them. I would set up Bind, httpd, ircd or a sendmail server and then toss them on my network and try to break them.
That concludes this week's Defender Spotlight. Have any other questions for Paul that we didn't ask? Connect with him on Twitter to ask more!
