Ashish Rajan: Hello, and welcome to another episode of virtual coffee with Ashish with clock security podcast Its a weekly video podcast where we talk about different topics in cloud security. Today’s topic is about container security in AWS.

Before I’m going to talk about, I think it’s worthwhile calling out that for the past few episodes, we’ve been talking about security leadership. We’re doing some testing in AWS and now we’re kind of going into deeper. Let’s say different technologies that affect cloud and containers is one of those from which has been really popular.

And I’m so glad I found Ben to come and talk about this. He’s done a few talks about this already, and I thought he was a perfect guest to bring in for people who know him already or it’s, I’m pretty sure there will be a lot of few because he’s been contributing in the open source I guess, free community space for awhile.

Sharing his knowledge, which has been great. Having people like him do public speaking really helps people like us and lot of others learn. So without further ado

Ben Tomhave: Thank you. Good to

Ashish Rajan: be here. It’s my pleasure to have you. And I’m so glad I got you in, cause I’ve [00:01:00] been getting a lot of questions about containers and container security, but it’s really easy to deep dive into this rabbit hole, I guess, for lack of quote-unquote. But before we get into this. Can you tell me a bit about how’d you gotten into the field?

Like, what was your path into cybersecurity and yeah, just tell us about Ben Tomhave who

Ben Tomhave: am I? Well, I’m so old that we used to just call it computer security back in the day. Yeah. Yeah, so I chose this field coming out of undergrad college, back in the nineties and , in the nineties, I helped put myself through college doing system administration.

And so it was at that time when the web was just emerging. And so I got to, for the college that I was at, I built the first web server and set up the first websites and did some, the initial hosting. And then towards the end of my college career, this thing called SSL came out and Netscape started talking about things like e-commerce and, you know, people started doing things like ordering pizzas online, and I very quickly when ha.

That’s going to be a [00:02:00] whole career field right there, securing all that. And what’s the interesting thing is I have a cousin who has much older than me and he’s been a developer and worked for, you know, Goodyear on classified projects for things like the Sr 71 Blackbird over the years. And then went to Silicon Valley and started doing a lot of software development.

And he told me I was stupid and crazy. For going, he’s like, yeah, security. Nobody’s going to care about that. I, you know, I’ve always had more of a systems and ops lean than a developer lean and he just thought I was stupid for doing this. I’m like, I am pretty sure this is going to be a decent career to go into.

It’s going to have a lot of, a lot of work for the foreseeable future. And. Turns out. It was right.

Ashish Rajan: Yeah. Years later, we’re still talking about cyber security and cloud security. It’s pretty good, man.

My second question for you. What does security cloud security mean for you? And I asked this to everyone because I feel like everyone has a different answer to this. So what’s your answer for what is cloud security

Ben Tomhave: yeah. I mean, I think cloud security fundamentally is applying all of the risk management and security management [00:03:00] principles and practices that we have developed over the last 20 years into a, an environment where we no longer control and own the physical fabric.

All right. Nothing else changes. No, that’s not true. But a lot of things, a lot of things change in the details, but at the high level, there’s a lot of commonality with how we used to do things as well.

Ashish Rajan: Yup. Yup. I’m glad you put it this way as well, because. I think when we were talking about this yesterday as well, that cloud security sound new, kind of like your point in computer security,

we’re still doing security work. It’s just that different name for similar topics in a different

landscape.

Ben Tomhave: Exactly. We’re all reducing risk to exposure for the business at the end of the day. Right.

Ashish Rajan: Yes, that’s right. And diving into the whole container space. Cause I’ve caught a few people who are new in this space and a new, I mean, I guess they have had security experience for some time and suddenly being introduced as well of containers and for people who have never heard of this.

Right. Like how do you explain containers to them? Like where do you even start? Yeah.

Ben Tomhave: There’s two prongs to the answer on that one. The sardonic answer is just ignore it. [00:04:00] Everybody’s going to serverless anyway. It’s going to come and go. But yeah, no, the more substantive answer is it’s the next evolution of essentially virtual machines without actually being a virtual machine.

And it’s the first incarnation of microservices architecture and really fundamentally we are in containers. We see. The actual first confluence that of what dev ops truly means, right. Were dev and ops come together and have to be together. Or they will fail anything up to this point. You could still keep dev and ops fairly separately, but when you get to the container world and even more so when you get onto them serverless, it’s very much has to be a dev ops world.

And if it’s not, then you’re missing at least half of the equation. If not more.

Ashish Rajan: that’s a really interesting point that you made about. Yeah. Cause DevOps wouldn’t really be a thing without automation and automation really wouldn’t happen without microservices. Then containers. Oh, I love how you define this, man.

So talk about deploying containers in AWS. And I just in general, like, what do you [00:05:00] see as some of the things that we are doing wrong as an industry? When we think about deploying container security or.

Am I better off starting with let’s talk from the beginning. What do you think about container security when you’re deploying into an AWS environment? Where does one start?

Ben Tomhave: Yeah, that’s a tricky question right there. There are a lot of potential entry points for starting with containers and container security.

I think step one is. From a security perspective, right? Step one is understanding that with containers. The entire app stack, the entire security stack matters all the way from secure coding down to app sec testing to solid base images, to container configuration, to whatever you’re using for container management, such as Kubernetes, that’s the predominant container management solution out there these days as well as how you protect those environments from an operational runtime perspective.

And so, you know, that. Means that you have to think about a whole lot of things all at one time with containers and Oh, by the way, that’s on top of all of [00:06:00] the additional complexity that comes from microservices, architecture, right? You’re no longer building this big monolithic thing. That’s going to sit on a big, iron server and talk to a big iron database.

And maybe have a firewall or proxy in front of it. Right now, you’re talking about taking every, sub routine component of that monolithic application and potentially containerizing every single piece and stitching it all together and potentially having lots of different data channels out the back end.

Maybe not even having one interface to the public, you may end up having multiple interfaces to the public. So you can’t think about like, , Oh, it’s just a nice thing. And we’re just going to pipe it all the way through. No, no, no, no. It’s like, Oh, it’s all coming in. Quote, the old George Bush, you know, thousand points of light.

Right. It’s all coming in on top of us and you have to go. I have to be here, here, here and here. And it gets complex real quick.

Ashish Rajan: I think API, Microservices has definitely changed the way security architecture has been thinking about this.

I was going to say, so now, since he’s spoken about, I guess, defining security architecture for a container, is [00:07:00] it different or it sounds is different obviously to your point, about the thousand points of light.

When you’re thinking about API, you’re thinking about microservices and designing containers. Are we deploying this like. Are we doing this, right? Like, are we containing being deployed incorrectly or

Ben Tomhave: yes, I think so let’s talk about immature organizations that are very new to it. And then let’s talk about far more mature organizations that are used to the concept of microservices.

Okay. I would say the organizations that have been doing it for awhile, I think hopefully understand it right. And are generally doing better. But when development teams and security teams, operation teams come to the new, one of the challenges is they misunderstand what a container is. They say, Oh, well, you know, it’s, it looks just like a VM, right?

A VM server that worker node looks like a VM server. And those containers look like VMs, So we’ll just spin up a generic, big, fat bloated base image. And we’ll just shove a bunch of stuff out there and. The actual very important practice that you have to achieve is one.

You have to have a good lean base OS right. [00:08:00] Something that’s purpose-built and hopefully purpose-built to a specific language base. Right. So don’t go deploy a generic base that has no Node and go and see C plus plus and Java. And, you know, you name it every single possible language you might Python, right.

Instead. You should have very, very lean S images that are purpose-built to a specific language framework. Right. And have very little overhead in them. And the reason for that is because again, we’re trying to secure stuff at scale. And so we need to make that surface area within each individual container.

Very small. So that is. One of the big misunderstandings that I think people have as they approach it as a VM, like they’re just going to deploy general purpose OS and you don’t do that. In fact, you don’t even go update the container running containers or you shouldn’t go update the running containers themselves.

Once they’re in production, you should always update the container in your build and then promote that up into your into your running environment. And that’s where we get into things like blue-green deployments, where you have your existing, and then you bring it down as you bring the new ones up. So and so forth.

Right? that is a fundamental shift from how we typically think [00:09:00] about traditional VMs and ECQ and stuff like that. It’s incredibly important because not getting that nuance. About containers takes you down a whole set of bad behaviors that end up keep expanding that surface area dramatically.

And now, again, not only are you securing that single pipe, but now there are multiple entry points potentially coming in that you have to secure. So it’s very important. You want to keep that as pinprick tight as possible. Coming in there.

Ashish Rajan: So you’re saying it’s wrong if I have a two GB Docker image.

Ben Tomhave: Yeah. and with persistent storage and all sorts of bad practices, right? I mean, you have to completely throw away kind of the old architectural model software architecture model, and it’s literally tiny bit of code running in a container that can scale horizontally as needed and no data in there.

It’s all data is all elsewhere.

Ashish Rajan: That’s awesome. So what you consider as an ideal size for a container image?

Ben Tomhave: I don’t do the operational side of things. I do the security side of, I’m not being able to give you a great deal of detail possible.

Yeah.

Ashish Rajan: As small as possible. That’s the answer I was offered because I’m pretty sure it’s definitely not in GBS. Definitely MBS. I imagine. [00:10:00] So the

Ben Tomhave: size of the code build, right. And, and it depends on where you are. Again, in the maturity process, you might take a large monolithic app and break it into some fairly large chunks and deploy those into containers.

Initially, just to start that shift over. To thinking about microservices. Now your next iterations may take those larger chunks and break them up into smaller pieces as well. Right? So there’s probably going to be an iterative process, but again, you need to know what that end point is. And go towards that direction rather than just saying, Oh, we’re just going to do a lift and shift from data center to AWS.

Now we’re going to do lift and shift from, from EC2 to into containers, because at some point you just that’s just disaster, right? Because you just can’t handle that.

Ashish Rajan: It’s a perfect segue into a question from Darpan here, hey Darpan. Nice to have you. He’s one of our previous guests as well, a regular listener Darpan is asking, would you say putting containers and cloud VM reduces.

That security complexity. I think it’s similar to what you were saying earlier about going containers.

Ben Tomhave: I think this is the key, right? Is. You got to stop and understand what the actual architecture, so using the Kubernetes [00:11:00] language, right? So you have a, what they call the worker node and off of the worker node, you’ll have several pods running off of that worker node, right?

And so all of these pods are your individual containers and they should be very lean, very small, very purpose-built. Now underneath your worker node is similar in concept to your VM server, but there’s a lot of nuance differences. For example. and this is something that a lot of security vendors haven’t quite figured out yet, or although they’re, I think they’re slowly catching on, you know, security vendors are, they tend to be about three years behind the curve on technology optimistically.

So within your worker node, you do not want to deploy things. Natively at the OS level, right? You want to actually use something called Damon sets to deploy essentially a package in a controlled fashion within that worker node. And that gives you a little bit better control and protection of the work in projects, because one of the problems that used to occur within legacy Docker based container environments is misconfigurations of that master node or the worker node where people would expose the control port.

And people could then log in with essentially full admin or [00:12:00] root privileges within that worker node and start doing all sorts of things, you know? And so we want to avoid that overall. looping it back, all this back, all the way to Darpan’s question then, too. Right. Just lifting and shifting into container doesn’t help anything.

If anything, it probably makes your life worse. You know, because you’re probably just going to have to figure out how to secure stuff without having even an O S level access to provide protection and things like that. Right. So you really need to be very intentional, very deliberate in how you move from what you have today to that containerized approach going forward.

Ashish Rajan: And talking about complexity also makes me thinks you’ve mentioned before, in some of your talks about the seven things or seven security challenges for introducing containers into an organization.

Think I’ve got them listed over here with, and I would love to kind of, if you can just like give almost like a intro into some of those like secure code. Application security based images. So is there like a sort of sequence that people should be thinking about this? And I think, I know you gave a whole talk about this.

I would love people to kind of go and check out the talk as well, but how do you kind of lay it out for people who may be [00:13:00] consuming this?

Ben Tomhave: Yeah, so the talk I did for IC squared security Congress as well as InfoSec world last year it was called seven layers of container insecurity. And I did actually start from source code and worked all the way down to production deployment.

Right? Because thinking about this and how this epitomizes DevOps, you can just take any one piece and be, Oh, we’re going to be really good at AppSec testing, but not do anything with secure coding, not do anything with configuration management, not do anything with runtime protection. Right because now, everything is in this nice tight package, hopefully, but it also means that now you have to check all of your boxes here.

So yeah, the seven layers I talk about are secure coding. Right. So working with developers to make good decisions AppSec. And that includes not just SAS and DAS type testing, you know, static, dynamic, but also looking at software composition analysis to identify components, libraries with known vulnerabilities, knock those out really quickly.

Pretty good base way to go base images. Right? So again, Focusing on lean images, maybe using [00:14:00] BottleRocket in AWS, maybe building custom Alpine Linux images to your needs container configuration. Right? So there are things you need to do at CIS benchmarks, provide a good starting point for container configuration to make sure you are.

Pushing out well, configured containers, things like eight EKS will help kind of manage some of that stuff for you too, and limit some of your exposure, but it’s still important to be aware of that. And then looking at your container management. So Kate’s config stuff as well, or if you’re using Docker swarm, and again, there are also CIS benchmarks for that.

And then the two last pieces are. Drifting and mobility. So when you deploy a container, you should think about it as being an immutable deployment. Meaning that once you push that container out there, nothing should change. And if you need to change something with that container, again, you build a new one and you deploy the new one and you take the old one offline.

The reason you want to do that, is it from a monitoring perspective, it makes it much easier to go, Oh, something unusual just happened in that container. Maybe an exploit occurred or, you know, maybe some sort of attack is being attempted that changed something on the system. And suddenly you can detect that much easier.

[00:15:00] And then the last one is runtime protection, and this is where. We’re really at a disadvantage in the security industry right now, there are really only a few security vendors who really provide runtime security production for containers it’s even worse for serverless. And the problem is that they tend to be very expensive.

I don’t know any really good free and open source solutions that do container runtime protection today. Nor do I know anything even for serverless for that matter as well. So. This is really problematic for all, but the largest enterprises, because, nobody can really afford to go.

Your average midsize enterprise. Can’t really afford to go drop 500,000 hundred to a million dollars per year on runtime container protection. And sure. It’ll pick up some of the other pieces, but really the primary focus there is the runtime protection, which you can’t get anywhere else.

Generally. Yeah. So there’s a huge gap there and even more so when you get into serverless, right. There’s even less out there in the serverless world too. So, you know, containers and microservices represent significant complexity and challenge. You made even worse by the fact that there’s just not much out in the [00:16:00] security solutions based on data to support it.

Ashish Rajan: That’s interesting. So does it change if you use, say, if you haven an ec2 with container like you basically install dock onto it versus using ECS or

EKS as well as what you see. So I’m trying to GKE and like all these names and

Ben Tomhave: ABCD EFT, right? Yeah. I think you have it was a GKE. You have a K S for Azure, Kubernetes, you have EKS for Amazon Kubernetes, and then you have ECS, which actually predates EKS, which is a, I think it’s the last to compete container services.

And of course, then within ECS, you also have far gate. Right where you don’t even, you basically just throw a container into the cloud and it runs, yeah, that’s right.

Ashish Rajan: Have another way to run farms of containers. So and I think this is really interesting, what you mentioned that because there’s a lack of security products trying to solve this runtime challenge in this space.

Do you feel being cloud native, does it make it easier to maintain security?

Ben Tomhave: It is why the entire chain is important. Right? If you do good secure coding, if you do good AppSec, if you, if you have a nice lean [00:17:00] OSTP and you have good config management control over all the pieces that you can control, at least and you and you at least minimally have immutability.

And drift detection. That at least gives you a pretty solid baseline starting point. Right. So if you can’t find or can’t afford true runtime protection, now I will say that some of the endpoint security solutions have started getting into this space as well and providing. Essentially what I would consider a lightweight runtime protection.

I know that CrowdStrike, for example, will do rogue process detection in your containers. And again, that kind of works hand in hand with that, that immutability principle. Right? So if you’re running a container and everything that looks good and you baseline it. And then other processes start popping up that gives you something to go.

That’s a red flag. You can interject that. And so that potentially gives you some options, but that’s not full runtime protection, right? It’s not doing full container monitoring. It’s not doing, you know, using the APIs to do actual operational reads off of all of the running containers. And so again, you can get very close without the big spin.

But you know, if you have a lot of high value stuff there or you, if you have concerns, [00:18:00] definitely a gap that needs to be potentially addressed.

Ashish Rajan: So when you look at securing containers in AWS. Are you thinking more from, to your point about the seven layers and you kind of would encourage people to kind of go through that one by one, because each one of those, no matter how fine tune you do it, that eventually means overall security posture has kind of improved.

Is that what you normally recommend even in the cloud native?

Ben Tomhave: Yeah. So one of the challenges, as you talked through, some of this is. When we talk about secure coding and app sec, that’s usually one whole position or role, right. And then when you’re talking about the actual Kate’s environment, you know, that’s oftentimes a whole other role.

You usually have some sort of like container platform, cloud engineering team that manages that or cloud security team. And then when you’re talking about runtime monitoring and protection, that’s oftentimes a whole other role or team. Right. And so this is, again, kind of that challenge with all of this is you’re really pulling everything together and it’s important to kind of hit all of these if possible, and staffing for it gets challenging and being able to actually do well in any of these is challenging on their own.

Not to mention staffing for them, you know, and being able to provide appropriate [00:19:00] support for it. So, yeah, it’s

Ashish Rajan: What do you reckon would be a low hanging fruit that people can start off with. Like, I think you hear examples of dont store secrets in images, what’s like a really hanging fruit that people listeners who are listening right now, , they understand the seven layers that you kinda to explain.

What’s the mini steps that they can take,

Ben Tomhave: right? What’s the what’s the name of the mountain in the Outback versus Everest, right?

Let’s go big,

Ashish Rajan: right? Yeah. Yeah. Let’s not, let’s not climb the Everest in one day because I would hate for them to kind of feel that they have to climb this Everest of GKE, ECE, or EKU and everything. That comes in. To your point about AB C s of cloud? Are there like some low hanging fruit that people can start off with based on your seven layers.

Definitely.

Ben Tomhave: Yeah. I think there’s a few things you can do and you can potentially, depending on language selection, even do them cheaper, easy One is software composition analysis. I’m a huge fan of software composition analysis. You can do it potentially for free with dependency check, which is an OWASP sponsored project.

Or you can do it with a variety of commercial [00:20:00] tools ranging from, , Snick to black duck, to soda type, lifecycle, and firewall. You know, it just depends on your ecosystem software composition analysis. To me. Especially for S for lower 30 organizations is hands down the best bang for the buck.

It tends not to be super expensive. And if you go aggressively after the findings, within the reports that are generated, or in the case of sort of type lifecycle and firewall implementing stringent policies and, literally just cracking stuff out of your artifact, repos, you can wipe out a bunch of the dangerous noise very quickly.

Now it requires good sponsorship to do so, but regardless that tends to be a pretty good starting point. So that’s a good one. I think lean images is also a good one, right? Don’t go grab a big fat Cintas image and put all 50 languages. You think you might someday support in it? We scanned an image once not too long ago.

It literally had three versions of node in it. And all three versions were old and had known vulnerabilities, pulling in that had go in. It had CC plus plus, and it had Java and, you know, don’t do that. Right. [00:21:00] Cause you’re expanding the surface area tremendously off of that.

So keep them nice and tidy. I think between those two things, you know, knock out the known vulnerabilities at the code level and start from a good clean, lean starting point. And then. If you can do it, find a way to implement immutability, drift detection, and again, a lot of your standard cloud scanning tools, you know, even stuff like Qualice right.

We’ll do drift detection to a degree where as long as you check the container in and it matches whatever is running in production. It’ll report off of any changes there. Right? So there’s three starting points that won’t break the bank and potentially get you force multipliers by you know, single purchase of a tool that gives you multiple things.

Right? So that’s typically where I would start. And then you can start working on AppSec programs, right? Always end up expanding it out, driving it back to the developers and you know, your infrastructure security. Eventually you’re going to want to make sure you have really good control over stuff.

But again, a lot of that’s potentially outsourced. If you do ETS or one of the other managed services, there actually isn’t a ton that you have to do around Kate’s config itself. There’s a little bit of tuning and tweaking you do, but for the most part that [00:22:00] you start from a pretty decent. Starting point.

Good, good security starting point. And then that leaves that runtime protection is kind of the big, expensive Beehag out there that you may or may not ever get to.

Ashish Rajan: But maybe in hopefully a pen test can solve I guess, problems. Yep. Yeah, for sure. Yep. Sweet. So to your point, We’ve kind of spoken about the low-hanging fruit.

What about doing this at scale? Like, you know, we’ve so say for example, you inspired a few people. Frank seemed to love your responses. Well, I’m sure others love your response as well. Like this like great idea. I want to go and do this. Are there some foundational pieces too? Cause we’re talking about AWS as well.

Right. And. Shit can just join and do like 50,000 servers in a matter of a few minutes. Yeah. So what do you recommend for a scalable solution for this? I guess

Ben Tomhave: it’s all about the pipeline. So bringing it back to dev ops, this is the epitome of dev ops and at the core of a really refined, really solid DevOps practice is a really good CICD.

Not [00:23:00] CI and CD, but CICD right. The tunes for never be separate. I keep finding organizations that are like, Oh, my development team wants to do their own CI. And then my ops team is going to do their own CD and maybe they’ll have a handoff in the middle, but usually they miss. And then it’s like, well, you’re punching yourself in the face while you’re doing that.

You know? So I know you really have to have an amazingly good CICD you have to have good maturity and then. You got to plug everything into it, right. So you mentioned automation very early on when we were talking about dev ops dev ops is not synonymous with automation. But dev ops does not work without automation.

Right. And so the entire purpose of the CI CD pipeline is to give you that here’s the source code. It comes in, here’s the bill, here’s the CI, right. Here’s where artifacts kick out all that stuff. Right. And all the opportunities where your Scott PO potentially comes in and starts knocking stuff off.

Right. All of the other checkpoints. But yeah, you got to have good CICD because once stuff gets out, you know, if you have one tiny vulnerability, but suddenly you scale horizontally to 500 containers to support load. [00:24:00] Now that’s 500 exposure points rather than just one , you got to make sure it’s all built in really well to begin.

And by the way, as yet another jab at the security industry, anybody who’s run static, application security testing or dynamic application security testing, especially SAS though. How long does that typically take to run four, a hundred thousand 500,000 lines of code? it’s just a blink of time, right?

we’re talking hours typically, right.

Ashish Rajan: So easy, easy. And this is like excluding the fact that you might have false positives,

Ben Tomhave: Yeah, exactly. So you can never put that into your CIC D so now you have to fork it. And run it in parallel and then provide feedback and then hope that, you know, three scrums from now three sprints from now, they’ll go, Oh yeah, yeah.

I see your findings that was already fixed because we threw away the code. Right. In many ways, it almost forces retirement of some of these old school desks, at least a little bit better. And I generally hear better. Experiences with IAAS. I’m still not convinced IAAW, is necessarily a real thing, but people seem to like it at least it’s it reduces the false positive rate.

So, Hey, that’s a win. Yeah. So, you know,

[00:25:00] Ashish Rajan: No, I, I love that you mentioned this as well, because I think that’s definitely, you know, been a lot of us would talk about like practitioners like you. And I would talk about, Hey, we should do SAAS. We should do software computer analysis some people may think it’s like a two minute job.

You switch it on and they go a lot. You have all these results. I mean, granted software composition analysis does happen fairly quickly to support a CICD pipeline as well. In most instances,

Ben Tomhave: Scott is fast, but here’s the problem with most software composition analysis. Right? So there’s two different kinds, right?

There’s two different ways to do it. So there’s the integrated way that, that Sonatype does it. Right. And that I really liked their approach. Right? Because they tend to be very developer centric, which is, they want to push it into the IDE.

They want to have a lot of checkpoints. They don’t want to let it into the repos at all. Right. That’s what firewall does it. You can just block it out. So there’s that approach or there’s the security industry approach, which is, Oh, we’ll just run a scan and produce a really lengthy report and you have to figure out what to do with it.

Right. And so, you know, there’s nothing worse than meeting with the dev team and going, here’s your list of [00:26:00] 5,000 findings from your Snick report or from your black duck report. Good luck fix it. And they’re like, well, I don’t even know what half this stuff is. What do you mean all these findings and this, this, by the way, it comes back to the importance of a Lean OS right?

Because a lot of these tools will also produce a lot of OS level. Library findings. It’s like, Oh, that, that OSTP wasn’t based on this image, wasn’t updated recently and there were 10 new findings in there or something, but yeah, it’s, it’s painful.

Ashish Rajan: I’m so glad you’re bringing this up, Ben, because I don’t think enough people are talking about this enough.

People are just trying to talk about the glorified world of. How application security is amazing. It is amazing for like it is required and people kind of are forgetting that it needs to mature and to continue to have kind of like what you’re saying. It needs to have all that in an open forum to kind of in this conversation, because going back to the whole container security thing as well, Containers was the reason people went down.

The container path was the whole microservices model I mean the similar to point, it’s should not be used in the same vein as a cloud or DevOps, but a lot of people still [00:27:00] think that or DevOps means containers or means Kubernites. And I’m automating a CICD pipeline.

Right. Yeah. To some extent you may think that they can work together and that’s what DevOps can look like in your organization, but that’s really half of the picture. And then you bring it back to the reality of what we’ve been traditionally doing insecurity for years. It worked because waterfall, I mean, I feel like I can tell you the waterfall method was great.

Yeah. Oh yeah. Oh yeah. The thing that the requirements document we had to fill out for what would be, what for the company get at the end of this project? One year later, I used to love that thing. You’re like, wow. We could decide one year later right now. And right now we don’t even know if tomorrow will existed.

So I’m so glad you brought that up. I’ve got a question here from Darpan here as well. Based on your experience, what are some strategies you think work better to create awareness about container security in traditional computer security team?

Ben Tomhave: Yeah. I saw his question sitting there and I’ve been [00:28:00] mulling over , there’s the carrot or stick, right?

You have, you really need to apply kind of both practices. It’s difficult. It’s very difficult. And let’s be honest as I’ve kind of jokingly alluded to a few times already. Security teams that are not dealing with container security today are already very late , serverless is already happening, right.

I’m already seeing dev teams like throwing all of their stuff away and just doing serverless, you know? And by the way, why is server-less popular? why are containers popular? If you’re a developer, you love the fact that I can write my code and. Build it into an image and throw it out in the cloud.

And I don’t have to deal with the infrastructure and serverless even more. So I forget about the container. I just have to write the code and I use same with CDK and then it’s out there. That’s all good and fine, except for one very tiny itsy bitsy tiny little problem, which is very few developers have even a remote clue about infrastructure.

They have very little understanding of architecture. They don’t get how data flows work. They don’t [00:29:00] understand security. So even though from an agile dev ops perspective, they like, I want it out there. I want it out there. Get it faster, faster, faster, faster. Okay. At the end of the day, they’re basically just tearing holes in the wall of your house and letting all the weather flow in and all the bugs you’re in Australia, right.

Everything down there will kill you. Right. And so you probably don’t want to have a lot of holes in your house to let you know, you don’t want to Rue to jump through and you don’t want the snake of the day to come in eat your baby and all that stuff. Right. You’re dealing with that just with your developer teams, right?

Yeah, and they at least at least understand the cutting edge technology. So now you have to go back to your poor old fashioned bedraggled security computer security teams, and you say, well, okay, think about it like this. You know how you have mainframes and you don’t run anything on the mainframe directly.

You run everything on the FIPs. Right. And so then you have to secure the access to the Phipps and you do all the access control and the FIPSE. So a worker node is your main frame and the container is your fat. And if you think about API architecture, you usually build some sort of, you know, one or two layers in front of those [00:30:00] FIPs usually.

Right. And the same thing with containers you oftentimes want to use. You’re probably using containers to do the front end. You know, AP central API, or maybe you aren’t doing a BFF model at all. Maybe you’re just letting everything be directly accessible out there. I say that tongue in cheek, right.

Very few people probably will actually get the mainframe references in an old guys. Yeah. I was

Ashish Rajan: going to say, I go to, because we were trying to help a bank move into AWS and yeah, it wasn’t really great. We only ended up doing the website because mainframe was just a no-go. So no one wants to deal with transaction at that point.

So I’m like, okay, fair enough. But I did want to add another thing that kind of has worked for me in the past. And I, if you don’t mind me adding in, I was going to say. From a, it’s usually a lack of education as well, because what I found running a lunch and learn session around the fact of, Hey, these are the kinds of containers you’re already running.

This is potentially the kind of exposure you don’t have to like, have like a company-wide presentation of lunch and learn even just within the security team. I found like a lunch. It’s usually like an education gap. Once this point started pointing out and then he started relating, Hey, you know how we have this hybrid world on the data center.

[00:31:00] We have this. Now, let me just map that over here, see how much exposed you are and then you go, like, I see those light bulbs go off and like, huh, we should talk more about this and then becomes like a bigger thing. That’s funny.

Ben Tomhave: Yeah, absolutely. I mean, I’ve done that as well. Right? Lunch and learns. I would say the biggest obstacle that we encounter is, especially when you get into larger organizations, you see a lot of specialization.

and I think the other trend I’ve noticed really lately, you know, of new people coming into security, as they seem to come into a specialization area, rather than coming into a general, you think about like 20 years ago, how we did security, right? It was usually, it was a security team and . Yeah, you did some IAM.

You did some AppSec, you did some security architecture. You did some OPSEC. Nowadays you come in, you’re a pen tester, or you’re a SOC person, or you’re doing AR forensics. So they’re missing the fact that. There’s this whole world, that’s all coming together in this one little container that you have to know, AppSec and secure coding.

You have to know infrastructure and OPSEC. You have to know IR and [00:32:00] forensics. You have to know monitoring, logging, monitoring, and runtime.

Ashish Rajan: I think what you’ve said is really interesting because. I’ve definitely heard about and seen the lack of networking knowledge and diverse and speeding it like as an obvious gap in that space where we’ve been talking about dev ops for so long, it gives the power to the developers without leaving, teaching them about the basics of network.

Hey, I can’t access my service. Like, yeah, that’s a networking. That’s a data flow. So that’s not Java, that’s not a C plus plus. Right. So similarly with this as well, I feel. A lot of these security screens are being told you don’t have to worry about cloud. And I remember sitting in this conversation, wherever the SOC team, and I told her, Hey, say, how are you preparing for this transformation to cloud and all that?

Like, it doesn’t really matter to us. That’s what the person said. And it just another landscape I got good. It is true. It’s like, it’s great. It’s just a lot of landscape. Right. So I guess it was like, no, no, no, no. we just monitor us, send us all the logs. Okay. I guess we’ll just send you all the logs, then it’ll be all good.

So that’s the kind of like, so, but to your point, [00:33:00] yeah, that’s fine. Then nothing happens. It just goes into this log. Aggregator could be called SEAM, which is

Ben Tomhave: probably without doing anyway. So change actually, you know,

Ashish Rajan: Yeah, that’s right. The reason I love that story is also because it ties in really well to what you just said.

A lot of people are being told to specialize in. Yeah. You’re a SOC person just focused on the soc thing. Don’t worry about container deployment. So I want less deployment who cares more, all that. That’s, that’s the architecture thing. That’s the solution problem. Not your problem. And maybe as an industry as well, we kind of need to rethink this as well and make that awareness.

Ben Tomhave: Yeah, I think there’s a lot of misunderstanding. We hear so much about, Oh, there’s a shortage of talent. Oh, there’s all this gap out here in the workforce. And I think there’s two flaws with that. One is there’s a lot of people looking for work. But they can’t fit into these massive oversized roles that have been defined that wants you to have expertise 50 years of expertise in every single domain with insecurity.

So that’s actually humanities. Oh yeah. Right, exactly. You know, so that’s a problem that, you know, these everything, but the [00:34:00] kitchen sink type roles, and nobody can qualify for them, you know? So that’s but the other thing I think that kills us too, is so much money has been invested into, Oh, we have to do cybersecurity training in college.

But it’s, it should not be to turn out better new security professionals per se. It should be to turn out better developers and better ops people. Right. And, and help, you know, sometimes there’s there’s call for specialization forensics programs. For example, going very deep in that forensic stuff is very important, but I think the rush to specialization ends up costing you a lot of stuff.

I mean, I think about the old NOK model, right? So prior to security and SOC, we had a NOC network operation center. And what did the N We did a lot of monitoring overall, right. Similar to what a SOC does. But what was different about a NOC? Is that in your NOC, you almost always had a junior CIS admin and a junior network administrator sitting side by side.

Right. And, or teams of them. Right. And maybe some IAM folks to help with access resets, but they were sitting there learning from each [00:35:00] other. And you could call into a knock and go, you know, and they also were trained typically in incident handling, right? If something went down, you know, a site is offline , I did for a large us insurance company back more than 22 years ago, a company I worked for redid their entire command center NOC environment, And all of the regional networks. To all of the agency locations and you could literally, they would keep a us map up and you could watch storms. You can see storms go from, from West to East, right across the country. And then next to it was our network map, our site map.

And you could see site outages roll across with the storms. Right? So as the, so did the network outages, and this is again, this was in the nineties, right? So a lot less tolerant, lot more likelihood of outages back then. Being able to look at that and understand the issue right.

And go, Oh, okay, well there’s weather happening. So as circuits down, it’s not that a firewall is off on. I have to go troubleshoot a firewall or a router or something like that. I just have to wait for it to come back online. Right. But [00:36:00] today, a lot of this, all of this nuance and understanding of the environment gets lost, right.

For a SOC person, somebody getting thrown into SOC for the first time. Oftentimes they’re not given much training at all. You know, and, and there seems to be an expectation that they will specialize from that position. I would much rather have, you know, developers and systems folks come into a SOC already having some technical experience.

And understanding about their domain and then start to understand the bigger security picture. And then from there as their experience grows, now you can start thinking about going back into specialization, right? Some of the best apps that people I’ve known were developers for 15 years before they then came in and said, Oh, I kind of liked the security thing.

I always found the security teams annoying, but I think we should write better code. Let’s talk about how we can do that. And then you can have that one that good direct conversation right. Overall. So I think, I think we’ve lost. Kind of lost our way in, how we develop and identify talent over the years.

Ashish Rajan: And now thanks for sharing that as well, man. I think it definitely agree on this and I think we kind of as an industry evolving, so [00:37:00] I’m glad. There are people who are sharing that information as well, because it, and saying that, you know, it’s not wrong to have a specialization.

Sometimes the scale of the work is so big. It’s, worthwhole specializing in something, but as you go up in experience, you kind of have to like, Oh, what else is out there? And you start looking at a cloud. Yeah, of course I’ve been mentoring a couple of people who are SOC analysts and the way we kind of worked on helping them get into cloud security has been more barn finding, Hey, what’s the, what’s the common ground that you already know.

If you’re already monitoring loGs for cloud, let’s start there. Like, what should we looking as threat vectors? And like, what’s the playbook we can define for this? Like, so there’s enough learning in that space as well. It’s just that A lot of people are told to not worry about it. And that scares me.

let’s see.

Ben Tomhave: It’s the old Hitchhiker’s guide to the galaxy SEP or somebody else’s problem. We need that SEP for a field generator, right? It’s like, Oh no, it’s not the job of the SOC. You know, we just look at things and we make the red turn, go back to green. That’s all we have to do.

Ashish Rajan: Yup. I’ve got a comment here from [00:38:00] Houston as well.

Who’s also a past guest and I think that, you know, from him and from the Slack community as well so Houston is saying Microservices a skill SAS. Can’t take that monolith of tech to container a serverless naturally. Input output are not sanitized how you use the services in many different places are where the controls are.

DAAS possibly better. I too, haven’t bought into the IIS with cloudy things, context matters more than ever. The role running the cloud is often more important than the code itself. Good stuff there. That’s a great comment. Any follow up comment to what Houston just meant? No, I

Ben Tomhave: think, I think he’s dead on, right.

And I think that’s the whole point of looking at This entire container security ecosystem, right? Is that you’re shifting to contexts, And you’re, looking at context and monitoring, right? That’s that’s where immutability and drift detection kind of come into play, right?

We’ve never been able to do that before. Right. Even in the VM world, you still had to have a fully functional OS that tended to be a little fat around the edges. Right. And you couldn’t just freeze it as is you, you [00:39:00] tended to upgrade in place containers. You don’t upgrade in place.

Right. And so that allows you. It totally changes your entire mindset. Now I think the other thing that’s interesting is anybody out there who has money to invest in something new for security, finding something to replace and kill. SAAS. all of these apps that testing stuff, nothing has fundamentally changed in more than a decade in how these tools function.

Right. I have to believe there’s going to be a better way to do it. It might be AI. Maybe know it might

Ashish Rajan: it’s Zero Trust. No, it’s not AI

Ben Tomhave: zero trust, secure your code. It just secures your network. Apparently I’ve, I’ve been zero trust networks, MFA is nobody’s ever broken MFA. That’s right,

Ashish Rajan: but I’m loving this conversation. I feel like you and I can go for hours about this. I’m so glad you brought this up as well.

Just coming back to the container topic, what you

Ben Tomhave: were talking about with containers.

Ashish Rajan: So I was gonna ask, is there like [00:40:00] a pattern that you’re seeing? I know we kind of touched upon a few of the, I guess, obvious gaps outside of that. Is there a pattern that you see is a complete, like, almost like an anti-pattern that people are applying in cloud security, which they should avoid any examples in that

Ben Tomhave: space?

W within cloud or within container, just to

Ashish Rajan: clarify containers in AWS. So from that cloud perspective, I guess,

Ben Tomhave: yeah, I think the anti-pattern I come back to this very simple concept, which is dev ops doesn’t mean the absence of ops because dev does everything. Right. Dev ops is dev and ops.

And people like to say dev sec ops. Right? So now we have, you know, Akuna Matata, what a wonderful thing. Right? So you know, everybody’s happy together, but no, I think fundamentally, even if they still function in autonomous groups, dev and ops, you still have to change the foundational culture. Of the organization.

Right? And so the anti-pattern is that if you move along [00:41:00] from traditional to containers to serverless and you make that move to microservices and then even, you know, extreme micro-segmentation and extreme microservices, If you don’t change how you do business as part of that, You have to change, you know, how do we collaborate?

How do we communicate? You know, how do you cross pollinate ideas? It actually raises the, import of security architecture and just architecture in general. People don’t like to hear about old school ideas, like, you know architectural committees or architectural, you know, centers of excellence or anything like that.

But fundamentally. You need those mechanisms to create the opportunities for conversations, for intellectual conversations and collaboration, and really also setting some standards and communicating them out, but not standards in the sense of. Oh, we’re going to write this policy. We’re going to write the standard and it’s going to go get checked into a SharePoint site that nobody ever looks at, right?

It has to be functional standards that are directly implemented and [00:42:00] ideally even enforced from a policy as code perspective as well within your pipeline artifacts. So I think that is the challenge is continuing to evolve the organization itself. I did research for a year for a company around dev sec ops in general, in.

That was the top takeaway I had, which is you cannot keep running organizations the way you did even pre 2010, you know, definitely not pre 2000, right? Pre pre bubble burst thing. You have to change it. And I would say one of the things I highly recommend that people read is Frederic Laloux reinventing organizations go out and read that here’s a guy who.

Did a bunch of research on organizational structure and management structure and what, you know, kind of, what are some of the next gen, if you will definitions and behaviors and values and things like that. And what’s interesting is, he’s not a technical guy and he’s not a dev ops guy. And yet everything he describes within reinventing organizations.

Is [00:43:00] really describing the evolution an organization needs to make as they move into agile and DevOps overall much flatter, much more collaborative people having a lot more authority themselves, getting away from a lot of the rigid hierarchies, you know? Cause if you still have that rigid hierarchy, if you’re heavily siloed, it’s going to be failure.

Right. And containers make that very evident. Right? Like I said, you were talking about at least three different security disciplines. Just right there within container security and then you have development and then you have operations and infrastructure, you know, and possibly two different layers of infrastructure.

Right? You may still have some form of network engineering to a degree as well as your cloud engineering and ops SIS offsite folks. So that’s a very challenging world and it’s not something. That most organizations, especially longer lived organizations have actually been built and structured for.

Right. We still continue to see this. And even we were talking about this with talent and you can get into, we could go on a whole tangent about education, right. But one of the common critiques of, contemporary education environments is that they were designed in the factory age. Right. The [00:44:00] manufacturing.

And you think about it as a manager. Yeah. But even through the eighties and the nineties for software development, seventies, eighties, nineties, right. For software development, we still followed that factory model. Right. I’m going to write my widget and you’re going to write your widget and it’ll plug together and then we’re going to go put it onto the platform and it rolls off the conveyor belt.

And it’s complete right into production. Yeah. Containers, change that model dramatically. There’s no longer a bunch of different people doing a bunch of different pieces. It’s it’s you, you, the developer are writing the entire thing and I might give you the base container that you can go run and build it on.

Yeah, that’s about it. But then it goes into the, into the CICD pipeline and poof suddenly ends up in production.

Ashish Rajan: Yeah. And it’s a huge responsibility for a developer to just be producing artifacts and to your point a security person. Yes. Great. But that’s just one discipline. It’s not the other parts of it.

You’ve made the artifact. You like, you’ve thrown it across the fence. Like good luck guys. we start getting into information, they’ll get the logs. And [00:45:00] then you’re like, Oh wow. This is I I’m, I do want to call it out. You’re not trying to paint a dark picture over here. Right. I think obviously this is just some of the gaps and these are some of the things that we need to work as an industry.

And these are just some of the patterns that you would you see, like in large and medium sized businesses. Well, quite a bit. Where there’s so many people and the scale of it is just so large, sometimes just does make sense to have those silos in a traditional context, but like even Facebook for that matter, from what I understand, I think one of our guests from Atlassian and it’s pretty big company as well, even, they have a cloud center of excellence.

Oh, yeah, you need have that common ground in a large organization for someone to go, Hey, I don’t want Ashish to be just creating containers, Willy nilly. I want him to get to these, have some standard. That way he can still grow at scale because he’s going by the standard that everyone agrees to. So you can just put that in CICD pipeline and just like, you know, just keep Godspeed and offer that

Ben Tomhave: right.

Well, and this, you know, this is again where. The maintaining the concept of cold [00:46:00] images, right. Is really critical. I don’t ever want a developer to go, Oh, I’m just going to go randomly grab whatever is out in AWS marketplace. No, no, no, no Docker hop, right? Yeah. That’s become kind of the yeah, the source code copy paste.

But I mean, if I provide you with the starting point. Oh, you want to write an app in Python? Here’s your Python image, right? You want to do something to go? Here’s your go, you know, so on and so forth. So giving them a starting point. I see typically two sets of responses to that one is, Oh, you’re limiting my ability to make choices.

That’s the negative read off the positive read off of it is I’m taking away all of the noise that you shouldn’t have to worry about. And by the way, as an aside, you’re not qualified. To know about anyway. Right? Really officially it’s not your job, right. You’re not taking that away from you. And, and in the true Steve jobs fashioned, right.

Where, you know, jobs went out and bought what hundreds of pairs of identical jeans and the [00:47:00] same shirt. Right. And the whole point of that was because he didn’t want to waste an ounce of brainpower thinking about what to wear today. Right. Well, for developers, it should be the same principle. You know, your biggest decisions should be what language and supporting frameworks best meet the needs of what I need to do.

Okay. Here’s the image. Go write your code. You worry about your code and make sure you write really good code and make sure it’s peer reviewed. Make sure you’re using EDD. Right? Write your tests then code to your tests. But I don’t want you to think about the S I don’t want you to think about scalability necessarily other than the fact that it should be autonomous and should be able to be horizontally scalable.

non stateful, basically type stuff. But that really is what we’re talking about. It allows us to kind of strip away some of those, those distractions, but at the same time also then inject really good security at the same time. You know? So it’s like, not only am I giving you a good base starting point, but is it known good, secure, starting point?

And so if anything gets screwed up, it’s your fault. You know, the other thing [00:48:00] is The importance of when you enable developers to, to have this autonomy, to write code and get it essentially shipped out to production. And by the way, it’s not that easy, right there are checks and balances along the way, especially I’ve had conversations with auditors early on a dev ops who freaked out, what do you mean a developer is writing code and pushing it straight to production.

And I was like, no, no, no time out there. There’s no straight to anything. They’re going through 20 different checks at least. Right. That code should have their name and contact information attached to it. And if once it gets into production, when the SOC sees it, there’s a problem with that code. They’re pulling that tag information down and going, Oh, I need to call Ashish.

You shouldn’t go. Oh, your code is broken, man. Hey bro. Fix your stuff. And by the way, this is how Netflix did it. Right? This is the Netflix story from the very early on. That everybody likes to hold up Netflix and looks at all the automation pieces, but that freedom and responsibility piece is really the big, big one, right?

That you, as a developer, not only have ownership, you have the responsibility to support it. [00:49:00] Once it’s into production,

Ashish Rajan: that is truly awesome. And I think that’s a great way to wrap the interview as well.

So for people who have follow up questions about container security and doing this at scale, we can then reach out to you. What social media can they reach out to you on?

Ben Tomhave: Contact you.

Ashish Rajan: Yeah, you can come to me and I’ll tell you go. Yep. Yeah. I

Ben Tomhave: would say LinkedIn or Twitter are probably the best bet I am.

Falcons view F a L C O N S V I E w all one word Falcons view on Twitter. Or you can find me on LinkedIn, several people have already done. So I would say if you’re sending me a LinkedIn connection, please send a note that at least said, Hey, catch you on, virtual coffee with Ashish, you know?

Yeah,

Ashish Rajan: yeah, yeah. So, so because nowadays LinkedIn has done into bit spammy. Sometimes I will say

Ben Tomhave: it’s insane, man.

Ashish Rajan: I’m looking forward to having you again, man. Thank you so much for taking the time out. I really appreciate that.

Ben Tomhave: Likewise. There’s a lot of fun.

Ashish Rajan: Awesome. All right. To Everyone else was listening. We’ll see you next week with another episode and stay safe out there.

‍