Kubernetes Security at Scale

[00:00:00]Ashish Rajan: Hello, and welcome to another episode of Virtual Coffee with Ashish and today is a really, how do I put this last week? We did container security. If you haven’t checked that video, you should check that out. It’s a great episode. This week we have kubernetes and I have a special guest for this.

He’s has a lot of conversations about this topic. He’s part of CNCF as well, and I’m not going to spoil his introduction, but I’m going to bring him in straight forward.

Hey, Michael,

Michael Fraser: the show. Thanks for having me.

Ashish Rajan: Oh man. So glad you came in, man. I think I’ve been getting kubernetes questions for so long, so I am so glad I could find you and bring you on the show for some conversations. All right. I’m going to start with something obvious, although I’ve known you for some time and I’m sure other people have as well, but some people may still not know who Michael Fraser is.

So how’d you get into cybersecurity, man? Like, what was your path into this?

Michael Fraser: Yeah. So I’m, I’m ex veteran myself ex air force. I like to tell people I started working on the physical [00:01:00] weapons systems. So I worked on F 15, C and D aircraft on a army weapon systems and then transitioned into becoming a cyber security engineer.

So working on digital weapons systems and. And really being able to focus on being a cybersecurity engineer when I was in. And then I’ve just been in cybersecurity for as a practitioner for about almost 20 years. Now. I can’t believe I can say that at this point. So about two decades, which is awesome.

And I’ve been, you know, in industry I’ve worked you know, primarily for myself, mostly a different. Cloud consulting slash cybersecurity consulting firms did a lot of stuff in the. VDI space as well. So it was pretty big into that. And then I got bored of that a few years back and decided full steam ahead on dev sec ops which is where I lived.

So that’s how I got into Kubernetes and cloud native and so on. And I did a brief period over it Optiv as well as a, a solution in the cloud security [00:02:00] architect. So I’ve been in industry for a long time and really started out at the very, very beginning in cybersecurity.

Ashish Rajan: That’s awesome, man.

And I think too, to your point, it’s really interesting for me personally, when I see a lot of veterans come into the safe space of cybersecurity, because it’s really very similar in the terms, what the jobs are. You’re trying to defend either a company or an organization or a whole country as well, by the way.

Thank you for your service, man. I really appreciate, I think I’m pretty sure a lot of people appreciate this over here.

I’m going to move on to the next question. What does cloud security mean for you?

Michael Fraser: So, as I said, I’ve been in cybersecurity for almost 20 years now and I’ve been in the cloud space actually. So. One of the first companies I had, I was doing private cloud for the SMB.

And this was back in 2009, 2010 timeframe. And so I’ve been in private cloud before public cloud and I kind of looked at cloud security is really. Kind of holistically covering security, not just in the public cloud, but also in the private cloud. And now that we’re getting into cloud [00:03:00] native and dev sec ops, I’m seeing more and more kind of looking at it as holistic automation.

That includes things like. Infrastructure is code cloud services that you can use both public cloud. And then a lot of these solutions out there also being used in, in private cloud as well. So there’s a lot of hybrid cloud stuff going on out there. In, from a cloud security standpoint, you see a lot of things out there, like.

Cloud security posture management systems and you know, cloud security tools that are focused primarily around the cyber security practitioner. And I, again, look at more of the holistic viewpoint of really wanting cloud security to be a piece of the overall solution that organizations are trying to drive outcomes.

To be able to do stuff that they’re trying to do and really increase their overall agility and cloud security is a a big piece of that. But I think it’s still a part of the overall kind of landscape of we’re trying to move from a technology standpoint and really. You know, it’s a very weighted term, cause it can mean a lot of [00:04:00] things.

A lot of people just like dev sec ops, right? So you can just look at it as, can you apply cybersecurity principles to public and private and hybrid cloud. And is that something that you can maintain more in kind of an agile type of approach? So it’s not just, Hey, I want to use a point solution. That’s going to go help me secure this one cloud.

We’re looking at all some more is multi-cloud. You may have even heard of the term now, poly cloud, where you’re using different services and different cloud providers to build a solution.

Ashish Rajan: And while we are on that topic do you want to quickly touch on what do you mean by devsecops as well? I know everyone has their own definition and probably is a good segway into kind of nailing that one as well.

What do you believe is Devsecops?.

Michael Fraser: So from a dev sec ops perspective, I view devsecops is holistic automation then includes cyber security baked into it. So you always hear the term out there. Shift left as well. And. If it requires a [00:05:00] cyber security shift left out. So have you DevSecOps a core piece of it as being a cultural shift, to enable collaboration between teams to where dev sec and ops can collaborate together and really start building solutions that are based around more of the modern principles that are out there, not just including cloud native, but really looking at if it becomes a software defined, say it’s infrastructure as code, I want to run an infrastructure as code scanner against it. That’s a piece of the overall pipeline and really the life cycle of what I’m trying to build as a solution and also building things more modules so that I can incorporate different cybersecurity tools into the process.

It really started building out more of a thinking about things more in an agile approach when it comes to building. Solutions. And really, so dev sec ops is holistic automation that includes multiple teams and has cybersecurity baked in

Ashish Rajan: and coming to the topic, then [00:06:00] bringing this back to Kubernetes, what is Kubernetes for people who don’t even know, like listening for the first time, like, why do I have this whole episode on kubernetes security? So what is Kubernetes for people who don’t know?

Michael Fraser: So Kubernetes was an open source project that came out from Google, pushed it out there. I think it was around 2014 and the name itself was actually based on Greek mythology.

it means essentially helmsmen or pilot. And it’s really about being able to create a portable and extensible open source platform. That’s very focused on. Container workloads and being able to incorporate things like infrastructure as code and automation into the process, to be able to push things like microservices and other things that you’re trying to accomplish when it comes to building out Kubernetes clusters.

It’s also a lot more of a modern approach to what we’re used to in say infrastructure’s code. And it’s also very. They had to say a bit complicated. So there’s a lot of moving parts when it comes to [00:07:00] Kubernetes and Kubernetes is just a piece of what you’re trying to use as an overall solution. And there’s a lot of things that go into it to be able to get the end result that you’re looking for.

Not just including. Cyber security into that process, but also thinking about what you’re trying to build with Kubernetes and what you’re trying to actually accomplish. And so it’s really also about the ecosystem around it. So there’s a huge community, huge ecosystem and it’s awesome technology.

And we actually use it in my startup not just inside of what you can build, but we also are on Kubernetes do with our platform and there’s some reasons why we went that direction which we can get into on some other questions you may have.

Ashish Rajan: Yeah, sure, man. I think that’s a great way to kind of get dive deep into this as well.

And I want to start with. Kubernetes has been here for a long time, but two, three years sounds like a long time in technology these days. So it’s been there for a long time. I’m going to say that

are we approaching Kubernetes the right way or is there not enough education on this at the moment?

Michael Fraser: So when I first got into Kubernetes and we were [00:08:00] building stuff out of refactor, we ran into it where Kubernetes is building, it was an enterprise product. It was, you know, something that Google had open-sourced.

And so there were a lot of things. Inside of Kubernetes when it came to security that we were looking at stuff like hard multi-tendency can we isolate each tenant in the platform? Are there tools that we can use to scan IAC or the Kubernetes configurations at the very beginning and the IAC that goes around it.

And then other tools to run around the deploy and the runtime environments as well. And there’s some amazing technology out there, but also say that. It’s very complicated to add security to Kubernetes. And I don’t think people understand how much work it is until you start getting into it.

And then thinking about other things around you know, NIST or the center for enhanced security frameworks to be able to use those are benchmarks. And what those mean for your overall process too? And so. That’s really one of the things about it is all the moving parts inside of your your [00:09:00] process to build out your Kubernetes solution and what all is actually going to go into it.

And then also trying to future proof yourself as well. Because there’s a lot of different tools and technologies out there and you really want to make sure, like, if you’re using an additional orchestration layer on top, like open shift or rancher, you want to think about as well, like what’s your coupling there.

Into that particular offering versus what may be required for an overall effort on moving to a different orchestration, Kubernetes orchestration tool, and then all of the security tools that go in there too, which again, stuff like, you know, all the way at the very beginning of your process, being able to scan your ISE all the way to the end, to your runtime, like what are you using for the different tools in that process and what really.

Makes sense for you. And then also, is there something that you can pull out and add in something else later? Cause another tool may be more applicable to your use case. So you really want to think about that too, to make it as easy as possible when you’re architecting out your Kubernetes pipelines

Ashish Rajan: thats [00:10:00] really interesting that you said that Cuban and he security is complicated, complex.

And I’m I going to repeat that myself, because I think it’s worthwhile calling out. And it is so different to when people just talk about cloud or, I mean, that thing in itself is like a beast. And then you almost have another beast. Pushed into this. I’m thinking of like your cloud native and I’m pretty sure there will be questions from people on this as well.

Doesn’t get any different if it is cloud native versus a say Kubernetes’s on a regular server versus EKS or GKE. And one of those ones that just cloud native to the cloud service provider, is there any different in terms of challenges or are they similar?

Michael Fraser: Well, first off there’s there’s trade offs. So when you’re looking at things like EKS or managed criminalities and AWS or HKS, same thing in Azure GKE, any of the manage offerings out there, you also have to look on what you have for frameworks and or benchmarks. And so a good example of that is it. A center for internet security.

So they have their base Kubernetes [00:11:00] benchmark, and then they have specific ones, the managed Kubernetes. And so there’s certain things that you can’t get access to in the manage. So you’re going to be relying upon the cloud provider to provide that, and they do a fairly good job when it comes to being able to provide certain security controls around , some of the infrastructure that the Kubernetes is in.

And then you also want to think about, and we’re actually going through this, the refactr we were engaged with the air force, with the platform one initiative, and one of the pieces of our silver phase two, a small business innovation research project is that it’s about being able to have Kubernetes that can run vanilla.

And then what does that also look like in the different managed Kubernetes offerings out there, and then who’s using it right and does it work, you know, both in the commercial side and on the did IC or Intel community side offerings from the cloud providers too.

So we think about that. And then the other piece is what are some of the dependencies that are required in order to be able to have a solution that’s using the [00:12:00] managed Kubernetes offering, and then what am I using to also scan there? So like, I see I’ll still be able to. At least be able to scan different things when I’m building out my containers and pushing them out to the registry and so on.

But then you also want to think about, you know, what kind of coverage do I have on the core infrastructure? And can I get any information about that from the cloud providers versus controlling that myself? And then the last part to that is am I looking to deploy this into my own private cloud? Am I going to use.

Things like PKS or you know, in VMware or, you know, like what does that look like? So do I need a flavor of Kubernetes that I can add dependencies to when I deploy it to the manage or I’m using other orchestration solutions like rancher, OpenShift. So I just want to be thinking about all of that when you’re thinking also about the security implications and what you’re going to be using tooling wise, and a lot of The automation that you can create for this. Do you just have to incorporate where in the process am I incorporating that and what do [00:13:00] I actually get access to? And so I always think about anything that’s going to come from the cloud. If I don’t have full control over all of the infrastructure, and I’m not even going to have control over, say the hypervisor other layers when it comes to this.

So do I control the VM? Am I deploying Kubernetes into. The VMs do I have control over that? Do I need to, you know, understand the security controls over that? And then also again, around the IAC, around the deploy and the runtime, what can I incorporate into there that I can use in whatever version of Kubernetes I decided to hone in on and decide that’s best for me in my organization,

Ashish Rajan: you dropped in so many gem there, man.

I hope people rewind this and listen to this again, but I’m going to touch on something that you did, which I thought is worthwhile calling out again. there would be some responsibilities that you probably hand off to the provider. And you mentioned that the U S air force is also trying to put Kubernetes and that’s pretty awesome.

For me, the way I listened to that is like, if I’m a regulated body listening to this, if I’m a bank or any other regulated body, I can [00:14:00] utilize Kubernetes as well . Actually, maybe that begs the question. Why would you. begin with. Is it the scalability, because you mentioned earlier Kubernetes is like an enterprise ready software, if you had a choice to build a solution, what would make you go towards Kubernetes, even from a security perspective or in general?

Michael Fraser: So first thing I would say is Kubernetes is not for everybody. So don’t, don’t just jump into Kubernetes just

Ashish Rajan: because I’m good.

Michael Fraser: Sometimes just deploying, you know, VMs is totally acceptable for your use case. So don’t like, don’t think that you have to go in Kubernetes because it’s the coolest thing that everybody’s getting into now. But that being said, I would say, you know, when you’re looking your architecture, when you’re trying to figure out what makes sense for the.

Applications, you’re building, there’s some definite benefits to Kubernetes from a scale perspective. And then there’s also benefits from, you know, again, what am I trying to accomplish? Am I trying to also, you know separate my state from my state lists, can I create immutable infrastructure? That’s really going to help drive what I’m trying to accomplish at scale.

And then the other piece of it [00:15:00] is really is. You know, what am I like for us in refactr, we looked at, you know, we needed to have an execution environment to be able to execute the tools that we support and what we call our runners. And so we were looking at Kubernetes is all right, is that something that we can use for the runner environment?

If that’s something we can also use on the application side, how many different services are we building out to be able to scale this thing out so that we can support, you know, the initial customer base and then also supporting. The customer base that scale. There’s also the compliance around it as well.

So one of the things that we looked at is we’re, we’re using AKS and Azure right now for our PAAS based version or platform as a service based version of the platform. And then we looked at, does it make sense for us to take. What we have and have it as a deployable Kubernetes version. And also does it make sense to support the different managed Kubernetes versions out there as well?

And so we were looking at a lot of these like EKS and EKS and GKE. They have various different supports on the government side [00:16:00] from the different impact levels that they support. And so it really depends on the end customer and what they can do on what type of version of Kubernetes. So can they use the managed?

Is this the boarding gov cloud at a particular ILO level? Do they have to support their own? On the DOD side, they’re very sensitive to needing, to obviously have control over anything that they deploy. They also have a cool registry container registry called iron bank where they push out. Certified containers that are available out there as well.

And actually you can go and check that out as well. A lot of this stuff is available. Just the general public, or you can go sign up and check out what they’re doing and. So your question about the air force side. So this is the one time, and I tell some of my air force counterparts.

You know, that I see that the, the DOD is on par with the commercial sector and usually they’re laggers behind. They want to make sure that technology is out there and completely proven. And it’s really interesting to see. How the government side is really pushing the envelope when it comes to being able to [00:17:00] support products like solutions like Kubernetes quicker, and really looking to use it to help, to modernize, to be able to support the more of the containerized approach to so that they can provide, you know, different applications and tools and really to build solutions that.

Work for the war fighter, but also on the commercial side, I’m seeing the same kind of push to, you know, can we pair down particular solutions that we’re creating, where we can have individual services that are provided on a per container basis. And so when you started thinking about that, you start going, okay.

Maybe Kubernetes makes sense. But scale is key. And then also what you’re trying to accomplish the complexity of the type of application that you’re creating and then assessing, does it make sense for me to use Kubernetes or not? And there’s a lot of enterprises out there that are using Kubernetes on their application side, because they need the scale.

They need the capabilities to be able to scale out their, their solution. You know, their application in a more of an elastic type way [00:18:00] that not just the cloud provider support, but that they can also get from being able to scale out with Kubernetes. Wow.

Ashish Rajan: That’s dude, you just keep dropping gems in there.

That’s pretty awesome. So again, I will hope people rewind this and just listen in all the comprehensive overview that you gave us on this. So I’m going to talk about more from a CICB pipeline. I know we touched on DevSecOps earlier as well, and we touched on how a A) Kubernetes is complex.

Security in Kubernetes is complex. You should only go down the path of doing Kubernetes. If you really feel you need to scale at that level, not just outside of a cloud provider as well. Now suppose I’ve made the decision yes. Kubernetes is the right choice for me. And as a security person, I’m going to start looking at,

okay, great. We have Kubernetes deployments. Now, what are some of the initial building blocks from a security perspective that I should be kind of looking at in terms of ticking off when trying to build this into a CICD pipeline, or maybe even start building the foundational pieces for this.

So how do I start deploying Kubernetes security, I guess.

Michael Fraser: So from a, Kubernetes perspective, you [00:19:00] also have to think about all the surrounding things that you need as well. So various things from say credentialing. Again, I see scan and keep bringing that up. Cause that’s very important.

All of the tools that you’re trying to scan your container then also being able to scan, run times as well. And then thinking about all of the other scans that you’re already trying to do with whatever you have inside of Kubernetes, because you’re using Kubernetes for a purpose, right. Which is to be able to push out applications.

And so what am I also doing from an application standpoint? So when I talk about the complexities of Kubernetes, you really have to think about it. Holistically and all of the pieces and parts that you also have to support from an enterprise perspective because your enterprise, you know, maybe you’re just going the latest, greatest stuff.

Like I can do my credentials and secrets from something like HashiCorp vault, which is really nice to add into a CICD pipeline, but I may be using other things like more traditionally on the cybersecurity teams, like using CyberArk for instance. And so I really have to be thinking about how do I incorporate what [00:20:00] the enterprise is using from a.

Cyber security tool perspective and a lot of this incumbent products that came about before we got into not just Kubernetes, but the CICD pipeline perspective. And how can I start building this? These are using these existing products that I have in my enterprise.

Inside of my pipeline and process. And then also thinking about what I’m trying to incorporate into that pipelining process and how I can not just build the infrastructure and incorporate the tools into that process, but also thinking about the implications of what I’m trying to accomplish from a. Whatever applications I’m using too, because I may end up using one particular CICD to do my build process.

And I may use another one. I, the organization do that, another one to be able to build the infrastructure and then combine them all together. So you may look at, GitHub or GitLab to be able to do the CII side as the organization’s using things like code fresh and harness on the more of the CD [00:21:00] or the continuous deployment or delivery side.

So it’s really about figuring what makes sense for your organization and also, you know, which tools make the most sense for when you’re building this out. And then also what is supported from a security tooling perspective, because you may want to use the full suite. You know, like Twistlock or you know, it’s called Prisma now by Paul for networks, but you can also look at like Aqua security and so on.

And I also liked this point out too. Like sometimes the open source version of tools is completely acceptable to start using out of the gate. And then when it makes sense for you to move over to the commercial tool, you know, the commercial offering alongside of that, but you’ve got to start from somewhere, which is really nice, you know, to be able to use some of these.

Open source version of these tools. And we kind of, I think that same thing, that approach at refactr where we believe it’s a mixture of open source and commercially available tools and it’s not skewed one side or another. There’s really a mixture in every enterprise that really make sense when you’re trying to incorporate these various tools into your [00:22:00] pipeline process.

Ashish Rajan: Right. This is making me think. Kubernetes has its own management platform layer or the management layer on it. And you can have multiple pods and it kind of goes into the whole service mesh and everything. if I were to decide on working on integrating Kubernetes security into my , CICD pipeline definitely makes sense.

If you already have security products in place, you just try and incorporate them and then overlay on top of that Kubernetes platform. But if I’m thinking about this and I’ve never done a CICD pipeline with Kubernetes before, am I better off thinking about my Kubernetes architecture first? It comes across like a chicken and egg situation. I know with the CI CD pipeline versus should I was talking with Kubernetes first or should I start with CICD pipeline first? If I don’t have either Kubernetes’s first or CICB pipeline first, what do you recommend?

Michael Fraser: I would recommend you figure out what you’re actually trying to accomplish.

First.

The base of what you’re trying to accomplish. Somebody actually asks the question. In your opinion, which is the best use case for Kubernetes. So that goes kind of hand in hand with what you just asked. When you’re first [00:23:00] off, using Kubernetes, especially if you already have existing applications is going to require a refactor.

So you’re going to have to look at, you know, how am I going to refactor this application to be able to be used within the architecture of Kubernetes. The second piece is if you’re starting from. Greenfield. You really want to look at first what you’re trying to build out from an application standpoint, how many different services that you have and does it really actually make sense to use Kubernetes?

Again, it may make sense to just use virtual machines. You don’t need everything that Kubernetes brings to the table. So you really want to assess the architecture of what you’re trying to build there. And the third piece is the question was when do we need to be able to scale.

So that’s really, when you’re looking at how many different services that you have and you’re building them out. And does it make sense to also have the ability to update individual services, be able to roll out as you expand user usage of your overall application? And then also, what does that look like from an infrastructure standpoint, as you’re [00:24:00] scaling out on the cloud provider.

And so you want to assess those things as you’re building out your application. And so I’ve seen. Organizations where they go straight to Kubernetes and it’s like, you didn’t need to do that. You should have really focused on like what you’re trying to build from a software standpoint first, before you get into like, you know, getting into Kubernetes, because Kubernetes actually requires a fair amount of management as well, to be able to support.

And so if you don’t have the support structure in place to, it’s going to be very difficult if you’re not thinking about those things as well, aside from the. The cyber security standpoint. And that would be the last thing I would say is. You know, it’s fairly easy to run a benchmark against a, you know, a rail seven VM and run some remediation against and make sure that it’s hard.

And then you have your application on it. It’s a little more difficult when you have all the different moving parts in Kubernetes to support your master nodes, your worker nodes, your overall cluster or your different pods, the container. So you have various layers of requirements of what you need to support from a [00:25:00] security.

Posture standpoint and then tooling around that. And so you really also want to think about the level of support that it’s going to take you to support from a cyber security tooling standpoint, let alone just supporting Kubernetes.

Ashish Rajan: I love the answer because I. Often find people underestimating the deployment of Kubernetes

I sometimes see that as like mini data center being deployed. It’s not like an EC2 instance. It’s not a server that you just bring up and shut down. It’s like a whole data center that you just bought in with its own management layer. And it could have sub containers like pods and every, Oh yeah, man.

I think. I’m so glad that you brought up. And I definitely emphasize on that for people who may be thinking about this, that right tool for the right job, but understand there’s a lot of work in supporting Kubernetes so whenever you put it in, there’s another layer to that. Sorry. Go on.

Michael Fraser: Yeah, no, I was going to say, and actually somebody else asked the question around applying runtime security controls on Kubernetes ingress controller versus using service mesh.

So that actually really is a good point, which is. The networking layer on [00:26:00] Kubernetes and also policy. So there’s a whole nother layer that you can have applied from a policy standpoint, using things like OPA or open policy agent or you know, hash HashiCorp, Sentinel, or, you know, various different tools that you can use for policy around your Kubernetes configuration to, or other things that work around there too.

So and then you can also manage policy in some regard in each of the cloud providers too. So you have to think again about the overall complexities of the different layers of when you’re building out Kubernetes, because it’s not just like, Hey, here’s a little infrastructure on the cloud provider and I’m good to go.

And then the other piece that you need to think about too is Kubernetes is. It can be somewhat difficult from a development standpoint too, when you’re trying to ensure that developers can have the Kubernetes cluster experience when they’re trying to deploy things as well, to go test things out as well.

So you always want to think about what that actually means to when you’re trying to build out your development workflow [00:27:00] too, because you know, and then where are you incorporating your. Cyber security tooling into the mix because that also can either slow down your development velocity or if you do it right, it could actually not slow it down at all.

If you have scans and radiation happening in certain places, I was like looking at where you can. Kind of decouple things from your process to make it a better developer experience. So it doesn’t cause any sort of issues with you know speed or velocity.

Ashish Rajan: That’s a really interesting, what are your thoughts on applying?

Runtime security controls on Kubernetes Ingress controller versus service mesh. Like, do you have a preference between using the ingress controller versus the service mesh for layer seven security policy control.

Michael Fraser: I don’t have a preference, but I would say that that’s again, kind of looking where additional management layer.

So if using something like Istio adding that additional layer to the mix, so why are you doing it? Does it make sense for you at scale? I mean, it’s gonna make sense for you at scale. And then also like where are you starting and what kind of [00:28:00] controls do you have in place? So being able to apply.

Security controls in the ingress controller is fine out of the gate. But then you really want to assess, you know, again, the level of complexity that you’re adding with service mesh, and then who’s supporting it. How are you going to support that and how you’re being able to support that at scale?

But a lot of organizations were using a service mesh because it does help to make it easier at scale, but you have to put a lot of time and effort in on the front end to do it right.

Ashish Rajan: Great point man. Frank here as well as mentioned, we started using Oscap and Claire and now evaluating Prisma and Aqua.

Oh, that’s a good one. I think that trend has something in the offering. Yeah. As well, but I thought claire was more containers or Kubernetes

Michael Fraser: Yeah, it is container . When you’re using container scanning, you push it out to a registry and then you use those containers in your recruiting Eddie’s cluster. So you want to think about the scanning process there. Open S cap is a. Is a, how do I put this nicely?

It’s a very cool things about it. That a lot of folks don’t [00:29:00] like using. So we actually, so we support OPNET staff at re factor. We also support Cisco cat assessor, which is a CIS or a tool by Senator for security and some of the nuances of how these tools are able to be utilized inside of your pipeline and process.

And then also how you’re scanning, not just when you’re doing the hardening of your images, but if you’re trying to run real time scans as well. So you really want to be thinking about that. And then also again, how much support is Claire also takes a fair amount of effort to get up and running in some form.

We actually integrated because it’s a better tool to start out with Aqua Trivy, which is an open source version from Aqua security, a great tool and great tool to get started with easier than Claire. Although Claire is fine from an open source standpoint. If you’re looking for a platform.

And an enterprise versions of those, I would definitely look at Prisma and Aqua securities enterprise version. But you also. [00:30:00] Want to start, if you can, with the open source version of some of these commercial counterparts so that you can start getting acclimated with their tooling and how that’s going to be incorporated into your process.

And then I would also kind of shifting more or less look at IAC scan or students who work with like bridge, cruise, checkoff, and a cure Exterra scan. And there’s some others coming out. Checkmarx just released the one, which I thought was very interesting. And so did And I think Checkpoint’s releasing one called shift left as well, but anyway, there’s more and more to be able to look at as well, even before you get to the actual scanning of your images and pushing them to the registry.

Ashish Rajan: Yeah. I think bridge crew has one as well for Kubernetes our teraform. I mean, yeah, there’s just plenty of people coming in the infrastructure as code scanning sort of space. So definitely good. Hopefully it gives you some insight, Frank,

okay, I’m going to revert to some of my questions because I’ve been dying to ask. So now we’ve spoken about right tool for the right job. Looking at Kubernetes from a deployment, whether you actually needed and the complexity around each layer, whether you can reuse some of the already existing security [00:31:00] products, is there like a benchmark or something that people can use to match, I guess their maturity against

Michael Fraser: absolutely. So. There’s been some amazing progress by organizations out there around being able to support benchmarking and being able to do assessments around that. So it’s a center for internet security has just a plain Kubernetes deployment.

Benchmark that you can do assessments against the benchmark itself is free. So you can actually go pull down a PDF on there. What I would say with that is there’s also benchmarks now coming out based support some around the managed Kubernetes again, kind of like. Understanding the differences and the nuances between not just the different managed versions, but also against the just the pure Kubernetes deployment, if you’re incorporating all the infrastructure as well.

And then there’s also NIST so NIST has. Also 800, I think it’s one 90, which covers some on the container side. And there’s a few others as well that you can go out there and look at as well. What I like to say about these though, is there a good way to look at a baseline, but you’re going to want to have some sort of tooling around it to be able to ensure that [00:32:00] you can run your assessments more in an automated way.

There’s a lot of vendors out there doing you know, the ability to scan around some of these benchmarks and assessments that you can use. Everybody from tenable to Tufin to, you know, there’s all kinds of vendors out there that have this type of. Functionality. And so again, you want to look though, like what works for me from a open source version perspective.

Can I use that tool? Does it make sense to use that to get started? And then what would make sense to make the investment into the commercial tool? Especially when you’re looking at scaling and what’s going to make sense there, and then what kind of coverage you’re trying to get as well. So some of these products out there have a lot of coverage around the entire life cycle of your Kubernetes deployment.

Some of them are just point solutions, just purely to do the assessment side. I also work with other companies like a customer of ours called mind point group that builds content out there as well around Ansible. So there’s actually a Kubernetes baselines, essentially remediation content out there too.

So again, you kind of want to look at, you know, the different pieces and parts and is there open source stuff [00:33:00] out there? I can use to make my own, am I gonna. You know, want to use the commercial version because it makes more sense for me to have that from the cyber security vendor. And so you just want to look and assess, you know, what makes the most sense for you, your organization.

And again, Start small. Don’t try to get all the way into the weeds, but plan accordingly. So you want to architect and start small, but you want to, when you’re architecting, you want to look at the big picture. You also want to look at, you know, from an architectural standpoint, what are some changes that would just not work when I scale this out?

So is it something that I’m going to have, you know, am I using a managed version of Kubernetes, but I, when I want to scale this out, I really need to have access to the underlying. You know, infrastructure because I have to do other things in my organization to know what my security posture is on that.

Or maybe I have to support it cause I’m, you know, VMware shop and I have to support it in my private cloud. So you just want to start thinking about things like that and how you’re building out your automation. And then the other thing on the CI CD [00:34:00] pipeline side is really think about.

Building things, modularly where you can run them against different things. And so an example of that is I can have certain scams run against Kubernetes. Can I use that? Both if I haven’t, you know, Kubernetes up and running in my private cloud, into my public cloud. And so you really want to think about reusability when it comes to creating your pipelines so that you can as we like to say in the software engineering world the dryer approach don’t repeat yourself, right?

You want everything to be. Automated from a repeatable standpoint and you don’t want to have to build things, you know, multiple times

Ashish Rajan: that’s a great answer. And for me, personally, as a CISO, I always feel my calls are usually if I’m going for an open source, which not minus the support part. If I don’t have the team with a skillset that can code and can deep dive into this, I feel like that should also be a consideration for a lot of people kind of considering, Hey, should we go open source or commercial?

Definitely need the understanding of the product space, but I feel if you don’t have the [00:35:00] right team to support you build that cybersecurity or kubernetes security at scale in a CICD pipeline, you should probably look at some kind of commercial offerings. What are your thoughts on that?

Michael Fraser: So that’s a great question because a part of what I say about open source too, is don’t underestimate how much time and essentially cost it is to the people supporting it in your organization.

And to the second point who in your organization is going to be supporting this. But also on the flip side of that, don’t think of that the commercial offering is just going to solve all of your problems too. So you really want to. We’ll look at how you balance it out. And somebody, I think Kevin said Oh yeah.

Use open source first to find easy fixes, which helps you become both more secure and then convince management to invest more into security. And I would also say like, Proving out that these tools will work for you on the open source side first to show something that’s quantifiable. So because you know, the executive levels who you’re trying to sell in your organization are going to want something that they can actually see, and that you’re able to actually prove to them [00:36:00] to be able to get the.

Commercial version of what you’re looking to do. So I would absolutely agree with that. And then I, I also agree with Kevin there’s free tiers, there’s community additions, there’s open. So look at what that means a lot of times community edition, and we have this a refactor too, with a hosted free version.

So you don’t have to spend all the time trying to set something up. You can get out there and start using it. But trying to still keep the, the level of effort on your end, as minimally as possible. Cause you’re really doing a proof of concept yourself when using open source tools to prove out on the front end when you’re building this stuff out.

And then CICD pipelines can be very complicated. So again, Start small when you’re trying to prove this out so that you can show something to your executive leadership to be able to get the investment and buy-in to this. And then the other piece that I would say too, is really look at it. Commercial tools that can also help you with collaboration because you really want in cybersecurity working together with your dev ops and cloud architects and so on, because this is a collaborative effort.

You’re building a solution. You’re not just building [00:37:00] something for one team. This is going to be something that you have to have multiple teams involved in. You know, and, and I said this on another talk last week, or it hashi talks, it’s very cliche, but. It’s really about breaking down the silos from a culture standpoint, because you can’t have cybersecurity siloed away from this because it’s an integral part of what you’re trying to build.

And it’s not just Kubernetes. It’s like an agile automation process where you’re trying to build a solution in Kubernetes can be one of those solutions in your process, but you really want to think about the culture shift and the culture change that you’re trying to get into an organization.

And it also requires the CTO and the CISO and the CIO to all be communicating and working together, not working against each other because, Oh, I want this for my budget, or I want this for my budget. It really has to be a collaborative effort at the top executive level when it comes to this culture change internally, which is what we’re talking about too.

Not just about Kubernetes, but. Around this whole modernization and where everybody’s going from a technology standpoint.

Ashish Rajan: So glad you touched on the [00:38:00] culture part and whether it’s DevSecOps, kubernetes or containers, all of that would fail. There’s no culture to support it. So very well said

it brings me to the next question. We kind of spoke about some of the challenges in CICD pipeline. Are there any current fires that you feel that are happening in the space that should be addressed in terms of kubernetes security? Like I think, or misconceptions that are there around in terms of reality?

What’s the expectation.

Michael Fraser: Yeah, I would take one step back from that. So we all know about the major solar winds, breach, which is really a supply chain issue. And I talk a lot now about a software bill of materials, really about the overall supply chain, knowing the integrity of what you’re trying to build, and that applies to everything.

Right. So that does apply to Kubernetes too. Because you have to understand what’s in your process at every step, whether it’s at the very beginning, when you’re building your source, go to all the way to the end, when you’re releasing a production and then anything that you’re building around it being.

Software defined [00:39:00] or as I like to call it, IT as code, but it’s like if it’s software, it’s in a pipeline, you need to know what’s going on in every step in your pipeline and process. And then also, you know, things like solar winds, the key there is to know the integrity of what your act what’s actually in your pipelining process, really focusing in on , can I have more evidence?

In this step that I can have available. Can I verify the integrity of these, you know, whether it’s external libraries or other things like, Hey, I’m pulling in a container image from a public repository, you have to think also about like, do I need to have my own private repository?

Do I want to be able to ensure. The integrity of this myself. So you have to be thinking about all of the things as you’re looking to build out your pipeline and process and Kubernetes as a piece of that process. But if you shift all the way to the beginning of that process, you want to incorporate the, SAAS and DAAS, and I scanning all the way at the very beginning of your process before you even start thinking about Kubernetes.

So it’s really, [00:40:00] again, holistically thinking about it as you’re creating solutions. I think Kevin’s said something else. Collaboration tools. Oh, you keep your audit risk team from sending. Yes. Everybody hates spreadsheets, actually I’ll hone in on another thing around this. So we actually center for in it security.

We work with them closely and we’re helping them to automate their benchmarking process because the benchmark is just that a spreadsheet right now, or the assessment for the benchmark process. And it’s funny how this process just generally out there in the industry. It’s still a, a lot of manual handoffs that exactly to Kevin’s point.

And so if we can start building automation around it, even if it’s for visibility to know what’s at every step of my pipeline process, do I know exactly what tools and automation, content and configurations in there? Then I can start also knowing if this configuration drifts or if somebody introduces a malicious you know, package or dependency, I can have visibility of that as quickly as possible.

And so it’s really, again thinking [00:41:00] about this holistically, because you really want to be thinking about every step inside of your process. and if it’s a spreadsheet. That’s a manual process and there’s a high likelihood that there’s going to be introduced some sort of human error or something happened.

Or somebody didn’t look at a particular control because it’s a manual process. So thinking about where you can automate it. But I also come from the other side, just like I’m saying with Kubernetes, from a complexity standpoint, you also want to think about automating a certain percentage. Don’t try to go out there and say, I’m going to automate a hundred percent of this.

If you can automate 50, 60% of it. It’s still okay to have some manual steps inside your process. The goal is that you’re making some progress to try to automate as much as possible so that you can be proactive in your process. Not reactive when something happens, there’s an incident that happens where now you’re spending a ton of time trying to figure out what happened.

And you have no visibility on what’s going on in your, your process, your pipeline, your process.

Ashish Rajan: And that brings me to the question of maturity. Then if someone listening to this has been doing [00:42:00] kubernetes deployment for some time in their environment. What’s an example of like a really mature kubernetes what would like a top-notch mature kubernetes deployment look like versus I guess it’s the beginning one is what we’ve been talking about.

Michael Fraser: That’s a good question. I guess the question is, are there mature Kubernetes deployments out there? I’d probably say. Probably some large organizations like you know, Netflix or capital one, people have been doing it for a long period of time. But I would also say there’s like the use case around, why do I need to be using it?

I need a scale. I need to be able to have the capabilities of a lot of the benefits that Kubernetes brings to the table from an automation and self healing standpoint. But I would also think about. From a maturity standpoint a lot of people are looking at Kubernetes because they’re moving from on-premise to, you know, the public cloud or cloud native.

And so it’s really thinking about. What am I trying to do when I’m re factoring my applications? What does that look like? And then the other piece is a lot of organization, you’re starting Greenfield or you’re a [00:43:00] startup. It’s going to be easier to assess. Does this make sense for me?

From the very first day that you’re building out your product. But for people that are refactoring their applications into cloud native, you really have to assess what that looks like from your own maturity standpoint. Do you have existing you know, a pipelining process in place?

You know, so does that make it easier to get where you’re trying to get with Kubernetes? And then also again, all of the things that go along with it. So from a cybersecurity tooling standpoint, like what do I need to have in there to just have my, my baseline or my MVP. And then over time, what does that look like?

From a maturity standpoint where am I trying to get with that? Because it’s going to be a consistent evolution of what you’re trying to do. And then also the thing to think about is net new products come out too. So you really want to be. Careful about locking yourself into any, and we’ve already talked about like vendor lock-in and at some point you’re going to pick pools.

You’re going to pick. A solution that you’re going to use, you’re going to have some sort of lock-in, but you want to think about the pieces and parts around your Kubernetes solution. And [00:44:00] can you decouple or pull that out and add in something else at a later point that makes it easy to change and, and slightly change the architecture, but also continues to help you evolve and mature your overall offering.

And I guess the last point I make is. I would say that most organizations are still not super mature when it comes to Kubernetes. So it’s still an ongoing evolution of just organizations embracing it and looking at it as a, as, as an option for them when they’re building out their applications.

Ashish Rajan: Right. And I want to interesting question here as well. Does Kubernetes security contribute to an organization’s overarching data loss prevention program? If yes. How

Michael Fraser: does it contribute to an overarching data loss or DLP program? That’s that’s a good question.

I mean it can, but if you’re just looking at it again, it’s something that you’re trying to support for the enterprise at large. So I would just look at it as. Another solution that you need to have coverage on and being able to know how that ties into your existing DLP [00:45:00] program, and then any tools that you’re using from that perspective tool.

So if you have you know, things in line that’s able to pull data out of, the network that you have around Kubernetes or the infrastructure that’s fine as for like anything else that you have. There’s the applications that are inside Kubernetes. You would still cover them as you would with any DLP program and tool set around that.

So I would just, you know, it’d be something else that you would just have coverage over in that program, but absolutely should be something you should be looking at to cover in there, especially if you’re supporting a product specific production workloads, but if you’re doing Kubernetes, you multiple have multiple environments you’re supporting as well.

Ashish Rajan: Awesome. Hopefully that answers your question, man. The other question that I have is from Ubaid and the question is what are your thoughts on striking the right balance between service velocity, scale architecture, security and running costs. As an example, when dev teams requests dedicated EkS clusters versus kubernetes multitenancy security and cost.

I love this question. What are your thoughts on that? Tough one though.

[00:46:00] Michael Fraser: So it is about striking the right balance and also like, what are the developers needing from you? You need an environment they can do with their day-to-day job.

And so the question there is, or are you building an environment for them that actually covers kind of what you’re talking about. So is there enough architecture in them for that and actually do their day-to-day job? As they’re building software from a security standpoint, like what are you securing in those environments versus what you have to secure in production?

And then it can be one on the same, but most likely you may have additional coverages that you have to have in production. And then what is the bare bone minimum that you can have to strike that balance? Or a dev environment versus a full production environment. So you really want to look at that. And then the other thing is I would use the public cloud wherever I could, if I can control, spinning up and spinning down these environments versus having them on my own infrastructure or having the same.

You know, layer the automation layer over that as well. Cause I want to make sure that I know how much uptime [00:47:00] that they require and make sure that infrastructure is only up for that time. And the last piece of that is from a security standpoint, where do I need to incorporate security scans into the process to ensure that I have the coverage I need as the enterprise, but also.

What is going to impede the developers speed and velocity. So can I do that, you know, off of particular branches that that’s not going to cause him to slow down and do I have particular scans when I’m trying to hardened around the infrastructure that they’re using too. So a lot of times you may not have fully hardened infrastructure for a dev environment that you’d have in production.

And then also, how are they accessing that environment too? So there are some. Really cool technologies out there. You can use also from a hosted standpoint coder.com is another one. Great interesting solution in there looking at like moving towards a hosted dev experience to kind of help solve for this.

So there are companies out there trying to solve for this too. Aside from you doing this yourself and having your own. Dev environment spun up. So hopefully I answered your question. [00:48:00] Thank

Ashish Rajan: you. I think Kevin just has another Gem he’s dropping in there. DevSecOps software lifecycle. Infinity diagram comes from us DOD.

I’ll definitely encourage people to Google that I’m curious myself. So I might look that up later. Thanks for that, Kevin, it’s brought another question on this right now, after listening to you, Michael people have gone pumped like, Oh my God, Kubernetes is the bomb. So is it a place where they, can probably kickstart this learning process of kubernetes as well?

Like I think it’s worth calling out. I feel like as security folks who are being introduced to containers, kubernetes, and cloud, and all this, there’s obviously an education gap as well.

Is there anything that you recommend as a good source for learning about kubernetes that you have found useful as well? And others may find as well from a security perspective.

Michael Fraser: So to Kevin’s point and I love that diagram too. And have been working closely with the dev sec ops initiative with the air force.

Now that’s spreading out with Navy and army and so on. There’s a lot of good resources out there [00:49:00] around Kubernetes and DevSecOps and what they’re trying to accomplish you know, from a, from a DOD perspective, but a lot of this information’s out there. That that’s totally accessible to anybody that wants to get into it.

So that would be a good place that I would start. There’s you know, the, the chief officer of the air force Nicholas, Julian, he has a, there’s a whole site about all this stuff. Some of the stuff I mentioned earlier around like iron bank and there’s a thing called depo one, which is, has all kinds of cool content out there.

Then there’s just a lot of presentations out there too, that I would go check out that you can get information about as well. , I’d say start off small, even stuff like, so we’re doing a refactor. We’re very focused on, you know, Hey, you should be able to start building out pipelines that maybe just spin up a managed Kubernetes cluster.

And then you can look at, okay, now, how do I incorporate these particular tools into the process? Then I can start adding in security into the mix, but you got to start out with knowing what. Kubernetes is and how it works. And then how are you going to apply security to it? And so you really want to start [00:50:00] small, get a Kubernetes cluster up and running.

I’d recommend doing a. Managed version of one of the cloud providers. And then from there, you can start incorporating different tools into the process, but start small.

Ashish Rajan: That’s awesome advice. , I just want to get your thoughts on where do you see the community space go?

And I know that some people have almost stopped talking about containers. It’s almost feels like no, one’s talking about containers. Everyone’s talking about kubernetes. Do you feel this is. Here to stay or where do you see the future? For kubernetes?

Michael Fraser: I do. I think that we’re, we’re just kind of on the verge of the, the overall opportunity for Kubernetes as a whole.

And then I would say one step above that from, from DevSecOps, which is where I live and breathe. And it’s only going to get more and more. It’s only going to grow bigger and bigger. So I would say that , it’s going to become a much larger market. The other thing I would say to that though, too, is we’re going to see a lot more.

AI ML type of offerings applied into there because there’s people are scaling and growing out the data sets on [00:51:00] what they’re creating from a Kubernetes cluster or crew 90 standpoint. There’s also going to be a lot of data that we can use to further enhance and make it better. So I’ll see that.

And then I’ll also say that Kubernetes is not the end all be all. I, we talk in two or three years from now. There’ll probably be something. New out there. That’s going to be, you know, it’s going to take over what Kubernetes is from a popularity standpoint where people are pushing, but I think it’s, you know, we’re just at the tip of the iceberg on what’s going to be possible in the growth from a from a kubernetes standpoint.

Ashish Rajan: Awesome. I’ve got my favorite person for Singapore here. Jerome Bonzoo USA. Casell run is a good illustration of how agile and DevSecOps bring value to organizations. They go another, another great resource.

Thank you so much for this man w people I’m sure I’ll be.

We want to get in touch with you and know a lot more about kubernetes and everything else that Michael and refactor are doing. Where is the best socials for them to reach out to you?

Michael Fraser: So you can look up. It is code. That’s kinda my trademark phrase right now. And that’s how you can find me on LinkedIn, Twitter.

I [00:52:00] think I’m on Instagram Facebook, but not because I want to be on there, but I’m on there as well. I’m actually on, I just joined a clubhouse not too long ago, too. So I’m on there. And actually I’m thinking about doing some going to do it. Some rooms. I have some other industry leaders. I want to start doing some conversations on there.

So that’ll be. That’d be pretty good too. So actually we should probably do something on clubhouse

Ashish Rajan: too. I know I was going to say, cause I moderate the rooms for a cyber security cloud security over there as well. So if anyone’s in clubhouse definitely feel free to connect and we should definitely do something over there for more conversations like this, man, dude, thank you so much for coming in and I definitely cannot wait to bring you back in because I’m pretty sure a lot of people would have once you know, the next layer of kubernetes as well, once he goes deeper into this and what else is out there from a.

Scaling perspective. So thank you so much for coming and I’m definitely looking forward to having you back again, man. Thank you.

Michael Fraser: Thanks for having me. Yeah. Looking forward to our next session and yeah, I could talk for hours on this, so I’m totally open for another episode of the future.

Ashish Rajan: Awesome. Awesome.

And for everyone else we will see you next week. [00:53:00] Thank you so much for your time. Peace .

‍