Ashish Rajan: Hello, welcome to another episode of Cloud Security Podcast. In today’s episode, we have Jimmy Mesta talking about Kubernetes best practices. Yes, this is the Kubernetes month. We are starting a new month, which is a Kubernetes month on October in celebration with KubeCon. That’s gonna happen later this month in the US. 


So, as I mentioned in the first episode, we’re talking about best practices. Jimmy was part of the OWASP top 10 for Kubernetes project with OWASP. We went through the top 10 that they had listed out as people should know about. We also spoke about what do you start with when you’re starting with Kubernetes today? 


We spoke about the difference between managed unmanaged and to the fact that there are so many bare metal Kubernetes still running on the internet, or, well, in some organizations. We spoke about some of the challenges around having a certificate authority, the secret management towards the end as well, and a lot more conversation around kubernetes best practices, not just from what the top tens are, but what you can also use to mitigate them as well. 


Hopefully enjoy this episode. As always, if someone who’s trying to learn about kubernetes security or is [00:01:00] excited about the KubeCon month, definitely share the episode with them so they get to know about Kubernetes. We have a few more interesting topics coming up for the rest of the month. So if you are someone who’s probably listening to the cloud Security Podcast for the second, third, or fourth time, I really appreciate if you can hit the subscribe or follow depending on where you’re listening on the podcast platform. 


If you follow us on YouTube and LinkedIn, follow us there as well. We definitely create a lot of content outside of the audio, which is your short form content on TikTok, Instagram and stuff as well. So yes, we are there everywhere talking about cloud and cloud data security basically. So I hope you enjoy this episode with Jimmy Mesta from KSOC and I will talk to you next episode 


Jimmy Mesta: by bringing developers and security together. 


You don’t have to choose between speed and security, develop fast, stay secure. 


Ashish Rajan: Definitely was while I get the hype going, then you gotta you gotta get the hype 


Jimmy Mesta: up. Yeah, that’s going, it’s going well. I have my afternoon coffee. The kids are asleep, [00:02:00] Napping life is, 


Ashish Rajan: Life is good., I’ve got my little cup of coffee as well, man. We are talking about kubernetes best security practices. So man, to start off with, for the one or two people who don’t know you in the crowd can you give us a brief intro on who you are and where you are today? Sure. Yeah. 


Jimmy Mesta: Well, thanks again for having me. I feel like kubernetes security is the gift that keeps on giving. 


My name’s Jimmy and I am the CTO and co-founder of software security platform called KSOC. , my journey started about 15 years ago, though. I inherited a kind of disentangled WAF program at a very large enterprise right outta college and kind of got into Appsec. 


And I actually have a degree in security that was sponsored by our friends at the NSA, which was an interesting story. But I kind of bounced around between offensive pen testing style roles and defensive, I guess, in leading compliance and security teams at various companies. But about six years plus ago, I took [00:03:00] a , a nose dive down in the rabbit hole of Kubernetes. 


At 0.8, the company I was at, we adopted it. We rolled our own Kubernetes using Kops , and chef and all sorts of hairy things, pre RBAC pre anything. And Yeah, I just kind of got very into the space, what it was doing to the software development life cycle, and ultimately kept building on that skillset , and build a company out of it. 


So here we are today. 


Ashish Rajan: Wow, man. I think very interesting to kind of go into this space as well from an offensive side as well. And , worthwhile calling out, I don’t know how many people out there understand Kubernetes as well. They kind of hear the word and because it’s a word that translate on something else as well. 


From a technology perspective. What is Kubernetes for people who, I mean, just explaining Kubernetes is like a thing. So how would you explain human to people who are in the tech space? 


Jimmy Mesta: Kubernetes as a household Word, has taken quite a few meanings and[00:04:00] it started as, , A container orchestration platform. 


If you go to kubernetes.io, that’s what you’ll see. I think what we could talk about today is like the nuance behind that. So Yeah. , to think about what a container is, right? Which is really a software packaging mechanism that shares, , the hosts kernel, and it really lets developers build and ship software in a very kind of modular approach, repeatable fashion. 


And Kubernetes was built to give you several APIs and guarantees around deploying, orchestrating, and managing multiple containers inside of a compute platform. , It’s like a data center inside of your data center, right? It does networking, it does secrets management, it does kernel level stuff. 


We have its own, it has its own access, control and, and authorization layers. It’s very much , a complex set of APIs that , you ultimately run software on and does a lot [00:05:00] 


Ashish Rajan: it. And I think that the data center within a data center is what makes people like get scared of it. It’s like, holy shit, there’s another data center inside a data center. 


Yeah. And you touched on containers as well. Cause another question that I get quite often is the whole Kubernetes in container. What’s the difference? 


Jimmy Mesta: Yeah, so I mean, a container is like the atomic deployable unit of, , your application representation inside of Kubernetes. You don’t need Kubernetes to run containers. 


There are other ways to make a container, , the process, start and serve traffic and do networking and things of that nature. Like you can just do docker, run and just open a port and serve your application that way. But the reason we have Kubernetes is really those strong guarantees, right? 


It’s saying, I want x, , end number of replicas. I want this pod to talk to this pod. I want, , environment variables to always be present in , this running container. It gives you [00:06:00] the utilities you need to run containers at scale. , I mean, even a small scale doing docker run doesn’t really work. 


So , for local development and things like that, , using a single container or even Docker composed, things like that, where you’re kind of, , merging multiple containers into one. System. That’s great. But Kubernetes , is when you need to , run it in production. 


Ashish Rajan: Right. And the other one, which is another popular question for us is the whole Docker space, cause especially after the news came out , that kubernetes is stopping support for docker and being like, Oh wait, so it’s different. 


So what is the relationship between kubernetes and Docker there? 


Jimmy Mesta: Yeah, so Docker is a, company right, to kind of, to start with, right? I think that’s easy to forget that like there’s actually like a for profit venture backed company. Now there are open source projects that Docker kind of consists of. 


And Docker kind of led the container renaissance, right? Like [00:07:00] it gave developers. This very clean CLI that stitched together a lot of underlying kernel technologies in abstracted way. The need to understand how to like create a container the hard way or from scratch and just let you run containers. And it’s relationship to Kubernetes really is Kubernetes runs. this component on every node called the kubelet. And the kubelet takes requests from the API and ultimately you have a container runtime interface. But does something like a docker run, right? So it’s not that we’ve abandoned Docker in general, we’ve established other patterns, right? Like cryo and container D docker’s pretty monolithic as a utility to orchestrate containers and create containers. 


When really you need to pull an image from a registry. Unpack it and run it ultimately using what we, , call run c that to actually run the container process. [00:08:00] And we have eliminated the need for all of the things that the entire Docker, Damon, and Docker ecosystem provides. But it’s not as scary as it sounds. 


Most of you won’t even notice it, right? Like you use Kubernetes, you’re not really. Dealing with it at that level. So it was sounded scarier than it was, I guess. Yeah. I 


Ashish Rajan: mean, I, the simplest way to, at least the way I explain to people is like the same as Google Chrome on your MacBook or wherever. If for whatever reason Apple decides to stop supporting Google Chrome doesn’t really mean you can’t use a browser anymore. 


You just use a different browser and Sure. , that’s like the simplest explanation I could come up with. But yeah, it’s not like your thing is gone, like the honest 


Jimmy Mesta: it and your docker file and your image and all the things that you’re used to doing stay the same. Yeah. Pretty much 


Ashish Rajan: the plumbing change. 


That’s right. The other question that I also get is, cause you kind of said this in the beginning when your first 0.8 that you were working on. Using chef and other things to kind of create [00:09:00] kubernetes, which I imagine was the bare metal kubernetes. Mm-hmm. Yeah. Is that still a thing? 


Like, and what is it, I guess, for people who already know what that is? 


Jimmy Mesta: Yeah., if you rewind in the early days of kind of Kubernetes as a open source project, we didn’t have managed Kubernetes like AWS and the big cloud providers , didn’t have these very like, easy to use, managed environments. 


So you built your own clusters, right. Kubernetes the hard way as our friend Kelsey High Tower would put it. And Yeah. That meant you needed to stand up nodes. You needed to place different components on those nodes, , certificate authorities, the kubelet like the container run time, like you had to deal with networking those nodes together. 


So you, , we use. Chef to do some of that. I mean, you could argue that wasn’t the best mechanism, but Kubernetes, , to your question, like on bare metal exists in a big way today and bigger than you would think. I think there’s really, there’s still a lot of [00:10:00] large organizations choosing to go into, , building their own clusters. 


It’s now, maybe it’s not quite as large as the managed space, but , there’s a lot of big companies that are all in on, on running their own kubernetes. We have the tools available too, like Ks , Cube adm, and , there’s Terraform modules. , there’s great bootstrappers out there that you can run your own Kubernetes now. 


You better have the expertise to deal with that. I guess that’s all that needs to be said, right? There’s there’s more to it , than running managed Kubernetes if you’re doing it yourself. 


Ashish Rajan: Yeah. To what you said, it’s a data center, insider data center. , you’re technically having a team skill set. 


We can run an entire data center for lack of better word, which is running, which is based off Kubernetes 


Jimmy Mesta: pretty much. Yeah. Now you, are lovely shared security model that the cloud provides is gone. Now you own every single flag that the API server consumes. You own the security of the kubelet, all the different things, networking that maybe you offloaded some of that to [00:11:00] AWS or something in the past. 


You don’t get to do that anymore when you build your own. 


Ashish Rajan: Yeah. Wow. And, for context of people, , what is a managed kubernetes as well? So we should probably clarify that cuz think most of us would know it, but some people who may be the first time may not know what it is. Yeah, 


Jimmy Mesta: yeah. It’s , pick your major cloud provider. 


The big three all have what they call managed Kubernetes. And essentially it is kind of two things. One,, under the hood, the Kubernetes API, if you were to run your own . cluster in your own data center, you are responsible for that instance or multiple like VMs , that manage the API server and the control plane manage Kubernetes is, you don’t have that ability. 


Your cloud provider is. Is responsible for that. They give you the API, , they give you some configuration. Yeah. There’s switches and toggles that, you can change as you bring your cluster up, but you don’t have access to ets e d really, you don’t have access to a lot of the [00:12:00] underlying control plane components, which is probably great for 99.9% of organizations. 


Right? Like to be honest, and, that mixed with easy, well, depending on the cloud, easy to spin up clusters, right? They’re part of node pools that give you container optimized like operating systems under the hood. The networkings kind of lay it out for you. You have a path to, , expose the Kubernetes API and, and they have deep integrations, which is another key point. 


Like AWS IAM is kind of like baked in Azure AD, like these things work well. Ish. With managed Kubernetes out of the box, when again, you build your own, you’re doing it like all by yourself. Wow. So like GKE inside of Google is a good example of like, they have tons of features that don’t even exist in Kubernetes proper. 


But, shielded nodes, workload identity , actually using the builtin kms functionality of your cloud to [00:13:00] encrypt secrets that then get persisted to c d. Those things are. Very hard to do in your own DIY clusters, but manage makes it easy, So, 


Ashish Rajan: Wow. Yeah. There you go. So well, Do you want the hard part or would you rather just like, use a share responsibility model than move on with your 


Jimmy Mesta: life? 


I think most companies want the easy button. I mean, they should. They should, right? 


Ashish Rajan: Yeah. Hope so. So this next question is from , what security plan you do to protect kubernetes cluster nodes and pods 


any particular security baseline you recommend? And I think the only, that’s a great question. I’ll, the caveat add is probably talk more from managed perspective as well. What’s there? 


Jimmy Mesta: Yeah, so the kind of expected answer here is like the benchmarks that exist, CIS NSA hardening guidelines they offer a good start. 


I will not go into a vendor pitch of why you should talk to me about our product, but aside there are lots of open source tools out there, and we’re even seeing more things like [00:14:00] RBAC police from, , Palo Alto Networks, which is like specifically looking at some obscure, RBAC configurations inside of the cluster. 


, kube bench is a classic CIS open source kind of CLI utility. There’s , in the world of managed Kubernetes, like each of them have their own CIS benchmarks, I believe. I know GKE has one. and eks, I believe has its own, I’m not positive on a AKS at the moment. But your cloud providers should be able to test for some of that too, right? 


Like if you use guard duty or at, , what is it, Azure defender and, and things like that. They’re gonna surface CIS managed Kubernetes specific issues. And if you’re rolling your own Kubernetes, there’s like 50 more checks that you need to do . Yeah, so I would say like also PCI has some stuff now, which is awesome. 


Oh, wow. Yeah. P C I DSS has recommendations pertaining to container orchestrators, Kubernetes yeah. So that Roy [00:15:00] McCune has written a few really great blog posts out there. Diving into each of those. Yeah, so there’s a variety of ways to get your baselines. And most folks start with CIS. 


Ashish Rajan: Sweet. I think hopefully that’ll answer your question as well Vineet. Great question related question from Ameya, which is OWASP top 10 and how they are related to Kubernetes. Or let me say, if I am using Kubernetes, then how can I remediate top 10 OWASP in my cluster? Cause the follow up also is that which one to follow. 


There are too many standards, so , which is 


Jimmy Mesta: we live in that’s the world we live in. Correct? 


Yeah. 


Ashish Rajan: So what’s your thoughts on the whole OWASP Top 10 and 


Jimmy Mesta: Kubernetes? Yeah, I mean, we could just get into it, I guess. Yeah, the, 


Ashish Rajan: or you get into it, would you say the OWASP top 10, which is the web top 10, Is there any relationship between that and the Kubernetes One? 


Because how people, when they look for OWASP top 10, they’re going like, Oh 


Jimmy Mesta: yeah, it’s always the web top 10. No, there’s really no, there’s relationship that it’s a top 10 [00:16:00] project within the OWASP Foundation. Right. But that’s about it. Like there’s no crossover. 


Ashish Rajan: But if there’s an application running on kubernetes which is a web application, I guess, then there you can use both, I guess. 


Jimmy Mesta: Sure. You would totally. So the OWASP top 10 The application security top 10 , has evolved a lot. It used to be very like cross-site scripting, SQL injection, Like very much like this is a vulnerability and here’s why it’s a problem. 


Now we see things like insufficient logging or broker authentication broken off. Yeah. Yeah. And there’s definitely crossover, right? Like I think, , the AppSec and infrastructure security are the Venn diagrams, like getting smaller and smaller. You can’t really just, I don’t feel like you can just do AppSec anymore. 


Like it’s, yeah, you can be a total expert, but it’s like the exploit almost always has something to do with the underlying infrastructure. SSRF right? Server [00:17:00] side request forgery. That is different in Kubernetes versus other systems. Mm. Remote code execution. Great. I found it in my web application. 


What am I gonna do with it? Well, in Kubernetes you can do some damage, right? So I think that there’s a convergence for sure happening today of AppSec and infrastructure and cloud security. You kind of have to know everything, unfortunately. Yeah. To 


Ashish Rajan: be, become a full stack person over there. Now you, it’s like a full stack developer, full stack security person now knows cloud and application security as well. 


Jimmy Mesta: Yeah. That’s how you make the big bucks, I guess. Yeah. . 


Ashish Rajan: Before we kind of jump onto the OWASP top 10 for Kubernetes, which you’ve been part of, I also wanted to kind of talk about if there are things. We may, building a hypothetical environment, the data center within data center, are there things within Kubernetes that could help build this thing or, I mean, I guess is, and, and maybe would you start with OWASP top 10? 


Is that where you would want people to start with when they’re trying to build a Kubernetes cluster or pod? 


Jimmy Mesta: Kubernetes as [00:18:00] a open source massive project offers the tools in, built in functionality to have a relatively hard end cluster, right? Like there’s a lot of really good stuff that you don’t need to go buy, bill, adopt other things, right? 


Right. So for, , things that come to mind are the now deprecated pod security policies, which are, , now pod security standards the network policy. API, the objects themselves. Like, you don’t need to use a service mesh, you don’t need to use these things that, , maybe are a little intimidating. 


There’s RBAC, right? Like RBAC is a security feature, but also when you do it incorrectly, it’s its own vulnerability. Kubernetes offers a lot out of the box. The problem is like a data center, right? Like you can build a, government, , style data center that’s super hardened and secure. 


Or you can have the most open Swiss cheese looking infrastructure possible, like everything in between. So like Kubernetes is [00:19:00] flexible. Yeah. Which means that you as the operator, you have to turn the knobs and the levers to get to a point where you’re comfortable. And I think those knobs and levers are. 


Complicated. Right? And to answer your question, the OWASP top 10 , for me to create it , was like step one on your journey, right? What’s happening here? Instead of the fire hose of everything all at once, how do I understand what’s going on in a cluster? 


What do I need to learn in depth , to manage this thing and feel confident doing it? That’s really the goal of the top 10. It’s a guide book. It’s a reference doc, it’s not a prescription, it’s to start your journey, 


Ashish Rajan: right? Cause to your point, then you can probably use the, OWASP Top 10 for Kubernetes along with say, CIS benchmark or the PCI benchmarks or whatever other benchmarks are there as a way to. 


So if I’m, I don’t know, starting today and I’ve been given a Kubernetes cluster, just one, not [00:20:00] multiple, just one kubernetes cluster. And lucky if you just, Yeah, yeah, yeah. I mean even that, I imagine this big thing, but and it, cause it could scale. I think what people underestimate is how big even one cluster can scale. 


It’s amazing. And I think if I were to start today with just securing one cluster, if I have to what Vineet said and what Ameya said, have a couple of security standards as a, okay, these are the baseline that gives me like a do this, do this, do this, do this. Then the OWASP top 10 for Kubernetes is like, oh, what should I be looking out for? 


Like the first one, which is the security misconfiguration. Sure. I mean, , Cause you have the cheat sheet as well, right? I think you help write the cheat sheet for OWASP as well. Yep. 


Jimmy Mesta: there’s , the OWASP itself, I think. Everyone assumes it’s all appsec all the time, but when you actually like, look at what OWASP as like an all the contributors, there’s a ton of stuff out there that has really, , not really appsec , it’s, there’s all sorts of top tens. 


and when I [00:21:00] first kind of published the Kubernetes top 10, there were definitely some people who were like, What? Like there’s only one. And it’s like, you should probably look in 2022. There’s like, there’s probably like 12 of them. And , that’s good. Like the world needs more, , educational reference material that we’re all working on together. 


Like yeah. So it came out with , the cheat sheet is held. Where to look, how to fix it, the details around, , some of these problems. And the cheat sheet is, there are so many cheat sheets. If you go to just like cheatsheets.owasp.org, there’s one for anything that you could imagine for the most part. 


So it’s cool. Like there should be there’s a docker one, there’s Kubernetes one, there’s serverless, there’s like all sorts of different things. , e every web framework out there. So yeah, the top 10 is, it’s good for things like, it’s the, , we can discuss the top 10 in the span of a podcast or, or screencast. 


And like you could, , cover a lot of ground and I [00:22:00] think that’s the point. 


Ashish Rajan: Yeah. Yeah. And think to your point then, I think for people who are looking at, because these are free resources as well by OWASP they don’t like, not like a paid wall or anything. And for people to be able to at least have access to a cheat sheet, A top 10 Kubernetes and a few benchmarks, that’s a good enough start for anyone. What the world needs more of it, I think I’ve got a comment from Ankit over here as well. Interesting topic definitely something industry needs more dedicated effort on. So thanks for sharing that as well. Ankit. But yeah, there’s definitely a lot of need I think, and for people in the cloud security space, there is an OWASP Top 10 for Cloud Security project as well. 


So if you’re in a contribute and maybe make it a thing and I believe it’s a project is still in draft, but to what Jimmy mentioned is a lot of projects there, which probably needs a breadth of fresh air at this point in time. So definitely check those out. Maybe we can use that as a good segway into switching to the OWASP top 10 Kubernetes. 


So the first one, which is in that top 10 Kubernetes one was security misconfiguration I know we only have half an hour to [00:23:00] kind of . Go through 10. So you’re gonna try and squeeze as many as we can. 


But for security misconfiguration, Kubernetes, , what do you think people should know more about? 


Jimmy Mesta: Yeah. So Kubernetes as a orchestration platform, it’s just one big collection of configuration. So it only really makes sense that it’s the first one that can go wrong because everything is a configuration. 


The intention here was really to call out. The flexibility of Kubernetes and what it gives you. So think about a container and how it can interact with the kernel. It can be privileged, it can have excessive system call capabilities that you don’t need. There’s file system configurations that, read, write, depending on how and what the container needs access to. 


There’s, what user is that container running as, and that all bubbles up into Kubernetes. So it makes, number one, because it’s hard, right? Like, it’s just [00:24:00] like there’s so many combinations of things that you’re bound to get something wrong, or that workload needs to run as privileged. How do you, , limit the blast radius? 


So the, the examples in the top 10 are just scratching the surface. But. You will see, , , imagine you have a big cluster and some web applications running. You’re bound to have a web app vulnerability. These misconfigurations are what happens next, right? I got a shell on this running container, what can I do? 


Well, if you look at the security context of that pod, it can tell you your path forward. If it’s privileged or if you have mounted the docker socket in into it you’re better off as an attacker. You’re gonna be able to traverse out of the context of that container. And that’s what number one’s all about. 


Ashish Rajan: And OPA is there as well somewhere. 


Jimmy Mesta: Yeah. So there’s two steps. One is finding this configured workloads, defining what that is. CIS does a good, [00:25:00] decent job of this. There’s lots of best practices, , that we can follow. But number one is like finding things that are running. , there’s also finding things in code. 


And then there’s also blocking these things from happening again, right? Mm-hmm. . So all three of those steps are really important. Open policy agent and, and, , kind of a complimentary project. Gatekeeper offer the ability to, number one, audit your cluster, right? Like in an open source way. 


, our platform does this as well in other ways, but the premise here being like, find misconfigurations, fix them and block them from happening again. And we use admission controllers typically to do that. So the Kubernetes API has the ability to stop things like privilege, workloads entering the cluster or particular game space. 


So it’s almost like a back to our firewall analogies. It’s almost like an API. It’s almost like a firewall to some degree. Except it’s not network traffic, it’s configuration , of Kubernetes objects. 


Ashish Rajan: Right. And [00:26:00] to your point then because s a declarative language or, I mean not language, I guess a framework, if your going to use that word use that word where you define what you want, it just creates it, manages it, so I think , the simplest definition was that. 


So to your point then, something like an OPA gatekeeper or admission control, like these things help you almost prevent more of these misconfiguration happen as well. 


Jimmy Mesta: That’s correct. Pod security standards as well. If you want to use the kind, the built in Kubernetes way, they offer three different like risk tolerance base lines. 


And you can choose what you’d like to be more permissive or like fully locked down. Yeah. There’s Kyverno you can write policies JavaScript, like you can get as creative as you’d like. The key is actually doing it though, right? It doesn’t necessarily matter what tool you pick, it doesn’t have to be open policy agent. 


There’s a lot out there that can help you. Yeah. 


Ashish Rajan: I’ve been told one thing, a lot of my people know this, but OPA gatekeeper is primarily on rego and everyone hates rego. It’s not the easiest language that’s like, [00:27:00] Kyverno is definitely getting a lot of popularity. We actually have someone coming talking about kyverno later in the month as well, so that’ll 


Jimmy Mesta: interesting. 


Yeah, no, I mean keep in mind, OPA and Rego are, Kyverno is Kubernetes specific. So if you’re just doing Kubernetes policy, great. If you need to do other policy things yeah. Rego’s not everyone’s favorite. But it, yeah, we use Rego a lot. It’s fine. It works, , it does what it needs. . To me, it’s still better than writing them in JavaScript, but that’s just my opinion. 


Ashish Rajan: I think the example that I gave to someone who did a whole entire talk on the whole Rego thing was when you have to kind of say you’re adjusting, , , you don’t mind, it’s like, it’s almost like drinking poison. You’re like, Oh, well I don’t have any of the choice, which is not going on that part. 


I, I’m just kidding, obviously. But I think I definitely find that. So the context of OPA is definitely beyond, Cause a lot of people use it for multicloud as well, so this, that, Yeah. It 


Jimmy Mesta: goes well beyond Kubernetes, which is, I mean, it goes into service to service [00:28:00] authorization. Like there’s a lot of interesting projects built on OPA and Rego that have nothing really to do with Kubernetes. 


Ashish Rajan: Yeah. So maybe kyverno for people who are just like, primarily to what you were saying, for all those people who have unmanaged like massive clusters 


Jimmy Mesta: what, what, Or just use pod security standards like. Start there. , like, just understand what those are, what they do. And you don’t have to learn any language 


Ashish Rajan: all day. 


And maybe to, That’s a good segue to the next one as well, just this whole supply chain as well for what you use, which has been the topic of, I don’t know, for how many years now, especially after the whole presidential order, supply chain has become like a thing. So supply chain, what’s that context in kubernetes space, supply chain, vulnerabilities 


Jimmy Mesta: yeah. I think it made the list because if it didn’t get yelled at in 2022, if you don’t talk about supply chain in SBOMs say SBOM three times and you Yeah. turn into a pumpkin or something. But the, the, it’s in, in Kubernetes [00:29:00] specifically, I think, and it trickles down into containers. , it’s, you’re, you’re, you’re running a ton of untrusted code, whether it’s Kubernetes, the platform, right? 


Mm-hmm. in all the different pieces. Like think about Argo cd, think about flux, think about helm. Think about like the way we, we bring in all like blindly random stuff into the cluster just because it’s there. We also build our software on top of base images that come from, who knows where those base images might have packages that we don’t use. 


Like it’s, it’s such a mess of third party stuff that it has to make the top 10. Kubernetes just makes it easier to run untrusted stuff. And Oh, right. , like it, , to me we actually have some. We have ways to make that better. Again, like admission control , in, , integrating that aha [00:30:00] control with, with, with things like co-sign image integrity, , writing policies that only allow images to come from certain places. 


We have great defensive mechanisms and they’re largely unused. So it’s a big deal, right? And like, building an image is hard, building an image that doesn’t have , extraneous packages or vulnerable packages. It’s a problem I see every day. So it makes sense that it’s only kind of multiplies in the context of Kubernetes in my mind. 


Ashish Rajan: is this where the whole, things like what libraries your containers using, that you’re using to build your kubernetes cluster. Like those kind of things come into this as well. 


Jimmy Mesta: Yeah, I mean, I think so, right? It’s a good place to, understand what’s running. 


Like, I think as an industry, we’re all still trying to figure out like what to do with this SBOM artifact. at the end of the day, like, yeah, God, that’s, I have the SBOM, I have lots of SBOMs. And , that is going to evolve. I think it’s going to evolve , in a big [00:31:00] way where auditors, , incident response folks, like people are gonna ask for SBOMs before they buy your product. 


Yeah. And that’s already happening, so we’re already hiding. I mean, if, yeah, you’ve heard a few different I mean, we all know the classic like vendor security questionnaire. We’ve all dealt with the vendor security questionnaire and I, , we all know that some of that’s broken, but, I have heard, and I don’t know the specifics, like there are companies now, there’re like, You want us to install your agent, your thing, your software, give us your SBOM, like and, and sign it and give it to us. 


Right? So, Wow. As you should, right? Like that’s a could. Before you install random stuff and your corporate IT infrastructure, you should understand what’s going in there. , and Kubernetes itself , as a project is also generating SBOMs for all the components and doing co-sign stuff. 


So , we’re seeing better supply chain emphasis inside of Kubernetes proper, which is also good. 


Ashish Rajan: Right. And the next few I was gonna combine, which is one was overtly permissive RBAC Configurations and the lack of [00:32:00] centralized policy enforcement. I kind of like combine those two. So , what’s the RBAC policy? 


So enforcement there. 


Jimmy Mesta: Yeah. So. RBAC. The Kubernetes API does authentication, authorization, and mission control. Authorization is RBAC. It’s combining roles, cluster roles, role bindings and cluster role bindings for humans and service accounts to give you access to do things, whatever it may be. 


Now RBAC is highly configurable, but it oftentimes, It’s like misunderstood. You’re as an operator, you’re like trying to combine these verbs with objects and like, and subjects and, and it, it ends up being I’m gonna give up and give you too much. That’s the tldr and we need 


Ashish Rajan: to not it. Yeah. Like too long. 


Don’t wanna read this. What do you need admin? Go on. Like go on with your life. That’s, 


Jimmy Mesta: but that’s, it’s no different in cloud, right? It’s just, , I just don’t think we’re doing much about it in Kubernetes [00:33:00] yet. Policy enforcement is, we already talked about that. , it’s a mission control. 


It’s saying I will stop things that look like this from entering this cluster. That’s actually taking the misconfigurations doing something about them. 


Ashish Rajan: Right. And your inadequate logging and monitoring is pretty straightforward, I guess is not enough. Is there like something from. Kubernetes that enables people to do logging and monitoring, or is that something they have to look for? 


Another, like a Kyverno is a project, like a policy management. But is there similar for logging and monitoring as well, or does Kubernetes by itself has a lot to offer there? 


Jimmy Mesta: Yeah, the Kubernetes API offers what is called Kubernetes API audit Logging. Yeah. It’s enabled not by default necessarily. 


If you’re using managed Kubernetes, it’s kind of a button you click to turn on. Yeah. Now it’s very valuable. And we do very little with it as a whole. insecurity. We kind of just place it into some backend system and hope for the best. And [00:34:00] we can do better. Like there, it, there’s very interesting logs in events happening at any moment. 


And otherwise like the, your applications typically standard error standard out, right? Like the apps themselves, there’s Prometheus metric exporters and things like that. If you want, , actual metrics. Yeah, there’s tons of stuff, like no shortage of logs. It’s just like we should do something with them. 


Ashish Rajan: Fair enough. The, the next one is also fairly obvious, but I think maybe people may not The broken authentication one. 


Jimmy Mesta: Yeah. I mean, access to the Kubernetes API, you can, it’s choose your own adventure. So you need to choose a path that isn’t a hard coded credential. You need to be able to revoke access, which like, , a client certificate doesn’t allow you to do it in Kubernetes. 


You need to not share credentials. You need to not use static tokens or service count tokens from the outside of the Kubernetes API. there’s a lot that can go wrong there. And it’s usually like, [00:35:00] use O I D C type things that you’re already using that support MFA that also have an r back policy that’s appropriate is the recommendation there. 


Oh, sweet. 


Ashish Rajan: But missing network segment approach is the next one. Sure. Is this where service mesh comes. Or no? It 


Jimmy Mesta: can, It can, Yeah. by default, Kubernetes is a flat network, so doesn’t matter what name space you’re in, want pod A can talk to pod B and it’s up to you to make something else happen. 


Right? And it’s typically not appropriate for every single workload to be able to have full network access to the next thing. So, yeah. Out of the box network policies, there’s overlays,, there’s things like flannel, calico, There’s silium there’s no shortage of like, fancy plugins that you can use to then build different style policies on top of service. 


Mesh is one of those right. As at layer seven. 


Ashish Rajan: And so, okay, the secret management failure, I think you [00:36:00] mentioned an example from Google Cloud earlier where we spoke about you can use the KMS pieces from the Google cloud space. So what’s the secret management failure? 


Jimmy Mesta: Yeah, I think it makes the list because secrets are hard always. 


How do you get a secret from point A to point B, and how do you promote it across environments? How is it encrypted at rest? And Kubernetes doesn’t make that easier because , everything comes back to RBAC, right? It’s like secrets in Kubernetes, if you choose to use them are Bay 64 encoded, , key value pairs that sit inside of c d, They’re accessible through the Kubernetes API. 


If your RBAC’s broken and everybody has access to list, git describe secrets, then you, doesn’t matter, like it’s, it’s open., it’s a blurry line. So you wanna make sure that you’ve at least thought about secrets management. Yeah, maybe Kubernetes et cd style isn’t the right place. Maybe you just use kms in your application is responsible for it. 


[00:37:00] That’s something you just need to put some serious thought into. 


Ashish Rajan: The next one is misconfigured cluster components. So this is obviously talking multiple clusters 


Jimmy Mesta: talking together. 


Yeah, it can be and you can kind of bundle nine and 10 together. A cluster has, , things like the kubelet, things like , the schedulers. , there’s lots of different. sub configurations within those components that you, as the operator might not even be aware of, you may not have control over all of them if you’re in a managed environment. 


There’s flags that you can pass the kubelet that make it, , susceptible to using anonymous requests and things of that nature. They’re outlined in the CIS benchmark, but they are certainly something to be aware of. And same with vulnerable components like Kubernetes has vulnerabilities. It, , there’s CVEs that come out with different components. 


Istio has CVEs, Argo, cd, like all this, this ecosystem is not like, , the CNCF landscape is new and busy and crowded and very complex and [00:38:00] vulnerabilities will exist with out of the box Kubernetes components. And you have to maintain that and keep track of it. 


Ashish Rajan: Yeah. And probably the, which kind of links three and I you to the last one as well, which is the outdated and vulnerable kubernetes component as well. 


Jimmy Mesta: Yeah. I mean, to that nature, like the Kubernetes API, if you run an old version, like there’s a live feed of Kubernetes CVEs on kubernetes.io, and you have to be aware that your cluster has a vulnerability, right? And it’s complicated. There’s just a lot of stuff running in there that could, that could have issues. 


Even back in like three years ago, there was like a, I think it was a run sea vulnerability that was, , critical cve. Every Kubernetes cluster is affected and you basically, Had a path to escalating out of a container onto the node. And that’s a thing, right? Like your run seat and container d and these things that power [00:39:00] Kubernetes are targets. 


So you gotta track and remediate. Yeah. 


Ashish Rajan: I guess to what you were saying earlier and kind of zooming out a bit onto the whole top 10 that exists at the moment. Mm-hmm. , there’s network, IAM vulnerable component supply chain. A lot of these as, best practice coming back to the whole topic that we are here for. 


The combination of using something like a benchmark and a, I think like a top 10 or cheat sheet, it’s probably a good way for someone to kind of have that baseline to understand, oh, this is a great way to do kubernetes maybe. Would this only apply for one cluster? How do people scale this to multiple clusters? 


Cause most companies would have multiple clusters, right? 


Jimmy Mesta: Yeah. So yeah, we work with companies that are, , approaching 200, 250 clusters and 50 clusters. Yeah. And wow, , there’s the other end of the spectrum where you have three mega clusters. They each have [00:40:00] 2000 nodes and each present different challenges because the security boundary of multiple clusters is like definitely strong. 


But then you’re dealing with like, you could have a hundred of the clusters could be running outdated Kubernetes, or you’re due for an upgrade, or you don’t have the right policy. On, , some subset of those clusters. If you do a mega cluster, then your RBAC needs to be tight, which is, we all know is hard. 


So applying this across multiple clusters is a challenge, right? And we at KSOC think about it one way. we definitely have the ability to carve things up, distribute policies to some subset of clusters. But if you’re doing it yourself, you’re gonna need number one asset inventory. Like, where are they? 


If you have over a hundred, you’re not gonna just like, no, , if you have three, maybe they’re kind of special pets, but if you have a hundred, they’re, it’s just all over the place. So you need an asset inventory dashboard. You need [00:41:00] policies because some, there could be 20 of them that are in scope for whatever, PCI or something. 


They are different and they have. At mission controllers and things that, , are very hardened and then you have dev clusters that maybe you relax the requirements on. So multi cluster is, it’s tough. And I think we didn’t, we weren’t ready for it. We probably weren’t ready for Kubernetes and one cluster yet alone, a hundred. 


So it is going to be our next big challenge, I think. Cuz now they’re in all clouds, right? Like you rarely, if you’re in the a hundred cluster club Yeah, you’re probably in all three and you probably have some bare metal stuff sprinkled in there. Right. So 


Ashish Rajan: wow. Wait, , so you’re saying this is on managed Kubernetes as well, so from someone Yeah. 


For Florida. Wow. Yeah. Damn. Cause the providers themselves, like Amazon, Azure, Google Cloud, they don’t really give anything for multi cluster management kind of thing for them. You, each one of them is an individual component[00:42:00] 


Jimmy Mesta: pretty much. Right? Yeah. And you, they could be in likely in many accounts, right? 


Yeah, you can have a hundred accounts with a hundred clusters. All different. IAM all different. I mean it’s, Yeah. 


Ashish Rajan: So to your point then, maybe for step one for people who are listening to this is like, do an asset inventory and find out how many Kubernetes clusters are running across your organization to even, that’s a 


Jimmy Mesta: conversation 


that’s a great exerCISe for all security professionals to partake in that easy though, right? Is that 


Ashish Rajan: easy to do other tools for? It, 


Jimmy Mesta: Well, we have a tool for it. But we, there aren’t, not many, like a CS P M is going to try to do some of that. Right. Like a cloud security, posture management tool. 


Yeah. The problem is that usually stops with like, I found an EKS cluster. Here it is, but you don’t really crack open what’s inside of it. You just see it as its individual piece. You could probably have [00:43:00] some scripts that do it constantly. You just daily run like an eks, get clusters command across all of your active accounts but then there’s like always the data science team is like, Hey, well we have four clusters over in GCP and then you’re like, Cool. I didn’t know about those for the last two years. Oh 


Ashish Rajan: yeah, that’s, that definitely so where, does this leave us in? 


I think Cause a lot of people are So maybe the old analogy of how to eat an elephant one bite at a time. Just do the asset inventory first and see how many clusters you find and then you channel match that with cheat sheets and CIS benchmarks and stuff. Yeah, 


Jimmy Mesta: I think our recommendation is always like, understand where your clusters are. 


Get the full picture. Yeah. Then you do, , the asset inventory inside, right? You’re, you’re looking for misconfigurations doesn’t have to be the entire CIS contrary to popular belief, like the CIS benchmark is good, but if you followed it step by step, you would spend the next three years trying to, , have this [00:44:00] checkbox it, you wanna pick the things you care about the most. 


And then ideally you want to squash categories of misconfigurations, not individual workloads. Right? Like, cause there’s tens of thousands of these. You’re not gonna just. Pick them off one at a time. You’re go, maybe, , this quarter, like we have, , 400 privilege workloads across all of our clusters. 


Like that’s where we’re gonna tackle and we’re gonna figure out how to get there. Because we know that privilege is bad and we can give an allow list of system calls or something of that nature, right? Like, you don’t want to just try to do everything on day one. And yeah, I don’t think training hurts, right? 


Like letting developers understand what these things are before you get in a situation where you have to do a firefighting exercise . Probably a good thing 


Ashish Rajan: too. Talking about firefighting. So, one question from Syed here. I have a case where deploying iot as a cluster for key management, Does it make sense? 


Use kms if using eks. 


Jimmy Mesta: [00:45:00] Sure. Yeah. You, I mean, you’ll have the one side of opinion is like, , people screaming about vendor lock in. , everyone’s like, I need my freedom and I don’t want AWS to control me and I don’t want to use what they have to offer. KMS is that example, but it, , also it’s easy and secure and backed by an hsm and it’s like, why not? 


Right? So So you’re deploying Internet of Things as a cluster. That’s good. Yeah. We’re talking to a set of meat packing facilities that all run Kubernetes inside of each of the facilities. All their iot devices report back to Kubernetes. It’s kind of cool. Wow. That’s pretty cool. Yeah, KS is great. 


You should use it unless you need to be truly portable, then you could consider something like Hash Corp Vault running inside of the cluster. Right. Or you could use Kubernetes Secrets, like that’s, it’s a threat modeling exerCISe. Right. The end of the 


Ashish Rajan: day. And it’s [00:46:00] got a follow up as well after that. 


Also, is it recommended to use a pki for cluster if I already have a PKI server management cert for sender? Lots of confusion here. Publicly infrastructure. Yeah. 


Jimmy Mesta: I, I think Kubernetes is gonna manage certs for you internally if you. I think there’s too much nuance to that question to, to answer with any 


Ashish Rajan: the general opinion on having certain management done on Kubernetes. 


Maybe we can have that 


Jimmy Mesta: as an answer. So, I mean, picking on like TLS certificates as an easy example typically , the pattern is to use something like the cert manager project, which is going to sit inside of the cluster, probably integrating with Lets Encrypt or something like that. And it’s going to revoke and refresh and deal with cert rotation, things like that for TLS termination at like an ingress perspective from like, call it engine X. 


That’s a fairly established pattern. I don’t think it is an external reliance though, right? You are not using baked in Kubernetes things, [00:47:00] certain managers. A third party component, sort of, kind of. So , you’re taking certs and trusting this thing, putting them into local storage typically and terminating TLS. 


there’s a ca, there’s a certificate authority in the Kubernetes API that’s kind of a different story where you’re minting client certificates for access to the cluster. Mm-hmm. , I don’t think you wanna yank that out and do it elsewhere. I mean it, because there’s too many internal complexities there. 


The kubelet needs the ca and things like that. I would pick, pick something. Yeah. Straying away from the pack. When it comes to cert management, PKI stuff is usually a bad idea. So , just use the defaults. Use your cloud provider. Offerings and use what Kubernetes has established. 


Ashish Rajan: Awesome. Great answer as well. Hopefully that answer your question side. That’s I think the final question that I had for the interview was just where can people learn about securing kubernetes Man, I think, is there any thoughts on where people can learn about [00:48:00] kubernetes and security within Kubernetes? 


Jimmy Mesta: Yeah, you can come to Kubecon and come to one of our workshops. You can go to kubernetes.io. There’s a lot of security content there and like, it’s a great reference. There’s the, the top 10 has lots of links that OWASP cheat sheet, you could follow me on Twitter at Jim Mesta. I post Kubernetes security stuff pretty much solely and like maybe some mountain biking things, but , there’s more and more out there. 


And yeah, I think. There’s a certification, The cks right? The certified Kubernetes or Right. Cks certified Kubernetes security. I guess that’s it. Yeah. Yeah. You could take that. Yeah, and, there’s more than ever. I, yeah, six years ago I was writing a lot of the stuff from scratch and it’s pretty amazing where we’ve come. 


There’s Hacking Kubernetes by the folks at Control Plane Andrew Martin. Oh yeah. And that’s a , great [00:49:00] reference. Yeah, there’s, Oh, I Shortage of Information Cupboard as well. There’s Kube goat. Yeah. Yeah. That’s Madhu’s project. Yeah, 


Ashish Rajan: that’s 


Jimmy Mesta: for many now. Yeah. We have a awesome Kubernetes security GitHub page. 


It’s Okay. I’ll send it to, It has a bunch of links. 


Ashish Rajan: Oh yeah. Sweet. Yeah. I’ll probably add another show notes as well. So just mindful. Yeah, I’ll probably share the link with the show notes. But where can people find you? Man, I think I’ve been taking way too much of your time as well, but I really appreciate this time spending with us. 


So where can people find you on the internet? They have more kubernetes best practice security 


Jimmy Mesta: questions. Yeah, just jim mesta on Twitter. You can hit me up on LinkedIn too. I saw you and Shilpi had some post today that has like 900 million reactions. I got like 45 LinkedIn requests today from that post . Yeah LinkedIn’s pretty good. 


If you could cut through the, cut through the noise and my dms there or Twitter is fine. Yeah. Or [00:50:00] jimmy@koc.com. Yeah, hit me up anytime. 


Ashish Rajan: I’ll lead those links in the show as well. But dude, thanks so much for coming on and sharing all this as well. And thank you for giving light to the Top 10 project as well for , at least now we to go to 


Jimmy Mesta: as well. 


Please contribute. It’s getting translated into different languages. We have like 10 active contributors, so I’m excited. Come on by and we’ll make it better over time. Awesome. 


Ashish Rajan: All right. Thanks everyone. Thank you for all the commenting questions as well. 


And I’ll see everyone else who was online as well for next episode on next weekend for Kubernetes again. But yeah, thanks so much for Jimmy and thanks. Thanks man. Talk to you soon. Bye. Thanks everyone.

‍