Ashish Rajan: well time to bring my guests on and it was our further I do.

Welcome to the show. I’m so glad you came in. . I was gonna say staying up late as well because you’re based out of Netherlands.

What time is it for you? I

Madhu Akula: almost midnight. It’s about morning 0.3 minutes.

Ashish Rajan: Fair enough. So I’m going to start with the obvious man for people who may not know who Madhu is. Can you just a bit about yourself and how’d you go from where you were to where you are in terms of professions today?

Madhu Akula: Yeah, sure. Thank you so much. First of all, for having me, myself Madhu Akula. So. I’ve been like doing mostly security from the beginning of the career, nearly from college, like where I have been doing started tinkering with mostly security and infrastructure and the operating system. Then I used to start doing mostly like a traditional security people, like finding bugs in most of the applications, whatever your name out there, like Google, Facebook, like these kinds of bug bounty stuff, then I thought, okay, maybe it looks interesting, then I can have a more diverse do it, like how it looks like.

And, then coming out of the college, like I joined a very [00:01:00] early stage company, like where I was doing consulting for a big giant companies mostly doing infrastructure security, learning more about like other things security. So then I started with parallely contributing, open source and also finding vulnerabilities in the software, which is used by the large enterprises, like ended up in G and WordPress and much of this ecosystem.

So it’s kind of from what made me more interesting and I thought, okay, now I have been there for learning security now. Maybe I want you to go bigger scope. Right. Then I joined Fortune one that is no bigger than that. So then I kind of building mostly network security and learning and tinkering with scale understanding systems, how it works in a bigger enterprise systems.

It’s kind of been there for some time then I thought, okay, maybe I’m missing the fun out of the game. And so I thought I’ll join against startup. So that’s where I actually joined another company like that’s one of my, most of the things has been changed. It’s an obstacle. Like I was been there as a second employee of the company, very early stages.

Did I kind of started tinkering more, actually this cloud [00:02:00] Kubernetes containers, even ecosystem, which is very early days of the Kubernetes and all this, then I kind of started, okay, let me learn all these things and share with the communities as well. So I started presenting my research and also written a book a co-authored with the Akash on security automation, with Ansible and some the other work.

Then it’s kind of more in more depth into Kubernetes because it’s quite interesting when I started then this is where I am, I think mostly doing things around cloud containers and Kubernetes security these days.

Ashish Rajan: I love it. How you’ve quietly shied about how much you have done in terms of public speaking as well?

I think, because you didn’t touch about blackhat as well. I thought you would have thrown that as well, that’s one that was well quickly.

Madhu Akula: Sure. So I have been actually speaking and trying to share my knowledge across the communities like a , Def con, or really use Nick’s GitHub chat, like some other conferences around the world.

And apart from that, I live, try to work with communities like All day, DevOps. I think this is the one which I think I’ve been working with them for past, beginning of the conference to moderate [00:03:00] their conferences as a speaking, and also with Null, which is saying yes, larger security community. Like these kinds of communities are kind of involved for quite some time.

Ashish Rajan: Yeah. I could not think of a better person to bring on the show at least to start the conversation about Kubernetes. And I’d love to get into Kubernetes Goat as well, a bit later. So, but for people who may be listening to this, and I think one of the ideas that we have with this month is we got Kelsey Hightower last month, at least starting off with why Kubernetes and why is it really getting popular?

Who can start to learn and where I want to get your take on Kubernetes as well. Like. What is Kubernetes for people who haven’t heard of this yet, who haven’t enlightened their worlds to this concept. And how is it related to cloud native?

Madhu Akula: Yeah, I think you kind of put so much pressure and we just know, like you said, Kelsey is there, which means he’s like, kind of working with Kubernetes for quite some time.

And he’s the best go-to person, anyone like you can ask, like to simplify, but I’ll try to cover what is my learning. That’s thinking of my approach, but course so for me to be honest, like [00:04:00] when I started thinking about Kubernetes, it’s like a kind of blue between the things. So Kubernetes is kind of a piece where it’ll help you to do all the things which you are doing for years, like a load balancing, scheduling things and the orchestration of the applications into the cloud, or the cluster.

So. It’s a mix of a few, but basically it’ll help you to automate these parts rather than being manually, in a declarative way. Like you mentioned that this is what I, the state I wanted to have. So companies will make sure achieve their desired state by, following whatever the definition you define, by using whatever the components it has.

Ashish Rajan: And how is that related to cloud native?

Madhu Akula: Ah, yeah, that’s quite a interesting, maybe I’ll just borrow the definition from CNCF only the CNCF is a cloud native computing foundation. So. CNCF says that a cloud native means where you can define the declarative API and which means the application. You write the code, which can be containerized and you can dynamically scale up and scale down.

And without much efforts from the teams, our [00:05:00] operations, where you can kind of manage these applications, effortlessly while making sure you can define them in a APA driven or some kind of a microservices approach.

Ashish Rajan: Interesting. And I was going to ask about learning as Kubernetes as a concept.

So I’m going to come to Magna’s question, cause this is all the question I’m going to get to that as well. I wanted to kind of lay some foundational pieces as well, because this is the first of the series and we didn’t go into a bit more layer deeper with Kelsey. So I’m hoping you can get, some more depth into the Kubernetes topics.

So we’ve spoken about. Where does Kubernetes are there like multiple types of Kubernetes or like malaise? Like what, what are the kind of Kubernetes am I looking at in the market.

Madhu Akula: So Kubernetes is just one thing. Maybe people used to use like things like Apache Mace’s marathon, which is like another lab used to do the same thing.

And I think now recently, nomad, which is another utility, like Kubernetes orchestrator, which is by Hashicorp, and so Docker itself made a Docker swarm, which is another orchestrator. Like these kinds of things are there, but when it comes to the Kubernetes, purely, especially [00:06:00] focused on Kubernetes, they do have like a managed to actions.

Like most of the cloud vendors has already there. Like Google had GKE. AWS has EKS with like Azure like similarly. So these are like managed by the vendors like a cloud providers and people can itself minis themselves also. So now it’s up to the situation. How do you want to offload, which kind of work to you are either cloud vendors like shared responsibility at the end.

Ashish Rajan: I think it’s probably important to kind of call that out because to your point, when people are listening to this and going, Oh, Kubernetes seems to be everywhere. And I seem to see it everywhere. Why do I care and what what’s an example of it.

So managed unmanaged is a good one, but I always wonder are people using this in production. Yes,

Madhu Akula: I have indeed. I would say I’ve been working with companies and I have been testing clusters for security purpose, for almost a years, I would say, but I definitely, I have actually recently worked with even bank where they have been using these in production, which means like banks serves the obligations and data or [00:07:00] Kubernetes clusters.

So, and I’ll say can maybe borrow a thing from CNCF, they did a survey recently. I think that adoption of Kubernetes itself almost raised 81% recently in the past year at section

Ashish Rajan: 81%. Yeah. People listening in definitely need to double down on learning Kubernetes and I’m glad we’re doing Kubernetes month, but , I’m definitely not plug in for people who may be interested in Kubecon is happening this month as well.

I had no idea before, so hashtag Kubecon. Thanks for supporting us as well. But. So it is in production. So it kinda makes me think that, okay, so this is getting to a point where I guess cloud was ages ago, like six, seven years ago when people were talking about cloud, Hey is it secure? It’s not secure.

And I feel like we’re at that stage with Kubernetes where that’s, where vulnerable by design kind of concept. I came in every time I would search about Kubernetes security, it would just come out. Is it vulnerable by design and funny enough, most of it was your talk as well. But before we go into that, is it really vulnerable by design,

Madhu Akula: I would say no.

So some of the things I think I would say it’s kind of functionality at the end, straight up [00:08:00] straight, like like one thing let’s maybe just start with the technical part, let’s take a networking part. Any part at any service to talk to any of the part of service within the question, which is like a functionality, something like where you can literally say it’s intended feature.

Right. But if you think the same thing in attacker’s perspective, like hacker specific, you, they can leverage this kind of features, by default because it’s. allows Everything. They can talk from one part to another part, which means there is no network segmentation or a firewall or something is that which means you can think like vulberable design, but if you think in other ways, like functionality, I’d like its intended feature effect.

So like these kind of things are, but yeah, it’s not truly vulnerable. Oh,

Ashish Rajan: so it’s more about, I guess, how you would abuse the function, which is supposed to be a feature, how you would abuse it as a hacker. I guess that’s kind of where it makes it, I guess. vulnerable

Madhu Akula: Yeah, indeed. And especially with the configurations, the, how you manage the configuration, like if you take a cloud as well, traditionally, right?

Like by default cloud, we’ll make sure when [00:09:00] you create an EC2 instance at something like AWS, they will say by default a denial, if you want to explicitly open these rules, like, Oh, go ahead and add this a firewall rule or security group or something like that. Right. So it’s all about configuration because at the end that they provided the future, they also given something called NSPs, which is network security policies.

Now it’s as a user at the operation, like a analyst or something security person, it’s your responsibility to say that, okay, this service, this application has to talk to this, another application. So that’s where it’s shared responsibility again. Right? Like you have to take that makes you think. Yeah.

Interesting.

Ashish Rajan: I think to your point it makes me almost go, okay. I get that. And it’s kind of like flipped on the head where AWS may say deny all to begin with but in a pod context is allow all to begin with you explicitly deny what you don’t want, but I guess that is quite small to start off with.

So node would not have that many pods shouldn’t have that many pods, but God knows how much stuff gets into a pod. So maybe let’s start with three security [00:10:00] design flaws that you’d see, or maybe. Misconfiguration that you see in Is Kubernetes maybe in production in some places.

Madhu Akula: I think again, maybe I can say did again, it may not be vulnerable by design it’s basically misconfigurations as you said, because yeah, definitely. I would say that I have seen quite by testing, seeing, working with the production clusters in general, one thing is definitely an NSPs. I hardly seen people will apply in network security policies, which means.

If you’ve got access to as an attacker, like one of the application or some kind of , which means by default, most of the places I have, it’s very difficult for a team stoplight network security policies, because even it’s for them to hard to understand which service which application is talking to, which one internally also, so to create a construct that network security policies it’s kind of pretty difficult.

And also like it has to be achievable over the time, like by monitoring what kind of connections it makes or something like that. So it’s a kind of one thing maybe I have seen mostly people who [00:11:00] did not apply network security policies, most of the cases. And of course there are quite a lot of security.

Misconfigurations I have seen testing, especially, most cases giving wide access, which is like a non-listed privileged approach right even till today. I see most times like whenever you get access to a pod and you are trying to look for a service account, like, which is like the access where you can get, it, might’ve given like star to the entire name space and which means other secrets or something, which the service second can access.

You can get access. They don’t restrict that privileged permission to specific thing. So it’s kind of wide open things. So this is quite a lot we see, testing the clusters and in general, and of course maybe definitely it’s immutable and fragile and then discover and decide a cloud computing.

It will come up a pod second and another second it may be destroyed. So the way to monitor detect these kinds of attacks as changes, if something happens within the pod or something is not a vulnerable way. Designer misconfiguration but it is definitely important in terms of security to monitor and understand, [00:12:00] what’s happening with within the cluster, what is happening in the pod or something like that.

Interesting. I say something

Ashish Rajan: useful. Yeah. Yeah. That’s great. Great examples as well. I think Magna is also added onto this as well. Just pretty good. Not vulnerable by design, but also not secure by default as well. So I think what was really well, I think it’s like semantics of design and default , but I think I totally agree with Magno and you, I do want to give a quick shout out to Magno , like, cause he has a CTF running on Kube con.

So I’ll the people I’ve whoever this thing, definitely check it out. Yeah, just definitely checkout Magno, CTF as well . We spoke about the design flows. How important is a Docker file in this context? It’s docker even related in this context of Kubernetes?

Madhu Akula: Yeah. So I think maybe Docker definitely be made the containers popular in general. That is one of the really good thing, which I think I feel also happy about because otherwise containers are like this thing there for years as per like see groups or namespaces assigned ch route and all this, but Docker made it so easy for developers so that you can go right at whatever your [00:13:00] code and package everything in there.

Your libraries, that application code and everything in, like zip file or something like that. Containerize, then you can run this anywhere in the world. Like developers used to say, it was working on my system. I don’t know what’s happening in this. However, this kind of, the problem that I solve, which is really good because at the end, once you package the Kubernetes disease, Orchestrated.

So the Docker will become run time. Now there are so many run times available, like run C, bunch of other. So now you package that container and run that across the scale, like distribute, make it like replica. So these are things handled by Kubernetes, but underlying it’s at the independent area, which is running.

Ashish Rajan: And so Docker file security then, what some of the examples.

Madhu Akula: I actually recently written a talk on a Dockerfile security, which is a, I think maybe I’ll link it later.

So basically definitely Docker files will be pretty useful, not only Docker because most of these monitoring fracture. Becoming codified, right? Which means you convert this into, infrastructure as a code somewhere, we choose single [00:14:00] source of proof. So Docker files will become definitely important because at the end that will be converted and built into a container and pushed into a registry somewhere it’ll get pulled into again, cluster or Kubernetes

so if there is a simple change, as attacker can control that container, which means literally DACA control image or some container is running in your cluster. So I think this would be in a larger scale, very large kind of attack surface when you look at a supply chain surface, so there can be quite a lot of possible attacks you can do with these things.

Yeah, I would say some of the tools I can think of is there are quite a lot especially lending is a really good to get started. There is I think handling is that, which is Linda, which you can just link across the files and also. Build the kit, Docker itself has a functionality when you are building Docker containers, how you can securely pass the socket and how you can secure your pass the secrets and all these things.

And there’s a really interesting tool I use actually , which helps us to use the CIS benchmarks on Docker files to run and [00:15:00] understand. Is there any misconfigurations and non best practices it’s following. And people even write the OPA, which is the open policy agent, which is custom, policies on top of this, let’s say, in your company that doesn’t want it to allow certain user or certain secrets out of certain environment variables.

So you can literally say that you can codify that and say that, Oh, go ahead. And each time, if your Docker file has pulling a image from certain registry apart from internal predators, go ahead and stop this bill, notified this bill ticket. So these kinds of, custom policies also you can drive, which kind of gives you more control over the container, the BT.

Ashish Rajan: Interesting. So keeping that in mind, if Dockerfile file is clearly quite important. And obviously that sounds like the foundational piece of what the entire thing will be built on. I mean, obviously Kubernetes has its own infrastructure components, API services, and everything. Is it like a sophisticated attack that you’ve seen in the Kubernetes space or are they all misconfigurations.

Madhu Akula: Yeah, I would say I have seen quite some of attacks, but I don’t [00:16:00] know if it is sophisticated, but yeah, definitely. I feel it still, and I believe, I think in future it’d one attack, which I can think is sophisticated with the supply chain attacks, because it’s, it will happen in a large scale if it happens.

Like I think once recently, not recently Docker hub got hacked, which is one 90 care for Docker hub official account also got hacked, right? What if one of the Ingenix image like which is official image, account also got attacked and their attacker can control that and put a back door, which means everyone in the world will trust Engenix sufficient because it’s official from engineers.

And I think most of the clusters might be running one of them right in the world. So if the mistake and liquid, because it’s a trusted and Able to run. It’s a, literally running a backdoor, which is in the entire clusters because it’s official docker. Right. So I think there is a quite large possibilities with supply chain attacks in this world.

I think I feel still that would be a big attack

Ashish Rajan: I was gonna flip this. What’s the simple attack that you can think of that people can actually go out there and check as well.

Madhu Akula: Hmm. I think looking [00:17:00] for the API server, which is exposed and if it is anonymous use that node I think mostly node in the modern days and maybe node ports, which is very interesting.

I seen, because they, especially with managed clusters, if they don’t have firewall enabled and applications exposed with nod ports, they literally can access. I think that is another thing. I know one of the coolest thing till previously, I used to find this quite a lot in pen testing engagements I’ll help if you are using , which is a Washington by default, it has insecure flat, which means anyone can access.

I mean, somebody followed has clustered in privilege. If you’ve got access to one of the cluster pod if they don’t enable like network security policies by default, it is used default configuration of help, basically that complete cluster admin access. You can own the complete cluster.

I used to find this bug in like almost so many testing

Ashish Rajan: enhancements. It’s out of the container at that point,

Madhu Akula: then you not only break out of the container, you can literally control the entire cluster, like any nodes, any color.

Ashish Rajan: Wow. Okay. That, that would be good. Interesting. Seems like Magno is releasing a blog next week on this as well.

So that’d be pretty cool as well. So okay. [00:18:00] I’ve got to understand the sophisticated attacks I’ve understood.

The simple attacks that people go for as well as security person Im listening to this, okay. Clearly this is, this is all bad news, right? This is like, it doesn’t sound like this is great. What’s the defense strategy and like what’s the blue team doing in this? Cause sounds like I just go on the internet, look for publicly open API servers and boom, I’ve got access.

I’m thinking of the Tesla example that happened where someone did crypto mining out, straight after. So what’s the defense. And is that even pops possible at the same level, as you were saying, like, dockers has come up and down? I have no idea if my security is going up and down with that as well.

I’m going to quote one of your talks and the automated defense that you use for one of your black hats, I though it was a great word. So what’s an automated defense look like in a Kubernetes world?

Madhu Akula: I mean some blue team security I think automated defense, I don’t know, like is it completely possible to defend for sure.

Reduce the attack surface as much as possible. But one thing I have seen, especially talking to so many security [00:19:00] police, they quite have a gap between the technical knowledge, because they’re really good at like security. And it’s not much difference from traditional world apart from maybe some learning curve, but if they don’t understand this technology itself, like how Kubernetes work works and what things then I don’t think so that they can add much value as from their experience or expertise.

I think it would be really good if people learn and understand the technology before even thinking about security, especially from the defender routing perspective. And apart from that, I mean, if you think about Kubernetes is not just a one could say it’s a component, but there is so many things which comes through the layers

I think have seen in the outside and I also still believe, it has to follow difference in depth layers, you have to think about like the supply chain the, before the build phase development are even into some of the infrastructure security. A run time security.

So these different layers, you have to think, which means you have to start from, like, before you ended up applying the application, like our traditional application security. Right. Which means if you advanced, it doesn’t matter if it is Kubernetes serverless or any of that [00:20:00] fancy tech, right. If you have application bug or a security issue, and then it get compromised and you can get access company bypass, all these layers.

So looking from that, like looking at application security issues, then moving towards the build time, how you make sure security package and build that containers. And from there, like looking at the registry. So how do you store these image majors? What kind of privileges you view then from there, like the infrared infrastructure.

The supply chain part from there to how you can pass these images to the clusters. Right. And also there is quite an infrastructure, because it kind of goes bigger and bigger when you started here, because if you are using cloud again, you have to think about cloud security also because some of the configurations you have to do in the cloud.

And again, you have to do for the Kubernetes but if you are being managed on your own, then maybe how focus about underlying infrastructure also, like maybe , like infrared sharing related vulnerabilities. So those can be handed over to cloud vendor. If you are using, like, let’s say, manage the Kubernetes and let’s say we have done all the good job and we [00:21:00] have to think about logging also because okay.

We built everything. We think that it is secured but, what if. Something new came out something which is already existing. We didn’t know. So having continuous visibility like monitoring is really important because we don’t know something gone wrong. It is. We need to know what kind of things happening by looking at the logs of the admission controller.

Like even the node level logs, pod level logs. So some kind of logging is really important. And also I think there is a really good toolsets also coming in really nicely and open source world. Like Falco is very useful for runtime security, how you can detect logs. And now, like in similarly in each list, there are quite a tool chain is coming up in open source.

I think that’s kind of really adds a lot of value to the community, how they can leverage these tools to do a defense or security for sure.

Ashish Rajan: Are there any things within say a managed Kubernetes by a cloud service provider? Are there things in there like, which are just kind of like what you mentioned vulberable by design, are there other gotchas like that as well that people should think about?

I mean, I [00:22:00] guess there’s a choice people would have to make between unmanaged and managed and sounds like the cloud option is the easiest one. Hey, I’m already in AWS or I’m already in Azure or Google cloud might as well use it. Is there, like, what do you see as, I guess the security default people should think about in their which may not be great out of the box, but they need to configure.

Madhu Akula: Yeah, I think a one attack used to be very popular quite recently which is nothing too, especially Kubernetes in general cloud based, which is server side request forgery, SSRF. It was super duper popular, even Kubernetes in early days and a really great example is Shopify. The, one of the researcher found in Shopify applications an SSRF and they can able to `compromise and bond the entire Kubernetes cluster because.

Using them, they can leverage and gain them, enter data and get the access. So these kinds of misconfiguration, it doesn’t matter. It is Kubernetes. It is something it’s compute level. Then you attach the instance profiles to the computer and they can able to query that, and able to get the meta-data credentials.

[00:23:00] So this was very good attack surface for a cloud source, even if you’re using managed. But I think nowadays it’s prevented, I think Google has introduced a workload entity and metadata concealment proxies. And I think even AWS introduced the MDSV2, which is version 2 of the meta data. So these were prevented, but I’m into this kind of thing.

So we do not like coming up. How do you configure it? Right. So these are really quite things you might want to look out because it kind of gives you much more like XL for assessment.

Ashish Rajan: Oh. So that’s to your point, there’s another additional layer. If you go down the path of cloud. For a managed Kubernetes cluster that people are to think about.

And to your point, it goes back to the shared responsibility that you were talking about. You almost have like three or four layers of shared responsibility. Now one is the cloud service provider. Then you have your own I guess, data that you have to manage and then own configuration that you’re asking Kubernetes to do.`

And then, then is managing something else inside of it. And there’s a pod, which there is another layer because you have supply chain sounds like the layers just keep adding up. It’s very [00:24:00] easy to kind of lose sight of that, even though there are so many layers, the scalability of it is what has made Kubernetes quite popular

Is Security.something which is possible to scale in this as well. Like, and have you seen examples of people scaling security for Kubernetes as well?

Madhu Akula: Yeah., I have seen some companies maybe I would say, where they really apply things. Cause yeah. And then it’s a security team.

So most of the places that may not be big teams, right. It’s hardly some number of people. So I think leveraging definitely our teams is really powerful because, we have seen people mostly we’ve worked with the DevOps team on SREs, whatever the name right. So we kind of embedded and help them to achieve these goals, like early stages.

So mostly applying, like from the beginning, like let’s say someone wanted to bring a new thing into the system or something like how we can threat model and understand so many attack surfaces you can leverage and, stop at at early stage. So in the design phase only we will understand like, okay, these are so many new components coming.

Maybe how we can. Think about implementing preventive controls as [00:25:00] much as possible then once is this dead, like, it kind of reduces a lot of attacks at first compared to which is going to directly then on top of that, how we can apply detective controls. Like on top of preventive. So somethings definitely one thing you have to keep all this, you have to automate as much as possible because you can’t keep looking at these things in a manual way especially, which is immutable as you come up and go.

So, one thing I have seen people does is try to understand the Critical assets are things. Some people may be really interested in their data, which is data security is primarily interesting. Some people may be getting access to the systems is different for the attacker’s perspective, right? So maybe that is what their critical assets.

So understanding that critical assets and trying to apply as much as defensive controls, they not to make it fashion like that would kind of reduce so much attack surface, which means it’s not completely eliminated. And on top of that, you can slowly bring it on some kind of automation and monitoring, which kind of adds a lot of with the smallest teams you can achieve more and more security [00:26:00] layers.

I think that has been tremendously added value. And also really good thing is if you can, enhance the knowledge or add much embedded security into your DevOps teams or whoever using that, building these platforms. It kind of saves a lot of time for security people because they itself make it so you’re hardly reviewing their code RPRs or some kind of changes, right?

So it really saves and built much more secure systems.

Ashish Rajan: So to your point, security team obviously is not as big as a development team in most of the companies. And for obvious reasons, is there like a example that you can share where people can think of and include that in there? I dunno, CICD pipeline or automation that they can use for that you can share that maybe people can listen to this and go, okay.

Maybe I, at least I can start with that today.

Madhu Akula: Yes, I can definitely share the problem with me is I throw so many tools, which is not good. So like, yeah. So one thing I would say, definitely start with understanding like let’s application is building, like start with the threat modeling. What is the thing which is going into the [00:27:00] cluster or something, then you kind of understand the attack surface, then when you are building applications.

Okay. You secure whatever the possible things like a Docker file or Docker containers, because that’s what you are packaging. And then you add as many things, least privileged approaches. So defense in depth that whatever the things into the layers. And then once you build the container, think about how much leash privileges each layer, it kind of applies some repetitive things, like, because.

I have seen, especially while I’m testing, maybe this is a really good scenario to give, from one of my engagement to when we got hacked into one of the clusters and the cluster has a pre-list even Pusha container to back to the Ridge, which means you can literally able to push a backdoor into the container labs is review and develop it.

It will pull the same container, which is going to run in and developer laptop because you’ve got application access in Kubernetes, you are literally backdooring into the developer workstation, which is in the . So, yeah, it’s a kind of a quite, interesting, so that’s what I’m saying. Sometimes it kind of backtrack that accelerator.

Ashish Rajan: So someone was able to use a Kubernetes cluster to go back to the [00:28:00] developers machine who put the thing in? Yeah,

Madhu Akula: the thing is the kubernetes cluster has the privilege or the permission, not only to pull the images from the private registry. Permission to push you up, which should not be there in that case.

So using that whoever got access, they can literally push it back door image with the same tag, that same thing. And because it’s internal image, I’ll pull this image and run locally.

Ashish Rajan: Yeah, that’s right. Because they would update to the latest version because that’s what everyone, I mean, I guess security team would have said as well, upgrade to the latest version always.

So yeah, I mean, I guess developers and security, we would say that. So you would unintentionally install a backdoor on your laptop and hence giving access to the Internal network. Yeah, the container next level.

Madhu Akula: Yeah. It’s the other way around, which we have seen in one hour, but it’s fun. In attackers perspective.

It is always fun to see those, but these things you have to control. Otherwise the small thing here is giving the proper permission set that registry back, which kind of leaked into so many things. So this month, misconfigurations can leverage into quite big attacks. So [00:29:00] I think one thing I would say just to CNCF did a really good work, especially the six security team.

They did a white paper actually, CNCF secret security white paper. I think that is really good. One I’m also following the CIS benchmarks also quite good to get started if someone and yeah, I think There are quite a lot of resources. I don’t know any specific ones. Exactly.

Yeah.

Ashish Rajan: Do you feel like CIS benchmark? And I don’t want to take a dig of CIS benchmark I’ve taken a dig on that for the cloud security pieces as well. Do you feel that’s, accurate? Like if I implement everything in CIS benchmark on my Kubernetes cluster, if would work because that doesn’t work in cloud, but I wonder if it works in communities.

Madhu Akula: Yeah. I wouldn’t say some of the things it’s quite difficult to implement in this. Like, as I said, network security policy it’s may not be hard every time you can implement them in some cases. Yeah. But it’s a really good base standard, like which kind of quite improves a lot of things like the host level hardening and also basic sanity checks like pods.

Our containers [00:30:00] checks. I think it’s a really good getting started point to, start the security hygiene. And on top of that, you can add more and more like layers of things that you wanted to add, by the way, I think those are good, but the tooling may not be good. Like if you wanted to implement those things it is quite hard.

And trying to understand based on your pentest also that’s a bit tricky because when we test with the tools, like whether is it following CIS benchmarks, like let’s take managed clusters or cloud clusters because you don’t manage the master node.Maybe you don’t have everything access to even understand what is doing there, right.

When you are testing for CIS benchmarks. But as we trust the cloud provider, because it’s a shared responsibility when you are taking a GKE, something. You basically handover the master node security to the cloud vendor, like cloud providers. So it’s kind of offloading the workload. So that way you kind of trust the vendor or cloud provider but I think CIS is okay.

I even say it’s a good starting point, but definitely you have add more if you want it to be more secure. I love the

Ashish Rajan: politically correct answer, [00:31:00] man. I love that. On that note, I want to switch gears and come to the question. Say Magno, thanks for your patience, man. I’ve got two questions here, which are asking about where you start learning about Kubernetes is one first one, obviously for Magno, who’s been waiting patiently.

How did you start learning Kubernetes? Which material did you use first? And then I’ve got the same question from Gerald as well. What resources would you be best to start learning Kubernetes? So let’s hear from the pro. What was your first resource and how can they use something similar?

Madhu Akula: Yeah, I think I heard about Kubernetes in 2016 from a DevOps friend that, like at that time it’s quite different. And things changed crazily and especially Kubernetes, this is the very interesting thing. Even if you learn everything, I still doesn’t know everything I’ve been said, anyone, but there is every three months new release.

It’s hard to catch up with the features and things

Ashish Rajan: every three months.

Madhu Akula: Yeah. I think it used to be, I don’t know if it had changed, but they used to,

Ashish Rajan: they’ve only just released one right now.

So how are you keeping up?

Madhu Akula: Yeah, it’s a bit hard for sure. I also don’t have complete knowledge, but the one thing I think they’re really good with this the documentation [00:32:00] is amazing. I think I would say if you wanted to learn and get started, then documentation is always good to refer and it’s very well documented and very user-friendly.

And another one is I would highly recommend is Katacoda, which is an online interactive playgrounds, which has a free labs. Even Kubernetes documentation itself suggested two tutorials to use that playground. So they have even scenarios where you can help you to try out and learn about in a practical way, how you can learn Kubernetes.

And when I was learning, I was actually follow the video series by Janakiran MSV, which has a YouTube series of webinars in that time, especially focusing on Kubernetes. An architecture and each services wise. But these are the things I followed, but now I feel still documentation is really great resources and the Katacoda playgrounds,

Ashish Rajan: right.

So someone trying to get into the space of Kubernetes security, they could probably start off. I mean, I’m definitely gonna include all of these links in the show notes as well. So people can check that on the website. And playground is an interesting one because actually, sorry, before we go into that magnet just [00:33:00] confirmed.

It is every quarter, but now it will be three times a year. So around four months. Damn so still, that’s a pretty quick, man. It’s like, it’s like, it’s not, it’s not like you’re updating I don’t know, adding a feature for a star. It may completely change how people use it. So thanks for sharing that as well.

Magno, the question that I had then is like, okay, I’m starting to play the playground. I’m going to do a quick plug or something. You created the Kube Goat why Kube Goat and what’s its relevance in people learning Kubernetes

Madhu Akula: yeah. So it’s quite interesting because I’ve been working with Kubernetes.For some time because I come from security background.

I always used to think anything. If I try a new tool in a two perspectives, how can I test think about security of that, like how I can secure that or how we can test for security vulnerabilities in that way. And then another one is I will use in other way, how can I use for security automation or security things, the new tool or technology I’m learning.

So these two questions I generally ask if I am trying out or something. So I’ve been playing very intentionally because I come from security background, but my [00:34:00] primary interested in, infrastructure and like Linux and playing with sys admin stuff. So this is where it made really good connection with me in this containers and Kubernetes.

So how been coming from security? Well, security has this popular notion called building vulnerable labs where you can, people can learn by understanding how vulnerability is existing so that you can learn about security, better, like web goat was there to teach web application security. Similarly there are quite a popular applications like cloud code for cloud security

yeah. That’s why. Yeah. And I started, okay, let me be build for Kubernetes because I have been learning, maybe I will share with the community so that they can learn about Kubernetes security by playing a kind of CTF or some kind of small challenges, some kind of scenario based. Then I built a lab around it with the bunch of scenarios I commonly while testing the Kubernetes clusters I have seen across the real world, people know or some research.

So I tried to put those in a scenarios where they can easily replicate them and play themselves to learn about kubernetes security.

Ashish Rajan: That’s pretty awesome. Good for Kubernetes. Goat, not to confuse [00:35:00] that with Kube Goat, cause kube goat to something else. I just to throw some stats in there, I saw the Docker pull requests on your Kubernetes.goat GitHub repository.

It was 37,000. That’s pretty incredible, man.

Madhu Akula: Yeah, I’m excited. I seriously still believe like it is super happy that I built that initial. I was like, why, what is the reason I should build this may not be useful. And especially if I really worry about the Kubernetes release process, even if I want to keep up with Kubernetes Goat

I don’t know what else things I have to add because I might have introduced new vulnerability that might might’ve fixed and properly updated in the new thing. But I seen a lot of people got benefited and I got really great feedback. Especially I built the Katacoda playground also for this also because they don’t want it focus on wasting their time setting up themselves.

They can kick in the browser and play and learn about Kubernetes security. So. So many people reached back and gave really great feedback and I feel super happy that it is adding a value to so many people, where they can learn and get started. And hopefully I’m digging to add more scenarios, especially both in offensive and defensive side [00:36:00] so that people can learn more in terms of Kubernetes. Security perspectives.

Ashish Rajan: Oh, okay. Cause , it’s always fascinating for me to kind of look at these things and go, huh. It was always fun to kind of think about from a learning perspective, I’m going to go on documentation, but this way you can actually get your hands dirty as well. I’ve used Katacoda you can probably go down the path of using a browser as a lab instead of trying to thatspot.

That’s pretty awesome, man. And good on you for doing that as well. I can imagine when you’re starting off, like no idea if it’s going to work or not, but you got 37,000. Cause how long have you been running this for.

Madhu Akula: I think it’s not hardly yet. Maybe I think about a year or something like that.

Ashish Rajan: That’s pretty good. Yeah. I’m just conscious that we are coming close to our hour Mark. So I’m just mindful of your time as well. So in terms of, I guess, there are different components. I’ll encourage people to check out Kuber netes goat, any big takeaway that you want to throw at people for who listen to all this and going, man, this is so complicated.

I don’t even know where to start. Before that hold that thought, I got a question from [00:37:00] RKB. Let’s say I have multiple clusters XYZ in same project. And is there any way we can control, give limited of fine grain access to users at cluster level?

Madhu Akula: If I understand that question correctly, I was thinking that they have three different clusters, and they wanted to how least privileged fine-grain controls, for the users within the cluster here. I’m not sure if it is. There are any specific solution or something, but in general, in a practical way, yes, you can do because at the end, yeah.

To create a MLR manifest with Rback, like role-based access, for what privileges we wanted to give, like, let’s say the user developer wanted to create a pod and delete a pod a watch a pod something like that, these three permissions. So you can create this manifest in a like that same definition you define these things and you have like, so you can reapply the same configure across the different clusters.

But I still feel that there might be done by tools like this, like where with the OPA had mentioned, like I think policies where you can define and check across these and apply to be [00:38:00] honest, I don’t know, but there might be even solutions for this. But I’m not sure, but in a simple way, yes, you can do with the back create manifest and apply the same thing across the clusters.

But in a central way, one place, if you want to manage across the clusters, I don’t know to be honest, but it might be there. Some solutions

Ashish Rajan: hope answered the question.RKB It’s an interesting question as well. I actually didnt really think about, would you have multiple clusters running or, well, I guess even if you have multiple clusters that will be talking to the same Kube API though, right?

So at that age normally, or would that be separate for each cluster?

Madhu Akula: I think it should be, if it just completely different cluster, it is completely different API, unless I think I’m not sure how things going with Federation. If they do Federation, which means one cluster has another cluster talking like, Ooh, and organizational units, like, like how AD is like domain controller like that.

But I didn’t ever work too much with federations, but in that way, maybe you can control with all these things.

Ashish Rajan: That might be interesting. Cool. All right. Well, if RKV knows the answer, feel free to share it, man, but coming back to my question sounds really complex? And I’m going, [00:39:00] Oh my God, I have to go learn so much.

And what’s that one thing that you want to, anyone who’s thinking about starting this, they’ve gone through your learning resources as well. Is there a bot path you recommend is use Katacoda to start off with, or try building a Kubernetes cluster or what’s an easy way to kind of, start their journey on Kubernetes according to you.

Madhu Akula: Yes. So I think one thing I would highly recommend is start learning, understanding the technology, like learning central, how it works in a basic, like pay to, apply simple lab or understand. Then once you understand how it works if you want to move to a security, then you can understand what are the components then slowly you can apply these principles.

If you are coming from security background, it doesn’t change much. Apart from maybe the terminology you use to clarify was that AP tables maybe here, like NSPs network security policies. So the terminology and the way you’d apply, things will change, but the fundamentals are still saying, if you don’t understand the technology, then you will confuse, okay, this is a new term.

I don’t know what is this in the is like, so that, that [00:40:00] way it is a, almost quite simple and the same standards, but when it comes from someone coming from DevOps background or operations, or like who is doing Kubernetes and wanted to move towards security, I think they might have quite understanding of working with systems and all of these things, maybe for them to understand, okay, what are the different components and which are the latest I can apply the security, like they think about like The container, maybe what all container security things I can apply, so they can think about these resources.

But I still say refer to the CNCF white paper and also there is a really good checklist in Kubernetes stash, security info, I think, which is maintained by these days. I don’t know if it is quite updated or not. I think Magno maintains when a GitHub repository, which is really quite up to date, which is awesome.

Kubernetes security. I think these are pretty good resources because as it’s fast moving, keep up to date with this. The one thing I would say, that’s kind of. One thing.

Ashish Rajan: So people should have that on their newsfeed in the morning. I guess I would love to see how we go. But yeah,

Madhu Akula: go on. Yeah. I would say one thing, keep follow the people who [00:41:00] does security around these areas like in the community and ultimately attend that CNCF six security, things maybe be really, definitely get a lot of value.

And the Slack communities are very helpful. CNCF has Slack and Kubernetes has security channels, feel free to ask any questions. And I think that really is very humble people and always happy to help. I think those are really great resources as if you house

Ashish Rajan: awesome. No, that’s, that’s a great answer, man.

And I think RKV just came back as well as thinking Rback combination with IAM GKE level. He is going to look at OPA and see if that fits into the bill. Thanks RK for the question, and I think Magna just mentioned network policies also help isolate in the clusters and namespaces so if people want to explore that further and Any thoughts and comments on the suggestions or comments are

Madhu Akula: good. Yeah. I think I had only one confusion when he asked multiple clusters.

Now I got the context because it is saying GKE, which means they wanted to run it Google cloud multiple clusters in the same single Google cloud project. Which means yeah, definitely in a cloud level, they can use the [00:42:00] IAM, which I think RKV only answered are bad control combination of, I am at a Google cloud level, which is definitely.

Nice way to control that. And yes Magno, as he says this, the Network security policies always helped to segregate the namespaces by default Lynch business doesn’t have any hard segregation. There are logical boundaries, right? You are applying network security policies on top of it. So that it’s segregating.

I think definitely those are,

Ashish Rajan: This has been awesome, man. Thanks so much for coming on the show as well.

And for I’m pretty sure. I could ask you questions for hours. And I think it’s just why I had to do a whole month of it. So next few weeks, I’m asking you these questions, but for people who are kind of cant wait , when I ask you a question today, or maybe next couple of days, where can they find you on social media?

Madhu Akula: Pretty much Twitter. I think I’m fine. I think even LinkedIn im active, but not that much, but Twitter is a pretty good, also they can drop me a message, even in my, website. There is a contact form. I’ll try to reply mostly those things all the time. If it is just in here.

Ashish Rajan: Ah, sweet. Okay. And Twitter and LinkedIn.

All [00:43:00] right. I’ll let the it’s been put out in the ether. Now, let people contact you and get it, find out more about Kubernetees, but this has been really amazing, man. And I’m so glad you clarified. The conclusion is it’s not vulnerable by design. It’s just vulnerable by default

Madhu Akula: it’s not actually vulnerable design. It’s more like a, it’s a misconfiguration are people does. This has to be securely configured that does that.

Ashish Rajan: There you go. Yep. So , make sure you check the defaults and make sure you check change your password. Like security awareness is what you’re creating for Kubernetes, but thanks so much for this man.

And for everyone else who’s been watching do follow us on all social media, like, and subscribe, feel free to leave a review of the podcast as well. We’d love five star reviews from wherever you are and it always helps us get more guests as well. I will see all of you next week .

Thanks so much again, Madhu. I was going to say, we definitely need to bring you back again and definitely need to have these conversations more often. So thank you again for taking the time out and staying up late

Madhu Akula: for us. Thank you. Thank you so much for having me.

Ashish Rajan: All right, everyone. Peace.