Ashish Rajan: [00:00:00] Do you know what a sidecar is in the container context? If you don't, this is probably the best episode for you. Because in this episode, we had Magno Logan from TrendMicro and he spoke about one of the most unique ways you can actually look at attacking a Kubernetes environment without actually being spotted.

We're talking about sidecar. Yes. The sidecar, which is used to monitor your Kubernetes environment for performance, for metrics, and anything else that comes with it as well, sometimes without making too much noise, how would an attacker make themselves sniff the traffic without being noticed. Essentially, this was a talk that Magno was giving at CNCF North America based on the research he had done.

He spoke about some of the common attack paths that you would hear about. And then we dive into how we can use sidecar as a way to sniff traffic and potentially make changes without having the need to make a lot of noise. If you know someone who's looking at testing the kubernetes security from a pentester perspective or research perspective, definitely share the episode with them if you find this valuable. If this is your second, third, or maybe even 10th or maybe 50th episode that you're listening to of Cloud Security Podcast or maybe watching on YouTube channel, and you have been finding us [00:01:00] valuable, I would really appreciate if you could take a few moments to drop us a review or rating on your popular podcast platform like iTunes or Spotify, that is, if you're listening to this, if you are watching this on YouTube or LinkedIn, definitely give us a follow or subscribe.

It definitely helps us spread the word. Let's other people know as well that we have a community that we would love to welcome them into. We are a growing community of over 50, 000 people so far. So we would love to keep growing that and keep spreading the good message of cloud security and how to do that right.

But maybe a good place to start is if you can share a bit about yourself.

Magno Logan: My name is Magno, I'm an information security specialist. I have a background as a developer. Right now I do cloud and container security research for the past few years and one of my topics has been around Kubernetes and Kubernetes security.

So that's pretty much it.

Ashish Rajan: What are some of the common things you've been seeing in your research for Kubernetes clusters as security threats that people should consider?

Magno Logan: One of the main issues is exploitation of the cluster, right? So once, once you compromise a cluster what can you do? We've seen many attacks in [00:02:00] Kubernetes clusters that potentially lead to crypto mining.

Crypto mining, it's a very noisy attack. It's just automated attacks to make easy money. But on the research side, we're always trying to find different ways to, okay, how can someone compromise a Kubernetes cluster and maybe stay hidden? Or, do other things besides crypto mining. So that's the idea.

Ashish Rajan: What are some of the initial entry points that you find people get into a Kubernetes cluster from? Whether it's managed or unmanaged.

Magno Logan: Sure. The three main ways, right? According to the MITRE ATT& CK for containers are either from an exposed application. Let's say you're running your application inside a Kubernetes cluster.

And then that application has a vulnerability, right? So that's one of the ways. You compromise the application. Let's say you get an RCE or command injection. And you are inside the container inside the pod. Another way is from exposed Kubernetes services, right? So before in previous versions, we had like the Kubernetes dashboard, which was enabled by [00:03:00] default, and that caused some issues.

Like for example, the Tesla hack that happened a few years ago, now that has been mitigated, it's not enabled by default, but there are other services that are still leveraging that can be exposed through your cluster. Like through your Kube API server, for example, and that could be a problem as well.

And the third item or the third point for initial access would be valid accounts, right? So if someone is able to get access to a developer's machine or access to a source code repo that has some hard coded credentials for a cloud account or a Kubernetes cluster. that could lead to compromise as well.

Ashish Rajan: Would that a bit different between a managed Kubernetes and a self hosted one?

Magno Logan: That's a good question. I don't think so. I think it's the same. The difference between manage and unmanaged. It's basically the shared responsibility model, right?

With the cloud, for example, with managed clusters, the provider is usually responsible for protecting your control plane, right? So the main components of your Kubernetes cluster. But the unmanaged Kubernetes, [00:04:00] you have to take care of everything. So it's like a two edged sword, right? Because like you do have control of everything in the unmanaged cluster.

You have to do more configurations. So it's easier to get things wrong. Yeah. So I think it's the same. You also can't trust the cloud provider and what they're doing, right? You have to trust, but verify, right? So if they're protecting the control plane, you still need to validate if they're doing a good job protecting the control plane of your cluster.

Ashish Rajan: Oh, okay. I like the shared responsibility model because it's good to remember that as well. Once you have access, like an attacker has gotten in, irrespective managed or unmanaged, I think now you're in the Kubernetes territory, once you have access, what are some of the ways you've found that you've learned that people maintain persistent access into the system?

Magno Logan: One of the main ways, or easiest ways, to get persistence into a Kubernetes cluster is to deploy a pod, or if possible, a privileged pod inside the cluster, right? But that all depends on the configurations and the [00:05:00] permissions that you have inside that cluster, right? From that compromised container.

It's all related to RBAC, or Role Based Access Control in Kubernetes, which is very tricky to get it right. But there are other ways as well. You can look for secrets inside the cluster. you can try to escape the cluster itself as well, if possible, through some kinds of vulnerability, depending on the versions and the applications that are being used.

So it really varies. But yeah, most of the time it's different techniques, either trying to break out of the container or break out of the cluster itself.

Ashish Rajan: Okay. And when you say break out of the container, is that the container escape? Yeah, container escape. And why is that bad, because people might just hear container escape you come out of cluster, what is that?

Why is it bad to have container escape?

Magno Logan: Yeah, because now the container is supposed to be this environment where you are segregated from the node, right? From the virtual machine or the EC2 instance or virtual machine there. So once you break out of the container means that you're out of the container and now you have access to [00:06:00] the host.

So now you can see much more, you can have more permissions and if you have like root privileges. you have ownership of that host, right? That node. And it has more resources, because it needs resources to run all the containers inside the node. It's easier it's actually better for someone for crypto mining, because instead of the container, they're now running on the node, for example.

And it makes it easier for them to hide their activities, because they could be able not just to now deploy new pods. But they might be able to run containers outside of Kubernetes, for example, just using Docker on any other runtime engine. Yeah. Yeah. So that's a problem as well.

Ashish Rajan: And they could even get access to the actual account as well, like Google account, database account, Azure.

Magno Logan: Yeah, exactly, right? So through the instance metadata APIs, right? Even inside the container, you're able metadata. Yeah. So that's a problem as well, which I don't think it should be the case. But yeah, once you break out of that, you can access that and if you have an IAM, the permissions attached to that instance.[00:07:00]

You could be able to access that as well and get the keys and everything.

Ashish Rajan: Okay, all so maybe moving that a bit further along down the attack path then, you've managed to get access, you have, okay, at least an understanding of what the threats would be. How would you persist your access without creating too much noise.

Because there's one way to just go, I'm going to go full bang, create a pod, as you said. But how am I going to make sure that I do it without creating too much noise?

Magno Logan: We need to understand that Kubernetes is a living system, right? It's always trying to achieve that desired state. Whatever is defined on your database, your etcd key value store, it reflects back to the cluster, right?

We need to understand that. If I need to maintain persistence without generating much noise. One of the techniques that I think would be ideal, and it's like you mentioned earlier, it's a low hanging fruit, would be messing with the sidecar containers, right? These are containers that are inside the same pod [00:08:00] of your main application containers, but they're not usually not monitored as much because they are second tier containers, right?

They're used for logging, for measuring purposes for telemetry and they're not the main core of the business, right? So that's the idea here of the sidecar injection vulnerability.

Ashish Rajan: And what are these sidecars normally used for? Does that be like logging or what are we using it for?

Magno Logan: So the idea of the container is that you only run one process per container, right? So if you have one container running your application, That's one container. That's one process. Yeah. So if you need to collect logs from that application and send to a centralized location, let's say a SIEM, for example, then you need, technically, you deploy a new container next to it.

That's why we call it a sidecar. Okay. It's like that motorcycle.

Ashish Rajan: Yeah, like a side, like a little thing, yes.

Magno Logan: Yeah. Same thing. And that other container's main responsibility would be just collecting the logs and sending to a centralized location.

Ashish Rajan: Would that be another pod? [00:09:00] Or would that be another node?

Magno Logan: It's another container on the same pod.

Ashish Rajan: Okay, on the same pod yeah.

Magno Logan: So you know that the pod is the smallest unit of a Kubernetes cluster. And one pod can have one or many containers, right? So the sidecar sits on the same pod and it shares like IP address and storage.

Ashish Rajan: So it can communicate, yeah, that's right.

So that's why it's problematic as well because technically to your point it's low noise because it's still in that same pod. So unless you actually turn off the sidecar or you do something dramatic with the sidecar, it should not be detected. Is that a good evasion technique as well for detection?

Magno Logan: That comes with the remediation techniques, right?

So if you have something like admission controllers, right? Which are like the bouncers of a nightclub, right? So they control who or which container is admitted to the cluster that can prevent any kind of issues from happening. If you need to deploy a new sidecar, for example, but also you [00:10:00] need runtime security.

So after your cluster is running production, if something changes, if a new process spawn or something happens inside your container that it's not expected, and we call this like drift and the technique it's drift detection. Detecting what change from the container image to the container that it's running in production that can so the runtime security would be able to detect that because they're also looking at least they're supposed to be looking to the sidecar as well.

They're looking to all the containers in the cluster.

Ashish Rajan: For people who are listening for the first time and going, okay, that sounds pretty problematic. Is there anything native in Kubernetes to look at Sidecar or is that they have to go for one of those CNCF projects to look at it?

Magno Logan: There are some admission controllers that are native to Kubernetes, you can leverage as well.

Yeah. But you can also use third party tools from CNCF, for example, Open Policy Agent or Kyverno. Yeah. And for runtime security, there is a great project called Falco that has a lot [00:11:00] of rules, a rule based engine for detection, right? Yeah. And will probably flag those sidecar containers that have been compromised if you have the right rules enabled and if you're monitoring the alerts as well.

Ashish Rajan: Yeah, sounds good. How would you go about, for people who are listening to this and wanting to understand how can they have detection rules for this or maybe even prevent this in the first place from happening. What can they do today? Should you look at doing the sidecar thing probably better?

Because I have to maybe to your point, if you have best practices in mind in general for how to go about security, I would love that as well. But maybe to start off with, just because specifically we're talking about sidecar. Is there ways to detect and prevent attacks from sidecar?

Magno Logan: Sure. I'm not saying sidecar is a bad thing, right?

It's one of the design patterns that people can leverage for Kubernetes. But you should really think, because once you start adding sidecars, it adds a lot of complexity, right? So whenever you had one container, now you have two. And so let's say when you scale that to hundreds of thousands of [00:12:00] containers, that can double or even more if you have more sidecars, right?

We're assuming just one here, but that can add a lot of complexity. My first recommendation would be just like what I say for Kubernetes do you really need Kubernetes to run your blog, your web application Think about, do you need sidecars to run this, system of monitoring, logging, service mesh, whatever? Or can you really leverage just one container for that?

Ashish Rajan: You know what happened? People unsubscribed after this. They're like, what do you mean Kubernetes is not required? Kubernetes can solve all problems. Yeah. Maybe to add another layer to it as well. Where can people learn about this as well? I feel like it's not enough spoken about in terms of how can you A, attack or defend against these kind of things.

Where do you recommend people can go to and learn more about this? Put a link to our talk as well so people can do that as well. But where can people learn more about this?

Magno Logan: About sidecar injection, for example, there is a separate to the MITRE ATT& CK for containers. There is a Microsoft Threat Matrix, which the MITRE ATT& [00:13:00] CK was based on.

So Microsoft published like the kind of their version of the matrix, what they were seeing in AKS, for example. And that has this technique, the sidecar injection, right? That's one of the only places that I found some documents. Oh, it's not popular enough. No, it's not really.

Ashish Rajan: I wonder if people are exploiting and they wouldn't even know that it's being exploited.

Magno Logan: Because it's if you think about it, yeah, you're just deploying maybe a new container or compromising a container that it's running. Yeah. But it's a sidecar container, right? It's something that at least from my perspective and what I've seen in production, people don't give the same amount of care and same amount of protection that they give to their main applications. I may be wrong, but that's my feeling as from a researcher perspective. So I think it's easier to stay hidden if you compromise a sidecar than if you compromise the main container of the application.

Ashish Rajan: Yeah, fair enough. That's most of the technical questions I had. I've got [00:14:00] three fun questions for you as well.

First one being, what do you spend most time on when you're not doing research on cloud native and Kubernetes and all of that?

Magno Logan: Recently, actually last year, I adopted a rabbit, so now I have a dog and a rabbit. Pretty much most of my time is taking care of both of them, besides family of course. Watching movies with my family,

Ashish Rajan: yeah. Awesome, and if you could have a superpower? What would that be?

Magno Logan: Given the current circumstances that we live in, maybe giving people empathy to understand each other's problems and each other's side, because sometimes most of the conflict and miscommunication that we have anywhere in the world today, it's about lack of empathy of seeing the other person's perspective.

Ashish Rajan: Yeah. Awesome. No, thank you. Very well said. And thank you for sharing that final question. What's the best part about coming to KubeCon?

Magno Logan: Best part about coming to KubeCon is I think it's seeing friends, seeing people that you haven't seen in a while, meeting new friends [00:15:00] as well, and being able to share what I learned and what I do as well.

Ashish Rajan: Yeah, that's awesome, man. Thank you for sharing. where can people find you on the internet if they want to talk more about sidecar injection, is that what you're calling it? Yeah. So I will obviously put the link to your talk in the shorts as well, but where can people find you on the internet to talk about this?

Magno Logan: The best way to reach out to me is through LinkedIn, Magno Logan on LinkedIn, best first off contact. And if need be, then I can share my email and other things. Yeah.

Ashish Rajan: Awesome. I'll put that link in the short. But thanks so much for coming in. Thank you. Thanks everyone for watching. We'll see you next episode.

Peace. I hope you enjoyed this episode and I will see you in the next one. Peace.

‍