Ashish Rajan: Hello. And welcome to another episode of Virtual coffee with Ashish, with the cloud security podcast. This is a weekly podcast where we talk about every topic under the sun about cloud security and everything that goes into it. Today’s topic, as you can see at the bottom is Kubernetes security explained by Kelsey Hightower.

Hey, how’s it going, man? Thanks for coming in.

Kelsey Hightower : Hey, no worries. Happy to be here.

Ashish Rajan: Oh, I’m so glad he came in, man. So morning to everyone who’s come in, our first question, I know you’re a very common name, but for people who have never heard of Kelsey Hightower, who is Kelsey Hightower and how did you get to where you are today?

I guess that’s like, let’s start there.

Kelsey Hightower : Yeah. I mean, I think of just a person trying to get into tech. So I kind of started with, you know, what I used to call self study, but of course, you know, reading textbooks and trying to learn this computer stuff on your own by just getting certifications. And I got into the industry.

And then once I got in, I kind of made the journey from system administration [00:01:00] to software developer. And I worked in all kinds of companies like enterprises, where you have the whole, it, everything takes long to provision that kind of thing. To like web hosting companies, where a company called server beach, they used to host YouTube before Google bought them.

So I have a lot of experience across lots of things, but I got my start in open source in the Python community. I used to speak at local meetups in Atlanta, and I remember starting to contribute to tools like cobbler. This was like provisioning red hat systems with kickstart files. And once I started contributing to open source, I really got into things like config management.

I used to work at puppet labs interfaced with the community. And I think as of late, most people may know me from my work in the early days of Kubernetes wrote a book Kubernetes up and running with my coauthors Brendan burns and Joe Beda. And so, yeah, I’ve been around Google cloud for the last five years.

So a lot of people may know me from the whole container space and now I’m making a slight transition to the serverless world .

Ashish Rajan: Interesting. So Kubernetes. there’s so much to unpack there and I’m going to definitely [00:02:00] going to touch on these because they definitely seem to be quite hot topics, but because it’s cloud security podcast, one of the questions that I ask and I always get a different answer keen to know what does cloud security mean for you?

Kelsey Hightower : , I think for a lot of people, there’s a split responsibility, right. You know, we assume that all the layers are doing their part. We assume that the cloud provider has given me shared infrastructure and they’re locking down the hypervisor. The virtualized stack I’m using is true. Multi-tenant so I don’t have to think about a noisy or malicious neighbor.

And then once you get up to the various abstractions, like if you get a VM and now you have this kind of share kernel. Now we talk about you’re part of this. You have to keep that kernel patched you’ll take those patches whenever your organization is ready. And then we have other systems like our databases, and we think about our networking things and those have configurations and you can be relaxing, your security, let everything in and deal with it at a different layer.

So I think it’s all about this split security model inside of the cloud. We try to do a good job with identity. This is one thing that I’ve seen a lot missing [00:03:00] from a lot of security practices where people will go get a firewall rule. Stick it in front of the thing and make sure that no bad person comes in, but you just kind of protecting ports and IPS.

Now where we’re talking about is identity. So can this VM with this identity talk to this particular database, and then we’re trying to put policy that’s a little bit more granular. Like, can it talk to this database on Wednesday from this region with this type of request? So I think we are getting really good at the identity part.

And I think we still have a lot of room to go in terms of, you know, off in is who you are obviously is what you can do. And that all Z requires a lot of granularity that needs support from the apps and the systems that backed them. So when I think about security, I just think of it as a practice. We’re trying to use the tools to the best of our ability to make sure that the only things happening are the things we want to happen.

So that’s the way I kind of look at this whole thing.

Ashish Rajan: That’s an awesome answer and well, because I guess how here as well, and you’ve co authored a book on Kubernetes. So [00:04:00] I have a lot of people still who asked me, where does this Kubernetes thing and why? But before we dig into that it what is this Kubernetes thing people who don’t know what it is, I guess, why is it so popular?

Yeah, I mean, I think on the simple surface, it’s a collection of patterns. People have been doing for about 15 years. This is why they call it cloud native. This idea that if a machine were to go away, you should be able to move the app to another machine. Why not this idea that we should be able to hook up our applications behind a low balancer using something like service discovery.

So if you’ve have five of this type of app, you can give it a service name. Like this is app Fu and food should be behind this particular route or domain. So that’s service discovery in a nutshell, and we’ve been practicing this kind of stuff for a very long time. But now that we started to automate these things right during the Dawn of configuration management, Maybe you’re used to infrastructure as code, right?

If you’re using puppet, use a puppet DSL to try to describe your systems. But I think Kubernetes does, it takes a step back. It takes all the systems, all the load, balancers, all the firewall rules, [00:05:00] all of that stuff, and just puts it behind one common API and it abstracts away your application by forcing you inside of a container image.

And so with this collection of things, we now get this new system we call Kubernetes. But if you look under the covers, there’s still a bunch of VMs is still a bunch of load balancers. And it’s still a bunch of the fundamentals that you knew already.

I must say Kelsey, one thing I love about the way you answer questions at least from the talk that I’ve seen, you given all the interactions that we have, our online had offline as well.

I love it. How you kind of break it down to fundamentals. And I think I’m kind of stealing your word here by breaking it fundamentals, but I totally get it. But. I sometimes find that senior leadership doesn’t really get Kubernetes. And I understand. And there’s a perspective to it.

I’m keen to know from your side, because my understanding is in your role, you’re talking to kind of broad breadth of people. You’re also talking to people who are I guess practitioners, you’re also talking to people who are decision-makers. So for decision makers listening to this before they [00:06:00] zone out, Oh my God, , these guys are gonna go into technical word Kubernetes security.

Why should they be considering like a senior tech lead, I guess, would he or she be considering kubernetes at all? And why should they think about it in this stack?



Kelsey Hightower : Yeah, I think when, when certain technology gets standardized, right? Like when the mobile space came, right, you could say, Oh, we don’t care about mobile.

All of our customers use browsers. And then there’s a certain point where. All of your customers start to have mobile devices. So now you’ve got to understand these frameworks to deliver a mobile experience. And then if you try to put your web experience with the mobile device, it’s not going to be really great in comparison to others who really understand how to leverage the technology for most companies, regardless of the marketing, Kubernetes is not going to be the missing piece.

So if you’re like, Oh, we got to transform our business or all of that stuff that you’re thinking about, like Kubernetes is not the thing. That’s like, if you add that boom, now you’re transformed. That’s not it. But it is a tool that if your team is ready to try to figure out how to increase resource utilization, [00:07:00] if your team is trying to figure out how to really get reliability yet by automatically failing over between systems.

So a lot of people are in the cloud, but they don’t really leverage the cloud. They treat the cloud like a local on-prem data center. They still open up tickets to provision virtual machines and package apps. So if you think about what Kubernetes is imagined a place where. You can hire people who have 10 to 15 years solid experience of running distributed systems, getting to those availability numbers, across multiple zones and regions.

That’s a skill that everyone doesn’t have, but guess what? All of those people have come together and they’ve serialized their experience in this project, we call Kubernetes the same thing was true for Linux, right? Like you could have had 10,000 other operating systems you could have used, but people rallied behind this open source project.

We call Linux. So I think there’s so many layers of the stack where we start to get the technology checkpoints. So news for everyone, just like virtualization was a technology checkpoint. The cloud was a technology checkpoint and Kubernetes represents [00:08:00] another technology checkpoint. So if you leverage it, you get to zoom into the future where everyone else is and get to just leverage those practices.

So all those people that you hire can go focus on the thing you care about most, which is the business. Most people aren’t doing that because they’re sitting around trying to reinvent some of these things that are already available, like in libraries and frameworks, like Kubernetes,

Ashish Rajan: that’s an awesome answer.

And that makes me kind of thing that, so now how you answered the question before, where Kubernetes is kind of as is a container management of , what fundamentals of because you, or fundamentals of compute kind of align for Kubernetes and he’s like for four I’m thinking of people who were probably from a security background have done Sysadmin for awhile, or maybe even done network security for a while.

They look at Kubernetes as this Oh my God, this is so complex. There’s so many layers to it as an API and then the, or Istio and all these, if you keep throwing words at them, it just kind of lose them really quickly. Because [00:09:00] we have you here and I wanted your help with. Stripping this down to the fundamentals of compute, I guess in the context of Kubernetes, when I’m thinking about this, as someone who probably has experience in networking and old school of vitual machines is Kubernetes I guess, compatible to compute, or is it just completely to your point, transformative technology

Kelsey Hightower : it may help to understand, what do you have to do if you don’t have Kubernetes?

Right? So let’s say your company wants to go into the cloud or even VMware on prem and something, you start with a bunch of virtual machines. So the first thing that team’s going to do is say, Oh, we need some automation so that you can go get puppet and then people can write a bunch of puppet code, or maybe you can reuse some people’s specific code.

And then you to say, what apps are we deploying say, well, we’re doing Java over here. We’re doing Python over here. So let’s make sure we right. Some automation code that supports all of those languages. Great. Okay. Now what do we need to do? Also we need to secure the system, right? So we need to go and put some agents on the server, decide what we want the [00:10:00] configuration to be.

Do we want to allow SSH? What does the tool need? Okay. Make those decisions then what about logging? Yeah. Some people want to log this way. Some people want a log that way, and then we should centralize the logs to a place where we can have our audit logs. So when people are using the system or if there’s a system crash, that way people can be alerted.

Great. Well, what happens if a system crashes? Oh, we’re going to have a process. If a server dies, we’ll just have multiple servers. So if one of them dies instead of having three, we’ll have two and that’ll buy us enough time to spin up the other server. Well, how are you going to spend at the other server or we’re gonna get this other automation tool that watches the other one.

So if one goes away, then that one will create another server, but that only works for VMware. But what about for the cloud? Oh, we’ll tweak it and we’ll configure that one too. So you do this for 10 years and your team is still missing something critical. When I want to launch a new application, we got to go through this process.

What server should we deploy on? Well, that one has three things running. So not those let’s go buy five [00:11:00] more servers. And it turns out the new app is not even using all five servers. You don’t care because you don’t really know that it’s not using, but it works. So now you’ve bought these servers. You spent maybe tens of thousands of some people, a million bucks, and it’s only using 5% of the resources.

And you keep repeating this. So now you have a bunch of underutilized resource. This is actually bad business, by the way, most businesses that run this way, where you buy more product than you can move, you will go out of business because that is not a very efficient way of dealing with your inventory.

Kubernetes sits on top and says, we know how to build what we call schedulers. Just describe the workload. Automation is built in. We will pick the right server because we already know what’s on everything. We know how to manage the memory. We know how to manage the CPU. If a machine were to fail, there’s built in auto provisioners for all major cloud providers, your team, isn’t going to go work on that layer of the stack security standpoint, the virtual machine, we lock it down.

Read only no one [00:12:00] logs into the server anymore because we’ve already having a decision around logging. We already have a decision around metrics. So these things become so standard that most people in the Kubernetes community can’t really think about going backwards and trying to make the same 100 collection of decisions and build their own Kubernetes like thing.

So that’s why it’s taken off. So it’s really more about without Kubernetes, what would you end up doing anyway? This is what you find inside of these, these open source projects. Right? So I think that’s what most people have to consider when they say, why are people so happy about Kubernetes? Well, the ones who’ve tried to build something like this before, know what they’re getting out of the box.

Ashish Rajan: That’s really? Yeah.When you’ve answered it. I’m just thinking that’s actually true. Because a lot is it is a kind of a spaghetti of different technology everywhere. It’s not that. You only have one cloud provider. It’s not that you only have, I guess, one application in your environment, you have multiple applications possibly distributed across global enterprise or global [00:13:00] edge centers as well.

So I guess to me, then I feel like there are kind of stages to challenge just you may face as you’re growing in. So, but Kubernetes seems to have abstracted a lot of I guess complexity or orchestration, distribution and generation. If I’m like a company, which is a startup and I’m trying to deploy Kubernetes and I’m starting with Kubernetes today, are there like different stages of maturity?

Oh, I guess say I’m a regular startup today. I’m going to be faced with tomorrow as I’m deploying kubernetes from stage one. When I was a startup to, I guess, fund I’m Facebook now, , is there like different stages of kubernetes security challenges that you kind of see it move across

Kelsey Hightower : for example, some people back in the day, you could buy parts and build your own server. I’m going to get this motherboard from here. Get the CPU from here, got to tie it together. I’m going to stall this operating system. Know what? I’m too good for a Linux distribution. I’m going to compile the Linux, kernel myself and keep up with all [00:14:00] the layers and changes, patches and bugs, and just be my own distribution where instead of using red hat, My team will make their own distribution because we can do a cheaper and more customized.

And then people don’t do that anymore. We just go get a Linux distribution. We’ll use that. We don’t buy and build service from scratch. Too many people don’t do that anymore. You go buy something that has already built and you get some model number and you roll with that. Same could be true for some people.

Like if you’re starting a company today, would you go and build a brand new data center? Probably not. You might go to a place that already has a data center and you just colo and get some colo space, or you may consider just using the cloud. Kubernetes is very much the same way. You know, four years ago, there were a lot more companies out there and customers who would go and get the source code for Kubernetes.

Build it themselves and then try to configure every aspect of it, including the security. Right? Some people didn’t do any security at all. They’re like, Hey, you know, that takes too much time dealing with SSL certificates to secure the connections [00:15:00] in between the various components. Some didn’t know how to configure security between the components.

So they just didn’t and then even know what they attack that theirs are because they were so new. So they were just installing it, they got it up and running. It’s like, Oh, this is amazing. Let’s make it available to the entire team. And then later on they’re mining Bitcoins because people are running their containers on their clusters because they didn’t know what to secure.

And then we get into a world where we are, I think as of a couple of years ago, where most major cloud providers will have a distribution of Kubernetes ready to go for you. Meaning that they’ve tried to do. As many of the integration things they’ve locked down, things they put security in between. They turned off the unsafe stuff inside of the box.

You kind of get more of an opinion. It a bit. If you were a startup. You have another option now, which is not even deal with the cluster at all. So even though you can click a button, pick how many nodes you want, pick what version that you want, maybe decide to turn on upgrades or not. And if you want to have the cloud provider automate the patching, now we have a thing.

We call our GK [00:16:00] autopilot and the concept there. And I think more cloud providers to be fair. Amazon also has a thing called far gate, which is pretty nice what they’re doing and what we’re doing. We’re making the cluster disappear no more Kubernetes cluster, but you keep the API. So what does that mean?

We lock everything down and limited to running only the workloads you want. So no need, do you need to figure out every layer? And this is very similar to what we did with the virtual machine hypervisor. That’s hidden from most people in the cloud, the switches, the network, all that stuff is hidden and you just create virtual machines.

So where Kubernetes is in 2021, you can now just say, here is my Kubernetes workload. And the cloud provider runs it and takes the responsibility of keeping the entire stack fully patched. And you just get the API. So as a startup, more than likely you’ll choose something like that.

Ashish Rajan: And as you’re growing from a startup to say a mid medium sized business, maybe a better way to ask [00:17:00] this question is if we were to kind of look at challenges in a, I’m going to quote unquote DevSecOps way for security and kubernetes , I’ve taken your advice.

I’ve used GKE or Fargate`, and I’ve gone down the part of deploying clear Kubernetes clusters. What are some of the anti-patterns that. If I’m starting today, I should avoid in terms of security. Like to your point, a lot of it is abstracted. So is it okay for me to asume that GKE and fargate or anything else on Azure has taken care of security?

Or is that something that’s like, what kind of security thing should I be looking at?

Kelsey Hightower : Yeah. So this is going to be two layers. One. There’s going to always be your app layer, right? You decided to expose your application to the world and that’s going to come with some trade-offs. And when you think about that particular trade-off the main thing to focus on is like all the common things you have to do.

So whether you have Kubernetes or not, the same security rules apply at the application level at the platform level, the biggest gotcha. Is any vulnerability inside of [00:18:00] Kubernetes itself is going to be an inherent vulnerability for you. That’s just how it’s going to work. You’re going to inherit that vulnerability.

What do I mean by that? A lot of times we’re going out to the third-party ecosystem, right? Because you want to buy and not necessarily build everything you want to use. Maybe you go find some database service or a monitoring agent that needs to run with elevated privileges, because it’s going to gather metrics from the low levels of the machine.

So it can report things like memory and CPU utilization. Typically what people are doing is just letting that thing run with route, right? Like all the privileges it wants. And when it does that, if there’s any vulnerability in the kernel, the, you may compromise not just the machine where other apps are running, but you may also compromise the ability to talk to the Kubernetes API and start doing things to other apps that you didn’t have permission to do before.

So I think this is where having a platform like this, that kind of runs in God mode. If you compromise it, you’re going to have a problem. This is just like the cloud. If you have a security token that allows [00:19:00] someone to log into your cloud provider, then they may be able to deal with all the databases and actually change your own security policies if they are able to gain that kind of access.

So you have to guard very carefully access to the platform as well. And that just think about the applications

Ashish Rajan: Thats Awesome. I’m going to switch gears, I guess, for lack of a better word. I’ve got to take some of the questions coming in from the stream. I have a Magno Logan asking, how do you deal with the high complexity to learn Kubernetes?

Kelsey Hightower : So like Linux, for example, most people don’t know Linux either and they use it every day. Most people doesn’t don’t know how the virtual file system works. They don’t know how IO works.

They don’t know how I notes work. Most people don’t know how Linux actually manages memory. They don’t have any idea how memory segments work. They have no idea how it picks an assigned CPU or how it prioritizes CPU caches or what a cache misses. Most people have no idea how the Linux kernel works, but they’re able to use it, right?

Because typically you use something like you bumped to a red hat and they assembled the [00:20:00] user land and kernel space and they put some defaults and they give you a system that works. And through muscle memory, somehow people remember how to use 7,000 tools, right? Cat proc file system pipe into tr Laura Case.

The second column, put it in this other thing. So I can see the process. That’s not easy. That’s crazy, but we know how to do it. So it doesn’t seem so scary because a lot of us have been doing it for so long that we think we understand like a bash shell versus all the other interfaces in the Kernel.

This stuff is just, it’s not even compatible. Right. We, we, we piece it together. So number one, Kubernetes is new. So if you’ve never seen Kubernetes before, and you really want to understand it at the degree that I’ve been talking about with the kernel, like how does CD work? That’s a whole distributed key value store built on the raft.

You know, a cluster protocol like that. That’s stuff aint easy to understand. Consensus is not an easy topic for many people to understand by itself. [00:21:00] Networking is not an easy copy to understand by itself. So I think what most people have to do is you have to give yourself the same amount of time. If it took you 10 years to get good with Linux, then come on, you gotta be fair.

It’s going to take you more than 10 minutes to get familiar with Kubernetes. So the way I would deal with this is start with learning how to use the criminals API, right? Like most people start system administration by SSH into a server, running some commands with bash in the Kubernetes world that would be creating your Yammel files and QC tail apply.

You may not learn how to roll your own Kubernetes cluster on day one. So imagine getting a fully managed Kubernetes cluster. When you click the button, you get an API and they may deal with things like auto scaling for you. Right? So at that point, you’re just learning the Kubernetes objects, how to store secrets, how to reference config, files, how to configure a load balancer using the Kubernetes API.

And I think for most people they can grasp like, Oh, there’s a configuration syntax. I’ll learn it. And I can configure it to run my apps in a container. [00:22:00] Now, when you go below that level, you start talking about managing Kubernetes itself. That’s where I think we got to exercise a little bit more patients to say, Hey, maybe I’m going to start by troubleshooting in the node.

How does Docker work? How do I manage CPU and memory? That kind of thing. But then the control plane is a whole different animal. If you want to learn how to Kubernetes scheduler works, that’s very different than learning how to create an app that runs. On Kubernetes. Most people don’t know how the Linux kernel scheduler works either.

So I think what we gotta do is make sure that we don’t try to lump in all aspects of Kubernetes on day one, use the API and over time learn how to manage the underlying system. If it ever comes to that.

Ashish Rajan: I love the answer. So you ready to API learn how to use that, to configure and achieve that deploying application and then maybe to your point, graduate to scheduler and everything else that comes with it almost seems like a natural progression where eventually we’ll all get to a point where we just have to use GKE or something similar instead [00:23:00] of worrying about, I guess, the complexity behind it.

Kind of like your example about Linux. Is that what you feel?

Kelsey Hightower : Yeah, my most people that went to the cloud, right? Like, imagine if you went to Amazon 12 years ago, you don’t, you’ve never seen the hypervisor before. Yup. All you’ve seen is the Amazon EC2 API for creating virtual machines. You don’t get to see the hypervisor.

Unfortunately, in the Kubernetes world, we started by showing people, not just the hypervisor, but the internals of the hypervisor. Right. So people, right. People thought they were, Oh, I need to learn it all. So it’s good that it’s there. It’s good that we have that kind of transparency, but that’s not the same.

It’s like, Oh man, cars are complex. And two, I learned how to rebuild a car. I’m not going to learn how to drive. Right. You got to separate those two.

Ashish Rajan: Yeah. And I think I find it hilariously funny because I remember doing a project once for a company where I remember this is like a few years ago. I exactly this, when you talk to a person who understands kubernetes or at least you think they understand kubernetes, and they’re going into these deep, deep detail of [00:24:00] carnival and everything that you have for build security from scratch, you’re even taking steps through like, Oh, how do I control this?

I find those really hilariously funny. That. So no one talks about this, which is what you just said, but there are two aspects to it. One is the complex side going deep kernel level. If you want to get into that and make your own kubernetes feelings, I guess. And the other one is just simple one. So a great answer.

Thank you. Magna, hopefully we answered your question as well. Does modern kubernetes make multi-cloud deployment management a reality?

Kelsey Hightower : Not by itself. You know, I know a lot of people will try to say that not by itself, right?

Because we could have asked this question 10 years ago, does Linux make multi-cloud deployment a reality? Right? All the cloud providers have SSH. All of them have pretty much the same version of these distributions. And if you think about it, I used to work at puppet labs and this was kind of the vision we had.

Right. Just put puppet on all of the servers. And then you can now have the same deployment tool across all the cloud providers. In [00:25:00] many ways, Kubernetes is very similar where we have a config language basically for running on top. So when you create a Kubernetes cluster, you’re taking all the VMs, depending on your distribution, it’s going to hook up all the networking stuff.

And it’s going to give you API. Sometimes they’re going to be similar APIs. Like if you want to create a load balancer between AWS and GCP, while you might be able to create the same object to create that load balancer all right. That’s cool. So we have a common configuration language across the cloud providers, but where does the extraction leak?

That’s the real important part. The abstraction tends to leak. Think about the metadata service. If you write an app using the S3 Amazon library that S3 Amazon libraries, going to assume you’re running on a VM or maybe a container with access to a metadata service where it can go and grab the credentials based on the service account and use that to connect to S3.

Well, if you take that app and you put it on GCP, number [00:26:00] one, the API and the metadata services, not going to be the same. And so that’s going to be a leaky abstraction. They’re not going to be the exact same maybe security configuration layer. So one thing I think it will help you do is you can keep a common workflow between the cloud providers, at least at the compute layer and for any of the layers, that Kubernetes is a Quip to configure.

Meaning if you can create a load balancer using the Kubernetes API, that’s going to be a portable workflow, not necessarily a hundred percent portable. Abstraction, right. Some load balancers have different features. So even if you use Kubernetes on AWS, let’s say you have some feature for routing. That’s not available in the other cloud provider.

Then you’re going to be in a situation where that same config won’t work on the other side. So I just want to be clear that that for the compute layer, that thing in the middle, you’re going to get a common workflow, common config language. But think about the databases. That’s a bit out of scope from the Kubernetes world.

[00:27:00] So dynamo DB is not available anywhere else. So if your application deployment uses dynamo DB, it doesn’t matter if you have Kubernetes in both cloud providers, the other cloud provider, you’re going to have to try to talk across the wire to that dynamo DB service. So just make sure we know what layers are going to be common and standard and make sure we understand where the abstractions start to leak.

Ashish Rajan: That’s really interesting because each cloud provider has their own proprietary databases as well. So is it right? Then people should look at Kubernetes as a compute rather than, well, I guess you can technically build a database if you want, but technically to your point, think about it as a VM, which can be distributed.

It’s probably a small, simpler way to put it.

Kelsey Hightower : I think what we’re getting to honestly is that, you know, there are some what we call operators, where people are taking things like Reddis and Mongo, DB, and Kafka and Cassandra, and they’re creating Kubernetes frameworks. If you will, meaning not only can it deploy into a side of a Kubernetes environment, [00:28:00] they can use the Kubernetes API to help manage it.

So this is kind of an up level over just raw virtual machine. So people are starting to look at this as an alternative to maybe a fully managed offering, right? So some people will say, well, instead of using dynamo DB, Maybe I’ll use Mongo DB or Cassandra or something else. And what I’ll do is I’ll just run that in Kubernetes.

Well, okay. In that particular situation, Kubernetes might become a very common layer that makes the cloud becomes something that’s very similar because compute networking, storage that’s roughly the same and Kubernetes does a good job abstracting those. So I think it’s really more about they’re going to be specialized services that are outside of the scope of Kubernetes, but Kubernetes is going to give you a very common compute substrate.

So think of it as like IAS V2, right? If IAS across all cloud providers had a more standard, common API that wasn’t focused on virtual machines, but was focused on containers instead. What would it look [00:29:00] like? I think we got our answer. I think it’s Kubernetes.

Ashish Rajan: Yeah, I appreciate that. So a couple of questions then we’ll come back to the podcast questions.

Silver Kumar is saying morning. How to manage secrets in cloud and hybrid Kubernetes world in a secure way, made tools available in the market. What’s the best way. Maybe another way to reword. This is what is the hybrid Kubernetes, I guess, to start with what is that for people who don’t know what it is?

Kelsey Hightower : There is no such thing as hybrid Kubernetes. That’s step one. There’s probably no such thing as hybrid. Anything. You can have your own data center. I have my own data center. Mine is called cloud is called GCP. And I share with other people you have your own, and you may use it only for yourself. We both have power.

We both have network connectivity and we both have some servers, memory and CPU. Yours is over there. Mine is over here. If you put stuff in mine, You need to connect it back to yours over networking. That’s not hybrid. That’s called two data centers with [00:30:00] networking. Now I think people consider hybrid to be an approach.

You know, in the cloud, everything is going to have an API, a billing model, and a certain set of expectations. You go on the cloud is going to be or on print and be like, well, we got some stuff. These people are using chef. These people are using puppet. We don’t have a common API for all the layers. So I get it.

It’s going to be a hybrid approach to solving this problem. But fundamentally, we’re talking about data centers here. Let’s not, let’s not get too carried away with this. So when you say hybrid Kubernetes, I might install a Kubernetes cluster in my data center and I’ll install one in the cloud. The API is going to be 80, 90%.

The same right to deploy workloads are going to be the same coop CTL apply commands. Now you might have an F5 load balancer and I might have Google cloud, loas balancer. And we might even be able to use the same config to get a base set of functionality, like attach these containers to this V host and give [00:31:00] it this domain with an SSL certificate.

You can do that in both places. Things start to fall apart where maybe you don’t have access to like an API for mounting data volumes. Okay. It might be slight differences there. So I think that’s the first thing we’ve got to make sure we understand. You can put Kubernetes in multiple places, whether it’s in the same cloud provider, two places would be one region, a different region.

If it’s your data center, it might be on your premise in mind, but we’re still going to have to be connected in networking somehow. That’s the first step. When you talk about secrets, I think the critical, critical thing to understand here, Kubernetes typically is not the source of truth for pretty much anything.

So what is a secret? A secret could be a private key inside of a, you know, TLS key pair. Right. Private key public key. The server needs the private key in order to serve encrypted traffic and I’ll serve the public key and you can connect trust my cert verified sign. Look at the chain. Boom, sweet. Now, how do I give [00:32:00] my app?

That’s running in Kubernetes, the public key and the private key. We’ve got to put it somewhere. So let’s talk about not Kubernetes, what’s your options. One option would be like what everybody does right now and they don’t want it admit it. You log into some server. I don’t care if you automate it, you log into the server and you copy the files.

In plain text on the server. That’s what you’re doing today. And that’s getting you fairground certified. You’re getting PCI certified by copying plain text files onto it. That’s what everybody’s doing. Don’t don’t talk to them going LA LA LA LA. Everybody’s doing that. And if you try to encrypt it, then you’re going to have to put the decryption key somewhere.

So the app can actually boot up or you’re going to be sitting there typing at the terminal while the app starts to try to unlock their key. So cool. So what does Kubernetes offer? Well, one, you can take those certs and you can put them in something like vault from Hashi Corp. So say, Hey Vault here’s a blob of data.

Key value. Call it app certificates. And now we have [00:33:00] to gain trust to vote. So how does the app ask for those certificates again? Well, one thing Kubernetes helps with is saying for every app, I’m going to give you a service account, which is basically a jot token. And that token is going to be signed by me the cluster.

And I’m going to tell you that this app is Fu and I’m going to make a jot token and I’m going to sign it and I’m going to give it to the app and they well-known location. It’s like VAR, lib security, key, or something like that. So you could take this job token and say, Hey, vault and vault actually works this way.

You can tell vault to trust Kubernetes as the signer of these chat tokens and use it as an authentication mechanism, container boots up, it gets this jot token gives it to vault. And then vault says, wait a minute, let me go ask Kubernetes, Hey, is this, is this legit? Did you sign this? It’s like, yes, I signed it for an app called Fu great.

I have a policy that says, Fu can get these certificates and I’ll pass it back over the request. So now the app has it in memory and the app [00:34:00] can go and configure the HTTP server with the key pair. Some people would say that is the ultimate security, right? You get your identity, you trade it for short-lived certificates and your identity shortly, and you’ve got to keep renewing it, but there you go.

That’s the best. Now let’s take a step down. One step down would be, well, my app doesn’t know how to talk to vault and I don’t want to configure it. Well, what we can do is delegate. There’s a thing called console template that can run as a separate container that has access to a shared file system between your app and this little.

Controller. So console template says, I’ll take the surface account. And those are the problem here. You’ve given somebody else access to your identity that hopefully they will only use it to talk to vault and nothing else, but you’ve made this trade-off right, because you don’t want to write any more code, fine console template grabs.

The token, gives it to vault and does the exact same thing. But except this time it writes it to the file system. So you can read it later, you and [00:35:00] every other container with access to that file system. But it’s better than copying it to a server. So we have an improvement and nonetheless scenario I’m going to talk about here is Kubernetes secrets.

This is when all the people fall out and lose their mind because they think that this is crazy. Typically you’re logging into the machine and you’re copying secrets on the file system. Stop playing around. That’s what you’re doing. So Kubernetes has a slightly better mechanism. Given the Kubernetes API is protected by our back.

We have an object type called secret. You can take the same blob instead of copying it into the server. You can give it to Kubernetes and have it be encrypted at rest. Then you can use our back to control, who is allowed to ask for these secrets, right? So you can lock it down. Think of Kubernetes as a cache.

It didn’t create the key pair, but it’s a cash because you don’t want all the apps going back and forth grabbing. So maybe you give it to Kubernetes. All right, Kubernetes, now that you have it, I don’t have to worry about network connectivity to vault. I don’t have to worry about people copying things to servers.

We have it in a clean [00:36:00] API. Now, how do we get it to the app? Well, in the Kubernetes config object, you have a section where you can reference the encrypted at rest and in transport secret. And say, Hey, Kubernetes, I want you to go to this secret and I want you to Mount it into the file system. Just like I would’ve, if I would’ve copied on the server, but guess what?

Every container gets its own file system. So in this way, I can say specifically, this app gets it and this other app does it and I can see it in the config. So we’re being very intentional about who has access and we have an audit log. So when you do your deployment, the agent on the server will read that you have access to that secret call Kubernetes and say, Hey, let me get that secret mounted in the right place and inject it into the running container.

So if you think about Kubernetes native secrets, it allows you to keep writing code the way you do don’t even know about vault and just inject it. And if you have both, you can use volt to periodically cache. Into all of your clusters. So you don’t [00:37:00] have to worry about vault being down when you need to access those secrets.

That’s the way you need to progressively think about secrets and Kubernetes. So all these people they’re saying, Oh, you know, it doesn’t rotate it. Doesn’t supposed to rotate secrets. That’s what vault does. That’s what these other tools do is typically a cache and distribution mechanisms based on the identity of the service that needs it.

That’s how you think about secrets with Kubernetes. That’s awesome. Great answer, man. And I think, cause it’s kind of, might’ve answered Magna’s question about encryption as developed as over here, out of curiosity you kind of mentioned control plane. So when as a security architect, looking at a Kubernetes cluster, there’s a data thing, control plane.

That’s the terminology they are familiar with. What do you consider as a control plane for Kubernetes? Is that there at that API or an cluster? Is that data plane, like are what, what are you out of control planes. So the main control plane is the API server. So by itself the API server needs a backend.

So MTD is its own key [00:38:00] value store. In many ways, it’s like its own database kind of control plane. You can store data in it. Then you put the API server on top, which understands the Kubernetes API and how to store it inside of NCD. And then you have nodes. The node has a control plane. We call it the kubelet it’s job is to talk to the API.

And once it gets to this configuration, it managed the nodes. What containers to run with secrets, to Mount, et cetera, et cetera. And then we have a scheduler. You can actually run a Kubernetes cluster with no scheduler. You can just put the workload there and there’s a little field called node name, and you can bind it yourself and it will work.

And in the note, we’ll start running it as if you had a schedule it, if you want to automate the scheduling process, then you can run the scheduler binary. And that little controller will look at the Kubernetes API server. List all the pods that have not been matched with the node and they will go and try to figure out what’s the best node based on available memory and CPU and other scheduler requirements.

When we think about control plan and [00:39:00] Kubernetes, when we’re talking about applications, we talk about the scheduler. There’s a thing called controller manager that creates those kind of secrets and various components, the AP side, NCD, all of those together. And there’s also a cloud provider thing that when you create load balancer, it knows how to really go configure the upstream load balancer all of those components together, we treat as the control plane, there’s even a DNS server in there like core DNS or kube DNS, all of that runs over there.

And then on the node, there’s fluid D there’s the kubelet there’s Docker, right? That could be the node control plane. And so when we think about these two things, that’s how Kubernetes tends to work, but you can extend the control plane. That’s where it gets interesting. You can create a new set of objects, configure it as like, Hey, I want to create databases.

All right, cool. You can create databases, design a config. We call them custom resource definitions. And now Kubernetes understands how to store validate and give you access to this object called databases. But now you need to build a [00:40:00] controller, a loop that can watch the API server, pull down the config and then go out and create databases just like Terraform does.

And there’s a lot of people who are using Terraform to create new extended control planes, to manage other quote unquote data planes.

Ashish Rajan: Interesting. So adding another layer to it and where I was going with this was it because now we understand control plane and the data plane. If I have policies, and I guess as everyone else, automation is a friend .

The one question that I had for everyone, and I would like you to consider is Kubernetes the next wave, or is it more the next transformative technology as people are calling it? Do you reckon serverless is the next wave after Kubernetes And I guess Kelsey touched on this as well. If you feel you feel starting today, and I dunno, think of a company that you’re starting today or a small startup, would you rather go Kubernetes or would you go serverless? What would make you go one over the other. I feel like I can talk to you [00:41:00] about Kubernetes and this for ages, man. So I, I definitely feel, I need to do a part two version. I made sure we’ve answered all the questions over there, but one question that I was talking to to the audience is about, and I’d love to hear your thoughts on this as well.

It seems like Docker has kind of, well, I don’t want to say disappeared, but it’s not. I hear more about Kubernetes and serverless among peers and other people who are in the industry then doc ker containers anymore. It’s almost like if you’re doing Docker container right now, you’re probably behind the eight ball.

What are your thoughts on that?

Kelsey Hightower : I mean, talk about another successful project that people don’t talk about that much anymore. Linux, right? Oh, yeah, it’s everywhere. It’s probably grown over the years. So w you know, even though it doesn’t get talked about, it’s still important. It’s still there. The internet also has this kind of thing, right?

Most people you say Netflix now, or you talk about the thing you’re using. You don’t say, Hey, let’s get on the internet. Right. You don’t, we don’t do that anymore. We talk about the thing above it. And I think in the case [00:42:00] of Docker, you know, most of it has been standardized the OCI container registries.

There’s other runtimes now like container D and so Padman from red hat. So when you think about it, even though a lot of people are creating Docker containers and images, we tend to interact more with things like Kubernetes and serverless platforms that take Docker images, but then hide everything else.

So I think for Docker Inc, the business, there are two big things are, you know, like Docker, desktop, the Docker engine, but on the server side, You don’t really need a mixture. You don’t even need them anymore. Right? Like you can literally just use standard implementations or alternative. So it just makes sense over time things evolve.

Right. I don’t even hear people say, git that much anymore. They say, github as if github is the whole thing. So I just think it’s one of those natural things that tends to happen over time. Technology gets introduced layers pile up on top, and then people talk about the thing at the top. And very rarely do [00:43:00] they mention the thing at the bottom.

Ashish Rajan: Ah, right. Okay. So it’s just abstracted to the point that now people are not kind of having that conversation. Cause he’s talked to a lot of executives as well and Kubernetes Serverless is kind of becoming like the next level of conversation, even at that stage. But a lot of people may not be, I guess, aware of it or understanding this, but a lot of technical lead wants to introduce having their companies.

What would you, cause I think I’ve heard some of your talks where you’ve actually helped. I guess executives get done in that space. So for people who are listening in, who are, I guess, technical leaders or security leaders trying to work in the Kubernetes and serverless space, what’s your approach to convincing an executive or someone senior for, Hey, we should give this a shot.

I know we’re asking you to invest money into this and rest in the future. But it’s like talking rockets science to, I guess, a layman,

Kelsey Hightower : I guess I don’t even try to convince, you know, I spend a lot of [00:44:00] my time working on serverless, meaning trying to get people to use something even more abstracted than Kubernetes does.

So a lot of times you ask the executives are number one. A lot of the pain that they express is because for, especially the longer they’ve been in business, they say, Hey, I got mainframe. I went all in 30, 40 years ago. Then people told me about, you know, getting these kinds of pizza box servers. And we did a bunch of that in automation.

And then people said, I need to have virtualization. Then they told me cloud. Then they told me this. So we bought everything. Everyone told me to do everything. Now I have everything I’m making money with everything and I can’t delete it all. And we allow people to like, some teams are using MuleSoft, this, this person using J boss and all the features.

And so they went so far in. That switching means rewriting code, and most people are not trying to re-write code that already works. So I think is it’s not about necessarily convincing. It’s more about what’s available. What is this Kubernetes thing? Is this a hugely for right? Remember people ignore [00:45:00] mobile for years, like five years.

Some companies like, ah, this is a mobile is a fad. We don’t have nothing to think about. We don’t worry about now. There’s some people that say, look, if you don’t have a mobile app, I can’t, I can’t deal with a coming no mobile app. I’m not going to buy an airline ticket using my browser. You kidding me?

You’re going to have mobile app. You mean you’re a bank and I can’t check my balance using my phone. This is not a real bank. So what executives are trying to figure out is Kubernetes at that point is Kubernetes a piece of technology that if they don’t get on board, they’re just going to get left behind.

Machine learning also worries them the same way. You mean to tell me that , my competitors are going to be making decisions, using different techniques and I might get left behind. So I think a lot of executives are saying, I’m hearing all this buzz. I need to know what the reality is. Is this going to give someone else a strategic advantage than I will ever have?

And the truth may be yes, in some cases, right? You may have some retailers who have revamped their whole e-commerce presence. They may have taken their best engineers and have them working on same day [00:46:00] delivery and not building some platform from scratch. And if that team tells you, you go meet that executives like, Oh yeah, my team doesn’t mess around with that layer anymore.

We’re just using Kubernetes. Prometheus we don’t have that discussion. We’re done. We’re just now building new things, growing revenue. Then you’re going to be asking questions and saying, Hey, why are you all still doing chef automation, writing scripts? Like I don’t, I don’t hear the value in that. So what I tend to hear from an executive is I just want to know.

What’s the state of the art, and then they can make an informed decision and sometimes the state-of-the-art is worth rewriting. Everything. There’s going to be cases where you say, wow, that would be a strategic advantage. And I’ll give you one example. There are people who do WordPress hosting, right? And word press hosting.

Typically was very efficient, right? They don’t do just VMs. They do like VBS, right? Like very small virtualized platforms where they try to restrict everything just to run WordPress. But even then they’re thinking like maybe they can do more features and get even more density by using Kubernetes and containerization.

And I’ve seen people [00:47:00] rebuild custom proxies in go to actually do a better job of multi-tendency than what Apache does out of the box for WordPress. So they said, wow, if we lose Kubernetes in a certain way, we can increase our density X and have a common API where we’re not dealing with low level things anymore.

So a lot of those providers have already started to rewrite some of their software because they saw the ability to leverage Kubernetes, to make a material difference in their business. Everyone doesn’t have that kind of outcome, but that’s the question that they’re asking

Ashish Rajan: great answer. I want to add something in that as well.

I don’t know keen to know your thoughts on this. I kind of, in my mental model, I put businesses articulate a feature, to convince them in two buckets. One is either they’re making money or they’re saving money. And I know we’re trying to relate to either of these objectives and it’s like almost irrespective of technology or what, I mean, put anything in that bucket.

They, everyone understands. It’s like, I even at personal finance, if [00:48:00] you’re telling me, Hey, you can Switch to this insurance and it’s remarkably cheaper. I will totally sign up like, I don’t need to know what they do in the background. That’s worked for me. I don’t know if something that you see if other others use it as well.

Kelsey Hightower : Well, I think most of the enterprises I deal with would be in the category of, I’d rather spend money if I can make more money by doing so, when you get to the point where you’re trying to save money, you have a different set of problems. So you’re right. There’s nothing wrong with saving money. Yeah.

Typically when a business is hyper focused on saving money, that means they’re in, they’re out of ideas in many ways. Yeah. You know what I mean? Because if you said that most thing you can do for me to save money, then now you’re like, wow, you have no idea how to get new revenue. Because most times what you’ll do is you’ll, overspend.

As long as you know that it’s going to grow you into a new revenue target that when that is all exhausted, Well, I think about blockbuster in the final days. Oh right. You’re thinking about consolidating stores. You’re thinking about like a Vinny machine outside of seven [00:49:00] 11, all to save money. And the truth is maybe the question should be is how do we turn this into what Netflix is today?

It means we got to spend a little bit more money to be able to distribute because it wasn’t cheap in the beginning for Netflix to get off the ground. So it’s typically a different conversation because saving money really is a different conversation. It’s more like we’re spending this, how do you get me to spend this?

And it’s like, well, maybe if you just use what you have a little bit better, it’ll be cheaper right now. But sometimes even just to go from VMs to Kubernetes, where in two years you’ll get a 30% saving because of Ben packing. But this year you need to double your costs.

Ashish Rajan: I love it. I love it. Now every time I see a company and if if they talk about, Oh, you’re trying to save money, I’m going to think about this conversation, but not, I appreciate that. . So for guess for people who are listening in, where can they, I guess, find you, where do you normally hang to? I guess, interact more.

Kelsey Hightower : Yeah, I pretty much completely on Twitter these days. So at Kelsey [00:50:00] Hightower, my DMS are open. I try to respond to everybody and sometimes I’ll make time for like one-on-one sessions like this as well.

Ashish Rajan: Awesome. I appreciate that man. And and I definitely want to bring you in because well, let’s just say the teaser is we’re trying to do something in I think may or June, I’ll let my producer confirm it, but we’re doing a month dedicated for Kubernetes coming up soon. So yeah, I’ll definitely bring you in for that part as well.

So thank you so much for coming in. I really enjoyed our conversation and I can’t wait to have you back again, man. This is really good. Awesome. All right, man later.