Katie Knowles: [00:00:00] There's been a lot of what they call LLM jacking going around recently. So there were some really good reports that have come out in the last year. If you look up LLM jacking, you'll find them where this is just basically another type of resource theft. If a resource exists and it's useful to people, somebody will want to steal that compute power from you.

And in this case, if they have the token, so like your authentication token for a model and access to that API model, the URL that it's. Living at which will be where it was deployed to when you created the resource, how you interact with it, then they can an attacker can basically slap this into a large infrastructure.

They have that creates a back end pool of. AI resources that they can send queries to so maybe

Ashish Rajan: Learning about azure security, especially incident response in azure is something that you're interested in. This is the episode for you I had katie knows from data. She's a security researcher in the azure space And she used to be an instant response person for azure So I thought she's probably the best person to talk about instant response in azure considering that's not a topic that is spoken about enough I thought this would be the [00:01:00] perfect topic for all of us to get an understanding of what it's like to have an incident and respond to an incident in Azure.

What are some of the things that probably should be taken care of? What are some of the fundamentals that you can overlap between Azure and what you probably already know from AWS or from on premise world? So overall, I think it's a great conversation. If you know someone who's trying to transition into the world where Maybe like me, you have just recently been part of a merger acquisition with a new company is working in the Azure land and you're trying to figure out how do I do instant response here?

What are some of the things that I need to know from a security perspective? This is definitely a great conversation for someone like that. So if you know someone who probably is looking into this, definitely share this episode with them. And as always, if you are listening or watching the episode of Cloud Security Podcast for a second or third time.

I would really appreciate if you can support us by hitting the follow subscribe button on the audio platforms like Apple or Spotify, or if you're watching some videos, YouTube and LinkedIn, please consider subscribing. That definitely helps more people know about us as well. Thank you so much for supporting us.

By the way, if you're attending DisObey, which is a security conference in the [00:02:00] Nordics, I will be there with Cloud Security Podcast. Definitely come and say hello. But if you are not, that's okay as well. I'll still bring insights for what it is to be working in cloud security in the European region.

All right. I hope you enjoy this episode. I'll talk to you soon. Hello. Welcome to another episode of Cloud Security Podcast. Today, I have Katie Knowles. Thanks for coming in, Katie.

Katie Knowles: Thanks for having me. Excited to be here.

Ashish Rajan: I am excited to have you as well about, because I haven't spoken about Azure for at least what feels like a while.

So today's conversation is very important from that perspective for me personally. But before we get into it, could you share a bit about yourself and your professional journey?

Katie Knowles: Yeah, absolutely. So I've done a little bit of everything, but I like it that way. I started out studying electrical engineering, realized that wasn't the most exciting career path for how many different things I like to study, all the different topics I love to get into.

So I ended up in security, thankfully. From there I've been a security engineer network and web app pen tester. Then I went over and did some incident response, which brought me to cloud along with threat hunting and detection engineering. And now I'm a security researcher at Datadog.

Ashish Rajan: [00:03:00] And instant response is what I would doubly click on today.

And this, at least in this conversation with them, many more to come on some of the other topics you touched on as well. Maybe just to give a lay of the land, because I think the way I've seen myself, I've come from that on premise background, did AWS first, did a bit of Azure and GCP.

But I've not done a lot of instant response, but To lay the ground, I guess even to lay a foundation for people who are probably coming from their on premise world. How different is, when I was studying for Ofsec or OSCP, whatever that is, that certification is still there. I don't know if it's still there, OSCP, but at least it used to be the cool thing was to get a domain admin.

In the Windows area and Okay. If you're domain admin in a Windows domain, like you're like guard level, what is that in Azure land? What's kinda like the lay of the land people care about in the Azure land when it comes to an incident?

Katie Knowles: Yeah. And it's interesting you mentioned the lay of the land from a domain admin perspective because it's still relevant.

O-S-C-P-I think is still relevant too, but the domain admin part [00:04:00] especially, there's a lot of attacks that can be used from on-prem, especially for the active directory Microsoft side to get to Azure. And there are equivalents as well. So pretty similarly to on premises. If you have global admin, you'll have access to all of the Azure resources.

You have a way to escalate into Azure permissions from your Azure Active Directory or Entra ID permissions. I might use both terms interchangeably. Entra is the new word. Azure ID is the old word, but people still say it. And it also has that relation where both are the core identity product. So there's a lot of cross between the two, a lot of what they call hybrid environments or hybrid deployments.

If I'm a big enterprise, I might've had active directory for 10, 20 years. Now I'm moving into Azure. There's ways to go back and forth between those two so that if I get permissions in either of those spaces that are significant it could be that I can move to on premises hosts or. From on premises hosts into the cloud and compromise cloud resources as well.

So the blast radius with [00:05:00] Azure is a lot more, I don't want to seem too grand doys here, but it's a little bit more significant than maybe AWS or even Google in some environments.

Ashish Rajan: Or because AWS and Google have their own identity thing that they're looking after, whereas Active Directory may be a single sign on or a SAML connection or whatever to the ADFS that may be.

Is it, is the relationship similar to, at least from what I remember, it used to be around group policies for, hey, Ashish has certain group policies or he or she gets access to Azure and certain services within Azure. But then you're saying the global admin and the role admin as an, as a, it's not just my on premise, but also my Azure one can translate back into my local on premise permissions as well.

Yeah, there's going to

Katie Knowles: be a lot of really complicated ways to do this. So let's maybe think about, we have an environment where. Let's say that we have Active Directory for on premises and we manage our devices using Intune. Now an attacker comes [00:06:00] in and gets access to our Azure environment, is able to escalate permissions in some way.

They might be able to manage those Intune devices and use a script to pop back onto the hosts and then start compromising the domain. Again, very generalized, but there's a lot of scenarios like that for both directions. Yeah, and then a lot of permissions within different applications, just like on premises, right?

When we talk about Active Directory hacking these days, it's getting into certificate services and exchange and all kinds of crazy backdoors and D. L. L's not obviously the deals don't apply in Azure, but there's similarly lots of different services. If I'm an exchange administrator, what can I do with that?

If I have application permissions to read a user's mail, what can I do with that? And I won't get too deep into that yet, but the types of permissions available are a little bit different than what's on premise as well. Mhm.

Ashish Rajan: Wow. Wow. The funny thing is Katie, I was thinking what happened to AWS land S3 bucket over to the internet.

You might say, what blob storage is my domain admin. Didn't realize there was a lot more depth to all this Azure land. Are those things. [00:07:00] I guess maybe is the whole and maybe to generalize the whole cloud security space a bit. A lot of the conversation always comes down to, Hey, you have an misconfigured resource accessible to the internet, probably after identity.

That's like the number two reason for why cloud security incidents happen in most scenarios. Is that still relevant for Azure or, cause it sounds like identity is already quite complex. Before we even jump onto local identity and all of that. So is that still things that happen? Or is that a common news in Azure world?

Katie Knowles: Yeah, public exposure is still a huge issue. I'd say that the context of what's exposed matters a lot. So if I've got a virtual machine that's exposed to the internet and I've just been testing with it. Which can happen quite a lot with Azure. There's a lot of MSDN, so developer network, it used to be called.

Now it's like a visual studio subscription. And Microsoft loves setting up a business to hand these out like candy and their main Azure tenant. And there's nothing wrong with a little experimentation, but if you get a certain number of users who've created demo [00:08:00] environments, maybe the demo environments got some sensitive data.

Or, they have just left it exposed and now. Even their free credits still not the greatest to get those crypto mind out on the company dime in some way. So a lot of that will happen in the context of what's exposed will be very important. If they're using the demo environment with, sensitive credentials or a user session tied to their Azure account, like they log into the Azure CLI on that VM, then that token could be used to pretty much access anything within.

Azure land you can use those tokens to exchange for a lot of different applications. It's called the family of client identities. Really good Azure, I can probably use that with, exchange or with Azure are back versus so the Azure resources versus ontra ID. Those are going to take different tokens, but I can exchange them back and forth.

So I've got lots of interconnectivity. I can work with.

Ashish Rajan: Interesting. And do you, and maybe just to double click on [00:09:00] the identity a bit more as well, because obviously we mentioned Active Directory, Entro ID as well. And a lot of enterprise obviously have been using, or even a lot of companies have used Active Directory for a long time.

That's where a lot of us started our careers for lack of a better word. If you're in, if you are in a tech company back in the day, you definitely have heard of Active Directory. Either for right reasons or wrong reasons, but I would enter ID. What's the identity interaction between resources? And I guess, is there more?

I know there's a whole I guess local users in AWS and local there's constantly called AWS IAM users, which is like, Oh, it's not just your Active Directory users. You have local users in Azure. What does the identity landscape look like in Azure world? Obviously you can't touch on the whole Active Directory users, which are your corporate users, but are there any specific in Azure as well, which may be relevant from an incident response perspective, because I imagine that's still number one in terms of reasons for why there's an incident in the first place.

Katie Knowles: Yeah, so there's going to be the user identities, I guess that we've been talking about. There's [00:10:00] also going to be service identities. So what would have been a service account in on premises? It's going to be a service principle, and those can cover a lot of different things. They're going to be paired with an application registration that can allow.

This is one of those cloud things right now. We're getting sassy in terms of interconnectivity. Maybe Adobe. Needs access to my environment to provide their super legit PDF services. Now they have an application registration that can be, they can add a credential to and use that app reg to authenticate to our environment.

The local service principle in our environment will perform actions for them. But basically they've got a backdoor with some kind of permissions that we've given to that account. So that can be really scary and very complicated. The ways that those are built out or managed identities can be associated to virtual machines.

They're another type of service principle and something that can be tricky from the research side that I deal with a lot these days is that. Microsoft will try to obfuscate the complexity of these different identities. So they'll say, oh, it's a managed identity. It's different than an application.

Okay, [00:11:00] it's still a service principle, and that's tied to an application registration. But if I search that, it's not in my tenant. So where is it? We don't need to know that. We're users. It's a managed identity. Don't worry too much. Versus an application. Which will be the service principle, or they'll call service principles, enterprise applications.

There's a lot of wording to look out for in that sense, but they'll also be resource specific permissions to sometimes. I know you mentioned with AWS, like local users, so maybe you've got kubernetes set up with like local users. Local are back, can set it up with Azure, but you can set it up locally as well.

And that will create more situations where an exposed resource may not be. As easy to control as you think it is by just using Entra ID, if not, everything is using that as the identity store,

Ashish Rajan: because you search your point then, because now we are in territory of I think we're talking about this from a user devices and work for identity perspective like the three broad categories.

but then where it [00:12:00] gets a bit tricky to your point, if you pull that back to native Kubernetes kind of resources, where now in this land of cloud native, there's like a data center within a data center. You think you have access. You don't really have access until you actually tried looking into it.

In terms of, the way I see it, RBAC. It's usually people from an on premise world and putting that hat back on. We usually believe that, Hey, you know what, Azure policy, I make sure that Ashish only has access to exchange. It only has access to, I don't know, SharePoint or perhaps even Azure portal for lack of better, and that's probably like the extent of it.

Is there something for us to understand from a perspective of third party as well? For example, I may have a CSPM or CNAP. How does those kinds of identities interact in Azure?

Katie Knowles: So there might be a, I think we might be talking about two contexts here. One would be the permissions that application has in our environment.

The other would be the permissions that it has through us. So [00:13:00] application versus delegated permissions ends up being a big thing in the Azure space with applications. The. An application can have permissions within Azure. So app roles would be the API word for it as opposed to the friendly word that give it the ability to do something like read our whole directory or modify our users.

If it's maybe we have a identity management solution that we want to use along with Azure. Or Entra ID, I should say. And in that case, the identity itself will have the ability to perform tasks without a user. But then we'll have something else called application permissions. And this is where we start getting into the type of OAuth illicit consent grants you might see or phishing.

Somebody says, hi, I'm SharePoint. Click this link, grant me access to see all your documents. And that's a little bit more, the user has the ability to consent to an app. And grant them the ability to perform certain actions on the user's behalf. So there needs to be that interactive component of a user saying, Hey, yeah, you can do this as me.

And then the app needs to perform a [00:14:00] impersonation flow basically to say, Hey, I've got, this token from the user. I'd like to exchange it for a token to do things as the user. Can you give that to me? And once they get that, you can have the. Point of, somebody reading your emails, modifying your files things like that to the point that they've introduced features now, where in Entra ID, you can, as an administrator set up certain high risk permissions that a user can't consent to without you approving first.

So that's different than me assigning permissions to the application as an admin, though, if I assign permissions at permissions to the application, it can do things without any user in the mix. But if I require my approval. A user can create a request in Entra ID and say, Hey, I'd like to grant super legit mail access to my calendars.

Is that okay? And you can say, no, that's not approved. That sounds very shadowy to me.

Ashish Rajan: Interesting. And I think there's a similar concept in AWS as well, because they call it instance profile roles or instance roles or whatever. But I think I was thinking more from a [00:15:00] perspective of, because these are important.

Is there a third layer for for example, I don't know, just think of a any product, performance management product or cloud security product that has to talk to Azure to find out, Hey, how many blob storages are exposed on the internet is that going to be used is going to, is that going to borrow my permission?

Or is that going to be a managed identity? Or would that be a, Service principle, because I, the reason I say that is because at least in the AWS land or even GCP for sometimes as well, you have the lay of the land from a, Hey now that I know the kind of identity, I've got local active directory. And then I take a step back, okay, now my workload itself can have an identity, which is what you would have framed as well with the app role, I think you called it.

And so there is that as well. The other thing that we will also talk about is the third, quote unquote, third party. Like for example, tomorrow, I don't know, some third party I've been using Okta I don't know, I don't want to name products. I'm like, if I. Octa doesn't have a breach right now before [00:16:00] people think there's a breach.

Like I'm just thinking about all these other products that we integrate into a cloud world, which may not be normal in an on premise context. Like in an on premise, I'm actually literally installing a software going next on a server. I know that's my Apache box. Every time I go to it, that's my web server.

But in the context of cloud, there's a lot of third party services that help a. Security person do security, whether it's your, how many blob storages across my entire subscription tenant or whatever is, how do they interact and kind of work in the Azure world.

Katie Knowles: Yeah. So in that case, you'd need to understand the scope of the permissions.

I've been talking a lot about the Entra permissions. The app role would be usually things that are being assigned within Entra, but you can also have permissions assigned in the context of other applications that offer their own app roles you can assign. So permissions that. I give an identity to perform in Microsoft Graph is usually how it works for Entra ID.

Most of the access to the data and the data modification in Entra is through Microsoft Graph [00:17:00] permissions. If it's in Azure though, specifically Azure, talking about, can I see all of the resources or all of the blobs associated with something? That can be a couple different ways. I don't know off the top of my head if there's app roles specific to Azure.

Permissions, but what comes to top of mind is Azure are back. So something to know is that those service identities can still have the same role assignments that a human identity can have. So I might have global admin. A service principal could have global admin. Those are on trip permissions. A service principal could also have something like.

Reader. So reader would be an Azure RBAC permission that lets me see everything in the scope of it. So like the subscription, I can see all the resources now. I can also assign that to a service identity just like I can a human. They just get extra special app roles as well to make things complicated.

Ashish Rajan: Yeah, I think I feel like there's a lot of Entra ID information that need people need to learn to understand the lay of the land from an incident perspective. I think this has been already going in a good direction [00:18:00] because. A, I'm able to draw some parallels between something that I already know with AWS and on premise, but also understand the complexity of there's a few more types of identity more than just your Active Directory domain account.

Or there's a global admin, the service principal, the users as well. There's AppSec, there are app roles as well. There are managed identity as well. In terms of, I guess the next component that I normally think about, and my hope is at least people who are listening or watching to this. They would be able to at least walk away drawing similar paddles between different parts of tech as we care about.

And we spoke about identity. The next obvious one that I come to mind is networking in terms of, how people have done networking in the on premise world, as long as two, two servers on the same box, sorry, as long as two servers are in the same network, they talk to each other. As long as the subnet is same and IP at the CIDR end or whatever is the same, they can talk to each other.

Now, obviously AWS and Google cloud have their own way of doing. networking on how that works. If I were to put that kind [00:19:00] of same networking question on the Azure land maybe first from a, what happens inside an Azure virtual network and then extend that to the on premise world. How's that usually I guess lay the land from that perspective.

If people are thinking about from an incident response perspective what's some of the things people should understand about the Azure networking?

Katie Knowles: Yeah, I think so. It's important to understand. And from the classical networking sense, the infrastructure networking for Azure is going to feel pretty similar.

There's going to be a network server. Sorry, network service security groups. There we go. NSG is I'm so used to the acronym. I forget the expansion. That's the Microsoft world for you. So you've got NSGs for handling network traffic. You've got your V nets and you can set up routing between them separately.

Things on the V nets can talk to each other. It's all pretty familiar. Yeah. Something to keep in mind is that they'll still be the scenarios you're used to seeing on premises. There might be a load balancer for an application. There might be a separate firewall configuration or different types of.

Networking [00:20:00] groups that you need to pay attention to. They have like application security groups, which do routing and more the application layer and then network security groups, which do the network routing. There might be a WAF involved. And if you're used to the on premises. Configuration of things.

I remember when I started getting involved with cloud. I was so upset because I'd had the opportunity to work with some Palo Alto's early in my career and the concept that you could control everything going out of the company was beautiful, right? Every request had to come through us. Yeah, we would try to support everyone to the best of our abilities, but we know we could see the config, you could access that and with Azure, because now all these little resources are there, they're like micro networking resources, and they're all split up around subscriptions and resource groups.

So figuring out how they map together can be really complicated. There's also going to be resources that are, look, nothing like what you're used to from on premises. Things like Platform as a Service, so Azure's database services, storage things like that. They're going to have custom networking rules, so they don't really have the idea of a [00:21:00] VNet.

You can use a private endpoint to change how things route with it, but they're still going to pay attention to their own Little networking rules that are going to be a subset of a special property of an object that you need to look for and know that is where the networking rules are kept for that type of resource.

So you might want to spend some time looking into just storage and some time looking into just like the. Databases or a certain type of database, or just how networking is configured in AI services. If you're starting to use AI in Azure and you have a deployment, where is the networking for that object set and how can you take a better look at that?

So it's going to be this constant evolution. I think if anybody's been listening and they're thinking, oh, my gosh this person's just telling me so many things in so many directions about Azure and I'm afraid and this doesn't sound fun. I would encourage you to at least try a little bit because learning little bit by bit in Azure is the way it's always gonna work.

There's still [00:22:00] stuff that I get reminded by the experts, no, Katie, this thing does exist or no. Actually, this feature has been out longer than you thought. It's been a risk longer than you thought. This is as a researcher, I've it. All I do is read Azure news right now. I'm telling you, I'm not quite crying about this every day, but all I do is learn about Azure day after day.

There's always something more to know, but it doesn't make it a hopeless quest, right? If you can be that subject matter expert for your company, this stuff is really complicated. And if you can be the person who's yeah, I know how to search the Azure portal. I'm not afraid. To touch that specific web UI, because people get allergic to it.

They're really terrified. If you're not afraid to interact with that specific portal, you're already a benefit to your company. I'm guessing if it's anything like where I've been before, nothing against any place I've been, just, there's so many clouds, so many things to do and somebody who's willing to really just start tinkering with cloud and looking into it will make so much of a difference to any environment.

Ashish Rajan: I think the networking one is definitely one of [00:23:00] those ones where it, to, to your point, when a lot of people move from on premises to cloud world, they may look at that and going, this does not make sense at all. Like to your point about the panels firewall, whether it's checkpoint firewall, parallel firewall, a lot of us have gone through our own versions of firewalls in our life, whatever we got access to first.

And we definitely, I guess there's definitely a difference in terms of how. Things are done in cloud because there's a whole, I think you did the whole you made a video on the was it a query that we, you could use for cloud security assessment? Was it?

Katie Knowles: Yeah, it's the resource graph query.

Resource

Ashish Rajan: graph, that's the one. I was like trying to think of a name for it. Yeah, but so for, I don't know, cloud security, you did a resource graph on how that could be a great starting point to even start talking about what is the baseline for security in Azure. And we just touched on identity and networking as the two components that maybe you don't have to start there.

You start from the net, the resource graph part, and maybe hopefully that kind of helps you identify. Oh, okay. [00:24:00] Before I start building a whole incident response plan, let's just see what is wrong with my environment in the first place to begin with, which kind of makes me also think, what are some of the common incidents in Azure that you probably have seen?

That maybe people can use as a starting point.

Katie Knowles: Yeah, so we talked a lot about exposure to the public Internet. You'll probably see a lot of that if you're in an environment where you're positioned on the security team, either like the. Security engineering. I was going to say blue team or incident response, but they're both blue team.

So either if you're trying to decide design more secure infrastructure, or if you're investigating compromises, you'll probably see a lot of, hey, something's been, hit with suspicious traffic. Microsoft might email you and say we're going to turn this off. If you don't stop your nonsense.

And you're like, what nonsense we're not, what? And then you look and it's nonsense. There might be some database exposures especially understanding how your company's big applications work can be very helpful or what experimental projects are going on. You might get alerts around crypto mining and maybe somebody's looking into it [00:25:00] legitimately.

So getting used to asking that context about different incidents is really helpful. Some other, I think the incidents are basically benign or we don't talk about that due to, the sensitivity of our employers. There can be, it can be a little tricky to find an in between. I will say that things that are really helpful to understand a bit more would be those OAuth illicit consent grants.

So it's very popular right now as an alternative to phishing. If I get somebody to click yes on my application and they're adding more protections around this in Microsoft, where you need to be either a verified publisher, which isn't that hard, but it's at least a bar, or you can only request certain permissions, maybe based on your set organizational settings.

Go understand what are those settings for your organization get the permissions you need to see those settings for your organization and to see the resources. I think you mentioned the resource graph query as well. I love that panel so much. So everything you have reader over in an Azure environment, you can query.[00:26:00]

Using that. So it's different from Microsoft Graph which is related to Entrez ID. It's different than the Azure AD Graph which is another different graph. This is the Resource Graph Explorer. There are certain words Microsoft loves. Graph is one of them. None of them are graph API like the graph APIs.

Yeah, it's just a, it's a graph. Call it graph. And in this case. Resource graph means a database of all of your resources that you can query like that database using KQL, Microsoft's query language. So there's a lot of guides on how to query it. I've got some example queries in the repository that I put up for Advent of Cloud.

So I definitely recommend playing around with those. There's Terraform code to spin up some resources too. Super easy if you've never played with Terraform before. There are no. Previous experience required on. Then you can see just how easy it is to say, okay, what public IPS do I have?

What, databases do I have? And how are they configured? Can they even access the public Internet? How is that set up? And then a little bit more complex. You [00:27:00] can chuck a big block of text. Microsoft provides to map IPS back to VMS, which is really helpful because the IP services. Public IP is a resource.

The network interface is a resource. VM is a resource. We need to associate these resources together just to understand if the computer is on the Internet. So it can be very helpful to get some baseline queries like that down as well.

Ashish Rajan: And would you say because we also because we live in the world of AI specific one that comes to mind as well for any of these common incidents in Azure?

Katie Knowles: Yeah, so I'm really glad you asked that actually. I have not been able to look into this too much yet. But there's been a lot of what they call LLM jacking going around recently. So there were some really good reports that have come out in the last year. If you look up LLM jacking, you'll find them where this is just basically another type of resource theft.

If a resource exists and it's useful to people, somebody will want to steal that compute power from you. And in this case, if they have the. Token. So like your authentication token for a model and access to that API model, [00:28:00] the URL that it's living at which will be where it was deployed to when you created the resource, how you interact with it, then they can an attacker can basically slap this into a large infrastructure.

They have that creates a back end pool of. AI resources that they can send queries to. So maybe I want to offer a cool AI service to my friends for nothing. Maybe I want to generate content that other engines won't let me. I'm just going to take a bunch of other people's AI models and chuck those into the background of this, and then.

I'm going to load balance between all those resources, and they've got great logs for how this has been investigated in AWS. Really interesting write up. Scary write up, but really interesting write up. But then for the Azure side there's code referencing that AI, Azure's AI models. AI models hosted in Azure, I should say, can be used in this way.

But I think there's been a little bit less done on the hunting front and the logging for that service specifically can be a little bit tricky to set up and get the depth that you would need compared to [00:29:00] AWS. So I would look out for it. If you've got a notification related to it, I would look into it certainly.

But there's been a little bit less specific research though. I would say all signs point to it's happening.

Ashish Rajan: Interesting. And maybe just on a similar note, anything on the whole privilege escalation as well? You had a talk at 4CloudSec as well about the whole privilege escalation. Any examples of privilege escalation that you could share as well?

Katie Knowles: Yeah, so I had a blog on privilege escalation. I had a talk on persistence. One day I hope to have a talk on privilege escalation but we'll see how that goes. So the privilege escalation that I published on a couple of months ago was around Key Vault. So going back to identity is a little bit, Key Vault can be configured to use Azure to allow a user to do things like view secrets, manage secrets, things like that.

And there's a couple of roles. The permission related to this role has been talked about quite a bit before. But the risk of the role wasn't very well [00:30:00] documented. So if a user has the ability to modify key vaults, they have the ability to modify access policies on key vaults. What are access policies?

They're another Alternative form of authentication will not or authorization. I should say authorization. Where if you're not using, if you choose not to use Azure RBAC for your key vault, you can use one of the two. If you don't use it, your key vault will use access policies and access policies are just saying this identity can read, write, edit secrets, certificates, whatever.

It's just a different model. But if you have a role that says you can't read Azure RBAC. Or, there was some particularly tricky wording, like the Azure RBAC. I can't remember it, but it was basically saying that you have no, no RBAC secrets access to Key Vault, but you can have access policy secrets access.

If the Key Vault is using access policies, if it's configured that way, you can modify an access policy, put yourself on it and access and modify all the secrets. So because of the way this is worded, it's not a vulnerability. It's two different [00:31:00] conflicting models of permissions, right? One allows you to, manage access policies.

And if you don't use access policies, that's not a problem. It won't be, it won't be helpful. But if you do use access policies, then you can use it to access secrets. So just being very careful on the wording, right? You can feel sometimes like you're in a lot of. You just dug deep into the riddles sometimes was trying to figure this out.

No, our back secrets access. Does that mean no secrets access? What does that mean? How can we articulate what that means better? And there's now a nice big warning page. The box is about this big on that role, which I'm very happy that they. Put in place, I reached out to Microsoft to MSRC about this the documentation was updated and then I got a response that because the document said the risk, it wasn't an issue, which, okay, all right.

Sure. It's on the page now. That's great. I like that.

Ashish Rajan: And what about the persistence that you were referring to as well? Yeah,

Katie Knowles: so the [00:32:00] persistence was around administrative units if we want to talk about that a bit.

Ashish Rajan: Yeah,

Katie Knowles: administrative units were introduced to allow scope of Entra ID role assignment.

So before, if I get help desk administrator. That's everybody. I can help desk everybody. You call me up. I'll reset your password. And they've changed the role, right? So I can't, really reset privileged roles anymore. Or I, if I'm a user authentication administrator or user, there's basically if I can manage user authentication.

It's going to be everybody. So an administrative unit was introduced to finally allow scoping of that. Maybe I only need to manage users who are in Boston. Because I am just the Boston IT person. I really don't need to reset everybody else in the company. So administrative units allow you to provide that scope.

I put a bunch of users into that administrative unit. And then I say, okay, Katie can reset passwords for these people. But I can't reset passwords for anyone else. And then there's this opposite property of them called a restricted management administrative [00:33:00] unit, which is newer which kind of is the opposite.

Logically, it takes a group of people. And unless someone has a role assignment over that group of people, specifically, no one can manage them. So if I take, CEO see and, our favorite. A high profile blogger who might be subject to attacks. I put them in an administrative unit and that's restricted.

And now I give we'll say Bob the access to manage their accounts. Bob can manage their accounts, but he's the executive it guy now. Nobody else can manage those accounts. So these are really great features. There might be ways attackers could use them in ways we're not looking for. Obviously I don't know what an administrative unit is.

If I don't know that. And then an attacker puts a bunch of users in a restricted administrative unit to protect them. I'm going to get really confused when I can't reset their passwords. My scripts might fail for containment. I might not notice. I might not know how to investigate it. There's another property called a hidden membership that wasn't super clearly documented, but basically.

Unless you had a [00:34:00] very privileged role, you couldn't see the membership of an administrative unit. So I could create a situation, these scenarios need super high privileges, I should specify. I would need to be global admin, privileged role admin as an attacker to perform them. But I could hide, the scope of a role assignment.

I give to myself, maybe I give myself a scoped role assignment over nothing to see here, the administrative unit. And then I put the CEO in there who, if he's not in a restricted administrative unit already, and then nobody can really investigate that. Maybe our, analysts don't have permissions to see into administrative units.

And then to clean that up, you also have to go to someone with very high privileges, like a global admin. I don't know about you. It's really hard to get their time. I've found so that can be pretty tricky as well. So I definitely recommend looking out for scoped role assignments related to that. And there's a couple other detections in the article that we put up around this.

Ashish Rajan: I'll put the link at the show notes and the description as well for this cause I think having this conversation is also making me think like now that we understand, okay [00:35:00] at least for people from a different background, they'll all understand the identity pieces, the networking pieces, probably they have, or they would have heard of examples already of the kind of scenarios people can see as well.

If you were to start building this and starting off from maybe just a simple playbook, usually a lot of people come up talk about the whole, Hey this is this is also the land when people go into the land of E5 licenses, E7 licenses, I have this security, that security, and it's there's every, there's just in time access and without adding all that complexity into this, that's just super simple scenario of if I wanted to just make a playbook, to your point, I saw a resource graph Explorer, and I saw a bunch of things.

I'm like, what would be level zero, for lack of a better word, for a playbook that I can start establishing today if I wanted to just, play, to your point. I come from an AWS land or GCP land or from an on premise land. I'm trying to just build my first playbook because my company acquired another company that is Azure and the cloud [00:36:00] security person for Azure.

Katie Knowles: Congrats. If so yeah, I would say, without taking into account, I'm glad you mentioned scheduled and just in time access, we didn't even get to touch on that, but that's a whole other can of worms, but. I'd start making sure that you've got the access and the logs that you need and that'll take you down some rabbit holes, honestly.

So if you use Splunk, they have a guide. For like tips and tricks on getting Microsoft data into Splunk, that was very helpful to me when I started looking into this. The logs, where they're stored and how to forward them can be different for different services. You likely won't have logs yet on things like who accessed secrets in a key vault who accessed or changed storage in a storage account.

For those you need diagnostic or resource logs. Those are gonna be resource specific. They're gonna be costly to implement. So start those battles now for your key resources, either key resource types or key projects. Making sure that you've got all levels of your Azure resources. Your Azure resource logs at [00:37:00] the control plane level.

So resources that were created and deleted resource group and subscription type stuff. That's going to be different than getting your Entra ID or Azure ID logs forwarded. So making sure that you're tackling both of those and then looking into the diagnostic or resource logs. You're going to need to make sure your team has access as well.

So getting people reader access on everything, if possible, is very helpful within Azure RBAC. But again, you're going to need a different permission within Entra ID as well to make sure that you can be a directory reader. You can use the Azure portal in some environments that's restricted or the Entra ID portals restricted.

And as you do this, write out some guides for people because you don't want to be the first. Maybe you're always going to be, the first person who gets called for this, but the more you can train people, the better. Especially things around like writing down what GUID corresponds to what there's going to be these long strings that represent object IDs and which object ID relates to what.[00:38:00]

Part of an identity can be really complicated. If I'm a user, I'll usually be referred to by my user principal name, which is going to be like Katie at tenant dot on microsoft. com. But if I'm using a service principal that could have an, a different identifier for the service principal object, the application registrations, ID, the application registrations, objects something else I'm forgetting and just getting all of that straight so that, whoever is looking at this ticket first doesn't completely lose their mind over this can be quite helpful. Which logs go to what resources? I'll say there's been some really good publishing. I saw an article just today from Invictus IR on roles you might need. It's pretty high level, but it's a good place to start.

They've got a log extraction tool as well, if you don't have logging set up already for M365 at least, which will get you your Entrez ID logs too. Microsoft has some IR playbooks and you should have all these links so we can share them. I put them down, but I found out Microsoft has some IR playbooks you can follow as well that are a little bit [00:39:00] more general, easy, easing into somebody got phished with an OAuth app.

Where do we go from there? And, maybe you thought you were going to be the cloud resources responder. You were going to respond to virtual machines and LLM jacking and really interesting cloud stuff. And now you get called every time something happens in Microsoft 365 or SharePoint is exposed to the Internet.

It's a slippery slope. They've got some playbooks. You can look at there. That might be helpful.

Ashish Rajan: It's an interesting one because I think that's where because there's obviously a shift happening in the cloud security landscape where a lot of the cloud security, I want to say logging and alerting for security is being sent over directly to SOC teams, not being, there's no middle person who's a Azure cloud security expert or AWS cloud security expert.

It's going straight from your resource graph. Or explore all the way to your Splunk or Slim or whatever the theme provider you have in front of a SOC operator who probably has, maybe has some idea about Azure, [00:40:00] maybe not some Azure ideas. Are we saying that because this Azure portal thing is this new way of doing cloud, at least in the Microsoft land, does M365 SharePoint is that obviously to your point, you need that because so you get the entire lay of the Microsoft landscape, or is that because how everything is connected in Azure? I guess where I'm going with this is that if I have just the logs for virtual machines. If I just have, I don't know, some kind of a reader access, I just say would I still need to, or does that, does it tend to happen that there are more issues usually on M365 compared to Azure or where should I focus my energy on?

Because I feel like there's a lot, because once you go into that world of Microsoft now, only to your point earlier, there's tenants, there's subscription, depending on how many subscription, how many tenants. And then, oh, I'm looking after M 3 6 5 and SharePoint and Azure AI Studio, and something else comes.

It [00:41:00] feels like the pod keeps growing. What's a good baseline to start building this? Cause I feel like a lot of people like me maybe feeling like what I am like, Oh shit, there's a lot to kind of start. What's my, what's a good level zero that I can start with in terms of research?

Yeah,

Katie Knowles: it is a ton. For me, I try to remind myself on the days that I feel like, Oh my gosh, I've been at this years and I still feel like an idiot. If you feel like. The spot you're in and security is messy. You're probably in a really good spot cause they must need a lot of security. I say that, both in a charming, but like a crying, laughing and crying way there.

But there's always something new to learn. That said do not pressure yourself to learn it all at once. For myself, I started out by learning just Azure Fundamentals and seeing how our coverage and security of those were at the role that I was at previously. Talking to people about, what does it take to secure this?

What does our, when we're following up on these incidents, what lessons learned are we taking away? For me, it ended up [00:42:00] Azure subject matter experts available, or if they were. They were a little bit more like managing Azure and that's okay. We get to know them, but in terms of securing it, there wasn't a set number of champions.

And I think this is just the case with Azure right now. It'll probably change in the future. I've noticed with AWS, you might have, 20 passionate folks that you're back in call to talk AWS and, tell you when you're wrong and things like that. I'm going to be honest when I joined in my current role.

People really like AWS where I am and, both internal and external to the company. I had some people who said, so you joined as an Azure, why, why would you do that? Yeah. I know my niche. So definitely start out understanding how does networking work? What are the general types of identities?

How do those interact? Great baseline for that is AZ 104. It's the Azure Associate, Azure Administrator Associate. They have free labs online that Microsoft will let you spin up. You just click a button. It makes the [00:43:00] lab. You do the thing it's telling you to do. You learn quite a lot by clicking through it.

And then it'll destroy the lab for you. You don't have to pay a thing. You don't have to set up anything complicated. And that's really helpful because I think especially when you're starting out. Telling somebody to, it's like telling someone to just install virtualization and start setting up their own, like custom arch Linux VMs.

When they first get into security, we don't need to hurt ourselves that much unless we want to. So Microsoft learns resources are fantastic there. Even if you don't take the test, it's a great way to get familiar with the terms. If you ever need video guidance, John Savile on YouTube is like the greatest, right?

He's, I always feel so happy every time I watch his content.

Ashish Rajan: Oh, Jesus. That guy is so dedicated. He's an

Katie Knowles: inspiration. Yeah. And he's always he doesn't know this, but he's always been there for me right through his videos. If I need to understand something he has a whole video on service principles and application registrations.

It's just him being like as the like video image, cause that's how it feels. [00:44:00] But he's there with you. It's okay.

Ashish Rajan: I think the only person who talks that much about Azure to your point about security, the champions for Azure, I think. There are definitely people who talk about different certifications, but in terms of I can maybe because he comes up from that solution architect background and teaching background, but he definitely covers in depth videos.

I think there was a. AI video, I think I saw from him on Azure, like a three hour video. This guy, three hour lecture. Go Jesus. I'm like, yeah, is there an AI to compile this? I can talk to it or somehow I'm sure there'll be a version for that as well. One day for John, whenever he is, we actually had him on the podcast a couple of years ago.

It's really interesting and I definitely find it to be great that you actually called out. You don't have to jump into the deep end straight away. There's actually easier part to just start scratching the surface, but just understand the fundamentals lay of the land as well at least.

At least from what I've heard so far, there is definitely a a foundation fundamental that is already a maybe people, most [00:45:00] people are aware of if they're coming from on premise or from AWS landscape, just basically how does that work in this particular world of Azure? Out of curiosity, I don't know, one of those controversial questions as well.

Can there be a multi cloud person? I feel I personally feel there can be, it's just you can be like jack of all trades. Yes, but would you be that good in AWS, Azure, GCP, Oracle, whatever else comes out tomorrow? And that's my personal opinion, but I'm just curious as to where do you stand on that?

Katie Knowles: I think you can be multi cloud. I think you just need to, for me, at least I always have to remind myself that. Like all things in time. So my past role, I was technically doing a cloud incident response across all 3 clouds. It was a very big organization. It's just that we had the most adoption that.

We could help respond or like business adoption for areas. We were focused in at the time we had a lot of that with Microsoft, which made it an easy place to focus. And then people start paying you for M three, six, five don't worry about that until you get [00:46:00] there. But it just happened. I think I also, you mentioned, everybody knows about active directory.

When I was a pen tester, I really wanted to be one of those cool active directory pen testers who knew everything I only got to I learned a lot, like I knew it was really rewarding to learn and there's more windows registry keys than stars in the sky. It's a rewarding practice, but there were a lot of people in that space.

So I think it wasn't really it's easy to go forge new paths and you had to get really deep into the internals to start. Having an impact. I like that. I get the Microsoft flavor in Azure. If you're used to the Microsoft flavor of, everything's expanding and there's lots of strange and interesting things happening.

You'll always get it with Azure. If you're more used to other clouds. You'll be able to relate it to other clouds and kind of wonder why on earth things happened a little bit differently and more strangely in that way. You can be, you can do multi cloud at a surface level for all clouds, [00:47:00] obviously, and maybe to a medium level.

I think if. Especially if the demand of your role gives you that, ultimately, you want to be able to line up what you're learning with what you're able give your company, not in a capitalist sense, but like you want to have impact, right? You want to be able to do stuff. Yeah. So if there's a truly multi cloud company, you can probably be a truly multi cloud person really easy, right?

So you'll get an extra, 40 ish hours a week to be a multi cloud person.

Ashish Rajan: But I think

Katie Knowles: the level of depth for each cloud. No that's not what I was sleeping.

Ashish Rajan: Yeah. I think that's where I feel. And I think, but the reason I wanted to cover this is because we had a lot of conversation around a lot of people want to have conversation with Azure folks in their organization and then, but then, or even communities folks for that matter as well.

Some people just want to get to a point where they can actually have a conversation without feeling lost. It's I have no I think from everything we have spoken about so far, I definitely feel I'm walking away from this conversation just going, Oh, I, at least I [00:48:00] know what some of the similarities between say an AWS type of incident versus an Azure type of incident would be, but I also understand the complexity that, Oh, okay.

Networking may be slightly different, but at least I know I have to focus on our identity, the kind of access and then build on my playbook. And obviously there are some examples that you shared as well. So I'll put in the link as well. That's most of the technical questions I had. I've got three more fun questions for you.

Hopefully I can get through them really quickly as well. Cool.

Katie Knowles: No rush.

Ashish Rajan: Yeah. So first one, what do you spend most time on when you're not working on doing the research with Azure or doing security of Azure?

Katie Knowles: So is this outside of work or what do I do inside of career? You can outside of work or

Ashish Rajan: work related as well.

Katie Knowles: Yeah. Outside of security research. So right now I've been with data dog about one year. I'm working a lot on deepening my product, my relationships with the product team to help influence our product. The reason I say this is that before my job was like all influence in politics because it was a very big organization.

Now I get a very deep. In depth technical role but those kinds of [00:49:00] relationships with teams where, you know, the Azure person to call, or somebody who says yeah, I can help you out with that. Or people who are looking for each other it takes a really long time to meet all those people and convince them that they should talk to you on a regular basis.

So I'm working on that. And also, not just security says as an influence, but how can we like work with people to give them the answers that they actually want from me. So that's a big focus inside along with the research, the speaking, the posting, stuff like that. And finding vols, which is great.

Love that. Really love that. Outside of work I started taking guitar lessons because research is a lonely life. This is the two sides is that if you work in a. Big company. You're never going to feel like I've found you're never going to feel like you get the technical depth you crave because you need to do so many things across so many surfaces as a business, but you'll have so many friends to go through it with.

You'll be in the trenches with so many people. If you get to talk to somebody and say, Hey, how's it going? They're like, Oh, going through [00:50:00] it. Remember that incident from last quarter? And you'll be like, Oh, I'm crying with you, buddy. With research. I have time to focus on Azure and learn Azure.

I love that. I love my, deep friendship with the cloud but I might go days where I'm not really talking very much. And so getting some music in my life has been very helpful for that. Some expressions, something to do. That's not screen oriented. So that's been my out of work focus right now, though we'll see how many weeks it takes me to get to Wonderwall.

I'll have to keep you posted on that.

Ashish Rajan: I guess for people who will watch this on YouTube or LinkedIn, they can see the guitar hanging at the background as well. So yeah, that one's

Katie Knowles: a ukulele. That one's a tiny guitar.

Ashish Rajan: Oh,

Katie Knowles: yeah. Yeah, I also enjoy noodling around with that. It's much less intimidating.

I highly recommend it. It's got four strings. It's about yay big.

Ashish Rajan: Oh, okay. There you go. I mean it looks pretty big from this angle, I guess because they're quite far

Katie Knowles: Maybe it's a little bit

Ashish Rajan: And the second question that I have for you is what is something that you're proud of that is not [00:51:00] on your social media

Katie Knowles: I don't know.

There's so many things Life's been pretty good, honestly. So I this is not like a, it sounds weird to say this is a pride thing, but I got married last year. I absolutely am so excited and happy about that. It's not easy to So we had a lot of long distance going on and then we had COVID, we were able to finally resolve that spend some time together and tie the knot and that's been a wonderful journey.

So I don't know that's like a pride thing, but I'm

Ashish Rajan: really happy about it.

Katie Knowles: Thank you.

Ashish Rajan: And maybe talking about things that you get to enjoy together. What's your favorite cuisine or restaurant that you can share with us?

Katie Knowles: Oh my gosh there's two go to's in the Toronto area, it's always going to be Thai or Indian.

Oh, nice. If anyone's in Toronto, I highly doubt you will be, but the Thai restaurant is Chiang Mai, the Indian restaurant is Qatar, they're both great, I recommend both of them.

Ashish Rajan: All right. I'll, I'm pretty sure people are I should probably make run [00:52:00] of an AI agent that just goes through all the episodes, finds out all the food places that are recommended and just puts them on a map somewhere.

If any listener or watching person comes up with the AI agent to do this, please do that. All the episodes are public, but Thai and Indian sounds like a great idea as well. And you can map that on the conference locations.

Katie Knowles: I said, if you can map that onto conference locations, you'll have a bestseller.

Ashish Rajan: Actually, yeah, that could be something even more, I thought of writing like a medium blog, but then I was like, ah, I don't know what medium blogs is who reads it, but then if it's a map, if it's interactive, just talk to it, chat GPT. Hey, I'm in going to be, I'm going to be in San Francisco for RSA. What are some of the food ways I can time?

I love that. It could be if anyone listening or watching wants to make this for us, it'd be amazing. Thank you. Thank you in advance. We can people can get to know more about your Azure research and get to connect with you as well, Katie.

Katie Knowles: Yeah. So I don't post much outside of sharing things, but if I'm sharing things on LinkedIn.

My name is Katie Knowles on Twitter [00:53:00] and BlueSky. I have handles as well that are some variant of Sigil, but I can provide you those links. People can follow those if they'd like to see more. I'm mostly just retweeting Microsoft things or cool research my colleagues have done, or things that I've finally gotten to publish.

Ashish Rajan: That's awesome. And I'll put those things in the show notes as well. But thank you so much for coming on the show and introducing us to the incident response world of Azure as well. I feel like there's a part two coming because I need to go deeper in some of these instances, maybe on a one of those practical workshop thing.

We'll get to that as well one day.

Katie Knowles: Here's hoping.

Ashish Rajan: Awesome. I'll keep that open deck. Thank you so much for Katie for this and I'll talk to you soon and I'll talk to everyone else as well. Peace.

Katie Knowles: Sounds great. See ya.

Ashish Rajan: Thank you so much for listening and watching this episode of Cloud2KT podcast. If you've been enjoying content like this, you can find more episodes like these on www.

cloud2ktpodcast. tv. We are also publishing these episodes on social media as well, so you can definitely find these episodes there. Oh, by the way, just in case there was AI. Cybersecurity. We also have a sister podcast called AI Cybersecurity Podcast, which may be of interest as well. I'll leave the [00:54:00] links in description for you to check them out, and also for our weekly newsletter where we do in-depth analysis of different topics within cloud security, ranging from identity endpoint all the way up to what is the CNA or whatever, a new acronym that comes out tomorrow.

Thank you so much for supporting, listening and watching. I'll see you next time.

‍