Managing Threats in AWS using AWS Tools

Ely Khan
Ely Khan
Principal Product Manager at AWS

▪️

July 21, 2021

About This Episode

Like this show? Please leave us a review here — even one sentence helps! Consider including your Twitter handle so we can thank you personally!

Managing Threats in AWS using AWS Tools

July 21, 2021
Season 2
Ely Khan

Ely Khan

Principal Product Manager at AWS

About this episode

Like this show? Please leave us a review here — even one sentence helps! Consider including your Twitter handle so we can thank you personally!

Episode Description

What We Discuss with Ely Khan:

  • 00:00 Intro
  • 2:51 Ely’s journey to his job at AWS
  • 4:46 Cloud Security in Govt vs Private Sector
  • 7:20 TTPs in AWS
  • 12:24 Are TTPs different for Small vs Big Organizations
  • 13:24 Starting point for building Threat Hunting in AWS
  • 16:39 Challenges that organisation will face with implementing AWS Security Tools
  • 20:47 the Customer Request Feature from AWS is Coming soon?
  • 23:24 Importance of Customer Feedback
  • 24:38 Security products other than AWS Security Hub
  • 28:53 Audit Manager Use Case
  • 31:31 VPC Mirroring with Zeek (open source networking tool)
  • 33:04 Do we need SIEM and AWS Security Hub?
  • 36:18 Is AWS Security FedRAMP Approved?
  • 37:22 Operationalizing AWS Security Products in a way that it scales
  • 37:51 Goldman Sachs Cloud Security Workflow
  • 46:35 Setting priority for Alerts being raised in AWS
  • 50:03 Mature Security Workflow with Event Bridge
  • 56:24 Fun Section
  • And much more…

THANKS, Ely Khan!

If you enjoyed this session with Ely Khan, let him know by clicking on the link below and sending him a quick shout out at Twitter:

Click here to thank Ely Khan at Twitter!

Click here to let Ashish know about your number one takeaway from this episode!

And if you want us to answer your questions on one of our upcoming weekly Feedback Friday episodes, drop us a line at ashish@kaizenteq.com.

Resources from This Episode:

Recommend a topic

Partner with us

Join the team

Share

Facebook
Twitter
LinkedIn
Pinterest
Reddit
WhatsApp
Email
Skype

Transcript

Ashish Rajan: [00:00:00] I have been following you for some time on LinkedIn. And so for people who may not know who you are. Can you tell us a bit about yourself?

Ely Khan: Yeah. Happy to thanks for having me on really excited to be here.

So let’s see here. My background. Well, currently I’m an, a principal product manager at AWS focusing on a product called AWS security hub. I’ve been with AWS about three and a half years. And prior to AWS, I co-founded a company called SQRRL, which AWS acquired in January, 2018. SQRRL became Amazon detective and I broke off from the SQRRL team and helped launch AWS security.

Yeah. Prior to AWS and SQRRL. I was in the federal government for about nine years doing cyber security strategy and policy mostly, and doing that a few different places first at the department of Homeland security. And then at the white house, national security council staff, I live in New York city at least for the next couple of weeks, then [00:01:00] off to the Connecticut suburbs.

Ashish Rajan: It’s a really interesting journey. You actually worked for the federal government in a way you were working in the government space or cyber security and it transitioned over to the private sector as well. So have your definition for cloud security changed between like moving from government to private?

Or what do you define as Cloud security?

Ely Khan: Yeah. Well, while I was in government, there wasn’t much usage of the cloud. It certainly has exploded in the last five years or so, but so my definition of cloud security is shaped by my experiences at AWS. So, you know, when we first come to AWS, we all learn about the shared responsibility model, you know, AWS responsible for everything hypervisor and below, and customer’s responsible for everything above the hypervisor.

Although I feel like that’s shared responsibility model is oftentimes misconstrued. And in that, you know, we’re not leaving customers to their own to figure out cloud security, you know, our goal and my specific goal AWS is to try develop tools and [00:02:00] services that make it really easy to get cloud security right in AWS.

But you know, we’re here where I’ve really been going and talking to a lot of customers about, you know, in terms of cloud security is really thinking about cloud security in different way as traditional on-prem security, where traditional on-prem security really was viewed as a cost center and something that slowed down.

Development teams, whereas with cloud security cloud security can really be an enabler to innovation and allowed those clouds development teams to move faster, build applications faster because the things can just be automated a lot more in the cloud than in traditional on-prem security processes.

So yeah, I think that’s, I think the great thing about cloud security is that it can not be viewed only as a cost center, but really as a way to push business. Interesting.

Ashish Rajan: And yeah, it seems like definitely you have a very Amazon definition of shift flexibility. It does definitely. Doesn’t leave you hanging for sure.

All right, so we’ve finally got the clock security pieces are, but people obviously want to know about threat detection and how do you kind [00:03:00] of manage this and cloud, especially AWS, I guess.

Before we start into this, I want to understand what are some of the common TTPs tools, tactics, that you kind of come across where you see quite common in the AWS customer space.

Ely Khan: Yeah. So you know, one of the really interesting offerings that AWS has is actually a professional services offering, not a, not a tool, but a, a pro serve offering called AWS zip-line.

And they help customers with security issues, security incidents, events. They do hundreds of these investigations every year. And they went through in mind all of their data on what are the top root causes of security. Events or incidents that they’ve seen. And there were several things that bubbled up to the top you know, really sort of in terms of those root causes.

One of the root cause isn’t, I don’t think these things are going to be a surprise for folks that are familiar with cloud security, but one of the top root causes is misconfigured resources and configure resources to [00:04:00] be publicly accessible when they shouldn’t be the most classic example about this was, you know, publicly accessible S3 buckets and folks scanning the internet for those and finding some time sensitivity.

AWS has now put in place a lot of different types of security measures, you know, different layers of security to make it really difficult to make S3 buckets public. But if that’s not the only resource that can be misconfigured, you know, it could be elastic search instance that’s made accidentally public.

You know, misconfiguring, those types of resources is still one of the top root causes of security incidents.

Ashish Rajan: Yep. So I’ll just quickly add that Amazon has made the S3 buckets are private by default. That’s really changed a lot of things for people.

Ely Khan: Yeah. And then there’s tools like Security Hub, Macie, and GuardDuty and IAM Access Analyzer, all that who are detecting open S3 buckets.

And then there’s of course the new Amazon S3 block public access feature, which you can just set a feature so that S3 buckets can never be made [00:05:00] public in an account. So that’s definitely a one primary root cause for security issues in AWS. You know, another important one is deploying insecure applications into AWS.

So, the top two use cases for AWS as a whole are one. Data storage, you know, people just plopping down their data into AWS and two is hosting web applications. So building applications hosting in AWS, and those web applications can be vulnerable. So you know, deploying those web applications with CVEs that are remotely exploitable you know, not checking your code for coming web application attacks like cross-site scripting or SQL injection attacks.

All these things can be exploited to cause a security issue. And then I would say you know, the last one and certainly not least is unintended disclosure of security, credentials and secrets. And the classic example [00:06:00] here is, you know, people baking access, keys into their code repos and then, you know, folks scanning code repos for access keys.

This is actually a service that we do for customers. Like we are always scanning github repos for ax, AWS access keys, and then you know, first turning off disabling those access keys, and then alerting customers of their unintended disclosure. But, you know, there are other types of credentials that could be unintendedly disclose.

The one of the worst ones would be your, your root credentials in AWS best practices to lock away those root credentials and never use them. So certainly unintendedly exposing those can cause major issues in terms of security events. But it’s really those top three things in terms of categories that are the root cause of most security events that, you know, our zip-line team investigates.

Ashish Rajan: Interesting. And actually as a good to know, the three categories that you guys kind of address these by now [00:07:00] this change between say like a smaller organization using AWS versus a bigger organization, or it’s just the scale of how big the buckets are, I guess.

Ely Khan: Yeah. It doesn’t, it doesn’t change much at all.

We see these being the root causes across all size of organizations, certainly the bigger organizations have bigger attack surfaces, so there’s more to secure, but you know, ultimately most attacks that most incidents that we see come down to those three root causes, regardless of the size or scale.

Of your AWS usage.

Ashish Rajan: Yeah. No, that makes sense as well. So what’s a good point for someone to start as an, when I say a good point, I mean, like if someone’s starting today and they want to, I guess, build a practice using AWS native tools, like the security hub of the world, and where does one start and what do you normally recommend for people to like, I’m going to build a practice for threat hunting today in my AWS environment.

But like what we’re doing that.

Ely Khan: When we work with customers who are getting started with AWS we help them think through the [00:08:00] build-out of their security capabilities, through the concept of epics and the first epic deals with building a strong identity foundation. So thinking through your strategy for how you want to build that iAM Roles optimally. You’re not using IAM users instead, you’re using a federated identity management system. So connecting up your IAM roles that you build out, you know, to something like active directory and federating in so that you don’t use long-term credentials. The next epic is building out a strong foundation in terms of network access.

So building out VPCs and network access controls and security groups to limit, you know, what type of traffic can access your resources. And it’s not till a later epic, like the third epic when we start saying, okay, once you have those foundational security capabilities in place, strong identity control, strong network controls, should you start thinking about putting in place, you [00:09:00] know, a threat detection and other security tools?

The security tooling that we recommend today to be on in every single accounts, regardless of what workloads you’re running, our guard duty which produces threat detections across a wide variety of, of threat actor TTPs tactics, techniques, and procedures, and security hub, which does a number of automated security hygiene checks, looking at your resources and ensuring that they are set up in a way that aligns a security best practice.

So those are the two ones that make sense to get started with. And then from there you should be expanding out. You should have a vulnerability scanner, you know, one of those key root causes. I mentioned, you know, having multiple web applications hosted in security up, you should have a vulnerability scanner.

There’s the AWS native version inspector. A lot of folks also use third party tooling like Qualis or Tenable Rapid7.

Ashish Rajan: People who may be starting this and want to build it for scale or so they might be a small organization at the moment. But [00:10:00] they obviously are hoping to make it a bigger organization.

Everyone wants it to be a big successful one. So are there any challenges that you hear from people when they’re trying to implement AWS security hub or any of the tools that you mentioned? Especially because, my understanding is AWS Security Hub needs to be enabled per region as well.

And it doesn’t centralize at the moment. So you kind of have kind of centralizes, but not really tough, but I’ll let you explain that. I wouldn’t find it.

Ely Khan: Yeah. So maybe I’ll just put it into the chat here. There is an awesome new document. They came out within the last month or so called the AWS security reference architecture.

And I would say that this is must read for anyone that’s in AWS cloud security. It is extremely comprehensive. And a lot of really smart folks spent a long time building this out, but, you know, I’ll try to summarize some of the key points around it. The when starting out with building out AWS security tooling, you should be thinking about this from a multi-account [00:11:00] multi-region perspective.

And we provide guidance about how to think about setting up a multi account structure for your AWS usage. It all starts though with organizations like everyone should be using AWS organizations, AWS organizations is a way of managing your accounts and AWS organizations has the concept of organizational units, and we recommend you building out a security oU

and in that security, OU you would have an account that’s dedicated to security logging. So that account would be where you be primarily aggregating all of your logs from your different tools. And then you should also have a security tooling accounts. And the security tooling account is in particular you would set up an administrator account.

And that would be where you have the administrator accounts for your various security tools like Security Hub GuardDuty Macie, Detective, et cetera. So I mentioned [00:12:00] administrator account. Let me back up and explain what I mean by that. For our various security tools that are integrated with organizations like security app and guard duty and Macy you have the ability to set up administrator accounts and then link member accounts to it.

So you’d want it deploy those tools, those services in every account that you’re using. And then you would designate one account as the administrator accounts. And obviously it’s the same administrator account across all of your security tools and that administrator account. Has permissions to see all of the security alerts across all of your member accounts and that’s automatic, you don’t need additional you know, I am roles or things like that.

Once you set up an administrator accounts, it automatically has permissions to see all of this security alerts across all the member accounts. And the nice thing about organizations is that it automates that whole process. So you can automatically say, Hey, I want this [00:13:00] administrator accounts. And I want all of my accounts in my organization to be linked or to be member accounts to that administrator account.

And then as new accounts are added to that organization, they automatically are enabled as member accounts. So you can sort of set in for you. Once you set up this integration between your AWS security tools in organizations any new account that comes online, that you know, that your application teams create that it is also automatically enabled with the security tools as you mentioned most of our security tools are not cross-regional yet.

So that’s actually our number one feature request for security hub and is something that we will be launching this year. It’ll give folks a similar type of capability in that you’ll be able to set a home region and then link regions to that home region in that home region. We’ll be able to see all of the findings across all linked regions.

[00:14:00] And you’ll actually be able to automatically keep your findings in sync between your home region and linked region. So if you make an update to a finding in a home region or in a linked region, that update will be automatically replicated across your regions. So if you ever need to switch back to a linked region for, you know, resilience type issues, you can, and it’ll have all the latest updates to your findings.

So that’s a really important one. You know, customers are really excited about that.

Ashish Rajan: Wow. Yeah, I think thanks for giving us a first shot at, at listening to this, maybe. So all the 10,000 guests, you are listening to this, probably get a brief intro I’m using reinvent, I guess is what you mean.

Ely Khan: You know, in general, we will release features when they’re ready. We typically don’t save features for re-invent. So, yeah, I mean, there’s certainly exceptions there in different teams and AWS operate in different ways, but I like to get value in our customer’s hands as soon as it’s ready. So I’m not [00:15:00] going to typically like sit on a feature for two months and wait for reinvents.

Like, if it’s ready, I want to get that in customer’s hands. As soon as it is, it is.

Ashish Rajan: That’s awesome. And which, you know, as well well, I guess, cause I’ve personally dealt with, had looked into it as well, so that’d be a great feature. So I wonder if one of my feature request made through for that as well, but definitely it’s definitely good to know that it’s considered that when people do put in feature requests, they though they do go into that pipeline where someone in AWS does see it, it’s not, and it is prioritized and it’s worked on as well.

So that’s the best way to know from asking

Ely Khan: so important. I mean, it’s, it’s the number one way we prioritize our roadmap is how many customer influences are on this feature. And it’s, it’s a really well-oiled process, you know? So every time an account team, here’s a feature request, they go into this database, they add a customer influence.

The product team is watching this in real time. As these customer influences are added in. You know, it would take a lot, there would be, have to be a really good reason to not [00:16:00] prioritize the things that are at the top of that that product feature request system in terms of which ones have the most customer influences.

Ashish Rajan: That’s even better to hear. So we spoke about our security hub as a great way to kind of manage the Threats you did mention other products as well, of AWS security. And I mean, I guess from an AWS security tool set perspective, like what are some of the other ones that you recommend people should be using?

You say all of them, but what are some of the easier ones to start off with? So they don’t get scared off by a lot of alerts to work with. I guess,

Ely Khan: you know, folks usually start off with Security Hub and GuardDuty, If you are hosting a web application inside AWS, it makes a whole lot of sense to switch on Shield Advanced. All customers get basic DDoS protection via Shield.

But if you want more advanced DDoSs production and essentially someone to call, if something bad happens you should turn on Shield Advanced. That’s a really important one. There’s a whole bunch of other relatively newer services that come out like not an innovation it’s been happening inside AWS [00:17:00] recently.

It’s really exciting to watch. We came out with a new AWS network firewall service recently that is using the Suricatta engine under the covers. We have a web application firewall service and with that web application firewall service, you can actually use rules. From other third parties.

So you can like use F5 rules in the AWS app, which is pretty cool. Interesting. And then we have our vulnerability scanner inspector. We have Macie, which is primarily a data classification service. So scanning your S3 buckets, looking for sensitive data. And we have another relatively new service Amazon Detective, which came out of the SQRRL acquisition, which, you know, of course I’m familiar with, and that is designed to be a threat investigation and threat hunting service.

So, you know, with threat hunting threat investigation, you know, usually that is for slightly more mature security, especially on the threat hunting side. No, usually it threat hunting is not [00:18:00] something that like the one or two persons security shop has been able to do. But, you know, as you expand out and you want to go beyond the security alerts that are being fed to you and start searching for the unknowns in your data, that’s where threat hunting comes into play.

Ashish Rajan: To your point, I guess you’re going to have to start somewhere. So I sounds like AWS kind of gives you the foundation to start off with it at least.

Ely Khan: , yeah, exactly. Then there’s some additional tangental services. So there’s AWS config, which does resource monitoring security I’ve actually uses config under the covers.

So the automated security checks that we do are powered by AWS config rules. You know, that’s all. Yeah, transparent to the customer, meaning the customer doesn’t need to configure those rules. We configure them on behalf of the half of the customer. There’s also another new service that launched last year called AWS audit manager, which is AWS is foray into the GRC world.

So that’s automatically integrated with security hub. [00:19:00] So any of the automated checks that we do are pumped into audit manager and you can produce cryptographically signed audit reports that you can then hand off to your audit team.

Ashish Rajan: Not sure if you’ve have any examples that people have been able to use Audit manager at all. There was a question asked. Audience members in the live stream last weekend I think and I think it does GRC done some compliance checks, which is what a lot of CSPMs or cloud security posture managers also do as well.

So if you’ve had a chance to look into like a comparison for if it does the same thing or if it’s a bit more. Yeah.

Ely Khan: Audit Manager has a few different types of data being fed into it. It can ingest security, hubs, security checks it can ingest configurable evaluations.

It also can ingest data from CloudTrail. And AWS license manager. And, you know, the way to think about some of these big regulatory frameworks is that there are some things in them that can be checked in an automated way. And that’s where like security hubs checks or configs rules come into play. And then there are [00:20:00] other things that require more manually collected evidence.

So like, you know, a classic item around that would be do you have an incident response plan in place? You know, that’s a control require by lots of different regulatory frameworks. There’s no way for security have to automatically check for that. So what Audit manager does is it gives you workflow to assign that particular control to someone to upload that to evidence.

And then it’s automatically collecting evidence for the controls that can be checked in automated ways, you know, via security hub. So yeah, it, it has support for, I don’t remember the exact number, but I think somewhere around 20 different regulatory frameworks now, and it has mappings to secure, you have checks and two other data sources to pop, to automatically populate evidence for those different controls in those different regulatory frameworks.

Deloitte put out a really nice white paper about how they use [00:21:00] audit managers. So like some of the big consulting firms that do these types of GRC edits is they’re now using auto manager as well.

Ashish Rajan: Interesting. I might links for those three resources in to the show notes as well.

I’ve got a question here. Buhari is F Bihari any advice on NIDS and VPC, traffic monitoring the Zeek. So Zeek’s like a network monitoring tool because opensource a network monitoring platform, for lack of a better word, I think for one understanding. And that kind of begs the question. How do we, how do you see people use traffic monitoring and network monitoring on AWS in any environment?

Ely Khan: I have not seen not seen a lot of it to be honest. Certainly. So we have the VPC, traffic mirroring capability that was launched I think about a year and a half ago. What I see more of is people using GuardDuty for threat detection and the new AWS network firewall service mainly, because.

Well, one guard duty is just stupidly, easy to use, you know, it’s like one click and that’s all you have to do. And [00:22:00] then with AWS network firewall servers, it abstracts away all the complexity of trying to do this yourself. So like, okay, how do I hook up Zeek with VPC, traffic mirroring? It allows you to have a fully managed experience to do network intrusion detection with all without all that heavy lifting.

So honestly, I don’t have a lot of advice about how to do it yourself. I think my advice is use our managed capabilities instead.

Ashish Rajan: I think we were talking about this last weekend as well. A lot of people ask about like the choice between security hub and a SIEM solution.

What does that really mean? Do we need a SIEM? If you already have adequate security hub , how do you compare the two

Ely Khan: you know, I wrote it internal FAQ about this after we launched security up, because it was one of the most common questions we got from customers, like is security, EPOS SIM.

And the short answer to that is no, it’s not a SIM. You know, there are some I actually think of security more as a CSP M tool, a cloud security posture management tool, but a cloud security posture management tool that also has some [00:23:00] SIEM like functionality. You know, there are two things that security hub primarily does.

One, it aggregates all of your security alerts related to your AWS environment, and it automatically normalizes them into a common data format. We call it the AWS security finding. And that has a lot of SIEM like characteristics or at least in terms of the value proposition it’s SIM like, but we’re not a SIEM in that one, we’re only focused on AWS security alerts.

So, you know, we’re not focused on trying to ingest alerts from your on-prem environments from your multi-cloud environments. And we’re not a general log ingestion tool like a SIEM, you know, we’re not ingesting lower level event logs like VPC flow logs or CloudTrail logs, really just ingesting the more curated alerts that are produced by other security tools.

The nice thing that we do that SIEMs don’t do is that, you know, with SIEM, you have to parse and normalize all that data. That’s coming into them. We really use [00:24:00] the advantage of being AWS to say, Hey, if you want to integrate with security hub, you have to send us the data. In an already parse and normalized state.

So we have this well, typed JSON Data format called security finding form. It has like a thousand fields in it and all the data comes to us already populated against that format which some of our bigger customers have, have I’m proud to say, I’ve said that’s really become the gold standard for how to normalize security alerts.

So we see a lot of customers getting value out of just that aspect. And then, you know, in addition to that, though, we do these automated security checks. Like you would see in a lot of CSPs cloud security, posture management tools. And from our perspective, if you really want to holistically understand your security posture, you need to do both.

You need to do those automate security checks, but you also need to look at those in combination with all of your other security data. And the alerts that are being produced by your other security tools. So we think it’s a really important combination of capabilities.

Ashish Rajan: Another question for Joe bourke [00:25:00] here is AWS security, FedRAMP approved.?

I’m assuming he’s talking about the tools and data security months. You mentioned so far. I believe they are,

Ely Khan: right? Yeah. So AWS security hub specifically is approved for both FedRAMP high and moderate. So that means it can be used in us-east-1 and 2 as FedRAMP moderate. And it can be used in AWS gov cloud it’s FedRAMP HIGH.

That’s also the case for GuardDuty. I’m not sure about all the other ones but definitely security guard GuardDuty are

Ashish Rajan: is a basic foundation pieces to start with would give people a good headstart on doing security in AWS. As we kind of go to different products, is there a, workflow, of a mature organization where people we would consider the benchmark , someone did to manage threats in their AWS environment, which is either fairly large, that you can operationalize this thing, I guess what we’ve been talking about, setting it up for how do I operationalize this?

Ely Khan: And actually it loops back to your SIEM question as well. So last year we did a really cool webinar with Goldman Sachs and their [00:26:00] security team, actually, when we publish it. A blog post here in the next week or so with some sample code that is aligned to what they laid out and rolled out. So it’s, it’s all, it’s really exciting to, to be able to speak publicly about a really marquee customer like Goldman Sachs but you know, what they’ve done with security hub, I would consider to be sort of like the peak of maturity.

And it’s a really amazing set workflow and set up that they have. So they’re using security hub as the central aggregation point for all of their AWS security alerts. And this not only includes the other AWS security services, but also includes third party tools and homegrown tools. So you know, they have a whole bunch of like homegrown analytics that they’ve built out over time.

And they’re also ingesting those into security hub. With security, have you can really ingest any type of data as long as it is properly formatted and as long as it adheres to that defining format. So that’s all coming into security hub in is automatically you know, in a, in a [00:27:00] normalized state as it’s ingested.

And then they’re actually using event bridge heavily, and they’re doing a number of enrichment workflows. Now some of these workflows will bake into the product as needed features. Like this is one of the things I love like watching how customers are using the tool and customizing it, and then baking those customizations in as need features over time.

But you know, one of the enrichments that they’re doing is they have a separate database that has all of their resource owner information. So for this resource, who is the person inside the organization responsible for this resource and they’re usingEventBridge. That calls out to a Lambda function that call to that database, and then writes that resource owner information back into the finding itself which is a really important piece of information.

If you want to operationalize that finding downstream, they’re doing some other types of enrichment as well. Like they’re customizing severity scores based on, you know, specific business logic that they have also [00:28:00] using event bridge rules. Like the whole, our integration of the event bridge is really critical way that people are operationalizing security hub.

And then, you know, once the state is enriched, they have additional event bridge rules that are auto remediation in auto response workflows. So like a typical auto resp, I think about those as different things, you know, fixing it versus, you know, taking the next step to address it. So, you know, in terms of auto response, you know, one of the things that they did is really cool is, you know, for certain types of findings, like a guard duty, finding, identifying unusual behavior on EC2 instances, there are automatically grabbing the memory from that EC2 instance and saving it for future forensic analysis.

So they’re dropping, they’re using a step function, which is the target of event bridge rule to go and grab that memory and chop it off for future forensic analysis. And then in terms of auto remediation, Yeah, one of the key things that they do, and frankly, [00:29:00] we even do this internally inside AWS. This is like how our internal AWS InfoSec team thinks about auto remediation.

In production accounts, you need to be very careful about auto remediating things when auto remediation means that destructive action. So you’re typically not going to want to enable any auto remediation that involves you know, terminating EC2 instance in a production account because you don’t want to break that production workload.

But there are plenty of things that you can auto remediate that are nondestructive in nature. Like CloudTrail logging being turned off. No, that should be never turned off in an account that should be automatically turned back on whenever it’s detected to be off. Oftentimes encryption can be nondestructive in nature.

Some encryption, you need to be a little bit careful about like turning on internode encryption on SageMaker that can actually slow down some of your training jobs. But like un-encrypted EBS volumes, things like that. Those are typically things that you can also auto remediate and wouldn’t be [00:30:00] considered a destructive action.

So thinking about destructive versus non destructive actions and thinking about prod versus non product accounts is sort of a key part of that process towards operationalizing security hub in a mature way. And so building out those automate automated remediation response workflows through those lenses is typically what we see customers do.

And then. Whatever’s left over, going back to the SIEM discussion, dump the rest of that into a SIEM. You know, hopefully you’re pushing less information. It’s your SIEM at that point? Like one of the other cool things that Goldman did is, and they exponentially reduce the volume of information going into their SIEM is one they’re auto remediating, as much as they can.

And to like the really high volume log storage stuff like CloudTrail logs or VPC flow logs, they’re actually just dumping those into S3 buckets and not putting that into their SIEM and relying on tools like GuardDuty to chew that data and spit out, you know, the really important [00:31:00] things, but you know, using, you know, Athena to query that data when they need to.

And reduce your SIEM bill. So all that data does need to be stored in the SIEM.

One last, I know this has been a mouthful, but one other really important piece to operationalizing your security findings, integrating with a ticketing system. Like that’s a must do. Because you know, this is also one of the really important differences, I think, between on-prem security and cloud security is that, you know, an on-prem security, you know, when you’re dealing with like lots of laptops typically all those alerts are pushed into a centralized security team, a security operation center.

Maybe if you’re a big organization, you have a traditional three tiered SOC operation with three tiers. But you know, with cloud security, you don’t really need to centralize all those security alerts within a SOC because a lot of these security alerts are about misconfigured resources or web applications that [00:32:00] have security issues that needs to be fixed.

It’s typically not a SOC that’s going to triage and fix those things. It’s typically the dev ops engineers, the application builders. So having a mechanism to push these findings out via tickets, back to the dev ops engineers application builders is critical. So that’s also another key decision points thinking about, okay, which of these issues do I want to push to my centralized security team?

Usually it’s threat detections, right? That guard duties detecting on. And which of the things do I want to. Create tickets for and push out to the dev ops engineers. And by the way, that’s how that resource owner enrichment step is really important so that you can make sure that that ticket gets assigned to the right person.

And and then being able to monitor progress about, you know, how well those tickets are being burned down.

Ashish Rajan: To what you said there might be some work required before they kind of go out on the journey as well to find the right resource owner. So, you know, probably overloading them as well.

So I imagine there’s a bit of [00:33:00] a prioritization as well, to the alerts coming in.

Ely Khan: All of our alerts come with default severity labels, critical high, medium, low, or informational. But we typically see customers, and this is also an area where you can expect to see some more native features around in the, in the near future.

But we typically see customers wanting to customize those severity scores. So, okay. This is a default medium severity issue, but when it happens in my production account, I want to bump it up to, you know, higher, critical. Yeah. That’s a pretty terrible thing. Or, you know, based on the resource tag or based on the account tag associated with this finding, I want to bump up.

The severity score two X that’s a, that’s also another good use case for event bridge rules today. But yeah, prioritization is really important inside AWS. This is sort of internally how we do things. We use the concept of campaigns, so it’s, it’s, it’s not typically possible to like, fix [00:34:00] everything all at once.

Right. You need to figure out priorities and we call those priorities campaign. And the goal will be to, you know, pick, you know, what the current priority or small set of priorities are . Make sure your resource owners are ticketed for any open issues against those priorities and then closely monitor the burndown of those tickets.

And the way that we like to do it is create cross-organizational visible dashboards. So for each of our VPs reporting up to our CEO, how many open issues do they have against that campaign? And each no VP wants to be the worst, explain to the CEO like why their organization is the worst at this critical security campaign.

And that information is visible across the organization. So like, you know, the transparently. Reporting out on this stuff across the organization, naturally dries activity to towards improved security posture.

Ashish Rajan: I love it. I love the approach. And I think that, that, that definitely makes me go. That will be [00:35:00] interesting because when you’re implementing this across the board, once you have the priority, right, the other challenge would be, how do you drive different teams to take action on this as well?

And that’s a great, that’s a great example. Just may increase the visibility of this in the circles of people that are hanging out. So that definitely goes a long way, man. So we’ve spoken about how AWS can be used or AWS security tools can use for threat hunting. We flop a compliance piece as well.

We spoke on the workflow as well. I think one of the examples that you spoke about with Goldman Sachs had done, and when bridge came in quite often, How do you see people use Event bridge, if you have some more insight on that, where in terms of security teams are triggering I guess there’s an event happens.

They, they have a trigger that launches a Lambda that an order remediation or like what we’re seeing or some mature workflow that be something that scales.

Ely Khan: . So EventBridge pages in an event driven workflow system, you know, think of it sort of like web hooks and nice thing about security hub is that in the administrator accounts, you all, for the event bridge feed associated with the [00:36:00] administrator account security hub, you have all of your findings across all of your all of your accounts.

So you can centrally build out event bridge rules in your administrator accounts, and you can build those out in such a way so that the execution. Of those rules. So the actual remediation or the response efforts can occur across accounts. So set up the rule in the administrator account, and then the remediation can execute in the member accounts.

You know, that, from our perspective, that’s the simplest approach, just in terms of configuring and managing rules, you people want to do that in one place as opposing to each account. We actually have an AWS solution that does this for customers today. So if you Google AWS security hub remediation solution, it’ll take you to the AWS solutions page.

And the way that that actually works under the covers is that we primarily use system manager. Runbooks. Although it could be a Lambda function or it could be [00:37:00] a step function, you know, all three of those things are typically used in remediation and all three of those could be the targets of the event bridge rules.

And we deploy those system manager, run books on your behalf into each member accounts. And then when the rule fires in the administrator accounts across account role , is used to actually execute the remediation and the member account. So in general, event bridge rules are a security team’s best friends.

They can not only be used for that automated response for remediation efforts, but it can also be used for any type of enrichment or finding modification efforts. If you want to add a note, if you want to change the workflow status, if you want to change the severity score that can all be done using an event bridge rule that calls security hubs, batch update findings, API, to actually make that change to the finding.

Ashish Rajan: That’s awesome. And I know we’ve been talking about a lot of ATM security products so far, so I do appreciate your patience on this one [00:38:00] as well, man. I think I, I think I got a lot of value out of it. I’m sure a lot of other people who listen got value out of it as well, but I’ve wanted one more section that I have on the podcast.

It’s called the fun section, non-technical, just three questions about yourself, this so that people get to know you a bit as well. What people should know a bit more about you, the first one being, what do you spend most time on when you’re not working on cloud and AWS security?

Ely Khan: The well, gosh yeah, it’s, it’s funny. I was telling someone before, like, you know, I thought I worked hard.

I worked very hard at SQRRL my startup but. The pace here, AWS, it feels like not much has changed year. Like my life is still definitely tilted towards towards work in terms of my work work-life balance. Definitely spending a lot of hours you know, trying to, to, to build out AWS security services, but you know, the things that I love outside of work.

So I have a new baby girl she’s a six weeks old, so that’s certainly been the, my number one focus outside of the being a new dad, which has been a lot of fun. Beyond that things that [00:39:00] I wish I spent more time on that I haven’t been spending a lot of time on recently. So I love boy Ty. So I’ve been practicing and for the last 10 years or so.

And and but that’s that’s been slipping as of late. I’ve been trying to balance out my fight sports with country clubs, sports. So also also enjoy golf and tennis and try to get out there to hit some balls whenever I can.

Ashish Rajan: Awesome. Well I’ve always get to meet another fellow more type person as well.

Although now I’m switching more to boxing more than more. It’s like, that feels more natural, but yeah, it’s a pretty, it’s pretty demanding sport as well, so understandable, man.

Ely Khan: Yep. Yeah. Yeah. It’s a, there’s a, there’s a certain Zen about certain Zen about

Ashish Rajan: it. I agree. I agree. And actually the next question on the, just, I guess the Zen thing reminded me of that question, that what’s something that you’re proud of, but it’s not on your social media.

Ely Khan: Oh man.

Something that I’m proud about. That’s not on my social media,

Ashish Rajan: Natalie or something else that they’ve done outside in terms of volunteering work, but something that comes to your mind for something that you’ve done. Yeah.

Ely Khan: I mean, I [00:40:00] imagine you could probably dig through this, my soul. Yeah. But yeah, I’m really proud of the friendships that I’ve been able to maintain over time.

I grew up in a small town. You know, about 6,006, 7,000 people. And you know, my closest friends are a lot of them are the ones that I, I started going to kindergarten with. It’s you know, it’s been, it’s been amazing, you know, seeing that group of friends evolve over the last, you know, 40, almost 40 years now.

So yeah, that’s something I’m really proud of that is, you know, a really, really close knit group of loyal friends.

Ashish Rajan: That’s pretty awesome. And I’ve grown here for maintain relationships since kindergarten man, a single friend who for like kindergarten. So, I mean, it’s a hard one to maintain for all, all these years.

A good one would only for doing

Ely Khan: that. Yeah. Preschool actually not even preschool, not even kindergarten. Yeah.

Ashish Rajan: Oh, right. So from preschool. Wow. Th I need some tips on you for how to do this and find some, a preschool friends of mine. I used to hang out with I’ve got one more question and we’ll close out for that.

We, what’s your favorite restaurant [00:41:00] cuisine that you can

Ely Khan: share? Oh gosh. That one’s a lot easier. Eh, w which Maybe it’s surprisingly easy given how many amazing restaurants there are here in York city, but there’s one in particular. That’s that’s been my favorite for many years. Joseph Leonard’s, it said a little tiny restaurant inside west village, not particularly fancy, but just has an amazing vibe.

If you’re in New York have a seat at the bar, it Joseph Leonard it’s like the service. There’s amazing, really cool people that work there and just I think the quintessential New York city dining experience.

Ashish Rajan: Interesting. Oh, what kind of food? Muslim, Italian

Ely Khan: food or? Well, it’s very standard. Just American.

Yeah. It’s everything that everything done very, very well. All right.

Ashish Rajan: I’m going, definitely gonna try it out. And the next time we can fly out of Australia and come to New York. This was, this was really awesome. And thank you so much for spending the time with us. Where can people reach out to you if they have any follow-up questions or we can, they, I guess, where do you hang on social media?

Yeah,

Ely Khan: LinkedIn’s usually the place I spend the most time, so yeah, definitely feel free to reach out to them.

Ashish Rajan: Awesome. And [00:42:00] for everyone else, who’s kind of judge on joining in. I’ll read it and I’ll see I guess thank you, Eli, and feel free to definitely connect with Eli after this. I’m sorry.

Bombard him with questions. We have any, but only, only the good ones obviously. And for everyone else, I’ll see you this weekend when we have a collaboration going on with other podcasts, but until then I will see you all soon. And thanks to you. I really appreciate this,

Ely Khan: man. Thank you.

Enjoying our content? Don't forget to subscribe!