Finding Security Bugs in Google Cloud

Ashish Rajan: welcome to the show, Dylan. Thanks for having me. Oh my God. I am so excited about this because I wanted to kind of get into the weeds of bug bounty for awhile. last year, we did one episode, actually, a couple of episode, one with bug crowd, which you’re also a member of and spoke about what does it mean?

W where does it like, and today I’m super excited about talking about what does it mean specifically for Google cloud and why not get a professional hacker come and talk about this as well.

So for people who may not know who dylan is let us I guess who are you? And how do you reach your professional hacker status?

Dylan Ayrey: Yeah, that’s a really good question. So for awhile, I was like a little bit under the radar. Didn’t have the social media wasn’t out there too much. I’ve done, a bunch of stuff in my career, but it’s kinda hard to find if you, if you try to Google me. It did a little bit of work in consulting.

I’ve done a little bit of work in politics. And then I was on the application security team at Salesforce for a little while. Most recently I was on the application security team at Netflix. [00:01:00] I’ve done a little bit of public speaking for security research that I’ve done, we’ve done a little bit of bug bounty aging.

And I also have open source to a whole bunch of tools. The most popular of which is a tool called truffle hog. And most recently I actually founded a company around that community.

Ashish Rajan: It’s pretty awesome. And we definitely to touch on truffle hog a bit as well, because it’s definitely relevant for a wider audience, not just the Google cloud audience as well.

I’m gonna start with the first question and this is something that we ask all our guests, and I love to hear people’s perspective on this. So I’m keen to know yours. What is cloud security for you?

Dylan Ayrey: Yeah, so for me, I really break it down to two major categories.

The first category being identity and access controls and the second category being secrets management. I know there’s a lot more to it. But those are like the two main ones that hit home for me.

Ashish Rajan: Perfect. And it’s a very short and succinct answer as well. I love it. I was going to ask, because you’re talking about finding bugs and it sounds like you’ve touched on bug bounty as well.

We may have a few people who do not know what bug bounty is and something got [00:02:00] a responsible disclosure considering all the movies that I have watched none of them talk about responsible disclosure. So what is bug bounty and what does it have to do with responsible disclosure?

Dylan Ayrey: Yeah. So I will touch on bug bounty from two different perspectives, both from the perspective of a company and from the perspective of a security researcher from a us perspective of a company it’s a mechanism to encourage researchers to go out and find vulnerabilities and to compensate those researchers for those vulnerabilities and in doing so, it also allows them to control the narrative and impact of those security vulnerabilities.

And then from the researcher standpoint, it’s basically just gig work. If you’re a hacker, this is a way that you can define your own hours and you can make some money on the side. You’re really good at it. There’s some people that make hundreds of thousands of dollars a year doing bug bounties. And if you’re pretty new to it, there can actually be a little bit of a learning curve there because there’s a lot of security researchers out there that are [00:03:00] incentivized and you’re competing against them.

Ashish Rajan: Awesome. So, and what’s the relationship responsible disclosure?

Dylan Ayrey: Yeah. So responsible disclosure is sort of the superset of bug bounty and other things. It’s basically just a mechanism that’s sort of endorsed by the company as a means to disclose security vulnerabilities. And so if a company has a bug bounty, then you can be financially compensated for responsible disclosure.

If a company doesn’t have a bug bounty, they may still have a mechanism for you to responsibly disclose vulnerabilities to them. And that might be an email address, or it might be a form submission.

Ashish Rajan: Wait, so is there an option like this for Google, cloud? Is that how we find bugs we start with, responsible disclosure for Google cloud?

Dylan Ayrey: Yeah. So Google cloud or actually Google as a whole. Has a mechanism for you to report a security vulnerabilities to them. And within that Google cloud has a piece of that carved out where they will financially compensate researchers for vulnerabilities that are [00:04:00] unique, that they triage and decide to action on.

Ashish Rajan: So I guess for anyone listening, who wants to probably kick it off a probably should lookout, if there’s a responsible disclosure program and then start exploring that a bit more, you seem to have shared quite a few wonderful, well, I guess, depending on how you look at it, I love when you call them bugs or vulnerabilitiesbut you find a few on, on the Google cloud site.

And I’m curious, what got you into looking for bugs in google cloud versus I don’t know, AWS or anything else, I guess.

Dylan Ayrey: Yeah, it was a little bit opportunistic. I mean, I guess the real story is there’s a friend of mine who works at bird, the scooter company named Marc Newlin. And one day I actually shared a Lyft ride with him home and he just told me a bunch of really eyeopening things about Google.

And he’s like, yeah, everything runs as admin by default. And I was like, what? And then he just started explaining more and more. And I’m like, this is not how I understand cloud’s work. Please tell me more. And then I eventually, you know, took it home and did more research on it and then pulled in [00:05:00] Alison and we did a whole bunch of research together on it and presented on a few times.

Ashish Rajan: All right. I had a chat with , Alison as well, who co-presented with you, if you have like a few of the bugs, but what’s the first one, which is an interesting one for me.

What was the metadata API one? What is the metadata API for people who don’t know, and this is the same as AWS.

Dylan Ayrey: Yeah, that’s a good question. the lead in here was I did a talk at, Bsides on the GCP metadata API at a high level cloud instances. So like And what I mean by that is when you assign a role. Or assign permissions to one of those instances. Lambdas or cloud functions or twos or virtual machines, they get their identity through an internal API.

That instance then is able to use those roles by making a request to an internal IP address fetching credentials, and then using those credentials to hit public API APIs. So the metadata API is basically just an internal IP address that you can hit [00:06:00] from an instance in the cloud that gives you credentials and those credentials match that access to those credentials match.

The role that you’ve assigned, the instance it’s very similar to AWS. They also have an internal API. I think the IP address is different, but other than that, the functionality is almost identical.

Ashish Rajan: Maybe it was while exploring why you maybe one of the reasons why they added headers into the Google metadata API.

Dylan Ayrey: Well, so actually let me clarify that. So there was somebody named who used to work at Netflix. Now he works at Hashi Corp, and I think you actually had the pleasure of interviewing him not too long ago. Awesome guy. Will sneaked in headers, which basically prevent a type of vulnerability called server-side request forgery most of the time.

But there was one exception to that, that you could still in Google cloud abuse through a headless browser. And to close that loop. What I did was I suggested they added host header validation. [00:07:00] Basically it’s a specific header that browsers can’t modify browsers can modify other headers, which is why wills protection didn’t quite cover this use case, but they can’t modify the host header.

And so what I was able to do is add the suggestion that they validate the host header so that we could close down that last avenue for SSR F or at least a known avenue, I should say.

Ashish Rajan: Yeah. And to your point, I think it’s worth calling out as well. Right? So pre adding the headers and one start people start using the API the whole conversation and the common use cases that you come across in the industry is someone leaves their credentials on a GitHub repository or a stack overflow forum somewhere.

And you kind of, once you identify them, the whole SSR of conversation was something that people would just like slam it everywhere. It wasn’t that straightforward, that, and I’m going to play a bit dumb here. So if I have access to someone’s keys in Google cloud to identity keys, I, or to a service account in a Google cloud, I [00:08:00] can use that to access, like in the previous version of non-hetero version, I could just.

Hit the API and get any information that’s related to that particular credential that I have. Would that be accurate description of the SSRF, and how it can go beyond

Dylan Ayrey: just that? Yeah, not quite. So let me let me rewind a little bit and define what SSRF for folks kind of new to AppSec.

And so this is kind of the confusing part is because a lot of my research has blended together application security with cloud. And so we need to kind of cover a little bit of both. And so SSRF is server-side request forgery. Basically. It is just a mechanism it’s a vulnerability that enables a anonymous user to send requests.

From the server on behalf of the user and example feature that could introduce the necessary F is something like a feature that allows you to upload images and you could specify a URL and they’ll fetch an image from that URL. And so the reason why it’s a vulnerability [00:09:00] is if the attacker specifies an internal IP address instead of an image, and then that server will go and fetch from that internal IP address and return it to the hacker.

Then. In this specific case of the metadata API, it allows that hacker to get access to credentials that are supposed to be internal only. You could also use SSRF to fetch just general sensitive data. If there are sensitive services behind a private network that the hacker can’t access, they can use this SSRF to jump into that private network.

But in a cloud setting it’s often scoped and referred to within the metadata API context. And Will Bengston did a bunch of work with Amazon to get them to add a required header. Which would not be present in the feature that I just suggested. So if you have a feature that you specify a URL that fetches an image that extra header wouldn’t be tacked on, and therefore the metadata API would not return an API [00:10:00] credential.

And so will, was able to close down a large swath of SSRF in the cloud context just by getting Amazon to add that header and then Google copied Amazon, and also added that header.

Ashish Rajan: Ah, right. Okay, cool. Thanks for explaining, so are we saying that the headbands browser wonderful, that you kind of explained in your talk that you referred to, is that completely gone to some header has been added, or actually maybe if you could explain that a bit more work was.

Dylan Ayrey: Yeah. So this is going to get really tricky because it’s going to go really in the weeds of application security. And I hope that’s okay. Basically browsers are beasts. They are really complicated beasts that have a long legacy of weird rules and restrictions. One of them is the same origin policy and I have to, I’m sorry, I have to dig in a little bit and explain what that is to be able to fully explain this vulnerability.

Basically, a browser is meant to be able to visit a website. And once that website runs JavaScript, the JavaScript on the page can [00:11:00] make that. User send HTTPS requests. But there are limits to what those requests can do and where they can go. And those limits are defined by the same origin policy.

Usually the rule is in less than other website allows you to, you can’t send a request and view the response from a different origin. So if you go on google.com, Google can’t on your behalf steal a bunch of your Facebook data and then do something with it. So there’s these, these rules, these limits on origins.

So there was a feature that a Google project manager did a blog post on saying you could run a headless browser in a cloud function in Google that allows you to basically take a screenshot of any website you want by providing a URL. And that cloud function will run that URL in a headless browser and then screenshot what’s ever there, that includes running all the JavaScript on that website, doing whatever that website says.

So, you know, you might think, okay, well that sounds a lot like [00:12:00] SSRF, let’s just specify the metadata URL and you’ll be done with it. But the problem there is it expects that header to be there. And so if you just type in a website in a browser and hit enter, it’s not going to set that metadata header and what gets screenshotted will not include a credential.

So we have to be a little bit more creative. And so what I ended up doing was I had this browser make a request to evil.com, we’ll call it. And then I served a bunch of evil JavaScript and what the evil JavaScript said was make a request to myself, to evil.com and then take that content and render it on the page.

And the reason why I did that. And this is again, deep in the weeds of browser security. The definition of an origin in a browser is the protocol the domain and the port. And the reason why I mentioned that is the IP address does not fall into the definition of a origin. And the reason why that’s [00:13:00] important is if when the browser does a look up.

To resolve the domain and it points to evil.com and then evil.com says, make a second request to myself. And the second time it makes a request. It does a second DNS lookup. And that time an evil DNS server responds saying actually my IP address is an internal metadata API IP address. The browser will be tricked into thinking, it’s talking to the same origin because IP addresses aren’t a part of origins and it will successfully be able to view the content from the metadata API.

And then we can render it onto the page. And I know that was a whole bunch and probably more than you bargained for. But that was the, that’s the crux of how that vulnerability was able to evade the protections that Will Bengston, installed it, AWS and the Google then copied around. That extra header check and the way we [00:14:00] protected against it at Google, the suggestion I made to them was to strictly validate the host header, which is automatically set by the browser.

And can’t be modified by JavaScript so that if they did make request to the metadata API, it would say evil.com and they would be able to check that and say, this is evil.com. This isn’t metadata API, let’s drop this request. And that was the extra protection that was added. And that was the loophole for us SSRF that was still possible in Google cloud.

When I reported the vulnerability,

Ashish Rajan: but it’s no longer there

Dylan Ayrey: anymore, as far as I know that loophole for browsers has been closed, but I definitely encourage other people to take a look because I can’t guarantee that every loophole is closed. There’s always fun research to be had.

Ashish Rajan: Yeah, that’s pretty awesome, man.

Thanks for, I guess. Laying it out. I know the whole, I definitely encourage people to check out the actual talk as well. I’m going to link that in show notes because I love to kind ofway you explained in a simple [00:15:00] way that you did just now, because there’s a lot to digest in there, but it was still a bit simple enough to know that, Hey, a browser should not be making a call to the back end, Google cloud, I guess, made a metadata API, unless you trust the origin, you should not be allowing the metadata to kind of respond to it as well.

I think I’m taking that. That’s the big takeaway from this. So glad to know it’s been primarily closed or kind of closed. We’ll encourage anyone who’s listening probably to just go in and well, I’m going to say tinker, but do what you think responsibly. I should be very careful when I mentioned this, . Thanks. So that was the first one. And I think the way I’m going to approach the show that we’re talking about a few more, just to kind of talk about what the bug was and hunting, how people can forward as well.

Maybe I guess, how people can limit the exposure. So the next one that came to mind was around the service accounts. I’m going to probably go into a bit more on the Google cloud territories for maybe we can dumb it down for people who may not have google cloud exposure, [00:16:00] but know AWS, which seems to be the bigger market player at the moment.

There is this whole concept of service account in Google Cloud, and there was a specific vulnerability that you spoke about from a Google cloud service account perspective and the permission it has and the owner. I would love for you to get, to get into, what did you identify as a use case forservice account and the owner and it kind of in a large organization, people just get lost in that, I guess.

Dylan Ayrey: Yeah. So I think, you know, Alison and I worked carefully on how to do the best introduction to Google cloud and where we settled was basically by just comparing it to AWS. Since most people had exposure to AWS and basically the way we broke it down is in AWS identity or access controls are identity centric.

And what I mean by that is you can answer questions like what does this identity have access to? It’s so that the policies for what can do what it’s centered [00:17:00] around an identity and Google cloud that’s flipped on its head. And access controls are actually centered around resources. So you can answer questions like who has access to this resource.

And the resource owner controls who can access that resource. The identity owner doesn’t have control over that narrative, which is the opposite in AWS, where the identity owner sets the controls for the identity in Google cloud. It’s the resource center sets the controls for the resource and all that’s a long way of saying in Google cloud, if you have an identity, you can’t answer questions.

Like what is the identity have access to it’s unanswerable by design because resource owners decide whether or not that identity can access the resource and you may not have permissions to be able to even see. What that policy is. And so you can answer questions like who has access to this resource, but you can’t answer questions.

Like what does this service account have [00:18:00] access to in AWS? It’s the other way around. If you have an identity, you can very easily be able to answer what is this identity have access to.

Ashish Rajan: Wow. Is that by design?

Dylan Ayrey: It is by design. Yeah. And I talked to some of their PMs about that, and I think the reason why they built it that way is just because that’s how they did.

I am internally at Google prior to Google cloud. And so they just copied the same access control model over.

Ashish Rajan: Right. So a resource owner can define who can access, but. Me with the, with the identity. I would not know what else do I have access to? Like how much do I have access?

Dylan Ayrey: No. So you won’t know, and you can’t know sort of by design there’s, there’s no way for you to know what that identity has access to because any resource owner in the universe of Google cloud, even if they’re outside of your organization can grant it, access to their resources and you won’t be able to see that if you don’t have permission to those resources, you can’t see the IAM for those resources.

So you really don’t know what your service accounts have access to.

Ashish Rajan: Wow. I mean, it sounds like, sounds [00:19:00] like the GCP permission model, maybe a bit more open than AWS

Dylan Ayrey: one. Well so, that decision , is important for understanding how GSPI and works, but that decision by itself, I wouldn’t say necessarily opens things up.

But there were a couple of other decisions that made Google much more open than AWS. One of them was what I mentioned before when Mark Nolan told me in a Lyft ride home that the defaults in Google cloud are just very permissive. If you spin up a VM instance or you spin up a cloud function by default, it’ll get an identity attached to it that has almost full administrative capabilities which is the complete opposite in AWS.

If you spin up a virtual machine or you spin up a Lambda function, it won’t have any identity attached to it.

Ashish Rajan: Yeah. Wow. I guess as a security person listening to this, you’re going, that’s a bad idea to begin with. Right. Just like Y okay. So I get a lot of permission that I probably may need, but not necessarily need, maybe [00:20:00] adding a use case to this end talking about service accounts and maybe if you want, we can go to take the GKE path as well, because we had last month was Kubernetes Month in Cloud Security Podcast so we did a full month episode of just kubernetessecurity in general.

And we had Kelsey Hightower from Google cloud as well. He came and gave a talk about the Kubernetes space in general GKE. Sounds like I’m taking what you just said. If I started GKE cluster, I’m getting a lot of permission that I should not really need. So and I think you did like a use case on this as well.

So keen to know if you can unpack that a bit for us as well.

Dylan Ayrey: Yeah. So first I’m glad you mentioned that you were in touch with some folks over at Google. I got a lot of friends over there and. You know, I think Maya was leading the Google security for GKE for a little while. She’s great. A lot of the security features they’ve added are awesome.

I think a lot of the things I’m about to speak to are inherited debt. And what I mean by that is when, when they created [00:21:00] GKE, there were constraints that were baked into the cloud. They just had to deal with. Some of them are like that default I just mentioned. And so I just wanted to, to caveat everything with that, that there’s great people over there and we shouldn’t just blame, you know, toss blame around , there’s lots of people that are making the story better.

But you know, to answer your question about GKE. By default, aGKE runs on top of a bunch of virtual machines. And so by default, those virtual machines will get that very permissive identity attached to it. And what that means is by default, if you run workloads in GKE, unless the default has changed since I’ve last looked into it, those workloads can.

Hit the metadata API because they just run on that VM. And so they have the same networking, unless you changed your networking by default for your workloads. And they can pull that identity and get lots of permissions to your project. I think they call them scopes that they drop onto the instance by default, that gives you lots of access to storage, but [00:22:00] maybe limit some of the other things you can do.

But those scopes are often opened up because people want their workloads to be able to access more API APIs. And then on top of that just, just full access to storage for every workload is kind of scary by itself. So the reason why I’ve had so many caveats in place is because they’ve implemented a lot of awesome features that lock that stuff down, but you have to opt into them.

And one of those features is at a high level for the entire organization. You can just turn off that administrative grant. For new projects, it’s not retroactive, it’s proactive. And so if you create new projects, that administrative account just won’t get attached to things like VMs by default, if you enable that policy and then within GKE.

You can enable something called workload identity, which for every single workload you have running in GKE, it then gets its own identity instead of using the notes identity. And that’s really powerful because if all the workloads are sharing the same identity, you have a whole bunch of developers that are [00:23:00] just slapping all kinds of permissions onto the node.

And you may have many different developers deploying many different workloads, and you end up with everybody’s sharing the same identity and the permission is just getting crazy. So they rolled out workload identity, something you can opt into, which allows each workload within the node to get its own identity, to get its own metadata, API credentials, and to be able to segment that.

But it is not enabled by default. You have to go in and enable those settings. And if you’re just playing around with GKE for the first time, chances are, you’re not going to get those things out of the box.

Ashish Rajan: Or interesting. So I love the context that, so you’re basically, and a few times back to what you were saying about identity earlier, where as an identity or not, I will not even know that you’ve given me permission to access the excessive permission.

I think I’ve got a question over here. If there’s a version of IAM permissions boundaries for GCP do you know the permission boundary? I can probably explain what it is and you can probably tell us about the GCP. version Essentially, it’s a permission boundary you can allow for, I [00:24:00] think Dylan’s role is often SRE and this is the maximum permission can that he can be allowed.

That’s the simplest way of putting the permission boundary? Is that a simple, similar thing in GCP?

Dylan Ayrey: The thing about GCP I am is it’s a little bit of a spaceship. There’s lots of controls, lots of levers, lots of dials. There is there’s one control called scopes, which allows you to limit what an identity can do once it’s attached to a resource.

So I kind of hinted at that before. And then there’s another control called conditionals, which basically let you do all kinds of stuff. On top of permissions, you can say Ashish on the third, Tuesday of every month is allowed to only access this one bucket. And so have to take all these things in tandem to really understand what your identities can do.

And a lot of them are like, Opt in, you have to be an expert. IAM figure them out and the defaults are open, but you have to kind of enable them after the fact. But I think those two are probably the closest analogy to the AWS feature you just described.

Ashish Rajan: Oh, right. [00:25:00] Perfect. Yeah, because I think AWS is something like conditions as well, but this time he makes it definitely, I don’t know if he can ask you to do the whole, every Tuesday of the month thing, but it sounds pretty awesome that he can actually go down to that kind of a, hopefully that answered your question.

A good question as well. Thank you. So we’ve kind of spoke once to GKE and all working permissive, I guess, ports or north nodes that can get access to, and I can kind of see this could, this could be quite how do I put this? It can be quite overwhelming cause you almost look at this and go, okay, do you see Google cloud?

Even though the security team in Google is trying to do their best, but they have a lot of tech debt that they’re trying to kind of work around while find to remain secure. There was another one that you mentioned, which was actually quite a surprise for me, which was the whole cloud build role.

And if you could touch on what that is and what was the scenario that you Amanda kind of spoke about it. That’d be awesome as well. Yeah.

Dylan Ayrey: So like I mentioned, there is a metadata API that instances have access to that. Allow them to fetch credentials, to get access to [00:26:00] you know, a credential that has permissions that enables it to hit a public API and do stuff.

Right. But what was interesting about cloud build and a couple of other API APIs is you know, cloud build is a build service forget. So if you have a get repository and you commit to it it triggers when you commit to it and then it runs a build based on what your build file tells it to do, but you don’t see a VM and you don’t define any permissions, just kind of magically in the background.

Something runs somewhere and it puts together whatever you tell it to put together, and then it can publish artifacts to your project. And so what we did was we said, okay, well, something’s running somewhere. And typically everything in Google cloud, when it’s running can hit a metadata API, let’s just add an instruction to the build instructions, to wherever this mystical thing that’s running.

Tell it to at the metadata API and see what comes back. And a credential came back, which was interesting because we set up cloud building a project with [00:27:00] no service accounts, with no roles, no permissions, nothing which has set up cloud, build attached it to get a repository. And then within the get repository, we just added an instruction and the build steps to.

Hit a magical metadata API that runs on some mystery instance that we never see. And all of a sudden a credential comes back. And so, you know, the first question we had is what the heck is this thing? And what does it have access to? And we, you know, we dug into it a little bit and we found that there’s something called a Google managed service account.

So they created it, they manage it and they created the permissions for it. And then when we looked at what those permissions were it could do a bunch of gnarly stuff, including stuff around storage and artifacts. Like, you’d expect this magic thing in the middle of nowhere that can write to artifact stores would be able to do.

And, and so it’s just, it creates this scenario where people are totally blind into it, where maybe you have this Gitrepository and yeah, maybe you’ve set up right protections around a sensitive branch. But you [00:28:00] still allow builds to happen on the nonsensitive branches and maybe you’ve in your build step said, okay, well, the sensitive branch that gets published to production and the nonsensitive branch, you know, everybody has the ability to create branches and write code and, and the build steps will still run, but there’s no publishing that happens.

But what this enables someone to do is if they can push code, even if it’s to the non-sensitive branch they can still fetch this credential. And then all of a sudden get access to storage for the entire project. It enables them to publish to production for those artifacts. And it also enables them to Snoop around on whatever else you have in storage for that project.

And that just is not at all transparent to anybody really. And so we really thought that this would be a vulnerability. And when we wrote it up the response we got back was it actually it’s not, they didn’t view it as a vulnerability. And so that, that capability is still there and we weren’t the first to report it.

And we weren’t the last to report it. There have been others that have done public write-ups. There was one rhino [00:29:00] security published on the exact same thing. And so there’s just identities that you can basically fetch out of thin air and that have weird permissions to stuff that, you know, undocumented and cloud build is one example, but there were a couple other examples we found as well.

Ashish Rajan: That’s awesome. And for folks who are listening in, who are from an AWS background and our something going, that’s, that name sounds very familiar. There’s an equal in service in AWS called Code Build and it exactly what pretty much what you mentioned, and I’ll encourage people to try and see if thecode build that they buildis making a call to the AWS API.

Metadata API, because I think it’s funny, but I’ll say one thing we didn’t mention with the whole metadata API is that people who were using the older version, they have to upgrade to the new version of the header. Right. It’s not like, oh, how does updated is isn’t that right? Like, I can’t just like say, oh, it’s also resort now.

So it’s not a

Dylan Ayrey: thing anymore. I feel like Will Bengston is the person who should tell the story, but I’m going to channel my best Will Bengston and try to [00:30:00] explain that the way I understand it worked out, basically we’ll push really, really hard on Amazon to get them to add this header and Amazon’s response, as I understand it was basically, but everybody’s using the old version of the old libraries.

It doesn’t have this, you know, everything would break overnight. Right. It’s just like you’re using an old library that doesn’t set the header, the Botto three library. You know how like it’s, wouldn’t set the header and cloud identity wouldn’t work anymore. And so Will as I understand, it sent whole requests to every library you can think of.

Adding capability to set this header, which was awesome. But the response from Amazon was basically, okay, now that all the libraries have the capability folks still need to go in and enable it and then activate that capability within the library, bump the version of the library, pulling the version that sets the header and all that.

And so that’s the way it worked out in AWS and then Google copied all of that functionality. So it was the same thing. They had a version one, then they rolled out a version two, but you had to opt into it basically.

Ashish Rajan: Yeah. And , it’s [00:31:00] one of those moments that you, as a security person go like, Ugh, it’s like this could have been, I’m so grateful by the way.

Shoutout to Will Bengston appreciate people giving him a shout out for was for the work he’s doing on making sure others are getting the benefits of the hard work he’s putting in for leader getting notice for it as well. So, no, thanks.Will if you get to see this I’m going to give them a shout out on the posting as well.

So. I’m glad we are kind of on this juncture where we kind of spoke about some of the bugs that you identified, and we’ve kind of given some inspiration for people to kind of check out some of the AWS equal in and see these vulnerabilitis, I guess weaknesses still exists because it is a legit thing to call metadata API, right?

It’s not illegal to call the metadata API. That’s why it does exist over there. They don’t really say, Hey, you can only call it from point a to point B only. There is no definition for it. So someone who’s a poking around in terms of whether these things are equivalently, I guess, available or possible in AWS as well.

Is this illegal or is like, what is this? Like? .

[00:32:00] Dylan Ayrey: Amazon has a responsible disclosure policy where they write up what you’re allowed to test and what you’re not allowed to test. But as far as I know, they have no public bug bounty for AWS. I think they might have one for amazon.com, but I don’t believe they have, I could be mistaken, but I don’t believe they have one for AWS.

I

Ashish Rajan: havent

Dylan Ayrey: seen one either

Ashish Rajan: so , I mean, cause I think there’s an AWS security email that you can send a bug to like if you go to the AWS security page and they talk about, Hey, if you do find a bug reported to this email, which is like, eh, yeah, sure. Like funny, then we get back from them after that I guess. But you’ll, you’ll see.

So I’m so glad we’ve kind of covered that. We kind of just crossing the halfway point as well. So I wanted to kind of touch on some of these things as well, because you do have exposure on the other side of. Bug bounty, I think as in part of running it and like from, from a blue team perspective cause a lot of people who may be listening to this, like, oh my God, these people are possibly great people to be hanging out with.

They should share more information. So keen to know from you, what was your experience on the other [00:33:00] side? Like the whole responsible disclosure what was your approach and what was your experience as being on the receiving end for a bug bounty?

Dylan Ayrey: Yeah. So AppSec usually sits at this weird. Purgatory, that’s not red team, it’s not blue team.

It’s just kind of its own thing. And AppSec also can often encompass a wide range of things from, you know, software development, life cycle to pen testing, to bug bounty. So when I was at Salesforce, I was on the app sec team and my team ran the bug bounty. I was a little bit involved, so they’re not as involved as the person that was primarily in charge of it.

And then at Netflix that responsibility was fully shared within my team. And so I would just rotate on a PagerDuty schedule and everyone would share, you know, this is your week where you’re running the bug bounty and then next week would be somebody else’s week. So I do have you know, a bit of experience triaging bugs and sitting on the other side of the table, kind of talking to researchers and those types of things.

Ashish Rajan: And this kind of brings me up and should everyone have a bug [00:34:00] bounty program?

Dylan Ayrey: So there’s some very good stories about certain companies that went full, crazy with a wide scope, public bug bounty. And it was too much for them. They received too much intake. It was too much spam and they ended up getting more bugs than they knew what to do with then some of them took years to close out and it created bad research experience.

So my recommendation, and this is from me. Me personally, if you’re thinking about bug bounty in your company you know, first and foremost, you need a responsible disclosure policy. You need a mechanism so that when people find vulnerabilities, they can disclose them. But if you’re thinking about bug bounty, if you’re thinking about compensating folks, I would recommend opening up what’s called a private bug bounty first with a limited scope.

So basically what that means is you have a small number of researchers that you allow to poke around a certain part of your infrastructure. So you can test the waters a little bit. You can see what comes back, how bad is it, how many vulnerabilities do you get? How noisy is it? Can you handle it? [00:35:00] Can you handle it with all your other competing priorities?

If your test, if your experiment there, if you try it and it goes, well, then you can gradually increase that scope, increase the size of the program. And over time you can work up bigger and bigger and bigger and bigger. And if you want it to, at some point, go crazy with it and create a public bounty then that’s an option for you, but I would not suggest just jumping right in to a public bounty with a wide open scope.

Cause you might find that it burdens your team more than you were thinking, and it creates a really bad experience for researchers and then having a bounty and then making it go away can actually also create a bad experience within the community.

Ashish Rajan: That’s awesome. And thank you for sharing that as well, because I think people may underestimate the overhead that may come with that as well.

To your point, you, I mean, when you were doing this Netflix, you guys were rotating it on an individual. This is like on a weekly basis as well. So you could be in a scenario where there’s just so much information coming through you going, oh my God, I didn’t expect this. Now I need to have a dedicated person.

Who’s looking at this.

Dylan Ayrey: Yeah. I mean, you [00:36:00] have to understand it. Netflix. They had a very large mature application security team that had a PagerDuty schedule and had people that are dedicated to doing nothing but security, but at a smaller company, that’s considering bug bounty. It could look very different.

You could just have people that are like you know, security is one of their responsibilities, but it’s not their entire responsibility. Or you might have one person that’s doing all kinds of different security, not just AppSec, depending on the size of the company and the maturity of their security program.

And so it’s really important to test the waters a little bit, to understand what that burden is going to be before you go crazy. And you start opening it wide up.

Ashish Rajan: Yeah. Awesome. And that’s just, we have clarified a few things. We’ve kind of talked, spoken about a few bugs that were there. And we’ve also spoken about some of the challenges that people may get into.

If they start a bug bounty program without limiting the scope and not keeping it private. I’m curious because a lot of the conversation that we had so far in terms of bugs have been primarily around [00:37:00] lateral movement. And I’m curious to know from your comparison to, I’m going to put the links to your talk onto the shownotes as well so people can go back to it and listen to the whole thing, because they’ve done a great video on the Def con thing, which is really interesting with the whiteboard and everything.

I see a few whiteboard behind you as well. So I would definitely encourage people to check that out. The question that I had is compared to traditional lateral movements between, Microsoft corporate office 365 or whatever you call it how different would this be?

Dylan Ayrey: Yeah. So in a traditional Microsoft environment, you have a sort of a domain controller and an attacker breaks in at some part of the network, maybe through spear fishing and they get access to a person’s endpoint. And then you would use something like pass the hash to use the local admin administrator account to jump from that computer to other local computers.

You might use something like a w pad poisoning attack to get credentials and jump that way. And you just kind of jumped, dropping malware to malware, to malware from system to system system. Until finally you find a credential that’s privileged enough. [00:38:00] You can take over the domain controller and compromise the entire environment in principle.

All the words I just said are totally different in the cloud, but the concepts are actually really similar. You start with a credential and it starts with presumably a low-level privilege and you jump from that to another identity, to another identity, to another identity, just slowly collecting credentials, like Pokemon cards, just like you.

Wouldn’t a Microsoft environment until eventually you find something that has full control of the organization and then you win the game. The big differences. There are no private networks in the cloud. Everything is through a public API. There are no malware implants through the mechanisms that we talked about.

You do everything by accessing that API with the credentials you find. And so it’s very interesting that like the traditional Microsoft ways of lateral movement are through grabbing credentials and then running arbitrary code on systems to grab more credentials, to run more code until eventually you compromise the entire environment.

[00:39:00] And we can do the, almost the exact same thing in a cloud environment without dropping them out. We’re implant without compromising running systems, all just hitting public APIs with static credentials that we find lying around. Oh,

Ashish Rajan: I love it. I love the comparison. Cause now it’s like old school meets new school in a cloud versus on premise, so.

Okay, cool. Wait, so does that mean, what we used to call network security bench testing. That should be just called identity security testing, I guess, in the cloud world. I mean, I don’t know. I’m just making up words here.

Dylan Ayrey: Yeah. I mean, so, you know, if, if you want like , a pen test, you could just call it offensive security, throw it all in one big bucket and not worry about it too much.

if you’re looking for an audit, then you might call it like identity or access control, audit, or cloud audit or something like that.

Ashish Rajan: Because to your point I guess what you are really trying to find is identity is becoming quite crucial in the cloud space versus say on an on-premise environment where network also is quite crucial because you have your private network close network

right. [00:40:00] Okay. Well I’m glad we kind of mentioned that as well, and maybe makes me think that that’s the vulnerability that you found and I’m sure if there’s a public writeup on the whole Google’s internal identity, I guess identity access management space. Does any of this give you access into the Google cloud network?

Like as in like their own network? Or is it like, how exposed are they with this?

Dylan Ayrey: Yeah, so Google has for a little while. And I’ve never worked at Google. All of the information I get about internal Google has fallen off a truck. We could say, certainly didn’t come from any, any, any friends or anything like that.

It just fell off this one truck. One time Google for a little while has been working towards the same called corporate cloud, where basically they’re trying to move a lot of their old corporate data centers into their own Google cloud ecosystem. They basically dogfooding their own cloud product is another way of putting it.

And we found from that truck I am policies for internal cork to cloud projects, and [00:41:00] we wrote about this in a, in a public write-up. We found. Evidence of all of the vulnerabilities and lateral movement issues that we talked about in their talk, apply to Google the organization. And so what that means is if an attacker wanted to attack Google they, from the limited snapshot that we were able to see of certain projects, that attacker would be able to use the same exact mechanisms that we documented work for every other organization we looked at and they’d be able to compromise Google through the same lateral movement techniques that we talked about.

Ashish Rajan: I just imagine insider threat profile being created in Google cloud for this, but it’s an interesting way to put it. So, because I’m actually one more point on just on that, because I think you guys mentioned this because you don’t see what permission you have access to, but there’s a, there’s a trick that you guys did.

You copied the permission off an existing Google managed role onto a custom role. Can you talk a bit about that as well? Yes.

Dylan Ayrey: That kind of speaks back to what we said about cloud build, where sometimes there’s these mystery identities [00:42:00] that just fall out of the sky and you’re like, oh, well, what the heck is this thing?

And what does it have access to? There was a trick that Alison found that allowed her to actually see what permissions that mystery identity had. And we talked about that in our Bsides talk and folks can check it out, but that’s how we figured out how to access to buckets and artifacts and things like that was through that trick.

Ashish Rajan: Yeah. Perfect if they allow for that, that already makes sense. It’s not technically hidden for quote unquote but so I’m going to take a pause there and I’m going to. Ask a very serious question now because now we have scared people of Google Cloud

so what cloud do you recommend from your security perspective out of all this?

Dylan Ayrey: Yeah, I get that question a lot actually. And you know, my answer to that is it’s kind of a trick question. And I’m going to share a little bit of a story on why I think it’s a trick question.

I know a friend of mine is an executive over it at Microsoft and I had a conversation with them. At some point I was talking with him about internal Microsoft IAM and you know, they’re very heavy Azure users as you’d [00:43:00] expect. And I said, you know, at some point the conversation came up about Google cloud and I said, that you have some people at Microsoft that are using Google cloud. And he was very adamant that that was not the case. He’s like flat where Microsoft, of course that’s not the case. We’re all Azure. And so I challenged them a little bit. I, I bet them on and I was like, I’ll bet if you go and search for it, you will find a bunch of people internally.

Like you’re gonna tell me no one at Microsoft needs to use big query. Like no one at Microsoft needs to use the AI pipelines that Google. I mean, I’ll bet if you go looking for it, you’ll see some teams have opted to use Google cloud. And he did. He went and looked and found that that was in fact, the case.

There were some people that just needed big query to do their job. They needed the AI APIs that Google offers to do their job. And so the reason why it’s a trick question is if a cloud provider is multicloud, right? If a cloud provider that is Azure is their service. If they can’t prevent other people from using other clouds, right.

Everything. Everything is going to be [00:44:00] multi-cloud if it’s not already and, and we just need to accept that. And so that means, unfortunately we all need to become experts in more than one cloud, and we need to be comfortable with our developers working in more than one cloud. And rather than trying to pretend that it doesn’t exist.

And then you end up with people throwing things on court cards, and nobody knows what’s going on. We just, we have to embrace it, enabled those work policies on day one set of policy for it, and just accept the reality that every everything is multi-cloud. Oh

Ashish Rajan: That is, it’s an interesting insight.

And I wonder, as you say this, someone’s using AWS and Google cloud because the competing engine is not good enough. I wonder if that’s the case as well, but i’ll let those people comment on that. I’m curious as well now, because we kind of spoke about a lot of the I guess to your point. Some, we went into some of the weeds of application security for some of these SSRF

we spoke about the blue team side as well. I’m curious, like people may be listening to this and thinking, Hey, I can do some bug bounty for Google cloud as well. And [00:45:00] where do I start? Like, what kind of skill set am I looking for? Do I need to be like this Netflix security expert as the Dylan was talking about that I’m looking for jumping from one network to another, or what are some skill set that people can look to start with and work?

How low is the bar? I guess

Dylan Ayrey: yeah. So I kind of hinted about this a little bit earlier. Unfortunately, bug bounties in general have a barrier to entry. If you’re not familiar with with security vulnerabilities and you’re just jumping in trying to do the, basic bare minimum against the biggest companies, you’re going to have a lot of competition and , it’s going to be tricky.

Somebody who I follow on Twitter Vickie Li just posted about a new book that she wrote. That’s meant to teach people how to do bug bounty. It’s called bug bounty bootcamp. And folks are thinking about getting into bug bounties. That’s a good place to learn the basics, but what I’ve found is if you really want to be successful, you have to find some niche that either speaks to your skill sets, some specialty that you [00:46:00] are really good about and just really research the heck out of that so that you are.

Looking at stuff that other people aren’t, or you need to take advantage of private programs. Once you do a couple of bug bounties, you’ll start getting invites to private programs and there there’s just much fewer people to compete with. And it’s just a little bit easier to find some of the low-hanging fruit.

So unfortunately, for folks that are new to bug bounty, currently there is a little bit of a, skill gap of like a barrier to entry. There’s some good resources out there to kind of catch yourself up. But my recommendation is speak to your own skills, whatever they may be, whatever you’re specialized at.

Do your security research around that thing because it’ll just help make you stand out and help you look at an area that other researchers probably aren’t taking as deep into.

Ashish Rajan: I love it. And I think it’s, what’s a lot of our. People may underestimate their skillset. I think Zinat earlier mentioned identity identity is in itself such such a broad space and so much that people can be doing.

And there is no, I think identity security in cloud has become a thing. [00:47:00] Credential security has become a thing as well. And I would love to talk to I guess if you can probably tell people about truffle security and what truffle hog was all about as well, I would love that as well then if you don’t mind sharing.

Dylan Ayrey: Yeah, no, I appreciate you asking. Truffle hug is a tool that I wrote a couple of years ago to help me do bug bounties is the long story short. It helps find credentials that are accidentally put into Git and I kind of just throw it up on the internet. After I wrote it, I, it helped me find a couple of bug bounties

I gave it back to the community hoping that there might be some other people that could make use of it. And I’m just so. Humbled by how many people have used truffle hog to find credentials and make money off bug bounties. And, and, you know, it’s, it’s been mentioned in like a book now, and I think it’s taught in some universities.

Truffle hog is an open source tool on GitHub that basically it helps you find credentials and get, and truffle security is a company that I recently built around it. That’s just expanding on that community and building off the top of, of all the awesome people in the community that have [00:48:00] used the tool.

Ashish Rajan: Awesome. And I’ll definitely recommend people to check that out as well because trufflehog it may sound very, how do I word this you may almost think that, Hey, who puts credentials in GitHub, but people will be surprised how often that happens.

Dylan Ayrey: Cloud security is identity and access controls and secrets management.

And the second part of that, you got leaky secrets all over the place. You know, walk down a street and in the San Francisco bay area, and you’ll run into a credential by accident there they’re leaking out left and right. So

Ashish Rajan: awesome. Well it seems like people should starting in bug bounty should walk down the streets of San Francisco bay area.

But I appreciate this man, and I know we’ve kind of gone quite technical and with, towards the end of our conversation as well. What’s important to be simple question for you. So hopefully they’re fun questions and this way people get to know a bit more about you as well. First question, what do you spend most time on when you’re not working on bug bounty or cloud or technology?

Dylan Ayrey: So I was warned that there would be three very tricky questions at the end. So [00:49:00] I’ve laid a trap if that’s okay. For every question you ask, I’d like to ask you a question back, is that okay? And I’ll answer it. All right. So the answer to the first question honestly, is I’ve been so involved with this community building and this startup that’s, that’s really taking up most of my time.

But otherwise it’s, it’s been sometimes doing work in politics sometimes going for hikes. It’s it’s been a good blend of stuff. All right. Now I’ve got I’ve got a question for you if it’s all right. Sure. My question is being a member of the Aussie Australian security community. Do you identify more as a flat duck or a tall duck?

Ashish Rajan: Oh I was going to, well, maybe I want to answer it. You should probably that people, can we tell what flat duck? I think there’s one piece of equipment that we, yeah. You can definitely talk about. I think I’m a fat document. Embarrassingly enough,

to tell people more. Do we have to tell people what it is?

Dylan Ayrey: I don’t even know what it is, to be honest with you,

Ashish Rajan: let people,

Dylan Ayrey: sorry. Oh, I just, I just know it’s a very Australian hacker scene thing that everybody has [00:50:00] picked their identity and it’s just, I’m very confused by it, but I love asking the question.

Ashish Rajan: Perfect. It’s a good question as well. Hopefully I didn’t embarrass myself public because unless the meanings have changed since the last time I heard about it, it’ll be because COVID has made people not come out in public anymore. So it’s been a year of not seeing person and hearing updated definitions of decisive, some terms.

So hopefully I didn’t embarrass myself called fat duck. The second question that I have is what is something that you’re proud of, but it’s not on your social media.

Dylan Ayrey: Yeah. So, I mean, I was not on social media for a really long time. I just created my Twitter recently. It’s just this kind of lonely over there just because it’s so new.

But I guess, you know, there’s so, so much of the research that I’ve done was prior to social media. So the community that was built around truffle hog, I’ve really enjoyed that community, but it wasn’t on social media at the time, the work I’ve done in politics, that companies that I’ve worked at all, all that stuff was really, before I created a Twitter before [00:51:00] I started posting things.

Ashish Rajan: Awesome. Well, I will definitely your Twitter accounts, you don’t feel that lonely anymore, man. I think I’m pretty sure after this conversation, people would find they definitely should follow and connect with you and find out all the other interesting things you find on Google cloud as well, or maybe other clouds.

Yeah. I

Dylan Ayrey: appreciate that. Nick. May I ask a question back at you really quick? Go for a bed? May we, and you can say no to this. Maybe we see your dog.

Ashish Rajan: Oh yeah, you could. I actually logged them out. I was, I was trying to see you locked him up. Don’t worry. He has he has an Instagram account, so he’s all over that.

I was thinking on my account. I’ll share, I’ll actually put them with the nursing as well. Cause I definitely feel he has been getting a lot of attention lately. Cause he’s one of those poodle mixes. So I love what you’re doing, man. I love the fact that you are throwing questions at me as well, so I’m ready for it.

Next third question. So what is your favorite cuisine or restaurant that you can check?

Dylan Ayrey: Yeah. So recently I’ve been on a little bit of an Ethiopian binge I know injera bread and everything. It’s I’m, I’m you know, you can’t eat enough injera I’m [00:52:00] always getting the extra and it fills me up.

It’s a sponge in my stomach, but it’s, it’s delicious. I really like Ethiopian food and lately I’ve been on Ethiopian Ethiopian streak

Ashish Rajan: to get that whole big plate and everything like

Dylan Ayrey: bread, you know, before COVID you get, you know, a whole bunch of your friends, everybody just shares this big, you know, family style, lots of different stuff on the big chair.

It’s great. It’s awesome. It’s delicious. It’s you know, it’s, it’s, it’s really special. So that’s, that’s, you know, that’ll, I’m sure the answer will change over time, but that’s that’s, that was a good

Ashish Rajan: answer, man, because I think what, before someone told me, and I did not realize this was, there’s a reason why a lot of the plates around north square, it’s supposed to be like everyone coming together as a community.

So imagine a square blade. There’s only four people sitting around it. But if you sit around in one, then it’s like, you’re, you’re bringing, you’re growing your circle. That’s your circle kind of six to nine. I’m like, yeah, that anyway, I’m wearing it a very different tangent. What’s what’s your third question then

Dylan Ayrey: my third, my third question for you is I heard that Melbourne has recently entered [00:53:00] lockdown.

So what is the first thing that you’re going to do? Once Melbourne gets out of lockdown, I’m going to

Ashish Rajan: go to a good coffee place and try. And we’ve gone. I’ve been making, because it’s Melbourne is known for coffee and food. And a lot of the restaurants are, so the restriction rules are, we can all go beyond a five kilometer or 10 kilometer from our house, which is probably the same as three to seven miles from the house.

Like a lot of people may not have a lot of things around the seven mile radius, not like a great restaurant sometimes. And unfortunately I’m in one of those suburbs, I guess there. I have a few places, but after a week you’ve tried all of them. So now you’re like you go into second week, you’re like, Ooh, I wonder what that place was like, which wasn’t downtown.

And it was pretty good, but it’s beyond that loneliness for me. So I’m looking forward to going for a good brunch so great, great coffee and hopefully a good brunch item as that’s what I’m looking forward to, but maybe you might feel for your people. Might just be interested in giving hugs and like, I’m so glad I could see another person after the log down [00:54:00] because you almost want to see many people on the road as well.

So you’re

Dylan Ayrey: just like straight for recommendation. Yeah. You’re like right. Priorities.

Ashish Rajan: I appreciate that. But a great question. And I love what you did with your approach as well. Matt, thanks so much for doing this. I’m conscious of your time as well. What can people find you if they want to have any more up questions with you?

What, where do you normally handling profit security as

Dylan Ayrey: well? Not just yourself. I’m on the I’m on Twitter now. Yeah. So, so insecure nature. Twitter is myself. I think we set up a truffle security Twitter, but I need to, I need to, I need to learn how to Twitter. I’m not, I’m not, I’m still learning the ropes.

I’m not new to it. I need somebody to help me. I need a shepherd to help me through the truffle security Val or the, you know, I mean the the Twitter valley I’m on LinkedIn as well. I used to not be active there at all. I finally like filled in the last glitch shops that I worked at and like actually did some stuff on LinkedIn I’m available now.

So folks are to find me on LinkedIn, find me on Twitter, find the, you know, wherever it’s convenient and I’m available. Awesome.

Ashish Rajan: Awesome, dude. Thanks so much for coming in, man. I really appreciate that call. I think the research that you’ve done [00:55:00] and and everyone that you’ve Alison and I think you mentioned Will as well.

It’s been really interesting to kind of hear. That, oh my God. As a community, we are trying to make these things better as well, even though the cloud service providers. I feel it’s, we just need more of these and I hope someone got inspired listening to this conversation and maybe reach out to you and or maybe even see it.

And other people would be just be able to kind of look into the space a bit more because how do you secure something without knowing how it can be exploited? Right. So I appreciate you spending time with us. Thank you so

Dylan Ayrey: much for having me. We’re all we’re all standing on the shoulders of giants. So like I’m only, you know, building on the top of previous people’s research and hopefully, like you mentioned, folks can build on the top of mine.

Thanks so much for having me this morning. It was a blast. Really enjoyed the chat we had.

Ashish Rajan: No problem. Thanks for coming in. And for everyone else feel free to. I guess follow it, subscribe to the channel, or if you’re on LinkedIn, follow the follow the channel because this is every post every weekend.

And I will see you all on the next episode until then stay [00:56:00] safe and enjoy my coffee. All right, bye.