Episode Description
What We Discuss with Jerome Walter:
- What is Security Chaos Engineering?
- What are Chaos Engineering Experiments?
- Example of a Chaos Engineering Experiment
- Does the Chaos Experiments need to take place in Production?
- Is Chaos Engineering and DevSecOps same?
- How do you know the maturity of a Chaos Engineering Practice?
- And much more…
THANKS, Jerome Walter!
If you enjoyed this session with Jerome Walter, let him know by clicking on the link below and sending him a quick shout out:
Click here to thank Jerome Walter on Linkedin!
Click here to let Ashish know about your number one takeaway from this episode!
And if you want us to answer your questions on one of our upcoming weekly Feedback Friday episodes, drop us a line at ashish@kaizenteq.com.
Resources from This Episode:
- Tools & services, discussed during the Interview
- Netflix Chaos Engineering
- Chaos Engineering Book – Casey-Rosenthal
- Cloud Security Academy
[00:00:00] Ashish Rajan: [00:00:00] Welcome Jerome! For people who don’t know, you should know this, that you’re awake at 5:00 AM for us. So I really appreciate that. Thank you. , I think everyone really.
We need so for people who don’t know Jerome, who is Jerome apart from this awesome energetic guy who wakes up at 5:00 AM and is ready to serve his community.
Jerome Walter: [00:00:22] Awesome am I Energetic guy, I don’t know, but who is Jerome? A , French guy working security,currently , working for VMware, on.
Everything we call modern application platform. So every thing that is cloud native platform. I hold an interesting title, Security Modernization. That’s really, that’s revealing on the fact that, basically I’m an advisor to customers. And if you look at what modern applications and cloud native platforms are doing, So they’ve changed the way we just changed the way we do it.
Has it enabled DevOps liberal or [00:01:00] a modern ways of developing and modern application patterns? And if you’ve done okay, come that there is a lot of opportunities for security by using ZOS, by using carnage platform, by using new technologies that allow us to do and resync. So we would do security. And my role is there just have been customers.
Usually when customers come to us, they are in their journey of transforming the application or that at depth thing and adult team DevOps practices. And so I have some trickled it down in terms of reviewing how we do security, but a lot of the security we’ve been doing for years is fairly traditional.
Like following your standards, making sure you have those controls in place and other things. Yep. And you just have to open a newspaper and you see that doesn’t work. Right. it doesn’t work and you just have to really send it to the developers. It unsecured each realize there is a shadow is a fight because between the different teams.
Yeah. There is opportunities in, in modern [00:02:00] practices to, to fix this. And that’s where, where I come in, where I am customers going and yeah. Bridging the gap between security and the rest of the world. Perfect.
Ashish Rajan: [00:02:11] And, we definitely need people questioned the traditional security as well , and I think it’s good that people like you exist, right.
You’re able to kind of convey the fact that. It’s okay. To be different. It’s okay. To not have something work, which is traditional. Let’s move on to the next thing. , I think the best example to this is that if we would have always continued using traditional parts, we would not have a car. We will still be riding horse carts.
. So somebody just needed to help in the transition. Then I’m glad people like you exist to help in the transition. the first question then from starting from this space is where does security kales in Jannie for people who don’t know. and I know it’s a very broad question. So what’s your definition of it?
Jerome Walter: [00:02:53] So you had Aaron in the past. So your probably had those three. Our definition of is three [00:03:00] hours picking like 10 times faster and, yeah, security goes and drinks is a, it’s an emerging practice and that’s really, tailoring on what Chaos Engineering is in general. If you think it’s a general, would be as a practice it is a practice of consciously testing that your security works by injecting failures into a tool by injecting potential interest into it.
Then making sure that, you don’t believe that your controls, security controls work, you actually test it and you fail dates. The fact that you’re able to detect and respond to potential issues. that’s the theory of it in practice. If you look at it, it’s really, and that’s why I am so sure. I’m such an advocate of it is that if you look at the human aspect, today’s is a gap we have between security between the two is the other teams and the practices.
The modern that taxi advance attack are usually not picked up by a sock, [00:04:00] or by traditional aesthetic attack. You need to understand what your application is doing to be able to find the anomalies and the way of making sure the different teams know how to work together, know how to find and spot, anomalies is to destined, to investigate infested.
So making sure that you create confidence. You created a confidence in your team and in your organization so that people can be bold and know that they have the, I want you to pick up the signal and then they try it. They can try to experiment with the risk of wrecking desecrated security controls. And
Ashish Rajan: [00:04:34] these, I they related to chaos engineering experiments.
Is that what we are really testing? And what is Experiments for people who don’t know experiments.
Jerome Walter: [00:04:43] It is a subset. If you look at callosum engineering was really born of the, with the goal of increasing the reliability of the system and it injection failure to make sure that one your system is able to withstand potentially each failure on itself, but also your teams are able [00:05:00] to pick it up and respond fast.
Right? So he’s creating these, He’s testing your muscle and making sure that you’re trained as an organism and not the system itself. And security goes in general. These are playing these principle of continuity failure, injection and experimentation on the system to the security outcomes. And if you look at a lot of the models, Oh, security practices, we’re moving towards, you know, DevSecOps security champions doing your threat model, or to be able to tailor what security controls should be.
We see what the obligation is supposed to do. Security injuring is natural is about, you identify 95, what you expect your application to do. What else? The security outcomes of your application. And you test this hypothesis by, not by making sure that and actually do it. So you will do expect some security controls.
You would have some potential attack, attack vectors, and you will test them by, like injecting basically, Potential failures into
Ashish Rajan: [00:05:58] it. [00:06:00] So I guess your point examples of this could, for example, be, I don’t know, your service should not be accessible on port 22 or four 43. What is one of those examples?
I’m just thinking,
Jerome Walter: [00:06:15] cause that wouldn’t be the most basic, but that’s not really a cows. because it’s just as well as captains. That’s what everyone does really, but let’s take an example of, That back, if you look at, for example, copy that one started. We just, yeah. A misconfiguration, someone trying to is someone stealing credentials.
That was another thing. So, you would start with identifying, what does it potential ways to get into your, to the crown jewels or you get to choose your objectives and you just started with, okay. Someone creates an account or are you important to detect if someone creates an account? Right. So. you eject the creation of an account in your own environment.
So are you able to pick it up, but also if someone texts suddenly you have a nodes it’s suppose to is [00:07:00] that you have credentials, it is not supposed to be used by just not Oh. You to detect that someone is accessing this credit service credential from somewhere different or on the different aspect. So you wouldn’t look at it, all the patterns and try to identify and inject.
Instead of just doing it Sierra ticketing, but you actually creates a . So you basically best so that you’re able to pick up any potential musical progression, any potential attack? Yeah. By testing it.
Ashish Rajan: [00:07:32] it’s a great example if I’m able to create an account and I’m able to validate whether I should be able to create an account, or is that something running, which is, which is running continuously or is that something like a point in time experiment that people run?
Jerome Walter: [00:07:46] Yeah, so usually in the same philosophies that, Galison Jungian, you don’t. But you could be, expecting it to do continuously, but you also want to give them Slack to your teams. so it’s mostly run as experiments or [00:08:00] the game days. So you pick a day where everyone is aware that you’re going to do today.
And, but you, you, because you don’t want to disrupt your production too much, something goes wrong. You need people to say, okay, well, let’s stop the game. But that’s really about everyone he’s alive. I would say let’s do a, let’s do a game day answers. They will connect yeah. To, to control the working and you have predefined scenarios and you experiment them.
So even when he’s aware, you don’t want to do that in the back of, In the back of people, that’s really a wine that distincts that change today. E and if you look a lot of modern security practices, the changes, it’s not the blame game, right? Their goal is not for security to find problem. And then you’re, you you’re bad.
Right? It’s a one we learn is the outcome. And we learn with a controls work. Well, what it does, it could be [00:09:00] improved. So everyone is here to learn and everyone is here too. To build confidence all the time. Right? So when you start, you would start very small and over time you build confidence. So you do more advanced, advents, Gable, experimentations.
And it’s really a very important aspects that it’s not a blue vs red or anything. Right. It’s, let’s go, let’s sit together. Let’s try to improve our knowledge and our experimentation. And obviously what the application is doing. One of the side effects of cloud and cloud is complexity, right? The, I talk about value to you very positively, but this would be very unethical when you change your mind, 60 or a hundred different microservices.
The, you show the effect of all the interaction makes it very difficult to apply a static security, right? So we do need to have teams that know very well. All things work, we’re able to respond to. [00:10:00] is that something we are going on when you have very complex system? Yes. It goes to engineering. Is this aspect of say we want his product team or the team that is behind this application too?
Not only no. When the note goes down when performance is done, but when something security revamped, it happens. We want them, and we want everyone to be able to pick it up a response. And you would have interaction required from. Product team being able to say, Hey, this is not normal. So this could be secretly let’s pick up the phone and get an expert to go into it or actually respond to it or the method that’s easiest, flex muscle that you’re trained to do.
And it doesn’t happen if your security controls, these are blame. If, if you just say you’re bad, you, you missed a controller. If you act like no one will ever trust you look at it positively. Right. So, yeah, it’s really, yeah. As well as a practice,
that’s
Ashish Rajan: [00:10:58] that’s a perfect segue [00:11:00] into a question that came from Arun
, how do we balanced between putting security controls versus developers convenience.
Jerome Walter: [00:11:08] Hi, if your security controls and let’s be honest, you’ve got security controls makes the life of your developer. Hell. Then they will never implement the controls and they will always find a way to bypass it. Or at one point that means you disagree control might not be the right one.
Any of it, you understand that the goal of security is not to make sure that nothing happens. Right? So we’re in an organization for the, yeah. For the business to flourish to serve the customers. And if your control, it goes against the business and. Maybe there’s other ways to do it. what we’ve seen through experimentation.
And, I may be from via drones through, through a company called people, told us that he is very, a pioneer in dev ops and Tonia and extreme programming. What we’ve seen is very often the same technology and the same practices that makes you look up productive actually [00:12:00] makes your security better.
Yeah, we don’t degrade the tools. And so that’s really where you want to see. Is there another way of doing your, your security controls that doesn’t go up against developers? is the developers nowadays as the one generating, I mean, for your company, right? Or the new services you’re developing, you’re trying to compete with or start a startup.
If you make their life. how and SMOs, and you know that the startup will take up your business in an extra year. So it’s really about finding the balance, but you wouldn’t see, I don’t want to go into the details, but very often the same tools and practice that makes you have the robust protein actually could be used for improving security.
Ashish Rajan: [00:12:38] Is there an example that you can share? And I think the reason I’m asking is because there’s another comment that came through that day. Who’s a regular listener as well by the developer, a piece of first before dropping a hammer.
Jerome Walter: [00:12:51] yes. Yes. Whether it’s a hammer, shouldn’t be on the developer should be usually when you have, when you have problems, it’s [00:13:00] rarely easy engineer is probably is.
As a doctor, as the organization is toxics and management has a wrong objectives, or you might have just. Very practically, very often. It’s just the objectives of security and developers are not the line. And there is no commitment of OKC is always a key. It seems where we need to be improved. we in people thought we had, we had a motto on our own internal cyber hygiene.
We call the three RS. It’s a good example of whore, Mitch you’ve halves. Right. We couldn’t just, easy ABI teach you repair to repave and to rotate diversity repair. Meaning when you find that averaging, how quickly are you able to fix it? And if you look at a giant and it’s all about making developers able to change code fast, but I made you to change goals also means a bat to fix things, right?
So if you work in a culture where people care about, it’s a great , [00:14:00] but you also have them. Develop faster and release faster. and well, there’s a number of things in the backend and you actually increase the speed at which you can fix monkeys. Yeah. And that’s, if you look in, develop it’s all about improving, there is a, the developers practices so that when they push comes into there’s something to production, you have enough there so that they push and they have a very low chance of breaking something.
Right. But the reliability of change. He’s one of the most important factors in DevOps, but I mean, as you revive me to have change, also give the confidence that you can put in a bag of patch without breaking it. So we have customers we’re able to apply the patches, even telling developers, I guess there’s always, there is a new version of Apache launch to production.
They have so much good desks that they are. They have a 1999% chance of. The government was at breaking and distinct. And so it’s all fully automated. And when you work towards making that [00:15:00] efficiency, then you have the developers, but you also have security. So that’s, repair. If you look at the ability to repay would choose instead of just thinking of server and just changing the code, adding things to it, just destroy the rebuild every yeah.
And that’s where microservices in cognitive is very useful. You run several nodes of routine is immutable and stainless. And you know that you deploy a new version of the app regularly several times a day. You just, instead of you just destroyed from creating UVM, recreate diversity. When it gives you from a security perspective is yeah.
Mixtures that you have no matter where into it, you’ve gotten you haven’t been bridged because you’re just three today. And so that gives a lot of benefits because that means if you do it regularly and you increase, you reduce the chance of having been breached, you reduce the attack window for that short moment.
But also that means that the day you have a breach, your secret your incident team can go and say, okay, let’s go to the data. And until we find. What was the reality of the zero [00:16:00] day, or we could bridge let’s repave, regular, you know, you can’t do it, right. So this is Brian rebill until we know how we got bridge, then we fixed it with empty.
Then it gives a render the confidence because it makes sense, increasingly explanation, the, expensive for attackers, right. And of those three RSPs rotate. And that’s really a very, very important, factor today. Increasingly bridges have to do with his credentials being sort of being served as code or credential or user credentials.
And if your automated or your pipeline so that you can change the credentials regularly. Yeah. All of them, you have two benefits. First fees. Usually that means the developer never needs to have the credentials of your database or your in the first place. You’ve just generated by the platform. so you remove the risk of malware B on the developers, workstation and the store, but also send means that if you rotate it every week, every month, so of someone [00:17:00] finding it and using it within this school, the window is very low.
It’s very small, right? So that’s an example of how you can use what the developers, so doing to actually improve security without buying additional security control. That goes in a way you can actually use the same practices. To reduce, to reduce your explanation. If you look at, there we go. Yeah.
Verizon repaired. There were yet five to 98% of Richie’s have to do with those three controls. if you do that well, if you do that regularly as a, as a list, do it regularly as a normal practice. Yeah. You actually reduced massively your risk of bridge and you haven’t just put anything. In the ways of developers, you actually have them because they say, Hey, let’s make you more productive.
Ashish Rajan: [00:17:44] It’s really interesting. That. You mentioned that the three Rs and rotating credentials, do you feel there’s a way, and this kind of goes back to the cold concept that even DevOps always expect of security to be part of it, right? There’s no.
Architecture that doesn’t talk [00:18:00] about security. Just security sometimes ignored by some people, but to your point, as you’re developing the three yards for rotating credentials, or it’s just a little, those little things can be put in in the beginning so that you don’t have to go back and go, well, that’s great.
Do you want to go to production tomorrow, but we don’t need, we need to put this in. So do you find, it’s I guess had nation had the foundational pieces set before you can kind of three yards, or is that something that. Like what, what’s an example. I mean, I guess you’ve mentioned pivotal. I think a lot of people, I don’t know how many people don’t know.
I would imagine everyone knows about it, but they, it was a really interesting company and what they were trying to do as well as it still exists. I believe talking about in their past tense was I loved the fact that, do you find it was a, this is an example of maturity that you can share. And I think, it would be really interesting to hear a simple example maybe that you can share of this.
Well, you were able to, if you have one I’m trying to, I may have put you [00:19:00] on the spot, but if you have an example, that would be awesome as well. No. I mean, let’s be honest
Jerome Walter: [00:19:06] is three things, simplicity. It requires a certain maturity to it. it also requires a certain mindset and a certain culture to it.
And that’s really what people always about that still exist. It’s just above and beyond where now? Yeah.
Ashish Rajan: [00:19:23] Sorry.
Jerome Walter: [00:19:24] Yeah. If you look, I mean, we have a number of customers. We’re very vocal about the benefits, those practices doing. And no one would tell you yet it’s you can go ahead and click, right? Well, usually when we talk about repaving yeah, won’t do that.
If you’re still running with it, physical servers or bare metal servers, dude’s stolen by hand. You wouldn’t reinstall regularly when you work on cloud a cloud technologies and creating a VM is two minutes, right. if you work with application platform or a moment platform, then deploying the application on it and rebuilding the [00:20:00] application on top is a matter of minutes as well.
And so yes, you consider it. You didn’t do that on never seen, it requires a certain level of donations that is linked to your DevOps capability, right? And so it requires pipeline. it requires. Cloud technology in your backend, let’s be very clear. Cloud native doesn’t have to be public cloud. It can be on premises or is doable anywhere.
So it requests something, but also it requires a certain maturity of developers practices. test driven development is very, very, a very big factor of, of push code who’s at breaking anything right. If you want to change several times a day without breaking every two times you push something, you need to have very good hygiene on testing and, making robust, close rates.
And so we go through this journey with our customers. we have customers, we’re open about it. if you look at, A Wells Fargo. So you actually [00:21:00] go publicly about it’s on holiday or to make it all the patching. so they have no human involved in patching now, and to do is repaving very regularly, several times a week.
Is this in devalue in terms of regions when an incident happens, but also from a security perspective? yeah, even defense sector, USF or something you’re pretty vocal about also his practices to them and hollow. is it costly? Great, more secure as in the, the old ways of changing, putting the air gap and taking six months off paper review, 4,000 pages of documents.
any change
Yes. So that’s, it’s interesting because very often when we, when we look at security, we’ve had years of security as required to control as a control function. but blaming the banks for that. But, as a control function, central require people to document and to write and to explain what they’re doing.
What’s it going to be an external party? We would go and approve it. But this actually goes into the way of developers [00:22:00] being. They want you to change fast rate. So, yeah. eating a number of things and it doesn’t happen in one day. It’s usually,
Ashish Rajan: [00:22:06] , we’re don’t need the mindset as well, to your point that everyone believes in the same mission.
I always find people are like, ah, , I don’t really like this, , thank you for taking the example. I think it’s pretty good because, there’s one other question, which is interesting.
Is there an element of pentesting insecurity, chaos engineering.
Jerome Walter: [00:22:23] Interesting. Is
there any of pen testing in red teaming in bug bounties and . You could say yes. as long as you so actively testing your predictions, she seemed to try to find a fault. yes, except that here is the infection is from the inside. so you, you inject internally. You’re not trying, I mean, you have privileged access already.
but yes, it would be, trying to reproduce the. They have the hand of an attack. So you could, you could say yes. it doesn’t, it’s not [00:23:00] the board’s candidate. No, just that, but yeah, you’re testing was a UI born to detect. It’s mostly about your testing, whether you’re able to detect if you’ve been here breach what happens.
Right. So, yeah, just doesn’t handle, pentesting. And, and, there is, Gentlemen and I am that was named and apologized to him as the founder of signal science. We used to be the CSO of,
Ashish Rajan: [00:23:22] Oh,
Jerome Walter: [00:23:23] yes, exactly. And so, yeah, he framed it very, very well, on the either approach with the bounty, right. It’s not about finding a prime mumbling.
Diversity’s about. Improving your capability to detect, and improve from that , it’s the same aspect and it’s complimentary to, and desperate bounties is really about continuously testing your environment to be able to know. To finding problems, but also even David, to create confidence, you want your teams to be confident in their system so they can be bought so that they can change things right.
Instead of black, or I don’t know if I change that I might break something you [00:24:00] want everyone to say, okay, we know that if we change something and if there is a mistake, we’re going to pick it up and worry about your respond and fix it within seconds. And sort of developers can be Boulder platform teams can go and either patches automatically.
I was updating developers on negotiating because they knows they’re going to pick it up, fast. Rachel, you take the application, you rebuild it with the latest version of Apache and then you’ve fast. Right. But it requires a certain confidence into it. And this is all complimentary it’s what makes principal Bendis and, it was a, your here to say, Hey, we’re going to find problems and this is wrong.
We’re going to block you from being too free training for find something. Always like this, this year for an experiment for us to learn and improve. Right.
Ashish Rajan: [00:24:41] Yep. Perfect. And a good answer. That’s going on? I think the way I probably, you the, I guess the nail on the head, I guess, for lack of a better word where, I guess all of it is around the fact that you can, I have to think like an attacker anyways, and Ben testing is the same security kills experiences are the same as well by the people who are joining in.
hi, Paul [00:25:00] and hi, Gerald as well. I was going to ask them, I did not know this, but you guys were watching, this can hit the Lake or the share, but nobody, I didn’t realize there was like these emojis in there that you gotta get, feel free to hit those so that LinkedIn can know that you guys are enjoying this.
Just saying that just a side note because yes, or even YouTube as well for that matter. Moving onto the next question. what, so you kind of spoke about the use cases as well. Now, if some of these people who are listening to this, I think hopefully are on God has answered as well. So some of these people who are listening to this, and I think I need to bring in my, a use case, or I’m going to do, I’m going to make a program for this.
Where do I start?
Jerome Walter: [00:25:42] So that’s the a million dollar question, dates, Jason, Sierra ticketing. You can do it anywhere. You can do it small, your baldness on making a program out of it would be very linked to a certain number of organizational behaviors. Firsties. [00:26:00] Your product owners so that if you look at DevOps being product teams, fully owning, whatever happens to a specific product or a specific application, so they need you, they need to have a common understanding that security is also part of that, right?
If you’re trying to, to build this culture of DevOps or a calcium generating. But they don’t take ownership of security outcomes as I approach things. And it’s going to be a NAZA, we’re going to break USA and you are going to believe you, and that don’t work. So as long as you have these understandings, that product teams are here too, own and Z life of their product.
Including ops, including security, and that security brings an expertise or knowledge of what does the different attack scenarios and other things then can have and compliments that we visit. Big time to stop then afters it’s always starts more one or two applications that, you know, you have a number of [00:27:00] patterns already.
I would say just would be the three hours. If you do that already, that gives it a certain number of confidence in your, in your ability to withstand and respond to SIS and then builds these flex muscle, that stuff with small, small scenarios that are not too complex. Creation accounts. Are you able to detect it?
Are you able to detective suddenly your content pops up in your environment and then you can be more specific on the, on the, on the application itself. And then you will see that as you build the confidence, you will be able to, to spread it. secretly champion, this would be very important because you do need to have a link between, between security and the developers team, but also you need to.
As you will take one or two obligations and learn from it. You need those people to be able to share with the rest of the argument or the organization to build their understanding and then not at trade. today’s there is no, I mean, it’s not a book, we’re writing a book about security because in general, but it’s not something you’re open and photo [00:28:00] or the procedures, because you’re trying to build a culture of experimentation.
So by definition, there is no book about how to experiment, But that’s really important to be able to create those things and inject and learn from it. At the end, end of the day, your security desk is, would be linked to what your application does. And the threat of that is the track model of this application.
So it’s very unique to your business. So that’s why it’s important to start and, and build and build from there. One recommendation. Don’t try to do that on the S 100, but we we’ve had questions like, Hey, can they do that on my core, on my core banking system that runs on the ACE pondered Sierra to get a, you can, but then honestly you will have less flexibility of injecting and recovering, and saw less confidence in Egyptian ferry.
you know, that’s a system. So start with the things that, you know, you can repay fast and that, you know, you can handle failure, risk [00:29:00] for you. Right.
Ashish Rajan: [00:29:01] I was going to say, would you do this in production? And that’s probably one of the most common questions I get about chaos engineering, where are we running these experiments in production?
And that would require a very different kind of approval for a lot of people. And maybe that’s why you should charge as Monica Fetterman. So is do, why didn’t you just do this in production or can I start in dev or staging
Jerome Walter: [00:29:21] then move to production? Where as in anything, would you start with, there’s a shotgun shooting.
Get there.
If you look at the goal, the goal is yeah. Three, you have the confidence that you can do it in production, right? So eventually you would want to run it your production at least to make sure that your systems are coughing and can handle that gracefully. you should probably start with a prediction if you’ve never done the cows experiments, that would be wise.
but eventually yes, you should, you should work your way towards being able to go to, to get your productions. I’d say I know [00:30:00] some customers who have. who were running it as a pre product validation. So when the runs the, on the, on the testing environment, when there is, they would run some callous experiment to make sure that the application is able to withstand that it’s more of a continuous vacation, that, that case, because it’s pretty fine, pretty fine scenarios.
So they wouldn’t just, yep. Inject failures, non failures, regardless. So the application business logic, but at least to create the ownership for the product team to say, if you want to go do prediction, the application needs to be able to withstand this kind of stuff. Right. And so that’s a good step, a good step in creating, and creating a baseline of, of resilience and security, a secret your controls, but eventually you would want to be able to produce things about.
Not pretty fine about data for the vacation and that are in production. don’t do that until you have a certain level of confidence into it.
Ashish Rajan: [00:30:54] Perfect. answer. And I think that’s a good segue into a question that came from Jalapa as well. I hope I pronounced that [00:31:00] correctly. hi Jerome.
Thanks for bringing together. Awesome topic and join this, question is chaos engineering and dev sec ops the same. If not, what is the difference?
Jerome Walter: [00:31:09] well, everyone who knows me knows that I’m going to have to bash on DevSecOps.
Ashish Rajan: [00:31:15] Well, let’s start with, what’s your definition of what’s your hatred towards DevSecOps about, I noticed that in the industry where people are like, it’s a great marketing term, but, what’s your, what’s your, what’s your story with DevSecOps
Jerome Walter: [00:31:29] and then
It’s it’s important as well. To me, there is no DevSecOps DevOps dumped the right way, because as you, when you do DevOps, we want to cover compliance risk and all the aspects and your finance aspects of you want to look at all the other functions so that your product use fully autonomous and able to deliver outcomes that are not impacting them that are integrating the rest of the organization.
Right. I was away to the beginning of your pastor production. You will have all [00:32:00] the hurdles. so it’s dev startups to me is really about getting people to every, every comes from function should facilitate the job of the developer when they diverse something compliance has taken care of, security’s taken care of risk is taken care off, and sort of, so you don’t have to have those long assessments and all those things of how you compliant because you know, it’s already built in, right.
So that’s more, made you have what. That startups is, that’s it. A marketing terms have a good value of now everyone older people who were kind of avoiding the topic, have no choice. You too. Take two greedy. Is there a success in a certain framework that comes into it? Any of, it started as basically security vendors trying to say, Hey, you can automate our tools in your pipeline.
but two days kind of, just kind of, okay, so, back to the question, East coast engineering and DevSecOps the same, [00:33:00] ZL not the same deal. E, I would say you can do it separately, but then DevOps is about the curvature, no commission, of, you know, DevOps is about getting product teams, autonomous and able to deliver outcomes and on from their systems and then from the customers and
customer demands to provide better services. Right. Kelson generating these experiment on making sure what these delivers is able to withstand potential figure, being security failures. they’d work out with aspects. You have. So chemist engineering could be seen as a subset of DevOps in a way as a, as a practice.
And I would say it’s in most organizations is bubbles SRE, which is one of the implementation of DevOps. So they’re very, very linked. Zola, you could do a cow’s engineering experiment without the dev ops. I’ve mentioned the year. It’s kind of dangerous if you don’t have donation. I wouldn’t stop there, but that was the same, but [00:34:00] very much about, I think.
Ashish Rajan: [00:34:02] Yeah. Perfect, great answer as well. I hopefully that answers your question Jalpa and I think it’s a, how do I say it’s taboo term in security, but it’s definitely something which is DevSecOps called at the time, started off as like, what, why aren’t we doing security in the first place? Why do we have to call it out?
But now I think to your point, so that vendors can get special. I guess because certain areas of devils were just being forgotten. So this is a great way. I think we I’ve even heard of deaf synopsis as well. There’s obviously these desktops, as you said,
Jerome Walter: [00:34:36] it’s interesting because in I’m at one of the grief I have about DevSecOps, is that in spenders, we’re trying to go into pipeline.
DevOps is not only about death each office as well. And in the DevSecOps model, there’s not much about the house, right? So he’s not much about learning and improving costs constantly. And that’s really where your calcium during yield, but Dantes. [00:35:00] And, it was a culture we put into play these at the end of the day.
Your team that runs your DevOps that runs and, run your system and it should be able to pick up any security issues into, into their system. Right. And so it’s a learning experiment. It’s about testing and creating just nerd learning. So if in what, they’re not should be, they have an art or the horse life cycle.
Cool. So then yes, it goes in during cigarette, it goes into journeys of their start ups. in the current definition that have been framed by most of the security vendors, which is mostly about just fine, said no secretly Gelsinger is not part of the pipeline. so he just mostly house, I think insecurity, we forget.
is that a life? The product starts when he goes to Prague. Yeah. You won’t get breached before you go to prom. And so still today, a lot of the, a lot of the controls are like [00:36:00] predefined barriers before you go to production. And we’re very, very loose and very, very weak in a sec ops. still, unfortunately even we develop serious student of scripture of
I can go back home and do my stuff. If you’ve ever seen the sock that knows call your business application, PA what is, what is the normal behavior or not? That can make a difference between a brute force and a guy who made a configuration mistake. Yep. Then they’re already pretty advanced. It’s not the case in, in many socks.
And that’s the bottom line of the issue here is there is this gap on, everyone’s saying the other one who’s taking care of, of security and. I’m an advocate of cows engineering because you can’t do. Well, let’s see cigarette. It goes into an angle without bring these, boasting together. And so,
Ashish Rajan: [00:36:53] yeah, it’s probably, it could be that bridge that can combine, maybe become an opportunity with developers [00:37:00] and security can work together as well.
I’ve got a few more comments coming in.
So going to the comments, Paul, is like, Oh, here we go. yeah. I love it as well. It’s like, there is no divorce without security. And this is a part of my favorite one. he said there is no death. There’s no divorce without security, but the problem is that lots of people claim success.
Jerome Walter: [00:37:21] Look at me, I’m doing the DevOps.
Ashish Rajan: [00:37:24] And if you’ll take a closer look, security is not a priority. So by using the term DevSecOps, you are emphasizing that the need for security has. Book ended up by Devin ops. And I guess the thing that he mentioned, dev kit cube, playbook, guardrail security by design ops. It just put everything in there.
So
Jerome Walter: [00:37:43] yeah.
Ashish Rajan: [00:37:45] Oh yes. AI is as well. AI ML is kind of what’s missing now. it’s, it’s funny. we’ve been talking about the chaos programs as well. Have you, I guess. if someone’s been doing some of the basic [00:38:00] experiments already, like none of the ones that you mentioned, where you have, you already have a tread model established and as part of the unit test, they may be doing this already.
They’re like, Oh, maybe after listening to you, they might say actually cloud native slows is, can allow me to do this even more. natively. Is there a, I guess the maturity level to this, like how do you know that you are mature? And is there an example of a mature person then? Yeah, he just realized ML, dev ops.
That’s like, yeah, that’s fine.
Jerome Walter: [00:38:32] On this one, the pin on this one and react to Paul’s comment. Everyone claims to do DevOps. And if I had to cut a, you had a very interesting the definition of agile. . So it was a very testing as well is, when people are saying, Hey, I may add drugs. Like, depends are you, can you push the button and take your whatever is in your code report right now and put it to prod.
And when you press the button right now, if you’re able to press a [00:39:00] button and send it your product without breaking anything, then yes you are. and. Going back to DevOps and the a and the sinks. that’s why he loves those three RS, because these are very simple things, but if you’re not able to destroy everything and rebuild a, deploying a new version and a whole stack of Sam young and really doing DevOps, right.
if your product team doesn’t own. See, the delivery and the outcomes of your, of your, then I’m doing DevOps. So I have a number, I made 12 metrics of what secret the outcome you expect from a developer, from other product teams. And that’s kind of a way to look at it, right? when you find them, how long it takes you to fix it is a good, good benchmark.
Right? If you, if you say I’m doing developer is to patch every two months. Yeah, maybe that, are you a bunch of destroying reveal, but do you do that regularly? how long is your workload living? Right, so long [00:40:00] Gary lives and that’s a change of mindset, right? The longer it leaves or logos, or is a change, there is something managers into it.
And so we moved from what we call the text to nurture for years to a cattle where you killed your product. So is ACC kind of, Seems you wouldn’t look at in terms of measuring, whether you’re really doing DevOps in the, in the early to make weights, you can still do the ops in the traditional ways.
That means improving delivery, improving, automating, but at the end of the day, if you really want to measure your maturities, there’s a certain aspect of those three RS that you will get into, you will need to look at. So that’s for a false comment. And then that remove the pin on. You mentioned about a use case.
If she wants to go out on the use case itself with a shameless plug, there is a book that is about to be released at all right. Is it the, there was a number of companies were doing calcium urine. you drew the, the [00:41:00] ones who do secretly go. So generic already doing cows in drain. So you just have to look at who’s bragging about a cows journey.
I know some of them make customers doing it. I’m not sure I can say that they are doing it. In, amongst the, as a company that we know about doing it a very open executive, I mean, for the company who started , he goes to generic as well. there is a number of financial institutions, in banking.
I know quite a number of, of sows who were doing coast and during that are starting to run some security experiment into it.
Ashish Rajan: [00:41:36] And to your point. so how do you measure the maturity of it though? At what point would you say, or this is like, when you look at it and like, Oh, this was a pretty mature chaos engineering practice.
Like, is there a point that someone can say that?
Jerome Walter: [00:41:51] Well, first is if you stopped injecting faders production, that means you already have a certain level of confidence into it. [00:42:00] I think there’s some tangible markers into it. yeah. Yeah, but I mean, it’s a number of experiments. Getting Brendan was at breaking scenes, a radius as a member of majority, it’s a primary series of various you bolt on intangible marker, which is the confidence of your team.
And if you look at the definition I was giving about HR, Oh, you have what you push, push your code base right now with your production and breaking nothing. I need to do it from a platform perspective. Are you going to rebuild completely your environment? If I asked you to rebuild that different data and rebuild and move.
Audio production from here to there. It’s a number of, I mean, it’s a Highmark, but that’s a confidence level, right? So you want to eventually all of that is to build confidence and to build it. It builds a relay, building a reliable system, one thing, but you also want to enable your business. I mean, your developers and the rest of the [00:43:00] business to go faster, to respond to the demand.
And that’s the intangible factor. in. Those are the ways I would say copied. Wasn’t very interesting. it covered was a very interesting case, for full IHI. and those markers people wrote looking at alcohol. Do we know whether they’re on a job, whether that’s simple, if your company was able to, was trying to. Struggles. And how do I get people to work from home and all those things, then you’re not really agile if your company was okay, our customers need to have new services today because of this situation, because they are at home, they need a different services.
How do we get to see what’s the need? And how would you develop those services to capture this moment? Right. That’s what HR is and what the goal of an organization needs evolve. And, in it goes in
but dependability is that. Can your customers depend on you when there’s a crisis? It’s not the only one you’re, there is a crisis on you, but when they are in crisis as well. And so all of these [00:44:00] reliability engineering, including Karastan, generic is all about how do I create dependability? How can, can I be relied upon when something happens?
And that’s really easy, is this market right? So you do have a number of. small markers along the way, but the crown is really on. I might able to keep up my business to capture as a moment of a crisis and to change product and service, depending on the environment. If you are immediately lovers, have the confidence, if your secret eating doesn’t shallot and saying, I have, we have some interment empties.
That means they’ve worked enough with the developers to enable this. And so, so on and so forth. Right.
Ashish Rajan: [00:44:42] As a, I think it’s a question that came through just probably related as well. How important is organization culture? When an organization becomes agile or DevOps,
Jerome Walter: [00:44:53] if you’ve read about dev ops and you know that you, the anagram use cabins, you know, and the first one is [00:45:00] future, right?
Yes, if you are, even the DevOps reports of Dora CD very well. You’d just look at the old West from model. If you don’t have a culture of, if you have a picture of shooting each other and security sees that you’re bad and developers trying to avoid security because they are in a way. and yeah, you, you have a problem.
You’re gonna have a hard time doing DevOps. You’re going Apple to make the number of sings. But that means that the day you have sings out of the books, sings would go, are going to go away. And chaos engineering is all about out of the books, right? When you don’t have an incident procedure, when you don’t have something that.
He’s already scripted. How is your organization responding? And we’ve seen it. I mean, I think there’s even studies about it. It’s all about, organizational resilience. She’s all about the future, right? It’s the nephew is the IBG of people to, to work together. In a very traditional ways, you [00:46:00] know that when you have a secret, the first few hours of use Bay to soft to create context, like what’s a normal coffee duration of this where’s this binary supposed to be there.
Yes, no. It’s like, when does the obligation doing, is it normal that this doing that and it’s unfortunate, right? Because these few hours are what you should be. planning, accurate responding to the incident and coerce engineering creates this knowledge, because if you’ve already done a number of drill, if your product team’s already knows how to do some things and can work with security experts on saying, okay, here we sing Gary something, and you go and do some forensics.
then you have a very different ways of handling incident and a faster ending incidents that, that knows why he is. That’s why one of the. Mike markers for maturity is your time to detect and time to respond to incidents. Right?
Ashish Rajan: [00:46:51] That’s really interesting. And I think, sorry, just to quickly add that as well, because she point, it’s also the maturity of [00:47:00] DevOps as well, then in a lot of ways.
Well, I don’t know if I use the word devil’s word more to your point. About how quickly do you respond? It’s kind of. That probably is a good marker to go that way. Can I, how quickly can I detect is probably the first one. Cause . That’s hard as well. That’s not easy, especially how big the organization is.
Jerome Walter: [00:47:18] Yeah. And that sorts of is a reason why Carol syndrome is so important to me is we keep saying, yeah, that is important, but we don’t test it right. A time to detect an outage. If you don’t pass it, or a many banks have an obligation to recover services because in one hour of flow, as you know, you have deal RTO.
But we never it. See, you’re just praying and praying all your home. The date happens and you recover within the polo is always in one hour. unless you test it continuously, then you don’t know. Right. And so calcineurin is one of those practice of not only besting it, [00:48:00] but creating the, reducing this time by just interest in failure.
So you first, you first. And you first inject figure figures that maximum would create is as you can recover within seconds or minutes, right? they won’t break anything so you can just come from us. But eventually what you want is to be able to have a team that is able to deal with any side effects and reconfigure and go back to normal or to a better solution if needed.
because it knows the system, whether it’s, it knows the security wetlands, it can react to it. Yeah.
Ashish Rajan: [00:48:36] And that’s probably a good segue into, I guess, if people hearing this and they want to start like, Oh, I want to start chaos experimentation today. Is there, is there a need for them to know tread modeling?
Cause we kind of spoke about quite a bit about threat modeling as well. Is there like what should they be looking at studying? I’m sure they can go to your book. They can go to Aaron’s book and like, there’s that as well. But while [00:49:00] we’re waiting for that, what, is there anything that they can learn? To kind of start working on this, just to build the foundational knowledge for it.
Jerome Walter: [00:49:09] One of the book is not yet out yet. So
Ashish Rajan: [00:49:13] while we wait for that, while we wait for that book, is there something that they can, should they do threat modeling or
Jerome Walter: [00:49:20] straight learning is a very, very important aspect of mobile security because it allows you to tailor your control for what your application is supposed to do instead of just.
Putting black, pre-canned a secret yellows. So if you want to do advanced scale experiment, yes. You will try to tailor it to the application. So it will kind of be linked to what is logical or what makes sense for this application. You wouldn’t do a, let’s do an SQL injection on the application that doesn’t have a SQL.
Right. but you can start, we can start with very simple stuff, right? you can stop these. I’m able to detect. If someone change there is a firewall configuration, [00:50:00] I’m able to detect them somewhere. it creates an account and I have my son coming in like easily. Is it a bit sleepy? It’s seven. Three now
Ashish Rajan: [00:50:15] I probably don’t think you belong as well.
No. Well, you can make it, bring them online, Mathew wine, , don’t literally happen.
Jerome Walter: [00:50:21] I won’t give him coffee, so, Oh, fair enough. Yeah, you should
Ashish Rajan: [00:50:24] probably shouldn’t like do to your point. You can start learning today by just starting the small experiments. I think a quick, quick comment from ’em. Paul has again, it’s like cheating on your taxes and hoping you don’t get audited. Pretty much sums up that on point, my friend.
Jerome Walter: [00:50:43] It’s like hiring a lawyer to test that your defenses are good enough that you can explain to the tax office in reasonable ways that you had a reason for doing it this way.
Ashish Rajan: [00:50:56] That’s true. That’s true. Like, Oh, but I, I had my lawyer look [00:51:00] into this. So it should be fine.
Jerome Walter: [00:51:01] I mean, it’s a very big common, if you look today where security is going, we do more and more crisis management drill, always executive management, where we, you know, hammer isn’t because you need to fix this muscle on being able to respond to those things.
But we hammer the executives on surgical scenarios and paper and tabletop scenarios. You also need to hammer your, your, your teams. It’s not about, you know, Being bad or something when you prep your CEO for process management is not, you’re not telling him you’re bad. You’re just making sure that it becomes better over time.
And that’s the same as a team, right?
Ashish Rajan: [00:51:37] Yeah. And I look at that as a fire drill light, everyone knows what a fire drill is and that we all do fire drills. I guess when we were all in the offices, we were doing fire drills when there is like, I mean, there’s, there’s all fire. The whole point is not because of the fire, but the point is when there is an actual fire, you’re able to respond to it effectively.
Right? This creating that behavior.
Jerome Walter: [00:51:59] look in [00:52:00] buildings also come and we will come and put some smoke on top of the detectors to make sure that all of them are working. They do announcement regularly to make sure that people can hear, so you, you need to test all of these, right. And that. In a way, like I was explaining is you would start with those testing each of the controls in individually until one day you take a small group, a smoke grenade, you throw it to where you see housing happen.
You wouldn’t know that on day one?
Ashish Rajan: [00:52:29] Yeah, probably not in the office and be bothered there as well. I guess that’s that’s production. That’s like production experiment where you just throw a smoke grenade inside a office where people are there
Jerome Walter: [00:52:42] of what you are aiming towards. making sure that even if you have a fire in one building or something, it doesn’t disrupt your business.
Right. You organize. So that. Even if you have some of this stuff or you’ll staff on the, on the stairs decades of building your approaches, it’s not going to go down. Yup.
[00:53:00] Ashish Rajan: [00:53:00] Yup. Perfect. And we kind of look towards the tail end, off the show as well. I’ve got some few questions, I guess I’ve worked three fun questions for you.
it shouldn’t be that many. Just three. What do you spend most time on when you’re not working on chaos engineering or technology?
Jerome Walter: [00:53:16] Ah, very good. when I’m networking on technologies, this one. Yeah.
I have a, I have a personal issue with, we do security, today, so I have a few bet. Perfect. bed. Perfect. I had a few companies as well. We were seeing color. The candidates are, is the right way. quite important to stay up to, up to speed and trained, to read sings that they would normally read that we didn’t normally read.
huh.
Ashish Rajan: [00:53:41] What are you reading?
Jerome Walter: [00:53:44] I mean, we know that they’re none of those, experiments it’s about, you know, Avoiding the bias and experimentation, you know, that human organization of it. reinforcing beliefs. Yeah. And so [00:54:00] I’m trying to challenge my own set of reinforcement of beliefs. it’s hard, but, you don’t want to create a dog, a dog, a new dog, man.
Right? It’s Or this agile DevOps go in general are very useful. It’s not applicable to everything, and it can be useful other things as well in the same mindset. But you, you, you also need to make sure that you’re mindful of not coming up with, like, this is a. This is a practice rate or the Bible that we have.
Otherwise we create what we have today. DevSecOps. We choose to do that, but they’ll take, you need to have your
in a bank vein. That’s not really, that’s where you need to be able to, to, to flex your muscles to be a little bit more.
Ashish Rajan: [00:54:46] Yep. Perfect. Perfect answer, man. And I love the fact that you’re reading. I needed to develop them myself as well. It’s so hard. what is something that you’re proud of, but not only social media?
Jerome Walter: [00:54:59] Oh, [00:55:00] My kids. My
Ashish Rajan: [00:55:03] awesome. And I think we should have probably gone over, had an intro with one of your sons, I guess, but, yeah, definitely a great answer, man. What is your favorite cuisine or restaurant that you can share with people?
Jerome Walter: [00:55:12] cuisine? Well, I’m married to a Japanese or I have a slight bias towards Japanese cuisine, so I’m French.
So I do like French cuisine. I I’m pretty open minded. my wife and I, we really liked it and he says, well, so flavors you have in those. So that’s a type of cuisine, restaurants. our favorite just do them today. A restaurant I went to in Hong Kong called bio innovation, very experimental, very experimental cuisine, mindblowing, and really it really enjoyed.
It’s a unpretentious, but very amazing food and very. I used to in all those things. So that was a very good experience, still exists. And I hope that is just go there. difficulty book. So we’re not the only ones to like them, but
Ashish Rajan: [00:55:59] right. [00:56:00] Okay. Last probably book in advance before you go to visit Hong Kong, I guess.
So if someone is watching from Hong Kong, just let us know what you guys think of that as well, but, where can people find you Jerome on the social media world?
Jerome Walter: [00:56:13] Yeah. And that’s that much of a social media guy. People can always LinkedIn me, can email me, Twitter. I look at it once in a once in a lifetime.
I know I should be better at that.
Ashish Rajan: [00:56:24] Yeah. You and I both just
Jerome Walter: [00:56:27] doing Facebook and I don’t even know what else the other social media,
Ashish Rajan: [00:56:32] wait,
Jerome Walter: [00:56:33] I work on deck, so I will not have to I’m of a generation gap was my kids were like, no, I can do half this generation gap. So that was the best ways LinkedIn still. I, I liked the fact that it’s super professional and, a little bit less, balance and what you get as a comment on Facebook.
So, yeah. Yeah.
Ashish Rajan: [00:56:53] I mean, the, the community is an example of it as well. Everyone who we have here is. either on LinkedIn or YouTube, [00:57:00] but I think we’ve been able to create a community where a lot of people are supportive. And as you can see from the questions as well, we’re all in a similar mind space about certain times, especially DevSecOps
Jerome Walter: [00:57:13] community is perfect for security goes from Janine like constructive mindset of not blaming each other.
This is a, I mean, this is a core of what DevOps is, is a building. Yeah, you’ve created an amazing community. So I
Ashish Rajan: [00:57:30] appreciate it, man. Thank you. Thanks. And thank you so much for coming in. I’m sure a lot of people got a lot of valuable information for at least going by the questions that came through. So thank you so much for taking the time out.
And I would Anchorage. We will reach out to you on LinkedIn and hammer you all the security, chaos, engineering questions, and experiments that they can be running.
Jerome Walter: [00:57:50] That’s smart to be a is engineering,
Ashish Rajan: [00:57:52] Ooh. Restaurants as well or more?
Jerome Walter: [00:57:56] No, no, no. Is there is more about Sigrid? Is it you [00:58:00] security practices, than just cloud architecture, a number of practices and a lot of value in.
When I go to DevOps, which includes objects for the volunteers, red teaming and all those things. so, I’m happy to have people, you know, reaching out, but also that behaves, this works, this doesn’t work. we’ve tried. We failed. there is nothing worse than save reinforcing by just looking at success, success stories and ignoring the failures.
Yeah. so I want to hear from people who say, Hey, this is a good way to ask a question, but also just emits. This is okay.
Ashish Rajan: [00:58:37] Perfect. And I would encourage people to reach out as vocal Jeremy’s is super, super nice. Dude has been like that from the day I’ve been speaking to him. He’s been super helpful. And Kelly, thank you so much for waking up early as well.
Like now finally, it’s 7:00 AM your time. So appreciate you woke up super early to kind of come on the show though, and I’m looking forward to kind of having you again and probably talk about it. Some of the other cloud native [00:59:00] aspects of, of, of, I guess, security as a, as you’re maturing in that space.
Thank you so much for coming in.
Jerome Walter: [00:59:05] Thank you. Have a good day, everyone.
Thank you.