Ashish Rajan: Hello, and welcome to another episode of Virtual Coffee with Ashish! And for those of you who are joining us for the first time, we are here talking about what the heck is CI/CD and, today’s episode, we’ve got Melissa Benua . But before we go into it, if you’re a first time, I feel free to hit the like button hit the subscribe button.

If you’re watching on YouTube, that’s where the audio version would appear in. Once we finished the audio.

Without further ado, let me just bring my guests. Hi, Melissa How are you?

Melissa Benua: Hi. Good. How are you?

Ashish Rajan: Good , before we go into, the, our history. so for people who probably haven’t heard of Melissa Benua, what was your path into this engineering world?

Melissa Benua: Yeah. so I came from a university that really emphasized software testing as a discipline of computer science. with, you know, if you notice our testing, it was under campaigner and Whitaker, but a bunch of people who, if you’re in the testing world, do you know who they are?

anyway, so I went from there to Boeing for a little bit, found that Boeing boring, doing, doing a software art test, not test automation, but [00:01:00] developing a virtual systems so that people could test. tasks control, mission control software, went from there to Microsoft started as an

rolled in from to, combine engineer slash developer when they started playing with this whole combined engineering fad slash universal truth.

Now. However many years ago that was, on a, on a product you may have heard of it’s called Bing.

Ashish Rajan: Oh, of course. the, the, the search engine that I always use,

Melissa Benua: I’m sure. I’m sure everybody’s definitely uses Bing all the time.

Ashish Rajan: The funniest thing was when, when people started paying people money to start using Bing, that was my saddest moment for Bing

Melissa Benua: like.

For my X-Box slash subscription with being rewards. It’s fantastic advertise, but

that the people who use being the most are Microsoft employee. So Bing is fantastic for looking up.net tool for looking up things about.net. Then that’s the,

Ashish Rajan: that, you [00:02:00] know, you’ll be surprised actually, because I’ve made friends with some people recently who are heavily in the.net. And that’s true. I did not know that.

So that, so is that a, that a thing that being is probably a better one.

We were talking about Bing and what was after Bing

Melissa Benua: so after Bing, I did a bunch of stuff I’m being, I shifted over to work on Cortana.

So I did some work on Cortana before Cortana was named Cortana as a part of, here, two audio fades with a slight delay.

Yeah. So they went to Cortana, was part of, we shipped windows phone, was part of the shipping team for X-Box one, the original one, worked on, the web socket. Yeah, they could the original and I still have a white one. It’s very fancy. X-Box

Ashish Rajan: well, you have one of those antique ones. That’s pretty cool.

Melissa Benua: Oh, they’re not going to give me a free one this time. It’s very sad for me. We joke. That was the whole reason that I went to join the X-Box team with just cause I wanted to be given a free

Ashish Rajan: Xbox. Oh, Oh, we could go into so many layers over there, [00:03:00] but I’m just going to hold myself up for my next conversation with you.

Melissa Benua: Oh yeah. So I went from there and so I did a lot of deep back in engineering. and then from there I went to join a startup doing, gaming services as a platform. I was there for a little while. It’s back at engineering it did all kinds of things. and that was where I started doing a lot of continuous integration and delivery.

and I’d been exposed to a lot of practices on Bing, Microsoft actually quite modern. so forward thinking practices, we were doing that. We were doing devops before devops had a name and we didn’t know what to call it. It was just the thing we did. and so I took a lot of those practices to my startup.

and then from there I left from that startup to a, to a bigger startup where I’m at now called mParticle. and I’ve continued bringing some of those practices here, right? Rolling. not just testing, but security and all kinds of other good things that you need in order to be able to ship code safely and securely and

Ashish Rajan: quickly.

Yeah, that’s awesome because I keep thinking that, you’ve had a really good, interesting history right. In Microsoft. And then you’ve been doing, and by the way, [00:04:00] I have huge respect for the gaming community because I feel gaming is the that’s the industry, but everything started even from security perspective.

Like that was an era where, you got, the gaming industry had to figure out how do we not. Get affected by cheat codes because everyone loves cheat codes and we won’t get more one extra point and they had to figure out security even before like cybersecurity or the thing I feel. So, I mean, sorry.

Melissa Benua: Yeah. It’s a hard problem.

Ashish Rajan: Yes. Yes. A hundred percent. A hundred percent. And I get surprised every time someone talks about it as well. So what is another, because we have the cloud audience over here. What does cloud security mean for you?

Melissa Benua: So for me, cloud security is a couple of dimensions. The first one is making sure that we haven’t shipped that we’ve done our best effort to ensure we’re not shipping insecure code, right.

That we’re not an insecure code. Doesn’t even make it into our main line branch, which is we’re putting a lot of checks early as early as possible. And then the second part [00:05:00] is assuming, okay, well we did our best, but nobody’s perfect. So let’s make sure that. If we do ship something insecurity to production, we’re going to know about it before somebody else does.

Ashish Rajan: Interesting it just to add to that. so a hundred percent, we should not be shipping any codes that is not secure and modern deployment practices have evolved. Like I think, I feel like it’s, it’s only been a few years but it feels like its been there for Yonks. Right. But I’m curious to know from your perspective, where does more than when people say modern deployment practices and they throw words like.

CI/CD trunk line deployment as a security person with some insight into this, what does it really mean for people who may not have heard this before?

Melissa Benua: Yeah. So continuous integration is the act of building and testing every single change, right? There’s no such thing as whoops. I broke, I landed my change and I accidentally made all of our code not compile, right.

That doesn’t happen. You may do it to yourself, but you [00:06:00] don’t do it to anybody else. continuous delivery is making sure that every change that just compiles and passes tasks, but is then put somewhere on, have deployed somewhere automatically. So it can be tested. doesn’t mean production just means in some way that it can be tested and continuous deployment is then every change that lands into your main line branch is automatically deployed live to production.

Ashish Rajan: Oh, so that’s a subtle difference. Cause I took deployment as well and go into production, but is there like a sequence to it? Yeah,

Melissa Benua: there is. So could you start? I mean, so the sequence is in when you’re, certainly when you’re starting implementing any of it, right. You always start with continuous integration because there’s no point trying to deploy garbage to production.

When you don’t know anything about just garbage, you don’t want to, you want to make sure that your code compiles, right. Step one of any quality, any quality pipeline just does your code compile.

Ashish Rajan: Yep. Yeah. A hundred

Melissa Benua: percent language.

Ashish Rajan: Interesting. Right. Because I imagine that this large scale integration is there obviously [00:07:00] clearly a lot of challenges and we’ll definitely dig into that a bit more as well, it’s always an interesting one for me because every time I’ve mentioned CI/CD, and you’ve mentioned this part as well, where, only good code should make it to production. Like what. Role does security play in your mind? And in some of these kids, like, I mean, you can go into each one if you want, or you can just give a holistic view if you feel like.

Melissa Benua: Yeah. So security is, is there should be right. first party citizen in the whole, CI pipeline. Right. Cause you can’t. You can’t ship code that doesn’t compile. You can’t ship code that doesn’t pass tasks. You shouldn’t ship code that has obvious vulnerability, whether it’s in vulnerable package dependency or has a static analysis issue.

Like you shouldn’t. You can’t ship any of that code. It’s easy to find out those things. It’s not like requires, you know, five PhDs and a team of 30 find new things out and its relatively fast. And so, no. you know, there’s not really a lot of excuse in this day and age to not be checking the basics.

It does know [00:08:00] that a lot of value in trying to do something esoteric and look for really difficult exploits. If, you know, for a fact that you’re using a super vulnerable package, that’s been exploited, actively being exploited in the wild right now.

Ashish Rajan: I’m going to touch a few nerves for security folks over here.

Does it mean in the new world? We security folks need to know how to code.

Melissa Benua: They don’t have to, I know plenty of security people who don’t, they probably know how to run a tool though.

Ashish Rajan: Oh yeah. no good to know. I was going to say like, where do I draw the line? Because every time I mentioned that, Oh, we might have to a bit more technical and not offense to people who make diagrams.

Right. I make diagrams myself, nothing wrong with it, but people’s ended up sometimes take offense to the fact when I say, ah, we kind of have to be a bit more technical, and have an understanding, but for people who are listening to this okay. CI/CD great. but what about, Starting off just by saying, where is like a super mature model of CI/CD like, are we talking like multiple deployments, what does that [00:09:00] look like from your perspective?

when you say modern deployment practices, how many times are we deploying? And it’s still security playing same part over there as well.

Melissa Benua: so an underpinning of any modern CI/CD practice is automation automating. I don’t know, just, I’m just doing tests. automation. I mean, everything should be automated as many things as possible should be automated and reliably automated and relatively quickly automated.

because the, the number one way you’re going to kill your CICB efforts is if you have a lot of manual pieces that rely on a human to do something, cause humans are slow. Compared to a machine and machines are really, really fast.

And so the underpinning of any successful CI/CD system involves automation, it means automating your testing, automating your integration, testing, automating as many, you know, as much of your security testing as possible, automating, you know, your infrastructure, your infrastructure set up and tear it down.

If you need to automating, not just having alerts, but having those alerts be automated, hyper, sorry, not just having metrics, but having those metrics be automated into alerts that [00:10:00] route to the right people. ideally having rollbacks automated if possible. because there’s just not time and every second, right?

Every second that you do a bad deployment and you’re, you know, God forbid your site is down or you’ve got major functional degradation. If you’re at scale, that’s hundreds of thousands to millions of users being affected. And so time matters and a human’s reaction time from the time of human hand, the alerts, you know, the metrics start rolling in and takes, can take three or four minutes if you’ve got a lag.

So the time a human is looking at the graph and the humans pattern recognition recognizes, Oh no, this is bad. And then the human goes and pulls up the page that tells them what to do. And Oh no, this is bad happens. And then the human has to connect to a machine. It has to type things. And then another human that’s the sign off of what they’re doing, right.

That can take a lot of time. Oh, it’s an automated system can go. Boop, boop, boop, boop done before. There’s a problem.

Ashish Rajan: So ready to start a scifi movie thing and we can, we can actually see this happen in real life where this seems just go one after the other. And it just happens [00:11:00] automatically.

Melissa Benua: Absolutely

Ashish Rajan: excellent.

Melissa Benua: A lot of times the biggest one that people say, I don’t want to do continuous deployment because I want to control when my roll ups happened because I don’t want one happening. Whenever in the middle of an ad campaign or in the middle of you on black Friday or whichever, which is fine. I don’t actually view that as a blocker to continuous deployment because realistically speaking, as long as you every change in your main line, branch could go live at any point in time.

it doesn’t really matter if each and every one literally does.

Ashish Rajan: Interesting. I’ll let other folks ask us questions on this as well, but I definitely feel it’s an interesting space, right? Where a lot of people feel that multiple deployments are offered for a day is always just a startup thing. Like you don’t see this in a mature organization, but I think Microsoft people, yeah, there you go.

That’s right. No, but that’s the thing though, right? Because people think that it’s for them, like it’s not [00:12:00] for people like us because we clearly have different problems to what those guys have. even though the scale is immense. So if I were to ask you, pealing a few more layers onto this, what are the, some of the building blocks?

Like someone who’s listening to this. maybe have one deployment every six months into production or one deployment every three months in your production. What’s your recommendation for what are some of the building blocks for this? , I think that’s a great question because that would, add another layer to, this is a question from Vineet is continuous monitoring part of CI/CDs security.

Melissa Benua: It is, absolutely it is

Ashish Rajan: as well.

Melissa Benua: I don’t think it would it would be continuous monitoring so much as monitoring.

Ashish Rajan: it’s the same thing, the same thing. It’s just, all right.

Melissa Benua: continuous monitoring usually implies that there’s a machine on the top end. Running alerts behind it because otherwise monitoring is a human looking at a TV that scrolls, you know, like with tickers or like the matrix.

so that would be non-continuous monitoring, I guess, because you don’t have humans

Ashish Rajan: all the time.

[00:13:00] Melissa Benua: but there’s really, so there’s really four dimensions you have to sort of think about when you’re. Thinking about transitioning a code base over, look at, at your source control practices, right? How mature are your source central practices?

Are you using, you know, Are you using branches? Do you have Gates on your main branch? Do you how do you make a production release? we used to look at your quality practices, right? How much automated testing do you do? Is it reliable? Does it run quickly? Does it run with every change? Does it run with every fifth change?

Is it all, some manual are sitting in a bathroom somewhere and you’ve got to wait on their reliable reports. deployment practices, right? How easy is it to make a deployment? Is deployment some dude tech taking a zip file from his machine and SSHing and the 15 other machines. And that’s what are your deployment is, are you containerized?

Are you running, you know, running Kubernetes, and finally monitoring, right? Do you have places adjust just as a state, you can do a machine and tailing, a log file. Do you have aggravated logging? Do you have metrics? Do you have alerting on those metrics? do [00:14:00] you have systemic organized metrics that you can make, systemic organized alerts or is it everybody names, whatever they need whenever they, so there’s, there’s really across those four different dimensions.

There’s different maturity levels. And then depending on your levels, you can determine how quickly you, how quick, how reasonable it is to deploy it with certain frequencies.

Ashish Rajan: Wait, so you’re saying all, all our problems can be solved by four pillars.

Melissa Benua: Basically, you know, there’s, it’s, it’s a little bit of a simplification and that those four pillars

Ashish Rajan: are massive.

I know how to drive, but depending on manual automatic, like this complexity, but to your point at a high level, yes. Sorry.

Melissa Benua: Yeah, he’s going to drive a semi.

Ashish Rajan: Yeah, that’s right. You don’t want to drive. but, but to your point, so as long as these four pillars have kind of been, taught off or taken care of, you should at least have the building blocks to start off your journey on doing continuous deployments, or continuous integration as well.

And I’m sure it goes into tools and everything. a [00:15:00] question that came from you on Jay? it’s more like great topic. So can you do CI/CD without knowing how to code?

where would someone start with that?

Melissa Benua: there’s a couple different ways to start. You can start with some open source solutions on Github, right? That they wrote the code for you. And you can look at what you know, what they did, and you can fork.

Their repositories Github actions, for example, is a free CI/CD system. you can go see how open-source flashy projects have set it up. there’s a lot of trainings and tutorials on YouTube. I actually am giving a training next week at a conference, which we’re teaching people how to set up a CA CD pipeline using a stock, using a stock Github refill thats on my Github

so they don’t have to write the code. The code works it’s, you know, it’s a silly application, but it’s the point is that it’s, it’s a backend slash front end application that. Goes through the CI/CD stages. It has unit tests and integration tests. It has a pipeline. so I don’t think you need to know how to code, you need code, but you don’t have to come up with it, like, cause you can’t say nothing, but you don’t have to write the thing as you do. [00:16:00]

Ashish Rajan: I’ll definitely recommend get GitHub I feel it’s like a treasure chest of so many people’s knowledge I feel like there’s so many people with so many great projects there, which are open source is, you’re just good enough to just leverage some of that and use that as a building foundation for what this could look like, or that you might have to vet, if there’s enough documentation and kind of layers as well, but I’m not going to go into that.

So The four pillars where we kind of said, okay, this is kind of what the basic pillars you need to cover. And I feel there begs a question as well because there’s some organization whom where I’ve gone. Oh, I’m only doing cloud first. Paul just mentioned that Bitbucket pipeline with GitHub actions. There’s so many complexity layers to this as well.

Melissa Benua: Yeah. We can argue about providers all day long. Right? I have a friend who swears by get lab, get lab is the number one way to do it? Another one does the best for you. I don’t like it. I don’t like the last changes they did to it, but. You know, there’s of providers circle C I will do it. Travis CIO.

Ashish Rajan: It’s [00:17:00] the same as Android versus Apple.

you can meet people. They would definitely have separate camps. And one of the hardest

Melissa Benua: versus PHP versus Ruby, you know, we’re

Ashish Rajan: Amount of Slack conversations that I’ve been part of it where people are just deciding to do, or should we do node JS or react? the big secret is we kind of had to go react because we couldn’t find developers for node JS at that point.

So we had to go for a guy that was like, even though everyone decided it’s gonna be nodes. Yes. But then we couldn’t find a developer and we had to go for a Ruby developer because that’s what we could find at that point. if they go cloud native for some of these deployments, Hmm.

And I’m sure they’re not mature enough to say as a GitHub or any other provider, which is our source control with all these integrations already there. but is there some thoughts on that, like it’s easier to start off on or are you guys using yourself as well? And do you like it or not like it personally?

Melissa Benua: So I’ve used Azure Azure service, for example, integrated really tightly with Azure Devops it’s. Okay. they pushed a recent [00:18:00] release to it lately that I don’t. It doesn’t work for a of problems last night. So I’m like freshly annoyed about it right now that I had filed, AWS has an offering as well.

It’s called code deploying. I it’s fine. You don’t always get as much control when you’re using some things. So when you’re using the cloud service providers, bonds, they generally tend to work really well. If you’re all in on just within that cloud and you’re doing a language that they primarily support, we don’t have any weird exception cases or any strange things you want to do, but as soon as you want to customize, usually it’s not as easy to do.

Ashish Rajan: Interesting. because even for people who would have been in the cloud journey for some time, because even with code deploy code build, they didn’t really come up a better version. They was still Beta for a while.

So all these people who had been writing code for 20 plus years, they probably should not just drop everything and move it from GitHub to a code deploy. Definitely not. But I think that you’ve raised [00:19:00] a very interesting point. whichever path you choose. You probably want to look at whether you already have a 20 plus years of code hanging around.

Do you want to just let go of that and move to a, probably a bit more restricted world of, cause it’s not the cloud’s main business is not doing deployments. Right

Melissa Benua: for the hosting, right. It’s all incidental to get your code up and running on services they can charge you for.

Ashish Rajan: Yeah. And I think, we may have a question over here.

what do you see as the difference between GitHub vs GitLab? in case someone doesn’t know what they are,

Melissa Benua: they’re competitors basically. There’s I don’t want to oversimplify this saying that basically the same thing, but they’re very, very similar source control providers that also provide some level of CI/CD system and some level of issue tracking.

So a friend of mine uses GitLab a lot and I use GitHub a lot. So we have interesting discussions. I definitely can’t tell you which one is better. It’s going to

Ashish Rajan: depend

I [00:20:00] think the way, the way I’ve explained it. And, you can add onto this as well.

Melissa cause GitLab is you can install it. You can deploy it locally inside the environment. Whereas GitHub is like a SAAS service as well. There’s a lot of that stuff. Oh really? Oh, there you go. Like don’t tell me that was the update last night that you are annoyed by.

Melissa Benua: So GitHub is own by Microsoft and GitLab isn’t, just slightly different ways of doing things. Different pricing models. GitHub is considered the industry leader. So they’re a little more expensive. but they provide more services, which you may or may not care about is I, I can’t really recommend it’s good.

It depends on your use case and you want it or not, and what’s your budget is and how many people, and if you’re going to be forking, there’s so many

Ashish Rajan: I was wanting to touch on something else as well. So we talked about the fact that, Oh, okay. So cloud obviously has limited capability in most cases for, from a deployment perspective.

They’re definitely trying to go into that part, but. When transitioning [00:21:00] say, we spoke about the building blocks and now if someone’s doing six months or seven months deployments and transitioning over to multiple cass or when I will have heard of cases where even if a deployment goes in and it fails, but it’s still deploying.

The changes that are going in into multiple times in production? Are they all big changes, small changes, high risk changes.

Melissa Benua: it’s the smaller, the better at big changes or we’re almost always the most dangerous changes and the bigger they are, the more dangerous they are.

Ashish Rajan: what do you do for scaling then? cause it sounds like big changes should go through a manual review every day. If I, if I can dare to say it,

Melissa Benua: it’s not even like that.

It’s more that you ship big, change it when you’re shipping a big change, right. You ship it incrementally, right? Say you’ve got a dev team and they’ve got six months of working on some huge new feature. with CIC, it was a true CD system. You don’t. Have them hold the [00:22:00] whole thing and aggregate it for six months and then released like 300,000 lines of code upon the world.

And everybody screams and whales and, you know, wrens their clothes. Instead, you ship it in little pieces, chat, preferably behind some kind of feature flag or some kind of gating. or in some small, incremental way that they don’t. So that by the time you’re done with six months, you end up in the same place.

You still have the feature ship, but you shift a thousand or 2000 little, like 50 line changes, a hundred line changes. And each one of those changes was tested. And then, you know, checkpoints, as you released enough pieces to make up a major functionality set, then you tested those major functionality sets, but at least, you know, that you didn’t break any distinct scenarios.

And all the time, if you’re shipping these little tiny changes, And then you can task the bigger changes together and aggregate to make sure that they’re doing what you think they do. so it’s not one heroic effort at the end. It’s just a lot of very measured and well-tested and secured pieces going out, very frequently.

Ashish Rajan: is this a change in a behavior in [00:23:00] how code is usually developed or is this more, like they need to learn a new, different kind of, Deployment methodology, like in some I’m thinking from a developer’s perspective, , I’m a developer listening to this and like, I have some interest in cloud, obviously doing cloud security, but, I’m a developer as well.

What does it really mean from like the skillset perspective? these smaller changes that putting in. My ID is also integrated somehow in doing test testing or like, what does that look like? I mean, obviously I totally, I don’t have an exact example. Like what’s the most generic version example that we can go for, I guess.

Melissa Benua: So it’s the mind, it’s certainly a mindset shift and a cultural shift for an engineering organization in general. Right. Especially if you’re one that’s used to shipping heroic efforts and then you’ve got, you know, three week long integration periods and then you’ve got bugs flying in and out of the integration period.

Great. If you use these long, massive development cycles, it takes, a lot of habit breaking and a lot of retraining to think of how do I break up? how do I break up this? You know, I got hit, did this big feature, [00:24:00] this big change. How do I break that up into small digestible pieces? some of it is tooling support, right?

We need to be able to support being able to run frequent tasks, make the changes worth their while. So a big issue. I see. happening with org they’re first considering this is that they’re build and test takes an hour. So it’s not worth the developers time to spend two hours writing code, put up a 15 line change, and then it takes two hours to get a response about if the code is good enough for that.

Nobody wants to do that. so the way I approach it is by making the behavior that I want really easy and making the behavior, I don’t want really hard. So when I started with my team, I said, I made sure that our builds that compilation build and unit tests never take more than five minutes. Five minutes.

because most modern languages are really fast. And if they’re not fast enough, I just throw a more expensive machine at it because what a high, like a high CPU count machine and AWS costs me like a hundred dollars a month. How much?

Ashish Rajan: Oh yeah.

Melissa Benua: I said, make sure they [00:25:00] get results back in five minutes. Cause anybody wait five minutes, you can wait five minutes for something. so that they’re not discouraged from doing lots of small rapid changes because that’s only five minutes. and I made changes to our poll request policy as well, where I, because the reality is if you’re right, if you’re reviewing other people’s changes, the more complex they are, the bigger, those changes are the longer it takes you to, to review them.

And it’s an exponential scale. I can review like 50 lines or less in like 10 minutes. It’s not that hard. You can get it. You can get your head around 50 lines of code, really easily. a hundred lines of code. I can read it like half an hour, 200 lines of code. It’s going to take me hours to get the whole thing through my head.

And I’m probably going to miss things. And more than that, like I can’t get through it. I know that I will miss things. Absolutely. A hundred percent because I can’t hold the whole complexity of that in my head. You know, I’m sure there’s some people who can, but most people. Yep. Right? So their full-time job is not just learning all this, knowing all they can learn about this one change.

Right? You’ve got other things going on. Most [00:26:00] people can’t hold that whole context in their head long enough to provide meaningful review, and find, you know, meaningful issues. So I set guaranteed review times. If your change is less than 50 lines of code, I’ll review it in the next hour. If your change is more than 500 lines of code, maybe I’ll get to you this

Ashish Rajan: week. .

Encouraging.

Melissa Benua: I made all the tools work really well for small changes. I removed as many roadblocks as possible for small changes and they made the big changes, more difficult with require different requirements, you know, different requirements for checks, different firing for people’s times.

and then we encourage a culture of small changes, breaking things up. We made the whole SDLC focus on small changes.

Ashish Rajan: Interesting. We definitely use a lot of words and I wonder how many people actually know about the SDLC. It’s basically what is a software development life cycle, [00:27:00] Melissa? Like what does that really mean for, for grads over here who listen to this?

And

Melissa Benua: so what that means is it’s thewhole way we ship code from the minute that a product owner comes up with a requirements, a need, right? We need to ship this new widget to the requirements of, this is what the widget should do. to the technical aspects of the widget seem to be built in react JS, and they need to have a dynamo DB backend to breaking that apart into deliverable pieces.

Well, first we need to learn how to talk to dynamo. And first we need to figure out how to make widgets, and then we make the specific widget we need, to figure out how we’re going to test those widgets, right? How are we going to test the connection to our database and how are we going to make sure that we can test, you know, that widget show up on a page, through to deployment through to, right to the developer writing code, how the developer writes code.

and then how they ship it, how they go through review. Right? Who reviews the code? how many reviews does it need? What does review look like? What do we expect to find and review? What are the automated checks? Right? What are the security bars it needs to pass? What are the quality bars? What are the [00:28:00] monitoring bars?

and then how it goes live. And then once it’s live, how do we make sure that the widgets are good and our customers like our widgets? How do we make sure the widget isn’t killing the performance of our feed? We shipped a widget. Now our page takes eight minutes to load, a software development life cycle.

Ashish Rajan: Yeah, I love how you explain it because I, it kind of also shows the importance where sometimes security tends to get, forget that aspect of it.

I definitely find myself guilty of doing this in the past where. You talk about changes into say, Oh, this cannot go into production or that cannot go to production, but some days are totally fine with, without even realizing that, Hey, what this really means is our, our app is going to load 10 times slower.

And the customer experience is going to be really, for lack of better word, it would not be pleasant or put MFA everywhere. Yeah. Great. We should put MFA everywhere for staff and everything, but does a end user really care sometimes? Like, is that a single sign on option? That we can go for it. [00:29:00] Like, you know, I

Melissa Benua: think your box is one you can’t log into.

Yeah,

Ashish Rajan: that’s right. Finding it like definitely, a lot of people are changing their mindset about this. They’re definitely saying no, there should be a better way to do this because I mean, at the end of the day, security is still a cost center. We’re all trying to all, we’re doing a security people are, we know what the risk is.

We’re just making people aware and we’re saying, how do we manage this risk in a way that we don’t harm our customers’ experience. That’s pretty much, I feel like that’s the definition of it. These days,

Melissa Benua: all of software engineering is really risk management, right? What is the risk for users? What’s the risk to our business?

What’s the risk of not testing this code? What’s the risk of spending too much time testing this code. What’s the risk of. You know, not monitoring, that’s the risk of deploying once every six months versus the risk of putting a bunch of investment to try to deploy every day. And it turns out we can’t, this is all, it’s all about risk management.

Ashish Rajan: And I, I find, I find it funny, right? Because when people talk about the fact that, Oh, [00:30:00] security is everyone’s responsibility, it is a risk management thing that everyone’s looking at. Like everyone, every time you deploy something, which has mistakes, five minutes longer, You’re risking the customers, you’re risking losing customers.

There’s all

Melissa Benua: major outage turning a minor outage into a major

Ashish Rajan: outage. Oh my God. I think there was a case. I remember this, someone was talking with this, example of theirs where automation is super great. But it’s not great. If the, if the whole thing takes, like we were talking about a container, which was a hundred MB container and I’m like, Oh my God, that’s a massive container load every time because, and they were doing daily deployments as well.

And that container would take almost half an hour because all these dependencies and everything. and this is by the way, they moved off from a five minute. deployment window to a half, an hour deployment window, because of the dependencies has got at it.

So exactly to your point. Great idea to add all the features, but if it slows down the entire process, it doesn’t really make sense at that point. [00:31:00] Yeah.

Melissa Benua: High high-level metrics like having guiding North star metrics to keep in mind as you’re doing this stuff helps. Right? Like that’s why I have my five minute build time.

My building unit test times, no matter what I’m doing, never switch out over five minutes. Not worth, it has to be another solution. Right? I can’t make primary building tasks go over five minutes. I can make subsequent jobs or I can make extra jobs or I make optional jobs, but I can’t make, build the unit tasks go over five minutes.

Ashish Rajan: Interesting. I love the theory that I’m definitely gonna, use it and talk to people about it because I feel, like because if we scale this to exactly the point every five minutes for every change that you do, but then if at any given point and under 500 changes happening in a company that’s 500 into five minutes, it’s 2,500 minutes of just.

Like, it’s almost like there’s so much, so much nuances to where it’s like, where does it really mean at scale at that point then [00:32:00] obviously there’s only one Melissa.

Melissa Benua: Right? You can absolutely do things that take longer. but after the initial check, and you need to be selective, right? You needed to be really sure that the value they’re providing, the extra time they’re going to take, you know, above and beyond five minutes.

Ashish Rajan: that’s a good segway into my next question, because we’re going to, talk about Nirvana state considering you’re from Seattle. And I did not know that Nirvana is from Seattle, but, what’s the Nirvana for a mature. continuous deployment or much mature software development life cycle.

Melissa Benua: so Nirvana is like, when things happen and nobody knows or cares, right. If you don’t know when a deployment happens, and it’s not because you don’t know anything about them, it’s because it doesn’t matter.

That’s sort of on a state, if you don’t care that you deploy in the middle of a huge demo, to a huge potential customer that serve on a state and those things become a non-issue and non-event event and deployment is just another whatever thing, because you’re at no point or you’re going to harm a customer.

[00:33:00] And, and you know that even if you were, you know, something would have slipped through, it would be caught and handled so quickly that it doesn’t matter.

Ashish Rajan: No. I think the coolest story would be when you’re in a customer meeting and you deploy change straight in your production because they said, Oh, I don’t like the widget that you deployed just before, like five minutes ago, like, Oh, that’s okay.

Let me just make that change right now. And you’d make a change that goes straight into production. Whoa. That could be a super cool.

Melissa Benua: That point actually. Yeah. They said that, just show the strength and value of the deploy. Exactly what we just talked about. They were showing, showing off something in the UI and they were, I think it was an investor meeting, actually.

It was an investor meeting, not a customer meeting and they wanted this for them. I took it as an example. The CTO was like, yeah, let me make a quick change for you. Here did it. we had walked him through the process, how quickly the build went, how quickly they ping somebody, got a PR, got to sign off and then click the button, push the whole thing.

Live.

Ashish Rajan: Wow. Yeah, no,

Melissa Benua: no, nothing.

Ashish Rajan: That is super cool. And, I [00:34:00] truly wish everyone kind of gets to go to that part as well, because that’s kind of where I feel scalability kind of becomes really, much more than just a dream. It becomes more of a reality at that point, because I feel like a lot of people have conversations where there are certain parts that you want to want to scale.

Want to do CI/CD but is there like a. Like people listening to this, right. they’re I’m sure they’re in different parts of maturity cycle. what do you use look use as a measure of metrics for how good it is, whether it’s quality or how do you measure maturity? Like what’s your scale?

Melissa Benua: That’s a good question. So it’s a couple of different dimensions. It’s how many deployments can you do in a week and what percentage of your engineering force is involved in that deployment? Great. Can you do a deployment every day, but it requires a quarter of your engineering team, or can you deploy to a deployment every, you know, once a week, but it only requires like a dude for an hour.

right. So how many deployments and how much of an investment are each deployment? things like bug bounty, bug return rate, right? How many bugs escape and are found in [00:35:00] production by your monitoring tools and how many bugs escape it and founder production by your customers. It’s one thing for a bug to escape in your monitoring tool or whatever immediately into your report.

It within a few minutes, something else of a bug goes out. And the next week you get a report from a customer support person telling you that the customers, a, B, and C encountered this bug. so there’s two diff two very subtly different types of escapes. And then the velocity, right? The velocity of which your development teams can get code out the door.

Right. Is it increasing or decreasing? I don’t care so much about the actual value of the velocity so much as it care about the Delta B, is it going up or you being able to ship code faster or is it going down so taking you more time?

Ashish Rajan: It’s interesting. So quality becomes really important at that point as well, because the more velocity you get the higher, the need to have.

I guess a strong, well, better quality, I guess not some strong quality, but better quality

Melissa Benua: automate quality doesn’t mean automated testing quality means quality. It means right. [00:36:00] How, how well is your product filling your users need it includes is your product secure? Is your product is a functionally, correct?

Are the requirements correct? , is it fast? Is it secure? Is it safe? It’s more. It’s not, it’s not just testing, right?

Ashish Rajan: No. And thank you for reminding me because I moved before moving into the product space. I had the general description of quality. Like the core secure is not quality and like

Yeah. I’m like, okay. Now, thanks for clarifying that. Cause I definitely had, had to be reminded of that. There’s a lot more to quality of in the development world. And you just say it’s all just about secure shipping, secure code we kind of towards the tail end of our, episode as well.

we can make it a course. If you want to like go on deployments. I have

Melissa Benua: done it already. You can find them on my website.

Ashish Rajan: Oh, perfect. I love the blog. I’m definitely going to include it aside in there. Cause I definitely encourage people to it out. I want to ask for people want to reach out to you, where can they find you online to connect

Melissa Benua: with you? Yeah. So, I have a [00:37:00] website it’s queen of code.net. I’m on Twitter somewhat frequently, also on queen of code.

my I’m on LinkedIn.

Ashish Rajan: you’ve got someone, who’s gonna check out the website as well. but I, I definitely feel, it’s, what’s why connecting that people should connect with you and get to know you outside of that, because you’ve done a lot of talks around the space as well.

And a lot of us feel for lack of a better word. The Nirvana would be to be doing multiple deployments a day. Everyone wants to get there, but I love the messaging that you’ve been trying to get it out there that it’s possible for anyone, even with mature code to be able to do it as well. I think it’s definitely a lessons there and I feel like a security kind of would automatically get covered in a lot of ways.

If you do, if you have a massive desk coverage, I think that’s what I’m taking away from this. If you have massive test coverage and you’re doing the right things, it, anyone, and everyone can be part of this, right.

Melissa Benua: Everybody can every it’s everybody’s job to contribute to the effort of shipping code multiple times a day, no matter what your job function is, [00:38:00] software, it is your job to ensure that your software goes live quickly and safely.

Ashish Rajan: Yeah. That’s what, and that’s how all of us get paid, right? people shouldn’t forget. Yes. You don’t get paid by, just. This is secure, yeah, that doesn’t pay my pay the bills. If you just said this is secure, but awesome. thank you so much for your time. Melissa, I definitely cannot wait to bring you back again, cause I, I love the perspective you brought in and I think we definitely can learn a lot from this episode.

So I do definitely appreciate all the gems that we’ve dropped into the episode for me.

Melissa Benua: Thanks. It’s great to be here.

Ashish Rajan: Thank you.

‍