September 11, 2020

Not too long ago, DevOps seemed like a fringe buzzword…

Now, it’s front-and-center.

So, what is DevOps and why should you care?

To answer, I invited Jon Bass, Co-Founder & CTO at Symops, onto the show. Jon’s expertise in the field makes him a perfect tour guide for the exciting — and often misunderstood — world of DevOps. 

Jon explains:

  • How DevOps moved from the fringe to the mainstream
  • How agile comes into the picture
  • What SecDevOps is and why it’s used

To hear this episode, and many more like it, you can subscribe to The Virtual CISO Podcast here.

If you don’t use Apple Podcasts, you can find all our episodes here.

Time-Stamped Transcript
This transcript was generated primarily by an automated voice recognition tool. Although the accuracy of the tool is 99% effective you may find some small discrepancies between the written content and the native audio file.

Narrator: (00:22)

John Verry: (00:25)
Hey there, and welcome to another episode of the virtual CISO podcast. As always, I’m John Verry and with me as always, the buzz to my [woody 00:00:34], Jeremy Sporn. Hey Jeremy.

Jeremy Sporn: (00:37)
Hey, good morning, sheriff Woody.

John Verry: (00:39)
So you had a chance to listen to my conversation with Jon. What’d you think?

Jeremy Sporn: (00:44)
We are talking about your conversation with Jon Bass, right? You’re not referring to yourself in the third person again?

John Verry: (00:52)
Yeah, that’s harsh [inaudible 00:00:54] I mean, I’m not going to deny it, I have a rather inflated opinion of myself, but I don’t think I’m guilty of actually referring to myself in the third party very often.

Jeremy Sporn: (01:04)
All right, fair enough. We’ll move on from that. I did listen to the conversation, there’s this commonality I find with all very technically brilliant people, we have a few of them on our team. When they are communicating with someone that they know is really not on their level, no, I’m not trying to hurt your ego any more than I already have, but I just think it’s fair to say that you guys are in different levels when it comes to this particular topic. Brilliant people choose their words really carefully to keep that conversation approachable for their audience, and I thought Jon Bass did an excellent job of keeping your far inferior brain into the conversation.

John Verry: (01:42)
You’re really playing this one up, aren’t you? So just so you know… And guys, stay tuned for next week, because we’ll have a different cohost on the show. So let’s say if you’re speaking about DevOps, guilty as charged, I cannot argue with the inferior statement. I mean, that’s why he came onto the podcast and I didn’t, but the idea behind the podcast was sort of a DevOps for dummies, and in this case it was me in the corner with a dunce cap on.

Jeremy Sporn: (02:09)
Yeah. At this point, I’ve probably listened to the conversation you guys have had three times and it is crazy how much I’ve learned. Application development and application security, in the security world itself is just its own beast. It’s got its own terminology. The people who live in there almost breathe differently. Internally now that we’re developing our own proprietary tools and I’m in charge of marketing those, it has been very helpful to listen to how developers think about getting their software out into the world safely, how they look at things like continuous improvement, really very insightful for someone who’s more on the business side of that software development.

John Verry: (02:49)
Yeah. You know what I was just thinking as you were chatting there, that we’ve really put together a number of really good podcasts for someone who’s starting to deal with this. And I’m thinking right away, like Jim Monaco and him talking about the software, the security training for developers and Andrew Vanderstock and Daniel Cuthbert. And providing the [inaudible 00:03:10] guidance to developers as well. So yeah, we particularly [inaudible 00:03:15] some good stuff here and I think that’s especially important because increasingly, there’s an adage and it’s true that the world runs on software. And a lot of that software is now being deployed in a cloud model using various levels of the DevOps that Jon and I talked about. So increasingly, DevOps and or SecDevOps, are critical to the security and privacy of the data that all these apps are storing, processing are transiting.

Jeremy Sporn: (03:40)
Yeah. It was very interesting to hear that hardware is now software in many cases, which was something I [crosstalk 00:03:45]

John Verry: (03:46)
Well, that’s really what DevOps is. I mean, in fact, when we were talking about the patents a few minutes ago, but what we’re talking about, we’re talking about the fact that with the click of a button, you are deploying servers, applications, databases, routing tables, firewalls. So yeah, software is a hardware.

Jeremy Sporn: (04:06)
Yeah. And Jon Bass, CTO at [SIMOPS 00:04:09], he is the guy that you want to listen to about this very experienced engineer that does a great job of explaining that very complex world of DevOps to people who just don’t live it every single day. You will definitely walk away with a fresh perspective on software development in today’s world, particularly. He also gives a pretty good history lesson, which I appreciated, and which is really built around this idea of aligning business inclined objectives with software development, which I just thought was super cool.

John Verry: (04:36)
Great. So with no further ado, let’s get onto the show. Jon, how are you, sir? Thank you for joining us.

Jon Bass: (04:47)
I’m great. Glad to be here.

John Verry: (04:49)
So as we always do, we start off easy. Tell us a little bit about who you are and what you do.

Jon Bass: (04:54)
Well, I’m John Bass, I’m the CTO at SIMOPS. We’re a startup based in Boston and San Francisco. We’re focused on helping companies build security and compliance workflows with automation that is perfectly matched to their environment. Our approach is really to create a developer focused, engineer focused toolkit that lets teams take our templates and build workflows that don’t just mostly match, but exactly match the use cases of each organization that we service.

John Verry: (05:31)
Cool. And before we get down to business, I always like to ask, what’s your drink of choice.

Jon Bass: (05:37)
Yeah. You let me know about this. I was thinking about it and I do-

John Verry: (05:40)
If you have to think about it, then it’s not your drink of choice. I mean, if it just doesn’t resonate then-

Jon Bass: (05:45)
Well [inaudible 00:05:47] the examples like this, I was trying to think of… the truth is, my drink of choice is coffee. So nothing my drink of-

John Verry: (05:53)
[crosstalk 00:05:53] nothing wrong with that.

Jon Bass: (05:55)
But I wanted to give something a little bit fun, but the reality is it’s coffee. Anyone who’s worked with me knows that I’m continually getting coffee, making coffee. And now that I’m home, we’ve got a nice espresso maker and that’s part of my morning ritual. And now it’s also to the point where if I don’t make the espressos for my wife, I get in trouble. So it’s part of what keeps me here.

John Verry: (06:21)
Nice. Whenever I talk with espresso people, it’s always, there’s the question, are you a Lowes person or a… what’s the other major one at the end of it? There’s two camps on espressos.

Jon Bass: (06:32)
Oh man. Well, I live in… I have been buying beans from this place in Brookline for years now that is… so I just get… I’m not sure, I just buy my beans there and even if I’m… when we were just on vacation and I mail ordered beans from them to where we were on vacation [crosstalk 00:06:53]

John Verry: (06:54)
I guess you like their beans. All right. So let’s get down to business. So obviously from your intro, somebody might understand why we asked you beyond this podcast, which we’re loosely calling podcast, DevOps for dummies.

Jon Bass: (07:10)

John Verry: (07:10)
Because it’s amazing to me how DevOps has moved from something which was, N years ago, where N is relatively short number, is something sort of fringe in a buzzword or something you heard, that to front and center. So what is DevOps and why did it move from being fringe to center?

Jon Bass: (07:31)
Yeah, I think, I mean, DevOps has come to mean a lot of different things. I think it’s almost now to the point where DevOps really can mean anything from almost classic system administration, to people doing sophisticated cloud architecture. I think probably if you had to say what is the ingredient where people feel like something is DevOps, it’s when there feels like there’s more of a coding or an engineering bend to operational activities. So, whereas maybe in a traditional IT or software organization, you had ops people and they had their practices and you had engineers and they had their practices and they were two different worlds. Literally as the name implies, DevOps, what I think would happen was the confluence of factors, both technical, cultural that led to some of the practices that engineering teams use to manage their software and applications, moving down the stack and over in the stack to operational practices as well.

John Verry: (08:37)
Let’s take that down or take it up a level, whichever way you look at it. So for like a business person that’s listening to this, and I’ll give you one example of where I was confused by DevOps, and why I almost feel like from a, not a guy that’s as experienced as you, I almost thought it should be called dev DevOps. Because what I used to think DevOps was, was the combination of development, like traditional development and operations. And it was for a while, I didn’t realize that DevOps was really the integration of development methodologies into the operations component that supported the development that we’ve always been talking about, the actual coding.

John Verry: (09:17)
So if you think about the whole set of pieces of this, we still have the conventional application development. Let’s say, we’re talking about [inaudible 00:09:25], you still got the guys that are developing the code that’s the underlying application, then you’ve got the guys that are developing the scripts and code that are responsible for the operational components of deploying and managing that stuff. Was that said reasonably well? Can you clean up what I said?

Jon Bass: (09:45)
I think what you said was great. I think for me, when I really started to get excited about DevOps and I think a lot of people that were early in DevOps got bummed out when they started to see job titles like DevOps engineer because… And since it’s a roundabout answer to your question, but the idea being that DevOps was supposed to be more of like a mindset and a methodology than a role. So the idea that I think you’re getting at, that the way… it’s the development practices, the way we think about our doing our work is part of what DevOps is, is that we’re moving from a more traditional siloed model of like one team does one part of the application development life cycle, and then another does another, and to looking at the delivery of software as more of a continuous flowing pipeline of changes.

Jon Bass: (10:44)
So it’s really the concept of like I think, to me, if I had to say what I think DevOps at its core is, it’s that it is that like it’s embracing the idea that in a good engineering culture, change is embraced and change is that constant… our organizational and technical processes are geared around how to make change easy and frequent and frictionless.

John Verry: (11:14)
Mm-hmm (affirmative).

Jon Bass: (11:16)
And that obviously, and we’ll get into it is that, especially in businesses that are regulated, the idea of frictionless free flowing change is a challenge.

John Verry: (11:26)

Jon Bass: (11:26)
Yeah. But I think the reason why it’s of interest and the reason… getting back to your why is this not a fringe thing, is because of the business value, because in the end, I think what people have found is that companies, organizations that are able to successfully unlock that more free flowing change model are just more effective, more successful. So I think these are older, early… like when WhatsApp got bought by for however many billions it got bought for, and it had like 30 employees, how are they managing change in their infrastructure to allow them to scale up to that?

John Verry: (12:07)

Jon Bass: (12:07)
Or, I think in the-

John Verry: (12:09)
Or Netflix, right. Isn’t Netflix one of the classic examples? I mean, how many containers did they push out a day… I mean, it’s like some staggering number, right?

Jon Bass: (12:18)
Yeah. I mean, Netflix they very early just made it an internal practice to invest in operational excellence and in using the cloud to its fullest and they clearly have a culture of… their culture is, I think there’s a couple of things going on there, it’s DevOps and then it’s also spending a ton of their own resources on tools, which is a real challenge. So if you can apply… if you have this DevOps mentality and you are now trying to figure out how do I embrace free flowing constant change in a way that makes sense for my organization, you have to start to invest in tooling to support that. And companies like Netflix can afford to create huge organizations internally, or like Google, to build the tooling to support that DevOps approach in ways that perfectly match their organization.

Jon Bass: (13:17)
I think if you’re not Netflix or Google and you then can run into the challenges of like, well, you want to apply their approaches to your business, but you can’t necessarily have a team of 20 engineers building tools for you in the same way. So you try to use the cloud, you try to use the services from the cloud, or you try to use services that like Netflix, open sources, for example, to replicate a little bit of what they’re doing.

John Verry: (13:43)
Got you. So I liked what you said there, and would this be fair to say? Is that DevOps is the way you aspire to operational excellence, right? By creating these repeatable enforceable processes, we can ensure not only that we’ve got effective change, the ability to effectively manage change, but we can do so in a way that, to what you guys do at SIMOPS, that does it in a way that ensures the security of the application, and ensures that the application maintains compliance with any of the regs that refer to it or any of the requirements that we specified. Correct?

Jon Bass: (14:22)
Yeah. I mean, a lot of the DevOps thought leadership really, where those folks took their inspiration from is like Toyota, like the Toyota production system.

John Verry: (14:33)
Yeah. So it’s almost like a six Sigma… it’s almost like a qua… In a weird way, is that a fair thing to look at it as well as almost like a quality management concept or related to development and deployment?

Jon Bass: (14:45)
Absolutely. Yes. I think absolutely. It’s about… that’s in the… Again, why is it not on the fringe? Because it makes-

John Verry: (14:55)
Too much sense.

Jon Bass: (14:57)
Well, it makes sense, but it’s that it makes value. It makes things better quality, it makes companies successful. So it’s like, it’s not about… I think it makes people happy when you’re working in that environment, but ultimately it’s a bottom line play. I mean, it actually makes companies more effective.

John Verry: (15:15)
All right. Do me a favor and unravel some of the key pieces or concepts that people have probably heard these, they might be buzzwords then they might be terms they’re familiar with. I mean, you hear agile and you hear workflows and you hear cloud deployment and Git and JIRA. Talk a little bit about, what does DevOps look like at a high level? What are some of the big pieces? How does it all fit together?

Jon Bass: (15:43)
Yeah, I think, okay. So if you say, well, okay, what we’re really trying to do here is figure out how do we make a fluid change management process for our organization? Well, and you start to think about, well, what are some of the tools that we’re going to need to help us do that? Well, we’re going… you mentioned JIRA, I guess, you’re going to need to have… Do you have the right visibility into what is the stream of work that’s happening on a day to day, week to week basis so that you can understand both what you’re planning to do this week and what you’ve delivered this week. If you’re… organizations now will routinely deploy daily, hourly, or certainly weekly. Well, if you’re going to do that but still need to have a quality and change control process in place, well, how do you know what you deployed this week? Do you have your… so you start to think about the tooling that you need to support that more dynamic situation.

John Verry: (16:50)
So is that like a… would JIRA be analogous to a project management tool for this process? Or how would you describe it more in layman’s terms?

Jon Bass: (17:00)
Well, I think JIRA would be a tool, yes. For project management for… I think the reason that JIRA has so much popularity is that can configure it in a lot of different ways and you can integrate it with a ton of different tools. So if you’re trying to manage a project, but you’re like, man, how do I track all the work that my team did this week? Well, JIRA will have plugins integrate with all the other different kinds of quality and development systems that exist. You can network together your tooling to get visibility at the right granularity to how your teams are functioning.

John Verry: (17:40)
Got you.

Jon Bass: (17:40)
And you mentioned Git, so that would be, Git, which is a way to manage your source code, Git would integrate with JIRA, but I think certainly another kind buzzword piece of the DevOps angle here is the idea that all the operational code and configuration for your software, that should be managed as software artifacts in the same way that the application that your users are clicking around in, they should be managed using the same processes and tools. System administrators and operational people for years, they wrote code, they wrote scripts, which were… scripting was maybe not… it was considered like a different thing than, I guess, quote unquote, engineering or coding in I guess, higher order languages like Java or Python or something. But the reality is that that’s… I think part of the DevOps mindset is realizing well, that’s software engineering too.

John Verry: (18:40)
Right. It needs the same level of version control, source code control that the conventional application code does, correct?

Jon Bass: (18:48)
Well, and especially in a cloud world where the hardware itself is all actually software.

John Verry: (18:58)
It’s true. Makes sense.

Jon Bass: (18:58)

John Verry: (18:59)
It’s true. So real quick for you there. So the code that we’re talking about, these scripts, this is the code that does what? It deploys, let’s say, server instances, it deploys [inaudible 00:19:13] talk a little bit about when you talk about that code, what is that code actually doing? It’s actually deploying what and deploying what, where?

Jon Bass: (19:23)
Yeah, I would say there’s a few aspects of it. There’s deployment, so deployment is, so when you’re building a software system you build artifacts which are executable at exe file or in a windows environment or like a Java archive or some sort of software package that needs to then get copied onto literally that file. A lot of times another way to reduce DevOps, is that DevOps is a process of copying files. So it’s just the art of copying files to different places in a [inaudible 00:20:10]

John Verry: (20:10)
In controllable way.

Jon Bass: (20:11)
Yeah. So deployment is, okay, I got my new version of my software, I need to copy that file onto my existing servers or on some new servers and it start it up, make sure it’s running okay. And if it’s running, okay, well, I’ll start to take down the old version of that software. So the most simple version of that, let’s say if you have a system that can support downtime, well, you just shut everything down. You say your team… the site is down for maintenance, you copy the new files on, you start it up again, and that would be the most simple way to deploy software. I think most users and companies now don’t… they expect system maintenance to be a rare or a special occurrence.

Jon Bass: (20:56)
So typically now teams, you wouldn’t do that. You would use your cloud, the fungibility of the cloud in a couple of different ways. Like you might say, well, all right, we’ve got a new version, we’ll just copy all of our infrastructure over to some new copies of all of our servers and we’ll copy the new application code to the new servers. We’ll make sure it all is running okay and then we can point our website, our URLs to the new stuff instead of the old stuff. So there’s different flavors of that. You can not actually create new servers, you can reuse some of your existing servers and just copy files to some of those new servers, make sure things are working.

Jon Bass: (21:38)
And people have invested a huge amount of time and sophistication and lots of different versions of what I’m describing. Meaning if you have a website with hundreds of… if you have a very scalable service, maybe you have hundreds of servers running your application code and you actually, when you do deployments, you have a very sophisticated rollout mechanisms, you go to five of those servers, if you don’t see any… And then you have monitoring systems in place to see, okay, if I don’t see errors from those five systems, then I deploy to five more. But as you can… so what I’m talking about there, that’s the stuff that like a Netflix is doing.

John Verry: (22:20)

Jon Bass: (22:21)
Where again, you can imagine, well, just the sophistication and the teams to build that software, those are products in of themselves, right?

John Verry: (22:29)
Yeah. Yeah.

Jon Bass: (22:29)
The system that can in a sophisticated way, roll out a new application to hundreds of nodes with anomaly and error rate detection to decide when to roll back and not like this is where the actual complexity of the tooling you build to manage your application can be just as complicated as [crosstalk 00:22:51]

John Verry: (22:52)
More complicated really. So what you’re saying is that through this DevOps process, I’m dynamically allocating all of the cloud resources that I need in a logical way that aligns with what I’m trying to accomplish, in a potentially fully automated way.

Jon Bass: (23:12)
Yeah. And I mean, we’ve talked a little bit here about deployment, so like deploying which makes sense. It’s like the first thing to think about, is like how do I get a new version of my software deployed? But in fact, the code that people write is not just the code around deploying the software, it actually is the code to define your infrastructure itself. So back to the fact that hardware is now software for most people if you’re working in the cloud, when you define your network and you define your server instances themselves, and all the your-

John Verry: (23:57)
Firewalls, web application firewalls. Everything is dynamically generated at this point, correct?

Jon Bass: (24:03)
So there’s a variety of different tools you can use to define all those components in various configuration languages. Basically you can check, manage those configurations in source control in a Git hub or Git in the same way that you’re managing the source code for your application features. So that’s a really important aspect of this too, is that now you have this opportunity, to kind of, if somebody wants to add a new internal network sub-net, you can review that change and deploy that change in the same way that somebody who’s adding a new tab to your homepage. So it’s deploying the software that your people are [inaudible 00:24:50] customers are using. And then it’s actually defining and deploying the underlying infrastructure that all of your services are running on. So that’s-

John Verry: (25:01)
Right. It’s almost as if you’re via software deploying everything from the actual application code itself, up through the server, through the network that it sits on, through the virtual data center that it’s sitting in, through the network infrastructure and security architecture that’s supporting it or protecting it, all in an automated way.

Jon Bass: (25:25)
Yeah. I mean, and it’s extremely powerful, but it creates a lot of complexity as well, like-

John Verry: (25:30)
Oh, amazing amount of complexity.

Jon Bass: (25:32)
Because each of these… Because it’s not like-

John Verry: (25:37)
I mean, if we lay containers into the conversation, it gets even more crazy because it almost feels like almost that movie inception, where there’s the dream within the dream, within the dream. I almost feel like when I talk with people about containers, it feels a little bit inception-ish.

Jon Bass: (25:53)
I think that’s a really good way to think about it because what happened with containers is, so, okay, we have the clouds and the clouds have ways to define your virtual machines and all these different network components that I’m talking about. And then containers are, to your point, it’s another layer on top of the clouds that lets you define your infrastructure in yet another way that is arguably a little bit more portable in that it’s not dependent on a specific cloud, like you don’t necessarily depend on specific Amazon features or specific Google cloud features. Some of these container technologies like Google and others have released a bunch of tooling around managing containers to get into the features I’m talking about with deployments in terms of the sophisticated rollouts and whatnot. So there’s tooling around supporting those, but on the flip side, what’s happened is that now as an operator, you have to think about both your AWS or your GCP layer of infrastructure.

John Verry: (27:00)
I’m sorry, what? GCP?

Jon Bass: (27:02)
Oh, sorry, your Google cloud layer or your Amazon [crosstalk 00:27:05]

John Verry: (27:04)
I just want to make… yeah, just right. I think most people recognize EC2, the GCP is one of those ones. I don’t know that everyone would recognize.

Jon Bass: (27:11)
So you still have to manage your cloud layer, but now you have a whole new layer on top of that, which is supposed to be… which has all the bells and whistles, but yet is a whole… all of the same challenges of infrastructure management are now at this container level, in addition to, it’s additive, it doesn’t necessarily remove complexity for you. It’s another layer for your engineering and your ops teams to think about, which is how do I manage the configuration and operation of this container layer in addition to my cloud layers underneath just it?

John Verry: (27:43)

Jon Bass: (27:45)
Yeah. I mean, I think the cloud people will say, well, there’s a vision like Google, but all the clouds have a container management as a service. So the idea being well, look, you don’t have to worry about any of those details anymore, you can just deploy your containers into our environment and not to think about any of this complexity. But in my experience is that most people don’t have that clean… Most people inevitably find reasons for their business, for legacy applications or for specific features that they need, where they have to go outside of the bounds of what the predefined containers as a service that the cloud providers offer can deliver. And so you end up having to think about not just the container layer, but these other layers beneath it.

John Verry: (28:33)
Yeah. Yeah. Okay. So quick question for you. So agile.

Jon Bass: (28:38)

John Verry: (28:38)
First off, what is agile? What are the basic concepts of agile? And second, you always hear agile. Well, I shouldn’t say you always, but agile seems to be inexorably linked to DevOps, does it need to be? Is that just the nature of… Is agile one of the drivers to DevOps?

Jon Bass: (28:53)
Well, I think in as much as I think, if you think of what the goal, to me, the goal of agile and the agile processes comes from similar motivations to how I tried to frame DevOps as this thing [inaudible 00:29:11] about a more of a continual fluid change model. And an agile similarly, I think it didn’t come from the technical operations side, it came from the more of the, how do we manage our engineering process side, but the idea with agile being you should create a culture where you have smaller, more manageable iterations of work and a culture where you share that work and tests that work with customers early and often. I mean, I think most people have been part of software projects where it’s easy to put your head in the sand and you’re behind, you want to get stuff done, you spend months and months building some software and then you release it and people don’t… they’re like, well, I don’t want any, this doesn’t meet my requirements anymore, or I don’t like using this.

Jon Bass: (30:01)
And I think agile was an attempt to inject more reasonable iterative methodologies into the way we think about a software. So like let’s not put it all our hopes around testing something with customers in two years. How do we plan out our work so that we can get value to customers in a short amount of time and in a few weeks or a few months. And how do we recognize the fact that we might think we know the requirements of our system now, but the reality is that in two months, we’ll have talked to 10 new customers and we’ll have realized, oh, well that wasn’t a good idea, or we actually have to do this other thing. And we don’t want to be locked into a plan where we can’t now react to those requirements for months on end. So to me, the reason that these concepts are linked is that it’s a similar… agile was reacting to a similar need to figure out how do we be more fluid? How do we create a process that embraces more continual repeated change than [inaudible 00:31:11] traditional software development methodology.

John Verry: (31:14)
Got you. So in a weird way, agile drove this idea of these faster cycles, the ability to manage change more effectively, which drove this need for operations to adapt to that, which in a weird way, drove DevOps a bit, it sounds like.

Jon Bass: (31:33)
I think that’s true. I mean, I think that is true for sure because I remember the first cloud migration I did was a situation where I was working with a team at my company’s internal data center where the process for years had been like, well, if you needed new software you’d fill out a spreadsheet… I mean, new hardware, you’d fill out a spreadsheet and you say, okay, I need, and you’d get it in the CapEx budget for next year like, here’s the periods with the components I need, here’s how much storage these servers are going to need. And then you would give them a… you’d work with them to schedule deployments and it’s like, you’d get somebody from the IT team to be in the data center to help you get your stuff deployed, and it was a big deal to change it.

Jon Bass: (32:20)
You had to get their time, oftentimes in an off hours because there weren’t necessarily processes in place to do rolling fast deploys, so you have to get somebody to be available off hours to do a deployment. And that was really stressful for everybody. And then as our team started to adopt more dynamic software development life cycle, and we were now trying to do deploys or initiatives like once a month, but that was really tough on the IT team, that was like that’s a lot of off hours time for them, but a bunch of teams are doing deploys that often and then once a week what happens when there’s a bug or there’s a hotfix process? Well, let’s get the hotfix team in place and we’ll get a review process for that hotfix.

Jon Bass: (33:10)
And it became unsustainable. And then I think at the same time, that’s also, as I mentioned, that was one of the… the same time, cloud became this viable option where engineering teams would be like, well, I could just program my way out of this. If I can securely define and create my infrastructure in the AWS cloud or in a VMware stack or something, then I can run this process, I can deploy when I need to, I can do the quality checks. I don’t need to have IT come in and do an off hours deploy for me. So yeah, they’re definitely linked.

John Verry: (33:49)
So yeah, it sounds like all of these things all happened as either some combination of a logical consequence of each other or arrived at the same time to enable them. Right? Because I do think you’re right. I mean, would DevOps truly be possible without new cloud technologies? The answer is probably no. Right?

Jon Bass: (34:08)
Yeah. I mean-

John Verry: (34:10)
I mean, I guess you could, but it’s certainly significantly enabled and simplified by Amazon, AWS and Azure and GCP.

Jon Bass: (34:19)
I think so, because not because then on-prem, couldn’t do it, but there wasn’t necessarily the business, like the clouds, they had to provide developers with these apps. They had to, they were financially mo… If you were running the data center, there was no real incentive for you.

John Verry: (34:34)
That’s right.

Jon Bass: (34:35)
But now there is. So now there are, I think there’s a whole industry around-

John Verry: (34:39)
Around creating almost private clouds with the same capabilities. That’s interesting.

Jon Bass: (34:44)

John Verry: (34:44)
That’s interesting. So in this brave new world, what are the key skill sets in the DevOps world? Is it knowledge of the clouds? So if I’m running a company, it’s a SAS organization, I know we’re moving towards DevOps and I need to make sure I’m hiring the right people, what’s the key skillset? Is it knowledge of the cloud that I’m going into?

Jon Bass: (35:06)
There’s a couple of things that… knowledge of the clouds, and particular with Amazon, it’s getting to the point where there are so many services, it’s overwhelming.

John Verry: (35:20)
It’s overwhelming. If you [crosstalk 00:35:21] it’s definitely overwhelming.

Jon Bass: (35:25)
There is almost this need certainly when you get to a certain footprint of deployment, you do need people that just know that vocabulary, who can spout off all those acronyms and know what makes sense, because there is a lot of real estate there. I mean, and the other clouds are catching up, but yeah, so there is some value in just people that know that landscape. From an application development perspective, I mean, it’s something that I found really that I learned the hard way, was that as… so I have an engineer, I was an enterprise Java developer for a long time. I did not come into the DevOps space from the operations side, I came into it by accident, really from the development side. And I always really enjoyed programming infrastructure, working with all these operational challenges.

Jon Bass: (36:22)
And one of the first opportunities I had to lead a team, I started to build a bunch of tools to help… I was trying to help an organization managed [shef 00:36:33], which is an infrastructure automation tool. And I created all these tools and readmes and documents and like, here’s how you can do this and [inaudible 00:36:42] you can do that. And what I realized is a lot of developers just don’t like that stuff. They were waiting for me or my team. They’re like, well, there are still remains… even though DevOps require… there’s a little bit of a, there’s an engineering skillset that is required, not every engineer, just like not every engineer likes to do front end development or back end development. Not every engineer is going to like working on those infrastructure and operational problems.

Jon Bass: (37:11)
So that I think, it should have been obvious to me at the time, but I had to learn that part of it is that you can’t just hire people that are good software engineers who have maybe developed great web applications or developed sophisticated algorithms and say, okay, well now you can apply your skillset to these operational problems. Because not everybody really… that doesn’t necessarily resonate with all people. You need to find people that have some of that engineering aptitude, but that there’s hints in their background that they’ve been the ones on their team that they’re the ones that built the internal tool to get the rest of the team unblocked, they’re the ones that we’re really, oh, I was so frustrated with our having to do these manual patch meetings every month, so I wrote this script to automate patching.

Jon Bass: (38:06)
You have to find people that have been those kinds of engineers on teams. Those people that get in there and maybe they’re a little OCD about how infrastructure gets managed and they’ve been the ones to build those tools before. So there you got to find hints in people’s backgrounds of that, about that stuff.

John Verry: (38:23)
That’s pretty interesting. Yeah. And I think you’re right. I mean, it’s somebody that gets jazzed and excited by that component of the development. Right? Because it’s… I mean, it is, it’s dozens of different skillsets through that whole life cycle. So where does security fit into this? You hear this term SecDevOps. Where does that fit in?

Jon Bass: (38:43)
Well, I think that a lot of it tied into the… we’ve been talking a bunch about complexity and you get into the now, okay, so now we’re in a world where our networks themselves, the actual, the network routing tables are defined in code, and our applications obviously are defined in code, but the ways they connect to each other, the way code gets deployed, the container level, the cloud level, you got so many layers upon layers of software that go into a modern application deployment. Reasoning about the system boundaries and who can do what, and who has access to what, properly logging and monitoring across that complex stack, that has just become more challenging.

Jon Bass: (39:39)
So I think SecDevOps has emerged as like, okay, we need people. So we’ve already talked about how it’s challenging to find people that are… you have to find people that like working on tooling and that stuff. Now we need to find people that are also going to maybe have a mindset where they’re thinking about in particular, the parts of this infrastructure stack that can maybe, well, put an organization at risk. The one you hear the most about, I feel like is still is that there’s, for Amazon at least is, access to resources in S3. So-

John Verry: (40:19)
Yeah. S3 buckets are definitely a challenge. I mean, and they shouldn’t be, but I mean, it’s remarkable how often they are.

Jon Bass: (40:25)
But I think the reason they are is because look, in the end, software business, the software does need to connect to the outside world, to deliver value. So something has to be something from S3 in order… In other words, [inaudible 00:40:40] the most secure business just doesn’t run.

John Verry: (40:42)

Jon Bass: (40:42)
So the reason that it’s not a solved problem is that there’s a ton of value in storing and getting resources from S3. You can’t just… so if it was as simple as just shutting it off, then so much software relies on data stored in S3 and you need to always be thinking about there… Amazon has now made it, they’ve gone as far as at your account level, you can just check a box to say nothing from the outside can access this S3 buddy. Okay. But [crosstalk 00:41:18]

John Verry: (41:18)
Yeah. You just broke everything. Right? You just broke the applications functionality.

Jon Bass: (41:22)
Yeah. But, okay, let’s say then you fix all that, but you still have servers that now the next vector of attack is okay, well, we can just get on to, how do we get onto some instance running? And I think there was a capital one breach a few years ago, which was, it wasn’t direct to S3, it was they got onto a virtual machine that could connect S3. So they were able to… and then so you have to… and I think the sophisticated and complex nature of the networks makes it just really hard to reason about and really hard to detect all the different potential access patterns where people cannot get access to these sensitive resources.

John Verry: (42:08)
Yeah, it’s an interesting challenge. When you think of DevSecOps, do you think of DevSecOps as being… like in my mind, I’ve got this, I’m going to call it, and you might disagree with me, a secure development life cycle methodology. Right? That in my mind goes from code inception, application, planning, all the way through continuous deployment. Right? So do you think of DevSecOps as being just the parts of that? So as an example, if I said that you were going to do unit code security testing, do you consider that part of DevSecOps, or do you consider that to be further upstream from DevSecOps in a more traditional SDLC?

Jon Bass: (42:52)
I mean, I think that if you think about like, well, anything where you’re developing tools as part of that life cycle that have a security focus, I feel like to me, they’d fit into that bucket. I think something that comes to mind for me, I think a lot of people have developed tools to try to help engineering teams understand cloud permissions models. If you’re trying to develop a service and you want to make sure you give it the least privilege, how do I properly configure and define my new application so that I’m only giving it the minimum amount of privileges it needs to run in the cloud? So people have done a lot of work… that’d be an example of where DevSecOps can come in with maybe tooling and support around okay, I see that you’re working on something that has new cloud permissions, something that can either review those, help you get those set up correctly early in the life cycle.

Jon Bass: (43:56)
That does feel like it’s in the purview of DevSecOps because there is a ton of complexity in properly defining these cloud permissions. So like AWS has identity and access management policies very complex. They’re hugely powerful, but it’s a discipline in of itself, even within the DevOps space of just understanding these policies, their capabilities, how to define them correctly. So I think organizations that are able to move the definition for as early as they can in the development of software, considerations around security policies are setting themselves up for success. Otherwise you end up when you get to the point where you’re like, all right, let’s deploy this to production, well, nothing works because I don’t know, we developed it in staging where everything had admin permissions, we didn’t have to… And then, so that’s a pretty typical pain point. And I think there’s a huge opportunity for DevSecOps to come in and help with that.

John Verry: (45:13)
Yeah. And it is to me, it gets blurry. You live in this world every day, I don’t, so to me it does get blurry. So would you consider [inaudible 00:45:20] so let’s say you’re running a workflow, something like Jenkins. And so it would be the DevOps people that would write the scripts that might, after data gets pulled from a code repository, when it’s getting prepped for deployment where it would be run through, let’s say a selenium or something of that nature, some type of security testing tool.

Jon Bass: (45:43)
Mm-hmm (affirmative).

John Verry: (45:44)
So is that an example of where it… that is DevSecOps from your perspective? Because it is reaching down into the SDLC a little bit.

Jon Bass: (45:52)
Yeah. I mean if the selenium… I think-

John Verry: (45:57)
And selenium might be, I think selenium’s got some code testing [crosstalk 00:46:04] doesn’t it?

Jon Bass: (46:05)
I think honestly, I don’t… I would say, you could say, I think, depending on how an organization structures this, the DevOps team, they would have set up the build pipeline that says, okay, when somebody checks something into Git, that that’s going to kick off Jenkins jobs that do a bunch of stuff, including just regular tests, regular packaging of the software or whatever. And then I could totally see well, the DevSecOps would say, well, we want to add a step to this pipeline that is we have this, I don’t know, we’ve developed a series of selenium scraps that try to compromise the web app in various ways, something like this.

Jon Bass: (46:54)
So, yeah, I mean, I think you could, depending on the maturity of your organization that easily could just be the DevOps team, but that certainly could be something you could say a DevSecOps role would specifically be like collaborating closely with DevOps who has this whole build and delivery pipeline to insert elements in it that are security specific. Sure. That could work.

John Verry: (47:19)

Jon Bass: (47:20)
I don’t know that the disciplines are so… the boundaries between those disciplines are so hard to set right now. I think just in the same way that in some organizations, you may not even have… the engineering team might set up that Jenkins job. They may not have a… that might not even be… they may not even have a separate DevOps resource to do that or the DevOps resource. [inaudible 00:47:47] they deploy Jenkins and they put in a Wiki page on here’s how you set up your jobs, everybody.

Jon Bass: (47:54)
And then everybody else has to read that Wiki page to configure their own jobs. So there’s a lot of [inaudible 00:48:03] I think, to these roles. I think because of the complexity of these cloud deployments, if you can label somebody as like, okay, you’re thinking about security throughout the lifecycle, that’s helpful just because… just in the same way it’s helpful when you want to make something important, you need to put a person on it. And so that is a… I guess that’s how I think about it.

John Verry: (48:30)
Yeah, you can see that the way… you can see you look at it from a perspective of a person who’s lived in that world. And I look at it from the perspective of the security gap, right? Because we sometimes get bounced around, you got to go talk to that person, you got to go talk to that person. That’s not my job. And from my perspective, security is everyone’s job and the further I can move security down that pipeline, and the more that I can minimize the chance that there’s going to be a gap between what this person is doing over here and this person is doing over here. So that’s why I was asking the questions the way that it was. So let me ask the question. So I’ve been hearing this term recently, RegTech, does your… SIMOPS, the stuff that you’re doing, would that be considered RegTech?

Jon Bass: (49:13)
Yeah. I think that actually, that makes sense. I think the reason that we started the company was our feeling that I spent several years as the VP of [inaudible 00:49:25] at a health tech startup. One of my co founders was the CTO of a health tech startup, we both felt like there’s such an opportunity to bring some of the DevOps principles and mindset to how we’re managing deploying our software to more of the compliance space. So like working in health tech, there’s so much that we needed to do to maintain a secure and compliant posture in terms of entitlement reviews, just the quarterly work we had to do to maintain our infrastructure. A lot of these things are workflows that we felt like, well, in the same way that people have invested in automating more of the way we deploy our infrastructure, the way we manage testing, we can apply some of these principles to the way we manage our compliance posture overall.

Jon Bass: (50:26)
So I say there’s definitely a new group of companies that are… we’re trying to help places when they need to achieve compliance to actually build with software and proper tooling into their actual cadence and the life cycle of how they work and build those workflows right in. And so you [inaudible 00:50:49] the beginning, we’re our approach is we’ve created an SDK that lets teams define security compliance workflows using our templates, but they match exactly with the environments that they have in place. So-

John Verry: (51:07)
Give me an example, just so people can understand what you mean by about security compliance. Give me an example of what a workflow template might do, and I’m assuming, and one other quick thing, so I’m assuming you’re also doing this. So to simplify an organization’s ability to become [SOC two 00:51:23] attested and maintain that SOC two attestation or ISO 27,001 or something of that nature. Correct?

Jon Bass: (51:30)
Yeah. So a good example for us, so a lot of our early examples, they start with access management workflows. So a typical situation in a cloud environment is because as I mentioned, it can be really challenging to figure out what cloud permissions are required sometimes, to manage your infrastructure. What ends up happening is there’s a lot of access drift in most organizations in terms of the amount of access that people have to infrastructure. So a typical workflow example for us is having a way for people to dynamically request access, to get elevated privileges to do something for some period of time. What we do to turn that into not just an access management workflow, but in our view, an actual compliance workflow is like, okay, so let’s say that I might have requested access to some infrastructure, but it’s like 2:00 in the morning, maybe at 2:00 in the morning, if you happen to be on call, you should be able to get that access to resolve an issue, but that’s really not good enough from a compliant… You can’t just say at 2:00 in the morning, people can do what they want. Right?

Jon Bass: (52:40)
So what we do is we provide a good way. Well, okay, if somebody gets emergency access, let’s make sure we kick off the followup workflow, so the next day there’s a proper followup and review. What was the emergency access for? Have you recorded the proper follow up actions from that emergency access so that they can be corrected in a non late night way, with the team as needed. And then in addition, we build in the reporting so we actually can see based on actual activity from the clouds, here are the key systems or components that were touched when this emergency access was requested. So-

John Verry: (53:28)
Got you.

Jon Bass: (53:29)
So there’s an initial pain point around, let me get access to this thing, but knowing that in our experience, that’s not enough, we can’t just say that, okay, there’s a process to get access, how do we show that we have the right controls around that access to really make it fly from a security compliance perspective, which in our view is, do we have the proper review process in place if we do have emergency escalations, is the proper followup in place? Do we have reporting and analytics that are tied to these events? So it makes it easy for my team and for another to see how we’re able to correlate workflow activity from access requests to what’s actually happening in the clouds. So that’s [crosstalk 00:54:20]

John Verry: (54:21)
It sounds like you enforce policy and procedure and ensure that you’ve got the artifacts to demonstrate that.

Jon Bass: (54:28)
Right. Which is what you… you need that whole package. If you want to go show we have a process in place, those are all the of components in our view that people are going to want to see.

John Verry: (54:41)
Got you. Just out of curiosity, is that done through the automation server, like the workflows, like Jenkins? Is that data stored in JIRA confluence? Where are you shimming in to their systems?

Jon Bass: (54:55)
Yeah. So there’s a couple of different ways that we do it. The vision with our SDK is that because it’s a toolkit, we’ll be able to integrate into basically anything that has an interface that supports programmatic interaction. For early customers, a lot of them either they’re kicking off these workflows in Slack, which a common way for teams to be communicating with each other. So they’ll kick off a workflow in Slack to request access to something. And then we basically have cloud integrations into customer accounts where basically we have a workflow engine that we host and then we have a serverless deployment model.

Jon Bass: (55:45)
So we’re able to basically deploy these small serverless appliances basically into our customer accounts, and those appliances can talk to… securely talk to our customer’s infrastructure to interrogate it to make entitlement changes in customer systems. So the benefit of this approach is we can manage for customers this nice workflow system that they don’t have to manage and maintain, but they don’t have to give us keys to the castle in order for that system to [crosstalk 00:56:21]

John Verry: (56:20)
Oh, I got you.

Jon Bass: (56:21)
So we could have these little appliances that are running where their access keys stay there in their accounts. And they get the benefit of the workflow and reporting and analytics living in our account, meaning the benefit of not having to maintain all that stuff.

John Verry: (56:40)
Got you. Yeah. That’s the problem with these… we’re running out of words to use. Like workflow, I’m immediately thinking you’re talking like a Jenkins workflow, you’re talking about a SIMOPS workflow.

Jon Bass: (56:50)
Yeah. I mean, and then reality is like, and we have a customer that’s like, we can have a SIMOPS workflow that will kick off your Jenkins workflow for [crosstalk 00:57:01]

John Verry: (57:00)
Exactly. Or vice versa. Right.

Jon Bass: (57:01)
That’s right.

John Verry: (57:03)
Yeah. And just generally speaking, where do people keep the evidence? So if you follow this process, is that going to be… I mean, I guess it can be anywhere, it could be a help desk ticket, it could be in JIRA, could be pretty much anywhere, because of the fact that you guys aren’t STK.

Jon Bass: (57:17)
Yeah. So I think we’ll see… a lot of people will want to ultimately create tickets in JIRA because a lot of people do have… that has become a little bit of like a system of record for these things. And then we have our own reporting platform, but to your point, I think we will… using our appliances basically, we can post the final artifacts to the systems that people need.

John Verry: (57:41)
Cool. Interesting stuff. It’s crazy that five years ago we probably [inaudible 00:57:48] whatever the number is, five years ago, there wouldn’t have been that many people that would be interested in this conversation and now it’s pretty much everywhere.

Jon Bass: (57:57)
We’ve been really… we’ve had such great conversations and there is a lot of interest. There’s just a lot of people that even five years ago, there was friction to, well, what are we doing securely in the cloud versus not? I don’t feel like people are still… everyone understands that it is possible to run securely in the cloud now, people don’t… Everyone understands that it’s hard and it’s challenging, people want help, but it’s not like… I don’t feel like… we’re not in the same conversations about like-

John Verry: (58:28)
Yeah. Is the cloud secure is no longer a conversation, it’s do we have enough knowledge to use the cloud securely, I think is now the question you need to ask. Right? Does my team have the knowledge? Does my team have the tools or do I need external support or external tools to be able to make the cloud secure? Because the cloud can be secure, there’s no doubt about it. In fact, a cloud arguably can be more secure than your on-prem stuff.

Jon Bass: (58:55)
I think you’re right, but I think you nailed it. Yeah. There’s a lot of complexity if you’re not Netflix and don’t have thousands of engineers that you can have investing in your cloud ops platform.

John Verry: (59:07)
If you’re not developing your own container system?

Jon Bass: (59:09)
Right. So that’s for a lot of engineers-

John Verry: (59:13)
That’s so many engineers that they have, that they looked at Kubernetes and they looked at Docker and said, no, we’ll build our own.

Jon Bass: (59:19)
Right. That’s why it’s hard to… it’s amazing what they’ve done, but they made [inaudible 00:59:25] It’s not necessarily a model that you… there’s a lot of risks to that model [crosstalk 00:59:30]

John Verry: (59:31)
Oh yeah. Oh yeah. But then if you have whatever it is, 20 million people giving you $15 a month, yeah, you can afford to do a lot of fun things. So is there anything that we didn’t touch on that you think we should touch on before we say goodbye?

Jon Bass: (59:48)
I think we covered a lot here [inaudible 00:59:48] conversation.

John Verry: (59:49)
I did too. I learned something which is always good for me. So we warned you, so I hope you’re prepared for this question. So we usually say an amazing or horrible CISO, I’ll say an amazing or horrible engineering manager or DevOps engineer, whatever. So what fictional character, real person do you think would make an amazing or horrible DevOps engineer and why?

Jon Bass: (01:00:13)
Well, I’ve been thinking about this [inaudible 01:00:16] so I decided to go amazing to keep it positive. And I think we talked about this. I mean, I’m a big Celtics fan, so-

John Verry: (01:00:23)
I like coach.

Jon Bass: (01:00:25)
I like the coach too. I’ve been thinking about, I’m going to go with Marcus Smart. I don’t know if you’re familiar-

John Verry: (01:00:30)
I like Marcus Smart.

Jon Bass: (01:00:31)

John Verry: (01:00:31)
He’s a great player.

Jon Bass: (01:00:33)
I think he’d be good. And I have a couple of reasons. So-

John Verry: (01:00:36)
Because he’s smart, obviously, that’s the first one, right?

Jon Bass: (01:00:39)
Well, he’s smart, but he is a… first of all, he’s a great individual defender. So I guess if I wanted… ultimately, I want [inaudible 01:00:49] or even I’d want somebody who could do the individual heroics when we needed them, when emergency struck, that he or she would be there for those heroics but he has a reputation as well for being a great communicator. And I think a lot of the CISO, or really a DevOps job is it’s not just the heroics, but are you able to effectively… I’m sure you [inaudible 01:01:14] resonated for you just, are you able to effectively communicate to people why it’s important for them to do these things? Which I think is often a real challenge that creates so much friction for a lot of this kind of infrastructure work.

Jon Bass: (01:01:30)
Why do we have to go through this process now? Why do I have to? What is it? So being able to communicate effectively, and in a way that people will respond to is really important. And then I think just in terms of he’s also a leader and I think back to the thinking about how the deployment tools, as if [inaudible 01:01:57] deployment tools can be more complicated than the application itself, the operations and security, I feel like are more and more what is going to differentiate organizations. There’s a lot of cool features out there, there’s a lot of companies doing neat things, but can you do them with operational excellence? It’s kind of like, so leadership from the ops side, from the security side, I think has huge value to organizations that everyone, it’s not that hard to have a good idea, but it’s hard to make that scale.

John Verry: (01:02:37)
Yeah. There’s the old adage, a great strategy with marginal execution will get trumped by a marginal strategy with great execution every time.

Jon Bass: (01:02:47)

John Verry: (01:02:47)
So I agree with you. The other thing which I liked that you said, and I think is really key here is that that communication, I think is so critical because you have such disparate audiences in the DevOps world. You’ve got the original engineering, the software engineers in the traditional sense, right? You’ve got DevOps, you’ve got security sticking their nose in there. And then you’ve got business leadership, which you need to get their buy in on all of this stuff. So, I mean, realistically, you need to be able to talk in multiple languages to disparate audiences in a way that rallies the troops and gains consensus. And I think that’s a hard thing to do. And that’s one of the reasons why the world that you live in every day is a challenging world to live in.

Jon Bass: (01:03:28)
Yeah. It can feel thankless, honestly, because it’s not… if you’re doing your job well, probably, you’re not getting a lot of applause, like-

John Verry: (01:03:36)
Yeah. It’s probably like being an NFL about an NBA ref. Right? I mean, the only time you recognize the NBA rep when he fails to swallow his whistle at the wrong time. Right?

Jon Bass: (01:03:45)

John Verry: (01:03:46)
All right. So last question, I guess. Nope. Just one quick thing. So farewells, before we say farewell, how can folks get in touch with you, Jon, if they’re wanting to chat with you?

Jon Bass: (01:03:57)
Well, I’m on Twitter. So first, my Twitter handle is Firstmorecoffee. So back to coffee being, so-

John Verry: (01:04:06)
It’s not Brookline beans. You know what I mean? I would’ve thought that would’ve been your handle because-

Jon Bass: (01:04:10)
No, no, but for a coffee, that’s my place. I’ll give them a shout out.

John Verry: (01:04:16)
There you go. I think you’re going to be looking for a cut on future revenues from them?

Jon Bass: (01:04:22)
Right now I just really want them to stay in business.

John Verry: (01:04:25)
Oh God.

John Verry: (01:04:28)
From your lips to God’s ears, that all of these small businesses that are getting killed in this current COVID thing are all healthy and States stay in business.

Jon Bass: (01:04:37)
Well, that’s what I think for them. And then is our company website, and you can reach out to us and [email protected], you can always reach out to me there too.

John Verry: (01:04:47)
Cool, John, thank you, sir. I appreciate you coming on. I had a lot of fun.

Jon Bass: (01:04:51)
Likewise, great talking to you.

Narrator: (01:04:54)
You’ve been listening to the virtual CISO podcast. As you’ve probably figured out, we really enjoy information security. So if there’s a question we haven’t yet answered or you need some help, you can reach us at [email protected]. And to ensure you never miss an episode, subscribe to the show in your favorite podcast player. Until next time, let’s be careful out there (silence).