EPISODE 1625

[INTRODUCTION]

[0:00:00] ANNOUNCER: This episode of Software Engineering Daily is part of our onsite coverage of AWS Reinvent 2023, which took place from November 27 through December 1 in Las Vegas. Today's interview, host Jordi Mon Companys speaks with Rob Zuber who is the CTO at CircleCI. This episode of Software Engineering Daily is hosted by Jordi Mon Companys. Check the show notes for more information on Jordi's work and where to find him.

[INTERVIEW]

[0:00:36] JMC: Okay. Rob, welcome to Software Engineering Daily.

[0:00:40] RZ: Thanks for having me. I'm excited to be here.

[0:00:41] JMC: I'm sure you've shipped applications that have been containerized, that were part of a container. But have you yourself been containerized ever? Because we are here at AWS Reinvent, it's a very silly joke. But yes, we are effectively recording this in a container. Have you ever been containerized?

[0:00:59] RZ: Well, I have not that I can recall being in a container. But I will say, we were just talking about Oakland, I live in Oakland, and Oakland is a massive shipping port. I see container ships all the time, and I'm fascinated by them. I think I'm fascinated as an engineer by the technology and the innovation of containers, and what it did for transportation, and shipping, and all these other things. So I love the metaphor in software, even with all the dumb jokes that we make.

[0:01:24] JMC: Now, I wonder because Docker's founder, Solomon Hykes, I think, came up with the idea of packaging containers in the nifty way that Docker did. When he was living in San Francisco, I think he was involved in a company that he also founded called Dark Cloud. You probably remember that better, but maybe he was inspired by the containers at the Oakland port or the San Francisco port, if they have any.

[0:01:47] RZ: Maybe. Yes, they would. I think Oakland is a little bigger, but they're on both sides of the bay, and you see the ships just drifting around waiting for spots or whatever they're called. Also, the tugboats are super cool. But now, we're way off topic. I don't know, but at least the designers were because a lot of their early logos, and I think some of them still had kind of this mash up of whales, and container ships, and stuff. 

[0:02:06] JMC: There's certainly a - at least in the design, and the communication style. There's a San Francisco landscape and ecosystem inspiration for sure, whether it's economical, nature wise. So tell us a bit about yourself, because you've been a longtime CTO of CircleCI, correct?

[0:02:23] RZ: Yes. I've been at the company over nine years, and most of that time, as the CTO came in through a small acquisition. The company was 14 people. We were three people like small on both sides. I had been working on CI/CD for mobile, again, with two other folks. Then, we joined CircleCI to combine that together with everything that circle CI had built at the time to build a complete solution across the board.

[0:02:45] JMC: What is CircleCI right now, by the way? It's acquired a few of the companies along the way.

[0:02:50] RZ: Yes. We did acquire two other companies in the last few years. If you're asking overall, as a company, we're between 400 and 500 in terms of people, so that's a lot of growth. A lot of growth, personally. Running a team going from 14, or whatever that was at the time to that size. It's been an exciting journey. We were just talking about containers. A lot has changed in how we deliver software in that time. We did our own containerization in the early days. Actually, even at the time that I joined in 2014, we were using LXC, which was an early sort of built into Linux. I think some of the early Docker stuff was built on top of that. We don't need to get into the technical details. But thinking about Docker, about Kubernetes. But how we deliver software, all of these things have changed so much over the course of that time.

Our core purposes stayed the same, right? Help our customers get early feedback, learn, deliver software, quickly feel confident in that. But how we've done that over that time has changed significantly. I got into doing that mobile CI/CD in 2014. Because we were shifting as a software community, I think, from Rails first. 2011, I was building a Rails app. Then by 2014, we were basically mobile first. Everyone was building iOS apps and Android apps. The tooling just didn't really exist for that. That was a big shift, and then the Docker shift, and containerization, and micro services. But at the end of the day, you're trying to build a product for your customer and get that out in front of your customer. So, I guess to round out about myself, I am a software engineer by background, not by schooling. But ever since school and love delivering great products. I love starting companies. I love thinking about what customers want. So supporting our customers to do that for their customers is just a really, really fun place to be.

[0:04:34] JMC: What does the CircleCI - this will be the end of the roundup of your description, professional description. What is the whole extent of CircleCI as current portfolio?

[0:04:44] RZ: Yes. I mean, we talked briefly about acquiring a couple of companies. Basically, we think about building and delivering software. We think about the lifecycle where that's happening, and getting feedback as quickly as possible. It starts on the developer's desktop, or laptop, or whatever the machine is they're using. I'm working in my IDE, and I want very fast feedback. We're now providing that right back down into your IDE all the way out. Part of that came through one of those acquisitions, and then all the way out into production, which is the other end, where we're now including deploy and release. 

So not just CI, which we've known for, for a long time, and CD, but release, which is going from, I have the software in an environment to, I'm exposing it to specific customers. Because it starts with the developer and the work they're doing. We think a lot about the fact that everything in software has changed, right? We just made analogies to other engineering disciplines, but in a lot of other engineering disciplines, you make a big plan, and you execute on the big plan. You build a bridge, you don't iterate, and sort of figure it out as you go.

[0:05:48] JMC: The waterfall works well for those civil engineering project.

[0:05:52] RZ: Exactly. Exactly. But we start with something tiny, and then we change, and change, and change. So we think about change flowing, again, all the way from the developer out into production. Until you've released that change to your customers, and seeing how it behaves, you can't truly feel good about it, right? A green CI build, my tests pass isn't really that green, if I then put it out in production, and I caused an incident, or customers can't use it, or even business metrics change. Like not as many people are signing up. Well, we obviously changed something. It does what we expected it to do, but it doesn't perform the way we hoped it would perform in front of customers.

And so, we want people to have that full confidence again, from the idea to the point where they're delivering value to their customers.

[0:06:32] JMC: I think that's something that although I've been monitoring CircleCI for a long time. I don't think that everyone is aware that the offering, the product itself covers the CD. Let's call it part of the software lifecycle. It does so for a while. It's been doing so. I just think that people tend to think of it as limited to a CI scope, and it's not completely true as you just described.

[0:06:52] RZ: I'm sure there's a good lesson in naming somewhere in there.

[0:06:55] JMC: Have you considered changing it by updating it?

[0:06:57] RZ: I would say for the nine plus years I've been at the company, we've been having that conversation, but it's a hard job. And at some point, you end up - yes, your brand might say something about, but ultimately, it's a brand that people just recognize the name, and then they go and use the tool. That's a tradeoff, and it's not the first problem we were trying to solve right now.

[0:07:15] JMC: I don't think it will change dramatically, anything. So one of the things that the industry as a whole, I've been having conversations here at Reinvent for the two days that I've been around is revolving around security, and CI systems, and build systems, both if there's any difference at all. I'm happy to hear your approach to it if it's worth it. Under the spotlight in terms of the tool, the space, the moment in the software delivery lifecycle in which most of the security critical measures should be in place. What are your thoughts on that? Do you think that that is a burden that should be put on the people managing CI? If so, how does CircleCI approach that?

[0:07:57] RZ: Yes. It's a rich and deep - it's deep, it's broad, it's a very big subject. There are a couple places that I think about a lot from a security perspective. One is what I often referred to as chain of custody, non-repudiation, basically, knowing that the thing that went into the build system is the thing that came out. That is an area where we are continuing to invest for our customers, but our customers have to do a lot of the work. I think one of the things that's interesting, challenging, but also helpful that we can solve for our customers is that there is a line that is the CI system platform that you use. And then, there's what you bring as a software team, software developer, right?

There are many points that we don't own where you need to secure the system, so we need to be aware of that. But what we can do is say the thing that you put in, meaning, what we pulled out of source control is the thing that you sent out. Some integrations around outside tools, like not all of that happens inside the system. But that is a key area in software delivery that you referenced SolarWinds earlier. It's an area that many folks are concerned with.

[0:09:06] JMC: Would you say that build SBOMs? The result of a build, the SBOM that describe the result of a build would actually solve that completely, partially, a big amount of it. Or actually, they are as many thing - potentially a waste of time, actually.

[0:09:22] RZ: Yes. I think that there are many facets. I think that sort of like law of constraints in many different places. If you sort of flip it around and think about a bottleneck, you're going to solve a problem and just move the problem somewhere else. So ultimately, if I go all the way out, and look at like a threat modeling perspective of security and step out of sight of CI/CD for a second. Then, the question would be, where are the biggest risks? I think people who are saying, "This is pointless" are probably identifying bigger risks, and that may be 100% true, right? Do I have any control over what's even going into my source control system. I can't say that for all software development shops. That's a problem that you have to solve, right? If you look at SolarWinds, as an example, which is the sort of - 

[0:10:10] JMC: The watershed moment. 

[0:10:12] RZ: Yes. It was the key point that raised everyone. Hey, this could be a real problem. The amount of work and investment that went into making that happen, if you look over the detailed timeline, it's multiple years. It is a very, very long game. In many shops, you can walk in through the front door. Why would I put all of the time and energy into that attack? Attacking, I don't know how many people think about this. I definitely think about this way. I used to work in email, and so I always thought about this was spam. Why do people send spam? Clearly, no one reads this.

The answer is, enough people read it to make it worthwhile. It's always an ROI cost. We don't think of attackers as MBA grads, but they're smart. They only invest the money when it's going to pay off. The question would be, what is the level of attack? Then, what is the return on that? A lot of this depends on what you have, what you have that's a value. I said earlier, I think about a couple of different points of view. The way that I would bring those together is, as a CI platform, we have a lot of access to other people's systems. So the place that we're really investing right now is getting rid of that.

Historically, when CircleCI was created, we talked about systems changing, and approaches changing. The only thing that we could do in order to help you deploy was hold on to tokens. I mean, we're sitting under an AWS sign. I'll say, AWS credentials. And AWS credentials, back in 2011, we're all powerful, but no termination, no expiry, access to everything. Those are super dangerous to have around.

Now, in 2023, you have OIDC, you have the ability to create a short-lived token, hand that off, use it for 15 minutes, and have all of its privileges removed. By creating better tooling to allow our customers to use some of these frameworks, we're allowing them to get away from these dangerous approaches to using CI/CD in the first place. Or, sorry, any integration, I think, is the best way of thinking about it. It's something that we've had to do in order to get to that deployment, but it's shifting rapidly. So, again, if you look for the sort of lowest hanging fruit or the first constraint, it's going to be something like that over. Because if I can get access to another system, I don't need all the complications of injecting undetectable binaries into your build system kind of thing, right? I think starting from that foundation, and working our way up is the right way to think of it, versus, here's a problem that I know how to solve. Therefore, let's go solve that one, instead of what is the biggest risk to us right now.

[0:12:40] JMC: It seems like, as we mentioned before, that the SolarWinds was a watershed moment and that in general, public organizations, governments have reacted in different ways. It seems that the EU follows a philosophy of, let's say, rule, law that is regulating. I personally like more the American approach, which is the American government will only buy software, and it's probably the biggest buyer of software, probably in the world, but definitely in the US, or one of the biggest. The American government will describe how it buys software, what kind of software it buys, or what kind of requirements that software will, and that will create enough demand for then that type of adaptation from that software to expand elsewhere.

This gives me the segue to ask you about FedRAMP? What kind of sort of like approach to circle CI have to it, and if those requirements are good enough, or are they a good incentive for the industry at large?

[0:13:39] RZ: Yes. I think it's a great question. I mean, we were the first CI/CD platform to be FedRAMP tailored certified. We do take these things seriously. We see value in them. But I think it's important to recognize, you describe that, as the federal government saying, this is the kind of software that we want to buy. I think that's good for the government to use that weight in that way. It's important to realize, though, that any kind of compliance framework is typically quite open, I guess, is the best way of describing. It's often, what are the measures that you are putting in place to account for this risk. And then, are you executing those measures effectively? It's possible to kind of end up on sort of different points on that spectrum from - we think about delivering software quickly and effectively, right? It's possible to put in a bunch of sort of unnecessary mitigations or unhelpful mitigations to try to meet those requirements, and then impede your ability to deliver.

On the other end, I think it's possible to describe things that aren't that effective, and follow them, but not actually be improving your security. So, I think it's important to drive from the perspective of security. I talked about asking yourself the question, "What's our biggest risk? What are the risks that we have in the way that we deliver software? How can we mitigate those risks?" I think if you do a good job of that, getting to compliance in many of these frameworks is fairly straightforward. If you try to drive from the problem of compliance, you often end up with a lot of complexity, and complexity actually breeds security gaps. It breeds quality gaps, security gaps. When your system is hard to understand, it's very easy to get it wrong. 

So striving for simplicity and security, I think ultimately gets you to the place where you can say, "Oh, no problem. We do meet all these requirements." I think there are some specific things, specific technology choices, standards, et cetera that are included in FedRAMP that are good to say, "Yes, that's great that you do it this way, but you must use algorithms that have met these standards." Basic stuff like that. But anybody that's writing their own algorithms for encryption is off to a terrible start, and is definitely not on a pathway to security anyway. So I think that's probably best to leave to someone else to say, "Yes, these are the ones that we currently consider acceptable."

[0:15:48] JMC: One sort of like philosophical doubt that I've always had with CI systems and build systems are the difference between those two? Would you agree that build systems are both the process that runs locally only to compile, transpile, interpret the source code into a binary. And that CI would be that same process, but hosted remotely? If you agree with that or not, I would love to know if you could just chip in your opinions about which one is better, or what would you recommend to anyone out there?

[0:16:20] RZ: Yes. I think you're serving different goals. So to your point, I think about build systems. I don't spend a lot of time thinking about build systems versus CI, but I think about build systems as the tools that we use to bring all the pieces together and compile the package. It's not always compilation these days, but construct the artifact that we're going to - 

[0:16:38] JMC: Would you say CircleCI is a build system?

[0:16:40] RZ: Well, in that definition, I'm thinking of things like -

[0:16:43] JMC: Basil.

[0:16:44] RZ: Make, Basil, Meson, like there's piles of different tools. For us, it doesn't matter. We want to create an environment in which you can use those tools collectively, right? With your colleagues, with your teammates, whoever you're integrating with. You're integrating your changes and other people's changes. Sometimes, now, those changes aren't even coming from anyone inside your organization. In fact, in a lot of cases, right? Someone's building a new model, changing some prompts, there's a new third-party library, the third-party service, we use this change. All of those things are different sources of change, that ultimately, you want to rebuild.

So now, you're reconstructing that package, and many build systems also, like I would use Make test, or Meson test, or whatever. Or Basil, you mentioned. Any of these, ultimately have some runners inside of them. What we're providing is, one, all of the automation and tooling to allow that to happen, regardless of who's making a change, and then to the scale to make it happen. So you don't have to worry about managing any of those systems, right? Whether it's, again, under the AWS, the bright AWS light here. Just managing your spend inside of your cloud provider, like spinning up capacity at the peak of your day, turning it all off at the end of the day is just a problem that most people don't want to work on. They're trying to build product for their customer. We take care of all that for you.

That's a little beyond just CI in general. But when I think about what a CI provider is doing, that's what we're doing, is, one making sure that it's happening every time anyone makes a change, and that that's happening constantly many, many times per day.

[0:18:18] JMC: Feedback is provided as immediately as possible, especially when it's negative, when there's a conflict or whatever.

[0:18:22] RZ: Right. Exactly. You want to make a small change, and I make a small change, and we find out our changes don't work together. Let's find out in an hour, instead of in six months, right? Because in six months, it won't just be that one tiny change. It'll be this mess that we'll never figure out how to untangle to get back to the place where we're shipping.

[0:18:38] JMC: So we were talking about before, starting the recording that it would be great if you could - I mean, not great, but you and I are having conversations at the forefront of software engineering sort of like landscape. And we talk a lot about AI lately, and we probably are a bit saturated with it, but it's unavoidable. So I guess, among your customers, are you seeing any change of patterns in the way they build software with CircleCI that has been provoked? I'm asking actually about causation not correlation. But although, probably, it's difficult to find the first one. Have you seen any effect of LLMs incorporating LLMs to build and so forth that has changed the pattern in which your customers, CircleCI's customers are building software?

[0:19:24] RZ: Yes. There's actually three ways that I think about it. One is the tools that they use to write software. So we're thinking about AI coding assistance, and stuff like that, which is generally producing more code. So you're integrating more often. There's people trying to build products that include AI supported capabilities. Features in their product that now depend on an LLM, and so trying to test that. That's a new and novel thing. People are used to testing this input creates this output. Now, it's this input creates an output roughly in this range.

[0:19:57] JMC: Maybe.

[0:19:58] RZ: Right. So helping them with things like understanding the concept of evals, and how do I do non-deterministic testing, and know that it's good enough, and what am I looking for. So that's number two. Then the third is we're actually putting capabilities in our product to help you get feedback faster. So we launched this error summarizer. And if you get a stack trace, we can summarize it for you and tell you probably what the fix is it, because often, maybe not the more senior engineers who've seen it a million times before, but a more junior engineer might look at that, and copy and paste it, put it in Stack Overflow, try to find out what caused that error, and then try to figure out how to fix it. If we can just tell you, "Go do this thing," and we've even tried just doing it for you, which we haven't nailed that at high scale, but we've made it work. To just say, "We know what the test is, the test failed. We'll fix the software and make it pass for you." Because we know all the details of the input, right?

I guess, our customers are changing the way they work, and then we're trying to help them work faster, get more feedback about what they're doing, so they can continue to deliver faster.

[0:20:56] JMC: [Inaudible 0:20:56] step down in the stack since we're at AWS. Are you seeing any changes in two areas in terms of speed, of power, of compute, in the hardware that your clients, CircleCI's clients are using for the built for CI process? And in terms of security, are they looking for the infrastructure that runs the build system, the servers, the CI servers? Are they looking for a particularly secure, more secure infrastructure than they had before? Are you seeing any of those trends be out there?

[0:21:31] RZ: Let's start with the security one. I think that that trend has been there for a long time. Yes, people are becoming more aware, but I think it's been consistent. The expectation that how we manage our systems with their stuff inside. I think, as I mentioned, using OIDC, as an example, that notion of getting away from storing as many secrets with us as much as possible, which is something we're trying to help them do. That's better for everybody.

[0:21:53] JMC: Makes them ephemeral for those -

[0:21:54] RZ: Yes, exactly. On the power and compute front, one of the areas where we've always invested is making available to our customers as many different options as possible. So if you're doing a very small thing, use a small cheap container to do it. Why pay us for more? Why would you want to spend more on that? But we are seeing larger and larger compute demands, including a lot of unsurprisingly GPU access, which is something that we started offering a few years ago. And we continue to expand with new classes of hardware that AWS provides access to new GPUs, so that more and more of our customers can use those to build. Because they're ultimately - maybe not everything is running on a on a high-end machine with a GPU, because that's expensive for everybody. But they can use the small cheap machines to do much of the work. But if they are integrating with a custom model that they've built, and they want to validate the model, and then validate the integration, they need each of those pieces. 

We try to make a large sort of offering or menu available to customers so that they can choose, like mix and match what they need for the purpose while managing costs for their overall capacity.

[0:22:58] JMC: I don't know of the role of chief architect, the responsibilities attached to architect fall under the charter of a CTO, in this case, the CTO of CircleCI with whom I am, is the guest of the show today. But if not, you're probably really familiar with this. The question is, how do you architect an application, a product like CircleCI to be able to adapt to the changes that we just described and others. Probably unexpected, because clients are - it would come with really weird requirements and surprising ones. How do you architect a product like CircleCI? How have you been doing that for the last years so that it evolves and reduces the amount of friction that architectures have naturally to changes?

[0:23:41] RZ: Yes. I think it's an excellent question. It is something that I spent a lot of time thinking about - have for many years and continue to. I think, actually, it's kind of in - the answers in your question, which is designing for change, like thinking about change as a core design principle. I already mentioned simplicity earlier. The more complex you make things, the harder they are to change. I think clear domain boundaries are really critical. So pulling together things that tend to change together. When change spans many domains, therefore, probably many teams, maybe many services, depending on how your system is built. It gets a lot harder than when it is sort of collocated or cohesive. 

So, I don't think the architectural principles are particularly surprising. I think, for example, AI and the mad scramble that everyone is going through of how do I integrate this thing that we never thought about before into my product is helping a few more people recognize the value of what folks have said in the past about designing for change. I think, many of the great writers in the world of software that people whose books you will have read, if you read anything about architecture, talk about change constantly.

They talk about simplicity, and they talk about change, right? So, I think that was is known phrasing in people's minds, but you really need one of these moments where it's like, "No, next week, we need to have something totally different." And you start to see the places where you're being impeded, that you recognize. "Oh, okay. That's what we've been talking about all this time." I think, great domain boundaries, I'm a big fan of domain driven design. Clear ownership, and cohesion, collocation of change, those things are the kind of the underpinnings that allow you to then in a moment like this say, "Oh, no problem." Now, we recombine these components in this other way and we've added this capability.

[0:25:38] JMC: I don't want you to give away any secrets, any particular competitive advantage, or whatever. But if you could give us a lay of the land of the architecture of CircleCI would be great. But the actual question is, what specific areas of the architecture, of the way you've designed, you and your team have designed this throughout the years? Do you feel particularly proud, because precisely, the existence of those features, of those design decisions rather have allowed you to adapt these years to make a product that is highly sought after? From what I have seen, and collected in the feedback of friends that have utilized it in clients is really highly valued. Again, if you could describe us in the simplest way, how it is architected, and what decisions do you feel particularly proud of throughout the years? 

[0:26:26] RZ: Yes. I think there's two things that I would call out. I mean, I'll try not to go through the entire architecture, especially just by waving my hands in the air. I don't think that would be particularly helpful. But a couple things that are probably not super surprising. One is an area that we call execution internally. I mean, it's effectively at the point we know we need to run a job, a piece of work. That might be many, many different jobs inside of a workflow, or an overall build for someone. We hand that off to an area that's responsible for finding capacity, determining the machine type that it's going to run on, finding capacity in that kind of machine, getting it scheduled and run, and doing all of that very quickly. We have high expectations for throughput and speed, and we talk a lot about fast feedback. It's not fast feedback if you're waiting for your build to run.

The group that the set of teams responsible for that, have taken a lot of time over the last few years to effectively cleanly partition that. Meaning, make a really simple API over it. So it's not as aware of how the rest of our system functions, and effectively simplify. I mean, talk about clean APIs, domains, and simplification. Simplify how we manage capacity, how we manage availability of VMs, of Docker containers, of whatever it is. How we connect into, we have a data center full of Macs, so we can do iOS builds, that kind of stuff.

[0:27:43] JMC: Yes.

[0:27:43] RZ: Do all that routing. That is something that then, when we say, "Oh, you know what we really need is GPUs." It's sort of plug and play. Versus, we never thought of this before. We don't have the extensibility for that. Once things are simple, and clearly marked off, sort of information hiding if you think about lower-level design concepts. So that team has done an amazing job of sort of carving that out, and making it really clear to allow them to make fast change in any of those areas. So that's one.

Another area that, again, that would not be surprising to people who know anything about our product, about CI/CD, is how we talk to the providers of source code, right? In the early days of CircleCI, there was one place online to put your source code, which was GitHub, mostly had source code. That's how you were driving change. Then, we talked about like Rails monoliths. It was a very different world.

[0:28:34] JMC: Wait a minute. Did CircleCI never worked with SourceForge? That's a joke. I'm sorry. I'm sorry.

[0:28:38] RZ: We did not. We did not. It comes up every once in a while, I'll admit.

[0:28:41] JMC: Oh, really?

[0:28:43] RZ: I mean, the number of sort of that era of technologies that we did support is, it's a long list. We can talk about it another time.

[0:28:50] JMC: Everything was in GitHub, yes.

[0:28:51] RZ: It was absolutely GitHub, and we were able to get to market faster by leveraging a lot of what GitHub had done. But, then people started to move things into Bitbucket and GitLab. Now, it's like, they're on hugging face, and they're on - like everyone's got a prompt hub of some kind. There's these different places that people are storing assets, and the changes to those assets are driving builds. We had done the work over the last few years, and it was harder than we wanted it to be, to really separate ourselves away from that. So that we could be independent of what was driving the change, to allow builds to run under any conditions. That is something that, I would say, we're proud of, but particular, we're mostly excited about. Because in this moment, where we've had to make those changes, we've already done the work instead of scrambling now. That actually would have been for us. The harder thing I think than the thing I was describing for the execution side to be like, wait a second, we need to be able to totally fundamentally change what we're based off of what we're integrated with. But we had already done a lot of the work. 

[0:29:56] JMC: I really don't know what changes you did for that to happen, but had I been asked, and I'm obviously not a CTO, nothing close to that, or an architect. But had I been asked a year ago, not five years ago, a year ago, that if I was the decision maker for the architecture of a CI system to incorporate other things that was not source code and was not Git based. That I should invest in making my system my product agnostic of the source of the changes to allow to accommodate for those changes coming from elsewhere, that the advice should invest in that, I would say no. 

I find that insight that you guys had is extremely - at least in common to say the least. But certainly, now, I'm sure it's paying off quite insightful, and quite intelligent. Because you guys are then ready for this new way of building applications that is cooperating, or source code, of course, from GitHub, GitLab, the Git gang, as I call them. Probably other source code management systems, but also, these new repos in which other things related to software, because I'm always hesitant to call neural networks and AI in general software, per se, or to inform the builds out for your client. I would feel proud of that. Yes, I would agree with you. It's quite insightful. 

How did you come across that data? What were you thinking when you were saying, "Yes, we need to be agnostic of this, and be as flexible as possible, make our product as flexible as possible with this"?

[0:31:25] RZ: Yes. I mean, I want to claim - 

[0:31:28] JMC: It was all your idea, right? 

[0:31:30] RZ: No, no, not even that. I will say, a couple of things. Organizationally, our CEO, Jim, came from the same acquisition as me. So we had worked together for many years before, and he was always the product person, and I was the tech person in the earlier days. So, just to say, he's a very strong product person, has equally insightful about where things are going. That's very helpful when you're making decisions about how to make big investments that you know are going to be expensive. For me, part of it was, okay, we're seeing - I don't know what it's going to be. I certainly didn't say, "On November 30th of 2022, the world is going to change, and the way that we think about building software is going to change." But we could see the signs, even again, third-party libraries, third-party systems, all of like - AWS is changing something about their EKS structure. That's going to impact me, so it wasn't just the source.

I think, for a while, we've been trying to cram all of our belief about our systems into source like GitHubs, infrastructure as code. I'm 100% in favor of those concepts, but they're limited, right? At some point, systems are changing in a way that we can't always reflect in source. They're changing whether we like it or not, and whether we decided to change them or not. So seeing that happen, and seeing that we were really still focused on the source code was a leading indicator. Again, it wasn't the - and then, generative AI is going to drop in our lap. But rather, this isn't the only thing. We were sort of like - we see all of these other signs of things changing in the world, and we're trying to fit it into this round hole, this square peg, whatever. How do we step out of that?

Then, we did a bunch of the work, partly, just honestly, it just made it easier for us to integrate with all the other Git providers. But as we did it, we did it with the belief that there was going to be a lot more than that. Then. this all landed, and we were like, "Oh, thank goodness. Thank goodness, we got ourselves here."

[0:33:28] JMC: How would this work in practice? Because bringing up GitHub, or a declarative representation of your system, whether it's the complete system, the infrastructure, the application layer, the data layer, everything in a data language. Let's call it, YAML, for the sake of the example. Store it, and get, and have something reproduce that, reconcile that, take it to production. Sounds ideal. I've worked at companies that proponed that. In fact, I've worked at - we've worked with a company that coined that term, if I'm not wrong. I've also, like you been a proponent of that system. But yeah, you pointed out a very strong shortcoming of that system, which is, you need the description of the system to be updated of the underlying system itself. What is CircleCI then proponing that CircleCI would be listening in a way to the system, the parts of the system, and updating itself with the events that come from them? Or how does this work?

I probably butchered the description. But could you describe as the difference between a GitHub's approach, everything as code, and the way CircleCI suggests works well with?

[0:34:34] RZ: Yes. I think that, ultimately, you need a place where you can see all these things. We've tried to do that. I mean, as an engineer, as someone who wants to know exactly what's in my system, I've tried to say, "Great" and I'm still a big fan of infrastructure as code, TerraForm, CloudFormation, whatever you choose to use there. But it's describing a subset of the system. We've seen a lot. I don't know how this plays out. That's the other thing is about a time like this is so many new approaches are coming out that it's impossible to make a bet. So you really want to be flexible. So we're seeing, as I said, like prompt hubs and places to store models, and people making modifications to open-source models, and pushing them back to places. They're doing their own custom training, their own fine tuning, whatever it might be. 

Each of those is independently stored, like people aren't taking their entire model, and all of their datasets, and putting that in source control. Some are trying, but it's not well-suited to the task. So, what we're ultimately getting is that, yes, listening to all of those sources of change, as we call them, and then being able to reflect to you, here's what changed in your system, here's the current state of your system. And, this went well, this went poorly, whatever that is. So that you have a view that represents what's happening in your production environment that came from multiple different places.

I mean, again, under the sort of trying to take a simple AWS example, you might decide and put in your infrastructure as code, and your repo. We're using RDS. But at some point, AWS is going to say, "Cool, we're upgrading Postgres. We're just upgrading the version." But that matters a lot to you, when it turns out that there's some minor version that actually is breaking to what you do. So, you need to know that, and you aren't necessarily consciously making that choice. You're getting an email notification or something. It's not - AWS is not coming along, and updating your repo for you. Knowing that these things are changing around you, and being able to at least react and say, "Oh, okay. Well, let's just rerun all of our tests against that version, and know that we have - like that we're still going to be good. Let's rerun them against a live RDS instance, which is the one that's been upgraded kind of thing, and know that it's still good."

[0:36:51] JMC: Oh, that's brilliant. I really liked that approach, to be honest. We're getting closer to the end of the interview. We're again, reminder that we are at AWS Reinvent 2023. Looking bit to the future, are you guys going to announce anything here? Have you already announced here? Or what is for coming to CircleCI in the next, let's say, year or so? 

[0:37:11] RZ: Yes. Well, I mean, you can tell what's top of mind, a lot of we've just announced some stuff this morning. We're here with a booth. I mean, more access to instances with GPUs, et cetera, so that folks can build and train. We've done some integration. I don't know if we announce it, but we're demoing it with sage maker, showing how you would use models that you've trained in SageMaker, introduced them, and release them confidently as part of a bigger product. Model training is one thing, but ultimately, you're integrating that into your product to deliver features. How do you push that out through our deploy and release capabilities, know that those things are good. Then, that view that - so you have all of these change sources in multiple different places. That view that shows, this is what it is, this is happening in your environment, and then this is how you can trace that back to where those changes come from if you need to get that feedback, or fix something. 

Bringing all of those pieces together is really what we're focusing on here, and helping people as we see those folks, or many of our customers, and new customers evolving to building product on top of AI capabilities. Not just MLOps, but I have a product, and I want to enrich that product with AI enablement. Helping them understand how to test that, how to know that it's good, and then how to roll it slowly into a production environment, release it to customers, and still feel good about it.

[0:38:29] JMC: I'm quite excited that AWS is actually investing in physical, in hardware. They are, they announced, I think. Before we invent that, they will be doing much like Apple, they will be providing the entire stack at some point. I think Microsoft has announced the same. So yes, new hardware is coming, new very powerful hardware is coming. I'm happy to know that CircleCI is actually incorporating more and more of those. I had a conversation this morning, and at breakfast, about a guy that his previous role was struggling to build computer vision pipelines to SageMaker, in turn could update their models that this guy was delivering to the client. I'm fairly sure that - I mean, that companies would benefit from what you just said about deployment pipelines and the better integration of the SageMaker.

So yes, I mean, it sounds like CircleCI is still at the bleeding edge of fast software delivery, which is - and I quote your colleague, Jim, that it seems like in the US, and we are in Las Vegas in the US. The key driver is speed, move faster. This is from a RedMonk article that Kate Holterhoff did in 2022. Is this still the case? Because I think so, because I think so, because yes, it's always been picture perceived as a fast delivery platform. Do you see all this demand still out there and you guys are serving that purpose?

[0:39:52] RZ: Yes. I mean, I think that the, again, a time like this where so many things are changing. People are clamoring for speed. But more generally, the thing I love about speed is it meets - as a software developer, I just get frustrated when things take time. If you completely ignore my customer, it's just frustrating to me. Now, as a product person, someone trying to build product, and put it in the hands of customers, I have exactly the same need. I want to get feedback. I want to build something small, put it in the hands of customers, and learn if it's working or not. If not, why not? How do I fix it? And then ship the next thing, right? Everyone benefits.

Take it down to just flow, like I'm staying in flow, and I'm getting fast feedback, and I'm working, and this is really exhilarating. Or I'm delivering great product, and I'm doing it faster than my competitors. I'm doing it in a way that's really finding the right fit in the market. It wins for everybody. And that's why we focus on it so much. And it's part of it is just the best hardware we can get. But part of it is, how do we get you feedback quickly? How do we tell you right away, this is what's wrong, so you can fix it and get on to the next thing. Across the stack, we absolutely continue to think about speed, and I think always will.

[0:41:03] JMC: That is developer experience, what you just describe. There's one phrase that I like to usually highlight, which is from Charity Majors that she says, and she means it with a different meaning is that, speed is security. This is not a direct quote. She means that, because adding more changes or fast changes to your product will eventually correct the bug, the fault, and so forth, and you will eventually make the product. But I also think that if you provide the developer the ability to go fast, and remain in deep work, and that mentality that you just described, this person will make software secure. Because it's the sweet spot of concentration that will allow him or her to be like aware of many things. So I really like this approach that you guys have to develop an experience and how you provide that. So with that, Rob, thanks for being with us and enjoy the rest of the week.

[END]