Kube Cuddle

Rich: Welcome to Kube Cuddle, a podcast about Kubernetes and the people who build and use it. I'm your host Rich Burroughs. Today I'm speaking with Gearhard Lazu, a Senior Software Engineer at Dagger. Welcome Gerhard.

Gerhard: Hi, Rich. It's nice to be here. It felt like it was only yesterday when the recorded the Ship It episode. So it's a deja vu moment. I really like it.

Rich: Yeah, for sure. So, uh, so for those folks listening, Gerhard, uh, hosts a podcast called Ship It! um, which is fantastic. And I, uh, had the privilege of being able to appear on it recently. And so, um,

We've recorded with the roles reversed, you know, and now, now I get to be the host, which is actually kind of where I'm the most comfortable. So I'm excited about this.

Gerhard: Yeah. I mean, I will, I will have to work really hard to not be behave like a host and be the guest this time, because I think this is the first podcast where I'm actually a guest, um, which is not part of Changelog, because even though I was a guest on the Changelog podcasts, on a few episodes before we started Ship It! with Jared and Adam, uh, I I'm like host mode by default.

So I'll try really hard.

Rich: So shipping is part of the Changelog podcast network, right?

Gerhard: Yes. And it started, so initially it was one Changelog episode every year where we go to talk about improvements that we did on the Changelog infrastructure. And that's where I would appear as a, as a guest and Jared and Adam would be the hosts. And, um, that went really well. And people said like, let's do more of this.

And then, you know, eventually we started Ship It! And then I fold it to host. And I think 42, 43 episodes later, 42 is the one with you, which is going to ship. Um, yeah, only host.

Rich: That's fantastic. I will definitely link to the episode with me, um, in the show notes so people can check it out. It was a lot of fun to record. We talked about the things that, um, we're doing, where I work at Loft Labs, we talked about ADHD, um, all kinds of fun stuff. So, uh, please do check that out.

Um, yeah, I just usually start off by asking people a little bit about their background. Like, how did you actually get started doing this computing stuff?

Gerhard: Okay. Now this will, this will be a surprise, uh, because I know that's like, most people don't expect this. So I got into computers without having a computer. And the way that looked, I used to used to go to the British library. So I grew up in Romania and, uh, I used to go to an English school, it was called William Shakespeare.

And, um, part of that English school, um, English was like, you know, everywhere and computers in my mind equal English. Right. So, if you know English, computers come easy. And th th the interesting thing is that English was actually my third language and it still is, English is my third language. So languages, you know, and, and that's, so what is like, you know, Java versus, I don't know, Rust, you know, when, you know, you know three actual languages or even more. Anyways, I used to go to the British library and I used to get the, I remember it HTML 4 book

Rich: Oh, wow.

Gerhard: And I would design my website in a notebook by writing HTML tags, like on paper. And when I would get a chance to, uh, a few years, I think two years later, when I got a computer, I used to transcribe all those notes from the notebook on the computer. And it mostly worked. I was surprised. CSS was not a thing.

Back then. CSS was like this cutting edge technology. So I started writing HTML on paper, learning about it without the computer. And that took me, I think, about like a year. Uh, I learned about JavaScript. Uh, there was no jQuery, jQuery wasn't invented, uh, um, uh, XHR requests were like the cutting edge. Like what's like the cusp of what's possible, you know?

So,

Rich: I remember those days. And, um, it's funny to look back on them. I actually did a lot of HTML back then too. And I worked as a kind of freelance web designer for a while, while I was in college, which is a, was a load of pain. Um, because, turns out people don't want to pay web designers much and they don't want to give you the check that they said they would give you.

So that was, that was fun. But, Um, the thing that I loved about those days is that since it was all so bare bones, like, like you said, you just had to know some HTML, right. So like it all was about the content, it all was about what you had to say and like, you know, you or me, like any of us, we could make a website as good As a website that Microsoft could make.

Right. Like, because it was just about what you had to say. And that was pretty amazing.

Gerhard: As long as they didn't involve Flash, I know Flash was, was the rage back then. And, uh, I tried it out and I hated it. It's like I hate ActionScript. Uh, like it was just so complicated. I really didn't like it. And I'm glad that, you know, JavaScript is still here. ActionScript, not so much. And when Apple decided to not have Flash on the iPhone, I was, I was saying, yes, that is like an amazing decision,

Rich: Yeah

Gerhard: because never, yeah, I never got along well with it.

So this was 20 something years ago when I first started. And from there I went to PHP. Um, I, I enjoyed that PHP, MySQL, learning about it. I still have, I still have the book, you know, in my library, which, which got me started with, I think it was PHP 3 at the time.

Rich: Yeah, same. This is really funny. We, we overlapped quite a bit in our experiences. I think.

Gerhard: And, um, from there I discovered Rails and I really liked it. I was in the Ruby ecosystem for maybe, maybe 10 years. I really enjoyed my time there, but, um, I got the whole deployment aspect of applications and of code really fascinated me. And that is just like a step away from infrastructure and operations and how to get code out there and how to not to break things.

I mean, I took production down so many times that's I like stopped getting concerned about it. Like, okay, I'll get it down and they'll fix it. What's the big deal, you know, there's no big deal. Uh, so I was also lucky, you know, from, from many respects, but, um, from there it was just a short, you know, like a short step to where it's Capistrano, towards Puppet, towards Chef, to where it's, uh, CloudFoundry, Ansible. And then Kubernetes came along and CI systems, and then the list goes on, but that was the, those were the early days, like, I think like the first decade, first, maybe decade and the bit,

Rich: Yeah. you don't hear about Capistrano anymore, but that was a really powerful tool. It was super useful.

Gerhard: Uh, it was, but it was so complicated to wrap your head around. So one of the things which like after trying it for years and trying to like to work with it and like, adding extensions to it was so complicated because of the DSL. It was really difficult to work with. And Chef's DSL was a little bit like that, but it was easier.

Uh Chef was also using Ruby. So, you know, writing, I forget what they're called now. Uh, I was gchef like Gerhard Chef, because that was, that was like the startup where I was like writing Chef cookbooks like crazy, the NGINX cookbook, and NodeJS cookbook and a bunch of other cookbooks. I think the org is still on GitHub. github/gchef. Yeah. It's still there. Um, I mean, they're really old at this point, but, um, the DSL, like the Ruby thing, that's how that's, it made it easy for me to transition into infrastructure and into, to understand Chef and play with Chef and Capistrano. And I got so frustrated by those tools that I decided to write my own.

Rich: What did you write?

Gerhard: It's called Deliver. And it was Bash all Bash. I was like, okay, this cannot be this complicated and, um, gerhard deliver its archived now. But that is the, uh, the project, which, which became really popular. It had like strategies, deployment strategies. It could be like multiple types of apps because I was part of a startup it's called Go Squared at the time.

That's where this was built. Um, and they were primarily NodeJS, but I was Ruby. So I was writing like EventMachine code and then Goliath and, you know, those sorts of things. And they were focused on NodeJS. And we were like, there was a bit of a competition between like, no, my Goliath is better than your NodeJS V8.

And, you know, so it was really fun, but we had to make those different types of applications work together. We had static apps that we have a bunch of apps and this was running on AWS. So the simplest thing it let's SSH into this machine, pull the code down, do a couple of checks and do blue-green restart, you know, without any downtime. Like how about that?

Like how difficult can it be? Obviously, I didn't know that Ansible was a thing because knowing this was using SSH as well, and on, on, on, under the hood, eventually when I discovered Ansible I switched to, to that, but Deliver was interesting because it was really simple. And like, how far can you take this idea?

And then there was a fork Edeliver, which was Erlang deployment. And that's how I got into Erlang ecosystem. And that's how Jared from Changelog started talking to me like, Hey, we have this app it's called changelog.com and we are trying to deploy it. Your lane name up, you wrote like a blog post about Chef for us.

Like, can you, can you help us like with his deployment, because they rewriting it at the time. This was, I think 2006 or 2016, I think 2016. Yes, 2006. That's right. Since 2016, they rewrote the app from WordPress to Phoenix, which is a framework similar to Rails, but it's running on the Elixir, which is a DSL running on the Erlang VM.

And either Deliver was one way that you can, you can deploy, you could deploy, um, Elixir apps and the Erlang apps. my name came up and then we started talking and then the rest is history.

Rich: Interesting. That's really cool. I, um, I don't have much experience with Erlang. I worked in one shop where we used it just a little bit, but I hear so many people talk about how amazing it is, especially in terms of, you know, distributed applications and, um, being able to handle lots of concurrent connections, uh, that kind of thing.

Gerhard: Yeah. That's right. So I'm not sure whether this came up in your research, but, um, I was on the RabbitMQ team for about seven years. So, um, I spent a fair bit of amount of time with, with Erlang and I think my, the contribution, which I'm, which I'm most happy with is my involvement with the RabbitMQ Prometheus plugin, which will was exposing Prometheus metrics from RabbitMQ, via plugin, built in.

And there's no need to know have like an extra exporter running and reading the API. And then, you know, so it was a lot more efficient. And part of, part of that though is like, how do you, how do you scale a RabbitMQ cluster? How'd you get the most out of it? And how does the Erlanng memory allocation work?

Because it's a fascinating system. And when these things crash, because it has like supervision, trees and all that, how does that work in a clustered system? You have like 3 nodes. Like what happens, what happens like when a queue moves around, they're all processes, all message parsing. Uh, it's a great technology.

I really, really like it and it can take some abuse seriously. Like it'll keep crashing and restarting and you can write some pretty bad code. Uh, and it's so resilient. I loved it for that. I love the resiliency of the Erlang language and the clustering. Now Mnesia not so much. It's like the built-in database, really old stuff.

Um, not great, but, um, that's what RabbitMQ had for a long time. I mean, for 10 years. Very difficult to change because it was like everywhere in the entire code base. But, um, apart from Mnesia I read like Erlang I have to say it's really, really nice.

Rich: We're here obviously to talk about Kubernetes. Um, we, we, uh, had a little chat before we started recording and, um, I, uh, I had asked you about whether you'd ever contributed to Kubernetes, um, and you have not. And, and I thought it was really cool that you are the first person who's come on that I really think of as a Kubernetes end user, as opposed to somebody who's like in a SIG or working for a vendor or something like that.

And, um, I even say in the intro to the episodes that it's a podcast about the people who build and use Kubernetes and, and you're one of those folks who use it. Um, so you mentioned that you, uh, had some reasons for why you hadn't become a contributor?

Gerhard: So I really like to play with tech. I really, really enjoy learning things by just trying them out and seeing what sticks. And with that mindset, you end up trying a lot of things. So like my, my, my breadth is huge. My depth, you know, not so much. So, um, for example, I haven't written a Kubernetes operator because there was no need to, I helped contribute to one, the RabbitMQ one. It's the joy of using the technology and finding combinations for me, that makes sense.

And it's the philosophy of Kubernetes and the ecosystem, which fascinates me and how it fits with everything else. I was doing Kubernetes before Kubernetes, which is, which is a bit weird because people don't realize CloudFoundry had a scheduler. And I spent like years in that ecosystem. Now, did I write the scheduler?

No, but I knew very well how it worked. I used to be called Diego and that was a rewrite of the previous Ruby one. I forget what that one was called. Um, so CloudFoundry was in a way an orchestration platform. Um, I, it was all about the containers and there was a runtime. And then all that. Um, no, I may be getting some of my details wrong, Diego, I think it was, that was the runtime, but I think was also the scheduler I have after that. I have to double check that anyways. Anyways. Um, there was also BOSH and BOSH came out of Borg. So how do you basically manage a large number of VMs? Right? Because it had like a CPI, which was like a cloud platform interface.

You could run it against AWS, GCP, a bunch of things. So it was orchestrating deployments, uh, compiling packages. You had the releases, we had like the, the director, which you can think of it, like the Kubernetes API. It was very similar. So that was BOSH, that was CloudFoundry. And there was Concourse and maybe people don't realize Concourse, it was a CI, but it also had a scheduler and everything was running containers. And, um, these three things that you were, I mean, Pivotal was like a, but a big driver. And I was, I was part of Pivotal for like a couple of years. Um, they were an alternative to Kubernetes, but obviously Kubernetes won. So Kubernetes was, was like old stuff to me because I recognized it from BOSH.

I recognize this from CloudFoundry from Concourse. So I was like, wow, this is amazing. So what else is there? So never spent enough time to go like, oh, let me contribute to this. Or let me contribute to that. Like, what else can I combine? What else can I pull in? What comes after Kubernetes, what should sit alongside it.

And there's like so many things. So how do you do for example, upgrades? Well, that's something which always fascinated me, um, being part of the Pivotal CloudFoundry services team, uh, we had to deal with upgrades a lot. Like how do you upgrade RabbitMQ? It's a stateful distributed system and things are really hard, in that world.

So Kubernetes was always like a playground for me and I, I love the people. I love the ecosystem and that's what attracted me. And that's what fascinated me. So I tried to do as much of that as possible, rather than writing code or documentation or going in depth.

Rich: Yeah. What you mentioned about CloudFoundry is super interesting. Um, I didn't ever have much experience with it, and I didn't really think of it as like a scheduler like that, but that totally makes sense. Um, there was also Mesos, know, and of course Borg, you know, which Google used internally.

And so these ideas didn't come out of nowhere,

you know? And that's one of the reasons that I like connected with Kubernetes so much in the beginning is that it was, it was very much an extension of the good operational patterns that were out there at the

Gerhard: Yes. Yes. I mean, when, when the health checks came about and, um, liveliness probes and readiness probes and all that stuff, like, yes, that's exactly what we were trying to build in BOSH, different checkpoints, different checks, you know, like jobs that run a design and Kubernetes had it. So the concepts that were there were not new, but the way they were combined.

It fascinated me and it's attracted me. So this is a very easy switch. No, it didn't take me long. And I'd liked how everything was coming together. And I love the ecosystem. I mean, all the tooling that was built around it, people say, well, do you need to run Kubernetes? Do you need Kubernetes? Well, no. But do you know about all the things that you need to do, whether it's Terraform or CI/CD system, like, so I'm trying to run as many things as I can in Kubernetes.

And it works very well. Like, you know, DNS updates and upgrades and, you know, syncing jobs, like Cron jobs, all that stuff, everything that, that would run in different systems in different ways. I just put it in Kubernetes and it works really well.

And you

Rich: yeah. Being Dave. Yeah. One of the things that just blew me away initially, and, um, it was Kelsey Hightower that I first saw talk about Kubernetes like a lot of people I'm sure. This was way back in like 2015 was, just being able to update that version number in a YAML file and hit save, you know, and suddenly you're running the new version of the app, you know, that was, um, that was something that, you know, like you, you mentioned, like how you had built a whole tool to like orchestrate deployments.

Right. And, and a lot of us were doing that same kind of thing before Kubernetes

Gerhard: right. I mean, BOSH and CloudFoundry was working exactly like that. They were continuously converging. Um, it was slightly different and there's like, there's features that people, I know they don't know even existed in these tools, you know? I mean, um, I don't think we'd like the best of jobs, like to, to make them popular.

And I think there were like some critical features which were missing. Like we had plugins, for example, for CloudFoundry, like, like, um, blue-green deploys. Blue-green deploys were not built into the platform or Kubernetes had it. Like, that's just how it works, you know? It's like, okay, you can configure it to not do that.

But by default, it'll try to do that because it's the right default. It's what most people want.

Rich: Absolutely. Yeah.

Yeah.

That was definitely the, the, um, the pattern at the time that, that everybody was kind of leaning towards. Yeah. I meant to watch the Kubernetes Documentary before we talked and I keep forgetting, um, as we discussed on Ship It!, I have ADHD. I need to like put a sticky note on my monitor or something to say, like watch that documentary, but I know that you saw it and you, um, enjoyed it quite a bit.

I think from, from what I heard, I'm wondering if, um, there are any things in there that got revealed that really surprised you and don't worry about spoiling me. That's

Gerhard: Yeah. I didn't realize how big of an effort it was and how many times, or how many points that were in, at the beginning, for, for it to never happen. It could have never happened a couple of times. And by some sort of luck, you know, chance, I don't know how to call it. It did, you know, things just kind of worked out in meetings and, um, you have to convince like this VP, that VP and, you know, some people don't like it. It's like, what is this thing is like a waste of time.

Like why, why, why even do this? So that's very, um, eventful and not eventful, let me try find a better word. Um, Uncertainty, the uncertainty around it.

uh, we in a different world, may not have had Kubernetes if it wasn't for specific people, specific moments. And I think that would have been a poorer world.

And I know that there's still many people that don't like it, and they think it's too complex. And that's a fair point, you know, no one, no one disagrees with that. But, um, if you try to do all the things that Kubernetes does, you will end up with the same complexity anyways, because the things are complex and it's trying to simplify it as much as it can, but they're inherently complex.

Um, you will have it's whether you want it or not. Uh, maybe not in all in one place for you to see, ah, this thing is complex and we have 10 things and it's easy to reason about them individually, but when you put them together, you end up with, I would say, a distributed complexity rather than all in one place, you know, like, oh, this thing is hard.

What thought? Like, what are CRDs? I don't know, 600 CRDs. It blows your mind. All, all those things. Anyways, the point being it could have not, it's sheer luck that it happened. Uh, some people worked really, really hard to see through. Um, it was a wild ride for many people, sacrificed a lot of free time, a lot of family time.

Uh, they poured themselves into the project and I'm very thankful to them because otherwise we would not have had it. Um, That is the thing, which makes me want to contribute to it, or like want to have like, been part of that. But I wasn't in that group. So I couldn't have done that at the time. I wasn't aware of these things like looking backwards.

It all makes sense. But at the time I wasn't aware of this, but the thing which stuck with me the most, and I think that the producers did an excellent job of this documentary, is what Kelsey said at the end. Uh, and he said, that's, it's not a zero sum game that all the, all the orchestrators, like Mesos versus Kubernetes.

I don't think he mentioned CloudFoundry specifically, but in my mind, I, I filled it in, like it was there. Um, People tried like to, you know, have winners and losers, but why, why think like that, you know, it's like an ecosystem that is becoming richer. Like the more perspectives you have. And I don't think that Docker was like a loser in that battle.

And there's like so much there that is so controversial. And working with Solomon and working with Sam, working with the people that were behind Docker and that were on the other side from some perspectives, it just puts a whole new spin to the whole story. So without going into too many details, it was fascinating to realize that Kubernetes as amazing as it is, uh, there was so much pain to get it out there and to, you know, sustain it, and the CNCF, I think they did an excellent job.

And I think there was like a couple of other things that happened, which would not have, um, which if which if they hadn't Kubernetes wouldn't have been where it is today, and I think

CNCF is a huge, huge part of it.

Rich: That's super fascinating. I'll definitely have to watch it.

I've even more excited to now.

Um,

one of the things that's kind of fascinated me, um, like over maybe the last couple of years is the fact that those early versions of Kubernetes that shipped like, there really weren't the multi-tenancy and security features that I think that people would expect of a container orchestration platform that you're going to run in production.

And it always, it's always kind of fascinated me because Google had years of experience running Borg internally before that. And so I'm, I've always been curious about the fact that, you know, I can understand wanting to ship an MVP or whatever. Right. But like Kubernetes is still catching up from the fact that, that those pieces aren't there.

Like they're, they're still constantly releasing new security

features to fill some of those gaps.

Gerhard: Um, the way I use Kubernetes is very different from how people think of Kubernetes. Like for example, I use single node. People go like, what? You're crazy, single node what that's like, no, it's just wrong. Well, it isn't once you consider how difficult getting the PVIs to work the persistent volumes, um, like the, um, I'm blanking, CSI.

So like the, the CSI that's, that's what I meant. Um, the container storage interface, the specific implementation of, for specific providers, it suffers from certain limitations, which are platform specific. What that means is that sometimes detaching volumes and reattaching them will result in more downtime, then, if you can reschedule like a whole new app without any persistent storage. If you can do that, and if he can restore from backup quickly, that may be the better option. It doesn't work for everyone. But knowing your use case is very important. And sometimes that is the better alternative, which is what we do for Changelog.

So we had so many issues when we were using Kuber Kubernetes, the way it was meant to be used. And we said, you know what? Let's not do that. Let's look at what we're trying to achieve and let's achieve that. And it doesn't matter what people say, Well, the best practices are, because they're not best practices for us.

And having that and the courage, the insight, the, I think you need to be a pro user.

Rich: I hate that term best practices and a lot of the folks that I know and respect do as well, because there are, there are things that are really good practices depending on your use case. Right. But I saw that joke somewhere recently where it's like, you know, the, the senior engineer or infrastructure person, you know, that their, their line is, It depends.

Right. You know, which is, which is always the case with me. It's, um, we've all got different use cases. And, um, for some people uptime isn't even that important. You know, it

Gerhard: yeah,

Rich: a lot on, on what it is that you're doing.

Gerhard: yeah. And if you can put a good CDN in front of it, maybe it's okay if the origin goes down. And how about you design a system where Kubernetes is just one of the runtimes and people are like multicloud. Well multi-platform is what I'm thinking. So what about using Kubernetes? And the PaaS so one instance of the app runs on Kubernetes, by the way, the Kubernetes API is amazing.

I mean, that's just like a killer feature and I love it. And like the whole control loops and how they work and the insights, and there's like a bunch of stuff. So, and that doesn't change whether you use three nodes or one though, it's still the same thing. So use Kubernetes use the platform as a service use, whatever else you need.

Like if you have a bare metal instance and there's a way of, of just Git pushing and the app update, the code update, makes it somehow on that instance, who cares, that's the whole point, right? You want the code updates to go out there as quickly as possible. You want to ship it as quickly as possible to see if it works right?

Because it may not. And as long as you optimize for that, forget all the other preconceptions, what you should be doing or what people think of it. I mean, if it works for me and it works really well and we didn't have any downtime in months now, um, I mean, okay. Uh, Fastly went down, I think last July, first time in five years.

That's I think that's the last time when we had like any significant downtime, you know?

Rich: That's super interesting. Um, so you mentioned that you're working with Solomon. Um, we'll talk about that a little bit more, um, about Dagger, but I w I wanted to ask you, you, um, you had been using Docker in production before, um, was that just with CloudFoundry or were you ever like running the Docker daemon, and, and, um, running, Um, like containers with.

Gerhard: Um, okay. So I need to, I need to start like with, uh, with like maybe take a step back and look at the high level. Regardless of what I'm using in production, I'm trying to do the opposite elsewhere. So I'm trying to always test my preconceptions. So I was thinking that's CloudFoundry is the best thing ever, but.

Can I see if Docker, for example, and Docker Swarm would work better. So we try that. What about Ansible? I mean, could Ansible be better than, than, than BOSH? In some ways it is, but how do you get that experience? And this is like, where, like my mindset is that user that tries as many things as possible. So to understand the trade offs, because every piece of technology, including Kubernetes will have trade-offs do you know what they are? If you think it's amazing and there's no issues with it well, you don't know it well enough. So let's talk when you do. And then you, and then you can use the right tech for the right job. I mean, it is what it is. And I was using Docker in production for my own projects. Um, like hosting my, I used to, I still host websites for like 20 years now for like customers.

That's how I started, like, as, as a freelancer, uh, designing websites, hosting websites, WordPress websites, a bunch of other things. So could we run them in Docker Swarm? And I still have an instance which runs Docker Swarm, which in production. And it's been running. I know for like six years now, eight years now, I forget how long it's been.

Um, and I think it's, I didn't run an Ansible run in maybe three years. It's bad. Sure. But you know, it's like, you know, it's, it's okay. You know, it's, it's okay. It's, you know, it's, it's not a problem. If, if everything is lost, I mean, there are backups and I can restore that. It's hasn't been a priority. The point is I have been trying different things out for a long time now, just to understand what works better when, and sometimes Docker was a better choice than Kubernetes.

And in some cases it may still be the case, you know, like maybe Kubernetes is too complex for it, great, maybe have a very simple setup. Maybe Docker is enough. Go for it. That's something that we did for Changelog as well. One thing, which they appreciate is that the every year we used to improve the platform. We used to run Docker swarm, and then we took the next step and went to Kubernetes.

And this, you will take a next step and they'll be the post-Kubernetes Changelog, but it's not like an, or it's not like Docker Swarm or Kubernetes. Now I'm thinking, how does an and proposition look like? Can we do Kubernetes and something else? I mean, that just like shows like a new level of understanding of maturity, of like, how do you orchestrate all those different types of systems?

So Docker, it was great for a long time and it's still great for some. Um, We don't use it for Changelog anymore, but I still used myself in certain places. I mean, I'm not going to tell you the Ubuntu version because it's so old, it's like out of support. I would love to give you the public IP address of, of that instance.

Oh yeah,

Rich: to this are just cringing

Gerhard: yeah. yeah. They will. They will not enjoy that they will not enjoy that. I mean, I think has a crazy uptime, you know what? This used to be a thing, like my it's, this has like a thousand days of uptime.

Rich: Oh yeah.

That used to be how you

measured, like how good of a sysadmin you were is, is, what the

Gerhard: Yeah. I mean, this is, this has like thousands days of uptime, which is not a good thing, but there you go.

Rich: But I, I really appreciate what you're saying. And, um, I feel the same way about Nomad, you know, like, Um, I know there are shops out there who use Kubernetes and Nomad both, right. It doesn't have to be either or, and they use Nomad for some of their use cases because it's a better fit. And I think for some shops, especially if you're really heavily invested in the other HashiCorp tools, as in like that whole ecosystem that using Nomad might make a lot

Gerhard: Um, I wish I had more time to play with all the, all the different technologies, seriously like Nomad I wanted to try for such a long time. I kept mentioning, I think when I interviewed Katie Gamanji , I was telling her that if I was to choose an alternative to Kubernetes, it would be Nomad. Um, this was, I think mid 2021.

I would still love to be able to try Nomad out, but there's just not enough time. Priorities. But Yeah,

Rich: Um, all right. so let's get to Dagger now. Um, so, uh, new startup, um, Solomon Hykes, is there one of the co-founders of Docker. Um, I'm super intrigued by this. I remember seeing Solomon tweet about it, and I went and looked at the website. Um, I'm taking it, you're one of the first few employees

Gerhard: Yeah, that's right. Yeah. That's right. So I mean, Solomon, he, uh, he, he's the one that, you know, it's it's, he is instantly recognized. Like everybody, or almost everybody knows Solomon for, for his involvement with Docker, but there's other people that are part of Dagger that were at Docker as well. I want to, um, mention Sam, uh, I want to S Sam Alba, let me, let me use full name.

Sam Alba, Andrea Luzzardi. People don't know him because he is not someone that you would know, but he was an essential person in creating Docker Swarm. And, know, like the whole architecture, how it all works. Um, he, he knows the most about Docker Swarm because he helped build it. And I don't want to like big him up too much.

I don't wanna say like he is the creator of Docker Swarm, but maybe, I mean, to some, from some perspective, he is, um, um, there there's, uh, who, who else? Uh, Joelle and Guillaume and Richard. And there's like so many other people have joined, like Helder and, uh, Comey. And, um, I think we're 13 now, 12, 13, um, it's, it's changed so quickly.

So I've been here for two months and there was always interviews and new people. So it's like, you know, it's like, I, I know that he, or she will be joining, but I can say just now. So there's like, in my mind, there's like a lot more people with us that there actually are, but, uh, all it's changing very fast. Um, It's an amazing team.

It's an amazing team on the, my, my guiding principle, when, when I, um, was thinking of what comes next for me after RabbitMQ and VMware, I was saying it will have to be a startup, but it will have to be a startup with a handful of people that I'm getting really well with that I enjoy just every single interaction with, and that I'm just like blown away every single time I get to talk to them.

And it took me a while, you know, to figure out who this group is and, um, Ship It! Was helping a lot, you know, in terms of having different conversations with different companies, with different people now, like finding what am I actually looking for? Like, what does this thing look like? Like what, what is important to me?

And I dunno, it was like, it's really difficult to describe it. Between Solomon's genius and vision. I mean, he's just, ah, I it's really hard to describe a person like him. I never met person like him. He's he's his mind is like a thousand miles an hour all the time. I think he's maybe 20 steps ahead. Most of the time, I don't even know what he's talking about, but two days later, three days later I get it like, oh yes, that makes starts making sense.

And two weeks later, okay, I get, I get it now, Solomon. So he's like so far ahead of everyone else. It's just unbelievable. Um, Andrea's so strong technically. And he has like a very good grasp of LLB and CUE and Docker and OCI and all that stuff. Um, um, Sam, I mean, he's, he's a great people person. He really is.

He really understands people and how to interact with them. And I had like such a, such an, I still have, uh, such a strong connection. Um, like on that side, like on, on like the soft skills sides, um, Eric, I, uh, he's, he's also great. I mean, I'm not getting to talk too much about that because he's like, almost like the, uh, I dunno, how shall I say this?

The, the, um, The almost like the king that no one knows about, but he's super important, you know? And he's like, you should know about him. I mean, the fact that he's there, it makes all the difference, but he's, he's really important. And this group of people that I worked with and, um, because we did like, um, a Ship It! Episode, I think it should be 23.

That was like, um, about Dagger, try and get out. Um, we did Ship It! Episode 33, which was like Christmas special, where we converted the Changelog. We migrated the Changelog pipeline from CircleCI to GitHub Actions using Dagger and trying these things out and working with different people in the Dagger team made me realize this might be it.

These could be the people that I've been looking for. And

a couple other things happened and everything just fell into place. It was just, it was just meant to happen. That's that's,

Rich: So I know. Yeah. So I know that Dagger isn't out there yet. Um, but there is a website that's got some kind of conceptual information about what's going on. Do you, can you describe it for

Gerhard: Yeah, sure. Okay. So Dagger is, I want to say it's a CI/CD toolkit because that's how it was known, but it's so much more than that. So let me talk about the things that I think are so difficult in the CI/CD ecosystem today. Um, because people will relate to this a lot more. One of the things is not being able to run your CI locally. GitHub Actions, any CircleCI you have to commit and push to see if it works well.

What if you could run it locally? What if you didn't have to commit and push, just make a change, you run it and see if it works. Okay. So that's the one thing. other thing

Rich: wow. Okay. That, that one right away has got my attention. Right. Because I think there's, there's, uh, our CEO and I have talked about this before our CEO at Loft, um, Lucas, um, about the fact that there's so many people out there who kind of rely on the results of the CI run, right. To know whether their code is, working or not, um, to they, they rely on that integration testing and all of that.

But, um, but as we know. like tightening those feedback loops is, is super helpful in terms of velocity. But, but also I think in terms of like making the developers feel good about the work they're doing, right? Cause like nobody wants to sit around for an hour and find out whether the thing they

Gerhard: Mm. Hmm, definitely. So no queueing no, or anything like that, you run it locally, as long as there is a Docker engine, um, Dagger can figure everything else out. Now, Dagger, it needs BuildKit which is like one of the super powers. And I'll get into that next, but I want to talk about, so first of all, there's like this local CI. The second thing is having a way of describing your pipeline, which think, Go format, think defining types and defining constraints.

So no more YAML, no more like, will this even work? Like, did I miss a space or did I maybe set the correct value or will 1.0 be converted to 1 and not remain 1.0 because you have to double quote it like, like, various YAML weird things.

Rich: I've I've never experienced anything like that. I've no idea what

Gerhard: Really. Okay. Yeah. It's so bad. You just want to forget about it. It's be the erase from your memory. Yeah. They later, right. A day later. Oh, damn it. The double quotes, you know, I don't let you read. So yeah, I mean, I spend a lot of time with YAML with, with, with BOSH, with CloudFoundry, with Kubernetes, it's like a decade of YAML and, uh, there is a love, hate relationship with it.

And when CUE came along, which is a configuration language, which was inspired by GCL, the Google Configuration Language, a lot of cool stuff coming from Google, like Kubernetes, GCL, and CUE is one of them. So what if you could declare things in a way that is so easy to validate in formats, like literally think Go code. You're writing your configuration and it's still configuration.

It's, you know, it's not like it, you're not programming. Uh, you're just writing your configuration. You can lint it. You can for, you can format it. It just has built in support, like in your code editors. And you know, that looks right. So you write CUE, you don't write, and this is like CUE that's, for example, CUE is, I think I have to double check this.

I'm I'm

Rich: so CUE is spelled

C U E for those folks listening.

Gerhard: Lang. Yeah. Configure, unify execute. That's what the three things stand for. Cue is a superset of JSON uh, what that means is that, if you're familiar with JSON and most people are, um, it will come really, really easy. So you can take CUE, like a CUE file, you can do cue export, whatever the file is dot q, and you'll get JSON spit out. So that makes it really, really easy to wrap your head around and work with.

But you have types, you have constraints, you have so many other powerful things. And declaring complex things like complex flows is really easy. It has like this, this, this concept of this junctions, which has combined things. And it was like, which branch to follow. And it's really powerful way of describing configurations.

Um, as much as I used to love uh, YAML um, I think after I discovered CUE I just want to write CUE everywhere. So how do we simplify it? How do you, because you, you, you can take you really far and like, it becomes, can become a bit cryptic, but I think it has the potential of being the next YAML. Like what comes after YAML.

I think this could be it. see. I mean, I may be wrong, but yeah.

Rich: I really need to take a look at it because, um, what you're, what you're saying is super interesting to me. And this is an area, you know. I did years of the configuration management world. Like you, you know. I was working with Puppet as user for a while, and then I worked there and, and, you know. it had its own DSL and I've worked with the YAML as well.

And, um, I feel like a lot is a lot of people's, um, relationship with YAML isn't love, hate, but it's maybe hate

Gerhard: yeah, yeah. I know,

I know. I know. Yeah. Okay.

Rich: pain people feel with YAML.

Gerhard: Yeah, for sure. For sure. Uh, yeah, I can definitely relate to that. Um, the other thing which I wanted to say, actually, actually, so, so, so there's, so there's, so there's the local CI aspect. There's a configuration aspect and then there's BuildKit. Okay. So BuildKit is how Docker internally executes the instructions in a Docker file.

So every single like Docker file line, and this happened like in the last, I know, three, four years, five years, um, it has this mode where it can parallelize instructions and every

single line is just like a way of running it. So you imagine this DAG which gets created from this Docker file,

Rich: So DAG is,

Gerhard: uh,

Rich: uh

Gerhard: DAG exactly like th Directed Ayclical Graph like a graph, which gets created from all the instructions in a Docker file.

And then off you go and you execute. And that, and they run in the LLB, the low-level builder, which is a part of BuildKit. And BuildKit is part of Docker Engine. But that is only 10% of what BuildKit can do. It has caching properties. It has volumes secrets, how to handle secrets securely so that they don't leak, which is something that we've been told not to do in Dockerfiles, because this was never enabled in Dockerfiles.

But this is possible. You can, you can get secrets through in your pipelines in a secure way. They will not. And they'll be discarded correctly and everything will be great. So you can get like secrets from like the secret store. Use them just like, you know, where they're needed and then securely discard them. Whether it's an environment variable, whether it's a file.

So this is what Kubernetes was not able to solve easily. Uh, because base 64 is like, let's be honest. It's like, seriously,

Rich: I, I prefer

Gerhard: Yeah,

Rich: myself. Um,

Gerhard: right. Sure. Okay. Okay. Anyway, so, so, so, so coming back, so, so BuildKit and LLB looks like some amazing, an amazing technology. And that's what Dagger used for the runtime. So all that CUE generates instructions for the LLB, creates this DAG, that LLB just goes and execute, executes, and it has built-in caching properties.

So, Hey, I've already ran this step. I don't need to rerun it. You have caching validation. So imagine pipelines that automatically know which steps have run and that they don't need to run, which makes things really, really fast, really

Rich: and from, from looking at the website, it looks like I can, I can describe what my pipeline is and it's portable. I can move it

Gerhard: Exactly. You can run it locally. Yes. Because you already do "dagger do" this is changing by the way, it used to be "dagger up" and it will just execute in Dagger. And it doesn't matter whether whether it's GitHub Actions, whether it's GitLab. It doesn't matter where this runs because Dagger internally, it interprets this CUE and it sends it to LLB which is like BuildKit. And BuildKit as long as there was like a Docker Engine, it will automatically spin up a specific version of BuildKit what it needs. It like pulls down a container or whatever, like figures, all that stuff out. So yeah. It doesn't matter which CI you're using.

Rich: That's really fascinating. That was the part that definitely leaped out at me because I'm, I'm sure that there are a lot of folks

who, you know, at some point or another, you may be outgrow your CI system or something else comes along that you want to have a look at. And, and that's one of those things.

Really, I don't want to say it's like a vendor lock-in thing because it's, it's really not the vendors who are choosing to do that. You know, it's not something intentional. But there are people out there still using Jenkins, Right. Because they've just invested so much time and energy into like building up this, this Jenkins infrastructure and, and building up, you know, um, test suites that run with it and all of that.

And, and that idea of, you know, that all being something that you could

just move to another CI system, when you see a better one is, is fascinating

Gerhard: Okay. Uh, permission to blow your mind and I'm going to use some of Solomon's.

Rich: okay. I

Gerhard: okay. If you want. I don't have to.

Okay.

So, okay. Let's do it. So, so there's like two other things and these are not obvious things. Uh, it took me a while to understand what Solomon was talk, was on about for months, but I think I get it now.

Okay. So do you like Makefiles Rich?

Rich: um, I mean, I can't say that I've used them a ton, to be honest, but, um, but sure.

Gerhard: I love Makefiles. Like most of Changelog infrastructure is managed with Makefiles, uh, including like kubectl installation, k9s installation, templating, pulling this dependencies, pulling that dependency it's like a lot of Makefiles in the Changelog repo which by the way is public. What if we could capture different targets as packages, as Dagger packages that others could reuse.

What about, for example, we have a package that knows how to template and configure and apply kubectl manifests, or Helm, whatever, without you having to worry about any dependencies on your host. So anything that you would put like in target, think about putting them in Dagger packages and then sharing them as you used to do with Docker images. Build a

Docker file.

Rich: the targets in the Docker

Gerhard: Exactly. GitHub Actions is doing a bit of that with the marketplace. Right? You have like all the GitHub Actions, but it's specific, to that CI system. CircleCI has orbs. What if there were these packages that everyone could help contribute and they would belong to the community? You know, do you know how to deploy Erlang and Elixir apps really, really well. So how bout we encode that in a package, we put it in the universe and it's actually called the Dagger Universe, think Docker hub, and anyone can use it. So you don't have to figure out what are the, I'm going to use best practices, but what is, what is the simplest way to deploy for example, a Elixir app, blue-green all the stuff.

The compilation of assets off like different types, the caching, all those things will be encoded in a package. It's out there, anyone and anyone can use. again, think GitHub Actions, Marketplace, uh, or circle CA Orbs that's everyone contributes, everyone uses.

Rich: that sounds fantastic. I'm a very, very much a, a fan of that pattern of, you know, these community, you know, kind of marketplaces or whatever where people can share code. Um,

absolutely.

Gerhard: the Docker images. That was, that was it. I mean, now it's just like, is there a Docker image for it? Let me just pull it. Like, I don't want to like, you know, figure out how to build this thing or how to start it or how to configure it. Um, just give me like the one line there and then off you go.

Rich: Is Dagger gonna be open source?

Gerhard: Uh, yes. Yes. So, uh, so what happens is like we keep saying, oh, we're going to launch like this week. Ah, there's like this other thing. Okay. So, and what about like this documentation page? Yes. We have to write this documentation page. So there's like the closer we get to launch date, we realize that, okay.

Maybe, maybe two more days, maybe like another week. And this has been going on for, I think maybe two, three weeks now. So it's, it should have been live, um, public, you know, open source, all that now. It's still in private beta, but it won't be for longer. It won't be for long.

Rich: Fantastic.

Gerhard: up to everyone.

Rich: Yeah. I mean, I have to say this, maybe contradicts some things I said in the last episode with Celeste, but honestly, that documentation stuff is overrated.

Gerhard: Okay,

Rich: can read the code, um, just push it out there.

Gerhard: well, you say that, but that's what the brief, the private beta had told us anything is that people need good. You like, how, how, like, if you don't know, that Dagger can, you can run it locally. And what is the simplest way of running it locally? There's no way you're going to look at the code and figure this stuff out.

You know, you need to take, and, and my perspective is taking people on journeys. So someone that has never seen Dagger before, what is the journey that you want to take that person on? So do that now. That is, that is the first documentation that we will be capturing. But as you write it, you realize actually this is way too complicated, a specific, you know, things that you're trying to do with Dagger.

So how do we simplify it? Can we do like a plan B like a workaround and we have a work around it's good enough. You know, it works. It's not pretty, but it works. Can we ship this? Okay. We can. So let's move on. And then you go through that process and you realize, yeah, this looks good. We can get it out there.

And that's what's happening right now. what's happening behind the scenes.

Rich: Well, I'm, I'm very excited to see it. Uh, I meant to sign up actually for the beta, but I haven't had a lot of time to look at it currently, but, um, it sounds super exciting. One of the things I want to ask you about before we go, um, I actually didn't end up getting the, um, the call-out for listener questions, so we don't have any of those.

Um, but I wanted to ask you about one of your controversial opinions that we've discussed. Um, you mentioned that, um, you don't like the GitHub sort of model of development where, you know, people have got their own feature branches and they open up a pull request and, and that gets merged. Um, us about why, why it is that you don't

Gerhard: Okay. So I see pull requests as a form of inventory. It's things that you have going on that you haven't finished. And it's like, this is like limbo state. It's not out there. You cannot learn from it other than, from a limited set of people. And you don't know how it integrates with everything else.

How many times do you have to rebase and do a lot of like busy work like that? And the longer branch goes and it will go. I mean, some branches you know, can go for weeks. It's it's okay. But what if we didn't have branches? What if we didn't have pull requests? What if every single commit that you did when straight out into production?

Um,

Rich: So this is very much the view of the Continuous Delivery book. Uh, this is something that Jez Humble talks about a lot. Um, um, it's, it's super interesting to me. And, um, I was, I was. I dunno, kind of surprised when you brought it up. Not, not that, um, I don't agree with that point of view, but, but it is just, so it's kind of become a contrarian point of view at this point, because everybody is so much ingrained in this GitHub workflow that, that so many of us

Gerhard: Yeah. Um, I, I understand that, understand the value of having an issue. And I understand the value of having a PR and then having all those comments and having all that context there. Um, I, you know, generate a lot of that type of content, comments and, you know, rebases and force pushes and squashes so that it all looks nice, and commit messages, and all that stuff.

But at some point I'm wondering, could there be an easier way, could there be a better way? And there are days when I think, this is just so much work. And I feel like it's busy work and it makes me want to do something else. What if I would just like push a commit and and it will just go out there. So maybe we need to centralize discussions and share context in a way that doesn't prevent us from getting code into production straight away, seeing how it works.

You have rewrites, you have like, oh, I hate rewrites because you don't know how well it's going to work in practice. And the longer the rewrite goes on for the higher the risk, it will not be what you want. It will not be what users want and you won't know until you get it out there and I forgot about this.

Or I forgot about that. And then you realize, oh man, it's like, I wish I knew this like two weeks ago or whatever. And when you have like, again, the larger the team, there's like so many things happening in parallel. And then by the point, by the time we integrate, it's like, ah, there's like so many problems.

And then this was like done in different ways and, you know, okay. And then you need like a refactor code. And um, so sometimes I wish we just had one branch, you know, every commit goes out there and there was different way of structuring the conversations, which didn't involve other branches. I think that's it because PRs is okay.

PRs don't need to be linked to branches in my opinion, I think that's,

Rich: Yeah,

Gerhard: issue comes from. PR means branch I don't think.

Rich: yeah. Yeah. I think that, um, what you would really need to go down that route is lots of safety around your deployment process.

Right. You know, and this is something that James Governor talks about a lot and other people as well, James uses the term Progressive Delivery. You know, that idea that, you know, you're maybe only you know, pushing stuff out to, uh, a small amount of canary hosts or something at first, you know, um, while you validate things. Netflix does a lot of this. They have a lot of, um, automated testing in place to make sure that the things that go out to canaries are okay to push farther out than that. Um, there's a lot of work people have been doing, uh, around those processes.

And, um, I'm sure there are lots of folks out there who have cried at some point when they were trying to rebase something and it wasn't working.

Gerhard: I know. I know. And then dependencies change and ah, and then, and then like you have like lint issue. Oh my goodness me, like, why couldn't I see this before? And other things like that. So you're right. You're right. There's like a bunch of things around integrating when you, when you go into everything goes into main and then that's where like the problem starts. And something else happens between you merging and someone else merging.

And you're like, ah, I didn't see this one coming. And if you like take smaller steps and constantly validate, am I going in the right direction? That's, that's all it is. Does this work? That's why shipping it often like every, like I wouldn't say every day, every hour, just get it out there and see if it works because it still is learnings, focus on the learnings.

Forget about everything else. Because you don't know. And start with that. Start that you don't know, even if you think you do, you don't, but try it. And based on what you see, based on the observation, and that's something else and not guessing. Like you did this and then someone did that. Okay. So maybe let's sh and then you made the change, but someone else said this, so you're just constantly iterating on, on that.

That's that's the way things should be.

Rich: All right. Well, uh, I think that we're going to wrap it up there. I want to thank you very much for coming on Gerhard. Um, especially in light of the fact that you have, the flu

Gerhard: Yeah.

Rich: So you, um, you

Gerhard: I have, I have a great voice. I have a great voice right now. So that's the positive. There's always a positive. So there's a trade-off. Yes, yes. I'm not feeling great, but I have a great voice and it's not the microphone. It's not like that, the mic preamp is just, you know, the flu.

Rich: If you, if you all listen to, if you all listen to our Ship It! episode, You'll

you'll

hear what he's talking about. He's about an octave lower right normal, um, Gerhard. Thanks so much for coming on. Um, are looking to find you on the internet, um, how would they do that?

Gerhard: um, @gerhardlazu Twitter. Um, that's my Twitter. I like it. You know, I'm, I'm, I'm not extremely active. I mean, nowhere near as active as you Rich but, um, I'm there it's it's uh,

Rich: you have like five hours a

Gerhard: No,

Rich: Twitter.

Gerhard: I don't actually, but yet, um, uh, but, um, so, so that's, that's one and the other one like changelog.com/shipit. Um, that's like where I share, like every week, you know, various conversations, uh, with, with other people. We need to get you like on a more regular schedule, Rich that's, that's what I'm thinking.

And we need to get more sponsors for your show. And that's the other thing which I'm thinking. And I was almost almost finished completing this Patreon, but I wish more people would sponsor the show because I think is great. Like, you know, it's a small donation, um, just like to help you with the show. Um, and I think if, if more people did that maybe you'd be able to produce more frequently.

I would like, I would like to

Rich: it's it's literally one of the Patreon goals to hit an amount where I can pay someone to edit the podcast, which is by far the biggest, um, the biggest chore of it right now. Um, it takes me probably like about six hours to edit an episode and, um, and fill out the transcript and all of that.

So, um, it's, uh, it's pretty time-consuming and if I could just have these wonderful conversations and.

And only do that part of it. Um, Yeah.

I definitely would be able to do it more often. So I really appreciate that. That's super kind of, you I'll definitely have a link to the Patreon in the show notes. Um, and, uh, yeah, if someone listening has a company that wants to throw some money

at it, Kubernetes podcast,

Gerhard: Loft.sh

Rich: sure my I'm sure my employer would if I asked them, but, uh, but yeah, um, feel free to hit me up if anyone wants to talk about sponsorship, but, um, thank you so much Gerhard.

That was very kind to say. And, um, it's been fantastic to talk with you yet again, um, wish you all the best with the, uh, with ship it and,

Gerhard: Thank you. Thank you. Rich it has been a pleasure to be here with you. Thank you.