Kube Cuddle

Adolfo García Veytia
===

Rich: Hi, this is Rich. Before we get to the conversation with Adolfo, I just want to give you all a quick update on some things. First, as you probably noticed, the show has been on a hiatus. I do Kube Cuddle in my spare time, and the last six months have been pretty weird. I was dealing with some burnout and I've also had a lot of other things going on. But I'm back now and excited to share some new episodes with you.

One note, this interview with Adolfo was recorded back before the hiatus. The one big thing that's changed since then is his employer. He was working at Chainguard when we recorded, but now he's at a company called Stacklok. Stacklok also works in the supply chain security space, and the co-founder is someone you might have heard of, Craig McLuckie, one of the creators of Kubernetes.

Also this episode is sponsored by me. I am looking for a new role in the Developer Relations space or potentially something adjacent. If you'd like to chat about that, my LinkedIn is in the show notes, or you can email rich at kubecuddle dot fm.

If you see this episode drop in your podcast app, thanks so much for sticking around. If you'd like to support the podcast, the biggest thing you can do is share it with some friends. Word of mouth is huge for podcasts, and that would be very appreciated. Okay. That's it for the admin talk. Let's get to my conversation with Adolfo.

Rich: Welcome to Kube Cuddle, a podcast about Kubernetes and the people who build and use it. I'm your host, Rich Burroughs. Today, I'm speaking with Adolfo Garcia Veytia, who you might also know as Puerco. Adolfo is a Staff Software Engineer at ChainGuard. Welcome.

Adolfo: Hey, Rich. I'm super happy to be here. I'm excited. I've always been a fan of the podcast, so I'm really glad that you invited me.

Rich: Well, I'm a fan of you. So, uh, I think that it's a, it's a mutual thing. You're somebody that I've known about in the while, for a while in the community. And, I think that you are someone who contributes a lot to, to Kubernetes and cloud native. So I'm super excited to talk to you about that. Before we get started, though, I wanted to ask you, so I'm sure that a lot of people who have interacted with you on Twitter or other places probably do know you as Puerco, and Puerco is Spanish for pig.

I'm wondering what the story is there.

Adolfo: It's, uh, well, it's, it's a really simple story, really. So when I, I've been married for 20 years now and I met my wife in junior high. And then we, I mean, probably this is true for other couples and we started like calling each other like cute names and like, Oh, fatty and love. And then at some point we, it evolved into like a Puerco and Marrana, which is pig and piglet, I guess, and it stuck.

Um, I will, I always been like, uh, fat or chubby or something in between. So it just stuck. And then when the internet happened and I started collecting, you know, signing up to services, it turns out that no one wanted Puerco in the beginning. So it was available everywhere. So, um, yeah, that, that's why I got it.

I, I signed up, like, I remember reading, Oh, this. Twitter thing launched and I was like, Oh, I'll sign up. And Puerco was free. And then I knew that was my, it was going to be my destiny.

Rich: All right. So usually where I start off with people is how they got into computing. What did that look like for you?

Adolfo: Oh, well, I My first experience with computers was my dad used to, my, my dad is a lawyer, but he used to, I think he has like a, some, a little bit of software engineer inside of him. So he used to like really try to computerize his, uh, his office really early on. And I started with the first Apple II computers.

And yeah, I got like, I, I started programming and since the interface was BASIC, uh, it was like straight into BASIC. I started, uh, trying to understand how to, how it worked. And then I discovered that you could program, program, the computer to do things. And I started programming like at seven, eight years old, uh, using that.

And then we jumped from those to the early, I think I never owned a 8088 or 86 or whatever that was, but I did have a 286 and I used it a lot. And yeah, from there,

Rich: Did you go to college for computer science?

Adolfo: So I, I started, uh, serving, uh, electronic engineering, but I dropped out. Uh, I started my company when I was in high school with my wife. And by the time I reached college, it was like, I mean, not super big, but it was already a business that, left us some money and I just couldn't keep up with business and, uh, college.

So I dropped out. And I did return to college when I was like 30, but it was not for computers, I actually, took up on a degree of history, so.

Rich: Oh, wow.

Adolfo: Yeah, I'm more formally trained as a historian than in computers.

Rich: Well, I think that that'll come in handy maybe. Like maybe someday you could do the Sigstore history or something. Um, there's, uh, so your business was, was doing computer stuff though? You were, you were programming for people or?

Done8
---

Adolfo: Yeah, no. So we started one of the early, early, early web hosting companies in the 90s. So, uh, it was, it was a great experience because I started programming all of our platform, you know, web and email, everything was house, built by, by us. I started programming that in Perl and yeah,

Rich: I laugh the laugh of somebody who has done some Perl back in the day too.

Adolfo: Yeah, I love it. I, to me it's, I, I don't know. It has a special place in my heart. Eventually we dropped all of the Perl systems and rewrote everything in PHP. Basically because, uh, we could find people who knew PHP, but no one really knew Perl, so,

Rich: Yeah. It was very much the, the tool of the sysadmin back then, but like not people doing like more front end stuff or things like that. They wouldn't be familiar with it. That's really funny. I come from a somewhat similar background. Like my first job was an internet provider in like '95, '96 and, and uh, yeah, dealt a lot with web hosting.

We, we did our own and we also handled stuff for customers. So yeah, funny. So what is it that brought you into like Kubernetes or cloud native?

Done1
---

Adolfo: Oh, well, it's, uh, basically like the next chapter in that story because we were, uh, so we were, uh, we had our own data centers, like small data centers, like we have, uh, here in Mexico City. It grew to about 20 racks. Uh, and then we have another, yeah, and we have another one. We actually opened a, an office in, in Miami, just to, um, you know, because our internet access was cheaper in the U S and we had grown to a point that it was cheaper for us to start moving everything to a physical location in Miami. But then, Katrina, the hurricane, hit Miami and the place where we had everything flooded and we, and so that was the end of our US adventure.

Rich: Oh no, that's terrible.

Adolfo: But, uh, and then we started playing with, uh, you know, virtualization and should we jump into that?

And, and then, uh, when containers and Docker started being a thing, I, I guess, just as a lot of other people, we said like, Oh yeah, containers are cool, but how do you manage them? And we actually wrote our own, we wrote our own Kubernetes in PHP. Like our container orchestration.

Rich: I've heard this story. I think.

Adolfo: And it's,

Rich: It's, funny enough. I can't remember who it was, but I think that I've talked to like, at least one other person who wrote their own Kubernetes. I think it was Justin Garrison. I think his was maybe in Bash. Um, but yeah, that's funny.

Adolfo: Exactly. Yeah. Yeah. I mean, it's still serving people today. Uh,

Rich: Oh, wow.

Adolfo: It's fully automated. The only thing we need to do is like reveal the, the, the base web server image for customers. And then we push newer ones and it's still coordinating and orchestrating all of the controllers.

So it's, it was, I don't know, it was a nice adventure. But then, uh, there was Kubernetes. And we actually like looked at Kubernetes early on. And it was like, Oh, this is way too complex. All of the networking, it's something we, complexity we don't need right now. And then we went our own way, but I started looking at the people who were forming that community and everybody seemed so nice and so friendly and always inviting. And I said, like, I need to get into that. I, I had been contributing to like minor contributions to open source before, uh, helping out in PHP. And before, uh, since we started like on bare metal, we, I, I used to like hang out in some of the kernel, Linux kernel mailing lists because of the SCSI drivers and all of that, that we, uh, were using, but never like full on as with Kubernetes.

And then, I, yeah, that's where I jumped to start collaborating with Kubernetes.

Rich: So that's another very welcoming community. The Linux kernel mailing list.

Adolfo: Um, uh,

Rich: That was sarcasm for anyone who, who might not know.

Adolfo: Yeah, I, yeah, I, yeah, I could see those horror stories unfolding with, because of, I asked for help with a driver and I didn't know where to ask for, and then the person who was in charge of that driver asked, like, like poked some people in the list. I mean, you can imagine.

And then he got like, you know, that famous, uh, I don't know. And I, actually that, that got me scared a little bit of contributing to open source. And then when I saw what was happening in Kubernetes, I said like, Oh no, it can, it can actually be a nice thing. I mean, I knew some, some nice people in PHP too, but nothing like, like here in Kubernetes cloud native.

Rich: That's, it's really interesting that you bring that up though about about the Linux contributions because I've definitely heard that same sort of story over and over and over, and it's always, it's been interesting to me looking back that like, I think that if something like Linux started today and they treated people, the way that they did back in the 90s and the 2000s, I think that it wouldn't take off, you know. I think that at this point, people who are in the open source community, they kind of expect more. They expect to be treated well, they expect to be respected and, I think that like, part of the reason why Linux was able to get away with that for so long was because it was kind of the big thing that was around, right?

And I think that, that a lot of people's thinking about things like diversity and, you know, inclusion and how to treat people have really evolved a lot since then. Yeah.

Done2
---

Adolfo: It's also, another factor is that interactions are completely different . And, uh, perhaps you remember that saying of, on the internet, no one knows that you're a dog. Uh, but I mean, today that's not, that's not true anymore. Today we have things like this, like we're interacting in real time.

We have video calls. And so it's more like being in person and it's, uh, it's easier to fix things as you go. And then, I think in the early, in the early days of open source, you had to, you know, be much more on point with your comments and with your interactions because communication was much less efficient.

So people's patience could, you know, I mean, today people, I've seen, uh, a number of people who don't have patience, especially with new contributors. But back then, uh, you needed to invest a lot more time in, in just, just interactions alone. So, people expected you to follow a certain protocol and then, and today it's more like, you know, walking into a room, uh, more similar to walking into a room with people just interacting.

Rich: So, I know you, through the SIG Release team in Kubernetes, and I imagine that that's how a lot of other people know you as well.

So, um, for those of you who might not be familiar, a SIG is a special interest group. So that's how Kubernetes is organized. around these different topics, and the SIG Release team deals with release engineering So so what brought you to that team?

Adolfo: Um, so before joining SIG Release, I have been trying to contribute to, uh, Kubernetes. My first approach was sending some PRs to what was then the K8s Infra Working Group. Uh, now it's a SIG, uh, but back then I was trying to, I mean, I, I tried to, because it was what I was familiar with, like, Oh, I know some infrastructure, I'll just fix some of the scripts and things like that.

But it wasn't really until I started hanging around SIG Release that things really kicked off. I, I actually shadowed in the release team early on. Well, not, not that early. I mean, um, but, um, I joined in, in, in my first release when I shadowed my, I think the lead was Lachie or Taylor Dolezal, one of them.

And then, uh, I started working my first, my first, uh, role on the release team was Release Note Shadow and then I joined and then I saw that tooling could need some help. And I started like sending PRs and it was much more easier because from the release team I have, uh, well, it was, it had two benefits.

The first one was I was forced to be in contact with, uh, some of the people in charge of those tools, and then second, it gave me a way of overcoming that imposter syndrome that is, sometimes it's hard to, to deal with when you're a new contributor. I guess other people have it too, but, uh, for me, it's always pretty bad.

I, whenever I, with new, let alone a job or something like this. With new people that I don't know, I'm always had like, Oh, I'm, I'm don't feel comfortable. Uh, but then I started interacting with, um, the people from SIG Release and it just, you know, it changed my life. Especially I was mentored really closely by Sascha Grunert, he's, I owe a lot to him.

He shaped me into a Kubernetes contributor. And then I, that's how I got to know Carlos Panato, who is now my partner in crime.

And then Steven Augustus um, taught me a lot on how to navigate the Kubernetes waters, and Tim Pepper, who is my spiritual guide of Kubernetes and open source and, and yeah, and now together with Jeremy Rickard and Veronica, who are my SIG Release family and I, I really, I mean, I love them and, I enjoy hanging around every, every day with them.

Done3
---

Rich: Yeah. Wow. That's a fantastic list of people that you just rattled off, a number of them are people I would consider pretty good friends too. And, I think that, um, to me, I, I think that one of the reasons why I sort of connected with that team specifically, you know, I mean, there's a lot of different SIGs, but like, that's the one that, that has always kind of stuck out to me and I think it's partly because I personally was in a lot of roles where, um, I wasn't a release engineer, but I worked very closely with them, you know?

So I was in roles where I was like deploying code, you know, and, and doing things like that. And so, the release engineers were, they were very important to me, right? Like they were the last step in the pipeline before I got the packages or whatever it was. And, and so, um, you know, I got to see a lot of their work and I know that it's something that I mean, I think it's, it's one of those things, like a lot of kind of operations, infrastructure work where it's, it's not something that calls a lot of attention to itself.

It's one of those things where, you know, people don't tend to notice it until something breaks. And so, I think of it as kind of thankless work, you know, and I mean, hopefully it's not that way for you all. Um, because I think we do have a really good community who appreciates people. But like, um, yeah, I specifically have like a lot of love for SIG Release.

So that was, I think, the way I got introduced to you was through that community. Um, so. One of the things that you've been working on with Carlos and other people is getting Sigstore integrated into the Kubernetes release process. And so I wanted to talk about that. Um, could you kind of give us a, an overview of Sigstore? Like what it is and um, why you want to use it?

Adolfo: Yeah, Sigstore is, well, it's a project now, I think now officially lives in the OpenSSF, the Open Source Security Foundation. And it's a project whose main aim is to sign software. So to solve a lot of the pain points that you have with key management, making verification easy. Um, and basically Sigstore pairs a bunch of tools with a transparency log, it has its own certificate authority and, there's a lot of cool innovation going on in the project and, uh, so open source software can really benefit, uh, from, signing using Sigstore because you don't have to deal with key management in, inside of your organization and other benefits.

And then we, we got to know Sigstore really early on when it was, um, starting really, I think, I don't know exactly what happened in the very, very beginning, but uh, I think Sigstore approached us. Um, and. one day, uh, Steven, Steven Augustus said like, um, I think all release managers, uh, all release managers should join this meeting that's about to happen.

And I jumped into it. I remember it was my birthday.

Um, and So I was like, Oh, let me see what's up with this. And I jumped and in that meeting were Steven, uh, a bunch of the people, uh, who started Sigstore and still work on it, then among them, my now boss, Dan Lorenc, and yeah, and they like did a presentation on what Sigstore was and how it could help us.

And we had an open issue to sign, uh, some of the Kubernetes artifacts for a long time. And, but no one really wanted to take on the complexity of handling keys for PGP and verification and key rotation, all of that. And then we was, when we saw that, like, Oh, this is perfect. Let's start doing some tests.

Carlos, who is really the engine of all of open source, that thankless work of open source is powered by Carlos. He started doing some, you know, maintenance work on the project really early on, and then we started, you know, collaborating and then building some proof of concept and collaborating with that community.

And some of us also became contributors to Sigstore and yeah, like it took us like a year, I think, until we finally have working signatures, but, uh, yeah, right now, uh, all of out binaries, our container images, and the security documents are signed with it and it's, uh, I mean, it's great.

Rich: So this, I think a lot of the focus on this idea of software supply chain security kind of came up in the, in the wake of the, the SolarWinds exploit, right? Like, like that really, I think, raised the temperature in the industry a lot on this idea that you could have bad actors who would get into your systems and basically send out their own version of, of your software that's got Trojans in it or, or whatever.

And, um, I think that, when you think about compromising software, you know, that, that sort of CI/CD part of the release process is, you know, if you can, if you can get ahold of those systems, you can do lots of things, and a lot of people started paying attention to this.

And I actually saw, you know, you mentioned Dan Lorenc. I saw him speak at KubeCon. I think that it was maybe LA. I think was.

Adolfo: Yeah we were sitting next, next to each other in that one.

Rich: Yeah. So he gave a, he gave a talk about Sigstore and that was pretty early on. I think that was the conference where they announced the company ChainGuard that they founded.

And, uh, I remember seeing the demo that they did of Sigstore. He had a co-speaker. I can't remember who it was. It was somebody. Oh, okay. Um, Yeah, yeah. Um, and I was so impressed with, with it, with Sigstore and, like you said, it's, it's not like, it's not like it invented the idea of signing software, right?

Like this is a concept that's been around for a long time and people did it in different ways. And, and yeah, you could wrangle up your own, you could write a bunch of code and do it with, you know, GPG or whatever, but, rolling all of that yourself is a lot of work, you know, and I had worked at places where, um, I, I worked at a place where, we had secrets for our Puppet code that were inside a repository and we would sign those and encrypt them with, with GPG. And all of those things that you mentioned, you know, like adding keys, rotating keys, all of these things are so difficult to do when you're rolling it all yourself.

And, and the thing that impressed me the most about SIGstore when I saw it at first was just how easy they made everything, . And not just signing the software, but being able to use the transparency log and verify a signature and, and all of that stuff.

Done4
---

Adolfo: Yeah, it was great. And actually during, yeah, I remember in, in, in that same KubeCon, the very first supply chain security con took place and, and Dan was the MC of, of that conference. And I remember that when he opened the conference, he said like, Oh, everybody's talking about, you know, doom and gloom about the state of supply chain, but there are good things happening.

And now let me show you some, and, uh, I think it was his first slide or the second one was a tweet of mine, like explaining the Kubernetes SBOM and I was like, Oh, this is fantastic. Yeah.

Rich: So SBOMs are a whole nother topic of, and, um, I guess maybe we could get into that. So explain what an SBOM is for folks who might not know about that.

Adolfo: Um, uh, an SBOM is at its core, a very simple concept. It's a file that lets you know the components that make up your software. So all of your dependencies, the language dependencies in container images, all of the, you know, OS dependencies, uh, and they can grow to group information about lots of other things, but at its core, that, that is it.

And those are important because you want to know, uh, especially when it's commercial closed soft software, what's in it?

Rich: Yeah. So, and that's not something that you would do with Sigstore, right? There's other tools out there that you would use to generate the SBOMs?

Adolfo: Yeah. I mean, we, so we have our own tool that we wrote for Kubernetes, uh, and it's, uh, it's called BOM and it's, it's, uh, available from one of the Kubernetes repositories. And you can use Sigstore to sign those SBOMs and attach them to container images.

Rich: And you and Carlos did a talk about that in KubeCon Amsterdam that I saw. Um, I'll go ahead and link to that in the show notes. It was pretty interesting to see like all the stuff you all had to build yourselves to, to make that all work. But, um, but sort of boil it down in terms of this stuff, there's, there's a few different issues, right?

The first issue is kind of like the identity of the person who's putting out the software, right? Like, how do you, how do you establish that this person is who they say they are? That this is really the company that makes the software, right? And then there's that issue of, you know, what's in it, which is the SBOM part.

And then you've got the signatures to, again, validate that, you know, uh, what, what's in it, . That this is the thing that's been signed by the people who are making the software. And then you've got that ability to verify those signatures.

Adolfo: Yeah. And there's, there's another very important component, which is, provenance. And that's where SLSA comes in. Uh, because it's one thing that I'm giving you a piece of software. Uh, you can verify that I am giving it to you via the signature and I can give you a document that says what's in it, but nothing really tells you if that SBOM is really t elling the truth and nothing really tells you what went into composing when, for example, when you compile software or compose an image or whatever, nothing really tells you in a truly verifiable way how that software came to be. And that's where provenance comes into play because it's, it's a document that captures the build environment, what went in, what were the transformations done to produce it.

And then you can verify, oh, this person pulled a dependency I don't want. This was built by someone I don't trust or was built in a different system. And that's, that's why the provenance document is also very important.

Rich: So, so if I'm somebody who's making open source software or even closed source, but especially open source, um, there kind of are two big concerns, right? There are like, like I wanna do this stuff for the software I'm making and distribute it to people. I wanna sign it and I want people to know what's in it.

But I also wanna consume that for the dependencies of things that I'm pulling in, right?

Adolfo: Yeah, exactly. That's, that's why it's a chain really. And as a consumer or builder of software, you're just, you could see it like yourself as the last link of the chain or one of the links of the chain. But the way, the way I always think about it is, we try to, in computer security, we try to like build these fortresses, uh, and we're like, it's like a bastion of well protected organizations.

And the supply chain is like, okay, so we're inside, we're protected. But are you checking the trucks that come to supply you? And then if you don't verify what's in those, then people can ship anything into you. And it's not just about owning your CI/CD systems. It's also about verifying the ingredients and making sure you're what you think you're pulling in.

Rich: Yeah. There's a lot that goes into that too. Like, Dan is somebody who, uh, he posts a lot on Twitter and, and LinkedIn about CVEs, you know, about security problems and, um, how hard it is to deal with that stuff. You know, like, like a lot of times the security scanners will give different results, depending on which one you look at and, and just even how you correct those things.

Like, you know, if you're pulling in a library that does like one thing, maybe you could just write that yourself, right? And not have to, not have to use that dependency at all.

Adolfo: Yeah. Some organizations actually do that. Like I've heard from stories from, I don't know if customers or just on the uh, working groups that I usually participate in, there are companies that have a no external dependency policies. Like everything has to be written in house.

Rich: Yeah, I mean that seems pretty extreme, but I mean, I think I could see it happening like , maybe dependencies you can't verify. Maybe the ones that aren't getting signed and aren't getting SBOMs and all of that.

Adolfo: Exactly.

Done5
---

Rich: Yeah. Interesting. So you mentioned chains and that leads us to Chainguard, um, your employer, which is, uh, we mentioned that, back in, in, I'm pretty sure it was at KubeCon Los Angeles where they announced the company.

Um, so it is, um, sort of the, the commercialization of some of this stuff, right? Like some of the Sigstore founders are there. Um, uh, but there's a lot of technology beyond that, um, beyond just SIGstore. So you worked there for, what is it like a couple of years now?

Adolfo: Yeah. I'm about to hit my two year mark.

Rich: That's fantastic. So, I've been really interested in it since I saw that first presentation about SIGstore, you know, because to me, it just, it just made so much sense. Um, again, because I'd been through trying to wrangle this sort of stuff in the past, you know, manually and, and the idea that like. You know, you could make it so easy for people to be able to sign things and validate signatures.

To me, that was kind of a no brainer, right, in terms of a company. And now, more other things have evolved. So, um, I don't want to turn this into necessarily a big ad for Chainguard, but there, you know, there are other pieces of this stuff that are open source too, right?

And so, so I want to talk about some of these other things. Um, so one of the big things that's come up that's super cool is Wolfi, which, uh, is open source and Wolfi as I understand it is basically, an operating system for building software.

Adolfo: Right. Exactly. We call it the, the Linux un-distribution., uh, because it's like a Linux distribution and it has everything except for Linux. So it doesn't ship with a kernel because it's, it's a distribution, uh, designed to be the base for container images. It's, uh, APK based, which is the same, uh, package manager that Alpine uses.

Uh, and yeah, we. package, basically thousands of, of the software packages. And, um, it's designed to be a critical piece of the supply chain of the secure supply chain. And all of the packages ship with SBOMs. Our tooling produces, uh, when you build container images, with our tooling, it gives you like full coverage of SBOM, it signs them, uh, they, uh, all of our images have provenance information. So it's, uh, we're trying to, get um, uh, artifacts that really demonstrate all of those technologies at work, uh, so that other people can, can use them and build a safe, uh, safe processes around them.

Rich: So I would use this again to build my own software if I was at a company that was making open source or just an open source project that maybe wasn't even a company. I could use Wolfi to like build our images that we distribute.

Adolfo: Yeah, for sure. So we have, there are two ways of doing it. One, you can use the same tool chain that we use for our own images. We have even software that will package your applications, uh, in an APK so that it can be installed inside of the, of a container, of a Wolfi container image.

But you can also, uh, benefit a lot if you use our own images as base images for your application. So, Chainguard distributes, uh, and this come, uh, from starting with a free tier to, you know, enterprise contracts. We, uh, have container images already built, which are almost drop in replacements from what you will find in Docker Hub.

And there are, uh, magnitude, magnitudes of order smaller, uh, and also come with a guarantee of almost zero CVEs. It's a, I don't know, it's something that we're really proud of, because part of, I mean, reaching, uh, that low level of CVEs in a software artifact, it's, most, most of what you, what you don't see there is hard work.

Engineers are constantly reviewing, triaging patches, uh, uh, triaging all of the automation we have to build them.

And while we have all of those signatures and SBOMs and provenance information, the images themselves capture hours and hours of work from our engineers who are constantly patching them. So it's, uh, you could say that our images are best practices as a service. Uh, so that's, that's, that's a really cool thing about them.

And I mean, the fact that you can use them, uh, you know, free, it's like, I don't know, pretty, pretty exciting.

Rich: I worked in a shop where, you know, for like 10 years where we had, uh, PCI audits every year. You know? And, and it's something where the security team or whoever is working with the auditor, there's a scan that gets done. Right.

And there's a list of like all the CVEs and then they kick that back to the engineering team and they say, Hey, here's a bunch of CVEs. We've got to get these fixed if we're going to pass this audit. Right. And then the engineers look at that and they're like, okay, well half of these don't apply to us, right?

There's CVEs that are there, but they don't apply to our use case, and so, we can write those off, but then we've got to patch all these others. Um, and, and I mean, when you think about an image that you're getting off a Docker Hub or some, something like that, I mean, again, it's not only the software that, that they built that they're distributing, but it's all of their dependencies too. So everything that's in that, in that image, and that's why you end up with these crazy high counts of like 150 or 300 CVEs. And, and as somebody consuming that, the process to, to go through and, and either mitigate those or, or show they don't apply to you in some way is just, it's just not realistic to do that for packages with 300 CVEs, you know?

Adolfo: Yeah, so, yeah, part, part of the problem of why, uh, triaging those vulnerabilities is so expensive is because of the noise. Uh, and when you have, so the way we're doing it is first minimizing those images, patching fast, patching early, and then even then you will still have some vulnerabilities that may not apply to you.

So that's, I'm working on another technology called VEX and that's kind of, uh, designed to quiet that last bit of noise remaining when your libraries have vulnerabilities, but you're sure that don't apply to you, you'll, you use VEX to quiet down. And then the dream is that one day we'll have a security scan that will tell you exactly what you need to focus on.

Rich: That is a dream that I'm sure a lot of people would love to see come to reality

Adolfo: Now we're getting there, we're getting there.

Rich: Yeah, cause this stuff is really, you know, it really impacts people a lot. I mean, not only, not only is it really time consuming to deal with this stuff as, as, you know, and, and I mean, as an organization that is trying to build and ship software, the time that you spend trying to like mitigate these CVEs and stuff is time that your engineers are not working on your actual product,

right?

So I wanted to talk a little bit about release engineering, because I think that it's a world that maybe not everybody understands, right? Unless they've been involved with it directly. Like a lot of engineers who are writing applications or services, you know, if they're more of the product engineers, they hand that stuff off and then they're done. Right.

Adolfo: Yep.

Rich: And they don't necessarily understand what it takes to, to get the software shipped from that point on. So I wondered if you could talk a little bit about what are some of the challenges that, that you think that folks in those kinds of release engineering roles face?

Adolfo: Yeah. Well, our mission is, I guess. to make that a reality, that, that exactly what you just mentioned. Like, Oh, I just push it and forget about it. So there are different, I mean, sizes and complexity in regarding projects, uh, but essentially release engineering deals with the same thing for all of them.

How do I create, systems, processes, software that make releasing software and like automating that process, easy, intuitive and efficient. And sometimes you can think like, Oh, I just build a binary and push it and that's it. And that may be true for simple projects, but when you start considering how big software projects can be, uh, it turns out that there's a lot of work that needs to go into building those systems.

But the rewarding part is that there's also a lot of innovation that happens in, in there.

Done6
---

Adolfo: So it's kind of thankless, as you said, uh, but it's. at least in my case, I'm there for the possibility of innovating and not really for, for the applause, right?

Rich: Yeah..

I mean, it's obviously a really critical thing for Kubernetes specifically, uh, as well as, you know, other organizations and projects, but, it's an incredibly complicated project. You've got a lot of different people contributing in a lot of different areas, you know, it's made up of many software projects itself, right? There's a lot of different things in there and, um, being able to pull all that together and push it out as like one release is, I'm sure it's, it's a ton of work.

Um, yeah.

Adolfo: Yeah the release engineering part handles the, you know, basically the automation. SIG Release also handles a release team, which handles like, like the human aspects. Interfacing with the other SIGs and making sure all the human parts are, are there. And release engineering takes care of Kubernetes itself, but we also publish utilities and tools for other projects and other projects, not just in Kubernetes, but, uh, we can, can leverage for the release processes. Uh, and it's like a really complex process. So when, when you think all the things that happen in a, when we kick off a Kubernetes release. We run all of the tests for Kubernetes, but also perform tests of ourselves, uh, from our side into the artifacts that we're building, ensuring that they, the integrity that the, all of the platforms are correct. We, uh, ship them to the final serving locations, to the buckets where people download them, push them to the registries, uh, tag all, all of the images. All of that happens in a process that takes about an hour and a half to run, uh, when we release Kubernetes, uh, and we keep adding more.

Rich: I, that's down. That's faster than it used to be. Right?

Adolfo: Yeah, yeah, yeah. We've, especially since we, yeah.

Rich: It used to be multiple hours.

Adolfo: I mean, it still takes several hours if you account everything that needs to happen. And especially since we have several phases and then we have to stop for those phases and interact with other people. So it, it takes several hours, but if you see the amount of time that takes the automation to run, it's like an hour and a half or maybe less.

Rich: Wow. That's really cool. What do you, what do you find rewarding about working on that stuff?

Adolfo: On Kubernetes specifically, or in general?

Rich: The release engineering part, like say really specifically.

Adolfo: Um, yeah, I mean, to me it's, yeah, the innovation. The fact that I get to try, especially when I pair it with, um, the supply chain security side, I get to build all of the systems that, uh, put in place those technologies who are, which are new and then we can, we can really give them a boost because I mean, it's the second largest open source project on the planet, right? So, so when we said, okay, Kubernetes will build an SBOM, it really gives SBOMs a boost. Same for SLSA, same for signing. And you see it when we put some of those new systems in place, the ecosystem starts to react around it.

You see people writing about it. You see people, uh, generating demos and tools. And so that's, that's what I see rewarding. So when, when you study the, in my case, the supply chain side of it and said, okay, we should maybe try this into Kubernetes and you kind of help bootstrap that ecosystem. So it's, that's, that's my, my biggest reward.

Rich: No, I definitely saw that when I was at your talk in Amsterdam, um, there were people there from other open source projects that were just starting to get into doing this stuff and, and they were like looking at the tools that you all were using. So, yeah, that's super cool. So just to kind of wrap up, um, I just wondered if like to kind of step back a little bit, if you could talk a little bit about what it's meant to you to be part of this Kubernetes community.

Adolfo: I think being part of Kubernetes has given me, you know, three things, basically. One, the most important one is the friends I've made along the way.

Some of my best friends are folks from the community. Um, And I, I honestly, honestly couldn't imagine my life today without them.

Uh, the second one is all of the opportunities to develop professionally.

And I, I'm not necessarily talking about, exposure and all of that, but the fact that you, so I, I live in Mexico, right? And I had my own business, but usually you are, your business will be a gap on how much you can evolve in your skills, in your knowledge, because the business will limit you to what the business needs. And when you're in, in the open source project, you can find areas that, all of Kubernetes needs help, right? So if you're like, for example, you said, I don't really care about storage, but maybe you care about networking and you can jump with, with the SIG Network folks, learn, learn a lot, fix some really important and possibly complex problems.

I mean, you will most surely find something that challenges you. So that, that lets you grow up a lot. Uh, and then it comes with a benefit that you know a lot of people and then everybody helps each other. So that's, that's the other one. And finally, to me, the most, probably, the most rewarding part is mentoring new contributors.

Like I really, one of the things that I enjoy the most is like, when new people show up in our meetings and you, like, welcome them and try to talk to them, find their areas of interest and help them grow in the community. That's, that's the best for me.

Rich: That's, that's awesome. Yeah. I think that you mentioned earlier that, that you were originally a shadow in the SIG Release team and that is, I don't know if that's something that every SIG does, but I know that SIG Release does it. And that there's a process where people could apply to be a shadow for like the next upcoming release.

And, um, I just want to encourage people if that's something that sounds interesting to you to, to maybe look at that. Because everybody that I talked to who've been through that, who've been one of those shadows has really gotten a lot out of it. Um, so,

Adolfo: Yeah, it's, uh, yeah, and also don't get discouraged. Like, right now we have. around 30 spots, for contributors every cycle. It's, it's going to, so we're merging some of the teams. So it's going to be a little bit less, uh, places for people to, to call, to contribute. But, uh, and we get like, I don't know, 200 applications or so every cycle.

Done7
---

Adolfo: So it's, it's hard to get picked up, to get picked for the release team. But SIG Release is still open for folks. But I really encourage people to sign up to the release team because it's like the least friction way of going into the community. Uh, the release team will get you, uh, will get you involved with all of the SIGs because you need to talk to a lot of people and you get to know who's working on what and what the SIGs does, do.

And so it's the best way of getting involved with it. So if you get, if you apply and don't get chosen, try again and then again and again. Oh

Rich: That makes a ton of sense. And yeah, there are all those other SIGs that are out there too, that need help as well. And, you know, there's been a lot of talk about maintainer burnout over the last few years. And I think that it's really good that there's like more focus on that.

And I just saw something the other day, I can't remember which SIG it was from, but they basically were saying, Hey, you know, we, we're handing this off now to the next generation of people. And, and, um, Oh, I think it was Bobby who posted that. Um, yeah. And, and I thought that was super cool. I was really happy when I saw that because, um, that's, to me, that's the system working as it should, right? Like when there are new people coming in who have energy, because eventually people are going to get burned out or they're going to, have family commitments or, you know, other things are going to come up and, and you can't expect people to hang around forever, you know?

And so, so that process of, um, you know, pulling in new people who have enthusiasm and, maybe a new way of looking at things too, that's really good. That's, that's how it should work.

Adolfo: Yeah, you get burned out. It's like, I think that's a topic that should be discussed a little bit more, uh, as a maintainer. Especially when you've been there for a long time. I mean, when the community keeps growing, it's easier, but when it stays, when it stays like constant, knowledge and responsibilities tend to concentrate on few, not few people and a few number of individuals. And then it's you, because we also have like day jobs, right? And then when both start to demand from you, you get like all of this pressure and stress and, uh, you still do it. And we support each other, but, uh, it can be, it can be stressful.

And, sometimes people can't handle it and quit the project. Kubernetes needs to unlock some of the knowledge that lives inside of a few individuals. So, yeah, and what, but what we need is that contributors show up. And show up constantly because it's not, you, you won't be able to, you know, be, uh, one of those replacements if you only show up one or two times. It needs like constant involvement and working and learning, and then you'll make other, some of the more experienced contributors lives easier.

Rich: Yeah. Well Adolfo, uh, I think that that is, uh, plenty for us to talk about. As I mentioned before we got started, my voice is about to give out. Otherwise I think I could keep talking to you longer, but, you know, you're someone who, um, I very much think of as a friend and I'm, I'm very glad that I met you through this community.

So, um, Um, and, and I really have a lot of appreciation for the work that, that you and the other folks do to, to help make the Kubernetes project, um, you know, more secure, make these images more dependable.

Adolfo: Yeah. Well, thank you. And, uh, I mean, it's, it, it's, it's always nice to hear appreciation for the job that we, that especially the release managers put into, and the release team put into, into all of this. So, uh, thank you. And I also, um, I mean, I also consider you a friend, Rich, and I'm always super happy when we meet at KubeCon and other events, like love it.

Rich: Yeah. I'll, I'll throw some links in the show notes. So folks know how they can get ahold of you. And yeah, uh, thanks a lot.

Kube Cuddle is created and hosted by me, Rich Burroughs. If you enjoyed the podcast, please consider telling a friend. It helps a lot. Big thanks to Emily Griffin who designed the logo. You can find her at daybrighten.com. And thanks to Monplaisir for our music. You can find more of his work at loyaltyfreakmusic.com. Thanks a lot for listening.