Redesigning the future of feature flags, with Ivar and Egil Østhus, co-founders of Unleash

Transcript

Beyang Liu: All right everyone. Welcome back to another edition of the Sourcegraph Podcast. Today I’m here with Egil and Ivar. They’re two brothers who have created a popular open source feature flagging tool called Unleash. It has around 4,000 stars on GitHub. They’ve also started a company on top of this, which sells Unleash enterprise for companies. And they’re here today to talk about the backstory behind the open source projects, the journey of the company, and all the various considerations and details that go into feature flags for modern applications. So without further ado, Egil and Ivar, thanks for joining us today.

Egil Østhus: Thank you. Great to be here.

Ivar Østhus: Yeah. Really happy to be here, Beyang. Thanks for having us.

Beyang Liu: Awesome. So to kick things off, before we get into Unleash and all the great work that you've done there, can you tell me how you both got into this business of computers and code in the first place?

Egil Østhus: Yeah. Maybe I can start. It's the regular story. I picked up or we picked up this Commodore 64 back ages ago. I think it was 1988 or something, I was eight years old. And I was playing the games, but I found actually coding Basic was more interesting. So it I'm a bit ashamed to say so, but Basic was actually my entry into software development.

Ivar Østhus: So I'm the little brother, so I was never allowed to use this Commodore 64, so I'm still pissed about that. But for me, I actually started a bit later, to be honest. So we had the computers around and Egil introduced that to the family. But for me, it really picked up when I started taking part in these data parties, these LAN parties that we did that I actually enjoyed hosting. I found it more interesting to host the party than to actually play the games, to set up the network and everything. So that's how I got into it, I guess.

Beyang Liu: That's awesome. I think Basic and gaming are probably the two most popular things that I've heard from guests on the show of how they got into computers.

Egil Østhus: I can imagine. And actually we got another very old computer later and I think we picked up or I picked up, I think you were not allowed to use that one as well Ivar. Fortran, which is another odd programming language, but it sort of... one programming language after the other, and eventually I figured out setting up the environment's much better for me than doing development, so I was much better doing other stuff. So I guess that's why we are still doing this together.

Beyang Liu: Yeah, that's awesome. So walk us through the story from those beginnings to the beginnings of Unleash, what's the progression of your journey as programmers leading up to that point?

Egil Østhus: Yeah. I think at some points, I figure out I was working at the company called Tandberg, acquired by Cisco, and I was fortunate to work with really skilled developers. I had this developer, actually his core duty was to take the C programming language after it was compiled to whatever assembly code or a machine code and optimize it even further. And at that point I just, yeah, understood that I'm never going to be at that level.

Beyang Liu: So he would hand optimize the outputted assembly?

Egil Østhus: Yeah.

Beyang Liu: Wow.

Egil Østhus: Yeah. And of course this was at the time where I assumed this wasn't wrong. This was the first attempt to really make video conferencing run on an internet processor. That was a really, really tough challenge back in the day. But my core focus was realizing I'm never going to be this good. I'm pretty good at all this stuff. I'm pretty good at making other people shine and get good at what they're good at, and I moved into more management stuff and luckily we have guys like Ivar still around that actually know how to do some proper coding.

Beyang Liu: Cool. Ivar, what about you? What's been your journey?

Ivar Østhus: Yeah, so I started out from the university. I started out as a consultant, as you often do for some reason. I don't know, that's at least very common in Norway. And I worked in the bank sector for a couple of years, banking finance. And I found it a bit depressing to be honest, because it was, maybe we were allowed to release every month. It was just terrible working like this. And I started looking for other opportunities and I was lucky enough to start in this company called FINN. And we will talk a bit more about that later, I guess. And there I've been on a very interesting journey and people were much more lean and agile and wanted to really make impact on the code I wrote and making sure that I actually sold a user need and getting it out there as soon as possible to the users.

So this was really an opportunity for me to learn a lot, to be honest and in FINN, I have taken many, many different positions. I started out as a consultant actually, but I soon realized that I just wanted to work there full time. So I switched and started out as a developer, and I worked up the ranks. I started to become a tech lead. And then I started as an architect and at some point I also got the chief enterprise architect leading architecture with FINN.

Beyang Liu: Wow. And what is FINN, for folks who are unfamiliar with it, because I understand it's quite big in Norway, right?

Ivar Østhus: Yeah.

Beyang Liu: Amazon in Norway.

Ivar Østhus: Yeah, not quite. So it's a marketplace, it's a classified marketplace. So I started out by allowing people to put ads online, selling new stuff, but it had progressed into a lot of markets like jobs and real estate and all those kind of things. So it's the number one page in Norway, actually owned by Shipster there, which owns a lot of classifieds and in Europe and also other places in the world.

Beyang Liu: Cool. So I guess it would be more similar to Craigslist here in the US.

Ivar Østhus: Yeah, exactly.

Beyang Liu: Got it.

Ivar Østhus: But it's a prettier version of Craigslist, you must say.

Beyang Liu: Got it. Yeah. So, I imagined it must be this quite complex code base with a lot of scale.

Ivar Østhus: Absolutely. And I was lucky enough to join at the time where it went from this monthly release to start to think about how can we improve on this? How can we move towards more releases whenever we want? At some point, I felt like we have implemented all the things that are talked about in the DevOps handbook. It felt like we had done all the things and we could release the production at any time. We have automated all the deployment and all the environments. And this was also at the time... So I was tech lead at the time, and this was actually the beginning of something like Unleash because I was working as a tech lead in one of the teams there.

We had these weekly sprints, but still we didn't push code into production every week. Or we didn't release features to end customers every week and this bugged me a lot. Why didn't we do that? So we had all these ideas and every time, "Yeah, we can do that in a week. Of course we can." But suddenly, you realize it's not possible because it's not fully tested. It's not all the way through all the details. It's not there. And you don't want to put out poor quality to the users. Of course, but still, they burden me because then we postponed the learning. We postponed the experience of actually putting code into a production environment, because there are all these small things that you don't think about.

It could be an edge case, it could be just a file that has not been opened. It could be anything actually. So I started exploring. There has to be a solution to this. People have to have encountered this before and people had, of course, and there were already people talking about feature toggles. You had Martin Fowler of course, and the famous blog article from him. And there were also open source frameworks at the time, but they were typically a framework within your programming language. It was like for Java only, and you had to use this database. Yeah. And this bugged me because we were already doing microservices. This was in 2014. We already had the polyglot programming model where we had some parts in Node.js, some parts in Python, some parts in Java.

And I needed something, or at least I wanted something that could work across all of this and still be scalable. And it shouldn't be a single point of failure that could take down everything. So I talked a bit about this problem, and I thought that there has to be ways to solve this. And this was actually before LaunchDarkly was a thing. So it was in 2004, at least it wasn't publicly known yet. And I thought that this is something that I can solve. It seems like something I need, so that I could change the behavior of the code in production, dynamically, in a very performant way so that they wouldn't hurt the performance of our applications. We were serving like 10K requests a second at peak. So I need to scale for that from the beginning.

And I thought that it needed to be this reverse dependency so that I could put something in the application that would do revelation, but I wanted the rules to be centralized so that I had the configurations centralized. It was pushed down to the applications and application could themselves decide whether a feature should be enabled or disabled for a particular user.

Beyang Liu: Got it. So you wanted a single dashboard where you could toggle these things on and off and change the behavior of the application without having to push code or make changes.

Ivar Østhus: Exactly.

Beyang Liu: Got it. And I find, you touched upon this a little bit earlier about the importance of feature flagging to engineering velocity. I think it becomes a lot more important, especially as you scale, because there's more and more quality considerations that you want to make sure that you meet before you release something to general availability. But then that very same QA enforcement mechanism becomes a blocker. You can kill a lot of good ideas before they're able to gain traction or prove themselves.

Ivar Østhus: Exactly, and I see this all the time and you have one extra dimension today and that's the privacy part because you cannot usually just have all the data you have in production available in a testing environment. You can of course have synthetic data that are similar, but it's not the same.

Beyang Liu: Yeah, totally. So what was the V1 of what you built, this feature flagging tool for internal use at FINN?

Ivar Østhus: So V1 one was a very simplistic version, I would say. You could turn something on and off. You could enable it for a specific user, but that's about it and then it's just adding functionality from there, I guess.

Beyang Liu: Got it. And so it evolved over time. At what point did you open source it?

Ivar Østhus: It was open source quite early. So I developed this or together with some others, we developed it in late autumn, 2014. And it was officially open source January 2015, but the repo was created in 2014, but we didn't push the code out before for January 2015.

Beyang Liu: Got it. And what was the process around open sourcing? Did you have to get organizational buy-in, agreement from stakeholders inside the company to open source it? Or was it relatively straightforward?

Ivar Østhus: I would say it was quite straightforward and there was actually an internal push in the company also that we wanted, or that FINN wanted, to show more of what they were doing to the world and share more with the world because FINN was also using a lot of open source. So why shouldn't FINN also contribute stuff back? So that was the philosophy back then and still is, I believe.

Beyang Liu: Yeah. And so when you open sourced it, what was the initial reaction? Did other people start using it immediately? Did it take a while for it to catch on?

Ivar Østhus: Yeah, I think it's like with every good thing, it takes a bit time for people to discover it. And I haven't, at least for many years, I didn't do any official commercializing or marketing or anything like that. I just open sourced it. And the reason for me to actually open source it from the beginning was to make it super easy for other teams at FINN to use it, because then I wasn't tied into some kind of legacy shared library or anything like that. It could be taken into any code base and anyone could start using it. And so how did it grow? This is how I observed it. It started growing within FINN. It was never a push to actually use it. It was more like people were using it and they used it also when they're doing the show offs, when they're showing new functionality. They used Unleash on the stage to show, "Well, no, we enable it for 10% of the users," stuff like that.

Beyang Liu: Oh, that's awesome.

Ivar Østhus: And then it started growing organically within FINN, but what I also noticed was that when developers quit FINN, one of the first thing they did in the new company was to introduce Unleash.

Beyang Liu: That's awesome. So you basically didn't do any selling or pitching or trying to get people to use it. You'd released it, made it open source, and then over time, it spread organically through word of mouth and people leaving FINN and taking it with them.

Ivar Østhus: Yeah. And of course I have been invited to conferences to talk about it. I have done that, but that was later years. So in the beginning, I didn't do anything like that. Of course. In hindsight, I should have, but I didn't.

Beyang Liu: What was the first point at which you said like, "Wow, this thing is actually being used by a lot of people, there's a lot of people relying on this?"

Ivar Østhus: Yeah. I started seeing a number of stars in GitHub started growing quite rapidly in late 2018. This is interesting, and also people started sending me emails personally, like, "Could you just host this for us? We can pay you for it?" It was like that. And that's a sign, I guess. So this was when I started talking to Egil about this, because I had actually never mentioned this to Egil. It was a side project I founded, like a fun program to work on by night and evenings, but it wasn't like... I had never intention of making more of it and that at the time, at least, and then like brothers do, we sometimes go out, have some beer, some food. And I just started talking about this project and I said like, "I don't know what to do about this. What do you think?"

Egil Østhus: Yeah, I remember. It's actually not true because I think you mentioned this to me back in 2016. At the time, I didn't really grasp what it was doing, to be honest and to be fair, but it stuck with me for some time that this was possible. And I remember that evening very well, and the interesting part, we discussed this, of course in more depth on what it's doing, what it's capable of handling. Ivar was coming from this environment, he was super on the edge with DevOps, with microservices. At the time I was, I would say working at a company, a software company. It's very successful, but it's more like an M&A company called Visma back in Europe.

And my duty, I was often put into an organization that they were doing legacy development, legacy engineering. The code base was not necessarily the most, I would say, on the edge. And interesting enough, I saw that this was actually solving a lot of my problems because as you say, and as you mentioned, this Q and A process with or without the data, just the fact that you're trying to replicate a very advanced Q and A test or pre-production environment, it's like super costly. It takes you a lot of resources, a lot of time, a lot of manpower to try to get it. And usually you don't really get it. And you still, at least those companies, some of those I was part of, was still having issues in production at the time, still launching a lot of new updates with tons of challenges.

And the time we spent just fixing up in production issues was ridiculous. And having this idea of being able to control everything in production at customer sites, which was also the case in this company, was really intriguing.

Beyang Liu: Yeah.

Egil Østhus: Well, it's really intriguing. And, as Ivar says, the fact that people are asking to get this product and pay for it is this obvious sign we just need to get something done here and start seeing what we can make out of this.

Beyang Liu: Yeah. That's awesome. So, at that point, it clicked for you given your experience with your existing company. Was there a point when the both of you finally said, "Okay, let's do it. Let's actually start a company around this."

Egil Østhus: If I may, I think it was an involving process for the both of us. Obviously, we were both having interesting jobs, well, paid opportunities. And frankly, it was a giant leap to leave that safety and move into something else.

So what we decided to do was, as a developer, as an engineer, do this iteratively, stating "let's do the simplest code to market and product wrapping we can do on this and just start learning." Obviously, I started to do some market analysis looking at who are like competitors, where is this possible positioning, and it was a very steep learning curve, to be honest. It was very interesting.

But we decided at the time to keep it as a side gig. I think we made a promise that we were never going to put any feature into the product because of a customer deal. I think we actually broke that promise, but we really wanted to just say, "This is the product." And if there is anyone willing to pay for it.

And actually that was the case.

I think we launched a version of our webpage in April or May? Maybe it was 1st of May? And I think we had the first signed up customer some time in June.

Beyang Liu: Wow. That was quick.

Egil Østhus: That was pretty quick. It was. It was amazing. And most of the first customers are still with us. So, I'm really, really happy with that.

Beyang Liu: Yeah. That's awesome. Is the paid version of the product, is it essentially Unleash, the open source thing running hosted, like someone else is hosted? Is there any difference or distinction that you make between the two versions?

Ivar Østhus: Yeah, there are a couple of things.

What we do is we do the open core model. We actually extend Unleash open source with more functionality. So, we are actually using the open source and adding more in the enterprise. Most of it is around user management, role-based access control, grouping stuff into projects, and having access control on those projects and stuff like that. So, it's more convenience for enterprises to manage access to who can do what.

But we also add some functionality or more advanced targeting rules to the pro edition or the commercial edition, making it even more convenient for the companies to control their rollout without having to write any code to do that, so that they can define different rule segments that they want to target with the new feature.

Beyang Liu: Yeah. That makes sense. This is a common pattern. Sourcegraph follows the same open core model where there's a core that's open source that provides, I think, most of the functionality, especially if you're an individual developer. But then there's these additional features that are enterprise specific that you build into the paid-for product.

Ivar Østhus: And, yeah. And it's super hard. And maybe you have experienced the same thing. But it's super hard where does the line goes.

And we try to be kind of as transparent as we can. And so far, and I think we will try to stick to this as long as we can, we have only moved functionality from the paid version into the open source. We have so far never done the other way. And I hope we never get there.

Our plan is to keep the best functionality, the newest, best stuff, the most shiny stuff, in the commercial version. And if we see that this is not a differentiator, or this could be really useful for a lot of people, we will actually move it to the open source.

An example of this is that we added more or easier control for your API, so that you can add API tokens and so forth, also in the open source. It makes sense. It doesn't make sense to require you to write some custom code to do that.

Beyang Liu: Yeah.

Egil Østhus: Yeah. And I think this open core model that you described, that you guys do as well, it makes total sense. I mean, it's really important that you have something for the individual developer that really is providing a lot of value because I don't think you will get away with just throwing out some open source kind of pro software out there.

That doesn't really make sense. It needs to provide some proper value.

And of course, as Ivar says, this is the interesting and challenging view of it. Where do you put the line? And I think that is also where we are iterating and learning every day.

Beyang Liu: Yeah. 100%. It's a tough line to draw. A lot of times it feels like you're trading off building something awesome that every developer can use and growing awareness as much as you can versus capturing some of that value to keep the lights on and keep growing as a business.

Egil Østhus: I think maybe if I may share this, Ivar, I think that this decision to open source the native apps, as the case, was probably one of the toughest ones we had, I would say. To me, it felt like, I know we have closed some business because of this and taking care of the commercial aspects of everything.

Ivar was really clear, and he's absolutely right, this is something that makes to sense to the individual developers. So, we need to be true to what we're saying.

But I need to admit, I met myself. I'm not sure if that is an English expression. I met myself in the door, saying like, "Okay, I need to listen to this, but I don't like it."

But we did. I think back in May, we open sourced those for native as the case with Unleash, and it seems to be picking up quite well.

Beyang Liu: That's awesome. Way to go Ivar for making the case for that and Egil for understanding where that was coming from. I think we have a lot of the same discussions internally at Sourcegraph, too, between-

Egil Østhus: Yeah. I can imagine. It's very tough. And I can imagine as you grow as a business, as well, there's more focus on the top line and revenue in this conversation.

For us, it has actually been a very cautious decision to make this commercial versus the open source from the day one, because it is difficult. It has to be difficult. And I truly believe that this is going to be why we are successful, because we are building this culture for really understanding where is the line. That needs to be in the doors, in the walls, and in the wood of the office spaces that we really know this. Everybody just knows that this is the decision. Everybody adapts to that, and we move forward.

So, we are training that every day. And it's hard, but it's necessary.

Beyang Liu: Yeah, totally. Can we dive a bit into all the stuff that falls under this umbrella of feature flagging? Because I imagine folks listening, they probably span the gamut from like, "Oh, feature flag, that's just a Boolean variable that you can toggle on and off that turns some minor part of your application on or off."

But there is actually a huge range and depth to the things that you can do with this tool. Can you talk about some of the more interesting, involved use cases?

Ivar Østhus: Yeah, absolutely. There is a lot of things and, of course, Booleans, on and off for everyone, is the easiest one and the most obvious one.

If you take that one step further, a very common use case is that you want to enable it for a percentage of your users. And one of the things that you have to consider then is how to make sure that it's a consistent behavior for an individual user and still you are not showing or treating the 1% the same across all your features. That could also be bad or maybe you want that. That's the configuration that you need to do. And maybe you want to feature targets to actually target the same users.

But this starts to add complexity when you're starting thinking about it.

And maybe you want to throw in that suddenly you think that you don't want to do 1% for everyone. Maybe you just want to do 1% for your gold customers or 1% of the customers in Norway, for that matter.

And when you start adding all these small things, all these attributes that you need, you start building up complex rules that could be hard to get right. And we also need to make sure that we do this right across all our implementations.

So, we have, as is the case for all the popular languages, at least official SDKs, there is also a huge list of more community-supported SDKs, and we also make sure that we do this correct, or the same way, across all these implementations. So, we do a hashing algorithm to make sure that all the SDKs are using the same hashing algorithm, making sure that, when we say 1% of the users, we mean the same 1% of the users in the Go client SDK or in the Java SDK.

And remember that the rules are evaluated in the application and not in the server API. So, we need to actually make sure that SDK does it correctly. So, to do this, we also have created a client specification framework so that we can use that to drive our integration test for all the SDKs to make sure that they evaluate the rules the same way.

Beyang Liu: Yeah. Makes sense. And I'm sure there's a ton of interesting tech there that I want to dive into in a little bit.

But before we get into that, Egil you mentioned doing some sort of competitive analysis at the very get-go. And I think a lot of people are probably trying to triangulate where Unleash stands with respect to other feature flagging tools out there, like LaunchDarkly or Split or other tools. Can you talk a little bit about what sets Unleash apart? Or why your customers decide to use you over other options?

Egil Østhus: Sure. Absolutely. And it's a great question. And as you mentioned there, this is a category that is growing. It's definitely an area that is getting a lot of attention at the moment.

Obviously, LaunchDarkly, looking at the funding and I believe also revenue, is the biggest player on the ground. They have been around about the same, as long as we have been around as an open source project, interestingly enough. But they have been more visible and more loud out there.

And I think that's the biggest. Of course, the most obvious one, we are open source and most of the other competitors are not.

I wouldn't say that is not necessarily a differentiator. It gives us some advantages for sure. I mean, just the fact that people are able to really dive into the source code, they are able to see what is happening, they are really getting this pulse of, "is this something that happens in this tool or not?" Quite a few customers, or let's say the developers, are coming back and saying, "We have looked at your open source repo. There is a lot of activity. We like that. We also see that there are contributors to this. This is something we believe in."

And I also believe it provides quite a high level of trust because we know that every developer are able to inspect the code. So, I trust the developer teams to be able to really be top-notch. And they are.

The feedback so far is that this is really, really rock solid.

And actually, a true story, we were talking to some friends of ours out of Sweden, Codecentric, I believe is what they are called. They were doing technical depth management and code inspection. And the CTO of the company was investigating our open source code because they wanted to identify some tech depth. And he came back and said, "Well, this is not interesting. This is just too good."

So, I think this is a good, valid feedback that there is some reasonable strength in the code. And obviously also we see that from the stability where we don't really see any issues with that.

But compared to LaunchDarkly and Split, I think that the biggest advantage we have is our extremely versatile or flexible platform.

Obviously, we have a lot of the same design decisions, the API first. Everything is an API. Everything is possible in or out of our core API, which makes it really attractive and adaptable to any platform team. And I see more and more large organizations that have a developer efficiency team or internal platform team. There are many, many names to the same responsibility, making sure that there is proper tooling and a developer platform available to the product team.

So, I see that talking to these guys. Unleash is really a perfect match. We can basically adapt to anything.

And part of the adaptivity is also, it's super easy for us to deploy to anywhere. I mean, private clouds, even on-prem is something that, for some customers, makes a tons of sense, or of course go with our hosting.

And Ivar says, privacy has been very focused from the day one. And I see that the focus of privacy is increasing in particular, in Europe. There is a lot of tension around privacy all over the world at the moment. I think there was last week or the week before, there was a long article around this verdict that was really going in, maybe we can say, an extreme part of the GDPR, but it was really putting extra responsibility and awareness, I would say, about data privacy.

And we see the same out of Australia. We see the same out of the California Act. And we see the same out of, I think, quite a bit of the Latin, as well.

So, privacy concern is probably one of those where it's really hard to compete.

But of course, that is a trade off. We can, we need to trade something off for that. So, what is sure is that data will never flow from our rest case into our platform. That is really something we take really serious. And in the healthcare, in the financial sectors and in a lot of those really heavily regulated industries, this just really makes a ton difference.

Beyang Liu: Yeah. Makes sense. So, with respect to the privacy question, the advantage of Unleash here is that you can run the server side of the feature flagging tool on your own infrastructure, within your own organization. It doesn't go towards a third party.

Egil Østhus: You may. If you, you want to go all the way to the extreme situation where everything is in your control, you can take everything back out.

But even if you want to stay consuming cloud operations or cloud business, like the host version, we have this signed protocol to never share any PII back from SDK back to our server side. So, we allow you to configure the rules and you have a ton of flexibility on this advanced segmentation that Ivar was talking about.

But everything happens server side in your part of the application. And you take care of your privacy. You should not have an extra layer of complexity because you are adding a wonderful tool like Unleash in to your product.

Beyang Liu: Got it. So, even if I'm using your hosted version, there's kind of special things that you built in to obfuscate or anonymize the data that-

Egil Østhus: Correct. It's never shared.

Beyang Liu: Got it.

Egil Østhus: It's built in to the application.

Beyang Liu: Cool.

Egil Østhus: Yeah.

Beyang Liu: You touched upon developer experience as a thing that you like to emphasize as well. Can you talk about, what is the developer experience like of using something like Unleash? How would I interact with it as a developer?

Ivar Østhus: Usually, you would use the SDK, of course. And we have emphasized a lot on also minimizing or making the test case very lightweight. This is something developers really fancy.

And for instance, the Java SDK, we only add kind of one dependency and that's JSON parsing because that's hard in Java. Besides that, we are only using core libraries, even though it makes our job a bit harder because it's harder to do ACDP and all of that.

Beyang Liu: So the SDK is a thing that you embed in the client. And then the server side is something that's running elsewhere and the SDK will talk to the server to determine which flags are on and how should I affect the behavior of the application.

Ivar Østhus: Yes, it will synchronize with the server in the background and also that part is very optimized. That's different per SDK, of course, per platform, in the Java SDK we will actually only use a single thread in the background to do all the communication with the Unleash API just to minimize our resource consumption on your application. And it's all those small details that are super important to be appealing to developers. You also have the APIs of course, where you can do anything you can do in the management UI, you can do also from the API. This is also something we have a lot of plans on improving, making it even easier to do patches and stuff like that. So that you can update just what you actually want to update and not send longer payloads. So we have a lot of plans on even improving the APIs to make it even simpler.

But you also really interact with the management UI to define your feature toggle... Actually, you don't have to actually manually define it because if you start using a feature toggle in the SDK it will actually detect it and it will prompt you to actually create that feature toggle in the management UI, if you haven't already.

Beyang Liu: Oh, that's cool.

Ivar Østhus: Because we will send usage metrics back to the API so that we know which apps are connected, how many instances are they connected from, and which feature toggles do they actually consume?

Beyang Liu: Interesting.

Ivar Østhus: So we will collect all that user data, and from that, we can make it easier to understand, is this feature toggle actually being used? Where is it being used? How often is it used in the last hour? Stuff like that. Because the other part of this is the technical debt side of things. This is also a huge problem with feature toggles. So you are creating a way, the long running feature branches that are never merged back to master because it's hard. And then you have the feature toggles instead, which actually are feature branches inside your code, giving you a lot of power. But then you have all these different code paths in your application. And for some reason, it seems like people don't clean up afterwards. A lot of developers are moving on, moving their focuses to new features and it seems like it's nice to just leave those feature toggles around. This bugs me a lot and I think we, as a tool provider, should also address this issue because the amount of feature toggles is not the point in itself.

Egil Østhus: Yeah, and I think actually that is a very important point. We have seen some of the competitors, I don't need to name them, have been referring to the number of feature toggles being served in the platform. Which I understand is, from a marketing point of view, it makes tons of sense because it gives you this impression of scale. It's big, it's really massive. Sure it is, but also it gives you, if you start thinking about it, it gives you the incentive to keep that feature toggle in that because that is part of what your... A north star metric for the company, which from engineering point-of-view, it doesn't make sense at all. So, actually that management was something we decided to build into this platform from day one because as you mentioned, coming back to the developer experience, it's really... Or at least my experience working with developers and developer organizations. Technical debt or anything that doesn't really meet this sales guys' requirement for the next nice-to-have feature, is really hard to get into the roadmap or into the sprints.

And by providing some visibility to provide some kind of dashboard or reporting telling that there is actually something that you should do here with your, whatever. And in our case, that's cleaning up your feature toggles. It's really valuable because usually it's not because we want to point fingers, it's more like opening up a conversation between the product manager or the marketing guy, so whoever is saying that. We have done it, but we haven't finished. We really need to clean up because otherwise we will impact our velocity and nobody really wants that. Everybody wants the next feature to be shipped as quick as possible. And we want to make it a bit easier for all of the tech leads to take that conversation and highlight that there is something that we need to do here.

Ivar Østhus: And interesting enough when we built this functionality, we were also provided access to a lot of open source users and their data on their usage to perform a large analysis. So we actually analyze the usage across more than hundred applications using Unleash. And we saw that 50% of the feature toggles were being used for 50 days. And after that, they have served their purpose, they were never changed anymore, they were usually just staying around. And some people turned them off or deleted them or deactivated them, but most just kept them around.

Beyang Liu: And that's all tech debt. That's a dead code path that could shoot your team in the foot down the road.

Egil Østhus: Yes, there are some nasty, nasty, nasty stories out there. And we don't want to put our customers in that position obviously.

Ivar Østhus: And of course, there are some valid use cases. You have the kill switches, for instance, that could be super convenient when you need them. But then you should actually invert them. You should actually have them default to disabled and you can enable them when you need them.

Beyang Liu: Yeah, that makes a lot of sense. Can you talk a bit about the people, both the paying customers and also the open source projects that are using you currently?

Egil Østhus: Sure. What can I say? Obviously, it's a wide range. Every business today is looking for developer efficiency one way or the other. So I think that what we are seeing here is, I think we have customer in all sectors at the moment, in all of the continents, obviously. And I think that what we are seeing is more on the engineering maturity, rather than an industry specific. Obviously, some industries is always a step ahead and I'm sad to say, you guys in the US tend to be slightly ahead of Europe and Asia on engineering culture. So we are picking up speed, but you're still a head or two in front of us there. But usually the customer is either already in the DevOps space, they're already set, they are just looking for our tool to go into the pipeline. Which we see is a bit more of a tougher sell, they want to do it.

And for one reason or the other, it's difficult. And I will say that actually developer efficiency is something that really puts into being a business problem. And sadly we see that often this initiative is driven by engineering. And quite a bit of the time, that also leads to the "business part" saying, "Well, we don't want to spend either time or money on this now. It sounds like a nice thing to have, but we don't have time to do this." So I think most of your listeners can recognize this one way or the other.

Beyang Liu: Yeah, totally. What are some of the big open source projects that use Unleash?

Ivar Østhus: That's a good question. Actually, in the public sector, you have something called NAV, which is the Norwegian labor authorities. And they have open sourced a lot of their code, they're not properly open source projects, I would say, but they have open sourced their code. And they use Unleash a lot actually, and they use the Unleash open source edition. So that's the biggest one that I know about at least. And they have open sourced more than a hundred applications, I think, that use Unleash. So that's quite interesting to look into actually. Besides from that, a bit unsure. I think some of our customers are doing open core, but I haven't prepared that list for you. So I'm a bit unsure to be honest.

Beyang Liu: Yeah, no worries. That's cool. Do you have an idea of how many applications use Unleash at the moment? What is the scale of the impact that you're having?

Ivar Østhus: Yes, so we are tracking downloads of the Docker image, so we know that the Docker image or the pools, it's more than 6,000 pools each day at the moment, that's the current rate of the Docker image. And it's growing, and growing quite nicely actually. And 4.2 million pull requests in total so far. So we have some callback functionality to check the version and stuff like that, and we know that there... But that was added quite recently in April, as you probably also know, is that open source tend to be more like long term support so everyone is not updating all the time. So we see that it takes some time for customer or users of the open source to actually update. But we have seen more than 30,000 installation in total, but they are not all active. So we have a hunch that it's a couple of thousand active installation based on the numbers we see.

Egil Østhus: Yeah. And also, you are paying attention to as soon as you put something out there, a new version, the download rate is quite good.

Ivar Østhus: I'm almost afraid because people are pulling down and installing the new version before I do. It's crazy.

Beyang Liu: That's awesome.

Ivar Østhus: So it should be high quality every time we release a new edition.

Egil Østhus: It is high quality.

Ivar Østhus: It must be.

Beyang Liu: So another side of this, in addition to the user community, is given that you're open source is also the contributor community. Can you talk about the open source contributions that have been made to Unleash as a feature flagging platform and what impact they've had on the capabilities of the product?

Ivar Østhus: Yeah, we have a couple of interesting contributions. Of course, the most popular place to contribute is on the SDK side. So we have a lot of community developed SDKs and they tend to pop up every now and then for a new programming language that is out. We have a community SDK for Rust or for Alexa, stuff like that, which is just awesome that people do that. So that's really interesting, but also on the Unleash project, we also have contributions, for instance, the import/export type of API, where you can jump to your full state and export that and you can import that into a new running instance of Unleash. That was fully contributed, the first version of that was contributed by an external contributor. And it has proven to be very valuable to us as well, at least when people want to convert from the open source into hosted offering. It's super convenient to have that functionality in place.

Beyang Liu: That's awesome.

Egil Østhus: Yeah, and I also remember when we put up this team around this commercial offering. I think one of the most interesting stories was, I think it was Christopher on our team, it was like stars in his eyes, he was coming through the stand up and was saying, "It's fantastic working for an open-source company because I'm writing this code and I'm getting feedback as I write the code before I even launch the feature by the community." So he was engaging with some of the community members and they were highlighting some, I'm sure it was some improvement points they've found in his source code. The feedback cycle, I've never seen anything like it, it's absolutely amazing.

Beyang Liu: Yeah.

Ivar Østhus: But I think it's safe to say that we have a very good user community and we have a very engaging community on Slack where people ask questions, they get answered and stuff like that. But I would say that the contributor community, of course it could be better. It's like, yes, we have a lot of contributors, we get contributions every week, but it's usually more towards creating integrations for other solutions, creating SDKs. And these are very valuable, but they don't drive the product by themself.

Beyang Liu: Right.

Ivar Østhus: And of course, that's a good thing also because we are doing this full-time and relying on the community could also actually slow us down. So I think it's a balancing act there.

Egil Østhus: Yeah, and I think it makes tons of sense to open up and make it super easy for the community to contribute in a cross/value add on top of the product. But we need to take good care of the quality of the core. Obviously, always welcoming anyone who'd want to contribute or improvement suggesting of these kind of things. But by the end of the day, it's our responsibility and our name and heads on the block to make sure it os top-notch quality. It has to be. So the cross and the value adds and one of the nice thing we haven't really started to market that is, we put up this add-on platform, so it's super easy to create these pre-curated add-ons to any third party. So we see that this is popping up. I think it takes with a couple of days if we are requesting from suggesting from the community that we should have this add-on for one or the other.

Beyang Liu: What sort of add-ons? Are these add-ons for specific languages?

Ivar Østhus: It would be to integrate with other solutions. For instance, we requested a Datadog add-on. It was nice to have, but we didn't use Datadog ourselves, so we just created the issue and said that this was a good first issue. And I took a couple of weeks and then suddenly someone jumped on it and they used a few days and created an add-on to Datadog so that in Datadog, you can actually get the event plug from Unleash and see if a feature toggle changes, you will see that in your Datadog graphs. So that you can correlate that if something suddenly start to fail.

Beyang Liu: That's cool.

Ivar Østhus: Similar with Microsoft Teams, Slack, stuff like that.

Beyang Liu: Yeah. It's really tough to strike the right balance between getting community contributions, especially to that core part of the code base that maybe is a little bit more difficult to dive into just because there's a lot going on and it's not adjacent to anything that an external user would know. It's not the SDK, it's the standalone service. And that's something that we struggle with a lot too, because on the one hand, you want to encourage contributions. I think there's a lot of knowledge out there that people can contribute to that part of the application that you wouldn't think of yourself. But at the same time, not having that deep context into that code base makes it difficult to check all the quality boxes to make a contribution.

Egil Østhus: Exactly.

Beyang Liu: Yeah. And we've actually been thinking a little bit about this at Sourcegraph or there's a feature idea that popped up and maybe we should feature flag this using Unleash, to create almost a notebook view where you can write out, here is how I would onboard to this part of the code, or something like that just to make it easier. An expert could write up this tour of that part of the code and you could hand that to beginners. Anyways. Now I'm venturing off into talking about Sourcegraph.

Egil Østhus: Makes tons of sense.

Ivar Østhus: But I think you are actually touching upon it and how can you encapsulate parts where it will make sense for multiple parties to actually contribute code without them having to understand everything.

Beyang Liu: Yep. And you almost do some of that by defining maybe like strong API boundaries. Because with things like the add-on, we have our own extension ecosystem. If you have a very firm, clear API boundary, then you can divide and conquer a little bit and say, "Okay, this part, we're going to have these rules, these norms for this part of the code base, and that's separated from this other part." And so we don't have to worry as much about the code becoming a spaghetti piled mess because it's isolated and contained. Anyways.

Egil Østhus: Yeah. Makes sense, and we have seen exactly the same thing as Ivar says, we need to make it very easy and isolated and self-contained as an issue, makes it much easier to get onboard them. And, and also if you can get this repeatable, and that's why we decided to build this out on framework, it's one pattern it's one way of working and it's one part of the documentation there to onboard our community to start creating this.

Beyang Liu: Yeah. Cool. From the technical side, what are some of the more interesting technical challenges that you've had to solve to enable feature flagging at large scale and across languages?

Ivar Østhus: I think you already said it there. I think at least in the beginning, that the hardest part was of course to add support for more programming languages, because we decided to put the logic inside SDK itself, the evaluation of the rules, and I cannot be an expert in all the programming languages out there. I'm not that smart unluckily, or sadly, I don't know.

Egil Østhus: I'm not sure about that.

Ivar Østhus: But, that's a good thing. And my strong point is that I know quite well a lot of Java, I know a lot of Node.js, that they're quite different as a programming languages. They're quite different projects, and for me to think about this problem and actually implement a solution in both of those, makes it more likely that it would be possible for someone to implement this in Python or in Go or any of the other programming languages. And it has proven to be quite a good model list. And now we are not always implementing in Java or Node.js first, we're actually... if someone comes to us, they have something they want to solve, and they are using, for instance, the Go SDK, we might implement a new functionality only in the Go SDK and add it behind some flags for the Unleash part to make sure that we can test it and verify it.

And if it turns out that this is a good idea, we start moving on and implementing it into more of the SDKs. And then we announce it that this is actually a functionality that you can now take advantage of from some version of the SDKs. And this has been challenging, but luckily, I have gotten a lot of help from the community on the SDK side, and we still actually closely work with some of the maintainers of some of the SDKs because they're good guys, and they actually want us to succeed and they want to help out, and we take some responsibility in actually implementing new functionality, but they will step up, they will help us code review the pull request and make sure that we do it the right way in the various programming languages, and it has been just very helpful to have them around, to be honest.

Beyang Liu: Yeah. I think you mentioned consistent hashing across languages as a challenge earlier, too. Can you talk a little bit about that? Why has that been a challenge?

Ivar Østhus: So, I started out not thinking that we needed... or I didn't realize that there was a problem in the first version, so I just did some regular object hashing in Java, and I found some other hashing for Node.js, and it worked, but then I realized that it was a problem because the user might enter into different parts of your stack, and you might use the same feature toggle in multiple places in various programming languages. And suddenly, if the same user would get in the Node.js application, they would get false, and in the Java application they would get true, and it would be confusing or it could be confusing. And suddenly you have an unpredictable experience for that user.

Beyang Liu: Yeah.

Ivar Østhus: So then I started thinking there has to be a better way and we needed to find some way of solving this so that it was consistent, and I already had... hashing was was very natural to me to do that, to just do the randomization, but I started investigating. And so what do others do again? And it turned out that most of the A/B testing tools seems to have standardized around something called MurmurHash.

Beyang Liu: Interesting.

Ivar Østhus: And it's known for having a very good random property and to be very fast. So it's not secure, but we didn't need that. We just need some easy way to randomize a number between zero and hundred and do it the same way across different programming languages, and it is a very small hashing algorithm, so there are already implementations in all the languages. So it was easy to just use that, basically. That was... I wouldn't say it was very challenging in terms of code you have to write, it was more like figuring out how to do it correctly.

Beyang Liu: Yeah. Yeah.

Ivar Østhus: Other parts that has been challenging... I think lately I have been struggling a bit with... we are doing a lot of heavy lifting inside Unleash itself where we do a lot of changes, but it's super important to me to be backward compatible. We should not break a user's applications, at least not in the first major version, so we need to make sure that we support our APIs, we support at least our SDKs for a long time, and it's a long time since I last broke the SDK API, because I don't think we should do that ever, or at least that's my goal, because it could be very unpredictable and you cannot expect customers or users to just update their SDK all the time to make sure that they are on the latest, greatest.

Beyang Liu: Yeah.

Ivar Østhus: And know which version they need to use to be very compatible. Of course, that require us to think a lot and have a lot of unit tests and integration tests to make sure that we don't break stuff, and it also makes it harder to introduce new concepts, but I think it's important though. So, I see now that if I just have talked about all the things we're going to build, it would be so much easier just to build it from scratch.

Beyang Liu: Yeah. That's another balance that's so hard to get right, because back compat is so important, especially when people start using your thing in production, but at the same time, it can become this restrictive constraint around innovation, I guess. That's why some people just do the version one, version two version break.

Ivar Østhus: You can do that. And actually, we did that once, but we added it to the payload itself and not as a part of the URL.

Beyang Liu: I see.

Ivar Østhus: But that was for the SDK part.

Beyang Liu: Yeah.

Ivar Østhus: It worked okay. I think the smartest thing that we did and it's looked a bit stupid for a couple of years... so I decided at one point that the SDK API should be a different API than the admin API, where we actually do the mutations.

Beyang Liu: Yeah.

Ivar Østhus: And in the beginning, they were exactly the same. It was the same payload. The client SDK could only support GET, but from the admin API, you could also do POST and mutate the data. And it was... felt like it was quite stupid in the beginning, but lately it has been paying off quite well, because now we can do a lot of larger changes on how you've worked with feature toggles, but we can still map that back to have a very consistent API for the SDKs.

Beyang Liu: Got it. So you separated out the two APIs so that you could have different rules.

Ivar Østhus: Yeah. And you can scale it differently, you can add more caching, maybe, to the client SDKs and make sure that they perform in a different scale, basically.

Beyang Liu: Yeah. Cool. So, what does the road ahead hold for Unleash, both the open source project and the company?

Ivar Østhus: So of course, it's world domination. Everyone should use Unleash and Unleash runs everywhere. But, we have a lot of ideas to get there, and there is a lot of things that we focus on. One of the things that is really important to us... and we want to solve real problems, so we actually have a Head of Customer Journey as part of the core team, and we are interviewing open source users and we are interviewing customers, and we do that regularly.

And we have done that for a lot in Q2 to prepare for a lot of improvements in Q3 in how we onboard new users. We want to make... so we have this idea that the first moment of, "Wow," this is the first time you actually understand the value of a tool like Unleash, and we want to make sure that you get that moment the first time you try Unleash, and not the second time or after talking to someone else that discovered it after a few months of hacking. So, we want that to be in the first session where you actually interact with Unleash.

Egil Østhus: Yeah. And I think what you say there, Ivar interviewing and getting use of feedback is... I would say we are very obsessed with that. Obviously, we have a direction where we want to go. We have quite clear thoughts on what is... where is the feature flag space going? I mean, feature flags or feature toggles to me is like a commodity. You need to provide something else. You need to provide some value on top of that, because it's... I mean, we understand that this is a system complexity that is interesting to solve, but it's also something, with all of the players in the market today, is going to be commoditized. So, we are very much focused on this, understanding these pain points, and I think we have as many customer interviews as we have customer sales conversation.

I mean, it's really getting to understand the pains, really understanding the situation, really understanding how there is any friction point that we can support or take away or even turn around as a competitive advantage there. On the business side, we are stepping up quite a bit on a commercial side. We just brought our head of sales onto the team based out of New York, so we are very excited to have the feet on the ground already in the U.S. And also, this means that we stepping up on all of those, what we expect to see on the customer success or getting the product properly supported into the organizations.

And also, we are looking into, where can we fit into supporting our customers, making developers efficient, either through partnerships or through integrations, or make it super easy for them to make Unleash part of their work platform or their development platform? And I think also, there is two big north stars. Ivar you have this... or at least you come back again and again to this phrase, you want to make sure that the world is Unleashing new features. So, that is the north star. We really want everybody to have access to Unleashing new features to the production because that's a huge difference. And the second part I was actually coming to... I just forgot now. Sorry.

Beyang Liu: No worries.

Ivar Østhus: And also to add to this, of course, because we are open source, we also have an open source or open public roadmap. So, if you're interested in what we are working on right now, or for the next quarter, you will find that on GitHub, as a project on GitHub on this repo.

Beyang Liu: Cool.

Ivar Østhus: So, we are open on everything we do. That's not... this is what we focus on right now and what our immediate plans are. Obviously, we also do a lot of other things than just what's on the roadmap, but that's at least for us to be transparent, and we will always adapt that roadmap into new learnings and what customer tells us, and what the customer interviews and user interviews tells us.

Egil Østhus: Yeah. And the idea came back to me there. And so, part of making this... everybody should have access to Unleashing a new feature, we are also see that we need to take this space from feature configuration into purpose driven feature management. Feature configuration doesn't make sense. It's just a step on the way to really start doing proper purpose-driven feature management.

Beyang Liu: What is the difference between...

Egil Østhus: Feature configuration is where you have your tech lead, your architect, your product owner, your product manager, whoever really, they start saying, "Okay, who should have access to this feature? Where do I need to enable this for whom and what and when, and these things." So, what you really should think about is so, by adding this feature, what am I trying to achieve?

Do I try to achieve a performance improvement? Do I try to achieve a conversion rate that is better? Am I doing an A/B test, maybe? And if so, what are the metrics, the key metrics in your business that you're trying to change there? And actually, we did some super interesting project together with Kevin on the open source project. I think it's supported by Dynatrace where we started to explore this as a proof of concept.

How can we start creating self-healing application, meaning you enable a feature and you see that some of the predefined metrics or metrics that you have in your operation dashboard starts to be badly impacted. Can you correlate that feature release into this impact on your metrics? And if so, where is the bar for this segment, this feature, and go back to the team and say, "Well, we need to do something here because it doesn't really work exactly as we say." So to me, that is a huge difference, and I see this space we have, there is so much talk about configuration and I mean, nobody really likes configuration, right?

Beyang Liu: Yeah. It's got a boring, slightly negative connotation.

Egil Østhus: Exactly.

Beyang Liu: Cool. As a final parting thought, if people are listening to this and they want to get started with Unleash, where should they go?

Egil Østhus: getunleash.io. That's the starting point. There, you'll find everything from the documentation, you'll find the links to download open source, and if they want to get in touch with us, there is also opportunities to reach out to our sales team and set up an appointment, or just request access to our paid offering, hosted paid offering.

Beyang Liu: Awesome. Well, Egil and Ivar thank you so much for joining us today. This was an awesome conversation.

Egil Østhus: It was a pleasure, Beyang. I really enjoyed being here.

Ivar Østhus: Absolutely. Thanks for having us.

This transcript has been lightly edited for clarity and readability.