Changing the web one tool at a time, with Kelly Norton, principal software engineer at Mailchimp and creator of open-source code search engine Hound

Transcript

Beyang Liu:

Hey, everyone. Welcome back to another installment of the Sourcegraph podcast. Today, I'm here with Kelly Norton. Kelly Norton has had a long and fruitful career working on developer tools and technologies. While at Google, he was a core contributor to Google Web Toolkit, which powered a lot of early interactive web apps. He also contributed to Chrome and worked on core browser performance, Google Wave, and a language that would eventually become known as Dart. While at Etsy, he created an open-source code search engine called Hound, and, along the way, he was a co-founder of FullStory. He's now a principal software engineer at Mailchimp. Kelly, welcome to the show.

Kelly Norton:

Thanks for having me. It's awesome to be here.

Beyang Liu:

Awesome. So, before we get into all that awesome corpus of work, let's go back to your kind of beginnings as a programmer. How did you originally get into coding?

Kelly Norton:

Yeah, so I used to think I had an original, somewhat original, story. But as I've listened to more podcasts and met more folks in the industry, I realize it's not all that unusual in that I grew up in a family that nobody worked in technology. Most of my family did not attend college. And they started encouraging me to get into computers at some point, which, strangely, I took them up on that suggestion, even though I knew very little about it.

And, man, the way I really got into programming is that it was just me and my mom in the house, and so she had to do something with me during the summers. One of the things she did, luckily, was she started sending me off to these summer camps. Most of them at the time were about beginning programming, mostly on Apple Basic, mostly graphics. So, fortunately, she sent me off to a couple of different summer camps where I was able to do my first programming.

And then it would be years before I even had my own computer. I would not even get my own computer until I was a sophomore or junior in college. But it stuck with me. I enjoyed it. I mean, it was great. You could sit down and you could create something, and it was very visual, and it was very rewarding. And it stuck in my mind as a thing I really enjoyed to do. And of course, I have been told by family, speculatively, that computers are the big thing. And so here we are.

Beyang Liu:

That's awesome. They had a lot of foresight, I guess.

Kelly Norton:

They did.

Beyang Liu:

Cool. So it was Apple Basic. You're coding away. This was grade school or middle school or high school?

Kelly Norton:

Yeah, it would've started in grade school. And then, as we got into high school, there were some Apple computers sort of littered randomly around the high school – part of some funding grant at some point, I'm sure. And so there would be little opportunities to change existing programs and make little graphics programs to entertain friends along the way.

Beyang Liu:

That's awesome. And did that naturally segue into studying computer science in college?

Kelly Norton:

Yeah, I actually started not in software. My thinking at the time was I like to design things, where I had under-defined design. Design meant making things become real in the world. And so, to me, I had heard that that aligned well with the title of engineer. And so I thought that I would just randomly combine the suggestion of ‘get into computers’ with ‘engineer’, and I would become a computer engineer. And, I joke, but the reasoning was not more sophisticated than that at the time. But engineering seemed interesting because it did have this property of you go and you sort of dream up a thing, design it, and then make it become real in the world.

And so I wanted to do computer engineering. Georgia Tech was the school, the in-state school, that was the good engineering school in Georgia. I was from Georgia. But my grades in high school weren't all that great. And so I looked at the very real possibility that I couldn't get into Georgia Tech at the time, and realized that Georgia had a wonderful engineering program where if you went to any of the schools in the state and maintained a GPA above, I think, 3.0, Georgia Tech would effectively guarantee you transfer into the engineering program.

And it also dovetailed with … at the time, Georgia was doing a really neat program called the Hope Scholarship, where they would basically pay a good chunk of your tuition. So I got to go to a small school until the Hope Grant went into effect, and then the timing just aligned beautifully.

Beyang Liu:

That's awesome.

Kelly Norton:

And I switched into Georgia Tech right at the time the Hope Grant went live. Did computer engineering, actually got a computer engineering degree, never worked a day on hardware. I think it's sad that I had to actually get a degree in order to realize that what I really loved was writing software.

Beyang Liu:

That's awesome. So were you coding during college, or was it after you graduated you realized that hardware wasn't for you and you switched over?

Kelly Norton:

Yeah, I think I was doing some coding during college. And I actually … after I finished my undergraduate, I decided to stay on for a little while and continue to work in the graduate program and computer engineering at Georgia Tech. And my poor advisor at the time was really patient with me because this is the time where I really did start to do a lot more software. I did a lot of side gigs. I actually wrote conference software for submitting papers for academic conferences at one point.

And I also, at that point, started connecting to some other ... There was a professor at the time that had left Georgia Tech and started his own business. I started talking to him and would ultimately end up going and working for him and leaving graduate school. So this is the point at which I began to realize not only is my interest in software, but I'm never going to work in hardware because I'm probably just going to continue to do this software thing.

Beyang Liu:

What was it, do you think, that drew you to software? Compared to hardware, what appealed to you about it?

Kelly Norton:

Yeah, I think it was, one, I was starting to realize that I did love the visual elements of programming. I got my start with the satisfaction of doing graphics. I had always sort of had a fascination with graphic design and all things visual art. So I think the fact that I could sit down and actually build something visual, very low cost, and it didn't really require a lot of ... I didn't have to go anywhere. I didn't have to have a bunch of pricey equipment. I just had to have the computer, which I had gotten by this point. I had my own computer by this point. So I think part of it was just the reward structure of fast iteration. I was already starting to sort of pick up on that from software.

And some of the early software projects I did were also pretty interesting in that you really started to get a sense that software had this property where you could sit down and you could construct something, and, in a very short time after that, it could actually help a lot of people. A lot of people could then just enjoy the rewards of your software, just from the copies.

And hardware just didn't have those properties. And the other thing is hardware companies also tended to be very conventional engineering cultures, and this sort of fast-moving software industry that was emerging pre .com explosion, I think, was very appealing.

Beyang Liu:

Yeah, got it. So you got into software. You mentioned that you joined your professor's startup, and then you eventually ended up at Google with a pit stop along the way at MIT Media Lab. Tell us about that journey.

Kelly Norton:

Yeah. So after I left grad school, I actually ended up starting ... This is another fortunate part of my career where some friends I made at Georgia Tech decided to start a business and asked me to come along and start the business with them. So technically … it's funny because for the longest time I wouldn't call myself a co-founder of this small company which we created just before 2000. But I realized later that, oh, I am a co-founder of that company. I just felt like I was so young and naive at the time that I did not deserve to call myself co-founder.

And that was mostly focused on sort of building software for admissions, college admissions. So one of the things I had said to myself when I started that company and when I joined these folks to start that company is that I ultimately wanted to go back to graduate school and I wanted to actually try to explore the whole visual part of my interest.

So I wanted to try to combine software with visual. And, at the time, the sort of pinnacle of this was happening at the MIT Media Lab, under John Maeda, where he had a group called the Aesthetics in Computation Group: a lot of data visualization, a lot of art through code. So I was really fascinated with this. And so after the company that we started around 2000 was winding down as part of a transition, I followed through and started trying to apply to graduate schools.

I got very lucky in that John Maeda ... the interests in his research group were changing such that it really aligned with me having basically spent some time starting a business. He was really interested in digital art markets at the time. So I think I just happened to magically align with his interests well enough that he accepted me into his group.

That was two years of the most intense, sleepless nights that I have ever spent. It was very, very intense, and I learned an awful lot. I actually really still love the MIT community. I still love the Media Lab community.

Beyang Liu:

Got it. Digital art, it sounds ... These days with NFTs and like the Metaverse, it sounds like that vision is coming to fruition in a very real way.

Kelly Norton:

Yeah. I mean, John Maeda actually tweeted recently pointing out that we had a project called Open Studio, which at the time was really focused on transfer of ownership while maintaining providence of digital art pieces.

Beyang Liu:

Sounds familiar.

Kelly Norton:

Yeah, it was very much a research project. There was no notion of decentralized ownership that you have through the blockchain. But it did have many of the properties of basically trying to keep these chains of ownerships and recognizing copies, and inherent values in a notion of ownership that is apart from physically owning and using a thing. I think, in hindsight, we look back at that project and go, "Wow. We were much closer to some interesting ideas than I think we even appreciated at the time."

Beyang Liu:

Very cool. Now, how did you go from there to landing at Google?

Kelly Norton:

Yeah, so after I finished up at the Media Lab … I'm from Georgia, I had been in Atlanta for many years. I wanted to get back close to family, but had assumed that my next step was probably going to take me to the West Coast.

In another fortunate turn of events, Google decides to start an Atlanta engineering office, mostly from an acquisition they had just done for Google Web Toolkit. And as I was wrapping up at the Media Lab, I heard about this. I actually reached out to some friends I had at MIT who had recently taken jobs at Google in Mountain View, asked them if they could confirm the existence of an Atlanta office. They said yes, but they knew nothing about it. Ultimately, I would get connected to the folks who were starting the Google Atlanta office.

And it was a weird time at Google. It was a fast growth time. I actually interviewed in the New York office for a position in Atlanta. I would do half of my interviews over VC: two with people in Atlanta and half with folks, who, many of which, I would never see again in my time at Google, doing the interview in New York.

But yeah, I ended up taking a position at Google Atlanta, moving back down to Atlanta, and spent about six years there. Most of the time was working on developer tools. And I think this is the point at where I really start to get into developer tools.

Beyang Liu:

Got it. I didn't know that Google Web Toolkit was an acquisition. What was the acquired company? What was their business model? Do you know?

Kelly Norton:

Yeah, the business model is fairly interesting looking back at it now. So they recognized all of these enterprise companies are being forced to move a bunch of software onto the web. And most of them, there was a lot of Swing UIs floating around in enterprise business at the time, just a lot of Java expertise. And the folks who had started this company, which were Bruce Johnson, Joel Webber, and Scott Voigt – their names will come up again as we talk about FullStory. What they realized is they'd been asked early, "Can you help us to move some software over to the web?" And they thought, "Well, I can understand why this is very hard for people. There's just a total lack of tooling to build these very complex…"

Particularly when you're porting existing, very complicated enterprise applications over the web, it's really hard to move a big app over with, at the time, so little tooling. I mean, we're talking about in the era of Chrome didn't exist. Firebug existed. Most people had switched over to Firefox. IE still dominated the market.

So that was their business model. And I think that they had already started to do this right in Java and compiled a JavaScript when Google got interested. And, in fact, I think the interest originated from the Google Maps team who ended up connecting to Bruce and Joel based on a blog post. And it would ultimately end in an acquisition and the formation of the Google Atlanta engineering office.

Beyang Liu:

That's fascinating. So what was it like working on that team? Because I feel like, in many ways, it was ahead of its time. It was the first real, principled, type-safe, web framework, right?

Kelly Norton:

Yeah. It was actually a complete mixed bag. So the project was kind of controversial. So there was already kind of a culture of building on the web and what that entailed: the sort of quick edit, refresh, see your changes. It really appealed to developers. There wasn't a history of having a lot of tooling on the web. So the expectations for getting help from engineering tooling were low anyway.

So Google Web Toolkit kind of divided into two camps, I think. I think most people ... Well, let me describe the camps and then I'll tell you why one was more visible than the other.

So the camps were: there were a set of people that came to Google Web Toolkit because they needed to write something on the web. They knew Java. They knew the Java ecosystem, and they wanted to use that in order to build web apps.

And then the second camp was a set of people who wanted to build for the web. They also wanted the tooling, but they weren't necessarily wedded to Java. What they really wanted is just stronger tooling for web languages, and largely, in a lot of ways, we sort of have that now. It's taken us a long time to get there.

And I fell into the camp of I really didn't care whether it was Java or not. I just wanted all the tooling. I wanted a static language. I wanted debuggers. I wanted a really good compiler. I wanted a compiler that I could change compile time constants and see 30% of the compiled code base disappear because the compiler can prove that code was unreachable. I wanted that type of engineering leverage on my web projects.

And so because of this, I think the project always sort of fought these ... These were sort of two competing camps. I mean, in some ways they were aligned, but it really, when it came to prioritizing the project, you can see how there was an opposition here. Because one camp would have you plow ahead trying to make more of the Java ecosystem real, where the other side would focus more on core tooling, and, in fact, making Java libraries that would be more web native than emulating Java, what you would find in Java.

Like a really great example of this, and I think one that always sort of underscores the point, is Java has longs. So it is a 64-bit integer. JavaScript has no such type. Well, not really. And so, this was very interesting.

Beyang Liu:

It doesn't even have a true integer type, right?

Kelly Norton:

Exactly. Yeah. That's right. In some places in the spec, it says that if you do certain operations on a number, it is guaranteed to become a 32-bit, but it doesn't really have anything that suggests a 64-bit integer. So the first versions of GWT really just assume that people knew that and would operate within the semantics of JavaScript, and they would not even expect their longs to work. Well, it turns out that the Java camp obviously expects their longs to work. In many cases, they were compiling existing Java code where they expected it to work.

Well, the JavaScript camp is like, "Well, we don't have longs, so it's unreasonable for me even to expect something that the platform doesn't supply." And so this comes to a question of: is it advantageous for the project to emulate longs in JavaScript? We ultimately did this. And it manifests in an expected way, which is that Java camp celebrated.

It became a bit of a landmine for the native web contingent because if you accidentally compiled in long emulation in your code, suddenly your compiled output went up by a considerable factor because it had to pull in the long emulation library. So like that's a really good example of where we really traded off between these two camps.

Beyang Liu:

That's fascinating. Okay, so you work on Google Web Toolkit for a while. That ended up powering AdWords 3, and also Google Wave, right? Did you work at the application level of either those applications, or were you mostly focused on the library stack?

Kelly Norton:

Yeah, so for both of those projects, I was actually pretty involved. We tended to have an embedded model where we would ... I mean, for one thing, we had met directly with these teams as they were considering using this tooling. And we were a bit of unknown for them. I remember them asking us, "AdWords is a really important project. Are you comfortable like basically the entire next version of AdWords depending on your project?" And I remember Bruce Johnson, who was the site director at Google at the time, said in the most confident voice, "Yes, we absolutely welcome this challenge." And I know that he must have been really nervous on the inside.

We were sort of embedded with them, and we would help with pieces of their app. Usually, though, it looked like, "Hey, we're having a problem here. Can you help us figure it out?" And this was both with the AdWords 3 front end and with Google Wave.

And Google Wave, by the way, was made by the Maps creators who were instrumental in the original acquisition to Google for Google Web Toolkit. So they had always been close friends. So when they came up with the idea of Google Wave, we were somewhat fortunate to get an early peek at what they were up to, and got led into the stealth project a little early, largely because they wanted to use GWT and they wanted our help in basically extending tooling to support them.

So yeah, we worked directly with them. This is actually how I ended up getting into web performance tools because, at the time, everybody's problem was that there were mysterious performance gremlins in their apps. Usually, the problems were on Internet Explorer, but not always. So I transitioned naturally from working on GWT mostly to basically building a bunch of hacked-together tooling to find performance issues in all of the various browsers. This is also how I got really interested in hacking on browsers in general.

Beyang Liu:

Interesting. Okay, so there's two parts of that. I want to get to both. The browser performance part, obviously super interesting, but before we get to that: Google Wave. What was it like working on that project? Because, in many ways, that was like way ahead of its time. And even though the product didn't end up succeeding, a lot of the concepts and ideas from it have made it into Google Docs, and also online collaborative editing tools in general now.

Kelly Norton:

Yeah, so there are a lot of interesting elements to that project. One, sort of in the simplest terms: they were in Sydney, we were in Atlanta. So that is probably the hardest time zone adjustment I've ever had to deal with. And I also went out to the Sydney office and it turned out to be just an amazing, amazing trip to go to Sydney. But also, the Google Wave team was really an amazing team.

And I think one of the unfortunate things that came out of Wave being both a controversial project internally at Google, but also not really meeting expectations in the market, was that that team probably did not get the recognition they deserved in terms of the caliber of team, and also the unusual circumstances in which they were operating.

I mean, the team was hired very quickly in Sydney. It was like most of the people were some of the top engineers that I met at Google. They put together the app under enormous pressure, in a very short period of time, while all keeping their sanity.

I remember Adam Schuck was the tech lead on the front end of Google Wave at the time, who I realized, I have an enormous respect for him. I remember him really influencing how I thought about tech leads, both in what he prioritized. I remember him telling me, "My team is operating at its absolute best when I'm working on the grossest stuff." So I always consider it a measure of success if I am over straightening out the build while everybody else is making progress. And I just have enormous respect for him. He would go on to Twitter and found some companies. And he's now back in Sydney, which is great.

Beyang Liu:

So they're building all this very forward-thinking, interactive, live editing and communication features on top of GWT. And you were responsible for making it all fast. I imagine they ran into all sorts of as-yet un-encountered performance issues. What was it like being the under-the-hood person and trying to make all that stuff fast for the end user?

Kelly Norton:

Yeah, it was interesting. So, one, the story we would tell about using tooling at Google Web Toolkit, they really embraced it. And this time zone difference sort of plays in here where I would show up in the morning, Eastern Time, only to find that they had sent me a lot of data overnight showing where there's mysterious, "Hey, here's a link to the code. This is taking a really long time. We're not really sure where the time is going." And so I basically had all day Eastern Time to try to make sense of what was going on and get back to them. And so I had started doing this.

There were a couple of things that really plagued web apps at the time, which people largely know them now. At the time, it wasn't so clear. So one of the things is that the browser layout engine is a little different. It basically has a notion of implicit layout. So in many UI frameworks, you build up your widgetry, and at some point you go, "Okay, I'm done. Go ahead and lay out everything where it goes on the screen." Well, JavaScript has a notion of, "Okay, go run some JavaScript. And when you're done with that, I'll compute everything and I'll lay it out." Except JavaScript exposes ways to ask questions, which basically need to know where things are on the screen. So one of the common mistakes for performance is that you get the order of operations wrong such that while your JavaScript is running, you ask for a size, and then you do something and destroy all the layout information, and then you ask for the size again.

And so I realized that many, many performance problems that we were finding were of this form. So then I had this quest of, "Okay, how can I modify browsers such that we can get that sort of data out?" So when the layout happens, I want to know when CSS selector matching happens. I want to know when garbage collection happens. And so I started like assembling these. I had like a fork of Firefox that had instrumentation built in. And then we had started this project ... I call it a project. We had started this directory of hacks to basically take Internet Explorer and hook into the DLLs.

There's a strategy called preamble patching on Win32, where you can basically intercept ... You basically take the memory map DLL, and you modify it right at function boundaries, even private function boundaries. And you can basically …

Beyang Liu:

Get your tracing hooks in.

Kelly Norton:

So if we were able to ... Exactly, so we were able to put the tracing hooks in. And so we basically ended up with a collection of tooling that worked on all of the major browsers at the time. So that was basically how we solved the performance issues for both AdWords 3 and Google Wave.

Beyang Liu:

So basically, you had this tooling. What was it? Did you ship a custom version of IE then, or is the mechanism that you're describing a thing that you can run on the side and it hooks in at startup time or something like that?

Kelly Norton:

So in the beginning we just used the tooling. We would eventually build an internal. Ultimately, we would turn this into a ... We would build the tooling directly into WebKit for Chrome. And that would basically … if you remember when the timeline panel and the Chrome developer tools first appeared, that was actually built on the instrumentation that we put inside of WebKit. And then, for IE, we ended up shipping, I think they're called Browser Helper Objects, BHOs. I may get the terminology wrong because it's been so long, but basically there was a way to create a DLL that integrated with ... I mean, it's basically all the hacks that we saw for Internet Explorer were generally these Browser Helper Objects.

So we basically shipped a Browser Helper Object that instrumented the current version of the DLL and extracted all the trace data and pumped it back to a UI that, ironically enough, was running as a Chrome extension inside of Chrome at the time. So it was quite a contraption, but it was super helpful because suddenly we could just give teams like AdWords and Google Wave these tools and they could just self-serve in terms of finding the performance issues.

Beyang Liu:

Got it. Got it. And this was Speed Tracer.

Kelly Norton:

This would become Speed Tracer. Okay. So it was known, this tool was known internally, as Speed Tracer and, and when we would ship it publicly, it would be just a Chrome extension showing Chrome performance. It did not have the Internet Explorer element. We briefly approached legal with the idea of shipping the Internet Explorer version, which actually depended on symbol server information for Microsoft, which showed what the offsets for all the functions are. This is typically what you would get to have stack traces for your data applications.

Beyang Liu:

Okay, got it.

Kelly Norton:

And they were not keen on the idea. So sadly, the idea of shipping an Internet Explorer tool was not favored by the legal team.

Beyang Liu:

Got it. Got it. You said something a while back that I want to drill into. So you're two geographically separate teams: one Atlanta, one in Sydney. And you said you'd wake up in the morning and they would send you links into the code that was slow. Were they sending you links to Google Code Search? Is that how the interaction played out, or was it they would just tell you where it was and you'd go navigate the code in your IDE?

Kelly Norton:

Yeah, so it was a little weird because Google Wave was on their own instance of Perforce, which was strange, but they were basically ... They weren't actually Google code searches just because of that. But, yeah, they were basically just links to the equivalent of a GitHub link within Google, but for their Perforce instance.

Beyang Liu:

Got it. So they had their own source browser that was web-based and they could link?

Kelly Norton:

Yeah. This was actually, I believe, this was prior to Grok showing up and many of the later tools. I could be wrong, but I don't remember Grok being present at the time.

Beyang Liu:

Grok would be the thing that powers the code navigation in Google Code Search now – that ‘jump to def’ and ‘find references’. Steve Yegge's thing. Cool. All right, so Google Web Toolkit, and somewhere along the line, Chrome and WebKit appeared, and my understanding is that you eventually became a contributor to Chrome as well.

Kelly Norton:

Yeah, mostly WebKit. So with this performance work that I had done, getting that tracing code into WebKit was sort of my first sort of entry into making WebKit changes and contributing to WebKit. And this was prior to Chrome forking Blink. So we were all contributing to the WebKit that is mostly centered around Apple at the time. And so through just getting all of that instrumentation landed, I would end up with committer status in WebKit.

And then, as Google Atlanta started to diversify the engineering teams that were there, one of which was a WebKit team. It was like the Chrome platform team, which mostly meant the people on the Chrome team who worked on the WebKit side.

So we would end up with a team in Atlanta working on the SVG implementation and WebKit for some time. And also under the Chrome umbrella, Dart. We actually had a big presence in Dart, as you alluded to earlier. And that's kind of an interesting story. Stop me if I ramble.

Beyang Liu:

No, please. What was early Dart like?

Kelly Norton:

Yeah. So Dart is interesting because it's actually a collision of a lot of cool ideas and Google trying to force them all into one space to operate. So obviously, in Atlanta we had Google Web Toolkit. We were really interested in compile-to-JavaScript languages. We were looking to not really change the platform. I mean, we weren't opposed to changing the platform, but we had just assumed that we would not be able to make enough changes in the platform to make a substantial difference.

So we saw that the place where we apply the most leverage would be to try to build an ecosystem around tooling. So we were really interested in, all right, so we had Java, we could compile Java to JavaScript. But, as I mentioned earlier, Java wasn't a really great ... It wasn't the best thing to be compiling to JavaScript because of a lot of baggage it brought.

And so we had this idea of, "Okay, well, if we were to just make a language from scratch that was very close to JavaScript semantics, so we didn't have to worry about things like long emulation, and it just didn't try to get over its skis what it promised to JavaScript developers, what would it look like?"

So we had forked off a team to work on that. And we were calling it, at the time, JS Prime. And the whole thing was a statically typed language that had structural typing. So you had an agile way of looking at types. You had a way to take a type declaration and promise the object I'm about to give you does match this. So it's sort of, if you're familiar with TypeScript, it's sort of the same as TypeScript interfaces. And so we had begun to work on this, writing a compiler and trying to rebuild a lot of the tooling infrastructure or ecosystem that we had in Java for GWT.

Well, at the same time, in Seattle, there was a team that was trying to do roughly the same thing, but very focused on the UI layer. They wanted a way to do UI widgets and they're trying to solve a lot of the same problems that would ultimately be solved by things like React. So what is the best way to build components that are reasonable inside of the web, since elements themselves weren't very composable in the way that people wanted them?

And then you had a third team. And that third team would come out of the V8 team and they were an Aarhus, Denmark. And they were really interested in a different thing, which is they basically wanted a new virtual machine. They wanted a new language that was a little bit more structured than JavaScript so that you could benefit both from a runtime perspective, but also from a tooling perspective.

And so these teams sort of got squished together by edict. And we were left to figure out what our priorities were. And that's not an easy task. I mean, we're essentially on divergent paths in what we think is important. And we're suddenly in a corner together debating which path we should take.

The truth is the team with all the power was the team at Aarhus. And I mean, they were an amazing team. V8 is an engineering marvel. And there was no reason not to trust that team. And so we would eventually merge as Dart.

We would set aside JS Prime and bring some things we learned from it into the early versions of Dart. And we would continue to move down this path where the folks in Seattle and Atlanta were really wanting, "All right, whatever we do, we're creating a VM over here, and we're making a language to compile for the web here. Let's don't compromise each other. Like, please don't do anything that would make it hard for us to compile to JavaScript."

And one of the very specific things that came up immediately is that many of the folks in Aarhus were Smalltalk folks, or Self folks. And so they loved many of the environments that had been created in Smalltalk. And they were really into things like ‘method missing’, where you … basically proxy calls, where you could call a method that was not declared and proxy it. And that is really not good if you're compiling the JavaScript and you want to be able to statically know where the static trace can go in every call site. So we would definitely have our debates about what the proper path forward was.

And Dart would continue to kind of be both for a while. And at this point, I've largely stopped following Dart, super interesting language. But going back to JS Prime, it was really heartbreaking to have to give up that project. I was so relieved when TypeScript appeared, because TypeScript was very much what JS Prime aimed to be. Probably the only big difference is TypeScript aimed to be a super set of JavaScript, which I think was super smart. Whereas JS Prime … we couldn't accept JavaScript that was untied. You had to do something to existing JavaScript to make it compile.

Beyang Liu:

Yeah. That's fascinating. It is one of those counterfactuals where what if those three projects had not been bundled together? Would they each have seen success on their own? And how would that have changed the evolution of the web landscape?

Kelly Norton:

Yeah, totally. I agree. I would have loved to have seen each of those projects go off on their own a little bit longer just to see where they landed.

Beyang Liu:

Yeah. Cool. So, you're at Google. Eventually you left Google and co-founded a company called FullStory, and you alluded this earlier, but it was with a couple members of the GWT team?

Kelly Norton:

Yep. Yeah. So I had mentioned Joel Webber and Bruce Johnson earlier. They were the ones who had started the Google Atlanta office. And I think we had reached the natural fatigue of ... I was at about six years at Google. They were a little longer. And I think we were excited about going off and doing something on our own. I think when you're at a bigger company, you spend a lot of your time thinking about what it would be like if you were at a smaller company, with a very small company. So we decided to do it. We all left at similar times and had no idea what we were going to build, and went immediately into just prototyping things. Most of the early things we would prototype, they were beautiful products. I was very proud of the craftsmanship that went into them, but they didn't have much of a market.

And interestingly, in sort of doing internal, not official, retrospectives, but basically sitting around on Fridays and thinking, "That product really … how did we mess so badly with that? It's very clear that it's not working." We started asking the question, "Well, if we could have seen what was going on in the product a little better, maybe we would have realized that it wasn't working sooner."

And that led to some of those early products. One of them was called Home Base. It's hard to even describe it. But I think one of the things we realized is that the fact that it was hard to describe: if we could have watched a few early users go through the product, it would be very obvious that they were super confused.

It would be very obvious that they would reach a part of the product and not know what they needed to do in that part of the product. And this is where FullStory actually came from. The idea was me and Joel, in particular, had spent a lot of time working on browsers. Joel was right there with me with all the stuff I just described. And so it occurred to us we actually have everything we need to capture what is happening in the browser in a way that you could go back and look at it.

And so the spirit of it was, "Okay, imagine user testing, but just on anybody actually using the application at any time." You could just say, "Well, here's a user who clearly was having issues and did not succeed. Let's just go back and see where they struggled and understand where the product really failed them." And that's basically where FullStory emerged. And to this day, FullStory's created a whole area of session playback where you can basically see what users are doing.

Beyang Liu:

It's really cool. It was kind of like a pioneer in that and mind-blowing the first time I saw what it was capable of doing.

Kelly Norton:

Yeah. We weren't sure it would work at first. And in fact, I remember it was a really ... I think those are one of those moments where the four people in a room are at their peak where Joel and I had discussed this idea. We talked about it with Scott Voigt, who'd become the CEO of FullStory. And we had said, "I don't even know if some of this stuff is possible. I mean, how can we send back all of the mouse movements? Do we want to send back all of the mouse movements? Can we approximate the curves of people's mouse movements? And would that be good enough?"

And so there were all of these interesting ideas and I remember Scott just saying, "Go prototype it." And that would be the last thing I did before I had to leave FullStory: is Joel and I just sitting in a corner, just banging out the first prototype of FullStory. And, yeah, not only was it possible, it turned out to be ... They have continued to make. It is a beautiful product.

As a person who doesn't work there now, I'm just proud of the team and proud to have been a part of it. It is an absolutely beautiful product, and I'm just so proud of how it turned out.

Beyang Liu:

That's awesome. So talk about going from FullStory to Etsy. What led you to take a job at Etsy?

Kelly Norton:

Yeah. After doing FullStory, as you can imagine, the good thing about being in a really small company with just people you work really well with is you get to work at peak production and peak creativity. The downside is you get paid nothing. And so I had run out of runway on the low pay, and actually had said from the beginning that “I'm in for about this long before I'm going to need to do something else.” And I reached that time.

And I sat down and I went, "What do I want from the next job that I have?" And one of the things I had picked up on is I talked earlier about how I had this decision where I assumed when I left Boston, leaving MIT, that I was going to the West Coast. And instead I ended up in Atlanta. And I wanted to stay on the East Coast.

And at Google, I felt that there was always this pull where at any point I could get moved to San Francisco, or I could see it that I was not going to improve my position in the company or get promoted or work on an exciting project unless I went to Mountain View. And so at this point, I was really determined to do something to ensure that I was not going to be forced off the East Coast.

And one of the things I decided to do is I said, "Well, I think that ultimately, we're going to hit a period of time where everyone is going to be exploring remote work. And I think my goal should be to become good at being remote, be good at understanding how to build a engineering team that can be remote. And I think the people who are doing that best is Etsy. And it's also a really cool company. So I would like to explore it." So I reached out to someone I knew who worked at Etsy, connected, and it was a pretty short interview process. And I ended up working on the search team at Etsy for a couple of years.

Beyang Liu:

Okay, cool. And while you were there, you created this thing called Hound, which is an open-source code search engine, which is really good. And I understand it was perhaps inspired by Google Code Search.

Kelly Norton:

Yeah. So Etsy is like a lot of places now who have adopted GitHub. And I will say that having ... Everyone is annoyed because Googlers love their monorepo. And I will tell you, I love a monorepo. I absolutely adore a monorepo. FullStory was a monorepo. I don't know if they still are, but that was a bunch of Googlers who are, from day one, there will be a monorepo. Etsy embraced the GitHub model. They used GitHub. The PHP monolith web app was in one repo. The iOS app was in another repo. The Android app was in another repo. And then any tooling was in its own repo. And as new things emerged, they were all in their different repos.

And so, at the time, GitHub search wasn't all that great either. And so we were constantly having these little fire drills of, "Hey, there's this feature flag, and it gets propagated around. And I don't know who all references it, but, boy, would I love to get rid of it." And it just wasn't really great way to answer this.

And at the same time, I had stumbled over Russ Cox's article about mapping regex matching to trigrams. And had just gotten interested in it from a personal standpoint. And as I continued to think about this problem, I realized that actually, Russ Cox put out the library, he put out a CLI version of this. I should just stitch a web UI on top of this. So in the background, I started just trying to stitch this together.

I was just going to glue it all together and make sure that it was as useful as I thought. And I had actually pitched this to a few other people and told them, "Hey, I'm thinking about making this web UI that sits on top of this old code search Go code." And I tried to pitch, "You'll be able to link to code and you'll be able to search code and it should be really fast. And we could include all of the repos." And I really got lackluster response. People were kind of like, "I don't know."

Beyang Liu:

They didn't get it.

Kelly Norton:

They're like, "GitHub is fine for me." I'm like, "But you don't have all the repos." They're like, "I mean, I don't know. I don't really feel the pain." But I stubbornly plowed ahead. And then I had reached the point where it didn't have a UI, but it had the API for the UI to call. So it's basically the search API that mostly exists now. You could basically pass it a regex and it would spit you back all the search results with some context.

And just as I reached that point, there was this huge internal email thread at Etsy where one of these fire drills had started. And people were trying to find, I think it was a list of about a hundred flags, or like, "We really need to know all the places where these flags get referenced."

And so I responded to the ... I basically quickly wrote a little program that just hit the API with all hundred flags, dumped the results. And I happened to capture the timing for how long it took to do the search. So I printed out the results in a nice format. I put it in the email. I responded and I said, "By the way, I know there'll be questions about what produced these results. I've been not really telling people yet that I've been working on this little tool, and I'll tell y'all more about it if you want to, but here's the two-sentence description of it." And people were quite excited to have the immediate answer in the email thread, but I also had people reach out to me very interested in working on the tool.

One of them was Jonathan Klein, who's the other person who worked on Hound with me, who was like, "I'm just going to build a UI. Is that okay?" I'm like, "Yes, please go build a UI." So Jonathan would end up building the first UI for Hound. And yeah, it got really popular in Etsy. And in fact, I still have folks from Etsy who reach out to me and tell me they still use Hound daily. MailChimp still uses Hound daily. And people in Mailchimp often thank me for it, which is really interesting because I feel like the tool is really more of hitting a sweet spot between what existed. Like it was just a quick path to take something that existed and to put a web UI on it that really captured the things that a web UI would give you: the ability to update on every commit to the repo and to link to search results. The linking to search results turned out to be way bigger than anyone expected. Like most of Hound's traffic is clicking on links.

Beyang Liu:

Yeah. Your experience resonates just so much from the kind of initial skepticism. Like we hear all the time. I almost divide the world into two camps of developers: those who get code search, and those who've never tried it before. And the ones who haven't tried it really just have a hard time comprehending how this is better than grep, or better than control-F in the editor.

And it is one of those things where, once you start using it, you start using it more and more. It's awesome to have all the code searchable in one place. And then the link sharing too, it just becomes almost this ... Like it starts out as a single-player tool because you're just searching for stuff for yourself. But then you start sharing links with colleagues. They can jump to what you're looking into and continue the exploration from there. It's kind of interesting that it's not more, I guess, universal at this point. I would say like Etsy and MailChimp are kind of the exception. Most engineering organizations don't have a code search tool, which is interesting, no?

Kelly Norton:

Yeah, I agree. And one of the things that I really love is the subtle little interactions that people develop with a code search tool. So, as an example, MailChimp is a PHP code base. We do have some static analysis through a tool called Fan that was created by some other folks at Etsy, and is now a part of the PHP ecosystem. But deleting code can actually still be pretty sketchy when you're in a PHP code base. I mean, there are some weird ways to call methods in PHP.

So there is a practice where if you are deleting a particularly important method in the MailChimp code base, and you feel like you've gotten all the references, you actually prove it to people by including the Hound, the basically ‘knows nothing found’ Hound link in your pull request. So we have a template, and one of them is you doing a quick risk assessment of your change. And people deleting code will often just post the Hound link as one of the risk analysis things they've done to show that the code is safe and that they've considered it. So I love little interactions like that because like those are the types of subtle interactions you can't really anticipate in advance.

Beyang Liu:

Yeah. It pops up everywhere. Just real quick, you mentioned your affinity for monorepos. I have to say I myself am in the monorepo camp. Can you talk a bit about why you prefer that model for tracking source code as compared to the multi-repo, more GitHub-driven model?

Kelly Norton:

Yeah. I think this really becomes acute the moment you introduce client libraries or protobuf into your ... So anytime you start defining the interfaces between things, boy, does it become acute really quickly. So I always have a joke that anytime the question of, "Hey, we're going to start doing services and we use a GitHub model." And I go, "But where are you going to store your protobuf definitions?" And inevitably, everyone laughs because there has been a long, drawn-out argument about where the protobuf definitions live. And I mean, my basic pitch for a monorepo is it's actually really hard to anticipate where the complexities are going to come in the future in terms of where code lives and what it takes to change something in one place and get it to the other place.

Monorepo is like, I mean, they basically solved that problem because everything's in the same place. And I think people ... I mean the argument I had heard against them after leaving Google was that, "Well the Google model really requires a lot of tooling." And originally, I was sympathetic to that. And I go, "Yeah, it did require a lot of tooling." And then after doing the non-Google model for a little while, I realized like, "Wow, we have a lot of tooling to support this sort of multi-repo world." So would really just rather have the simpler mental model about how to make code changes and just invest in the tooling because I invested in the tooling anyway; we still built a ton of tooling.

To give you a concrete example inside of Mailchimp: we have some PHP libraries and we distribute them via Composer internally. So, sadly, even though fortunately Composer exists – and Composer is kind of like the package manager for PHP, and it's good; it does its job well – it's really intended for pulling in open-source projects, but it does the job. But it just adds so much additional complexity to our changes to our shared libraries because we have to make a change to the library. And then we actually have automations that then push pull requests to all the various apps that use that library.

And again, this is all tooling that was required to support this. And somebody has to maintain that tooling. And it's not like it makes things easier because we use a continuous deployment model. So if your change to that shared library doesn't land well in the main app, the best course of action is to revert it out all the way back to the library so that you're not blocking deploys for other people, which is a lot of work. I mean, it really complicates some day-to-day flows. So I continue to ... The more I use the non-monorepo world, or the more I'm sort of in the non-monorepo world, the more I love the monorepo world.

Beyang Liu:

Yeah. That resonates with our experiences as well. It's just like you're basically just introducing moving parts when you have multiple repositories. And then, as you said, when you update a dependency, you got to update everywhere that dependency is called, and it essentially removes this atomicity from the change. They have to do it in multiple steps. And then you sometimes don't know if the first step is going to screw up the second step. And then if it screws up the second step, which is updating the dependent part of the code, then you have to go back and modify the dependency. But you're doing both things independently now instead of just verifying it all works nicely together and making the change all at once.

Kelly Norton:

Yeah. And in fact, we use a lot of feature flagging at Mailchimp. And one of the things ... And there's no mechanism built in to flag within some of the core libraries. So what you end up having to do is you have to kind of create a mechanism, like a static method to change behavior in the library, then get that version into the Mailchimp app. And then you set up a feature flag in the Mailchimp app, ramp that feature flag or control it. And you ultimately have to go back and delete the code you set up to flag changes and delete the old code. So it basically turns one change into five, in many cases. And, sadly, it discourages people from making the changes to the core libraries, which is not a good incentive.

Beyang Liu:

Yeah. Now I think a lot of people maybe convolve or confuse the monorepo setup with having a monolithic architecture. But that's not always the case. So my understanding is Mailchimp has a more modular architecture, right? Can you talk about that choice of architecture and what sort of tools that you've built to support that?

Kelly Norton:

Yeah. So Mailchimp, like so many places, Mailchimp started as a monolithic PHP application, and now we're heavily motivated to move to, I don't know what other people call it, but it's basically a hybrid architecture that evolves over in time into a service-oriented architecture. So a lot of people talk about breaking up the monolith. There are various approaches. And the approach that … so we've invested in a couple of things.

One, we're moving a lot of ... We're creating new services. Most of those services will live inside a Google cloud. We still have our monolith. Monolith runs in data centers, physical hardware that we can walk up to in many cases. It will continue to live in our data centers for some time.

And we have new functionality that's evolving, which, just naturally, we are inclined to build service things that are more service-oriented for those new things. And then we have some things that need to come out of the monolith. And I'm really just so fascinated by the different approaches that I see people take. Like you had Kirsten …

Beyang Liu:

Westeinde, yeah, from Shopify.

Kelly Norton:

Westeinde, yes, from Shopify. And she had mentioned that they had taken an approach that is ... I was really glad to hear that there was someone else on the other side of this argument. So there's the argument of just go in fast and furious, break up your monolith, tear it apart, get it into services. And I think the downside of that approach is, in so many cases, it confuses the value of encapsulation with physical isolation and operational independence. You can have good encapsulation in a codebase, even a monolithic codebase, I mean, just as part of software design.

And I thought that was one of the things that Kirsten was really hammering on is in a lot of cases ... And we're in a similar boat. We're a PHP, but we're built on an internal framework that is very, very rails-like. So we have many of the problems that a rails app has.

We have models that are chatty with one another. And, basically, I assume they know everything about what the tables look like for another model. I mean, everything is just twisted together. So, in most cases, the real challenge in breaking up the monolith in these codebases is it's not tearing things out and sticking them in another service. You have much deeper problems.

The biggest problem is that your separations of concerns don't exist. You literally have to shovel some chunk of code over to the side and build a clear interface to that portion of code. And the first step in all of these breakups, I have to imagine, is doing just that because it's not like we can move a portion of the code to a completely separate service if the boundary is not even defined.

And so anyway, for that reason, I was really excited to hear her talking about their struggles in defining the boundaries more than spinning up the services because that has totally been our experience there. We're way more focused on, "Hey, when we get into this portion of the code, let's worry less about where it's going to live and more about what the boundary is going to be and how we're going to enforce that boundary until it does get moved elsewhere. Because we really have to start to build these separations of concerns within the codebase, even within the monolithic codebase, if we have any chance of ever sticking some of this core functionality into a separate service."

Beyang Liu:

Totally. So we are coming up on the end of time here, but I was wondering as, the last parting thought, do you have any thoughts or maybe prognostications that you'd like to leave listeners with about the future of developer tools, or web development, or maybe just the path forward at Mailchimp?

Kelly Norton:

Yeah, I mean, so developer tools are really interesting because I would just say that … I don't know if this is a forward-looking statement or not, but when we were doing Google Web Toolkit, and when we did things like Speed Tracer, and we really focused on developer tools, and then we left Google intent on starting a business, we intentionally avoided developer tools because we felt that having worked in that space, that it was a thing that nobody paid for.

I look around these days and I am just ecstatic that Sourcegraph exists. LaunchDarkly exists. Many of these tools that are things you would build yourself and never possibly outsource are now businesses. And I think while having these things be totally separate does introduce some complexity into our lives, building a software business, having the software industry make a pivot where they are willing to actually pay and recognize the value of developer tools is really exciting.

So in general, I'm just ecstatic to see that there are developer tool businesses that are doing very well. And so, I don't know, I now wonder if the market had been like it is now when we left Google, intent on starting a business, would we have been more eager to start a dev tools business since that was our passion? I mean, I love what we created, but boy did it take us a long time to get to that, largely because we were operating in spaces we didn't understand, whereas we absolutely did understand developer tools and loved it.

Beyang Liu:

Yeah, the world has changed a lot. I remember we started Sourcegraph back in 2013. It was very different then ... You'd say the words ‘developer tool’ to an investor, and they'd almost get up out of the seat and like run away because the consensus was there wasn't a viable business strategy for it. But now, it's completely different, so many awesome companies, a lot of which we feature on this podcast, fortunately enough. And it's just great to see the innovation that's advancing developer productivity and spreading it across all the companies building software today.

Kelly Norton:

Yeah, I totally agree.

Beyang Liu:

Cool. Well, thanks so much, Kelly, for being part of the show. This was an awesome conversation and really appreciate you taking the time coming on.

Kelly Norton:

Thank you for having me. It was great.

Beyang Liu:

The Sourcegraph Podcast is a production of Sourcegraph, the universal code search engine, which gives you fast and expressive search over the world of code you care about. Sourcegraph also provides code navigation abilities, like jump-to-def in references, in-code review, and integrates seamlessly with your code host, whether you're working in open source or on a big hairy enterprise code base. To learn more, visit sourcegraph.com. See you next time.

This transcript has been lightly edited for clarity and readability.