Transforming delivery time at a pivotal moment with Jack Humphrey

Rebecca: Hey Jack, how's it going?

Jack: It's going great. Glad to be here. Thank you.

Rebecca: You and I worked at Indeed together and that's what we're going to talk about mostly today. But what are you up to now?

Jack: For the last nine months or so, I'm at LinkedIn where I'm an engineering director and the chief of staff for an engineering organization in LinkedIn that's about 700 people.

Rebecca: Got it. Can you tell me anything about what the engineering organization does?

Jack: Absolutely. Yeah. We're the talent solutions organization, and so we build everything related to the talent marketplace that LinkedIn has. How we get jobs on LinkedIn, how we help people apply to jobs, how we work with enterprises to get jobs posted on LinkedIn, but also the learning side, the LinkedIn learning business is part of our group.

And that's really cool. There's a lot of really great synergies there between people's careers and how they use LinkedIn to find their next play and also what skills they're building. And so we're doing a lot of really fun stuff with skills-first hiring right now.

Rebecca: I wanted to chat today about the time that we spent at Indeed. You were certainly there far longer than I was. I was there for about five years and you were there for 11, which is just mind-boggling. I've never had a job that long.

But the focus of everything that you worked on — I'm sure there were other things in your world — but to me a big thing that you worked on was driving productivity improvements at Indeed. And you know, when I wrote you to get this set up, I said, “That story just becomes more and more impressive to me the longer ago it was,” as I watch people today still trying to wrap their brains around how to address problems like that.

Maybe before we jump into to that though, and that's what I want really dig into, but first just talk me through, 11 years is a long time to be at a company and you were there not quite at the beginning, but pretty early on.

Can you just tell me a little bit about the change that you got to see and some of the challenges that you ran into?

Jack: Sure, yeah. I joined in 2009. There were just over a hundred people in the whole company and roughly 15 engineers in the whole company in 2009. So yeah, very small.

Rebecca: And you joined as an engineer?

Jack: I joined as an engineering manager. And so 10 of those 15 engineers were reporting to me and I was working closely with the VP of engineering who’d started the year before me. And we had a great partnership through the years. I actually reported to him for the entire 11 and a half years that I was at Indeed. Even though both of our jobs changed multiple times, somehow he was my manager the entire time.

Rebecca: That is absolutely unheard of.

Jack: So shoutout to Doug Gray. So people would ask through the years, you know, I don't know what number of years it starts to impress people that you've been somewhere, but they would start asking what's it like to have been there from that early?

And I would say it's like working for multiple different companies because you go through a lot of changes and a lot of iteration. And we went from, like I said, a hundred people to past 11,000 when I left in 2020. So it's a couple of orders of magnitude of growth.

And it didn't happen, you know, one steady straight, smooth line. There were periods of hypergrowth, which were… what's the word? “Learning experiences” where you do your best to kind of just hang on while it’s growing. But it was a fantastic opportunity, because I got to go from being an engineering manager to being a director, to being a VP.

Over my time there, I got to work on the core, jobseeker-facing products for roughly half the time I was there. And then I got to change jobs and do something completely different inside Indeed to lead this, what we originally called Engineering Capabilities, eventually became known as Engineering Platforms.

And yeah, it was a lot about productivity, but also it was just about satisfying the needs of a large engineering organization and making sure that engineering organization could have a high degree of excellence. Whether that was about delivery speed or just how things got built, craftsmanship, things like that.

Those were the areas that I got really passionate about in the second half of of my 11 years there.

Rebecca: I was talking to a fellow software engineering friend about how there needs to be a Bingo card for capabilities and platforms and core and I don't know, we came up with a whole list of them over lunch the other day. Yes, naming things is fun.

So a hundred to 11,000 people that's pretty mind-bending to think about all the kind of variations of a company that you got to see during that growth. I joined in 2016, I think. And I'm just curious: what was going on then? I know I got invited to a really cool all-day offsite around that time and got to be a part of a lot of great conversations.

But what were you being charged with at that time to be thinking about?

Jack: Yeah, that was a really interesting moment, late 2016 because it was right around the time that I had said I want to do something different. I want to figure out how we get better at building things.

And my initial focus, which you and I talked a lot about, was how we built our front-ends, how we did front-end engineering. As I dug into it, I got interested in a lot of different aspects and the more I talked about it with people, the more I thought, we need to lay out a three-year vision.

We were probably at that point, maybe a few hundred engineers. I don't remember the exact size, but it was clear that just going along with the status quo we would probably end up with a lot of things that we didn't like as we scaled up more and more.

We were at the tail end of four years of hypergrowth. Or maybe not quite four years, but it sort of felt that way. And we were still planning to grow a lot more in the next three years, and so we wanted to set ourselves up for success as best we could.

So one of the things that we did was gather people together. I think I tapped about 30 people around engineering and other parts of the technology organization, and we got together. You were part of that two-day offsite that we did, and the idea was to think about all aspects of Indeed engineering: how we built things, how we communicated, the technology choices we were making, technology roadmaps, and come up with what we wanted the future to look like.

It was a lot of fun and we actually came up with a really big list of things that we wanted the future to have.

Rebecca: And it's amazing to look back and think a lot of those things actually happened. It wasn't just stuff we talked about at an offsite one day.

Jack: Actually, with some prompting from a colleague, I did a three-years-later blog post where I sort of graded us on all the things that we'd set out to do. We didn't do them all, but we did a surprising number of them, I would say.

Rebecca: Yeah. A surprising number of them really stuck. So was it kind of broadly understood that Indeed was maybe slowing down in delivery or were there pockets of people who felt that way? And how did the hypergrowth play into the delivery challenges you were seeing?

Jack: Great question. To set the stage for that I would say from the early days of the company, we had very much a kind of get things done quickly, try out ideas, iterate, move fast, kind of culture.

We also had a culture that was very oriented towards “What's gonna have the biggest impact?” How do we orient everyone towards coming up with ideas that are gonna make an impact on jobseekers and on the business? We also instilled this ownership mentality early on of like, you should not just wait around for somebody to tell you what to do, but figure it out, right?

And all those things, as you grow, they work really great to a point. And I think we were starting to see the early signs of what that was gonna feel like in 2016 when you joined. And then three years later, despite us having accomplished a lot of those things from the offsite, I would say two years later was when it really became a big conversation. And it was kind of an executive level conversation. Because our executives, a lot of them had been there from when we were much smaller, and they were feeling like we used to get things done faster.

And engineers were saying, yeah, we used to get things done faster. I think because eventually you can grow, grow, grow, and you can give ownership and you can kind of try to just sort of have a horizontal approach of growth of teams own their parts of things and they go, go, go, and they do their best.

But you know, eventually you've gotta start coordinating all that. And then you've got coordination overhead. And that's a headwind. You've got to plan together. You don't have to, but it's a good idea to start leveraging work across teams. That’s a good idea.

And so you start to slow down. You start to see diminishing returns of adding an engineer or 10 engineers or a hundred engineers doesn't do what it used to do, or it feels that way. But also this all sort of happens gradually. And it's not like you drive off a cliff and you're like, what happened?We were great and then we weren't.

You just sort of gradually slowed down and everybody's figuring things out and trying to get better, but nobody knows how to address this kind of core challenge of the coordination headwind and I think there's the things that you can't control at the team level, like we've gotta coordinate with other teams, we've gotta depend on other teams.

I think what can happen is you start to confuse that with the things that you can control and make better. And you sort of just accept like, well, things take a long time so everything takes a long time. That's what I've observed happening, talking to people at other companies and arriving at other companies.

I've seen evidence that this is not a unique to Indeed kind of experience with companies that have grown larger over time. But we still had a good core culture. And it was worth trying to take this on. And again, it was the end of 2018 when the CEO turned to Doug Gray and said, what are we gonna do about this?

And Doug Gray turned to me and said, Jack, I've got a project for you: make engineering at Indeed go faster. And he didn't say it quite that way, but he was like, you know, you should tackle this. So it became my problem in late 2018 and it was daunting to say the least with probably a product and technology organization that was somewhere around 800 to 1,000 people. And I'm gonna solve the everybody go faster. That sounds hard, right?

Rebecca: Well, and you know, you were just saying coordination headwinds, that sounds like a coordination challenge to get a whole ProdTech org pointed at this. And especially the prod part of ProdTech, product, where it could be really easy for product to feel like working on this is going to take away from working on something else.

So how did you get the organization to act on such a large change? And maybe even before you answer that, I'm skipping ahead a little bit, but what goals did you end up arriving at for this undertaking and then how did you actually make that happen across the ProdTech org?

Jack: Yeah, I think that the process was sort of a lot of conversations with people who had thought deeply about this, both in engineering and product organization at Indeed. And first off trying to say like, okay, how are we going to set a goal?

Because we're not going to affect any change unless we can set a measurable goal. It's a very, very data-driven culture, and so, I didn't want to roll out a program to sort of support people in getting faster or say, “Do this to go faster,” which I've seen happen in a lot of companies. I don't know if a lot of people can probably relate to the scrum mandate or something that their company implemented so that everybody would go faster, right?

I didn't want to take that kind of approach. I wanted to start with a goal and say if we can measure it, then we can know we've improved it. And so to have a goal, I had to have a metric. Having a metric seemed incredibly daunting in itself. And that became a multi-week research project for me where I was able to, sort of take a lot of the ideas that I'd been talking with people about and look at the data that we had and try to say, “What can I actually look at here?”

And that's where I got very lucky. The way that I got lucky with the metric was in 2008–2009, probably before I even joined with a dozen-people engineering team or so Doug Gray had said, we're all gonna use Jira for everything we do. And you know, this small scrappy startup was like, “Oh, great, the new VP's coming in and making us use Jira, right?”

He's like, “No, no, no, it's gonna be important. We're gonna need to look back and understand why we made the changes we made. And so everybody's going to document their work in Jira and we're going to use this workflow.”

And he set up custom workflows that kind of matched how the team worked and that sort of got encoded in the DNA of how engineering got done.

So generally speaking, for any change that reached production, we had the Jira trail of what happened. Let me elaborate what I mean on Jira trail, I mean like state transitions through like this is a feature or a bug and now it's accepted by a developer and now they've started work on it and now they've resolved it and now it's in production.

I’m skipping a few steps there. But, so we had that. The other piece of a thing that I had done, really not sure why I was doing it, but it turned out to be incredibly useful was that I did a little skunkworks project where we kind of fully instrumented the Jira into our analytics platform so that we could run queries that were over all actions taken in Jira over all time.

And that was a fun little project that people told me I shouldn't do because it wasn't possible or it was gonna break our Jira instance, or there were lots of reasons not to do it, but I did it anyway. And really by, I did it, I should say another engineer, shoutout to Kevin Binswanger, who was the engineer and an intern working with him.

They did it, they got it working and it gave me this ability to do real-time interactive querying of this data set of what happens in Jira. And included in that was in a state transition on a Jira issue, how long was it in the previous state? Okay, now I've got something really tangible to work with.

Which is, how long does it take to get something out into production? And so what I was able to measure with some degree of fidelity was, from the time an engineer starts work on something, how long is it until it's in production? And I'd been looking a lot at cycle time and the research around looking at cycle time and optimizing cycle time.

And there's a lot of great industrial research that had happened over the years. And people are saying, is this going to be a cycle time metric? And I was just like, well, let's call it a lead time metric, because it's the lead time from a point in the work process to it being in production.

So we called it delivery lead time. I built a dashboard around it. I started socializing the dashboard, and it had org by org team by team what are people's mean delivery lead times.

Rebecca: I remember even that was a whole discussion. The mean versus the median. I remember all of this so clearly. I remember that email from Kevin saying this tool was available and that was such an eye-opening moment for me that this is a thing that you could do. And I also remember you starting to talk about delivery lead time. I know I was really skeptical, resistant, I don't know what word you might use, just based on “What is this going to do to quality? Are we going to go faster and ship worse things as a result?”

Can you talk a little bit about, on the one hand you had this goal to improve delivery lead time, or DLT, which was all the rage for a couple of years? On the one hand, you had this goal to materially improve DLT. On the other hand, how did you balance the concerns from people like me?

Jack: You were not the only skeptical reaction. I would say there were different flavors of skepticism, but it was rare to have somebody just say, “That sounds great.”

And I loved working in that kind of culture where everybody is like, “Okay, Jack, but what about this and what about that?” And, it's a great forge for an idea, right? Like the more you get tested on, is this actually going to work? There were a lot of — I’ll get to the quality one — but there were a lot of different concerns I heard from people. As I started socializing this with leaders and the executive team, I remember CEO saying, “Is this really what we want to measure? I don't like it. Wouldn't it be better to measure from the time we have an idea to the time we've implemented the idea?”

And I was like, “That sounds amazing, and I have no idea how to do that.” So this is something I can get my hands around and that I can tell you has meaning when I give you the measurement. We'd have to make major process changes in the way we log early stage work to do that other thing. But you know, maybe we can do that later, but it's not really addressing the problem of how do we know that we're getting better at delivering faster?

And so that was one set of concerns. I also had the frequently asked question, or frequently offered criticism, is that a thing? FOC? Which was, “Aren't people just gonna game the stats and, and haven't you heard that it's terrible to turn something like this into a target because then it ceases to be a useful measurement?”

And as you know, that was a valid concern I had to address. Because you know, how we were going to use this and what kind of incentives we were going to tie to moving this metric would make a big difference in terms of how much people would actually try to game it.

So I can talk more about that, but the short version is we tried really hard not to create incentives to game it. And also to say like, we can slice and dice the data finely enough that we can actually tell when people are gaming it. We can exclude data that looks like something fishy happened, right? And we in fact did that because we had to. Because occasionally somebody forgot to edit their JIRA issue and it looked like it took them two minutes to go from starting work to in production. And that was unlikely.

Then, there were concerns about quality. If we focus on delivering faster, are we going to compromise quality? And, and I think I had sort of two answers to that. One is, well, let's not do that. Let's not say that we're going to consider a valid way to speed up is to cut corners on quality.

We had a really good, I would say a very excellent quality process in general. An expensive one, but it was good. We relied a lot on people and we did want people to invest in more automation to help go faster. But we didn't want to do that at the expense of our users, our jobseekers and employers.

And the other part was to say, look, we've got quality metrics. These are going to be our health metrics and we'll make sure that as the delivery lead time metric improves we don't see correlations in negative quality metrics. And so that reassured you, maybe, when I told you that. You can tell me whether or not it reassured you or not.

But that was my approach to that concern. And there's probably some other concerns that I'm even forgetting about now. But those were a few of the kinds that came up.

Rebecca: Yeah, I mean, I think the other really obvious one is, “How is this going to impact my performance review?”

Jack: Right. Yes.

Rebecca: Or, “Is this going to impact my performance review?”

Jack: Yes. Is this going to be weaponized against me and my team if we are a slower team? That one is particularly tough because it depends on having a culture where there's trust. Like if I say we're not going to use this metric at the individual level, and we're not going to reflect in your performance review how well your team did on improving this metric.

Or, it’s all upside. It's either, you know, you get credit for getting better at this metric, but you're not gonna be penalized if you didn't. If there's not trust, they’re still gonna be either anxious or not want to take part, or try to game the system. And so, like I said, we were data driven and we had done things in the past where we had produced even individual-level metrics to help managers have conversations with people.

And I think actually, that was even more controversial than this, I would say, because of the way it was rolled out. And I was involved in that and we had kind of lived up to our promise to make this about inputs to conversation and not judgements. So we had credibility, which I think helped a lot.

I'm not gonna say that everybody loved this and was super excited necessarily, but I think that if people brought me concerns individually and trusted me, I knew a lot of people and a lot of people felt comfortable coming to me and telling me how much they hated it. So I had a lot of conversations and I think most of the people went away with… If they started at hate, they went away with grudging acceptance at least. Because it's like, “Okay, I sort of trust that this isn't gonna be a weaponized metric.”

Rebecca: Right. And back then, Indeed still was doing quarterly performance reviews too. So you were able to pretty quickly prove that — the feedback loop there was short, which is interesting. When you have annual reviews, that feedback loop can be a lot harder to establish.

Jack: Yeah, that's a great point. That's a positive thing. There's so many negative things about doing quarterly performance reviews, but being able to show very quickly how you do it and how you don't do it is a benefit.

Rebecca: You'd mentioned like top down, “do scrum”, “be better” doesn't make sense. How did it get done? Spoiler, at the end of a year you had, the organization as a whole had made just massive changes in its delivery lead time, like days of improvement. So how? How do you roll it out across an engineering organization that large?

Jack: Having the metric was my first step in kind of socializing the metric and showing people that dashboard. But then I did even more what I called “what if” scenarios, because my product partner, Tom, said, “What if we just set a goal to double velocity, to cut delivery lead time in half?”

And I was like, “That sounds really good. That's a great tagline for the presentation to the executive team. So I'd really like that to be the goal.” So I went and tried to convince myself with data, like, OK, what sort of scenarios would actually make that possible? And at the end of that analysis where I crunched a bunch of numbers, did a lot of spreadsheets, I said, “You know, this isn't crazy. That's not a crazy goal.”

If everybody's bought in and everybody's doing things I think I had some scenario that — I don’t know, I don’t remember — it was like, 80% of issues have a 30% DLT improvement or whatever.

I just remembered another concern that I had to answer. So just quick sidebar, another concern I had to answer was, “Won't this just mean that I'll just break my work up into smaller issues?” That was my favorite one, right? Because I was like: “Yes. And that's a good thing and you should have been doing that already.”

So some of our wins were less real in terms of the doubling velocity, probably, because we had outliers that were like a 30-day time, because it was just too big of an issue or something like that. Anyway, there weren't too many of those and I convinced myself that there weren't too many of those, but it did affect the numbers.

So. Okay, so sorry. Back to the story. We proposed this goal, we kind of vetted it with the executive team. I went around to all the engineering leads of the different parts of the organization and said, here's where you are now. Here's the goal that I would like you to set, that if you hit you'll be doing your part, right, towards our cut-it-in half goal.

That was helpful too, because it was not like saying everybody needs to double. Because I think our company-wide DLT was 15 days when we measured it on a trailing 12 weeks. And you know, if a group had eight days, I wasn't going to ask them for much. I was going to ask them to do a little.

So our overall goal was to get down to seven days. I wasn't gonna ask a team that had 18 as their average to get down to seven. Right? So just very realistic goal setting with the leaders, getting them on board with the goals and then they cascaded that out to their organizations saying, “Hey, this matters. There's going to be more coming on how you can set a goal at the team level and the kinds of things you can do to hit it.” That was all easier than you might expect because everybody felt this. And everybody wanted to have the tools to be able to deliver faster.

And I think everybody appreciated that we were going to put ownership at the team level to say, you know, you actually get to decide how you're going to improve your delivery process and move faster — or if you even need to. It was totally valid for a team to say we're good. We don't need to go faster.

I mean, we'd wanna see the data to support that, but again, that “We're not going to do one silver bullet” thing was appealing to the engineering leaders, appealing to the teams. Then we, and this is crucial, we actually gave them tools and we gave them support. And by them, I mean the teams themselves that actually were going to do the work of process improvement.

And the tools were things that we'd been building, or I'd say we'd been prototyping up to that point. We actually delivered to teams a tool called Peregrine, you might remember, that let teams dig into their delivery lead time and look at all the cycle time analysis and decide like, “Oh, hey, code review is the big thing for us. If we can make code review 20% faster, we'll be doing great.” Or QA verification, what can we do to make that faster? Or, you know, whatever the case may be. Or maybe something else in the process. So we gave them the tools to see where their bottlenecks were.

Huge, huge help to teams in actually moving this goal forward. And then we did a lot of communication about helping people understand the metric, how to learn to use the tools. And then very crucially, ideas for process improvement. Like, here's a workshop on reducing work in progress. Here's a workshop on test automation. Here's a workshop on Kanban versus other methodologies, just showing people different ways of working. And those workshops are just led by senior engineering leaders who stepped up and volunteered to do it. And we coordinated it as a program and we tracked it, and we moved the needle.

We didn't hit our goal, but we we got down to I think something like 8.1 or 8.2 days from 15. So I declared that an unqualified success.

Rebecca: Yeah. And I remember some outlier teams saw some really substantial improvements and that was a big deal. It still just really strikes me that it takes courage as a leader to give this problem to teams and trust that they will do the right thing. And was that scary? Was that obvious?

Jack: No, it was terrifying. The whole thing was terrifying. I'm always a little bit more confident, a little less afraid when I can lean on data. So I think that's what kind of gave me the courage: all the data analysis that I was doing and having the metric.

But yeah, I mean, at the end of the day, I think anything you do as a leader, you're sort of taking a gamble. If you can take an approach that's much more heavy handed on “everybody's gonna start doing this,” that can be successful or that can completely blow up in your face. You know, you can take this more like, you might call it laissez-faire approach, that I took and I did this over and over, right?

In the organization that you were in with me, we would agree on an objective and a result. And I keep this book right beside me, because Measure What Matters really influenced me. Like I didn't want to dictate, and I felt like if you hire smart people, point them at a problem, define what success is, and let them figure it out.

So that was… I mean, it was scary, but it would've been more scary to me to say, “I know, I've figured it out. Here's what everybody needs to do. Everybody needs to do WIP reduction and Kanban, and that'll solve all our problems.” And maybe that would've been successful too, you know?

But it would've all been scary and it would've all been risk of complete failure. But really at the end of the day, I had great partners and great collaborators who made sure that it was successful. So that made it a little less scary too.

Rebecca: I already know the answer, but I just want to hear you say it out loud. Anything you would do differently if you had to do it over again?

Jack: This may be the only project in my entire career that I don't have any regrets on. I've tried really hard to think of what are my regrets? I mean, I guess I could say I regret not making the goal like 8.5 so that I could blow away the goal or whatever.

But that's not even really true. I sometimes worry that I'll never have anything that feels like as much of a success as this project. Again, I hope I do because it's a really good feeling and it's fun. It's fun to talk about even years later now, it's fun to talk about with people.

Rebecca: Yeah. It was a fun time. It's also so interesting to me, this was 2016, 2017, 2018, and Indeed started in 2004, right?

Jack: Yeah. 2005-ish.

Rebecca: There about, yeah. I can never remember the number. So it's also interesting to me, the circumstances, the environment in which you had to solve this problem, and you were solving it for a company that had been around for 10 years and made 10 years of technology decisions.

And I'm seeing a lot more younger and smaller companies talk about this problem. Is this something that you wish you had done something about sooner, or do you feel like this happened at approximately the right time?

Jack: Yeah, that's a good question. I actually feel like it happened at the right time in some sense in terms of everybody sort of feeling the need for it.

That is sometimes the challenge with like big sweeping company-wide project is if you've got 20% of the company who feel like we need to do this, then you're dragging along the other 80% sort of kicking and screaming. I mean, I suppose in theory we could have done more as we were smaller and grew around process excellence that would've meant that we never needed to do it because we would've grown up with this built in culture of here are the ways to deliver software really quickly. But again — I don't know if this is a controversial statement or not — but I don't think there's a one size fits all for process excellence.

I think you've gotta look at it in the context of what it is you're building, who you're building it for, and you know what your technology choices are, how old your codebase is. Everything factors into what's right for a given team. And so that's why I think the approach of let the teams figure out how to go, how to improve their process is, is the right one.

And I recommend that to anybody I talk to. But you've got to give them support. You can't just say, “Hey everybody, figure out how to make your process better.”

Rebecca: That decision by Doug in 2000 whatever, to kind of enforce Jira. That in a way was a pretty powerful decision early on that probably had a lot of benefits to Indeed over the years. Even before you got to this project. Yeah. I miss it. I miss — that level of Jira rigor was so powerful.

Jack: Me too.

Rebecca: If you don't get it in place and you've got 10 engineers, I don't know how you pull that off later. Well, Jack, it has been really, really great to talk to you and to reminisce about those olden days.

Jack: It was fun. We could go on and on and on I imagine.

Rebecca: Maybe I'll have you on again to talk about more of this.

Jack: Anytime, Rebecca. Absolutely.

Rebecca: This has been great. Thank you so much.