Pulling off a multi-month tech debt paydown project

Rebecca: Alice, hi!

Alice: Hi!

Rebecca: Thank you so much for coming to talk with me. I saw your– well, I've known you for years, but I've also saw your lead dev talk from last year and really wanted to talk to you about the stories that you told in that talk and just in general about your experience with FT because when I first met you, you were working on design system stuff. We didn't really talk about platforms back then so much. And now I think you're pretty solidly in a pretty platform-y world in your current role if I understand correctly.

Alice: Yep. That's pretty right. Yeah.

Rebecca: Tell me a little bit first, just about your journey at the FT. Eight years is a long time to be somewhere. I've never done it.

Alice: No. So I'd never worked anywhere longer than three years before joining the FT, and I didn't know where to go after working at the government digital service. And I was like, well if I join the FT, I can work with my former boss, Andrew Betts, who is now there. He's really, really smart, and I will just learn stuff and be cleverer. And I don't know that the FT is where I'm going to spend a lot of my time because I didn't really resonate with the Financial Times as a product at that time. But I thought I'll get to work with smart people on interesting problems, so what's the worst that could happen?

And then, yeah, it just was way, way different to what I thought it was going to be like. Much, much better. Much more inclusive, much more accepting of me as a woman, and now it's sort of the thing that keeps me is every time I go and try and work somewhere else, I look at the workforce, and it's all these dudes, and they're saying things like, “yeah, I work flexibly. I mean, my wife does all the childcare, but yeah, it's great.” And I'm like, “Well, that's not gonna do it for me, pal.” So I'm kept because I do now think that the Financial Times is a great product, but I'm also kept because the culture is just amazing and exactly what I want and just great. It's just brilliant. And everywhere else looks kind of (unsatisfied noise) to me.

Rebecca: So when you started the FTE, you were working on the origami design system. Did we even call them design systems yet? Had we made up that word yet?

Alice: No. And there was a bit of a like, “Is origami a design system? Because where the heck are all the designers if that's the case?” That was my initial challenge with that title. I was calling it a front-end component system because of the absence of design from it. The thing I'll say is, there were no designers and one of the things I did was move myself and all of the team into the design team. So I reported into the head of design instead of someone in technology. And I just viewed that as the quickest, most efficient, and impactful way to get design into the picture.

Rebecca: It's interesting because, like I said, I was working in a similar space back then, and I don't know that that word ‘design systems’ had really gotten the traction that it has today. But also this concept of platform didn't have the traction it has today, I think. Or maybe I've just grown up. I don't know. But it seems like there's a whole lot more talk about thinking about building things for other engineers and making things easier for other engineers, which you've been doing kind of your whole career with FT, but we haven't necessarily had a name for it.

Alice: At the FT, there was a platform. But the platform was APIs for content, metadata, membership, subscriptions, payments. It was a series of Java spring boot APIs that still exist. And that was the platform. And then what I think has changed is we now have the idea of much more front-end platforms.

Rebecca: And when you say front-end, you don't mean in terms of pixels and designs so much as producing the front-end, or how do you mean?

Alice: I mean that there's a platforms team in customer products, which is the team I'm now the tech director of. And the platforms team is doing all of the shared stuff, so they manage the Fastly config, they have created a platform for people to build other stuff on, but also that there is another entire platform group called FT Core who build the APIs. And that's also, in a sense, a platform, but the two teams are doing quite different things.

Rebecca: I forgot the most important question about the FT, which is what is the deal with salmon colored paper? Which probably no one sees anymore, but that's what I think of when I think of the FT.

Rebecca: Yeah, well, I tell you what, it's annoying in digital products because you're trying to create accessible pages and you have less color contrast to work with because you have to have a pink page. And also, screens look different to real-life paper. So, the color pink is different. But, originally, it was a cost-saving measure because the pink paper was cheaper than the other paper because it's unbleached, and it is now not cheaper anymore, but we're staying with it because, obviously, the brand. It was a double hit because it was cheaper, but also it gave us a lot of brand recognition because if you're a man in the 1920s, then you're trying to pick a newspaper to read, the pink one you can see easily.

Rebecca: Well, and what a statement too, to all the people around you on the tube or whatever, you have the pink paper.

Alice: Well, I keep references of the pink paper appearing in popular culture. And I have to say that the most common times you see a pink paper in a film, although this does say a little bit about the types of films I watch, probably, is when the person is evil. So in Legally Blonde – which I'm sure everyone has watched many times – in Legally Blonde, the professor who sexually harasses Elle, he reads the FT. And this probably goes back to why I said earlier I wasn't sure the FT was for me. It's because culturally, if you don't know much about it and you're a 26-year-old idiot like I was, then you'd just think the FT is for men who care about finance, and it's much more than that, and it's become more than that as well, I think, in the last eight years, but yeah, unfortunately… Because it's pink as well, and distinct from other colors of newspaper, it is quite a good shorthand, apparently, in films – I don't agree with this statement – “this guy's a bit of an arsehole.”

Rebecca: All right. Well, on the more technology history side, kind of going back to what you said, where you weren’t sure if this was a place for you, and is it all just going to be rich white men trying to get richer? Did the FT have a pre-internet history of innovation or embracing new tech, or is this something new?

Alice: Difficult to say. I think the FT had a pre-internet history of making quite risky decisions, but then I've got one, which is the fact that the paper's pink, and then nothing seems to have happened for about – what are we looking at there? 70 years. And then a lot happened. So I don't know, you would have to ask someone a bit more of an FT historian than me, what we were doing in those 70 years. Certainly creating a very successful financial paper. But then thinking when things are being disrupted is where you get people taking more interesting and bigger risks.

And so, we launched an online presence in 1997, which had a little price and earnings calculator and today's main stories. And I think that probably was just, if I can imagine it, one man in editorial FTP-ing stories to a server somewhere. And that was all that really happened for like two years. And then we got more into publishing stories and more of our content. And these websites had the disclaimer, “to view this site properly, you will need a web browser that supports tables, forms, and backgrounds with auto-load images turned on, e.g., Netscape or Microsoft Internet Explorer.” So this was quite a long time ago.

Rebecca: Yes. And you were the first paywalled paper as well?

Alice: I think we were one of the first, yeah. The FT content was free to begin with, like everything, and then we put up a paywall quite early on in the “publisher wars.” We were one of the first, we have a very successful business model with a paywall. The CEO, John Ridding, that we have currently, has been CEO about 15 years, so one of his big things was paywall. He just had a real conviction that people would pay when a lot of people didn't think they would.

And he did quite a lot of quite spicy things like taking the FT app out of the App Store and just making it a web app only, which was a big thing maybe 13 years ago. Yeah, it would be 13 years ago because the app is 13 years old. We still have the app, it's still going. So there have been, not so much innovations, I think, but brave decisions by people who aren't technologists.

Rebecca: But they made big decisions about technology and they paid off. Of course you weren't there from the very beginning, but I'm curious from your perspective today, as the tech evolves at FT, are there areas of clear underinvestment in hindsight? Like “oh, I wish we had realized this would be important, or I wish we had done this differently.”

Alice: I think the FT suffers from the same problems that all media companies that are of a similar age have, and probably all companies of a similar age and size have, which is that it's really easy to make things and it's really hard to kill things. So, I don't think we underinvest particularly in technology, but it is very expensive and I think we have a lot of, even just down to features, the features add up to a large maintenance overhead. And if we could be a bit more brave about turning off features, then we would probably be able to more smartly invest in different things.

But we're currently investing a lot in AI and ML stuff. And I think, when there's a significant enough reason to do so, the FT will pony up some money for some stuff. I think, in general, we could be better at turning things off. And the problem is when you try and turn off a feature or kill a product, there's someone somewhere on an email thread going, “absolutely not!” And then you have to really understand their objections and get to the bottom of them and it's a really complicated, long-running project to do that kind of thing, whereas building something is easy. People love to be building things and they love to see things being built.

Rebecca: Yeah. Maybe you can also talk about what's changed organizationally in the time that you've been at the FT as far as the engineering organization, what structures have come and gone.

Alice: Yeah. So I think the main thing that's happened is Agile. When I joined, the origami team was just four people. So the process was a weekly meeting and we would decide what to work on and we would have a Slack channel that people would raise support requests or give us ideas, but it was super light.

And in that role, I would be wearing a bit of a delivery manager hat, but there was no need to really report anything, so the delivery manager role was me just saying, “why is this going slowly? Can I do anything?” And unblocking things to the person who is sat next to me. A bit of product manager, a bit of tech lead, a bit of DevRel sort of advocacy, but it was really just me seeing problems and trying to make them better whilst everybody else was– I mean, I was writing some code, but I was also trying to just use my time as effectively as possible.

But then the other bigger, more established proper teams at the FT were building ft.com, rebuilding it, which was a £10 million project that took five years. And then the platforms teams that were doing the APIs, they had a bit more of a– so the website that we were replacing at that time had three monthly release cycle. So it was quite waterfall-y and it certainly wasn't continuous. CI/CD, right? So that's a big thing, and we started by doing that on ft.com with this new website, which is codenamed Next. But then that way of working leaked out into everywhere else.

So I'd say that when I joined, there was a bunch of very seemingly quite old-school practitioners of software development and now everybody is working in a much more modern way.

Rebecca: So, in your talk, you talked about this “amalgamation of systems” for taking data and making it show up on a screen somewhere. I definitely recommend that people watch the talk because there's some great takeaways in there and it's also some great storytelling because you're awesome at that.

I wanted to dig into it from a different angle and understand, what did it take media, but really everybody who's existed for 20 years who's been doing tech things, they've accumulated exactly that kind of debt, a series of decisions that were totally rational in the moment but that added up to an incredibly irrational system. We all have those systems that we work with. You managed to fix one. How did that happen? How did you get the organizational buy-in?

Alice: Yeah. So the challenge was, my colleague Kara has summarized this as sort of spaghetti which we wanted to make into lasagna. Maybe it's useful to talk about side effects here, but one of the side effects of this was when there was a rendering bug with some content on, say, an article page or a stream page or the homepage or something, it was really hard to tell where the bug was in the content because the content was being transformed in loads of different places.

And so we kind of knew that we had this architectural problem for a while and people have been talking about it for a while as an issue and so someone – not me – wrote a proposal for it. And I was actually not very interested in working on this problem because I regarded it as being incredibly hard to solve and also exactly the sort of tech problem/software project that we all know about where it takes a really long time. No one really understands it. And the people who want it to be over so that they can get their engineers back working on these product features are just continuously looking at their watches and tutting.

And also, what if we built the wrong thing in the software? What if it sucked? So I was minding my own business, hoping no one was going to assign this to me, thinking that it was a problem that some of my peers, other principal engineers, were much more interested in doing. And then the tech director at the time, Anna Shipman, was like, “Alice, I want you to work on this because it really needs excellent communication and that's one of your strengths.” And I was like, “Okay…” And, so then I thought, “well, okay, I will say yes to that, obviously. My job is to say yes.” So I started working on it and what I realized was that it did really need good communication, but it needed some razzle dazzle as well.

All I had was a proposal and it didn't have a proposal of how many people or anything like that. It was actually quite a technical thing about what we should be aiming for and the problem to solve. So I said “Give me… what we need is…” And I picked a team of four for six months because it felt like a six-month team of four kind of thing and I knew if we did– Three months is never long enough to do anything at the FT. It's a too big of a company, there's too much going on. Well, three months is good if you know the domain and the team is already working, but if you need to set up a team and get something going, six months is minimum.

And I knew it needed to be four people because I didn't want it to be two and then have one of them take a load of holiday and then you've got one person on their own. Small teams are incredibly brittle. So I said, it needs to be four. And I said we're going to just give them a problem to solve. And the problem is, “look at this spaghetti, look at how hard it is to solve, to understand what our content is doing, understand these rendering bugs, and then go away and improve it.”

Rebecca: Make the lasagna.

Alice: Yeah, make the lasagna. So the first thing I did actually was decide who I was going to ask to be the tech lead. Once the funding had been agreed theoretically – and actually I do talk about this in talk, but the first thing anybody says at the FT when you ask for anything is, “Yes, you can do it, but do it with half the people or in half the time.” So I knew that was going to happen because it always happens. And I said, “Well, no, if we can't afford to do it for four people for six months, there's no point doing it. If you half the team size, it doesn't go – your listeners will know this – half as slow. It goes slower and then you get less done.” So I had a lot of conviction in the thing I was asking for and I was also perfectly happy to say, “Okay, we just shouldn't do it” because I really believe that it would be one of those projects that goes on forever. And everybody's just like hoping we'll end soon.

So, once I got that part agreed, I knew who I wanted to be the tech lead. It's Kara Brightwell. She's got an expansive and brilliant mind and she's been complaining about the rendering for years. And I was like, “Kara, you're up. Do you want to do it? Who else are we going to get?” And then we started thinking about who we would be able to attract to this team to have the energy and excitement that they would actually be able to solve the problem in that time frame and not hate the work or make it a real mess – over-engineer it, make it really hard to maintain. I wanted people who understood what it was to maintain software as well as just build it, because the project would take six months and then we would have to have this software for the rest of our lives.

Rebecca: The rest of your careers, perhaps. So the first thing I wanted to talk about is just the fact that you were chosen for your communication. Of course you were, you're a great communicator, you're an excellent communicator. But also just the importance of that skill, even back when you were just a line manager on the origami team, the importance of that skill. You're, I'm sure, also a great engineer, but that was something that differentiated you and sounds like continues to differentiate you. It was just your ability to communicate. Is that something that you have tried to foster or you've been doing this since you were three?

Alice: Actually, I've been doing this since I was three. I have a twin brother and my mum always comments on how great I was at talking to grown-ups, getting them to listen to my stories and explaining things. And if Tim and I would have an argument – Tim's my brother – it would be really unfair for him because I would just be able to articulately put my point across and explain what happened, and Tim was much less good with words and couldn't explain. So yeah, I think actually there's a bit of natural talent and I do just love talking. I am an extrovert, that's definitely true.

Of course, communicating is not just talking, it's also listening, and that is something I've had to learn and work on. But I think there's a natural thing there. And it did take me maybe a little while to be comfortable with that, because your prototypical engineer is not a great communicator. They're this brilliant mind that knows everything. And that's not really me. But once I knew I was going to be running this, the thing I also knew was that I needed a brilliant mind.

And that's why I asked Kara, because I thought she and I together, I’ll be able to keep up with Kara, sure. And I'll be able to steer her a little bit on what our principles are for building this system, but she will be bringing the heavier engineering skills, definitely. Especially in this space.

My background is front-end, properly in browser front-end engineering, which doesn't really exist anymore since Node collapsed all of these roles into JavaScript developer. That's the thing about leadership, isn't it? You have your strengths and then you compensate effectively for the things you're not so good at.

Rebecca: Speaking of communication, what stories did you tell to get buy-in for this project?

Alice: So the first one was this byline thing, right? I thought if I could show how much happens to a byline through all of these systems– the byline is the smallest unit of content, right? It's really hard to do it with articles or full article bodies or tables or anything that appears in an article because there's so much stuff happening. But a byline is really easy to understand for anyone.

So the first thing I did was trying to illustrate. Instead of just showing an architecture diagram, I wanted to take the byline and push it through the architecture diagram so that people could see what was happening. And then I would push it through the architecture diagram of the improved system so people would see how it was better. And when I was doing that, I realized there was this bug with apostrophes, which is sort of the basis for the whole talk at LeadDev.

But, initially, all I was trying to do was give a five-minute update on the work, or introduction to the work, actually, at our team Snapshot meeting, which has all of the engineers, all of the product managers, all of the delivery, all of design. It's a big team meeting, about 70 people, and I just wanted a quick video of me explaining the architecture that would be no longer than three minutes to show what we were doing in a way that they could relate to.

Rebecca: And is that the same video that was in your talk? Because that was amazing.

Alice: No, that video is similar, but that video was several rounds. I also created a makeshift sound booth out of my daughter's duvet cover. No, that one, the production quality is a lot higher, I think is what I'm saying.

I am such a communication perfectionist when it comes to giving talks. Not like this kind of thing where it's a bit free-flowing, but I recorded it the first time and there was a tiny echo in the mic and I was like, “This will not do.” And so then I did it a second time in a den, effectively.

Rebecca: If I understand right, you have the old system that is taking content from some back end and, ultimately, through a series of magic that happened in lots of different places, HTML would come out on the other end. And, I'm imagining, as is usually the case with this sort of thing, that there was a time when it was okay to just make another content transformer, where that was exactly the right thing to do. Or maybe not, maybe this was bad from the very beginning. But I'm curious, was there a tipping point where it became like “we got to do something”?

Alice: Yeah. So I think that was fine. And it wasn't so much that making another transformer is the wrong choice, it was that we didn't have all of our transformations in the same place. So you might be doing a transformation on the byline in one application and then a transformation on the article body in another one. And maybe a downstream application. But if you have another application that wants to talk to the upstream application, not the downstream one, then you have to recreate. That's where you start doing this kind of duplication of work.

So, for our world where we have the app and we have ft.com, and, prior to this work, we have a CMS that Editorial use and their preview of an article is an entirely differently rendered article. They had recreated every single part of the article page in the CMS, and now they are all using the same rendering engine so that your preview is actually a preview. And you don't have the situation where we create a new component, a special type of chart widget or something, and you have to wait for it to be implemented in the CMS, in Spark, before you can see what it looks like on the website.

So this work did unlock loads of really amazing stuff like that, and that wasn't even the intention at the beginning. And in fact, the only reason we realized we could do that is because I was communicating with all these people about what we were doing and someone went, “Oh, hang on, could we just plug the CMS into that? And then we wouldn't have this really annoying problem.” And I was like, “Yeah, probably, I think we could.”

That's one of the massive benefits of going around talking to everybody all of the time and being good at communicating is that these opportunities arise, they just present themselves to you and you're like, “Great, let's do that. That makes a lot of sense.”

Rebecca: So in a world where you are dealing with decades of decisions that were made by people with the best of intentions, but are maybe obviously bad now, how do you keep a project like this blameless and focused on the future and not on being mad about the past?

Alice: I think the FT's culture is already like this, so it wasn't hard. It is like a self-correcting thing at this point in time. It's quite a mature engineering team, and we've all made the “bad decision” at some point, so we all know that that's what engineering is. This is Agile manifesto stuff, I think, but you do the best with the information you have available at the time. And I think because everybody knows that was them, and in some case, literally, it was them as well, actually, I think it just means that we're not going around pointing fingers. And if anybody, a new person or a person having a bad day, did try and point fingers, quite quickly, someone – not me, I'm not there going, “No, we must be nice to everyone all the time!” – but someone in the room will go, “well, we've all done it, haven't we?” So it becomes self-correcting, that blameless culture.

I got everyone to read The Pragmatic Programmer, which is 20 years old, and we did the 20th-year reissue. That has a whole thing about blameless stuff in it, and it's very reminiscent of what Etsy were doing 20 years ago. And so yeah, it was a bit of fun to get the team, although a lot of people already had all of the Pragmatic Programmer stuff in their head, for them to read where the origin of the phrase ‘rubber ducking’ comes from, which was popularized by that book, was quite a cool ‘learning about our culture past’ moment.

Rebecca: So you talked a little bit about this, but I'm curious how you communicated or the team communicated throughout the course of this project, precisely because you steal four engineers for six months and people want to know that that's going towards something worthwhile. So how did you keep the engagement, keep the interest and keep the commitment going over that time?

Alice: So there's a couple of things we hooked into, and it's partly just understanding not what you want to tell someone, but what they want to know, and then saying that about your project. And so, for the other engineers, we did week notes, which are meant to be sort of irreverent little weekly diary entries about what has happened in the project. Sometimes it would be funny, sometimes it would be more serious, sometimes it would be very fact-heavy.

We also created decision documents for every decision and some of them got quite long because we kept the ESM versus– oh no, what's the other thing called? I can't even remember now. Node modules, maybe. That decision doc flip-flopped between the two several times, but we have the whole thing captured now. And so every big decision, or even quite small decision, was captured there. And that was, again, for engineers.

These things hang around for such a long time and it can be really hard to find. In code, great code should be sort of self-documenting. You should have some comments in the code if something strange is happening, or your Commit messages should tell the story of your project, but it helps to have, where there's a big decision, it's in a Confluence page or something. And then, for the people who weren't engineers, we have this monthly meeting called Snapshot, which is where we would give updates on what we had achieved and how it was going.

And then we have the usual things, Office Hours, for anybody to come and ask. And the reason we really went hard on attracting other engineers to understand the project is because it's a platform, so we need the apps team engineers to understand what's happening over here, because at some point we're going to come to them and be like, “you now need to adopt this software.” But we also wanted everybody to be looking at this project going, “oh, that seems really fun and interesting and I'm excited for it” and not like, “Oh.” You want them to be excited about the good project coming to you, not worried about the cursed project where everybody's really stressed and bored.

I don't mean that we were pretending it wasn't fun, I mean that we were making sure that everybody knew that it was fun, because I wanted the good energy to be obvious. And if it hadn't been fun, I'd have done things to make sure that it was fun. I worked hard to convince the engineers with a lot of energy and enthusiasm for this project to come on board. It meant that actually it really went incredibly smoothly, the actual build, even when the work was very hard in some places.

Rebecca: And I'm sure communicating with other engineers, they become advocates for the project because they can imagine how this will be good for them, how this will help them do their jobs better.

Alice: Yeah. And they bring important challenges as well. “Have you thought about this?” “Oh, God, no, we haven't. Thank you.” So that was essential, because there's the communal community or shared collective knowledge of how to render a web page in the customer products engineers is pretty significant.

Rebecca: The code doing these transformations was owned, I'm guessing, by the teams that owned the application where the code was. Is that right?

Alice: So there's the platform team in customer products who looked after the elastic search cluster that had all of the content before, and we replaced it with GraphQL and a couple of APIs. But maybe originally, yes. There wasn't a platform team, so I don't know who created the content APIs, but that information pre-dates me.

Rebecca: I was mostly asking because you can create this new, single, beautiful system with four engineers having a great time for six months. That's awesome. But then how do you take care of that system in the long run if those four engineers go back to whatever they were doing before?

Alice: So what we did was made sure that two of the four engineers were from the platforms team. When I say it took six months, the four engineers working on it – and I think it was actually five – five engineers. I pitched for four, but I got five.

Rebecca: You're just that good.

Alice: Well, this is a pro tip. Sometimes you just need to look for opportunities and be patient and wait for the opportunity. And I found an opportunity, which was an engineer needed a project to work on, and I was like, “Well, here's a project. Don't you think you should just come here?” And that's just how that shook out.

So some of the engineers were from the platforms team, which meant that they could easily take that work back into the platforms team. And then, what I was originally planning was for one of the other engineers who had been working on it to go into the team that runs the article page, because that's where most of the implementation of this new project was going to be. But they didn't like that idea, so it didn't work out like that, but the work did go into the article page.

Because of all of the noise and positivity around it, they already knew quite a lot about it. They'd been going to the office hours stuff, they'd been making suggestions. So there was a very heavy involvement from everybody.

Rebecca: In the end, do you feel like you actually accounted for “all use cases” or are there edge cases that are just going to live forever?

Alice: So the way the project started was, we were like, “we need to audit every single component.” So, in an article, you could have charts, graphs, big number, for example. That's just a big number, a number written big, bigly, in an article. You could have quotes, obviously, a couple of different styles of quote. You can have bylines, different styles of header, all kinds of things, hundreds of things, video.

And so we started trying to audit them and work out where they were transformed, and how and whatever, but it was just an awful, awful slog. And so I said, after we'd done about 60 percent of them, “We just got to get going. I mean, we got to do some stuff. We can't audit everything forever.” And so we kind of worked that way and made some big strides in the architecture through what we had already. But I think a BA would have been a wise addition to the team, to be honest, because QAing this was really hard.

We launched it a few times and broke very random bits of articles that we just hadn't quite accounted for. And I think we've caught all of them now, and this is kind of how we work anyway. It's okay to create bugs, in my opinion. If you're shipping software that's a hundred percent perfect, then it's too perfect. You're taking too long to do it. So it's better to ship something and then find that you've missed something than it is to ship the most perfect. ‘Cause you'll never do it. You'll never deliver anything if you live that way. And if you have a blameless culture where you get together and do the incident retro afterwards and learn from them what you missed, then we're good. That's great. That's perfect.

Rebecca: Did you have to do any expectation setting around, “there will be bugs”, or was that just well understood?

Alice: No, that's well understood. And we have a really, really phenomenally good postmortem operational response team here. I do now as tech director have to give the same speech every three months, pretty much, which is, “Incidents are good.” The reason we don't track the number of incidents and then say, “hey, this number's going up” is because incidents are so related to what we're shipping, how much we're shipping.

And sometimes, you can look at an actual specific incident and be like, “This shouldn't have happened.” But mostly, I would rather we were having incidents, because it means we're shipping code and we're making improvements to the website, than have zero. Because there will always be incidents, even if your code is perfect, because someone will delete an AWS bucket or all of the East region in Amazon. And that's a bit of a deep cut, I think that probably happened like eight years ago at this point.

But there will always be incidents, even if they're outside of your control. So we need to have this high degree of understanding on what we do when there’s an incident and being able to easily roll things back and all this stuff. And so, I think the team was happy with the way it rolled out and there were some slightly stressful moments.

Rebecca: But I think that's so true. Like you could have spent months QA-ing it instead and delivered none of the value in the meantime.

Alice: I mean, we did do quite a lot of QA-ing it. Definitely. I don't want to say that none happened at all, but if you've got proper monitoring in place, the bugs you can really quickly find if you just turn the fire hose of traffic at something and then just see. As long as you can turn it off again quickly, it's much better to just quickly do something like that if you're reasonably confident than to worry yourself to death. You'll still miss something anyway on this type of project.

Rebecca: Thank you for telling that story. People are always asking, “I want to hear like real stories of people doing real engineering leadership work.” And this is it. So I really appreciate you sharing that. I wanted to just close. I'm expecting that you are a person who has some hot take about the tech industry right now. What is your current tech hot take?

Alice: I mean, it's a very interesting time, isn't it? I'm trying to hire at the moment and the caliber of candidates is really high. And the FT's had a very stable– we haven't made any layoffs and I don't think we will, but it's very interesting watching people who haven't struggled for jobs ever come in and interview with the FT and be so bad at interviewing because, despite their employment history being like, “Amazon, Microsoft, blah, blah, blah, blah.” And I'm like, “Wow, this person's going to be great.” And then they are just not good at interviewing and not able to explain what they did or why in any of their jobs. And that's very interesting to see. An interesting problem as a hiring manager as well, actually.

Rebecca: Yeah, because they never, I don't want to say never had to, but, I think it is really interesting right now. I'm one of these people who got opportunities that I wouldn't have gotten in this market for a period of time. Just because the tech market was so hot. And now, thankfully, I have learned to be a good storyteller, but that was just sort of a ‘nice to have’ and now it's kind of an essential skill to have, to be able to tell the story of why you are more employable than the hundred other people whose resumes look a lot like yours.

Alice: Yeah. And I think also, although I want the people who've worked in these top-tier tech companies, I also worry that they won't actually be able to enjoy the work at the FT because the technologists are not the crown princes of the organization. That's the journalist. So when you are here, you do need to be able to put your point across. You can't just walk into a situation and expect everybody to go, “Oh, well, the engineers say this, so it's gotta be this.” You do have to be able to communicate well and influence and explain things to people who don't know how software works. So I don't know. Is that a hot take?

Rebecca: It's hot enough. It's a take.

Alice: It’s a take! Well, we'll take the take.

Rebecca: It's a professional observation that you have made. Well, Alice, oh my gosh, it is so good to chat with you and I'm just dying to listen to this and hear your accent more. It's been such a treat and so fun to get to see you rise through the ranks and stuff. So, congrats and here's to another eight years if you want them.

Alice: Yeah, maybe.

Rebecca: We'll see. No commitments.

Alice: I need people to think I could quit at any moment because otherwise they'll begin to take me for granted.

Rebecca: Well, again, so great. Thank you for doing this and look forward to talking to you again soon.

Alice: Yeah, okay!

Rebecca: And that's the show.

Engineering Unblocked is brought to you by Swarmia, where we're building the tools that help the best software teams get better. With Swarmia, modern engineering organizations get the visibility they need to build the right things faster. You can find more episodes at unblocked.fm or wherever you get your podcasts.

See you next time.