Understanding "Inconsistent Chaos" at Facebook with the 3X Model - A talk by Kent Beck

This year I attended my fist YOW! Conference in Melbourne on 6th & 7th December. I hope to share my experiences in this writeup and encourage as many developers as possible to attend this conference in future. It was the best conference I’ve attended to date and the speakers were an absolute treat. I enjoyed all but one of the fourteen talks I attended and I was blown away by the amount of speakers that travelled from The US and are really well-known in our industry.

The About page of YOW! gives a nice overview of what it’s all about:

The aim of YOW! is to bring together internationally recognised speakers and developers to encourage excellence and innovation in the local development community. We cover the emerging technologies and best practices in the software industry – regardless of technological platform or language – without commercial hype.

Since launching in Australia over 30,000 people have attended YOW! events and we are now the largest independent event organiser in Australia for software development. We are able to achieve this through support from loyal company partners, over 100 usergroups, media outlets, government organisations and publishers.

The person who I was most looking forward to seeing was Kent Beck (yes, the Kent Beck). I had high expectations… I hadn’t heard of Kent Beck until earlier in 2018 when I started my first job as a developer and I noticed all the software Architects and Principal Developers in my office always threw in a reference to this ‘Kent Beck’ guy every now and again. It was mainly in the context of helping us junior developers focus on modern development practices using things like Test Driven Development (TDD) and Pair Programming. Little did I realise, these concepts came out of the Extreme Programming (XP) movements of the late 1990’s and early 2000’s when Kent’s “Extreme Programming” and “Test Driven Development” books became some of the most highly regarded industry literature in our generation. And in my opinion, I think it’s because Kent and his co-authors talk in the sense of ‘you should do x, because it will make your life better as a developer’. They talk from experience, from mistakes they’ve made in the past, and it doesn’t feel like typical ‘computer-science-in-academia-101’.

3X: Explore, Expand, Extract

Kent Beck’s talk at YOW! 2018 was my number one talk and the best speaker I’ve seen in my experience to date. I’ve followed Kent on Twitter for a while now but seeing him in person was an unreal experience. He’s quick-witted, succinct and brutally honest. No inexperience noted. I raced in as soon as the doors opened and I was fortunate enough to sit in the second row of his keynote titled ‘3X: Explore/Expand/Extract’. It was described as follows:

3X: Explore/Expand/Extract Before you can evaluate a method, you have to understand its goals. Before you can evaluate a style of software engineering, you have to understand its goals. Quick execution of experiments? Rapid scaling in the face of unexpected bottlenecks? Sustained, profitable growth? Each goals requires a different style and yet we talk about software engineering as if it should be one thing. This talk introduces 3X and the ways software development, quality assurance, design, management, financing, planning, and staffing change depending on the goal of development.

On the face of it, the description sounds like a fluffy load of garbage; the last thing you expect from someone so revered. The talk probably should have been called Kent Beck: An analysis of my time at Facebook. Entering a global giant’s software engineering team as a best-selling software engineering practices author and nobody knows who I am and they’re not following anything I write in my books, and hey WTF… they’re still killing it?

Kent Beck speaking at YOW! Conference in Melbourne, 2018
Kent Beck speaking at YOW! Conference in Melbourne, 2018

Kent’s talk was very conceptual. He had big ideas in his head based on the way he sees how the world of software works and in this talk he explained some of those thoughts using visual representations based on some behavioural economics principles he’d learned recently. He described this talk as his way of trying to make sense of software development at Facebook in what seemingly appeared as ‘inconsistent chaos’. But he knew there was a pattern and he had to understand it.

Kent began by discussing what it was like to start at Facebook in 2011: a place where this ‘cowboy culture’ was rife and nobody writes tests. He reminisced about how he initially thought he could have an impact at Facebook by teaching people TDD but nobody wanted to hear from him. To my surprise Kent said that if he was going to try to make it as a coder at Facebook, he would have failed miserably compared to others, but as a coach? Well, maybe that could be something.

“I had this puzzle in my head. These Facebook engineers are not doing the stuff I write about in my books and they’re being very successful. How does that work? I don’t mind if you don’t do the stuff in my books. I just mind if you don’t do the stuff in my books and you’re successful. That pisses me off.” - Kent Beck

So he decided that perhaps he needed to change his attitude; he was going to, in a manner of speaking, ‘un-learn’ TDD and just go with the flow. He wouldn’t correct people if they suggested not writing any tests for their code, and he just went along with it. His key learning from this? Long story short: Tests are a form of feedback. Don’t write tests if you have other forms of feedback. Because we often do; we have deployment pipelines and other monitoring tools, some of which you might custom-build for your own codebase. So it might be considered over-engineering when we write tests we don’t even need if other feedback mechanisms will tell us when something goes wrong.

“If the cost of writing tests is really high, and the likelihood and consequences of failure are really low, consider this trade-off and then don’t write tests (as long as you have other feedback loops).” - Kent Beck

A convex world or a concave world?

Kent then went further into how he tries to understand this behaviour by Facebook engineers, borrowing a concept from the world of finance. He explained the convex and concave ‘payout’ models and related these to software development project experiments and the payoff from those experiments.

Kent Beck's 3X Concave Convex Model

First, the convex model: Is your company in a convex world? He talks about how with many small, cheap investments that you can do, you sometimes get a huge payoff. It’s hard to analyse and predict the future in this model (a Product Manager’s nightmare) but this is where you should try a little bit of everything; experiment lots and throw it away and move on if it doesn’t work.

Compare this to the concave model: Is your company in a concave world? Think of the banking industry here; large investments of time with big downsides if they fail. Here it’s good to “systematise” and reduce risks wherever possible. At the end of the day, it’s all going to payout the same amount anyway, most of the time.

Now, we get into the real meat of the talk with an explanation of what he calls the three E’s: Explore, Expand and Extract. It’s a thought which originated by combining the concave and convex models together.

Kent Beck's 3X Explore Expand Extract

Explore

Beginning with Exploration Mode: he tells us a story about co-creating the hugely popular Java test library Junit. It wasn’t about risk analysis or planning loads in advance. It was just about using the combined experience of two guys to build something useful. All of a sudden, unpredictably, a lot of people wanted their hands on Junit. The key here was the unpredictability. There’s just no way they could have planned that it would be as successful as it was. At that point, it’s growing. “The Exploration phase is over, you won!” Time to move to Expand mode.

Expand

Expand Mode: This is where we need to have focus and discipline. At Facebook, their engineers talk about going into lockdown and finishing something, pushing all meetings and other tasks aside. It’s where he says you’ll find it’s ‘one damn thing after another’ in this mode. Many more hours of working overtime actually do payoff in this mode, but ultimately it’s not sustainable.

Extract

Finally, from Expand Mode we move to Extract Mode. Here, we’re thinking long-term and we’re thinking about scale. Here, functional reporting becomes useful. This is where management decisions are critical: do you pull engineers away from other tasks and get them to focus on this, do you want to ‘feed’ the Extract Mode or not? Don’t pull people away in this critical time. It’s all about engineer incentives and management decisions.

Kent discusses that in each of these phases, our business and product functions all need to be different, although it is definitely possible to do them all at the same time. This is where innovation comes from. You can, like at Facebook, successfully run Expand projects and Extract projects at the same time, for example.

The ‘tug of war’

Now that we’re comfortable with these ‘three modes’ as a concept, this is where we go deep. He talks about adding two conceptual loops between this curve, one on the left and one on the right, both spinning in different directions. There is a ‘tug of war’ between these two loops. This is really just a way of visualising the following sentence: “The bigger you get, easier it is to get bigger”.

“Behaviours change when you have a lot to lose.” - Kent Beck

“The mode you’re in should drive your behaviour. If you’ve got nothing to lose, then don’t estimate. If you’re in Extract and you need funding, you need an estimate. If I have nothing to lose and the experiment will run super quick, don’t write tests! It’s going to be deleted anyway. It will allow you to explore faster. If the code will live for many years, of course, write tests!”

What’s wrong with Facebook engineering?

An era of experimentation is dying because software engineer incentives are wrong. Some things can’t be measured with numbers. At Facebook, he mentions that engineers are rewarded if the work that they were particularly doing led to an increase in users on the platform. It’s why, he jokes, that we see more product features released by Facebook every December and June when everybody is going through their company performance review.

Kent worries for the future. He says that it’s a really good thing for an engineer to say, ‘Hey, I worked on this feature for a few months, we realised it didn’t work, so we’re scrapping it’. And that’s not happening at Facebook from what he saw. Engineers are in fear of ‘inadequate’ experiments because they might not lead to more users on Facebook. I feel depressed about capitalism at this point.

Watch the talk: You can view the entire 3X talk by Kent Beck on YouTube now. (Note: this is not the exact talk I attended; this talk was recorded at Brisbane YOW! conference and seems to be slightly different from the Melbourne YOW! conference talk I attended).

“It’s really easy to exceed expectations when people think you’re an idiot.” - Kent Beck