The ghost of Henry Ford is ruining your development team

People talk a lot about "lean". Maybe they've read The Machine That Changed The World and were inspired by how lean manufacturing transformed the motor industry. They read how Japanese companies became incredibly efficient and blew the competition out of the water, and then they took that to their software development team.

This sounds good, but too often, I feel like people miss the real genius of lean manufacturing and continue to run their organisation with a top-heavy command and control structure; the only difference is a constant message of

Be more efficient! Stop being wasteful! LEEEEEAN

Agile, DevOps and other buzzwords get a lot of their inspiration from lean manufacturing, and these ways of working gain their efficiencies (and other, more significant benefits) broadly by empowering development teams and giving them responsibility.

A primer on mass production

Before Henry Ford’s assembly line, cars were made to order by craftsmen. It took a long time to make them, and they were reserved for the wealthy.

The assembly line made it possible to make cars cheaply, which made them accessible to a far broader market.

How?

It’s a big subject, but here are some pithy bullet points:

• To reduce costs, he needed to produce cars at high volume with low-skilled workers, not craftsmen.
• The assembly line was a collection of expensive, inflexible machinery that scaled to volume and could be operated by cheap labour doing very simple tasks.
• Emphasis on standardisation, being able to replace one worker with another cheaply and easily.

Problems

The assembly line was very intolerant to disruption, small problems could cause big problems and stop the line. To mitigate this, they:

• Had to add lots of buffers, such as storing supplies, space and additional workers to smooth production
• preferred to accept defects and fix them after the car was made, rather than to stop the line

The work was very dispiriting. Ford even passed on some of the savings and profits to his workers by heavily increasing their salaries, but people just didn’t want to do the same tedious task 100s of times a day.

The assembly line is heavily optimised for volume with expensive, specific machines meant it was very inflexible, so creating different cars was a very difficult and costly endeavour.

When designing new cars, it took what we would recognise as a very waterfall approach, of a project being passed around many departments and problems being unsurfaced way too late due to poor feedback loops, and not a good structure of ownership.

Lean manufacturing

Japanese carmakers studied the production line and tried a different way. When you hear about Kanban and Kaizen in agile circles, it all came from here.

From Wikipedia

For many, lean is the set of "tools" that assist in the identification and steady elimination of waste. As waste is eliminated, quality improves while production time and cost are reduced.

There isn't a canonical "lean" way of working, but the one that resonates strongest with me is "The Toyota Way."

The Toyota Way, in which the focus is upon improving the "flow" or smoothness of work, thereby steadily eliminating mura ("unevenness") through the system and not upon 'waste reduction' per se... The implementation of smooth flow exposes quality problems that already existed, and thus waste reduction naturally happens as a consequence. The advantage claimed for this approach is that it naturally takes a system-wide perspective, whereas a waste focus sometimes wrongly assumes this perspective

How does Lean try to reduce waste?

Another big subject and there are good resources on it with a Google search. The main drive of it from my perspective, is actively trying to identify waste and then fix it. Lean describes 3 types

• Mura: Waste due to changes in demand. Lean advocates a "pull" system rather than "push". So you only make a car if there is demand for it rather than trying to forecast demand which is often difficult and makes supply inconsistent. In software, this would be akin to perhaps making a cheap prototype to prove an idea before committing to a bigger endeavour.
• Muri: Waste by trying to do too much stuff at once. This is why if you're doing Kanban and not observing work-in-progress limits, you're not doing Kanban. It can sometimes feel counter-intuitive that you can go faster by doing less work, but you've probably worked in a situation where a team is working really hard, and everything feels chaotic and unreliable. This is also why smaller teams are generally preferable.
• Muda: Non-value-adding work. A classic example in software would be developers constantly struggling with flaky builds and not addressing the underlying causes.

Lean emphasises the idea of "flow", optimising the processes for delivery and reducing interruptions in the production process. (think about trunk-based development vs pull requests here)

Quality

For quality, you adopt Kaizen, striving for perfection and hunting down the root causes of issues and fixing them. You embrace a culture of quality across the value stream, not after the product is "finished".

Command and control vs autonomy and empowerment

What I mainly took from the book "The Machine That Changed The World" was how it contrasted mass manufacturing and lean manufacturing with respect to management.

Lean manufacturing acknowledges that it's impossible to heavily micro-manage the production of a complicated product like a car with thousands of people working on it.

The best way for Kaizen to work is to coach and empower the do-ers to find the best ways to deliver.

Signs you're not in a lean company

People with job titles of "manager" or "leader" but are nothing more than coordinators

When designing new cars, there is the idea of the Shusa (not sushi!), who in short, was actually in charge of things. They could bring people to help and make decisions and were not worried about being overruled by the whims of upper management; they were empowered to make the project work. People would say that these people actually made their mark on a project "Oh, you can tell that's a Dan car because of X, Y and Z".

Contrast that to places I've worked where people have had very lofty job titles but no actual empowerment and were nothing more than a communication funnel between the people who actually make things and the people who could actually make decisions.

How common is it in our industry for tech leads to be miserable simply because they're promoted into what they'll actually call a "shit shield". Seriously, can we stop saying that as if it's a good thing? It's a problem.

A leader should be able to lead

The development team has no say in priorities

Lean famously relies on Kaizen, the concept of continually striving for perfection. In lean plants, workers can stop the plant if they find a problem, and they work hard to make sure it doesn't happen again.

If you see something technically wrong and slowing your team down, how empowered are you to fix it? Do you work with a scrum master who will humour you but probably thinks "technical stories" are a "waste"?

Being lean and efficient is not simply done by only working on the most important feature at a given time. It's about the builders working hard to create a system and environment that can be worked on efficiently. That takes the form of the usual best practices; tightening and improving feedback loops, continuous delivery, easy to set up, simple delivery pipeline, excellent code, etc.

The lean plants worked hard to reduce problems and defects and to give themselves flexibility. This only happened because the workers were given the autonomy to make it happen, not micromanagement from above.

How does your team compare? Can you fix problems when you see them (which is the best time), or do you have to wait for the "technical debt sprint"?

All teams have a standard, immutable way of working

Toyota didn't just make up lean and be successful; they had to practice and iterate at it to figure out how to be good. Your IT teams need to practice and work at being lean.

You also have to understand that we don't have teams of resources; they are individuals working on different problems. If you want to get the most out of people, you need to give them autonomy to figure out how they can work together most effectively.

How does a team get better at working together if they can't gather feedback by retrospecting and improving how they work?

Larger organisations often talk about standardisation to allow developers to move between teams. That in itself isn't a bad thing, but it's not essential as autonomy.

If you want teams to be empowered enough to practice Kaizen and let them find how to work efficiently, you have to accept that teams will work differently.

This is fine, and in my experience, moving between teams is always a challenge, even if everyone works the same way because teams are more complicated than a set of tools and processes.

The people who write the software are not responsible for QA or running it in production

In mass manufacturing, the assembly line optimised itself for volume and would defer any quality problems to the end of the line, where another team would pick things up and fix them. The issue is that the feedback loop didn't exist, and systemic problems would never get addressed, which brings waste.

I consider a QA column in a Kanban wall as a smell. QA in lean worlds is a part of the entire development process, not something tacked on at the end. QA is a responsibility and an inherent quality of a good team.

From the continuous delivery website

It’s much cheaper to fix problems and defects if we find them immediately—ideally before they are ever checked into version control, by running automated tests locally. Finding defects downstream through inspection (such as manual testing) is time-consuming, requiring significant triage. Then we must fix the defect, trying to recall what we were thinking when we introduced the problem days or perhaps even weeks ago.

If you're throwing your software over a wall to an ops team, how do you expect to write a sound system?

Boredom

If you're trying to turn your development team into a homogeneous assembly line pumping out JIRA tickets at volume you might start to bore some people (and probably build some crappy products).

You probably pay your engineers a lot of money because they're presumably quite clever, so why are you denying them the space to think and take responsibility?

I've worked on projects which should be, to be honest, quite boring but turned out to be great fun because I worked in an engaged, empowered and fun team who took pride in doing things the "right" way and practised Kaizen.

It was excellent for the team, and we delivered the project, which was initially seen as very risky and tricky, with very little stress. Not only that but what we learned with respect to shipping software was adopted by other teams in the organisation.

Kaizen is fun, rewarding and will improve your outcomes.

Too scared to release and get feedback

In the book Lean Thinking: Banish Waste and Create Wealth in Your Corporation, it says

Identify value from the customer's perspective

Doing this with software products is notoriously difficult, but it's made even more difficult if you don't have the courage to release your software to the world and gather feedback.

Too often, the software is managed as a project which "ends" (this is nonsense, obviously), and no one takes any lessons from the customers. This is the ultimate form of waste that Lean strives to fight against.

The transformational lean technologies and methodologies

I think it’s interesting to think about the transformative technologies over the past few years and how they relate to lean. I think a lot of things appear big and transformative but are they solving real problems? Are they helping you be lean?

Docker

Docker is an empowering technology allowing developers to use technology far more freely. Rather than asking for permission to use technology and have it installed in production etc., we can use whatever we like because we just ship containers.

The main benefit for me, though, is it has made the difference between a local computer and production much smaller, which allows developers to take more responsibility for the quality of their software.

Synonymous with moving away from highly specialised, inflexible tooling in the mass manufacturing world

DevOps

DevOps is lean. It is about tightening feedback loops, improving the quality of our tooling and processes and taking responsibility for our software. Read The Phoenix Project. Just do it.

Cloud computing (e.g. AWS)

Again this is another empowering development which lets the shop floor workers quickly adapt their technology landscape to solve problems. Synonymous with having more flexible machinery in the lean car factories as opposed to highly specialised ones from mass manufacturing.

Wrapping up - we're not building cars.

Agile and DevOps have taken a lot of lessons from the lean manufacturing of cars. All of them are about building complicated things with lots of people, which requires a different way of thinking vs mass manufacturing if you wish to do it efficiently with a happy workforce.

But when you're reading about lean, it's important to remember that we software developers have different goals.

• We're not trying to build software at volume.
• We're trying to deliver the promise of software, products that are malleable and can change to users' needs quickly and cheaply.

A big thing about lean is identifying value from a customer's perspective and eliminating anything that doesn't contribute to it. People take this lesson in software too aggressively to deprioritise anything that isn't a feature.

The thing is, this is just a very immature way of looking at what you're building. Not only are you making a product, but you're building a system that your developers work on. It's akin to not allowing your factory workers to improve the assembly line.

The factory is as important as the cars, and the system is as important as the software product.

The "real" customer does benefit from the following:

• Making it more operable so it has less downtime.
• Improving the test suite so the software can be changed easily to meet customer needs quickly
• etc. etc

Examining the "value stream" for waste is excellent; nothing more dispiriting than working on something for months with no objective evidence it'll benefit anyone.

Too often, though, this is led by "product" people with little collaboration from the technical side. The team cannot practice Kaizen and wastes its time working around technical debt, slow builds, poor feedback loops, manual testing etc. This results in a product becoming too hard to change and can't react to what the user wants in the long run. A far cry from the flexible, malleable and efficient development system we're all supposedly striving for.

This comes back to what I wrote earlier, is your tech lead empowered to lead? Or is she just making sure you put estimates on JIRA tickets? Is she treated as an equal when deciding priorities for work? Does she have to argue constantly for scraps of time for the team to practice Kaizen whilst passive-aggressively being accused of being wasteful for even suggesting non-feature work?

Final final thought

From Wikipedia again

The cultural and managerial aspects of lean are arguably more important than the actual tools or methodologies of production itself. There are many examples of lean tool implementation without sustained benefit, and these are often blamed on weak understanding of lean throughout the whole organization.

Replace the word lean with "agile", and I think you'll have a lot of software developers nodding with knowing looks on their faces.