Chris James - Software Engineer and other things

The ghost of Henry Ford is ruining your development team

31 October 2019

People talk a lot about "lean". Maybe they've read The Machine That Changed The World and were inspired by how lean manufacturing transformed the motor industry. They read how Japanese companies became incredibly efficient and blew the competition out of the water and then they take that to their software development team.

You keep saying lean, i do not think it means what you think it means

This sounds good but too often I feel like people miss the real genius of lean manufacturing and continue to run their organisation with a top-heavy, command and control structure; the only difference is a constant message of

Be more efficient! Stop being wasteful! LEEEEEAN

Agile, Devops and other buzzwords clearly get a lot of their inspiration from lean manufacturing and these ways of working gain their efficiencies (and other, bigger benefits) broadly by empowering development teams and giving them responsibility.

A primer on mass production

Before Henry Ford’s assembly line, cars were made to order by craftsmen. It took a long time to make them and were reserved for the wealthy.

The assembly line made it possible to make cars cheaply which made them accessible to a far broader market.

How?

It’s a big subject, but here are some pithy bullet points

Problems

The assembly line was very intolerant to disruption

The work was very dispiriting. Ford even passed on some of the savings and profits to his workers by heavily increasing their salaries but people just didn’t want to do the same boring task 100s of times a day.

The assembly line being heavily optimised for volume with expensive, specific machines meant it is very inflexible so creating different cars was a very difficult and costly endeavour.

When designing new cars it took what we would recognise as a very waterfall approach, of a project being passed around many departments and problems being unsurfaced way too late due to poor feedback loops and not a good structure of ownership.

Lean manufacturing

Japanese car makers studied the production line and tried a different way. When you hear about kanban and kaizen in agile circles, well it all came from here.

From Wikipedia

For many, lean is the set of "tools" that assist in the identification and steady elimination of waste. As waste is eliminated quality improves while production time and cost are reduced.

There isn't a canonical "lean" way of working, but the one that resonates strongest with me is "The Toyota Way"

The Toyota Way, in which the focus is upon improving the "flow" or smoothness of work, thereby steadily eliminating mura ("unevenness") through the system and not upon 'waste reduction' per se... The implementation of smooth flow exposes quality problems that already existed, and thus waste reduction naturally happens as a consequence. The advantage claimed for this approach is that it naturally takes a system-wide perspective, whereas a waste focus sometimes wrongly assumes this perspective

How does lean try to reduce waste?

Another big subject and there are good resources on it with a google search. The main drive of it from my perspective is actively trying to identify waste and then fix it. Lean describes 3 types

Lean emphasises the idea of "flow", optimising the processes for delivery and reducing interruptions in the production process. (think about trunk based development vs pull requests here)

Quality

For quality you adopt Kaizen, striving for perfection and hunting down the root causes of issues and fixing them. You adopt a culture of quality across the value stream, not after the product is "finished".

Command and control vs autonomy and empowerment

What I mainly took from the book "The Machine That Changed The World" was how it contrasted mass manufacturing and lean manufacturing in respect to management.

Lean manufacturing acknowledges that it's impossible to heavily micro-manage the production of a complicated product like a car with thousands of people working on it.

The best way for Kaizen to work is to coach and empower the do-ers to find the best ways to deliver.

Signs you're not in a lean company

People with job titles of "manager" or "leader" but are nothing more than coordinators

When designing new cars there is the idea of the Shusa (not sushi!) who in short was actually in charge of things. They could bring people to help and make decisions and were not worried about being overruled by the whims of upper management; they were empowered to make the project work. People would say that these people actually made their mark on a project "oh you can tell that's a Dan car because of X Y and Z".

Contrast that to places I've worked where people have had very lofty job titles but no actual empowerment and were nothing more than a communication funnel between the people who actually make things and the people who could actually make decisions.

How common is it in our industry for tech leads to be miserable simply because they're promoted into what they'll actually call a "shit shield". Seriously, can we stop saying that as if it's a good thing. It's a problem.

A leader should be able to lead

The development team has no say on priorities

Lean famously relies on Kaizen; the concept of continually striving for perfection. In lean plants workers can stop the plant if they find a problem and they work hard to make sure it doesn't happen again.

If you see something that is technically really wrong and is slowing your team down, how empowered are you to fix it? Do you work with a scrum master who will humour you but probably thinks "technical stories" are "waste" ?

Being lean and efficient is not simply done by only working on the most important feature at a given time. It's about the builders working hard to create a system and environment that can be worked on efficiently. That takes the form of the usual best practices; tightening and improving feedback loops, continuous delivery, easy to setup, simple delivery pipeline, excellent code, etc.

The lean plants worked hard to reduce problems and defects and to give themselves flexibility. This only happened because the workers were given the autonomy to make it happen; not micromanagement from above.

How does your team compare? Are you able to fix problems when you see them (which is the best time) or do you have to wait for the "technical debt sprint"?

All teams have a standard, immutable way of working

Toyota didn't just make up lean and be successful, they had to practice and iterate at it to figure out how to be good. Your IT teams need to practice and work at being "lean"

You also have to understand that we dont have teams of resources, they are individuals working on different problems. This means if you want to get the most out of people you need to give them autonomy to figure out how they can work together most effectively.

How does a team get better at working together if they cant gather feedback by retrospecting and then improving the way they work?

Larger organisations often talk about standardisation to allow developers to move between teams. That in itself isn't a bad thing but it's not important as autonomy.

If you want teams to be empowered enough to practice Kaizen and let them find how to work efficiently you have to accept that teams will work differently.

This is fine and in my experience moving between teams is always a challenge even if everyone works the same way, because teams are more complicated than a set of tools and processes.

The people who write the software are not responsible for QA or running it in production

In mass manufacturing the assembly line optimised itself for volume and would defer any kind of quality problems to the end of the line where another team would pick things up and fix them. The problem of this is the feedback loop just didn't exist and systemic problems would never get addressed, which brings waste.

I consider a QA column in a kanban wall as a smell. QA in lean worlds is a part of the entire development process, not something tacked on at the end. QA is a responsibility and an inherent quality of a good team.

From the continuous delivery website

It’s much cheaper to fix problems and defects if we find them immediately—ideally before they are ever checked into version control, by running automated tests locally. Finding defects downstream through inspection (such as manual testing) is time-consuming, requiring significant triage. Then we must fix the defect, trying to recall what we were thinking when we introduced the problem days or perhaps even weeks ago.

If you're throwing your software over a wall to an ops team then how do you expect to write a good system?

Boredom

If you're trying to turn your development team into a homogeneous assembly lines pumping out JIRA tickets at volume you might start to bore some people (and probably build some crappy products).

You probably pay your engineers a lot of money because they're presumably quite clever, so why are you denying them the space to think and take responsibility?

I've worked on projects which should be to be honest quite boring but turned out to be great fun because I worked in an engaged, empowered and fun team who took pride in doing things the "right" way and practiced Kaizen.

It was nice for the team and we delivered the project which was initially seen as very risky and tricky with very little stress. Not only that but what we learned in respect to shipping software was adopted by other teams in the organisation.

Kaizen is fun, rewarding and will improve your outcomes.

 Too scared to release and get feedback

In the book Lean Thinking: Banish Waste and Create Wealth in Your Corporation it says

Identify value from the customer's perspective

Doing this with software products is notoriously difficult but it's made even more difficult if you dont have the courage to release your software to the world and gather feedback.

Too often software is managed as a project which "ends" (this is nonsense, obviously) and no one takes any lessons from the customers. This is the ultimate form of waste that lean strives to fight against.

The transformational lean technologies and methodologies

I think it’s interesting to think about the transformative technologies over the past few years and how they relate to lean. I think a lot of things appear big and transformative but are they solving real problems? Are they helping you be lean?

Docker

Docker is an empowering technology allowing developers to use technology far more freely. Rather than asking for permission to use a technology and have it installed in production etc we can use whatever we like because we just ship containers

The main benefit for me though is it has made the difference between a local computer and production much smaller, which allows developers to take more responsibility for the quality of their software.

Synonymous with moving away from highly specialised, inflexible tooling in the mass manufacturing world

Devops

Devops is lean. It is about tightening feedback loops, improving the quality of our tooling and processes and taking responsibility of our software. Read The Phoenix Project. Just do it.

Cloud computing (e.g AWS)

Again this is another empowering development which lets the shop floor workers easily adapt the technology landscape to solve problems. Synonymous with having more flexible machinery in the lean car factories as opposed to highly specialised ones from mass manufacturing.

Wrapping up - we're not building cars

Agile and DevOps have clearly taken a lot of lessons from the lean manufacturing of cars. All of them are about building complicated things with lots of people which requires a different way of thinking vs mass manufacturing if you wish to do it efficiently with a happy workforce.

But when you're reading about lean it's important to remember that we software developers have different goals

A big thing about lean is trying to identify value from a customer's perspective and eliminating anything that doesn't contribute to it. People take this lesson in software to aggressively deprioritise anything that isn't obviously a feature.

The thing is, this is just a very immature way of looking at what you're building. Not only are you building a product but you're building a system that is worked on by your developers. It's akin to not allowing your factory workers to improve the assembly line.

The factory is as important as the cars and the system is as important as the software product

The "real" customer does benefit from:

I think examining the "value stream" for waste is a great thing, there's nothing more dispiriting than working on something for months with no real evidence it'll bring any benefit to anyone.

Too often though this is led by "product" people with little collaboration from the technical side. The team is unable to practice Kaizen and instead wastes its time working around technical debt, slow builds, poor feedback loops, manual testing etc. This results in a product which eventually becomes too hard to change and cant react to what the user wants in the long-run. A far cry from the flexible, malleable and efficient development system we're all supposedly striving for.

This comes back to what I wrote earlier, is your tech lead empowered to lead? Or is she just making sure you put estimates on JIRA tickets? Is she treated as an equal when deciding priorities for work? Does she have to argue constantly for scraps of time for the team to practice Kaizen whilst passive-aggressively being accused of being wasteful for even suggesting non-feature work?

Final final thought

From Wikipedia again

The cultural and managerial aspects of lean are arguably more important than the actual tools or methodologies of production itself. There are many examples of lean tool implementation without sustained benefit, and these are often blamed on weak understanding of lean throughout the whole organization.

Replace the word lean with "agile" and I think you'll have a lot of software developers nodding with knowing looks on their faces.