Chris James - Software developer

The weight of AI on engineers

Chris James — Fri, 06 Mar 2026 00:00:00 +0000

Accountability has a weight limit

If the AI deletes all your business-critical data, you can't fire the AI. If an AI controlled missile launcher accidentally shoots an ally, you can't send the AI to prison. AI can act, but only humans can be accountable.

As a rhetorical device for this post, let's assume that a single engineer in the old days could complete one story a day, but now they can, and are expected to do 10; because AI makes writing code and building systems 10x faster, right?

Of course, measuring productivity by lines of code per day is stupid and flawed, and user stories have a similar problem. A user story should be valuable, but it might not be. Sometimes our knowledge can be imperfect, and not quite what the user wants, or they could be catastrophically wrong. (See: the $327m Mars Climate Orbiter, lost because one team used metric units and another used imperial.)

Is it realistic though for an engineer to be accountable for 5-10x the amount of change they were dealing with before? How can they possibly understand 10 user stories, and the code that was written to fulfil the requirements. Does that sound realistic?

The AI maximalists argue you no longer need to understand the code, just focus on the outcomes and let the AI deal with that weird code stuff. But nonetheless, whoever is accountable has to understand how the system will behave; and that's not abstract: would you really be comfortable taking accountability for something like the Therac-25 (a radiation therapy machine whose software bugs delivered fatal overdoses to patients) if it was generated entirely from a prompt, without understanding medicine?

We've been solving the wrong problem

The code was never the bottleneck. Not a new take, I appreciate. Pedro Tavares put it well recently:

The marginal cost of adding new software is approaching zero, especially with LLMs. But what is the price of understanding, testing, and trusting that code? Higher than ever.

Now people believe reviewing is the bottleneck. What can we do? Every day stories of teams with 10s, 100s of pull requests stuck in review.

In truth, reviewing was a bottleneck for teams before the AI boom, they just didn't realise it. Many companies that adopt a pull-request model with individual contributors created a bottleneck, but it was one of their own doing.

To deal with this, techniques from extreme programming (XP) are having somewhat of a renaissance, even if people don't want to admit it. XP Explained was published in 1999.

Hey everyone! I've invented spec-driven development, totally a new thing, buy my book!

— AI influencer, 2026

These techniques will help, breaking problems down into smaller pieces, encoding small behaviour changes in automated tests reduces risk and cognitive load, but it still feels like it's attacking the wrong problem, it's treating a symptom, not the cause. Again, can a human realistically, deeply understand 10, albeit smaller problems every day?

Presence in the problem

My teams do pair programming. Where review is an ongoing part of the process. Two of you are in the context together. You might do an informal review together before committing, but it is nothing like the pull request model where you must read someone else's git diffs, out of context, when you have a spare 30 minutes. So, for my teams reviewing wasn't a bottleneck.

Two humans pairing with an AI and watching what it does, does not feel like pair programming, I think we might look back on this as a daft moment. Unthinkingly using the same tool or technique in a different context is often a bad decision. With pairing, it's like you've introduced the worst part of pair programming; the demanding, present, high-energy way of work, with the boredom and difficulty of looking at git diffs written by someone else.

When you're pairing, or if you're writing code by yourself the engineer's presence is in the problem, not just the solution. The code is a manifestation of your business domain, it represents your business, and how it should behave. When you're writing code, you're trying to add capability to the business, or change how it works. Doing this, helps you understand it, but now with AI writing the code, the human is taken out of this process.

There's a pattern every engineer recognises. A stakeholder comes in confident: "it's simple, I know exactly how it should work." And on the surface, it is simple. But the moment you start building, the gaps appear. What happens when the user has two accounts? What does "current" mean when the time zone changes? These aren't edge cases, they're the domain, and they only reveal themselves under the pressure of implementation. This is the engineering. Not the code, the thinking. The code is just what's left behind when the thinking is done. Coding agents don't help here, if anything they let you paper over the gaps faster.

Review is still necessary. The question is what good review looks like when the code wasn't written by a human. Reading git diffs was always a blunt instrument, but we tolerated it because the human who wrote the code had already done the hard cognitive work; the diff was just a record of their thinking. When an AI writes the code, that thinking never happened on the human side. The diff is all there is, and it tells you surprisingly little about whether the system now behaves correctly.

diff + const r = await db.query(`SELECT * FROM orders o LEFT JOIN users u ON + u.id = o.user_id LEFT JOIN products p ON p.id = o.product_id WHERE + o.status NOT IN ('cancelled','refunded') AND (u.flags & 0x4) = 0 + AND p.region = $1 GROUP BY o.id, u.id HAVING sum(o.qty * p.price) + > $2 ORDER BY o.created_at DESC`, [ctx.region, threshold]) + if (r.rows.length) await notify(r.rows.map(x => transform(x, opts)))

LGTM

Engineers don't think in text files. We build mental models of flows, of state, of how components talk to each other. Maybe the right review primitive isn't a diff at all. Sequence diagrams, visual system maps, behaviour traces -- tools that show you what the system does rather than what the code says. This feels like an open problem the industry hasn't seriously tackled yet, and I suspect whoever cracks it will matter more to the future of AI-assisted engineering than whatever coding agent is flavour of the month.

Left of dev

So, if the AI has freed up time, what's it freed for? I think, given the cognitive burden, perhaps churning out more stories per day is not the right approach. How about the engineers focus on making stories valuable in the first place. The stuff that happens "left of dev". Sitting with users and watching them use your product, taking part in research, spending more time with stakeholders and so on.

Perhaps the real unlock of AI isn't this "10x" thing, it's that engineers finally have time to deeply understand what they're building, and why. That may result in some real productivity gains, more valuable stories gained from deeper insights; and it helps the people working on these systems feel more comfortable with the accountability on their shoulders.

Some nice feedback on LGWT

Chris James — Sat, 21 Sep 2024 00:00:00 +0000

William Martin forwarded me some written feedback on Learn Go With Tests from Kynan Ware which made my day.

I get a lot of friendly feedback from people on the work, but this is one of those where the author has taken the time to explain why they enjoyed the book more specifically.

Before reading Learn Go With Tests, I read a Go different book that only had a chapter on the standard library’s testing suite. To say Learn Go With Tests expanded on that chapter would be an understatement. Getting hands on with the standard library and how it works was great and informative. Keep in mind that I do not have really any background in testing anyways, so I needed to be told both what features and patterns exist in a general sense and how they are used; Learn Go With Tests did just that and more.

I enjoyed seeing the TDD feedback loop in a hands-on way with concrete examples. The author took their time to explain why TDD is valuable in a “show, don’t tell” way. Initially, the TDD loop seemed tedious and monotonous, but as I progressed through the book past the simple examples and into more complex territory, I found it to be actually fun, interesting, and very productive to get instant feedback on your work.

(The emphasis is mine). I have spent a lot of time directly and indirectly (e.g the book) coaching TDD and this mental leap is often the hardest one. Many people find it boring, many find it "unrealistic" and bemoan how TDD tutorials are often too simplistic, and don't apply in the real world.

TDD does work in the real world, but it takes time and practice. In my mind, one goal of TDD is breaking down work well enough so your real work, feels as simple as the examples you learned it with.

This was new to me (surprise!), but I like the idea that TDD is a design tool very much. I’m not going to repeat or summarize the book’s details on why or how it is used as a design tool, but I’ll just say that I ingested this idea and vibe with it very much.

It's gratifying this message is coming through to people who read the book, because this is often such a missed point of TDD. Sadly, many believe TDD is merely about achieving test coverage, but in The Why of TDD I write at length how it is so much bigger than that.

The author was also particularly good at answering questions that might come up as they come up. The moment I had a question in my mind, the next sentence addressed it. I thought it was beautifully written. I didn’t have a place for this point in another heading, but I also wanted to mention how much I enjoyed just the overall introduction to testing patterns in general.

Learn Go with Tests reaches 20k stars

Chris James — Fri, 18 Aug 2023 00:00:00 +0000

I made The first commit for Learn Go with Tests (LGWT) on the 2nd of March 2018. I was hungover and frustrated at the lack of high-quality test-driven development articles I had seen in the Go community, so I decided to give it a Go.

Whilst I understand that GitHub stars don't mean anything, they are still an indicator that the project is well received. I'm not going to engage in false modesty; I'm proud of this project; it's probably one of the best things I've accomplished in my career.

However, it would be wrong not to mention that it has been a community effort driven by 212 other contributors. I truly appreciate every contribution you have made.

With each star, fork, and contribution, the project's reach has expanded, reaching corners of the world I could have imagined. Between 2018 and now, there have been 1299 more commits and translated into five other languages.

The repository's impact has even been recognised beyond the GitHub platform, with mentions in articles, forums, books and tech conferences.

The rest of this post reflects on the experience, lessons learned, and plans.

Be nice

I never intended to write much more beyond the hello world chapter, but when my friend messaged me saying I was top of the orange site, I couldn't quite believe it. The comments were positive, and my GitHub feed was alive for the first time. This drove me to keep writing more; without the positive feedback, it's unlikely LGWT would exist at all.

The feedback confirmed for me that the idea was a good one. I could use Go, a simple, easy-to-understand language and ecosystem, as a vehicle for teaching TDD properly.

The takeaway here is, be nice! If you use something someone provides or read something interesting, tell them. It will mean a lot and may act as a motivator for them to keep them driving forward.

Learning by writing learning go with tests

I wouldn't describe myself as a Go expert when I started the project. I only had a few years of experience with Go, although I did have around ten years of software development experience.

Whether you're writing open-source code or blogging, if it gets attention, you will have people giving you feedback on your work. Sometimes it is done in a constructive, friendly manner, and some people are just angry someone said something wrong on the internet. Sadly, you have to build a bit of resiliency for this, but I definitely learned a lot from the community and hopefully taught a lot back to it.

I've always relied on writing to organise my thoughts to communicate better, and the five years of LGWT have helped me practice this critical skill. Having to write these posts has helped my TDD coaching skills tremendously. I am a much better teacher of TDD as a result.

Work in progress

Delightfully, the Go language is very conservative concerning language design. This means the content I wrote five years ago is still broadly correct. Some tooling changes have happened (I try to resist adding stuff about that anyway), but the language is highly stable. As new features land or exciting additions to the standard library are made, I will always try to bring a chapter in to cover it.

That said, I would hope by the time readers have made it to the end of the book, they have learned how to fish. That's why you'll find the most recent chapters of LGWT, are not specific to Go, but more around software design and approaches.

To that end, I am working on a chapter called "Living without mocks", which goes into contract testing, testing databases and some commentary on designing your software for testability, especially with respect to fast and accurate feedback loops. No promises when it'll be done, as I do have bills to pay and other hobbies to enjoy.

Learn `X` with tests

Many have requested that someone make equivalent resources for Rust, JavaScript and others. As I mentioned earlier, if I have done a good job with LGWT, you can read that and apply the same method for learning other things; you don't need someone to write something for you.

You can find me using tests to try and get a handle on a few programming languages to varying levels of expertise at:

They don't feature any commentary, just the code.

I would love to write a "proper" book in the LGWT format sometime in the future. On my list of "maybe, one day, if I win the lottery and don't have to have a full-time job"

Learn Rust with Tests
Learn Ocaml with Tests
Learn Svelte with Tests

It's not a proper book.

I do feel sad that I haven't gotten a physical book edited by a real publisher due to all this effort. Mainly to give to my parents as I think they would understand the accomplishment better!

I have had a couple of publishers approach me, but none of them has been able to work with the constraint of keeping material open-source, which is understandable as they need to make money. Realistically, the only way to accomplish this dream is to find the time to do it again with a different programming language.

Comments

In one final ego boost, I did some internet searches to try and dig up some pleasant comments for LGWT.

One of the things that took my programming know-how to the next level was going through a Test Driven Development guide that used Golang

Learn Go with Tests is incredible

@quii

you've written one of the best programming books I've ever used. "Learn Go with Tests". Enjoying it very much. Thank you.

As a beginner, I found Learn Go with Tests extremely helpful. Thanks a lot

@quii

for putting up such great learning content! :)

Read and typed along with an early edition of this a few years back and it has continued to be one of my favorite technical tutorials of all time, far surpassing lots of other golang material that I've paid money for.

I decided to learn and use Go for my recent job hunt, and this site was just the best. I used the structure of the code in the "build an application" section in every take home coding challenge I wrote.

Learn Go with Tests is the best programming language course I've ever gone through.

https://github.com/quii/learn-go-with-tests

That's all. Just sharing that the methodology worked so well. I felt like I could legitimately jump right into a Go developer team.

....

I can't upvote this enough.

I am going through this course- and it is fabulous. Covers tests, and even though the topic names might seem easy or trivial(I mean there is only so many ways you can write loops or define arrays), they include a lot of "extras" that make it fun- for example one of the topics might include details about how to write doctests and docs, another one might introduce table driven tests and provide advice on when to use them. Overall it is great.

Practical ways to shift QA left

Chris James — Sat, 12 Aug 2023 00:00:00 +0000

“Inspection to improve quality is too late, ineffective, costly. Quality comes not from inspection, but from the improvement of the production process.”

― W. Edwards Deming, Out of the Crisis

Many prescribe shifting QA left, which sounds clever, but it's not very actionable advice, more an aspiration. This post describes concrete things quality assurance engineers can do to help increase quality and reduce waste within their teams.

The QA role in a continuous delivery and deployment world

The State of DevOps Report asserts that profitable, high-performing teams deploy software tens of times per day. The days of QAs signing off work before it is deployed are gone for elite teams.

Even with feature flags to decouple feature release and software deployment, the code is still being deployed, and it will affect the system's quality, even if it hasn't gone through the JIRA theatre of moving into the done column.

Nonetheless, quality is still essential! The report also asserts that high-performing teams have a low defect rate; throwing things into production and hoping for the best is insufficient. High-performing teams don't sink lots of time fixing bugs because the quality of their releases is excellent.

It's common for teams that are only releasing bi-weekly to have automated tests to give confidence that the system does what it's supposed to, which is good. However, as Deming said, this is still inspection after the work is done.

It's an improvement; at least it's automated, but these tests are rarely run in prod and say little about the system's quality, just that it behaves in specific ways in certain conditions in a non-production environment.

We only really appreciate the quality of a system, when it is in production and used by real people. Beyond trivial systems, what you'll discover can be very unpredictable and interesting. This is when you really start to understand the quality of your system and what needs improving.

The QA role must move away from inspection of features, which can be largely automated, and instead focus more on understanding what the system is doing in production.

Why is quality important?

Sounds obvious, but often teams (or management) make decisions to trade off quality for speed. This is a false economy. Speed and quality are correlated.

Quality is what helps teams continue to deliver, especially over time.

The Accelerate authors have data that shows significant correlations with much more important things.

For example, organizations made up of high-performing teams, based on this model, make more money than orgs that don’t. Here is data that says that there is a correlation between a development approach and the commercial outcome for the company that practices it.

It also goes on to dispel a commonly held belief that “you can have either speed or quality but not both.” This is simply not true. Speed and quality are clearly correlated in the data from this research.”

~~ Dave Farley, Modern Software Engineering

It is too familiar a story for teams to struggle with bugs, support requests and other issues around quality, reducing their capacity to evolve their system further. Many teams will be in this situation, frustrated that they don't have enough time to further iterate on their system to respond to customer demands.

As with all things around agile, DevOps, etc., they all have their roots in lean manufacturing. Lean prescribes a zero-defect culture and reduces waste by increasing quality. By reducing waste, we increase our capacity to do valuable work for the customer. This allowed Toyota et al. to outcompete their American competitors (who deferred quality checks to the end of the assembly line), building cars more quickly and cheaply.

The cost of fixing an issue and its impact increases the further right in the process, and the longer it is in production. Shift-left means a team will catch problems sooner, reduce waste, and increase capacity.

Teams should aim to have zero support requests landing on their plate. Too often, teams accept a normalisation of deviance and become a boiling frog, losing their ability to iterate and improve the product they work on.

A quick waste rant

As I mentioned in The Ghost of Henry Ford is ruining your development team, many managers think about lean in fairly short-sighted ways. They will focus on waste only through the lens of:

Only do work that generates "value".

This can take the form of:

Analysing stories very deeply, so no wasted effort is made on features that might not be valuable.
Deprioritising any tasks around technical debt, toil, etc

This does not work. Analysis paralysis is a form of waste in itself, and not letting teams improve the quality of their system also generates huge waste.

Moving on...

Shifting QA left, and improvement of the process.

The ultimate in shift-left are QAs influencing the formulation of user stories, where they are armed with a deep knowledge of how the system works, what problems commonly occur, its dependencies and so on.

The QAs are equal partners in prioritisation discussion, raising issues that should be addressed, affecting the system's quality and producing waste.

QAs often are too focused on the "outside" of the system (the features), but if they get involved in the innards by looking at logging, metrics, pair-programming and others, they can understand how the system works better. This will help them be more effective on the analytical side of work, allowing them to be more effective at quality recommendations such as "We should write a test for this scenario" and "What should happen if this fails?".

This helps QA distinguish their role from:

business analysts, whose job is to develop a deep understanding of the domain, but not necessarily how the system works on a technical level
product owners who understand the customers, and how the system can help them

...when analysing work right at the start of the value chain.

This will further help improve quality and increase capacity by reducing the chance of waste earlier in the pipeline.

Things to do

Metrics

Metrics give you an overall view of how your system performs in response to requests. When done well, they help you understand how the system works better and can give you warning signs of incoming problems.

This is the one gap I've noticed with many QA teams I've worked with; they are very focused on how a feature behaves but not how it performs. Slow performance is annoying for users and tends to result in more errors because of timeouts and resource contention. Beyond performance, error rates when calling APIs help you understand areas of your system where you need to add more resiliency.

Logs / Tracing

Metrics give you an overview of the system, whereas logs & tracing show specific events. Analysing them can help you identify concrete areas where the system hasn't behaved as expected, common failure patterns, etc.

The QA can then work with the developers to prioritise work to fix systemic issues that could otherwise get ignored, and end up pushing support costs.

The team should pursue a “zero-defect culture”, where the logs are silent unless there is an actionable error. Too often, systems will log errors that are common failures that cannot be acted on, and they end up creating noise, masking actual quality issues.

When you release code and it causes more errors, if you already have 100s, or thousands logged daily, you won't notice it, but your users might.

Tests, pipelines

Flaky tests are a clear quality issue. Engineers will often lazily re-run pipelines, but that is ignoring the systemic issues. Flaky tests can be signals that your system isn't as resilient as you think. QAs can help diagnose these problems to support the developers in fixing them.

QAs should be very aware of the testing strategy of the system and offer statistics and guidance when the suite is not fit for purpose. Maybe the 20-minute functional test suite had some value a few years ago, but you know it is stable, and we should remove some of them in favour of unit tests.

The speed and reliability of the pipeline are critical and are prevalent forms of waste. It also has a heavy influence on your lead time.

QAs and developers should feel ownership in making the pipeline as high quality as possible. It is akin to the factory worker in lean settings, ensuring it is as efficient as possible to reduce waste when manufacturing.

Automating other quality checks

Some tests are part of the development process, which, again, tend to show you that the feature works in given conditions, and other lower-level tests like unit tests. However, there are numerous different kinds of tests that QAs can introduce to the system to help ensure quality.

With their expert knowledge of the system's constraints, how it works, and what it needs to do, they can use their knowledge to tighten the feedback loop on the system's quality.

Other tests/checks QAs can introduce:

Performance (both backend and frontend)
Accessibility
Real user monitoring (RUM)

Trends with bugs, support

QAs should be involved with every support ticket and work with the rest of the team to determine what changes need to be made so the support never arrives on the development team's radar again.

Visibility on common support issues, bugs and so on will help you identify the quality hot spots in your system, to help you prioritise proper fixes to reduce the waste surrounding these activities.

Support tickets are a result of a quality failure. Of course, it has to be expected mistakes will be made; no system is perfect, and we'll never have perfect knowledge, but that shouldn't be used as an excuse not to strive for perfection.

Support issues often form before a line of code is written:

Insufficient
- domain knowledge
- customer knowledge
Not understanding the "real" problem to solve
Not thinking enough about non-happy path scenarios.

If everyone is on the ball, though and shifting left, they can be entirely avoided, but issues can also occur during the software being written and released.

Insufficient test automation around unexpected areas
Incorrect test strategy (e.g. loads of unit tests, no integration tests - meme)
Bad modelling allowing invalid domain states

Conclusion

This can be a touchy subject for some QAs as it can be perceived as making their role redundant. But I have seen first-hand how the good QAs who understand their role as "helping to ensure quality" and not "manual tester", can embrace this change and make their role not only more useful to the team, but also more fulfilling for themselves.

The post's title frames this as a QA issue, but quality is a team-wide issue in practice. Teams that write rubbish, ill-thought software and throw it over the wall to a QA department are doomed to create bad products with lots of waste issues.

If you are a team concerned with its capacity to do meaningful work, you need to be concerned with quality.

Poor quality introduces waste, which is a drag on your capacity. Productivity killers include bugs, support requests, noisy error logs, and slow and flaky pipelines. The further "right" these issues are, the more significant their impact and the costlier they are to fix.

Whilst I have written some specific things you can do, to truly be effective at this, you need to wear a lean hat, practice Kaizen and think about quality across the whole value chain of your work, which will always be context specific.

If something feels harder to do than it should, or you are frustrated at how you feel like you are working hard but not having the impact you'd hoped for, listen to that signal, and look for waste.

HTMX is the Future

Chris James — Fri, 05 May 2023 00:00:00 +0000

The current state of web application development

User expectations of the web are now that you have this super-smooth no-reload experience. Unfortunately, it's an expectation that is usually delivered with single-page applications (SPAs) that rely on libraries and frameworks like React and Angular, which are very specialised tools that can be complicated to work with.

A new approach is to put the ability to deliver this UX back into the hands of engineers that built websites before the SPA-craze, leveraging their existing toolsets and knowledge, and HTMX is the best example I've used so far.

The costs of SPA

SPAs have allowed engineers to create some great web applications, but they come with a cost:

Hugely increased complexity both in terms of architecture and developer experience. You have to spend considerable time learning about frameworks.
- Tooling is an ever-shifting landscape in terms of building and packaging code.
- Managing state on both the client and server
- Frameworks, on top of libraries, on top of other libraries, on top of polyfills. React even recommend using a framework on top of their tech:

React is a library. It lets you put components together, but it doesn’t prescribe how to do routing and data fetching. To build an entire app with React, we recommend a full-stack React framework.

By their nature, a fat client requires the client to execute a lot of JavaScript. If you have modern hardware, this is fine, but these applications will be unusable & slow for those on older hardware or in locations with slow and unreliable internet connections.
- It is very easy to make an SPA incorrectly, where you need to use the right approach with hooks to avoid ending up with abysmal client-side performance.
Some SPA implementations of SPA throw away progressive enhancement (a notable and noble exception is Remix). Therefore, you must have JavaScript turned on for most SPAs.
If you wish to use something other than JavaScript or TypeScript, you must traverse the treacherous road of transpilation.
It has created backend and frontend silos in many companies, carrying high coordination costs.

Before SPAs, you'd choose your preferred language and deliver HTML to a user's browser in response to HTTP requests. This is fine, but it offers little interactivity and, in some cases, could make an annoying-to-use UI, especially regarding having the page fully reload on every interaction. To get around this, you'd typically sprinkle varying amounts of JS to grease the UX wheels.

Whilst this approach can feel old-fashioned to some, this approach is what inspired the original paper of REST, especially concerning hypermedia. The hypermedia approach of building websites led to the world-wide-web being an incredible success.

Hypermedia?

The following is a response from a data API, not hypermedia.

{
  "sort": "12-34-56",
  "number": "87654321",
  "balance": "123.45"
}

To make this data useful in an SPA, the client code must understand the structure and decide what to render and what controls to make available.

REST describes the use of hypermedia. Hypermedia is where your responses are not just raw data but are instead a payload describing the media (think HTML tags like <p>, headers, etc.) and how to manipulate it (like form, input).

A server returning HTML describing a bank account, with some form of controls to work with the resource, is an example of hypermedia. The server is now responsible for deciding how to render the data (with the slight caveat of CSS) and what controls should be displayed.

<dl>
  <dt>Sort</dt><dd>12-34-56</dd>
  <dt>Number</dt><dd>87654321</dd>
  <dt>Balance</dt><dd>£123.45</dd>
</dl>
<form method="POST" action="/transfer-funds">
  <label>Amount <input type="text" /></label>
  <!-- etc -->
  <input type="submit" value="Do transfer" />
</form>

The approach means you have one universal client, the web browser; it understands how to display the hypermedia responses and lets the user work with the "controls" to do whatever they need.

Carson Gross on The Go Time podcast

...when browsers first came out, this idea of one universal network client that could talk to any application over this crazy hypermedia technology was really, really novel. And it still is.

If you told someone in 1980, “You know what - you’re gonna be using the same piece of software to access your news, your bank, your calendar, this stuff called email, and all this stuff”, they would have looked at you cross-eyed, they wouldn't know what you were talking about, unless they happened to be in one of the small research groups that was looking into this sort of stuff.

Before the World Wide Web, before web browsers, the prevailing patterns of apps were bespoke, and often "thick", clients.

Whilst ostensibly, people building SPAs talk about using "RESTful" APIs to provide data exchange to their client-side code, the approach is not RESTful in the purist sense because it does not use hypermedia.

Instead of one universal client, scores of developers create bespoke clients, which have to understand the raw data they fetch from web servers and then render controls according to the data. With this approach, the browser is more of a JavaScript, HTML and CSS runtime.

By definition, a fatter client will carry more effort and cost than a thin one. However, the "original" hypermedia approach arguably is not good enough for all of today's needs; the controls that the browser can work with and the way it requires a full page refresh to use them mean the user experience isn't good enough for many types of web-app we need to make.

HTMX and hypermedia

Unlike SPAs, HTMX doesn't throw away the architectural approach of REST; it augments the browser, improving its hypermedia capabilities and making it simpler to deliver a rich client experience without having to write much JavaScript if any at all.

You can use whatever programming language you like to deliver HTML, just like we used to. This means you can use battle-tested, mature tooling, using a "true RESTful" approach, resulting in a far more straightforward development approach with less accidental complexity.

HTMX allows you to design pages that fetch fragments of HTML from your server to update the user's page as needed without the annoying full-page load refresh.

We'll now see this in practice with the classic TODO-list application.

Clojure HTMX TODO

First-of-all, please don't get overly concerned with this being written in Clojure. I did it in Clojure for fun, but the beauty of this approach is that you can use whatever language you like, so long as it responds to HTTP requests with hypermedia.

Your browser doesn't support HTML video. Here is a link to the video instead.

Nothing special here, but it does feel like a SPA. There are no full-page reloads; it's buttery smooth, just like all the other SPA demos you would've seen.

The difference here is:

I did not write any JavaScript.
I also didn't cheat by transpiling Clojure into JavaScript. (see ClojureScript)

I made a web server that responds to HTTP requests with hypermedia.

HTMX adds the ability to define richer hypermedia by letting you annotate any HTML element to ask the browser to make HTTP requests to fetch fragments of HTML to put on the page.

The edit control

The most exciting and impressive part of this demo is the edit action. The way an input box instantly appears for you to edit and then quickly update it again feels like it would require either a lot of vanilla JS writing or a React-esque approach to achieve, but what you'll see is it's absurdly simple.

Let's start by looking at the markup for a TODO item. I have clipped the non-edit markup for clarity.

<li hx-target="closest li">
  <form action="/todos/2a5e549c-c07e-4ed5-b7d4-731318987e05" 
      method="GET">
      <button 
        hx-get="/todos/2a5e549c-c07e-4ed5-b7d4-731318987e05" 
        hx-swap="outerHTML">
        📝</button>
  </form>
</li>

It maybe looks a lot, but the main things to focus on for understanding how the edit functionality works:

On the <li>, an attribute hx-target tells the browser, "When you get a fragment to render, this is the element I want you to replace". The children inherit this attribute, so for any HTMX actions inside this <li>, the HTML returned will replace the contents of the <li>.
hx-get on the edit button means when you click it, HTMX will tell the browser to do an HTTP GET to the URL and fetch some new markup to render to the <li> in place of what's there.
The form is not essential for the example, but it allows us to support the functionality for non-JavaScript users, which will be covered later.

When you start working with HTMX, an easy way to understand what's going on is to look at the network in the browser's developer tools.

When a user clicks the edit button, the browser does an HTTP GET to the specific todo resource. The server returns a hypermedia response, which is a representation of that resource with some hypermedia controls.

<form action="/todos/45850279-bf54-4e2e-a95c-c8c25866a744/edit"
      hx-patch="/todos/45850279-bf54-4e2e-a95c-c8c25866a744" 
      hx-swap="outerHTML" method="POST">
  <input name="done" type="hidden" value="false"/>
  <input name="name" type="text" value="Learn Rust"/>
  <input type="submit"/>
</form>

HTMX then takes that HTML and replaces whatever we defined as the hx-target. So the user now sees these hypermedia controls for them to manipulate the resource, instead of the row pictured before.

You'll notice the form has a hx-patch attribute, which means when it is submitted, the browser will send a PATCH with the data to update the resource. The server then responds with the updated item to render.

Embracing the web

There's more to HTMX, but this is the crux of the approach, which is the same as the approach that most websites were made before SPAs became popular.

The user goes to a URL
The server returns hypermedia (HTML), which is content with controls.
Browser renders hypermedia
Users can use the controls to do work, which results in an HTTP request sent from the browser to the server.
The server does business logic, and then returns new hypermedia for the user to work with

All HTMX does, is make the browser better at hypermedia by giving us more options regarding what can trigger an HTTP request and allowing us to update a part of the page rather than a full page reload.

By embracing the hypermedia and not viewing the browser as merely a JavaScript runtime, we get a lot of simplicity benefits:

We can use any programming language on the server side.
We don't need lots of libraries and other cruft to maintain what were basic benefits of web development.
- Caching
- SEO-friendliness
- The back button working as you'd expect
- etc.
It is very easy to support users who do not wish to, or cannot use JavaScript

This final point is crucial to me and to my current employer. I work for a company that works on products used worldwide, and our content and tools must be as usable by as many people as possible. It is unacceptable for us to exclude people through poor technical choices.

This is why we adopt the approach of progressive enhancement.

Progressive enhancement is a design philosophy that provides a baseline of essential content and functionality to as many users as possible, while delivering the best possible experience only to users of the most modern browsers that can run all the required code.

All the features in the TODO app (search, adding, editing, deleting, marking as complete) all work with JavaScript turned off. HTMX doesn't do this for "free", it still requires engineering effort, but because of the approach, it is inherently simpler to achieve. It took me around an hour's effort and did not require significant changes.

How it supports non-JavaScript

Your browser doesn't support HTML video. Here is a link to the video instead.

When the browser sends a request that was prompted by HTMX, it adds a header HX-Request: true , which means on the server, we can send different responses accordingly, very much like content negotiation.

The rule of thumb for a handler is roughly:

parseAndValidateRequest()
myBusinessLogic()

if request is htmx then
    return hypermedia fragment
else
    return a full page
end

Here's a concrete example of the HTTP handler for dealing with a new TODO:

(defn handle-new-todo [get-todos, add-todo]
  (fn [req] (let [new-todo (-> req :params :todo-name)]
              (add-todo new-todo)
              (htmx-or-vanilla req
                               (view/todos-fragment (get-todos))
                               (redirect "/todos")))))

The third line is our "business logic", calling a function to add a new TODO to our list.

The fourth line is some code to determine what kind of request we're dealing with, and the subsequent lines either render a fragment to return or redirect to the page.

So far, this seems a recurring theme when I've been developing hypermedia applications with HTMX. By the very architectural nature, if you can support updating part of a page, return a fragment; otherwise, the browser needs to do a full page reload, so either redirect or just return the entire HTML.

HTML templating on the server is in an incredibly mature state. There are many options and excellent guides on how to structure and add automated tests for them. Importantly, they'll all offer some composition capabilities, so the effort to return a fragment or a whole page is extremely simple.

Why is it The Future ?

Obviously, I cannot predict the future, but I do believe HTMX (or something like it) will become an increasingly popular approach for making web applications in the following years.

Recently, HTMX was announced as one of 20 projects in the GitHub Accelerator

It makes "the frontend" more accessible.

Learning React is an industry in itself. It moves quickly and changes, and there are tons to learn. I sympathise with developers who used to make fully-fledged applications being put off by modern frontend development and instead were happy to be pigeonholed into being a "backend" dev.

I've made reasonably complex systems in React, and whilst some of it was pretty fun, the amount you have to learn to be effective is unreasonable for most applications. React has its place, but it's overkill for many web applications.

The hypermedia approach with HTMX is not hard to grasp, especially if you have some REST fundamentals (which many "backend" devs should have). It opens up making rich websites to a broader group of people who don't want to learn how to use a framework and then keep up with its constantly shifting landscape.

Less churn

Even after over 10 years of React being around, it still doesn't feel settled and mature. A few years ago, hooks were the new-fangled thing that everyone had to learn and re-write all their components with. In the last six months, my Twitter feed has been awash with debates and tutorials about this new-fangled "RSC" - react server components. Joy emoji.

Working with HTMX has allowed me to leverage things I learned 15-20 years ago that still work, like my website. The approach is also well-understood and documented, and the best practices are independent of programming languages and frameworks.

I have made the example app in Go and Clojure with no trouble at all, and I am a complete Clojure novice. Once you've figured out the basic syntax of a language and learned how to respond to HTTP requests with hypermedia, you have enough to get going; and you can re-use the architectural and design best practices without having to learn a new approach over and over again.

How much of your skills would be transferable from React if you had to work with Angular? Is it easy to switch from one react framework to another? How did you feel when class components became "bad", and everyone wanted you to use hooks instead?

Cheaper

It's just less effort!

Hotwire is a library with similar goals to HTMX, driven by the Ruby on Rails world. DHH tweeted the following.

Hotwiring Rails expresses the desire to gift a lone full-stack developer all the tools they need to build the next Basecamp, GitHub, or Shopify. Not what a team of dozens or hundreds can do if they have millions in VC to buy specialists. Renaissance tech for renaissance people.

That's why it's so depressing to hear the term "full stack" be used as a derogative. Or an impossible mission. That we HAVE to be a scattered band of frontend vs backend vs services vs whatever group of specialists to do cool shit. Absolutely fucking not.

Without the cognitive overload of understanding a vast framework from the SPA world and the inherent complexities of making a fat client, you can realistically create rich web applications with far fewer engineers.

More resilient

As described earlier, using the hypermedia approach, making a web application that works without JavaScript is relatively simple.

It's also important to remember that the browser is an untrusted environment, so when you build a SPA, you have to work extremely defensively. You have to implement lots of business logic client side; but because of the architecture, this same logic needs to be replicated on the server too.

For instance, let's say we wanted a rule saying you cannot edit a to-do if it is marked as done. In an SPA world, I'd get raw JSON, and I'd have to have business logic to determine whether to render the edit button on the client code somewhere. However, if we wanted to ensure a user couldn't circumvent this, I'd have to have this same protection on the server. This sounds low-stakes and simple, but this complexity adds up, and the chance of misalignment increases.

With a hypermedia approach, the browser is "dumb" and doesn't need to worry about this. As a developer, I can capture this rule in one place, the server.

Reduced coordination complexity

The complexity of SPAs has created a shift into backend and frontend silos, which carries a cost.

The typical backend/frontend team divide causes a lot of inefficiencies in terms of teamwork, with hand-offs and miscommunication, and makes getting stuff done harder. Many people mistake individual efficiencies as the most critical metric and use that as justification for these silos. They see lots of PRs being merged, and lots of heat being generated, but ignoring the coordination costs.

For example, let's assume you want to add a new piece of data to a page or add a new button. For many teams, that'll involve meetings between teams to discuss and agree on the new API, creating fakes for the frontend team to use and finally coordinating releases.

In the hypermedia approach, you don't have this complexity at all. If you wish to add a button to the page, you can add it, and you don't need to coordinate efforts. You don't have to worry so much about API design. You are free to change the markup and content as you please.

Teams exchanging data via JSON can be extremely brittle without care and always carries a coordination cost. Tools like consumer-driven contracts can help, but this is just another tool, another thing to understand and another thing that goes wrong. The overheads, support and complexity of API versioning are inherently not an issue with a hypermedia approach, you are free to change the markup as you please.

This is not to say there is no room for specialisation. I've worked on teams where the engineers built the web application "end to end", but we had people who were experts on semantic, accessible markup who helped us make sure the work we did was of good quality. It is incredibly freeing not to have to negotiate APIs and hand off work to one another to build a website.

More options

Rendering HTML on the server is a very well-trodden road. Many battle-tested and mature tools and libraries are available to generate HTML from the server in every mainstream programming language and most of the more niche ones.

Wrapping up

I encourage developers looking to reduce the costs and complexities of web application development to check out HTMX. If you've been reluctant to build websites due to the fair assessment that front-end development is difficult, HTMX can be a great option.

I'm not trying to claim that SPAs are now redundant; there will still be a real need for them when you need very sophisticated and fast interactions where a roundtrip to the server to get some markup won't be good enough.

In 2018 I asserted that a considerable number of web applications could be written with a far simpler technological approach than SPAs. Now with the likes of HTMX, this assertion carries even more weight. The frontend landscape is dominated by waiting for a new framework to relieve the problems of the previous framework you happened to be using. The SPA approach is inherently more complicated than a hypermedia approach, and piling on more tech might not be the answer, give hypermedia a go instead.

Check out some of the links below to learn more.

How to set a WIP limit

Chris James — Sat, 14 Jan 2023 00:00:00 +0000

A colleague of mine is venturing into the magical world of Kanban, and one of the essential parts of Kanban is setting a "work-in-progress (WIP) limit".

A WIP limit dictates the number of items of work you can have in progress at a time. It dictates your maximum work capacity.

In this example, the team can only have 2 items at play at a time. Even if you have a "spare dev", she cannot start a new piece of work until one of the items in progress moves to Done.

You're misinformed if you're not setting WIP limits and claiming to be doing Kanban.

But what should your WIP limit be?

The Goal

The Goal (highly recommended reading), describes a poorly performing manufacturing part. Imagine an assembly line, where parts of items are assembled from raw materials, which then get passed down to the next part of the line to be further assembled.

It could look like this:

Steel -> Bonnet -> Car

Other parts of the plant will deal with building wheels, seats, fuzzy dice, etc.

The performance of these parts of the plant is tightly micromanaged and appraised based on their output, e.g, how many bonnets can we produce per day?

Most of the units could describe how they were performing well, yet the overall output of the plant was expensive and slow, constantly missing targets, and the profits were tumbling.

When you look at the plant, you could see piles of inventory, waiting to be converted into real things that could actually be sold. The plant was producing lots of stuff but not delivering real value to customers.

The moment you buy some raw materials and start converting it into something you intend to sell sell, you've taken liquid cash and turned it into inventory.

Inventory is waste until it delivers value.

This is where the plant's management had lost sight of The Goal. There's no point efficiently producing thousands of bonnets if they're not for the right car, or there are other bottlenecks in the plant for making wheels or whatever. The bonnets are dead weight, waste.

So what if instead of worrying about local efficiencies, you started focusing on the actual company goal? What if you said you'd only produce the number of bonnets you needed for a particular day, or better still just-in-time?

Surely this is terrible?! Our bonnet-making machine is just sitting idle! Remember, there's no point churning out tons of inventory if there's no immediate need for them. It's actively harmful, you're making more illiquid assets that need storage, insurance, security, etc.

So how do you make more money? You must identify the bottlenecks, and improve them. Only once you've improved the capacity of your bottlenecks, can you increase the output of the other silos; and only if market demand dictates it wants more of your product, the market is also a bottleneck.

One of the main messages of The Goal is this. There's little point worrying about local efficiencies, it can actually make things worse sometimes! Only The Goal is important; you need to look at the system as a whole and identify where best to improve.

Any improvements made anywhere besides the bottleneck are an illusion.

~~Goldratt

Lean is all about making work smooth, with flow, minimal bottlenecks and reduced waste.

In software

Setting a WIP limit puts a hard constraint on what work your teams can do at once.

You might be asking yourself

Seriously Chris? So if a developer is working on a ticket, but she can't finish it because deployments are blocked because another team's API is down, she has to sit there idle?

As a manager, your job is not to ensure everyone looks busy. Your job is to fulfil The Goal. Every time you park a ticket and pick up a new one, you choose to avoid dealing with a bottleneck. This bottleneck will keep harming your team and the company and slow its progress.

People love talking about breaking down silos in meetings; this is what should drive it. What if your engineer climbed out of the artificial play den you constructed and helped the other team relieve the bottleneck? Setting WIP limits give you clear indications as to what your bottlenecks are.

So much of "the DevOps movement" (which has its roots in lean manufacturing) is centred around automation, shift-left and breaking down silos. By making it, so a team owns not only the writing of the code but also deploying, testing and running it, you increase flow and reduce inventory.

The Phoenix Project writes about how not adopting this approach is harmful. The company invested millions of dollars in the project, writing code for years. Then it was thrown over from the development team to the operations team to release to the customers. The execution was catastrophic, but the subtler point was that the company had piled up millions of dollars worth of inventory over many years without delivering any value.

Inventory?

Inventory: Something "paid for" not yet put to use (delivering value). If we write something in a backlog, write code, do testing, think about the thing, or do anything, it is something we have paid for. If we have not yet received value for it - it's inventory

~~ Woody Zuill

When you start writing code, it is inventory until it delivers value. It is something that costs to write, maintain, test, etc. When teams have piles of PRs waiting for approval, all work is invested in but not delivering value.

It's inventory. What's worse is in software, we typically don't really know if our work is valuable until it is in the user's hands, and we want to collect feedback loops so that we can improve it to extract more value from the work. The longer your work sits as inventory, the longer you delay that crucial feedback loop.

If your team views work as "done" when a PR is raised, you are not looking at the overall system, just like the plant in The Goal. The back-and-forth review, perhaps manual testing, and all the steps between that and it being used by a customer are all bottlenecks you're choosing to ignore.

Monitoring pull requests raised/closed is akin to measuring how many bonnets you produce.

Mature teams that achieve a good flow of value will review code quickly (ideally on-demand with mobbing or pair-programming), automate all tests and have a reliable deployment pipeline. Hence, inventory never piles up, and they can deliver value quickly.

Definition of done

This is why it's essential to clearly understand what "done" means, and a story shouldn't move into the "done" column of a Kanban board until it meets your done criteria.

These criteria should include deployed to production, used by real people and generating value. It should include other things, which mean it's done to sound quality, such as testing, observability, etc.

When something is done, it should not require any more planned work from the team. Otherwise, it's still a work in progress.

But Chris, the marketing team want us to hide it behind a feature flag for a few months before they get a campaign ready.

It's still WIP! It is still unvalidated ideas and work that is not delivering value. Why is marketing a bottleneck? What could you do as an organisation to be more nimble concerning marketing? You're still thinking locally. Think about The Goal!

Too often, software development organisations celebrate the amount of work they've done; it may even be deployed to production, but if no one uses it, it's still inventory and waste.

Be honest about what "done" means; don't be content with being an efficient bonnet-making silo within the plant. Thinking about "done" will make you answer some tough but essential questions about the way your organisation tries to deliver value.

How to keep WIP low

Reduce toil. A system that is simple to work on will have a smooth flow of work. Flaky tests and slow builds all contribute to work stalling.
Automate all the things. You will always suffer from bottlenecks if you have to wait for someone to test something or do a security review manually. See: "shift left".
Reduce scope. Work in small, achievable steps. Think of ways to frequently deliver small amounts of value rather than "big bang".

Focus on the goal, and work as a team. Encourage a culture within your team of helping each other keep WIP low. Once you finish a task, rather than starting a new one, help your colleagues finish their tasks too. If you find dependencies on other teams are causing bottlenecks, try to help them too.

The perils of high WIP

There is a very real sense of chaos. Many plates are being spun, lots of context switching. Your team will work hard but feel inadequate regarding the actual value (The Goal) being delivered. This will hurt morale. Having high effort but low outcome/reward is commonly linked with burnout.

A recent, sad tale

I've seen in the past year how a combination of socio-technical, and organisational issues have caused subtle, but very harmful bottlenecks for the team I've managed.

Writing this stuff is simple, but I won't pretend it's easy to solve! Nonetheless, you must try.

I needed to communicate these issues more effectively and get sufficient buy-in to deal with them appropriately.

Capturing data sooner would've helped build a more coherent case. When planning work, identify potential bottlenecks and find ways to measure them so you can find the problems quickly with concrete data.

So what number should you set?

As with all things process and methodology, you should not cargo-cult and get lost in a particular implementation, but instead, understand why the practice is essential. This will allow you to tailor it to your specific context, and I hope this post has given you an understanding of what setting a WIP limit accomplishes.

The specific number is insignificant; what's important is the flow of work and listening to the signals the limit provides you, mainly what your bottlenecks are.

If I'm starting with a new team, my rule of thumb is roughly (Number of devs / 2) + 1, and I take it from there.

If work is sitting in progress or parked for long periods, your "inventory" is high, and you likely have bottlenecks you're not addressing. Setting a WIP limit lets you "see" these bottlenecks more clearly. If you choose to ignore these bottlenecks, though, your team will suffer.

If work goes from idea to done (and remember, actually done and delivering value) smoothly and quickly, you probably have your WIP limit at the right level.

My job advert for 2023

Chris James — Mon, 02 Jan 2023 00:00:00 +0000

I'm looking for a new software developer/engineering manager role, and this post will summarise me for potential employees.

This blog is a fair reflection on my general attitudes towards software development, what's important to me, what I feel works, and what doesn't!

My hope is someone will read this and think.

Chris's experience, values, and work methods align with what we need, and we should chat with him.

If I look like a good fit, contact me and dig into other posts here to see more of what makes me tick. If you wish to prepare for our interview, I already have my questions written down for you.

The headlines

Happy to be either an engineering manager or a developer within a team.
I'm happiest and probably most effective when I am coaching and pairing with engineers & teams.
Around 17 years of professional software-engineering experience
My most recent experience is with Go, but I have shipped numerous production systems in Type/JavaScript, Kotlin, and Scala.
I've worked on various systems during my career, from client-side heavy, niche data exploration tools and backend APIs to websites serving hundreds of requests per second across the globe.
I am the author of the popular resource Learn Go with Tests.

Stuff I like

Trunk-based development
TDD
Continuous delivery (10 deploys per day, DORA metrics, that kind of vibe)
Autonomy with responsibility. You build it; you run it.
Kanban (not scrum)
Iterative, lean development practices
XP
Stream-aligned teams (think Conway's Law and topics discussed in Team Topologies)
Learning, coaching and mentoring. A curious and learning-focused engineering culture is very important to me. Every place I work I try to encourage and facilitate book clubs, katas, lightning talks, etc.

My answers to the usual questions

What are you most proud of professionally?

Writing Learn Go with Tests and seeing what a positive impact it's had on many engineers.
My conference and meetup talks (Gophercon UK, London Gophers) and being invited to Go Time a few times
Participating in several junior engineers' journeys, helping and guiding them to well-deserved promotions and pay raises. The thing I am probably most proud of is hiring a group from Makers Academy, a coding boot-camp, and fostering a fun, curious and effective engineering team.
Helping teams become more effective in their development and release practices, making work more straightforward, less stressful and more effective. We're a good fit if you have heard of DORA metrics.

What are you looking for?

I am happiest when I am in a team that is constantly growing. I love coaching, guiding, and learning from people smarter than me. I want an environment where engineers are humble and curious about software engineering. My dream role is to be a coach.
Go, ideally, but I also like Kotlin and JavaScript (mostly). I enjoy programming, and if you're happy to accept additional ramp-up time for me to learn something new, I'm more than excited to do that. Any Rust roles?
A team of diverse backgrounds. I've been lucky to work in teams over the past five years with people from various backgrounds, making me a better engineer and person.
I love seeing metrics fly up that represent actual people using the software. Feeling a connection between the work to tangible outcomes is fantastic.

Can you complete these data structures and algorithms / cracking the coding interview / leetcode exercises?

I'd rather not. I last thought about bubble sort et al. around 2002.

I've designed a number of interview processes, securing some excellent colleagues. They've always involved a small but practical homework that has helped us understand the quality of the code they write in a team setting (think testing, coupling/cohesion, separation of concerns, etc.), rather than whether they can invert a binary tree or not.

Look at my GitHub if you need clarification on whether I can write code, or look at Learn Go with Tests as I clearly explain my thought process.

Values and principles

Quality and delivery speed are correlated, not traded-off

Dave Farley's book Modern Software Engineering encompasses my views of what practical engineering looks like. If you've read that book and it resonates, I'd love to work with you. One topic he covers is that teams often take shortcuts and technical debt to go faster, but when we bring evidence to the picture, we find it's a false economy.

The State of DevOps reports debunk the idea we trade speed and quality off; they are correlated.

If we're honest, we all know this to be true. How many times have you opened a garbage codebase and wasted days or weeks trying to make a minor change?

The way to go fast is to write high-quality, small increments, get feedback, change direction, and repeat. See also: How to go fast.

Empowered teams

I'm not too fond of the idea of story robots. Programmers on an assembly line picking up tickets from JIRA, mindlessly doing them and throwing them into "done". Never questioning the work and never working with real users.

I have no interest in leads, seniors or BAs sitting in ivory towers writing tickets and throwing them over the wall to some poor, demotivated programmers who are judged by how many tickets they complete or how many PRs they approve per week.

I want my teams to be given a real user problem and for them to work together to figure out how to solve it. They should focus on actual business outcomes, not "velocity".

Velocity is an abstract concept; if a team has a "good" velocity, it doesn't mean they're solving real problems. High velocity is only useful if you're going in the right direction.

Work-life balance

I feel very blessed to have a profession that I enjoy, but that doesn't mean I will sacrifice my personal life for it.

The need for long hours and heroism often ruin work-life balance. Heroism is a symptom and a bandaid for organisations that fail to manage work correctly, execute poorly on a technical level and not having the courage to say "no".

(Another good sign of compatibility is if you know what I mean by "That looks like a Brent", referencing The Phoenix Project)

Occasional acts of heroism are sometimes necessary, I'm never jobsworth about my role and only work strict hours, and it can feel good to save the day; but it should be the exception, not the norm.

When I run teams, I optimise the way we work so that we try and do things properly and not be woken up at 3 am, and if work is taking longer than people hoped, we deal with it, find ways to improve, cut scope, etc., but not demand engineers work late nights which for the most part, tends to make things worse.

If your job description includes phrases like a fast-paced environment or working well under pressure, that gives me some red flags.

Honesty, with respect and empathy

Honesty is extremely important, and when I build teams, I do my best to try and build a safe environment for people to express their views so we can grow.

"Constructive" feedback is an act of kindness if it is done kindly. Honesty does not have to come at the expense of respect and empathy.

Learning culture

I love learning and improving my craft. By our industry's standards, I am very experienced, but I am still excited to challenge myself to improve and learn new things.

So many organisations bemoan the output of their engineering teams but are unwilling to do anything concrete to increase capacity beyond cracking whips. I want to work in organisations that invest in their teams to help them improve; things like 10% time are good examples of this.

It also should be happening on a day-to-day basis. Pair programming, mobbing, retrospectives, etc., are all important tools to help teams and people improve. A safe environment where people can be fearless to try things without fear of reprisal if something goes wrong is key to growth and innovation.

This also relates to my first point: quality and speed are correlated, not traded off. The better we get at writing software, the more we understand the domain, the higher the quality of the systems we build and, therefore, the more value we can deliver. If this doesn't align with you, it will be difficult.

Other practicalities

Remote?

Even though I am an introvert pretending to be an extrovert, I like going to the office and seeing people face to face. That said, depending on where in London your office is, it might take a long time. Either way, it is very expensive (I live just outside London), so I prefer to go 1-2 times per week maximum. I can make occasional exceptions; for instance, if you're doing a week-long project inception, that's fine.

I am open to being 100% remote, but so far, my one experience (during Covid) could have been more enjoyable. I feel I suffered from "Zoom fatigue". Maybe our environment did remote work wrong, though, and I appreciate lots of people swear by it.

I also am okay with travelling abroad from time to time.

When?

I prefer to start around March, but have some flexibility.

Salary?

Nice try. You tell me.

PS, everyone has a "competitive salary".

2022

Chris James — Sun, 11 Dec 2022 00:00:00 +0000

Work has been a mixture of a lot of highs and lows. My current place is quite chaotic; there have been leadership changes that bring in different ideas and expectations. Re-orgs often tend to be tough, where you find different people's values and ways of work clashing. Over this year, I've had three different line managers, which has sometimes caused problems. These are all fairly normal growing pains, and in a lot of ways things have improved as the year has gone on.

I have learned that expectation management is an essential skill for an EM (engineering manager), it's not enough to assume people will trust you, and people need clarity as to why something is taking longer than they expect. Explaining why a complex system (both in terms of tech and the social side) is taking "longer than expected" has been difficult and stressful for me. Not something I've enjoyed, and it has made me question my skills and motivations for being an EM.

I need to reflect on how I can explain these ideas better if I can protect my team better and, on a personal level, have a healthier work-life balance. On previous projects I leveraged "sliders" to get buy-in and clarity around the kinds of trade-offs teams need to make with work. I think it was a big error on my part not to use this tool and get blessing from senior management.

So there are a lot of things to stress about, but importantly, my team are delivering, and it all seems to be more-or-less coming together as the year finishes. Our part of a large and complex system is rolling out across Europe for thousands of customers. My team has been put under a lot of pressure, which I have tried to manage as best I can, not always successfully, but I hope they recognise their achievements. It's sometimes easy to get lost in the dozens of bits of work we need to do, which will seriously help the business in tangible ways.

No matter how it all pans out, I'm also extremely pleased and proud of how the team has grown over this year. We've got much better at writing software as a group. We've learned many essential concepts in writing sound systems, particularly automated tests, TDD, contract testing, London vs Chicago, and many more. Our code-review sessions, mobbing and pairing, have brought up some fascinating ideas, precisely what I'm looking for when trying to build teams. I'm proud that members of my team have gone to other teams to help coach these modern engineering practices to good effect and have a wider impact to the engineering organisation. To top it all off, 2 on my team got well-deserved promotions; I'm thrilled for them.

Open-source / community things

Given how hectic work has been, it was surprising to look back and see I still found the time and inclination to write five new chapters for Learn Go with Tests this year.

I was also lucky enough to be invited for my 3rd episode of Go Time. In this episode, we discussed TDD with someone who is not a fan.

Goals for next year

Not going to be too hard on me if I don't achieve these, but it's nice to think ahead.

Professional

Keep on top of Learn Go with tests.

It's cool and appreciated how many issues and PRs are raised, but I feel guilty about not addressing them quickly; this will be a constant struggle.

I have a few new chapter ideas that would be nice to try out:

Living without mocks
Nested testing
Fuzzing
Browser testing

Back to a simpler world with web development

I am thrilled that technologies like htmx and hotwire are gaining traction. I have long expressed how much I hate the proliferation of client-heavy SPAs, about how much complexity they bring to making simple websites.

With the likes of htmx, we can go back to a world of rendering HTML on the server, which is extremely simple and battle-tested; but still have a smooth client-side experience for the user. It also liberates from using Node on the server; if your $FAVOURITE_LANGUAGE can respond to HTML requests, you can make great websites.

I experimented with this during some time off, and many people were engaging with it on Twitter.

If I find time, I think I can write a lot of material around web development and Go using this way of working.

Speak at GopherCon US

I was lucky enough to speak at Gophercon UK a few years back. A while ago, I applied to talk at a few state-side conferences and didn't get a response. I want to try again, as it is a personal ambition to speak at a conference outside the UK that I'm not quite willing to give up on just yet.

Maybe the simpler web development approach would make a decent conference proposal.

Personal

You can probably stop reading here; this is just for me :)

April this year, I weighed 107kg (I am 6'2" tall), which was terrible. The pandemic and work led to a lot of inactivity and stress-eating, which is a bad combo. I went back to basics, tracking calories and exercising, and seven months later, I am 91.4kg! Very pleased with this progress; I fit into all my clothes again. I want to push on next year and get myself down to around 85kg.
As part of the above, I got into "couch to 5k" and have since completed a few 5k runs. I am unfit still and relatively slow; it takes me 33 minutes. I want to try and push this to 30 minutes, but honestly, I have no idea if that is good/impossible. Either way, something to improve at.
I waste so much money on computer games that are probably really good, but I don't have time—budgeting myself a max of 4 new games next year to force myself to at least make choices not to be so impulsive.

Speaking at GopherconUK

Chris James — Mon, 29 Nov 2021 00:00:00 +0000

For several years, I had an ambition to speak at some kind of Gopher conference event, especially to talk about testing, which is a subject that is so misunderstood. I've been doing the whole testing thing for quite a long time now, and I still find lots to learn. So, I get irritated when developers of 2 years experience tell me, so full of confidence, that "no-one does TDD in the real world".

A few years ago, to work toward this goal, I set myself a shorter-term goal of speaking at London Gophers. It went pretty well, I was proud of the talk and how it went. However, when I went on to apply at various call-for-papers for Gophercons around the world, well, I didn't get anywhere. My rational brain knows it was very competitive and was pretty natural to not get any responses at all, but I still took the failure a bit personally and gave up on the idea.

So, I was pretty thrilled to get the opportunity to speak at GopherconUK and even more excited that I roped Riya Dattani in to speaking with me.

I won't go too much into the material, watch the video on YouTube.

But I will say it's a combination of many thoughts I've had in general with testing, from reading things like Continuous Delivery, GOOS and even everyone's favourite novel; The Phoenix Project.

Testing has a profound effect on your ability to ship software effectively, without toil, and it even has an influence on how your team works and how it solves problems. It's a far bigger subject than merely preventing bugs.

Reflections on the experience

I definitely have a new-found respect for good conference speakers. It is hard work. We worked really hard on it, for almost 2 months (not to mention the years of studying, experience and reflection beforehand!).

The first month was "just" making the slides, which was actually harder work than I thought it would be. Trying to come up with a talk that had to be around 40 minutes but still have a coherent narrative and flow is tough.

We had a lot of support from friends and colleagues, which helped a ton.

On the day, we were obviously nervous, but once we were on the stage it was honestly straightforward. It was like we had done it 100 times before because we had.

Tips

Before committing to doing a talk at a conference, know that it will suck up your life for 6-8 weeks if you're going to do it properly.

I remember saying to Riya:

I don't want us to not try hard enough and then use that as an excuse if we don't do well. Let's stick our necks out a bit, and take advantage of this opportunity and do a good job.

Other side-projects and things will probably need to be paused for a while. The effort on preparation will pay dividends, though. It's not that we did it perfectly, but it went as well as I hoped it would, and I enjoyed the experience.

I'm not a prolific public speaker, I've only done it a few times, so take the following with a pinch of salt. However, Lara Hogan has an excellent free book - Demystifying Public Speaking which is definitely worth a look.

Maybe there are naturally gifted speakers who can just wing some slides on the plane, for the rest of us though, this is what I'd recommend.

Practicing effectively

I committed to practicing my part of the talk at least 2 times a day. I wanted to avoid relying on presentation notes too much, and I wanted to be fluent at what I wished to communicate on every slide. Perhaps that's over the top, but it gave me confidence when I was on the stage.
Practice out-loud. Reading your slides is not practicing what you're going to do on the day.
Practice your talk standing up. Your posture, breathing e.t.c. is just different compared to sitting, and you want it to be as close to the real thing as possible.
Try mixing it up, so you practice each section effectively. For instance, practicing doing slides X to Y a few times over, rather than the whole thing.
If you're inexperienced with public speaking, you might want to start by speaking with your colleagues first. At my previous and current job, we've set up a lightning talks initiative where people regularly do short talks to the rest of the group to share knowledge and practice communicating effectively. With a healthy feedback culture, your team/company can get better at sharing ideas, and it can set you up for public speaking.
Record yourself, and listen to it.
Get feedback from your friends/colleagues early, don't be afraid to show something rough.

Preparing the talk

Start early. You'll never regret being over-prepared.
Try to identify a central narrative first. Think about what the audience will specifically take away from the talk. Ask yourself what an engaged audience member would do when they get back to the office.
Find time to keep researching for ideas and perspectives. Even though I had a pretty good idea of what I wanted us to convey, by reading some other books whilst writing the talk I learned some different ideas and incorporated them into the talk.
People like stories and anecdotes, try to weave them in to your talk. I think Riya's experiences of how BDD helped her as a junior dev in particular were a different and interesting take on the material that I hadn't heard before.

On the day

Remind yourself that you've done this 100 times before.
Remind yourself that people want to hear a good talk. You're not in high-school any more, people won't taunt you for messing up.
It doesn't need to be perfect, it doesn't matter if you briefly lose your way, or misspeak.
Your talk won't be for everyone. Some people won't like it, and that's ok.
You're going to get to talk about something you're passionate about to a captive audience, this is cool and a real privilege.
Try to enjoy it, you'll probably find yourself nervous for the first few minutes, then you'll get into your rhythm, and it'll be over before you know it.

Reduce WIP by practicing trunk-based development, rather than pull requests

Chris James — Sat, 26 Jun 2021 00:00:00 +0000

Poorly managed work in progress (WIP) is a common cause for low performance in development teams. When you have a team juggling too much work, it engages in frequent, expensive context switching, which reduces quality and slows delivery. Your team is working hard, but not delivering.

The reduction in quality has a compounding effect on WIP as you will get unexpected bugs, re-work, and outages; which in itself causes more WIP. Unplanned work is poison for productivity and predictability.

Managing WIP involves looking at how work flows through your system, trying to make it go quickly, smoothly and predictably. Identifying bottlenecks & constraints allows you to manage them more effectively and focus your efforts on improving them and making sure only the highest priority work goes to them. Any efforts to improve efficiency outside these constraints is waste because it’s the bottlenecks in your system that dictate pace. With a focus on quality you can reduce the amount of unplanned work which helps you manage WIP.

There is no one true way of working, the best method depends on your team and the context they’re in. Managing WIP, and having a smooth flow of work, is not a negotiation if you are serious about managing an effective team.

Pretend you are the lead of a 4-person development team. How your team approaches creating value obviously has a big impact on its ability to deliver it, yet many teams copy other team’s rigid models, rather than thinking about the context they are in and tailoring the process to it.

The open source, pull-request model

This is an often-cited, popular way for teams to work.

Each developer picks up a ticket/story/task
Developer writes code on a branch. This allows them to work isolated, focused, and not affect other team members with their work which may not be completed.
When the developer believes the work is finished, they raise a pull request. When another developer is free, they can take a look at the work and then initiate a feedback cycle with the person who raised the PR until it is of sufficient quality.
The code gets merged and deployed to customers

Often the reasons people adopt this model is it allows a degree of gatekeeping and socialising knowledge. It’s argued that it can help prevent mistakes from being shipped and maintain code quality.

High WIP

For a team of 4 developers, having at least 4 tickets in play at a time feels like high WIP. Chasing 100% “resource” allocation is not a desirable management goal because all systems need a level of slack to cope with unexpected work. If a team is running at full-capacity it is inflexible, and inflexible things tend to break under pressure.

If you raise a PR and none of your colleagues are free, you may be tempted to pick up another ticket. The WIP of the team has now increased to 5. You can easily imagine, and have probably experienced, scenarios where this is a slippery slope to climbing WIP.

How many times have you heard:

It’s done, it just needs to be reviewed

In software development, the word “just” is often doing a lot of heavy lifting. The work is not “done” until it is actually delivering value.. The work is still contributing to the team's WIP.

The feedback cycle in the PR can take some time with asynchronous back-and-forth, so you may end up having to switch context between multiple tickets at a time. In lean terms, the process has caused a bottleneck.

Even with a team of only 4 you can easily end up in a situation where they’re context switching frequently. Between the story they’re assigned to, but also reviewing lots of other code written by their team. The personal WIP for each team member can climb steeply depending on how back-and-forth the review process is. As the WIP increases for each person, the ability to get code reviewed diminishes.

PRs are remote-friendly due to their asynchronous nature. This is true, but you have to accept then you will have a lot of unfinished work in play, which will raise your team’s WIP.

If I were to be managing a team working like this, I think it would be prudent to ensure that you only have 3 developers working on stories at a time, giving your system some slack. The other developer can help reviews happen quicker, find ways to improve flow, increase quality in other ways (improve tests, logging, metrics, e.t.c.), and you’ll have more slack to deal with unplanned work.

Slow integration

The longer your code lives on a branch, the less integrated you are with the rest of your team. Whilst the 4 of you are described as a team, you are all working on different versions of the code, as individuals.

If you find it hard to get your colleagues to review your changes, this slow integration compounds, and teams that adopt this system often have to deal with challenging merge conflicts. In the previous example this way of working can lead to multiple PRs that need merging which can result in multiple merge conflicts having to be resolved. This is waste.

Unpredictable flow

The review back-and-forth can last minutes, hours, days, or even weeks. In the worst cases, it involves a lot of re-work. Once the developer falls in to the trap of picking up another story whilst their other is in review, WIP can quickly spiral out of control.

Plenty of successful teams ship software like this, it’s not unworkable, but I’m not convinced it’s simple or efficient. I think even when it is managed extremely well, it still causes a lot of waste.

Trunk-based development (TBD) and pair programming

What is the most obvious way to reduce WIP?

Do less work concurrently
Concentrate on finishing work
Increase quality to reduce unexpected re-work

So instead of having 4 developers work on 4 tickets, let's instead have 2 developers pairing per ticket.

By having 2 heads work on a problem, it liberates us from having to gate-keep the main branch. We should trust that 2 developers on our team can take ownership of a problem from idea to production. Therefore, they can commit their code straight to main. This simplifies the flow of work. We no-longer have a wasteful bottleneck to shipping value.

No, I can’t trust 2 developers to take an idea from kick-off to production.

How many then? 3, 4? Or worse, can it only be put into the main branch if it is blessed by the lead? If you’re a lead making yourself a bottleneck, you’re not doing a good job. You’re not spreading your knowledge, you’re not coaching well enough, you’re increasing bus factor, and you’re hurting the productivity of your team.

Improved flow, higher quality and less re-work

By integrating frequently through the day, the chance of merge conflict is low and even when it happens it is typically trivial to fix. Far easier than having to merge a week’s worth of changes back into main.

This is true continuous integration, the team can consistently see what everyone is working on, which helps collaboration. No longer will you hear excuses like:

I’d like to refactor the blah blah, but I can’t because I know Alice is working on a PR related to it.

By removing the barriers to changing code and having a tight integration with your team (rather than disparate branches) you can now fearlessly refactor.

It’s an assumption, but not an unfair one, that the quality of code should be higher if 2 people are working on a change. High internal-quality makes changing our system simpler and should reduce the chance of unexpected re-work.

If you practice continuous deployment, it means you’ll release smaller changes frequently throughout the day. This reduces risk and makes recovery easier compared to diagnosing a pull request which may have been worked on for a number of days (or weeks!) rather than hours.

The idea of code-review and socialising ideas is now constant and easily available. Developers don’t have to write large amounts of code and then inconvenience another developer, forcing them to context switch to give feedback on your ideas. Instead, you have a colleague that you can easily bounce ideas with.

To make sure the socialising of code is spread amongst all developers, we ensure that we rotate the pairs frequently. We’ll also do a “code review” session every week for a couple of hours when we can talk about our code in broader strokes, in a more holistic way; rather than focusing on an isolated change in a PR.

Risk

If the idea of pushing directly to main is uncomfortable for you, it’s likely you’re working on an unsafe system. For TBD to work, you need to engineer your system so that it is safe to work on. You’ll need things like:

Good alerting
Easy or automatic rollbacks
An excellent test-suite
Observability
“Test in prod”

If you aren't testing in prod you aren't testing in reality -- just a weak dime store knockoff. Then you light the fuse and walk away with your fingers crossed. Test in prod. Please.

— Charity Majors

Gatekeeping around PRs and releasing is often reliability-theatre. Nothing can save you from mistakes, they’re inevitable. What you need to focus on is your mean time to recovery.

To commit to main directly, safely & frequently you need an excellent, fast test-suite and a well configured CI-pipeline. Developers need to be disciplined and only commit code they’re happy to put on the main branch.

Every developer is touching mainline, so all features grow in the mainline… which acts as a communication point. With CI, the mainline must always be healthy, so in theory (and often in practice) you can safely release after any commit.

— Martin Fowler

To live with the constraints of wanting to continuously integrate but not make “bad” commits, you need to find ways to break your work down into small, achievable pieces. This discipline makes work more achievable and will reduce risk. By working like this, I’d expect our 2 pairs of engineers to push several times per day.

Trying TBD

If you do decide to try TBD, be prepared for some friction and growing-pains if you have a team unfamiliar working in this way. A few years back, I wrote about how I introduced TBD to a team that was previously familiar with a PR-based approach.It will be a journey of uncovering issues in quality, bottlenecks and collaborating to find ways to improve them.

TBD may expose some uncomfortable shortcomings in your system, and the quality of your code and architecture. I’ve noticed that often a PR flow masks problems like:

Developers finding it hard to break problems down in to smaller, less risky changes. They have not learned how to work in small batches.
Low internal quality. If your system suffers from a lot of inappropriate tight coupling, you’ll tend to trip over each other’s toes when working together on the main branch.
Poor/inadequate tests. Lots of teams use the “CI server” as a crutch for poorly written tests that they run slowly and infrequently. Every so often, the system is designed in such a way that important tests are impossible to run locally.
- If you’re pushing every hour, you need to be able to test locally and be confident your change is shippable and won’t affect your colleagues negatively.

This is a journey that many factories (real factories!) have gone through. To work leanly, with excellent flow, managing WIP requires you to look at how you do work and commit to increase quality from start to finish as a team.

Which is best for your team?

TBD is not niche or unusual. Plenty of big businesses, in well-regulated industries, work with pairs of developers committing many times a day to do the main branch. Chromium releases every day and does not use branches.

In this short video, Dave Farley shows a fin-tech company in a well-regulated space working with TBD

Usually, these high-performing teams ship these changes extremely frequently to customers, so they can get real feedback quickly and build superior products.

Whilst pairing may seem like an expensive frivolity, the workflow is simpler, more focused and of higher quality. You’ll find that this approach can out-perform the same team working as individuals on branches. There is plenty of evidence to back up this assertion.

Five years' worth of academically defensible State of DevOps research that shows causality of feature branching with poor performance.

— Daniel Terhorst-North

PRs optimise for individual resource allocation, but introduce bottlenecks in your system which drive up WIP, and make the flow of work less smooth than it could be. High WIP diminishes a team’s productivity and is a management dysfunction, making work harder than it needs to be.

To those unfamiliar with TBD, it may seem risky and even unprofessional, but it forces you to work with more discipline, adopt best practices which will make your system more resilient and the work simpler.

(Mainline development) is an extremely effective way of developing, and the only one which enables you to perform continuous integration.

— Continuous Delivery: Jez Humble & David Farley

Many arguments that dismiss TBD as impractical for their team often are centred around fear, and a lack of trust. If you’re a leader of a team, don’t dodge these issues with walls and process. Deal with it.

Pull-requests work excellently in the open-source, where you can't trust external contributors. For your closed-source, day job though?

Despite abundant evidence that trunk-based development practices contribute to better software delivery performance, some developers who are used to the GitHub-recommended workflow remain skeptical

— 2017 State of DevOps Report

Think about your context, examine how work flows through your system, set a WIP limit and stick to it. Find ways to improve the flow and your team’s performance will improve.

The TDD Thinking Hats

Chris James — Fri, 25 Jun 2021 00:00:00 +0000

When working with people who are learning test-driven development (TDD), I have a habit of talking about “the hat we’re wearing”.

This comes from reading the book Six Thinking Hats.

The premise of the method is that the human brain thinks in a number of distinct ways which can be deliberately challenged, and hence planned for use in a structured way allowing one to develop tactics for thinking about particular issues.

When a group wears a particular hat, they engage and discuss an idea in a deliberate way. For instance, if the group wears the yellow hat, they should think positively about an idea, even if that feels unnatural to them. This means everyone explores positives in something, even if individuals within the group have some biases against it.

I’ve seen developers who mix up the steps in TDD, where they are refactoring whilst the tests are failing, and they are trying to introduce new behaviour at the same time. These same developers frequently skip the explicit refactoring step too, so they make the whole process more difficult and still end up with code that requires a lot of improvement.

I find trying to have a more focused, deliberate mindset during the TDD steps helps me practice it more effectively, simply, and avoid some common TDD pitfalls.

This posts attempts to describe the hats you should wear during the TDD steps.

TDD feels right when you’re not wearing any single hat for a long time. If you’re switching through them frequently, that’s telling you that you’re iteratively making positive progress on the system.

When writing a new test: The green, ambitious hat

Write the test you want to see

In this situation, we should be consumer focused, and optimistic about the kind of code we’re going to write.

It can be helpful to ask yourself:

If this code already existed in the open-source world, how would I want to use it, and how should it behave?

Try to be ambitious, you get to design a great API by describing how it would be used by a consumer, in this case your test.

Take pride in the test you write, keep asking yourself whether the test describes the what and why effectively. Have empathy for your colleagues who’ll be reading and maintaining these tests.

At the same time, think about how you can cut scope sensibly, so you are confident that you’ll be able to make the test pass and have this positive change committed to source control in a reasonable amount of time. Have an eye on the end-state, but keep thinking about each small step toward that goal.

Aim to wear this hat frequently, but for short periods, so that you’re working iteratively with fast feedback loops on your decisions.

But, I don’t feel confident

This is normal! The first step, is often the hardest. There can be numerous reasons why writing a test is difficult:

The existing test suite is a hot mess

Adding a new test to an existing suite can be difficult for various reasons. Perhaps the existing tests are poorly written. Often they are a reflection of poor design, such as lots of complicated mock setup.

This might be time to reflect on the design of the code, and it may be prudent to pick up the blue hat to update the design of your system to facilitate the next change.

I don’t really know what I’m trying to make

Talk to your colleagues and stakeholders. Ask them what they think might be the first valuable step. If you don’t understand the problem you’re trying to solve, it’s going to be impossible to write a test for it.

You may wish to invest some time moving up a few abstraction levels to create an acceptance test to give you a more clearly defined north-star.

When the tests are failing: The red, uncomfortable to wear hat

When practicing TDD effectively, you’ll wear this hat frequently, but it should feel deliberate when you do, and you should never be wearing it for very long.

When the tests are failing, you should feel uncomfortable. Your software is not working correctly.

Endeavour to get the tests passing (green) as quickly as possible.

You’re not safe to make changes that are not directly related to passing the test, as you’ve lost your safety net. Any further changes not dedicated to making the tests pass can make the situation worse by adding confusion and complexity.

Make the test work quickly, committing whatever sins necessary in the process.

-- Kent Beck

It’s important to be thinking about good source control practice. Once you get the tests passing again, commit your changes. If you ended up wearing this hat during refactoring, it’s often better not to be precious about the code you wrote, just revert it.

Stop wearing multiple hats at once

In my experience, many developers wear too many other hats whilst wearing this one in particular, and can make life difficult for themselves because introducing more changes tends to make the failing test situation worse.

Too often, developers worry too much about writing "beautiful code" and get distracted from our short-term goal, which is to make the software work. We’ll have time to make the code great once we’ve made the test pass.

When you’re refactoring: The blue, confident hat

We are free to be creative! Maybe you were thinking about a design with the Red hat or the Green hat on - now is the time you can try it out.

We feel safe to make improvements because we get fast and reliable feedback that whatever changes we’re doing are valid. This allows us to aggressively refactor and take pride in our work.

Make a positive change
Verify the change is valid by running your unit tests.
Commit the change to source control
Go to 1

All our efforts are backed by tests against real, desired behaviour, so design decisions we make will be “real”, rather than for imaginary concerns.

When wearing this hat we get to improve the code, experiment and learn what works. Be brave and try different approaches to improve the code. If it doesn’t work out, it’s not a problem, as you’ll always be able to get back to working code with a git checkout src/whatever.

Wearing this hat frequently through your day will steadily make you a better developer because you get more chances to experiment and learn.

If you have an idea for an improvement that you think will take many hours, pause for a moment. It might be worth going through a few more TDD cycles before making this commitment. By adding more tests, and therefore more behaviour; you are giving yourself more concrete information to make better design decisions.

Like the other hats, don’t let your head get hot wearing it for many hours in a row, as you might end up in a refactoring rabbit-hole. Little and often is the key.

Wearing the hats while pairing/mobbing

Six Thinking Hats says that everyone should be wearing the same hat at the same time, so that the group remains focused on the particular task, and you don’t end up moving in different directions and getting distracted.

For instance, the group could be wearing the red hat, but you can see a way the code could be improved. Don’t break the group’s flow, just take a note and bring it up when you’re all wearing the blue hat. Remember, if you’re practicing TDD well, you’ll be switching hats frequently through the day.

If you’re a more experienced member of the group, perhaps you can try using this thinking tool as a way of moderating how your team works through a problem more deliberately, and with better focus.

Do you wear these hats?

To those who feel confident and proficient with TDD, do these feelings and behaviours resonate with your experience? Do you disagree or do you have anything that’s missing here? Please reach out to me on Twitter.

Go Time - Giving TDD a Go

Chris James — Thu, 24 Jun 2021 00:00:00 +0000

I was lucky enough to have a second appearance on the Go Time podcast a few weeks back. Riya, Dave, Mat and I talked about TDD.

I enjoyed it! Don't think I waffled too much but as Mat said, it felt like we only scratched the surface.

You can have a listen here

The Three Books

Chris James — Sat, 19 Jun 2021 00:00:00 +0000

David Wickes wrote an excellent post called Three Books, which claims that any senior developer should be able to recommend three books to a less experienced colleague. There are various reasons for this and some nuance, so please read the post!

I wish I had been lucky enough or had the foresight to read these books when I was 25. I’ve tried to pick three varied books that give you solid foundations to be a good software developer.

The Phoenix Project

This helps you understand there’s so much more to shipping software than raw technical skills.

Making anything non-trivial requires teamwork. This brings social and organisational challenges.

These are different from what you learn in a computer science degree. This book helps you see these issues, the dysfunctions organisations can get into, and how to get past them.

If you understand the concepts in these books, you’ll be able to help your team work better and be more likely to have successful projects.

After reading this book, if you enjoyed it and found it helpful, I’d go straight into reading The Unicorn Project, which is the successor, but from a developer's point of view.

The Unicorn Project is the organizational civil-war novel that every technology and business trailblazer must read.

Christopher O’Malley

Growing Object-Oriented Software guided by Tests (GOOS)

I read this book long after working with ThoughtWorks (TW), a software consultancy company. For many years they worked with the company I was in, coaching us on how to write software the modern way.

Before this, I was mainly a solo developer, bashing code out and copying it to prod on small websites. TW taught us how to work on big projects, which requires you to work entirely differently due to the different challenges working as scale brings.

Reading GOOS, it’s apparent that it’s a blueprint for what TW taught me and what I have been coaching my team.

It would be a long post if I were to talk about everything they introduced, but some core values were:

Working on big things is hard. So cut scope, break the problem down and work iteratively.
TDD
Fast feedback loops
Pair programming
Trunk-based development
Heavy emphasis on automation

Yes, it’s a book about TDD, but it feels more significant than that. It’s a practical book showing how to build a real, complicated thing in what I would consider the right way. Many people lament that too many TDD articles are simplistic and not “real world”. The answer to that whining is this book.

Refactoring (2nd ed)

I have no evidence for this, but anecdotally, most developers don’t know how to refactor very well. Many don’t even know what refactoring is. Most aren’t refactoring anywhere near enough.

The good news is, if you study this relatively short manual on refactoring, you’ll level yourself up as a developer. Once you understand what refactoring is and practice what is explained in this book regularly, you’ll improve the code you work on. You’ll increase your and your team’s productivity, and refactoring will reveal better designs for your system.

If you’re a developer who claims, "I don’t have time to refactor", you’re wrong. Your definition of refactoring is likely incorrect, and you certainly don’t know how to practice it effectively.

Honorable mentions

These were books that were very influential in my development. However, I recognise my juniors are very time-starved and suffer from vast information overload. These books are great, but they should probably only be picked up if they catch the individual’s eye.

The C Programming Language

This may be a function of sentimentality, as C was the first language I properly studied.

It’s a wonderfully short book and easy to read, but at the same time, it goes deep into how your computer works.

The big thing you’ll have to overcome is the concept of variables that are the actual data and variables that point to data. This requires you to leap across a somewhat abstract thinking barrier, but it’s essential to make it, and it’s applicable outside of C.

Functional Programming in Scala

Writing your abstract data types like Option, List etc., will help you realise that a lot of functional programming (FP) is not all that complicated. It’s a fun book and taught me much of my FP knowledge.

Practical Object-Oriented Design in Ruby (POODR)

Sandi Metz teaches what it means to be a good developer from a technical and behavioural standpoint.

We are similar in respect of not tolerating bad excuses for poorly written code. Her discussion about the importance of exemplary code has stuck with me.

She shows you how to write good, object-oriented code throughout the book. Like refactoring, there’s a lot of talk about OO from thought leaders on the internet, but most of the time, it’s clear that the person’s idea of what OO is wildly different from what is written by experts, like Sandi.

OO gets a lot of negative press these days, mainly due to the faddy nature of our industry.

Function good, class bad.

This book is relatively short and gives you an excellent intro to OO.

Don’t Make Me Think

A common sense approach to web usability

If you wish to describe yourself as “full stack”, you should acknowledge that you won’t be an expert at everything but should appreciate all aspects of your craft.

This book will help improve your empathy for users and give you some practical ways to ensure the website you’re making works well.

In this second edition, Steve Krug adds essential ammunition for those whose bosses, clients, stakeholders, and marketing managers insist on doing the wrong thing. If you design, write, program, own, or manage Web sites, you must read this book."

Jeffrey Zeldman

Release It!

I read the first edition of this book a while ago; a 2nd edition was released in 2018.

For someone who had previously been manually copying code into production without a second thought, it taught me what it takes to release and operate systems at scale.

Writing code is one thing, but creating sustainable and straightforward systems is a skill that needs to put more effort into learning and improving.

It introduced me to the circuit breaker pattern, which is a pattern for isolating failures. It keeps small failures small rather than escalating and taking down your entire system. You'll find this pattern in most modern, web-scale systems, and often now it's built into things like Istio

There are many more gems like this; it's worth reading.

Interview questions as a candidate

Chris James — Fri, 04 Jun 2021 00:00:00 +0000

No matter what my role is within a company, I want to work in an empowered, trusted, team that works together to deliver things we can all be proud of.

Old-fashioned “command and control” hierarchies or completely misunderstood applications of agile have caused misery for people working under them, and do not result in successful products. I want to avoid working in those kinds of environments.

These questions won’t always be relevant for every company. A 2-person startup probably doesn’t have the notion of a product owner (the 2 people are implicitly the POs!) but I feel the principle of most of these questions stand.

Who decides priority of work?

Ideal answer

A product owner, guided by research and UX practitioners, assigns hypothetical value to stories — but the team decides what to work on, so they can decide how best to deliver value. That might include playing spikes, addressing technical debt etc.

It should be a collaborative relationship between product people and engineers.

Warning signs

Teams with no autonomy eventually fail. Software engineers are paid the mega-bucks to solve problems, not blindly work through tickets. Without autonomy the wrong things are prioritised, productivity tumbles and systems will fall apart.

PO as manager
Feature factories
Spiralling technical debt
Bad relationship between engineering and product

How are users’ problems solved?

Ideal answer

The end user is at the centre of what the team does. User stories are user stories, a brief amount of text trying to convey what problems the user is trying to solve and why.

People from all disciplines get together, discuss it, figure out what needs to be done to try to solve the problem and figuring out ways to know if we’ve succeeded. Ideas include,

UX team reaching out to users for interviews
Developers creating cheap prototypes along with thinking about possible long-term solutions
The team going back and admitting that they don’t understand the problem rather than just blindly building something
A story typically is not finished with a release of software. Many iterations are made to gather feedback to learn how to solve the problem better for the user.

This style of working gets everyone invested in the project and feeling involved.

This environment is where real innovation comes from because smart people have a better understanding of the domain and have the space to solve real problems.

Warning signs

A select group of individuals looks at user’s problems and generate numerous tickets in JIRA for technical people to administer. Developers have little to no idea the purpose of the work and cannot offer feedback on what is being worked on.

Story robots
Feature factories
Bad products

The tickets have very little resemblance to stories and are instead barely validated, overly specific, solutionised tasks for developers to work on.

Work “happens”, arbitrary targets met, but both users and development teams are not happy.

What’s the general makeup of experience? Do you hire many juniors? How heavily white and male is your engineering department?

Ideal answer

Hiring from a diverse range of experience gives us a diverse team, which is a good thing in terms of productivity, environment and being the right thing to do.

Seniors should feel proud to help less experienced people grow, and the juniors help challenge assumptions.

An environment of continuous learning with coaching and mentoring is just nice.

Often people who have worked for less than <5 years have had a bigger impact on me and the teams I’ve worked in than the engineers with 10 years.

Warning signs

Don’t have time to train people. What else do they not have time for? We all need training and space to think regardless of experience.

I want to work in an environment that encourages and supports people from any background. If your engineering teams do not have any female leaders for instance, that is a bad sign.

How often do you release? Who releases?

Ideal answer

Feedback loops are so important in software and product development. Whether its feedback given by customers, or realising something you’ve shipped has introduced a bug; the sooner you get this feedback, the cheaper and easier it will be to act on it.

Developers should be in charge of releasing code, and because of that they’re forced to write good automated tests and having excellent operability. They should be focused on optimising for a reliable, quick release pipeline so they can make changes to the software frequently and safely.

Product owners can be in charge of features becoming visible to users if that’s needed (e.g., use feature flags), but copying code from one server to another is a technical concern, not a product one.

You are continuously delivering code to production on green builds. This question will also tell me about your testing approach (or lack of!).

Warning signs

Manual testing

Points to a lack of thought as to how to deploy software safely.
Usually coincides with poor engineering, lack of automated tests begets lack of high internal quality.
Doesn’t scale.

Gatekeeping (release managers, or some kind of sign-off step):

Lack of trust.
Empire building.

Lack of DevOps culture

If you have a “DevOps team” who are in charge of releasing software then I suggest you read a book about DevOps because you have missed the point. Focus on the principles, not the tools.

Do you have delivery / project managers? What do they do?

Ideal answer

Project managers (PMs) can be extremely useful with coordination across bigger sets of people, helping to understand X team dependencies. They often do a lot of “glue work” for the team to help them work more effectively.

Warning signs

"Striving to ensure that no resource be idle is the biggest generator of waste." —Eli Goldratt

PMs are incentivised to “maximise developer output” and the company has no idea what the value of slack is. If your team is always running at 100% capacity, how do you expect it to improve? Hint, repeatedly cracking a whip will not improve your team.

Slack is critical and is widely written about. It’s weirdly ignored by a lot of project managers. Teams running at full capacity all the time are brittle.

“Projects” are poison to good software. What happens after your deadline?

Implies no feedback loop with customers, just ship stuff for hitting deadlines sake rather than doing something useful.
Bad incentives. Celebrate hitting deadlines but won’t talk about whether the product is actually worthwhile.

If they dictate priority they’re just as bad as POs as bosses.

Do you have QAs? What do they do?

Ideal answer

If you do, they are there to act as coaches in respect to how to build high-quality systems (think testing, operability, etc). They also might be individuals who have time and space to do exploratory testing to find new insights about the system.

QA should be a responsibility shared by the team, not a role.

Warning signs

Separate individuals writing tests
Gatekeepers
Army of manual testing which leads to slow releases, slow feedback loops, etc

Do you have UX engineers? What do they do?

Ideal answer

UX are researchers and educators.

They reach out to users and learn about their needs, often they’ll bring other disciplines such as engineers with them, so they also gain first-hand appreciation of the problems you’re trying to solve.

From there they work with the team to figure out how to solve these problems.

You’ll often find the UX people sat with engineers scribbling on paper to agree upon cheap experiments to try to understand these problems.

The UX team revel in learning, striving to learn more about users and then documenting and presenting their findings.

Warning signs

The discipline of UX is conflated heavily with visual design.

Rather than actually thinking about the user’s problems they are trying to solve, they spend more time worrying about making pixel-perfect designs for the product owners to rubber-stamp. This is a real waste of talent and energy, they’re UX experts to serve the users, not the whims of POs.

This way of working leads to costly feedback loops where engineers have to implement these pixel-perfect solutions only to find they are impractical in some way, or don’t actually meet the user’s needs.

Often they seem not attached to the software team, rarely collaborating or learning with the team. Instead, they’re working very far in to the future making mockups for software you may not even write.

All of this looks like siloed work in a waterfall project.

How do you validate your work? What makes you decide to start initiatives? What makes you end initiatives?

Ideal answer

You have a vision to guide your company but acknowledge there’s lots to learn. Mistakes are fine so long as you learn from them.

You have a learning environment, you’re always pushing for faster more effective feedback loops. A part of this is trying to find ways to cheaply validate ideas, and you lean on your software development teams to help you do this.

You’re not afraid to cancel an initiative or change it radically if you need to. Plans are changing all the time, driven by these learnings.

Empowered and involved teams bring ideas that are also prioritised

Good ideas don’t arrive during quarterly planning

Customers are involved and we work with them closely to help build a great product.

Warning signs

Immutable product roadmaps
Long backlogs
Optimising for predictability over good products
Development team feeling disempowered and uninspired
Lack of feedback loops
Lack of end-user involvement, if at all.

Can we stop selling people things that aren't possible to deliver? Please? Users and customers need to be involved in development of their software. If you don't want to be involved, you can't complain when the resulting software doesn't work for you.

https://twitter.com/ChristinGorman

Do you have fixed deadlines?

Ideal answer

Obviously, we don’t have infinite money but working to a strict deadline implies a lack of feedback loops.

We prefer to keep iterating quickly and keep gathering feedback. If we learn that what we’re doing is useful, we’ll keep investing until we (and the customers) are happy with it. Has the Twitter “project” finished yet?

Rather than deadlines instead think:

How much of our budget are we willing to spend on this idea?

Warning signs

Whilst a bit of pressure is good, tight deadlines are toxic. It usually results in teams rushing out poor quality systems that are difficult to maintain and because of the fixed nature of work the technical debt will remain unaddressed.

Inevitably the system needs changing at some point (by another absurd deadline, again), repeat cycle of doom forever. Engineers get further blame as things become more and more difficult as technical debt spirals out of control.

Who is supporting the software after you release it?
Is it really worthwhile to build if you’re not going to change it to adapting customer needs? Again, is the Twitter project finished? Is the Facebook project finished?
Hitting deadlines is not a goal in itself. Too many companies fetishise hitting deadlines (which are often totally arbitrary ) over building good products and having healthy, meaningfully productive teams.
If you use tight deadlines as “motivation”, you have big problems. Why aren’t people intrinsically motivated at all? Why can’t you treat your colleagues as adults and trust them? Can the management seriously not think of other ways to motivate people?

Do you have 10% time?

Ideal answer

Yes, we give space for engineers to learn because we recognise that self-improvement is vital to the success of the business.

In addition, we believe it helps with diversity. A young white male with no responsibilities can spend his free time learning the latest, greatest programming languages, but that’s not going to be true for a single mother.

Warning signs

Lack of time? Again, why are projects so poorly managed that there’s no time for the team to improve?

How do you expect the team to improve and be able to hit your aggressive deadlines if they’re always working at full capacity? You should view your development teams as important assets that you want to invest in.

Who is in charge of process? Is there a standard way across teams?

Ideal answer

The team(s) are in charge of their process. Teams are groups of different individuals working on different problems and if they have a positive, learning-led culture they will retrospect and refine the way they work.

They are trusted and incentivised to find better ways of working. Kaizen is at the heart of every team’s culture and a result of that is inevitably teams will work differently.

Some standardisation is helpful though, maybe every team is obliged to showcase their work every week for instance. However, standardisation never trumps a team’s ability to improve the way it works.

Warning signs

An answer of “Yes there is a standard process”. Teams are different and need to be empowered to improve their working practices, or they wont be able to adapt to changing circumstances.

Whilst developers being able to move between teams is nice, prioritising it over autonomy will be extremely costly. Think about what a standard process involves. There has to be some kind of governance for it. If a team needs to change the process to work more effectively, how time-consuming will that change take and how chaotic will that be for the other teams?

It’s pretty laughable how many companies will blindly follow ceremonies like retrospectives and simultaneously not allow them to act on whatever they uncover because of the process.

Quickfire questions about managers

Do they enjoy the job? Or were they pressed into it. What do they like?
Do they have access to training? Management isn’t an inherent skill, it is something that needs to be studied and practiced.
How often do they have 1 to 1s, skip level meetings?
Does your manager care about you? Have they helped move your career forward?
Has your manager said no to you?
Can your manager actually make decisions? Or are they just a communication funnel?
How do you grow your reports?
Why did you become a manager?
When has a manager batted for you?
Has your manager ever given you hard to hear feedback?
Talk about a team member’s growth they are proud of
Talk about when they got negative feedback from someone on their team, and how did they deal with it?
What’s the biggest challenge your team faces, and what are you doing about it?

What if you answered a lot of the questions above “wrong”?

Does your organisation have the maturity and honesty to see its own short-comings? How willing is it to change? Who is sponsoring this change? Are they actually in charge, can they make decisions?

I’ve often found there are numerous people who are asking the right questions, but they’re at the wrong level at the organisation. They’re often fighting bad incentive structures. For instance, when an organisation hires and incentivises people to prioritise hitting arbitrary deadlines at the cost of everything else it’s going to be very hard to improve things.

I’ve worked in environments that have answered many of these questions poorly, but they have a good enough environment and autonomy to let us all make positive changes. So, I’m not looking for 10/10 but a willingness to improve.

Being involved with teams that have driven positive change have been the source of a lot of pride for me. The nature of our industry means all teams/orgs have to constantly adapt to deal with new circumstances. The open-minded, honest, diverse organisations are the ones that are best equipped for this reality.

The Why of TDD

Chris James — Fri, 28 May 2021 00:00:00 +0000

Many people have false starts with test-driven development (TDD). When I discuss it with people, many have some misunderstandings around the method, in particular the motivations behind it.

I believe if developers have a better understanding of why to use it, they’ll have a better chance of practicing TDD more effectively and get more value out of it.

This posts explores the reasons for using TDD and offers some guidance as to how to practice it more effectively.

I was invited to speak about this topic at the GoSG, a meet-up for the Go programming enthusiasts in Singapore. You can find the recording on YouTube.

This post quotes Growing Object-Oriented Software Guided by Tests (GOOS) (written by Steve Freeman and Nat Pryce) extensively as it aligns very closely with how I was taught TDD. I strongly suggest grabbing a copy and giving it a read. Ignore that the examples are in Java, the main lessons from the book are broadly language-agnostic.

Introduction

If you’re new to software development, there’s several knowledge hurdles you need to leap over. The one you’ll be judged on the most is your ability to use the tools in front of you to get stuff working and get it to the customers. How well you do this and how delightful the result can be referred to as external quality.

Once you get proficient at this, the main challenges facing mid-level and above are to do with internal quality. This focuses on your ability to work on and design systems so that the cost of change is reasonable. This is important because if a system is useful, it will have to change over its lifetime.

When talking to non-technical folk about this, you can use an analogy of the quality of a factory, and its equipment. If parts of your machinery are unreliable or hard to deal with, the factory will begin to grind to a halt. This is also true of software. If internal quality declines, the ability to affect the external quality of the software will decline too.

There are a lot of software developers out there who can bash the code out enough to make useful software. A problem in our industry is we’re often incentivised and praised by our ability to work through “tickets” quickly and trying to hit (often arbitrary) deadlines. Frequently though, the internal quality gets neglected, and we hear these stories over and over again:

Why is the estimate for changing the system 3 weeks? It should be 3 days!

We need to spend a month re-writing the Foobar module.

I can’t stand to work on this legacy system again. Please let me move to a different project, or I’ll find somewhere else to work.

There are fewer developers who can write software that can live for many years without painful re-writes and staggering maintenance costs. The reason simply is, it’s really hard. It requires effort and study. I’ve been writing software for over 15 years and I still find it hard and feel the need to keep studying and reflecting.

There are developers and stakeholders who have advanced skills in justifying reasons for not caring about internal quality. They’ll say things like:

The only thing that matters is working software

What about time to market?

All worthwhile systems have to be changed (see Lehman’s laws of software evolution). “All that matters is working software” can’t be used as an excuse for poor internal quality today when you will have to change the external quality tomorrow. People tend to under-estimate the costs of poor internal quality and over-estimate the cost of keeping internal quality high.

You need to be thinking about the design of your system because the problem you are solving is different to everyone else’s. Domain knowledge (or lack-of) has a giant impact on what designs are suitable, so no single blog post can tell you how to design your system. It’s your responsibility as an engineer.

Many developers work in environments which have a real fear around speed of delivery, and you can get tempted in to short-term promises of productivity. A new programming language! A shiny new framework. Even folder structures in the Go community seem to be endlessly discussed as some kind of silver-bullet to good software. Chasing these things is a distraction and is not equipping you to strengthen your skills at writing software with high internal quality, you’re just learning another thing that will be unfashionable in a few years.

This is why I think TDD is such a valuable method. It gives you a structured way to confront and think about the internal quality of your system. TDD is a skill which can work in any context, even on Mars. It is an excellent tool for training yourself and your team to design better software.

TDD can be seen as a developer frivolity that slows teams down. This is true when you are first learning it (like any skill) but with practice and training, you’ll find yourself working at a sustainable pace and consistently delivering value. High internal quality will facilitate improvements to external quality.

Adam Luzsi points out:

… most people who start to learn TDD, are also in the process of learning software design implicitly through TDD. And this causes the feeling of "this slows me down".

High internal quality is not exclusive to practitioners of TDD but if you’re on the fence about TDD I hope this post encourages you to take another look. Maybe at least make you reflect on your own methods for writing great software.

What is TDD?

Test-driven development is conceptually simple to follow:

Write a test for a small, desired behaviour
See it fail
Write just enough code to make it pass
Refactor
Go to 1 for next increment of behaviour

Yet, there appears to be many developers diverging a lot from what was described by Kent Beck and others.

This inconsistency makes discussing the merits of TDD difficult because every so often what someone believed to be a typical TDD process seemed to be following a TDD-like process but getting key parts of it wrong.

Why does this happen? I think tech in general suffers a problem of being extremely faddy and people not really understanding why they are adopting a particular framework or technique. If you don’t understand what TDD is for, you’re unlikely to practice it effectively.

So, why test-driven development?

Here are the typical answers you’ll get:

Prevent defects
Good test coverage
Confidence to refactor

Rightly some will argue:

These benefits are not unique to writing tests first

Why write tests first?

It’s worth exploring the why of TDD in detail as I believe it helps clarify how to use the method effectively. Many people have false starts with TDD.

GOOS picks up on this:

“We’ve seen teams that write tests and code at about the same time where the code is a mess and the tests just raise the cost of maintenance. They’d made a start but hadn’t yet learned that the trick … to let the tests guide development.”

You have to know what behaviour you’re trying to build to write a test

Developers go too fast, diving in to code and letting their imaginations run wild.

Jo Crossick on Twitter

When I write tests first, it's because I know that I'm itching to write over-complicated, over-ambitious code that is destined to become a hot mess, and tests is the only thing that can hold me back from creating instant technical debt

A lack of method to the design process can lead to software designed against loose, ill-defined requirements. This design can quickly become a drag on the team’s productivity. It can confuse developers, baking-in to your system assumptions that are untrue and behaviours that aren’t actually needed.

Writing a test first demands you precisely define, and focus on what you’re trying to achieve; this is a key part of the design process

Writing a test first means you have to:

remove any ambiguity, meaning you have to really understand the behaviour you need
cut down the scope to a small piece of useful functionality, liberating you from having to design the whole system

Rather than having to be somehow clever enough to know all the requirements the system would ever need and committing to a design, instead you design code for behaviour you have precisely defined. You design, for one thing at a time, and you do it properly.

It offers a fast and continuous feedback loop on your design

Writing a test gives you a sense of the design of a particular unit at the beginning. Many developers will dive in, creating types and abstractions for hours on end and only when they plug it all together and run their application do they begin to see the flaws in their design. Until code is executed and the behaviours observed, the design has had no real validation.

With TDD you get feedback at the start and as you iterate through more behaviour you will keep getting guidance from your tests. There is no faster way of getting feedback on your design than by declaring what success looks like at the start.

You will have a sense of progress as you take small, validated steps toward your goal. If practiced well, you won’t end up in situations of having to throw hours or days worth of code away.

Whilst not being unique to test-first, tests reflect your design back at you. If a test is difficult to read or write, it is telling you something. A key skill for practicing TDD well is to listen to your tests.

It gives you momentum and by focusing on internal quality will help you continue to change your software cheaply.

Many developers “spin many plates”, darting from one part of the code to the other, changing different bits of code and in general working in a fairly chaotic way. This way of working makes life difficult and more prone to error and re-work.

Writing a test first acts as a stabiliser. This sets you a clear, short-term objective and lets you focus on doing one thing well, rather than worrying about competing concerns.

The refactoring step improves your chances of maintaining good internal quality which is essential for a system to be malleable. We’ve all worked on systems where the internal quality is terrible and how frustrating it is how difficult it can be to make a seemingly simple change.

The iterative nature of TDD means you keep working on achieving one small improvement at a time.

It does not slow you down

https://www.geepawhill.org/2018/04/14/tdd-the-lump-of-coding-fallacy/

GeePaw has an excellent video which dispels the idea that TDD slows you down. In fact, it's the opposite when you forget about the silly idea that software development's bottleneck is simply typing code. Thinking is the bottleneck, we work in a knowledge-based field and TDD offers us a way to think with clarity.

It’s true that when you first practice TDD you will go slower than before. In cases where maybe your design skills aren’t as strong as you think, TDD may feel frustrating, but you can use it as a tool to improve your ability to write modular code.

It’s a skill that has to be learned and practiced but with time you’ll find yourself working quicker, confidently and with more focus.

The qualities required to make code unit-testable are good qualities in themselves

GOOS:

“…the effort of writing a test first also gives us rapid feedback about the quality of our design ideas—that making code accessible for testing often drives it towards being cleaner and more modular.”

Modular code is:

Easy to repurpose and re-use within different contexts
Focused
Simple to test
Small, with clear responsibility
Easy to understand

Many developers have a healthy skepticism to premature abstraction, but it can go too far.

It is not good to have that what and the how of code mixed up within a function featuring 100s of lines of code, with mixed up and complected concerns. Rich Hickey covers the topic of easiness vs simplicity in his incredible talk Simple Made Easy. Having all your code in one method is “easy”, but it is not simple.

Systems built up of modular, cohesive units are simpler to understand and easier to change.

It’s easier to write good tests beforehand

Writing tests after the code is written is challenging to get right. You can end up with tests that may not fail when you want them to, have cryptic error messages or simply not have good coverage. You’ll also fall in to the trap of testing your design, rather than the behaviour (more on this later).

Writing tests first and seeing them fail helps you validate their usefulness and helps you focus on testing from the consumer’s point of view, rather than the person who wrote it.

It requires you to constantly refactor

A key, but often skipped step by novices is the refactoring step. It is far easier and more sustainable to do frequent, small refactors around the code you’re actually working on throughout the day, rather than leaving it for “a refactoring/technical-debt sprint”.

Ill-factored code makes the system harder to re-design. Often the act of refactoring reveals designs. As you consolidate code and DRY things up you start to “see” abstractions that can simplify matters.

Digression: Refactoring sprints are an absurd idea and point to some real team disfunction

You want the benefits of increased internal quality now, not in 3 weeks time.
Everyone knows they are hard to do and frequently fail to give the pay-off you’d hope for.
It’s your job to maintain internal quality because that’s what helps you deliver external quality at a reasonable cost. Don’t ask for permission to do your job.

How to use TDD

Work iteratively, in small positive steps

Working on a single, small, well-defined thing is simple, and you’re more likely to succeed. You’ll get fast feedback on whether you’re going in the right direction.
Working on many, large, loosely defined things at once is difficult, and you’re more likely to make mistakes. The feedback loop is slow.

When practicing TDD you should always be thinking about how to reach the vague end-state with specific, achievable steps. If you’re writing a unit test that you know will take hours of effort to make it pass, you need to rethink your approach.

It can be challenging to break work down into small steps, and it will also challenge your design skills but working this way means you are more likely to succeed, and you’ll reduce waste.

When TDD feels right

Constant, small progress throughout the day. Never feels slow, feels deliberate.
Frequent commits to source control per hour.
Work feels “safe”, less chaotic than maybe you’re used to.
Mistakes are rarely large. Happy to revert changes if you realise you’ve gone down a wrong path.

Writing the test first

There’s a difference between knowing to write a test first and doing it in a way that helps you design your software.

GOOS:

“We want each test to be as clear as possible an expression of the behaviour to be performed by the system or object. While writing the test, we ignore the fact that the test won’t run, or even compile, and just concentrate on its text”

Rather than worrying about how the code will work at this point, concentrate on the behaviour, particularly from the user’s point of view. It’s essential you block out implementation detail from your head at this point in the process because you’re just trying to nail down exactly what you want to achieve.

The what and the how are two separate design activities which are too often mixed up, which makes coherent decision-making more difficult than it needs to be.

Focus on being precise and limited in scope, this will help you uncover assumptions and help you keep focused.

TDD is an iterative approach to development. When you are writing your tests you should be mindful of how to keep the work moving with fast feedback loops and small positive steps.

Start from the top down for new features

TDD is focused on letting you design for the behaviour you precisely need, iteratively. When starting on a new area you need to identify a key, important behaviour and then aggressively cut scope.

From there you want to take a “top down” approach, starting with an acceptance test (AT) that exercises the behaviour from the outside. This will act as a north-star for your efforts. All you should be focused on is making that test pass. This test will likely be failing for a while whilst you develop enough code to make it pass.

Once you have your AT set up you can then break into the TDD process to drive out enough units to make the AT pass. The trick is to not worry too much about design at this point, just get enough code to make the AT pass because at this point you’re still learning and exploring the problem.

Taking this first step is often bigger than you think, setting up web-servers, routing, configuration e.t.c, which is why keeping the scope of the work small is important. We want to make that first positive step on our blank canvas, have it backed by a passing AT so that we can then continue to iterate quickly and safely.

As you develop, listen to your tests, and they should give you signals to help you push your design in a better direction but again, anchored to the behaviour rather than our imagination.

Typically, your first “unit” that does the hard work to make the AT pass will grow too big to be comfortable, even for this small amount of behaviour. This is when you can start thinking about how to break the problem down and introduce new collaborators.

This is where test-doubles are very useful because most of the complexity that lives internally within software doesn’t usually reside in implementation detail, but “between” the units and how they interact with each other.

/*
The design "outside the braces" is what often has 
the biggest impact on internal quality
*/

func NewStockCheckHandler(
    stockChecker StockChecker, 
    productStore ProductStore,
) http.Handler {

/* 
implementation detail inside the braces is usually 
low thrills and simple to test
*/

}

We can use test-doubles to explore the interactions between our collaborators with a much tighter feedback loop than if we tried to design and develop units isolated and then integrate them later.

For further iterations we may not need to write a new AT, we can simply add new behaviours to our existing units at a faster pace than the first one. We’ve made our first step, and we’re confident that we’re not developing them blindly because the units are integrated with the rest of the system.

On “TDD-ing a design”

Too often people assert that TDD is only compatible with a “bottom-up” approach where you imagine a bunch of abstractions up-front and then you “TDD them”.

This is called “TDD-ing a design”. You’re not using TDD as a design method, merely a way of writing tests around some code. This often results in classes or functions being overly developed in isolation for extended periods and then only when you try to integrate them into the system and use them you realise the design is wrong.

This is the point where people declare TDD to be a poor design tool when in practice, they already had a design in their head and just added a very long feedback loop to it, exhaustively testing code and various edge-cases before using it in their actual system.

Remember that TDD demands we start with a test for a desired behaviour, not a design. This is very much “top down” development. When done correctly you stop “over-designing” and you write and design the code you actually need.

By TDD-ing a design you can fall into the trap of tests being tightly coupled to implementation rather than behaviour. This becomes extremely problematic when you want to change your implementation detail and your tests will become a burden. I talk about this in detail in “The Tests Talk”.

Listen to your tests

If you follow the TDD process well you should end up with modular, loosely coupled, cohesive units within your system. This journey is rarely smooth. As requirements and knowledge changes your design will have to evolve with it.

Tests can offer you a focused lens into areas in your system where you can view the impact of your design isolated, which makes it far easier to appraise. Tests reflect your design back at you.

Here are some common test smells to look out for. Commit them to memory, and you’ll be able to raise some interesting questions about your design to answer.

High number of test-doubles (mocks), complicated setup

This is telling you that the unit under test has to collaborate with many things, this pain in your test world will be multiplied in all the cases you want to use this unit.

This pain can sometimes feel hidden in the “real” usages but comes apparent when you start to change things and feel the pain of the tight and inappropriate coupling. You should be asking yourself why this unit has to collaborate with so many other things.

It points to leaky abstractions which could be remedied by:

Look at the test doubles. Are you having to mock out lots of methods? Perhaps you can consolidate this interaction into another more cohesive unit
Maybe the unit itself has too many, unrelated behaviours. See if you can come up with a more cohesive design.

Unclear intent

Does your test really explain the code’s behaviour and why?

GOOS:

A test called testBidAccepted() tells us what it does, but not what it’s for.

If the test is unclear it could mean the responsibilities of the unit being tested is unclear and warrants a redesign.

Wide scope

This points to a lack of focus. Perhaps you need to break the problem down in to smaller parts.

Value the quality of your tests

Start being more attentive about the quality of your test code. This will result in you writing tests that are cheaper to maintain and will challenge you to write more focused, modular code resulting a better designed system.

Writing high-quality tests is cheaper than maintaining poorly written ones

In a high-performing team where it must respond to client needs quickly and cheaply, good tests will enable them to have a productive experience. The opposite is also true.

Write the tests you want to see

The easiest, most generic feedback I give to my colleagues is asking them to explain their tests out-loud. Often it’s harder than they imagine, even when the tests appear terse; and then an interesting conversation about the design almost always follows.

Reflect on the tests you have now. Do they express the intent of the behaviour well? Do they describe the why clearly? If not, what would it take? Is it a simple refactoring of your tests or does it require some redesign?

Be attentive when you watch the test fail

Be sure it fails the way you expect and that it fails with a clear message. If you skip this step you may be glossing over design decisions, or perhaps you don’t understand the requirements clearly enough.

Usually making the test fails “nicely” is a multi-step process, but it’s important to go through the steps until you get a satisfactory failure message due to your code not exhibiting the failure you need.

View being in “the red” as something to get out of as soon as possible

You should be running your unit tests extremely frequently, which is why it’s important they run quickly and don’t depend on slow things like the network or file-system.

If at any point your tests are failing (so you’re in the red), you should be thinking about the shortest path to get back to “green”.

Often the safest way to do this is to revert your changes. By working in small increments this should never feel like a big deal.

It is forbidden to refactor or redesign code in the red state because your feedback loop as to whether the changes are valid is broken.

Commit your work frequently

To be able to use your tests to help you guide your design they need to be green, but it’s inevitable that sometimes you’ll go down the wrong path and want to get back to safety. It’s therefore critical that you practice good source control practices and commit your work frequently.

I usually commit my work once I’ve made the test pass and that means if I get into a bad state during refactoring I can easily get back to working software again.

So much of applying TDD is working in small, safe steps, so your mind is freer to think about the design of your system rather than juggling multiple concerns at once.

Refactor aggressively when in the green

Most of the time your system should be in a state when you can refactor freely.

Ill-factored code is problematic because it becomes hard to understand, which makes it harder to re-design. The process of refactoring in general will reveal improvements in the design for you.

Read Martin Fowler’s refactoring book and get in to the habit of applying common refactors. A lot of code I observe is not well-factored, but the good news is it’s a relatively easy skill to learn and once you’re proficient takes a matter of seconds or minutes to perform; in particular with help from your IDE.

Wrapping up

If you don’t practice TDD, what is your method for designing software? You do you! Can you describe your method to someone else, so they can follow a similar approach? It’s healthy to retrospect on how you work, to see if there are ways to improve.

No matter your design method, internal quality is essential for the productivity of your team. So many systems suffer from poor productivity and high costs because some developers and teams will neglect internal quality. It’s a false-economy to ignore internal quality and the pace at which you will be able to affect the external quality of the system will decrease quickly if you do.

It is of course possible to have software with high internal quality without practicing TDD, but I am not especially clever and want an easy life.

When I design up front:

it’s often wrong
I have to think about too many things at once
I design for things I don’t need

TDD offers a methodical, incremental way of designing code which feels like a simpler, more evidence-based way of creating useful, malleable software.

Stick to the simple TDD process, listen to your tests and practice. This should result in you designing software, verified against concrete desired behaviours; rather than your imagination.

My experience of Introducing a team to Go

Chris James — Fri, 16 Oct 2020 00:00:00 +0000

I was lucky enough to be invited on to the Go Time podcast to talk about Introducing your team to Go. It was my first time on a podcast and whilst I was a bit nervous and waffled a little, it was a lot of fun because Jon and Mat made me feel very welcome and comfortable.

Introducing a new technology to an organisation is not only about technical merit. Even if it were, we’re all terrible objective judges of how good or bad programming languages are.

Change represents opportunity but also risk and if you want to succeed you need to build trust and consensus.

This requires you to find and influence the right people so you can get help, act with empathy to understand the concerns of a diverse range of roles and people, and work hard to bring people on the journey.

Background

Around 4 years ago I joined a new company. Before I joined they had experimented with a few technologies and then seemed to settle on writing everything in JavaScript.

I’m very keen on Go and felt it would be a far simpler and more effective tool for a number of problems that we had to face. What follows is my story around bringing Go into the company, what worked, and what didn’t.

Bringing it up, badly

A few months in to my new job we were starting a new project so we were in an inception. An inception is where various stakeholders and disciplines around building products get together to try and figure out on a high-level what we’re going to build and discuss a little how.

During the architectural session I sensed an opportunity to stick my oar in and suggest Go for one of the systems that was being proposed.

This was shut down, very firmly by a project manager. Paraphrased:

We write JavaScript here. Using the same languages and tools lets us be efficient. We spent a lot of time experimenting with languages and now we have to focus. We won’t be using any new languages.

At the time I did not take this well. I bit my tongue but inside I was thinking

Who is this PM to decide how we should write software? Stay in your lane. I know I’m right here.

In truth, it was poor judgement on my part to bring it up. It wasn’t the right forum as many people would just see this as me being technically frivolous and I didn’t think hard enough about the motivations of the people around me.

PMs are incentivised and judged on how effectively they deliver projects. They will of course be very conservative and resistant to anything that would add risk to a project.

Lessons learned

Be aware of the room

Do you need to bring up contentious subjects up with all of these people? Do you think it’s likely that people will agree?
Think about the motivations of the people you’re talking to and tailor your message accordingly.

Chatting with the right people

Hopefully you’re not naive to think that you can merely suggest an unfamiliar language and everyone jumps on board. If one of my engineers said to me

Hey CJ can we write this in OCaml ?

It would be pretty irresponsible for me to go

Why yes, let’s go for it. More fodder for my CV!

You need to build confidence within your organisation that you’re not going to build some systems that only a couple of people are able or willing to maintain.

You’ll also need a critical mass of people who are skilled and enthusiastic about the language, with some real working software already running on your platform to prove it’s viable to work with.

Influence the right people

If you’re the only one pushing an idea you’re likely to fail. You’ll need to get to know your colleagues, try and identify the ones that are able to hold a captive audience and influence people. If you can convince the energetic influencers half the battle is won and you’ll have other people to help you.

Speak positively about Go, rather than criticising whatever is used now

It’s not productive to harshly judge the current tools. At best you’ll get agreement but at worse you’ll

Insult previous decisions made, which is especially bad when you’re unlikely to understand the context behind them.
Alienate yourself from people who do like whatever it is you’re criticising.
Come off as a jerk and get into pointless arguments.

If you like Go so much, it should be enough to speak positively about it but try to be realistic about it. Yes, Go does have downsides and it isn’t going to make your company 10x more productive overnight, don’t insult the intelligence of the people you’re talking to.

To influence you need to act with empathy.

You won’t convince everyone

Often developers can be inflexible about their favoured tools. Don’t waste too much time on these scenarios because they are a path to negativity and conflict. Usually there’s enough space for some people who like X and people who like Y.

Getting organisational confidence

Hopefully you’ve got a number of people with at least some interest in Go and perhaps 2 or 3 people who you can rely on to help you push forward.

Whoever is in charge should be thinking about the following when looking to adopt a new tool

Do my engineers want to write it?
Are they able to write good, production-ready systems that they are happy to support?
Does it fit into our existing infrastructure and tooling?
Can we hire for it?

How did we address these concerns?

Book club, and learning at people’s pace

We thought we could tackle this lunch & learn style where once a week we had our lunch in a meeting room and invite people to learn some Go.

We started by reading through The Go Programming Language as a group. We hoped to learn a concept together and then set a “homework” of completing some exercises at the end of each chapter.

What actually happened, and what was learned

We had a diverse group, it ranged from engineers with over 10 years experience to only just out of a bootcamp. One major stumbling block was the idea of using a statically typed language. A number of our engineers had only ever worked with Ruby and JavaScript so this was quite a strange thing to get over at first.

You can demonstrate the strengths of static typing quite easily by showing how it stops you passing a string to a function that requires an int. I also related production issues we had with our JavaScript code which would be impossible in Go. It is an act of faith though, if you haven’t got a lot of experience building larger systems it feels like needless ceremony.

The trouble you may run in to is you have to go at a pace everyone can work with. This may frustrate more experienced developers. They may have a temptation to dominate the group and show-off somewhat which leads to others feeling lost and lose confidence. Try to remind them that the goal of the group is for everyone to get up to speed with the programming language and this is an opportunity to practice their coaching and mentoring skills.

Only a few people would do the homework which at the time felt frustrating but what you need to understand is

People have lives. Not everyone can commit time to homework outside of office hours, especially parents.
Some people want/need to be guided along in a more hands-on fashion
Some people just aren’t as enthusiastic as you. That doesn’t mean your efforts are doomed to failure! Just accept this.

Go by example

The book approach wasn’t really working for the group so we thought we’d scrap homework and the quite detailed book in favour of studying Go By Example

We’d work through each page, discussing each small concept in detail and playing around with the code in the Go playground to try to understand the inner-workings better.

Any temptation of seniors trying to show-off and skip-ahead were stamped out.

We’re studying for-loops right now, not channels. We’ll get to that later.

Studying a small part of the language as a group without other distractions worked really well for us. They were comfortable bite-sized chunks that the less experienced could get a hold of, but the seniors could also have a bit of time to explore the topic a little more deeply to understand it better.

Making it more concrete

Whilst the lunchtime club got people more excited about Go and starting to understand how to write it we had other activities to support out goal.

Dedicated Slack channel

This was somewhere to post interesting links, share code, articles etc.

Developers would also show little Go projects they were working on to get feedback and ideas on what they’re working on.

This helped foster the little mini-community that was forming and as the numbers grew the legitimacy of the idea increased.

Group activity

Most of our work broadly fit into 3 categories

Building fancy, interactive websites
Building web servers which mostly call N APIs to get data, do something useful and put that data somewhere else (e.g a database) or make it available over HTTP
Churn some data from a message queue

What I did was designed an exercise to showcase Go’s strengths in building robust, high-performing web-servers.

The exercise involves building a web server which takes a request containing N URLs (e.g http://quii.dev, http://news.bbc.co.uk). The server should respond with a “stitched” response of the two websites response bodies.

The caveats I added to this were you were not allowed to have the WiFi on to complete this exercise, which forces the developer to

Use the built-in documentation
Use the standard library
Use httptest to test the needed functionality (you wouldn’t be able to make a passing test that went to a real website)

The group was really successful in implementing it. The lesser experienced devs could make the required functionality, and the more senior ones could make the exercise more interesting by making the HTTP calls concurrent.

I feel this exercised showed the developers that - They are capable Go developers - They can do more than FizzBuzz, they can do their jobs in this language

10% time

We were lucky enough to have 10% time when every other week we could work on whatever we liked. Naturally a number of us would spend time reading more tutorials and starting to build web servers to do useful things.

I felt like we reached a real tipping point when we had a “Hello, world” server building and deploying through our Jenkins pipeline and deployed to live. This required us to work a bit on the companies’ tooling but finally we could point at something real.

Opportunism

This process was never going to be done in a few weeks but after a few months of 10% time, exercises, prototypes and lunch & learns I felt we were at a point we could build something real, but what?

In truth a lot of the time writing a web server in Node or Go isn’t going to be radically different in terms of developer productivity or even performance. There’s so many things that influence these things and programming language usually isn’t the highest one.

I was working on a system which had a micro-services approach to it. It would process thousands of messages from a message broker and for the most part Node was standing up quite well.

However, we started writing a part of the system which was very CPU bound, crunching a number of strings to build new strings. Node can struggle with CPU bound tasks, and it did for us. Our systems are horizontally scalable, so we went from 3 instances to 10 and increased the memory each needed to 512mb but it still struggled.

We had a pair of engineers spending a few weeks picking at low hanging fruit and trying different approaches to improve performance but it just wasn’t happening

Pitching it

I felt this was the right time for us to try re-writing this one service in Go. I spoke to the tech lead and chief architect and outlined the following

Go should have much better performance in these kinds of tasks
We have invested a lot of effort in getting a pool of enthusiastic developers who want to write Go
Writing code isn’t the hard part here. Understanding the requirements is the hard part and we now have them codified for us to work from. I believed it wouldn’t take long to do properly
We had technology-agnostic black-box-tests that we could run against the new version of the service to verify it

The result

As much as we attempted to practice with “real world” exercises as part of our learning, writing real systems is always different and things come up. It wasn’t the smoothest ride but we replaced the Node service with the Go one in around 2 weeks.

It was around 10 times faster than the Node service, with only 3 instances using 256mb of memory

When we deployed it, I thought we had made some kind of mistake as I just couldn’t believe the metrics I was seeing.

Around 4 years later I asked my colleague how the system has held up and he said it has become a very resilient part of the system, now handling double the load it had when we originally made it with very few problems.

The kudos

Presenting our accomplishments to the rest of the development team and CTO was a real highlight of my career. It felt like I had stuck my neck-out a bit and worked hard and this was the pay-off.

When you’re thinking about introducing Go to a company, just remember that while programming languages are fun, they are a means to an end. Think about those ends and how you could get recognised for meeting them.

Our bosses shouldn’t care what programming language we use. Managers care about how we delivered useful software, smashed our performance targets, reduced our AWS bill and had fun. Think about what you’re really trying to achieve and use that to motivate your efforts.

Go specifics

Almost everything I’ve written here is not Go specific. You could take the same approach if you wish to introduce any language to your organisation.

Go does have some particular strengths that you can use to persuade people to use it

Backward compatibility

In enterprise this is especially important. Upgrading a codebase from Scala 2.8 to 2.10 was one of the most tedious activities in my career. Upgrading Go versions is just a configuration change.
It means tutorials written 10 years ago are still relevant and correct today and this is one of the reasons it’s an easy language to pick up.

Stripped back, simple language

In a previous life I was a Scala developer and the amount of bike shedding around approaches is incredible. You may find yourself frustrated at writing for loops but soon you’ll realise how liberating it is to have only one way to iterate over a collection of things and do stuff.
Much less to learn
Much less to read

Built for software engineering

All the things you learn about writing software to a high standard in a team environment, not a 1 person project that only works on one laptop is well catered for with Go.

Built in testing
Built in benchmarking
Easy to document code in a consistent manner with runnable examples
go fmt for consistent readability
Compile to practically any platform you’d want to use and extremely easy to deploy
Fast compilation time for excellent feedback loops
Excellent IDE support
Stable and rich standard library

Wrapping up

Trying to advocate for something new that you feel strongly about can be difficult and stressful. Work hard on your people skills to win-over helpful allies and build trust in your idea.

Think of it as a marathon, not a sprint and keep retrospecting over the course of the journey. As I’ve said context is so important and only you can figure out what exactly is right for your own situation.

Remember that programming languages are not a goal in themselves, they are a means to an end. If you’re thinking about championing Go in your job, try to put on your bosses hat and think about what she’s trying to achieve.

It’s unlikely she's judged on what programming language is used, it’s more likely to be things like whether they’ve delivered a good product, improved uptime or other things like that. If you can then relate Go to how those goals can be achieved you’ll have a better chance of convincing people.

People will respect you for sticking your neck out and standing by your principles so long as you do it in a mature and respectful manner.

How my website works

Chris James — Thu, 27 Aug 2020 00:00:00 +0000

I recently redesigned this website. It’s not exactly a technical marvel, but I feel that is its strength, and I wish more projects I observed prioritised delivering useful content rather than expensive to implement designs. Most people just want to read some text and look at some pictures and it’s extremely easy to do this if you don’t tie yourself up in flavour-of-the-month technologies.

On the way I got a chance to brush up on some of the more modern CSS techniques which was a lot of fun. People who complain about modern-day web development don’t know how lucky they’ve got it.

There’s so many fantastic techniques and tools built into web browsers compared to 10 years ago that don’t require a lick of JavaScript.

The approach

Under-the-hood not much has changed.

The server has a posts folder which contains markdown files for each post.
The web server is written in Go. When it starts up it opens up every markdown file, converts it into HTML and puts it in an in-memory map.
When it gets requests, the server simply looks at the path and matches it to a key in the map of posts, returning the pre-rendered post.
The frontend is plain HTML and CSS because it’s the best, simplest, most performant way of delivering textual content to users.
It is deployed to Heroku and sits behind CloudFlare CDN.

I’ll use the simplest tools I can to deliver a good experience. This will sometimes rely on using CSS techniques that won’t work on older browsers. This is fine so long as my content is still accessible and looks ok. This is called "graceful degradation".

Graceful degradation is a design philosophy that centers around trying to build a modern web site/application that will work in the newest browsers, but falls back to an experience that while not as good still delivers essential content and functionality in older browsers.

New things I learned

CSS Variables

On the face of it they seem like a nice way to DRY up some code and give some semantics to things like colours.

:root {
    --nav-colour: hsla(354, 100%, 71%, 1);
    --link-colour: hsl(354, 100%, 71%);
    --bg-colour: hsla(30, 100%, 99%, 1);
    --code-colour: hsla(327, 24%, 35%, 1);
    --text-colour: hsla(280, 12%, 20%, 1);
    --header-colour: hsla(280, 12%, 20%, 1);

    --font-size: 2.2rem;
    --line-height-multiplier: 1.5;
}

time {
    color: var(--code-colour);
}

I think they’re an interesting way of identifying the complexity of your design. If you give yourself the constraint that the only place you’re allowed to declare a colour value is within your variables then you’ll have a well-organised list of colours on your website. If that list gets long, or it becomes hard to name your variables ask yourself

Do I really need all these colours for my website?

You can take the same line of thinking in respect to the variables you use with your typography and your general spacing. By using variables it forces you to think and will help give your design a more consistent and less cluttered feel.

Media query for dark mode

Dark-mode feels pretty faddy to me but I am pleased there is a nice progressive way to change the style of your website without relying on any wonky JavaScript.

If supported, my website will automatically change into dark-mode with no prompting from the user. In OS X you can set dark mode to automatic which will go from light mode to dark through the day, and my website will respect the user’s wish.

This feels much better than expecting users to have to update some kind of setting on every website they visit.

This works especially well with CSS variables. To enable dark-mode all I had to do was change the value of my variables.

@media (prefers-color-scheme: dark) {
    :root {
        --nav-colour: #15B097;
        --link-colour: #3282b8;
        --bg-colour: #051923;
        --code-colour: #15B097;
        --text-colour: #EDF6F9;
        --header-colour: #1C77C3;
    }

    img {
        filter: brightness(.8) contrast(1.2);
    }

    body {
        font-weight: 350;
    }
}

Easy!

Fallback for older browsers

Without any effort, my website is still useable for users of Internet Explorer (and probably Netscape Navigator).

Web browsers are very forgiving and will do their best effort to render even if they see syntax they don’t understand. As I am relying on robust, battle-tested technology (HTML & CSS) it "works" - the content is still accessible.

If I had done some react-whizz-bang framework single page app I would have to jump through a lot more hoops to reliably deliver some text.

Fallback techniques

Including this in the head of the document means this CSS file will only be loaded by users of IE9 and older.

<!--[if lte 9]>
    <link rel="stylesheet" type="text/css" href="/static/fallback.css">
<![endif]-->

The following media query lets me style for IE10 and IE11

@media all and (-ms-high-contrast: none), (-ms-high-contrast: active) {
    body {
        /* IE 10 & 11 specific style declarations */
    }
}

I don’t anticipate many users of IE11 or under being especially interested in a programmer’s blog, but it can’t hurt to help them out.

It's not pixel-perfect, but it doesn't have to be. I have seen teams spend days fixing minor pedantic styling issues with Internet Explorer and I can't imagine that's the best way to spend expensive developer time. Are you building websites to satisfy your users or designers?

Performance

All the content delivered and styled in 40ms with under 5kb of payload (excluding post-specific images, which I try to minimise)

All these fancy computers we use have an impact on the environment. By taking a minimalist approach my website's carbon footprint is cleaner than 99% of the websites tested on websitecarbon.com

Big deal Chris, it’s only HTML and CSS, it’s not a real website

Yeah, it’s only HTML and CSS, brilliant isn’t it. Easy to make, accessible and performant. I made my first website around 21 years ago, and I find it quite comforting the same technology I worked with then works now.

I wish more websites were written like this rather than clogging my network and computer with megabytes of useless JavaScript written by developers running transpilers over transpilers on frameworks built on top of other frameworks and for what?

Fancy transitions?
The client not doing a full page refresh going between pages?

Do your users actually care about this? Click between links on my archive page and tell me it’s too slow and has a bad user experience.

There are projects where SPAs are a good fit, but many developers are poor at making this decision and add a huge amount of complexity for no real reason.

Wrapping up

Making a static website does not require a fancy blog generator built on top of a gig of node_modules. You do not need special tools to make a great website.

Even if you’re not a confident “backend” developer you don't have to write a web server like I did; you can still… create websites with plain old HTML files, written by hand!

This is a totally legitimate way of making many kinds of websites and will most likely result in the best user experience for the least effort.

Start naming your test doubles correctly

Chris James — Thu, 30 Jul 2020 00:00:00 +0000

A lot of developers are very vague when they name their test doubles which makes tests needlessly more difficult to understand.

In conversation there’s also often ambiguities which cause problems. I overheard on a podcast:

Have you ever used a mock where you make assertions on the call?

This post will hopefully illuminate why that sentence is a bit silly. By making a little more effort to be precise with your naming, you can communicate the intent of your tests and the design of your system more effectively

Quick primer on dependencies

To separate concerns, we often make objects that will “depend” on other objects.

We “inject” or “pass in” these dependencies when we construct them, so they can do a job. This is called Dependency Injection.

Example

A ShoppingCart sends an email when you call its sendOrder method.

We don’t want ShoppingCart to have to know how to send emails as well as everything else so we give it an Emailer when we create it.

new ShoppingCart(new MailgunEmailer())

It can then use the emailer without having to know how emails are sent, and you have your separation of concerns.

sendOrder() {
    // interesting domain logic
    this.emailer.send(confirmation)
}

This approach has implications for our tests. With tests, we prefer not to use real objects for dependencies because we want to be able to control them, so we use something different:

It would not be desirable to send a real email every time a test is run.
We sometimes depend on things that are slow, but we like our tests to be fast because feedback loops are extremely important
It would be hard for us to exercise failure scenarios using real dependencies. How do you get a 3rd party email service to deliberately fail?

Use “Test Doubles”

Not specifically mocks. A test double is what you inject into an "object under test" (OUT) to control a dependency; Like a stunt double!

Using the precise words for test doubles helps reveal intent and documents your design more effectively.

It’s not about being big and clever, it helps you communicate unambiguously. Software development is a team-sport so clear communication will help your team work better.

The different kinds of test doubles

Dummies

Sometimes you have arguments to functions that you don’t care about. Dummies are the arguments you pass in.

const dummyLogger = jest.fn()

Stubs

Some objects will query dependencies for data.

Stub the dependency

const stubUserService = jest.fn(() => 'Miss Jones')

Spies

Sometimes you want to call a method (or more abstractly send a command) to a dependency.

It can be hard to tell in a test this happened from the outside as it’s an internal detail.

If you want to verify these commands, use a spy.

const spyEmailSender = jest.fn()
// do some stuff
expect(spyEmailSender.mock.calls.length).toBe(1)

Fakes

Usually a “real” implementation but constrained.

Like an in-memory database.

Not as popular anymore due to things like Docker which let you spin up local versions of databases without the risk of there being differences between a fake and a real implementation.

Mocks

Mocks are a kind of test double! Mocks are precise and encapsulate the detail of your interactions more explicitly. You describe upfront what calls you expect on your test double

Order
Arguments
Specific return values

If the object under test doesn’t call things as expected the mock will error.

So returning to

Have you ever used a mock where you make assertions on the call?

What is being described is just a mock. If you're not worried about the calls, you're talking about a stub.

Why “mocking too much is bad”

The risk is your tests become needlessly precise and coupled to implementation detail. This is also true of spies as they have a similar nature.

If you're overly specific with your mocks, when you want to refactor things you may find tests failing for annoying reasons that have nothing to do with behaviour.

Why “having too many test doubles is bad”

Listen to your tests if they cause you pain. If you have many test doubles in a test that means your OUT has many dependencies.

Does that sound right to you? It shouldn’t and it means you probably need to take a look at your design.

An anecdote

Yesterday I reviewed a test which had a test double named

getterFn

I had to spend some time reading the test to understand what was going on, it turns out it should've been called

spyCSVFetcher

Once this was ascertained it turned out the writer had actually neglected to spy on the number of calls, which was actually a core part of what was supposed to be tested.

If the double had been named correctly in the first place (which is very little effort) I would know to look for assertions on the way it was called to help me understand what the intent of the code is. Instead, it took some investigation.

Summary

Consistently using the recognised names for test doubles will help you be precise when communicating with other developers. It's a low-effort way to improve the way you work.

For instance if you name something a dummy rather than a mock the readers of your tests can forget about that particular test double and focus on the important collaborators.

Tests should act as documentation so describing your test doubles clearly will help reveal the design of the test for maintainers.

So start naming your test doubles correctly, today!

Quiz

What does a test with lots of doubles tell you?
How do you improve the design of a test with many test doubles?
Name all the kinds of test doubles
How is a mock different from a spy?

Work 2019

Chris James — Sat, 28 Dec 2019 00:00:00 +0000

This post is a summary of work stuff over the past year which covers my first adventures into

Technical leadership and line-management
Hiring and building a brand new team

I think I've helped build a pretty rad team, so if you're interested in that kind of thing, read on.

Rough start

After working in one of the most productive, interesting and fun teams I've ever had the pleasure to be a part of, the team has to be disbanded due to complicated (and boring) politics.

We were given freedom to move in to almost any team within the organisation but I really didn't take our team being disbanded well at all so I think in my eyes every opportunity looked boring and unfulfilling. It was never going to work out.

New job

Happily a new opportunity came about with my previous employer. I'd worked there for just under 2 years; it was really enjoyable, I liked the environment and I'm proud of the work we did and how we did it. It felt like we stood behind the good DevOps principles and it worked out well for us.

This would be a different role though, I would be a technical lead. I have been a technical lead before but in a different environment and without the line management responsibility needed for this role.

I was a little tentative about this at first. I hadn't been a line manager before and everyone I've spoken to about it to implies it's horrible. My future boss spun it pretty well though:

I'd get to hire my own team
I'd get to run it more-or-less how I like (so long as we deliver, obviously).
- He knows I have strong opinions about how teams should be run and is happy to give me the autonomy to try to make it happen.

I deliberated on this and thought about leads who had worked well with me and those who hadn't. A couple of people who've had the dubious pleasure of managing me have had a real positive effect on my career (and life, I suppose). They pushed me into things I never saw myself doing and gave me confidence to stand behind my convictions.

When I put all that together with someone giving me this rare opportunity to pay it forward and to help some people find the joy I get out of software-engineering; I had to give it a go. I may never get this kind of opportunity again!

My personal goal for this job

Learn how to create great teams who not only deliver good software with minimal stress, and actually enjoy it.
Grow some engineers

Hiring

The initial suggested makeup for my team was:

A senior
A mid-level
An associate (junior)

Now where I work isn't as glamorous as one of these fancy fin-tech startup whatevers, but it is a good place to work. The environment is great, we're empowered and work in a pretty cutting edge way. We go to DevOps conferences and hear about best practices and we can honestly say we do a lot of them. We deploy many many times per day, deployment is a non-event for us.

Getting that across to the highly competitive London job market I guess is hard because I had to do a lot of recruitment effort to eventually get the right people.

Diversity and all that

I was determined not to hire 3-4 clones of myself. I am infuriating to work with. Studies show diverse teams have better results. Also, hiring diverse teams is the right thing to do.

We approached Makers Academy to help us hire as their raison d'etre is bringing people from different backgrounds into software development. I've had a very positive experience working with Makers graduates over the past 5 years, they've always made the teams and people around them better so I wanted more of that.

It was a fun and intense experience of pitches, chatting to lots of people and interviewing a number of great candidates. If it were up to me we would've hired more but in the end we hired 3 excellent developers who are thankfully nothing like me.

The team

The stars aligned quite nicely in that after months of recruitment going nowhere my team of 3 associates and a senior more or less joined at the same time. It was super exciting to have a new team suddenly appear in the office.

Having a brand new team, and role suddenly land on my lap all at once was pretty intimidating at first but the people I hired made it a lot easier because they're all excellent people and I had good support from other people in the department.

How I tried to make it work

Before leaving Springer Nature I had a meeting with one of the managers which was more impactful than he probably realises. He said

You need to take learning management as seriously as you do learning software development. In the end you have a direct effect on people's lives

OK, it's a bit dramatic but it is true. I reflected on managers who hadn't worked for me and how frustrated and annoyed I was about the situation even outside of work.

It's definitely a problem that a lot of people seem to think that management comes naturally to certain people and is not a skill to learn and practice.

So I picked up some books and watched some talks in the hope I could actually achieve my goals and not let my team down.

The Phoenix Project & The Machine That Changed The World

These are two books that talk about the problems you have when trying to build big complicated things with lots of people. They're very different books,

The Phoenix Project is described as a "DevOps Novel" which sounds lame as hell but is actually really enjoyable.

The Machine That Changed The World talks about the approaches to manufacturing cars. From artisan bespoke cars, to mass production and then lean manufacturing. It sounds far away from software development but as you read it you find yourself nodding along, drawing parallels with our industry.

There's loads to take from them and here are the main bits I got:

Command and control definitely does not work. You need to empower the people who actually create things to make decisions as they're the people with the most up-to-date, relevant knowledge.
You must respect work in progress (WIP) limits. It's way better for a team to properly finish tasks and only work on a small number of things than hurriedly work through dozens of tasks. Disrespect of WIP will slow you down.
A better understanding of "lean". Once you read these two books you have a very clear idea of where agile gets a lot of its ideas from.

I summarise these ideas in more detail in The ghost of Henry Ford is ruining your development team

Resilient management

This is a very practical book on technical leadership, it's quite short and I have found it invaluable. It has given me concrete things to do to help build my team.

Growing people

It has some fascinating insights between the terms "coaching" and "mentoring" which I share with basically everyone because it really has changed the way I try to help people.

Mentoring Giving advice and perspective to help someone solve a problem. Quite a direct way of giving information. Good for overcoming roadblocks. Focused on the problem and solution.
Coaching Asking open questions to explore a topic more. I like to see it as trying to help someone "join the dots" of an idea. This style will have a longer term impact on people and will help both parties explore ideas more.

Having these ideas explained has made me more conscious about how I try to help people solve problems. I find myself looking for coaching opportunities more when working with people and have tried to take that into our workshops by encouraging coaches to draw on real-life analogies to help explain technical ideas (David Wickes is extremely good at this).

Self reflection

The book discusses how when people go from engineering to leadership need to understand their "default work mode" and how it might not be compatible with different goals as a lead. I think my default work mode is "just fucking do it" and "ask for forgiveness, not permission"; this has broadly worked out quite well for me as an engineer in teams. However as a lead I need to check myself.

Sometimes I just want to open my IDE and refactor some code I dont like and git commit -am "refactor". On a high level that is the right thing to do but how does that help my teammates in the long-run? I should see these things as a coaching opportunity and instead find time to bring it up with people and explore the space with them so that in future everyone else can see the problems in our code better.

The book asks you what you think your management style is so you can reflect on it. Just asking that question of myself was weird and difficult, this is what I came up with:

No micromanaging. Feel strongly about empowerment, ownership and trust. I have no interest in denying clever people the opportunity to use their skills to achieve our goals.
Reliant on honest feedback, sometimes I don’t observe myself not being great
Hope to focus more on growing people than “getting shit done” but I will need to check myself because that is my “default work mode”.
Want to act as a sponsor, finding opportunities for people to grow

I like to think that I've stuck to my guns with this style and it seems to be working out OK.

Setting direction

The book also instructed me to write a vision (or North Star) for the team

Be an amazing team that is continuously improving that ships well-engineered software with minimal hassle and stress by relying on craft and taking advantage of our autonomy to apply industry best practices.

We will take pride in our actual agility to confidently change and evolve our software as and when we need to.

If we get good at creating great systems, our stakeholders will always be happy with us and will continue to give us autonomy.

And a mission, which is a more grounded version of it

Openly and honestly communicate with each other to guide ourselves so we can grow
A culture of learning and sharing knowledge
Confidence to suggest changes to improve our ways of working and our code
Continuously tightening feedback loops to guide improvement

And a strategy to get there

Finds ways to break problems down into smaller chunks so we’re never working on something that feels too intimidating. Perfect is the enemy of good, better to ship 80% of the feature quickly than 90% in 3 months time.
Low WIP (work in progress) so we’re not spinning too many plates and that allows us to focus on small tasks that we can execute well.
Culture of feedback. Practice saying something nice every day! Remember that giving someone “negative” feedback is an act of kindness (so long as it’s done kindly).
Practice continuous integration. Both in respect to code and knowledge.
Pairing, mobbing. Even if you work on a task by yourself it should be considered very unusual not to seek feedback on your work as you do it.
Give ourselves time to research, in particular pre standup research times
Share interesting things you learn.
Ensure everyone has time to work on all aspects of our code and development practice, actively fight any silos of knowledge forming.
Everyone on the team contributes in forming how we work.
We should take pride in our work. If you’re not proud of something, we discuss how to make it better.
Identify any single points of failure we may have in our ways of working and work to spread the knowledge & responsibility

If nothing else it felt cathartic to write this down and I've encouraged the team to contribute to the document.

I sometimes feel my role, if anything is to keep pushing the team in the direction this describes so that we do become that team that I hope we will be.

A talk from Makers

A few of us went to a Leading a Diverse Team workshop at Makers. It was interesting and one talk resonated with me a lot. It discussed how often as a leader we project our own ideas as to how to accomplish a goal based on our own experiences. This is fine and well intentioned but it's from a position of bias and you may be unintentionally shutting out different ideas.

A better, more empowering way of trying to achieve what I'm trying to achieve is setting a vision but then encouraging the individuals in my team to think about the unique ways they can contribute toward that vision that I may not be thinking of.

My way of achieving goals is one means to that end but it doesn't have to be the only one; and there's probably better ones that my team can think of based on their own background, skills and experience. It's my job to encourage people and build a great environment so they can explore their own ideas.

Lessons learned, things to improve

Introvert Chris

I am one of those introvert-ish people who can seem relatively outgoing most of the time but I definitely suffer from running out of social energy quite quickly and just wanting to get a break from people (no offence, team).

This role definitely has taken its toll on introvert Chris at times and I need to find ways of managing that better but I dont have many good ideas yet. I think perhaps now my team is settled more I shall try and delegate a few more meetings to people as it will save me some social energy and is an opportunity for others to contribute in those areas.

You cant just mob

I hear a lot of great things about mobbing and I am totally on-board but for whatever reason when we tried it did not go well. I suspect it was a function of the team being very unfamiliar with each other and the nature of the work (it was very new-project, infrastructure work).

Still, like leadership it's probably something you have to study and not just an innate skill. I hope to read more about this and try it again next year.

General time management

I'm not happy with the amount of time I've got to pair with everyone. I'd say I do it quite often but not enough. The times where I paired I've been able to give much more useful feedback in our one-to-ones.

When the role was first offered to me it was proposed I would manage 3 people. That quickly turned to 5 and at the start of 2020 7. To serve all these people well I am definitely going to have to think carefully about what I am doing every day and try to find opportunities to delegate useful tasks that will help people grow and free my time up at the same time.

Study more

The few books I got through gave me some excellent ideas and set me up well. I want to continue reading this subject to get better. I'm currently reading An Elegant Puzzle which is very interesting but seems maybe better suited to a head of engineering role rather than the one I'm in right now.

The next books will be:

Final reflections

I'm glad I didn't try and "wing-it" with management. I'm still learning though and there's lots of room for improvement but I think the upfront study I did helped grease the wheels for my team and get it off to a good start.

During hiring I was very upfront about what my expectations and aspirations for the team were which resulted in a group of people joining who seem to be bought into the idea and are contributing toward it.

I'm really pleased and proud of how things are going. I shouldn't take for granted that we managed to turn a group of strangers with various backgrounds and experiences into a great team. The culture is fun, open, curious and we take pride in our work; which is honestly exactly what I wanted.

I feel everyone has become better engineers since joining and I am confident we'll do great next year so long as we keep pushing ourselves to get better and better. (that felt like a bit of a lame manager-esque rallying call, sorry)

I think the most gratifying moment for me was a small one. A few weeks after the team formed I went on holiday. When I came back one of my team mates told me they had changed the process. I replied:

Good

It's the team's process, not mine. I want them to be comfortable making decisions about how we work and voicing strong opinions about the way we do things, and this demonstrated we are heading in the right direction.

Gamifying Continuous Integration

Chris James — Thu, 28 Nov 2019 00:00:00 +0000

If you read the internet, you'll soon realise that a large number of software developers don't understand what CI is and why it's important.

It's important to state that continuous integration is not a server.

Like the word DevOps, people seem to think these important principals of software engineering are about tools and products. Perhaps misinformed managers think if they buy the right magic box they can finally deliver their projects on time.

Continuous integration is continuous integration

Read the words.

It's about making sure that when you are working on code, it is the most up to date version of the entire team's code.

If you have multiple versions of code checked out on different branches or machines, then you are not collectively integrated in respect to code. The code you see, is different to the code your colleague sees.

In an ideal world if you could somehow all work on same computer at the same time comfortably, that would be pure CI.

If you've ever used VS-code's Live Share plugin, that's as pure a CI experience you can get in a remote environment. If you've not used it, it allows you to connect to a colleague's VS code and edit the code together as if you're working on the same computer. The changes you make are reflected (or integrated) to everyone else in the session - the feedback is immediate.

A much simpler experience of CI is mobbing. Everyone is working from the same version of the code together.

How do you know if you're not practicing CI well?

One obvious sign is developers wasting lots of time resolving large merge conflicts. Merge conflicts are a direct result of you working on a version of the code that isn't up to date. Your team are not integrating their changes continuously and therefore as they make changes the risk of making invalid changes increases.

The time wasted on merge conflicts has compounding effects as it increases the delays of integrating your own changes to the system.

One might argue you can resolve this by structuring work differently to avoid conflicts, even going as far as to re-architect your system to avoid merge conflicts. An often touted reason to go for microservices is it allows teams to work independently.

Reader, I have worked on a team with 20 odd developers, shipping frequently on a monolith and very rarely did we have merge conflicts. We also didn't do pull requests, we committed to the main branch.

If you don't practice CI you are not working with your team as well as you could

Changes to the system are invisibly stored on your colleague's computers where they are of no use to you.

You could be imagining some great abstraction given the code in front of you but unfortunately the code in front of you is not the source of truth because unknown to you another developer merges a gigantic change after spending a week on a feature.

People become reluctant to refactor parts of the code because someone is working on a branch that is vaguely near the code you want to change so you wait a few days and then forget about it.

How do you CI well?

The simplest way is for everyone to commit to main (or master), frequently. It really is that simple!

To be good at continuous integration, you should continuously integrate your changes to main. That way everyone will be working with the same version of code for the most part.

It's important that you respect your colleagues and run your tests before pushing. If the build is broken the team should stop committing unless it's to directly fix the problem. If people keep committing whilst the build is broken it will become very difficult to diagnose what the problem is and the new commits may compound the issue.

The process, in full

git pull -r
Make a positive change to the system
git commit -am "added blockchain terraform machine-learning module 2.0"
git pull -r
./build.sh && git push

This practice is simple to follow, doesn't require elaborate review systems, branch strategies etc, and forces good habits.

In order to push frequently, you need to pull changes frequently
You can only push frequently if you work on small tasks, iteratively. There are many, many reasons why this is a wonderful, less risky and easier way of working. (see TDD, XP, any decent agile book, etc etc)

It wont entirely stop merge conflicts occurring, but they should be very infrequent and because you're doing small changes they're typically trivial to resolve.

But what about code review?

The industry seems to have conflated reviewing code with a very specific, relatively new process of looking at code when a pull request is submitted; usually too late once all the code is written.

Did you know that you can talk about code, at any time during the day?

Did you also know that the correct abstractions and patterns may not be apparent when the code is "done" and you'd be better off encouraging an environment where people are free to refactor code when they want to, rather than having to go through a laborious process every time they want to change the code. I've worked on and observed projects that practice code review very strictly and yet the code is still not great.

Relying on a process of checking code before it is merged is not going to result in a healthy codebase alone.

With this way of working I strongly encourage pair programming because people rightly assert that as a lone developer it is very easy to do something wrong if you don't gather feedback. Pair programming facilitates a constant feedback loop on what you are writing.

What if someone commits something wrong/broken?

First of all, no process in the world can prevent this. Stuff does go wrong, and you would be better asking yourself:

How can we detect and recover from problems easier?

Secondly, this approach does require a healthy and collaborative team that talk to each other regularly about what they are working on. They should be talking to each other when they start a bit of work about what they are going to do, how they will release it safely, what kind of tests, etc. Couple that with working on small things iteratively the risk of pushing something catastrophic are very low.

If someone does push some code that maybe works (so the tests pass, monitoring is all good, etc) but is actually "bad". Well so what? We can refactor it if we don't like it.

What if I, a tech lead want to check every change before merging

I don't want to work with gatekeepers.

I want to work on a team that trusts one-another.

Perfect is the enemy of good.

So is a system of pull requests bad?

Not at all, they're perfect for open source projects where you want to welcome contributions from other people but dont have implicit trust. In that case you will want to review changes that go into your precious project.

For a team where you trust the developers all it adds is overhead.

Liberation

We should be removing the barriers to changing software, not adding walls and walls of bureaucracy.

In this kind of environment people fix code when they see it needs fixing.

This way of working tightens feedback loops and increases actual meaningful agility.

CI is not just about the code

It's important to integrate knowledge of domain, work etc. (so important not to have silos of work with just the BAs/UXD etc)
Who is working on what (standups, kick-offs, etc)

Gaming it

I feel very strongly about working this way over a pull request approach when you are working on a team you trust.

Most of my team until recently were very unfamiliar with this way of working so I made a silly dashboard on one of the TVs which showed who committed to the main branch most over a given week. It penalises you if you push a change which breaks the build.

After just a few weeks this slight nudging has improved the team's approach to CI, we've had significantly less merge conflicts, people are refactoring more often and the number of failed builds has also dropped.

The code for it is on github

And if you have docker, you can run it with

$ docker run -p 8000:8000 quii/ci-league

If you want it to look at a private repo you'll need to supply a github personal access token as an environment variable

$ docker run -p 8000:8000 -e GITHUB_TOKEN=supersecret quii/ci-league

It doesn't do any fancy auto-refreshing but most browsers have extensions to auto refresh a tab, we set it to update every 10 minutes here.

Wrapping up

As usual in software, we seem to suffer as an industry of remembering the why of all of these fancy terms.

If you appreciate the why of CI you may not fall in to the trap of blindly applying best practices that work in a different context to the one you are working in.

Pull-requests in an open source project make total sense and work great. In a different context it can add overhead.

You should instead focus on what your environment your team is working in, the constraints you have and use principles like CI to guide what process best fits.

The carbon footprint of your website

Chris James — Wed, 13 Nov 2019 00:00:00 +0000

Stumbled across https://www.websitecarbon.com which measures the carbon footprint of your website.

I've always believed in lightweight, efficient websites to deliver content. I firmly believe in not using client-side JS unless absolutely necessary because not everyone has an Internet Telephone Professional Maximum on a 4g connection.

I ranted about this and riled up the community about this subject here

My website is 94% cleaner than other websites tested

What's your score?

Given the huge amount of evidence pointing to how website performance effects users and the environment, what are you going to do about it?

For more on this check out the slides of this excellent talk https://twitter.com/hanopcan/status/1187675166236643329

The ghost of Henry Ford is ruining your development team

Chris James — Thu, 31 Oct 2019 00:00:00 +0000

People talk a lot about "lean". Maybe they've read The Machine That Changed The World and were inspired by how lean manufacturing transformed the motor industry. They read how Japanese companies became incredibly efficient and blew the competition out of the water, and then they took that to their software development team.

This sounds good, but too often, I feel like people miss the real genius of lean manufacturing and continue to run their organisation with a top-heavy command and control structure; the only difference is a constant message of

Be more efficient! Stop being wasteful! LEEEEEAN

Agile, DevOps and other buzzwords get a lot of their inspiration from lean manufacturing, and these ways of working gain their efficiencies (and other, more significant benefits) broadly by empowering development teams and giving them responsibility.

A primer on mass production

Before Henry Ford’s assembly line, cars were made to order by craftsmen. It took a long time to make them, and they were reserved for the wealthy.

The assembly line made it possible to make cars cheaply, which made them accessible to a far broader market.

How?

It’s a big subject, but here are some pithy bullet points:

To reduce costs, he needed to produce cars at high volume with low-skilled workers, not craftsmen.
The assembly line was a collection of expensive, inflexible machinery that scaled to volume and could be operated by cheap labour doing very simple tasks.
Emphasis on standardisation, being able to replace one worker with another cheaply and easily.

Problems

The assembly line was very intolerant to disruption, small problems could cause big problems and stop the line. To mitigate this, they:

Had to add lots of buffers, such as storing supplies, space and additional workers to smooth production
preferred to accept defects and fix them after the car was made, rather than to stop the line

The work was very dispiriting. Ford even passed on some of the savings and profits to his workers by heavily increasing their salaries, but people just didn’t want to do the same tedious task 100s of times a day.

The assembly line is heavily optimised for volume with expensive, specific machines meant it was very inflexible, so creating different cars was a very difficult and costly endeavour.

When designing new cars, it took what we would recognise as a very waterfall approach, of a project being passed around many departments and problems being unsurfaced way too late due to poor feedback loops, and not a good structure of ownership.

Lean manufacturing

Japanese carmakers studied the production line and tried a different way. When you hear about Kanban and Kaizen in agile circles, it all came from here.

From Wikipedia

For many, lean is the set of "tools" that assist in the identification and steady elimination of waste. As waste is eliminated, quality improves while production time and cost are reduced.

There isn't a canonical "lean" way of working, but the one that resonates strongest with me is "The Toyota Way."

The Toyota Way, in which the focus is upon improving the "flow" or smoothness of work, thereby steadily eliminating mura ("unevenness") through the system and not upon 'waste reduction' per se... The implementation of smooth flow exposes quality problems that already existed, and thus waste reduction naturally happens as a consequence. The advantage claimed for this approach is that it naturally takes a system-wide perspective, whereas a waste focus sometimes wrongly assumes this perspective

How does Lean try to reduce waste?

Another big subject and there are good resources on it with a Google search. The main drive of it from my perspective, is actively trying to identify waste and then fix it. Lean describes 3 types

Mura: Waste due to changes in demand. Lean advocates a "pull" system rather than "push". So you only make a car if there is demand for it rather than trying to forecast demand which is often difficult and makes supply inconsistent. In software, this would be akin to perhaps making a cheap prototype to prove an idea before committing to a bigger endeavour.
Muri: Waste by trying to do too much stuff at once. This is why if you're doing Kanban and not observing work-in-progress limits, you're not doing Kanban. It can sometimes feel counter-intuitive that you can go faster by doing less work, but you've probably worked in a situation where a team is working really hard, and everything feels chaotic and unreliable. This is also why smaller teams are generally preferable.
Muda: Non-value-adding work. A classic example in software would be developers constantly struggling with flaky builds and not addressing the underlying causes.

Lean emphasises the idea of "flow", optimising the processes for delivery and reducing interruptions in the production process. (think about trunk-based development vs pull requests here)

Quality

For quality, you adopt Kaizen, striving for perfection and hunting down the root causes of issues and fixing them. You embrace a culture of quality across the value stream, not after the product is "finished".

Command and control vs autonomy and empowerment

What I mainly took from the book "The Machine That Changed The World" was how it contrasted mass manufacturing and lean manufacturing with respect to management.

Lean manufacturing acknowledges that it's impossible to heavily micro-manage the production of a complicated product like a car with thousands of people working on it.

The best way for Kaizen to work is to coach and empower the do-ers to find the best ways to deliver.

Signs you're not in a lean company

People with job titles of "manager" or "leader" but are nothing more than coordinators

When designing new cars, there is the idea of the Shusa (not sushi!), who in short, was actually in charge of things. They could bring people to help and make decisions and were not worried about being overruled by the whims of upper management; they were empowered to make the project work. People would say that these people actually made their mark on a project "Oh, you can tell that's a Dan car because of X, Y and Z".

Contrast that to places I've worked where people have had very lofty job titles but no actual empowerment and were nothing more than a communication funnel between the people who actually make things and the people who could actually make decisions.

How common is it in our industry for tech leads to be miserable simply because they're promoted into what they'll actually call a "shit shield". Seriously, can we stop saying that as if it's a good thing? It's a problem.

A leader should be able to lead

The development team has no say in priorities

Lean famously relies on Kaizen, the concept of continually striving for perfection. In lean plants, workers can stop the plant if they find a problem, and they work hard to make sure it doesn't happen again.

If you see something technically wrong and slowing your team down, how empowered are you to fix it? Do you work with a scrum master who will humour you but probably thinks "technical stories" are a "waste"?

Being lean and efficient is not simply done by only working on the most important feature at a given time. It's about the builders working hard to create a system and environment that can be worked on efficiently. That takes the form of the usual best practices; tightening and improving feedback loops, continuous delivery, easy to set up, simple delivery pipeline, excellent code, etc.

The lean plants worked hard to reduce problems and defects and to give themselves flexibility. This only happened because the workers were given the autonomy to make it happen, not micromanagement from above.

How does your team compare? Can you fix problems when you see them (which is the best time), or do you have to wait for the "technical debt sprint"?

All teams have a standard, immutable way of working

Toyota didn't just make up lean and be successful; they had to practice and iterate at it to figure out how to be good. Your IT teams need to practice and work at being lean.

You also have to understand that we don't have teams of resources; they are individuals working on different problems. If you want to get the most out of people, you need to give them autonomy to figure out how they can work together most effectively.

How does a team get better at working together if they can't gather feedback by retrospecting and improving how they work?

Larger organisations often talk about standardisation to allow developers to move between teams. That in itself isn't a bad thing, but it's not essential as autonomy.

If you want teams to be empowered enough to practice Kaizen and let them find how to work efficiently, you have to accept that teams will work differently.

This is fine, and in my experience, moving between teams is always a challenge, even if everyone works the same way because teams are more complicated than a set of tools and processes.

The people who write the software are not responsible for QA or running it in production

In mass manufacturing, the assembly line optimised itself for volume and would defer any quality problems to the end of the line, where another team would pick things up and fix them. The issue is that the feedback loop didn't exist, and systemic problems would never get addressed, which brings waste.

I consider a QA column in a Kanban wall as a smell. QA in lean worlds is a part of the entire development process, not something tacked on at the end. QA is a responsibility and an inherent quality of a good team.

From the continuous delivery website

It’s much cheaper to fix problems and defects if we find them immediately—ideally before they are ever checked into version control, by running automated tests locally. Finding defects downstream through inspection (such as manual testing) is time-consuming, requiring significant triage. Then we must fix the defect, trying to recall what we were thinking when we introduced the problem days or perhaps even weeks ago.

If you're throwing your software over a wall to an ops team, how do you expect to write a sound system?

Boredom

If you're trying to turn your development team into a homogeneous assembly line pumping out JIRA tickets at volume you might start to bore some people (and probably build some crappy products).

You probably pay your engineers a lot of money because they're presumably quite clever, so why are you denying them the space to think and take responsibility?

I've worked on projects which should be, to be honest, quite boring but turned out to be great fun because I worked in an engaged, empowered and fun team who took pride in doing things the "right" way and practised Kaizen.

It was excellent for the team, and we delivered the project, which was initially seen as very risky and tricky, with very little stress. Not only that but what we learned with respect to shipping software was adopted by other teams in the organisation.

Kaizen is fun, rewarding and will improve your outcomes.

Too scared to release and get feedback

In the book Lean Thinking: Banish Waste and Create Wealth in Your Corporation, it says

Identify value from the customer's perspective

Doing this with software products is notoriously difficult, but it's made even more difficult if you don't have the courage to release your software to the world and gather feedback.

Too often, the software is managed as a project which "ends" (this is nonsense, obviously), and no one takes any lessons from the customers. This is the ultimate form of waste that Lean strives to fight against.

The transformational lean technologies and methodologies

I think it’s interesting to think about the transformative technologies over the past few years and how they relate to lean. I think a lot of things appear big and transformative but are they solving real problems? Are they helping you be lean?

Docker

Docker is an empowering technology allowing developers to use technology far more freely. Rather than asking for permission to use technology and have it installed in production etc., we can use whatever we like because we just ship containers.

The main benefit for me, though, is it has made the difference between a local computer and production much smaller, which allows developers to take more responsibility for the quality of their software.

Synonymous with moving away from highly specialised, inflexible tooling in the mass manufacturing world

DevOps

DevOps is lean. It is about tightening feedback loops, improving the quality of our tooling and processes and taking responsibility for our software. Read The Phoenix Project. Just do it.

Cloud computing (e.g. AWS)

Again this is another empowering development which lets the shop floor workers quickly adapt their technology landscape to solve problems. Synonymous with having more flexible machinery in the lean car factories as opposed to highly specialised ones from mass manufacturing.

Wrapping up - we're not building cars.

Agile and DevOps have taken a lot of lessons from the lean manufacturing of cars. All of them are about building complicated things with lots of people, which requires a different way of thinking vs mass manufacturing if you wish to do it efficiently with a happy workforce.

But when you're reading about lean, it's important to remember that we software developers have different goals.

We're not trying to build software at volume.
We're trying to deliver the promise of software, products that are malleable and can change to users' needs quickly and cheaply.

A big thing about lean is identifying value from a customer's perspective and eliminating anything that doesn't contribute to it. People take this lesson in software too aggressively to deprioritise anything that isn't a feature.

The thing is, this is just a very immature way of looking at what you're building. Not only are you making a product, but you're building a system that your developers work on. It's akin to not allowing your factory workers to improve the assembly line.

The factory is as important as the cars, and the system is as important as the software product.

The "real" customer does benefit from the following:

Making it more operable so it has less downtime.
Improving the test suite so the software can be changed easily to meet customer needs quickly
etc. etc

Examining the "value stream" for waste is excellent; nothing more dispiriting than working on something for months with no objective evidence it'll benefit anyone.

Too often, though, this is led by "product" people with little collaboration from the technical side. The team cannot practice Kaizen and wastes its time working around technical debt, slow builds, poor feedback loops, manual testing etc. This results in a product becoming too hard to change and can't react to what the user wants in the long run. A far cry from the flexible, malleable and efficient development system we're all supposedly striving for.

This comes back to what I wrote earlier, is your tech lead empowered to lead? Or is she just making sure you put estimates on JIRA tickets? Is she treated as an equal when deciding priorities for work? Does she have to argue constantly for scraps of time for the team to practice Kaizen whilst passive-aggressively being accused of being wasteful for even suggesting non-feature work?

Final final thought

From Wikipedia again

The cultural and managerial aspects of lean are arguably more important than the actual tools or methodologies of production itself. There are many examples of lean tool implementation without sustained benefit, and these are often blamed on weak understanding of lean throughout the whole organization.

Replace the word lean with "agile", and I think you'll have a lot of software developers nodding with knowing looks on their faces.

Go back to basics with MVC

Chris James — Thu, 25 Jul 2019 00:00:00 +0000

This is another follow up post to an episode of Go time I listened to the other day which seems to get my creative juices flowing.

It talked about web development in Go and a few things stuck out to me

A cautionary tale of generating HTML

Early on in the episode it was described how the built-in template library wasn't very expressive and described that the Go Buffalo framework uses Plush

Powerful, flexible, and extendable, Plush is there to make writing your templates that much easier.

Enter Jade

8ish years ago I worked on a pretty major project which was written in Scala. We decided to use the Jade template engine to generate our HTML which, like Plush allows you to do a lot of clever coding inside your templates.

This felt amazing at first but quickly became a nightmare. With the flexibility Jade gave us we made a bit of a mess. Our templates became ugly, hard to understand and difficult to test as more and more business logic leaked into our templates

In software I often see flexibility as rope that a team can hang itself with and I tend to prefer very opinionated and constrained things (like Go!).

What did we learn?

Software engineers are doomed to relearn things. We had forgotten about separation of concerns, in particular the guidance the Model-View-Controller design pattern prescribes.

MVC

It's quite surprising how badly people misunderstand MVC. If you go on Reddit or Twitter you will hear people telling other developers that "MVC isn't suited to Go".

Often people seem to think MVC is about folder structures, class names, or other fairly superficial concerns. So when developers from say a Rails background show their Go code with an array of folders they get derided and thrown to the lions.

MVC does not prescribe folder structure and I worry a lot of people seem to miss the point of the principle.

Model–View–Controller (usually known as MVC) is an architectural pattern commonly used for developing user interfaces that divides an application into three interconnected parts.

Controller

Intercepts "requests" (such as HTTP), parses them and then calls the appropriate "Model", getting some kind of data as a result and then sends it to a "View". In the Go world, that's usually a http.Handler

Model

Loosely, it's where your domain logic lives. I personally think it's quite poorly named but the important thing is the separation of concern. It must be decoupled from the Controller and View, so if you see anything HTTP related passed through from the controller it's likely you're violating that decoupling. They should also know nothing about views.

View

It doesn't know anything about controllers or models. It should probably get passed to it some kind of ViewModel which is just a collection of data it needs to render a view. A well designed view is simple and doesnt have domain logic; because templates are very hard to test cheaply.

What do these principles buy you?

Your domain code is cleanly separated from the rest of the system so it is easy to test and can be "plugged in" to different uses beyond your web server; for example a CLI tool, or just as a package for others to use.

It is trivial to create and edit views as they are merely mappings from a bundle of generic data into HTML (or whatever). It also makes them more accessible for frontend developers et al.

Your controller has clear concerns and are also easy to test.

Circling back to the start, I would be very cautious about something that allows me to write very expressive code for my view. These kind of tools trick developers with promises of convenience and power but often lead to leaky abstractions unless you are very disciplined.

Now have a think about all the tutorials you may have read around HTTP servers and Go. The good ones will recommended all of these principles without perhaps just calling them out. Rather than the community dancing around the idea of MVC, just embrace it. It's a tried and tested pattern and most people advocate it anyway, just not explicitly. If we were explicit then maybe there wouldn't be a new post every week about how to structure a HTTP server!

How to structure your web app

Understand the idea of MVC.
Your http.Handlers are your controllers, make sure they only do things controllers do according to above.
All your other business logic (Model) lives elsewhere, in packages centered around real things. If it's a bank i'd maybe have packages for Account, Currency, Customer e.t.c. Your controller will call things in your packages to do useful stuff.
Depending on what you're building you can just use the encoding packages to spit out XML or JSON, or if it's HTML just use template/html.

Functional programming and Go

Chris James — Mon, 10 Jun 2019 00:00:00 +0000

The Go Time podcast on functional programming (fp) was an interesting listen and as someone who has done many years of both Scala and Go it's a topic I'm interested in.

The show talked about the stuff you'd expect, pure functions, immutability, higher-order functions etc. Here's some thoughts I'd like to add on top.

Errors

Let's have a look at a function written in a few programming languages

func Divide(x, y int) (int, error)

Scala

def Divide(x: Int, y: Int): Either[Exception, Int]

Java

public Int Divide(int x, int y)

What does the Go and Scala one have in common that Java does not?

You can trust the function from the signature. You do not need to read the function body to understand that it can fail.

In Java it would not be surprising for the function to throw an exception. Unless the author decided to use checked exceptions it means the user has to inspect the function body to see if anything potentially bad can happen.

At first throwing exceptions can seem convenient but in the long run become a pain.

FP values explicitness and and so does Go. It is recommended that you do not write packages that panic because that makes your code more difficult to work with and trust.

Errors as values

Go (and Scala) treat errors as values just like any other data in our system.

Crucially there is no special special syntax to work with errors. Error is just an interface that everyone happens to use as it's built-in.

Go 2 has proposals to make working with errors more convenient because error handling in Go right now can be fairly verbose.

a, err := Divide(2, 0)

if err != nil {
    return 0, fmt.Errorf("oh no %v", err)
}

b, err := Divide(2, 2)

if err != nil {
    return 0, fmt.Errorf("oh no %v", err)
}

return a + b, nil

For Go 2, we would like to make error checks more lightweight, reducing the amount of Go program text dedicated to error checking. We also want to make it more convenient to write error handling, raising the likelihood that programmers will take the time to do it.

The main problem with errors in Go is they do not safely compose and this is an important property (and advantage) of FP

Let's see how Scala let's us safely compose functions that can sometimes fail

val result: Either[Exception, Int] = for {
  a <- Divide(2, 0)
  b <- Divide(2, 2)
} yield(a + b)

Explanation

In this case, the 2nd Divide would not run and result would equal Left[ArithmeticException("Cannot divide by 0")]
If we changed the first divide so it would work it would run the 2nd Divide and then in the yield it is able to used the "unwrapped" a and b to simply add them as if they are two normal Ints

I need to stress this isn't special syntax, the for syntax works with any type that has map and flatMap. (cough monads cough)

Go 2, errors and generics

I worry about the proposals around errors in Go 2 because they are making errors "special". It is adding ad-hoc syntax and that doesn't feel very "Go like" to me.

Go 2 also has generics under consideration but the two topics seem to be discussed in separation a lot of the time. FP has identified patterns and datatypes for safely and easily working with errors; and they almost all rely on generics.

Instead of

type Reader interface {
        Read(p []byte) (n int, err error)
}

Maybe we could have

type Reader interface {
        Read(p []byte) Either<Int, error>
}

With a good generics solution we wouldn't have to have special syntax that only work with errors and errors can retain their non-special status they have right now.

We'd still need some kind of new syntax to help with composition (like the Scala one earlier) but it wouldn't be tied down to a specific datatype; and could be based on proven and existing patterns from other FP languages.

(Incidentally, I find Kotlin suffers from trying too hard to not be like Scala and instead invents syntax and idioms which are not as portable as learning about say.... monads.)

Wrapping up

Go already values a lot of principles from FP. Rather than creating new syntax that makes errors somehow different and special, wouldn't it be better if Go continued to lean on FP principles to help Go programmers work with errors a little more elegantly?

I could be totally off the mark (this is very likely, I am not a language designer). It can definitely be argued that bringing in more elements from FP is also not "Go like".

Generics are simple but not always easy. Adding ad-hoc syntax to work with errors is probably easy but not simple.

Go errors right now are simple and easy. If we dont want to tack on more complexity just for the sake of making code less verbose... maybe dont bother and leave errors as they are?

What's the difference between simple and easy?

Simple made easy is one of my most favourite tech talks.

Rich Hickey emphasizes simplicity’s virtues over easiness’, showing that while many choose easiness they may end up with complexity, and the better way is to choose easiness along the simplicity path.

How to go fast

Chris James — Sat, 02 Mar 2019 00:00:00 +0000

This article is for anyone who works in software and wants to keep life simple, less stressful but still keep their employer happy. You'll see how by making simple choices and sticking to well-understood, proven principles you can ship software and not burn everyone out along the way.

The business have arbitrarily given your team a year to make a website thing and they've got many grandiose ideas. Don't be tempted to spend lots of time planning up front imagining big architectures and backlogs around all this ambition. This is the path to failure, premature optimisation and broken promises.

The only promise you'll make is you will showcase what you've done every 2 weeks and accept steering from the business on actual problems and not feature decisions.

We know going fast is important but so is the quality of the system we're making and the mental health of our team. We will go really fast but we will always be doing it in a sustainable way.

We don't want to be stressed out about big-bang releases, having to do big scary refactors or obsessing over irrelevant estimates for work that we may not ever actually do.

Get things clear

Often things fall apart quickly when there isn't good alignment as to what the point of the work is. It's essential that everyone on the team (not just the blessed leaders) understand:

Who the user is
What problem they have that we're solving
What are the non-goals (to increase transparency, manage expectations, e.t.c.)
How do we know this is worthwhile and how do we validate our efforts?
How do we avoid the sunk-cost fallacy? What would make us radically change approach or cancel the project?

At all points everyone should understand the why of what they are doing. You're paying developers megabucks to think and if they don't understand the goal they won't be thinking about it. Instead, they'll either not think at all or focus on the wrong things.

Start writing software and iterate quickly

Start with hello world, fully deployed with continuous delivery.

This means if a commit goes to the main branch and passes the tests it should be deployed to production, ideally on a public URL that you can share with your users and stakeholders to get feedback quickly on your product.

We always want a small and focused vision for the team to be working on. Developing small things is easy and likely to succeed.

Thinking about delivering tons of features causes teams to over-complicate and makes decision-making difficult.

With lots of features up in the air there's the temptation to expand the team and this also slows things down. See: The Mythical Man-Month

Instead, we work on the next most important thing, based on business expertise, research, developer know-how and feedback from our users.

We will carry on continuously delivering small features iteratively indefinitely.

Rules and principles

1 Small co-located team

Small teams deliver quicker than big teams because the lines of communication are small and the amount of work being done at once has to be small. Large teams working on lots of things breeds confusion and complexity.

Having multiple small teams working on clearly separated things can work but requires good coordination. It's a lot easier to define those teams on architectural lines once you have a mature product. I think it's a mistake to take a new project with lots of unknowns and try and divide it amongst many teams because the architecture needs to be very flexible at this point and therefore the teams wont be able to work independently, they will be coupled.

In my experience being able to very easily talk to someone is important. The informal chats you have by just turning to your colleague is invaluable in breaking down the barriers of sharing ideas and building relationships. To me this is why I prefer to work in a co-located team because it minimises friction in communication.

If it is possible make sure the decision makers, such as the product owner be sitting with the team too. I wrote how work would be showcased every 2 weeks as a ritual but what's even better is quickly trying out an idea and getting immediate feedback from the stakeholders. This levels up the teams collective knowledge of the domain and massively reduces waste.

Pair programming most of the time

Pairing helps you to write better software and helps cross-pollinate skills, ideas and domain knowledge. It will help grow your team and will improve relationships which will in general make everyone happier!

Don't mandate it from 9-5 though as it becomes very tiring very quickly. People need time and space to think.

No branches and no pull requests

You have a small team that are pairing. You should be talking to each other all the time so you don't need pull requests to check code is OK. You're better off shipping it and getting feedback. You should be able to trust two people to get things right most of the time.

Pull requests are great for open source projects where you don't have implicit trust in every contributor. In a small team you don't need it and you need to take advantage of very tight feedback loops by reducing ceremony in writing software.

A big theme of these rules is removing barriers to feedback loops and this means we can change software easily. Someone shipping code that isn't ideal is not the end of the world, just fix it and move on - don't introduce complicated processes that slows everything down

Have a minimal deployment pipeline

Maintaining pipelines and various non-live environments can become burdensome. At least at the start of the project all you really need is:

Run tests
Deploy to live
Run smoke tests

If you find yourself trying to justify a non-live environment ask yourself

What is it that I cant test locally or in production? Why?
What are the costs and ramifications of being only able to test a thing in a particular environment?
What delays does this cause to shipping to production?

Most of the time you probably wont have good answer to these questions.

Continuously deliver to live on green builds.

This gives you:

Fast feedback
Force you to automate tests
Force you to write actually shippable code
Forces you to have good monitoring
Makes releases a stress-free, non-event

This has been written about a lot.

And is proven to be how high-performing teams work

Manually shipping a week or two weeks worth of work takes time and is risky; releasing small things constantly is easy and less risky.

Write tests

They enable you to iterate quickly with confidence. Cant do continuous delivery without them and you cant go fast if you have to manually check things all the time.

Monolith, not microservices

Writing distributed systems is a pain in the arse and I'd rather avoid that pain until I actually need to distribute my system. YAGNI in other words.

Refactoring in-process method calls is far easier than changing API calls. Deploying one application is much easier to deploy than orchestrating many system deployments etc etc.

I've worked on microservice projects in the past but my last project was a monolith and we moved so much faster because we cut out a huge number of issues to contend with.

In order to do microservices you have to think how you will break your system up into small components which leads nicely to...

Don't design up front

Just ship some code, if you make something useful and keep refactoring and iterating a design will emerge from reality rather than developers arguing over a whiteboard.

Refactor continuously

Bad code makes you go slow. Bad code tends to get worse. So refactor continuously, don't have a refactor sprint and don't put it off. Don't ask for permission, if it's bad just fix it. You want the pay-off from refactoring now, not 3 months later.

Half the perceived problems of monoliths come from people writing bad code. If you maintain the quality of the code real abstractions based on real usage will emerge and then if you need to break up your monolith there will be obvious ways to do.

Avoid writing an SPA, embrace progressive enhancement

SPAs bring a mountain of complexity that most projects don't need. You're probably not building Gmail so stop pretending you need to write an SPA. You can get React on your CV at 10% time.

Instead prefer to use progressive enhancement techniques by writing semantic HTML, decorated with CSS and a sprinkle of JavaScript to enhance the experience a little when necessary. Even better, don't write any client-side JavaScript if you can.

Don't obsess over pixel-perfect design

Writing HTML and CSS can be challenging and very time-consuming if you have a designer being overly precious about their design looking pixel perfect in every browser.

We are writing code for users not designers and most of the time they wont care if it looks a bit different in Firefox compared to Chrome.

Save time, effort and stress by challenging what is good enough for your users.

Simple designs are simple to implement, elaborate designs are complicated to implement. If your designer is giving you difficult to implement web pages, remind them we're trying to go fast.

Ship the first 80%, polish it later

Related to the above it is a total waste of time to really perfect a certain feature in respect to design and UX if it turns out that it is a feature that isn't needed.

I have seen a team spend weeks polishing a search feature on a website in respect to its design and its JavaScript only for it to be thrown away a few months later because the fundamental idea was wrong in the first place.

Ship enough to get feedback and prove the idea, once you've done that you can then spend time polishing it.

The people who write the software are the ones that are on support for it

This responsibility must be coupled with the empowerment to make the system maintainable.

If the people creating the system are the ones who have the threat of getting called at 3am because it is down they are more likely to ship more robust code.

Use mature, proven tools

It doesn't matter if you go fast with shiny thing if you then leave and no one can build or understand your code anymore.

Get a consensus in your team of what you'd all enjoy writing with that is appropriate for the task at hand. Don't pick up a new programming language that is unfamiliar to everyone - remember we're trying to go fast.

It must be simple to build the software

There should be a minimal amount of setup required to run and test the project. It is totally unacceptable to be unable to build a project just because it hasn't been touched for a few months or a key person has left. If it takes an individual more than an hour to setup the project you have failed hard.

With technologies like Docker it is trivial to configure a project so it has all it's 3rd party dependencies such as databases available locally.

It should not be reliant on a particular setup of a computer (beyond installing a few key things such as the JVM, Go or whatever). It must not be reliant on a 3rd party system for it to run & test locally.

To help enforce this, configure whatever CI tool you are using to automatically build the software every morning; even if no one has committed to it.

If the build is red, stop

The tests must always be passing. If you have unreliable tests, stop and fix them. Flaky tests have cost me many hours of my time in the past.

Actually sit with users and watch them work

What the business thinks the users needs vs what the user actually needs is often very different. I have been lucky enough to witness this first hand where we saved ourselves a ton of work by actually watching a user at work and we were able to identify a very cheap and simple solution that they actually want vs the extremely costly solution the business proposed.

User stories are a conversation starter

If a user story prescribes exactly what software to write, who made that decision? Why? How did they know they were correct?

This kind of approach of a "tech lead" or whatever deciding how to write the software creates disillusioned developer robots who have no sense of ownership and stories that are "blocked" because they have not been blessed by some over-controlling leader.

User stories should describe a user problem and when it is picked up the team figures out a way to do it and then they execute that plan, simple as that. This has a higher chance of success because more people are thinking about the problem and makes people feel involved.

Each ticket have some kind of measure of success that should be validated when enough data/research is gathered.

Understand that we are all software developers

In my imaginary company if you consider yourself a "backend" developer and consequently get blocked because the "frontend" developer is sick consider yourself fired. Same if the UX designer is away and you don't want to take a punt at how a particular interaction should pan out.

You should have an appreciation of all aspects of software development. That's not to say you have to be an expert in everything but being blocked because a particular person is away is unacceptable.

You should feel empowered enough to give whatever task a try. It doesn't matter if it's not perfect because it's better to ship something that is 70% good and then when the expert returns you can then iterate on the software to make it better.

When I said pairing earlier I didn't just mean "backend developers" or whatever. Everyone should be involved. If a UX designer is interviewing users it is their responsibility to bring along someone else to get experience of this skill and learn about the users. This kind of knowledge sharing is vital to make individuals that feel invested in the project and the team.

Have 10/20% time

Allocate time for everyone to work on whatever they want and be strict about it.

We work in a creative profession and we need some space to breathe, learn things and try out different ideas.

This is a great perk for the team, can help avoid burnout and can help bring new ideas that can bring great improvements to your systems down the line.

It's also important because it allows everyone to spend time developing their skills, not just people who happen to have a lot of free time and little responsibility. 10% time is important for diversity.

Don't plan extensively beyond 6 weeks

Having a vision of the future is fun, but keep it vague and aspirational. Remember a team focused on one or two features is far more productive than one which is worrying about 10 imaginary requirements.

No Jira worship, no estimates, no timekeeping, no burn-up charts

All a massive waste of time, causes friction and I haven't seen a single burn-up chart in my career deliver any kind of actual useful information.

Remember we have committed to showing real working software frequently, not imaginary charts which are at best educated guesses. Finger in the air guesses are fine, adding up story points for tickets that have no hope of being actually looked at in the next 3 months is worthless.

Individuals and interactions over processes and tools

By all means have a simple, physical wall with the columns of ready for dev, in dev & done but much more than that it just becomes silly.

JIRA, or any other tool should be used to help your process; not define it.

Keep work-in-progress (WIP) low

It's important to understand what WIP really means. I've worked in teams where a story is "done" but not actually in the hands of users. If you've written code that's not being used by your customer, that code is work in progress.

Ever worked in a team with many stories in progress, and the team seems to be in a perpetual state of chaos with no clear value being actually delivered?

This usually stems from:

Stories being parked due to being "blocked"
Not "real" stories, but technical tasks that may have some hypothetical value someday
Stories with unclear problems to solve
Developers not empowered to solve problems, but are rather seen as JIRA ticket robots

Over time high WIP causes tons of work to not being finished properly which will add a ton of friction to your team and reduce productivity.

Optimise for flow, not resource allocation

When you start work have a clear understanding of how it will get into the hands of users and if it's unclear, don't even start it.
Be wary of depending on other "silos" and if you can find ways to remove them.
Teams can also form inner silos which you need to avoid, don't be in scenarios where only certain developers can do certain tasks.

If you empower developers and make them responsible for delivering value they will find ways to unblock themselves and improve flow.

Wrapping up

The way to move fast is not imagine the perfect product, architecture, design a backlog and then execute it.

The way you move fast is by iterating quickly with low-effort, proven methodologies and communicating with each other as you learn through your iterations.

Optimise your processes so that your team can fearlessly and quickly change software. This way you will be able to build a good product guided by constant feedback loops.

There is no one true way to create great teams who can ship great products but in my experience so far these principles seem to stand up well. Your way might be different and that's fine but ask yourself if your team is as good as it could be. Retrospection is always important to help your team adapt to new conditions.

Footnotes

This article has been translated into Korean

Work 2018

Chris James — Sun, 30 Dec 2018 00:00:00 +0000

This post is a summary of work stuff over the past year which covers

The ways of working, agile and innovation
Microservices vs monoliths
Kotlin
Writing a book (kinda)

The team and the way we work

I rejoined Springer Nature (SN) August 2017 on the exciting premise of being in an "innovations team" where we would attempt to maybe make scientific publishing a bit better using technology.

I was a little sad to be moving from a role mainly writing Go to Kotlin but programming languages aren't everything.

What mainly drew me to the role was the understanding that we would be empowered to work in the way we felt was best.

I am a big believer in "agile"; the actual agile, not the project management nightmare. Stuff like

Individuals and interactions over processes and tools

and

Responding to change over following a plan

We are a small team of 3 developers, UX and a BA. We work very collaboratively with little ceremony. To give an idea our "stand up" is having a chat about what were working on and our general thoughts on the work. By that definition we have at least 3 standups a day.

The responsibilities within the team are more or less shared and we rely on each other's relevant expertise to fill in the blanks when needed. I wont be blocked if our UX is away at a meeting, I'll just take a punt knowing it probably wont be that bad.

We talk a lot and we iterate quickly and actually talk to real users, rather than fighting through many layers of hierarchy.

With all of us feeling empowered and involved it means ideas come to us quickly and if we think we have a good idea we'll discuss it and try it out. We dont worry about 2 week long inceptions and prioritising backlogs; if we think its the best thing for us to try now, we'll do it.

If we get things wrong, we dont sweat it; we can change software. Someone very clever said

A prototype is worth a thousand meetings

The other key to us moving quickly is really acknowledging the rule that the last 10% of work takes 90% of the time and just avoid doing it (at least in the short term).

We often say

Perfect is the enemy of good

We would rather ship something that we hope is good, get feedback and iterate on it rather than tying ourselves in knots over perfection.

In terms of the way we work, I couldn't be happier.

But what are we working on?

The work

We were connected with a journal who had some exciting ideas around data and community building. We thought we could create a minimal viable product (MVP) peer-review system (PRS) to experiment around these topics to hopefully learn how SN can help in these areas.

We thought Trello was cool and we had actually interviewed someone who managed a journal entirely in Trello so it seemed like a nice way to bootstrap a "system" quickly.

Our first iteration was a simple HTML form for an author to submit a manuscript which would create a card in Trello. An editor could then inspect the details and then manage the whole process of peer review manually themselves (for now); when they were happy with the manuscript if they dragged it into the column "Ready for publication" our web hook would listen to that event and then create an XML payload for publishing. We turned this around in about a week and felt really rad.

From there we kept iterating, adding a simple peer reviewer search and inviting mechanism (still in Trello).

Eventually though Trello's limitations in terms of UI were becoming burdensome. It was great to prototype and we learned some interesting things from the users about their processes but ultimately we needed to control the UI. We didnt throw it all out and start again though; we just kept iterating, creating new web pages to address our new needs until eventually we didnt need Trello at all.

We "launched" with our MVP submission system for the journal in September. Since then we have carried on iterating based on user feedback and have been working to try and make the arduous quality assurance process for a manuscript a more pleasant experience.

Innovation vs invention and politics

What exactly is an innovations team? Some would think that we should be using blockchain terraform kubernetes machine-learning wizzpops to change the world. I have often wondered why what we build isn't massively transformative.

The talk Building a winning product and UX strategy from the Kano Model by Jared Spool gives a lot of clarity for me.

The real tension is between invention and innovation. Innovation is where we improve a customer's experience so that they are "delighted". It's moving the needle from basic expectation to something that's awesome.

Invention is when you are launching moonshots at ideas that are wild and likely to fail but if they work, could radically change the business.

When I think about how software teams should be working, they should all be empowered to be innovating. This to me is what agile is all about. You should be shipping a product, collecting feedback and as a team understanding it as best you can and then trying out new ideas to move the needle.

This is exactly how we work and I feel this is how every team should work. You dont work on a project (because there's no such thing as projects in software), you work on a product. You start small and keep iterating for as long as the product lives.

As your product grows so does your team, in terms of improving the way it works, how it collects information and ideas around a domain so that it can deliver better ideas quicker.

Maybe software wouldn't be so expensive if we treated it properly rather than shipping on a particular date and then letting it rot; then re-writing it all over again forever and ever.

My conclusion for all of this, is the whole concept of an innovations team is an anti-pattern. Which is a bit weird to say given that was what was sold to me when I rejoined SN.

The joy of monolith

Until this year I have mainly worked on microservices systems for a good 4-5 years. I really like the approach of microservices in terms of creating small systems within bounded contexts and then gluing them together.

I've always understood that the cost of this is in terms of increased complexity in terms of continuous integration (CI), deployments and making sure your system actually works with unreliable network calls and evolving APIs so they dont break each other (e.g. consumer driven contracts)

This year we monolith'd our system and it has been incredibly liberating. I sometimes sit in meetings with other teams as they discuss their technical stuff and I had forgotten how much of a pain building distributed systems is.

This has really cemented the idea that you should always start monolith when building a system.

YAGNI is getting ignored too much these days, every recruitment email tells me how its a "greenfield microservices project" - sounds awful!

It's far easier and quicker to build your monolith and then you get to see the actual areas in your system that could be separate services when the API is a bit more mature.

A lot of the perceived downsides of monoliths are often fixed with just good programming practices.

At some point a system does get too big and you'll need to break it up, but maybe we should worry about building a successful system before breaking it up?

Kotlin

After writing Kotlin for about 18 months I conclude it's alright.

It feels like a nice happy medium between Scala and Java, but I do miss a number of Scala features such as for comprehensions and pattern matching.

These features being missing often makes Kotlin code more complicated than the Scala equivalents which flies in the face of the opinion of Kotlin being simpler than Scala. Abstractions can and should bring simplicity when used correctly.

Scala for better or worse gives you a lot of abstraction power which most developers will make a mess with if I'm being honest; often myself.

The Kotlin compiler is still quite slow; especially compared to Go but its much faster than Scala.

Ad-hoc Kotlin-isms

I'm personally not a fan of the ad-hoc syntax it has added for commonly known problems. It feels like to me they have really tried to avoid acknowledging monads and the like but have invented different things to learn instead.

I've run into occasions where I've wanted to refactor code from a T? (nullable T) to a List<T> which is harder than it should be.

In Scala the abstraction is clearly modeled so you can call map on them in a consistent way. So if you change the type there's very few code changes to make.

In Kotlin if it is nillable you do things like myThing?.let{it + 1}. But with lists its myList.map{it + 1}. To me it's annoying that they're different. A nullable thing should be seen as a collection of one or zero items in it.

Ecosystem

From what I read a lot of people writing Kotlin still want to feel the comfort of Java land in respect to relying heavily on frameworks like Spring. I really dont want to use a massive framework and neither does anyone in our team.

Thankfully SN has invested a lot of time and effort in allowing some developers to create a lovely library called http4k which offers a simple API to create web servers.

I still feel Go is probably my weapon of choice for a general purpose language but it's certainly not a deal-breaker for me to write Kotlin.

Writing Learn Go with tests

SN is nice enough to have 10% time. In March I thought it would be fun to do a TDD guide of a "Hello, world" function in Go.

I stumbled on a nice approach of starting with a simple function and iterating on it, adding new features of Go as the software grows and gets refactored which results in quite a nice flow of learning.

It got a lot of attention on the social media thing, getting top of hacker news and lots of internet points in Github stars.

This led to me feeling very motivated and since then I've written a lot of content covering most of the Go programming language syntax by TDD-ing useful functionality.

You can find it all on Github

For me the most amazing thing is some members of Chinese Go community have translated it into Chinese!

A few chapters have been contributed by other members from the community and a lot of my terrible grammar has been fixed. Overall there are 39 different contributors to the project.

Using gitbook it has a website and I used Pandoc to generate an epub file from the markdown that the content is written in.

The feedback from the community has been invaluable and very rewarding. Committing knowledge to paper is challenging and trying to explain what you know helps solidify your ideas and I am sure the whole experience has improved me as a developer.

I am immensely proud of this work and intend to press on with it to add more content in 2019.

Summary

Small empowered teams can deliver useful software quickly
Innovations team is an organisational smell
Build a useful, well-engineered monolith first. Then you'll see how you can break it up into smaller pieces as you feel the pain of a big codebase. Taking on the challenges of microservices will just slow your project down at first.
Kotlin is pretty good, but like all programming languages has its annoyances
Writing down stuff you're interested in is challenging and rewarding. When people say nice things about your work its very nice, so if you like someone's work do tell them as it'll make their day.

The Tests Talk

Chris James — Tue, 18 Dec 2018 00:00:00 +0000

This month I had the opportunity to present some ideas floating around in my head around tests, refactoring and design at the Go London User Group.

Here's a link to the video

Here are the slides

If you're not into videos, here's wordy version of it.

Software

The promise of software is that it can change. This is why it is called soft ware, it is malleable compared to hardware. A great engineering team should be an amazing asset to a company, writing systems that can evolve with a business to keep delivering value.

So why are we so bad at it? How many projects do you hear about that outright fail? Or become "legacy" and have to be entirely re-written (and the re-writes often fail too!)

How does a software system "fail" anyway? Can't it just be changed until it's correct? That's what we're promised!

A lot of people are choosing Go to build systems because it has made a number of choices which one hopes will make it more legacy-proof.

Compared to my previous life of Scala where I described how it has enough rope to hang yourself, Go has only 25 keywords and a lot of systems can be built from the standard library and a few other small libraries. The hope is that with Go you can write code and come back to it in 6 months time and it'll still make sense.
The tooling in respect to testing, benchmarking, linting & shipping is first class compared to most alternatives.
The standard library is brilliant.
Very fast compilation speed for tight feedback loops
The Go backward compatibility promise. It looks like Go will get generics and other features in the future but the designers have promised that even Go code you wrote 5 years ago will still build. I literally spent weeks upgrading a project from Scala 2.8 to 2.10.

Even with all these great properties we can still make terrible systems, so we should look to the past and understand lessons in software engineering that apply no matter how shiny (or not) your language is.

In 1974 a clever software engineer called Manny Lehman wrote Lehman's laws of software evolution.

The laws describe a balance between forces driving new developments on one hand, and forces that slow down progress on the other hand.

These forces seem like important things to understand if we have any hope of not being in an endless cycle of shipping systems that turn into legacy and then get re-written over and over again.

The Law of Continuous Change

Any software system used in the real-world must change or become less and less useful in the environment

It feels obvious that a system has to change or it becomes less useful but how often is this ignored?

Many teams are incentivised to deliver a project on a particular date and then moved on to the next project. If the software is "lucky" there is at least some kind of hand-off to another set of individuals to maintain it, but they didn't write it of course.

People often concern themselves with trying to pick a framework which will help them "deliver quickly" but not focusing on the longevity of the system in terms of how it needs to evolve.

Even if you're an incredible software engineer, you will still fall victim to not knowing the future needs of your system. As the business changes some of the brilliant code you wrote is now no longer relevant.

Lehman was on a roll in the 70s because he gave us another law to chew on.

The Law of Increasing Complexity

As a system evolves, its complexity increases unless work is done to reduce it

What he's saying here is we can't have software teams as blind feature factories, piling more and more features on to software in the hope it will survive in the long run.

We have to keep managing the complexity of the system as the knowledge of our domain changes.

Refactoring

There are many facets of software engineering that keeps software malleable, such as:

Developer empowerment
Generally "good" code. Sensible separation of concerns, etc etc
Communication skills
Architecture
Observability
Deployability
Automated tests
Feedback loops

I am going to focus on refactoring. It's a phrase that gets thrown around a lot "we need to refactor this" - said to a developer on their first day of programming without a second thought.

Where does the phrase come from? How is refactoring just different from writing code?

I know that I and many others have thought we were doing refactoring but we were mistaken

Martin Fowler describes how people are getting it wrong

However the term "refactoring" is often used when it's not appropriate. If somebody talks about a system being broken for a couple of days while they are refactoring, you can be pretty sure they are not refactoring.

So what is it?

Factorisation

When learning maths at school you probably learned about factorisation. Here's a very simple example

Calculate 1/2 + 1/4

To do this you factorise the denominators, turning the expression into

2/4 + 1/4 which you can then turn into 3/4.

We can take some important lessons from this. When we factorise the expression we have not changed the meaning of the expression. Both of them equal 3/4 but we have made it easier for us to work with; by changing 1/2 to 2/4 it fits into our "domain" easier.

When you refactor your code, you are trying to find ways of making your code easier to understand and "fit" into your current understanding of what the system needs to do. Crucially you should not be changing behaviour.

An example in Go

Here is a function which greets name in a particular language

func Hello(name, language string) string {

  if language == "es" {
     return "Hola, " + name
  }

  if language == "fr" {
     return "Bonjour, " + name
  }

  // imagine dozens more languages

  return "Hello, " + name
}

Having dozens of if statements doesn't feel good and we have a duplication of concatenating a language specific greeting with , and the name. So I'll refactor the code.

func Hello(name, language string) string {
    return fmt.Sprintf(
        "%s, %s",
        greeting(language),
        name,
    )
}

var greetings = map[string]string {
  es: "Hola",
  fr: "Bonjour",
  //etc..
}

func greeting(language string) string {
  greeting, exists := greetings[language]

  if exists {
     return greeting
  }

  return "Hello"
}

The nature of this refactor isn't actually important, what's important is I haven't changed behaviour.

When refactoring you can do whatever you like, add interfaces, new types, functions, methods etc. The only rule is you don't change behaviour

When refactoring code you must not be changing behaviour

This is very important. If you are changing behaviour at the same time you are doing two things at once. As software engineers we learn to break systems up into different files/packages/functions/etc because we know trying to understand a big blob of stuff is hard.

We don't want to have to be thinking about lots of things at once because that's when we make mistakes. I've witnessed so many refactoring endeavours fail because the developers are biting off more than they can chew.

When I was doing factorisations in maths classes with pen and paper I would have to manually check that I hadn't changed the meaning of the expressions in my head. How do we know we aren't changing behaviour when refactoring when working with code, especially on a system that is non-trivial?

Those who choose not to write tests will typically be reliant on manual testing. For anything other than a small project this will be a tremendous time-sink and does not scale in the long run.

In order to safely refactor you need unit tests because they provide

Confidence you can reshape code without worrying about changing behaviour
Documentation for humans as to how the system should behave
Much faster and more reliable feedback than manual testing

An example in Go

A unit test for our Hello function could look like this

func TestHello(t *testing.T) {
  got := Hello(“Chris”, es)
  want := "Hola, Chris"

  if got != want {
     t.Errorf("got '%s' want '%s'", got, want)
  }
}

At the command line I can run go test and get immediate feedback as to whether my refactoring efforts have altered behaviour. In practice it's best to learn the magic button to run your tests within your editor/IDE.

You want to get in to a state where you are doing

Small refactor
Run tests
Repeat

All within a very tight feedback loop so you don't go down rabbit holes and make mistakes.

Having a project where all your key behaviours are unit tested and give you feedback well under a second is a very empowering safety net to do bold refactoring when you need to. This helps us manage the incoming force of complexity that Lehman describes.

If unit tests are so great, why is there sometimes resistance to writing them?

On the one hand you have people (like me) saying that unit tests are important for the long term health of your system because they ensure you can keep refactoring with confidence.

On the other you have people describing experiences of unit tests actually hindering refactoring.

Ask yourself, how often do you have to change your tests when refactoring? Over the years I have been on many projects with very good test coverage and yet the engineers are reluctant to refactor because of the perceived effort of changing tests.

This is the opposite of what we are promised!

Why is this happening?

Imagine you were asked to develop a square and we thought the best way to accomplish that would be stick two triangles together.

We write our unit tests around our square to make sure the sides are equal and then we write some tests around our triangles. We want to make sure our triangles render correctly so we assert that the angles sum up to 180 degrees, perhaps check we make 2 of them, etc etc. Test coverage is really important and writing these tests is pretty easy so why not?

A few weeks later The Law of Continuous Change strikes our system and a new developer makes some changes. She now believes it would be better if squares were formed with 2 rectangles instead of 2 triangles.

She tries to do this refactor and gets mixed signals from a number of failing tests. Has she actually broken important behaviours here? She now has to dig through these triangle tests and try and understand what's going on.

It's not actually important that the square was formed out of triangles but our tests have falsely elevated the importance of our implementation details.

Favour testing behaviour rather than implementation detail

When I hear people complaining about unit tests it is often because the tests are at the wrong abstraction level. They're testing implementation details, overly spying on collaborators and mocking too much.

I believe it stems from a misunderstanding of what unit tests are and chasing vanity metrics (test coverage).

If I am saying just test behaviour, should we not just only write system/black-box tests? These kind of tests do have lots of value in terms of verifying key user journeys but they are typically expensive to write and slow to run. For that reason they're not too helpful for refactoring because the feedback loop is slow. In addition black box tests don't tend to help you very much with root causes compared to unit tests.

So what is the right abstraction level?

Writing effective unit tests is a design problem

Forgetting about tests for a moment, it is desirable to have within your system self-contained, decoupled "units" centered around key concepts in your domain.

I like to imagine these units as simple Lego bricks which have coherent APIs that I can combine with other bricks to make bigger systems. Underneath these APIs there could be dozens of things (types, functions et al) collaborating to make them work how they need to.

For instance if you were writing a bank in Go, you might have an "account" package. It will present an API that does not leak implementation detail and is easy to integrate with.

If you have these units that follow these properties you can write unit tests against their public APIs. By definition these tests can only be testing useful behaviour. Underneath these units I am free to refactor the implementation as much as I need to and the tests for the most part should not get in the way.

Are these unit tests?

YES. Unit tests are against "units" like I described. They were never about only being against a single class/function/whatever.

Bringing these concepts together

We've covered

Refactoring
Unit tests
Unit design

What we can start to see is that these facets of software design reinforce each other.

Refactoring

Gives us signals about our unit tests. If we have to do manual checks, we need more tests. If tests are wrongly failing then our tests are at the wrong abstraction level (or have no value and should be deleted).
Helps us handle the complexities within and between our units.

Unit tests

Give a safety net to refactor.
Verify and document the behaviour of our units.

(Well designed) units

Easy to write meaningful unit tests.
Easy to refactor.

Is there a process to help us arrive at a point where we can constantly refactor our code to manage complexity and keep our systems malleable?

Why Test Driven Development (TDD)

Some people might take Lehman's quotes about how software has to change and overthink elaborate designs, wasting lots of time upfront trying to create the "perfect" extensible system and end up getting it wrong and going nowhere.

This is the bad old days of software where an analyst team would spend 6 months writing a requirements document and an architect team would spend another 6 months coming up with a design and a few years later the whole project fails.

I say bad old days but this still happens!

Agile teaches us that we need to work iteratively, starting small and evolving the software so that we get fast feedback on the design of our software and how it works with real users; TDD enforces this approach.

TDD addresses the laws that Lehman talks about and other lessons hard learned through history by encouraging a methodology of constantly refactoring and delivering iteratively.

Small steps

Write a small test for a small amount of desired behaviour
Check the test fails with a clear error (red)
Write the minimal amount of code to make the test pass (green)
Refactor
Repeat

As you become proficient, this way of working will become natural and fast.

You'll come to expect this feedback loop to not take very long and feel uneasy if you're in a state where the system isn't "green" because it indicates you may be down a rabbit hole.

You'll always be driving small & useful functionality comfortably backed by the feedback from your tests.

Wrapping up

The strength of software is that we can change it. Most software will require change over time in unpredictable ways; but dont try and over-engineer because it's too hard to predict the future.
Instead we need to make it so we can keep our software malleable. In order to change software we have to refactor it as it evolves or it will turn into a mess
A good test suite can help you refactor quicker and in a less stressful manner
Writing good unit tests is a design problem so think about structuring your code so you have meaningful units that you can integrate together like Lego bricks.
TDD can help and force you to design well factored software iteratively, backed by tests to help future work as it arrives.

The Web I Want

Chris James — Mon, 20 Aug 2018 00:00:00 +0000

I originally posted this at Dev.to, where you can see some comments

This post will very much sound like I want you all to get off my lawn - because I do.

How has the web become like this?

The other day my partner was browsing the web on her two-year-old Chromebook, it struggled to run a number of sites as they loaded ads, gifs, videos and tons of JavaScript to do absolutely nothing of value.

Every day I am frustrated at my phone as it chugs along trying to display a blog post. The post is there behind a sea of JavaScript and autoplaying videos.

This all sounds a bit first world problems, but it's far, far worse for those in developing nations. Do you know how bad satellite Internet is compared to what most of us have here in the west? It is awful.

Graph of mobile connection speeds

The World Wide Web is supposed to be a leveler, something that brings knowledge everywhere and yet developers every day are making it harder for those who need the Internet to work well the most.

Let's take a look at how things have "progressed" with the Internet.

20ish years ago

I knew nothing about programming or the web, but using HTML that was returned in response to HTTP requests, I managed to build a website.

It delivered as much content as most websites today. It was more accessible, ran faster and easier to develop then 90% of the stuff you'll read on websites teaching about web development. It did not need a build tool, transpilation, a framework or any of the other nonsense that is now required to build a website.

10-15 years ago

People were fed up with nested tables and spacer images. The web was losing its roots of being a content delivery platform.

I was working on websites like the above in my placement year on my degree. I started reading articles on A list Apart about how we should be pushing for semantic markup, where HTML simply describes a document of content, and then it is styled using this thing called CSS.

Eventually I ran into CSS Zen Garden which is a website that showcases what you could do with CSS.

The idea is the markup is the same and the website has submissions from developers showing different designs purely using CSS.

The HTML remains the same, the only thing that has changed is the external CSS file. Yes, really.

This website had a profound impact on my attitude to web development. It clicked. HTML for content, CSS for style. No JavaScript required, no tables needed for layout, no spacer images.

Just pure HTML decorated with CSS.

It felt exciting to be part of a community that took real pride in delivering beautiful looking content in the leanest, simplest and most accessible way possible.

Not only did this make websites more accessible and fast to run but in some ways made them easier to develop. Suddenly generating the markup from the server wasn't horrible, because you didn't worry about style, just the content and the hypermedia controls.

Still things were quite difficult. CSS support wasn't amazing, and we still didn't have a lot of semantic elements with HTML4 so there were a lot of divs. Firebug had just come out which was a huge boost, but it was still hard to make a consistent experience.

I seem to recall Gmail coming out around this time, and it was amazing. Suddenly the web could be actual applications. This was interesting and groundbreaking, but had a real negative impact too. Soon enough people thought that the promotional website for their local pub had to have as much JavaScript as Gmail.

5 years-ish till present

I now browse the Internet with a few tabs open and I have somehow downloaded many megabytes of data, my laptop is on fire and yet in terms of actual content delivery nothing has really changed.

Here is a not exaggerated summary of today's attitudes

Scores of people who just want to deliver their content and have it look somewhat nice are convinced you need every web technology under the sun to deliver text and some pictures.

You're not a frontend developer unless you know Vue/React/whatever. You're mocked if you dare to just write the bare minimum JavaScript you need for your website

The page refreshing is seen as a massive problem for users, and it must be avoided at all costs.

For some reason people are building tons of SPAs (single page web apps). The reasons stated are for speed and ease of developer use. (I don't buy sending tons of JavaScript to a browser will ever be as fast as just some damn HTML. Also, it's not as easy as putting some HTML files on the Internets)

You see these laughable posts where developers jump through dozens of hoops to make their website "fast and performant". They struggle because of the underlying technical choices, and then I'm still downloading half a megabyte of data to read 500 words. It's embarrassing.

There are a few things I want you to take away from this post

Most websites are about delivering and exploring content. HTML is amazing for this, and you don't need JavaScript.
In fact JavaScript is actively harmful. It is poison for low power devices, horrible for people on low bandwidth and requires extra effort to keep it accessible and fast.
So much effort poured into all this JavaScript. You seem to have to learn so much, new frameworks, new build tools, new ways of testing; and most of the time it is making the experience worse for your users. The churn is unreal too, every year it feels like there is a new framework or tool that is going to change everything.
In practice your website would do everything it needed to with some HTML files linking to some CSS.

So what is The Web I want?

Semantic, accessible, HTML. This means it works for everyone, is fast to download and is less effort than writing React components.
CSS to make it look pretty. Don't use JavaScript for visual effects.
Small images, when needed. Enough of these hilarious 4mb GIFs.
Most websites should be ready to read after a 10kb download.
If I cant use your website without JavaScript being turned on then you don't know what you're doing.

So no JavaScript, really?

JavaScript of course still has its place and when used tastefully can improve the usability of a website. However, think carefully about the libraries and frameworks you pull in. Maybe you can accomplish what you need without bringing in JQuery, modern browsers have excellent APIs built in these days. Aim for progressive enhancement with JS. Your website should still work without JS turned on.

Regarding single page apps (SPAs), I genuinely believe too many people are making them. GMail is a web application and as such deserves a framework. Your blog platform? Not so much. Remember these frameworks not only put a lot of strain on user's experience, but it's also just a lot to learn and maintain. Maybe your time could be served better.

But my product owner says we need all these bells and whistles!

As a professional, it is up to you to take a stand. You are the expert, not the customer.

You should be informing them how Amazon learned that every 100ms in page load cost them a 1% of sales.
You should be informing them that making a carousel of their product catalogue is slow and annoying and expensive to make compared to just listing them in a <ul>.
You should let them know that if they really want their groundbreaking poetry to reach around the world that they better make sure their website works on old hardware with slow Internet connections.

If I told the builder of my home to make it out of straw, I would hope she would convince me otherwise.

Of course, circumstances are tricky and sometimes people will ignore you but when you look at the state of some popular websites today you can only conclude that people either don't know what they're doing or are just not pushing back on bad requirements enough.

Wrapping up

Take a look at the performance tab in developer tools for your website. Does it make you proud? If not, take some pride and start cutting away the cruft you thought you needed.

Let's take pride in making lean, accessible, simpler to execute websites by using simple technologies that work everywhere. There are moral and technical arguments for taking this approach.

Remember that the goal of most websites is delivering useful textual content, and all you really need for that is HTML.

Then we might not have to buy new laptops every 2 years and people less fortunate than you have a chance of actually using the web; the way it was intended.

Why you should deploy on Friday afternoon

Chris James — Mon, 28 May 2018 00:00:00 +0000

I originally posted this at Dev.to where lots of people agreed and disagreed.

Don't deploy on Friday afternoons!

This expression is taken as programmer wisdom but I hate it. I'm going to try and kill it, with words and experience.

The motivation behind it is sound. I don't want to spend my Friday nights debugging a production problem either.

To me the expressions smacks of unprofessionalism. Software development as an industry has a poor reputation and phrases like this do not help.

If you went to an ER on a Friday afternoon and was turned away because the doctors don't trust their own tools and know-how, how would you feel?

If we want people to take our craft seriously we need to own it and not give the impression that we dont understand the systems are we making enough to make changes at the end of the week.

Why a lot of people don't want to deploy on Fridays

You don't understand your system. You're making changes but you're not 100% sure it'll be OK. Ask yourself why this is. Why are you scared of changing your software?
Poor monitoring. If the users are the first to tell you if something is wrong, that feedback loop spills into time at the pub, rather than when you deploy.
Overly-complicated rollback/branching. How is your disaster recovery? How easy is it to push a fix once you solve a bug?

Continuous Deployment/Delivery (CD)

I have worked on teams that have deployed new versions of various services in a distributed system multiple times at 4:30pm and not broke a sweat.

Why? Because deployment is fully automated and is a non-event. Here is the groundbreaking process.

Write some code
git commit -am "made the button pop"
git pull -r && ./build && git push
Make a cup of tea
It's now live.

Not so long ago it was considered normal for there to be releases every 6 months, or even just one at the end of a project.

The forward thinkers in that age saw problems with this

Poor feedback loops
Stressed development teams
Catastrophic failures.

So the industry as a whole worked on lots of tooling, techniques and best practices to allow us to release software far quicker.

Recognising that releasing often reduces risk is generally accepted nowadays but teams still often settle on weekly or fortnightly releases; often matching the cadence of their sprints.

Which CD?

Annoyingly, continuous delivery and continuous deployment are very related concepts, with the same acronym; but refer to slightly different things:

C Delivery means you've set up your system so that software is very simple to deliver your software to users. Think pipelines, tests, automation e.t.c.
C Deployment means if a commit is pushed to your main branch, and it passes all the checks, it is automatically deployed to live.

For the purpose of this post, when I say "CD" I mean "continuous deployment".

What are the problems with weekly/fornightly releases?

The feedback loops are still not great. If you do your release there can be quite a lot of commits going live and if something is wrong it can be challenging to figure out exactly what broke. Especially if you wrote it 2 weeks ago.
Still overly reliant on manual processes. I have seen teams actually skip a release because a QA was on holiday. This is surely unacceptable in 2018. Manual testing does not scale into the future. People leave, things get forgotten, etc.
Let's you fall into the trap of writing stories that are dependant on other stories being finished in a "sprint". When they aren't things can get very complicated.

With CD we recognise that we can go further, deploying new software to live every time the build is green. This has some amazing benefits,

Extremely fast feedback loops. No longer do you have to think about code you wrote 2 weeks ago when there is a problem in live.
Forces best practices. In order to be able to deploy to live on green you need excellent monitoring and tests. These are all good things in their own right.
Reduces stress. "Releasing" is no longer a thing any more. You can be confident in writing your software again!
Vastly improves agility. Found a bug? Just fix it! This encourages a more lean way of working vs lots of upfront planning. There isn't even an option for a convoluted release process, you have to keep it simple.
Forces you to work on stories that are actually releasable. Not dependent on story x y and z. Forces the best practices on user stories that everyone acknowledges but often people ignore.

But what if you break things?

Often people say with CD

Yeah it's nice but what if it breaks? We should have a QA check things over

Here's the thing, no process in the world prevents bugs. You will ship broken code. What's really important is how quickly you can detect and recover from it. Hoping manual testing will catch everything is wishful thinking.

How to CD on a new project

It is much easier to do CD on a new project since you can start small and evolve.

Generally your work should be centered on delivering the most valuable user journeys first, so this is an excellent chance to practice how to ensure that feature works without any humans checking anything.

Write an end-to-end test. These are expensive to write and run and should only be reserved for your most important journeys
Have monitoring with threshold alerts for when things go wrong
Set up your pipeline so that when your code is pushed all the automated tests are run, if they pass go to production.
Have some kind of green/blue release mechanism. Run your automated tests on the deployed release candidate and if they dont pass, dont ship it.

For each subsequent story ask yourself

How will we know this is working? (monitoring)
What tests do I need to have enough confidence this will work without any humans checking. Not every story needs a full end-to-end test on a huge distrubuted system but obviously you'll need some tests.
Is this story as small as it can be? If your user stories are massive they are more likely to go wrong. If the story takes a week then that's back to slow feedback loops.
If you cant answer these questions then you need to rethink the story. Notice these are all just basic agile principles for user stories. Continous delivery forces you to adhere to the principles that so often get ignored

How to CD on an existing project

Peel away at the manual process

You may have some kind of "run book" that is used when shipping the software. See what you could do to automate it.
Find out all manual processes are happening. Ask why they are needed and what could be done to automate them.

CD up to staging.

Some companies have many environments in their delivery pipeline. A good first start is to automatically ship all the way up to the environment before live. A better step is remove as many of them as you can. It's ok to have some kind of "dev" environment to maybe experiment with but ask yourself why cant just test these things locally in the first place.

Identify a sub-system you could work with as a starting point

If you're working with a distributed system you might be able to identify a system which is easier to CD than the rest. Start with that because it'll give your team some insights into the new way of working and can help you begin to break the cultural barriers.

CD is a cultural issue as much as a technical one

Roles and responsibility

Often a product owner or project manager wants to be the one who is in charge of releasing.

There are circumstances where exposing features to users should be controlled by a non-technical member of your team, but this can be managed with feature toggles.

But the copying of code from one computer to another is the responsibility of the developers on the team. After all we are the ones who are responsible for making sure the system works. It is a technical concern, not a business one.

What do QAs do now?

CD is actually liberating for QAs

Rather than spending their time manually testing poorly tested systems they can now focus on a more holistic view of the system, trying to facilitate an environment for CD so that the whole team can be confident things are working.
QAs spend more effort helping developers define what needs to be tested and monitored for a story to be written.
More time for exploratory testing

Re-evaluate your tolerance for defects

Lots of companies think they cannot have any defects and will spend a lot of time and effort on complicated, time consuming (and therefore expensive) processes to try and stop them.

But think about the cost of all this? If you push a change to production that isn't covered by tests, perhaps a CSS change; consider if it's really catastrophic if there's a small visual fault for some browsers

Maybe it is, in which case there are techniques to test specifically for this too.

Recovery

Each release you do with CD will have the following qualities

Plenty of tests
Good monitoring
Small scope
Still "fresh" in the developer's mind

So in my experience fixing anything that falls through the cracks is easy. It's much less complicated than trying to look through 2 week's worth of git history.

I would recommend in most cases not rolling back (unless it's really bad), but just fixing the issue and releasing it. Rollback is sometimes not an option anyway (e.g database migrations) so the fact that your system is geared to releasing quickly is actually a real strength of CD.

Other quick tips

Fast feedback loops are key. Measure your git push to live time and keep it low.
If things are getting slow re-evaluate your end-to-end tests. If you removed a test or refactored it to be a unit test, would you be any less confident? If not, then refactor away
You may need to invest some time in making your components more testable to avoid writing lots of slow end-to-end tests. This is a good thing.
Feature toggles are a useful tool but can become messy, keep an eye on the complexity
Celebrate it. Moving from one release every 2 weeks to 50 a day feels great.

Wrapping up

This has been a small introduction to CD, it's a huge topic with plenty of resources to investigate.

Continuous deployment is a big effort both technically and culturally, but it pays off massively.

I have worked on distributed systems with over 30 separately deployable components with CD, and it has been less stressful than another project I've worked on that had just a handful of systems but a ton of process and ceremony.

Being able to release software when it's written puts a higher emphasis on quality and reduces risk for the team. It also forces you into using agile best practices like testable, small, independently releasable user stories.

Maybe most importantly, it demands that you understand your system and that you're not deploying to production and crossing your fingers. That feels more professional to me.

Tis the season to write Clojure

Chris James — Sat, 02 Dec 2017 00:00:00 +0000

[:fa (take 9 (repeat :la))]

This post is aimed at beginners who want to take a look at Clojure, a super-interesting language which is probably quite different to what you’re used to.

You will learn some basic concepts and see how to write a Secret Santa assigner because IT’S CHRISTMAS!!!!! :D

Assumptions

You are somewhat familiar with programming, understand concepts like flow control (if, when, for, etc), variables, functions, & methods. If you’ve done a bit of Ruby or Javascript, you’ll probably be fine.
Everything you do for this post will be done through the terminal, so it’s assumed you have some familiarity and confidence with it.

What is Clojure?

A general purpose language like Java, Ruby, et al
Can also compile to Javascript so you can use Clojure to write client-side code for your website as well as the backend; so you can write a rich website end-to-end with one language

Getting started

Here’s a link to install a tool called Leiningen which lets you easily create Clojure projects.

If you’re on a Mac and have homebrew installed you can just run brew install leiningen. If you dont have homebrew installed, you should install it as most useful coding tools are installed using it.

Check your installation is working by typing lein in your command line, you should see a load of blurb.

The first time you run leinit may take some download the stuff it needs. It won’t be that slow on subsequence runs.

If you see it complaining about JDK

Clojure is compiled using the Java Development Kit which you might not have installed on your computer. On Mac you can use brew to install it.

brew update && brew cask install java

Assuming it’s all working you can now type lein repl, this is a great way just to experiment with Clojure as you can just type code and try it out.

The last line in the terminal should read something like user =>. If you ever wish to quit the REPL just type exit and hit return.

Type (println "Hello, world") and hit return and you should see

Hello, world
nil

What’s with the nil?

Every time you execute some Clojure like (println "hello world") it will always return a value. They are called “expressions”. This is a strength of Clojure and adds to its simplicity via consistency.

Think about some functions you have called in other programming languages, sometimes they return a value and sometimes they don’t.

Given this Javascript

function myAdder(x, y) {
     x + y
}

Oops no result! You forgot to put return.

This approach results in Clojure being more terse, you don’t have to worry about explicitly returning, imagine how rubbish this would be (return + 20 (return - 50 20).

In the case of println it can’t return anything meaningful as it just prints to the screen so it return nil which means “nothing”. The REPL always prints the result of the expression you run, so in the last case you see our println being printed, and then the result of the expression being printed (nil).

If you try (+ 20 50) you will see it just prints => 70.

REPL tips

Inside the REPL if you’re not sure what a function does type (doc function-name) e.g (doc println)
Often you can use tab to get auto-complete¡
Press the up arrow key to cycle through previous expressions

Clojure vs Javascript

Here’s some examples of some common operations you’d do while writing an application in both Clojure and Javascript. Try out the Clojure examples in the REPL and get in the habit of using doc

Note the variety of syntax for doing these tasks in Javascript compared to Clojure.

This isn’t picking on Javascript in particular, this is true of most mainstream programming languages.

Adding

Javascript: 1 + 1

Clojure (+ 1 1)

Calling a function

Javascript alert('Chris')

Clojure (println "Chris")

Flow control

Javascript

if(CHRISTMAS){
    return “PUT NUTMEG IN EVERYTHING”
} else {
    return “PUT SORROW IN EVERYTHING”`
}

Clojure

(if :CHRISTMAS "PUT NUTMEG IN EVERYTHING" "PUT SORROW IN EVERYTHING")

Doing stuff with lists

Javascript ["chris","ruth"].foreach(name => println('hello' + name)

Clojure (doseq [name ["chris" "ruth"]] (println name))

How is Clojure different?

You should notice that the syntax for all of this stuff is the same

(function argument1 argument2)

And you can build more complicated expressions, just like you do in maths by nesting.

(+ 20 (- 30 50))

The inner expression gets evaluated leaving us with

(+ 20 -20)

Which is 0.

Unlike in Javascript and most other mainstream languages where arguments and function names can appear in different orders and positions, you also need to know symbols like return and =>. It’s no wonder learning programming is so difficult.

What I find the most interesting thing about Clojure is that the syntax is completely uniform. The barrier to entry to learning how to write valid Clojure is very low. Writing Clojure that actually works is still a challenge just like any other language, but the syntax is very easy to pick up.

An interesting side-effect of this approach is that you will come to understand that the code in clojure is data. When you are constructing functions like (+ 2 2) that is actually a list of data, containing functions and arguments (just like a list of names like ("chris" "ruth")).

Accomplished clojure-ists use this to write code that parses and manipulates code called “macros”. Try not to worry too hard about this right now, but it is a very interesting property of Clojure.

Secret Santa time

Now you probably feel like a Clojure expert so lets ship some epic code.

Values

When you write code you generally need to store useful information in variables for later use. To do that in Clojure, there is no magic special syntax (as promised!) just call def.

In the REPL try (def my-santas ["Chris" "Ruth" "Turner" "Hooch"]).

The only new syntax you’ve seen here is [ ] which creates a data structure called a Vector which for now you can just think of as an Array like in Ruby or JS.

If you want to get to your variable, just type my-santas into the REPL.

How do you Secret Santa?

We need some kind of algorithm to pair up people for Secret Santa. When thinking about this stuff it’s often helpful to think about how would you solve the problem in real life.

We need a way of representing in data people being paired up; x gives to y, y gives to z
We need to randomise the people

How I have done this is just one of many ways of doing Secret Santa. I fully encourage you to try different ways!

When tackling a problem you’re not sure about it’s always best to break the problem down into smaller and simpler chunks.

Representing our data

For a starting point, we need a way of representing our pairing up of Santas. The map data type is perfect for this. If you’re unfamiliar, a map is a data structure like an Array but it has a set of keys with corresponding values.

They are called HashMaps in Ruby and Objects in Javascript. You can broadly think of them as dictionaries.

We can represent our result for Secret Santa in a map (syntax for maps are {key1 value1, key2 value2})

{"chris" "ruth", "ruth" "hooch", "hooch" "turner", "turner" "chris"}

i.e Chris gives to Ruth, Ruth gives to Hooch, etc..

Creating our first representation

zipmap is a function in Clojure that takes two vectors and “zips” them into a map (funnily enough).

So try in the REPL (zipmap my-santas my-santas)

You should get {"Chris" "Chris", "Ruth" "Ruth", "Turner" "Turner", "Hooch" "Hooch"}

OK, so everyone is just giving a gift to themselves but our basic data structure is there and we can see if we can just adjust the second argument to zip-map in such a way that everyone is moved along one, it could work!

Try experimenting with zipmap, what do you think happens with (zipmap my-santas [1 2]) ?

Rotate the list

We don’t want Ruth give a gift to Ruth, if we can take our second copy of my-santas and change it then everyone will give to someone different

Imagine our room of Santas standing in a line Chris Ruth Turner Hooch

Then you clone them (like our zipmap does)

Chris Ruth Turner Hooch

If we just shifted the cloned row to the right, and put the person on the end back to the beginning…

Chris Ruth Turner Hooch

Hooch Chris Ruth Turner

Everyone could take their gift and give it to the person in front of them and we would have our Secret Santas.

How do we do that?

Again, try and simplify the problem

Make a new vector
With the last one as the first item
Then the rest on the end

Clojure’s amazing standard lib again provides us with last to get the last element of a vector and butlast to get everything apart from the last item. Along with the function cons which lets us create new vectors let’s stick it all together.

In the spirit of breaking things down first try (cons (last my-santas) (butlast my-santas))

Then put it together

(zipmap my-santas (cons (last my-santas) (butlast my-santas)))

You’re allowed newlines to let the code breathe a bit

(zipmap my-santas 
         (cons (last my-santas) (butlast my-santas)))

Which results in {"Chris" "Hooch", "Ruth" "Chris", "Turner" "Ruth", "Hooch" "Turner"}

Awesome! Chris gives to Hooch, Ruth gives to Chris, etc.

Randomise

This is nice but it gives us the same result every time. We can make our algorithm even better by adding some randomisation.

Shuffle

Try typing (shuffle my-santas) in the REPL and you’ll see it takes a Vector and shuffles it. This could be useful for us to make our expression a bit more exciting.

Let

let is a function that allows you to declare values that can be used inside the function passed to it

(let [x 1, y 2] (+ x y)

(Remember this might look a bit weird but it still follows the convention of (fn arg1 arg2). It's just for let its 1st argument is a vector, and then an expression to run)

In Javascript it’s:

x = 1
y = 2
return x + y

The utility is the same, when writing non-trivial code you need to capture values and name them to improve readability. Coming back to our case we want to create a variable called random-santas that stores our shuffled Santas for the rest of the code to use.

Stick it all together

Let's use these functions to make our algorithm less predictable and store some values to let our code read easier.

(let [random-santas (shuffle my-santas)
   shuffled-santas (cons (last random-santas) (butlast random-santas))]
   (zipmap random-santas shuffled-santas))

The code now reads relatively ok!

Randomise the santas list
Create a copy of that list called shufled-santas, where everyone has moved to the right and we put the last Santa at the front
zip them together to create our pairings of Chris gives to Ruth

Challenge for yourself

What if you want to re-use our amazing algorithm with different sets of Santas?

You probably want to define a function, so you can use it just like we’ve used functions like shuffle and zipmap.

The basic syntax for defining your own function is

(defn name-of-function [arg1, arg2, etc] function-body)

Example

(defn add-2 [number] (+ number 2))

And then you can call it just like any other function (add-2 10)

Give it a go! Write secret-santa.

We saved Christmas!

That was probably a lot to take in but I hope it has been at least interesting. Maybe try to re-implement it yourself and see how far you get. Can you think of different ways to write it?

What might be more interesting is to implement the Secret Santa in a language you’re more familiar with. Think about how different your implementation is to the Clojure one, especially in regards to potentially all the different kinds of syntax you faced vs (fun arg1 arg2).

Developers are stakeholders

Chris James — Mon, 19 Jun 2017 00:00:00 +0000

When it comes to prioritising work you may find yourself in an environment where it can feel hard to justify playing "technical" stories. These are stories raised by developers and typically cover

Raising the quality of the code
Fixing technical/architectural debt
Addressing edge-cases
Making the system easier to operate

The usual excuse for not playing this work is that a team should only ever be delivering "business value" but that is almost always defined as customer-facing features.

At the same time developers will be responsible for making sure the system works, often obliged to be on call using their precious free time. This is a responsibility which is usually uniquely assigned to developers. You will rarely find a project manager being on call.

This isn't an issue of fairness, on a practical level it is unrealistic for anyone other than developers to fix most issues.

But given developers are responsible for not only delivering but also maintaining these systems it is only fair that they are also treated as stakeholders and need to have a say on what non-shiny work is being done.

But what if the developers spend all their time over-engineering everything?

This is nothing more than a matter of trust. If you do not trust your developers to make pragmatic choices for delivering sustainable systems then you have big problems.

Presumably you are doing showcases every one or two weeks so it should become very clear quickly if your team isn't delivering stuff as fast as you like.

Why didn't they make it sustainable and reliable in the first place?

Remember that whole agile thing of build->measure->learn? And the prime directive? It is unrealistic to expect all software to be perfect when it is first built. Only when you have shipped some code and seen it used do you really start to understand the flaws in the implementation.

If you think you can just ignore the developers and entirely micromanage their time then they will resent the project and especially resent being woken up at 3am to fix a problem which they had anticipated but you didn't believe them.

If you value the following attributes of your system:

How easy it is to change
How reliable it is

Then you have to acknowledge that the work developers wish they could do has business value.

So talk to your developers, trust and empower them.

Testable Systems and Continuous Deployment

Chris James — Tue, 21 Mar 2017 00:00:00 +0000

People talk a lot about how to write testable code but testable systems are what you need if you are going to be able to practice continous deployment.

Writing testable systems takes practice and can be quite domain specific, but there are some general rules that you can apply.

You may encounter resistance to making this extra effort so along with some guidance I will discuss how I justify this approach.

Some positive traits of a mature system

Being able to improve the system without worrying about breaking it.
Releasing features as soon as they're ready, not tied down to release schedules.
Fast feedback loops, knowing when there are problems and being able to fix them quickly.

What impedes a mature system from being fun to work with

Lack of confidence in the system so lots of manual testing.
This leads to fewer releases.
Which leads to slower feedback loops which makes it harder to fix problems and you are slower in delivering value.
Lack of confidence results in fear-driven development, where people become incredibly risk averse, taking decisions which are well intentioned but can end up complicating matters.

So what do you need?

Releasing as often as possible, preferably every commit to master is going to live.
Absolutely no manual tests so you have "acceptance tests" running on each deployed component, including live and some end to end tests for some key scenarios.
These tests are running periodically irrespective of releases so you know your system is working in live

Writing testable code can be more effort but it's worth it and the same is true for systems. With some effort you can make sure the project you're working on is pleasant to work with and you can actually start looking forward to releases rather than worrying about whether it is going to work.

"Just write a ticket and we'll let the product owner prioritise it later"

In a common agile scrumban development environment where everything is focused on "business value", "story points" you must assert that the development team are stakeholders too.

If your system is useful you will continue to work and maintain it so it's important to do work and add features which might not be for the direct benefit of the actual customer but help the development team.

Every job spec in the world talks about developers being empowered and responsible for the systems they create so make sure you call your company out. Fight for this if needs be otherwise you're going to have a miserable time.

Some rules to live by

When you are working on a new feature or system ask yourself

How can I be confident this will work that does not require me to manually test it in live?

Then figure out these answers, the answers may not always be as straightforward as you hope but by putting yourself in this mindset you will write your system in a way that is testable from the start, rather than trying to retrofit it.

Involve your colleagues, in particular QAs as to how we can be sure that when the commit is merged that it will go to live and deliver high-quality value reliably

Just like how you don't worry about breaking the internals of your system because of unit tests you also wont worry about deploying new versions of your system because everything is built with testing in mind.

Like the tests within your system you should be striving for your tests to be reliable and fast to run.

If your tests are proving problematic and are being flaky then you should stop working on features until it is sorted.

Over recent years I have been working with this mindset and I am going to describe a number of techniques to help make your system testable

Write good, testable well separated code

An obvious one but if you maintain good separation of concerns then it's much easier to "open up" your systems for tests.

Avoid the browser

For a web app you will need some tests in the browser but keep them to a minimum. Follow Fowler's test pyramid because otherwise you will have a slow, flaky test suite which becomes a big maintenance overhead.

Good code design (separation of concerns especially) will allow you to test a lot of functionality without the need to run a web browser.

A trick I have learned to avoid web browsers but still test content that is being delivered is by adding some functionality to the web app so if you make a request with an Accept header of application/json it returns the view model representation as JSON rather than the rendered HTML. This allows you to test with simpler HTTP tools rather than firing up a browser which will be much faster to run and more reliable.

Idempotence

When writing systems if you see an opportunity for an operation to be idempotent then go for it. This makes writing tests a lot simpler as you dont have to worry about the previous state of the system which is often a cause of complication when writing system tests.

Open up your system

Your deployed system may get it's input from sources that are not as easily accessible as you'd hope, for instance maybe a message queue or a file system on a secured server.

This is less easy to test than say a system that is interacted with over HTTP, with which there are tons of simple and quick tools to use.

An option for this is to create an internal endpoint with an API that allows you to work with the system as if you are a file-system, message queue or whatever.

Not only does this make writing tests very easy but in my experience the QAs have found these endpoints invaluable for demos and exploratory testing.

Good separation of concerns should make this relatively simple to accomplish but you may wish to employ some kind of 'isTest' flag to let you separate out that from normal data if you need to.

A more concrete example

Recently I worked on a service that took its input from a message queue in the form of some JSON, perform some business logic on it and then put it onto another exchange for other parts of the system to consume.

The JSON is a payload which is to be rendered as a HTML email. It also calls some other services to help it create this payload.

Taking a very pedantic approach to developing this would've resulted in a tricky system to write a test for in isolation. Obviously we had some unit tests with some fakes services wired up but we still didn't feel super confident.

Thankfully the domain logic was really nicely separated out so we were able to expose the logic over HTTP so that you can POST to the service and then get the rendered HTML back. Coupling this with PhantomJS we could take screengrabs of the rendered HTML and use image diffs to test for any regressions.

The other benefit was the BA and QA could also use this endpoint to try it out with other datapoints very easily so they could demo it; and of course they found some bugs.

Code just for tests? Wtf

Yes, I've read The Lean Startup too.

Again developers are stakeholders. No one bats an eye when you write logging or metrics code, probably because it is not "seen" and it is perceived as low effort. In addition no one complains when you write unit tests (I hope) even though it's not directly aiding the customer.

Writing testable systems is often just an extension of this logic, everyone in the business should value the maintainability, extensibility, releasability and reliability of your system and these kind of measures facilitate these traits.

Straight to live? But what about bugs

No process in the world will truly stop all bugs apart from very expensive means and 99% of developers don't need to produce bug free systems (despite what their product owners might like to think!)

Perfect is the enemy of good

However when you adopt these methods you are more likely to find bugs quicker but most importantly deploy a fix within potentially minutes rather than on some weekly release schedule.

Benefits for the QAs

By automating away the manual testing process our QA now has more time for exploratory testing and the testing endpoints we add to our system helps facilitates this.

When it comes to starting a new piece of work the QA can come armed with a number of suggestions to help ensure the quality of the system; rather than spending their time re-testing software.

I have also found that QAs will spend more time thinking about the quality of the test suites and deployment process etc, making sure we as a team have quality for the whole system rather than just the end product.

It's 2017: Do continuous deployment (for new projects at least)

When you have your tests in such a state that you are confident to release with every commit a tremendous burden is lifted from your team and it also pays off years down the line. How many times have you had to work with a "legacy" system where it's tough to even deploy the code in the first place and you have to spend hours doing manual tests.

With CD not only are you guaranteeing that you can ship new changes when they are ready quickly but in a few years down the line when some poor sap has to work with your no longer flavour of the month programming language, they can at least be confident that they can make the necessary changes and get it shipped with little fuss.

If you do encounter resistance then you are not working in an agile organisation. You are supposed to be in a position where you are empowered to improve processes and software to make delivering value as quick and as painless as possible. There is a difference between continuous delivery and continous deployment, the latter should not even be up for discussion. Deployments are the team's concerns, showing features to users is usually product owner's concern.

But still, if the company you work for are throwing terms like lean and agile around it should be in their interest to get features, even if they're imperfect infront of users as quickly as possible.

Fun with Bash

Chris James — Mon, 20 Mar 2017 00:00:00 +0000

I was given a fairly mundane task at work and when this happens I try and find ways to make it interesting. I will usually find a way to learn something new to do the task so I can always get something out of the work I do, even if it's dull.

I would quite like to get better at using the legendary tools available to me on the command line. I have never felt super confident doing text processing there, often giving up and just going to my editor instead.

I am not going to exhaustively document every command I do, if you're curious about the details consult man $command just like how I did! This post will just be illustrating how by doing a little research you can quickly do some rad text processing.

The task

I am trying to analyse how much memory we are provisioning compared to actual usage for some of our services to see if our AWS bill can be cut down a bit.

The data

I thought it would be fun (??) to use CSV, as it feels like CSV like data. Here is a sample of the data I captured.

name, usage (mb), allocated (mb), CPU %, containers
dispatcher, 150, 512, 40, 10
assembler, 175, 512, 75, 10
matcher, 85, 512, 15, 10
user-profile, 128, 512, 40, 5
profile-search, 220, 512, 80, 10
reporter, 90, 512, 40, 10 
mailgun-listener, 90, 512, 10, 5
unsubscribe, 64, 512, 3, 5
bounce, 8, 128, 0.5, 3
legacy-reporting, 30, 512, 15, 3
content-store, 80, 256, 30, 10
legacy-alert-poller, 64, 256, 1, 1
migrator, 80, 256, 10, 5
entitlements-update, 150, 256, 70, 3

Display it nice

This is nice and easy column -s, -t < data.csv

-t determines the number of columns the input contains and combined with -s specifies a set of characters to delimit by. If you dont specify a -s it defaults to using space.

name                  usage (mb)   allocated (mb)   CPU %   containers
dispatcher            150          512              40      10
assembler             175          512              75      10
matcher               85           512              15      10
user-profile          128          512              40      5
profile-search        220          512              80      10
reporter              90           512              40      10 
mailgun-listener      90           512              10      5
unsubscribe           64           512              3       5
bounce                8            128              0.5     3
legacy-reporting      30           512              15      3
content-store         80           256              30      10
legacy-alert-poller   64           256              1       1
migrator              80           256              10      5
entitlements-update   150          256              70      3

Sorting by usage

cat data.csv | sort -n --field-separator=',' --key=2 | column -s, -t

name                  usage (mb)   allocated (mb)   CPU %   containers
bounce                8            128              0.5     3
legacy-reporting      30           512              15      3
legacy-alert-poller   64           256              1       1
unsubscribe           64           512              3       5
content-store         80           256              30      10
migrator              80           256              10      5
matcher               85           512              15      10
mailgun-listener      90           512              10      5
reporter              90           512              40      10 
user-profile          128          512              40      5
dispatcher            150          512              40      10
entitlements-update   150          256              70      3
assembler             175          512              75      10
profile-search        220          512              80      10

--key=2 means sort by the second column

Using awk to figure out the memory differences

What we're really interested in is the difference between the amount of memory provisioned vs usage.

awk -F , '{print $1, $3-$2}' data.csv

Let's pipe that into column again

awk -F , '{print $1, $3-$2}' data.csv | column -t

name                 0
dispatcher           362
assembler            337
matcher              427
user-profile         384
profile-search       292
reporter             422
mailgun-listener     422
unsubscribe          448
bounce               120
legacy-reporting     482
content-store        176
legacy-alert-poller  192
migrator             176
entitlements-update  106

This is nice but it would be good to ignore the first line.

awk -F , '{print $1, $3-$2}' data.csv | tail -n +2

tail -n X prints the last X lines, the plus inverts it so its the first X lines.

Sort mk 2

Now we have some memory differences it would be handy to sort them so we can address the most inefficient configurations first

awk -F , '{print $1, $3-$2}' data.csv | tail -n +2 | sort -r -n --key=2

And of course use column again to make it look pretty

awk -F , '{print $1, $3-$2}' data.csv | tail -n +2 | sort -r -n --key=2 | column -t

legacy-reporting     482
unsubscribe          448
matcher              427
reporter             422
mailgun-listener     422
user-profile         384
dispatcher           362
assembler            337
profile-search       292
legacy-alert-poller  192
migrator             176
content-store        176
bounce               120
entitlements-update  106

WTF

There it is! The utterly indecipherable bash command that someone reads 6 months later and scratches their head. In fact there has been 2 weeks since I wrote the first draft of this and I look at the final command and weep.

It is very easy to throw up your hands when you see a shell script that doesn't make sense but there are things you can do.

Remember that the process will usually start small like we have here, starting with one command, piping it into another, into another. This gives the lazy dev like me the perception that it is a complicated command but all it really is a set of steps to process some data. So if you're struggling you can wind back some of the steps for yourself by just deleting some of the steps and see what happens.

If it is an important business process that needs to be understood for a long time, you're probably better off writing it in a language where you can write some automated tests around it.

But a lot of the work we do does involve doing things that are ad-hoc and doesn't reside in "the codebase" where you can easily get into a TDD rhythm to accomplish something shiny. Often you have to do something a little boring and sometimes the tools available on your computer can really help you out. They're so old and well established that you can find tons of documentation and tips so dive in!

If you need more reasons to get to know the shell better read how command line tools can be up to 235x faster than your hadoop cluster

Testing Asynchronous Code In Go

Chris James — Thu, 29 Dec 2016 00:00:00 +0000

Sometimes you may need to test that an asynchronous event has been triggered in your code but you can end up compromising on the quality of your tests when doing so.

This post will show one technique to test this behaviour whilst keeping your tests clean and not complicating your production code

It's assumed you're familiar with writing tests, interfaces, go routines, channels and select from Go.

The code to test

We are writing a service which has a dependency of a repo passed to it. Plain and simple DI for separation of concerns.

The service has a method FaveAndSave which takes a string and it will do some amazing business logic and then save to the repo.

type Repo interface {
    Save(string)
}

type Service struct {
    repo Repo
}

// FaveAndSave does something great with your string and saves it
func (s Service) FaveAndSave(x string) {
    go s.repo.Save("☆" + x)
}

The code calls the repo in a new go routine which makes testing a little tricky.

Let's test it

The benefit of using DI here is that we can now stub our Repo, make it behave how we like and see how it's called.

type repoStub struct{
    saveCalled bool
}

func (s *repoStub) Save(x string) {
    s.saveCalled = true
}

func TestItTriggersSave(t *testing.T) {
    repo := new(repoStub)
    service := Service{repo}

    service.FaveAndSave("Cat")

    if !repo.saveCalled {
        t.Error("Save was not called")
    }
}

This test fails wrongly because the save event is fired off in a separate go routine which ends up being executed after our assertion.

This means the saveCalled value is still set to false when we make our assertions.

Solution 1 - Sleepy time

func TestItTriggersSave(t *testing.T) {
    repo := new(repoStub)
    service := Service{repo}

    service.FaveAndSave("Cat")

    time.Sleep(10 * time.Millisecond)

    if !repo.saveCalled {
        t.Error("Save was not called")
    }
}

We can add a sleep into our test to let the go routine finish before making assertions. But there are a few problems:

The sleep length I put in is fairly arbitrary and it's hard to know precisely what value it should be. You dont want to set it to be too long because then your test will be slow, but too short will make the test flaky.
It's noise in our test. In this simple example it doesn't seem so bad but it is ultimately more cruft which we need to try and avoid.
Sleeps always feel yucky!

Solution 2 - Use a channel in our stubs

We can embrace the asynchronousness of our production code so we dont have to rely on guessing how long the stub takes to get called.

type repoStub struct{
    saveCalled chan bool
}

func newRepoStub() *repoStub {
    s := new(repoStub)
    s.saveCalled = make(chan bool)
    return s
}

func (s *repoStub) Save(x string) {
    s.saveCalled <- true
}

func TestItTriggersSave(t *testing.T) {
    repo := newRepoStub()
    service := Service{repo}

    service.FaveAndSave("Cat")

    if saved := <- repo.saveCalled; !saved {
        t.Error("Save was not called")
    }
}

Rather than setting a normal value we have a channel which gets written to when the stub is called. This means our test can wait on a value to appear on the channel, rather than using sleeps and hoping the value gets set.

This too has problems:

If the stub isn't called like we expect then the test will fail, but because of deadlock/test time out. This means our test output is less helpful then it should be.
The test now has channels and general asynchronous behaviour which is implementation detail that we dont really need to care about, like the sleeps it is noise.

Solution 3 - Make the stub's value "blocking" with a timeout

It seems these days many developers automatically recoil in disgust when you say "blocking" but it can often be simpler than the alternative.

I write a lot of Javascript in my day job and 99% of tests are littered with promise, done, return and generally a lot of asynchronous cruft because everything is asynchronous.

That can make writing tests quite taxing, especially if you are new to it. Common gotchas include tests not actually running assertions or tests running forever.

Let's keep our tests clean by putting just a little bit of code in to our stubs to make them block for the event we care about

type repoStub struct{
    saveCalled chan bool
}

func newRepoStub() *repoStub {
    s := new(repoStub)
    s.saveCalled = make(chan bool)
    return s
}

func (s *repoStub) Save(x string) {
    s.saveCalled <- true
}

func (s *repoStub) WasSaveCalled() bool {
    select {
    case <-s.saveCalled:
        return true
    case <- time.After(1 * time.Second):
        return false
    }
}

func TestItTriggersSave(t *testing.T) {
    repo := newRepoStub()
    service := Service{repo}

    service.FaveAndSave("Cat")

    if !repo.WasSaveCalled() {
        t.Error("Save was not called")
    }
}

We have moved the responsibility of "Has the stub been called in a timely manner?" to the stub rather than the test by simply adding the method WasSaveCalled.

The code takes advantage of Go's select syntax to block until the value is written or until the timeout occurs.

This feels like a good separation of concerns. Now if we look at the actual test we can see it reads very clearly and has no cruft around go routines or channels.

The code in the stub maybe looks a little involved but you will definitely move these into separate files, perhaps even go generate can be used to auto generate them like goautomock.

Summary

This post was motivated by some work I am doing where we are writing a "listener" to a RabbitMQ channel. The listener takes a chan Message, gets fired off in a go routine and processes messages as they come in to the system.

We went through similar iterations described above until we arrived at solution 3.

Is this idiomatic Go? I'm not sure I know what idiomatic Go is yet but this solution to me ticks a number of boxes I care about in writing software irrespective of the programming language.

Easy to read and write test code
Separation of concerns in regards to "has this function been called?"
Good test output on failure

Hopefully this has been interesting. Know a better way? Flame me on twitter @quii

Footnotes

For the sake of terseness this example just sees if Repo is called, but you can of course see what it was called with. In the real world Repo would probably have return values that we'd stub for tests too.

I like to pass t.Log to my stubs so they can log diagnostic info (such as "I timed out waiting for x"), it has the added bonus they only appear if the test fails

Property-based testing in real life

Chris James — Sat, 27 Aug 2016 00:00:00 +0000

Most property-based test examples you see on the web are contrived scenarios and people can be dismissive of them, claiming that they aren't actually practical. This post will have a simple example applied to one of my projects which will hopefully illustrate their value.

This is a follow up to Property-based testing in Go and if you're not familiar with these kind of tests I suggest you read that first, it's a quick read.

Although the examples are in Go there are QuickCheck frameworks for most mainstream programming languages and the concepts described here will still apply.

This post will:

Briefly explain the domain
Describe writing a real world property-based test
Summarise my conclusions

mockingjay server

It's important to have a good knowledge of the domain when writing property-based-tests, the following is an overview of the domain we will be testing.

mockingjay-server (MJ) is an application which is a lightweight HTTP server driven from configuration.

---
 - name: My very important integration point
   request:
     uri: /hello
     method: POST
     body: Chris
   response:
     code: 200
     body: '{"message": "hello, Chris"}'
     headers:
       content-type: application/json

MJ has two "modes":

Server mode

Serves responses matched to requests that you define.

Compatibility or CDC mode

Take that same config, executing each request against a given URL and checking the response is compatible with the response in configuration. This is known as a consumer-driven contract (CDC).

Why?

When writing integration tests against a HTTP service you will usually make a fake server (or some kind of stub around HTTP) to write tests against. You will then make your tests pass and be happy right?

But:

Fakes dont always correspond with the real behaviour
Fakes might behave correctly at first, but if the downstream service changes the fake could be wrong

In both of these situations your build would be green but your software is broken.

mockingjay allows you to make a server and easily verify that it is equivalent to what you're testing against. You can then distribute this configuration as a CDC for the maintainer of the downstream service, so they dont accidentally break the service for your use case. The wiki goes into this more.

There is an inherent coupling between HTTP integration tests and consumer driven contracts. MJ leverages this in a single config.

A property of mockingjay

To write a property-based test we need to identify a property to throw lots of auto-generated data at, to make sure the property holds true.

A property of MJ is:

MJ should always be compatible with itself

Here's my thinking:

Given Config A
When You start an MJ server with config A
And you run a CDC check using config A with the URL of the running MJ server
Then the CDC check should pass

If it doesn't it either means there is a flaw in the CDC algorithm or in the way the server is behaving.

I have lots of example based tests for this but if I invest time writing a property-based test I can be really confident MJ is working.

Create a generator

For all but the basic types you will need to create a Generate method for the input type in your test. This will allow the quickcheck package to create thousands of different data points to check the property against.

I made a simple one to start with and other HTTP things like headers, forms, etc can be added later.

func (r FakeEndpoint) Generate(rand *rand.Rand, size int) reflect.Value {
    randomMethod := httpMethods[rand.Intn(len(httpMethods))]

    req := Request{
        Method: randomMethod,
        URI:    "/" + randomURL(rand.Intn(maxURLLen)),
    }

    res := response{
        Code: rand.Intn(599-100) + 100,
    }

    return reflect.ValueOf(FakeEndpoint{
        Name:     "Generated",
        Request:  req,
        Response: res,
    })
}

FakeEndpoint is a representation of the config from earlier.

Test the property

Write a function which takes the randomly generated endpoint
Start an MJ server using that endpoint
Take the same config and run the CDC against the server
If there are any errors then the check fails

Here's what the test looks like

func TestItIsAlwaysCompatibleWithItself(t *testing.T) {

    compatabilityChecker := NewCompatabilityChecker(noopLogger, httpTimeout)

    assertion := func(endpoint FakeEndpoint) bool {

        // Start an MJ server with the random configuration
        mjSvr := NewServer([]FakeEndpoint{endpoint}, false, ioutil.Discard)
        svr := httptest.NewServer(http.HandlerFunc(mjSvr.ServeHTTP))
        defer svr.Close()

        // Run CDC against "itself". An MJ server should always be compatible with itself.
        errors := compatabilityChecker.check(&endpoint, svr.URL)

        if len(errors) > 0 {
            t.Log("It wasn't compatible with itself")
            for _, err := range errors {
                t.Log(err)
            }
        }

        return len(errors) == 0
    }

    config := quick.Config{
        MaxCount:1000,
    }

    if err := quick.Check(assertion, &config); err != nil {
        t.Error(err)
    }
}

When I ran the test I was pleasantly surprised in that the CDC check failed.

Couldn't reach real server: Post http://127.0.0.1:41006/snipped-really-long-random-url: 303 response missing Location header

This means the CDC tried to POST to the configured URL and Go's HTTP client returned an error.

This is quite exciting, I haven't seen MJ configured with any 3xx response codes so this points to some naivety in my code.

HTTP 303 is "See other" which relies on a location header to get redirected to another resource.

Investigating Go's HTTP client

Before writing any code for this I wanted to have a look at the Go source of http.Client to see exactly how it works.

By following a few of the function calls we can see if it's a POST or a PUT and the status is a HTTP Found 302 or See Other 303 then it will expect a location header that it can parse with req.URL.Parse.

I will have to add some additional validation to the configuration so that these rules can be respected or make Go's HTTP client not follow redirects.

Conclusions

The costs of writing property-based tests are low (this took me about 10 minutes) and can help give you a lot of confidence in your code (or not!).

What's great is that even on a very simple generator implementation the tests uncovered some bugs.

This style of testing can:

Improve knowledge of the problem domain
Find bugs you wouldn't have thought of otherwise

These things prove that property-based tests can improve the quality of your software.

Footnotes

Other examples of this style of testing in the real world:

QuickChecking Riak

In this talk John Hughes shows us how QuickCheck helped us to model Riak’s behaviour, improving understanding and revealing the occasional bug.

John Hughes - Testing the hard stuff and staying sane

Taking 3k lines of specification to create 20 lines of QuickCheck to test 1 million lines of code from 6 different vendors. Some real war stories in this video.

Magic

Chris James — Fri, 25 Mar 2016 00:00:00 +0000

Magic is often used to describe confusing code. Often we cannot explain how something works so magic seems like an apt description.

The problem with magic is that it’s an entirely subjective term which can be used as a very annoying stick to hit useful abstractions.

An anecdote

A while ago I wrote something along the lines of this in Scala.

val transformedCollection = collection map transformer

For those not familiar with the map construct, this will run the transformer function on each item in the collection, returning a new collection of transformed items.

Another nameless programmer chastised me

"I like to be pragmatic"

(Who exactly doesn’t like to be pragmatic?)

"This is magic, why not just write a for-loop, it’s clearer"

Apart from the pragmatic comment it’s actually a reasonable thing to say, if you don’t understand how something works then it’s magical!

Understand the magic

Just because something seems magical doesn’t mean it’s wrong. The problem with the term is it’s often lazily used when someone doesn’t understand an abstraction, or worse - doesn’t want to understand.

So what is magic?

For me I get the feeling of dark arts surrounding me when there is a lack of explicitness. Generally stuff changing “underneath” me based on some kind of state of the system. It often feels like the code is working by accident rather than deliberately.

Anti-patterns like global imports and setter-based DI makes the system feel magical in a very ugly way.

Choose your spells

From time to time all of us look at some code and throw our hands up in the air when we don’t initially understand something. Rather than unsheathing our anti-magic sticks make an attempt to understand it first.

There’s nothing more annoying than someone describing your code as magic when you feel you put some effort into a useful abstraction. So try to be considerate when faced with unfamiliar constructs.

That being said it is important for programmers to share their knowledge when introducing something new. Be sure to explain your sleight of hand at code review and use abstractions tastefully and keep intent clear at all times.

tl;dr

Don't lazily conflate the unfamiliar with magic. It's short-sighted and annoying.

Property-based testing in Go

Chris James — Mon, 07 Mar 2016 00:00:00 +0000

This posts hopes to illustrate

What property based testing is and why it is an important tool to compliment your existing tests
How easy it is in Go

Your current tests

The usual examples illustrating TDD could be described as example based tests; where you write tests by providing example inputs and expected outputs.

func TestAddingExample(t *testing.T) {
    result := add(3, 2)
    if result != 5 {
        t.Error("3 plus 2 is 5 but i got", result)
    }
}

When you realise that they are just examples, you see your code perhaps isn’t as well-tested as you think.

Wouldn't it be nice if it were easy to provide thousands of different test inputs for your tests?

You may have already heard about property-based testing and think its about testing your code with randomly generated data to exercise different scenarios; but it actually requires a shift in mindset to get the most out of it.

Specifications, not just lots of data

Property-based testing is called that because in order to write the tests you have to think about the properties of your domain in order to write effective tests.

An example with addition

We will test an add function, which just adds 2 integers together.

func add(x, y int) int {
    return x + y
}

The tricky thing when you feed in random data, is how do you know what your expected result should be?

func TestAddition(t *testing.T){
  x := getRandomInt()
  y := getRandomInt()

  expected := x + y

  if expected != add(x, y){
    // oh no..
  }
}

You may be tempted to do the above, but in effect you’re cheating because you’re using the implementation to verify itself.

Pretend that you’re testing a more complicated function. You know you shouldn’t do the following:

func TestComplicatedThing(t *testing.T){
  x := getRandomInt()
  y := getRandomInt()

  expected := complicated(x, y)

  if expected != complicated(x, y){
    // oh no..
  }
}

This isn’t really testing anything, if you change the implementation of complicated it will continue to pass its tests even if its behaviour is incorrect.

(This is a common problem with bad tests where the writer has re-implemented the unit they are testing, in the test!)

But what do you do? How can you verify your function on random inputs?

The properties of addition

What is there to addition anyway? Well, there’s a few things:

when you add zero to a number, you get the same number back again
the order of the inputs does not matter

Let’s write some property based tests based on these laws of addition.

func TestAddingZeroMakesNoDifference(t *testing.T) {
    /*
    Create an assertion, which is a function that takes N inputs of
    random data and returns true if the assertion passes. 

    In this case, we're saying take any random integer (x)
    If you add 0, it should equal x
    */
    assertion := func(x int) bool {
        return add(x, 0) == x
    }

    // Run the assertion through the quick checker
    if err := quick.Check(assertion, nil); err != nil {
        t.Error(err)
    }
}

func TestAssociativity(t *testing.T) {
    assertion := func(x, y, z int) bool {
        return add(add(x, y), z) == add(add(z, y), x)
    }

    if err := quick.Check(assertion, nil); err != nil {
        t.Error(err)
    }
}

The quick.Check function will run your assertion 100 times (by default), feeding in randomly generated data for the inputs required. If it fails at any point then you will be told for what inputs your function fails on so you can go reproduce it and fix.

Notice how:

By asserting on properties rather than known values we don’t have to "cheat" on the tests like I did earlier.
We have now described what addition is in terms of its properties, which is more expressive than the example based test from earlier.
We now have a more robust test suite, able to run thousands of different inputs in milliseconds.

Thinking abstractly about your domain

Once you take a step back and really think about the domain of your code, you may discover some unexpected properties that the examples you use in your general TDD flow wouldn’t find.

I imagine this is how a lot of QAs think. They tend to be a bit detached from the code we write and really think about the domain and how things can fall apart. Property based testing allows us to capture these kind of rules as tests.

Going beyond simple types

Property-based testing relies on generators for types to create random data for your tests. It’s simple to imagine how this works for integers but what about your own types?

type LatLong struct{
    Lat float64
    Long float64
}

func (l LatLong) Generate(rand *rand.Rand, size int) reflect.Value{
    randomLatLong := LatLong{
        Lat: rand.Float64(),
        Long: rand.Float64(),
    }
    return reflect.ValueOf(randomLatLong)
}

func TestLatLongIsAlwaysTrue(t *testing.T){
    assertion := func(x LatLong) bool {
        t.Log("Random LatLong:", x)
        //todo: Do some interesting assertions!
        return true
    }

    if err := quick.Check(assertion, nil); err != nil {
        t.Error(err)
    }
}

All you have to do is implement the Generator interface for your type. Go will provide you with a rand to help you generate random data.

Conclusion

To write effective property-based tests you have to take a step back and think deeply about the rules of your domain.

Once you gain this deep understanding you can then express them very easily with property-based tests which will help you find bugs before you get to production.

The tooling in Go facilitates easily writing these specifications, even for your own types so there’s no excuse not to try it in your current project.

Notes

You can check out the code on github
I learned how to do quickcheck in Go by RTFM. I suggest you do too because it offers a number of options and configurations I didn't cover here.
The talk that really got me interested in property-based testing

Estimates

Chris James — Tue, 29 Dec 2015 00:00:00 +0000

Why do developers whine about doing estimates? All the business wants is just some kind of assurance that their important project will be done by a given date. What’s the big deal?

Let’s be clear here, estimates are always wrong

That’s not the worst of it. When you lock your development team in a room for a few hours to try and estimate 100 stories you are now suddenly back in the waterfall world. You are committing yourself to a set piece of work that cannot change.

Which is what I thought as an industry we had got past. These are inarguable facts about software development

Priorities change
Knowledge (about a system being developed) changes
Teams change (they get better, usually)

Not only is it impossible to say “40 features will be delivered by April”, but you shouldn’t want to either!

In X weeks time, you will have a better understanding of what you want and the development team will be better equipped to help you make those decisions. Upfront estimations take you away from this.

The estimates I do like

A finger in the air estimate of a bunch of high-level features can of course be given. These are ok because everyone knows that they’re vague and subject to change
Relative weighting of features. This is actually the only estimate that’s important because it helps the business make a decision based on a sense of reality.

Estimation sessions

“Estimation sessions” where a gigantic pile of stories are dropped at the feet of developers locked in a room for a few hours, points to big flaws in your process.

It doesn’t acknowledge that change happens

Only when you actually start doing the work do the real problems you’re facing become clear. There will be stuff you missed which will effect your to-the-day timescales.

It implies you’re doing mini waterfall, not agile

I am often astonished when I see these huge piles of cards. Where have they come from? What happened to the principles of lean-ness and doing just the work you need and then moving your priorities as and when you need them?

You have done the “analysis” stage of waterfall already it seems. You have looked into your crystal ball and worked out all the features that your customers need.

The software project fallacy

You have a deadline and I understand this. I hope you understand that if your software is worthwhile that it will still need work on it after your deadline. If it’s not worthwhile, why are we building it?

At the very least it will need maintenance but if your software is actually good, by setting a hard deadline you are really shooting yourself in the foot.

People often forget that what’s really good about software is you can change it. The industry has dedicated so much energy in trying to make software malleable, yet so often we are poor at selling this to the people paying our wages.

By setting hard deadlines you are creating a precedent that you know what your customers want and that their needs never change. Quite the claim.

Go back to basics

Build, measure, learn over a small handful of the most important features. Discover what features are actually important and get it right. Everyone will learn about the product and you may develop something completely unexpected.

If you do this in a collaborative way with the business you will get out of the mindset of software being a static thing that you make and throw away.

If the parties engage together they can make something better. The software development team is not seen as a cost-centre but an asset that can actually respond to changing needs.

Summary

Fine-grained estimations disregard many of the good principles around lean, agile software development. Instead of trying to predict when a set-in-stone list of features will be done, acknowledge the unknown of business needs and if you create a truly collaborative team you can build a better product than you imagined.

You cant escape deadlines. But you should be focusing your efforts on building a team that works hard at becoming efficient at writing maintainable, useful software by working with the business/customers and responding to changing needs.

There is some interesting debate around this on twitter

Terror Handling

Chris James — Fri, 20 Nov 2015 00:00:00 +0000

The Happy path of writing software is usually the easiest part and coincidentally is the the thing focused on most during estimation. A lack of thought behind error handling can have big consequences on the long term health of a code base.

I am going to contrast on the 3 main methods of handling errors that I have come across, exceptions, product types (i.e tuples) and sum types (Try, Either, et al).

The problem with exceptions

On the face of it exceptions seem convenient but most people understand that you need to apply a lot of good practice for them not to be a pain. There is countless literature about the misuses of exceptions, from Pokemon exception handling to using exceptions as control flow.

The best use-case for exceptions is when you “know” it can’t be recovered from. The problem is exceptions tend to be conflated with normal errors that you do want to recover from.

When you use exceptions like this then it impacts code re-use. By throwing exceptions you are losing referential transparency, which means you cannot trust the type system; getFoo might not return a Foo for all inputs (it is not a total function). This means that reasoning your code becomes more difficult (because you need to check the source to see the real behaviour) and it is harder to simply plug functions together.

Product types to the rescue?

The debate around error handling is fairly prominent in the Go community, mainly because it doesn’t really* have exceptions.

The language supports tuples so the convention is to simply return the error to the caller if there is one.

package main

import "fmt"

func printer(f func() (string, error)){
    result, err := f()
    if err != nil {
        fmt.Println("Fail!", err)
    } else {
        fmt.Println("Success!", result)
    }
} 

func niceFunc() (string, error){
    return "Whoop", nil
}

func main() {
    printer(niceFunc)
}

In this case niceFunc is referentially transparent and the higher order function lets us know that it wants a function which could return an error and will act on it.

This explicitness means you don’t have to look into the implementations of functions to check for exceptions being thrown and the compiler (and tooling) will complain if you try and pretend a function could never return an error.

There is a better way

In simple examples it doesn’t seem so bad, but a lot of Go code is littered with these kind of checks. The main problem is that it is very difficult to compose potentially failing functions into new functionality.

package main

import "fmt" 

func func1() (string, error){
    return "Whoop", nil
}

func func2(input string) (string, error){
    return "", fmt.Errorf("Bwahahaha")
}

func func3(input string) (string, error){
    return "Yippee", nil
}

func main() {
    // We want to chain func1, func2, func3, bailing if we get an error
    res1, err := func1()

    if err != nil{
        return
    }

    res2, err := func2(res1)

    if err != nil{
        return
    }

    finalResult, err := func3(res2)

    if err != nil{
        return
    }

    fmt.Println(finalResult)
}

Scala (and others) lets you write sum-types which allow you to encapsulate these very common computations in the type system in such a way that you can be explicit and still write your code in a declarative and convenient way.

case class DomainError(context: String, error: String)

// this is using scalaz's either type
def func1(): DomainError \/ String = ??? // so in this case it can return either a DomainError or a String
def func2(x: String): DomainError \/ String = ???
def func3(x: String): DomainError \/ String = ???

// see how easy it is to compose functions
def functionsComposed: \/[DomainError, String] = func1 flatMap func2 flatMap func3

// take our newly composed function and pattern match
def process = functionsComposed match{
  case -\/(err:DomainError) => // do something with error
  case \/-(result: String) => // do something with result
}

As you can see, i am declaratively gluing the functions which could fail together. If they fail at any point then the result becomes a DomainError and doesn’t call the following functions.

It’s important to note that the respective functions don’t “care” about this composition, i didn’t have to do anything special other than declare that there might be an error using the type system.

This approach has the referential transparency of Go whilst being more convenient and declarative

Summary

When you raise errors to a first class citizen in your code by asking the compiler for help, you improve the re-usability and understandability of your code; not to mention it will be more robust as you won’t be having uncaught exceptions flying around.

As your type system becomes more expressive the perceived convenience of exceptions over explicitness disappears.

The lessons I learned from SpringerLink if i was making a website from scratch again

Chris James — Wed, 01 Jul 2015 00:00:00 +0000

SpringerLink is the biggest project I’ve been involved with. It was completely different to every other project in terms of scale and complexity. It was my first real introduction to agile software development and writing software that was actually used by lots of people.

I learnt a lot of random lessons from working on a system developed by hundreds of people and I thought it was an interesting thought experiment to think about what I would do if I were to start again

So in no particular order…

Agile

We were at our fastest when we were developing simple, well understood features that could be done in a week. Even if it’s hard it’s worth the effort to try and trim the fat off a feature. Usually features that take > 2 weeks don’t go well.
Done is when it’s live, if you have a big pile of stuff that’s no longer worked on but is on a branch or not released than that is a huge problem that gets worse with every commit. The reason is to do with feedback loops. If you find a problem in a new feature it becomes more and more difficult for a team to fix it the longer it’s left alone.
We like to see stories going from left to right (i.e, analysis, in dev, qa, live, etc) but it is not the end of the world if stories move backwards. In fact, that usually means you’ve learnt something. At times I have seen situations where stories have been “not ready” for a very long time. Some upfront analysis is important to cut waste but there comes a point where you just need to try it and then you’ll learn rather than speculate.
Continuous delivery (i.e green build = live) is a really nice thing. It’s a shame that releases are still a “thing” for us. The moment software starts going to live on successful builds is the moment people take more ownership of code and you get feedback.
Remember that agile development is meant to be iterative. Get basic skeletal features to live and then improve them. Don’t turn it into a waterfall project delivering in increments.
UX research should not be a recipe for paralysis. If you don’t know, take a punt and then test. If you are in a state where you think changing UX would be too hard so you need to do a lot of upfront research then maybe something else is wrong. One of the reasons why Agile came about is because upfront analysis is almost never as good as build, measure, learn. This does apply to UX.
Tech tasks are important, but can expand to fill the volume you give them. Make the developers feel exicted and interested in the project as a whole so they can strike the right balance in their own minds.
“YAGNI” is a useful stick but don’t use it too often. My rule of thumb in general is do the minimum to get the minimum functionality and at that point you can make a more informed choice as to how you gild the lily.
Actually measure build time and don’t let it get awful. Report it, every week.
Refactor. If the business doesn’t let you, just fucking quit. Also refactoring shouldn’t be a thing. What I mean by that is developers should be empowered to write good software and not have to report every little detail.
Don’t have stats on a new feature? Why not? How do you know a feature is successful?
Why aren’t you deleting features?
NFRs (Non-functional requirements) are important and should be considered for every story, it doesn’t need to take up a lot of time. Too often we have gone live with something without logging and metrics to help support the change.
Retrospectives are really useful but can sometimes go down focusing on a small issue which ends up being blown out of proportion. Needs strong facilitation with someone who has good context on the group. Everything that goes awry does not need a process change. It’s like dropping a footballer every time he misses a pass.
On the other hand, especially in a stable team retrospectives can become too “safe”, not tackling real incumbent issues that maybe the team takes for granted. You should always be asking yourself how can we go faster/better?. A good facilitator can really help this, usually by structuring the retro in a different way to try and make a team think about things differently. Have a look through Game Storming for ideas.
Meetings are important and can be very productive but they need good facilitation and most importantly a focused goal. I have been in too many meetings that have drifted and dived into irrelevant tangents.
Everyone wants a culture of developers “just fucking do it”, but that requires trust in devs and less micromanagement of their time. This contradicts “process” sometimes and is a hard balance to get right.
If your PM insists on story points, dont make it an issue - but don’t report them upwards as it just causes games which is an inefficiency.
That being said, everyone ultimately knows estimates are bollocks. Just refine your process and measure how long real features take to deliver and that gives you your cadence for future planning.
Why are you estimating work that won’t happen for > 6 months? It’s a waste of time for everyone.
Why are you talking about work at any level of detail that won’t happen for 6 months?
Document decisions somehow. 2 years down the line you will struggle to justify your decisions but you probably were doing the most rational thing for the given situation/knowledge.
Stakeholders need to feel involved, informed and consulted. But it is the product team’s job to make the detailed decisions.
Process change is sometimes important, but you need to stick with it for longer than a month to understand if it works. Constant process change just pisses everyone off and slows everything down.
“Process” is often synonymous with micromanagement (in my brief experience) which most developers hate. They almost always boil down to trying to paper over cracks in communication. With an engaged and enthusiastic team who understand the value of all roles and the vision of the product you will get all this for free.
It’s so much harder to refactor code 3 months later; even if the code around it doesn’t change because people and context change. Don’t pay lip service to this, this has cost us lots of time and effort. Just give space between stories to tidy up properly and it will save lots of money
Related: “big refactors” are very dangerous things, especially when new people come in and want to change everything to suit the paradigms they are used to. How do you measure the success of this? That’s your warning, for literally everything in software. Pick your measurements carefully too. If you speed up the build but make the code hard to follow, that is not a success.
Measuring success is hard but it doesn’t have to necessarily be a number so long as you’re honest with yourselves and have a good idea of what success is. We have done work which we know has helped in the long run but you can’t possibly put a number to it. The trouble happens when there isn’t a consensus as to what success looks like, which hurts motivation and is wasteful.
Even though you’re developing software incrementally there should be an overall plan in terms of technical direction. In addition you must include everyone or people will absolutely feel left out and will resent that.
Performance can become a second class citizen which gets delayed until the end of a milestone. Try and incorporate performance metrics sooner rather than later.
As a developer working closely with your BAs and QAs is invaluable. Don’t become one of those devs that just mindlessly picks up stories without talking to people. Remember it is everyone’s responsibility to deliver features end to end. You can help clarify the goal of a story with the BA and QA. You can help make the story easy to test. You can help the BA use technology to measure the value of the story. It shouldn’t be a chore, you’re wrong to think it is and you’re wrong to think it takes too much time.
Put less diplomatically: - ALL FUCKING TALK TO EACH OTHER ABOUT THE WORK. SERIOUSLY. WHY WOULD A STORY BE PLAYED WHEN NO ONE HAS SAT TOGETHER AND TALKED ABOUT IT AS A GROUP. WHYYYY?

People

The right amount and kind of pressure is good. Too much is demoralising and destructive. None at all is also demoralising, and will lead to drift. Set near-term customer-facing objectives that are just challenging enough.
Motivation is hard but visibility of value being delivered really helps.
Don’t be a bastard. People (including yourself) take shortcuts for legit reasons a lot of the time. Most especially everyone (you) makes mistakes or is less experienced with what’s going on. Everything is fixable, eventually. Only be a bastard to people who won’t listen.
A noisy, jolly standup is not necessarily indicative of a productive team. Neither is a quiet one. Don’t judge a team on such superficial things. What does their product owner think?
In the same way, one team works with paradigm/language/framework X. You use Y. Why do you care? Are you all delivering good software? That’s all that matters.
You’re probably doing better than you think as it’s easy to focus on the negatives. Make sure you take the time to celebrate the good stuff the team has done, especially in an organisation with lots of teams.

Technical

Read The 12 factor app. It helps you write software that works beyond your desktop.
Be really strict with statefulness. Things like feature toggles, properties and stuff should either require a server restart or be managed on a request based level. This makes testing simpler and the code much easier to write. If you see tests which are setting and unsetting things, then your server has state. Stop it.
Just use environment variables for config. You don’t need any more complexity than that most of the time. Managing configuration outside the app then becomes language and platform agnostic. Your software is then more likely to be compatible with most PaaS solutions.
No logic in templates, seriously. This has caused us no end of pain.
Don’t use some complicated build tool to do basic things bash can do. Such as… making a zip file.
If your template language doesn’t look like HTML, don’t use it. It might seem fancy and, darn closing those tags is annoying but having to mentally parse template files years down the line is really painful.
Controllers should all look the same. They have some dependencies injected in. Each endpoint parses a request, calls a service and then calls a template with a view model. Anything else is bullshit and is a barrier to understand how particular requests in your system are processed.
Beware of The Onion. Code should read in a declarative way so you understand on a broad level what is happening. This doesn’t mean don’t do encapsulation. Encapsulation means hide how something happens. If it’s hard to understand what is going on then you just need to fix it. The Onion is when you have to dart between a lot of files just to understand what is happening in a system and eventually you have a stack overflow in your brain where you cant handle the call chain anymore.
Logging/events. Don’t have static imports, services should delegate to a “listener”. Logging/metrics is important and should be treated as something that needs to be testing. It seems odd at first but really it isn’t much effort at all and then you can be confident of finding information at 3am when everything blows up.
Trying to encapsulate loads of behaviours in libraries is almost always a bad idea. Saying “doing HTTP and getting circuit breaking, metrics, exceptions yada yada should be standard” is nonsense. Even if it is standard, some day it won’t fit into what we need to do. Composition FTW.
Try not to couple different concerns into one piece of data. In our case having “has Access” coupled with the content meant we cannot cache it.
Follow general good OSS principles. It should be clear how to build the software from a README.
If you use a whacky language like Scala where it’s not very opinionated then a CONTRIBUTING.txt is a useful way to get to a consensus as to how your code should look.
Really think about how you work and what inefficiencies there are and build tooling to overcome it. It was silly how long it took us to have a tool to copy a document from live on to the local box.
If you’re using a statically typed language refuse to use “escape-hatches”. In Scala doing things like “asInstanceOf” is a real code smell and is asking for a runtime error to occur.
If you have a language with a half-decent type system, use it! Code re-use and maintainability can be significantly improved by reflecting everything in the type system. The moment you dont have this (exceptions,nulls,etc) is the moment you have to employ "defensive" (paranoid) programming.
Feature toggles are really useful for degrading the system gracefully in live and also can help testing. In the past we have suffered from having to set specific access rights to test features that have little to do with access rights. If only we had an “all access” feature those tests would be quicker to run and simpler to write.
Making feature toggles a request level concern means you can do things like set a cookie to test a feature in live.
As a functional programmer I really love purity and you can still apply these principles in imperative languages but also in services (i.e APIs). Idempotency is a really powerful thing which makes error handling and general understanding much better. You can always be confident of doing a PUT request more than once wont have some crazy side-effect. Strive to make the endpoints of your APIs as idempotent as possible.

Tests

Make sure everyone is onboard with what kinds of tests you do and what they’re responsible for. For instance:
- Unit tests should be really fast and should not be reliant on anything external (file system, network, etc). Start your project by running these in parallel and then that puts a good restriction as to what you can do. Trying to make them parallelisable a year later is often already a lost cause.
- Integration tests should be testing how your system communicates with another and that’s all. Resist the temptation to start re-testing the downstream service, which complicates your mocks. They should only test “does the system do the correct request and can it parse the responses it might get. Separate out everything else.
- Functional tests are expensive in terms of time and maintenance and therefore should be kept to a minimum. Try and cover key scenarios and the rest should be tested lower down in unit and integration tests. Do they need to test specific rendering of things? Should your functional test fail if the markup changes from a <strong> to an <em>?
Functional tests don’t have to be done through a browser. Why not make your page have a JSON equivalent view that is easier to test. That will also prove you have decoupled your data from your view (i.e a View Model).
Just use fake servers rather than a generic stub framework. The lack of flexibility with fake servers is usually a good thing. You should not have logic in your fakes.
Run your consumer driven contracts against both your fakes and the real downstream service. That way you can be sure your integration tests reflect reality. Perhaps you should use something awesome like mocking-jay
Visualise your test pyramid and try and make it look like the one described by Martin Fowler
If lots of tests unrelated to your change fail then you have some nasty coupling going on there.

Books I've read recently

Chris James — Mon, 22 Jun 2015 00:00:00 +0000

I can’t sell these books well enough, which is why I’m not an author or a salesman. I did really enjoy them, you might too.

The Nature of Software Development

If you’ve practiced agile software development for a while this will feel like an echo chamber, there’s nothing really new here. But it’s a reassuring echo chamber when things aren’t quite going as well as you’d like.

It reminds you how simple it can all be and that is reflected in how easy it is to read. It can easily be read over a weekend.

The Lean Startup

A bit of a classic, the book directs you to measure the success of every feature you make and see if it’s valuable, to hopefully point you toward a successful product.

It’s very tightly bound with agile software, build measure learn.

It can sometimes be quite hard to put into practice but every time you don’t follow it’s principles you know you’re making excuses.

“Make useful products” - harder than you think.

Outliers

This book discusses how people become successful or not. It tries to debunk the romanticism that is put behind entrepreneurs and instead analyses the situations that surround them and how they really benefit them, or not.

That’s not to say talent doesn’t play a part, just that it’s only part of the equation. It’s a fascinating book with some great stories of success but really makes you think about how society is so potentially wasteful of talent.

The Power of Habit

Yet more pop psychology, but a really fascinating read on how prominent habit is not only your own life but in organisations. It’s a fun look at human nature and how it is possible to change those bad habits, and where people/companies get it wrong.

Reality is broken

This is a lovely book on gamification, talking about how people get into the zone and become productive. It’s easy to think about this as a guide book into making cheesy apps to try and make a quick buck but it has also made me think about how my own productivity is effected by things such as distractions or the way the work is laid out.

Thinking fast and slow

This books teaches you to think about the way you think. This is quite important because you are biased. It essentially weighs up your fast brain (instinct) and slow brain (deeper thinking) and when you should trust them.

It aims to make you better at decision making, which everyone should be interested in.

In progress

Cakes, custard and Category Theory

I wish Eugenia Cheng had been my maths teacher. In light of reading Outliers I wish she was everyone’s maths teacher.

She really “sells” mathematics beautifully in a way that just makes you want to do it all over again.

I think my favourite point she has made so far is how people often accuse abstraction as being too divorced from reality to be useful. When actually, abstraction is zooming in on the essence of ideas. Maths has always had a close relationship with programming but I don’t think I’ve appreciated it as much as I have now that I’ve read only 25% of this book.

The Glimpse of Truth: The 100 finest short stories ever written

In my attempt at being slightly more cultured I thought I’d read something that isn’t:

Tech
Fantasy
Pop psychology
Distopia
Steinbeck

The price of inequality

Being a bleeding heart liberal I get wound up easily over my perceived injustices in the world. I truly believe that if we could open up opportunities for everyone (hello again, Outliers) we can all benefit. This book, by Nobel Prize winning economist Joseph Stiglitz, describes how we’re totally fucking it all up.

And books on my list…

Lean Enterprise
The little ML-er
Learn You a Haskell For Great Good (attempt #3)

Improving the quality and performance of your code, guided by the tooling in Go

Chris James — Sat, 04 Apr 2015 00:00:00 +0000

What’s rad about writing Go is how the tooling is really well catered to every day software development concerns.

I recently did a lunch and learn at Springer on why I think Go is great and did a case-study showing how Go makes it easy to create consistent looking, performant and reliable code.

Goal

We are going to write a function which calls two APIs, one is the “Hello” service and the other is a “World” service. Our code will stitch the results together to return the string “Hello, World”.

Iteration 1

To start off we’ll just get the signature of our function together so we can write a failing test against it.

func Stitcher(hello_url string, worldURL string) string {
    return ""
}

Before we get too excited though, let’s run golint. Go Lint is a tool which points out style inconsistencies in your code. One of the key goals for the language is that all the code should feel recognisable and consistent, we can see that I have already made some mistakes.

stitcher.go:7:1: exported function Stitcher should have 
comment or be unexported
stitcher.go:7:15: don’t use underscores in Go names; 
func parameter hello_url should be helloURL

Whether you think comments are good or not, what’s nice about this message is that it makes you think about the public surface of your library. Does every function need to be public? Probably not. However if you do cave in to the linter’s demands, the GoDoc tool will make some lovely documentation for your code; so it’s worth the effort.

// Stitcher calls the Hello and World APIs to create a very useful string
func Stitcher(helloURL string, worldURL string) string {
    return ""
}

Admittedly a small improvement in this context, but in larger code bases consistent naming and well documented public methods will help you out. The linter tool, like all Go tools are very fast so there’s little reason not to add it to your build scripts.

Now that our stub code is linted we can write a test.

package main

import (
    "fmt"
    "net/http"
    "net/http/httptest"
    "testing"
)

func Test_it_calls_the_services_and_concatenates_the_results(t *testing.T) {

    helloAPI := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprint(w, "Hello")
    }))

    worldAPI := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprint(w, "world")
    }))

    result := Stitcher(helloAPI.URL, worldAPI.URL)
    expected := "Hello, world"

    if result != expected {
        t.Errorf("Stitcher failed, expected [%s], got [%s]", expected, result)
    }
}

Calling APIs and creating new functionality is a very common task for many developers and Go makes testing these operations a breeze all within the standard library. This test fails so now we can fill out our function with some real code that we can be reasonably confident will work and is safe to refactor.

package main

import (
    "fmt"
    "io/ioutil"
    "net/http"
)

// Stitcher calls the Hello and World APIs to create a very useful string
func Stitcher(helloURL string, worldURL string) string {

    helloResponse, _ := http.Get(helloURL)
    defer helloResponse.Body.Close()
    helloContent, _ := ioutil.ReadAll(helloResponse.Body)

    worldResponse, _ := http.Get(worldURL)
    defer worldResponse.Body.Close()
    worldContent, _ := ioutil.ReadAll(worldResponse.Body)

    return string(helloContent) + ", " + string(worldContent)
}

We know we haven’t finished yet as there is some naivety to the code in a number of places, most especially error handling, but we will get on to that later. For now we can celebrate the awesomeness of our first pass.

Iteration 2 - Performance

The cowboys we outsourced the two APIs to have said that the APIs cannot return responses quicker than half a second. As we’re calling both APIs we can assume our awesome function will take at least a second to complete.

Intuitively we know that we could do the HTTP calls concurrently which might halve that time.

We shouldn’t just dive in straight away and use Go’s concurrency tools because like all things we need to validate our assumptions otherwise we could be wasting our time. We must be able to prove the code we write adds value in a repeatable way.

Let’s use Go’s benchmarking tools to simulate this and then fix the problem

func Benchmark_the_stitcher(b *testing.B) {
    slowHelloAPI := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        time.Sleep(500 * time.Millisecond)
        fmt.Fprint(w, "Hello")
    }))

    slowWorldAPI := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        time.Sleep(500 * time.Millisecond)
        fmt.Fprint(w, "world")
    }))

    b.ResetTimer()

    for i := 0; i < b.N; i++ {
        Stitcher(slowHelloAPI.URL, slowWorldAPI.URL)
    }
}

The benchmark tool will run the loop N times until it thinks it has a consistent benchmark time. Running the tool confirmed our assumption, the function takes ~1 second.

// Stitcher calls the Hello and World APIs to create a very useful string
func Stitcher(helloURL string, worldURL string) string {

    helloChannel := make(chan []byte, 1)
    worldChannel := make(chan []byte, 1)

    go func() {
        helloResponse, _ := http.Get(helloURL)
        defer helloResponse.Body.Close()
        helloContent, _ := ioutil.ReadAll(helloResponse.Body)
        helloChannel <- helloContent
    }()

    go func() {
        worldResponse, _ := http.Get(worldURL)
        defer worldResponse.Body.Close()
        worldContent, _ := ioutil.ReadAll(worldResponse.Body)
        worldChannel <- worldContent
    }()

    return string(<-helloChannel) + ", " + string(<-worldChannel)
}

This code has successfully halved the execution time and I have the code to prove that’s the case and that it still works as intended from the test we had earlier.

That being said, the code is starting to feel a little unwieldy, there is a lot of repetition so lets use the safety net of our tests to refactor the code a bit before moving on.

const wordSeparator = ", "

// Stitcher calls the Hello and World APIs to create a very useful string
func Stitcher(helloURL string, worldURL string) string {

    helloChannel := make(chan []byte, 1)
    worldChannel := make(chan []byte, 1)

    go getStringFromAPI(helloChannel, helloURL)
    go getStringFromAPI(worldChannel, worldURL)

    return string(<-helloChannel) + wordSeparator + string(<-worldChannel)
}

func getStringFromAPI(ch chan<- []byte, url string) {
    response, _ := http.Get(url)
    defer response.Body.Close()
    content, _ := ioutil.ReadAll(response.Body)
    ch <- content
}

Iteration 3 - Error handling

Now that we have a performant happy path, we should tackle the fact that HTTP calls can and will fail; so let’s write a new test to simulate this scenario and that can help us make the code more robust.

The product owners are a fairly vindictive bunch and have told us that if either API fails we are to default to a string which says “ALL OTHER SOFTWARE TEAMS ARE USELESS, OUTSOURCING IS A SHAM”

Here is our test simulating the hello API returning a 500 which results in our code panicking.

func Test_it_handles_non_ok_from_hello_api(t *testing.T) {

    helloAPI := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        http.Error(w, "oops", http.StatusInternalServerError)
    }))

    worldAPI := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprint(w, "world")
    }))

    result := Stitcher(helloAPI.URL, worldAPI.URL)

    if result != errorMsg {
        t.Errorf("Stitcher didnt fail properly, expected [%s], got [%s]", errorMsg, result)
    }
}

We have a test which proves this problem so we can fix the code and be confident that this particular bug will not come up when we go to production.

const wordSeparator = ", "
const errorMsg = "ALL OTHER SOFTWARE TEAMS ARE USELESS, OUTSOURCING IS A SHAM"

type apiResult struct {
    content string
    err     error
}

// Stitcher calls the Hello and World APIs to create a very useful string
func Stitcher(helloURL string, worldURL string) string {

    helloChannel := make(chan *apiResult, 1)
    worldChannel := make(chan *apiResult, 1)

    go getStringFromAPI(helloChannel, helloURL)
    go getStringFromAPI(worldChannel, worldURL)

    helloResult := <-helloChannel
    worldResult := <-worldChannel

    if helloResult.err != nil || worldResult.err != nil {
        return errorMsg
    }

    return helloResult.content + wordSeparator + worldResult.content
}

func getStringFromAPI(ch chan<- *apiResult, url string) {
    response, err := http.Get(url)

    if err != nil {
        ch <- &apiResult{"", err}
    } else {
        defer response.Body.Close()
        content, err := ioutil.ReadAll(response.Body)

        if err != nil {
            ch <- &apiResult{"", err}
        } else if response.StatusCode != http.StatusOK {
            ch <- &apiResult{"", errors.New("Non 200 response from API")}
        } else {
            ch <- &apiResult{string(content), nil}
        }
    }
}

This code makes our new test pass, hooray! Some Go experts might have some better ways to write this and that's fine because we now have a suite of tests to make sure no matter how much someone tinkers with the implementation they can be confident they have not broken existing functionality.

Summary

Even though this is a contrived example I hope this demonstrates how Go’s ecosystem and tooling can really help to create robust, well tested and readable software.

Go makes it really easy to write maintainable open source software, if you are mad enough to want to use something that blindly concatenates two successful HTTP calls, all you need to do is

go get github.com/quii/go-perf-test-example

and then just check the almost entirely auto-generated go doc to see how to use it.

The final codes are available here: https://github.com/quii/go-perf-test-example

If you have any comments on feedback, please do get in touch with me.

I originally posted this at spiking the solution

What is a typeclass and why should you care

Chris James — Tue, 01 Apr 2014 00:00:00 +0000

Type systems are great because they protect you from lots of errors at compile time. However, sometimes it feels like it gets in your way and can make your code feel less flexible when you are trying to extend certain types that you may or may not have defined.

In dynamically typed languages this is less of a problem because of duck-typing. However, you get no safety here; if you pass in something that doesn’t quack you will get a runtime error.

Duck typing doesn't fix everything though. A human class might be able to quack with a "say" function; you would need to give your program some help in these instances

If you wish to write a generic function that operates on a variety of types, you may be tempted into writing an interface and ensuring all the types you care about implement them.

But this requires changing potentially lots of code and adding in a commonality that only you care about to solve a particular problem. Ask yourself, is this polymorphism truly useful for other people? Or are you polluting the namespace for your convenience (hint: the latter).

The GoF have a solution to this too, the adapter pattern. But every time someone wants to use your library they will have to take their type and wrap them up in an adapter. This feels clunky to me; and of course there is a better way.

Type classes to the rescue!

A type class T is an interface that defines a set of behaviours that a type Foo must implement in order to be of type T. Great, so what?

If you wish to have a polymorphic function, you can require evidence of a type class for the type which is being passed in. This can be any type because a developer can provide this evidence wherever they like; i.e not within the type itself.

This is a great strength of type classes because it addresses the flaws of the other approaches:

Any arbitrary type can use your library if they implement your typeclass; no need for access to the source code.
No need to pollute namespace of types only for your needs. Clear separation of concerns and still polymorphic
No wrappers needed. Scala's implicit means you can pass in your type to the library function without noisy adapters; makes your code easier to follow.
Type safety.

Example

// Pretend this is your awesome library code that you want others to use freely
object MyAmazingSerialisationLib{

  trait Serialiser[A]{
    def toJson(x: A): String
  }

  def writeJsonToDisk[A](item: A)(implicit serialiser: Serialiser[A]){
    def writeToDisk(x: String) {println(s"I saved [$x] to disk, honest!")}
    writeToDisk(serialiser.toJson(item))
  }
}

object TypeClasses extends App {

  import MyAmazingSerialisationLib._

  case class Dog(name: String)

  case class ComputerMachine(id: Int, complexity: Int)

  implicit val dogSerialiser = new Serialiser[Dog] {
    def toJson(dog: Dog) = s"""{"name":"${dog.name}"}"""
  }

  // compiles because we have evidence of serialiser
  writeJsonToDisk(Dog("Spot"))  

  // doesn't compile because we dont have a serialiser in scope
  writeJsonToDisk(ComputerMachine(2, 20)) 
}

Scala provides sugar to make it so your function's type can reflect it's needs, rather than having it as a parameter. You can replace the function from above with the code below and it will continue to work.

// [A: Serialiser] means the compiler expects an implicit Serialiser 
// needs to be available for A
def writeJsonToDisk[A: Serialiser](item: A){
  def writeToDisk(x:String){println(s"I saved [$x] to disk, honest!")}

  // Access the parameter with implicity
  writeToDisk(implicitly[Serialiser[A]].toJson(item))
}

This syntax can provide more flexibility in terms of not having to pass implicit parameters to other sub-functions; your function just requires "evidence" of a type-class.

However, people in your team may be more familiar with an implicit parameter so be sure to explain things whenever you introduce exotic symbols into your code-base.

Conclusion

Type classes are easy to use and an interesting functional design pattern for your tool-belt.
Enables “ad-hoc” polymorphism but with type safety.
Allows users of your useful functions to use them with any type without having to change their own types.
Easier to separate concerns without making lots of types/adapters.
Essential for making your library code re-usable and extensible; a very common approach in the standard Scala libraries and popular open source projects.

Unused features are technical debt

Chris James — Mon, 09 Dec 2013 00:00:00 +0000

Technical debt is often described on a low level, where developers will take some kind of shortcut in order to deliver a feature quicker; at a cost to future development which must be "paid" sometime.

However technical debt can manifest itself in many different ways and one way it has reared it's head on a project I am working on is by unused features.

Delivery culture

Agile development has many flavours and in my experience it has been broadly about improving a team's capacity to release software by improving processes.

For the team I work on, it has worked out great. We generally deliver things in good time and we work well in a productive and enjoyable environment.

Right from the very top of the business at even the board level, the whole company is focused on delivering software and features to the areas of the business who need it. In our company we have a number of departments who fight for the features they want on our main website; plus a number of other products. These requests eventually filter down to the development team and we build them.

From there, that's more or less it; and that's the problem.

When a company becomes delivery focused, it forgets that it should be value focused.

Build - Measure - Learn

I've been reading Lean Startup and it focuses heavily on validating your assumptions when building products. It seems obvious really; you need to verify that the features you make actually have value.

From my (admittedly limited) experience with agile, teams invest a lot of time in terms of reviewing processes in order to deliver things more efficiently; but it doesn't matter how efficiently you deliver a feature that isn't used; you are wasting money by doing so.

This can sometimes be a hard sell to the business. People have forged jobs for themselves by making decisions on what should be built; based on their previous successes. It takes quite a culture shift for a business to acknowledge that it's assumptions about it's customers could be wrong.

However unless you validate your learning with actual real hard numbers, all that you are doing is guessing.

When you consider how expensive software development is, guessing is not good enough.

This is not to say that we shouldn't build anything without some peer reviewed, scientific thesis. In fact, building something and learning that customers don't want a feature is actually a really valuable piece of knowledge for a business to learn about their customers.

Lean startup talks about this in greater detail and I really do encourage you to check it out.

So why is it technical debt too?

We have established that building unused stuff doesn't make business sense (duh); but it has many other unintended side-effects, which I will illustrate with an example.

I am currently working on a project which is taking some useful code we have within our big-blob mothership product and trying to extract it into smaller useful services for the rest of the business.

While doing this we invested a lot of time trying to figure out how we can refactor certain areas of functionality so that they carry on working as we break up the system.

Of course, a BA comes along who happens to look at Google Analytics and then tells us "oh well only a tiny percentage (like 0.0001%) of our users use that anyway".

facepalm

When you don't measure the success of features but leave them in the system, developers have no choice than to treat all the features on the product equally.

Developers have to be mindful of these features when trying to change the system to meet the business's needs; and this can have a considerable cost.

I felt frustrated, and I'm sure the business would too, knowing the time and effort wasted; simply because we don't know our product (and customers) well enough.

Side effects of unused features

I am quite confident that the business realises that a number of features on the site are much less important than others; but probably think "well what's the harm in it being there?". I shall try and debunk this line of reasoning here.

Every unused feature effects a businesses' ability to deliver value and makes the product worse.

Build time

Builds can often slow down the velocity of a team. For us functional tests with Selinium have often really slowed us down. Unused features are a pointless way of slowing down your build.

Refactoring

The best commits are the ones where you delete code as it's less code to maintain.

These so-called harmless features can reduce the malleability of your product because every time someone wants to change code they might have a dozen tests fail for a feature no one cares about.

Additionally, developers might think that generalising code to fit a number of different features could be worthwhile effort; but what if the dev could just delete them instead?

More to learn and understand

By having lots of features in a product, it simply becomes harder to understand; this again makes it more difficult for your team to deliver. In particular when making larger changes.

Demotivating

Knowing that you have to work with code which isn't actually useful, is a frustrating and demotivating thing for developers to face.

On occasion we have measured some feature's success and it was really exciting to see when a feature we made was really successful.

Every story should be like that.

Performance

A number of features we know intuitively (or with analytics) that are not used, are slowing down the delivery of features which are actually useful because they are on the same web page and happen to be backed by slow queries.

UX

Simple is good right? Why are you cluttering your search page with features no one uses? Unused features make your product harder to use and distract users from what you actually want them to do.

Moving on

There are plenty of tools now to validate learning. Metrics are trivial to assign to features and A/B testing frameworks can help you compare features and how a certain configuration can be more beneficial than another.

Rapid prototyping is an excellent tool to quickly test a business's hypothesis before investing too much development effort and as I said, every failed hypothesis is still a positive thing for the business as it improves it's understanding of it's customers to help it make more informed decisions later down the line.

I feel with the advent of agile development, teams took on more of a responsibility for delivery in exchange for empowerment. I think the next step in the maturity of an organisation is for the people responsible for picking features to work closer with the development team; which have the tools and expertise to help them learn more about their customers and make better decisions.

If you can create a learning culture within your business you can improve the motivation of your organisation, learn more about your customers and be more efficient.

Intangibles

Things become more difficult when you are unable to measure the success of a feature reliably.

This might be because of a salesman promising X sales if feature Y is implemented on the website. The development team has little choice and makes the feature but then sometimes this "vital" feature isn't used.

This usually comes about by customers not knowing their own needs, thinking they want a feature and demanding it from a salesperson; but they don't need it.

Even though the feature is unused, it still technically has given value to the business; at the customer's expense.

But eventually, the customer's perception as to how valuable your product is, may diminish. When you base your business on these intangible things, you are running a risk.

In these cases it would be nice to try and work with the customer to try and find features that they want and that they will actually use, as chances are other customers might find them useful which could then lead to more sales than if you just left the customer with their pile of unused features.

It would be exceptionally brave to contact a customer and say:

"You demanded and paid for these features but we notice you haven't used them, is there any way we can make them useful to you?"

but it could lead to some valuable insights about your customers and open up new opportunities.

Other examples of intangibles

"It'll improve SEO"
The person purchasing is not the end user, but is buying on behalf of the users.

Conclusions

Software development has almost always been trying to become more efficient. Whether it is with different programming languages, new development techniques and different styles of project management.

Producing only what is valuable is vitally important but in many domains trying to define what is valuable can be hard, but still very important. Without this understanding the business will find it hard to find new ways to appeal to customers; which is too short-term of a viewpoint.

Thankfully these techniques are widely known these days and many companies can attribute their successes to them. In Lean Startup, it discusses how Facebook really understood their users and that helped them become what they are today. I have been to talks where engineers talk about how they often experiment and hypothesise what their users will like and measure it's success.

It is up to development teams to evangelise this kind of approach upwards, to show how the business can get more out of it's development; if it is willing to work more with developers.

Scala - Just enough rope to hang yourself

Chris James — Mon, 02 Dec 2013 00:00:00 +0000

Last week Dan and I did a presentation at the Springer offices for the Functional Media meetup about our experiences with Scala. We thought it would be interesting to talk about the mistakes we had made with Scala as we thought that would be more interesting than rehashing a number of other talks about why Scala is or isn't good.

You can find the slides here

Scala is a very feature rich language and because of this there are many ways of doing things. This was how Scala was designed, to allow developers to express themselves in a flexible way in comparison to languages like Java.

This flexibility is a double-edged sword and I think this not only effects our team but can also effect the OSS community around Scala. Some developers see the power in being able to write DSLs or construct cool type systems which they think will be really useful but aren't actually as generalised and good as they think and can hamstring other developers. As Jeff Atwood says writing re-usable code is hard and that is especially true when with a flexible language like Scala.

This immediately comes to mind when I read this post by Giles Alexander.

I can actually appreciate some of the author's sentiments. If I started Scala without the aid of working with some very experienced programmers in a pair programming environment I imagine I would find it a nightmare too.

His complaints about def and val to me seem like inexperience in his team with Scala, which then means the resulting code is difficult for everyone. Again, this is Scala's flexibility hurting a team. I actually think good Scala code reads really well, but it takes some discipline and knowhow. Scala can tempt you into writing code you think looks great but is very hard for other people to read.

Lessons from our talk

If you want to have a productive team working with Scala, we feel you need

An enthusiastic team - A team that really wants to learn Scala where individuals demonstrate new powerful ways of getting stuff done to the rest of the team
Pair programming - This helps newer people get up to scratch with Scala.
Code review - You want to encourage your team to learn how to weild Scala's power but at Springer we make sure we have weekly team code review where we go over code so everyone understands
Question things - "The functional way" is not a catch-all argument winner. Leveraging Scala's power is great but readability is the most important factor in code for us.
Monitor build times - You cant really avoid Scala's compilation problems. But if you invest a little time with CI to measure it, you can take steps to manage it.

Generalists

There seems to be a demand on a lot of developers to be "generalists" who know a number of languages and can be productive with them. This is a perfectly valid goal as it means developers generally have a broad and open mind about technical approach.

I'm not entirely sure if Scala is well suited to this as to write good Scala requires you to put effort into learning and appreciating the language. It is a power tool and it means you have to spend time to know how to use it responsibly so you can write concise, readable code.

Some would say this is a bad thing, but after working with Scala for a number of years now and if I was to change language I would want to make sure I picked a language which offers the same power and flexibility of Scala; otherwise I would probably feel like I am fighting the language to get things done.

I totally understand when people accuse Scala being cryptic and bloated, but I think that mainly comes from either using a horrible library of just being inexperienced. I think the true power of Scala is that it gives me the opportunity to express the intent of my code without the need of complicated boilerplate. However, it takes time to get good at it and to be honest that's what I like about it, there's always more to learn.

The Chris Validation Pattern

Chris James — Sat, 30 Mar 2013 00:00:00 +0000

After nosing through some Scala FP books I have been trying hard to write small and simple functions and then compose them together. This post will show an example of where this has taken some needlessly complex code into something terser and easier to understand.

Previously

When getting our feet wet with Scala we would more or less still write fairly traditional OO code with some nice functional goodies mixed in.

This is one of Scala's great strengths and it creates a relatively low barrier of entry for OO programmers; but there is a wealth of cool stuff you can achieve with it without having to blow the dust off your copy of The Gang of Four.

In this scenario we are re-implementing an old piece of functionality, which is user registration validation. The old Scala code to be fair it's completely fine. It's unit testable, fairly easy to understand and gets the job done.

My colleague and I (also named Chris); when re-implementing this feature for a new shared service thought we could use our growing Scala & FP skills to make something a bit nicer; and to show off. I think it's important not to rest on your laurels with Scala and really try new things out when possible. It's not just Java without semicolons.

The Problem

You wish to apply a number of validation rules against something and have a list of error messages to return eventually to a client.

def validatePassword(password: String): Set[ErrorMessages]

The old code

For the first iteration of code there was a class for validation which had one public method which took the item you wished to validate. Within the class were a number of private functions that took the item and checked to see if the item was valid and returned a sequence of errors.

The respective functions are all called at the public function and the sequences are mashed together into one sequence of errors.

class PasswordValidator{

    private def valFunc1(password: Password): Set[Error]

    private def valFunc2(password: Password): Set[Error]

    etc..

    def validate(password: Password): Set[Errors] = {
        valFunc1 ++ valFunc2 ++ valFunc3 ++ etc..
    }
}

The problems with this are :

The validation functions have two responsibilities, applying the business rule and then returning a collection of errors when appropriate. By coupling these responsibilities the functions become less re-usable.
Imagine if the private functions were public so you could unit test them. To unit test the sub functions which contain the actual business logic, you need to check for the error message returned, rather than whether the field passes the validation or not.
It doesn't look enough like a regular expression, which is how you know you are winning at Scala :)

Our Approach

When reading most FP books, they usually emphasise the idea of creating small "pure" functions which you then compose together to achieve greater pieces of functionality.

The ultimate solution: The Chris Validation Pattern

As I said, the colleague I did this was also called Chris so it's not an entirely egotistical name. Anyway ...

By thinking in terms of functions as being "pure" and only having one purpose, the resulting code was created fairly organically.

The first goal was to make it so the validation functions only dealt with applying a rule and letting the caller know if it passed or not

type StringValFunc = (String) => Boolean

def passwordIsTooShort:StringValFunc = 
    _.size &lt; MIN_SIZE
def passwordIsTooLong:StringValFunc = 
    _.size &gt; MAX_SIZE
def passwordNeedsInteger:StringValFunc = 
    !_.exists(_.isDigit)

//etc..

This means that unit testing these functions becomes very easy and not dependant on the error messages. Now that they are decoupled from any domain specific error messages; you could see how they could be refactored into something more generic, such as "stringIsTooShort".

The next task is coupling each function with an error message, which is obviously done by creating a map.

val passwordValidation = Map(
        passwordIsTooShort -> TooShortError,
        passwordIsTooLong -> TooLongError,
        etc
)

You then simply run all the functions against the value you are testing, filter out ones that didn't pass and then take the right side of the Map; which results in your list of error messages.

passwordValidation.filter(_._1(input)).map(_._2)

Ok, so to a non-scala developer it may not be completely obvious what that line does; but if you read into it, you can see it's only really performing two operations over a map.

Granted, this isn't the most impressive piece of code but I think it illustrates well how by being really strict in the responsibility of your functions and decoupling things you can compose useful functionality elegantly.

Edit

Thanks to Dan for pointing out that you can use the values method of a map to retrieve the values rather than the wonky .map(_._2). I also found that filterKeys can make this code more readable too.

So, ultimate version:

passwordValidation.filterKeys(_(input)).values

Understanding Javascript Objects

Chris James — Wed, 26 Sep 2012 00:00:00 +0000

I have been reading the book Javascript Patterns, which is fairly easy reading and I would recommend it if you wish to understand Javascript a little better.

This post will describe JS objects, which is something you'll come across when you're writing almost any kind of JS.

Objects in Javascript are essentially hash tables of key value pairs. The values can be almost anything; strings, functions, other objects, etc.

It is worth noting that these objects are mutable which can offer some cool features but obviously has it's drawbacks. This is why understanding scope is very important because otherwise you can get some unexpected behaviour when your object gets changed by something at runtime.

Here's a basic example of this mutability, where we define an empty object and then add a property.

var person = {};
person.name = "Chris James";

Object literals

Objects can be defined upfront like this with a JSON like syntax:

var person = {
    name: "Chris James",
    sayHello: function(){
        console.log("Hi, my name is " + this.name);
    }
};

person.sayHello()

Constructor functions

Although there are no classes in Javascript you can create constructor functions to make objects in a standardised way which makes sense for your application.

var Person = function(name) {
    this.name = name;
    this.sayHello = function() {
        console.log("Hi, my name is " + this.name);
    };
}

var chris = new Person("Chris James");
chris.sayHello();

New

Using new will implicitly return the this object that you create in the function. You dont need to do var this = {}; - that is also done for you.

In other words what happens is similar to the first example in this post where it makes a new empty object and then adds the properties defined in the function.

Most importantly, the newly created object will also inherit the prototype of the function you defined.

Prototypes?

Javascript uses prototypal inheritance. A prototype simply defines methods that an object referencing it can run.

When you try to run a method of an object, if it cannot find the method locally to the object it will then try to find it in the object's prototype.

Like most things in Javascript, prototypes are mutable which can offer some interesting (and insane) power to a programmer. You could have a bunch of widgets on a web page and based on an event, such as a user action change the functionality of all of them by changing the definition of their prototype.

Returning to the Person constructor function before, I defined a function called sayHello. Every time this constructor function is called and an object is made, a copy of that function will be made inside that object. This is inefficient and should be refactored to use a prototype instead.

This will potentially make unit testing easier too as you can simply create a mock object which has the prototype you wish to test.

var Person = function(name) {
    this.name = name;
}

var chris = new Person("Chris James");

chris.sayHello(); // will fail

//The following is defined after "chris" is made
Person.prototype.sayHello = function(){
    console.log("Hi, my name is " + this.name);
};

//But chris will have access to it by virtue of it's prototype
chris.sayHello();

//Prototypes are mutable...
Person.prototype.sayHello = function(){
    console.log("Hola, me llamo " + this.name);
};

chris.sayHello(); // Spanish!

Therefore, if you think your object will be created more than once and has methods you should definitely consider including the functionality in a prototype to save memory.

Dont forget new

Unfortunately forgetting new will not cause any obvious errors. What it will do is bind new to the application's Global object, rather than the "new" this which would normally be implicitly returned. In a browser, this would be the window object; fail!

This behaviour is fixed in the strict mode of ECMAScript 5 but this is not always available so it's better to be safe than sorry.

Coffeescript constructor functions

If you use Coffeescript's class syntax, it will take care of prototyping methods for you:

class Person
   constructor: (@name) ->
   sayHello: -> console.log("Hello #{@name}")

chris = new Person("chris")
chris.sayHello()

Results in

var Person, chris;

Person = (function() {

  function Person(name) {
    this.name = name;
  }

  Person.prototype.sayHello = function() {
    return console.log("Hello " + this.name);
  };

  return Person;

})();

chris = new Person("chris");

chris.sayHello();

However, as far as I can tell it wont protect you against not using new

Constructor functions vs Object literals

So it seems there are two ways to create objects but which should you use?

If you wish for an object to just contain data then you should use object literals. It is also acceptable to use object literals if you are sure that you wont make another "instance" of your object to have behaviours attached to it.

If you feel you will have more than one instance of your object and in particular if you wish to add behaviour and alter it across all instances then create it using a constructor function.

The Web for non-techies

Chris James — Wed, 11 Apr 2012 00:00:00 +0000

A number of my friends are cool enough to take an interest in what I do and even go as far as to want to try and understand it.

This post is for my less techy friends to help gain an understanding on how when you type an address in your browser that you get something useful displayed to you.

Getting the information to render a page

When you tell your web browser to go to a website, an awful lot happens and it all begins with something called a HTTP request.

A HTTP request contains the information required to get a web page from a server.

Some of the information included is: - Your computer's address, so the reciever knows where to send the information back to - The destination address; i.e Facebook.com - Other information required to complete the request, such as your log in details.

This request is sent to the interwebs where it uses something called DNS to translate your request's address from "facebook.com" to an IP address. An IP address is synonymous to a phone number and this allows your HTTP request to be transported to a server.

Servers

A server is nothing special, they're usually just like any other computer, just configured in such a way that make them suited to the task of working out what information to serve.

When a server recieves a HTTP request it will use code written by programmers to try and understand your request and respond to it correctly.

Examples

Show the home page
Register a new user to the website
Show the product information for a particular product id

Along with code, servers typically store other assets required to make a website, such as images, video and often a database to store information. The aforementioned code will usually deal with working out what information to pull out of the database, or update; depending on the circumstances it has to deal with.

The information to be shown to you

Ultimately the code's goal is to return you a HTTP Response which will contain the information needed for your browser to display something useful to you.

Usually, the information returned is HTML. You may have heard of various dialects of HTML, such as XHTML, HTML4, HTML5. Dont get bogged down by this, they all share the same goal; to describe the information to display to the user.

If you right click on a blank area of a website, your web browser should have an option called "view source". If you do this you will see the HTML used to show you your web page.

You may notice things such as

<h1>The title of the website</h1>

HTML uses "tags" to describe different kinds of content, in this case a H1 means "header 1" or main header. H2, is "header 2", a sub header, h3 a header below that, and so on. A p tag dennotes a paragraph.

You may also notice a tag like this:

<img src="https://www.google.co.uk/images/srpr/logo3w.png" height="95px" width="275px" />

This tells your web browser to go to a server and retrieve an image from the web server called cat, and display it to you.

Obviously the browser has to read through the HTML first to understand what other assets, such as images it has to retrieve for you. That's why you might notice when your connection is slow that things like text render quickly but images appear later.

Making it look nice

Most modern websites use a technology called CSS; Cascading Style Sheets. This rather fancy name is actually just a list of rules to describe how a page should look.

They describe all aspects of style, such as the colour of particular HTML elements, their font, how big they are and where they are positioned on the screen.

For instance if you wanted to make all h1 tags on a website red and bold, you would write in your CSS file something like:

h1{
  color: red;
  font-weight: bold;
}

To attach a CSS file to a website, it is referenced in the HTML using a special tag, just like images. When your web browser sees this tag it again goes to a server, just like it would any asset on the page and retrieve it. Once the browser retrieves the CSS file it then applies it to your page and makes it look pretty; yay!

Javascript

Javascript can also be attached to the HTML you recieve from a server. Javascript is code which is run on your web browser, not on a server. This is called "client side" code, as it just runs on your computer, the client.

Javascript is used to manipulate what the page displays without having to go to the server. It is usually used to make a website feel more dynamic and responsive. By doing code locally on your machine it can do things a lot quicker than going back and forth from a server.

Javascript works by changing the HTML on your browser that is being used to display information to you. For instance there could be Javascript code to update:

<h1>The time is 8:00pm</h1>

<h1>The time is 8:01pm</h1>

Javascript has many applications, in fact new uses for it are found every day, some typical examples are

Validating user input. When you fill out a form incorrectly it will often give you a warning before sending your data to the server, which would be slower.
Updating the page without the need for user input. For instance, a news ticker on a sport website.
Animation on a page, such as "show all" on a large list of items

Huh?

I hope this has given you an appreciation of how websites work. For my role as a programmer I am involved in both the server side code and the client side code. This means when I am writing a website I am trying to write code which understands the various requests a server gets and then doing something useful for the user.

The range of challenges this poses is quite immense and I am lucky enough that technology is going forward at a very fast pace and new ways to interract with the web are being discovered every day; so it's always fairly new and exciting. For me, making a website which is fun and easy to use is a very satisfying activity.

If you have any questions please dont hesitate to contact me on twitter as I do love to chat about this sort of thing :)

Coffeescript Presentation

Chris James — Mon, 09 Apr 2012 00:00:00 +0000

I recently presented to the team an introduction to CoffeeScript (CS). CS is a Ruby-like language which compiles into Javascript.

You can find the presentation here (you may need to refresh the page a few times as google docs seems dodgy). Obviously not as good as the real thing but you'll get the idea.

The code examples I used during the presentation can be found here, if you have a browse through these you should have a good idea of the language features.:

https://github.com/quii/CoffeeScript-snippets

If you are involved in any kind of web development, I encourage you to give CoffeeScript a go. It doesn't take a great deal to set it up and once you have Cake (build system like Make, Rake et al.) automatically compiling your code, it becomes just like normal Javascript development.

I have made a basic skeleton project to get you going, with a Cake file which will automagically compile your CS into JS as you make changes.

https://github.com/quii/Coffeescript-web-application-example

Simply clone that and then run

cake watch

My experience with Coffeescript has been generally very positive. For Springer's innovation week I have built a pretty cool web application entirely with Coffeescript, using Node.js for the server with a MongoDb for persistence. I will blog about that soon but suffice to say being able to do the whole project in one language felt great.

New Blog

Chris James — Sun, 08 Apr 2012 00:00:00 +0000

Welcome to my new blog! I have finally got round to finishing it off. I have had it half finished for around 6 months and today I took the plunge, tidied it up and put it on the interwebs.

My old blog Purple haze used Wordpress. This one is written myself using Ruby with Sinatra. The reason I decided to roll yet another blog was because: - I wanted to learn some Ruby - I felt very discouraged to customise my old blog because I really cant stand PHP

The implementation of this blog is perhaps a little weird. Rather than using a database I am using the filesystem. I write my posts using my favourite markdown editor; IA Writer and Ruby just serves up the HTML generated by markdown.

Stay tuned :)

Chris James - Software developer

The weight of AI on engineers

Accountability has a weight limit

We've been solving the wrong problem

Presence in the problem

Left of dev

Some nice feedback on LGWT

Learn Go with Tests reaches 20k stars

Be nice

Learning by writing learning go with tests

Work in progress

Learn X with tests

It's not a proper book.

Comments

Practical ways to shift QA left

The QA role in a continuous delivery and deployment world

Why is quality important?

A quick waste rant

Shifting QA left, and improvement of the process.

Things to do

Metrics

Logs / Tracing

Tests, pipelines

Automating other quality checks

Trends with bugs, support

Conclusion

HTMX is the Future

The current state of web application development

The costs of SPA

Hypermedia?

HTMX and hypermedia

Clojure HTMX TODO

The edit control

Embracing the web

How it supports non-JavaScript

Why is it The Future ?

It makes "the frontend" more accessible.

Less churn

Cheaper

More resilient

Reduced coordination complexity

More options

Wrapping up

Further reading and listening

How to set a WIP limit

The Goal

In software

Inventory?

Definition of done

How to keep WIP low

The perils of high WIP

A recent, sad tale

So what number should you set?

My job advert for 2023

The headlines

Stuff I like

My answers to the usual questions

What are you most proud of professionally?

What are you looking for?

Can you complete these data structures and algorithms / cracking the coding interview / leetcode exercises?

Values and principles

Quality and delivery speed are correlated, not traded-off

Empowered teams

Work-life balance

Honesty, with respect and empathy

Learning culture

Other practicalities

Remote?

When?

Salary?

2022

Open-source / community things

Goals for next year

Professional

Keep on top of Learn Go with tests.

Back to a simpler world with web development

Speak at GopherCon US

Personal

Speaking at GopherconUK

Reflections on the experience

Learn `X` with tests