Work 2018

This post is a summary of work stuff over the past year which covers

• The ways of working, agile and innovation
• Microservices vs monoliths
• Kotlin
• Writing a book (kinda)

The team and the way we work

I rejoined Springer Nature (SN) August 2017 on the exciting premise of being in an "innovations team" where we would attempt to maybe make scientific publishing a bit better using technology.

I was a little sad to be moving from a role mainly writing Go to Kotlin but programming languages aren't everything.

What mainly drew me to the role was the understanding that we would be empowered to work in the way we felt was best.

I am a big believer in "agile"; the actual agile, not the project management nightmare. Stuff like

Individuals and interactions over processes and tools

and

Responding to change over following a plan

We are a small team of 3 developers, UX and a BA. We work very collaboratively with little ceremony. To give an idea our "stand up" is having a chat about what were working on and our general thoughts on the work. By that definition we have at least 3 standups a day.

The responsibilities within the team are more or less shared and we rely on each other's relevant expertise to fill in the blanks when needed. I wont be blocked if our UX is away at a meeting, I'll just take a punt knowing it probably wont be that bad.

We talk a lot and we iterate quickly and actually talk to real users, rather than fighting through many layers of hierarchy.

With all of us feeling empowered and involved it means ideas come to us quickly and if we think we have a good idea we'll discuss it and try it out. We dont worry about 2 week long inceptions and prioritising backlogs; if we think its the best thing for us to try now, we'll do it.

If we get things wrong, we dont sweat it; we can change software. Someone very clever said

A prototype is worth a thousand meetings

The other key to us moving quickly is really acknowledging the rule that the last 10% of work takes 90% of the time and just avoid doing it (at least in the short term).

We often say

Perfect is the enemy of good

We would rather ship something that we hope is good, get feedback and iterate on it rather than tying ourselves in knots over perfection.

In terms of the way we work, I couldn't be happier.

But what are we working on?

The work

We were connected with a journal who had some exciting ideas around data and community building. We thought we could create a minimal viable product (MVP) peer-review system (PRS) to experiment around these topics to hopefully learn how SN can help in these areas.

We thought Trello was cool and we had actually interviewed someone who managed a journal entirely in Trello so it seemed like a nice way to bootstrap a "system" quickly.

Our first iteration was a simple HTML form for an author to submit a manuscript which would create a card in Trello. An editor could then inspect the details and then manage the whole process of peer review manually themselves (for now); when they were happy with the manuscript if they dragged it into the column "Ready for publication" our web hook would listen to that event and then create an XML payload for publishing. We turned this around in about a week and felt really rad.

From there we kept iterating, adding a simple peer reviewer search and inviting mechanism (still in Trello).

Eventually though Trello's limitations in terms of UI were becoming burdensome. It was great to prototype and we learned some interesting things from the users about their processes but ultimately we needed to control the UI. We didnt throw it all out and start again though; we just kept iterating, creating new web pages to address our new needs until eventually we didnt need Trello at all.

We "launched" with our MVP submission system for the journal in September. Since then we have carried on iterating based on user feedback and have been working to try and make the arduous quality assurance process for a manuscript a more pleasant experience.

Innovation vs invention and politics

What exactly is an innovations team? Some would think that we should be using blockchain terraform kubernetes machine-learning wizzpops to change the world. I have often wondered why what we build isn't massively transformative.

The talk Building a winning product and UX strategy from the Kano Model by Jared Spool gives a lot of clarity for me.

The real tension is between invention and innovation. Innovation is where we improve a customer's experience so that they are "delighted". It's moving the needle from basic expectation to something that's awesome.

Invention is when you are launching moonshots at ideas that are wild and likely to fail but if they work, could radically change the business.

When I think about how software teams should be working, they should all be empowered to be innovating. This to me is what agile is all about. You should be shipping a product, collecting feedback and as a team understanding it as best you can and then trying out new ideas to move the needle.

This is exactly how we work and I feel this is how every team should work. You dont work on a project (because there's no such thing as projects in software), you work on a product. You start small and keep iterating for as long as the product lives.

As your product grows so does your team, in terms of improving the way it works, how it collects information and ideas around a domain so that it can deliver better ideas quicker.

Maybe software wouldn't be so expensive if we treated it properly rather than shipping on a particular date and then letting it rot; then re-writing it all over again forever and ever.

My conclusion for all of this, is the whole concept of an innovations team is an anti-pattern. Which is a bit weird to say given that was what was sold to me when I rejoined SN.

The joy of monolith

Until this year I have mainly worked on microservices systems for a good 4-5 years. I really like the approach of microservices in terms of creating small systems within bounded contexts and then gluing them together.

I've always understood that the cost of this is in terms of increased complexity in terms of continuous integration (CI), deployments and making sure your system actually works with unreliable network calls and evolving APIs so they dont break each other (e.g. consumer driven contracts)

This year we monolith'd our system and it has been incredibly liberating. I sometimes sit in meetings with other teams as they discuss their technical stuff and I had forgotten how much of a pain building distributed systems is.

This has really cemented the idea that you should always start monolith when building a system.

YAGNI is getting ignored too much these days, every recruitment email tells me how its a "greenfield microservices project" - sounds awful!

It's far easier and quicker to build your monolith and then you get to see the actual areas in your system that could be separate services when the API is a bit more mature.

A lot of the perceived downsides of monoliths are often fixed with just good programming practices.

At some point a system does get too big and you'll need to break it up, but maybe we should worry about building a successful system before breaking it up?

Kotlin

After writing Kotlin for about 18 months I conclude it's alright.

It feels like a nice happy medium between Scala and Java, but I do miss a number of Scala features such as for comprehensions and pattern matching.

These features being missing often makes Kotlin code more complicated than the Scala equivalents which flies in the face of the opinion of Kotlin being simpler than Scala. Abstractions can and should bring simplicity when used correctly.

Scala for better or worse gives you a lot of abstraction power which most developers will make a mess with if I'm being honest; often myself.

The Kotlin compiler is still quite slow; especially compared to Go but its much faster than Scala.

Ad-hoc Kotlin-isms

I'm personally not a fan of the ad-hoc syntax it has added for commonly known problems. It feels like to me they have really tried to avoid acknowledging monads and the like but have invented different things to learn instead.

I've run into occasions where I've wanted to refactor code from a T? (nullable T) to a List<T> which is harder than it should be.

In Scala the abstraction is clearly modeled so you can call map on them in a consistent way. So if you change the type there's very few code changes to make.

In Kotlin if it is nillable you do things like myThing?.let{it + 1}. But with lists its myList.map{it + 1}. To me it's annoying that they're different. A nullable thing should be seen as a collection of one or zero items in it.

Ecosystem

From what I read a lot of people writing Kotlin still want to feel the comfort of Java land in respect to relying heavily on frameworks like Spring. I really dont want to use a massive framework and neither does anyone in our team.

Thankfully SN has invested a lot of time and effort in allowing some developers to create a lovely library called http4k which offers a simple API to create web servers.

I still feel Go is probably my weapon of choice for a general purpose language but it's certainly not a deal-breaker for me to write Kotlin.

Writing Learn Go with tests

SN is nice enough to have 10% time. In March I thought it would be fun to do a TDD guide of a "Hello, world" function in Go.

I stumbled on a nice approach of starting with a simple function and iterating on it, adding new features of Go as the software grows and gets refactored which results in quite a nice flow of learning.

It got a lot of attention on the social media thing, getting top of hacker news and lots of internet points in Github stars.

This led to me feeling very motivated and since then I've written a lot of content covering most of the Go programming language syntax by TDD-ing useful functionality.

You can find it all on Github

For me the most amazing thing is some members of Chinese Go community have translated it into Chinese!

A few chapters have been contributed by other members from the community and a lot of my terrible grammar has been fixed. Overall there are 39 different contributors to the project.

Using gitbook it has a website and I used Pandoc to generate an epub file from the markdown that the content is written in.

The feedback from the community has been invaluable and very rewarding. Committing knowledge to paper is challenging and trying to explain what you know helps solidify your ideas and I am sure the whole experience has improved me as a developer.

I am immensely proud of this work and intend to press on with it to add more content in 2019.

Summary

• Small empowered teams can deliver useful software quickly
• Innovations team is an organisational smell
• Build a useful, well-engineered monolith first. Then you'll see how you can break it up into smaller pieces as you feel the pain of a big codebase. Taking on the challenges of microservices will just slow your project down at first.
• Kotlin is pretty good, but like all programming languages has its annoyances
• Writing down stuff you're interested in is challenging and rewarding. When people say nice things about your work its very nice, so if you like someone's work do tell them as it'll make their day.