My experience of Introducing a team to Go

I was lucky enough to be invited on to the Go Time podcast to talk about Introducing your team to Go. It was my first time on a podcast and whilst I was a bit nervous and waffled a little, it was a lot of fun because Jon and Mat made me feel very welcome and comfortable.

Introducing a new technology to an organisation is not only about technical merit. Even if it were, we’re all terrible objective judges of how good or bad programming languages are.

Change represents opportunity but also risk and if you want to succeed you need to build trust and consensus.

This requires you to find and influence the right people so you can get help, act with empathy to understand the concerns of a diverse range of roles and people, and work hard to bring people on the journey.

Background

Around 4 years ago I joined a new company. Before I joined they had experimented with a few technologies and then seemed to settle on writing everything in JavaScript.

I’m very keen on Go and felt it would be a far simpler and more effective tool for a number of problems that we had to face. What follows is my story around bringing Go into the company, what worked, and what didn’t.

Bringing it up, badly

A few months in to my new job we were starting a new project so we were in an inception. An inception is where various stakeholders and disciplines around building products get together to try and figure out on a high-level what we’re going to build and discuss a little how.

During the architectural session I sensed an opportunity to stick my oar in and suggest Go for one of the systems that was being proposed.

This was shut down, very firmly by a project manager. Paraphrased:

We write JavaScript here. Using the same languages and tools lets us be efficient. We spent a lot of time experimenting with languages and now we have to focus. We won’t be using any new languages.

At the time I did not take this well. I bit my tongue but inside I was thinking

Who is this PM to decide how we should write software? Stay in your lane. I know I’m right here.

In truth, it was poor judgement on my part to bring it up. It wasn’t the right forum as many people would just see this as me being technically frivolous and I didn’t think hard enough about the motivations of the people around me.

PMs are incentivised and judged on how effectively they deliver projects. They will of course be very conservative and resistant to anything that would add risk to a project.

 Lessons learned

Be aware of the room

  • Do you need to bring up contentious subjects up with all of these people? Do you think it’s likely that people will agree?
  • Think about the motivations of the people you’re talking to and tailor your message accordingly.

Chatting with the right people

Hopefully you’re not naive to think that you can merely suggest an unfamiliar language and everyone jumps on board. If one of my engineers said to me

Hey CJ can we write this in OCaml ?

It would be pretty irresponsible for me to go

Why yes, let’s go for it. More fodder for my CV!

You need to build confidence within your organisation that you’re not going to build some systems that only a couple of people are able or willing to maintain.

You’ll also need a critical mass of people who are skilled and enthusiastic about the language, with some real working software already running on your platform to prove it’s viable to work with.

Influence the right people

If you’re the only one pushing an idea you’re likely to fail. You’ll need to get to know your colleagues, try and identify the ones that are able to hold a captive audience and influence people. If you can convince the energetic influencers half the battle is won and you’ll have other people to help you.

Speak positively about Go, rather than criticising whatever is used now

It’s not productive to harshly judge the current tools. At best you’ll get agreement but at worse you’ll

  • Insult previous decisions made, which is especially bad when you’re unlikely to understand the context behind them.
  • Alienate yourself from people who do like whatever it is you’re criticising.
  • Come off as a jerk and get into pointless arguments.

If you like Go so much, it should be enough to speak positively about it but try to be realistic about it. Yes, Go does have downsides and it isn’t going to make your company 10x more productive overnight, don’t insult the intelligence of the people you’re talking to.

To influence you need to act with empathy.

You won’t convince everyone

Often developers can be inflexible about their favoured tools. Don’t waste too much time on these scenarios because they are a path to negativity and conflict. Usually there’s enough space for some people who like X and people who like Y.

Getting organisational confidence

Hopefully you’ve got a number of people with at least some interest in Go and perhaps 2 or 3 people who you can rely on to help you push forward.

Whoever is in charge should be thinking about the following when looking to adopt a new tool

  • Do my engineers want to write it?
  • Are they able to write good, production-ready systems that they are happy to support?
  • Does it fit into our existing infrastructure and tooling?
  • Can we hire for it?

How did we address these concerns?

 Book club, and learning at people’s pace

We thought we could tackle this lunch & learn style where once a week we had our lunch in a meeting room and invite people to learn some Go.

We started by reading through The Go Programming Language as a group. We hoped to learn a concept together and then set a “homework” of completing some exercises at the end of each chapter.

What actually happened, and what was learned

We had a diverse group, it ranged from engineers with over 10 years experience to only just out of a bootcamp. One major stumbling block was the idea of using a statically typed language. A number of our engineers had only ever worked with Ruby and JavaScript so this was quite a strange thing to get over at first.

You can demonstrate the strengths of static typing quite easily by showing how it stops you passing a string to a function that requires an int. I also related production issues we had with our JavaScript code which would be impossible in Go. It is an act of faith though, if you haven’t got a lot of experience building larger systems it feels like needless ceremony.

The trouble you may run in to is you have to go at a pace everyone can work with. This may frustrate more experienced developers. They may have a temptation to dominate the group and show-off somewhat which leads to others feeling lost and lose confidence. Try to remind them that the goal of the group is for everyone to get up to speed with the programming language and this is an opportunity to practice their coaching and mentoring skills.

Only a few people would do the homework which at the time felt frustrating but what you need to understand is

  • People have lives. Not everyone can commit time to homework outside of office hours, especially parents.
  • Some people want/need to be guided along in a more hands-on fashion
  • Some people just aren’t as enthusiastic as you. That doesn’t mean your efforts are doomed to failure! Just accept this.

Go by example

The book approach wasn’t really working for the group so we thought we’d scrap homework and the quite detailed book in favour of studying Go By Example

We’d work through each page, discussing each small concept in detail and playing around with the code in the Go playground to try to understand the inner-workings better.

Any temptation of seniors trying to show-off and skip-ahead were stamped out.

We’re studying for-loops right now, not channels. We’ll get to that later.

Studying a small part of the language as a group without other distractions worked really well for us. They were comfortable bite-sized chunks that the less experienced could get a hold of, but the seniors could also have a bit of time to explore the topic a little more deeply to understand it better.

Making it more concrete

Whilst the lunchtime club got people more excited about Go and starting to understand how to write it we had other activities to support out goal.

Dedicated Slack channel

This was somewhere to post interesting links, share code, articles etc.

Developers would also show little Go projects they were working on to get feedback and ideas on what they’re working on.

This helped foster the little mini-community that was forming and as the numbers grew the legitimacy of the idea increased.

Group activity

Most of our work broadly fit into 3 categories

  • Building fancy, interactive websites
  • Building web servers which mostly call N APIs to get data, do something useful and put that data somewhere else (e.g a database) or make it available over HTTP
  • Churn some data from a message queue

What I did was designed an exercise to showcase Go’s strengths in building robust, high-performing web-servers.

The exercise involves building a web server which takes a request containing N URLs (e.g http://quii.dev, http://news.bbc.co.uk). The server should respond with a “stitched” response of the two websites response bodies.

The caveats I added to this were you were not allowed to have the WiFi on to complete this exercise, which forces the developer to

  • Use the built-in documentation
  • Use the standard library
  • Use httptest to test the needed functionality (you wouldn’t be able to make a passing test that went to a real website)

The group was really successful in implementing it. The lesser experienced devs could make the required functionality, and the more senior ones could make the exercise more interesting by making the HTTP calls concurrent.

I feel this exercised showed the developers that - They are capable Go developers - They can do more than FizzBuzz, they can do their jobs in this language

10% time

We were lucky enough to have 10% time when every other week we could work on whatever we liked. Naturally a number of us would spend time reading more tutorials and starting to build web servers to do useful things.

I felt like we reached a real tipping point when we had a “Hello, world” server building and deploying through our Jenkins pipeline and deployed to live. This required us to work a bit on the companies’ tooling but finally we could point at something real.

Opportunism

This process was never going to be done in a few weeks but after a few months of 10% time, exercises, prototypes and lunch & learns I felt we were at a point we could build something real, but what?

In truth a lot of the time writing a web server in Node or Go isn’t going to be radically different in terms of developer productivity or even performance. There’s so many things that influence these things and programming language usually isn’t the highest one.

I was working on a system which had a micro-services approach to it. It would process thousands of messages from a message broker and for the most part Node was standing up quite well.

However, we started writing a part of the system which was very CPU bound, crunching a number of strings to build new strings. Node can struggle with CPU bound tasks, and it did for us. Our systems are horizontally scalable, so we went from 3 instances to 10 and increased the memory each needed to 512mb but it still struggled.

We had a pair of engineers spending a few weeks picking at low hanging fruit and trying different approaches to improve performance but it just wasn’t happening

Pitching it

I felt this was the right time for us to try re-writing this one service in Go. I spoke to the tech lead and chief architect and outlined the following

  • Go should have much better performance in these kinds of tasks
  • We have invested a lot of effort in getting a pool of enthusiastic developers who want to write Go
  • Writing code isn’t the hard part here. Understanding the requirements is the hard part and we now have them codified for us to work from. I believed it wouldn’t take long to do properly
  • We had technology-agnostic black-box-tests that we could run against the new version of the service to verify it

The result

As much as we attempted to practice with “real world” exercises as part of our learning, writing real systems is always different and things come up. It wasn’t the smoothest ride but we replaced the Node service with the Go one in around 2 weeks.

It was around 10 times faster than the Node service, with only 3 instances using 256mb of memory

When we deployed it, I thought we had made some kind of mistake as I just couldn’t believe the metrics I was seeing.

Around 4 years later I asked my colleague how the system has held up and he said it has become a very resilient part of the system, now handling double the load it had when we originally made it with very few problems.

The kudos

Presenting our accomplishments to the rest of the development team and CTO was a real highlight of my career. It felt like I had stuck my neck-out a bit and worked hard and this was the pay-off.

When you’re thinking about introducing Go to a company, just remember that while programming languages are fun, they are a means to an end. Think about those ends and how you could get recognised for meeting them.

Our bosses shouldn’t care what programming language we use. Managers care about how we delivered useful software, smashed our performance targets, reduced our AWS bill and had fun. Think about what you’re really trying to achieve and use that to motivate your efforts.

 Go specifics

Almost everything I’ve written here is not Go specific. You could take the same approach if you wish to introduce any language to your organisation.

Go does have some particular strengths that you can use to persuade people to use it

Backward compatibility

  • In enterprise this is especially important. Upgrading a codebase from Scala 2.8 to 2.10 was one of the most tedious activities in my career. Upgrading Go versions is just a configuration change.
  • It means tutorials written 10 years ago are still relevant and correct today and this is one of the reasons it’s an easy language to pick up.

Stripped back, simple language

  • In a previous life I was a Scala developer and the amount of bike shedding around approaches is incredible. You may find yourself frustrated at writing for loops but soon you’ll realise how liberating it is to have only one way to iterate over a collection of things and do stuff.
  • Much less to learn
  • Much less to read

Built for software engineering

All the things you learn about writing software to a high standard in a team environment, not a 1 person project that only works on one laptop is well catered for with Go.

  • Built in testing
  • Built in benchmarking
  • Easy to document code in a consistent manner with runnable examples
  • go fmt for consistent readability
  • Compile to practically any platform you’d want to use and extremely easy to deploy
  • Fast compilation time for excellent feedback loops
  • Excellent IDE support
  • Stable and rich standard library

Wrapping up

Trying to advocate for something new that you feel strongly about can be difficult and stressful. Work hard on your people skills to win-over helpful allies and build trust in your idea.

Think of it as a marathon, not a sprint and keep retrospecting over the course of the journey. As I’ve said context is so important and only you can figure out what exactly is right for your own situation.

Remember that programming languages are not a goal in themselves, they are a means to an end. If you’re thinking about championing Go in your job, try to put on your bosses hat and think about what she’s trying to achieve.

It’s unlikely she's judged on what programming language is used, it’s more likely to be things like whether they’ve delivered a good product, improved uptime or other things like that. If you can then relate Go to how those goals can be achieved you’ll have a better chance of convincing people.

People will respect you for sticking your neck out and standing by your principles so long as you do it in a mature and respectful manner.