We learned this lesson already
In Why "Build vs Buy" is the Wrong Question I explained how you should categorise your systems into appropriate "subdomains" (a la Domain-Driven Design), to help better understand whether to build or buy.
- Core - your business's unique competitive advantage, keep in house and invest.
- Supporting - "boring" software that exists to serve your core. Because it supports something unique to your business, it tends to need to be custom built - but it's not worth over-investing in beyond what your core needs.
- Generic - domains where external vendors have deep expertise you don't need to replicate. Payroll, HR, email infrastructure and so on.
The key message being that whilst you absolutely should buy generic systems, you mustn't mistake a supporting subdomain for a generic one. Trying to vendor a subdomain that is supporting your core can lead to many issues.
This post is about what happens when you do misclassify, and how the process makes it worse.
An introduction to RFP
An RFP (Request for Proposal) is a formal document sent to vendors inviting them to bid on supplying a product or service. In the context of software, it typically involves specifying your requirements in detail and asking vendors to respond to each one.
Unlike in-house software, a vendor commitment is relatively immutable. You need to fully understand what you're buying upfront, otherwise you risk an expensive and difficult mistake to unpick.
This should sound familiar
Specifying everything upfront, then committing. The Software Crisis, Waterfall, decades of failed projects.
This is not conjecture. One widely cited analysis of 5,400 IT projects found that 17% went so badly they put their company's reputation at stake, and on average IT projects run 45% over budget. Waterfall-based approaches were a key contributing factor.
The industry's response was "agile", iterative development, fast feedback, and perhaps most importantly - acknowledging that you cannot fully specify software up front. Agile is a rejection of the idea that you can know everything.
The Agile Manifesto puts it plainly
(we have come to value) Responding to change over following a plan
This instinct to specify everything before building anything keeps resurfacing in new forms. A recent rebuttal of Spec-Driven Development (SDD) puts it well:
The what is discovered by building, not defined before it.
The RFP process makes the same assumption SDD does, that the what exists in advance, fully formed, waiting to be written down clearly enough. For your generic domains, probably. For the others, unlikely.
By committing to fixed scope, you're letting go one of the fundamental benefits of software - if you own the code, you can change it. The product might be right today, but it will likely need to evolve in the future if it's to continue to be useful.
The RFP process is waterfall in disguise
To vendor responsibly, you must specify the requirements exhaustively upfront - you are being asked to make your most consequential decisions with your least amount of knowledge. Consider the software products you admire most. They were not built from a requirements document - they were built iteratively, through cycles of building, measuring, and learning. The RFP process asks you to skip all of that.
And what's the cost of being wrong? A huge waste of engineering time spent working around the constraints of a commitment you can't easily unpick.
The RFP process takes time - significant time:
The average RFP timeline spans 6 to 10 weeks for most organisations, though this can extend to 12+ weeks for complex projects.
People agonise over spreadsheets for hours on end, hypothesising use cases that may never see the light of day. And as decades of software development have shown us, no matter how long you spend analysing, you're likely going to be wrong anyway.
It's tempting to look at a 10 week timeline and assume there's bureaucratic fat to trim. But the length isn't a function of inefficiency - it's a function of the task. Real understanding of a problem comes from building, shipping, and getting feedback from the people who actually use it. The RFP process replaces all of that with the opinions of a handful of stakeholders who won't write a line of code or click a single button in the finished product.
Even the fear of being wrong, corrupts the process. While staring at the spreadsheets, you may lean into MoSCoW prioritisation (Must have, Should have, Could have, Won't have). This model exists to manage uncertainty around scope and helps prioritise. Even though it predates "agile", it leans into one of its tenets, that not everything can and should be delivered.
In this context though, declaring something as a Should is a risky move. What if it's actually a Must and the vendor can't do it? Under that pressure and fear, everything becomes a Must. The tool designed to help make smart decisions around scope, gets crushed by the fear of making the wrong vendor choice and made worthless. If everything is a must, nothing is.
Once you have collated your list of hundreds of requirements, you now have to paw through the vendor responses to them, becoming a temporary domain expert in multiple different products, learning their terminology, their constraints, their edge cases and trying to understand how they would fit into your core. Once you pick a vendor, the other accumulated knowledge becomes waste.
Meanwhile, the understanding you could have built by actually starting to solve the problem remains at zero.
The custom code double standard
To be fair, the RFP process does ask the vendors to identify what requirements are "native" and what ones would require "custom code". This is a good thing to identify. What's revealing is the double standard. Before the commitment, engineering effort is something to be minimised - surely a vendor exists so we don't have to write this ourselves. After the commitment, engineering effort spent making the vendor fit is quietly accepted.
If the amount of custom code is large, this is a clear sign the decision to go to market was wrong. But imagine the optics of that. 2-3 months of RFP, all the time and effort, the calls, the politics of getting everyone "aligned". After 3 months, no one will want to change course, the sunk cost fallacy will override the difficult decision. So you pick the least worst option and hope the integration costs don't hurt too much (they will).
RFP is tolerable for generic subdomains
The definition of generic subdomains is they tend to have fairly stable, well-understood requirements and they are not core to your business. It's not unreasonable in these spaces to have a good enough understanding of what your business needs.
HR software doesn't need to flex with your business, the business will bend itself around it, because the cost of trying to customise it is not worth it.
If the RFP process gets it slightly wrong, it's unlikely to have a big impact. No one likes HR software, but you live with whatever your company happens to use as it tends to be good enough. You work around its limitations and move on.
But this is dangerous when it's near your core.
Supporting subdomains are called supporting for a reason, they support your core business, they are built in service to it. As the core changes, you'll sometimes have to change how its supporting subdomains work, but that's ok, it's usually trivial.
When you vendor a supporting subdomain, its data model and API become your constraint. Instead of a supporting subdomain serving the core, the core has to work around the vendor's limitations.
Software products don't reach a point of completion and stop. Uber hasn't finished and disbanded its engineering team. Like all successful products, it keeps evolving in response to users, competition and market changes. When your core is entangled with a vendor's data model and API, every evolution your core needs has to negotiate that constraint first.
And the consequences of getting this wrong at scale are well documented. The NHS National Programme for IT (NPfIT), launched in 2002, was the largest civilian IT project in the world - an ambitious attempt to digitise patient records across the entire health service. It was scrapped in 2011 having cost over £10 billion, delivering a fraction of its intended value.
One key contributing factor among many: the programme moved into large-scale vendor contracts with the assumption that requirements could be stabilised upfront. They couldn't. The NHS is not a static organisation, healthcare evolves, policy changes. A system specified in 2002 was always going to be wrong by 2005, let alone 2011.
The custom software trap
Any experienced engineer who has worked in this situation knows this story. You end up taking on masses of accidental complexity, workarounds and stress, just to make the core work with the vendor's constraints.
Comically you may end up building supporting subdomains around the vendor system in order to make your core system work!
This is not a new observation. Architecting for Flow explicitly warns that over-customising off-the-shelf products risks accidental complexity and erodes the very efficiency you were seeking. The book's case study is a conference event planner, using a messaging subdomain as its canonical example of a supporting domain - describing it as 'simple CRUD with straightforward input validation and minimal business logic.' If that description fits your subdomain, it's a strong signal you don't need to go to market.
As you load on more custom workarounds and software to work against the vendor's constraints, the more locked-in you become. Before you know it, you have entire projects dedicated to upgrading from v1 of a vendor's api to v2. Entire projects! And this is supposed to be saving us time and effort.
Wrapping up
A lot is made of "Big A Agile", around its processes and certifications - nonsense. Real agile has the clue in the name, it's about agility, being able to change direction. In your case, it's the adaptability of your core subdomains that translates to successful business outcomes. For a worthwhile, non-trivial problem, you cannot specify that up front.
A common response to examples like NPfIT is 'they just didn't analyse well enough.' But this misses the point entirely. The problem isn't the quality of the analysis - it's that no organisation can fully specify a complex, evolving system upfront, regardless of how carefully they try. Requirements change because the world changes. The RFP/Waterfall process assumes a stability that simply doesn't exist.
Supporting subdomains may be simple, but they are not immune to the need to adapt. They play an important role in your business and need to remain flexible to serve your core.
If you miscategorise a supporting subdomain as generic, you lose that malleability. Even if by some miracle, you contradict the decades of experience that tells us you can't possibly know every requirement up front - and successfully create the perfect requirements document for today - it will not be suitable in the future.
If your supporting subdomains become stuck, your core business will forever incur unnecessary costs building around constraints made by the vendor - rather than having the flexibility of a custom built supporting subdomain. The irony is that in trying to avoid caring about a 'boring', 'surely commodity' subdomain, you end up caring about it enormously - just in the worst possible way.
References