The rot of the internal library - Dependency hell & inability to automate

It’s a shame, really when I came across a series of shared libraries that my team built up over three or more years, only to be sadly neglected like an unwanted child.

Code maintenance and hygiene

The thing that annoys me most is that almost no developers talk about maintenance and hygiene, everyone wants to be building a new greenfield project and people rarely care about the cleanliness of our internals. Code that is blatantly rotting away. I recently came across a shared library where we had Spring Boot dependencies from over two years ago. To ensure we were “up to date” with security vulnerabilities, developers had just bumped the sub-dependencies of Spring Boot individually, overriding what that version of Spring Boot was bringing in, one by one.

Okay for the once-off occasional thing, fair call, you have a production release next week and you needed to fix something, only to realise your CVE checker found a new vulnerability so you patched it to move on quickly. But for goodness sake, after that don’t forget about it only to be found by the person who has to do the same thing again next time, and again, and so forth.

DependencyCheck for Gradle

In my current organisation, we have a wonderful-yet-annoying tool that checks the latest security vulnerabililties (“CVE’s”) from a public database during every build from DependencyCheck If any security vulnerabilities exist not only in our Gradle dependencies, but the dependencies of our Gradle dependencies, then the build will fail. Hence, sometimes it is annoying and it blocks you from releasing if you absolutely need to get something out and a vulnerability that isn’t even applicable to your application fails your build. However, it is wonderful in the sense that it keeps you on your toes.

When I came along I planned to do a major version release of our internal shared library so that we could remove the dependency-hell of patches that had been done over years, I brought it to the team and they agreed it would be a good idea to be up to date and clean this up, even though no one wanted to admit they never got around to cleaning it. A shame, really. Where’s the pride in your work? And why was I the first one to bring this up as a topic? What happened before I joined? No one discussed it and it was swept under the rug? I’m a little frustrated because I think to myself ‘surely I am not the only one that cares about hygiene and cleanliness of our code out of a whole bunch of engineers of varying skill levels?’ and yet the evidence shows that I am.

My next thoughts were around how we prevent this in the future. Of course, there is the well-known Dependabot which will update all our Gradle dependencies by automatically raising a pull request as they are found. I’ve used this at a previous organisation with great success (just make sure you don’t let the automatic pull requests it creates get forgotten, because that is a hygiene task in itself).

This is a beautiful solution if your organisation uses Github. Alas those companies like where I am currently on Bitbucket do not have such a luxury. There is a similar tool called Snyk but apparently it’s still not up and running yet from our platform team and approval has taken longer than expected.

Long-term hygiene

The long term-picture is dire if these practices become the norm. You can end up in a state of neglect where:

  1. There are now only a handful of engineers that know how the system works because the tech is so old that you need specialised skills that are really hard to find in the market; and
  2. You get to a point where doing an update becomes a major task affecting so many different people and teams because things might break and you aren't sure exactly of the consequences.

Maybe that’s a little extreme but there are examples of this all around the world in many organisations.

So what now? I mean it becomes a manual process, right?

Perhaps, even though that really sucks. I think one consideration first (that we never discussed) is how bleeding-edge we want to be as a team in regards to these major versions, because we might not want to always be on the latest release of a particular dependency. Perhaps we are okay being one or two minor versions behind, but always on the latest major version. Sounds reasonable, as long as we make time for it. I won’t make that call on my own because I don’t want to be a dictator but I want to raise it with my team and bring a few suggestions to how we can manage this and avoid letting it slip through the cracks.

That is the key really, allocating time for hygiene tasks which don’t have a meaningful impact on delivery. It’s tech debt in a way. Or perhaps, it’s the neglect that leads to tech debt over time that then makes it difficult to release at any point in time in the future. Boy oh boy, keep your services clean.