The Absurdity of Modern Tools

It was one of those nights when I decided to quickly do something that has been sitting on my todo list for long time. It was a rather mundane task, involving some conversion of repetitive data which should not take any more than ten or maybe fifteen minutes in total. However, after close to forty-five minutes into only the data conversion part I ended up finishing my todo roughly one hour later. It was late that night, what might have had an impact on my focus, but that certainly wouldn’t have accounted for the simple task to evolve into an hour-long journey of anger and fury. Instead, the reason for this was due to the tool I decided to use for solving this problem, namely converting the data: jq.

I’ve been using jq in the past. It’s available for most systems, it can be built as a statically linked binary and it’s small in file-size. While I never found its syntax to be especially pretty or intuitive, it usually worked for the things I needed it for. However, this time it was different.

The data conversion I needed to do required me to do a little more advance things like merging of values different JSON arrays of the same structure, doing some lookups on other properties and combining all that into a specific output-format. There were a couple edge cases, which I also had to consider, but all in all nothing that I probably couldn’t have done manually in under half an hour. Since I’m not a huge fan of repetitive work and thought that jq could do that effortlessly, I decided to go that way. This turned out to be a big mistake.

After browsing through 1251 lines of jq’s documentation, trying one query after the other without the expected results and getting more and more impatient, I eventually gave up and wrote a 6 lines long Deno Typescript in little over 8 minutes that performed the conversion I needed. This situation left me wondering: What has happened with tools these days?

One of the key principles in UNIX design philosophy is to have tools that do very few things, very good. “Do One Thing and Do It Well” is a well-known mantra. However, apart from the amount of functionality, the philosophy also states that such tools ought to be easy to use. In his book The Art of UNIX Programming Eric S. Raymond summarised the philosophy as KISS principle and provided a set of design rules:

• Build modular programs
• Write readable programs
• Use composition
• Separate mechanisms from policy
• Write simple programs
• Write small programs
• Write transparent programs
• Write robust programs
• Make data complicated when required, not the program
• Build on potential users’ expected knowledge
• Avoid unnecessary output
• Write programs which fail in a way that is easy to diagnose
• Value developer time over machine time
• Write abstract programs that generate code instead of writing code by hand
• Prototype software before polishing it
• Write flexible and open programs
• Make the program and protocols extensible.

Two of these rules are especially noteworthy: Write simple programs and Build on potential users’ expected knowledge. If we take only these two rules and see how jq compares to them, we can see a terrible discrepancy between the classic design principles and the way jq is implemented. Not only isn’t jq simple by any means – especially considering that it was created in a time in which C++ as well as Go and even Rust were already available and viable alternatives to C – but it also isn’t built on its potential users’ expected knowledge at all. JSON is a data interchange format extended from JavaScript and widely used for mostly web-related things (like APIs), meaning that it’s likely that (apart from the die-hard UNIX admins or rocket scientists) a good amount of potential users might be front-end and API developers that usually live in expressive and descriptive environments, technology-wise. And while it can surely make you reach a new level of 1337 to show your friends that you have bubble-sorted a JSON with jq, I’m not certainly convinced that this is what the large part of its users aspire to do with it on a daily basis.

However, the ability for jq to perform these kind of fancy tricks comes at a very high price: Complexity. In order to make those things possible, the syntax needs to be implemented in a specific way, introducing and following specific rules, raising the degree of complexity for even just simple tasks quite noticeably. At some point, one might be wondering, whether the ability to implement sorting algorithms in a tool like jq – which describes itself as a lightweight and flexible command-line JSON processor – even makes sense in first place. Going back to the UNIX design principles, Brian Kernighan and Rob Pike provided a very well-formulated description in their paper called Program Design in the UNIX Environment:

Much of the power of the UNIX operating system comes from a style of program design that makes programs easy to use and, more important, easy to combine with other programs. This style has been called the use of software tools, and depends more on how the programs fit into the programming environment and how they can be used with other programs than on how they are designed internally. […] This style was based on the use of tools: using programs separately or in combination to get a job done, rather than doing it by hand, by monolithic self-sufficient subsystems, or by special-purpose, one-time programs.

By outfitting a tool like jq with the ability to create such complex, special-purpose, one-time programs it basically defeats the whole idea of being a lightweight […] command-line JSON processor. And while that almost certainly has not always been the case for jq, it seems like it evolved from a lightweight tool into something that isn’t a tool anymore. The correct description for jq these days would rather be something like a lightweight domain-specific language designed for processing JSON. Calling jq a lightweight and flexible command-line JSON processor is therefore the equivalent of calling gcc a lightweight and flexible instruction-set generator.

So, what has happened with tools these days? Turns out, jq isn’t the only tool that seems to have evolved into something it initially wasn’t intended to be. Obviously jq is just a scapegoat here; There are plenty of tools that suffered the same fate. Take Caddy for example. Initially, Caddy was a super lightweight, single-binary tool that was the perfect fit for whenever you needed to quickly fire up something that could speak HTTP. Fast-forward to today and you’ll find it to have evolved into a powerful, enterprise-ready, open source web server with automatic HTTPS written in Go that takes care of TLS certificate renewals, OCSP stapling, static file serving, reverse proxying, Kubernetes ingress, and more. And more. What a time to be alive, isn’t it?

This trend isn’t exclusive in tools however. The over-complication of single software pieces is an issue across many different areas. Have you tried building a website recently? It seems that we’re at a point at which we have all those great tools, libraries and frameworks and instead of stopping to teach them new tricks and continue to implement new things on top of them, we just keep iterating on that one thing we’ve built until it becomes this feature-creature, this Frankenstein that eventually grows so big that it even gets its own conference and certifications. This lifecycle is also one of the reasons why some tech-communities (e.g. the front-end/JavaScript community) have become known for notoriously re-inventing the wheel. Only to eventually realise that all these features and extensions lead to things becoming less usable for everyone – the people building things with these tools and the people using those things in the end.

Getting back to the matter at hand, with jq not actively supporting me in performing the required task in a simple fashion, I ended up doing exactly what the UNIX philosophy oughts to protect us from: I wrote a one-time program. Since it was easier for me to re-use existing knowledge in a very common area like JavaScript/Typescript over extending my limited knowledge of a very specific niche language like jq’s, I wrote a one-time solution to this problem. What would have helped in this situation would have been a more abstract tool, that implements popular features – maybe even on top of jq as a runtime – using either a common syntax or no syntax at all, rather than a tool that provides a powerful set of buildings blocks in an unintuitive format that is not common knowledge and that requires me to implement things all by myself. In the end, when I need to process JSON by implementing complex logic, I would very certainly pick whatever common tooling is available to me in that specific environment and situation (like a Python or Ruby interpreter) over installing additional software. Especially, when that software won’t help me avoid implementing a solution and on the contrary will require me to learn yet another way of doing just that.

Even though I keep using jq as a scapegoat to make my case, this is true for other tools and frameworks as well. In concluding, I would state the following:

• We don’t need existing tools to implement more features, we need more tools leveraging and building on top of existing tools and features.
• We don’t need different tools that let us solve similar problems differently - we need different tools that solve different, more specific problems for us.
• We need these tools to focus on only these specific problems instead of continuously evolving them into oversized Swiss Army knives for solving a broad spectrum of problems.

As for my specific case, I should have used gron which happens to do a great job at following the UNIX design principles and further more even descriptively brags about it in its examples:

▶ gron --json testdata/two.json | grep likes | gron  --json --ungron
{
  "likes": [
    "code",
    "cheese",
    "meat"
  ]
}

gron is a pure tool, by allowing you to leverage its capabilities to other tools and vice-versa.

With this in mind, brew uninstall jq (and other oversized Swiss Army knives)

Note: As mentioned repeatedly I only used jq here as a scapegoat for a whole range of tools that seem to sell a lightweight and simplistic lifestyle while having evolved or being in the middle of evolving into oversized Swiss Army knives. There might be a niche for jq where it makes sense to use it as the domain-specific processor that it is, just like there was (and probably still is) a niche that successfully builds and runs things on top of awk. jq, just like awk and similar tools are very respectable works, built by very intelligent people. Nevertheless, my reason for writing this post is to point out how such software is increasingly becoming the new black these days, with large parts of the tech community continuing to push overly-complex or bloated solutions over minimalistic and simplistic ones to fix what usually turn out to be quite trivial problems. As I continue to find out how a wide range of tools that I use(d) to keep around can be replaced with smaller, more focused ones that help me accomplish things with less effort, I will continue to post valuable insights here.