Moxie Marlinspike on Decentralization

(Image from CCC recording of 36C3, The ecosystem is moving)

Moxie Marlinspike is a well known computer security researcher, cryptographer and overall a highly respected person in the spheres of secure, end-to-end encrypted communications. He gained popularity back in 2010, when he was the CTO and co-founder of Whisper Systems, the mobile security startup that developed andoperated RedPhone and TextSecure – the products that ultimately lead to what has become Signal today.

Back in 2019, Marlinspike was a speaker at the 36C3, the 36th CCC congress, and held a talk about the considerations for distributed and decentralized technologies from the perspective of a product that many would like to see decentralize.

In his talk Marlinspike pointed out what (from his perspective) the advantages and disadvantages of centralized and decentralized systems are, with him clearly favoring the former. I would like to reiterate on these points and evaluate how these statements have aged.

Marlinspike begins by saying that due to an ever evolving ecosystem, software cannot be written today and enjoyed in the same way 20 years later. He’s making the point that rapid evolution of things is in contradiction with decentralization. He’s asking the question, why that would be, because the internet is decentralized and that seems like a dynamic, evolving rapidly kind of place. He then goes ahead answering that question by pointing out to IPv4, which we presumably unsuccessfully tried to replace with IPv6 for 20 years. Next he points to HTTP 1.1, to which we are stuck until now. After that, Marlinspike continues throwing up acronyms, SMTP, IRC, XMPP, DNS, stating it’s all the same, they’re all frozen in time, specifically circa in the 1990s. His explanation for this: Once you decentralize a protocol it becomes extremely difficult to change. He continues by stating that meanwhile, centralizing protocols has become a recipe for success. You know, it’s what Slack did with IRC, it’s what Facebook did with e-mail, it’s what WhatsApp did with XMPP, according to Marlinspike. He continues: In each case, the decentralized protocol is stuck in time, but the people who centralized them have been able to just iterate super quickly and develop these products that are extremely compelling to people.

Marlinspike is arguing under the pretext, that every protocol has to be able to quickly iterate and that decentralization would not allow this, while centralization does. However, there are multiple flaws in this way of thinking.

First of all, let’s have a look at the definition of what Marlinspike refers to as protocol:

A communication protocol is a system of rules that allows two or more entities of a communications system to transmit information via any kind of variation of a physical quantity. The protocol defines the rules, syntax, semantics and synchronization of communication and possible error recovery methods. Protocols may be implemented by hardware, software, or a combination of both.

…

Internet communication protocols are published by the Internet Engineering Task Force (IETF). The IEEE (Institute of Electrical and Electronics Engineers) handles wired and wireless networking and the International Organization for Standardization (ISO) handles other types. The ITU-T handles telecommunications protocols and formats for the public switched telephone network (PSTN). As the PSTN and Internet converge, the standards are also being driven towards convergence.

Source

A protocol basically defines the laws of communication between e.g. a server and its clients. Within the first chapter we also found out that, just like there are regulatory entities that maintain the actual laws that humans live by, there are similar entities that take care of handling communication protocols, namely the IETF.

Similar to how laws provide us with a framework for things we can and can’t do, and ultimately gives us reliability and stability, the communication laws (a.k.a. protocols) do that for services like the ones mentioned by Marlinspike. We as humans trust the laws we agree upon to be steady and durable. We would not be able to find reliability and stability in an environment of ever changing laws.

For example, what worth would banking laws be, if one day the framework would guarantee the bank to handle the money on your account responsibly, while the next day these guarantees would be gone and the framework would look differently. The basic laws that a bank has to obey do not change on a daily basis. They do not change on a weekly, nor a monthly basis. Heck, changes to these kind of frameworks are in fact so rare that when things change it’s usually a big deal, with plenty of people involved, media coverage, and bells and whistles. The fact that the framework rarely changes however does not mean that banks are unable to iterate on the products they offer, nor that they can’t offer new products. In the past 20 years alone, which was the time frame repeatedly used by Marlinspike, we went from paying with cheques to having our bank accounts in our pockets wherever we go.

But wait, how did we do that, if neither IP nor HTTP have changed a single bit since the 1990s?

It turns out, the protocols that Marlinspike is referencing are not the limiting factor when it comes to innovation velocity. The examples he gave might appear to make sense from a layman’s perspective, however if you consider what he’s actually suggesting it becomes apparent that his argument is flawed. Slack did not centralize and reinvent the wheel on IP, HTTP, DNS or any of the fundamental laws/protocols that Marlinspike was referencing. Neither did Facebook. Neither did WhatsApp. In fact, all these companies still very much use the exact same protocols that Marlinspike claims to be stuck in the 1990s to build their products on. Slack, as well as all the others are presumably using a ton of HTTP requests between individual internal services to operate their platform, just like Facebook uses fundamental protocols like DNS to do name resolution or load balancing or SMTP to send mails to their users.

All these companies, including Signal Messenger LLC, are working within the framework that we as humans have built and agreed upon, whether that be actual laws or rudimentary communication protocols. None of them reinvented or revolutionized a protocol all by themselves and is now using it in a way it gave them competitive advantage over others. Instead, they made use of these pillars of our digital civilizations to build their own products on top of them. Has Slack managed to offer a more appealing solution (to specific user groups) for real-time communication than any IRC network before? They surely did. Is the reason for that because they centralized the IRC protocol and were able to iterate more quickly on it that way? Not at all. I would argue that the protocol itself does not even matter for what Slack has to offer to their clients. On the contrary, I think Slack wasn’t even a good example to begin with, given that they had a big head start and financial firepower but have now been caught up with by alternatives like Microsoft’s Teams, Discord and even open source software like Mattermost and Element. Even with all these new products, the decentralized protocols that Marlinspike is referring to are still in use and very much alive today – even the IRC.

Marlinspike continues by stating that he’s hosting his own e-mail services – something he says he wouldn’t wish upon anyone, for reasons he doesn’t care to explain – and that, quote, the fact that e-mail is decentralized is also what means, that my e-mail is not encrypted and never will be, because it’s too difficult to change at this point, it’s out there in this world, and making that change, it’s probably not going to happen. Unfortunately, this statement is also flawed on different levels.

The fact that e-mail is decentralized is also what means, that my e-mail not encrypted is suggesting that just because a protocol is decentralized, it cannot be encrypted. I’m sure that this was just an unfortunate way to put it, and he doesn’t actually suggest that decentralization conflicts with encryption. I’m also assuming that even though he’s constantly referring to encryption, what he’s actually saying is end-to-end encryption/E2EE.

However, as a matter of fact, e-mail is encrypted, as well as end-to-end encrypted – to certain degrees. If we’re talking pure content, there is TLS for transport encryption (SMTP/TLS), as well as GPG, for end-to-end encryption. If, however, we’re looking at metadata, e-mail will definitely not provide a sufficient level of privacy and security. Marlinspike nevertheless seems to be intentionally unspecific, in order to be able to use these arguments for his narrative.

He goes on by pointing towards WhatsApp, stating that unlike e-mail, WhatsApp is centralized and it’s presumably end-to-end encrypted for billions of people by default. All of the sudden, the vague encryption has shifted into a very precise end-to-end encrypted by default statement. And thanks to the centralized nature of WhatsApp …

[…] they were able to just, you know, roll out a software update

Now this is something that we need to pick apart in more detail. Please bear with me here.

Marlinspike is suggesting, that a centralized system can iterate quicker and with less constrains, due to the fact that it is the single source of truth. A centralized service like WhatsApp can make changes to the protocol or the client application and have them trickle down to its users in a more efficient way.

In theory this makes sense. A system in which updates are being dictated by a single leader, that has full control over every instance running within the system/network, might be able to introduce new changes more easily, especially if these might be critical changes to the system’s core or even breaking changes on a protocol level.

However, what Marlinspike is missing to point out here is the fact that WhatsApp, as well as his own system (namely Signal), despite being centralized services, do not have full control over every step of the trickle down process. Unlike for example Apple’s Messages application, that is operated by Apple, as well as tightly integrated into Apple’s operating systems (iOS/macOS), Signal’s (and WhatsApp’s) control stops at the point of app distribution, which is their website and the app stores they are using. While Apple is able to force updates throughout every instance, down to their users – by including new features and even protocol changes for Messages directly into the system their users’ devices operate on – Signal can only assume that their users will update the Signal app regularly.

This small but important detail leads to interesting effects. For example, take a situation in which there is a group of people using Signal across a vast variety of devices, like Android and iOS smartphones, tablets and computers. While Signal is able to push new features to their apps as well as their underlying services quite easily, it does not mean that these updates will necessarily trickle down to every device of the said group of people.

Let’s assume Signal has introduced a core change to their underlying service, which depends on a recent version of their apps, presumably due to changes within the protocol between the Signal service and the apps. In such a case, Signal needs to take care to orchestrate the deployment of the services, as well as the client apps in a way, in which their users are able to update the apps before Signal rolls out this change on their service. The argument Marlinspike is making against decentralized systems is actually the same argument one can make against a centralized system with decentralized clients, a.k.a. clients the centralized system has no way to enforce changes on. That is, what Signal (or WhatsApp) is.

Signal has to effectively wait for a majority of their users to update to a specific app version, before they can consider rolling out the new change across their services. For an app update to reach market saturation can take a long time. Alternatively they could invest enough time and effort into anticipating such changes on the architectural and software development side, so that they would have a smooth upgrade path in place – in which case, they would basically already be dealing with similar things like decentralized platforms deal with (namely version compatibility).

With this in mind, let’s have a look at what actually happens in Marlinspike’s centralized system, when Signal has decided to push core changes to their services under the assumption that every participant has updated the Signal client application to the latest version, or by simply disregarding the remaining portion of people that have not yet updated:

The group conversation is blocked in this case, supposedly due to a rolled out software update introducing a new feature that requires upgrading the group. There is no more input field to continue communication using the legacy group – which would be helpful for letting members know that an upgrade is required and that everyone should update their Signal app to the latest version in first place. Not only is this a bad approach to upgrading existing features, it also leads to further issues down the road.

Let’s try to first understand what this upgrade means, by looking at the Learn More link from the note:

At this point, our users are getting confused. It seems this group cannot be upgraded after all, even though the note on the bottom of the group said “Upgrade this group to activate new features […]”? What is it now?

Apparently, it’s not an upgrade but rather a completely new group that has to be created, with each of the previous members presumably being invited into it automatically. To quickly reiterate on this: We have a centralized system that is presumably under full control of Signal Messenger LLC, which however is not able to provide a smooth upgrade path that would allow people to continue using their old group until they collectively decide on upgrading.

Let’s click the Continue button underneath the note and try to create a new group.. or.. upgrade:

Pressing the Continue button shows this message, which again changes the story, talking about the group being upgraded after all. Putting aside the confusing user experience due to back-and-forth in regard of the terminology, that makes it unclear to users what will happen and whether it’s possible to upgrade or not, this message points out something more important:

One member of the group will be removed from the group. Why? Well, apparently because it’s not as easy as Marlinspike puts it in his talk, even when you have a centralized service. The user in question apparently has not yet upgraded to a recent version of Signal app. In a decentralized system, this would usually not be an issue, as upgrades would likely be coordinated with the subset of users impacted by the new version upfront and – what’s even more important – most upgrades that might break compatibility with client applications would usually be limited to the local cluster of users and would not immediately have a global impact.

To understand this, let’s take Mastodon as an example – even though Mastodon is not decentralized, but federated.
When a new version of the Mastodon instance software is released, administrators of individual instances coordinate the update with their users. They will likely inform the users of any breaking changes upfront – in case there should be any, because so far, most decentralized systems have had a pretty good track record of keeping things backwards compatible – and would eventually perform the upgrade on their instance, and their instance only. Incrementally the network would roll out the new version, giving its users time to adapt while limiting the impact that bugs and other issues might have.

Fun fact: Even centralized systems use this approach, although not on an independent, per-instance level like decentralized or federated systems do, but instead on internal clusters, on which service administrators perform a so called blue-green or red-black deployment. To put it in layman’s terms, it’s essentially the synthetic implementation of a similar path that a decentralized service (like the Mastodon example above) will naturally take. Android updates are a pretty good example that show how centralized systems (e.g. Samsung’s or Google’s Android branches) perform staged rollouts by clustering users and making the software available to each group successively. So not only are the approaches for releasing changes similar in both cases, the centralized systems also usually have to provide and maintain backwards compatibility for users that haven’t updated to the latest version.

However, going back to the actual topic, let’s reiterate on the situation we’re in:

We are now left with a group, in which members cannot communicate with each other anymore, because the text input has disappeared and the group only shows a note, followed by a big Continue button. This is presumably because Signal has already changed the underlying protocol and services, that they – being a centralized system – are in control of. Due to this change having already occurred on Signal’s end, it appears that it is not possible for our group to communicate anymore. Instead, we are now forced to reach out to each member individually and coordinate an upgrade of the Signal app for the specific platform that they’re using, which might not even be possible due to various reasons (e.g. them not being able to perform the update at this point in time, or them using a legacy device that doesn’t even retrieve updates anymore).

Signal’s centralized service, paired with what feels like a disregard for users that are not able or willing to always upgrade to the latest client app, has led our group to become completely unusable for all members within a matter of months. The only option we have at this point is to accept that one member will not be able to continue the conversation and move on with all the other members. All that for being able to have features like @mentions and group admins, as the text mentions, which might not even be a necessity for this specific group in first place.

Centralized systems like Signal try to force things upon their users – as they also did with MobileCoin – but might fail to do so due to unforeseen circumstances on the users’ ends, or simply out of their users’ unwillingness to accept the newly introduced changes (again, MobileCoin). Unlike in a decentralized system, where users have full control over every component that is required for interacting with each other (e.g. the clients as well as the service instances, in case there are any), and where users can choose to either accept the changes or continue running their current version of the service, in a centralized system like Signal, users could end up being forced out of the platform.

Additionally, I’d argue that the fact that Signal is a centralized system has presumably led to their architecture, as well as software development being lazy, in a way in which they seem to have taken shortcuts in order to not have to deal with a slightly more complex upgrade path that might have helped keeping our example group backwards compatible in first place. Something that would have been a must for decentralized systems – backwards compatibility – seems to have been deliberately sidestepped in case of Signal, under the assumption that being centralized they are presumably in control of the whole system and might not need to put up with the extra effort.

Many things Marlinspike mentions in his talk are inaccurate and even incorrect. It is interesting (and slightly ironic) how time has proven him wrong even with his own app, Signal, while decentralized services like e-mail or matrix.org on the other hand have solved the things that he pointed out, as well as the example I just gave, far better than Signal did.

While on Element (a matrix client) groups/chat rooms periodically receive updates as well, these are not being forced upon users. Older versions of the groups/rooms, as well as older versions of the Element client, remain perfectly functional to this day. Surely it does make sense to keep up with updates for the sake of retrieving security fixes, however, in many cases security related updates could be made separately available from feature updates (e.g. long term support versions). Besides, decentralized services that aren’t available to the whole internet but instead only serve a group of people could be secured by other means (e.g. authenticated access, VPNs, …), to deal with situations in which services would not be able to receive updates for prolonged periods of time. Especially since protocols between instances (e.g. ActivityPub, S2S XMPP, SMTP, …) rarely change significantly, most instances will remain compatible to newer versions for a good amount of time. Most breaking changes, if any, usually only occur between clients and instances in first place. The advantages that Marlinspike argues for in centralized systems unfortunately don’t always play out that way in real life, as can be seen with the example above. If Signal would not have taken shortcuts implementing their new groups, they could have provided the same upgrade path and backwards compatibility a decentralized system naturally offers. In fact, I’d argue that in most cases, in which Marlinspike refers to centralized systems being quicker (iteratively), it’s only due to shortcuts that were taken while nobody is watching.

Marlinspike’s assumption that he can’t have encrypted e-mail, because e-mail is stuck in time due to its decentralized nature is also not correct. E-mail is a valid example of a protocol that has been around for decades and is still functional up until this day for every participant, even with new features (take SPF and DMARC for example) having being introduced over the years. Is it a perfect example of decentralization? Surely not. Could things have been done differently/better in hindsight? Of course. Yet the whole world is still able to use e-mail for communication, no matter whether participants are using a latest generation Apple device or an old Intel Pentium machine from the 90s. A decentralized system like e-mail still works, while a centralized service like Signal has become dysfunctional after only a fraction of the total lifetime of e-mail, as demonstrated above.

And it’s not just e-mail, it’s similar with XMPP, what Marlinspike had also pointed fingers to in his talk. Take any ejabberd from the mid 2000s and try to connect to it using a reasonably well maintained client and you’ll have no issues with that. Does stability of the protocol mean that XMPP is stuck in time? Certainly not. While you could still use OTR, most XMPP clients have moved to OMEMO these days. Does that mean you cannot use XMPP if your client does not support OMEMO? Not at all, it only means that this specific feature will not be available to you. Heck, if XMPP is stuck in time, then tell me how come Jitsi Meet, which is using the decentralized XMPP protocol, can do video calls with 100 participants while Signal’s video calling only supports 40 people and WhatsApp’s group calling only 8 at most?

Another great example is BitTorrent. According to Marlinspike’s arguments, BitTorrent must have died a long time ago, due to all the pain points and disadvantages decentralized systems supposedly have over centralized systems – slow to iterate, hard to maintain, no sufficient censorship resistance, not easy to deploy, yada-yada-yada. Yet, BitTorrent has had its fair share of changes to its wire protocol as well as its services (e.g. DHT), and its popularity has been steadily growing over time. 21 years after its initial creation, BitTorrent’s monthly users are estimated to be more than a quarter of a billion, without any indication of things changing for the worse. And regarding compatibility between versions, well, it seem like they’re doing a far better job than Signal or WhatsApp so far:

It is possible to include both a v1 (btih) and v2 (btmh) info-hash in a magnet link, for backwards compatibility.

…

All new features in BitTorrent v2 that are not backwards compatible have been carefully given new names, to allow them to coexist with the v1 counterparts. Hence, it’s possible to create hybrid torrents. That is, torrents that can participate in both a v1 and a v2 swarm at the same time, serving the same files.

Source

Turns out the world has solved many of the issues that Marlinspike attributes to decentralized systems. With plenty examples of successful and sustainable decentralized services (Mastodon, Scuttlebutt, Tor, etc.) being around today, we can prove Marlinspike’s views on a supposedly moving ecosystem that decentralized protocols can’t keep up with as incorrect. In particular, statements like these make one doubt that these are honest and carefully thought-through opinions on the topic of decentralization:

• Decentralized systems are not inherently encrypted, in fact most decentralized systems are not end-to-end encrypted by default.
• Things like metadata protection are going to require new techniques. And that those new techniques are more likely to evolve in centralized rather than decentralized environments, because centralized environments are were things tend to change.
• […] meanwhile my e-mail still isn’t even end-to-end encrypted and never will be.
• Decentralized systems aren’t inherently going to give us the privacy properties that we necessarily desire, and that it’s more likely that we can develop technology to, you know, offer what it is that people want in centralized environments
• Everytime there’s like an outage people are, like, you should decentralize, you know, you wouldn’t have as many outages. But I think the reality is that you’d just have more outages.

Marlinspike also presents individual features of Signal and how they were built, suggesting that these things could only be built within a centralized system – which is certainly not true. While it does take more effort to build things like zero knowledge proofs and metadata encryption in a decentralized system, it is certainly possible to do so, and there are projects today that I’ve covered previously on other pages of this journal, that are actively doing that.

Additionally, Marlinspike appears to have a different approach to censorship resistance within decentralized systems, than people who actively work with decentralized systems tend to have. His idea of having users on different servers and moving them from a censored server to other servers is inherently flawed. He even correctly identifies the idea as a game of whack-a-mole himself, only to suggest doing exactly that in a centralized environment a moment later, but instead of calling it server, he then refers to the instance users connect to as ingress point. Marlinspike doesn’t mention at all what would happen if the actual server behind his ingress point would get taken down – which, in censorship scenarios, is far more likely, unless you’re outside of the jurisdiction that is trying to censor the service in first place. Clearly, Marlinspike does not consider the USA, which is the jurisdiction he operates in, to ever be censoring Signal Messenger LLC services. On the other hand, why would they, if they could kindly ask for access to the servers and proxies instead?

Coming to an end of his talk, Marlinspike starts randomly poking into topics like Yahoo Mail, control, bitcoin, open source, financial incentive and how decentralized systems make everything harder, in a world where he feels like we should be making things easier for people to deploy. Well, let’s take a look at the Signal-Server repository to see how easy it is for people to deploy that. Oh well.

One specific quote from the waffling, that I would like to highlight is this:

And then, even in distribut.. eh.. like, pseudo-distributed kind of models like bitcoin, I think that the control that people have, it’s sort of interesting, that the control that people are seeking has manifested in the form of forks, you know. So that when there’s a disagreement, people just, you know, starting a new network, you know, like, you know, eh, bitcoin cash, or you know various alt coins, and people take like the existing code base and just start a different service. Ultimately I think that has led to a lot of confusion in terms of users that are engaging with these networks, […]

Fast forward little over one year:

MobileCoin, a cryptocurrency that has received technical guidance from Moxie Marlinspike, the creator of private messaging app Signal, has raised $11.35 million in fresh venture funding across two rounds from Future Ventures and General Catalyst.

As previously mentioned, from my point of view Marlinspike’s opinions on decentralization and crypto currencies are inconsistent and conflicting. While I respect the work that he did in the areas of cryptography, to me it doesn’t seem like the opinions he upholds around decentralization and crypto currency are valid. Especially after the MobileCoin situation, it seems like his views change depending on the entity he was/is acting on behalf of, whether that be Marlinspike as a person or Marlinspike the (former) CEO of Singal Messenger LLC, supporter of Facebook’s efforts for implementation of WhatsApp’s E2EE, and technical guide for MobileCoin. To me, the talk at the 36C3 appeared to be more of a sales pitch for capitalism-driven, centralized platforms than a well thought out and honest talk about decentralized systems.

As demonstrated above, centralized systems have their own set of quirks that in a many cases are being dealt with in a rather poor fashion. Considering the big steps the decentralized community has made over the past two years in particular, one might be thinking that maybe Marlinspike has come to the conclusion that, after all, decentralized systems are here to stay.

Unfortunately, even with a fair amount of competition from the decentralized camp these days, the Signal founder still appears to think that people are trying to standardize sharing video over IRC, that after 30+ years his email is still unencrypted while on the other hand stating that he was contemplating to create a filter that would make incoming GPG-encrypted e-mails bypass his inbox entirely, and that the reason WhatsApp went from being unencrypted to E2EE in a year is not related to funding.

UPDATE 2022-08-15: The news about Twilio’s incident today showed once again that centralization – whether that’s platforms or, as in this case, identifiers – is rather a thing of the past than the future. With this in mind, did you know that Decentralized Identifiers became a W3C recommendation nearly a month ago?