Crush or Crash: The Possible Fates of MuskTwitter, Its Distributed Systems, and a Commentary on Freedom of Speech

I launched this site just north of two and a half years ago. And in that time, I have only posted about Current Events once. But here I am, posting about Elon Musk and Twitter. And trust me, I do my best to stay out of the swirling shitstorm that is Current Events Commentary in 2022. This is a tech blog, after all.

But I am doing it anyway.

Why?

Well, first of all, as comical as it all may strike you, what’s going on with MuskTwitter is also mind-numbingly tragic: despite its flaws, a lot of very good, hard-working people depend on Twitter for their business. Activists of many stripes organize events through it. Real-time information that benefits society at large is disseminated (and yes, I am aware there’s a lot of disinformation too). So Elon Musk’s decisions have real impacts on real people.

And secondly, Twitter is a phenomenon where I have both technical knowledge about, and in which I have an albeit passing but vested interest. I am somewhat active on Tech Twitter (lurking is activity, right?), and there are some legitimate ideas exchanged there. I don’t like what’s happening to Twitter right now, and this is how I record my take. And for the record, I have been following this since even before Elon Musk created the dumbest visual “Let that sink in” pun in human history by walking into Twitter HQ carrying a sink.

Plus, let’s face it, it’s still a topic that can get some traffic to my site, so, why not?

Anyway, I hope to get to the technical breakdown of what might be happening at Twitter as soon as I can, but I do feel compelled to set the stage for it with some commentary about the business and economic context here, because those are big factors overall.

Let’s Get the Whole Elon Musk Thing Out of the Way

I don’t know much about Elon Musk. I mean outside of the obvious Tesla/SpaceX/Starlink thing, I know he awkwardly smoked pot on Joe Rogan’s podcast. I saw the episode of SNL he hosted. I don’t Follow him on Twitter, but I know he’s also a douchebag internet troll, which is something most people should grow out of by their early 20’s. Elon Musk, is currently . . . ** looks at notes ** . . . 51 years old.

At best he is over-hyped. At worst he is a far too ubiquitous narcissistic man-baby with way too much money to burn.

But, that’s another blog post altogether.

Appropriate to the topic however, I also know that, based on his many tweets, Elon Musk doesn’t know shit about the various technologies, processes, infrastructure or development methodology on which Twitter is built. I am not going to take the time to link those tweets here. If you want sources and commentary, contact me.

And the killing joke about all that, is that he doesn’t need to know the technology in order to run Twitter properly. He could have simply trusted whatever leadership is (was?) in place to collaboratively alter Twitter to his vision, but in a way that doesn’t just set shit on fire. More on that later.

What about All the Layoffs?

The feathers of Tech Twitter started to ruffle (get it?) once Elon Musk almost immediately laid off . . . fired? . . . over 50% of the staff at Twitter once he took over. The numbers vary depending on the source, but that’s the prevailing amount. And there will be more to come who will leave or have left of their own volition. The number is much higher in some circles. Either way, considering the law of averages, that’s a lot of technically adept people that he just blindly set out to pasture. And for what?

Of course, the Musk fantheys have come to his defense, saying that, “Tw1tt3r wuz t00 bl0ated anD ov3rstaffed anywayz, lollerskates!”

Was it? I have no idea. I have never worked there. Even if it was, first-year MBA students would know that you should do that shit strategically, not all at once through a series of, “my way or the highway” tweets. It’s reckless, to put it lightly.

Why does that all matter? Well, here we get to the technical part:

Distributed Systems and Twitter: The Possibilities

If you’re a first-time reader here (welcome!), I feel compelled to fill you in on some of my technical creds. I started my career as an Infra Engineer in 2012. I spent 8 years dealing with Containerization, and 2 years implementing Kubernetes at a large nationwide retailer here in the US. I was responsible for complex infrastructures at Enterprise scale. I dealt with Private/Public Cloud infrastructure all the way up to the Application stack. I have never been a Dev, but I was neck deep in all the modern buzzwords: Iac, DevOps, InfraOps, CI/CD and so on. I gave input about decisions and architecture not unlike what is run at Twitter, just not at that scale.

The small leap of faith I am asking you for here, is to understand that Twitter is not immune to many of the challenges any Enterprise runs into, no matter what the size.

I also wrote a primer on distributed systems on this very blog.

So what’s all the hullaballoo about, anyway? Well, simply put, with that many technical staff now gone (it wasn’t all tech staff who were given papers, by the way), there remains one very pertinent question:

Will Twitter crash without all of those people maintaining it?

First, let’s get the easy scenario out of the way: It’s possible that Twitter will be just fine. The personnel remaining could work miracles, reverse engineer all Twitter systems, hack the passwords, do the needful, and MuskTwitter emerges victorious . . . functionally, anyway.

It’s already had problems though. MFA stopped working a few days after the layoffs. Twitter’s copyright strike system failed, so people have been posting entire movies on Twitter. I have noticed lag in loading times for content, which is unusual for me. Others have noticed the same thing. This is Elon Musk’s actual Twitter profile (took longer than usual to load):

These are not good things . . . like, at all. MFA going down in particular violates a lot of best practices in security that could cost people money, create outages, or cause security breaches.

All the while, the Musk fantheys continue to say stupid shit, which is usually something like, “El0n iz aweS0me becUz all tHat DeaD wiaght iz g0ne anD Tw1tt3r iz st1LL rUnning, LOLZ!”

I added the bold part because that’s the rub: “Twitter is still running.”

Yes, genius, you’re right, Twitter is still running. But something “still running” is meaningless in the world of IT. Is it stable? What line of sight does anyone have about impending failures? Did a clock of impending failures start with people being let go like that so carelessly, and if so, does anyone still there know about it, or even care? It’s a big gamble and there is way too much at stake to just do things all willy-nilly.

What about Legal and HR? Those have an impact on Twitter’s technology and features. Who’s managing that?

With MFA failing and the copyright strike system failing we already know the answer to these, not to mention things we don’t know about, or that can come up over the next few weeks.

So it should be clear now that if you’re in the “Twitter is still running lolz!” crowd you have no idea what you’re talking about. Or you’re trolling.

Either way, what are you, 12?

If you want some additional insight on the possible failures that could occur with Twitter, read this.

Diving deeper, complex distributed systems at scale suffer from a host of human problems, much less technical ones, no matter the size. These systems are interrelated with a lot of dependencies, especially over time. These interdependent systems are susceptible to a host of human problems:

1. “Good enough” solutions that were never revised. Related to that are MVP solutions that never got out of MVP status.
2. Shitty vendors.
3. Bad platform and design decisions.
4. Internecine politics that act as barriers to execution.
5. Deadlines that were too short so project code was rushed.
6. Flat out incompetence (hopefully not very much).
7. Portions of the infrastructure that had to be migrated to inferior solutions.
8. Workarounds that became permanent.
9. Manual processes that, despite one’s best efforts, have to be done by humans.

. . . and so on.

So, yes, Twitter is built on resilient infrastructure, so it’s still running. But the fact remains: it won’t just run forever on its own without being monitored and maintained.

Have you ever seen the movie Passengers?

There’s a part in the movie where the main characters are able to see what’s happening to the ship’s system failures. The best pop culture reference I can give you about the possibilities at Twitter is this scene.

You need to watch that to understand why Tech Twitter is theorizing. Go ahead. I’ll wait.

If you didn’t watch it or didn’t pick up on it . . . read this:

A major system went down somewhere.

Everything else on board is trying to pick up the load, but the load’s too heavy.

Whatever started this, we got to find it.

And fix it.

Gus Mancuso

. . . “BuT teh sHip wuz sTill m0vinG!”

What About This “Freedom of Speech” Thing?

A third possibility is that it’s not the technology that brings Twitter down. It might just be the changes about who gets to use the platform, which, by the way is also something with which Tech People or people who crack open a book every once in a while are familiar. Mr. Musk has made it obvious that he wants Twitter to be a 4chan-esque platform for Freedom of Speech, whatever the fuck that means.

I don’t make any claims about being a Constitutional Scholar, but I will say this:

Freedom of Speech just means that The State cannot arrest you for what you say. It does not mean that there are no consequences for saying horrible shit that society at large deems as abhorrent.

MuskTwitter is not The State, and like it or hate it, MuskTwitter can make its own decisions about which users get banned and what can or cannot be said on the platform. I have my own opinions about that too but . . . say it with me: That’s another blog post altogether.

To sum up, what could happen to MuskTwitter?

1. Technologically, MuskTwitter could be just fine.
2. Technologically, MuskTwitter could crash and die forever.
3. Technologically, MuskTwitter could sputter and cough while the people left figure shit out and re-staff it with Musk fantheys and clones. And then it will be “just fine” but profoundly different.
4. Politically, MuskTwitter could be just fine.
5. Politically, MuskTwitter will just become 4chan and shift its user base while more and more people just go to Mastodon or LinkedIn.

Bottom line, the choice is up to Twitter’s users, which begs the question: What am I going to do?

I don’t know yet. Twitter is a train wreck I can’t stop watching. I know some who have “jumped ship” and gone to Mastodon . . . but I am still Thinking Out Cloud about it.

Let that sink in.

Hit me up on twitter @RussianLitGuy or email me at bryansullins@thinkingoutcloud.org. I would love to hear from you!