Announcing the Great Rewrite Project

In the world of programming nothing strikes fear in the hearts of programmers more than this phrase:

"We're announcing a complete rewrite of our product."

The battlefield of corporate programming is littered with the bodies of teams and projects that have been destroyed by The Great Rewrite. Nothing goes to plan, everything takes longer than anticipated, and all the wrong people in middle management use The Great Rewrite as an opportunity to expand their power base at the expense of the company's profits.

And I'm going to attempt one, live.

How The Big Rewrite Goes Wrong

There are many reasons why a rewrite of an existing system fails, but these are the major causes in my experience:

1. Moving Target: The new project is announced and suddenly the old project has to have new features added. I'm not sure why companies do this--and I usually suspect sabotage--but it creates a moving target for the new project.
2. Second System Syndrome: Even worse is if the new project tries to add new features while also replacing the old one. This is also a moving target, since announcing new features invites Middle Manager Lampreys (MMLs) to latch onto the new project to make their name. The Second System projects always end up insanely over architected and prone to failure. A great example of a Second System being worse is Stripe's New API.
3. Middle Manager Lampreys: You know exactly who I'm talking about. Those rent seeking middle managers who try to latch onto the project to get credit for its success. Just like a Lamprey they attach to the body of the project, sucking its life out by suggesting random features they can take credit for. Even worse, they might work to secretly sabotage the project so that their previous work on the old project is still a viable excuse to go golfing with the CEO.
4. Alpha Coder Language Zealot: The second system is announced and the programmer in charge declares they're using a "way better language." No actual analysis goes into finding the best technology for the job. The only criteria for the new language is because Alpha Coder wants to use the new language, or thinks it will get them a job when the project fails, or because they watched on Youtube video and the guy who made the language said it'll make Alpha Coder look smart if he used. The project picks Haskell, OCaml, or Lisp and dies because nobody wants to piss off the Alpha Coder by abandoning the failed path.
5. Submarine Consultants: The second system is announced and the company decides to hire consultants to do it. No, your employees can't be trusted and the Consultants Brought Cocaine and Hookers to the sales call. Did you bring Cocaine and Hookers? No? Then Consultants it is. The Consulting firm milks it for everything it's worth. They put substandard programmers on the project, a team of 10 people will have 9 who can't code and 1 who can. Oh sorry, a team of 20 since they also do pair programming. They have 5 "architects" who dictate to the programmers insane designs that will never be finished and that's how they like it because they only make money if the project is never finished.

There's many more causes for sure, but these are the big ones I've ran into in the past and thankfully I don't have to deal with most of these. I don't have consultants, I'm the only "manager", so I only have to deal with potentially being the Alpha Coder who picks Go and invents a crazy Second System that's doomed to fail.

That won't happen. Right?

Why I'm Attempting One Anyway

Everything on my site mostly works, so why would I rewrite it? I actually rewrote it 2 other times before:

web.py

I did a first version in web.py to get things going. I found web.py to be a really good web framework, but decided to switch to Django for educational purposes.

Django

Next I rewrote it in Django, and it took me about 3-4 months but I ran into all kinds of issues hosting Django. This will be a repeating theme in this saga, but Django just doesn't scale at all. I ended up having to up my hosting to $300/month from $20/month just to run enough processes to handle my very moderate load. Once it was running I mostly left it alone, but the Django project kept advancing and I found myself constantly updating random dependencies. One day I simply couldn't upgrade smoothly without major surgery so I decided it'd be easier to rewrite in JavaScript.

Express.JS and Svelte

In the Express.JS rewrite I made the classic blunder of both Second System Syndrome
and Alpha Coder Language Zealot. I was working on my JavaScript course and for various reasons decided to make my entire business only JavaScript. I then added a ton of features and tried to make the site code base also educational and full of features someone needs to make a site. The problem is, I didn't test that this was even possible for a mere mortal and painted myself into a dark corner I couldn't escape. There were all kinds of things Express was terrible at that I should have noticed, like Authentication or properly reporting an exception. Sending emails was a major pain and still is to this day. I also decided to use Svelte (which is an amazing system and my favorite reactive system), but Svelte's tooling is some of the worst in the industry. Builds of my site would take 10 minutes to deploy, I had to get esbuild for any speed, there's no error reporting when there's a failure, and then there's the JavaScript community's constant forced upgrades and failure to understand semver. The last straw was when everyone decided that adding an @ to their projects name would totally solve their huge supply-chain attack surface and renamed their projects, but didn't update the version number according to semver. I came into work one day and suddenly I'm having to rewrite whole chunks of random stuff just to support some JavaScript dude's fantasy that an @ solves his problems.

As you can see I've encountered a few of these causes in my own little world of solo-business ownership, but have I learned my lesson? Will this time be different?

How Will Go Be Different?

I think the biggest feature Go has that makes it superior to all of the previous solutions is Go's brutal simplicity means it's stupid easy to work with and stupid easy to deploy. That's my top #1 A+ gold star motivation for using Go. I do not ever want to run my site inside a container just because some jackass might update one dependency in the 2.5GB node_modules. I don't want to have to run Docker, or update millions of packages because idiots can't pad a string to the right, and that's a hard requirement.

With Go the deployment story is much simpler. You have on binary, you can use systemd easily, or simple containers. Nearly every piece of Go software I use is a dream to deploy. In fact, if I run into a Go server that requires Docker I laugh and point at their project like a bully in the locker. By comparison both my Django and Express projects needed extensive gear to keep them running smoothly, and they eventually failed because of random upgrade paths I couldn't complete.

Managing versions of dependencies seems to be easier, but I've adopted a stance with Go of, "If you piss me off I replace you." I've already done this with my Hugo replacement and my Air replacement. Go makes it easy to write an alternative in a few days for almost everything I plan on using.

The remaining thing I have to do is keep everything simple, simple, simple. No new features. No grand redesigns. I'm going with a dead simple web development strategy I call "Page then Play" that is the sweet spot between "Slow Jank Templates in the Backend" and "Crazy SPA the Never Works." I also just deployed a version of the old site that removes features and simplifies the UI. I'm stripping down everything I can before the rewrite and then only doing a direct copy with not one more thing added at all.

I swear! No, for real! Why are you laughing?! Stop it! It'll totally work!

The Strategy

My working strategy so far is this:

1. Spend time evaluating Go and creating a base web dev kit that works. I tested it by creating various test sites with similar features and then refining it further. I will have to do a few more things but this is the start.
2. Strip out as many features as possible from the old site to reduce the surface area of the work. My goal is to remove everything but what's essential.
3. Plan no new features at all. The new site will work the same as the old, and only have upgrades when I get them for free. For example, ditching Svelte to use plain old pages with a bit of JavaScript means free No Build.
4. Extract the old site's test suite into a separate project to use as a "validator" of the new project. I already have a fairly robust test suite, so I can hopefully point it at the new site to confirm everything is working.
5. Create the new one, testing it as I go and working lock-step one page a time until it's ready to do initial deployment testing with users.

Once the new site is in place and the old one is dead I can then start working on new features.

Why My Situation is Not Realistic

Obviously I'm not a realistic demonstration of this problem, but that should be one more reason you should most likely never do this. I'm able to maybe do this because I don't have managers, politics, budgets, consultants, or any external forces you have. I do have the constant problem of trying to do more than I can, which is the only realistic part of this scenario.

If you're in an organization and thinking of doing a Great Rewrite than you should at least learn about how to maybe do it right, and also learn that there's no way your organization could do it. You probably can't stop all features in the old system. You can't prevent new features in the new system. You can't keep rogue managers out of the project. You may even be forced to hire consultants.

If that's your situation then we're nothing alike. Don't do it. It's death...I mean...unless you're a manager.

Adding Risk to The Equation

I want this project to be a close simulation of doing a rewrite at a company, and the missing component is risk. Imagine if a company decided to rewrite their main product following the ideal (and impossible for most) situation:

1. They remove all unused features and code before the rewrite.
2. They prevent all new features in both the old and new system. It's a direct copy.
3. They stop all Alpha Programmer and Manager Lampreys from interfering.
4. They do this until the new project has completely replaced the old product.

In this situation the company could potentially be losing tons of money from the old product's customers. If they aren't allowed to add features they also probably can't bring in new business or convince existing customers to spend more. They also would run the risk of losing customers if it's a subscription model. If they try to communicate to the customers about the rewrite then there's a risk customers leave if/when the rewrite takes too long.

My situation isn't very close to this since I can keep my old site running and still acquire new customers while I work on the new version. In fact, starting this project might increase the number of people purchasing my courses.

I want this rewrite to be more realistic, so I'm going to add a risk by only selling the Epic Supporter Pack while I'm doing the rewrite. This will have two effects:

1. I'll have removed one big feature (multiple products) from the rewrite by targeting only one product until it's done. This will simplify the new site's development.
2. I'll potentially make less sales than before so I'll have to move quickly so I can start offering the other courses as well.

Obviously if I'm not able to complete the rewrite before going bankrupt then I'll bring back the other products, so my risk is lower, but this should raise the stakes and motivate me to not screw around.

I'll have another announcement about this change later this week.

Where To Watch the Series

I will be doing the development live on My zedashaw Twitch so if you're keen to watch then just wait for any sessions with "Great Rewrite" in them. I also keep my VODs up so you can see them any time. I stream 10AM/10PM EST every day.

I will also post the videos to My Youtube channel when I find the time, so if you prefer Youtube then go there.

Finally, I'll post the videos as a series to my X/Twitter since I guess X/Twitter loves my videos so I can't pass up 500k views per video...even if they have zero traction outside of that weird platform.

Wish Me Luck...Or Cheer My Downfall

As mentioned before, this is an experiment and I hope you get something out of it. What I really hope is that it's far more difficult than I anticipated so if you are ever tasked with a rewrite you can show your boss my retrospective to warn them. If this project serves as a warning to others then I'll feel I've done my job. If it helps people better plan for their rewrite, even more better.

But the most important thing is...man am I tired of managing scripting languages.

More from Learn Code the Hard Way