A Whole New Video Infrastructure

You can read this email at https://learncodethehardway.com/emails/24-a-whole-new-video-infrastructure/

Hello once again. This is a journal email describing upgrading my video serving infrastructure to use DASH and webm instead of mp4 and WebTorrent. The project I created to do this is open source and the first batch of videos are live now on the Learn Go the Hard Way course.

In this email I'll recap some of the things I've done to make this happen, and if you want to see the whole saga live you can follow this thread on X:

Vidcrunch Saga

I'll see about getting this series on Youtube (even though Youtube will show it to 5 of you while X shows it to 80k of you).

I also stream this work (and a lot of other fun stuff) on Twitch:

https://twitch.tv/zedashaw

Part 1: Why Webm?

I'm always looking for ways to make it possible for small creators like myself to host their own websites. I do this mostly for my own education but also because it's way cheaper and far less likely to bankrupt me on accident.

Normally hosting only text wouldn't be a big problem, but once you host videos you run the risk of getting "server-lessed." You wake up one morning, and suddenly you owe Amazon or Google or Cloudflare $129,000 because one video was hotlinked for 20 seconds. Those of you saying, "I set a max limit on my account," should know that all of those services ignore your limit until it's too late.

For this reason I have to use some secret sauce ffmpeg settings I've divined over the years to get the size of my videos down while keeping the quality up. Mostly the settings amount to doing multi-pass encoding and tuning for animations since that seems to make the text in my code easier to read.

I've wanted to switch to webm/vp9 for years but it wasn't supported well in Safari browsers and the encoding process for it was insane. It would take sometimes whole days to encode just a 30 minute video.

Recently though ffmpeg and the speed of encoding improved to where it's only 1 hour to encode 3 minutes. With hardware acceleration it could be even faster (which I'm looking into next). The end result is a smaller video, but more importantly, the resulting smaller video has better quality for the size.

Part 2: Enter Vidcrunch

Previously I used a jank JavaScript program to convert my video to smaller .mp4 files, but the modules I used in that program were abandoned. Since I just finished Learn Go the Hard Way I decided to create a Go replacement for it that'd be more stable and easier to extend.

The result of this work is at:

https://lcthw.dev/go/vidcrunch

The code currently has all of the features I'm describing in this email, and if you need an easy way to compress a bunch of videos and create the DASH .mpd for them, then this will do it.

WARNING This tool is current undocumented but that'll improve shortly when it's closer to complete.

Making Vidcrunch turned out to be really easy, and I'll reiterate that Go is super good at processing tools like this. Vidcrunch mostly just takes a .json file and crafts a bunch of ffmpeg calls to make it happen. Most of this though is based on my years of experience wrangling ffmpeg into submission.

Part 3: vp9 or av1

Once I had a first version of Vidcrunch working I started experimenting with vp9 vs av1. Both are better supported today but the big blocker on av1 is that it takes WHOLE DAYS to encode even small videos. Yes, a 30 minute video could take 18 to 24 hours to encode at moderate settings.

I already have a hard enough time getting videos out so having to way possible weeks before a release is not going to work. For that to be viable av1 would have to give me 4K quality in the size of an Atari 2600 FMV.

Vp9 is also a little tricky, with some settings still taking 2x or 4x the video length to encode. That means if you have a 30 minute video you're looking at 1-2 hours of encoding time. Not ideal but the results are good for the time it takes.

The only problem with vp9 is that testing it with these hour long encoding times is a real pain. I found the best way to test conversions with vp9 is to use the -deadline realtime option. This makes ffmpeg do a "realtime streaming" encoding which can encode extremely fast. In my testing a 30 minute video would take about 2-3 minutes to encoding with realtime.

With that out of the way I loaded my new stack of webm videos onto my site and...disaster.

Part 4: Blame it On The Chrome

The first problem I ran into is that the .webm format played just fine in Firefox, but failed to play until fully downloaded in Chrome. I tried every setting you can possibly think of, delving into the deep depths of every bug tracker on the internet with no luck at all. Nothing made these stupid video play in Chrome.

After a couple of sessions on Twitch trying to solve it I realized I hadn't tried one simple thing:

Stop Using WebTorrent

I use WebTorrent not really as a P2P offloader (since it doesn't really work for that anyway) but instead as a "Poor Man's CDN." I mark a bunch of my cheap high-bandwidth servers as "peers" and then webtorrent grabs all the chunks from all of the servers in a distributed fashion. This keeps the videos quick to load and my costs way down.

It turns out though, that WebTorrent's jank finally caught up with me and for some reason Chrome simply won't play a .webm video until it's totally downloaded by WebTorrent.

Part 5: Goodbye WebTorrent

If I'm being honest, I never liked WebTorrent. It's trying to do too many things at once. If all it did was "poor man's CDN" across a bunch of cheap servers it'd work way better. Instead it's trying to be a Cypherpunk's wet dream of anti-censorship in the browser and also CDN capabilities. Doing this all in JavaScript also means using a bunch of web assembly parts just to get at WebRTC and it's just a mess.

Rather than keep using WebTorrent I just decided it was time to try out DASH. DASH is a next generation protocol for distributing media across a ton of servers. I say "protocol" but really it's just an XML file that says where all the chunks of a video are located, and more importantly at what bitrates and sizes.

Companies like Netflix and Youtube use DASH to both break videos up across many servers but also to downgrade a video in realtime to fit your bandwidth and device. If you're on a little phone with a 4G network then you get a tiny 360p video which is fine for your phone. If you're on a 4K TV with a 1Gig fiber network then you get a 4K video.

Part 6: Making DASH Work

DASH is really complex standard, and it's kind of obvious it was made by a conglomerate of Go and Java programmers. Lots of XML with lots of nested CamelCase tags. But, ffmpeg has a reasonable output directly to .mpd format, and there's a Go library that can load the format for modification.

The biggest hurdle to getting DASH to work on the browser is, sadly, you have to use another JavaScript/WebAssembly shim:

https://dashjs.org/

Thankfully though the dash.js API is actually quite reasonable, and my biggest struggle was working it into my existing Svelte video playing code.

Part 7: It's Live

Currently this is all encoded in Vidcrunch, and vidcrunch can now create multiple encodings of a single source video plus create the .mpd file so it plays in DASH dynamically.

The current test of the new DASH-WEBM video setup is live in Learn Go the Hard Way:

https://learncodethehardway.com/courses/learn-go-the-hard-way/

If you try it let me know if it doesn't work. I've tested it on iPhone Safari and Chrome, Linux Chrome and Firefox, and Windows Chrome and Firefox. (If you're running Iceweasel in a QubesOS nixflakes sandbox then it's up to you to make it work).

Future Features

The only remaining feature I need is the ability to specify multiple servers for targets of the downloads. Once that's working I'll create full docs and probably provide an update.

If you try out Vidcrunch and think it needs something then feel free to email me (just hit reply).

Thank you so much,

Zed

More emails we've sent.