Skip to content

Scalable systems

Tech journalism can be full of crap at times. I've just read an article with a video about Blockchain & Ethereum entrapping creators into using words, then beating them with their words.

It's title "Ethereum Founders Admit Blockchain Never Designed for Scalability" piqued my interest. I've been telling people that blockchain evangelism is more of an abstract thing like OLAP cubes, which by design cannot get past a certain point for years. For years I've had some look at me like I'm out-of-order, being negative or just plain wrong, but the real-world is showing that blockchain isn't a magic bullet it's made out to be. That is just fine.

It's okay not to be perfect

My actual problem with blockchain isn't the tech at all. Of course emerging tech is not going to solve complex problems immediately. It's the faith and reckless abandon with which people place all their hopes and dreams in the new shiny that bothers me.

A fundamental problem with any transaction log, even without cryptographic assurances is that without pinning, at which point it becomes a system of trust, there is at least a linear cost involved best-case. That means the running costs get more expensive as time goes on. Anyone who has dealt with logs even in passing knows this to be true.

It's okay to not be built for "Scale" hype

Most of my dealing with systems at scale has been in the billions of actions in a very short time-period. Even more of that is in test-scenarios built to produce those results. It's expensive to run something which can handle a billion operations per second. Most businesses including my main job right now don't need that volume.

I designed and built an ERP connector which handled > $3m revenue within a few days for a small growing business. This is not record breaking by any means. It's not even the largest system I've built, it's just the one where I have access to the numbers to see actual dollars received rather than some puffed up investor report.

When that workload passed through my system the ERP system connected to a human warehouse. They were unable to fulfill the orders because they didn't have enough people. They could not scale dynamically. There is no way I've come across during that time to overcome physics.

Some of the design considerations for that system were how to empty logs to a backup cold-storage at intervals so that the logs didn't fill the computer. It's trivial to build a system which flushes content to a file well into the gigabytes of size. It's less trivial to read that, and it's often not even desirable. Amazon Cloudfront logs when exported are 32MB. I'd suggest this is with good reason and may even hint at a CPU-cache-bound workload.

There are places you can learn this

Grace Hopper does a famous talk where she shows people a nanosecond. In some versions of the presentation she brings a millisecond with her. It's a lot more wire for a millisecond, and as Grace says "There are a lot of nanoseconds between here and the moon". These represent the distance covered, over copper; I believe by a signal in that space of time.

Of course these people that invented a protocol and software suite, didn't solve fundamental physics, or computing problems. They were working on other things. They may not need to scale into the billions of operations in a short space of time.

Mechanical systems often struggle to get past the thousands of operations per-second. All a fan does is spin, like a traditional hard-drive. Their speeds, I've not seen anything top 20,000 rpm. All it does is spin!

Being reasonable

According to investopedia blockchain writes are < 10 operations per second. That's a lot slower than most web-apps, it might at glance seem terribly slow given the mechanical example I gave of fans spinning and hard-drives. It's a heck of a lot faster than most staff members for the same task.

When I left school systems operating in the hundreds of operations per second were being sold as innovative; the future. Quarter by quarter the bar rose. It takes time for things to be blisteringly fast. It generally takes a margin of error too.

The actual problem space of serving a billion anything is aggregate requests / time. Of course a mean average is a blunt instrument, but using analysis techniques it's possible to identify peaks, dips and regular load. The problem then is dividing that load between a number of machines.

Something doing a billion things a second is limited to pass-through with perhaps basic transformation; with some tooling and technologies it's not even possible. Home computers often have < 100GB/s full-throttle throughput speed limited to a number of operations. Intel publishes this and still I deal with people acting like enough good-will changes hard-facts.


  • News headlines and articles can be a trashfire
  • Few people have to design scalability from the ground up
  • Blockchain is not a magic hat
  • You never should have thought it was
  • Laws of physics, not thought constrain digital
  • Really fast things have to do less, carry less
  • Compared to people, mechanical, past systems things are fast