Reimagining Infrastructure for a Distributed Era

Computing has historically had alternating eras of centralization, distribution, and re-centralization. Now we seem to be at the dawn of a new distributed era of computing. What is involved, and how must infrastructure be reimagined to meet the needs? Come, join me in the exploration! This post is the first in a series of five posts, giving an overview of the topic, which will then be elaborated in the subsequent posts.

The first era of computing was of course mainframes and supercomputers, which were completely centralized and batch operated. Then came mini-computers surrounding the mainframes, with online access via terminals, which gave rise to a degree of de-centralization. After that came personal computers and PC-LANs, which completely exploded the whole concept of centralization. And now, of course, the internet, e-Businesses, smartphones, and IOT have made computing ubiquitously distributed!

But then the advent of cloud computing has swung the pendulum hard back to centralization! Public clouds are becoming the infrastructure of choice for most data-center based applications with e-Businesses and even enterprises increasingly adopting them for new apps. And, even legacy enterprise apps are increasingly getting migrated to public IaaS (a phenomenon known as “lift-and-shift”). Gartner group estimates that about 18% of legacy apps have already been migrated, and more are on the way.

While the foreseeable future may still be hybrid cloud, the trend is towards increasing public cloud centralization, with the growth rates of Amazon AWS and the revitalization of Microsoft via its cloud services being strong indicators. Gartner forecasts cloud system infrastructure services (infrastructure as a service or IaaS), to grow 27.6% in 2019 to reach $39.5 billion, up from $31 billion in 2018. Many enterprises (even some fortune 500 firms) have initiatives in place to shift most of their non-business-critical applications to public clouds over the next few years.

SaaS is also a form of centralization: instead of many instances of apps at customer data center, it is a multi-tenanted centralized delivery model now. Much of that SaaS also is being delivered from public cloud IaaS. Any new SaaS company almost always begins its market journey initially on one or more of the public clouds. According to the Gartner group, SaaS is the largest segment of the cloud market, with revenue expected to grow 17.8% to reach $85.1 billion in 2019.

Not just infrastructure, but Applications also have traditionally had a form of centralization in the form of being a monolithic, tightly bound development and deployment model. On the data side, data lakes are starting to create a centralized data store model (never mind that many companies have had huge relational databases anyway for decades!). And trust models have historically been tightly centralized with clearing houses and market exchanges holding the “truth of state” in one place, and everyone else relying upon them.

All that is starting to change, and how! On every single front — infrastructure, application development, and deployment, data, trust all of them — we are starting to see decentralized and distributed models of development, deployment, and trust.

In the subsequent posts, I intend to lay out the thesis that we are the advent of a new era of glorious distributed computing of many kinds, and therefore infrastructure has to be reimagined to meet the needs and challenges of such an era. In the next part of the series, we will explore how de-centralization via micro-services and external services is impacting application development and deployment.