For the past 100 years of "modern business", we have been capturing what happens in our organizations. The main goal has been to support the operational side of the business; the day-to-day operations of providing products and services. The focus for many organizations has been on keeping an accurate system of record, a representation of the state of our organization.
While the idea of gathering intelligence out of business operations dates back well into the 18th century, the introduction of computers really kicked off the analytical revolution. The sheer processing power at our disposal allowed us to start collecting and interpreting business events in order to deduce insights from them.
The two-pillars of the status quo
The tools we use have changed over the years, but the general idea has been that a system of record - the transactional side of the business - feeds the analytical side of the business; both sides being clearly separated from each other.
Expensive jobs would run during the night with the sole purpose of shipping data from one side to the other.
But with analytical data about customers becoming ever more connected to how businesses operate, there came a need for that information to flow back from the analytical side into the transactional one. Many organizations figured out trivial solutions in shipping insights back from the analytical side to the transactional one just as they did the other way around. But the overall delay introduced by this game of back-and-forth has finally become too much for modern businesses.
We just can’t wait a day, or even an hour for that matter, for the insights we need to steer our organizations
For example, even with the introduction of machine learning and AI, most companies are only leveraging historical data to train their models—effectively "offline" data. And then copying the result over to the operational side to be used for inference.
While piggybacking on previous solutions certainly has its merits, it also amplifies the weaknesses of previous iterations.
Nightly jobs only work if your company actually has real "non-business" hours. With many companies working at a global scale, this isn’t the default anymore and insights, models, and analytics are increasingly required to be continuous. Even for the organizations which can structure their jobs around a clear distinction between online and offline hours, the amount of offline time available just isn’t enough to deal with the workload at hand.
The speed of business and the expectations of stakeholders are only going up
Sometimes it is best to take a step back and think through what we actually want to accomplish given the changing requirements we are faced with. That way, we can figure out what the end goal should look like and start drawing a map to get there.
We can’t draw a complete map for you, since it is highly dependent on how your organization functions and relies on data today. What we can do however, is provide a glance into what end goal we think you should be aiming for.
This series will explore a fundamental shift into how we build organizations to use data (and vice-versa). Spoiler alert: it has more to do with mindset than tools. Obviously we think NATS is a strong candidate to be the foundation for this new kind of organization and we are pretty sure we can convince you about that.
However, this journey is meant to be an informative one for anyone out there who feels there is something off with the current state of affairs. And it is meant to be an interactive one; yes we will publish articles but at the same time we would love to hear from you and what you think.
I hope you enjoyed this introduction as much as I enjoyed writing it. But it doesn't stop there. Hop into the NATS Community Slack channel for this series and share your thoughts. I would love to hear from you.
Up next: getting into the weeds on systems of record. Coming Friday.
Until next time!
About the author
Daan has been both a Chief Data Officer and a startup founder, and now works on all things data at Synadia. He's been at the forefront of the data landscape since the early days of the data (r)evolution designing and implementing data-driven architectures for startups, telcos, airports, and banks. While the early focus was on the usual big data suspects, the focus has shifted in the last decade toward data streaming and event-driven systems.
