In September, I had the great opportunity to attend the Big Data Ignite 2017 conference in Grand Rapids, Michigan. The three day conference carried a theme of “The Age of Exponential Intelligence” and offered a chance for me to take a step back and get a broader view of the big data and analytics marketplace. I used the time to connect and learn from others who are down in the trenches, making these big data platforms work at a handful of large enterprises here in West Michigan. It was invaluable to talk to people that understand the various platform vendors out there and learn about the tool choices driving their IoT, data, and data lake infrastructure.

Over the three days, a few themes solidified into some interesting trends:

Enterprises are running after Big Data and Analytics aggressively, and, not just from within IT
While at the conference, I observed presentations and participated in numerous hallway conversations with team members from Meijer, Steelcase, Amway and many others. These multi-billion dollar enterprises are working hard to find the right talent to bring onboard and then giving them the freedom to experiment. The hope is that the freedom will lead to a successful big data program to sit at the center of core business operations. As a result, these teams are driving change end-to-end within their organizations as they uncover what data is available, how best to centralize it and ultimately opening it up for others to derive value from. These “data lakes” and “enterprise data marts” are collecting everything from system logs to user-generated social content and IoT connected asset data streams from manufacturing operations.

At the heart of digital transformation is data and analytics
We hear a lot about digital transformation in the press today. As with many significant trends, the phrase has become overused and misunderstood. It was clear at Big Data Ignite that the heart of any digital transformation effort is a core focus on data and analytics. It’s not about just building a digital product or opening a new digital channel. Digital transformation is about starting at the core foundation of a company, and shifting the focus to collecting and utilizing data to make better, faster, more informed decisions. The progressive companies we talk with are looking at how to make data accessible to anyone in the organization. They’re aggressively shifting from a command-and-control IT organization, to one of enablement. Meijer has even gone to the extent of establishing development “sandboxes” where anyone within the company, who has attended their “data bootcamp”, is permitted to work with production level data to evaluate hypotheses or solve a line-of-business problem.

The tooling market is complicated and fraught with duct tape
As I sat through various vendor presentations and hallway conversations, I started to see the broader landscape and gain a better understanding for the massive amount of suppliers flooding into the space. I was blown away by a number of bolt-ons you could tack onto your Hadoop cluster or how many ETL’like tools available to flow data into your data lake. Things are simplified if you choose to live within the Microsoft Azure or Amazon AWS ecosystems, but as you stray out of those environments, the verity and options are endless. In the grand scheme, this space is still evolving, and there will be winners and losers. We will soon enter into a period of vendor consolidation that will help to simplify the options, but for now, the space remains very thick with new tools and piecemeal architecture.

Everyone is still trying to figure this out
We are still in the early times of large enterprises adopting and implementing big data infrastructure and enterprise wide analytic platforms. We’ve moved past the point of proving the technology can scale and is viable and now have transitioned into how these data sources can add exponential value to the bottom line across the entire enterprise, not just siloed Business Intelligence groups. As this transition continues to evolve, we’re starting to see more questions arise around best practices. Where should data scientists “live” within the organization? What data is providing value and what value is just excessive? And, ultimately how do you transform and optimize existing products and services to take advantage of these newfound insights?

The reach of big data and analytics is helping break down the silo’ed walls of traditional IT and help drive more value to the bottom line. At Collective Idea we understand and believe that not to far in the future, all companies of scale will need to become data driven. Their volume, velocity, and variety of their data will become a core part of their competitive advantage. How are you getting started today?

What do you think about the pace of adoption of big data within your company and how do you deal with the variety of choices out there? We’d love to hear from you — tweet us @macfowler and @collectiveidea.