Can AI Put an End to Big Data’s Boulevard of Broken Dreams?

Way back in 1914, engineer and data visualization pioneer Willard Brinton observed that while companies went to considerable effort to analyze and catalog their information, most failed to draw useful conclusions from it. “Known facts cannot be marshaled and presented in such [a] manner as to be effective,” he wrote more than 100 years ago. Yes, that’s how long companies have struggled to make sense of and extract value from what’s now called big data — datasets so large they can only be analyzed via computer.

Today, advancements in artificial intelligence (AI) and AI-era data platforms may be the key that finally unlocks the value of big data — not by, yet again, reinventing the way data is processed and stored, but by reimagining the way it’s accessed and disseminated throughout an organization. 

For example, the unique approach of Data Cloud not only connects all your big data, regardless of source, but makes it accessible and actionable to all business users in the apps they use every day. This relevant, trusted data is automatically sent to an AI model, giving it the context it needs to create high-quality outputs. 

All your data is, at long last, unified, harmonized, and accessible to every business app and user. It’s a far cry from where we’ve been.

How did we get here?

In the 1990s, when the term “big data” was first popularized, the hope was that organizations could analyze this data to uncover valuable customer and market insights, deliver hyperpersonalized experiences, forecast demand more accurately, and enhance product development, among other things.

Big data grew so fast and became so unwieldy that an entire industry emerged to help companies parse and analyze it. Enterprise customers continue to spend mightily on big data solutions like data lakes and data warehouses. But major challenges remain.

“Owning lots of data doesn’t solve the big data problem,” said Andy Cotgreave, senior data evangelist at Salesforce. “Throughout human existence, we’ve used technology to create datasets, which have given us ideas beyond the limit of what the data and technology can do.” 

Part of the problem is that data lakes and data warehouses are designed for technical users with some knowledge of SQL, a computer language for manipulating databases, and semistructured data like emails and web pages. For this reason, salespeople can’t use them to move deals forward. Service representatives can’t use them to resolve customer issues. And retailers can’t use them to create personalized product promotions. 

For all intents and purposes, these massive repositories only trapped a company’s data inside increasingly sophisticated silos. 

“Let’s face it. Our customers have done a lot of work to pull what they thought was trapped data and put it into these really excellent data platforms. Except it was still trapped,” said Steve Fisher, executive vice president and general manager, next-generation CRM and unified data services at Salesforce. 

The big data solution

Data platforms like Data Cloud, coupled with AI copilots like Einstein Copilot, solve this problem. They connect all your disconnected internal and external data, giving siloed areas of your business a complete view of customer activity and data. And they go much further. These freshly minted AI-powered data platforms connect information from your data lake or data warehouse and connect it directly to grounded AI, informing daily workflows within your business apps. 

As Fisher explained in this post, “Data Cloud provides the data and grounding for Einstein Copilot, our new generative-AI conversational assistant, as well as Einstein Copilot Builder, a new way to build and tailor AI assistants, to deliver a new generation of AI-powered apps. With Data Cloud, we can create a data graph that provides a real-time, consolidated view of a customer or any entity. And with just one click, customers can send all the relevant data to the prompt that then feeds the large language model (LLM). You don’t need to send SQL queries or create data joins manually.” 

These capabilities are available to the entire organization, not just the data experts, giving everyone the power to make better decisions and communicate with customers in more meaningful ways. 

In real-world terms, this means a salesperson can now get insights from a prospect’s interaction with marketing campaigns and social media activity to identify the hottest leads. It means that a marketing pro can quickly integrate a wealth of data, including information from marketing, commerce, sales, and service teams, to build more targeted audience segments. Then they can seamlessly activate them across channels. 

The role of AI and AI copilots

AI copilots can also help you get more out of big data. They’re simple to use because they’re built into the business apps you use every day. You ask a copilot to perform a task — say, “find the three best sales leads for the week, and tell me why they’re the best” — and it does. How? Behind the scenes, the bot uses LLMs to parse immense amounts of data on your behalf. 

This is what’s possible when your AI copilot has access to all the data in your organization. While many companies have different types of data from different sources siloed in different departments, those that can unite this data in one organized system will reap the full benefits of AI copilots — and their big data. 

The split-second answers you get from your copilot are grounded in business data from across your organization, not just sales, marketing, or customer service systems. Tasks built for copilots have access to all this business data, plus data that couldn’t previously be analyzed, like PDFs, web pages, and emails. 

Big data and AI – Like peanut butter and jelly

Many factors have contributed to the incredible growth in data over the past decade, including cheaper storage, the proliferation of online services, and the internet of things. The amount of data created, captured, copied, and consumed will reach 147 zettabytes (that’s a one followed by 21 zeros) this year — almost double 2021 levels. 

This data is a treasure trove of information and insights, provided you have the tools to easily access, process, and integrate it into your day-to-day work. Finally, with AI and data platforms built for the AI era, you do.

As David Schmaier, Salesforce president and chief product officer, told MarketWatch recently, “You don’t have to be a data scientist to harvest all your data across the enterprise.”