The Data-Driven Delusion

Why Being Data-Driven Alone Won't Save Your Business

Imagine it's the early 2010s. You're an early employee of a high-flying startup (been there, ahem, NOT done that) with some influence over the company's leadership and how things should be run. The primordial soup that would birth the modern data stack is brewing. You're inadvertently fed a diet of "data-driven" success stories at social networks and marketplaces built by entrepreneurs barely a few years out of college.

You regularly receive effusive newsletters about digital transformation and vague analogies between data and fossil fuels. Your CEO, as CEOs should, is obsessing about survival and winning and is already pitching investors on how data would be the cornucopia that seals the deal for your fledgling company. There's a vision deck listing you as an author (a reluctant one, but unbeknownst to others) on how data will enable your company to crush the competition while creating joyful product experiences for users. And there is that insane Moneyball story of how data transformed baseball or was that story overblown?

So over the next months and years, you do what's important for the sake of preserving access to your keto snacks and to boost the value of your stock options. You hire a couple of talented full-stack data scientists who, like you, are gluttons for punishment. These data scientists work 16-hour days setting up data pipelines and reporting for executives, stand up metrics and pressure test assumptions, and you lay the foundations for experimentation and lay the groundwork for the use of modeling (ML in today's parlance) in pricing decisions.

Despite resource constraints, you manage to set up a "data-driven" culture: leaders routinely ask questions along the lines of "what does the data say?" Experiments are run to test assumptions, and you keep track of ROI from performance marketing dollars through dashboards. Your data team has grown, and you're commended for your efforts. You now sport stylish silver strands that add to your look.

Then things start turning south for your company—slowly at first, then suddenly. The cost of customer acquisition creeps up, and the company tries a soft pivot or two (data-driven ones, to their credit). But the dreaded writing is on the wall. Your investors and vulture capitalists all start questioning strategy and share the epiphany that you may not have product-market fit at all. 

Meanwhile, another fledgling company in the same neighborhood has eschewed investments in data but seems to be experiencing brisk growth (okay, they have some dashboards, but they also have a rockstar CMO who has cracked the code). As the sun sets on this wild adventure, you arrive at a reductive conclusion: "data-driven" does not guarantee an edge in business success.

The ROI of "Data-Driven": Beyond the Hype

The point of this semi-apocryphal (well, some of these were experienced firsthand, the rest are embellished for effect!) tale is to highlight the question that irks many a business leader and data leaders in particular. What is the ROI on these investments to become data-driven? What's the point of it all if it doesn't give the business an edge?

There are, of course, obligatory data investments that every business needs to make to ensure accurate financial reporting and observability. After all, a skilled Alaskan bush pilot still needs basic instrumentation to inform their altitude, speed, fuel levels, and whether or not the engine is on fire. Nor will anyone question the ROI on "data products" that are core components of a business's value proposition. Imagine a last-minute food delivery platform that doesn't provide an ETA for your order!

But what about the other seemingly fancy and expensive people, tooling, and process investments—your pesky insights experts (data scientists), your data lakes and warehouses, and BI tools, that lofty experimentation platform, your lifetime value models, and revenue/loss forecasts—the significant investments that need to be made in building out programs and systems for data-driven decision-making?

Having spent many a "sleepless" night ruminating about this question (and I believe I'm not the only one), I now have a mental framework to think about this question that I'd like to share.

Alpha, Beta, and the Pursuit of Systematic Advantage

Early in my career, before I embraced data as my vocation, I spent a few years in what's called quantitative or systematic investing. I was part of a motley crew of engineers, mathematicians, physicists, etc., who had invaded Wall Street in our quest to act like financial wizards with math hats. We built fancy number-crunching models to predict market moves, keep the risk goblins at bay, and wrote code so that trading would be as automatic as your morning coffee. The idea was to take the "I hope this works" of investing and turn it into "I've got the numbers to prove it!"

While we were at it, I learned some useful Greek alphabets—the first two in particular—alpha and beta—that are widely used and abused but nevertheless have had a role to play in thinking about "data-driven."

The terms "alpha" and "beta" have become part of folklore, especially after Google decided to rebrand itself as Alphabet (which some said was a "bet" on "Alpha" (?)). Anyway, "alpha" and "beta" are like the dynamic duo of portfolios. "Alpha" is the superhero of investing, showing how much extra return your portfolio makes compared to the market. If the market is the standard and you make more money, that's your alpha! As life would have it, most portfolios end up underperforming the market, and you would therefore report negative alpha and then slap on relevant commentary in the investor newsletter blaming it on China or some unpredictable Gen Z spending habits.

"Beta" on the other hand, is your portfolio's sidekick, measuring how much it swings with the market. A high beta means your portfolio dances more wildly with market moves, while a low beta means it's a bit more chill. Together, alpha shows if you're adding (or detracting) value, while beta tells you how much market risk you're taking on.

The astute reader is probably catching my drift—a business as a portfolio of bets is not very different. Let's say you are a grocery retailer—the winds in that market are moving to more online spending and customers opting for healthier choices, etc. Your grocery business moves, be it favorably or otherwise, with those big secular changes in the market. The impact of those market shifts will vary from grocer to grocer—so the "beta" may vary, but you can't ignore beta. On the other hand, imagine a grocer who's ahead of the curve with smart carts, omnichannel distribution, and loyalty programs that offer targeted discounts (maybe bankrolled through a profitable side hustle running cloud services!)—and this I would categorize as "alpha."

What Does "Data-Driven" Have to Do with Any of This?

The Wall Street quants will tell you that not all "alpha" is created equal, and that there is discretionary alpha (also termed "idiosyncratic" to the more cynical quants) and systematic alpha.

The success of a business most often comes down to adroit humans intuiting their way through the complex multi-dimensional space of business decisions. You could imagine the grandiose high-risk decisions to change product direction, or enter new markets, or it could be way more primal as in the case of Airbnb. Airbnb's founders—Chesky and Gebbia—noticed that the early Airbnb listings looked more like a haphazard college dorm room than chic travel digs. They didn't just sit around and hope for a data-driven solution to this quality of listings problem. Nope, they rolled up their sleeves and decided to be their own roving photographers, snapping high-quality photos of properties. The rest, of course, is history.

So if this discretionary alpha is what makes or breaks businesses and portfolios, systematic alpha is the seemingly smaller but steady advantage that businesses build through robust processes, sound management principles—yes, and most importantly in the context of this blog—data-driven decision-making.

These systematic advantages don't yield dramatic turns of fortune. But the true and quite sizable power of embracing and building these sources of systematic alpha is realized in the long term through the eighth wonder of the world—compounding. So while setting up a steady yield on your portfolio won't bend the curve when you're on the verge of bankruptcy, a steady compounding return can be a potent source of prosperity if leveraged early and consistently.

Illustration of the long-term hypothetical edge for a business from being data-driven

The Nuances of "Data-Driven": A Balancing Act

So can being “data-driven” drive business success? I would argue IT CAN  if a business is willing to put in and stick with the investments in tooling, data quality and access, culture, and expertise. This doesn’t mean that being data-driven can provide an escape hatch to the headwinds in your market (“beta”). Nor does it mean that it can be the cure for strategic missteps (“discretionary alpha”). Case in point are companies like Stitch Fix and Lyft that built some of the best data teams in the industry but yet continue to struggle. 

But much like your daily workout,  the significant dividends of being  data-driven accrue to businesses that are willing to put in the time and resources to build and maintain great data, smart tools, and skilled expertise. So if you are a data leader, I urge you to brace yourself for the grind - the wins will be small and hard-fought but keep the faith in the power of compounding

A word of caution here,  building dashboards that no one uses or writing a big check to Snowflake or Databricks each month does not make you data-driven. That would be the equivalent of having a shiny Peloton and the complete Alo Yoga collection - looks fancy but hardly drives any meaningful change. 

Yet another caveat. Not every business has the same opportunity to milk the systematic data-driven advantage. As I’ll probably explore in my next blog, it comes down to leverage (the information content in your data) and the number of times you get to use it - stay tuned!