You're Probably Not [Really] Doing Data-Driven Decision Making
Why Artificial Intelligence and Machine Learning Need Decision Intelligence
Many organizations claim to make data-driven decisions. But are they really? When I took a break last week for Thanksgiving, at least for the writing, I was still busy following my plan to shore up any gaps that I may have in my professional and educational experience when it comes to incorporating Artificial Intelligence into any company’s strategy.
The last time that I wrote, I wrote about JP Morgan’s journey to becoming one of the leaders of AI in the financial sector, though not the first. JP Morgan has been on a bit of a future-proofing journey for over 10 years, so adopting AI into their business applications and technology stack was a common-sense next step. As illustrated below: they started with monolithic, on-prem systems and broke those apart. They created the leadership position of a Chief Data Analytics Officer and stood up an AI/ML Center of Excellence. Finally, they are diligently training employees to ensure adoption and a realization of value.
As I continue on my journey towards AI Strategist, I am focusing on technology products, the ever-changing regulatory landscape as well as different AI applications that can and will be impactful to business and government entities in the coming years.
Enter Decision Intelligence: Beyond Basic AI/ML
I took a class by Dr. Lorien Pratt about Decision Intelligence (DI). If you are not already familiar with Dr. Pratt, allow me to give you a short introduction to this computer scientist. She is the inventor of machine learning transfer and decision intelligence. Machine learning transfer is definitely an important stepping stone from generic machine learning to the more sophisticated types of AI and deep learning algorithms that have gotten so many of us exciting these past few years. Dr. Pratt is a former professor at the Colorado School of Mines as well as the founder of Quantellia.
So, what is decision intelligence? Decision Intelligence is a discipline that helps organizations make better decisions by combining human expertise with data science, social science, and managerial science. It provides a framework for understanding how actions lead to outcomes through complex chains of causality.
When I first picked up her book, The Decision Intelligence Handbook, I became extremely interested in how it can be used to truly do data-driven decision making. Of course, DI is so much more than that.
One of the first things that I learned was that DI helps teams and siloed organizations align more effectively so that they can collaborate on complex decisions. With the growing complexity of the business environment, it is critical that teams speak the same language and understand how their decisions contribute and impact the overarching goals and objectives of the organization.
The larger the organization, the more difficult it is to do this.
Why Traditional AI/ML Falls Short
Some of the outstanding problems in ML today that remain include:
ML is single-link: it does not solve action-to-outcome problems
ML needs better orchestration
ML is missing a requirements discipline
Machine Learning Does Not Solve Action-to-Outcome Problems
Machine learning helps us to see patterns in data, but it does not necessarily tell us what to do. When we talk about data-driven decision making, most often we think of graphs and data that come out as part of PowerBI-sorts of graphs and charts. Sure, these charts will give you insights into your data, but it won’t necessarily tell you the best decision to make next…and how many external and other dependencies could impact the outcome of that decision.
One thing that I have seen in my conversations with business leaders and was validated during the class is that our diligence and structured processes goes down as people become more senior. And, sure, as we become more experienced, we have years of lessons learned and outcomes and certain choices seem to be second-nature. However, the decisions become increasingly complex due to the changing nature of the business environment.
This is particularly concerning because senior-level decisions often have the most complex ripple effects throughout an organization. While experience is invaluable, the increasing complexity of our business environment means that even seasoned executives need structured frameworks to fully understand the cascade of consequences their decisions might trigger.
For instance, ML might tell you that customer churn is likely to increase by 15% next quarter based on current patterns. But it won't tell you whether to address this through pricing changes, product improvements, or enhanced customer service - or how these different actions might interact and affect other business goals.
Machine Learning Needs Better Orchestration
As a bank, maybe you deploy some fraud-detection algorithms. The number of fraud-related incidents go down by about 14% from the previous years. That’s great. However, that is a one-outcome scenario.
Consider instead a strategic decision facing JPMorgan: Should we acquire a major fintech company for $500M? This requires orchestrating multiple ML models and data sources to evaluate:
Market impact analysis (competitive positioning, market share models)
Technology integration feasibility (legacy system compatibility, security risk assessment)
Customer behavior prediction (retention modeling, cross-sell opportunity analysis)
Regulatory compliance risk (automated policy analysis, jurisdiction-specific ML models)
Financial performance forecasting (synergy modeling, revenue/cost predictions)
Talent retention likelihood (employee satisfaction models, compensation analysis)
Each model provides critical insights, but the complexity comes in weighing conflicting indicators. For example, the technology integration model might flag high risks while the market impact analysis shows significant opportunities. Decision intelligence frameworks help leaders synthesize these competing signals into actionable insights.
The framework must also account for timing - moving too slowly could mean losing the opportunity to competitors, while moving too quickly without proper due diligence could lead to costly integration failures. This is where decision intelligence differs from simple automation by:
Quantifying uncertainty across multiple scenarios
Modeling cascade effects across business units
Incorporating expert judgment at key decision points
Providing clear audit trails for regulatory review
Adapting recommendations based on real-time market changes
The real challenge comes in modeling unintended consequences. Take JPMorgan's hypothetical fintech acquisition:
Primary Decision Models:
Market impact analysis shows 30% market share gain
Technology integration feasibility predicts 18-month timeline
Revenue forecasts project $750M annual increase
But what about the cascade effects?
Competitive Response: The acquisition could trigger aggressive moves by Goldman Sachs or Bank of America, potentially nullifying projected market share gains
Regulatory Scrutiny: The deal's size could prompt enhanced oversight of all JPMorgan's fintech activities
Culture Clash: Integration friction could lead to key talent exodus, with competitors poaching top performers
Security Vulnerabilities: Rapid integration pressure could create cybersecurity blind spots
Brand Impact: The fintech's younger customer base might view the acquisition negatively, leading to customer churn
This is where decision intelligence provides real business value - by modeling these interconnected ripple effects and their probabilities. The framework must capture how one decision triggers multiple consequences, each spawning their own effects.
Machine Learning is Missing Requirements Discipline
Often, data scientists and AI/ML practitioners are given a set of data and a leader may request something along the lines of: “what does this data showing us? Are there any patterns that we should be aware of?”
The team can develop a model and do some predictive analytics. The model can help you see a short way into the future, but it will not tell you what to do. It will not help you understand other causal chains that may interfere with the data you are seeing today, the decisions you make tomorrow and the outcomes your company is striving for.
This "exploratory" approach contrasts sharply with traditional software engineering, where requirements are rigorously defined upfront. Without clear requirements, ML projects risk:
Solving the wrong problem
Missing critical business constraints
Overlooking regulatory compliance needs
Creating models that don't integrate with existing systems
Producing insights that aren't actionable
Decision intelligence frameworks bridge this gap by starting with the decision requirements first, then determining which models and data are needed to support those decisions. This ensures ML efforts align with business objectives rather than just finding interesting patterns.
So Many Outcomes, Many Undesirable
As organizations face increasingly complex decisions in an interconnected world, the ability to model and understand these intricate cause-and-effect relationships becomes critical. Decision Intelligence provides the framework needed to augment human judgment with data-driven insights while accounting for the full scope of potential consequences - both intended and unintended.
Here is an example Decision Model that I created to figure out how a company may choose to hire a marketing consultant to help drive growth and sales. Note, this shows the levers and actions that someone who owns the decision and the budget has control over. The externals represent things that are outside of the decision-maker’s control. It is a simplified example, but it provides a template for where you could begin to identify data and models to measure the likelihood of the different outcomes.
The more I look at it, the more I want to make adjustments (especially to the intermediates and how they are measured), but this provides an overview.
Are you interested in learning more about Decision Intelligence? I recommend you check out Dr. Lorien’s Website. It is chock-full of information and one that is on my list to follow. I’ll also share in my notes if I hear of an upcoming live class. This will not be the last time I discuss DI - it has a lot of promise and there is still a lot of info that I am pouring through as I review and reflect on the class.
What are your thoughts on Decision Intelligence and data-driven decision making? What does it mean to you? Let me know! I’d love to hear from others.
How to get rid of procrastination.
That's the question.
Can you recommend any book?