I tried automating and integrating AI
into almost all workstreams relevant to me.
Here’s what I found.
Six findings on where AI actually changes the work, and where human judgment becomes more valuable, not less.
There is a particular anxiety running through analytics and commercial teams right now. We are operating inside a function that looks like it could be deeply transformed by AI, while still struggling to fully grasp how AI-integrated workflows will impact commercial analytics, especially as we continue to navigate regulatory and compliance constraints.
So I stopped waiting and ran an experiment using synthetic data that mimics real-world patterns. I built agentic workflows on top of it, stress-tested them against experienced analysts and junior staff, observed where AI broke down and where humans overestimated their own edge, and drew conclusions to help reduce the uncertainty around how the future of work in commercial analytics is likely to evolve.
Key finding
AI can answer the question. The problem is that without the right context, it answers the wrong question, because nobody taught it how your business actually defines its own data.
The argument
The integration most commercial teams have built is a dashboard with pre built views. The dashboard answers the questions you thought of when you built it. It cannot handle the question you ask at 11 PM when the client pushes back on a number.
What changes that is a connector agent with a business language translation layer underneath it: a maintained document that maps natural language business questions to precise query logic. Not just column definitions. The domain specific shorthand: how a market segment is defined in the source data versus how the commercial team refers to it in conversation. What a lapsed customer means in your CRM versus what marketing means when they say it.
When the agent carries that context, it stops producing outputs that are technically correct and commercially misleading. When it does not carry that context, it produces exactly those outputs, confidently and with no flag.
The structural shift is bigger than the time savings. When natural language access is available to any stakeholder, the analytical function is no longer the bottleneck. It becomes the function responsible for framing better questions, a different job with a different skill profile, and most teams are not deliberately building toward it.
In practice
The analytical function is no longer the bottleneck between a business question and a number. It becomes the function responsible for asking better questions.
How it works
Before: analyst as bottleneck
After: business language mapped into data language
Analyst time allocation: before and after
Indicative reallocation based on analyst interviews post deployment.
Technical detail
Key finding
Getting an AI output is not the hard part. Knowing which of the ten embedded judgment calls in that output were made correctly, and being in the loop at each one rather than just reviewing the final result. That is the hard part.
The argument
The common mistake is treating multistep commercial workstreams as prompt engineering problems. They are process formalization problems that happen to be executed through an AI interface. Demand forecasting, territory sizing, pricing scenario modeling: these workstreams have a dozen judgment calls embedded in them, each calibrated over years of domain experience.
The architecture that works has three layers. A template: the fixed skeleton, covering calculation logic, data requirements, output format. A decision node map: every judgment call written down explicitly, each with a framework for making it and the standard it is held against. And mandatory human checkpoints at each node, not a single sign off at the end.
A 2023 JMIS study found that reviewing complete AI outputs end to end was associated with higher dissatisfaction than either manual work or structured incremental collaboration. Correcting a finished analysis built on a wrong assumption is expensive and demoralizing. Approving or adjusting at a checkpoint takes thirty seconds.
The second order benefit: decision frameworks are not workflow specific. The logic for handling thin comparator market data appears in forecasting, territory sizing, and pricing scenario work. Once formalized, each framework composes into any subsequent workflow that hits the same decision type. Each new project gets cheaper and more consistent than the last.
In practice
The real long term asset is not a library of workflows. It is a library of decision frameworks that assemble into workflows.
How it works
The map: what runs on its own, and where someone decides
How to read this
A repeatable workflow is not a straight line. It is a set of connected blocks, dozens of small variables, and a handful of real decisions. The craft is knowing which decisions change the path — and which just tune the numbers.
The four kinds of decisions
in plain terms
Which path?
Direction decisions. Choose differently here and everything after it changes.
What matters here?
Scope decisions. Pick which local factors — segments, channels, constraints — belong in the work.
How much?
Tuning decisions. Set the thresholds, ranges, and cutoffs. The structure stays the same.
Too close to call
Judgment calls. The data doesn't decide; an experienced person does.
Every decision answers five questions
When it's unclear
When the data is clear, sign-off takes seconds. When it isn't, slow down: show the trade-off, hand it to a person, and write down why they chose what they chose.
The three layer implementation architecture
The fixed skeleton: calculation logic, data requirements, output format. This is the part that does not change across markets or business cycles.
Every judgment call written down explicitly, with the framework for making it AND the standard it is held against. The second element is almost always missing in practice.
Mandatory approval at each decision node, not a single sign off at the end. Reviewing at a checkpoint is fast. Auditing a finished output built on a wrong assumption is not.
A checkpoint in action: what the human actually approves
Note: output is sensitive to this assumption. A 4 point shift changes projected revenue by ~$2.3M.
Why decision frameworks compound: an indexed library
Once a decision framework is written down and indexed, it gets reused in every future workstream that hits the same decision type. Each new project gets cheaper and more consistent than the last.
Technical detail
Key finding
When a senior analyst says something feels off about a number, that reaction is worth investigating. It is pattern recognition built over a decade, not vague discomfort. Build workflows that capture it.
The argument
The instinct in most AI augmented workflows is to treat human intuition as a corrective mechanism, the one you invoke when data runs out. This is too narrow.
In 1966, Michael Polanyi articulated something organizational theory has spent decades formalizing: "we can know more than we can tell." Human expertise is built around a layer of tacit knowledge, pattern recognition developed through sustained exposure to outcomes, a form of knowledge that resists full articulation. It operates below the level of explicit reasoning.
Gary Klein's research on naturalistic decision making found that experienced practitioners rarely compare options. They recognize a situation as similar to past situations, generate a course of action, mentally simulate its outcome, and act. For practitioners in domains with consistent underlying patterns, this process is faster and often more accurate than deliberate analysis, because it draws on a pattern library no explicit model has yet captured.
The Kahneman and Klein paper (American Psychologist, 2009) drew the boundary condition clearly: expert intuition is reliable when the environment has valid regularities and the practitioner has had sustained exposure to outcomes. Commercial analytics meets that condition.
Designing workflows to elicit that signal, building in moments where practitioners articulate what they are noticing, not just approve or reject, is the quality mechanism most likely to surface tacit knowledge in a form that eventually improves the model itself.
In practice
“We can know more than we can tell.” — Michael Polanyi, 1966
How it works
Recognition Primed Decision
Deliberate Analysis
When is expert intuition reliable? (Kahneman and Klein, 2009)
The domain has consistent underlying patterns. The practitioner has seen enough outcomes to have a calibrated experience library. Commercial analytics qualifies.
In novel markets or first-exposure outcomes, human judgment becomes more important. Humans adapt faster, develop expertise from new feedback loops more quickly than AI, and can reason through context the model has not seen.
Technical detail
Key finding
Give junior analysts AI before they have learned to spot a bad output, and you get confident, wrong answers produced at scale. The tool is not the problem. The order of introduction is.
The argument
The problem is not that AI is present. The problem is that AI removes the friction that creates learning.
Judgment in analytics does not develop through training. It develops through repeated exposure to wrong answers and the process of diagnosing why they were wrong. A junior analyst who gets an implausible output has to reason about what went wrong: check the population logic, interrogate the assumptions, trace the error back to its origin. That reasoning process is the skill. If the agent catches the error first, or the analyst accepts the output without interrogating it, the reasoning never happens.
This is identical in structure to how numerical intuition develops, or fails to. Children who develop arithmetic sense before using calculators use calculators as accelerants. Children who reach for a calculator before they have number sense cannot catch the calculator's errors. The sequence determines the capability.
In practice: AI access to well specified query and retrieval tasks before it is extended to judgment dependent workflows. Deliberate exposure to cases where AI is wrong and the analyst must diagnose why. Review of AI outputs treated as a reasoning exercise, not a formatting task. This is not a training philosophy; it is a risk management requirement for any organization expecting to maintain output quality as the ratio of AI to human work shifts upward.
In practice
The sequence is the intervention.
How it works
Wrong sequence
Right sequence
The calculator analogy
Technical detail
Key finding
The firms chasing efficiency through AI are running the same race slightly faster. The firms asking which previously impossible problems are now viable are running a different race entirely.
The argument
The efficiency framing rests on two assumptions that frequently fail. First, that AI costs are negligible. Second, that being faster than competitors creates durable separation. Both assumptions deserve scrutiny before being embedded in a strategy.
On costs: running AI at scale across real workstreams, with the data infrastructure, model management, and quality control it requires, is more expensive than the productivity narrative suggests. Companies that have deployed widely have found that compute and infrastructure costs can erode the efficiency gains substantially, sometimes entirely, especially in the first two years before workflows are well optimized. The bet that AI makes you cheaper is more fragile than it looks.
On competitive advantage: even where efficiency gains do materialize and hold, they commoditize. If every firm on the same technology stack captures the same productivity improvement, the competitive landscape shifts but no individual firm gains separation. You are all faster. You are still running the same race.
The asymmetric opportunity is elsewhere. Every previous analytical technology wave, spreadsheets, database software, BI platforms, expanded the frontier of what was commercially viable to attempt, not just what was cheaper to execute. The firms that captured value in each wave were not the most efficient producers of the old work. They were the ones that identified the new problem categories the technology unlocked.
AI does the same thing. There is a backlog of commercially valuable problems that were never started because the human time required exceeded the value they would generate. Those problems went into a drawer. Some of them are now worth pulling out.
In practice
Go back to the drawer. Find the problems that were commercially dead. Recalculate. Some of them are now alive. That is a different question than asking how much faster you can run the ones you were already doing.
How it works
Each technology wave expanded which problems were economically viable
Problems back out of the drawer: recalculate the economics
Illustrative estimatesThe directional logic is real; the time estimates are order-of-magnitude guesses, not benchmarks. The structural shifts (unviable to viable, post-meeting to live) are the more honest claims. Suggested method: take the last 3 years of declined projects, rerun the economics at lower delivery costs, and see what changes.
Technical detail
Key finding
The work being automated is the work of producing the answer. What is left is harder: deciding what question to ask, structuring the logic, and knowing when the output is wrong.
The argument
The question used to be: can you produce an accurate output efficiently?
The question now is: can you design the decision frameworks the AI runs through, surface what experienced people know but cannot easily articulate, develop the judgment of people below you, and find the problems the business did not know it could afford to solve?
The first question was always going to be automated. The second one is not.
The analysts who will fare well are not those who resist AI; they are those who use the efficiency gains to build capabilities AI cannot replicate: judgment, architecture, knowledge elicitation, problem discovery. The window to build those capabilities while they are still in demand is not infinite.
Organizations that restructure roles without creating conditions for that practice, giving analysts AI tools but keeping them in execution heavy workflows, will not build the capability they need. The diagnostic question is what percentage of analyst time goes toward designing the frameworks and logic the AI uses, versus running outputs the AI could run autonomously. If the answer is less than 30%, the transition is not happening.
In practice
The first question was always going to be automated. The second one is not.
How it works
The old question
"Can you produce an accurate output efficiently?"
The new question
"Can you build decision frameworks, surface what experienced people know implicitly, develop judgment in the people below you, and find problems the business did not know it could afford to solve?"
Where skills are moving
Technical detail
Conclusion
The analyst role is not ending.
It is being restructured
around a harder question.
These findings come from running the experiment and not theorizing about it, from building the workflows, watching where they failed, and drawing conclusions from what actually happened.
About this experiment
The workstreams described here are drawn from commercial analytics in healthcare, a domain where the gap between a technically correct output and a commercially useful one is wide, assumptions are consequential, and market contexts are specialized enough to stress test these principles properly. That specificity is what made the experiment useful. The patterns generalize. The examples came from there.