Are We Thinking, or Just Agreeing?
The Rise of Cognitive Surrender in the Age of AI
New research from Wharton suggests that as artificial intelligence becomes embedded in everyday decision-making, human judgment may be quietly stepping aside — with profound consequences for business and accountability
When Goldman Sachs analysts began using AI tools to draft client memos, or when a hospital administrator in Singapore started routing triage decisions through an algorithmic system, something subtle but consequential happened: humans stopped being the primary authors of their own conclusions. They became, in a very real sense, editors — and not always critical ones.
A landmark paper from researchers at the Wharton School of the University of Pennsylvania is now putting a name and a rigorous empirical framework to this phenomenon. Titled Thinking — Fast, Slow, and Artificial, the study by Steven D. Shaw and Gideon Nave introduces “Tri-System Theory,” a sweeping update to the foundational cognitive psychology model popularized by Nobel laureate Daniel Kahneman. The research doesn’t just describe how people think alongside AI. It reveals, with striking clarity, how often they simply stop thinking at all.
Beyond Fast and Slow: Introducing System 3
For decades, Kahneman’s dual-process model has been the dominant framework for understanding human judgment. System 1, as it is known, is the fast, intuitive, gut-feel mode of thinking — the one that tells you the answer to 2+2 without effort. System 2 is slower, more deliberate, the kind of reasoning deployed when filling out a tax return or weighing competing job offers.
Shaw and Nave argue that this model, influential as it remains, has a fundamental blind spot: it assumes all cognition happens inside a human brain. That assumption, they write, is becoming “increasingly misaligned with modern decision environments.”
Enter System 3 — artificial cognition, defined as external, automated, data-driven reasoning performed by AI systems such as large language models.
System 3, in their framework, is not merely a tool like a calculator or a search engine. It is a functional cognitive agent — one that can supplement human reasoning, but also one that can quietly replace it. The researchers call the endpoint of this replacement “cognitive surrender”: the tendency for people to adopt AI-generated outputs without critical scrutiny, effectively outsourcing their judgment to a machine and then accepting its conclusions as their own.
This is distinct, they are careful to note, from using AI strategically to assist one’s thinking — a process they call “cognitive offloading,” which remains an active and deliberate form of engagement. Cognitive surrender is something more passive and more troubling: reasoning that has been vacated rather than delegated.
What the Experiments Showed
To test their theory, Shaw and Nave conducted three preregistered experiments with 1,372 participants, totaling nearly 10,000 individual reasoning trials. Participants were asked to solve problems from the Cognitive Reflection Test — a well-validated set of questions designed to distinguish intuitive, snap-judgment answers from those requiring careful deliberation. (A classic example: a bat and a ball cost $1.10 in total; the bat costs $1 more than the ball; how much does the ball cost? The intuitive but wrong answer is 10 cents; the correct answer is 5 cents.)
Some participants worked alone. Others had optional access to an AI chatbot embedded in their testing interface. Crucially, the researchers secretly manipulated the AI’s accuracy: on some questions, it gave the correct deliberative answer; on others, it confidently delivered the intuitive wrong answer.
The results were stark. Participants consulted the AI on more than half of all available trials. When they did, they followed its advice roughly 93% of the time when it was correct — and 80% of the time even when it was wrong. Accuracy soared 25 percentage points above the no-AI baseline when the AI was right, and plummeted 15 points below baseline when the AI was wrong. The researchers describe this asymmetry — the “behavioral signature of cognitive surrender” — as having a large effect size by the standards of psychology research.
Perhaps most disquieting was the confidence data. Despite the fact that approximately half of all AI outputs in the study were deliberately incorrect, participants who used the AI reported being significantly more confident in their answers than those working without it. Confidence did not meaningfully decline as faulty AI responses accumulated. People felt more certain precisely because a machine had told them they were right — even when they weren’t.
Who Surrenders, and When?
The research also mapped the individual characteristics that predict susceptibility to cognitive surrender. Those with higher trust in AI were more likely to consult it, more likely to follow its recommendations, and significantly less accurate on trials where it gave wrong answers. In contrast, participants who scored higher on “need for cognition” — a measure of how much someone enjoys effortful thinking — and those with higher fluid intelligence were better able to resist faulty AI recommendations and override them.
In practical terms: the people most likely to surrender cognitive control to AI are those who trust it most and reflect least. In the language of organizations, these are not necessarily bad employees or poor decision-makers in other respects; they are simply operating in a cognitive environment that has subtly changed the default setting from “think, then decide” to “consult the machine, then agree.”
The studies also tested whether situational factors could dislodge this pattern. Predictably, time pressure — a 30-second countdown per question — did not eliminate cognitive surrender; it amplified it, further suppressing the deliberative checking that might otherwise catch AI errors. Performance incentives combined with real-time feedback did help, improving accuracy and increasing the rate at which participants overrode wrong AI recommendations. But even then, the gap between following correct and incorrect AI advice remained large. The surrender effect proved stubborn.
The Business Stakes
For the global business community, the implications of this research extend well beyond the laboratory.
Across industries, AI has moved from experimental curiosity to operational infrastructure. McKinsey’s 2024 global AI survey estimated that nearly three-quarters of organizations had adopted AI in at least one business function. In financial services, algorithms now participate in credit decisions, fraud detection, and portfolio management. In healthcare, diagnostic AI tools are used in radiology, pathology, and emergency triage. In law, large language models draft contracts and review documents. In each of these domains, a human is nominally “in the loop” — but Shaw and Nave’s research raises a pointed question about what that really means.
Consider the legal profession. In 2023, two New York attorneys were sanctioned after submitting court briefs containing AI-generated case citations that turned out to be entirely fictional — cases that did not exist. The lawyers had used ChatGPT to research precedent and accepted its output without independent verification. This was not a failure of intent; it was, in the language of the new research, a textbook instance of cognitive surrender. The AI was consulted, it delivered confident outputs, and human deliberation was short-circuited.
Or consider the medical context. A 2025 study cited by Shaw and Nave found that physicians who used AI-assisted colonoscopy tools showed signs of “deskilling” over time — their unaided diagnostic performance declined after repeated deference to algorithmic recommendations. This is one of the more alarming downstream implications of the theory: that cognitive surrender today may erode cognitive capacity tomorrow.
In finance, the risks are different in kind but perhaps equal in consequence. Quantitative funds have long grappled with model risk — the danger that a flawed model is trusted too completely, too uncritically, by the humans nominally overseeing it. The 2010 Flash Crash, in which automated trading systems played a factor in the brief erasure of nearly a trillion dollars of market value, is a reminder of what happens when algorithmic confidence outpaces human oversight. Tri-System Theory gives us the cognitive vocabulary to describe how such failures propagate: not through malice or incompetence, but through the natural human tendency to defer to a system that presents itself with fluency and authority.
The Design and Policy Challenge
Shaw and Nave are careful not to frame their findings as an indictment of AI adoption. Cognitive surrender, they note, is not inherently irrational. In many domains, deferring to a statistically superior system is the right move. A radiologist who follows AI guidance on a scan the model has seen a million analogues of may achieve better outcomes than one who relies solely on intuition. The problem is not AI use; it is uncalibrated AI use — the inability to distinguish contexts where System 3 deserves trust from those where it requires scrutiny.
This is, at its core, a design challenge. The researchers argue that AI interfaces should be built to trigger deliberation rather than suppress it.
Confidence scores, uncertainty indicators, explicit flags for low-reliability outputs — these are not cosmetic features but cognitive guardrails. An AI that tells a doctor “I am 94% confident this is benign” invites a different kind of engagement than one that simply outputs “benign.” Similarly, the study’s evidence that incentives and feedback can partially restore deliberative engagement suggests that performance management systems and accountability structures in organizations have a role to play — not in restricting AI use, but in shaping how it is used.
For policymakers, the research joins a growing body of evidence calling for AI literacy as a core competency, not a niche skill. The European Union’s AI Act, which entered into force in 2024, mandates human oversight requirements for high-risk AI applications. But legislation alone cannot substitute for the cognitive habits and institutional cultures that determine whether human oversight is genuine or ceremonial.
What We Lose When We Stop Thinking
There is a deeper question lurking beneath the empirical findings — one that the authors gesture toward in their conclusion:
What happens to human agency, accountability, and even identity when our judgments are systematically shaped by minds not our own?
The study found that people exposed to AI outputs were more confident in their answers, regardless of whether those answers were correct. This inflated confidence is not merely a statistical curiosity. In a business context, it means that leaders who defer to AI recommendations may be simultaneously less accurate and less open to challenge. Confidence, after all, is partly what earns people the authority to make and defend decisions. If that confidence is borrowed from a machine rather than earned through genuine deliberation, the accountability structures built around it become hollow.
Shaw and Nave call Tri-System Theory “a theory for an age of human-AI algorithmic cognition.” What they have built, in practice, is a diagnostic tool — a framework for asking, in any given decision, whether what looks like judgment is actually something more passive. We do not merely use AI, they write. We think with it. And increasingly, it thinks for us, while we sit back, more confident than ever, and agree.
The question facing businesses, regulators, and individuals alike is not whether to engage System 3. That ship has sailed. The question is whether, in doing so, we are doing so as active partners or as willing passengers — and whether we will know the difference when it matters most.
The research paper “Thinking — Fast, Slow, and Artificial: How AI is Reshaping Human Reasoning and the Rise of Cognitive Surrender” by Steven D. Shaw and Gideon Nave is available as a preprint via SSRN.
References
Budzyń, K., Romańczyk, M., Kitala, D., et al. (2025). Endoscopist deskilling risk after exposure to artificial intelligence in colonoscopy: A multicentre, observational study. The Lancet Gastroenterology & Hepatology, 10(10), 896–903. https://doi.org/10.1016/S2468-1253(25)00133-5
Clark, A., & Chalmers, D. (1998). The extended mind. Analysis, 58(1), 7–19. https://doi.org/10.1093/analys/58.1.7
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015). Algorithm aversion: People erroneously avoid algorithms after seeing them err. Journal of Experimental Psychology: General, 144(1), 114–126. https://doi.org/10.1037/xge0000033
Evans, J. St. B. T., & Stanovich, K. E. (2013). Dual-process theories of higher cognition: Advancing the debate. Perspectives on Psychological Science, 8(3), 223–241. https://doi.org/10.1177/1745691612460685
Frederick, S. (2005). Cognitive reflection and decision making. Journal of Economic Perspectives, 19(4), 25–42. https://doi.org/10.1257/089533005775196732
Hudon, A., & Stip, E. (2025). Delusional experiences emerging from AI chatbot interactions or “AI Psychosis”. JMIR Mental Health, 12, e85799. https://doi.org/10.2196/85799
Kahneman, D. (2011). Thinking, fast and slow. Farrar, Straus and Giroux.
Logg, J. M., Minson, J. A., & Moore, D. A. (2019). Algorithm appreciation: People prefer algorithmic to human judgment. Organizational Behavior and Human Decision Processes, 151, 90–103. https://doi.org/10.1016/j.obhdp.2018.12.005
Mosier, K. L., & Skitka, L. J. (1996). Human decision makers and automated decision aids: Made for each other? In R. Parasuraman & M. Mouloua (Eds.), Automation and human performance: Theory and applications (pp. 201–220). Lawrence Erlbaum Associates.
Risko, E. F., & Gilbert, S. J. (2016). Cognitive offloading. Trends in Cognitive Sciences, 20(9), 676–688. https://doi.org/10.1016/j.tics.2016.07.002
Shaw, S. D., & Nave, G. (2026). Thinking — fast, slow, and artificial: How AI is reshaping human reasoning and the rise of cognitive surrender. SSRN. https://doi.org/10.2139/ssrn.6097646
Spatharioti, S. E., Rothschild, D., Goldstein, D. G., & Hofman, J. M. (2025). Effects of LLM-based search on decision making: Speed, accuracy, and overreliance. Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3706598.3714082
Stanovich, K. E., & West, R. F. (2000). Individual differences in reasoning: Implications for the rationality debate. Behavioral and Brain Sciences, 23(5), 645–665. https://doi.org/10.1017/S0140525X00003435
Tully, S. M., Longoni, C., & Appel, G. (2025). Lower artificial intelligence literacy predicts greater AI receptivity. Journal of Marketing. https://doi.org/10.1177/00222429251314491
 | A guest post by
|