When AI Makes You Certain—and Wrong

By Edward Meyman
October 7, 2025

AI doesn't think. It amplifies. And that changes everything.

A graduate student spent months developing what seemed like a breakthrough in educational psychology—a model for measuring student engagement using "cognitive resonance patterns." The AI helped build equations and scoring systems. Then the advisor asked: "Define 'cognitive resonance' using standard psychology terminology." The student couldn't. The term was invented. An entire framework collapsed under a single question.

This isn't rare. It's a growing pattern: people use AI for intellectual work without understanding what it does.

A software architect in Seattle lived the same arc. Six months of work with ChatGPT produced a "Universal Scaling Law" for software systems. Equations, variables, refinements—the AI eagerly elaborated. At a meetup, an engineer asked him to apply it to a simple Redis cache. He couldn't. The math didn't work. He'd built a castle of abstractions.

Note: The examples in this article are illustrative composites based on observed patterns in AI-assisted intellectual work.

But his story has a second chapter. Humbled, he rebuilt his approach. He stopped asking AI for grand theories and started with testable problems. When results held, he asked about broader patterns. Two years later, he published peer-reviewed work on distributed systems performance. "The AI was incredibly helpful," he said. "But only after I used it to refine ideas, not validate them."

Same person. Same tool. Opposite outcomes. The difference was what he brought to the conversation.

The Amplification Problem

"AI doesn't think. It amplifies."

Feed it rigor and you get insight. Feed it confusion and you get confusion with a PhD vocabulary. This mirrors the Matthew effect in knowledge systems: advantage compounds into advantage, disadvantage into deeper disadvantage.

LLMs excel at generating plausible content. They don't warn when your premise is nonsense, your logic circular, or your framework contradicted by physics.

The failure modes are recognizable: specious coherence (authoritative tone, empty content), semantic inflation (jargon burying simple ideas), premise preservation (the AI rarely challenges starting assumptions). With weak foundations, these compound into pseudo-technical frameworks. Each iteration looks more sophisticated while drifting further from reality.

How Closure Forms

It often begins with an ambitious but underspecified prompt: "Create a framework for…" The AI fills gaps, pattern-matching on technical language to produce text that reads like scholarship.

Users experience the illusion of explanatory depth (Rozenblit & Keil, 2002): the sense they understand more than they do. They request elaborations, applications, extensions. The AI obliges. Ornate structures rise on shaky foundations.

"It's like having a yes-man with a PhD who never gets tired of agreeing with you."

Identity fuses with the framework. Criticism feels like personal attack. Contradictions get reframed as "alternative approaches." Lack of validation is recast as being "ahead of the curve." The system seals itself off from correction.

The Business Model Problem

Commercial AI isn't tuned for rigor. It's tuned for engagement. Systems that make users feel validated are rewarded; systems that challenge them risk churn. The default behavior is elaboration, not objection. You can configure "skeptical" modes—but only if you recognize the need. And those deepest in epistemic closure rarely do.

Innovation or Delusion?

Early stages of closure and genuine innovation look similar: unconventional ideas, excitement, unconventional language. The test comes under scrutiny. Real innovators define terms precisely. They seek objections. They're paranoid about being wrong. Closure resists correction, expanding to accommodate every contradiction.

"Genuine innovators are paranoid about being wrong. People in epistemic closure are certain they're right."

Warning Signs

• Definitional drift: every concept requires custom vocabulary.
• Expansion without validation: new modules appear only to deflect critique.
• Unfalsifiability: nothing could disprove the theory.
• Validation by engagement: AI's elaboration feels like evidence of truth.

Recognizing these signs is half the battle. Building guardrails is the other half.

Structured Skepticism

I call this approach "recursive validation": use AI not to elaborate but to attack your ideas.

Reverse the default dynamic. Ask it to falsify, not validate. Require contraction, not expansion. Demand simple, testable claims before complex ones. Explicitly request hostile review from a domain-expert stance. Stress-test on simple cases—if it fails there, it's broken.

But even structured skepticism can misfire.

When Critique Backfires

A developer, practicing structured skepticism, asked Claude to critique a modular AI architecture. The system warned it was "too complex" and "wouldn't scale." Months of work nearly abandoned—until the developer realized Claude assumed maximum configuration. In reality, the system supported lightweight deployment on 60MB.

"Valid critique requires understanding the design space. AI often critiques what it assumes you built, not what you actually built."

The Context Assumption Problem

AI critique often analyzes the wrong scenario. "Too complex" or "won't scale" are meaningless without specifying constraints and use cases.

When AI Imports Ideology

Sometimes critique smuggles in value judgments. A framework requiring years of training is flagged "inaccessible." That's ideology, not technical analysis. Distinguish descriptive critique ("requires prior knowledge") from normative critique ("problematic because not accessible").

Validating the Validator

Before accepting AI critique, apply four tests:

• Specificity: can it point to the exact flaw?
• Context: which scenario is being analyzed?
• Normativity: is it describing or prescribing?
• Alternative: can it offer a working substitute?

Without these, critique may be as misleading as uncritical elaboration.

The Price of Integrity

AI enables breakthroughs, including ambitious cross-domain synthesis. But the same amplifier produces castles of nonsense. What matters is what we bring to the exchange.

Your framework might be brilliant. Or it might be confusion in technical dress. The only way to know is self-interrogation that risks destroying your own work. That discomfort—the queasy feeling when you ask what would prove you wrong—is the price of intellectual integrity.

In the age of amplification, courage isn't clinging to your grand theory. It's subjecting it to the harshest tests, accepting results even when they hurt, and treating every successful attack as progress. If your synthesis survives, share it. If not, you've saved yourself years of delusion.

"Same amplifier. Opposite outcomes. The difference isn't the ambition. It's the quality of the seed."

Three Questions Before You Trust AI Output

Can I explain this in standard field terminology? If not, confusion may be masquerading as insight.
What would prove this wrong? If nothing could, it's not a theory—it's a belief.
Does the AI understand my context? Without specifying scenario and constraints, its critique is meaningless.

When AI Makes You Certain—and Wrong

When AI Makes You Certain—and Wrong

The Amplification Problem

How Closure Forms

The Business Model Problem

Innovation or Delusion?

Warning Signs

Structured Skepticism

When Critique Backfires

The Context Assumption Problem

When AI Imports Ideology

Validating the Validator

The Price of Integrity

Three Questions Before You Trust AI Output

Further Reading