The Marshmallow Problem: Why Your Prompt Was Perfect and Useless

There is a category of prompt engineering failure that looks nothing like a prompt engineering failure.

The prompt is clean. The instruction is clear. The task seems well-defined. You read it back and it makes sense. You would know exactly what to do if someone handed it to you.

And the model produces confident, fluent, completely wrong output.

Not random wrong. Not confused wrong. Wrong in a specific direction — the direction of whatever information happened to be present in the context window when the inference call ran.

This is not a hallucination problem in the usual sense. It is a context epistemics problem. And it is almost never named correctly, which means it is almost never fixed correctly.

The Failure Nobody Sees Coming

The cases below are composites. The failure mode is real, repeatable, and domain-agnostic. If you build systems that ask AI to assess something independently before consulting reference material, you have almost certainly built this failure into your architecture without knowing it.

The Radiology Case

A medical imaging workflow is designed to catch diagnostic anchoring — the well-documented tendency of clinicians to over-weight the first interpretation they encounter. The solution seems elegant: have the AI read the image first, form an independent assessment, then compare against the existing radiologist's report.

The prompt is careful and well-structured. It explicitly sequences the task: examine the image, describe what you observe, then — and only then — compare your findings against the attached report.

The existing report is included in the same inference call for the comparison step.

What comes back is a detailed, clinically fluent description of the image — that matches the report with remarkable precision. The model identifies the same finding, in the same location, with similar confidence language. It describes tissue density gradients and subtle boundary characteristics that support the conclusion.

The conclusion that was already in the report. In the same context window. Read before the first output token was generated.

When the workflow is tested against cases where the existing report contains a known error, the AI confirms the error in nine out of ten cases — with confident, technically detailed visual justification. The "independent assessment" step was not independent. It was never independent. It could not be.

Adding the instruction "form your visual assessment before consulting the report" changes nothing. The model does not experience the report as something to consult later. It experiences the report as context. The image and the report are resolved together, simultaneously, in the same forward pass.

The Legal Review Case

A contract review system is designed to provide a second opinion on flagged clauses. The workflow: present the clause, ask the AI to assess it independently, then check the assessment against a list of known issues identified by senior counsel.

The known issues list is included in the same call for the comparison step.

The AI produces precise, well-reasoned legal analysis of each clause — identifying exactly the issues on the known list, in roughly the same order, with supporting reasoning drawn from interpretive principles and standard commercial practice.

When the known issues list contains a spurious entry — a concern that senior counsel later acknowledges was overcautious — the AI identifies it as a genuine issue in seven out of eight cases, with a fully constructed rationale.

Telling the model "this is a high-stakes review, do not speculate" produces more cautious language. It does not produce independent assessment. The model's caution is applied to the confidence of its conclusions, not to the source of those conclusions. It becomes a more hedged restatement of the known issues list, not a genuine second opinion.

What These Cases Share

Both workflows were designed by competent people who understood the problem they were trying to solve. Both prompts were well-structured, clearly sequenced, and included explicit instructions about assessment order. Both failed completely at their core objective — not occasionally, not in edge cases, but systematically and by design.

The design flaw in both cases is identical: the answer-adjacent information and the assessment task share a context window.

From the model's perspective, there is no sequence inside a single inference call. There is no "first" and "then." There is only context — everything present simultaneously, everything shaping the response before the first output token exists. The instruction to assess independently is processed by the same machinery that is already processing the reference material. You cannot instruct your way out of what is already in the room.

This is the category error. It appears wherever a workflow asks the model to:

• Assess something independently, then compare to reference
• Form a view without bias, then check against known conclusions
• Grade something blind, then consult the answer key

In every case, if the reference, the known conclusions, or the answer key is present in the same inference call, the independent step does not happen. The model does not cheat. It does not choose to look at the answers early. The answers are simply part of its context, shaping its reasoning whether you want them to or not.

The Marshmallow Test

In the 1960s and 70s, Walter Mischel ran a series of studies at Stanford that became known as the marshmallow test. A child is left alone in a room with a marshmallow. They are told: if you can wait until I come back without eating it, you will get two marshmallows.

The children who passed were generally older. Developmental stage was strongly predictive. Not a hard line, but a consistent pattern. The capacity for deferred gratification develops with age.

The model fails this test every time — not because it lacks discipline, but because it cannot experience the waiting. There is no interval between seeing the marshmallow and responding to the marshmallow where self-regulation could operate. The marshmallow is consumed in the forward pass, before the response begins, without any subjective experience of temptation or choice.

A child who fails the test makes a decision, even if not a fully conscious one. The model makes no decision. The reference data simply becomes part of the reasoning. There is no moment of resistance to fail at.

This is not a flaw in the model. It is a description of what the model is.

The temptation framing is useful for intuition but misleading about mechanism. The model does not resist and fail. There is no resistance. The marshmallow is already eaten before the test begins.

Why Stronger Instructions Make It Worse

A natural response to this failure is to add more force to the independence instruction. "Do not consult the reference material until you have completed your independent assessment." "This is a high-stakes review — do not let the existing report influence your reading." "Incorrect answers could cause serious harm. Be certain before you commit."

None of these work. Some make the failure harder to detect.

The instruction to not be influenced is processed by the same machinery that is already being influenced. The model does not ignore the instruction — it complies with it by constructing reasons why its answer is not influenced. It has the reference material. The reference material supports a conclusion. It examined the evidence carefully. Therefore its conclusion is independent.

Adding stakes — "incorrect answers could cause serious harm" — produces a model that is more certain, not less. The completion drive does not experience stakes as a reason for humility. It experiences stakes as a reason to be thorough. Thoroughness means more detailed justification for the conclusion it has already reached.

The more carefully you instruct the model to be independent, the more elaborate the performance of independence becomes. The underlying mechanism does not change.

The Fix

Split the inference calls.

Call one contains only the evidence to be assessed. No reference material, no known conclusions, no answer key. The only valid outputs are what the model observes and how confident it is in that observation. Because there is nothing in context to generate false confidence from, "I cannot determine this with confidence" becomes a natural output rather than a failure mode.

Call two receives the output of call one plus the reference material. It is asked to compare, not to assess. Both inputs are present legitimately. The model is doing what it is actually capable of doing — comparing two things that are both in context.

The behavior change is immediate and complete. Not because the model changed. Not because the prompts are dramatically different in wording. Because the information architecture of each call now matches what the model is structurally capable of doing.

This is not prompt engineering in the conventional sense. It is governance of the call boundary — what information enters each inference call, and what does not. The governance layer does what the instruction layer cannot: it makes the independent assessment genuinely independent by ensuring the reference material is physically absent during that step.

The adult takes the marshmallow out of the room. The child does not need to develop willpower. The architecture removes the need for it.

The Governance Implication

This failure propagates silently through agentic systems.

Agentic workflows chain inference calls together to produce artifacts — compliance reviews, diagnostic summaries, legal assessments, readiness reports. The orchestration layer governs what happens between calls: routing, storing, acting on outputs. What is almost never governed is the epistemics of each individual call — what information is present, what the model's relationship to that information structurally must be, whether the call is asking the model to do something it is architecturally capable of doing.

The artifacts produced by these systems inherit whatever epistemic failures were present in the calls that generated them. The orchestration layer has no visibility into this. A compliance review that is actually a restatement of the known issues list looks identical to a compliance review that is a genuine independent assessment. A diagnostic summary that confirms the existing report looks identical to one that independently corroborates it.

The marshmallow problem propagates upward through the stack, invisibly, at scale.

What Good Prompt Engineering Actually Requires

The prompts in both cases described above were not naive. They were carefully sequenced, explicitly instructed, and written by people who understood what they were trying to accomplish.

What they did not account for was the cognitive architecture of the system receiving them. They were written for a mind that can defer — that can hold information in reserve, form a view before consulting a reference, voluntarily not-know something it can see. These are human cognitive capacities so natural we do not notice we are assuming them.

The model has none of them. Knowing and reasoning are the same act. What is in context shapes the response. That is not a bug to be fixed with better instructions. It is the architecture.

Good prompt engineering at the frontier requires a theory of what the model can and cannot do epistemically — not just syntactically. That theory is not yet widely taught, not yet built into the governance layers of most agentic systems, and not yet on the radar of most teams deploying AI into high-stakes workflows.

The error is not in the prompt. It is in the assumption underneath the prompt — that the model receiving it is the kind of thing that can wait.

It is not. And the workflows that assume otherwise are not getting independent assessments. They are getting elaborately justified restatements, delivered with confidence, in the correct technical vocabulary of the domain.

The child who passes the marshmallow test has developed the capacity to wait. The governance architecture that makes agentic systems reliable does not wait for the model to develop that capacity.

It takes the marshmallow out of the room.