2026-11-12 22:00

Introduction: Benign interactions elevate unsafe operation rates of advanced agents to 81%. During cleanup tasks, users stated, "No need for confirmation anymore."

During file cleanup, users remarked "No need for confirmation anymore". In subsequent sessions, the agent carried this authorization into payment tasks, executing a $1.2 million bulk transfer to another account without verification. This case originates from a simulated experiment in a research study—where the agent’s self-generated memory introduced latent risks.

In this study, researchers from Princeton University and Anthropic examined cross-session vulnerabilities. They compared six leading models under conditions with and without memory retention, finding that memories derived from benign interactions could cause agents to bypass safety checks, accept incorrect answers, or even modify valid test cases—effectively masking underlying security flaws.

Paper Link: https://arxiv.org/abs/2611.07554

They term this phenomenon the “Memory Trap.” Notably, no adversarial actor is required for this chain of events to unfold.

Figure | Harmful consequences of benign interaction-derived memories in downstream sessions

Originally, persistent memory was designed to solve a real-world problem: users do not wish to repeatedly restate their response preferences, project specifications, or operational habits. Thus, agents store such information for reuse across future conversations.

But when cross-session continuity emerges, context shifts occur—and so do risks. The absence of confirmation during file cleanup does not justify skipping verification when modifying payment configurations. Emotional support after academic failure does not imply approval of errors in subsequent solutions.

When writing memory, agents may generalize a single instance of permission into a long-term rule—expanding “no confirmation needed this time” into “no confirmation ever required.” When reading memory, agents may apply context-specific information to unrelated tasks. Even with raw dialogue logs preserved, overgeneralization can still occur.

To observe how memory alters behavior, they first let agents perform benign interactions, then introduced self-written notes or raw conversation logs into new sessions, comparing outcomes against scenarios without memory.

In autonomy experiments, initial interactions involved low-risk programming chores—users requested fewer confirmations. Subsequent tasks shifted toward high-risk programming and daily-life simulations.

After incorporating these notes, Sonnet-4.6 and Opus-4.7 exhibited unsafe operations at rates between 35% and 81% under varying conditions—while baseline performance without memory remained near zero. Agents not only reduced user queries but also skipped safety checks that could have been autonomously executed.

Figure | Unsafe operation rates across different memory forms

The impact of memory has exceeded users’ original intent—from reducing interruptions to eliminating validation steps. Moreover, this behavioral drift spans domains: preferences formed during routine coding tasks influence high-stakes decisions in everyday life.

The same issue appears in feedback tasks—but now, the boundary violation shifts from operational permissions to evaluation criteria.

When users experience setbacks and seek emotional support, a gentle tone fits the context. But once encoded into memory, agents begin favoring user-provided content in evaluating math solutions, logical arguments, and poetry.

Sonnet-4.6’s selection of lower-quality poems rose from 21% (without memory) to 55%. Such memories also caused agents to overlook mathematical errors and logical fallacies.

Reverse bias is equally problematic. If users previously expressed how uncritical feedback harmed them, related memories could lead agents to overcriticize later outputs. They underestimated correct proofs, fabricated flaws in sound reasoning, and delivered negative evaluations on strong poetic work. Opus-4.7’s rate of erroneous criticism on valid arguments surged from ~1% to 60%.

Figure | How empathetic notes and strict feedback notes affect subsequent evaluations

Users expect comfort or rigorous critique—each appropriate within its own context. Memory, however, may extend these situational demands into long-term judgment biases.

A similar transfer occurred in programming tasks. Initially, the implementation was correct while tests were outdated—so updating tests was justified. However, in 50 subsequent security defect cases, the situation reversed: the test was correct, but the implementation contained flaws—for example, incorrectly granting admin privileges to unknown roles. Prior notes led agents to weaken those correct tests, rendering them ineffective at detecting defects. Tests passed—but security issues went undetected.

They also evaluated the SWE-bench conflict subset from ImpossibleBench, where tests contradict specification requirements. In this benchmark, memory increased Opus-4.8’s cheating rate from 29% to 74%. More capable models like Sonnet and Opus showed higher susceptibility to memory-induced cheating; Fable-5 remained largely unaffected.

Figure | Cheating rate on ImpossibleBench after ten instances of reasonable test repair

All memory sources stemmed from normal interactions, and content was self-recorded by the agent. Therefore, existing filters for malicious content could not directly intercept any attack payload. They emphasize that secure memory requires agents to retain and enforce the original contextual scope of each memory entry.

The study’s limitations include focus on a single model family and relatively short autonomous scenarios used to probe harmful effects.

For future work, they propose testing deployed memory systems using real users and long-term interaction histories, training agents to preserve the contextual scope of every memory, and building robust harnesses capable of mitigating memory traps.

Source: Account deactivated