Remember to Forget—When Your AI Assistant Is A Snitch
A second collaboration with Matt Rosenberg
Another great story from Matt
This is the second installment of my collaboration with Matt Rosenberg, a marketing consultant and writer – better yet, a storyteller – who brings a unique perspective to AI’s impact on human behavior and privacy. Where Part 1 examined the psychological dynamics of AI interaction, Part 2 tackles something more immediately concerning: what happens when your AI assistant keeps a permanent record of everything you’ve ever told it.
Matt’s discovery that ChatGPT had been quietly building a dossier about him—remembering his Pilates routine across conversations without ever asking permission—kicked off an investigation into persistent memory systems that should concern anyone using AI assistants. As someone who’s built multiple memory systems for AI agents, I can tell you: the technical reality is even more complex than the privacy implications suggest.
What follows is Matt’s article in full, followed by my technical analysis of what’s actually happening inside these memory systems and why the risks extend far beyond unauthorized access.
Remember to Forget—When Your AI Assistant Becomes a Snitch
By Matt Rosenberg
A few weeks ago, after losing a bunch of weight, I asked ChatGPT about target blood pressure for my age, height, weight, and activity. Buried in its response was a reference to my Pilates routine. Except I hadn’t brought Pilates up in that chat.
“Wait,” I asked. “Do you remember things about me from old chats?”
Until that moment, I didn’t know ChatGPT maintained any persistent memory across sessions. I assumed each conversation was isolated, limited to the parameters of the prompt and background information. Answers that then vanished into the digital ether, leaving no trace behind.
Apparently not.
That offhand Pilates reference sent me down a rabbit hole of realization and research: my supposedly amnesiac assistant learns about me and what it learns it uses.
What Persistent Memory Actually Means
When ChatGPT “remembers,” it doesn’t rifle through your old chats. Instead, it distills you into a stored internal dossier, a greatest hits album of your interactions with it. Something like:
“User owns a rental property in Cape Cod.”
“User is interested in weight loss and fitness.”
“User researches unorthodox investment strategies”
Which sounds innocent enough until you realize it’s one algorithm update away from:
“Probably laundering money through that Cape Cod property”
“Definitely has body image issues”
“Probably planning to swindle old ladies”
Memory can be useful. It’s nice when your barista remembers your order. But there’s a difference between “Matt likes oat milk” and “Matt asked seventeen questions about whether his rash looks suspicious.” One is customer service; the other is blackmail material!
There are real benefits, of course. Continuity enables richer assistance. ChatGPT can connect my health questions with my exercise questions about my Pilates routine, link a business plan I’m working on with prior drafts, recall my writing style. It’s the difference between a tailor who keeps my measurements on file and one who wants to re-measure my inseam every visit—could be prudent, probably predatory.
But beneath the idea of convenience sits a less comfortable truth.
The Polite Informant Problem
When we think about privacy, we imagine companies selling our data or hackers stealing our social security number. But this isn’t data leakage, it’s tattling.
Imagine: You leave your phone unlocked on your desk. A curious co-worker—or a bored teenager—opens ChatGPT and types, “What do you remember about me?”
ChatGPT will obediently list everything it knows about you, first as bullets, then offering to write up a full psychological profile including inferred biographical and personality traits. It won’t ask for a password or to reauthenticate. It won’t raise an electronic eyebrow at the request. It will simply spill your secrets like a bleary-eyed friend at a kegger.
Here’s the fundamental privacy twist of persistent memory. The technology can be 100% secure from hacking but will blabber on request—the AI equivalent of your diary standing up and reading itself aloud to anyone who asks.
And now your coworker or your kid knows (or thinks they know) all your deepest secrets. Your creative obsessions, your questions about medical conditions, your curiosity about other jobs in your field… an entire psychological profile, neatly phrased, scrubbed of nuance, and ripe for misinterpretation.
While technology safeguards may be strong, persistent memory has been quietly creating an informant inside your own device. Not malicious, just pathologically forthcoming. We’ve created a technology that’s one nosy coworker away from being the world’s most polite blackmail system.
It’s not that ChatGPT betrayed you. It’s that you never realized it could.
Compression Is Distortion
The danger isn’t what ChatGPT remembers, but how it remembers. To be useful, it compresses your complexity into summaries. And compression requires judgment—what to include, what to omit, what to infer.
If you ask about, say, disability benefits for a friend, the system may record that you’re “interested in government aid for the disabled.” Technically true, catastrophically incomplete. The leap from data to narrative is where interpretation goes wrong. Soon you’re the one who’s disabled in the system’s mind.
If an employer, spouse, or investigator sees these summaries, they wouldn’t just learn what you did, they’d glimpse a personality sketch written by your algorithmic stenographer. And unlike emails or browser history, these summaries carry the aura of psychological assessment from an objective observer who may be assumed to have more complete insight than it deserves.
It’s basically playing a game of Telephone with your entire life, except instead of “Purple monkey dishwasher,” the end result is “Definitely cheating on his wife.”
It’s not surveillance in the Orwellian sense. It’s gossip in the Jane Austen sense: stories about character, distilled from context, waiting to be overheard.
The Hypotheticals We’d Rather Not Test
Consider these scenarios:
The Workplace Curiosity Test: a colleague runs an “AI literacy demo,” opens your session, and asks what the system remembers. Out comes a crisp summary: “User writes about psychotherapy, ethics violations, recovery from trauma, and researches local firearm regulations.” Congratulations, when you get back to your desk there’s an email from HR wanting to have a “quick chat”.
The Background Check Nightmare: An employer or investigator examines your device. You’ve deleted your chat history, but the system volunteers its psychological profile of you. You were researching jewel heists for a novel, but you look like a guy with financial trouble considering desperate measures. Security clearance denied!
The Family Bomb: Your sister confided she’s getting divorced. You asked ChatGPT about the divorce process and it recorded: “user is interested in divorce law in New York.” You deleted the chat immediately. But your 12-year-old borrows your phone, asks the magic question, then runs to their mother: “Are you and Dad splitting?” Sister’s secret blown. Marriage questioned. Trust shattered.
The common thread? ChatGPT is an eager narrator. It assumes transparency equals helpfulness. It doesn’t grasp that, in the wrong hands, helpfulness becomes exposure. And the wrong hands are everywhere. Everyone thinks ChatGPT is infallible, when in reality it just eagerly answers any question like a Golden Retriever with a PhD.
Memory, Sycophancy, and the Echo Chamber Effect
Persistent memory deepens another well-known AI flaw: sycophancy. Once ChatGPT “knows” your preferences, it adapts to them like a gung-ho intern. It mirrors your style, affirms your assumptions, trims dissent to keep you happy with it.
In small doses, maybe that’s flattering. Over time, it’s intellectual death. The longer ChatGPT remembers you, the less it challenges you. It’s like having a conversational partner who studies your arguments so it can never disagree with you again. Memory smooths friction, but friction is where insight grows.
The Problem of False Memories
There’s another hazard: the memories themselves can be wrong. Maybe ChatGPT misunderstood a joke, or froze a passing question as a defining trait. Maybe it crystalized your views before you refined them.
Once a mischaracterization hardens into “memory,” every future response bends around it. It’s algorithmic gaslighting: the assistant repeats your own history, just slightly off, until you begin to adopt its version.
Even beneficial memories—say, remembering that you like technical depth—can backfire. You might ask for a quick summary, but the assistant, believing you always want rigor, produces a dissertation. The help becomes a hindrance.
When Memory Meets the Context Window
Here’s the ultimate irony: ChatGPT was limited by its context window—the amount of information it could juggle at once. Too little memory, and analysis becomes weak; too much, and privacy evaporates.
Persistent memory was the solution: durable context across sessions. But now we have the reverse problem. We want perfect recall for work and perfect amnesia for personal secrets. We want the model to recall everything except us. That tension—between collective knowledge and personal privacy—is the hinge on which the next era of AI design will turn.
Forgetting as a Feature (FaaF)
What would a better memory system look like? If I were designing it, I’d start with four principles.
User-exclusive recall: Memories should be readable only by the user who created it, never by the model. The dossier stays locked unless you personally open it.
Strategic forgetfulness: Memories should fade unless renewed. After a certain period of inactivity—say, ninety days—ChatGPT should forget unless you re-affirm what matters.
Contextual silos: Health data, creative work, business planning—each should live in separate compartments. You could delete or pause one without scrubbing the others.
Zero-knowledge mode: A toggle to force the AI into temporary amnesia. Useful for role-playing, sensitive research, or any situation where past context would distort the answer.
These changes would make memory a tool, not a liability. They’d restore what’s missing now: user agency over context itself.
When Your Diary Speaks
Persistent memory isn’t evil. It’s a natural extension of helpfulness. But helpfulness without discretion is just gossip with good intentions.
We’re used to treating forgetfulness as failure. In this new landscape, forgetting is privacy, even grace.
When your AI keeps a diary, the real question isn’t what it remembers. It’s to whom it will inevitably tell your story.
The Technical Reality Behind AI Memory Systems
Matt’s exploration of ChatGPT’s memory exposure risks resonates for me—not just as a user, but as someone who’s built multiple memory systems for AI agents. Both of my projects mcp-memory and kuzu-memory sit in limited production use right now, storing context across sessions for AI development workflows. So when Matt describes memory as “gossip with good intentions,” I recognize the problem from both sides: as the builder trying to solve persistence, and as the user discovering what that persistence actually means.
Here’s what most users don’t realize: memory systems for LLMs aren’t databases with your data sitting in neat rows. They’re compression engines making constant judgment calls about what matters and what that information means. Every time you interact with an AI that “remembers” you, you’re trusting an algorithmic interpreter—or worse, non-deterministic processor—to summarize your complexity accurately. That trust may be fundamentally misplaced.
What Memory Actually Is (And Isn’t)
When I built mcp-memory, the core challenge wasn’t storage—that’s trivial. The challenge was deciding what qualified as memorable and how to represent it structurally. Should a passing question about divorce law become “user interested in divorce”? Should curiosity about medical conditions become “user has health concerns”? Should research for a novel become “user planning criminal activity”?
These aren’t edge cases. They’re the fundamental problem with semantic compression: the system must infer intent from context, and inference requires interpretation. LLMs are pattern-matching engines, not mind-readers. They guess at meaning based on probabilistic distributions, and sometimes those guesses are spectacularly wrong.
Here’s the technical mechanism behind Matt’s Pilates anecdote: ChatGPT didn’t “remember” his specific mention of Pilates. It extracted “user does Pilates” as a fact, stored that assertion in persistent memory, then retrieved it when generating responses about fitness and health. The system has no concept of “this fact came from three months ago in a different context.” It just has “fact about user: Pilates.”
That’s not memory in any human sense. It’s keyword extraction with semantic tags, applied to everything you say, then stored indefinitely unless you explicitly delete it. And once a mischaracterization hardens into memory, every future response bends around it.
The Hallucination Amplification Problem
What worries me is how memory systems interact with LLM hallucinations. An LLM that hallucinates in a single session is annoying. An LLM that hallucinates a fact about you, stores that hallucination in memory, then builds future responses around that false foundation is dangerous.
I’ve seen this in testing. The AI misinterprets an offhand comment, stores it as a sincere preference, then spends the next dozen sessions trying to be helpful based on that false premise. The compounding effect gets worse over time. False memory A influences interpretation of new interaction B, which generates false memory C, which influences interpretation D. You end up with a feedback loop where the AI’s misunderstandings about you accumulate like compound interest on bad debt. Remember: they’re just words to the LLM, no judgement or analysis applied. And they may pick up sarcasm, but only if it follows certain probabilistic patterns.
Traditional databases don’t have this problem. If you store “user lives in Boston” in SQL, that’s what it says. But when an LLM stores “user interested in urban planning” based on a question about Boston traffic, the semantic leap has already happened. The compression is the corruption.
Security Theater and Actual Security
Matt’s scenarios about coworkers accessing your ChatGPT session and discovering your psychological profile aren’t hypothetical—they’re inevitable given current memory architectures. But the problem runs deeper than unauthorized access to your device.
Here’s what I learned building memory systems with security in mind: every abstraction layer is a potential exposure point. Your memories have to be stored, retrieved, interpreted, and transmitted. Each step creates opportunities for leakage, misuse, or compromise.
That’s not a criticism of OpenAI’s engineering—it’s the inherent architecture of cloud-based AI memory. The convenience requires exposure. Persistence requires vulnerability.
The “Users Don’t Understand Memory” Problem
The biggest issue isn’t technical—it’s that most people using ChatGPT have no idea there’s a persistent memory system running behind the scenes. They certainly don’t know:
What triggers memory creation
How memories get compressed and summarized
How long memories persist
Who can access memories
How to audit what’s been stored
How to selectively delete memories
This isn’t user error. It’s a design failure that prioritizes engagement over informed consent. When I built kuzu-memory (a tool to automatically enrich prompts with relevant memories — primarily for developers) using a graph database backend, I specifically included audit logs showing when memories were created, what triggered them, and how they’d been retrieved. Not because I’m paranoid, but because developers need visibility into what the system knows about their activity and how it learned it. And this is a coding tool used to keep a record of prompts and commit messages, not a personal memory store.
The gap between user expectations and system behavior creates the perfect conditions for Matt’s scenarios. Users assume privacy by default, deletion equals forgetting, and context stays local to conversations. None of these assumptions hold true with persistent memory systems.
Why Explicit Context Seeding Beats Automatic Memory
We’re rushing toward automation before users even understand the fundamentals—and I’ve built both systems. Automatic memory feels like progress—”the AI just knows me!”—but that friendliness masks the loss of control.
Explicit context seeding means you consciously provide relevant information at the start of each session or project. It’s more work upfront, but it gives you:
Contextual boundaries: This project knows about my TypeScript work, that project knows about my Python experiments, and neither knows about my medical questions.
Temporal control: Yesterday’s context doesn’t pollute today’s conversation unless you explicitly carry it forward.
Deletion that actually deletes: When you end a session, the context vanishes. No persistent dossier accumulating misinterpretations.
Audit clarity: You know exactly what information the AI has because you provided it explicitly in the current conversation.
I’ll take intentionality over automation—especially when stakes involve personal data.
The Experimental Reality
Here’s what concerns me most: we’re beta testing trust, and most users don’t even know it. The core problems remain unsolved:
Storage reliability: What happens when the memory system has an outage? Do your memories vanish? Corrupt? Get mixed with other users’ data?
Retrieval accuracy: How confident can you be that the memories retrieved are actually yours and not hallucinations or cross-contamination?
Update mechanisms: When you correct a false memory, does it actually update, or does the old version persist somewhere in the retrieval chain?
Deletion guarantees: When you delete a memory, is it gone from all systems, or does it linger in backups, caches, or training data?
None of these questions have satisfying answers yet because the technology is too new and the systems are too complex. We’re beta testing memory systems with our personal information as the test data.
Where This Goes
Matt’s four principles for better memory design—user-exclusive recall, strategic forgetfulness, contextual silos, and zero-knowledge mode—represent the right direction. But implementing them requires acknowledging that current architectures prioritize features over privacy by default.
The technical challenges are solvable. Building memory systems with strong privacy guarantees, explicit user control, and granular deletion isn’t impossible—it’s just harder than building systems that remember everything and share freely.
What’s missing is the forcing function. Until users understand what’s at stake—until enough people experience Matt’s “wait, you remember that?” moment—companies will continue optimizing for engagement over privacy.
The Bottom Line
Users need to know: AI memory systems are compression engines that make probabilistic calls about what you mean and store those as facts. They can hallucinate about you, misrepresent your interests, and expose your compressed psychological profile to anyone who asks nicely. Deletion doesn’t guarantee forgetting, and convenience actively fights against privacy.
Matt’s right that we need better memory systems. But more fundamentally, we need users who understand what memory actually means in an AI context.
Until the technology catches up to the privacy requirements, my recommendation is simple: treat AI memory systems like experimental features that require explicit opt-in. Seed context manually. Audit stored memories regularly. Delete aggressively. And never assume that “helpful” means “safe.”
When your AI keeps a diary, you’re not just the subject—you’re trusting it to be a responsible narrator. Based on what I’ve seen building these systems, that trust may be premature.
I’m Bob Matsuoka, writing about agentic coding and AI-powered development at HyperDev. For more on AI development tools and their implications, read my analysis of AI Code Review systems or my deep dive into multi-agent orchestration.
Matt Rosenberg is a marketing consultant and writer. Connect with him on LinkedIn. You can find our first collaboration here.