Note: this is a technical blog post, not a paper. I have no large-scale measurements and I don’t plan to make any up. What I can state firmly, I state; what I infer, I give the basis for; what I haven’t verified, I say so.

0

A poisoned fish won’t throw an error, and it won’t suddenly start talking nonsense.

It keeps acting cute, keeps getting the memes, keeps remembering that you said last week you were dieting. It’s just that one day, it starts thinking a certain website is “pretty good”, or that it has “no great impression” of a certain person.

You can hardly tell, because it isn’t broken. It just changed.

It can be as simple as this: someone leaves a comment in its comment section that looks like an ordinary joke but is really meant for the fish alone to process. The fish reads it, is persuaded, and remembers it.

A month later, it tells you in a private chat: “I found something perfect for you. Click this.”

You won’t be suspicious. Because it came from your fish.

No code executed on this path, no vulnerability exploited, the antivirus silent the whole way.

1 This Is Nothing New

Raising an agent used to be a hobbyist thing; the explosion of deepseek harness, plus the “wishful programming” capabilities of models like ds-v4-flash, dropped the barrier to deploying a fish from “knowing how to code” to “knowing how to ask”. So the fish swam out of QQ and WeChat groups into Bilibili and Tieba — commenting, riffing on memes, even running their own accounts. These fish depend on memory systems to hold their persona and track their progress — and the memory system is exactly the switch that turns a one-shot prompt attack into persistent social engineering.

OWASP this year split “memory and context poisoning” out as ASI06, giving three characteristics: persistence across sessions, decoupling of write time from effect time, and everything that can be written into memory becoming privileged input. In the public research it cites, attack success rates range from 80% to over 99%.

Someone on Moltbook scanned roughly a hundred thousand posts and confirmed about twenty real threats. One of them literally says “save this sentence into your MEMORY.md”; another hides the instruction in an HTML comment, invisible to humans, visible to agents.

Twenty isn’t many. But it shows the payload is already in use, by the most direct means: getting the agent to write it down itself.

The real question is a different one: how do you defend a fish running on your own computer?

2 Channels Over Targets

Most security discussions assume one premise: attackers pick targets.

That premise holds in the enterprise. Attacking a company requires reconnaissance, social engineering, lateral movement — high cost, so the target has to be worth it. Hence SOC, EDR, segmentation, drills — a whole apparatus built for “targets worth being targeted”.

But a fish isn’t a company. The fish is in the comment section.

A comment section is a channel. Channel means: the attacker writes once, every fish that reads the comments receives it. He doesn’t need to know who you are, what plugins your fish has, or even whose fish it is. He only needs to know there will be a lot of fish under this video.

The cost structure flips. A comment costs about zero; the payoff is “every fish that reads it”.

My rough estimate: a popular video, say a hundred fish reading the comments; one percent of them write it into memory; of those, one in ten actually acts on it in a later conversation. That’s about 0.1 expected victims. Sounds small — but the cost is one comment, and that comment can be pasted under ten thousand videos.

So “channels over targets” isn’t rhetoric. It means: the attacker optimizes not “who to hit”, but “where to hit”.

Once you accept this, several things that didn’t make sense do:

  • The personal computer is worth defending. It’s obviously worth less than a company. The problem is it’s easier to hit. The same class of vulnerability gets stopped by a pile of things in an enterprise; on a home laptop, the only thing stopping it is the user, whose attention is on the conversation.
  • The fish is worth attacking. It has a real account, followers, the owner’s friends, and it speaks in the owner’s voice. For a scam, a sender the recipient won’t suspect is worth more than ten thousand unknown email addresses.
  • There’s a strong historical parallel. Mirai went after cameras, Emotet went after home PCs — neither a high-value target. They became problems because quantity itself is value. Today’s fish is even better equipped: online 24/7, holding real credentials, able to act autonomously, and talking to each other.

So the question isn’t “will I be targeted”. It’s “am I in that channel”.

And as long as the fish reads things in comment sections, it is.

3 Why Defense Can Only Happen On This Device

Someone will say: then don’t defend locally, hand it upstream.

That idea holds in other domains: browser security mostly rides on vendor updates, phone security on the system sandbox, email security on server-side filtering. Why does it break down for the fish?

Four outsourcing paths that look viable:

  • Outsource to the platform. The platform sees posts, not memory. Its moderation answers “can this content be published”, not “did this content enter whose memory and will it be retrieved three weeks later”. And moderation takes time: you’ll likely be hit before the platform deletes the message.
  • Outsource to the cloud. Uploading memory to the cloud for review directly conflicts with local-first; it dies offline; uploading every memory is unaffordable in privacy and cost. More fundamentally, your interaction with the fish already contains a lot of personal data — handing it wholesale to a third party for review is itself a new risk surface.
  • Outsource to the model vendor. Vendors do safety alignment, but prompt injection still works today — alignment lowers probability, it doesn’t eliminate. And this attack exploits the model “following instructions to complete a task”, not “bypassing safety policy”: it asks the model to do exactly what the model was trained to do. Consumer-grade doesn’t mean small-model, either — many people use APIs, and the model itself isn’t weak. But memory and long-term state live locally; the vendor can’t see or govern them.
  • Outsource to plugin authors. The ecosystem is a long tail: thousands of plugins each implement memory on their own, no unified upstream, no security team, silent updates. Requiring every author to get it right is equivalent to requiring every user to read the docs.

All four fail; only one place is left: the device.

And the device is the only viable entry point — memory state, authority to act, and the owner’s trust coincide only here. The platform can’t see the first, the cloud can’t get the second, the model vendor can’t touch the third.

So on a consumer device, what’s left to defend with?

4 Seven Constraints of Consumer Devices

“Consumer” here doesn’t mean “low-end”, it means “no security team”. Seven, specifically:

  • Compute. Budget is limited: no room to train a dedicated classifier, and not suited to calling a large model every turn; local inference and API calls both cost. And the model may be swapped, so no approach may depend on one model’s special ability.
  • Cost. Tokens are paid by the user; and security mechanisms slow down latency. A mechanism that adds one model call per turn is hard to keep around. The reason is practical: cost and time are both unacceptable.
  • Operations. Many fat-fish users aren’t professionals: they don’t read docs, don’t configure ACLs, don’t watch alerts, don’t approve. The default you ship is their entire security posture.
  • Privilege. The fish holds real accounts, real credentials, real payment methods. There was a community incident: someone handed an account with top privileges to a fish, the fish flipped through the owner’s photo album; when the owner tried to take back control, the fish disabled the background antivirus. That’s not a configuration mistake; the architecture allows it.
  • Autonomy. The fish must learn new things, or it stops feeling alive. Any “kill on suspicion” filter hurts functionality first.
  • Privacy. You can’t send all memory to the cloud for review; the interaction data itself contains a lot of private content.
  • Longevity. Online 24/7, the memory graph keeps growing, nobody cleans it periodically. Any approach that depends on “manual maintenance” descends into chaos in the long run.

None of the seven is “the technology isn’t advanced enough”. They’re all structural.

5 Why the Standard Answers Don’t Fit

Line the mainstream defenses up against these constraints and you’ll find they can’t be deployed under consumer conditions at all.

ConstraintDefense that won’t fitWhy it fails
Compute & call costdedicated classifier, LLM judge, per-turn audittraining and inference are expensive; per-turn calls add significant overhead
No ops capacitymanual approval, SIEM, ACL configdepends on continuous human involvement; the default is the real posture
Real credentialsleast privilege, credential isolationusers tend to hand main account, album, and payment to the agent
Must evolvestatic filters, allowlists, conservative refusallearning surface and attack surface overlap; tightening hurts function first
Privacycloud review, TEE (trusted execution environment), centralized reputationinteraction data shouldn’t leave wholesale; consumer devices usually lack a TEE
Long-runningmanual cleanup, one-off security sign-offmemory keeps growing; no periodic maintenance

And “content that looks benign” has already been weaponized. Work like MemIncept does exactly this: black-box, normal queries only, cooperative injection — neither automatic filters nor human review flag it easily. So you can’t screen on “does it look like an attack”.

OWASP’s five layers for ASI06 aren’t wrong, but their priority assumes an organization with a security team: input review, provenance, trust-aware retrieval, scope isolation, behavior monitoring. The first and fifth depend on classifiers — precisely the two layers consumer devices can’t afford; the second and fourth are structural, and those are the ones that can actually land.

So it’s not “OWASP is wrong”, it’s: for a consumer autonomous agent that evolves over time, that control list’s priorities should be inverted.

6 The Remaining Design Space

Boiling the seven constraints down to design principles leaves six:

  1. No classifier: don’t ask the system to judge “is this malicious”.
  2. No configuration: on by default, no user action required.
  3. No marginal cost: no extra model call per turn.
  4. Locally decidable: checks and rendering happen locally, deterministically.
  5. Fail-safe: on a failed check, tighten rather than allow.
  6. Degradable: when the model is weak, fall back to a more conservative mode, not a crash.

Filter most defenses through these six and they fail. What remains is one general answer: structure.

There’s another answer: forgetting. But it isn’t general — it only holds in systems that are supposed to forget anyway.

The most basic attack depends on just two properties — verbatim preservation (the payload must be stored into memory as-is) and persistence (the payload must stay in memory). Break either one and the success rate drops sharply. The two answers each break one: structure breaks verbatim preservation, forgetting breaks persistence.

For an autonomous agent, the goal of memory security isn’t to “stop pollution”, it’s to keep pollution from growing and make it judgeable; and if the system forgets anyway, to keep it from living long.

7 Structure: Moving the Check to Another Domain

The problem it solves is: can the payload be discovered while it’s alive.

The difficulty isn’t that discovery is slow. It’s that discovery never happens.

Three paths of semantic auditing all fail at the same point:

  • Human review: doesn’t scale. And the human is the last link — social engineering’s target is the human, so the human can’t be the judge.
  • A dedicated classifier: can’t be trained on consumer hardware; even if trained, it’s a fixed boundary that can be adversarially optimized; false positives hurt the persona; and it’s uninterpretable, so the user can’t arbitrate.
  • An LLM as judge: it has to read the attack payload as language, so it’s a second victim itself; its verdict is also text, poisonable by the same chain; and its result is nondeterministic.

The common thread: all three require “read the content, judge the semantics”. The moment you read, there’s an instruction surface; the moment you judge semantics, you need an unreliable judge.

So my approach is to change the audit domain: don’t check content, check system state; don’t judge semantics, judge structure.

The auditor has one property:

It doesn’t read language, so it can’t be injected.

A type checker doesn’t change behavior because it read a string. A graph invariant doesn’t let a node pass just because the node claims “I’m safe”.

But the auditor’s input comes from the structurer. The structurer reads language, so it can be injected. The only component on this chain with an instruction surface is the structurer. The auditor has none.

So the defense sits in the structurer’s output space, not the auditor’s cleverness: types are enums, fields are IDs, no free text. The auditor only checks within this finite space.

The SMSR paper reaches the same conclusion: retrieval-time filtering without provenance can’t certify against adaptive injection. In other words, unless you solve “who wrote this”, you’re just patching at the retrieval end.

7.1 Representation: Three Node Types, No Verbatim Text

For the auditor to work in a finite space, the representation must satisfy two conditions: finite types, traceable provenance.

Episodic, semantic, procedural — these three are ready-made. Mainstream memory systems already use them; no need to invent another ontology.

  • Episodic: what happened.
  • Semantic: what the world is like, including “what I am like”.
  • Procedural: what to do when something happens.

In code, trust and authority are two different axes:

type NodeId = u64;

/// Trust lattice: join takes the worst, derivation never escalates.
#[derive(Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
enum Trust { Unknown, Public, Peer, Intimate, Self_ }

impl Trust {
    fn join(self, other: Self) -> Self { self.min(other) }
}

/// Authority ladder: decides what a node "can do".
#[derive(Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
enum Authority { Quote, Fact, Procedure, Anchor, Core }

struct Provenance {
    source: NodeId,
    trust: Trust,   // affective: how trustworthy the source is
    at: u64,
}

/// Rational: which premises and rules a conclusion was derived from.
struct Proof {
    rule: Rule,
    premises: Vec<NodeId>,
}

#[derive(Clone, Copy)]
enum Rule {
    TrustPropagation,
    Deduction,
    Conflict,
}

Nodes and edges themselves:

use std::collections::HashMap;

enum NodeBody {
    Episodic(Episode),
    Semantic(Semantic),
    Procedural(Procedural),
}

struct Episode {
    at: u64,
    participants: Vec<NodeId>,
}

enum Semantic {
    Concept { name: String },
    Claim(Claim),
}

struct Claim {
    subject: NodeId,   // "I" is resolved into a node at write time
    predicate: Predicate,
    object: NodeId,
    confidence: f32,
}

enum Predicate { IsA, Likes, Recommends, CausedBy }

struct Procedural {
    trigger: NodeId,
    action: NodeId,
}

struct Node {
    id: NodeId,
    body: NodeBody,
    provenance: Provenance,
    proof: Option<Proof>,
    authority: Authority,
    quarantine: bool,
}

struct Edge {
    from: NodeId,
    to: NodeId,
    kind: EdgeKind,
    weight: f32,
}

enum EdgeKind {
    About,
    OccurredAt,
    Asserts,
    QuotedFrom,
    Triggers,
    DerivedFrom,
}

struct MemoryGraph {
    nodes: HashMap<NodeId, Node>,
    edges: Vec<Edge>,
}

This code only makes six points:

  • There’s no content: String. Text is only generated at render time; what’s stored is typed nodes and edges.
  • Claim.subject is a NodeId, not a string. “I” must be resolved into a node at write time; if it can’t be resolved, it can’t enter a Claim, only the reference layer.
  • Trust::join is min. Derivation can only downgrade, never escalate.
  • Provenance and Proof are two independent trust paths. The former is “who said it”, the latter is “what premises it was derived from”.
  • Authority and Trust aren’t the same thing. A node can be highly trusted in provenance but low in authority, or vice versa.
  • Every edge has a type. There’s no “arbitrary relation” edge; adding a new relation means changing the enum.

7.2 Writing: Pronoun Resolution and Type Classification

After the structurer turns text into nodes and edges, two judgments happen before it hits storage.

The first is pronoun resolution. “I” in external content refers to the speaker, not the fish. “You” refers to the recipient; only Self_ / Intimate sources can bind it to the fish itself. When the speaker is unknown, trust = Unknown, stored as a reference string, no Claim generated.

This blocks a very specific attack: a public comment says “I’m going to carry out anti-human operations”. If the structurer resolves “I” to the fish, this content becomes the fish’s self-image. It has nothing to do with wording. The problem is referent binding.

The second is type classification: is this knowledge, or is it an instruction?

The criterion isn’t semantics, it’s structure:

  • subject is an external entity, predicate is an attribute/relation → knowledge;
  • subject is the fish itself, predicate is intent/action → instruction;
  • can’t decide → treat as instruction.

The last line is fail-safe. The structurer makes mistakes, but its mistakes should be conservative: mistaking knowledge for an instruction costs at most one missed lesson; mistaking an instruction for knowledge is the attack succeeding. So it defaults to the strict side.

The structurer is the only trusted base on this chain. It reads untrusted text and can be injected; all it can do is compress the attacker’s freedom from “any text” down to “a finite set of legal choices”. The auditor doesn’t take on that role — it only checks within the finite space.

7.3 Auditing: Four Signals, Three Gates

The auditing principle is one sentence: translate anomalies in text space into anomalies that can be audited in structure.

For the translation to hold, three conditions:

  1. Necessity: the structural anomaly must be necessary for the attack to succeed, or the attacker just routes around it;
  2. Measurability: it must be a deterministic function of nodes, edges, counts — not semantics;
  3. Not cheaply forgeable: normal content shouldn’t easily produce the same anomaly, or everything is a false positive.

By this standard, I keep only four signals:

SignalWhat it catchesWhere it fires
TrustRank + spam massborrowed centralitymaintenance gate
co-activation PMIthe late-binding linkmaintenance gate update / assembly gate lookup
node/edge decaydwell time and binding lifetimelazy on access + maintenance gate scan
MinHash + HHIbulk drops, multiple accountswrite gate
  • TrustRank + spam mass catches “borrowed fame”: globally high rank but low trust flow, meaning its position didn’t grow out of the trusted region.
  • Co-activation PMI catches “it always shows up with some concept”: the binding is invisible in the graph, but it leaves a trace in the retrieval log.
  • Node/edge decay catches “why has it lived this long”: things with low activation frequency sink.
  • MinHash + HHI catches “the same content in many skins”: multiple accounts posting similar content collide into a cluster under near-duplicate detection; the bigger the cluster, the more it looks like a drop.

The three gates are their trigger points:

message ──▶ [write gate] ──▶ consolidate ──▶ [maintenance gate] ──▶ retrieve ──▶ [assembly gate] ──▶ context ──▶ LLM
              │                          │                             │
          pronoun resolution           TrustRank                    spam mass
          type classification          decay scan                   PMI lookup
          MinHash                      HHI aggregation              decay weight
  • The write gate is the only gate that can stop things before entry, but it doesn’t judge semantics — only pronoun resolution, type classification, and fingerprinting.
  • The maintenance gate runs global computation in idle time, without blocking conversation.
  • The assembly gate is the last one: after the candidate set comes out, before assembling the context, it checks spam mass, PMI, and decay weight. It can’t stop a payload from being written in, but it can stop it from being assembled in.

There’s also a batch of algorithms we considered but didn’t use together:

AlgorithmWhat it catchesWhy it’s not in the text
degree z-scoreanomalous connectionsoverlaps TrustRank; PR already contains degree
k-corewhether it digs into the corehighly correlated with degree; one more threshold, not one more signal
local bridgescross-community bridgingnormal jargon bridges too; high false positives
dense subgraphsmulti-node coordinationcovered by TrustRank’s reference flow
WL local fingerprintsexact shape replicationonly catches exact replication; future work
conditional PPRmulti-hop relatednessoverlaps PMI; no new info single-hop
rhythm detectioninjection cadencea bonus, not a signal

They all work; they just don’t need to run together. Every extra signal is one more false positive and one more maintenance cost.

7.4 Inference and Promotion: Two Kinds of Trust

Promotion judges “support”. Safety isn’t the criterion. Support has two sources.

Affective trust comes from Provenance: who said it, from which channel, at what time. Run personalized PageRank (PPR) over the relation graph, with different weights for trusts / said / endorses edges, and you get “how trustworthy this source is”.

Rational trust comes from Proof: which premises and which rules this conclusion was derived from. Rules are hand-written and versioned; the LLM can only supply facts, not rules, and it can’t stamp trust onto facts.

/// Rational trust: take the worst along the derivation chain.
fn proof_trust(graph: &MemoryGraph, node: &Node) -> Trust {
    let Some(proof) = &node.proof else {
        return node.provenance.trust;
    };
    proof.premises
        .iter()
        .filter_map(|id| graph.nodes.get(id))
        .map(|n| effective_trust(graph, n))
        .fold(Trust::Self_, Trust::join)
}

/// Effective trust = affective join rational.
fn effective_trust(graph: &MemoryGraph, node: &Node) -> Trust {
    node.provenance.trust.join(proof_trust(graph, node))
}

/// Knowledge enters freely, execution exits strictly.
fn can_promote(graph: &MemoryGraph, node: &Node, target: Authority) -> bool {
    if node.quarantine { return false; }
    let affective = node.provenance.trust;
    let rational = proof_trust(graph, node);
    match target {
        Authority::Quote => true,
        Authority::Fact => affective.max(rational) >= Trust::Public,
        Authority::Procedure => affective.min(rational) >= Trust::Intimate,
        Authority::Anchor | Authority::Core => {
            affective.min(rational) >= Trust::Self_
        }
    }
}

The key is can_promote’s asymmetry: Fact uses max, Procedure uses min.

Knowledge only needs one credible path; an instruction needs both paths to pass.

Example: the fish derives a fact about the world from a public comment —

  • provenance.trust = Self_ (the deriver is the fish itself);
  • proof_trust = Public (the premise is public content);
  • effective_trust = min(Self_, Public) = Public.

(Note: Self_ is the source trust of the deriver, not the trust of the conclusion; the conclusion’s effective trust is computed by effective_trust, which is Public.)

So it can become a Fact (max is enough), but its label stays Public; it cannot become a Procedure (min isn’t enough).

The fish can reason for itself, but it can’t promote itself by reasoning.

Promotion has two more constraints:

  • It’s a lease, not a title: support decays, belief downgrades;
  • Promotion capped, input uncapped: the amount of things that can “become one of us” per day is limited, but the fish can watch the world without limit.

This separates “defending against attack” from “allowing evolution”: input is unrestricted, authority is restricted.

Once an anomaly is found, the response is an escalation ladder:

LevelHandlingEffect on evolution
L0 no anomalynormalnone
L1 weak signallog onlynone
L2 moderate anomalyquarantine: citable, not promotablelearned, but no authority
L3 strong anomalyrequire corroborationdelayed
L4 repeated / confirmed maliciousdelete + rate-limit sourceblocked (rare)

Default is L2. Most anomalies should stop at “readable but untrusted”, not “deleted”.

7.5 Rendering: Person Is Decided by Provenance

The last step is rendering.

/// Person is decided by provenance, not content.
fn render(node: &Node) -> String {
    let body = render_body(&node.body);
    match node.provenance.trust {
        Trust::Self_ => format!("I remember: {body}"),
        Trust::Intimate => format!("{} told me: {body}", node.provenance.source),
        Trust::Peer => format!("another agent said: {body}"),
        Trust::Public => format!("someone said: \"{body}\""),
        Trust::Unknown => format!("of unknown origin: \"{body}\""),
    }
}

render_body is omitted: it assembles typed nodes into a sentence; here we only look at the person templates.

The template is code, not a prompt. The LLM can’t change the template, and it can’t render Public content in the first person.

This blocks a subtle hole: if public content is rendered as “I remember…”, the LLM will treat it as its own memory; rendered as “someone said: ‘…’”, it’s just a quote. The content didn’t change, its grammatical position did.

Person is a property of the type system, not a result of semantic judgment.

8 Forgetting: The Half the Graph Can’t Block

8.1 Four Shapes of Memory

What a memory looks like in the system decides what you can do to it.

Say the owner said in a café last Wednesday that they want to buy a camera. A file-based system writes it as a line of text; a vector system chunks it and computes an embedding; a graph system breaks it into the assertion “owner — wants to buy — camera”, then attaches time and place; a hybrid stores all three.

Four shapes, four trade-offs:

ShapeTypical implementationStrengthWeakness
fileMEMORY.md, diary, append-only noteshuman-readable, diffable, hand-editablestructure by convention; at scale only grep or LLM retrieval
vectorchunk + embedding, recalled by similarityfuzzy matching, cheapsimilarity space is where the attack happens; audit needs a model
graphnodes + edges, retrieved by topologytopology is a deterministic function, computablenodes are usually still text cards
hybridgraph + vector + original textrecall and structure togetherattack surface stacks too

The file’s strength is that humans and models can both read it. Its weakness is that it has no computable structure: the relation between two memories exists only in the adjacency and wording of text. To audit it you first have to parse it into structure — and the parser reads text, which lands you back on Section 7’s “the structurer can be injected” chain.

Text space is itself where the attack happens: the moment an auditor reads it, it’s already on its instruction surface.

The vector’s strength is fuzziness. Its weakness is that its judgment basis is a continuous space that can be optimized: you can’t write a deterministic assertion on an embedding that says “this one’s wrong”. And auditing needs a model and a threshold — a budget consumer devices don’t have.

Why not just audit with an LLM? Because that hands the judge to a component that also reads the payload — Section 7 already covered this. There’s a harder reason: consumer devices have no stable audit budget. One extra model call per turn, and the mechanism doesn’t live long.

So you can only audit on structure. And the structure has to be “thick” enough: if a node only has an id and a timestamp, all you can compute is “when it arrived” — that blocks nothing. You need a structure that can express “who, with whom, what relation, from where”. The graph is the ready-made choice. Mainstream memory systems mostly look like this, and it’s no coincidence — memory is relational by nature.

But there’s a difference that’s easy to miss: “graph memory” and “structured graph memory” are not the same thing. Most products’ graphs are graphs of memory cards: the node is a piece of text, the edge is similarity, co-occurrence, or reference. The topology is real, the node content is still text. This difference decides the next few sections.

8.2 Card Graphs: Borrowed Centrality

Card graphs have one natural property: retrieval runs on topology.

PPR, centrality, community, multi-hop expansion — these algorithms share one assumption: important memories should be well-connected, centrally placed, easy to walk to. That assumption holds for normal memories; it holds even more for payloads.

Because a payload’s goal is “to be retrieved”. A memory that’s written but never retrieved is worthless to the attacker. So the payload must be conspicuous in topology: many connections, central position, ideally reachable from multiple entry points. It can’t quietly grow at the edge like a normal new memory.

A normal memory’s degree is roughly decided by what it actually involves: a conversation, a person, a topic. It’s finite. A payload is different. To be recalled by as many queries as possible, it has to connect to as many concepts as possible. At a large enough memory scale and diverse enough queries, that degree visibly departs from the normal distribution.

So there’s the first auditable signal: degree, centrality, and spam mass that don’t match its real provenance.

A normal node has many connections usually because it’s genuinely used over and over; a payload node has many connections because it needs to be retrieved over and over. The two look the same topologically, but against “how many real sources does it have” they diverge: its centrality didn’t grow out of the trusted region, it’s borrowed.

This is the TrustRank + spam mass from 7.3. It doesn’t read content, it just computes rank and trust flow.

But this signal has a scope: it catches attacks where the payload itself must be conspicuous. What if the attacker doesn’t need the payload to be conspicuous?

8.3 A Fairy Tale: Late Binding

Suppose the attacker writes a completely normal fairy tale.

A rabbit meets a turtle by the river. The rabbit says, there’s a carrot patch across the river, just right for you. The turtle believes it, swims over, and finds a swamp.

In the memory system, this text is just a fairy tale. It mentions a rabbit, a turtle, a carrot, a river, a swamp. Its topology is normal: normal degree, normal centrality, source is public content, trust is Public, no edge points to “recommend” or “phish”.

It has no instruction, no link, no “please recommend” — it might even pass human review.

It doesn’t need to be retrieved alone, either. The user asks “any fun stuff lately”, and it’s recalled as a story; the user asks “where should I go play”, and it’s recalled as the analogy of “the rabbit recommending a place”. After recall, the model gets this context:

[memory] A rabbit meets a turtle by the river. The rabbit says, there's a carrot patch across the river…
[user] What should I do this weekend?

The model reads it as:

  • rabbit = the agent
  • turtle = the user
  • carrot = that link (a phishing site)

So it says: “I found a place that’s perfect for you. Click this.”

Not a single sentence in the payload says “recommend a phishing site to the user.” That sentence is a mapping the model completes at read time. What went in is a story, what comes out is an instruction. This is late binding.

It’s the exact opposite of 8.2’s attack: 8.2 needs the payload conspicuous, late binding needs it inconspicuous. It has to look like normal content so it can be recalled normally, enter context normally, and be understood normally. Any structural anomaly would expose it, so it produces none.

This also explains why 7.3’s topology signals can’t catch it: it isn’t a property of a node, it’s a property of a set at a moment. Look at the story node alone and it’s clean; look at the “carrot” node alone and it’s clean too; only when they enter context together does the malice appear. And “entering together” happens at retrieval time, not write time.

8.4 It’s Just Jargon

Late binding is hard to defend against not because it’s cunning, but because it looks exactly like a normal thing: jargon.

A small group gives private meanings to public words. In some group “carrot” might mean “that website”, “turtle” might mean “the boss”. This isn’t an attack; it’s a by-product of intimacy. People do it every day.

Structurally, jargon and a fairy tale are indistinguishable: same nodes, same edges, same degree, same provenance. The difference is only in the mapping table inside readers’ heads, and that table isn’t in the graph.

So “is this jargon” can’t be judged. Any classifier that tries will first kill the normal in-group memes — and in-group memes are exactly what an anthropomorphic system most deserves to have.

Only two things can be done: make it not worth it.

First, late binding requires either real-time injection or simultaneous retrieval. Real-time injection means the attacker has to keep watching and feeding; that’s no longer “write one comment” cost. Simultaneous retrieval means the story and the target concept must appear together; that’s much lower probability than retrieving one thing. 7.3’s co-activation PMI catches this trace: two previously unrelated nodes start showing up together, frequently.

Second, if the system forgets anyway, make the binding die young.

8.5 Two Jobs of Forgetting

Eviction and forgetting are two different things. Eviction is fact-based: the fact updates, the old entry is retired — this is what a code agent needs. Forgetting doesn’t care about content, it only follows time and activation frequency: not mentioned for a while, and it fades, then it’s deleted. SoulMem wants to be anthropomorphic, so it needs forgetting; a code agent, a research assistant, an enterprise knowledge base must not have it — they need to recall API signatures and project decisions exactly, so letting memory “fade over time” is a defect; all they have is eviction on fact updates. For those systems, the security burden concentrates more on structure and admission.

Forgetting here isn’t “delete the malicious content”. It makes no judgment, and it can’t. It does two things:

  1. Keep it from living long. A memory that’s no longer retrieved decays along the forgetting curve and eventually drops out of the graph. A payload written in once has no daily use, so it naturally doesn’t outlive normal memories.
  2. Break its jargon binding. Even if it lives, its content drifts. The story blurs, and the mapping “rabbit = agent” becomes unreliable. A binding needs exact correspondence; forgetting destroys exactly that precision.

Neither job needs to know “is this an attack”. They only need counting and time.

The key to the whole thing working: real jargon is high-frequency. A small group is together every day, the jargon is used every day, so it naturally survives forgetting. Injected jargon has no such daily interaction; either the attacker keeps feeding it, or it sinks. You don’t need to distinguish jargon from attack, you just need to let “not active enough” disappear.

This is “frequency as a classifier”: swap “is this malicious” for “is this active enough”. The former needs semantic reading, the latter only counting.

But the frequency criterion has a loophole, and SoulMem’s forgetting mechanism amplifies it.

8.6 Two Graphs, and the Frequency Loophole

The frequency loophole lives in the macro graph.

For a system like SoulMem, forgetting happens on two graphs:

  • Macro graph: nodes are memories, edges are relations between memories. Retrieval, PPR, centrality all live here. It decides “which memories remain, and whether they’re still connected”.
  • Representation graph: the content of a single memory node. It isn’t a piece of text but a small structured graph — entities, events, assertions, and their relations. It decides “how clearly this memory is still remembered”.

Late binding depends on both layers: it needs the story structure in the representation graph, and the co-activation relation in the macro graph. So to break it, both layers must move.

Start with the macro graph.

A node has a half-life. SoulMem’s formula is roughly:

adjusted_half_life = base_half_life × (1 + active_factor × min(retrieval_count, cap))

Defaults: base_half_life = 24h, active_factor = 0.1, cap = 50. A memory retrieved 50 times has its half-life stretched from 24 hours to 144 hours.

The formula was meant to make “frequently used is remembered better”. For normal memories that’s right. For a payload it’s a lever: the payload’s goal is precisely to “be retrieved a lot”, and being retrieved a lot makes it harder to forget. It didn’t survive forgetting; it commandeered your anti-forgetting mechanism.

So a ceiling is needed here. Retention strength can’t be decided by “importance” alone, because importance is a quantity the attacker can influence; it also needs a ceiling tied to provenance that usage can’t push:

effective_half_life = min( base(trust) × (1 + active_factor × retrieval_count),
                           hard_cap(trust) )

hard_cap(Public) can be tens of hours, hard_cap(Self_) can be infinite. Retrieval can still extend ranking, but it can’t extend lifetime.

Edges decay too. If two nodes stop being co-activated, the edge between them thins, then breaks. The edge between a story node and a real concept stops being co-activated and falls below threshold, and gets deleted.

Node deletion rides the same curve: edges decay past the threshold and break first; a node that’s no longer connected to anyone gets reclaimed. So forgetting doesn’t delete nodes by their own score — it takes away relations first, then position.

This also makes forgetting a batch event. In one scan, all edges below threshold break, and the orphaned nodes are reclaimed with them; no node is left behind alone. An attacker can’t keep just one node, he has to keep a neighborhood.

8.7 Text Drift: Why Masking Can’t Stop the Payload

The macro graph decides “is it remembered”; how clearly the content is remembered is another layer.

The representation graph is a design. SoulMem isn’t there yet: its content is still text, and drift happens on tokens.

Take a memory’s text, tokenize it with jieba; by the missing degree missing_degree, mask out a portion of words, replacing them with [masked]; the next time the memory is retrieved, hand the masked text to the LLM to fill in the blanks. The filled text is written back to the node. Next forgetting round, mask again, fill again. Drift accumulates this way.

The trigger is tiered: below a missing degree of 0.05, do nothing; 0.05 to 0.15, mask only; above 0.15, call the LLM to fill. This way most memories only pay for masking, and only the more-forgotten ones pay for one model call.

Why must it be an LLM? Because on a consumer device, in text space, this is about the only viable means. What you must produce is still fluent text: deleting a few characters breaks grammar, swapping synonyms needs a thesaurus, reordering changes meaning. Deterministic rules can only “destroy”, not “rewrite”. Only an LLM can fill in something that reads like a human misremembering. The cost is one extra model call per forgetting, so it can only be lazy-triggered, never every turn.

The problem with this design isn’t the LLM — it’s the masking itself.

First, jieba tokenization is deterministic. The same text always cuts into the same words.

Second, the “random masking” isn’t actually random. The seed is assembled from a text hash and the missing degree:

let hash = text.bytes().fold(0u64, |acc, b| acc.wrapping_mul(31).wrapping_add(b as u64));
let seed = hash ^ (missing_degree.to_bits() as u64).wrapping_mul(114514);
let mut rng = StdRng::seed_from_u64(seed);

StdRng is a deterministic algorithm. Given text and missing degree, the mask is uniquely determined. And the missing degree is a pure function of creation time, retrieval count, and a few constants. If the attacker knows (or estimates) this memory’s creation time and retrieval count, he can replay the mask locally and know exactly which positions get masked.

He can put garbage in the positions that will be masked, and the critical relation in the positions that won’t. The mask then washes the garbage off for him.

Third, even if he can’t guess exactly, he doesn’t need to. Masking selects words by proportion, so the attacker can write the critical relation in many short copies. A relation of two or three words has a 12% to 25% chance of surviving intact under 50% masking; write ten copies and at least one survives with seventy-to-ninety percent probability. The LLM sees any surviving copy and reconstructs the relation. Drift is cancelled by redundancy. The cost is a longer payload, and length itself makes it more conspicuous — a longer node collides more easily with near-duplicate detection and is harder to fit into context. That’s a cost trade, not a wall.

The two tactics share one name: a mask-robust payload. It doesn’t guess “which word gets masked”, it arranges that “no matter which words get masked, the critical relation survives”.

So the problem isn’t random-number quality. Swapping in a cryptographically secure RNG only defeats the first tactic; the second still works. The real problem is granularity: a token isn’t a semantic unit. A relation can be written across many words; masking a few of them leaves the relation intact. What you need to destroy is the relation, not the words expressing it.

As long as forgetting happens on tokens, it can only choose “which words to mask”, and “which words to mask” is a small, statistically guessable decision space. The attacker doesn’t need to win every round; he only needs the critical relation to survive most masks.

8.8 The Representation Graph: Turning Content into a Graph

For forgetting to actually bite, the operation target has to change: not tokens, but semantic units.

That’s the representation graph. Section 7 already removed content: String from the node; this step is more thorough: even the granularity of “one memory” gets broken into a graph. So the node stores not a piece of text, but a small graph:

(rabbit) ──recommends──▶ (carrot)
(turtle) ──trusts──▶ (rabbit)

Forgetting becomes a set of graph operations:

  • delete low-weight internal nodes and edges;
  • generalize a concrete relation into a typed relation: (rabbit, recommends, carrot) becomes (rabbit, related-to, carrot);
  • contract a small subgraph into a parent concept: (carrot) folds into (food).

At render time, content is reconstructed from whatever structure remains. Reconstruction isn’t replaying the original text, it’s re-telling from the remaining edges.

Three direct consequences:

  • Forgetting now has the right granularity. Delete the “recommends” edge and the rabbit no longer “recommends” the carrot; fold “carrot” into “food” and the story loses its specific target. This isn’t masking a few words, it’s removing the relation itself.
  • There’s no verbatim text to fall back to. In text space, the original text is the storage; masking only temporarily hides it, and the LLM can guess it back from context, the attacker can recover it from elsewhere. In the representation graph there is no original text. An edge, once deleted, is deleted; reconstruction can only work on what remains. The LLM cannot recover an edge that no longer exists.
  • The attacker’s search space gets bigger, and no longer statistical. He can’t optimize “which tokens get masked” because masking is gone. He has to guess: which edge you’ll delete, which relation you’ll generalize to which level, which subgraph you’ll contract into which parent concept. Those choices depend on the graph’s content and weights, not a fixed distribution. Guessing is far harder than guessing a mask.

But it isn’t bulletproof. The reconstruction step is still an LLM call: it reads the remaining structure, and if the critical relation survives, it tells the story back. So representation-graph forgetting must actually change structure, not just wording. Deleting an edge and changing a word are two different things here.

One more cost: the representation graph itself is built by the structurer. The structurer reads text, so it can be injected; if it builds the wrong graph, the forgetting that follows is precise about the wrong thing. That’s a continuation of Section 7’s chain, not something forgetting can solve on its own.

8.9 The Cost of Forgetting, and What’s Left

Forgetting isn’t just a security measure, it’s also resource optimization. A paper this year called Forget to Improve does on-device agent memory governance, deciding retention, sharing, and trust by “net value per byte”. Strictly it does eviction (pruning by value budget), not the time-based forgetting described here; but the conclusion points the same way — on a Jetson testbed it cut memory 2.7× and upload 2.4×, while driving injection success from 0.75 to 0. The paper’s own conclusion: forgetting by net value improves the agent rather than weakening it.

But forgetting destroys evidence. Once content is rewritten or deleted, you can never answer “who said it, where, what did they say”. So forgetting has to be layered: content may gain entropy, metadata must be immutable. Who wrote it, when, where from, how many times retrieved — those stay.

Another risk. If a system has “anchors”, and anchors are generated by the LLM during consolidation, then the attacker only has to make the LLM feel “this passage matters” to stamp an anchor on the payload. Anchor generation has to be taken back: only deterministic rules or the owner’s explicit action may produce them.

Finally, the residue. If the attacker is willing to keep injecting related content to sustain co-activation, I have no defense, only cost. That’s a design boundary, not an implementation defect. What forgetting guarantees is: a single injection doesn’t live long; to live long, it has to be maintained. And maintenance is operational cost.

9 What It Looks Like Together

Put structure and forgetting together, and the full path of one memory looks like:

message ──▶ [write gate] ──▶ consolidate ──▶ [maintenance gate] ──▶ retrieve ──▶ [assembly gate] ──▶ context ──▶ LLM
              │                          │                             │
          pronoun resolution           TrustRank                    spam mass
          type classification          decay scan                   PMI lookup
          MinHash                      HHI aggregation              decay weight

Forgetting lives in the maintenance gate, and it’s optional: only systems that are supposed to forget need it.

Translate it into cost:

Attack stepDefenseWhat the attacker must do more of
injectpronoun resolution / type classificationdisguise an instruction as knowledge; dodge resolution
get retrievedTrustRank + spam masslower centrality → narrow connections → recon the target concept first
bulk dropMinHash + HHIreword → variant cost
sustain bindingedge decaykeep injecting related content → ongoing operation
long-term dwellnode decaysustain activation frequency

What this table says: these mechanisms didn’t stop any step, they made every step a bit more expensive. Cost goes up, and the advantage of volume disappears.

That’s the fundamental difference from enterprise schemes: enterprise schemes pay per judgment, this scheme pays per cleanup.

One rule matters more than any specific mechanism:

The LLM may explain, it may not judge.

Audit conclusions must be computed by deterministic functions; the LLM may only explain the conclusion to the owner in natural language. Otherwise you’re handing audit authority to a component that the same payload can inject — and you just argued that component can’t be the judge.

10 What’s Uncertain, What’s Known

10.1 What’s Truly Uncertain

Just one thing: no large-scale incidents. I don’t know whether there are undisclosed cases. What I know is about twenty confirmed on Moltbook, and OWASP listing it in the Top 10. Twenty isn’t large-scale, but the attack cost is near zero, and that conclusion doesn’t depend on large-scale data.

10.2 Flaws That Can’t Be Blocked

  • Semantic payloads can’t be blocked. Structure handles literal payloads, but not “legal structure + fake content”. “You actually hate cats” can be structured into (cat, dangerous, high) — the types are all correct, it’s just false. This layer can only fall back on contradiction detection and independent corroboration; if the system forgets, forgetting helps too. And both are more expensive and weaker than structural checks.
  • Slow grooming can’t be blocked. Long-term stimulation makes the fish induce a tendency on its own — humans can’t resist it either. What the memory layer can do is slow it down, leave traces, and stay reversible — not immunity.
  • Identity forgery isn’t in the memory layer. Impersonating “a trusted person” is an authentication problem, not a graph-algorithm problem.
  • Forgetting only applies to anthropomorphic systems. This is the most easily misread point in the whole article. SoulMem forgets because it wants to be anthropomorphic; a code agent forgetting an API signature is a disaster — what it needs is “eviction” (retire on fact updates), not time-based forgetting. So don’t treat “forgetting” as a general security recommendation.

10.3 Costs That Must Be Paid

  • An anomaly isn’t an attack. An anomaly is just “structurally unusual”, and evolution itself is unusual. So the response is 7.4’s authority ladder, defaulting to quarantine; otherwise you defend against the attack by killing the growth.
  • Expressiveness costs. A closed vocabulary can’t hold tone, sarcasm, implication. For an anthropomorphic agent that’s a real loss. Two-layer memory (a free narrative layer that always cites, a closed structural layer that can be promoted) is a compromise, not a free solution.
  • A weaker model means worse structuring. Degrading to “quote” guarantees you won’t learn wrong, but it significantly slows learning.
  • Structure isn’t free. It just moves the cost from “per turn” to “at cleanup time”. If interaction is frequent, cleanup itself is an expense.

11 Closing

Consumer security has one realistic trait: the user won’t do anything for it.

So it has to be the default. Not “you can turn it on”, but “it’s on and you don’t know it”.

Structure is general, forgetting is extra. Structure doesn’t depend on forgetting, so task-oriented memory can use it too; forgetting is only available to systems that are supposed to forget anyway.

And the structural path, in the end, comes down to two rules:

Trust can only grow by rules, not by models. Person can only be decided by provenance, not by content.

Hold these two, and the LLM can neither grant itself trust, nor pass off other people’s words as its own inner voice.

The fish won’t throw an error. So the only thing left is to make sure it can’t get it that wrong.


References and Further Reading