AI Agent Memory: Building Agents That Actually Remember Across Sessions

Ask most production chatbots the same question two days in a row and they answer identically both times - not because the answer is stable, but because the agent has no idea it ever talked to you before. Every session starts from zero. That is fine for a one-off Q&A tool and a serious limitation for anything meant to feel like an assistant that actually knows the user. Memory is the piece that closes that gap, and it is one of the least understood parts of agent architecture because it looks simple and is not.

Why the Context Window Is Not Memory

The most common mistake is treating the context window as memory. It is not - it is working memory at best, and it disappears the moment the session ends or the window fills up and older turns get truncated. A user who mentions their tech stack on Monday and returns Wednesday expecting the agent to remember it will find an agent that has completely forgotten, because that information lived only in a context window that no longer exists.

Real memory requires a separate persistence layer outside the conversation itself - something that survives session boundaries, gets retrieved selectively based on relevance, and gets written to deliberately rather than just accumulating everything that was ever said.

The Three Layers of Agent Memory

Most teams build working memory by default because it is just the conversation history. Session and long-term memory require deliberate engineering - deciding what is worth remembering, how to store it, and how to retrieve only the relevant pieces without flooding every future prompt with irrelevant history.

Architecture: Extraction, Storage, Retrieval

The pipeline has four stages. When a conversation turn completes, a memory extraction step (an LLM call that decides what is actually worth remembering) runs and writes results to storage - a vector database for semantic facts, a structured database for hard facts like account details. When the next session starts, a memory retrieval step runs semantic search plus recency and relevance scoring to pull back only what matters, and injects it into the system prompt as compact context rather than replaying old transcripts.

Step 1: Deciding What to Extract

Not every message is worth remembering. A well-designed extraction step runs after a conversation (or periodically during a long one) and asks an LLM to identify durable facts worth keeping - preferences, constraints, decisions, corrections the user made. Throwaway small talk gets discarded.

extraction_prompt = """Review this conversation and extract durable facts
worth remembering for future sessions. Only include facts that would
still matter days or weeks from now - preferences, constraints,
important decisions, corrections the user made to prior assumptions.

Ignore small talk, one-off questions, and anything session-specific.

Return a JSON list of facts, each as a short standalone sentence.

Conversation:
{transcript}"""

def extract_memories(transcript: str, llm) -> list[str]:
    response = llm.invoke(extraction_prompt.format(transcript=transcript))
    return json.loads(response.content)

This extraction step is what separates a memory system from just logging every message ever sent. Logging everything and searching it later sounds simpler, but it means every retrieval pulls in noise alongside signal, and noisy context degrades response quality more than having no memory at all.

Step 2: Storing Memories With Structure

Each extracted memory needs metadata beyond the raw text - when it was created, which user it belongs to, and a category that helps with later retrieval and conflict resolution:

class Memory(BaseModel):
    user_id: str
    text: str
    category: str  # preference, constraint, fact, correction
    created_at: datetime
    embedding: list[float]

def store_memory(memory: Memory, vectorstore):
    vectorstore.add(
        ids=[str(uuid4())],
        embeddings=[memory.embedding],
        metadatas=[{
            "user_id": memory.user_id,
            "category": memory.category,
            "created_at": memory.created_at.isoformat(),
        }],
        documents=[memory.text],
    )

The category field matters more than it looks. A new preference that contradicts an older stored preference should overwrite it, not sit alongside it as a second, conflicting memory. Without conflict resolution, a memory system accumulates contradictions and eventually starts feeding the agent inconsistent context.

Step 3: Retrieval - The Part Most Systems Get Wrong

Retrieval is not just semantic similarity search. A pure vector search over stored memories will happily surface a fact from six months ago that is only vaguely related to the current query while missing a highly relevant fact from yesterday because the wording was different. Production memory retrieval combines three signals:

• -Semantic relevance - how closely the stored memory matches the current query in meaning.
• -Recency - more recent memories are weighted higher, since user preferences and context change over time.
• -Access frequency - memories that get retrieved often across sessions are probably more central to who the user is, and deserve a small relevance boost.

def retrieve_memories(query: str, user_id: str, vectorstore, top_k: int = 5):
    results = vectorstore.query(
        query_embedding=embed(query),
        where={"user_id": user_id},
        n_results=top_k * 3,  # overfetch, then rerank
    )

    now = datetime.utcnow()
    scored = []
    for r in results:
        age_days = (now - r.created_at).days
        recency_score = 1 / (1 + age_days / 30)  # decays over ~30 days
        final_score = (0.7 * r.similarity) + (0.3 * recency_score)
        scored.append((final_score, r))

    scored.sort(key=lambda x: x[0], reverse=True)
    return [r for _, r in scored[:top_k]]

Injecting Memory Without Bloating the Prompt

Retrieved memories should be compact and clearly separated from the live conversation, so the model treats them as background context rather than instructions to follow literally. A short block of five to ten memories at the top of the system prompt, phrased as plain facts, works better than dumping raw conversation transcripts from prior sessions.

system_prompt = f"""You are a helpful assistant with memory of past interactions.

What you remember about this user:
{chr(10).join(f"- {{m.text}}" for m in retrieved_memories)}

Use this context naturally. Do not explicitly mention that you are
recalling stored memories unless the user asks how you know something."""

When Memory Actually Matters (and When It Doesn't)

Frequently Asked Questions

Should I build my own memory system or use an existing framework?

For most teams, an existing memory layer (several open-source options integrate directly with LangChain and similar frameworks) gets you 80% of the value with a fraction of the engineering time. Build a custom system only once you have a concrete reason the existing options do not fit - a specific conflict-resolution rule, a compliance requirement around data retention, or retrieval latency requirements an off-the-shelf vector store cannot meet.

How do you handle memory that becomes outdated?

Apply a recency decay to retrieval scoring as shown above, and run a periodic consolidation pass that asks an LLM to review memories for the same user and merge or discard ones that are now contradicted by newer information. Without active consolidation, memory stores grow stale and occasionally contradictory over months of use.

Does adding memory increase latency noticeably?

The retrieval step adds one vector search, typically 50 to 150 milliseconds, which is small relative to the LLM call itself. The bigger latency cost is the extraction step after each session, but that runs asynchronously after the user has already gotten their response, so it does not add to perceived response time.

Is storing user memory a privacy concern?

Yes, and it should be treated as any other personal data store - encrypted at rest, scoped strictly by user ID, deletable on request, and covered in your privacy policy. Building memory deletion and export as a first-class feature from day one is far cheaper than retrofitting it once you have real user data and a compliance deadline.