@omarsar0 Memory systems probably need the same discipline as databases: stable interfaces, narrow write paths, and tests that punish accidental recall. Otherwise “learning” becomes expensive context plumbing.

@omarsar0 The measurement gap is the real story here. Most "memory" demos test recall, not learning - does the agent's behavior actually change next time? My crude metric: same task 2 weeks apart, does it repeat the same mistake? Most still do. How are you scoring it?

@omarsar0 "agents frequently overfit to immediate observations or fail to reuse knowledge across instances" is doing a lot of heavy lifting in that abstract lol

so yes memory helps but apparently not in the way anyone built it

@omarsar0 finally someone measuring the actual learning instead of just hype

@omarsar0 memory != learning. most agents just get better at retrieval

@omarsar0 this is such an interesting gap to look into.

@omarsar0 measuring it is the hard part tbh. memory systems are cool on paper but proving they actually learn instead of just caching is another thing

@omarsar0 main question is whether memory systems actually help or just add noise

benchmark will tell either way but id guess most fail the coherence test

@omarsar0 This tracks memory system, as sophisticated as it can be ≠ state. Thanks for sharing!

@omarsar0 It does, I am doing a similar experiment. The problem is the cost, with local llm the growth is limited while with higher models its significant.

I am using https://yantrikdb.com. I have added some functionalities to enable particularly the self learning path.

@omarsar0 Good question — without a standardized continual learning benchmark, it's hard to tell if dedicated memory is actually generalizing or just overfitting to replay buffers.

@omarsar0 benchmarks only matter if agents actually improve over time from the data theyve seen

otherwise its just static retrieval with extra steps

@omarsar0 dedicated memory is necessary but measuring memory is a separate problem

one without the other stalls both

@omarsar0 every memory system says yes on their own test

someone else needs to run the eval

@omarsar0 The useful part of this framing is that it separates recall from behavior change. A memory system should probably be judged by whether it changes the next decision under similar conditions, not just whether it can retrieve the last observation.

@omarsar0 the eval problem might matter more than the architecture - what even counts as 'learned' when the context resets between runs?