Problem
Debugging an LLM agent means answering questions a normal log can't: which tool call went wrong, what the prompt looked like three versions ago, and whether a change actually improved retrieval or just moved the failure somewhere else.
Approach
AgentOps is the tool Hasan uses to trace his own agent work:
- Full traces of every tool call, with failed runs replayable step by step.
- A retrieval evaluation harness that scores chunking strategies against a labelled question set.
- A streaming trace viewer in React with virtualised rows, so runs of more than ten thousand spans stay navigable.
- A single-container deployment with ClickHouse for span storage and sub-second queries.
Stack
Python · FastAPI · ClickHouse · React
Outcome
Placeholder — AgentOps is a working side project rather than a released product. If it grows into something public, adoption and benchmark notes go here (SPEC §13).
It exists because the alternative was reading raw JSON logs, and the evaluation harness is what makes prompt changes measurable instead of vibes.
Links
- Repository — to be added once the repo is public