LLM 0.32: reasoning traces in the log DB, server-side tools, and the eval determinism problem

Short version: LLM 0.32 is worth the upgrade for the logging alone. Reasoning traces landing in the same SQLite database as everything else means you can finally diff a bad answer against the thinking that produced it without bolting on your own wrapper. But server-side provider tools are not a drop-in replacement for a local eval harness, and the reason is determinism, not features.

I upgraded with `uv tool upgrade llm`, ran the schema migration on an existing logs.db with about 40k rows, and nothing blew up. That's the boring part, and boring is the correct outcome for a CLI you've had in your shell history for two years.

What the reasoning traces actually give you

Reasoning traces in LLM 0.32 are stored alongside the response, not printed and thrown away. That's the whole point. Before, if you wanted to inspect a reasoning model's intermediate output you either watched it scroll past in the terminal or wrote your own capture layer around the API client.

Now `llm logs -n 1 --json` gives you a record you can grep, and because the store is plain SQLite you can just open it in Datasette or query it directly. I ran a batch of 60 classification prompts through a reasoning model and pulled the traces into a scratch table to see which failures shared a bad intermediate assumption. Three of the five wrong answers had the same misread of the instruction in the trace. That took a `SELECT` and about four minutes. Previously it was a Python script I'd rewrite every six months.

One thing to keep in the front of your head: reasoning tokens are billed tokens. Having the traces visible in your log DB makes the cost side legible for the first time — you can actually sum reasoning tokens per model per week instead of squinting at a provider dashboard. That's a bigger deal for budget conversations than it sounds.

Where the plugin surface bites

The plugin ecosystem is LLM's best feature and its most reliable source of upgrade pain. Provider plugins pin against the core's model and tool interfaces, and a release that adds a new capability class — server-side tools, in this case — means plugins that haven't been touched in a while will either ignore the new surface or throw on it.

My advice: after upgrading, run `llm plugins` and actually look at the list, then re-run one representative prompt per provider before you assume your scripts still work. I had one third-party plugin that loaded fine and then failed the moment a tool was attached, which is the worst failure shape because it passes your smoke test.

Do server-side tools replace your eval script?

No, and I'd push back on anyone who says otherwise.

Server-side tools through the OpenAI Responses API mean the provider executes web search or code execution on their side and hands you the result. Great for interactive use. Bad for evals, because you lose the interception point. My eval harness works by mocking every tool call so the same input produces the same output on every run — that's the only way regression numbers mean anything. When the tool runs on the provider's infrastructure, I can't mock it, can't replay it, and a live web search makes yesterday's baseline meaningless.

So the split I've settled on: server-side tools for exploration and one-off research prompts, locally-defined tools via `-T` for anything that feeds a scored eval. The upgrade didn't replace my eval script. It replaced the logging half of it, which was the ugly half anyway.

Verdict

Upgrade. The trace logging pays for itself the first time you debug a reasoning model failure, and Simon Willison calling it the most significant release since launch is not overstated — [Willison's release notes](https://simonwillison.net/2026/Aug/4/new-release-of-llm/) are worth reading in full before you touch your plugin set.

Just don't rip out your eval harness. Keep local tools for scored runs, use server-side tools for the interactive lane, and budget an hour for plugin breakage.

FAQ

Does upgrading to LLM 0.32 break my existing logs.db?

No. The upgrade runs a schema migration in place. I migrated a ~40k-row logs.db with no data loss, but back the file up first — it's one `cp` and the database is a single SQLite file.

Can I use server-side tools in an automated eval pipeline?

You can, but you shouldn't for scored runs. Server-side tools execute on the provider's infrastructure, so you can't mock or replay them, and any live tool like web search destroys run-to-run determinism. Use locally-defined tools for evals.

How do I inspect reasoning traces after a run?

Query the log database. `llm logs -n 1 --json` returns the most recent record including the trace, and because storage is plain SQLite you can also point Datasette or any SQL client at logs.db and aggregate across runs.