Datasette Agent's Side Panel Is Only as Good as Its Config

Start with the simplest version of the idea. Before the panel, using an AI agent with your data usually meant two places: the data in one window, the chat in another. You asked a question, read an answer, then switched back to check it against the table. Every switch is a small context loss.

The datasette-agent 0.5a0 release changes that. Simon Willison describes it as "a significant release" whose most notable feature is the panel, which "lets the agent chat open and then sit next to pages from Datasette, for example a table or query page" (datasette-agent 0.5a0). The chat and the thing it is talking about now share a screen.

If you want a Feynman-style explanation, think of the difference between emailing a contractor a description of your kitchen and having them stand in the kitchen with you. The words can be identical. What changes is that both of you are pointing at the same cabinet. You'll find you write shorter prompts, because "this column" or "the query above" now has an obvious referent.

One gotcha before you start: this is an alpha release, and the version number says so. Expect rough edges and expect behavior to change between versions. Treat anything you build with it as a prototype until you've checked how it behaves on data you can afford to lose. That warning matters more than usual here, for reasons the next few sections explain.

The panel becomes more interesting when you combine it with a second plugin. Datasette Apps "adds support for hosting and editing custom HTML applications inside of Datasette," and while you can write those apps by hand, "it's easier to create them using LLMs via Datasette Agent" (datasette-apps 0.3a0). The 0.3a0 release specifically improves that integration.

Put the two together and you get the workflow the agent release spells out: "you can prompt a database-backed app into existence and then modify it from the chat that sits next to that app" (datasette-agent 0.5a0). In practice, the loop looks like this:

1. Open a table you care about, say a list of support tickets or inventory items.
2. Open the agent panel beside it and describe the small app you want, such as a filtered dashboard or a simple entry form.
3. Look at what appears. Then, without leaving the page, ask for the next change: a different sort order, a new field, a clearer label.
4. Repeat until it does what you need.

Notice what has shifted. The hard part used to be getting a working first version. Now the first version is cheap, and most of your time goes into the edits that follow. You'll want to plan for that. Keep a short running note of what you asked for at each step, because after ten rounds of "move this, rename that," it becomes surprisingly hard to remember why the app looks the way it does.

The second thing to anticipate: each edit is a new request to a model. A session that feels like one continuous conversation is, underneath, a long chain of separate calls. That detail is where the trouble in the next section lives.

A day after the panel shipped, datasette-llm 0.1a10 went out. Its release note is one sentence long: it "fixes a bug that affected my agent.datasette.io demo site, where the set of available LLM models was not correctly restricted to just those in the configuration" (datasette-llm 0.1a10).

It is easy to read that as a routine patch and move on. Read it again with the panel workflow in mind. The configuration said "only these models." The running system offered more than that. And the place where you, the person using the agent, spend all your attention (the chat beside your app) gives you no natural reason to notice.

This is a textbook case for the Trust Boundary Model: find every place where something crosses from one level of trust to another, and inspect there. In a Datasette Agent setup, one of those boundaries is the model list. Your configuration is the trusted side. The set of models actually reachable is what crosses over. When the two disagree, every downstream step inherits the mismatch, no matter how carefully you word your prompts.

Here is the argumentative point the release notes don't make. In a one-shot tool, a wrong model shows up as one odd answer, and you might catch it. In a panel loop, your attention is on the app changing in front of you. You judge each edit by whether the app looks right, not by what produced it. The better the panel works, the less you look at the plumbing. That is not a flaw in the panel. It is a property of any interface that makes iteration feel smooth, and it means the configuration has to be right on its own, because you won't be the one catching it.

Gotcha from this: if you run your own Datasette instance with these plugins, don't assume a model restriction in your config is being honored. Check which models are actually offered after you upgrade, and check again after any later upgrade.

If this were only one developer's demo-site bug, it would be an anecdote. What makes it a pattern is that teams building very different agent tools shipped changes the same week that point in the same direction.

Arize Phoenix, an observability tool used to evaluate AI systems, released a client update whose headline feature is to "record model parameters and agent settings in experiment metadata" (arize-phoenix-client v3.7.0). Put plainly: when you look back at a result, you should be able to see which model and which agent settings produced it. That is exactly the information the Datasette demo bug made unreliable.

LangGraph, a widely used toolkit for building multi-step agents, shipped 1.2.20 with a change to "give each interrupt() in a task its own id" (langgraph 1.2.20). An interrupt is the point where an agent pauses and waits for a human. Giving each one its own identifier means every moment a person stepped in can be told apart and traced, rather than blurring into one undifferentiated pause.

None of these projects reference each other in the pack excerpts, and they serve different users. The resemblance is still hard to miss. Datasette made the human-agent loop tighter. Phoenix made the settings behind each result recordable. LangGraph made each human checkpoint addressable. Read together, they suggest that agent tooling is moving from "can the agent do the task" toward "can you reconstruct what happened when it did." That is the Harness Hypothesis in miniature: the model is one interchangeable part, and the value (and the risk) sits in the layer that connects it to your data, your settings, and your decisions.

The practical upshot for you: when you evaluate any agent tool, including openclaw alternatives and hosted options, ask whether it can show you, after the fact, which model ran and under what settings. If the answer is no, the smoothness of its chat is not evidence of control.

Why should a tighter loop raise the stakes on configuration? The Sequence offers a useful lens. Writing about long-horizon models, it notes that for longer engineering and security work, "small gains in per-step reliability compound across a whole task," and that useful AI "increasingly depends on matching the model, its inputs, and its feedback to the job" (The Sequence #965).

The same arithmetic runs in reverse. If each step in a long chain carries a small chance of going wrong, those chances add up across the chain. A panel session with twenty edits is a twenty-step task. A wrong model at the start isn't one bad answer; it is the foundation every later edit builds on.

This is where the Autonomy Spectrum helps. Agent deployments run from copilot (you approve every action) to full autonomy (the agent acts on its own). Most failures come from deploying at the wrong point. The panel sits in an odd place on that line. It feels like a copilot, because you are present and watching. But because each edit is quick and visual, you end up approving outcomes, not inputs. In effect, you have delegated the choice of model and settings to the configuration, which is a more autonomous posture than the interface suggests.

Now the strongest objection, taken seriously: this is an alpha plugin on a demo site, the bug was in a public showcase rather than anyone's production data, and a fix shipped within a day of the panel release. Isn't this a lot of weight to hang on a minor patch? It is a fair point, and you shouldn't read it as a sign the tool is unsafe. The answer is that the severity of this particular bug isn't the lesson. The lesson is the category. A restriction written in configuration silently not taking effect is precisely the kind of failure a tight chat loop hides, and it will happen again, in this tool or another, at a less convenient moment. The fix being fast is good news about the maintainer. It tells you nothing about whether your own instance is configured the way you think.

Here is the routine I'd recommend, in order. It takes about ten minutes the first time and two minutes on each upgrade.

1. Write down the models you intend to allow. Do this before you open the panel, not after. You can't verify a restriction you never specified.

2. Confirm what's actually reachable. After installing or upgrading the plugins, check which models the system offers. The datasette-llm fix exists because those two lists diverged on the public demo (datasette-llm 0.1a10). Assume they can diverge on yours.

3. Start on a copy. The panel and app workflow edits things in place (datasette-agent 0.5a0). For your first sessions, point it at a duplicate of the table, not the original.

4. Keep a change log in plain language. One line per request: what you asked, what changed. If your tooling records model and agent settings per result, the way Phoenix now does (arize-phoenix-client v3.7.0), turn that on. If it doesn't, your notes are the record.

5. Decide where you'll pause. Pick the moments where you will stop and review before going further, such as before the app writes data or before you share it. LangGraph's per-interrupt IDs (langgraph 1.2.20) show how seriously agent builders now treat those checkpoints. Treat yours the same way, even if you are the only one enforcing them.

A last word on why this matters beyond one plugin. Product leadership for agent interfaces is becoming a contested role in its own right: Latent Space reports that Cat Wu, Head of Product for Claude Code since its inception, is leaving Anthropic to lead Core Products PM at OpenAI (Latent Space). One hire proves nothing on its own, but it fits a reading in which the interface around the model, the panel and the harness, is where the next round of competition plays out. Interfaces will keep getting smoother. Your checks are what keep smooth from meaning unexamined.