Benchmark
01Abstract
The top entries on text-to-SQL benchmarks like Spider 2.0 are built around a semantic layer. But no benchmark tests how well that layer is built. They either give the agent a ready layer or none at all. We wanted to measure the layer itself.
We took a Shopify schema and its dbt project: 118 tables and 2,358 columns. Schema and dbt alone miss how the data is used, so we added dashboards and queries by hand, including:
- Queries that forget critical filters
- Competing definitions of the same metric
- Rules recorded in one comment and nowhere else
Each of these became a test case: a trap the model has to avoid, or a filter that has to be attached to the table it belongs to.
This is the size of a mid-sized company's warehouse, and the full corpus still fits in Claude's context window.
We tested several Claude models running in Claude Code, and Shanc. Each built a context layer of table and column docs, metrics, joins and filters.
02Accuracy drops as scope grows
We started building the benchmark with Opus 4.8. On a narrow slice (rules and metrics only), it passed 100% of the tests. As the scope grew to the full warehouse, it started trading coverage against depth without saying so. With one paragraph of the prompt changed, one run passed 78% of the tests but documented only 22% of the columns; the other documented every column and passed 73%. Across ten runs, 78% was its best, even though the full corpus fit in the context window. We expect bigger drops once the corpus is larger than 1 million tokens.
We then ran the full benchmark on Opus 5, Opus 5.5 and Shanc:
| Opus 4.8 | Opus 5 | Opus 5.5 | Shanc | |
|---|---|---|---|---|
| Accuracy | 78% | 87% | 85% | 99% |
Shanc splits the corpus into smaller tasks along the schema's relationships. For example, tables that are used together are summarized together. Each task stays small enough for Claude to handle well, and this runs automatically, with no prompting when something changes.
03Claude doesn't notice changes
Given a real changed file, Claude updated every affected page, rejected a change that didn't apply and flagged a contradiction we had missed.
The problem is that nothing tells it to look:
- No trigger. Someone has to notice that the dbt model changed.
- No completeness guarantee. It reconciles what it reasons about, but it can't prove it found everything.
- No lasting record. Each edit is one-off, so next quarter you start from zero.
You can build the trigger, scope resolution, scheduling, review step and version history yourself. At that point you're maintaining an internal SaaS.
04Unstructured docs can't be enforced
Prose has no schema, so nothing can check it.
When we ran the same prompt twice, the two documents disagreed. Across identical runs the customer filters were missing, then correct, then written in the wrong place. Two runs disagreed on what a base table is. Someone reading those docs can't know which version they got.
In three separate runs, when asked for full coverage, the model stopped writing prose and built itself a harness: dictionaries for each table and column in /tmp, plus templates to render them. It generated the documentation from that store. Then it deleted the store and shipped only the markdown.
To hold every fact, the model needed a structured store, and the markdown was just output. Shanc keeps that store instead of deleting it.
05Claude works well on smaller projects
The only run that ever scored 100% was the smallest one. There was less to know, and it avoided every trap.
That result holds, and it's what we recommend. When there's little to write down, one careful run gives you a good context layer.
As the knowledge grows, at some point Claude starts making mistakes, and you have no way to know when.
The limit is the system around the model.