Spec-Driven Agentic Engineering for Engineers

Every structural engineer already knows how to trust work they cannot fully re-check.

You don't re-derive every equation in the code, and you don't hand-verify every number a finite element package returns. What you do instead is bound the uncertainty: you know roughly how wrong each input can be, you know your margins, you check the answer against a hand calc or a sanity limit, and you get a second set of eyes on the parts that matter. Trust, in engineering, is a system for catching the failures you can't personally inspect.

AI-assisted development has the same problem in a different costume. A language model writes code that compiles, passes lint, and looks right. Sometimes it is right. Sometimes it is confidently, quietly wrong, in a way that looks exactly like the times it was right. If you accept its output on vibes, you will ship the wrong thing eventually, and you won't know which time it was.

Most people respond to this by either trusting the model too much or refusing to use it at all. Engineers have a third option, because we already own the discipline for exactly this situation. The habits transfer.

Spec-driven agentic engineering is what I call that discipline applied to software built with AI agents. You write down what a correct result looks like before you prompt, you keep a set of cases with known answers and grade every change against it, you have a separate evaluator do the grading blind, you spend the capable model only on the step that governs the answer, and you tune parameters as a parametric study against the same cases. The agent does the work, and the spec and the checks decide whether the work is any good. The rest of this essay walks through each habit as it showed up in ETABuddy, a tool I now rely on every day.

Write the spec before the prompt

The first mistake is prompting before you know what "correct" means.

In design, nobody sizes a member before they know the loads, the code, and the governing case. The spec comes first, because it is what lets you tell later whether the answer is any good. Software with an AI in the loop is no different, and it is more tempting to skip, because the model will happily produce something plausible from a vague ask.

For ETABuddy, the spec was concrete before I wrote a single prompt. The tool diagnoses ETABS warnings. So: what is a correct diagnosis? It names the warning, states the likely root cause, and gives a fix an engineer can act on. What are the inputs? Real warning text, in the messy formats ETABS actually emits. What is out of scope? Anything that requires opening the model file. Writing that down changed what I built. A vague "help with ETABS warnings" would have wandered. A spec gave every later decision something to be measured against.

The rule is the same one you already follow: if you can't say what correct looks like, you are not ready to start.

Build a golden set

Once you know what correct looks like, you write down the cases where you know the answer.

Every engineer keeps a mental library of check problems: the cantilever whose tip deflection you can do in your head, the simple beam whose moment you know cold. When a new analysis method or a new tool shows up, you run it against the case you already know before you trust it on the case you don't.

That is a golden evaluation set. For ETABuddy I collected real warnings and wrote the known-correct diagnosis for each one by hand: the warning, the cause, the fix I would give a colleague. Not many at first, a couple dozen, but real ones, chosen because I already knew the answers. Every change to the tool got run against that set. If a "clever" improvement to the search made three golden cases worse, it wasn't an improvement, no matter how good the idea sounded.

This is the habit non-engineers skip most often, and the one with the most leverage. Without a golden set you are tuning on feel, and feel is exactly what the model is best at fooling. With one, every change produces a number, and you argue with the number instead of with your own optimism.

Get an independent review, and make it blind

You do not check your own drawings and call it reviewed. Someone who didn't do the design looks at it, precisely because they don't share your blind spots.

The trap in AI-assisted work is grading the output with the same model, or the same person, that produced it. It agrees with itself. So I made the evaluation independent: a separate agent, given the golden answer and the tool's answer, asked to judge whether they match, with no stake in the result and no memory of having produced it. A blind third party.

That one move caught things I would have waved through. When the author and the reviewer are the same process, "close enough" wins. When the reviewer is blind to who wrote the answer and is only comparing against the known-correct case, close enough stops being good enough. It is the software version of handing your calcs to the engineer across the hall who has no reason to be kind to them.

Route the work by how hard it is

You don't send every question to the senior principal, and you don't hand a bridge to a co-op student. You match the reviewer to the difficulty of the problem. That is triage, and it is a design decision, not a slight.

Models come at different capability and different cost. A fast, cheap model is fine for the easy, high-volume steps: normalizing a warning, deciding which bucket a question falls in. The expensive, capable model earns its cost on the hard step: the actual diagnosis, where getting it wrong is the whole failure. Routing by difficulty means the tool spends capability where it changes the answer and saves it where it doesn't.

The engineering instinct here is exactly right. Effort is a budget. You spend it on the governing case. ETABuddy runs the same way: cheap models do the sorting, the capable model does the judgment, and the budget cap on the whole thing means a diagnosis can't silently run up a bill. A diagnostic tool with no cost ceiling is a tool you stop trusting the first surprising month.

Let the search run overnight

Some things you don't solve by thinking harder. You solve them by running the case a few hundred times and reading the results in the morning.

The hybrid search in ETABuddy has parameters: how much weight keyword matching gets versus embedding similarity, how the two scores get normalized so they're comparable, how results get fused. There is no first-principles value for these. You could guess, or you could measure. So I let it optimize overnight: try combinations, score each against the golden set, keep what wins. I woke up to a configuration I would not have guessed and could not have argued my way to, backed by numbers on cases I already trusted.

This is a parametric study, the same thing you'd run to find the wall thickness that governs across a range of loads. The computer is patient and you are not. Set up the sweep, define the objective, and let it grind while you sleep. The only prerequisite is the golden set, again, because an overnight search without a trustworthy objective just finds the fastest way to fool your metric.

Where it still bites

None of this makes the model correct. The narrower, more honest claim is that the wrongness becomes catchable.

The golden set only covers the cases I thought to include. A warning I've never seen, in a format I didn't anticipate, can still draw a confident wrong answer, and the blind evaluator can only judge against cases it has. So the set grows. Every time ETABuddy surprises me, the surprise becomes a new golden case, the same way a near miss on a project becomes a new line on your personal checklist. The system is a ratchet that makes the tool a little harder to fool each time it fools me.

That reframing matters. Engineers don't claim their structures cannot fail. They claim the failure modes are understood and bounded. A tool built this way earns the same modest, defensible claim. When it is wrong I will probably catch it, and the next version will be wrong less often. That is the most you can honestly promise about work you cannot fully inspect, and it is enough to build on.

Why the habits transfer

All of this came from the way I already checked structures, carried across to software.

Every one of these habits is something a competent engineer already does, because being confidently wrong is expensive and you cannot personally inspect everything you are responsible for. AI-assisted development puts you in exactly that situation, and our profession is built around it.

The people struggling with AI tools right now mostly have the technology and lack the discipline. They have a powerful, plausible, occasionally-wrong assistant and no system for catching the "occasionally." Engineers have had that system for as long as the profession has had to sign off on work it could not inspect in full.

Plenty of things use language models. ETABuddy is reliable because I ran the same checks on it that I run on a set of drawings I can't fully inspect. Those checks are the transferable part, and the part worth writing down.