The Founder's Field Guide to Shipping AI That Works

Most AI deployments fail in a boringly predictable way: a demo that dazzles, a pilot that posts great numbers, and a rollout that quietly never happens — or worse, ships and then bleeds money in ways nobody modeled. The failure is almost never the model. It's the process around the model: the wrong use case, no real success criterion, an eval that was three cherry-picked examples, a human placed where they add friction instead of where they carry liability, and an ROI number extrapolated from a curated slice.

This is the playbook I use to avoid that, built from shipping AI into two businesses with opposite risk profiles — Velya, where AI agents qualify leads and handle intake for clinics, and Kommerce, a cash-on-delivery commerce OS where a wrong automated decision can put a real package on a truck to an address that can't pay. Eight phases, in order, each with a decision criterion or a checklist. The through-line: you are not building a feature, you are building a system that has to be right often enough, cheaply enough, on the cases that actually occur. Work the phases in sequence. Skipping one is how the demo-to-production gap gets you.

Phase 1: Find the high-leverage use case#

Before anything technical, you are doing triage. Most tasks are bad AI candidates and you want to reject them fast, on paper, before they cost you a sprint. Score each candidate task on five factors — call them the AI Leverage factors — from 1 to 5.

Factor	Question	Low leverage (1)	High leverage (5)
Repetition	How often does this task recur?	A few times a month	Hundreds of times a day
Data availability	Is the context the task needs reachable?	Scattered, tacit, in someone's head	Structured, logged, queryable
Error tolerance	What does a wrong answer cost, and can you undo it?	Expensive and irreversible	Cheap and reversible
Decision value	Is the output worth acting on?	Marginal, nice-to-have	Directly drives revenue or cost
Verifiability	Can you cheaply check if an answer was right?	Only obvious months later	Checkable in seconds

Two factors deserve more weight than the rest: verifiability and error tolerance. A task you can cheaply verify is a task you can eval, monitor, and safely automate. A task where errors are cheap and reversible is a task where the AI can be wrong sometimes without wrecking you. When both are high, AI pays even at mediocre accuracy. When both are low, a great demo is a trap — you won't catch the failures until they've compounded into real losses.

At Velya, "qualify an inbound lead for a clinic" scores high across the board: it recurs constantly, the context lives in the chat transcript and a form, a misqualification is recoverable (a human re-reviews before anything irreversible happens), and correctness is checkable against whether the lead actually booked and showed. At Kommerce, "decide whether to auto-confirm a cash-on-delivery order for shipping" scores dangerously: high repetition and decision value, but a wrong confirmation ships a physical package to someone who won't pay, which is expensive and only partly reversible. Same company, opposite verdicts. The score tells you where to point the effort.

Substitution vs. compounding#

One more filter, and it's the one that separates a cost line from a strategic asset. Ask whether automating this task substitutes for labor once, or compounds.

• Substitution: the AI does a task a human did, you save that labor, and that's the end of it. Real, but linear and bounded — you can't save more than you were spending.
• Compounding: doing the task generates data or capability that makes the next task better. Every lead Velya qualifies produces a labeled example of what a good lead looks like, which sharpens qualification, which improves the eval set, which lets the agent run on harder cases. The task feeds the system that does the task.

Prefer compounding use cases. A substitution win caps out at the salary line; a compounding win bends the curve. If two candidates score similarly on the five factors, the compounding one is worth more even at a lower initial accuracy, because it gets better on its own exhaust.

Reject criteria — kill the candidate on paper if any of these hold:

• Verifiability is 1 and error tolerance is 1 (you can't check it and mistakes hurt).
• Repetition is 1 (you'll spend more building than you'll ever save).
• The data the task needs doesn't exist in a reachable form and won't within the pilot.

Phase 2: Scope a narrow pilot with a written, testable success criterion#

You picked a task. Now cut it down until it's almost embarrassingly small, and write down — in one sentence — what "working" means, in terms you could hand to someone else to test.

The discipline here is specification, and it's harder than it looks because vague goals feel like progress while guaranteeing you can never declare victory. "Automate lead qualification" is not a criterion. This is: "On our last 200 real inbound leads, the agent's qualified/not-qualified decision agrees with a senior rep's on ≥90% of cases, and on disagreements the agent is conservative (flags for human) rather than wrongly qualifying, ≥95% of the time." That sentence is testable, has a number, and encodes the asymmetry you care about (false-qualify is worse than false-flag).

If you can't write that sentence, you don't understand the task well enough to automate it — and as I've argued in You Can't Evaluate an Agent You Can't Specify, the specification is the hard part. The model is not your bottleneck. Your inability to say precisely what correct behavior is, is the bottleneck.

Good success criteria share four traits:

1. A distribution, not an anecdote — "on our last 200 real cases," not "on this example."
2. A number — a threshold you commit to before you see results.
3. An asymmetry — which kind of error is worse, and by how much. Almost every real task has one.
4. A tie to money or risk — the criterion should map to a decision you'd actually make.

Scope the pilot so that hitting this criterion is a week or two of work to test, not a quarter. Narrow beats broad: one clinic vertical, one order type, one language. You can widen later. You cannot un-spend a quarter you burned on a vague target.

Phase 3: Build the eval before you build the feature#

This is the phase teams skip, and skipping it is why their pilots don't survive production. Before you write the agent, build the test that will tell you whether the agent works. Not after. Before.

An eval is not a handful of demo prompts. It's a trajectory-level acceptance test on a distribution of real cases. Two words carry the weight: trajectory and distribution.

Distribution means you assemble 30 to 50 real cases — pulled from actual logs, including the ugly ones — with known correct outcomes. Cherry-picked happy-path examples tell you nothing, because a single green demo is one sample from an unknown distribution. The rule of three from applied statistics is worth internalizing: after n clean trials, a 95% upper bound on your failure rate is roughly 3/n. One successful demo (n=1) is consistent with a true failure rate up to about 95%. Thirty clean cases put your ceiling near 10%. That's the difference between "it worked once" and "I have evidence."

Trajectory means you score the path, not just the final answer. An agent that reaches the right conclusion via a wrong retrieval got lucky, and luck doesn't generalize. Where you can't enumerate the single correct path — and for open-ended tasks you usually can't — specify invariants and forbidden states instead: the agent never acts outside the authorized account, never fabricates a citation, never confirms an order above the auto-approve limit, never qualifies a lead the transcript explicitly disqualifies. Those are testable regardless of which valid path the agent took.

Build the eval like this:

1. Pull 30–50 real cases from your logs. Deliberately over-sample the weird ones.
2. Label each with the correct outcome and, where relevant, the acceptable paths or the forbidden states.
3. Write the scorer — automated where you can (exact match, invariant checks), human-graded where you must (was the reasoning sound?).
4. Run your current process (human, or old system) through it to get a baseline. If you can't beat the baseline, you have no story.
5. Commit the threshold from Phase 2 into the eval as a pass/fail gate.

Now you have a regression suite. Every prompt change, every model swap, every tool tweak runs against it. This is the single artifact that most separates teams who ship from teams who demo. It converts "seems better" into a number, and it catches the silent decay that happens when you "improve" one case and break four others.

Phase 4: Design the human-in-the-loop and the checkpoints#

Now you can build — but the architecture question comes first, and it's governed by one fact: reliability decays multiplicatively across steps. An agent that's 95% reliable per step, run over twenty independent steps, is 0.95²⁰ ≈ 36% reliable end-to-end. Push each step to 99% and you get 0.99²⁰ ≈ 82% — better, still not something you'd let touch money unattended. This is the compounding-error problem, and it dictates the whole design. Long autonomous horizons are where reliability goes to die.

Three levers fight the decay:

• Shorten horizons. Fewer sequential steps between checkpoints. Break a twenty-step task into three short chains with verification between them, and the math stops working against you.
• Verification gates before consequential actions. Right before any step that spends money, sends an external message, or writes an irreversible record, check the state. Cheap check, big payoff — it's the difference between catching a bad early step and executing fifteen flawless steps on top of a wrong customer record.
• Retries on recoverable steps. If a step is idempotent and cheap to verify (a lookup, a parse, a search), let the agent detect failure and retry. Recovery is what pushes the real-world number back above the naive independent-step estimate.

Then place the human deliberately. Not everywhere — a human on every step destroys the economics and trains everyone to rubber-stamp. Put the human at the liability-bearing decisions: the actions that are irreversible, expensive when wrong, and hard to verify after the fact. Let the agent run freely on everything reversible and cheap-to-check.

At Kommerce, the agent drafts the order-confirmation decision, checks fraud signals, and assembles the case — all reversible. The human touches exactly one thing: approving orders above a risk threshold before they ship. At Velya, the agent qualifies and books, but any message that gives clinical-adjacent guidance routes to a human, because that's the liability-bearing surface. The rule: automate the reversible middle, gate the irreversible edge.

Human-in-the-loop checklist:

• Every irreversible action has a gate in front of it.
• The human sees the agent's reasoning and confidence, not just its output — so they can catch bad paths, not just bad answers.
• Escalation carries clean context (the human doesn't have to reconstruct the case from scratch).
• The agent's uncertainty is calibrated enough that "I'm not sure, escalate" actually fires on the hard cases.

Phase 5: Build the environment, not just the prompt#

Founders over-invest in prompt wording and under-invest in the world the agent acts in. Yet an agent is a reasoning loop wrapped around tools, and an agent is only as good as its tools. A perfect prompt over a broken tool surface produces a confident, well-reasoned wrong answer. Fix the environment and mediocre prompting works fine.

Three properties make an environment agent-ready:

1. Clean tools. Each tool does one legible thing with a narrow, well-typed interface. A get_customer_orders(customer_id) that returns exactly that beats a query(sql) that can do anything and fails a hundred ways. Every extra degree of freedom in a tool is a new way for the agent to go wrong.
2. Legible state. The agent can observe the current state of the world before it acts. If it can't read whether an order is already confirmed, it will confirm it twice. Make state queryable, not implicit.
3. Actionable errors. When a tool fails, it must say why in a way the agent can act on. Error: 400 teaches the agent nothing. Error: order already confirmed at 14:32, cannot re-confirm lets it recover. Error messages are not for your logs; they're inputs to the agent's next decision.

The test for this phase: could a competent new human employee operate your business using only the tools and error messages you're handing the agent? If a smart human would be confused by a tool's output or stuck by an opaque error, the agent will be too — with less judgment to recover. Most "the model isn't smart enough" complaints are actually "the environment is illegible" complaints in disguise.

Phase 6: Plan for the exception tail#

Your pilot will run beautifully on the routine cases and that will feel like victory. It isn't, because cost is not uniform across cases. Sort your cases by the human effort they actually consume and you get a steep curve: the easy 80% are cheap, the hard 20% are brutal — ambiguous, high-stakes, slow. The tail is a fifth of the volume and often the majority of the labor. This is where agent ROI goes to die, and it's the most reliable optical illusion in enterprise AI.

Do the arithmetic. Say easy cases average 4 minutes and hard cases average 30. Then:

• Easy: 0.80 × 4 = 3.2 minutes per average case
• Hard: 0.20 × 30 = 6.0 minutes per average case

The tail is 20% of cases and 65% of the labor. Automate the easy 80% and you remove 3.2 of 9.2 minutes — 35% of the cost, not 80%. And you still staff the whole team for the tail, plus you now pay to detect which cases belong to it.

The implications reshape how you measure and build:

• Measure cost-weighted automation, not volume-weighted. Weight each case by the fully loaded human effort it consumes, then ask what fraction of that the agent removes. "Clears 80% of tickets" is a vanity metric if it's 35% of the cost.
• Design the handoff as a core product surface, not an afterthought. The value of an agent on hard cases is mostly the quality of its escalation: calibrated uncertainty, clean context transfer, a fast lane to the right human. Build that before you optimize the happy path.
• Don't try to automate the tail to death. The last few points of coverage cost more than the whole rest of the project and usually shouldn't be attempted. Let the agent be excellent at triage and handoff, and let humans own the tail — that's the profitable equilibrium, not a failure.

Phase 7: Measure ROI honestly#

This is where good engineering meets bad statistics and the business case dies on a spreadsheet that lies to you. Two traps, both fatal.

Trap one: linear extrapolation. Your pilot ran on a curated slice — probably the clean cases — and Phase 6 already told you why that slice is unrepresentative. Pilot economics are an optimistic ceiling, not a forecast. Discount them, and model the tail explicitly.

Trap two: attribution without a control. You turn the agent on, the metric improves, you credit the agent. The metric might have moved because of seasonality, because the team was trying harder under observation, because you rolled out to your best segment first, or because of plain regression to the mean. This is the causal-inference problem hiding inside every AI decision: a before-and-after comparison measures time, not treatment. Correlation between "turned on AI" and "number went up" is exactly the kind of confounded evidence that survives a demo and collapses in a controlled test.

The fix is a holdout. Split comparable cases into treatment (agent) and control (status quo), randomly, running at the same time. The difference between them is your real effect. It is almost always smaller than the naive before-after gap, and the size of that shrinkage is the size of the mistake you avoided.

Honest ROI checklist:

• Effect measured against a concurrent randomized holdout, not a before/after.
• Costs modeled on the full case distribution including the tail, not the pilot slice.
• Cost-weighted automation, not volume-weighted.
• All-in cost counted: inference, the humans still on the tail, the detection/routing overhead, and the engineering to maintain the eval and the tools.
• A confidence interval on the number, not a point estimate you'll be held to.

Phase 8: Decide — scale, iterate, or kill#

You wrote a success criterion in Phase 2 and built the eval in Phase 3 precisely so this decision is made against a pre-committed bar, not against the sunk cost and the excitement in the room. Hold the line.

Result vs. criterion	Decision	What it looks like
Meets the bar, honest ROI positive on a holdout	Scale	Widen scope one increment (next vertical, next order type), keep the eval running as a regression gate
Misses, but the gap is a known, fixable cause	Iterate	One more cycle with a specific hypothesis and a deadline — not open-ended "keep tuning"
Misses with no clear path, or ROI negative after honest accounting	Kill	Shut it down, harvest the eval set and the labeled data, redeploy the team

Killing is a legitimate, high-status outcome. A use case that failed against a real criterion taught you something concrete and cost you two weeks, not two quarters — because you scoped narrow and built the eval first. The teams that can't kill are the ones that never wrote a criterion they could fail. Precommitment is what makes the kill decision cheap and clean.

The one-page pre-flight checklist#

Before you write a line of agent code, every box should be checked:

• Task scores ≥3 on repetition, verifiability, and error tolerance (reversibility).
• It's a compounding use case, or a substitution one big enough to justify itself alone.
• Success criterion written in one sentence: distribution, number, asymmetry, tied to money.
• Eval built first: 30–50 real cases (over-sampling the ugly ones), trajectory-scored, with a baseline.
• Invariants and forbidden states specified for the open-ended parts.
• Horizons short; verification gates in front of every irreversible action; retries on recoverable steps.
• Human placed at the liability-bearing decisions only.
• Tools are clean, state is legible, errors are actionable — a smart new hire could operate on them.
• Exception-tail handoff designed as a product surface; automation measured cost-weighted.
• ROI plan uses a concurrent holdout and models the full distribution.
• Scale/iterate/kill thresholds committed in writing, in advance.

Top failure modes and their fixes#

Failure mode	Symptom	Fix
Demo-driven development	Dazzling demo, pilot never ships	Build the eval before the feature (Phase 3)
Vague success criterion	"It's getting better" forever, never done	One-sentence testable criterion with a number (Phase 2)
Long autonomous horizons	Confident, catastrophic end-to-end failures	Shorten horizons, add verification gates (Phase 4)
Prompt obsession	Well-reasoned wrong answers	Fix the tools and errors, not the wording (Phase 5)
Happy-path economics	ROI evaporates at rollout	Model the tail, measure cost-weighted (Phase 6)
Before/after ROI	Overstated gains that don't replicate	Randomized concurrent holdout (Phase 7)
Sunk-cost persistence	Can't kill a losing use case	Pre-committed thresholds (Phase 8)

Start here Monday#

Pick your top three candidate tasks and score them on the five AI Leverage factors — 1 to 5, in a shared doc, this afternoon. Take the winner and pull 30 real cases from your logs into a spreadsheet with a "correct outcome" column. That spreadsheet is your eval, and building it will teach you more about whether this task is automatable than any model benchmark ever will. If you can't fill in the "correct outcome" column with confidence, you've found your real problem — and you found it in an afternoon instead of a quarter. Build the eval first. Everything else in this guide is downstream of that one habit.