⚡ TL;DR
Amazon reported millions in savings after swapping a Sonnet 4.6 call for a $0.0005 decision layer. Here's the 5-step pattern to replace your own LLM calls.
Amazon Swapped Sonnet for Rules: When to Replace Your LLM Call
In November 2026, the Strands Agents team published a post that should make anyone with a per-token bill sit up. One of Amazon’s largest internal ticketing systems ran a Claude Sonnet 4.6 call on Bedrock to label tickets into 20 categories. They swapped that call for a “decision model” that never writes a sentence, and now project millions of dollars of token-cost savings in a year (Strands Agents blog, Nov 2026). The uncomfortable part isn’t the money. It’s the admission that the model was doing a job a lookup table could do.
The lesson transfers. If you ship LLM features, you almost certainly have call sites shaped like this one: closed outputs, stable criteria, real traffic. This post is the pattern I’d use to find them, price them, and replace them without breaking anything.
TL;DR
- Amazon replaced a Sonnet 4.6 call (20 ticket categories) with the Strands decider: cost per ticket fell from $0.0222 to about $0.0005, a 98% reduction, roughly 44x cheaper (Strands Agents blog, Nov 2026).
- Latency per call dropped from 4,215 ms to 897 ms and tokens per ticket from 6,724 to 2,010. Lambda durations fell 74% the day Sonnet was switched off.
- The rollout was boring on purpose: a three-way config flag (Sonnet only, dual, decider), both systems compared on live traffic, and a 45-second rollback path.
- The same decision layer pattern travels: decision table first, LLM fallback on low confidence, for any call that returns a structured choice instead of prose.
Contents
- Why Would Amazon Drop an LLM Call That Already Worked?
- Which of Your LLM Calls Don’t Need a Model?
- What’s the Traffic Worth? Price the Call Before You Cut It
- How Do You Build the Decision Layer?
- How Do You Prove It Before You Switch? Shadow-Test It
- What Do You Watch After Cutover? Drift
- FAQ
- Verdict: Audit First, Swap Second
Why Would Amazon Drop an LLM Call That Already Worked?
The old setup worked, and that’s the trap. Given a ticket message, Sonnet picked one of 20 categories. Output was never prose, options never changed mid-week, criteria fit in a sentence each. So Strands moved the job to their decider, a small model (a Gemma 4 E4B build) that takes a question and options, then returns a probability and confidence score for each. It can’t generate text. It doesn’t need to (Strands Agents blog, Nov 2026).
The economics, straight from the post: about $22,200 per million tickets became about $500. A single SageMaker ml.g5.xlarge at $1.408 per hour handles roughly 5,600 decisions per hour at about 640 ms each. Nothing about the product changed. The 20 label descriptions just became one- or two-sentence criteria, plus a “none” option.
When the post hit Hacker News, the top thread wasn’t shock. It was “we’ve come full circle” (Hacker News, Nov 2026), with commenters who built the same kind of ticket classifier from SVMs years ago. Others defended the LLM as the cold-start tool: an okay classifier in five minutes, replaced once real traffic shows you the distribution. Both camps are right. That’s a lifecycle, not a contradiction.
Which of Your LLM Calls Don’t Need a Model?
Sort every call site in your codebase into three buckets.
- Genuine reasoning. The answer depends on context your option list can’t anticipate: drafting, summarizing, novel debugging, writing code. Keep the model here.
- Structured decision. The model picks one of N known options against stable criteria: intent classification, agent routing, tool selection. This is where Amazon’s savings lived, and where a decision layer wins.
- Fixed mapping. Same input class, same answer, every time: error code to action, keyword to category. An LLM call here isn’t an architecture, it’s a bug.
Most of what gets shipped as “AI” is bucket 2 wearing bucket 1’s badge. The signal categories below tell you which bucket a call belongs to.
| Signal category | What it looks like | Rules or model? |
|---|---|---|
| Fixed mapping | Deterministic answer per input class, few dozen options at most | Rules, every time |
| Structured decision | Closed choice from a stable catalog; criteria fit in a sentence | Decision table or decision model, with LLM fallback |
| High volume, low stakes | Thousands of calls a day; a wrong pick is cheap to correct downstream | Rules with sampled audits |
| Low volume, high stakes | A handful of calls a day; a wrong pick costs real money | Model, or a human |
| Genuine reasoning | Output depends on context the catalog can’t anticipate | Model |
| Tone and judgment | Nuance, empathy, negotiation of intent | Model |
What’s the Traffic Worth? Price the Call Before You Cut It
Before writing any replacement code, measure per call type: volume, tokens per call, blended price. Sonnet on Bedrock ran $3 per million input tokens and $15 per million output, and the labeler burned 6,724 tokens per ticket at $0.0222 per call, about $22,200 per million tickets (Strands Agents blog, Nov 2026). Hacker News commenters worked out that “millions saved per year” implies tens of millions of tickets annually; ex-Amazonians in the thread called that plausible (Hacker News, Nov 2026).
Your numbers will be smaller, but the formula doesn’t care: calls per day times tokens per call times blended price equals daily burn per call type. Provider dashboards rarely group tokens by call site, so tag every prompt with a route name and log it. Most of the self-hosted MCP gateways I compared expose per-route token and latency metrics, which makes the grouping almost free. The same audit applies to developer tooling; I tracked what three coding assistants cost over three months in Claude Code vs Copilot Agent vs Cursor: 3 Months of Real Work, and per-task routing mattered as much as model choice.
How Do You Build the Decision Layer?
Two implementations share one shape: a cheap scorer runs first, and the generative model only fires when confidence is low. Amazon picked a decision model on dedicated hardware. The version you can build in an afternoon is a decision table. Here’s the pattern you’d implement, in Python:
"""Decision-table router with LLM fallback.
Contract: every path returns {"category", "confidence", "source"}.
The LLM is the fallback, not the default.
"""
import logging
import re
from dataclasses import dataclass
log = logging.getLogger("decider")
@dataclass(frozen=True)
class Rule:
pattern: re.Pattern
category: str
weight: float
CONFIDENCE_FLOOR = 0.85 # keep this in config in production
RULES = [
Rule(re.compile(r"\b(500|503|timeout|connection refused)\b", re.I), "incident", 0.9),
Rule(re.compile(r"\b(refund|invoice|chargeback)\b", re.I), "billing", 0.9),
Rule(re.compile(r"\b(sso|saml|locked out|2fa)\b", re.I), "access", 0.85),
]
def llm_classify(text: str) -> dict:
"""Your existing model call, unchanged. Same output contract."""
# resp = bedrock_client.invoke_model(...) # the call you already ship
# return {"category": resp.label, "confidence": resp.score, "source": "llm"}
raise NotImplementedError
def rule_classify(text: str) -> dict | None:
scores: dict[str, float] = {}
for rule in RULES:
if rule.pattern.search(text):
scores[rule.category] = scores.get(rule.category, 0.0) + rule.weight
if not scores:
return None # no match = not confident
category, confidence = max(scores.items(), key=lambda kv: kv[1])
return {"category": category, "confidence": confidence, "source": "rules"}
def classify(text: str) -> dict:
"""Serve path: rules first, model only under the floor."""
hit = rule_classify(text)
if hit and hit["confidence"] >= CONFIDENCE_FLOOR:
return hit
return llm_classify(text) # same shape, higher cost
def shadow(text: str, llm_result: dict) -> None:
"""Dual-run helper: log agreement while serving the model's answer."""
hit = rule_classify(text)
agree = bool(hit and hit["category"] == llm_result["category"])
log.info("shadow agree=%s rules=%s llm=%s", agree, hit, llm_result["category"])Details that matter: both paths return the same shape, so callers can’t tell which one answered. The confidence floor is config, not code. An empty rule match means “not confident” and routes to the model instead of guessing. Amazon’s equivalent was a “none” option in the decider’s list, so ambiguous tickets got an explicit answer rather than a forced one (Strands Agents blog, Nov 2026).
How Do You Prove It Before You Switch? Shadow-Test It
Amazon’s rollout was deliberately dull. A Lambda read a three-way AppConfig setting per message: BEDROCK (Sonnet only), DUAL (both systems compared in parallel), or DECIDER. Dual mode ran on live traffic until the comparison was trusted, and rollback took about 45 seconds (Strands Agents blog, Nov 2026).
Copy that shape: serve the model’s answer while you build the table, compute the rule answer alongside, and log agreement. Keep dual-run sampling on live traffic after cutover too. Set a bar first; I’d want 99% agreement on a labeled sample before flipping the flag.
Shadow mode catches landmines a notebook never shows. Amazon hit one worth memorizing: the decider’s output probabilities shifted by up to 0.19 depending on JSON key order in its input, so keys are now sorted before every call. That bug survives unit tests. It doesn’t survive two weeks of dual runs on real traffic.
What Do You Watch After Cutover? Drift
The work doesn’t end at the flag flip. Three metrics, checked on a schedule:
- Fallback rate. The percentage of calls hitting the model path. A sudden jump means traffic shifted or a rule stopped matching.
- Sampled agreement. Keep a small dual-run sample (1% is plenty) forever. If table and model disagree more each month, the criteria drifted.
- Catalog churn. A decision table is frozen judgment. The day someone adds a 21st category, it’s stale, and only a human knows it. Someone has to own the criteria file.
That last point is the whole job now. Werner Vogels framed it in November 2026: agents and automated layers build and check most software, “with engineers deciding what gets built and when the work is correct” (Werner Vogels, All Things Distributed, Nov 2026). A criteria file is exactly that artifact. It encodes judgment until the world moves and nobody re-freezes it.
This idea didn’t start with Amazon, either. Neubird has argued for decision models over general LLMs on closed-choice tasks, and Towards Data Science has covered cost-aware routing on the same grounds. The Strands post matters because it puts a public number on the pattern, and gateways with per-route observability make the swap measurable at your scale.
FAQ
Did Amazon really replace Sonnet with plain rules?
No. It replaced a generative Sonnet 4.6 call with the Strands decider, a decision model that returns probabilities over fixed options and can’t generate text. The reported result was a 98% cost reduction per ticket (Strands Agents blog, Nov 2026).
What is the Strands Decider, exactly?
A small model (a Gemma 4 E4B build on SageMaker) that takes a state, a question, and options, then returns a probability and confidence for each. It never generates prose. The post reports about 640 ms per decision and roughly 5,600 decisions per hour per instance.
Which LLM calls should I replace first?
High-volume calls that return a closed choice from a stable catalog: routing, intent classification, tool selection. Amazon’s win came from ticket labeling with 20 fixed categories at millions of calls per year. Start where volume multiplies a simple decision.
Is a decision table as good as a decision model?
For a small, stable mapping, yes, and it’s cheaper to debug. A decision model handles more options and fuzzier criteria. Either way, keep the LLM as the fallback under a confidence floor, as Amazon kept Sonnet one config flag away.
How do I make the switch safe?
Run both paths in dual mode on live traffic and log agreement rates until they cross your bar. Flip traffic with a config flag, not a deploy. Amazon’s rollout kept a 45-second rollback path through a three-way configuration setting.
Verdict: Audit First, Swap Second
- Inventory every LLM call site and sort it into genuine reasoning, structured decision, or fixed mapping.
- Price the structured-decision bucket per day. Amazon’s gap was 44x; yours only needs to be 5x to fund a week of work.
- Build the decision table (or adopt a decision model) and keep the LLM call as the fallback under a confidence floor.
- Shadow-test on live traffic until agreement crosses your bar, then flip with a config flag and keep the rollback alive.
- Monitor fallback rate and sampled agreement weekly, and re-freeze the criteria whenever the catalog changes.
Most of your token bill is calls that were never reasoning. Price them, then take them away from the model.

Written by Mihir Pipermitwala
Senior Full Stack Consultant & Mobile Engineering Specialist building user-focused applications in healthcare with Flutter, Dart and React Native. I write about what I actually build and test — every tutorial here comes from real project work.
More about Mihir →