Amazon Swapped Sonnet for Rules: When to Replace Your LLM Call

By Mihir Pipermitwala · 11 min read

⚡ TL;DR

Amazon reported millions in savings after swapping a Sonnet 4.6 call for a $0.0005 decision layer. Here's the 5-step pattern to replace your own LLM calls.

Amazon Swapped Sonnet for Rules: When to Replace Your LLM Call

In November 2026, the Strands Agents team published a post that should make anyone with a per-token bill sit up. One of Amazon’s largest internal ticketing systems ran a Claude Sonnet 4.6 call on Bedrock to label tickets into 20 categories. They swapped that call for a “decision model” that never writes a sentence, and now project millions of dollars of token-cost savings in a year (Strands Agents blog, Nov 2026). The uncomfortable part isn’t the money. It’s the admission that the model was doing a job a lookup table could do.

The lesson transfers. If you ship LLM features, you almost certainly have call sites shaped like this one: closed outputs, stable criteria, real traffic. This post is the pattern I’d use to find them, price them, and replace them without breaking anything.

TL;DR

  • Amazon replaced a Sonnet 4.6 call (20 ticket categories) with the Strands decider: cost per ticket fell from $0.0222 to about $0.0005, a 98% reduction, roughly 44x cheaper (Strands Agents blog, Nov 2026).
  • Latency per call dropped from 4,215 ms to 897 ms and tokens per ticket from 6,724 to 2,010. Lambda durations fell 74% the day Sonnet was switched off.
  • The rollout was boring on purpose: a three-way config flag (Sonnet only, dual, decider), both systems compared on live traffic, and a 45-second rollback path.
  • The same decision layer pattern travels: decision table first, LLM fallback on low confidence, for any call that returns a structured choice instead of prose.

Rules-based decision layer replacing an LLM call in a ticket routing pipeline

Contents

Why Would Amazon Drop an LLM Call That Already Worked?

The old setup worked, and that’s the trap. Given a ticket message, Sonnet picked one of 20 categories. Output was never prose, options never changed mid-week, criteria fit in a sentence each. So Strands moved the job to their decider, a small model (a Gemma 4 E4B build) that takes a question and options, then returns a probability and confidence score for each. It can’t generate text. It doesn’t need to (Strands Agents blog, Nov 2026).

The economics, straight from the post: about $22,200 per million tickets became about $500. A single SageMaker ml.g5.xlarge at $1.408 per hour handles roughly 5,600 decisions per hour at about 640 ms each. Nothing about the product changed. The 20 label descriptions just became one- or two-sentence criteria, plus a “none” option.

When the post hit Hacker News, the top thread wasn’t shock. It was “we’ve come full circle” (Hacker News, Nov 2026), with commenters who built the same kind of ticket classifier from SVMs years ago. Others defended the LLM as the cold-start tool: an okay classifier in five minutes, replaced once real traffic shows you the distribution. Both camps are right. That’s a lifecycle, not a contradiction.

Which of Your LLM Calls Don’t Need a Model?

Sort every call site in your codebase into three buckets.

  1. Genuine reasoning. The answer depends on context your option list can’t anticipate: drafting, summarizing, novel debugging, writing code. Keep the model here.
  2. Structured decision. The model picks one of N known options against stable criteria: intent classification, agent routing, tool selection. This is where Amazon’s savings lived, and where a decision layer wins.
  3. Fixed mapping. Same input class, same answer, every time: error code to action, keyword to category. An LLM call here isn’t an architecture, it’s a bug.

Most of what gets shipped as “AI” is bucket 2 wearing bucket 1’s badge. The signal categories below tell you which bucket a call belongs to.

Signal categoryWhat it looks likeRules or model?
Fixed mappingDeterministic answer per input class, few dozen options at mostRules, every time
Structured decisionClosed choice from a stable catalog; criteria fit in a sentenceDecision table or decision model, with LLM fallback
High volume, low stakesThousands of calls a day; a wrong pick is cheap to correct downstreamRules with sampled audits
Low volume, high stakesA handful of calls a day; a wrong pick costs real moneyModel, or a human
Genuine reasoningOutput depends on context the catalog can’t anticipateModel
Tone and judgmentNuance, empathy, negotiation of intentModel

What’s the Traffic Worth? Price the Call Before You Cut It

Before writing any replacement code, measure per call type: volume, tokens per call, blended price. Sonnet on Bedrock ran $3 per million input tokens and $15 per million output, and the labeler burned 6,724 tokens per ticket at $0.0222 per call, about $22,200 per million tickets (Strands Agents blog, Nov 2026). Hacker News commenters worked out that “millions saved per year” implies tens of millions of tickets annually; ex-Amazonians in the thread called that plausible (Hacker News, Nov 2026).

Your numbers will be smaller, but the formula doesn’t care: calls per day times tokens per call times blended price equals daily burn per call type. Provider dashboards rarely group tokens by call site, so tag every prompt with a route name and log it. Most of the self-hosted MCP gateways I compared expose per-route token and latency metrics, which makes the grouping almost free. The same audit applies to developer tooling; I tracked what three coding assistants cost over three months in Claude Code vs Copilot Agent vs Cursor: 3 Months of Real Work, and per-task routing mattered as much as model choice.

How Do You Build the Decision Layer?

Two implementations share one shape: a cheap scorer runs first, and the generative model only fires when confidence is low. Amazon picked a decision model on dedicated hardware. The version you can build in an afternoon is a decision table. Here’s the pattern you’d implement, in Python:

Copy
"""Decision-table router with LLM fallback.

Contract: every path returns {"category", "confidence", "source"}.
The LLM is the fallback, not the default.
"""
import logging
import re
from dataclasses import dataclass

log = logging.getLogger("decider")

@dataclass(frozen=True)
class Rule:
    pattern: re.Pattern
    category: str
    weight: float

CONFIDENCE_FLOOR = 0.85  # keep this in config in production

RULES = [
    Rule(re.compile(r"\b(500|503|timeout|connection refused)\b", re.I), "incident", 0.9),
    Rule(re.compile(r"\b(refund|invoice|chargeback)\b", re.I), "billing", 0.9),
    Rule(re.compile(r"\b(sso|saml|locked out|2fa)\b", re.I), "access", 0.85),
]

def llm_classify(text: str) -> dict:
    """Your existing model call, unchanged. Same output contract."""
    # resp = bedrock_client.invoke_model(...)  # the call you already ship
    # return {"category": resp.label, "confidence": resp.score, "source": "llm"}
    raise NotImplementedError

def rule_classify(text: str) -> dict | None:
    scores: dict[str, float] = {}
    for rule in RULES:
        if rule.pattern.search(text):
            scores[rule.category] = scores.get(rule.category, 0.0) + rule.weight
    if not scores:
        return None  # no match = not confident
    category, confidence = max(scores.items(), key=lambda kv: kv[1])
    return {"category": category, "confidence": confidence, "source": "rules"}

def classify(text: str) -> dict:
    """Serve path: rules first, model only under the floor."""
    hit = rule_classify(text)
    if hit and hit["confidence"] >= CONFIDENCE_FLOOR:
        return hit
    return llm_classify(text)  # same shape, higher cost

def shadow(text: str, llm_result: dict) -> None:
    """Dual-run helper: log agreement while serving the model's answer."""
    hit = rule_classify(text)
    agree = bool(hit and hit["category"] == llm_result["category"])
    log.info("shadow agree=%s rules=%s llm=%s", agree, hit, llm_result["category"])

Details that matter: both paths return the same shape, so callers can’t tell which one answered. The confidence floor is config, not code. An empty rule match means “not confident” and routes to the model instead of guessing. Amazon’s equivalent was a “none” option in the decider’s list, so ambiguous tickets got an explicit answer rather than a forced one (Strands Agents blog, Nov 2026).

How Do You Prove It Before You Switch? Shadow-Test It

Amazon’s rollout was deliberately dull. A Lambda read a three-way AppConfig setting per message: BEDROCK (Sonnet only), DUAL (both systems compared in parallel), or DECIDER. Dual mode ran on live traffic until the comparison was trusted, and rollback took about 45 seconds (Strands Agents blog, Nov 2026).

Copy that shape: serve the model’s answer while you build the table, compute the rule answer alongside, and log agreement. Keep dual-run sampling on live traffic after cutover too. Set a bar first; I’d want 99% agreement on a labeled sample before flipping the flag.

Shadow mode catches landmines a notebook never shows. Amazon hit one worth memorizing: the decider’s output probabilities shifted by up to 0.19 depending on JSON key order in its input, so keys are now sorted before every call. That bug survives unit tests. It doesn’t survive two weeks of dual runs on real traffic.

What Do You Watch After Cutover? Drift

The work doesn’t end at the flag flip. Three metrics, checked on a schedule:

  1. Fallback rate. The percentage of calls hitting the model path. A sudden jump means traffic shifted or a rule stopped matching.
  2. Sampled agreement. Keep a small dual-run sample (1% is plenty) forever. If table and model disagree more each month, the criteria drifted.
  3. Catalog churn. A decision table is frozen judgment. The day someone adds a 21st category, it’s stale, and only a human knows it. Someone has to own the criteria file.

That last point is the whole job now. Werner Vogels framed it in November 2026: agents and automated layers build and check most software, “with engineers deciding what gets built and when the work is correct” (Werner Vogels, All Things Distributed, Nov 2026). A criteria file is exactly that artifact. It encodes judgment until the world moves and nobody re-freezes it.

This idea didn’t start with Amazon, either. Neubird has argued for decision models over general LLMs on closed-choice tasks, and Towards Data Science has covered cost-aware routing on the same grounds. The Strands post matters because it puts a public number on the pattern, and gateways with per-route observability make the swap measurable at your scale.

FAQ

Did Amazon really replace Sonnet with plain rules?

No. It replaced a generative Sonnet 4.6 call with the Strands decider, a decision model that returns probabilities over fixed options and can’t generate text. The reported result was a 98% cost reduction per ticket (Strands Agents blog, Nov 2026).

What is the Strands Decider, exactly?

A small model (a Gemma 4 E4B build on SageMaker) that takes a state, a question, and options, then returns a probability and confidence for each. It never generates prose. The post reports about 640 ms per decision and roughly 5,600 decisions per hour per instance.

Which LLM calls should I replace first?

High-volume calls that return a closed choice from a stable catalog: routing, intent classification, tool selection. Amazon’s win came from ticket labeling with 20 fixed categories at millions of calls per year. Start where volume multiplies a simple decision.

Is a decision table as good as a decision model?

For a small, stable mapping, yes, and it’s cheaper to debug. A decision model handles more options and fuzzier criteria. Either way, keep the LLM as the fallback under a confidence floor, as Amazon kept Sonnet one config flag away.

How do I make the switch safe?

Run both paths in dual mode on live traffic and log agreement rates until they cross your bar. Flip traffic with a config flag, not a deploy. Amazon’s rollout kept a 45-second rollback path through a three-way configuration setting.

Verdict: Audit First, Swap Second

  1. Inventory every LLM call site and sort it into genuine reasoning, structured decision, or fixed mapping.
  2. Price the structured-decision bucket per day. Amazon’s gap was 44x; yours only needs to be 5x to fund a week of work.
  3. Build the decision table (or adopt a decision model) and keep the LLM call as the fallback under a confidence floor.
  4. Shadow-test on live traffic until agreement crosses your bar, then flip with a config flag and keep the rollback alive.
  5. Monitor fallback rate and sampled agreement weekly, and re-freeze the criteria whenever the catalog changes.

Most of your token bill is calls that were never reasoning. Price them, then take them away from the model.

Mihir Pipermitwala

Written by Mihir Pipermitwala

Senior Full Stack Consultant & Mobile Engineering Specialist building user-focused applications in healthcare with Flutter, Dart and React Native. I write about what I actually build and test — every tutorial here comes from real project work.

More about Mihir →
Scroll down to load comments...
Related Blogs
AI-Generated Code Is Fragile: Engineering Habits That Fix It

AI-Generated Code Is Fragile: Engineering Habits That Fix It

AI-generated code gets fragile fast: GitClear measured an 8x rise in duplicated blocks in 2024. Five engineering habits, with code, restore maintainability.

AI TOOLSDEVELOPER TOOLSPRODUCTIVITY

November 16, 2026

Amazon Swapped Sonnet for Rules: When to Replace Your LLM Call

Amazon Swapped Sonnet for Rules: When to Replace Your LLM Call

Amazon reported millions in savings after swapping a Sonnet 4.6 call for a $0.0005 decision layer. Here's the 5-step pattern to replace your own LLM calls.

LLMOPSAI TOOLSDEVELOPER TOOLS

November 15, 2026

Apple Harbor: What the New Mac App Means for Local AI Models

Apple Harbor: What the New Mac App Means for Local AI Models

Apple's reported Harbor app aims to make local AI models easy on Mac, maybe across multiple Macs. Plus the 3 commands to run local models yourself today.

ON DEVICE LLMAI TOOLSAPPLE

November 14, 2026

Go JSON v2: A Practical Migration Guide to encoding/json/v2

Go JSON v2: A Practical Migration Guide to encoding/json/v2

Migrate a Go REST API to encoding/json/v2 in 7 steps: struct tags, omitempty, unknown fields, error handling, and jsontext streaming, plus a v1 vs v2 comparison.

GOTUTORIALDEVELOPER TOOLS

November 12, 2026

Related Recommended Services
Visual Studio Code for the Web

Visual Studio Code for the Web

Build with Visual Studio Code, anywhere, anytime, in your browser.

IDEVISUAL STUDIOVISUAL STUDIO CODEWEB
Renovate | Automated Dependency Updates

Renovate | Automated Dependency Updates

Renovate Bot keeps source code dependencies up-to-date using automated Pull Requests.

AUTOMATED DEPENDENCY UPDATESBUNDLERCOMPOSERGITHUBGO MODULES
Best XML Formatter and XML Beautifier

Best XML Formatter and XML Beautifier

Online XML Formatter will format xml data, helps to validate, and works as XML Converter. Save and Share XML.

XMLXML BEAUTIFIERXML CONVERTERXML FORMATXML FORMATTER
Kubecost | Kubernetes cost monitoring and management

Kubecost | Kubernetes cost monitoring and management

Kubecost started in early 2019 as an open-source tool to give developers visibility into Kubernetes spend. We maintain a deep commitment to building and supporting dedicated solutions for the open source community.

CLOUDKUBECOSTKUBERNETESOPEN SOURCESELF HOSTED
Related Recommended Stories
How GitHub reduced testing time for iOS apps with new runner features

How GitHub reduced testing time for iOS apps with new runner features

Learn how GitHub used macOS and Apple Silicon runners for GitHub Actions to build, test, and deploy our iOS app faster.

IOSGITHUBTESTINGRUNNER
5 ways to transform your workflow using GitHub Copilot and MCP

5 ways to transform your workflow using GitHub Copilot and MCP

Learn how to streamline your development workflow with five different MCP use cases.

AGENT MODECODING AGENTCOPILOTFIGMAGITHUB
One weird trick for powerful Git aliases

One weird trick for powerful Git aliases

Advanced Git Aliases

ALIASALIAS TEMPLATEATLASSIANBITBUCKETGIT
Awesome Python

Awesome Python

An opinionated list of awesome Python frameworks, libraries, software and resources

AWESOMEAWESOME PYTHONCOLLECTIONSGITHUBPYTHON
Related Recommended Tools
Find out what websites are built with - Wappalyzer

Find out what websites are built with - Wappalyzer

Find out the technology stack of any website. Create lists of websites and contacts by the technologies they use.

ADD ONSANALYTICSAPP STOREAPPLEBOOKING
Sourcetree | Free Git GUI for Mac and Windows

Sourcetree | Free Git GUI for Mac and Windows

A Git GUI that offers a visual representation of your repositories. Sourcetree is a free Git client for Windows and Mac.

GITGITHUBGITLABATLASSIANBITBUCKET
Related Recommended Videos