The situation#
The business sells the same products in five places at once: our own storefront, Amazon, several eBay accounts, Walmart, and Newegg. Each place charges different fees, ships a different way, and competes against different sellers. For years, pricing was rules, spreadsheets, and judgment applied by hand. That worked when the catalog was small. As it grew, it quietly broke.
The immediate push was tariffs. Costs were jumping faster than anyone could reprice by hand, and a stale price is a quiet leak: sell too low and the profit is gone, sell too high and the product stops moving. The first version of this system was an emergency response to that. What it grew into is the interesting part.
The constraint#
Pricing this catalog is genuinely hard, and the difficulty is not arithmetic. A competitor's price means one thing when you are nearly out of stock and something else entirely when you are drowning in it. The same product sells as a single unit and as a three-pack, and those two listings must not undercut each other. Sales look slow right after a stockout, but not because demand died. And sometimes the “competing” listing is not even the same product - it is an accessory that happens to match the search. No formula holds all of that at once.
Giving the agents real autonomy makes the quality of their information and reasoning consequential. A confident wrong answer can become a real price, and the system has fewer deterministic guardrails than many people expect. I chose that deliberately. The job was to give the agents enough business context to make good decisions, keep the few limits that truly cannot be crossed in code, and measure whether each model deserves the authority it receives.
System boundary#
The central design rule: the agents never see the production database. All raw data retrieval is ordinary code. What reaches the AI starts as object models built from the bookkeeping reports - structured Python objects that encapsulate the raw records behind each report. On top of those, the data is projected and attributed along multiple dimensions (by SKU, by channel, by time period) to build both detailed and summary reports. The agent receives the data models themselves, and the summary report usually answers the question directly. But when it doesn't - when the agent needs to dig - every underlying record is already loaded in its environment as part of those models. It can explore the data interactively, thoroughly, without a single database call. No credentials, no supplier records, no unrelated data ever cross the boundary. If a source system changes its format, the code that gathers data breaks loudly before anything strange reaches the model.
The stages, in plain terms: one agent sets a baseline price for the product. Separate agents adjust for each sales channel - listing prices deviate from the baseline for a variety of reasons, and the system works through an iterative process of pricing, refinement, and approval before converging on a final number. A consistency stage makes sure the single unit and the three-pack never drift into prices that contradict each other. A small set of business limits remains in plain code, and publishing runs as its own controlled stage. Most of the pricing decision belongs to the agents.
A decision trace#
Here is one product moving through the pipeline, end to end. BRKT-42 has 18 units on the shelf and 60 more on a boat. Recent sales look slow, but only because it was out of stock for part of the month. It sells as a single unit from our warehouse and as a three-pack through Amazon. Search results surfaced one “competitor” that is actually a different product - an accessory. And the landed cost just went up.
| Stage | Inputs | Decision / output | Why |
|---|---|---|---|
| Derived metrics | raw inventory, orders, landed cost | 0.6 months of stock left → critical band; sales rate corrected for the stockout days | raw sales history understates real demand, and weak-looking numbers must not trigger a panic markdown |
| Curated view | metrics, pack links, band | a clean summary with no supplier or account data | the agent sees the decision, not the plumbing |
| Baseline agent | product economics, portfolio state | raise the base price slightly | stock is critical, cost just rose, and the margin floor still holds |
| Listing agents | channel fees, fulfillment, competition | warehouse single: raise; Amazon three-pack: hold | the three-pack still has competition; the single is what is running out |
| Consistency check | both listing prices | adjusted the pack gap | buying three singles must never cost less than the three-pack |
| Research | competitor listing | rejected | source check: the “competitor” is an accessory, not this part |
| Policy gates | proposal vs. floors | final price inside the allowed band | plain code enforces the limits the business will not cross |
| Audit output | all stages | a readable record of the reasoning | you can see why the price moved, not just that it did |
A worked example with synthetic data, showing where plain code ends and AI judgment begins.
Business policies as code#
The business rules are written down as code with named conditions, so they run the same way every time: low stock, healthy stock, overstock, protecting a best-selling listing, keeping pack prices consistent. Two examples:
- Running out. When stock drops below the critical line, the system protects what is left: prices move up, discounts stop, and how long restocking takes enters the reasoning. A slow seller and a product about to stock out can look identical in a simple sales report; the stock band is what tells them apart.
- Pack consistency. If the single unit's price moves but the three-pack's does not, the consistency stage compares them and stops the two listings from drifting into a gap that shoppers would spot within minutes.
The boundary in practice#
The boundary is not a separate gate layer bolted on after the fact - it is the shape of the data itself. The object models the agents receive are built by ordinary Python that reads the bookkeeping reports and packages each one into a typed, self-contained structure. The agent gets sku.landed_cost, sku.units_available, sku.trailing_30_day_units - plain attributes on plain objects, already loaded. When the agent wants to answer a follow-up question, it explores those objects directly. It never writes a query, because there is nothing to query against.
The publishing side works the same way. An agent can propose a price, and the code that applies it is ordinary Python that checks the business floors before anything touches a marketplace API. The agent's proposal is input to a function, not a command that executes itself.
Evaluation discipline#
I compare model changes against a fixed set of realistic pricing situations. These evaluations do not make a bad decision impossible; they show whether the model is still making decisions well enough to deserve production authority:
- six test scenarios covering healthy stock, low stock, and distressed stock;
- each answer is scored against the business rules it was supposed to respect;
- the score weighs reasoning quality alongside what the model costs to run;
- in one benchmark run, a cheaper model matched or beat the expensive one on four of five test tiers - so we switched and saved the difference;
- every migration must also pass a suite of 484 automated regression tests.
Numbers are from benchmark runs during model evaluation and migration testing.
Failure modes and safeguards#
| Failure mode | Safeguard |
|---|---|
| The AI invents a competitor or a price that does not exist | every piece of market evidence must name its source, and can be rejected if the source does not check out |
| A source system changes its data format | the code that gathers data fails loudly before anything malformed can reach the agent |
| The single and the three-pack drift into contradictory prices | the consistency stage compares related listings and forces them to agree |
| A markdown triggered by sales that only look slow because of a stockout | the metrics correct for stockout days before any agent sees them |
| A new model is quietly worse than the old one | candidate models replay the fixed scenarios so I can compare their judgment before switching |
| The AI bill quietly grows | evaluations weigh cost alongside quality, so cheaper models win when they reason just as well |
| An agent makes a poor pricing decision | the decision remains visible in the audit output; recurring failures become evaluation cases or, when the limit is absolute, a narrow code rule |
Key takeaways#
What makes this work: the business knowledge lives in the data the agents see, the guardrails that do exist are plain code, and the agents have real decision autonomy inside that. The system has fewer deterministic rails than you might expect - on purpose. I trust the agents to make good decisions, even though they are technically capable of making bad ones, because the curated data they reason over is the same data a good merchant would use. The evaluation scenarios and regression tests exist to catch it when a model change makes that trust less deserved.