How it works

What “the agent” is

The agent is a large language model (LLM) driving a real Chromium browser — the same engine behind Chrome. It is not a scripted bot following a fixed sequence of clicks written for your specific site. It's given a goal in plain English (“buy a black t-shirt under ₹1,500”), and on every turn it decides, on its own, what to do next: click a link, type into a search box, select a size, scroll down. Nothing about how it acts is hand-coded per store — the same agent, unmodified, is pointed at all 50 storefronts.

This is deliberate: a benchmark that scripts its way through steps written for a specific site would be measuring the script, not the store. Using a general-purpose agent means the difficulty a real shopper (human or AI) would hit is the difficulty the benchmark measures.

What “the harness” is

The harness is the loop that connects the language model to the browser. Neither one can talk to the other directly — an LLM can't see pixels or click a mouse, and a browser doesn't understand English goals. The harness bridges that gap, once per turn:

1. 1Distill the page. The harness scans the current page and lists every interactive element it can see — buttons, links, text fields, dropdowns — with enough detail (its label, position, what it does) for a model to reason about, plus a screenshot for visual context.
2. 2Ask the model for one action. The model sees that list plus the shopping goal and picks exactly one thing to do next — click element #7, type “kurta” into element #3, and so on.
3. 3Execute it in the real browser, the same way a human's click or keystroke would land, then check whether that action moved the journey to its next checkpoint (cart, checkout, and so on).

This is why it's called a DOM-distillation loop rather than a browser “script”: nothing about it is written against any particular site's buttons or layout. It reads whatever page is actually in front of it, the same way a person would.

The six checkpoints

Every journey is measured against the same six-stage path a real purchase takes. Reaching a stage earns credit; the stage the agent gets stuck at — and why — becomes the store's finding. (This is the sequence itself, not a result — see the leaderboard for real per-store outcomes.)

Design principles

• Rules before judgment. Wherever a stage transition or a failure can be detected with certainty — a URL pattern, a cart counter, a payment form appearing in the DOM — the harness uses that deterministic signal instead of asking a model to guess. The model is only asked to judge ambiguous cases, and its verdicts still have to line up with what's actually on the page.
• Claims are verified, not trusted. When the agent says it completed a stage, that claim is checked against the page before it counts. An agent that says “added to cart” while the cart is still empty doesn't get credit — it gets told why, and one chance to recover.
• Agent mistakes aren't store failures. Sometimes the agent itself stalls — clicks the wrong thing repeatedly, misreads a price, gets confused by a variant picker. That's a limitation of the agent, not evidence the store is hard to use, so those runs are voided and re-run rather than counted against the store.
• Three attempts, majority rules. Every store is run at least three times. A finding is only published when at least two of three runs agree — a single unlucky or unusual run can't define a store's score.

Under the hood

The benchmark runs on open-weight models, split by job. That's a design choice, not a budget accident: it keeps the results reproducible, and it keeps the cost of re-running the entire fleet low enough that nothing here ever has to go stale.

• Two models, split by job. The driver does the expensive work — multi-turn reasoning over a live page. The classifier is only consulted for judgments the deterministic rules can't settle (see the principles above), and those calls are short and rare. Most stage transitions never touch a model at all: a URL pattern or a cart counter settles them for free.
• Open weights, on purpose. The driver was picked on independently verified agentic-browsing benchmarks, not brand. Scoring stores with models anyone can run means the leaderboard isn't hostage to a single lab's closed API — and a future rubric version can be re-validated against the very same weights.
• Every model call is metered. Tokens and cost are recorded per turn, per run, per store, with hard budget rails at both the run and fleet level. Total model spend behind everything published on this site — every journey, every verification, every re-run: under $10.