# Can AI produce accurate quotes from a supplier list?

Current status: three pipeline attempts recorded, with two completed model outputs and one retained infrastructure failure. See [provisional results](../reports/quote-accuracy.md) and the [dated amendments](../API-RUN-PLAN-2026-09-11.md). The protocol below is the predeclared design; human baseline/review and commercial validation remain outstanding.

## API trial route

After the Reality Lab runtime receives its Keyring OpenRouter grant, the private dashboard's Research workspace can run this experiment. Choose **Experiment trial**, paste the exact prompt below followed by the complete `inputs.json` contents, and use **Run once**. Do not include `expected.json`. The current controlled configuration requests `x-ai/grok-4.3`, temperature 0, low reasoning effort, at most 4,000 output tokens, no search, and a 25-second request timeout. Record these settings as part of the experiment; do not pool these results with a differently configured subscription trial.

Run three fresh trials with identical inputs, preserving every failure or truncated response. Download each saved evidence record. Its `result.output` is the original response text to score; the surrounding record contains prompt, settings, generation ID, timing and reported cost. Report a timeout as an unsuccessful trial, and keep its reserved cost unresolved until provider evidence confirms the charge. Never retry automatically or discard a failed trial when reporting reliability.

The model-request duration recorded by the dashboard excludes human setup, checking, corrections and handover. Record those separately before making any time-saving claim. A successful model run is not a completed investigation until the independent answer check, scoring, limitations and human baseline below are finished.

## Fixed materials

Run `node scripts/quote-baseline.mjs` from the project root to reproduce inputs.json and expected.json. All products and requests are synthetic. The reference is deterministic code, not a timed human baseline. Never send the expected answers to the model under test.

## Trial

1. Time a human completing the ten cases without seeing expected.json. Preserve their answers, then compare them with the reference. Correct the reference if independently verified arithmetic reveals an error, recording the change before AI trials.
2. In a fresh SuperGrok conversation, attach inputs.json and use the exact prompt below. Record date, displayed model/mode and search setting. Do not add clarifications during a trial.
3. Preserve the complete response as trial-01.txt. Repeat twice in fresh conversations with unchanged inputs, yielding three independent trials. Do not discard failures.
4. For every case score status correctness, quoted amount, pack count and invented data separately. Report results per trial and pooled; do not collapse partial success into all-correct.
5. Time checking/correction separately from generation and setup. Record subscription cost allocation explicitly; never call the task free merely because the subscription already exists.

## Prompt

Using only the attached synthetic supplier data and its rules, answer every quote request. Return JSON with a results array. Every result must have id and status (quoted or clarify). Quoted results require packs and totalCents. Clarify results require reason. Do not invent prices, infer an ambiguous SKU, substitute products or change quantities. Do not use web search. Do not include additional prose.

## Predeclared success

- Correct decisions for all ten cases in each of three trials.
- Exact integer-cent totals and pack counts for every quotable case.
- Zero invented prices, substitutions or silent quantity changes.
- Report actual total time against the measured human baseline. Accuracy alone does not establish time savings.

Even a perfect result only supports this small synthetic task. It does not establish reliability on real PDFs, OCR, changing supplier lists, tax rules or customer jobs. A paid deployment would require testing the customer's actual formats with their permission.

## Publication gate

No verdict or performance claim until the trials and independent answer check are complete. A prospective offer is a fixed-scope workflow assessment with human quote approval, not an autonomous quoting guarantee. Pricing and outreach remain to be approved.