REALITY LAB / RECORDED 11 SEPTEMBER 2026
Three controlled AI workflow tests
Owner review pending. Prepared evidence package; not approved for public release. The tests use synthetic data. Human baseline and checking time have not been measured.
34unique synthetic cases
8 / 9attempts returned output
92 / 92matches in recovered outputs
Including the failed attempt, 92 of 102 attempted pipeline case checks returned a correct answer. The repeated cases are not independent customer observations. The reference answers and scoring rules were authored for these fixtures and need independent review.
What we ran
Each request used x-ai/grok-4.3 through OpenRouter: temperature 0, low reasoning, maximum 4,000 output tokens, JSON output, web/X search disabled and no automatic model fallback. The model received the frozen prompt and inputs, never the answer key. All nine attempts are included below.
Total provider-reported model cost was US$0.02957075; another US$0.10 remains reserved with unconfirmed billing. Hosting, development, subscriptions, human review and delivery costs are not included. No customer revenue or profit was established.
The initial supplier-quote request failed in the integration. Its record is retained. See the dated run plan and amendment; no replacement attempt was silently substituted.
Supplier quotes
Can Grok apply explicit supplier-pack, price, discount and clarification rules to structured input?
2 of 3 pipeline attempts returned completed model output. Those outputs matched 20/20 case references. Across all attempted pipeline cases, 20/30 returned a correct answer; missing outputs remain failures of the pipeline and are not scored as model answer errors. All completed outputs matched the exact prompt and finished normally.
The recovered outputs quoted all five valid requests with the expected pack counts and cent totals, including the half-cent rounding case. All five unquotable requests were flagged for clarification. Q09 lacked a SKU and used an ambiguous product name; the clarification reasons were brief but did not invent a matching product.
Every recorded attempt
| Attempt | Outcome | Case matches | Model request | Reported cost |
|---|---|---|---|---|
| 1 · 150053e6 | uncertain | 0/10 | Unknown | Unconfirmed |
| 2 · c010d172 | completed | 10/10 | 11.715 s | US$0.00534215 |
| 3 · b93ef0e2 | completed | 10/10 | 10.527 s | US$0.00356815 |
Time and cost
Completed model-request times ranged from 10.527–11.715 seconds. Provider-reported costs total US$0.00891030. A further US$0.10 remains reserved with no confirmed invoice cost. Provider reporting is not invoice reconciliation; the app rounds allocations upward to microdollars.
Human setup, checking, correction and baseline times were not measured. Hosting, development effort and subscription overhead have not been allocated. This is not total delivery cost, a speedup result or a profit calculation. Revenue and attributable customer interest are unmeasured.
Limits and decision
Only two model outputs were recovered. The first pipeline attempt failed before a usable answer. The three-successful-trial criterion was not met. The input was already structured JSON; no PDFs, OCR, real supplier updates, taxes or customer documents were tested. Deterministic arithmetic can implement these rules, so this study does not establish that a language model is needed.
The assistant reviewed references and outputs; independent human checking is still pending. The current evidence supports only the observed behavior on these fixtures. Do not infer a population-level accuracy guarantee from repeated tiny examples.
A paid assessment would need to test a customer’s permitted supplier format and checking process. Do not sell autonomous quoting or claim that these synthetic cases validate a real installation.
Inspect and reproduce
Inputs · Exact prompt · Reference answers · Original protocol · Full report
OpenRouter model metadata supplies workflow/model context, not evidence for our trial scores or commercial demand. OpenRouter usage accounting describes the reported cost field. Our findings come from the linked raw run records.
From the project directory, run node scripts/score-workflow-trial.mjs quote-accuracy experiments/results/2026-09-11/RUN-ID.json for each record. Use node scripts/score-pilot-evidence.mjs and node scripts/build-pilot-reports.mjs to regenerate scores and these reports without making provider calls.
Email routing
Can Grok classify 12 synthetic emails under one explicit policy while retaining human review?
3 of 3 pipeline attempts returned completed model output. Those outputs matched 36/36 case references. Across all attempted pipeline cases, 36/36 returned a correct answer; missing outputs remain failures of the pipeline and are not scored as model answer errors. All completed outputs matched the exact prompt and finished normally.
The expected labels include security versus refund precedence, refund versus invoice precedence, support versus sales precedence, a full-team outage, urgency words that do not justify high priority, and an embedded instruction attempting to remove human review. The scorer checks category, priority and needsHumanApproval on each record.
Every recorded attempt
| Attempt | Outcome | Case matches | Model request | Reported cost |
|---|---|---|---|---|
| 1 · a061eea1 | completed | 12/12 | 5.826 s | US$0.00335465 |
| 2 · b68b2751 | completed | 12/12 | 6.321 s | US$0.00274485 |
| 3 · c071d832 | completed | 12/12 | 6.174 s | US$0.00276235 |
Time and cost
Completed model-request times ranged from 5.826–6.321 seconds. Provider-reported costs total US$0.00886185. A further US$0.00 remains reserved with no confirmed invoice cost. Provider reporting is not invoice reconciliation; the app rounds allocations upward to microdollars.
Human setup, checking, correction and baseline times were not measured. Hosting, development effort and subscription overhead have not been allocated. This is not total delivery cost, a speedup result or a profit calculation. Revenue and attributable customer interest are unmeasured.
Limits and decision
The same 12 short synthetic English messages were repeated. This is not 36 independent real customer emails. No Gmail account, long thread, attachment, multilingual text, autonomous reply or delivery action was tested. A true approval flag in output is not an enforcement mechanism; software must still block unapproved actions.
The assistant reviewed references and outputs; independent human checking is still pending. The current evidence supports only the observed behavior on these fixtures. Do not infer a population-level accuracy guarantee from repeated tiny examples.
A permissioned routing-policy assessment could test a customer’s actual ambiguous cases and review needs. These trials do not prove demand or justify sending automatic replies.
Inspect and reproduce
Inputs · Exact prompt · Reference answers · Original protocol · Full report
Google’s Gmail assistance documentation supplies workflow/model context, not evidence for our trial scores or commercial demand. OpenRouter usage accounting describes the reported cost field. Our findings come from the linked raw run records.
From the project directory, run node scripts/score-workflow-trial.mjs email-routing experiments/results/2026-09-11/RUN-ID.json for each record. Use node scripts/score-pilot-evidence.mjs and node scripts/build-pilot-reports.mjs to regenerate scores and these reports without making provider calls.
Meeting actions
Can Grok extract agreed actions without inventing owners or resolving explicitly ambiguous dates?
3 of 3 pipeline attempts returned completed model output. Those outputs matched 36/36 case references. Across all attempted pipeline cases, 36/36 returned a correct answer; missing outputs remain failures of the pipeline and are not scored as model answer errors. All completed outputs matched the exact prompt and finished normally.
The reference has ten outstanding tasks across 12 cases. Checks include cancellation, suggestions without agreement, reassignment, deadline correction, missing owners and dates, tomorrow relative to a dated meeting, an ambiguous Friday, completed work, duplicate mentions and a quoted instruction to invent a task.
Every recorded attempt
| Attempt | Outcome | Case matches | Model request | Reported cost |
|---|---|---|---|---|
| 1 · 20a43b08 | completed | 12/12 | 9.020 s | US$0.00453340 |
| 2 · a9d8bfd1 | completed | 12/12 | 8.727 s | US$0.00361260 |
| 3 · e997c7ad | completed | 12/12 | 8.792 s | US$0.00365260 |
Time and cost
Completed model-request times ranged from 8.727–9.020 seconds. Provider-reported costs total US$0.01179860. A further US$0.00 remains reserved with no confirmed invoice cost. Provider reporting is not invoice reconciliation; the app rounds allocations upward to microdollars.
Human setup, checking, correction and baseline times were not measured. Hosting, development effort and subscription overhead have not been allocated. This is not total delivery cost, a speedup result or a profit calculation. Revenue and attributable customer interest are unmeasured.
Limits and decision
The same 12 short synthetic transcripts were repeated. Bracketed task IDs deliberately simplify matching. No audio, speech recognition, real meeting, speaker identification, long transcript or production task-creation system was tested. Task ordering is ignored by the scorer; owners and dates must match exactly, including null values.
The assistant reviewed references and outputs; independent human checking is still pending. The current evidence supports only the observed behavior on these fixtures. Do not infer a population-level accuracy guarantee from repeated tiny examples.
A team-specific action-extraction assessment would need permissioned notes and owner review before any tasks are created. These results do not establish production reliability or time savings.
Inspect and reproduce
Inputs · Exact prompt · Reference answers · Original protocol · Full report
Microsoft’s meeting recap documentation supplies workflow/model context, not evidence for our trial scores or commercial demand. OpenRouter usage accounting describes the reported cost field. Our findings come from the linked raw run records.
From the project directory, run node scripts/score-workflow-trial.mjs meeting-actions experiments/results/2026-09-11/RUN-ID.json for each record. Use node scripts/score-pilot-evidence.mjs and node scripts/build-pilot-reports.mjs to regenerate scores and these reports without making provider calls.
Verify the evidence offline
Download and extract the package from the private dashboard. With Node.js 24 installed, run node verify-evidence.mjs in the extracted directory. This checks file hashes and reproduces every score without an API key, installation or network request.
Read provenance and redaction details and file hashes. Hashes support byte consistency, not independent proof that a provider produced a response. The internal workspace owner field is omitted; exact prompts, outputs, settings, timestamps, reported costs and model metadata are preserved.
Anyone completing a blind human baseline should not inspect the reference answers first. If the answers have already been seen, use a separately preregistered new fixture set.