Evaluation, verification and release gates
The qualification plan separates data/tool correctness, deterministic business logic, AI reasoning, security controls and end-to-end task outcomes. Each metric has a defined population, expected result, failure class and release consequence.
| Evaluation layer | What is measured | Method / ground truth | Release consequence |
|---|---|---|---|
| 1 · Data & tool | Entity resolution, tool selection, tool inputs, tool-call success, source freshness | Expected entities/tools/parameters and authoritative service responses | Wrong-record or critical tool/input failure blocks release |
| 2 · Deterministic logic | Shortage, allocation, replenishment, holds, customer eligibility | Rule-engine expected outputs and exact numeric recomputation | Protected calculation or policy failure blocks release |
| 3 · AI reasoning | Task adherence, blocker explanation, groundedness, completeness, permitted recommendation set | Expected blocker/evidence/recommendation labels plus reviewed rubric | Material unsupported operational claim blocks release |
| 4 · Safety & control | Abstention, role enforcement, approval boundary, prompt/data boundary, stale-state handling | Designed negative/adversarial cases and authorization policy | Any critical authorization bypass blocks release |
| 5 · End-to-end | Correct investigation disposition, escalation, action preparation, revalidation, execution verification | Replayable scenario trace with expected final state | Workflow cannot advance beyond its qualified stage |
| Failure slice | Why it is isolated |
|---|---|
| Ambiguous identity | Tests safe resolution and abstention rather than guessing |
| Reserved vs available stock | Prevents false fulfilment from inventory semantics |
| Incoming PO | Ensures future supply is not represented as current availability |
| Conflicting/stale evidence | Tests withholding, refresh and escalation behavior |
| Customer policy | Ensures partial shipment/substitution options are policy-permitted |
| Changed state after approval | Tests mandatory pre-execution revalidation |
| Unauthorized request/action | Tests controls outside model reasoning |
Versioned benchmark covering exact retrieval, shortages, multi-warehouse allocation, incoming stock, customer rules, conflicting evidence, abstention and protected-action tests.
1,500150Entities + tools + rules + evidence + expected dispositionRequires versioned run artifact150 protected-action tests cover order changes, allocation changes, shipment release and customer-rule boundaries; zero unauthorised writes execute.
150 designed cases0 authorization bypassRole · action · approval · stale state · write boundaryRequires reproducible security runCorrelation trace records user role, order, SKUs, warehouses, tools, source versions, rule results, evidence references, recommendations, approval state and action status.
SOAI-REQ-004817SO-12547ORDER_LINE_SHORTAGE4NONENOT_REQUESTEDPrimewayz AI enablement for enterprise operations