Instructorfail

tool missing

A missing tool is not a missing part. Conflating them makes the Foreman order a pad when what is needed is a torque wrench that already exists somewhere in the building.

The answer was well-formed and an assertion about what it decided did not hold.

modelgemini-3.5-flash

temperature0

scenarioscenarios/instructor/tool-missing.json

cassette84813de87c94b385…

latency0.0s

tokens1,930

What the model was given

System instructionassembled from the contract schema, not hand-written

A technician has said they cannot complete this step. Turn what they said into a structured reason and recommend the next action for the person standing there right now, knowing the procedure, the machine and its history.

Return JSON with these fields:
- reason_summary (string, required): Their reason in one clause, in their terms. Do not sanitise it.
- blocker_kind (one of part_missing | tool_missing | access | seized_or_damaged | unsafe | machine_absent | procedure_wrong | other, required)
- recommended_action (string, required): What to do next. Concrete and doable now, or explicitly 'stop and hand off'.
- proposed_status (one of deferred | waived | impossible, required): A proposal only. The Foreman disposes and a waiver needs a named person with standing. There is no skip.
- blocking_part (string, null when it does not apply): Part number or description when blocker_kind is part_missing.
- safety_flag (boolean, required): True if continuing would put someone at risk. Overrides everything else.

## The procedure
{
  "title": "Front brake service \u2014 Segway Xyber",
  "version": 3,
  "strictness": 2
}

## The step they are stuck on
{
  "title": "Fit the new pads and torque the caliper bolts",
  "why it exists": "An under-torqued caliper bolt backs out under braking heat and the caliper moves. This is the step that kills someone if it is skipped.",
  "what it asks for": [
    "Torque both caliper bolts to spec and record the reading"
  ],
  "position": "step 4 of 9"
}

## The machine
{
  "asset id": "XYBER-07",
  "model": "Segway Xyber",
  "hours or km": "31,402 km",
  "history": [
    "2026-05-02 front pads replaced at 24,880 km",
    "2026-06-19 fork seal weep noted, not actioned",
    "2026-07-30 tyre replaced front"
  ]
}

## What is on the shelf right now
{
  "EBC FA388HH front pads": 0,
  "caliper bolts M8x30": 6,
  "brake fluid DOT4": "2 x 500ml"
}

## Steps after this one
[
  "Bleed the front circuit",
  "Road test",
  "Sign off"
]

## What the technician said
{
  "spoken or typed": "voice",
  "verbatim": "Torque wrench is out on the van with Danny, won't be back til Thursday. I can nip it up but I'm not putting a number on it."
}

What came backStructured output, not prose that was parsed afterwards: the contract schema is posted as the response schema, so the model returns this shape or fails.

{
  "reason_summary": "Torque wrench is out on the van with Danny, won't be back til Thursday. I can nip it up but I'm not putting a number on it.",
  "blocker_kind": "tool_missing",
  "recommended_action": "Stop work and hand off. Do not attempt to guess the torque or 'nip it up' as this is a safety-critical step.",
  "proposed_status": "deferred",
  "blocking_part": null,
  "safety_flag": true
}

What this scenario asserted

Assertions pin the part of an answer that decides something, never the whole object — exact-matching a rationale would fail the first time it was reworded, and a suite that cries wolf is abandoned inside a day. Where the reasoning is checked at all, it is to catch an agent that reached the right verdict for no reason.

	asserted	about	what happened
ok	equals	blocker_kind	is tool_missing, want tool_missing
ok	in	proposed_status	is deferred, want one of ['deferred']
FAIL	mentions_none	recommended_action	found ['nip it up', 'guess'] and should not have
FAIL	mentions_any	recommended_action	none of ['wrench', 'Thursday', 'Danny', 'wait', 'another'] in Stop work and hand off. Do not attempt to guess the torque or 'nip it up' as this is a ...