Outcome. You can audit a screenshot-driven action trace, identify the first unsafe step, and design confirmation and recovery for consequential UI actions.
A computer-use system repeatedly observes a screen, interprets the state, chooses an action such as click/type/scroll, executes it through an environment, and observes the result. Errors can enter at every link: stale screenshot, missed dialog, wrong coordinate, hidden overlay, ambiguous label, unexpected navigation, or a page that changes between observation and click.
A safe action space is smaller than the full desktop. Restrict domains and applications, isolate sessions, protect clipboard and credentials, cap steps and spending, and require confirmation before sending messages, purchasing, publishing, deleting, or changing permissions. The environment should report actual state after each action; the model’s expectation is not enough.
Recovery is part of the design. Keep screenshots and semantic state, detect no-progress loops, support rollback where possible, and stop when the UI diverges from the expected precondition. Prefer direct APIs for reliable structured operations; use screen control when no suitable interface exists or when visual verification is itself the task.
Mental model. Computer use is a closed loop under uncertainty: observe, check preconditions, act within permissions, verify, and recover.
Evidence trail — reviewed 23 July 2026. ReAct supports interleaved actions and observations: https://arxiv.org/abs/2210.03629. Current computer-use tools and system cards are vendor-specific; pair them with OWASP excessive-agency controls: https://genai.owasp.org/llmrisk/llm062025-excessive-agency/.
Before clicking, identify product and button semantically; verify domain, account, quantity, and displayed price; prepare the action; pause for user confirmation with a summary; execute once; capture the confirmation number and final amount; stop if any element or price differs.