Computer use

Open in product: preshos.com/agent-ops/computer-use

Computer use lets a session operate a computer the way a member would: opening applications, driving their interfaces, and carrying multi-step work across them. It runs through a loopback desktop bridge on the host backend — not screen-scraping over VNC — inside the session’s sandbox boundary. It applies where no connector or tool reaches: the long tail of software that only has a screen, not an API.

When it applies

• The target has no connector and no usable tool, and the work still needs doing inside a governed session.
• The work is visible and step-by-step: open, read, fill, confirm — sequences a reviewer can follow in the transcript.
• The environment allows it. Computer use runs inside the sandbox under the session’s environment policy, not on the member’s device.

How it drives: AX-first, not VNC

Computer use is not VNC and not screen-scraping. Control runs accessibility-first (AX tree): the agent addresses named interface elements — windows, buttons, fields — through the loopback desktop bridge on the host backend, rather than guessing pixels from a screenshot. Screenshots backstop verification (did the screen show what the action claimed?), but they are not the control plane.

Concrete examples, same shape every time:

• Open the invoicing app, read the open-invoices list, and draft the overdue summary for review — reads flow, sending anything pauses at the gate.
• Open the desktop spreadsheet, fill the named columns from the approved source rows, and save a draft copy — fills are drafts until a human approves the outcome.
• Reconcile two windows: read the record in the CRM client, confirm the matching row in the reporting app, report the mismatch — cross-app reads stay reads; any write pauses first.

Each step names its target (which app, which element, what value). A step that cannot name its target is not ready to run.

A concrete computer_use attempt, including the guardrail firing: the agent tries to launch TextEdit via computer_use twice and both attempts return the same block — “User is active; pause and observe before retrying” — because the system detects live user activity and stops the automated app launch as a safety measure. The transcript keeps the raw action_result JSON and the honest close-out: status Blocked on com.apple.TextEdit, no shell, AppleScript, clipboard, or other tools used, no existing documents touched.

• Per the run’s own rule (“if a command fails twice, report the error and stop”), it stops rather than retrying blind — retry once the desktop is idle, or bring TextEdit forward and continue from Command-N.
• Blocked is a verdict, not a stall: read what completed, what was in flight, and what never started before re-running.

Stopping: Stop cancels

Stop cancels the run. A stopped computer-use session halts further actions and records what completed, what was in flight, and what never started. Partial writes stay partial — review the transcript and the target app before re-running, and never assume a cancelled fill “probably finished.”

What a failed run reports instead of progress: a Run failed banner with the gateway error, per-runner Error rows repeating it, Open logs for the detail, and a retry composer below (“send a message to retry or add context”). No desktop-bridge status panel is shown here — a session whose host has no bridge is a scoping question for the tenant admin, not a wording problem.

• Failed after retries means stop and read: what the transcript proves happened decides the next move, never assumed device state.
• Gateway and runner errors are backend failures, not wrong prompts; retry with context or re-scope the host.

A halt you do not have to press: when the desktop is busy, the run stops itself instead of pushing through. This transcript shows a chrome_browser step failing at step 1 with “User is active; pause and observe before retrying” — no Stop button is shown here, the guardrail is the stop. The close-out names exactly what never started (snapshot, fill, save, reload, read-back, screenshot), so the next run knows what is still untouched.

• A user-active pause is a stop with a resume condition: wait for an idle desktop, then re-run — do not retry blind into a busy screen.
• Read the never-started list as the re-run plan: everything named there is still to do, nothing half-done to untangle.

Plugin install path

Computer use arrives as an installed capability, like any other: install the package from the plugin registry, grant it to the agents that need it, and confirm the session environment policy allows it. The Manage modal shows Installed status with its skill and computer_use tool, pointing back to the agent’s Plugins tab to enable it. Until all three hold, the session panel will not list it — see Plugins in session. The host backend needs a desktop bridge available; a session whose host has none is a scoping question for the tenant admin, not a wording problem.

The grant itself lives on the agent, per agent: on the agent’s Tools and Plugins Plugins tab, each row shows workspace status with an Enabled toggle. Here Computer Use is on for the Active workspace — alongside Chrome Browser — while uninstalled rows (Drive, HubSpot, Images) stay off. This tab shows the grant only; it says nothing about environment policy, which is confirmed separately.

• An agent never granted the plugin cannot carry computer use into a session, whatever the registry shows.
• Grant scope is per agent: the same plugin can be on for one agent and off for another.

Guardrails

• Approval gates apply exactly as elsewhere: reads and drafts flow, irreversible or cross-system actions pause for a human decision.
• Watch it work. Screen-driven sessions drift into wrong windows and wrong records faster than tool-driven ones — read the transcript as it goes.
• Credentials stay vaulted. Sign-ins the session needs arrive as grants from Secrets; values are never typed into chat, reasoning, or screen fields from this wiki.
• One goal per session holds double here. A computer-use thread doing three jobs in three apps is three sessions.

What next

• Browser use: scoped web work, the narrower sibling.
• Chrome browser: local-Chrome control through the extension — a separate surface with its own rules.
• Sandbox: the isolated space computer use runs inside.
• Approving agent actions: the gate computer-use actions pause at.