Computer use
Open in product: preshos.com/agent-ops/computer-use
Computer use lets a session operate a computer the way a member would: opening applications, driving their interfaces, and carrying multi-step work across them. It runs through a loopback desktop bridge on the host backend — not screen-scraping over VNC — inside the session’s sandbox boundary. It applies where no connector or tool reaches: the long tail of software that only has a screen, not an API.
When it applies
Section titled “When it applies”- The target has no connector and no usable tool, and the work still needs doing inside a governed session.
- The work is visible and step-by-step: open, read, fill, confirm — sequences a reviewer can follow in the transcript.
- The environment allows it. Computer use runs inside the sandbox under the session’s environment policy, not on the member’s device.
How it drives: AX-first, not VNC
Section titled “How it drives: AX-first, not VNC”Computer use is not VNC and not screen-scraping. Control runs accessibility-first (AX tree): the agent addresses named interface elements — windows, buttons, fields — through the loopback desktop bridge on the host backend, rather than guessing pixels from a screenshot. Screenshots backstop verification (did the screen show what the action claimed?), but they are not the control plane.
Concrete examples, same shape every time:
- Open the invoicing app, read the open-invoices list, and draft the overdue summary for review — reads flow, sending anything pauses at the gate.
- Open the desktop spreadsheet, fill the named columns from the approved source rows, and save a draft copy — fills are drafts until a human approves the outcome.
- Reconcile two windows: read the record in the CRM client, confirm the matching row in the reporting app, report the mismatch — cross-app reads stay reads; any write pauses first.
Each step names its target (which app, which element, what value). A step that cannot name its target is not ready to run.
A concrete computer_use attempt, including the guardrail firing: the agent tries to launch TextEdit via computer_use twice and both attempts return the same block — “User is active; pause and observe before retrying” — because the system detects live user activity and stops the automated app launch as a safety measure. The transcript keeps the raw action_result JSON and the honest close-out: status Blocked on com.apple.TextEdit, no shell, AppleScript, clipboard, or other tools used, no existing documents touched.
- Per the run’s own rule (“if a command fails twice, report the error and stop”), it stops rather than retrying blind — retry once the desktop is idle, or bring TextEdit forward and continue from Command-N.
- Blocked is a verdict, not a stall: read what completed, what was in flight, and what never started before re-running.
Stopping: Stop cancels
Section titled “Stopping: Stop cancels”Stop cancels the run. A stopped computer-use session halts further actions and records what completed, what was in flight, and what never started. Partial writes stay partial — review the transcript and the target app before re-running, and never assume a cancelled fill “probably finished.”
- Hide keeps the run alive behind the preview; reopen the session to watch again.
- Hiding the preview never pauses the run; the transcript keeps recording.
- Stop from the preview chrome cancels the run; the close-out names completed / in-flight / never-started.
- Never assume a cancelled fill finished; partial writes stay partial.
Hide and Stop
Section titled “Hide and Stop”Hide and Stop are different verbs — do not mix them. Hide dismisses the run surface (the preview, the panel) while the run keeps going wherever it is allowed to; nothing is cancelled, nothing is paused, the transcript keeps recording. Stop ends the run: further actions halt, and the close-out names what completed, what was in flight, and what never started.
- Hide when you need the screen back but the run should continue — for example hiding a floating preview to reclaim space while background control carries on. The run is still live; reopen the session to watch it again.
- Stop when the run should not continue — wrong app, wrong record, changed mind. Read the close-out before re-running: partial writes stay partial, and a cancelled fill never “probably finished.”
- Treat the product UI as the source of truth for the exact Hide / Stop placement and labels.
What a failed run reports instead of progress: a Run failed banner with the gateway error, per-runner Error rows repeating it, Open logs for the detail, and a retry composer below (“send a message to retry or add context”). No desktop-bridge status panel is shown here — a session whose host has no bridge is a scoping question for the tenant admin, not a wording problem.
- Failed after retries means stop and read: what the transcript proves happened decides the next move, never assumed device state.
- Gateway and runner errors are backend failures, not wrong prompts; retry with context or re-scope the host.
A halt you do not have to press: when the desktop is busy, the run stops itself instead of pushing through. This transcript shows a chrome_browser step failing at step 1 with “User is active; pause and observe before retrying” — no Stop button is shown here, the guardrail is the stop. The close-out names exactly what never started (snapshot, fill, save, reload, read-back, screenshot), so the next run knows what is still untouched.
- A user-active pause is a stop with a resume condition: wait for an idle desktop, then re-run — do not retry blind into a busy screen.
- Read the never-started list as the re-run plan: everything named there is still to do, nothing half-done to untangle.
Background control
Section titled “Background control”- Background keeps the run alive without holding the foreground desktop.
- The user-active guardrail still fires on the controlled surface; the transcript stays the record of what ran.
Background control lets the run carry on without holding the foreground desktop hostage. The member gets the screen back for other work while the session keeps driving its own controlled surface in the background.
- Background does not relax the safety rules. The user-active guardrail still applies on the controlled surface: if live user activity is detected there, automated actions pause and the transcript records the block — same “User is active; pause and observe before retrying” verdict as the foreground runs above.
- The transcript stays the record of what ran while you were looking elsewhere. Read completed / in-flight / never-started before acting on anything the run claims it did — background runs are the easiest to assume about and the worst to assume about.
- Treat the product UI as the source of truth for how background is entered and exited (toggle, per-run setting, or session default).
Floating live preview
Section titled “Floating live preview”- The preview is observe-only — watching is not steering; keep it visible on screen-driven runs.
- Drift caught early is a Stop, not a re-prompt.
The floating live preview is a small always-on-top view of the desktop the agent is driving. It answers “what is it doing right now” without surrendering the whole screen to the run.
- The preview is observe-only: watching is not steering. To intervene, use the session controls — Hide to dismiss the preview, Stop to cancel the run. Do not try to click through the preview into the controlled desktop.
- Keep the preview visible on screen-driven runs. Computer-use sessions drift into wrong windows and wrong records faster than tool-driven ones — drift caught in the preview is a Stop; drift caught three apps later is cleanup.
- Treat the product UI as the source of truth for preview placement, sizing, and dismiss behavior.
Agent cursor
Section titled “Agent cursor”- The agent cursor marks the agent-driven pointer versus the member pointer; follow it in the preview to track intent step to step.
- A cursor that cannot be told apart from yours is a product bug — report it.
The agent cursor is the distinct pointer the run drives on the controlled desktop — visually separate from the member’s own pointer so there is never doubt about who moved the mouse.
- Follow the agent cursor in the floating preview to track intent step to step: which app it entered, which element it aimed at, what value it set. A step whose cursor you cannot account for is a step to re-read in the transcript.
- A cursor that cannot be told apart from the member’s own is a product bug, not a documentation gap — report it with the session transcript rather than working around it.
- Treat the product UI as the source of truth for the exact cursor treatment.
Plugin install path
Section titled “Plugin install path”Computer use arrives as a granted capability: on default Agent Ops agents the package is installed and granted at provision time, like the rest of the foundational set — see Plugins in session. The plugin’s Workspace settings choose what the agent can access on the Mac — App access scope, what the agent can do (Make changes), Show agent activity, and Allow screenshots — pointing back to the agent’s Plugins tab to enable it. On custom agents, or where the grant was switched off, the manual path still applies: install the package from the plugin registry (preshos.com/ai/plugins), grant it to the agents that need it, and confirm the session environment policy allows it. Until grant and environment policy hold, the session panel will not list it. The host backend needs a desktop bridge available; a session whose host has none is a scoping question for the tenant admin, not a wording problem.
The grant itself lives on the agent, per agent: on the agent’s Tools and Plugins Plugins tab, each row shows workspace status with an Enabled toggle. Here Computer Use is on for the Active workspace — alongside Chrome Browser — while opt-in rows (Drive, HubSpot) stay off. This tab shows the grant only; it says nothing about environment policy, which is confirmed separately.
- An agent never granted the plugin cannot carry computer use into a session, whatever the registry shows.
- Grant scope is per agent: the same plugin can be on for one agent and off for another.
Chrome extension update notice
Section titled “Chrome extension update notice”- The stale-extension message names the outdated helper; update the extension before re-running.
- A run that fails after the update names the transcript verdict, never assumed state.
An outdated Chrome extension can surface an update message inside a computer-use run. Treat it as environment maintenance, not a run failure: update the extension first, then re-run. If the run failed behind the notice, read the transcript verdict for what completed before anything else — same rule as any failed run, never assumed device state.
- Treat the product UI as the source of truth for the exact update message and update path.
Guardrails
Section titled “Guardrails”- Approval gates apply exactly as elsewhere: reads and drafts flow, irreversible or cross-system actions pause for a human decision.
- Watch it work. Screen-driven sessions drift into wrong windows and wrong records faster than tool-driven ones — read the transcript as it goes. The #1998 floating preview plus agent cursor exist for exactly this: keep the preview up and follow the agent cursor step to step.
- Background is not unsupervised. A run carrying on behind your other work still pauses on the user-active guardrail and still needs its transcript read before you act on what it claims.
- Credentials stay vaulted. Sign-ins the session needs arrive as grants from Secrets; values are never typed into chat, reasoning, or screen fields from this wiki.
- One goal per session holds double here. A computer-use thread doing three jobs in three apps is three sessions.
What next
Section titled “What next”- Browser use: scoped web work, the narrower sibling.
- Chrome browser: local-Chrome control through the extension — a separate surface with its own rules.
- Sandbox: the isolated space computer use runs inside.
- Approving agent actions: the gate computer-use actions pause at.