Desktop control with a memory.

Short answer.

OpenCode Computer Use is an OpenCode plugin that registers a local MCP server named cua_repl. The model receives two tools: js for persistent JavaScript and js_reset for a clean JavaScript session. Native operating-system permissions still control desktop access.

Plugin

The package loaded by OpenCode. It manages registration and lifecycle.

MCP server

The local stdio server named cua_repl.

Model tools

js and js_reset, not raw desktop tools.

Native runtime

open-computer-use, running separately with OS accessibility and capture permissions.

Supported hosts

OpenCode 1 and OpenCode2 on Linux, macOS, and Windows where the native runtime and permissions are available.

Install once. Keep the session.

This repository is a standalone OpenCode plugin. It contains the plugin entrypoint, the local MCP server, and the native runtime integration. The plugin registers cua_repl; the model sees only js and js_reset.

git clone [email protected]:rashidtvmr/opencode-computer-use.git
cd opencode-computer-use
npm install

For a published-package install, use:

npm install @frontendxlab/opencode-computer-use
# pnpm install @frontendxlab/opencode-computer-use also works

The package postinstall step detects OpenCode 1 versus OpenCode2 and local versus global installation. It edits JSONC without discarding comments, avoids duplicate entries, and prints both manual formats when the choice is ambiguous. Run it explicitly with npx opencode-computer-use-setup, or add --local, --global, --v1, or --v2 when you want to choose explicitly.

Recent pnpm releases can require explicit lifecycle-script approval. If pnpm reports ERR_PNPM_IGNORED_BUILDS, run pnpm approve-builds @frontendxlab/opencode-computer-use once, then rerun pnpm install. If scripts are disabled, run pnpm exec opencode-computer-use-setup manually. CI environments skip setup automatically.

Desktop onboarding is part of installation from v0.1.5. When package lifecycle scripts are allowed, postinstall first configures OpenCode and then runs opencode-computer-use-permissions --install. On macOS it runs the native doctor and opens the Open Computer Use permission onboarding when Accessibility or Screen Recording is missing; Apple still requires you to approve the protected toggles. On GNOME Linux it best-effort enables toolkit accessibility before the AT-SPI/session check. Windows validates that UI Automation is running in an interactive signed-in desktop session. Set OPENCODE_COMPUTER_USE_SKIP_PERMISSIONS=1 to opt out.

{
  "$schema": "https://opencode.ai/config.json",
  "plugin": [
    ["@frontendxlab/opencode-computer-use", { "backend": "auto" }]
  ]
}

opencode mcp list

{
  "$schema": "https://opencode.ai/config.json",
  "plugins": [
    {
      "package": "@frontendxlab/opencode-computer-use",
      "options": { "backend": "auto" }
    }
  ]
}

Start with discovery.

Use the first call to see what the native runtime can actually see. Do not guess an app name or assume that a screenshot is available.

var apps = await cua.listApps({ emit: false });
var chrome = await cua.getApp("Google Chrome");
await chrome.getAXState({ textLimit: 12000 });

The generated snippet is intentionally small. Bindings persist between calls, so use var when a later call may need to reassign the same name.

One surface. Three layers.

1. OpenCode plugin
   Loads the local package and registers the MCP server without exposing raw desktop tools.
2. cua_repl MCP server
   Provides a persistent JavaScript evaluation surface, output bounds, request validation, and worker isolation.
3. Native runtime
   Runs open-computer-use as a separate child process. OS accessibility, screen capture, and input permissions remain there.

The Codex-shaped API.

The model receives a small async API inside persistent JavaScript:

• cua.getState(options?)List the current desktop apps and emit bounded output.
• cua.listApps(options?)Return structured app records without emitting by default.
• cua.getApp(nameOrPid)Bind an app and emit its initial accessibility state.
• app.getAXState(options?)Read a fresh semantic tree before choosing an element index.
• app.getScreenshot(options?)Return local image bytes and optionally emit them with nodeRepl.
• app.click(target, options?)Use an element index first, or a bounded coordinate when necessary.
• app.typeText(text)Send literal text through the native runtime.
• app.pressKey(key)Send a named key or supported modifier chord.
• app.performSecondaryAction(index, action)Invoke an advertised accessibility action such as a context action.

Use nodeRepl.write(value) for extra text and await nodeRepl.emitImage(bytesOrDataUrl) for local images. Both are bounded before crossing the worker boundary.

What it can — and cannot — do.

This is the complete model-facing computer-use surface of the current cua_repl adapter, plus its intentional exclusions. A green check means the adapter/native runtime exposes that path; it does not mean every application or desktop session will accept it.

WebMCP, when available.

The documentation page includes a small, read-only WebMCP enhancement based on the current document.modelContext.registerTool() draft. Browsers without the preview API continue normally; no desktop-control or mutating tool is exposed through the page.

const tools = await document.modelContext.getTools();
// Preview-only discovery: search_docs, get_install_config, get_api_reference

Preview boundary: WebMCP support is experimental and requires a supporting browser, a secure context, and the tools Permissions Policy. Cloudflare Browser Run lab sessions can be used to test discovery. This page does not claim stable or universal browser support.

Make risky capabilities explicit.

{
  "backend": "auto",
  "serverName": "cua_repl",
  "safetyMode": "full",
  "appAccess": {
    "default": "allow",
    "allow": [],
    "deny": []
  },
  "timeoutMs": 30000,
  "maxTextChars": 200000,
  "maxImageBytes": 2097152,
  "allowGlobalPointerFallbacks": false
}

• backendauto, native, or a fixed custom command.
• serverNameRename the generated MCP server when a project needs isolation.
• safetyModefull or read-only. Read-only blocks mutating CUA methods and changes MCP annotations.
• appAccessOptional exact-name and ID allow or deny rules, evaluated before native access.
• timeoutMsBounds JavaScript evaluation. A timeout resets the worker session.
• maxTextCharsAggregate text budget for one tool result.
• maxImageBytesAggregate image budget for one tool result.
• allowGlobalPointerFallbacksOff by default. Enable only when system-level pointer movement is explicitly intended.

Observe, act, verify.

1. Read getAXState() before selecting an element.
2. Prefer a fresh element index over a remembered coordinate.
3. Batch deterministic actions and the next state read in one js call.
4. After navigation, dialogs, resizing, or rerendering, read state again.
5. Use js_reset when bindings need to be discarded, not when an app needs restarting.

var app = await cua.getApp("Example App");
await app.getAXState();
var state = await app.click(12);
await app.getAXState();
nodeRepl.write(state);

Know what the host can do.

macOS generally needs Accessibility and Screen Recording. Windows needs UI Automation in the interactive desktop session. Linux needs AT-SPI plus the native runtime's input and capture services. A secure desktop, lock screen, UAC surface, or missing permission is not bypassed.

Installation walks you through desktop access.

The v0.1.5+ npm/pnpm install path runs the same desktop prerequisite helper automatically. It checks the current OS, runs the native doctor, and gives platform-specific guidance. The helper can enable non-protected settings such as GNOME toolkit accessibility, but it never bypasses protected OS consent. On macOS, the native onboarding opens the correct System Settings panes and waits for you to approve Open Computer Use.app — not Terminal or OpenCode.

Manual permission and verification commands.

# Run/re-run our guided installer check
npx opencode-computer-use-permissions --strict

# Native status + onboarding UI
open-computer-use doctor

# Open the exact System Settings panes manually if needed
open "x-apple.systempreferences:com.apple.preference.security?Privacy_Accessibility"
open "x-apple.systempreferences:com.apple.preference.security?Privacy_ScreenCapture"

# Verify after approving both toggles
open-computer-use doctor
open-computer-use call list_apps

# Run/re-run our guided installer check
npx opencode-computer-use-permissions --strict

# Native runtime/session notes
open-computer-use doctor

# GNOME only: manually enable and verify toolkit accessibility
gsettings set org.gnome.desktop.interface toolkit-accessibility true
gsettings get org.gnome.desktop.interface toolkit-accessibility

# Inspect the session the runtime inherited
printf "desktop=%s  session=%s\n" "$XDG_CURRENT_DESKTOP" "$XDG_SESSION_TYPE"
test -n "$DBUS_SESSION_BUS_ADDRESS" && echo "D-Bus session available" || echo "D-Bus session not exported; runtime will try discovery"

# Verify real desktop visibility
open-computer-use call list_apps

# PowerShell
npx opencode-computer-use-permissions --strict
open-computer-use doctor

# Confirm this shell belongs to an interactive user session
Write-Host "SESSIONNAME=$env:SESSIONNAME"
(Get-Process -Id $PID).SessionId

# Verify UI Automation can see desktop applications
open-computer-use call list_apps

How to turn conditional actions into reliable ones.

• Fix the host firstRun the guided permission check and make sure the native runtime can actually list the target app before debugging clicks or typing.
• Prefer semantic actionsUse a fresh accessibility element index for click, scroll, set-value, and secondary actions. This is more portable than remembered coordinates.
• Observe after mutationsRead a new accessibility state after navigation, dialogs, rerenders, or an uncertain timeout. Never assume a reported input event changed the UI.
• Use screenshots as a fallbackWhen the accessibility tree is incomplete, a screenshot can recover visual context if the OS capture backend permits it.
• Opt into pointer fallback only when neededallowGlobalPointerFallbacks can recover coordinate-only cases, but it may move the real pointer or change focus and stays off by default.
• Choose the right Linux sessionAT-SPI works best when the process can reach the logged-in desktop bus. Wayland may restrict capture/input; semantic actions are the preferred path.

Small surface. Strong boundaries.

• No arbitrary shellThe model-facing API does not expose run_shell or a host command escape.
• Restricted VMFilesystem, process, fetch, dynamic import, code generation, and host built-ins are blocked.
• Worker isolationCPU-bound JavaScript can be terminated without wedging the MCP server.
• Read-only modeAn opt-in safety profile blocks clicks, keys, text, scroll, drags, and secondary actions before native execution.
• App policyExact-name and ID allow or deny rules provide adapter-level defense in depth.
• Bounded transportInput frames, output text, images, content items, app bindings, and native responses are bounded.
• Lifecycle cleanupTimeouts send native cancellation, and shutdown sends a best-effort turn-ended notification.
• Approval preservingConsequential actions still pass through OpenCode and OS permission boundaries.
• Safe defaultsGlobal pointer fallback is disabled until explicitly enabled.

Read SECURITY.md before using the session against banking, password managers, administrator tools, or unattended workloads.

When the session is quiet.

Start with the same three checks on every platform. They separate plugin registration problems from native desktop permission/session problems before you retry an action.

opencode mcp list
npx opencode-computer-use-permissions --strict
open-computer-use call list_apps

Permission failures by operating system.

open-computer-use doctor
open "x-apple.systempreferences:com.apple.preference.security?Privacy_Accessibility"
open-computer-use doctor

open "x-apple.systempreferences:com.apple.preference.security?Privacy_ScreenCapture"
open-computer-use doctor
open-computer-use call list_apps

tccutil reset Accessibility com.ifuryst.opencomputeruse
tccutil reset ScreenCapture com.ifuryst.opencomputeruse
open-computer-use doctor

sw_vers -productVersion
open-computer-use --version

gsettings set org.gnome.desktop.interface toolkit-accessibility true
gsettings get org.gnome.desktop.interface toolkit-accessibility
printf "desktop=%s  session=%s\n" "$XDG_CURRENT_DESKTOP" "$XDG_SESSION_TYPE"
printf "runtime=%s\n" "$XDG_RUNTIME_DIR"
test -n "$DBUS_SESSION_BUS_ADDRESS" && echo "D-Bus session available"
open-computer-use doctor
open-computer-use call list_apps

npm install [email protected]
open-computer-use --version
open-computer-use call list_apps

open-computer-use doctor
Write-Host "SESSIONNAME=$env:SESSIONNAME"
(Get-Process -Id $PID).SessionId
open-computer-use call list_apps

General failures.

Frequently asked questions.