Computer Use

Computer Use lets Klaus see the screen, move the mouse, and type on the keyboard the way a person would, so it can operate desktop apps that have no API. You need to grant Accessibility and Automation permissions first (see System permissions).

Its shape is different from an ordinary toggle:

• To the engine it is a built-in MCP server, and its switch and configuration surface live in the native capability card on the MCP page. Once enabled, the session gains a set of desktop control tools: list the apps on your computer, read the current window state of an app (with a screenshot and the accessibility tree), click, run context menu actions, fill values, select text, scroll, drag, press keys, type text, read and modify the app policy described below, plus three recording control tools. (Browser control used to be another native capability card; it now belongs to a built-in plugin, and its configuration surface is on the plugins page.)
• The actual work is done by a small native helper process shipped with the app, which Klaus drives over standard input and output. The helper calls system interfaces directly to perform actions: the Accessibility API to read the control tree, graphics event APIs to inject mouse and keyboard input, and the system screen capture framework for screenshots. No extra runtime needs to be installed. System permissions are recorded against this helper process, which is launched by the properly installed Klaus.
• Once the connector is enabled, all local apps are eligible for approval. Klaus asks before first using an app; choosing Always allow saves the decision. Existing saved entries become always-allowed apps and no longer exclude other apps. Explicitly blocked apps remain blocked, and friend tasks remain limited to their separately authorized apps.
• Settings > Computer Use lists always-allowed apps. Choose Add app and enter an app name or full application path to save it without another app question. Remove an entry and confirm to restore the app question next time. The connector switch stays on the connectors page. Plain policy entries match by normalized containment; bundle: and name: entries match exact identifiers and names, and app:self identifies the current Klaus installation.
• It does not ask again before every click: the current computer use session remembers your acceptance. Before each round of actions it must still read the window state, with a screenshot attached.
• The lists are read live from settings on every action, with no caching. If you ask it to change the list during a conversation, the next action already uses the new list — no reconnect and no restart needed. Likewise, a request like "don't touch Mail from now on" edits this list directly and takes effect immediately.
• It yields while you are using the computer: before every write action (clicks, typing, and the like) it checks whether anyone has touched the keyboard or mouse recently — input within the last 1.5 seconds that is more recent than its own last action counts as you. That action is then refused outright, and the model is told to wait 2 seconds and retry, instead of fighting you for the mouse.
• It stops by default when the screen is locked: with the screen locked, it simply tells the model "let the user unlock it themselves". Automatic unlocking is optional and deliberately conservative: the login password is only ever typed into the system dialog and stored only in the system keychain, never entering the conversation or the logs; and if anyone appears to be physically present (keyboard or mouse activity in the last few seconds), it will never unlock automatically. Also note that macOS turns on secure input at the lock screen, so keystrokes injected by an ordinary signed app are rejected by the system — automatic unlocking will most likely fail and fall back to asking you to unlock it yourself.
• System permissions such as Accessibility and Screen Recording can only be granted by you in System Settings; no program can write them on your behalf. And because macOS caches permission decisions for already-running processes, restart Klaus after granting them.

Demo recording (you do it once to show it):

• The entry point is "Record skill" on the skills page: it opens a session in which Klaus starts the recording. What gets recorded is your own actions — which control you clicked, what text you typed, which window you switched to — captured as accessibility semantics (app name, control role and title) rather than pixel coordinates, with no screen video recorded.
• While recording, a small floating panel appears on screen with only two buttons: stop and discard. A single recording lasts at most 30 minutes and wraps up automatically if you forget to stop it (a crash of the connecting process loses nothing either — events already written to disk can still be finalized later).
• After you stop, Klaus reads the event stream and turns it into a skill (~/.klaus/skills/<short-name>/SKILL.md, always containing an overview, prerequisites, reproduction steps, verification, and items to confirm). So the next time it "does it again", it follows this written procedure rather than replaying a video — it can handle moved windows and pick a more suitable tool instead of a hard click.
• Privacy: password fields are never recorded (secure input is probed on every keystroke), credential-like text is redacted before it hits disk, and apps on the deny list are not recorded — "never touch this app" also means "don't record it either".