SwiftAutoGUI

Requirements

• macOS 26.0+
• Swift 6.0+

Installation

Install from the marketplace

Inside Claude Code:

/plugin marketplace add NakaokaRei/SwiftAutoGUI
/plugin install swift-auto-gui@swift-auto-gui

This installs the macos-control skill, which is invoked as /swift-auto-gui:macos-control. The skill walks Claude through installing the sagui binary the first time it's needed (Swift 6.2+ toolchain required).

Example Usage

If you would like to know more details, please refer to the DocC Style Document.

Basic Usage

import SwiftAutoGUI

let backend = OpenAIVisionBackend(apiKey: "sk-...", model: "gpt-4o")
let agent = Agent(backend: backend, maxIterations: 15)

let result = try await agent.run(goal: "Open Safari and search for Swift")
print("Completed: \(result.completed), Steps: \(result.iterationsUsed)")

Basic Usage

import SwiftAutoGUI

// Execute single actions
await Action.leftClick.execute()
await Action.write("Hello, World!").execute()
await Action.keyShortcut([.command, .a]).execute()  // Select all

// Build and execute action sequences
let actions: [Action] = [
    .move(to: CGPoint(x: 100, y: 100)),
    .wait(0.5),
    .leftClick,
    .write("Hello, SwiftAutoGUI!"),
    .keyShortcut([.returnKey])
]
await actions.execute()

Complex Automation Example

// Example: Copy text from one app and paste to another
let copyPasteWorkflow: [Action] = [
    // Focus on source app
    .move(to: CGPoint(x: 100, y: 200)),
    .leftClick,
    .wait(0.5),
    
    // Select and copy text
    Action.selectAll(),
    .wait(0.2),
    Action.copy(),
    .wait(0.5),
    
    // Switch to destination app (using keyboard shortcut)
    .keyShortcut([.command, .tab]),
    .wait(1.0),
    
    // Paste the text
    .move(to: CGPoint(x: 500, y: 400)),
    .leftClick,
    Action.paste(),
    
    // Save the document
    Action.save()
]

await copyPasteWorkflow.execute()

Demo

AI Agent that autonomously observes the screen and executes actions to achieve a goal.

sagui agent "Open Safari and search for Swift"

<img src="https://github.com/user-attachments/assets/0b501faa-ff8a-4fee-8bf7-9db49bbd6766" alt="Demo: sagui agent" width="900">

With options

sagui agent "Click the trash icon" --model gpt-4o --max-iterations 15 --delay 2.0

Using environment variable for the API key

export OPENAI_API_KEY=sk-... sagui agent "Open Terminal" ```

Swift Package Manager

SwiftAutoGUI is available through Swift Package Manager.

in Package.swift add the following:

dependencies: [
    // Dependencies declare other packages that this package depends on.
    .package(url: "https://github.com/NakaokaRei/SwiftAutoGUI", branch: "master")
],
targets: [
    .target(
        name: "MyProject",
        dependencies: [..., "SwiftAutoGUI"]
    )
    ...
]

Claude Code Plugin

SwiftAutoGUI ships as a Claude Code plugin so Claude can control macOS GUI applications via the sagui CLI — taking screenshots, clicking buttons, typing text, scrolling, and more.