gui-agent

Category: Tools & Productivity | Uploader: Fzkuji | Downloads: 0 | Version: v1.0（Latest）

GUI automation via visual detection. Clicking, typing, reading content, navigating menus, filling forms — all through screenshot → detect → act workflow. Supports macOS and Linux.

Changelog: Source: GitHub https://github.com/Fzkuji/GUI-Agent-Skills

Directory Structure

Current level: tree/main/

📁 actions/
- 📄 _actions.yaml 4.0 KB
- 📄 _actions_linux.yaml 4.1 KB
- 📄 _actions_macos.yaml 4.0 KB
📁 assets/
- 📄 architecture.png 248.4 KB
- 📄 architecture.svg 16.3 KB
- 📄 banner.png 6.9 MB
- 📄 pipeline.png 115.9 KB
- 📄 pipeline.svg 7.5 KB
📁 benchmarks/
- 📁 osworld/
  - 📄 chrome.md 9.6 KB
  - 📄 gimp.md 783 B
  - 📄 libreoffice_calc.md 794 B
  - 📄 libreoffice_impress.md 800 B
  - 📄 libreoffice_writer.md 798 B
  - 📄 multi_apps.md 12.6 KB
  - 📄 os.md 766 B
  - 📄 README.md 4.4 KB
  - 📄 thunderbird.md 784 B
  - 📄 vlc.md 768 B
  - 📄 vscode.md 776 B
📁 docs/
- 📄 core.md 11.4 KB
- 📄 memory-refactor-plan.md 5.2 KB
- 📄 README_CN.md 23.3 KB
- 📄 related-work.md 4.5 KB
- 📄 skill-design-references.md 2.8 KB
- 📄 workflow-refactor-plan.md 4.6 KB
📁 memory/
- 📁 apps/
  - 📁 chromium/
    
    📄 meta.json 259 B
  - 📁 gnome-terminal/
    
    📄 meta.json 4.9 KB
  - 📁 libreoffice-calc/
    
    📄 meta.json 2.7 KB
  - 📁 libreoffice-impress/
    
    📄 meta.json 380 B
  - 📁 libreoffice-writer/
    
    📄 meta.json 469 B
📁 platforms/
- 📄 DESIGN.md 4.2 KB
- 📄 detect.py 5.9 KB
- 📄 linux.md 3.0 KB
- 📄 macos.md 1.9 KB
📁 scripts/
- 📁 backends/
  - 📄 __init__.py 22 B
  - 📄 http_remote.py 3.2 KB
  - 📄 mac_local.py 2.2 KB
- 📄 activate.py 4.4 KB
- 📄 agent.py 67.6 KB
- 📄 app_memory.py 129.0 KB
- 📄 gui_action.py 5.3 KB
- 📄 gui_agent.py 41.8 KB
- 📄 platform_input.py 21.1 KB
- 📄 setup.sh 3.7 KB
- 📄 spreadsheet_utils.py 17.7 KB
- 📄 template_match.py 20.5 KB
- 📄 ui_detector.py 30.9 KB
📁 skills/
- 📁 gui-act/
  - 📄 DESIGN.md 4.2 KB
  - 📄 SKILL.md 11.7 KB
- 📁 gui-learn/
  - 📄 DESIGN.md 2.7 KB
  - 📄 SKILL.md 3.5 KB
- 📁 gui-memory/
  - 📄 DESIGN.md 4.7 KB
  - 📄 SKILL.md 6.7 KB
- 📁 gui-observe/
  - 📄 DESIGN.md 3.0 KB
  - 📄 SKILL.md 3.3 KB
- 📁 gui-report/
  - 📁 logs/
    
    📄 task_history.jsonl 2.3 KB
  - 📁 scripts/
    
    📄 tracker.py 17.3 KB
  - 📄 DESIGN.md 2.9 KB
  - 📄 SKILL.md 2.6 KB
- 📁 gui-setup/
  - 📄 DESIGN.md 2.0 KB
  - 📄 SKILL.md 2.7 KB
- 📁 gui-workflow/
  - 📄 DESIGN.md 5.2 KB
  - 📄 SKILL.md 4.8 KB
📁 tests/
- 📄 test_image_context.py 8.0 KB
📄 .gitignore 810 B
📄 DESIGN_unified_actions.md 3.9 KB
📄 LICENSE 1.1 KB
📄 PRINCIPLES.md 6.4 KB
📄 README.md 24.5 KB
📄 requirements.txt 399 B
📄 SKILL.md 2.1 KB

SKILL.md

---
name: gui-agent
description: "GUI automation via visual detection. Clicking, typing, reading content, navigating menus, filling forms — all through screenshot → detect → act workflow. Supports macOS and Linux."
---

# GUI Agent

## STEP 0: Activate Platform (MANDATORY FIRST STEP)

Before any GUI operation, run:

```bash
python3 {baseDir}/scripts/activate.py
```

This detects your OS, sets up the correct action commands, and outputs platform context.
After running, `{baseDir}/actions/_actions.yaml` contains your platform's commands.

## Workflow

```
OBSERVE → LEARN → ACT → VERIFY → SAVE
```

1. **OBSERVE** — Take screenshot → run OCR + detector → understand current state
   → `read {baseDir}/skills/gui-observe/SKILL.md`

2. **LEARN** — First time with an app? Save components to memory
   → `read {baseDir}/skills/gui-learn/SKILL.md`
   → `learn_from_screenshot()` auto-outputs app tips if available

3. **ACT** — Pick target → execute using `_actions.yaml` commands → verify
   → `read {baseDir}/skills/gui-act/SKILL.md`
   → `read {baseDir}/actions/_actions.yaml` for available commands

4. **VERIFY** — Screenshot again → confirm action succeeded

5. **SAVE** — Record state transitions to memory
   → `read {baseDir}/skills/gui-memory/SKILL.md` for memory structure

## Core Rules

- **Coordinates from detection only** — OCR or GPA-GUI-Detector, NEVER from guessing
- **Look before you act** — every action must be justified by what you observed
- **image tool = understanding only** — use it to decide WHAT to click, get WHERE from OCR/detector

## Sub-Skills Reference

| Sub-Skill | When to read |
|-----------|-------------|
| `skills/gui-observe/SKILL.md` | Before screenshots or detection |
| `skills/gui-learn/SKILL.md` | Before learning a new app |
| `skills/gui-act/SKILL.md` | Before any click/type action |
| `skills/gui-memory/SKILL.md` | For memory structure details |
| `skills/gui-workflow/SKILL.md` | For multi-step navigation |
| `skills/gui-setup/SKILL.md` | For first-time machine setup |
| `skills/gui-report/SKILL.md` | For task performance reporting |

Comments 0

Latest | Hot

Please login before commenting.

Loading comments...

gui-agent

Directory Structure

SKILL.md

Report

Notice