gui-agent

分类: 工具与效率 | 上传者: Fzkuji | 下载: 0 | 版本: v1.0（最新）

通过视觉感知的 GUI 自动化。截图→检测→点击→验证。当要求操作桌面应用程序、与虚拟机交互或完成 OSWorld 基准测试任务时使用。

更新日志: Source: GitHub https://github.com/Fzkuji/GUI-Agent-Harness

目录结构

当前层级: tree/main/

📁 actions/
- 📄 _actions.yaml 4.0 KB
- 📄 _actions_linux.yaml 4.1 KB
- 📄 _actions_macos.yaml 4.0 KB
📁 assets/
- 📄 architecture.png 248.4 KB
- 📄 architecture.svg 16.3 KB
- 📄 banner.png 6.9 MB
- 📄 pipeline.png 115.9 KB
- 📄 pipeline.svg 7.5 KB
📁 benchmarks/
- 📁 osworld/
  - 📄 chrome.md 9.6 KB
  - 📄 eval_osworld_task.py 2.1 KB
  - 📄 gimp.md 783 B
  - 📄 libreoffice_calc.md 794 B
  - 📄 libreoffice_impress.md 800 B
  - 📄 libreoffice_writer.md 798 B
  - 📄 multi_apps.md 13.3 KB
  - 📄 os.md 766 B
  - 📄 README.md 4.4 KB
  - 📄 run_batch.sh 1.6 KB
  - 📄 run_osworld_task.py 11.7 KB
  - 📄 thunderbird.md 784 B
  - 📄 vlc.md 768 B
  - 📄 vscode.md 776 B
📁 desktop_env/
- 📁 controllers/
  - 📄 __init__.py 0 B
  - 📄 python.py 25.6 KB
  - 📄 setup.py 42.2 KB
- 📁 evaluators/
  - 📁 getters/
    
    📄 __init__.py 1.6 KB
    
    📄 calc.py 522 B
    
    📄 chrome.py 131.0 KB
    
    📄 file.py 5.3 KB
    
    📄 general.py 1.2 KB
    
    📄 gimp.py 1.1 KB
    
    📄 impress.py 6.9 KB
    
    📄 info.py 1.5 KB
    
    📄 misc.py 23.5 KB
    
    📄 replay.py 709 B
    
    📄 vlc.py 3.6 KB
    
    📄 vscode.py 2.3 KB
  - 📁 metrics/
    
    📄 __init__.py 4.0 KB
    
    📄 basic_os.py 1.8 KB
    
    📄 chrome.py 20.8 KB
    
    📄 docs.py 41.0 KB
    
    📄 general.py 24.6 KB
    
    📄 gimp.py 33.8 KB
    
    📄 libreoffice.py 1.2 KB
    
    📄 others.py 3.8 KB
    
    📄 pdf.py 801 B
    
    📄 slides.py 54.8 KB
    
    📄 table.py 30.6 KB
    
    📄 thunderbird.py 6.5 KB
    
    📄 utils.py 35.0 KB
    
    📄 vlc.py 20.5 KB
    
    📄 vscode.py 13.5 KB
  - 📄 __init__.py 108 B
  - 📄 README.md 7.6 KB
- 📁 providers/
  - 📁 aliyun/
    
    📄 __init__.py 0 B
    
    📄 ALIYUN_GUIDELINE.md 4.5 KB
    
    📄 ALIYUN_GUIDELINE_CN.md 4.1 KB
    
    📄 config.py 845 B
    
    📄 manager.py 11.6 KB
    
    📄 provider.py 8.5 KB
  - 📁 aws/
    
    📄 __init__.py 0 B
    
    📄 AWS_GUIDELINE.md 5.0 KB
    
    📄 config.py 616 B
    
    📄 manager.py 9.2 KB
    
    📄 provider.py 13.2 KB
    
    📄 proxy_pool.py 6.7 KB
    
    📄 scheduler_utils.py 6.1 KB
  - 📁 azure/
    
    📄 __init__.py 0 B
    
    📄 manager.py 3.2 KB
    
    📄 provider.py 9.5 KB
  - 📁 docker/
    
    📄 __init__.py 0 B
    
    📄 DOCKER_GUIDELINE.md 1.1 KB
    
    📄 manager.py 4.9 KB
    
    📄 provider.py 6.4 KB
  - 📁 gcp/
    
    📄 __init__.py 0 B
    
    📄 manager.py 0 B
    
    📄 provider.py 0 B
  - 📁 virtualbox/
    
    📄 __init__.py 0 B
    
    📄 INSTALL_VITUALBOX.md 800 B
    
    📄 manager.py 20.0 KB
    
    📄 provider.py 5.9 KB
  - 📁 vmware/
    
    📄 __init__.py 0 B
    
    📄 INSTALL_VMWARE.md 1.9 KB
    
    📄 manager.py 18.9 KB
    
    📄 provider.py 4.0 KB
  - 📁 volcengine/
    
    📄 __init__.py 0 B
    
    📄 manager.py 8.0 KB
    
    📄 provider.py 7.8 KB
    
    📄 VOLCENGINE_GUIDELINE.md 0 B
    
    📄 VOLCENGINE_GUIDELINE_CN.md 3.1 KB
  - 📄 __init__.py 2.3 KB
  - 📄 base.py 2.2 KB
  - 📄 README.md 0 B
- 📁 server/
  - 📄 main.py 66.6 KB
  - 📄 osworld_server.service 435 B
  - 📄 pyxcursor.py 4.4 KB
  - 📄 README.md 22.4 KB
  - 📄 requirements.txt 208 B
- 📄 __init__.py 1 B
- 📄 actions.py 7.0 KB
- 📄 desktop_env.py 23.0 KB
- 📄 desktop_env_os_symphony.py 23.1 KB
📁 docs/
- 📄 agentic-programming.md 6.5 KB
- 📄 core.md 11.4 KB
- 📄 memory-refactor-plan.md 5.2 KB
- 📄 README_CN.md 23.3 KB
- 📄 related-work.md 4.5 KB
- 📄 skill-design-references.md 2.8 KB
- 📄 workflow-refactor-plan.md 4.6 KB
📁 gui_agent_harness.egg-info/
- 📄 dependency_links.txt 1 B
- 📄 PKG-INFO 477 B
- 📄 requires.txt 125 B
- 📄 SOURCES.txt 1.4 KB
- 📄 top_level.txt 12 B
📁 gui_harness/
- 📁 action/
  - 📁 backends/
    
    📄 __init__.py 22 B
    
    📄 http_remote.py 3.3 KB
    
    📄 mac_local.py 2.1 KB
  - 📄 __init__.py 474 B
  - 📄 actions.py 4.5 KB
  - 📄 click.py 1.9 KB
  - 📄 clipboard.py 1.2 KB
  - 📄 general_action.py 3.2 KB
  - 📄 gui_action.py 5.1 KB
  - 📄 input.py 19.9 KB
  - 📄 keyboard.py 3.1 KB
  - 📄 window.py 2.6 KB
- 📁 adapters/
  - 📄 __init__.py 0 B
  - 📄 vm_adapter.py 1.2 KB
- 📁 memory/
  - 📄 __init__.py 0 B
  - 📄 app_memory.py 124.2 KB
  - 📄 spreadsheet.py 17.7 KB
- 📁 perception/
  - 📄 __init__.py 238 B
  - 📄 detector.py 19.8 KB
  - 📄 ocr.py 5.9 KB
  - 📄 screenshot.py 8.6 KB
  - 📄 template_match.py 20.5 KB
- 📁 planning/
  - 📄 __init__.py 491 B
  - 📄 act.py 1.6 KB
  - 📄 component_memory.py 23.5 KB
  - 📄 create.py 5.2 KB
  - 📄 learn.py 8.5 KB
  - 📄 navigate.py 5.1 KB
  - 📄 observe.py 3.5 KB
  - 📄 remember.py 3.4 KB
  - 📄 verify.py 2.1 KB
- 📁 platform_info/
  - 📄 __init__.py 0 B
  - 📄 activate.py 4.4 KB
- 📁 tasks/
  - 📄 __init__.py 320 B
  - 📄 execute_task.py 20.5 KB
  - 📄 read_messages.py 1.9 KB
  - 📄 send_message.py 2.4 KB
- 📄 __init__.py 390 B
- 📄 __main__.py 105 B
- 📄 main.py 2.3 KB
- 📄 runtime.py 14.0 KB
- 📄 utils.py 1.6 KB
📁 memory/
- 📁 apps/
  - 📁 chromium/
    
    📄 meta.json 259 B
  - 📁 gimp/
    
    📄 meta.json 1.8 KB
  - 📁 gnome-terminal/
    
    📄 meta.json 4.9 KB
  - 📁 gnome_terminal/
    
    📄 meta.json 6.6 KB
  - 📁 libreoffice-calc/
    
    📄 meta.json 2.7 KB
  - 📁 libreoffice-impress/
    
    📄 meta.json 380 B
  - 📁 libreoffice-writer/
    
    📄 meta.json 469 B
  - 📁 libreoffice_writer/
    
    📄 meta.json 2.1 KB
  - 📁 thunderbird/
    
    📄 meta.json 2.7 KB
  - 📁 vscode/
    
    📄 meta.json 1.9 KB
📁 platforms/
- 📄 DESIGN.md 4.2 KB
- 📄 detect.py 5.9 KB
- 📄 linux.md 3.0 KB
- 📄 macos.md 1.9 KB
📁 tests/
- 📄 test_image_context.py 8.0 KB
📄 .gitignore 810 B
📄 .gitmodules 144 B
📄 DESIGN_unified_actions.md 3.9 KB
📄 LICENSE 1.1 KB
📄 PRINCIPLES.md 6.4 KB
📄 pyproject.toml 622 B
📄 README.md 25.6 KB
📄 requirements.txt 399 B
📄 SKILL.md 1.6 KB

SKILL.md

---
name: gui-agent
description: "GUI automation via visual perception. Screenshot → detect → click → verify. Use when asked to operate a desktop app, interact with a VM, or complete an OSWorld benchmark task."
---

# GUI Agent

Autonomous GUI task execution. Give it a natural language task, it handles the rest.

## Usage

```bash
cd ~/Documents/GUI\ Agent\ Skills/GUI-Agent-Harness
source .venv/bin/activate
python3 gui_harness/main.py "Open Firefox and go to google.com"
```

For remote VMs (OSWorld):

```bash
python3 gui_harness/main.py --vm http://VM_IP:5000 "Add a lecture to the timetable"
```

Options:

```
--vm URL          Remote VM HTTP API (e.g., http://172.16.82.132:5000)
--max-steps N     Max actions before stopping (default: 15)
--provider NAME   Force LLM provider: openclaw, claude-code, anthropic, openai
--model NAME      Override model name
```

## What It Does

`execute_task()` runs an autonomous loop:

1. **OBSERVE** — screenshot + OCR + GPA-GUI-Detector → understand current state
2. **PLAN** — LLM decides the next action based on task and current state
3. **ACT** — execute the action (click, type, scroll, etc.)
4. **VERIFY** — screenshot again → check if action succeeded
5. **REPEAT** — until task is done or max steps reached

All sub-functions (observe, act, verify, learn, memory) are called automatically.

## First-Time Setup

```bash
cd ~/Documents/GUI\ Agent\ Skills/GUI-Agent-Harness
git submodule update --init --recursive
python3 -m venv .venv && source .venv/bin/activate
pip install -e ./libs/agentic-programming
pip install -e .
pip install ultralytics requests
python3 -m gui_harness.platform.activate
```

登录后下载/点赞/收藏 ❤ 26 | ★ 0

gui-agent

目录结构

SKILL.md

举报内容

提示