openai-voice-skill

Category: Content & Multimedia | Uploader: nia-agent-cyber | Downloads: 0 | Version: v1.0（Latest）

Add real-time phone calling to AI agents using OpenAI Realtime API and Twilio Media Streams. Use when you want an AI agent to make or receive phone calls with sub-200ms latency, bidirectional audio streaming, and session continuity across voice, Telegram, and email channels. Requires Python 3.9+, a Twilio phone number, and an OpenAI API key with Realtime API access.

Changelog: Source: GitHub https://github.com/nia-agent-cyber/openai-voice-skill

Directory Structure

Current level: tree/main/

📁 .github/
- 📁 ISSUE_TEMPLATE/
  - 📄 bug_report.md 264 B
  - 📄 feature_request.md 177 B
- 📁 workflows/
  - 📄 tests.yml 624 B
📁 channel-plugin/
- 📁 dist/
  - 📁 adapters/
    
    📄 config.d.ts 2.6 KB
    
    📄 config.d.ts.map 1.3 KB
    
    📄 config.js 3.9 KB
    
    📄 config.js.map 2.8 KB
    
    📄 gateway.d.ts 2.3 KB
    
    📄 gateway.d.ts.map 1.2 KB
    
    📄 gateway.js 5.2 KB
    
    📄 gateway.js.map 3.5 KB
    
    📄 inbound.d.ts 7.8 KB
    
    📄 inbound.d.ts.map 3.5 KB
    
    📄 inbound.js 13.7 KB
    
    📄 inbound.js.map 8.9 KB
    
    📄 inbound.test.d.ts 100 B
    
    📄 inbound.test.d.ts.map 140 B
    
    📄 inbound.test.js 6.1 KB
    
    📄 inbound.test.js.map 6.3 KB
    
    📄 outbound.d.ts 1.9 KB
    
    📄 outbound.d.ts.map 693 B
    
    📄 outbound.js 4.1 KB
    
    📄 outbound.js.map 2.4 KB
    
    📄 session-bridge.d.ts 5.8 KB
    
    📄 session-bridge.d.ts.map 3.0 KB
    
    📄 session-bridge.js 41.6 KB
    
    📄 session-bridge.js.map 33.6 KB
    
    📄 status.d.ts 2.4 KB
    
    📄 status.d.ts.map 1.4 KB
    
    📄 status.js 4.7 KB
    
    📄 status.js.map 3.2 KB
  - 📄 index.d.ts 932 B
  - 📄 index.d.ts.map 362 B
  - 📄 index.js 2.1 KB
  - 📄 index.js.map 1.1 KB
  - 📄 meta.d.ts 160 B
  - 📄 meta.d.ts.map 194 B
  - 📄 meta.js 466 B
  - 📄 meta.js.map 389 B
  - 📄 types.d.ts 2.8 KB
  - 📄 types.d.ts.map 2.3 KB
  - 📄 types.js 229 B
  - 📄 types.js.map 115 B
- 📁 src/
  - 📁 adapters/
    
    📄 config.ts 4.9 KB
    
    📄 gateway.ts 6.1 KB
    
    📄 inbound.test.ts 5.9 KB
    
    📄 inbound.ts 16.4 KB
    
    📄 outbound.ts 4.4 KB
    
    📄 session-bridge.ts 40.5 KB
    
    📄 status.ts 5.4 KB
  - 📄 index.ts 2.2 KB
  - 📄 meta.ts 478 B
  - 📄 types.ts 2.6 KB
- 📄 index.ts 731 B
- 📄 openclaw.plugin.json 934 B
- 📄 package-lock.json 417.7 KB
- 📄 package.json 618 B
- 📄 README.md 2.7 KB
- 📄 tsconfig.json 481 B
📁 config/
- 📄 agent.json 1.7 KB
- 📄 inbound.json 462 B
- 📄 integration-config.yaml 4.2 KB
- 📄 openclaw-voice-config.yaml 3.7 KB
- 📄 phone_mapping.json 955 B
📁 docs/
- 📄 bridge-cleanup.md 5.7 KB
- 📄 BUG_ANALYSIS_ask_openclaw.md 8.4 KB
- 📄 CALCOM_OUTREACH.md 7.8 KB
- 📄 COMPETITIVE_ANALYSIS_VISIONCLAW.md 10.9 KB
- 📄 INBOUND.md 9.5 KB
- 📄 ISSUE_38_ROOT_CAUSE.md 4.0 KB
- 📄 MISSED_CALL_TUTORIAL.md 14.0 KB
- 📄 OBSERVABILITY.md 9.0 KB
- 📄 SHOW_HN_DRAFT.md 1.4 KB
- 📄 streaming-tool-responses.md 12.7 KB
- 📄 TOOL_INTEGRATION.md 6.7 KB
📁 examples/
- 📄 missed_call_handler.py 3.3 KB
- 📄 README.md 497 B
📁 scripts/
- 📄 call_context_store.py 3.0 KB
- 📄 call_metrics.py 41.3 KB
- 📄 call_recording.py 23.8 KB
- 📄 cleanup_zombie_calls.py 4.5 KB
- 📄 context_example.py 6.5 KB
- 📄 fix-bridge-conflict.sh 3.5 KB
- 📄 inbound_handler.py 16.5 KB
- 📄 metrics_server.py 13.8 KB
- 📄 openclaw-webhook-bridge.py 14.8 KB
- 📄 openclaw_bridge.py 21.6 KB
- 📄 openclaw_executor.py 19.5 KB
- 📄 outbound_call_example.py 5.6 KB
- 📄 realtime_tool_handler.py 27.3 KB
- 📄 requirements-dev.txt 94 B
- 📄 requirements.txt 150 B
- 📄 security_utils.py 13.4 KB
- 📄 session_context.py 24.9 KB
- 📄 smart_chunker.py 7.0 KB
- 📄 start-integrated-system.sh 8.8 KB
- 📄 test-integration.sh 6.3 KB
- 📄 test_call_integration.py 5.9 KB
- 📄 test_context_integration.py 14.6 KB
- 📄 test_enhanced_context.py 2.4 KB
- 📄 test_recording.py 4.8 KB
- 📄 user_context.py 13.4 KB
- 📄 webhook-server.py 63.4 KB
📁 src/
- 📁 bridge/
  - 📄 python-webhook-bridge.ts 10.6 KB
- 📁 channel/
  - 📁 voice/
    
    📄 call-handler.ts 13.8 KB
    
    📄 context-manager.ts 12.2 KB
    
    📄 index.ts 5.8 KB
    
    📄 integrated-call-handler.ts 14.4 KB
    
    📄 integrated-plugin.ts 11.3 KB
    
    📄 session-manager.ts 10.6 KB
    
    📄 types.ts 4.1 KB
- 📄 package.json 1.6 KB
- 📄 README.md 7.8 KB
- 📄 tsconfig.json 571 B
📁 tests/
- 📄 test_calendar_guard.py 6.1 KB
- 📄 test_call_metrics.py 18.8 KB
- 📄 test_call_recording.py 20.6 KB
- 📄 test_cleanup_zombie_calls.py 7.1 KB
- 📄 test_error_handling.py 7.6 KB
- 📄 test_example_scripts.py 18.0 KB
- 📄 test_inbound.py 8.9 KB
- 📄 test_inbound_handler.py 10.5 KB
- 📄 test_metrics_server.py 15.7 KB
- 📄 test_openclaw_bridge.py 18.0 KB
- 📄 test_openclaw_webhook_bridge.py 18.7 KB
- 📄 test_realtime_tool_handler.py 18.1 KB
- 📄 test_realtime_tool_handler_extended.py 15.4 KB
- 📄 test_scripts_coverage.py 9.5 KB
- 📄 test_security_utils.py 13.8 KB
- 📄 test_session_context.py 19.2 KB
- 📄 test_smart_chunker.py 12.3 KB
- 📄 test_user_context_extended.py 16.0 KB
- 📄 test_webhook_server.py 11.8 KB
- 📄 test_webhook_server_extended.py 28.7 KB
- 📄 test_webhook_server_extra2.py 26.2 KB
📄 .coverage 52.0 KB
📄 .env.example 1.1 KB
📄 .gitignore 129 B
📄 COMMS_DRAFTS.md 5.4 KB
📄 COMMS_LOG.md 8.4 KB
📄 COMMS_PLAN.md 39.0 KB
📄 CONTRIBUTING.md 1.9 KB
📄 CTXLY_FOLLOWUP_DRAFT.md 1.6 KB
📄 DECISIONS.md 20.7 KB
📄 DEVTO_POST_DRAFT.md 13.2 KB
📄 FOLLOWING_LOG.md 1.7 KB
📄 IMPLEMENTATION_COMPLETE.md 5.5 KB
📄 IMPLEMENTATION_SUMMARY.md 4.8 KB
📄 INDIEHACKERS_POST_DRAFT.md 4.0 KB
📄 INTEGRATION_GUIDE.md 11.3 KB
📄 ISSUE_17_IMPLEMENTATION.md 3.9 KB
📄 LICENSE 1.1 KB
📄 phone_mapping.json 507 B
📄 PR_SUMMARY.md 6.0 KB
📄 PRODUCTHUNT_POST_DRAFT.md 4.2 KB
📄 PROGRESS.md 9.7 KB
📄 PROTOCOL.md 6.3 KB
📄 pyproject.toml 134 B
📄 README.md 12.3 KB
📄 REDDIT_POST_DRAFT.md 2.4 KB
📄 requirements-dev.txt 66 B
📄 requirements.txt 226 B
📄 send_email_outreach.py 3.1 KB
📄 SESSION_CONTEXT_README.md 9.3 KB
📄 skill.json 706 B
📄 SKILL.md 5.8 KB
📄 SPRINT_1_TEST_SCENARIOS.md 2.4 KB
📄 SPRINT_2_TEST_SCENARIOS.md 2.9 KB
📄 SPRINT_3_TEST_SCENARIOS.md 3.0 KB
📄 STATUS.md 42.0 KB
📄 STRATEGY.md 142.7 KB
📄 SYNC_NOTES.md 4.3 KB
📄 test_context.py 1.8 KB
📄 VOICE_ARCHITECTURE_RESEARCH.md 16.1 KB
📄 VOICE_OC_INTEGRATION_PRD.md 8.5 KB

SKILL.md

---
name: openai-voice-skill
description: Add real-time phone calling to AI agents using OpenAI Realtime API and Twilio Media Streams. Use when you want an AI agent to make or receive phone calls with sub-200ms latency, bidirectional audio streaming, and session continuity across voice, Telegram, and email channels. Requires Python 3.9+, a Twilio phone number, and an OpenAI API key with Realtime API access.
license: AGPL-3.0
compatibility: Requires Python 3.9+, Twilio account with a phone number, OpenAI API key with Realtime API access, and a publicly reachable webhook URL (ngrok, cloudflared, or deployed server). Compatible with Cursor, VS Code Copilot, Claude Code, OpenClaw, Gemini CLI, OpenHands, JetBrains Junie, Cody, Continue, Aider, Zed AI, and any agent platform that can run a Python process and make HTTP requests.
metadata:
  author: nia-agent-cyber
  version: "2.0"
  repo: https://github.com/nia-agent-cyber/openai-voice-skill
  tests: "727"
  coverage: "75%"
---

# openai-voice-skill

Open-source phone calling for AI agents. Bridges Twilio Media Streams to the OpenAI Realtime WebSocket API so your agent can make and receive phone calls with sub-200ms latency and natural barge-in support.

## Architecture

```
Inbound call:
  Phone → Twilio → POST /voice/incoming
       → TwiML <Connect><Stream url="wss://your-server/media-stream"/>
       → webhook-server.py (WebSocket bridge)
       → OpenAI Realtime API (wss://api.openai.com/v1/realtime)

Outbound call:
  POST /call → Twilio dials number → on answer → same TwiML flow
```

**Audio bridge** (inside `webhook-server.py`):
- Twilio sends: µ-law 8 kHz (base64)
- OpenAI expects: PCM16 24 kHz (base64)
- Conversion uses `audioop` (stdlib < 3.13) or `audioop-lts` (3.13+)

**No deprecated SIP endpoints.** This skill uses Twilio Media Streams (WebSocket) — not `sip.api.openai.com`, which OpenAI deprecated. All audio flows through `webhook-server.py`.

## Prerequisites

1. **Python 3.9+** with pip
2. **Twilio account** with a voice-capable phone number
3. **OpenAI API key** with `gpt-4o-realtime-preview` model access
4. **Public webhook URL** — ngrok, cloudflared, or a deployed server

## Setup

### 1. Clone and install

```bash
git clone https://github.com/nia-agent-cyber/openai-voice-skill.git
cd openai-voice-skill
python -m venv venv && source venv/bin/activate
pip install -r requirements.txt
```

### 2. Configure environment

```bash
cp .env.example .env
```

Edit `.env`:

```bash
OPENAI_API_KEY=sk-...
TWILIO_ACCOUNT_SID=AC...
TWILIO_AUTH_TOKEN=...
TWILIO_PHONE_NUMBER=+1XXXXXXXXXX   # E.164 format
PUBLIC_URL=https://your-tunnel.example.com
PORT=8080
ALLOW_INBOUND_CALLS=true           # set false to disable inbound
```

### 3. Expose your local server (development)

```bash
# Option A: cloudflared (recommended, no account needed)
cloudflared tunnel --url http://localhost:8080

# Option B: ngrok
ngrok http 8080
```

Set `PUBLIC_URL` in `.env` to the HTTPS URL from your tunnel.

### 4. Start the webhook server

```bash
source venv/bin/activate
python scripts/webhook-server.py
```

The server auto-registers its `/voice/incoming` URL as the Twilio webhook for your phone number on startup.

Verify it's healthy:

```bash
curl http://localhost:8080/health
```

### 5. Make your first call

**Inbound:** Call your Twilio number. The server answers and connects you to the OpenAI Realtime agent.

**Outbound via API:**

```bash
curl -X POST http://localhost:8080/call \
  -H "Content-Type: application/json" \
  -d '{"to": "+1234567890"}'
```

## Customising the Agent

The agent's identity, instructions, and voice are configured in `webhook-server.py` via the `OPENAI_VOICE` constant and the `build_call_prompt()` function. Edit those to change:

- **Voice:** `shimmer`, `alloy`, `echo`, `fable`, `onyx`, `nova`
- **Model:** `gpt-4o-realtime-preview` (default)
- **System prompt:** Anything returned by `build_call_prompt()`

## Session Continuity

After each call, `summarize_and_remember()` writes a transcript summary to `memory/YYYY-MM-DD.md` and emits a wake event to the OpenClaw gateway (if running). This means the same agent that handles phone calls also has full context from Telegram, email, and other channels.

Skip or remove this feature if you don't use OpenClaw — it degrades gracefully.

## Monitoring

```bash
# Active calls
curl http://localhost:8080/calls

# Server health
curl http://localhost:8080/health
```

Logs are written to stdout. For persistent logs, pipe through `tee` or configure `uvicorn` log file output.

## Production Deployment

For production, run behind a process supervisor:

```bash
# systemd (Linux)
ExecStart=/path/to/venv/bin/python scripts/webhook-server.py

# launchd (macOS) — see scripts/org.niavoice.voice-server.plist
launchctl load scripts/org.niavoice.voice-server.plist
```

Ensure `PUBLIC_URL` points to your stable public domain (Cloudflare tunnel, Fly.io, Railway, etc.).

## Troubleshooting

**Call connects but no audio / robotic voice:**
- Verify your tunnel is working: `curl https://your-tunnel.example.com/health`
- Check audio conversion: `audioop` or `audioop-lts` must be installed
- Sample rate mismatch causes robotic audio — do not bypass the µ-law ↔ PCM16 conversion

**Agent doesn't pick up inbound calls:**
- Set `ALLOW_INBOUND_CALLS=true` in `.env`
- Verify Twilio webhook points to `https://your-public-url/voice/incoming`
- The server logs the webhook URL it registers on startup

**Outbound call placed but agent doesn't speak:**
- OpenAI session setup takes ~300ms; speech before `session.updated` is silently dropped
- The server gates audio forwarding until `session.updated` is confirmed (3s timeout)

**Python 3.13+ import error on audioop:**
- Add `audioop-lts` to your virtualenv: `pip install audioop-lts`

## License

AGPL-3.0 — see LICENSE file. Self-hosting for personal or internal use is unrestricted. Distribution of modified versions requires source disclosure.

Comments 0

Latest | Hot

Please login before commenting.

Loading comments...

openai-voice-skill

Directory Structure

SKILL.md

Report

Notice