69 lines
3.6 KiB
Markdown
69 lines
3.6 KiB
Markdown
# Voice & WebRTC
|
|
|
|
> **Status:** Active
|
|
> **Last updated:** 2026-07-05
|
|
|
|
## Overview
|
|
|
|
Voice channels, camera, and screen-share use direct **WebRTC** peer connections between clients. The signaling server relays SDP offers/answers/ICE and **voice presence** (`voice_state`); media never flows through the server.
|
|
|
|
## Responsibilities
|
|
|
|
| Layer | Owns |
|
|
|-------|------|
|
|
| `infrastructure/realtime/` | WebRTC sessions, negotiation, data channels, RNNoise worklet |
|
|
| `voice-connection` / `voice-session` domains | Facades, workspace UI, settings, multi-device ownership |
|
|
| `screen-share` domain | Source picker, quality presets (Electron) |
|
|
| Signaling server | RTC relay + `voice_state` / `voice_client_takeover` broadcast |
|
|
|
|
## WebSocket signaling types
|
|
|
|
| type | Purpose |
|
|
|------|---------|
|
|
| `offer`, `answer`, `ice_candidate` | WebRTC negotiation relay to `targetUserId` |
|
|
| `voice_state` | Voice roster (mute/deafen/speaking, channel id); sets `voiceActive` on one connection per user |
|
|
| `voice_client_takeover` | Notify sibling tabs to yield voice ownership |
|
|
|
|
Relay rules: RTC messages require shared server membership (except DM-specific types). See [signaling.md](signaling.md).
|
|
|
|
## Multi-device voice
|
|
|
|
- Only one connection per `oderId` may be `voiceActive`; RTC offers route to that connection (fallback: any open connection).
|
|
- Other tabs show passive UI and may send `voice_client_takeover`.
|
|
- `clientInstanceId` in `identify` and voice payloads distinguishes tabs.
|
|
|
|
## Media pipeline (client)
|
|
|
|
- **Voice:** `getUserMedia` → optional RNNoise AudioWorklet → gain → same-room peer routing only.
|
|
- **Camera:** separate video track; same-room filter.
|
|
- **Screen share:** on-demand via data-channel `SCREEN_SHARE_REQUEST`; platform-specific capture (browser `getDisplayMedia`, Electron picker, Linux PulseAudio routing).
|
|
|
|
## P2P data channel
|
|
|
|
Carries voice/screen control messages, chat, attachments, and state sync — not server-relayed. Data-channel failure triggers peer renegotiation or full rebuild (see realtime README). When a failed control channel is replaced (`replaceDataChannel`), the old channel is closed first so its SCTP resources are released.
|
|
|
|
## Media memory invariants (2026-07-14, RAM investigation)
|
|
|
|
- `removePeer` / `closeAllPeers` clear **all four** remote stream maps, including `remotePeerCameraStreams` (previously leaked per departed peer).
|
|
- Video tiles (`voice-workspace-stream-tile`) pause and null `srcObject` in `ngOnDestroy` (`voice-workspace-stream-video.rules.ts`) so Chromium releases decoder/frame buffers immediately.
|
|
- `debug-network-metrics` drops a peer's entry when the peer is fully removed and caps the store at `MAX_TRACKED_DEBUG_NETWORK_PEERS` = 200 (oldest evicted).
|
|
- Electron registers `setDisplayMediaRequestHandler` once per app run (guarded in `create-window.ts`), not on every window recreation.
|
|
|
|
## Mobile / Capacitor
|
|
|
|
Background voice uses Android foreground service + iOS audio/CallKit bridges — [mobile-capacitor.md](mobile-capacitor.md). Screen share is limited on mobile WebViews.
|
|
|
|
## Related
|
|
|
|
- [signaling.md](signaling.md) — envelope catalog
|
|
- [direct-messaging.md](direct-messaging.md) — private calls share `PeerDeliveryService`
|
|
- [`toju-app/src/app/infrastructure/realtime/README.md`](../../toju-app/src/app/infrastructure/realtime/README.md) — negotiation, recovery, RNNoise
|
|
- Domain READMEs: `voice-connection`, `voice-session`, `screen-share`
|
|
|
|
## Changelog
|
|
|
|
| Date | Change |
|
|
|------|--------|
|
|
| 2026-07-14 | Media memory invariants: camera-stream map cleanup, tile `srcObject` release, debug-metrics cap, single display-media handler registration, replaced data channels closed |
|
|
| 2026-07-05 | Initial cross-context voice/WebRTC contract |
|