myxelium
e49b3ec112
- `LIVE_RELOAD=false npm run dev` keeps the renderer alive across a machine suspend; the reload client otherwise destroys the session under test. - `dev-peer.sh` plus a separate userdata dir runs a second local peer. - `tools/voice-probe.js` samples peer state and RTP counters from a live window, persisting to localStorage so a renderer reload cannot erase it. - e2e helpers for voice pairs, peer-role election, and a TURN relay. - Electron single-instance and dev-client-load decisions move into rules files with colocated specs.
131 lines
5.5 KiB
Markdown
131 lines
5.5 KiB
Markdown
# 06 — Voice & WebRTC bugs
|
||
|
||
> **User theme:** “Voice is broken / one-way / connecting forever / works until reconnect.”
|
||
> **Severity:** P0.
|
||
|
||
---
|
||
|
||
## How voice is supposed to work (user view)
|
||
|
||
1. Click a voice channel (or answer a call).
|
||
2. Mic turns on (unless muted); you appear in the channel roster for others.
|
||
3. You hear everyone in **that** channel; they hear you.
|
||
4. Mute / deafen / camera / screen share behave Discord-like.
|
||
5. Navigate away → floating controls; voice continues.
|
||
6. Joining another voice target auto-leaves the previous one.
|
||
7. Second device shows you’re in voice elsewhere; Join takes over.
|
||
8. After a short network blip, voice returns without restarting the app.
|
||
|
||
Voice is **not** supposed to depend on the data channel for audio itself — but DC carries control/state; broken DC recovery currently **rebuilds the whole peer**, which drops media too (`07`).
|
||
|
||
The users should always be able to hear each other and see each other in joined calls and voice channels. (Not 1 out of 6 cant be heard for 3 users!)
|
||
|
||
---
|
||
|
||
## How voice is supposed to work (system)
|
||
|
||
| Piece | Responsibility |
|
||
|-------|----------------|
|
||
| `voice-session` | Session metadata, floating UI, settings, exclusivity, takeover rules |
|
||
| `voice-connection` | Facade, VAD, per-peer playback gain |
|
||
| `MediaManager` | getUserMedia, RNNoise, gain, same-channel track routing |
|
||
| `PeerConnectionManager` | RTCPeerConnection, negotiation, ICE |
|
||
| Signaling | `offer`/`answer`/`ice_candidate` relay; `voice_state`; `voice_client_takeover` |
|
||
| ICE | STUN defaults; TURN optional via settings — **no bundled TURN** |
|
||
|
||
Initiator election: deterministic compare of local id vs peer `oderId` so only one side offers.
|
||
|
||
Audio routing: attach/detach mic based on matching `voiceState.roomId` + `serverId`. Playback similarly scoped.
|
||
|
||
---
|
||
|
||
## Failure modes
|
||
|
||
### 1 — Cross-signal initiator / politeness uses home id
|
||
|
||
**Symptom:** Peers never connect or glare forever when users have different home servers / foreign actor ids in the room.
|
||
|
||
**Mechanism**
|
||
|
||
- Presence peer ids = **per-server actor** `oderId`.
|
||
- `getLocalOderId` / polite-peer path often uses `getIdentifyCredentials()` → **home** id (`signaling-transport-handler.ts`, `realtime-session.service.ts`, `negotiation.ts`).
|
||
- Election `localOderId < peerId` inconsistent across clients → dual offer, dual wait, or stuck non-initiator.
|
||
|
||
**Fix direction:** elect and politeness using **per-signal-url** identify credentials (`getIdentifyCredentialsForSignalUrl(peerSignalUrl)` / room source URL), not home-only.
|
||
|
||
### 2 — Voice allow-list misses peer map key → one-way / silence
|
||
|
||
**Symptom:** Connected peer, speaking UI maybe wrong, no audio out or in.
|
||
|
||
**Mechanism:** `syncOutgoingVoiceRouting` / playback only recognizes certain aliases (`id` / `oderId` / `peerId`). If `activePeerConnections` key is another alias → track detached.
|
||
|
||
### 3 — Silent give-up after reconnect budget
|
||
|
||
**Symptom:** After ~60s of failures, voice never returns; no error toast.
|
||
|
||
**Constants:** `PEER_RECONNECT_MAX_ATTEMPTS = 12`, interval 5s. Tracker deleted; no UI.
|
||
|
||
### 4 — Transient signaling drops offers during WS reconnect
|
||
|
||
Offers/ICE classified transient may be deferred/dropped while socket reconnecting → half-open peers; depends on fallback offer timers (`USER_JOINED_FALLBACK`, non-initiator give-up 5s).
|
||
|
||
### 5 — DC recovery tears down media
|
||
|
||
Closed control channel → `removePeer` + full reconnect (`peer-recovery.ts`). Docs claim soft `replaceDataChannel` preserving AV — **code does not do that** (`07`, `10`).
|
||
|
||
### 6 — Auth / presence missing
|
||
|
||
If identify/join failed (`04`/`05`), no RTC relay eligibility / no peer discovery → empty voice.
|
||
|
||
### 7 — Same-channel filter false negatives
|
||
|
||
Remote `voice_state` missing/stale channel ids → locally mute peer while UI still lists them in channel.
|
||
|
||
### 8 — No TURN by default
|
||
|
||
Symmetric NAT / strict firewalls fail ICE with STUN-only. User-configurable TURN exists; many installs never set it. Product decision needed: ship defaults vs document limitation.
|
||
|
||
### 9 — Multi-device ownership races
|
||
|
||
Takeover / heartbeat / `voiceActive` routing wrong → offers hit passive device; active device silent.
|
||
|
||
---
|
||
|
||
## Code vs docs
|
||
|
||
| Doc claim | Code |
|
||
|-----------|------|
|
||
| Soft DC renegotiation preserves media | Always full peer recreate on closed DC |
|
||
| Deterministic initiator from logical peer ids | Uses home identify credentials in several paths |
|
||
| TURN supported | Configurable only; not default |
|
||
|
||
---
|
||
|
||
## Key files
|
||
|
||
- `domains/voice-session/`, `domains/voice-connection/`
|
||
- `infrastructure/realtime/media/media.manager.ts`
|
||
- `peer-connection-manager/**`, `negotiation.ts`, `peer-recovery.ts`
|
||
- `signaling-message-handler.ts` (server_users / user_joined offers)
|
||
- `ice-server-settings.service.ts`, `realtime.constants.ts`
|
||
- Feature: `agents-docs/features/voice-webrtc.md`
|
||
|
||
---
|
||
|
||
## Proof of done
|
||
|
||
1. Two users same home signal, same voice channel: bidirectional audio < 5s after both join.
|
||
2. Two users **different home signals**, same foreign-hosted room (both provisioned): bidirectional audio.
|
||
3. Toggle mute/deafen; camera; screen share request path.
|
||
4. Kill Wi‑Fi 15s: recovers or shows actionable error (not silent forever).
|
||
5. Second client takeover: first stops transmitting; second owns mic.
|
||
6. Regression test for initiator election with mismatched home vs actor ids.
|
||
|
||
---
|
||
|
||
## Interview prompts (when implementing)
|
||
|
||
- Soft DC replace vs keep full rebuild but fix media reattach + UX error?
|
||
- Ship public TURN defaults or document “requires open NAT / user TURN”?
|
||
- Instrument-only first week vs behavior fix first?
|