myxelium
e49b3ec112
- `LIVE_RELOAD=false npm run dev` keeps the renderer alive across a machine suspend; the reload client otherwise destroys the session under test. - `dev-peer.sh` plus a separate userdata dir runs a second local peer. - `tools/voice-probe.js` samples peer state and RTP counters from a live window, persisting to localStorage so a renderer reload cannot erase it. - e2e helpers for voice pairs, peer-role election, and a TURN relay. - Electron single-instance and dev-client-load decisions move into rules files with colocated specs.
100 lines
4.7 KiB
Markdown
100 lines
4.7 KiB
Markdown
# 07 — Data channel drops & recovery
|
||
|
||
> **User theme:** “Connection drops / chat and files die / voice dies after a blip.”
|
||
> **Severity:** P0/P1 — shared control plane for chat sync, attachments, emoji, screen control, and currently media (because recovery rebuilds the whole PC).
|
||
|
||
---
|
||
|
||
## How the data channel is supposed to work (user view)
|
||
|
||
- Once you’re in a community with other people, messages, files, emoji, and many live updates “just sync.”
|
||
- Brief network glitches should self-heal.
|
||
- You should not need to restart the app to get chat syncing again.
|
||
- Voice should ideally survive control-plane blips (product docs claim this; code currently does not).
|
||
- Users should sync all but only load a portion into ram for viewing so the app doesn't crash of high ram usage.
|
||
---
|
||
|
||
## How the data channel is supposed to work (system)
|
||
|
||
- Single **ordered** RTCDataChannel per peer pair (label typically chat/control).
|
||
- Carries: chat events, inventory sync, attachments, avatar/emoji chunks, voice/screen control, pings, plugin bus, game activity, etc.
|
||
- Back-pressure: high 4MB / low 1MB watermarks.
|
||
- Ping every 5s for RTT.
|
||
- On failure, recovery should restore control **and** ensure inventory/resync runs when channel reopens.
|
||
|
||
### Documented recovery (README / voice-webrtc.md)
|
||
|
||
1. Non-fatal error on **open** channel → request voice-state snapshot on same channel.
|
||
2. **Closed** channel → initiator renegotiates **new DC on existing PC** (preserve AV); non-initiator waits then full rebuild if missing.
|
||
3. Closing-but-not-closed → short grace (2.5s).
|
||
4. `replaceDataChannel` adopts the new channel.
|
||
|
||
### Actual recovery (`peer-recovery.ts`)
|
||
|
||
1. Closed → `repairUnavailableDataChannel` → `removePeer` + `attemptPeerReconnect` / `schedulePeerReconnect` (**full PC teardown**).
|
||
2. Closing → wait `DATA_CHANNEL_RECOVERY_GRACE_MS` (2.5s) → same full recreate.
|
||
3. `replaceDataChannel` exists on the manager and is wired into handlers, but **recovery path never calls it**; specs assert it is **not** called in several cases.
|
||
4. After 12 reconnect attempts (~60s): abandon **silently**.
|
||
|
||
**This is a documented lie** — treat README paragraph as aspirational until code matches or docs are corrected in the same PR as a deliberate decision.
|
||
|
||
---
|
||
|
||
## Failure modes
|
||
|
||
### 1 — Full rebuild drops audio/video on every DC close
|
||
|
||
User hears a “drop” even when ICE media might have survived. Cascades into voice bug reports.
|
||
|
||
### 2 — Silent abandon
|
||
|
||
No toast, no “Reconnect” CTA, no automatic retry on later `user_joined`. Mesh looks permanently broken until navigation/restart.
|
||
|
||
### 3 — Live chat limp vs history dead
|
||
|
||
While DC down, `chat_message` WS fallback may still deliver **live** text. Inventory sync is **DC-only** → late joiners / catch-up fail until P2P returns (`08`).
|
||
|
||
### 4 — Large payloads kill shared channel
|
||
|
||
Custom emoji / attachment floods can stress or close the shared ordered channel (lessons). One feature outage becomes total control-plane outage.
|
||
|
||
### 5 — Attachment announce vs message ordering
|
||
|
||
`file-announce` on DC can beat `chat-message` on WS → auto-download gives up unless re-queued on message bind (lesson; verify still present when touching attachments).
|
||
|
||
### 6 — Replacement channel race
|
||
|
||
If soft-replace is reintroduced, must close old channel to release SCTP (voice-webrtc changelog). Current full rebuild avoids that class but at higher cost.
|
||
|
||
---
|
||
|
||
## Key files
|
||
|
||
- `peer-connection-manager/messaging/data-channel.ts`
|
||
- `peer-connection-manager/recovery/peer-recovery.ts` (+ specs)
|
||
- `peer-connection.manager.ts` (`replaceDataChannel`)
|
||
- `realtime.constants.ts`
|
||
- Consumers: chat sync effects, attachment transfer, custom emoji chunking, screen-share request
|
||
|
||
---
|
||
|
||
## Fix directions (interview)
|
||
|
||
| Option | Idea | Tradeoff |
|
||
|--------|------|----------|
|
||
| **A (align code to docs)** | Implement true soft DC replace on connected PC; full rebuild only if PC not connected | Harder; matches user expectation for voice survival |
|
||
| **B (align docs to code)** | Keep full rebuild; fix reattach + force inventory on reopen; **surface give-up UX** | Faster; still interrupts voice |
|
||
| **C** | Separate unreliable channel for bulk (files/emoji) vs reliable small control | Larger design |
|
||
|
||
Recommend starting with **instrumentation + B’s UX/resync**, then **A** if voice drop rate stays high.
|
||
|
||
---
|
||
|
||
## Proof of done
|
||
|
||
1. Force-close DC in debug: control messages resume; inventory runs; user sees progress or success.
|
||
2. If soft-replace chosen: audio continues through DC replace (automated or manual with metrics).
|
||
3. After max attempts: visible error + manual retry works.
|
||
4. Attachment + emoji transfer during recovery does not deadlock the mesh.
|
||
5. Update `realtime/README.md` + `voice-webrtc.md` in the same change set so they match behavior.
|