Files
Toju/emergency-fix/07-data-channel-drops.md
T
myxelium e49b3ec112 chore: dev-stack switches, shared e2e harness, and desktop shell rules
- `LIVE_RELOAD=false npm run dev` keeps the renderer alive across a machine
  suspend; the reload client otherwise destroys the session under test.
- `dev-peer.sh` plus a separate userdata dir runs a second local peer.
- `tools/voice-probe.js` samples peer state and RTP counters from a live
  window, persisting to localStorage so a renderer reload cannot erase it.
- e2e helpers for voice pairs, peer-role election, and a TURN relay.
- Electron single-instance and dev-client-load decisions move into rules
  files with colocated specs.
2026-08-14 03:19:29 +02:00

4.7 KiB
Raw Blame History

07 — Data channel drops & recovery

User theme: “Connection drops / chat and files die / voice dies after a blip.”
Severity: P0/P1 — shared control plane for chat sync, attachments, emoji, screen control, and currently media (because recovery rebuilds the whole PC).


How the data channel is supposed to work (user view)

  • Once youre in a community with other people, messages, files, emoji, and many live updates “just sync.”
  • Brief network glitches should self-heal.
  • You should not need to restart the app to get chat syncing again.
  • Voice should ideally survive control-plane blips (product docs claim this; code currently does not).
  • Users should sync all but only load a portion into ram for viewing so the app doesn't crash of high ram usage.

How the data channel is supposed to work (system)

  • Single ordered RTCDataChannel per peer pair (label typically chat/control).
  • Carries: chat events, inventory sync, attachments, avatar/emoji chunks, voice/screen control, pings, plugin bus, game activity, etc.
  • Back-pressure: high 4MB / low 1MB watermarks.
  • Ping every 5s for RTT.
  • On failure, recovery should restore control and ensure inventory/resync runs when channel reopens.

Documented recovery (README / voice-webrtc.md)

  1. Non-fatal error on open channel → request voice-state snapshot on same channel.
  2. Closed channel → initiator renegotiates new DC on existing PC (preserve AV); non-initiator waits then full rebuild if missing.
  3. Closing-but-not-closed → short grace (2.5s).
  4. replaceDataChannel adopts the new channel.

Actual recovery (peer-recovery.ts)

  1. Closed → repairUnavailableDataChannelremovePeer + attemptPeerReconnect / schedulePeerReconnect (full PC teardown).
  2. Closing → wait DATA_CHANNEL_RECOVERY_GRACE_MS (2.5s) → same full recreate.
  3. replaceDataChannel exists on the manager and is wired into handlers, but recovery path never calls it; specs assert it is not called in several cases.
  4. After 12 reconnect attempts (~60s): abandon silently.

This is a documented lie — treat README paragraph as aspirational until code matches or docs are corrected in the same PR as a deliberate decision.


Failure modes

1 — Full rebuild drops audio/video on every DC close

User hears a “drop” even when ICE media might have survived. Cascades into voice bug reports.

2 — Silent abandon

No toast, no “Reconnect” CTA, no automatic retry on later user_joined. Mesh looks permanently broken until navigation/restart.

3 — Live chat limp vs history dead

While DC down, chat_message WS fallback may still deliver live text. Inventory sync is DC-only → late joiners / catch-up fail until P2P returns (08).

4 — Large payloads kill shared channel

Custom emoji / attachment floods can stress or close the shared ordered channel (lessons). One feature outage becomes total control-plane outage.

5 — Attachment announce vs message ordering

file-announce on DC can beat chat-message on WS → auto-download gives up unless re-queued on message bind (lesson; verify still present when touching attachments).

6 — Replacement channel race

If soft-replace is reintroduced, must close old channel to release SCTP (voice-webrtc changelog). Current full rebuild avoids that class but at higher cost.


Key files

  • peer-connection-manager/messaging/data-channel.ts
  • peer-connection-manager/recovery/peer-recovery.ts (+ specs)
  • peer-connection.manager.ts (replaceDataChannel)
  • realtime.constants.ts
  • Consumers: chat sync effects, attachment transfer, custom emoji chunking, screen-share request

Fix directions (interview)

Option Idea Tradeoff
A (align code to docs) Implement true soft DC replace on connected PC; full rebuild only if PC not connected Harder; matches user expectation for voice survival
B (align docs to code) Keep full rebuild; fix reattach + force inventory on reopen; surface give-up UX Faster; still interrupts voice
C Separate unreliable channel for bulk (files/emoji) vs reliable small control Larger design

Recommend starting with instrumentation + Bs UX/resync, then A if voice drop rate stays high.


Proof of done

  1. Force-close DC in debug: control messages resume; inventory runs; user sees progress or success.
  2. If soft-replace chosen: audio continues through DC replace (automated or manual with metrics).
  3. After max attempts: visible error + manual retry works.
  4. Attachment + emoji transfer during recovery does not deadlock the mesh.
  5. Update realtime/README.md + voice-webrtc.md in the same change set so they match behavior.