diff --git a/docs/design/afv-mesh-pr10.md b/docs/design/afv-mesh-pr10.md index a5508f7..2b786c5 100644 --- a/docs/design/afv-mesh-pr10.md +++ b/docs/design/afv-mesh-pr10.md @@ -5,9 +5,10 @@ | **Title** | AFV multi-node mesh — directory sync + AT frame relay | | **Author** | openfsd design (implementer-ready) | | **Date** | 2026-07-28 | -| **Status** | **Memory path Implemented** (PR-10 on `dev`: framing, Interest, AudioRelay, MemoryMesh e2e Cases A–F, fail-closed `ENABLED` without TCP). **TCP mesh = PR-10b** (not in production binary). | +| **Status** | **Memory path Implemented** (PR-10: framing, Interest, AudioRelay, MemoryMesh e2e Cases A–F). **Production mesh = PR-10b hybrid TCP control + UDP voice** (see `docs/design/afv-mesh-pr10b.md`; TCP-only voice **superseded**). | | **Parent** | `docs/design/afv-server.md` (Clustering, KD-8/9/16/17, PR-10) | | **Contract** | `docs/design/afv-mesh-implementer-prompt.md` | +| **Production transport** | `docs/design/afv-mesh-pr10b.md` | --- @@ -16,9 +17,11 @@ | Item | State | |------|--------| | Framing, Interest, remote directory, AudioRelay, MemoryMesh e2e | **Landed** (`internal/afv/mesh_*.go`, Cases A–F) | -| Fail-closed `AFV_CLUSTER_ENABLED` without TCP | **Landed** | -| Production `TCPMesh` | **PR-10b** — not in binary; enable fails with clear error | -| Operator wiki for multi-node AFV | Single-node ops documented; multi-host voice deferred until TCP | +| Fail-closed incomplete `AFV_CLUSTER_*` | **Landed** | +| Production mesh transport | **PR-10b hybrid TCP control + UDP voice** (TCP-only voice **superseded**) | +| Operator wiki multi-node AFV | Deliverable of PR-10b stack (Configuration + Deployment) | + +> **Callout (PR-10b):** Production **AudioRelay (type 20) is UDP only**. Dual TCP voice+control queues apply to **MemoryMesh** tests only for production semantics. See `afv-mesh-pr10b.md`. ## Overview diff --git a/docs/design/afv-mesh-pr10b.md b/docs/design/afv-mesh-pr10b.md new file mode 100644 index 0000000..84540cd --- /dev/null +++ b/docs/design/afv-mesh-pr10b.md @@ -0,0 +1,1206 @@ +# AFV Multi-Node Production Mesh — Hybrid TCP Control + UDP Voice (PR-10b) + +| Field | Value | +|-------|--------| +| **Title** | AFV multi-node production mesh — hybrid TCP control + UDP voice | +| **Author** | openfsd design (implementer-ready) | +| **Date** | 2026-07-28 | +| **Status** | **Implemented** (rev 3 — H-20 multi-host fix + H-19 rate; hybrid stack landed) | +| **Parent** | `docs/design/afv-mesh-pr10.md` (PR-10 Memory path landed; TCP-only voice plan **superseded**) | +| **Also amends** | `docs/design/afv-server.md` §Clustering (production mesh transport) | +| **Contract** | `docs/design/afv-mesh-implementer-prompt.md` (framing + Interest invariants remain; transport revised here) | +| **In-repo target** | `docs/design/afv-mesh-pr10b.md` (land during implementation) | + +--- + +## Supersedes / amends PR-10b TCP-only voice + +This document **supersedes** the production-mesh transport plan in `docs/design/afv-mesh-pr10.md` that placed **all** mesh traffic — including high-rate `AudioRelay` (type 20, opaque Opus + TX geometry) — on a single **TCP** connection with dual outbound queues (voice depth 256 drop-oldest; control depth 64). + +| Topic | PR-10 / original PR-10b (superseded for production) | This document (normative for production) | +|-------|------------------------------------------------------|------------------------------------------| +| Control plane | TCP: Hello, Heartbeat, Trx*, SessionLeave, Interest | **Unchanged** — TCP only | +| Voice plane | TCP: `AudioRelay` on same conn, voice queue 256 | **UDP only** — `AudioRelay` never on TCP | +| HOL / retransmit | Voice subject to TCP HOL and retransmission delay | Voice is lossy, best-effort, no HOL with control | +| `AFV_CLUSTER_LISTEN` | TCP listen for entire mesh | TCP **control** listen only | +| Voice endpoints | N/A (piggybacked) | `AFV_CLUSTER_VOICE_LISTEN` + peer voice addr derivation | +| MemoryMesh | In-process dual queues (voice+ctrl) | **Unchanged** — tests-only; may keep in-process queues | +| Framing codecs | `EncodeAudioRelay` / `DecodeAudioRelay` | **Same payload codec**; UDP datagram carries type-20 frame | +| Fail-closed enable | `ENABLED=true` without TCP → error | `ENABLED=true` without **hybrid** production mesh → error | +| Import graph | `internal/afv` ↛ `internal/cluster` | **Unchanged** | + +**Parent doc patches (normative — land with PR-10b.1, do not wait for 10b.5):** + +When landing `docs/design/afv-mesh-pr10b.md` (even as Draft), **immediately** patch: + +1. `docs/design/afv-mesh-pr10.md` implementation status table: + - Production mesh transport → **PR-10b hybrid TCP control + UDP voice** (TCP-only voice **superseded**) + - Operator wiki multi-node AFV → deliverable of PR-10b stack +2. `docs/design/afv-mesh-pr10.md` TCP mesh / dual-queue sections: add a callout that production AudioRelay is **UDP only** per `afv-mesh-pr10b.md`; dual TCP voice+control queues apply to **MemoryMesh** tests only for production semantics. +3. `docs/design/afv-server.md` §Clustering normative transport row: + - **Control** = TCP between static peers + - **AudioRelay (type 20)** = UDP mesh voice (not TCP) + - Point to `afv-mesh-pr10b.md` + +**What remains valid from PR-10 (do not re-litigate):** + +- Sticky home clients; AEAD keys never leave home; Opus never in rqlite +- Interest filter; max 4 remote peers; static `AFV_CLUSTER_PEERS` +- Hello CT-PSK; post-Hello Snapshot+Interest (M-16); peer death 15s (M-9) +- `isATC` / `isXC` on AudioRelay; `routeSyntheticTX`; KD-16 deadlock rule +- MemoryMesh e2e Cases A–F as the in-process correctness bar +- `Mesh` interface surface already landed in `internal/afv/mesh.go` + +--- + +## Overview + +openfsd AFV is a single-binary VATSIM-shaped voice service (`internal/afv`: REST + UDP CryptoDTO). PR-10 landed the **mesh data plane logic** in-process: length-prefixed framing, Interest, remote directory, AudioRelay re-encrypt path, and MemoryMesh e2e (Cases A–F). Production enablement is fail-closed: `AFV_CLUSTER_ENABLED=true` without a production mesh returns `errClusterTCPNotBuilt` from `NewDefault` (`internal/afv/bootstrap.go`). + +This design specifies the **production multi-host AFV mesh** as a **hybrid transport**: + +1. **TCP control plane** — Hello (first-frame + constant-time PSK), Heartbeat, TrxSnapshot, TrxDelta, SessionLeave, Interest. Reliable-ish, low rate, dual-queue control path depth 64 drop-oldest. +2. **UDP voice plane** — AudioRelay **only**. Lossy, drop-oldest / best-effort per peer, no head-of-line blocking with control. Payload codec is the existing `EncodeAudioRelay` / `DecodeAudioRelay` in `mesh_frame.go`. + +AFV mesh remains a **separate fabric** under `internal/afv/`. It must **never** import `internal/cluster` (copy patterns only from `internal/cluster/tcp_mesh.go`). Clients stay sticky to a home node; mesh is **inter-node only**. + +--- + +## Background & Motivation + +### Current state (on `dev`) + +| Layer | Path | State | +|-------|------|--------| +| Framing + codecs | `internal/afv/mesh_frame.go` | Landed; Hello, HB, Trx*, Leave, AudioRelay, Interest | +| Queues | `internal/afv/mesh_queue.go` | Landed; voice 256 + control 64 drop-oldest | +| MemoryMesh | `internal/afv/mesh_memory.go` | Landed; tests-only via `SetMesh` | +| Interest / remote dir | `mesh_interest.go`, `mesh_remotedir.go` | Landed | +| Server hooks | `mesh.go`, `server.go`, `udp.go`, `api.go` | Landed; `EnqueueAudioRelay` after local AT | +| Bootstrap | `bootstrap.go` | `ClusterEnabled` → always `errClusterTCPNotBuilt` | +| Config stubs | `config.go` | `AFV_CLUSTER_{ENABLED,NODE_ID,LISTEN,PEERS,PSK}` | +| Production TCP/UDP mesh | — | **Missing** (`mesh_tcp.go` / voice UDP not in tree) | + +### Pain points with TCP-only voice (original PR-10b sketch) + +- **Head-of-line blocking:** a single lost TCP segment delays subsequent AudioRelay frames even when control traffic is healthy. +- **Retransmission delay:** voice that is already late is worse than dropped; TCP insists on delivery order. +- **Queue coupling risk:** even with dual queues and control priority, one writer on one stream cannot fully isolate real-time audio from control burst / congestion response. +- **Product requirement:** do **not** piggyback UDP-nature voice over TCP. + +### Why hybrid preserves PR-10 invariants + +The hard invariants are **data-path** properties (keys, Opus storage, sticky home, Interest, import graph), not transport properties. Hybrid transport only changes **how bytes leave the process** for type 20 vs control types. MemoryMesh continues to exercise the same `Mesh` interface for in-process e2e without sockets. + +--- + +## Goals & Non-Goals + +### Goals + +1. **Production hybrid mesh** constructible from valid `AFV_CLUSTER_*=true` config in `NewDefault`; attach via `SetMesh` (or internal equivalent) before `Run`. +2. **TCP control** with Hello first-frame rule, CT-PSK, Heartbeat, directory, Interest, peer death purge. +3. **UDP voice** for AudioRelay only; same `EncodeAudioRelay` / `DecodeAudioRelay` payload; Interest-filtered fan-out; drop-oldest depth 256 per peer. +4. **Two real processes on localhost** can complete A2A via mesh (control TCP + voice UDP). +5. Fail closed: wrong PSK, non-Hello first TCP frame, peer death purge, empty Interest → no flood. +6. Config surface for voice endpoints with a **single primary ops scheme** and sticky-LB notes. +7. Unit coverage floors in the spirit of PR-10 Testing Plan for **new** modules; race/import/hygiene/build green. +8. Operator wiki: Configuration + Deployment multi-node AFV sections. +9. Update status tables in parent designs (or superseding addendum pointer). + +### Non-goals + +| Non-goal | Notes | +|----------|--------| +| Merge AFV mesh into FSD `internal/cluster` | KD-8; import allowlist | +| Reimplement P0 single-node AFV | Already on `dev` | +| Implement PR-9 cross-coupling XC | Keep `isXC` on AudioRelay; never XC-again | +| Global callsign claim | Dual-login across nodes remains allowed | +| Dynamic peer discovery / membership | Static peers only | +| Mesh AEAD / TLS on control or voice | Private-network + PSK trust root (same as FSD mesh) | +| Client CryptoDTO or REST changes | Sticky home unchanged | +| >4 remote peers | Cap remains 4 | +| Opus or session keys in rqlite | Unchanged | +| Browser E2E / Playwright | House standard forbids by default | + +--- + +## Key Decisions + +| ID | Decision | Rationale | +|----|----------|-----------| +| **H-1** | Production transport is **hybrid**: TCP control + UDP voice | Product mandate: no TCP HOL/retransmit for real-time audio | +| **H-2** | **AudioRelay (type 20) is forbidden on TCP**; if received, log + close that peer control conn (fail closed) | Prevents accidental dual-path and keeps control stream low-rate | +| **H-3** | UDP datagram wire = **full mesh frame** bytes (`EncodeMeshFrame` → type 20 + AudioRelay payload); **strict** one frame per datagram (`decodedFrameBytes == n` from `ReadFrom`) | Share decode path; prevent multi-frame smuggling | +| **H-4** | Voice endpoint config: **`AFV_CLUSTER_VOICE_LISTEN`** local bind + peer voice = **same host as peer TCP, port = TCP port + 1** by default; optional peer suffix `id=host:tcpPort/voicePort` | Clear ops model; no post-Hello discovery complexity; sticky LB notes unchanged | +| **H-5** | UDP accept only from **allowlist of resolved peer voice addrs** while peer is **control-authed**; canonicalize keys per H-16 | Spoof mitigation without per-packet crypto; private net assumption | +| **H-6** | Type name: production type is **`HybridMesh`** (implements `Mesh`); file split `mesh_tcp.go` (control) + `mesh_udp_voice.go` (voice) under package `afv` | Matches PR-10 file plan; no new importable subpackage | +| **H-7** | Control outbound: per-peer **ctrl queue depth 64 drop-oldest** with **per-peer TCP writer**; voice outbound: per-peer **UDP send queue depth 256 drop-oldest** with **per-peer voice drainer** (not a shared global buffer) | Preserves M-6; avoids cross-peer buffer races | +| **H-8** | Liveness is **TCP-only**. Peer death triggers: (a) 15s since `lastHB`, (b) TCP read error/close, (c) TCP write error. `lastHB` updated on **every successfully decoded post-Hello control frame** (not only Heartbeat). UDP errors never alone mark death. Death → purge remote dir + Interest; stop voice; re-init empty queues | Matches FSD `tcp_mesh.go` spirit; single liveness signal | +| **H-9** | Dial TCP only if `NodeID > self`. On second live conn for same peerID: **close previous**, reset queues, re-run Hello auth sequence on the new conn | Avoid dual control conns; FSD replace pattern | +| **H-10** | `NewDefault` with valid cluster config constructs `HybridMesh` and sets it on Server (PR-10b.4). Until then keep fail-closed sentinel. After 10b.4: delete `errClusterTCPNotBuilt` or redefine message to hybrid-start failures only | Acceptance bar; no grepping confusion | +| **H-11** | MemoryMesh stays tests-only; production never auto-wires MemoryMesh | M-11 unchanged | +| **H-12** | Named constant **`maxMeshVoiceDatagram = 16 KiB`** is the **only** accepted UDP voice size; reject larger **before** `DecodeMeshFrame`. Do **not** use `MaxMeshPayload` (1 MiB) for UDP reads. Encode path also rejects if encoded frame > this cap | DoS footgun; concrete test lock | +| **H-13** | No registry/remote-dir lock across mesh enqueue, TCP write, or UDP `WriteTo`. Peer-map locks: short hold for enqueue/map ops only; never across I/O | M-14 / KD-16 | +| **H-14** | Post-Hello sequence still Snapshot then Interest on **TCP only**. HybridMesh applies inbound Interest **on the TCP reader** (replace set for origin). Interest empty → no AudioRelay fan-out | M-16 + no flood; production is async (not MemoryMesh M-17 sync) | +| **H-15** | Peer voice address resolution is **static at Start** from config (not rewritten from TCP RemoteAddr). Log TCP RemoteAddr host ≠ configured peer TCP host as warn only | NAT/LB: operators configure true reachability | +| **H-16** | UDP allowlist: resolve each peer `VoiceAddr` at Start; key = `net.JoinHostPort(ip.To16().String(), port)` after normalizing IP; **IPv4-mapped IPv6** (`::ffff:x.x.x.x`) is **accepted as equal** to the IPv4 form (map both keys to same peer). Multi-A/AAAA: allowlist **all** resolved addrs for that peer; **send** to first successful resolve order (prefer IPv4 if both). One unconnected `PacketConn` so source port == `VOICE_LISTEN` | Consistent anti-spoof + dual-stack | +| **H-17** | Peer death / `lastHB` / dual-conn: see H-8 / H-9 (normative algorithms below) | Implementer lock | +| **H-18** | `maxMeshVoiceDatagram` formula + `maxMeshVoiceTxRadios = 64` (see Constants) | Bound geometry DoS | +| **H-19** | Inbound UDP: after allowlist, soft **per-peer inbound rate limit** (default **5000 datagrams/s**; excess dropped + counter). Cap is **anti-runaway**, not a capacity plan — sized for multi-talker fan-in (see Constants). Unknown sources never call `OnAudioRelay` | Compromised-peer / bug flood without clipping normal busy mesh | +| **H-20** | Config fail-closed: duplicate peer `VoiceAddr` (string form); `ClusterVoiceListen` ≠ `UDPListen` (normalized); local voice **full `host:port` string** ≠ any peer `VoiceAddr` (no port-only wildcard rule); ports ≠ 0 | Avoid allowlist ambiguity + client/mesh socket footgun **without** rejecting multi-host same port numbers | +| **H-21** | Parent transport row patched in **PR-10b.1** (with design file), not deferred to docs-only 10b.5 | Prevent implementers reading parents as TCP-voice | + +--- + +## Proposed Design + +### High-level architecture + +```mermaid +flowchart TB + subgraph node1 [AFV Node n1] + C1[Client A REST+UDP CryptoDTO] + R1[Registry + router] + H1[HybridMesh] + TC1[TCP control listen] + UV1[UDP mesh voice listen] + C1 --> R1 + R1 -->|local AR| C1 + R1 -->|EnqueueAudioRelay| H1 + H1 --> TC1 + H1 --> UV1 + end + subgraph node2 [AFV Node n2] + C2[Client B REST+UDP CryptoDTO] + R2[Registry + router] + H2[HybridMesh] + TC2[TCP control] + UV2[UDP mesh voice] + C2 --> R2 + R2 -->|local AR| C2 + H2 --> R2 + H2 --> TC2 + H2 --> UV2 + end + TC1 <-->|Hello HB Trx Interest| TC2 + UV1 -.->|AudioRelay type 20 only| UV2 +``` + +```text +Client A ──REST+UDP──► AFV n1 (home A) + │ local AT→AR for local RX + │ TCP: TrxDelta / Interest / HB + │ UDP mesh: AudioRelay (opaque Opus + TX geo) + ▼ + AFV n2 (home B) + │ routeSyntheticTX + encrypt AR with B keys +Client B ◄──UDP AR───────┘ +``` + +### Sequence: cross-node A2A (hybrid) + +```mermaid +sequenceDiagram + participant A as Client A + participant N1 as AFV n1 + participant TCP as TCP control + participant UDP as UDP voice + participant N2 as AFV n2 + participant B as Client B + + Note over N1,N2: Mutual Hello + Snapshot + Interest on TCP + A->>N1: UDP AT (home AEAD) + N1->>N1: Decrypt; routeAT local + N1->>UDP: Enqueue AudioRelay (Interest filter) + UDP->>N2: datagram type=20 + N2->>N2: routeSyntheticTX; re-encrypt AR + N2->>B: UDP AR (B home keys) + Note over TCP: Directory/Interest/HB independent of voice path +``` + +### Package / file layout + +All under package `afv` (no new importable subpackage): + +```text +internal/afv/ + mesh.go # Mesh interface, Server hooks (existing) + mesh_frame.go # Codecs (existing; minor: maybe MeshType assert helpers) + mesh_queue.go # dropOldestQueue (existing; reuse for ctrl + voice) + mesh_memory.go # tests-only (existing) + mesh_tcp.go # NEW: HybridMesh TCP control listen/dial/Hello/HB/ctrl queues + mesh_udp_voice.go # NEW: HybridMesh UDP voice listen/send/allowlist + mesh_hybrid.go # NEW (optional): HybridMesh constructor + Start/Stop orchestration + # OR fold into mesh_tcp.go if single type owns both + mesh_tcp_test.go + mesh_udp_voice_test.go + mesh_hybrid_e2e_test.go # two-process or dual-bind localhost A2A + + config.go # add ClusterVoiceListen; extend peer parse + bootstrap.go # construct HybridMesh when enabled + # server/registry/udp/api — minimal or no change if Mesh interface holds +``` + +**Do not** add mesh types to `pkg/afvprotocol`. **Do not** import `internal/cluster`. + +### Conceptual mirror of FSD cluster (analogy only) + +| FSD (`internal/cluster`) | AFV hybrid mesh | +|--------------------------|-----------------| +| `tcp_mesh.go` single TCP stream | TCP for control only; UDP for voice | +| `peerConn.outCh` depth 256 | `ctrlCh` 64 + `voiceCh` 256 **separate transports** | +| Dial if peer ID > self | Same for TCP control | +| Hello first frame + PSK | Same on TCP; CT-compare already in `VerifyHelloPSK` | +| Interest AABB | AFV `(freqHz, CellKey)` (already landed) | +| No Opus | AudioRelay on UDP | + +--- + +## Configuration surface + +### Env vars + +| Env | Role | Required when `AFV_CLUSTER_ENABLED=true` | +|-----|------|------------------------------------------| +| `AFV_CLUSTER_ENABLED` | Master switch (default false) | — | +| `AFV_CLUSTER_NODE_ID` | Stable node id (string) | yes | +| `AFV_CLUSTER_LISTEN` | **TCP control** listen `host:port` | yes | +| `AFV_CLUSTER_VOICE_LISTEN` | **UDP mesh voice** listen `host:port` | yes (new) | +| `AFV_CLUSTER_PEERS` | Remote peers (see grammar) | yes (≥1, ≤4) | +| `AFV_CLUSTER_PSK` | Shared Hello secret | yes | + +**Unchanged client-facing env:** `AFV_UDP_LISTEN` / `AFV_UDP_ADVERTISE_IPV4` remain **client** CryptoDTO endpoints (sticky home). Mesh voice UDP is a **separate** socket from client UDP. + +### Peer string grammar (H-4) + +```text +AFV_CLUSTER_PEERS = + peer ( "," peer )* + +peer = + nodeID "=" hostPort [ "/" voicePort ] + +hostPort = # must be valid for net.SplitHostPort after /voice strip + host ":" tcpPort +| "[" ipv6 "]" ":" tcpPort +``` + +Examples: + +```bash +# Default +1 voice ports +AFV_CLUSTER_PEERS=n2=10.0.0.2:17000,n3=10.0.0.3:17000 +# → n2 control 10.0.0.2:17000, voice 10.0.0.2:17001 + +# Explicit voice port (non-contiguous or shared-host multi-process) +AFV_CLUSTER_PEERS=n2=127.0.0.1:17000/17100 + +# IPv6 (brackets required) +AFV_CLUSTER_PEERS=n2=[2001:db8::1]:17000 +AFV_CLUSTER_PEERS=n2=[2001:db8::1]:17000/17001 +``` + +### Normative peer parse algorithm (Issue 1) + +Replace today’s `id=addr` + blind `SplitHostPort(addr)`. **Order is mandatory** so `/voice` is never left inside the TCP port field (`SplitHostPort("10.0.0.2:17000/17001")` would otherwise yield port `"17000/17001"`). + +```text +parseClusterPeers(s) → []ClusterPeer | error: + +1. Split s on ",". For each non-empty trimmed part p: +2. Find first '='. If missing or at ends → error "invalid entry". + id = trim(p[:eq]); rest = trim(p[eq+1:]) + if id == "" or rest == "" → error +3. Split rest into hostPort and optional voicePort: + a. If rest contains '/': + - Find last '/' (only one slash form is allowed; extra '/' → error) + - hostPort = trim(before last '/') + - voicePortStr = trim(after last '/') + - if hostPort == "" or voicePortStr == "" → error "empty voice" + - voicePort = parseUint16Decimal(voicePortStr); on fail → error "non-numeric voice" + b. Else: + hostPort = rest + voicePort = unset (derive later) +4. Validate hostPort with net.SplitHostPort(hostPort) → (host, tcpPortStr): + - On error → "want host:port" (covers missing brackets for IPv6) + - host must be non-empty + - tcpPort = parseUint16Decimal(tcpPortStr); reject non-numeric + - if tcpPort == 0 → error "port 0" +5. If voicePort unset: + - if tcpPort == 65535 → error "voice port overflow" (tcp+1) + - voicePort = tcpPort + 1 +6. if voicePort == 0 → error "port 0" +7. VoiceAddr = net.JoinHostPort(host, strconv.Itoa(int(voicePort))) + Addr = hostPort // original SplitHostPort-valid form (preserve brackets) +8. Append ClusterPeer{ID: id, Addr: Addr, VoiceAddr: VoiceAddr} +9. After all peers: enforce unique IDs (existing), unique VoiceAddr strings + (string equality on JoinHostPort form — H-20; Start-time resolve may add more checks) +``` + +**IPv6:** operators **must** use bracket form. Unbracketed IPv6 is rejected by `SplitHostPort` — treat as invalid entry (do not invent a second parser). + +**Accept / reject fixtures (required unit tests):** + +| Input peer entry | Result | +|------------------|--------| +| `n2=10.0.0.2:17000` | OK; Addr=`10.0.0.2:17000`, VoiceAddr=`10.0.0.2:17001` | +| `n2=10.0.0.2:17000/17100` | OK; VoiceAddr=`10.0.0.2:17100` | +| `n2=[::1]:17000` | OK; VoiceAddr=`[::1]:17001` | +| `n2=[::1]:17000/17050` | OK; VoiceAddr=`[::1]:17050` | +| `n2=10.0.0.2:65535` | **Reject** overflow | +| `n2=10.0.0.2:17000/` | **Reject** empty voice | +| `n2=10.0.0.2:17000/abc` | **Reject** non-numeric voice | +| `n2=10.0.0.2:17000/0` | **Reject** port 0 | +| `n2=10.0.0.2:0` | **Reject** port 0 | +| `n2=10.0.0.2:17000/ 17100` | OK after trim → 17100 | +| `n2=::1:17000` | **Reject** (no brackets) | +| two peers same VoiceAddr | **Reject** (H-20) | +| spaces around id/addr | trimmed OK | + +### Validation (`ValidateCluster` extensions) — fail closed (H-20) + +When `ClusterEnabled`: + +1. Existing: node id, TCP listen, PSK, peers ≥1 ≤4, no self, unique ids, TCP host:port. +2. `AFV_CLUSTER_VOICE_LISTEN` non-empty; `SplitHostPort` OK; **port ≠ 0**. +3. Local TCP listen port ≠ local voice listen port (string port compare after SplitHostPort). +4. Peer parse algorithm above succeeds for all entries. +5. **Unique peer `VoiceAddr`** (exact string after JoinHostPort) — **fail**, not warn. +6. **`ClusterVoiceListen` ≠ `UDPListen`** after normalize: both SplitHostPort; compare `host:port` with empty host / `0.0.0.0` / `::` / `*` treated as wildcard-equal on host **if ports equal** → **fail** (client CryptoDTO must not share the mesh voice socket). Same for concrete equal host:port. (This is **local-only** socket collision — not peer comparison.) +7. **Local voice vs peer `VoiceAddr` — full host:port equality only (no port-only wildcard rule):** + - Fail closed **only** when the configured local voice listen string’s `JoinHostPort(host, port)` **equals** a peer’s `VoiceAddr` string (after the same parse/JoinHostPort normalization used for peers). + - Examples that **must accept**: + - Local `0.0.0.0:17001`, peer `10.0.0.2:17001` — **OK** (recommended multi-host layout: same port number, different hosts). + - Local `0.0.0.0:17001`, peer `10.0.0.2:17000/17001` → VoiceAddr `10.0.0.2:17001` — **OK**. + - Examples that **must reject**: + - Local `127.0.0.1:17001`, peer `127.0.0.1:17001` — same host:port (same-machine multi-process must use distinct ports as in the localhost example: `17001` vs `17011`). + - Local `10.0.0.1:17001`, peer `10.0.0.1:17001` — equal concrete endpoints. + - **Do not** treat wildcard local host as “port equals any peer port.” That incorrectly rejects production multi-host meshes where every node uses voice port `N+1` on different IPs. + - **Optional same-host tightening (implementer may add, not required):** if peer host is loopback (`127.0.0.1` / `::1`) **and** local host is loopback or wildcard **and** ports equal → fail (catches localhost multi-process when local binds `0.0.0.0:17001` and peer is `127.0.0.1:17001`). Wiki still documents distinct ports for multi-process on one host. +8. Local TCP listen must not equal any peer TCP `Addr` by the **same full host:port equality** rule (no port-only wildcard). Self must not appear in peers (existing). + +**Defaults:** no silent default for `VOICE_LISTEN` when cluster enabled — **require explicit bind**. Recommended multi-host convention: every node `AFV_CLUSTER_LISTEN=0.0.0.0:N`, `AFV_CLUSTER_VOICE_LISTEN=0.0.0.0:N+1`, peers `id=otherHost:N` (voice derives `otherHost:N+1`). + +### Config struct + +```go +// config.go additions +ClusterVoiceListen string `env:"AFV_CLUSTER_VOICE_LISTEN"` + +// mesh.go — ClusterPeer extended +type ClusterPeer struct { + ID string + Addr string // TCP host:port (control) — SplitHostPort-valid + VoiceAddr string // UDP host:port (voice) — derived or explicit +} +``` + +Update `parseClusterPeers` to populate `VoiceAddr` via the normative algorithm. + +### Sticky LB ops model (unchanged, reinforced) + +```text + ┌──────────── sticky REST ────────────┐ + Client ──HTTPS──► │ LB / DNS per-node hostname │ + └───────────────┬─────────────────────┘ + ▼ + AFV node home (mint keys) + │ + Client UDP ─────┘ AFV_UDP_ADVERTISE_IPV4 only + + Mesh (inter-node only): + TCP AFV_CLUSTER_LISTEN ↔ peers' TCP + UDP AFV_CLUSTER_VOICE_LISTEN ↔ peers' voice addrs +``` + +- Mesh does **not** replace sticky affinity for clients. +- Operators must open **both** control TCP and voice UDP between node security groups. +- Client UDP port (`AFV_UDP_LISTEN`, often 50000) is **independent** of mesh voice port (often 17001). + +### Example two-node localhost + +```bash +# Node n1 +AFV_CLUSTER_ENABLED=true +AFV_CLUSTER_NODE_ID=n1 +AFV_CLUSTER_LISTEN=127.0.0.1:17000 +AFV_CLUSTER_VOICE_LISTEN=127.0.0.1:17001 +AFV_CLUSTER_PEERS=n2=127.0.0.1:17010/17011 +AFV_CLUSTER_PSK=dev-shared-secret +AFV_UDP_LISTEN=127.0.0.1:50000 +AFV_UDP_ADVERTISE_IPV4=127.0.0.1 +AFV_API_LISTEN=127.0.0.1:8080 + +# Node n2 +AFV_CLUSTER_ENABLED=true +AFV_CLUSTER_NODE_ID=n2 +AFV_CLUSTER_LISTEN=127.0.0.1:17010 +AFV_CLUSTER_VOICE_LISTEN=127.0.0.1:17011 +AFV_CLUSTER_PEERS=n1=127.0.0.1:17000/17001 +AFV_CLUSTER_PSK=dev-shared-secret +AFV_UDP_LISTEN=127.0.0.1:50001 +AFV_UDP_ADVERTISE_IPV4=127.0.0.1 +AFV_API_LISTEN=127.0.0.1:8081 +``` + +--- + +## HybridMesh detailed design + +### Type and construction + +```go +// mesh_hybrid.go / mesh_tcp.go + +type HybridMeshConfig struct { + NodeID string + ListenTCP string // AFV_CLUSTER_LISTEN + ListenVoice string // AFV_CLUSTER_VOICE_LISTEN + Peers []ClusterPeer // remote only; TCP + Voice addrs + PSK string + // HeartbeatInterval default 2s + // PeerDeathAfter default 15s + // CtrlQueueDepth default 64 + // VoiceQueueDepth default 256 + // MaxPeers default 4 + Logger *slog.Logger +} + +type HybridMesh struct { + cfg HybridMeshConfig + // TCP + ln net.Listener + // UDP voice + voicePC net.PacketConn + // peerID → peer state + // callbacks / providers same as MemoryMesh + // implements Mesh +} + +func NewHybridMesh(cfg HybridMeshConfig) (*HybridMesh, error) +``` + +`NewDefault` path: + +```go +// bootstrap.go (normative) +if cfg.ClusterEnabled { + if err := cfg.ValidateCluster(); err != nil { + return nil, err + } + peers, _ := parseClusterPeers(cfg.ClusterPeers) // already validated + hm, err := NewHybridMesh(HybridMeshConfig{ + NodeID: cfg.ClusterNodeID, + ListenTCP: cfg.ClusterListen, + ListenVoice: cfg.ClusterVoiceListen, + Peers: peers, + PSK: cfg.ClusterPSK, + }) + if err != nil { + return nil, err + } + srv := New(cfg, ...) + srv.SetMesh(hm) + return srv, nil +} +``` + +Replace: + +```go +if cfg.ClusterEnabled { + return nil, errClusterTCPNotBuilt +} +``` + +Keep `errClusterTCPNotBuilt` (or rename to hybrid-not-wired) through PR-10b.2/10b.3 so partial HybridMesh is never production-attached. Delete or redefine to `afv hybrid mesh failed to start: …` only in PR-10b.4 when `NewDefault` wires HybridMesh. + +### Constants (H-12 / H-18 / H-19) + +```go +// Normative constants (mesh_udp_voice.go or mesh.go) + +// maxMeshVoiceDatagram is the only accepted UDP voice size (bytes). +// Do NOT use MaxMeshPayload (1 MiB) for UDP reads. +const maxMeshVoiceDatagram = 16 << 10 // 16384 + +// maxMeshVoiceTxRadios caps TxRadios on mesh AudioRelay encode/decode for voice path. +const maxMeshVoiceTxRadios = 64 + +// meshVoiceInboundRatePerSec soft cap per authed peer (datagrams/s). Excess dropped. +// Anti-runaway only — not a capacity plan. Default 5000: see H-19 capacity note. +const meshVoiceInboundRatePerSec = 5000 + +// Formula (documentation / tests — must fit in maxMeshVoiceDatagram): +// frameOH = 4 + 1 +// strings = 2 * (2 + maxMeshString) // origin + callsign worst case +// fixed = 4 + 3 + 4 + maxAudioRelayBytes + 2 // seq, bools, audio, nTx +// radios = maxMeshVoiceTxRadios * 30 // u16+u32+3*f64 +// worst = frameOH + strings + fixed + radios +// = 5 + 2052 + 8205 + 1920 = 12182 ≤ 16384 +// Encode rejects if encoded frame length > maxMeshVoiceDatagram even if audio ≤ 8192. +``` + +UDP read buffer: allocate `maxMeshVoiceDatagram+1`; if `n > maxMeshVoiceDatagram` drop and count (same pattern as client UDP oversized drop in `udp.go`). + +**H-19 capacity note (inbound rate default):** + +| Factor | Order of magnitude | +|--------|--------------------| +| Opus / CryptoDTO TX rate per talker | ~20–50 datagrams/s (codec-dependent) | +| Concurrent remote talkers whose Interest hits this node (busy event) | tens (not thousands) on ≤5-node mesh | +| Mesh fan-in from **one** peer node | sum of that node’s concurrent TX that pass Interest | + +- **200/s was too low:** a few concurrent talkers on a peer can exceed it → silent drop under normal load. +- **Default 5000/s per peer:** headroom for ~100 talkers × ~50 pps, or fewer talkers with burst/retransmit noise, while still bounding runaway (buggy peer spinning TX). Still cheap vs per-AR encrypt + client `WriteTo`. +- Cap is **not** a guarantee of voice quality under overload; it is fail-soft anti-DoS. Not an env product knob in first cut (constant; change in code if ops proves need). +- Formula alternative if implementers prefer: `max(5000, min(20000, MaxSessions*50/4))` — optional; **normative default remains 5000**. + +### Mesh interface compliance + +`HybridMesh` implements the existing `Mesh` interface in `mesh.go` without signature changes: + +| Method | Behavior | +|--------|----------| +| `Start` | Resolve peer voice addrs; listen TCP + UDP voice; accept/dial; HB loop; post-auth Snapshot+Interest when each peer Hello completes | +| `Stop` | Close listeners/conns; wait workers | +| `PublishTrxSnapshot/Delta/Leave/Interest` | Encode payload; enqueue **control** job per live **authed** peer (drop-oldest 64). Outbound Interest does **not** mutate local `peerInterest` | +| `EnqueueAudioRelay` | Normative MemoryMesh-compatible filter (below); never touch TCP | +| `PeerWants` / `InterestedPeers` | Read `peerInterest` map (filled **only** from inbound TCP Interest apply) | +| Callbacks | `OnAudioRelay`, `OnPeerDead`, `OnDirectory`, providers | + +### Per-peer state + +```go +type hybridPeer struct { + id string + tcpAddr string + voiceAddrs []*net.UDPAddr // all resolved at Start + sendVoice *net.UDPAddr // WriteTo target (prefer IPv4 if dual-stack) + conn net.Conn + ctrlQ *dropOldestQueue[meshCtrlJob] + voiceQ *dropOldestQueue[voiceJob] + lastHB time.Time + authed bool + stop chan struct{} + // inboundRate: token bucket / sliding window for H-19 +} + +type voiceJob struct { + // exclusive copy of full mesh frame bytes for this peer only + packet []byte +} +``` + +**Deadlock / concurrency (H-7, H-13):** + +- **Per-peer TCP writer** drains `ctrlQ`; **per-peer voice drainer** drains `voiceQ` and `WriteTo`s (not a global shared buffer). +- **Packet copy at enqueue** per peer — forbid sharing one `[]byte` across peers or concurrent `WriteTo`. +- Peer-map mutex: short hold for lookup/enqueue/conn swap only; **never** across `conn.Write` or `WriteTo`. +- Server already unlocks registry before `EnqueueAudioRelay`. + +### Interest apply on HybridMesh (H-14 — Issue 3) + +MemoryMesh applies Interest **synchronously inside `PublishInterest`** on the peer map (M-17) and does **not** re-apply from control drain. HybridMesh **must not** copy that sync path. + +**Normative inbound Interest (TCP reader, type 30):** + +```text +on TCP DecodeMeshFrame type == MeshTypeInterest: + p, err := DecodeInterest(payload) + if err != nil: death path (malformed post-Hello control) + if p.NodeID != fromPeerID: close peer // must match Hello-authenticated id + set := make(map[FreqCell]struct{}, len(p.Entries)) + for e in p.Entries: + set[FreqCell{FreqHz: e.FreqHz, Cell: CellKey{ILat: e.ILat, ILon: e.ILon}}] = {} + // REPLACE entire set (not merge). Empty entries → empty set (clears Interest). + lock; peerInterest[fromPeerID] = set; unlock + // no callback required; PeerWants sees new set after unlock +``` + +- Entry cap remains the **publisher’s** job (`interestMaxEntries`); receiver uses existing decode caps. +- Full replace under one assignment — no partial merge. +- `EnqueueAudioRelay` copies matching peer IDs under `RLock`, unlocks, then enqueues (H-13). + +**Outbound `PublishInterest`:** encode + enqueue control to each authed peer only. Does **not** write `peerInterest[self]`. + +### EnqueueAudioRelay / Interest fan-out (Issue 9 — match MemoryMesh) + +Must match landed `MemoryMesh.EnqueueAudioRelay` (`mesh_memory.go`): + +```text +EnqueueAudioRelay(relay): + if OriginNode == "": OriginNode = self + Audio = append([]byte(nil), Audio...) + TxRadios = append([]RelayTxRadio(nil), TxRadios...) + + snapshot authed peerIDs under RLock; unlock + for each peerID: + want = false + for each tx in relay.TxRadios: + ck = CellKey{ + ILat: geo.CellIndex(tx.LatDeg, geo.DefaultGridCellDeg), + ILon: geo.CellIndex(tx.LonDeg, geo.DefaultGridCellDeg), + } + if PeerWants(peerID, tx.FreqHz, ck): want = true; break + if !want: continue // empty Interest → no flood (Case F) + payload, err := EncodeAudioRelay(relay) // max audio; enforce maxMeshVoiceTxRadios + if err != nil: drop; continue + packet = EncodeMeshFrameBytes(type=20, payload) + if len(packet) > maxMeshVoiceDatagram: drop; continue + peer.voiceQ.Enqueue(voiceJob{packet: copy(packet)}) // drop-oldest 256 +``` + +### TCP control plane + +#### Listen / dial (H-9) + +- `net.Listen("tcp", ListenTCP)` +- Dial peers with `peer.ID > self.NodeID`; backoff 200ms → cap 5s (FSD pattern) +- Accept handles reverse +- Both sides send Hello immediately; first frame must be Hello + +#### Hello (existing M-3) + +- `VerifyHelloPSK`; nodeID ∈ allowlist and ≠ self +- Reject wrong PSK / unknown / self / non-Hello first → close, Warn without PSK +- On success: + 1. If existing live conn for peerID: **close old**, stop old workers, **recreate empty** ctrlQ/voiceQ + 2. Install new conn; `authed=true`; `lastHB=now` + 3. Ensure per-peer TCP writer + voice drainer running + 4. Send TrxSnapshot then Interest (M-16) + +#### Allowed TCP frame types + +| Type | Allowed on TCP | +|------|----------------| +| 1 Hello | First frame only; post-auth unexpected Hello → **close** | +| 2 Heartbeat | Yes | +| 10–12 Trx*/Leave | Yes | +| 20 AudioRelay | **No** — close peer (H-2) | +| 30 Interest | Yes → apply replace | +| other | Close peer | + +#### Control writer + +- Per-peer goroutine drains `ctrlQ` (depth 64 drop-oldest) +- `EncodeMeshFrame(conn, typ, payload)` +- Write error → **death path** (same as read failure) + +#### Heartbeat / death / lastHB (H-8 — Issue 4) + +**`lastHB`:** advance on **every successfully decoded post-Hello control frame** (any allowed type), not only Heartbeat — FSD `tcp_mesh.go` pattern. + +**Death triggers (any):** (1) `now-lastHB > 15s`; (2) TCP read error/close; (3) TCP write error. UDP errors alone never kill the peer. + +**Death procedure (idempotent):** + +```text +authed = false +close conn; stop writer +peerInterest[peerID] = empty set +re-init empty ctrlQ + voiceQ (drop stale) +OnPeerDead(peerID) // Server → remote.RemoveNode; local sessions stay +slog.Warn +// dialer keeps dialing; acceptor waits +``` + +**Reconnect:** Hello success → replace conn + queues + Snapshot+Interest. + +#### Inbound TCP reader + +- Read deadline ≈ `peerDeathGrace + 5s` +- Success → `lastHB=now`; dispatch directory / Interest / HB +- Never `OnAudioRelay` from TCP +- Error → death procedure + +### UDP voice plane + +#### Listen / socket invariant (H-5 / H-16) + +- **One** unconnected `ListenPacket("udp", ListenVoice)` for mesh voice +- All `WriteTo` on that socket → **source port == bound VOICE_LISTEN port** +- Client CryptoDTO stays on `AFV_UDP_LISTEN` (different socket) + +#### UDP allowlist canonicalize (H-16 — Issue 2) + +```text +At Start, for each peer VoiceAddr: + resolve host → one or more IPs (literal IP → single) + for each IP: store UDPAddr{IP: IP.To16(), Port: voicePort} + sendVoice = first IPv4 if any, else first + for each resolved addr: + insert allowlist[canonicalUDPKey(addr)] = peerID + if To4() != nil also insert IPv4 string form (and v4-mapped form) + so ::ffff:127.0.0.1 and 127.0.0.1 match same peer + if key maps to different peerID → fail Start + +canonicalUDPKey: JoinHostPort(ip.To16().String(), port) (+ dual keys for v4) +``` + +Inbound: lookup src → peerID; require `authed`; else drop + counter; **never** `OnAudioRelay` for unknown sources. + +**IPv4-mapped policy:** **accept as equal** (dual-key insert). Unit-test both forms. + +#### Outbound path + +See EnqueueAudioRelay above. Voice drain **per-peer**: + +```text +WriteTo(job.packet, peer.sendVoice) +// err → udpSendErrs++; Debug only — no death +``` + +#### Inbound path (strict framing Issue 14 + flood H-19) + +```text +n, src := ReadFrom(buf sized maxMeshVoiceDatagram+1) +if n > maxMeshVoiceDatagram: drop; continue + +peerID = allowlistLookup(src) +if peerID == "" || !authed: drop; continue // no OnAudioRelay + +if !inboundRate.Allow(peerID): drop; inboundRateDrops++; continue // H-19, default 5000/s + +frame, err := DecodeMeshFrameExact(buf[:n]) +// require 4 + int(be_u32 length) == n (zero trailing; no multi-frame) +if err != nil || frame.Type != 20: drop; continue +relay, err := DecodeAudioRelay(frame.Payload) +if err != nil || len(relay.TxRadios) > maxMeshVoiceTxRadios: drop; continue +if relay.OriginNode != peerID: drop; continue +onAudio(peerID, relay) // after releasing mesh locks +``` + +### MTU / IP fragmentation (Issue 6) + +- Real Opus frames expected **well under path MTU** (typical AFV voice ≪ 1500). +- `maxAudioRelayBytes=8192` is a **safety clamp** matching client `AFV_MAX_DATAGRAM`, not a target size. +- Full mesh frame + max clamp **can** IP-fragment; fragment loss drops the whole relay. +- **Accepted risk** on private mesh. Wiki: verify UDP path; avoid hostile fragment-dropping paths. +- **Non-goal:** PMTUD, smaller default clamp, multi-datagram reassembly. H-3 unchanged. + +### MemoryMesh relationship + +| Concern | MemoryMesh | HybridMesh | +|---------|------------|------------| +| Production `NewDefault` | Never | Yes when enabled (10b.4+) | +| Transport | In-process channels | TCP + UDP sockets | +| Interest apply | Sync in `PublishInterest` (M-17) | **Async TCP reader replace** only | +| Voice queue | depth 256 in-process | depth 256 → per-peer UDP WriteTo | +| E2E barrier | `PeerWants` immediate | Poll `PeerWants` with timeout after remote publish | +| Tests | Cases A–F required regression | Localhost hybrid e2e (T11) | + +No requirement to change MemoryMesh dual-queue design; it remains a valid `Mesh` for unit/e2e without ports. + +--- + +## API / Interface Changes + +### Client-visible API + +**None.** REST and CryptoDTO unchanged. `PostCallsignResponse.addressIpV4` remains this node’s client UDP advertise address. + +### Internal + +| Surface | Change | +|---------|--------| +| `Config.ClusterVoiceListen` | New env field | +| `ClusterPeer.VoiceAddr` | New | +| `parseClusterPeers` / `ValidateCluster` | Normative voice peer parse + H-20 fail-closed checks | +| `NewHybridMesh` + `HybridMesh` | New production `Mesh` | +| `NewDefault` | Construct hybrid mesh when enabled (PR-10b.4 only) | +| `errClusterTCPNotBuilt` | Keep until 10b.4; then delete or redefine to hybrid-start errors (`AFV hybrid mesh failed to start: …`) | +| `Mesh` interface | **No change** (already sufficient) | +| `DecodeMeshFrameExact` | New helper: require full datagram consumed | +| Package comments | `mesh.go` / `mesh_frame.go`: say HybridMesh production, not “TCPMesh is PR-10b” only | + +### Bootstrap / test surface (Issue 11) + +| Today | After 10b.4 | +|-------|-------------| +| `TestNewDefault_ClusterEnabledFailsClosed` asserts error contains `"TCP"` | Replace with: valid cluster env **including `AFV_CLUSTER_VOICE_LISTEN`** → `NewDefault` returns server with **non-nil `Mesh()`**; incomplete config still errors; `ENABLED=false` → nil mesh | +| `errClusterTCPNotBuilt` | Remove from success path | +| Comments referencing TCPMesh-only production | Update to HybridMesh | + +Until 10b.4, `NewDefault` remains fail-closed (10b.2/10b.3 must not claim production readiness). + +### Before / after bootstrap (10b.4) + +```go +// BEFORE (today through 10b.3) +if cfg.ClusterEnabled { + return nil, errClusterTCPNotBuilt // or errHybridMeshNotWired +} + +// AFTER 10b.4 +if cfg.ClusterEnabled { + // ValidateCluster already called (includes VOICE_LISTEN) + hm, err := NewHybridMesh(hybridConfigFrom(cfg)) + if err != nil { + return nil, fmt.Errorf("afv hybrid mesh failed to start: %w", err) + } + s := New(...) + s.SetMesh(hm) + return s, nil +} +``` + +--- + +## Data Model Changes + +**None for SQLite/rqlite.** Mesh state remains process-local: + +| Structure | Lifetime | +|-----------|----------| +| Local registry / AEAD keys | Process | +| `remoteDir` | Process; purged on peer death | +| Peer Interest maps | Process | +| TCP conns + queues | Process | +| UDP voice socket | Process | + +No migrations. No Opus in DB. + +--- + +## Alternatives Considered + +### A1. TCP-only mesh with dual queues (original PR-10b) + +| Pros | Cons | +|------|------| +| One port; simpler firewall | HOL + retransmit delay on voice; product rejected | + +**Verdict:** **Rejected** for production. Superseded by this document. + +### A2. QUIC single connection (control + datagrams) + +| Pros | Cons | +|------|------| +| Multiplex + unreliable datagrams | New dependency; ops complexity; overkill for static ≤5 node private mesh | + +**Verdict:** Deferred. Hybrid TCP+UDP matches ops skill set and FSD-like static peers. + +### A3. Post-Hello voice endpoint advertisement + +| Pros | Cons | +|------|------| +| Flexible behind NAT rewrite | Extra state; bootstrap race; less predictable firewalling | + +**Verdict:** Rejected for first cut. Static config (H-4) is primary. May revisit as optional Hello extension without changing type numbers. + +### A4. Raw Opus UDP without mesh frame wrapper + +| Pros | Cons | +|------|------| +| Slightly smaller packets | Fork decode path; lose type guard | + +**Verdict:** Rejected. Full mesh frame on UDP (H-3). + +### A5. Derive voice port only from `AFV_CLUSTER_LISTEN+1` without `VOICE_LISTEN` + +| Pros | Cons | +|------|------| +| One less env var | Forces same bind host semantics; harder multi-process colocated tests | + +**Verdict:** Rejected as sole scheme. Explicit `AFV_CLUSTER_VOICE_LISTEN` required; peer default +1 remains. + +### A6. Merge AFV into `internal/cluster` + +| Pros | Cons | +|------|------| +| One mesh ops surface | Violates import graph; rate domain pollution (KD-8) | + +**Verdict:** Rejected. + +### A7. DTLS or AEAD on mesh voice + +| Pros | Cons | +|------|------| +| Stronger on hostile nets | Key distribution; latency; not required under private-net + allowlist trust model | + +**Verdict:** Deferred. Same trust root as FSD mesh PSK + static peers. + +--- + +## Security & Privacy Considerations + +| Threat | Severity | Mitigation | +|--------|----------|------------| +| Spoofed UDP AudioRelay | High | Canonical allowlist (H-16); require control-authed peer; `OriginNode` must match peer id; unknown sources never `OnAudioRelay` | +| Unauthenticated TCP peer | High | Hello first-frame + CT-PSK + allowlist (existing) | +| Type-20 on TCP as smuggling path | Med | H-2: close connection | +| Multi-frame UDP smuggling | Med | Strict `DecodeMeshFrameExact` (consumed == n) | +| AEAD key exfiltration | High | Keys never on mesh wire (PR-10 Case E); hybrid does not add key fields | +| PSK brute force | Med | Private network; CT compare; no PSK in logs | +| Outbound amplification | Med | Interest filter; max 4 peers; drop-oldest voice; max audio 8192 | +| Inbound flood from authed peer | Med | H-19 per-peer soft rate limit (**5000/s** anti-runaway) + counters; residual under private-net trust | +| Oversized UDP DoS | Med | `maxMeshVoiceDatagram` (16 KiB) — never `MaxMeshPayload` for UDP | +| Ambiguous allowlist (dup voice) | High | H-20 fail-closed on duplicate VoiceAddr | +| Client/mesh socket mix | High | H-20 fail if VOICE_LISTEN == UDPListen | +| Peer IP mismatch (NAT) | Low | Configure reachable VoiceAddr; log TCP RemoteAddr ≠ config as warn only (H-15) | +| Sticky LB bypass | Med | Ops docs; per-node advertise IP; mesh is inter-node only | + +**Trust root:** operators who know `AFV_CLUSTER_PSK` and appear in `AFV_CLUSTER_PEERS` — identical operational model to FSD mesh. + +**Privacy:** AudioRelay carries callsign + geometry + Opus; treat mesh network as trusted as voice content. No client passwords/JWT on mesh. + +**Inbound flood policy (Issue 8 / H-19):** private-net trust assumes peers are operator-controlled. Soft rate limit is **anti-runaway** (best-effort CPU protection), **not** a capacity plan and **not** a DDoS product. Default **5000/s** is sized so normal multi-talker fan-in is not clipped; residual risk remains if a peer is compromised and floods above the cap. + +--- + +## Observability + +| Signal | Mechanism | +|--------|-----------| +| Control listen / voice listen up | `slog.Info` with addrs at Start | +| Hello success / fail | Info / Warn (no PSK) | +| Peer death | Warn + purge | +| Voice queue drops | `atomic.Uint64` + periodic Debug | +| Control queue drops | atomic + Debug; Snapshot resync on reconnect | +| UDP allowlist drops | atomic + Debug (rate-limit log spam) | +| Inbound rate drops (H-19) | atomic + periodic Debug | +| Oversize UDP drops | atomic | +| TCP type-20 reject | Warn once per peer episode | +| Interest truncate | Warn (existing) | +| **Silent UDP mute** (Issue 15) | **Required:** periodic `slog.Debug` (≤1/min) if any authed peer has non-empty Interest **and** both `voiceTx==0` and `voiceRx==0` for that peer for **>30s** | + +Optional later (not merge-blocking): Prometheus counters for relay in/out/drop, peer RTT via HB timestamps. + +No new metrics subsystem required for acceptance. Wiki checklist (10b.5) remains mandatory for operators. + +--- + +## Testing Plan + +### Gates (every hybrid PR) + +```bash +go test -race ./internal/afv/... +bash scripts/check-coverage.sh 80 +bash scripts/check-import-graph.sh +bash scripts/check-hygiene.sh +gofmt -l . # clean on touched files +go build -o openfsd ./cmd/openfsd +``` + +### Coverage floors (spirit of PR-10 M-18) + +| Scope | Bar | +|-------|-----| +| New TCP control modules | ≥90% statements | +| New UDP voice modules | ≥90% | +| Peer parse / ValidateCluster voice extensions | ≥95% | +| Frame path reuse (type-20 on UDP encode/decode integration) | table-driven | +| Overall `internal/afv` | do not regress below soft 80% | + +### Mandatory tests (new / extended) + +| # | Test | Notes | +|---|------|-------| +| T1 | Peer parse fixtures (IPv4 default+1, explicit `/`, IPv6 brackets, overflow 65535, empty/non-numeric voice, port 0, dup VoiceAddr) | `mesh_config_test.go` | +| T1b | `ValidateCluster`: require `VOICE_LISTEN`; voice≠TCP; voice≠UDPListen; full host:port local≠peer VoiceAddr; port 0 reject; **accept** multi-host `0.0.0.0:17001` + peer `10.0.0.2:17001` | same | +| T2 | TCP Hello wrong PSK → no authed peer; no voice accepted | `mesh_tcp_test.go` | +| T3 | Non-Hello first TCP frame → close | net.Pipe or localhost | +| T4 | Type 20 on TCP → peer closed | unit | +| T5 | UDP from unknown source: drop; **OnAudioRelay never called** | `mesh_udp_voice_test.go` | +| T6 | UDP from known peer while not authed: drop | unit | +| T6b | IPv4-mapped `::ffff:127.0.0.1` allowlist accept-as-equal | unit | +| T7 | Interest empty → zero UDP sends under load | unit | +| T7b | Interest **replace** (not merge); empty Interest clears prior set | TCP reader unit | +| T8 | Voice queue drop-oldest 256 under backpressure | queue + hybrid | +| T9 | Control queue drop-oldest 64 | same | +| T10 | Peer death (read error + 15s timeout): purge dir; stop voice; local sessions alive; queues re-inited | hybrid | +| T10b | Dual conn replace: second Hello closes first | unit | +| T10c | `lastHB` advances on TrxDelta (not only HB) — death timer reset | unit | +| T11 | **Localhost two-node hybrid e2e:** two `Server`+`HybridMesh`, REST+UDP clients, A→B AR | `mesh_hybrid_e2e_test.go` | +| T12 | Wrong PSK two-node: no A2A | e2e | +| T13 | `NewDefault` valid cluster+`VOICE_LISTEN` → non-nil `Mesh()`; incomplete still errors; disabled nil mesh | replace fails-closed-TCP test | +| T14 | Import graph: no `internal/cluster` | CI | +| T15 | MemoryMesh Cases A–F still green | regression | +| T16 | UDP oversize `n > maxMeshVoiceDatagram` dropped before decode | unit | +| T17 | Strict framing: trailing bytes after frame → drop | unit | +| T18 | Inbound rate limit: delivers under **5000/s** default; drops excess above cap | unit | + +**T11 detail:** prefer in-package dual HybridMesh on `127.0.0.1` with free ports (`:0` bind then discover) inside `go test` — no external processes required if both meshes run in one test binary. Optional stress: `exec` two `openfsd` binaries in integration tag later (not required for merge). + +--- + +## Rollout Plan + +1. **Default off:** `AFV_CLUSTER_ENABLED=false` — P0 single-node unchanged. +2. **Land hybrid code behind enable flag** with tests green. +3. **Staging:** two hosts / two localhost processes; open TCP+UDP mesh ports; sticky REST; verify A2A + peer kill + wrong PSK. +4. **Operator wiki:** Configuration + Deployment multi-node AFV sections (ports, sticky LB, example compose). +5. **Rollback:** set `AFV_CLUSTER_ENABLED=false` or stop peers; local voice continues; no DB rollback. +6. **Parent doc status:** mark PR-10b hybrid landed; strike TCP-only voice. + +### Feature flags + +No additional flag beyond `AFV_CLUSTER_ENABLED`. Voice and control always pair when enabled. + +--- + +## Risks + +| Risk | Severity | Mitigation | +|------|----------|------------| +| Operators open TCP but forget UDP firewall → silent cross-node mute | **High** | Require `VOICE_LISTEN`; wiki checklist; startup Info both binds; **periodic Debug** if Interest non-empty and voice rx/tx zero >30s (Issue 15) | +| TCP port +1 collision with unrelated service | **Med** | Explicit `/voicePort`; validate ≠ control port | +| UDP spoof on shared L2 | **Med** | Canonical allowlist + authed gate; private net | +| Interest race before first publish | **Med** | Post-Hello Snapshot+Interest; PeerWants poll in e2e | +| Dual death signals if UDP errors used | **Low** | H-8: TCP-only death | +| IP fragmentation of large AudioRelay | **Med** | Expected Opus ≪ MTU; max 8192 is clamp; accepted private-mesh risk; wiki note (Issue 6) | +| Authed peer inbound flood | **Med** | H-19 rate limit; residual under private-net trust (Issue 8) | +| Allowlist dual-stack mismatch | **Med** | H-16 dual-key IPv4-mapped accept; Start resolve all A/AAAA | +| Config ambiguity (dup voice / client socket) | **High→mitigated** | H-20 fail-closed | +| Port `:0` in production confusion | **Low** | Fixed ports in docs; tests may use `:0` | +| Scope creep into QUIC/DTLS | **Low** | Non-goals | + +--- + +## Open Questions + +| # | Question | Disposition | +|---|----------|-------------| +| 1 | Voice port discovery via Hello advertise? | **Closed for PR-10b:** static config (H-4); advertise deferred | +| 2 | Per-packet voice auth token? | **Closed:** allowlist + control auth sufficient first cut | +| 3 | Keep `errClusterTCPNotBuilt` name? | **Closed:** keep until 10b.4; then delete or redefine to `afv hybrid mesh failed to start: …` (H-10 / Issue 16) | +| 4 | Single file vs `mesh_tcp.go` + `mesh_udp_voice.go` | **Closed:** split (H-6) | +| 5 | Type 20 on TCP ignore vs close? | **Closed:** close (H-2) | +| 6 | IPv6 peer strings | **Closed:** brackets required; normative parse algorithm + fixtures (Issue 1) | +| 7 | `lastHB` only HB vs any frame? | **Closed:** any successfully decoded post-Hello frame (H-8 / Issue 4) | +| 8 | Inbound rate limit value? | **Closed default 5000/s** (H-19); anti-runaway with multi-talker headroom; not a product env knob in first cut | +| 9 | Wildcard local voice vs peer same port number? | **Closed:** full host:port equality only; multi-host same port **accepted** (H-20 rule 7 rev 3) | + +--- + +## Observability of acceptance (operator checklist) + +- [x] Both nodes log control TCP listen + voice UDP listen +- [x] Hello success both directions +- [x] Interest non-empty after clients bind +- [x] Cross-node AR heard +- [x] Kill one process → peer death warn; survivors keep local A2A +- [x] Wrong PSK → auth fail; no voice + +--- + +## References + +- `docs/design/afv-mesh-pr10.md` — Memory path + framing + Interest (parent; TCP-only voice **superseded**) +- `docs/design/afv-server.md` — KD-8/9/16/17, Clustering +- `docs/design/afv-mesh-implementer-prompt.md` — implementer contract +- `Agents.md` — package ownership; `internal/afv` ↛ `internal/cluster` +- `internal/afv/mesh.go` — `Mesh` interface, `ValidateCluster`, Server hooks +- `internal/afv/mesh_frame.go` — `EncodeAudioRelay` / `DecodeAudioRelay` / `VerifyHelloPSK` +- `internal/afv/mesh_queue.go` — drop-oldest queues +- `internal/afv/mesh_memory.go` — tests-only reference implementation +- `internal/afv/bootstrap.go` — `NewDefault` fail-closed today +- `internal/cluster/tcp_mesh.go` — dial rule / peerConn patterns (**do not import**) +- `scripts/check-import-graph.sh` — hard import edges + +--- + +## PR Plan + +Incremental, independently reviewable PRs. Each keeps `go test -race ./internal/afv/...`, import graph, hygiene, and build green. + +### PR-10b.1 — Config: voice listen + peer voice address parse + parent transport pointer + +| | | +|--|--| +| **Title** | afv mesh: AFV_CLUSTER_VOICE_LISTEN + peer voice grammar + design land | +| **Files** | `internal/afv/config.go`, `internal/afv/mesh.go` (`ClusterPeer`, `parseClusterPeers`, `ValidateCluster`), `mesh_config_test.go`, **`docs/design/afv-mesh-pr10b.md`**, patch status + transport row in `docs/design/afv-mesh-pr10.md` and `docs/design/afv-server.md` (H-21 / Issue 12) | +| **Depends on** | None (PR-10 already on `dev`) | +| **Description** | Normative peer parse algorithm + fixtures; H-20 fail-closed (VOICE_LISTEN, voice≠TCP, voice≠UDPListen, unique VoiceAddr, **full host:port** local≠peer VoiceAddr — multi-host same port number OK). **Do not** construct production mesh — `NewDefault` still fail-closed. Land this design file and **immediately** correct parent “TCP carries all types” rows. | + +### PR-10b.2 — HybridMesh TCP control plane + +| | | +|--|--| +| **Title** | afv mesh: HybridMesh TCP control (Hello, HB, directory, Interest) | +| **Files** | `internal/afv/mesh_tcp.go`, `mesh_tcp_test.go`, optional `mesh_hybrid.go`; reuse `mesh_frame.go` / `mesh_queue.go` | +| **Depends on** | PR-10b.1 | +| **Description** | TCP listen/dial (ID>self), Hello first-frame + CT-PSK, control queue 64, Snapshot/Delta/Leave/Interest/HB, **lastHB on any post-Hello frame**, death on read/write/timeout, dual-conn replace, Interest **replace** apply on reader, reject type 20 on TCP. **`EnqueueAudioRelay` is a no-op** (preferred — Issue 13); do **not** buffer voice without send. **Not production-complete:** `NewDefault` remains fail-closed (`errClusterTCPNotBuilt` or renamed “hybrid not wired”). Tests T2–T4, T7b, T9, T10/T10b/T10c control-side. | + +### PR-10b.3 — HybridMesh UDP voice plane + +| | | +|--|--| +| **Title** | afv mesh: UDP AudioRelay voice plane + Interest fan-out | +| **Files** | `internal/afv/mesh_udp_voice.go`, `mesh_udp_voice_test.go`, integrate Start/Stop / `EnqueueAudioRelay` | +| **Depends on** | PR-10b.2 | +| **Description** | Bind VOICE_LISTEN; H-16 allowlist; MemoryMesh-compatible fan-out; `maxMeshVoiceDatagram`; strict exact-frame decode; per-peer voice drainer + copy; H-19 inbound rate; no death from UDP alone. Tests T5–T8, T6b, T16–T18. Still **not** wired in `NewDefault`. | + +### PR-10b.4 — Bootstrap production wiring + localhost hybrid e2e + +| | | +|--|--| +| **Title** | afv mesh: NewDefault HybridMesh attach + localhost A2A e2e | +| **Files** | `internal/afv/bootstrap.go`, `mesh_config_test.go` (replace `TestNewDefault_ClusterEnabledFailsClosed`), `mesh_hybrid_e2e_test.go`, package comments in `mesh.go` / `mesh_frame.go`, delete/redefine `errClusterTCPNotBuilt` | +| **Depends on** | PR-10b.3 | +| **Description** | Valid `AFV_CLUSTER_*=true` **including VOICE_LISTEN** constructs HybridMesh via SetMesh; non-nil `Mesh()`; e2e two nodes A2A; wrong PSK; peer death; MemoryMesh A–F still pass; T13. | + +### PR-10b.5 — Operator wiki + ops checklist + +| | | +|--|--| +| **Title** | docs: multi-node AFV hybrid mesh ops | +| **Files** | Operator wiki Configuration + Deployment multi-node AFV; firewall/sticky-LB/MTU notes; mute-debug checklist | +| **Depends on** | PR-10b.4 (or draft after 10b.1; “landed” status after 10b.4) | +| **Description** | Ports, peer grammar, sticky LB, two-node env example, “open both TCP and UDP”, fragment/MTU note. Parent status “implemented” only after 10b.4 (pointer rows already fixed in 10b.1). | + +### Suggested merge order + +```text +10b.1 → 10b.2 → 10b.3 → 10b.4 → 10b.5 +``` + +## Implementation notes for engineers + +1. **Copy, don’t import** FSD dial/backoff/accept/replace-conn patterns from `internal/cluster/tcp_mesh.go`. +2. **Reuse** `dropOldestQueue`, `EncodeMeshFrame` / `DecodeMeshFrame`, `EncodeAudioRelay` / `DecodeAudioRelay`, `VerifyHelloPSK`. +3. **KD-16 / H-13:** registry locks released before mesh enqueue; HybridMesh unlocks peer maps before `onAudio` / I/O. +4. **Client UDP vs mesh UDP:** two sockets; never multiplex client CryptoDTO on mesh voice port (H-20 validates). +5. **`SetMesh` before `Run`:** production sets mesh in `NewDefault` (10b.4); tests may inject MemoryMesh. +6. **Interest dirty / publish loop** already in `server.go` — HybridMesh only transports frames; apply inbound Interest on TCP reader only. +7. **MemoryMesh Interest path is not the Hybrid path** — do not sync-apply on `PublishInterest` for HybridMesh (Issue 3). +8. When landing design: **`docs/design/afv-mesh-pr10b.md`** + parent transport row in **same PR as 10b.1**. + +--- + +## Acceptance checklist (product gate for PR-10b stack) + +- [x] `NewDefault` + valid `AFV_CLUSTER_*=true` (incl. `VOICE_LISTEN`) constructs production hybrid mesh and attaches via `SetMesh` +- [x] Two HybridMesh nodes on localhost A2A via TCP control + UDP voice +- [x] Wrong PSK / non-Hello first frame / peer death purge / Interest filter (no flood) +- [x] AudioRelay never sent or accepted on TCP (close on receive) +- [x] Normative peer parse fixtures green; H-20 config fail-closed +- [x] `maxMeshVoiceDatagram` oversize drop; strict one-frame-per-datagram +- [x] Unit coverage floors met for new modules; MemoryMesh A–F green +- [x] `go test -race ./internal/afv/...`; import graph; hygiene; `go build -o openfsd ./cmd/openfsd` +- [x] Operator wiki Configuration + Deployment multi-node AFV +- [x] Status + transport rows in `afv-mesh-pr10.md` / `afv-server.md` updated; design at `docs/design/afv-mesh-pr10b.md` +- [x] AEAD keys never leave home; Opus never in rqlite; no `internal/cluster` import +- [x] Sticky LB ops note present; silent-mute Debug signal present + +--- + +## Document control + +| | | +|--|--| +| **Draft location** | session temp path (this file) | +| **Final in-repo path** | `docs/design/afv-mesh-pr10b.md` | +| **Supersedes** | PR-10b TCP-only AudioRelay transport in `docs/design/afv-mesh-pr10.md` | +| **Does not supersede** | PR-10 MemoryMesh, framing, Interest, Server hooks | + +## Revision history + +| Rev | Date | Notes | +|-----|------|-------| +| 1 | 2026-07-28 | Initial hybrid PR-10b design | +| 2 | 2026-07-28 | Design review: normative peer parse, UDP allowlist, Interest apply, death/lastHB, maxMeshVoiceDatagram, MTU, H-20 config, inbound rate, PR plan sharpening (Issues 1–18) | +| 3 | 2026-07-28 | H-20 rule 7: drop port-only wildcard peer compare (multi-host same voice port OK); H-19 default **5000/s** with capacity note | diff --git a/docs/design/afv-server.md b/docs/design/afv-server.md index ca3d57a..2754f20 100644 --- a/docs/design/afv-server.md +++ b/docs/design/afv-server.md @@ -5,7 +5,7 @@ | **Document** | AFV (Audio for VATSIM) server — full feature design | | **Author** | _(design author / implementer)_ | | **Date** | 2026-07-28 | -| **Status** | **P0 Implemented** (single-node REST + UDP on `dev`; design history retained). Mesh: MemoryMesh path landed (PR-10); production TCP = PR-10b. | +| **Status** | **P0 Implemented** (single-node REST + UDP on `dev`; design history retained). Mesh: MemoryMesh path landed (PR-10); production hybrid mesh = PR-10b (`docs/design/afv-mesh-pr10b.md`). | | **Project** | openfsd | | **Target land path** | `docs/design/afv-server.md` | | **Related** | `Agents.md`, `docs/design/distributed-openfsd.md`, `internal/auth`, `internal/geo`, `internal/cluster`, `internal/db`, `cmd/openfsd/main.go`, [AFV-Native](https://github.com/xsquawkbox/AFV-Native) (BSD-3) | @@ -20,8 +20,8 @@ |-------|--------| | P0 single-node (`pkg/afvprotocol`, `internal/afv`, `-afv`) | **Landed** — REST auth/callsign/transceivers; UDP H/HA + AT→AR; range model; reaper; empty stations route | | PR-10 mesh (MemoryMesh + framing + Interest + AudioRelay) | **Landed** (in-process / test mesh; e2e Cases A–F) | -| PR-10b production TCP mesh | **Not landed** — `AFV_CLUSTER_ENABLED=true` fails closed without TCP in binary | -| Operator docs | Wiki Configuration/Deployment AFV sections; root README | +| PR-10b production hybrid mesh | **Landed** — TCP control + UDP AudioRelay (`HybridMesh`); see `docs/design/afv-mesh-pr10b.md` | +| Operator docs | Wiki Configuration/Deployment AFV sections (incl. multi-node hybrid mesh); root README | Design body below is the engineering record (decisions, protocol, PR plan). Prefer wiki + `internal/afv` for day-to-day ops. @@ -891,26 +891,27 @@ Why separate from `internal/cluster`: import allowlist, rate, interest model. **Goals:** in-process / localhost **memory mesh** or TCP between two nodes; directory sync; AT relay; e2e two clients on different nodes hear each other in range. -##### Normative mesh framing (KD-17) — first cut +##### Normative mesh framing (KD-17) — production transport (PR-10b) | Item | Spec | |------|------| -| Transport | TCP between static peers (`AFV_CLUSTER_PEERS=id=host:port,...`) | -| Frame | `[u32 BE length][u8 type][payload]` where length = 1+len(payload); max payload **1 MiB** | -| Auth | First message must be Hello with HMAC-SHA256(PSK, nodeID\|\|nonce) or simple constant-time compare of shared `AFV_CLUSTER_PSK` on Hello (document chosen scheme in PR); reject peers failing auth | -| Types | `1=Hello`, `2=Heartbeat`, `10=TrxSnapshot`, `11=TrxDelta`, `12=SessionLeave`, `20=AudioRelay`, `30=Interest` | -| Heartbeat | every 2s; peer-death after 15s → drop remote directory entries for that node; local sessions unaffected | +| **Control transport** | **TCP** between static peers (`AFV_CLUSTER_LISTEN`, `AFV_CLUSTER_PEERS=id=host:tcpPort[/voicePort],...`) | +| **AudioRelay (type 20)** | **UDP mesh voice only** (`AFV_CLUSTER_VOICE_LISTEN` + peer voice addrs); **never on TCP** | +| Frame | `[u32 BE length][u8 type][payload]` where length = 1+len(payload); TCP max payload **1 MiB**; UDP voice max **16 KiB** datagram | +| Auth | First **TCP** frame must be Hello; constant-time compare of shared `AFV_CLUSTER_PSK`; reject peers failing auth | +| Types | `1=Hello`, `2=Heartbeat`, `10=TrxSnapshot`, `11=TrxDelta`, `12=SessionLeave`, `20=AudioRelay` (**UDP only**), `30=Interest` | +| Heartbeat | every 2s on TCP; peer-death after 15s (or TCP read/write error) → drop remote directory entries for that node; local sessions unaffected | | Interest | each node advertises set of `(freqHz, cellKey)` for local RX; rate ≤ 2 Hz; cap **4096** entries (drop coarsest cells if over) | -| AudioRelay payload | msgpack/json binary: originNode, callsign, seq, audio, last, isXC, []{freq, lat, lon, alt, txID} — **never client AEAD keys** | -| Relay queues | per-peer outbound channel depth **256**; **drop oldest AudioRelay** under pressure; never block UDP hot path on full mesh queue | -| Deadlock | never hold registry lock while sending on mesh TCP | -| Max peers first cut | **≤ 4** (document; like FSD mesh caution) | -| Sticky LB | required: REST sticky by CID or per-node API only advertises local UDP | -| Dual-login | per-region uniqueness only in first cut; no cluster-wide claim | +| AudioRelay payload | originNode, callsign, seq, audio, last, isATC, isXC, []{freq, lat, lon, alt, txID} — **never client AEAD keys** | +| Relay queues | voice depth **256** drop-oldest (UDP); control depth **64** drop-oldest (TCP); never block client UDP hot path | +| Deadlock | never hold registry lock while sending on mesh TCP/UDP | +| Max peers | **≤ 4** remote | +| Sticky LB | required: REST sticky by CID or per-node API only advertises local client UDP (`AFV_UDP_ADVERTISE_IPV4`) | +| Dual-login | per-region uniqueness only; no cluster-wide claim | Keys never leave the home node. Local node re-runs range model for **local** RX only and encrypts AR with local keys. -**PR plan:** PR-10 is **framing + memory mesh two-node e2e only**. Production TCP polish / interest caps / ops docs may split to PR-10b after a short design addendum if scope grows. Full FSD-mesh-scale work is **out of band** relative to P0. +**Normative production design:** `docs/design/afv-mesh-pr10b.md` (hybrid TCP control + UDP voice). MemoryMesh remains tests-only. --- diff --git a/internal/afv/bootstrap.go b/internal/afv/bootstrap.go index 6c0cec5..304fcd6 100644 --- a/internal/afv/bootstrap.go +++ b/internal/afv/bootstrap.go @@ -77,13 +77,23 @@ func NewDefault(ctx context.Context) (*Server, error) { if err := cfg.ValidateCluster(); err != nil { return nil, err } - // M-11: ENABLED=true without TCP mesh in this binary fails closed. - // Never wire production NewDefault to MemoryMesh. + + s := New(cfg, repos.UserRepo, repos.ConfigRepo, jwtSecret) + + // Production hybrid mesh (PR-10b): TCP control + UDP voice. Never MemoryMesh. if cfg.ClusterEnabled { - return nil, errClusterTCPNotBuilt + hmCfg, err := hybridConfigFrom(cfg) + if err != nil { + return nil, fmt.Errorf("afv hybrid mesh: %w", err) + } + hm, err := NewHybridMesh(hmCfg) + if err != nil { + return nil, fmt.Errorf("afv hybrid mesh failed to construct: %w", err) + } + s.SetMesh(hm) } - return New(cfg, repos.UserRepo, repos.ConfigRepo, jwtSecret), nil + return s, nil } func resolveJWTSecret(ctx context.Context, cfg *Config, kv db.ConfigRepository) ([]byte, error) { diff --git a/internal/afv/config.go b/internal/afv/config.go index 6899a62..1357e01 100644 --- a/internal/afv/config.go +++ b/internal/afv/config.go @@ -37,6 +37,7 @@ type Config struct { ClusterEnabled bool `env:"AFV_CLUSTER_ENABLED, default=false"` ClusterNodeID string `env:"AFV_CLUSTER_NODE_ID"` ClusterListen string `env:"AFV_CLUSTER_LISTEN"` + ClusterVoiceListen string `env:"AFV_CLUSTER_VOICE_LISTEN"` ClusterPeers string `env:"AFV_CLUSTER_PEERS"` ClusterPSK string `env:"AFV_CLUSTER_PSK"` diff --git a/internal/afv/mesh.go b/internal/afv/mesh.go index 7db813a..a5a0fda 100644 --- a/internal/afv/mesh.go +++ b/internal/afv/mesh.go @@ -1,6 +1,6 @@ package afv -// AFV multi-node mesh (PR-10 / KD-17). +// AFV multi-node mesh (PR-10 / PR-10b / KD-17). // // Ops: sticky LB required — each node mints unique ChannelTag + AEAD keys and // advertises its own AFV_UDP_ADVERTISE_IPV4. Clients must use the same node for @@ -8,14 +8,15 @@ package afv // replace sticky affinity. AEAD keys never leave the home node; cross-node // audio is re-encrypted at the listener's home with local ClientRxKey only. // -// Production: AFV_CLUSTER_ENABLED=true without TCP mesh in this binary fails -// closed at startup (MemoryMesh is tests-only via SetMesh). +// Production: HybridMesh (TCP control + UDP voice) when AFV_CLUSTER_ENABLED=true +// with valid VOICE_LISTEN + peers. MemoryMesh is tests-only via SetMesh. import ( "context" "fmt" "log/slog" "net" + "strconv" "strings" "time" @@ -24,7 +25,7 @@ import ( ) // Mesh is the AFV inter-node fabric. Implementations: MemoryMesh (tests); -// TCPMesh is PR-10b / optional. +// HybridMesh (production: TCP control + UDP AudioRelay). // // All publish methods must be non-blocking w.r.t. the UDP hot path (enqueue or // drop). Callers must not hold registry locks across Publish*/Enqueue*. @@ -103,7 +104,7 @@ type RelayTxRadio struct { HeightM float64 } -// MeshConfig is shared construction config for MemoryMesh / future TCPMesh. +// MeshConfig is shared construction config for MemoryMesh. type MeshConfig struct { NodeID string PSK string @@ -115,17 +116,19 @@ type MeshConfig struct { const ( interestMinInterval = 500 * time.Millisecond // ≤ 2 Hz defaultPeerDeathGrace = 15 * time.Second + defaultMeshHeartbeat = 2 * time.Second ) // interestMaxEntries is the Interest set cap (M-5). Var so tests can lower it. var interestMaxEntries = 4096 -// --- Cluster config validation (M-11) --- +// --- Cluster config validation (M-11 / H-20) --- -// ClusterPeer is one static remote peer (id + host:port). +// ClusterPeer is one static remote peer (TCP control + UDP voice). type ClusterPeer struct { - ID string - Addr string + ID string + Addr string // TCP host:port (control) — SplitHostPort-valid + VoiceAddr string // UDP host:port (voice) — derived or explicit } // ValidateCluster fails closed when cluster is enabled with incomplete config. @@ -140,9 +143,32 @@ func (c *Config) ValidateCluster() error { if listen == "" { return fmt.Errorf("AFV_CLUSTER_LISTEN required when AFV_CLUSTER_ENABLED=true") } - if _, _, err := net.SplitHostPort(listen); err != nil { + tcpHost, tcpPortStr, err := net.SplitHostPort(listen) + if err != nil { return fmt.Errorf("AFV_CLUSTER_LISTEN: want host:port: %w", err) } + tcpPort, err := parseUint16Port(tcpPortStr) + if err != nil || tcpPort == 0 { + return fmt.Errorf("AFV_CLUSTER_LISTEN: invalid port") + } + _ = tcpHost + + voiceListen := strings.TrimSpace(c.ClusterVoiceListen) + if voiceListen == "" { + return fmt.Errorf("AFV_CLUSTER_VOICE_LISTEN required when AFV_CLUSTER_ENABLED=true") + } + voiceHost, voicePortStr, err := net.SplitHostPort(voiceListen) + if err != nil { + return fmt.Errorf("AFV_CLUSTER_VOICE_LISTEN: want host:port: %w", err) + } + voicePort, err := parseUint16Port(voicePortStr) + if err != nil || voicePort == 0 { + return fmt.Errorf("AFV_CLUSTER_VOICE_LISTEN: invalid port (port 0 rejected)") + } + if voicePortStr == tcpPortStr { + return fmt.Errorf("AFV_CLUSTER_VOICE_LISTEN: must differ from AFV_CLUSTER_LISTEN port") + } + if strings.TrimSpace(c.ClusterPSK) == "" { return fmt.Errorf("AFV_CLUSTER_PSK required when AFV_CLUSTER_ENABLED=true") } @@ -157,7 +183,11 @@ func (c *Config) ValidateCluster() error { return fmt.Errorf("AFV_CLUSTER_PEERS: max 4 remote peers, got %d", len(peers)) } self := strings.TrimSpace(c.ClusterNodeID) - seen := make(map[string]struct{}, len(peers)) + seenID := make(map[string]struct{}, len(peers)) + seenVoice := make(map[string]struct{}, len(peers)) + localVoiceNorm := net.JoinHostPort(voiceHost, voicePortStr) + localTCPNorm := net.JoinHostPort(tcpHost, tcpPortStr) + for _, p := range peers { if p.ID == self { return fmt.Errorf("AFV_CLUSTER_PEERS: must not include self node id %q", self) @@ -168,15 +198,97 @@ func (c *Config) ValidateCluster() error { if _, _, err := net.SplitHostPort(p.Addr); err != nil { return fmt.Errorf("AFV_CLUSTER_PEERS: peer %q addr want host:port: %w", p.ID, err) } - if _, ok := seen[p.ID]; ok { + if _, ok := seenID[p.ID]; ok { return fmt.Errorf("AFV_CLUSTER_PEERS: duplicate peer id %q", p.ID) } - seen[p.ID] = struct{}{} + seenID[p.ID] = struct{}{} + if _, ok := seenVoice[p.VoiceAddr]; ok { + return fmt.Errorf("AFV_CLUSTER_PEERS: duplicate peer VoiceAddr %q", p.VoiceAddr) + } + seenVoice[p.VoiceAddr] = struct{}{} + // H-20: full host:port equality only for local voice vs peer voice. + if p.VoiceAddr == localVoiceNorm { + return fmt.Errorf("AFV_CLUSTER_VOICE_LISTEN equals peer %q VoiceAddr %q", p.ID, p.VoiceAddr) + } + // Optional same-host loopback tightening. + if sameLoopbackOrWildcardVoiceCollision(voiceHost, voicePortStr, p.VoiceAddr) { + return fmt.Errorf("AFV_CLUSTER_VOICE_LISTEN collides with peer %q loopback VoiceAddr %q", p.ID, p.VoiceAddr) + } + if p.Addr == localTCPNorm { + return fmt.Errorf("AFV_CLUSTER_LISTEN equals peer %q Addr %q", p.ID, p.Addr) + } + } + + // H-20: ClusterVoiceListen ≠ UDPListen (client CryptoDTO socket). + if err := rejectSameSocket(c.ClusterVoiceListen, c.UDPListen, "AFV_CLUSTER_VOICE_LISTEN", "AFV_UDP_LISTEN"); err != nil { + return err } return nil } -// parseClusterPeers parses "id=host:port,id2=host:port2". +// sameLoopbackOrWildcardVoiceCollision catches localhost multi-process when local +// binds loopback/wildcard and peer is loopback with the same voice port. +func sameLoopbackOrWildcardVoiceCollision(localHost, localPort, peerVoice string) bool { + ph, pp, err := net.SplitHostPort(peerVoice) + if err != nil || pp != localPort { + return false + } + if !isLoopbackHost(ph) { + return false + } + return isLoopbackHost(localHost) || isWildcardHost(localHost) +} + +func isLoopbackHost(h string) bool { + h = strings.TrimSpace(h) + if h == "localhost" || h == "127.0.0.1" || h == "::1" { + return true + } + ip := net.ParseIP(h) + return ip != nil && ip.IsLoopback() +} + +func isWildcardHost(h string) bool { + h = strings.TrimSpace(h) + return h == "" || h == "0.0.0.0" || h == "::" || h == "*" +} + +// rejectSameSocket fails if two listen specs collide (wildcard-equal hosts if ports equal). +func rejectSameSocket(a, b, nameA, nameB string) error { + a = strings.TrimSpace(a) + b = strings.TrimSpace(b) + if a == "" || b == "" { + return nil + } + ah, ap, errA := net.SplitHostPort(a) + bh, bp, errB := net.SplitHostPort(b) + if errA != nil || errB != nil { + return nil // other validators handle format + } + if ap != bp { + return nil + } + if ah == bh || isWildcardHost(ah) || isWildcardHost(bh) { + return fmt.Errorf("%s must differ from %s (same host:port / wildcard port collision)", nameA, nameB) + } + // concrete equal already covered by ah==bh + return nil +} + +func parseUint16Port(s string) (uint16, error) { + s = strings.TrimSpace(s) + if s == "" { + return 0, fmt.Errorf("empty port") + } + n, err := strconv.ParseUint(s, 10, 16) + if err != nil { + return 0, err + } + return uint16(n), nil +} + +// parseClusterPeers parses AFV_CLUSTER_PEERS with optional /voicePort (H-4). +// Grammar: id=host:tcpPort[/voicePort],... — strip /voice before SplitHostPort. func parseClusterPeers(s string) ([]ClusterPeer, error) { s = strings.TrimSpace(s) if s == "" { @@ -194,19 +306,65 @@ func parseClusterPeers(s string) ([]ClusterPeer, error) { return nil, fmt.Errorf("AFV_CLUSTER_PEERS: invalid entry %q (want id=host:port)", p) } id := strings.TrimSpace(p[:eq]) - addr := strings.TrimSpace(p[eq+1:]) - if id == "" || addr == "" { + rest := strings.TrimSpace(p[eq+1:]) + if id == "" || rest == "" { return nil, fmt.Errorf("AFV_CLUSTER_PEERS: invalid entry %q", p) } - out = append(out, ClusterPeer{ID: id, Addr: addr}) + + hostPort := rest + var voicePort uint16 + voiceSet := false + if slash := strings.LastIndexByte(rest, '/'); slash >= 0 { + // Extra '/' in the hostPort side is rejected by SplitHostPort or empty checks. + if strings.Count(rest, "/") > 1 { + return nil, fmt.Errorf("AFV_CLUSTER_PEERS: invalid entry %q (extra /)", p) + } + hostPort = strings.TrimSpace(rest[:slash]) + voicePortStr := strings.TrimSpace(rest[slash+1:]) + if hostPort == "" || voicePortStr == "" { + return nil, fmt.Errorf("AFV_CLUSTER_PEERS: empty voice port in %q", p) + } + vp, err := parseUint16Port(voicePortStr) + if err != nil { + return nil, fmt.Errorf("AFV_CLUSTER_PEERS: non-numeric voice port in %q", p) + } + if vp == 0 { + return nil, fmt.Errorf("AFV_CLUSTER_PEERS: port 0 rejected in %q", p) + } + voicePort = vp + voiceSet = true + } + + host, tcpPortStr, err := net.SplitHostPort(hostPort) + if err != nil { + return nil, fmt.Errorf("AFV_CLUSTER_PEERS: peer %q addr want host:port: %w", id, err) + } + if strings.TrimSpace(host) == "" { + return nil, fmt.Errorf("AFV_CLUSTER_PEERS: peer %q empty host", id) + } + tcpPort, err := parseUint16Port(tcpPortStr) + if err != nil { + return nil, fmt.Errorf("AFV_CLUSTER_PEERS: peer %q non-numeric tcp port", id) + } + if tcpPort == 0 { + return nil, fmt.Errorf("AFV_CLUSTER_PEERS: port 0 rejected for peer %q", id) + } + if !voiceSet { + if tcpPort == 65535 { + return nil, fmt.Errorf("AFV_CLUSTER_PEERS: voice port overflow for peer %q", id) + } + voicePort = tcpPort + 1 + } + if voicePort == 0 { + return nil, fmt.Errorf("AFV_CLUSTER_PEERS: port 0 rejected for peer %q voice", id) + } + voiceAddr := net.JoinHostPort(host, strconv.Itoa(int(voicePort))) + // Preserve SplitHostPort-valid form of hostPort (brackets for IPv6). + out = append(out, ClusterPeer{ID: id, Addr: hostPort, VoiceAddr: voiceAddr}) } return out, nil } -// errClusterTCPNotBuilt is returned when ENABLED=true but TCP mesh is not available. -var errClusterTCPNotBuilt = fmt.Errorf( - "AFV_CLUSTER_ENABLED=true but AFV mesh TCP is not built in this binary (use tests with SetMesh(MemoryMesh) or enable PR-10b TCP)") - // --- Server mesh hooks --- // SetMesh injects a Mesh (tests: MemoryMesh). Must be called before Run. diff --git a/internal/afv/mesh_config_test.go b/internal/afv/mesh_config_test.go index 6239987..facf9ce 100644 --- a/internal/afv/mesh_config_test.go +++ b/internal/afv/mesh_config_test.go @@ -13,11 +13,13 @@ func TestValidateCluster(t *testing.T) { t.Fatal(err) } base := &Config{ - ClusterEnabled: true, - ClusterNodeID: "n1", - ClusterListen: "127.0.0.1:9100", - ClusterPeers: "n2=127.0.0.1:9101", - ClusterPSK: "psk", + ClusterEnabled: true, + ClusterNodeID: "n1", + ClusterListen: "127.0.0.1:9100", + ClusterVoiceListen: "127.0.0.1:9101", + ClusterPeers: "n2=127.0.0.1:9110/9111", + ClusterPSK: "psk", + UDPListen: "127.0.0.1:50000", } if err := base.ValidateCluster(); err != nil { t.Fatal(err) @@ -29,16 +31,25 @@ func TestValidateCluster(t *testing.T) { }{ {"no node", func(c *Config) { c.ClusterNodeID = "" }, "NODE_ID"}, {"no listen", func(c *Config) { c.ClusterListen = "" }, "LISTEN"}, + {"no voice", func(c *Config) { c.ClusterVoiceListen = "" }, "VOICE_LISTEN"}, {"no psk", func(c *Config) { c.ClusterPSK = "" }, "PSK"}, {"no peers", func(c *Config) { c.ClusterPeers = "" }, "PEERS"}, - {"self peer", func(c *Config) { c.ClusterPeers = "n1=127.0.0.1:1" }, "self"}, + {"self peer", func(c *Config) { c.ClusterPeers = "n1=127.0.0.1:9110/9111" }, "self"}, {"too many", func(c *Config) { - c.ClusterPeers = "a=1:1,b=1:2,c=1:3,d=1:4,e=1:5" + c.ClusterPeers = "a=1.1.1.1:1/2,b=1.1.1.1:3/4,c=1.1.1.1:5/6,d=1.1.1.1:7/8,e=1.1.1.1:9/10" }, "max 4"}, {"bad peer", func(c *Config) { c.ClusterPeers = "notvalid" }, "invalid"}, - {"dup", func(c *Config) { c.ClusterPeers = "n2=1:1,n2=1:2" }, "duplicate"}, + {"dup", func(c *Config) { c.ClusterPeers = "n2=1.1.1.1:1/2,n2=1.1.1.1:3/4" }, "duplicate"}, {"bad listen", func(c *Config) { c.ClusterListen = "not-a-hostport" }, "host:port"}, {"bad peer addr", func(c *Config) { c.ClusterPeers = "n2=nohostport" }, "host:port"}, + {"voice same port as tcp", func(c *Config) { c.ClusterVoiceListen = "127.0.0.1:9100" }, "differ"}, + {"voice equals peer", func(c *Config) { + c.ClusterVoiceListen = "127.0.0.1:9111" + }, "VoiceAddr"}, + {"voice equals udp", func(c *Config) { + c.ClusterVoiceListen = "127.0.0.1:50000" + }, "UDP_LISTEN"}, + {"voice port 0", func(c *Config) { c.ClusterVoiceListen = "127.0.0.1:0" }, "port"}, } for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { @@ -52,21 +63,107 @@ func TestValidateCluster(t *testing.T) { } } -func TestNewDefault_ClusterEnabledFailsClosed(t *testing.T) { +func TestValidateCluster_MultiHostSameVoicePortOK(t *testing.T) { + // H-20 rev3: local 0.0.0.0:17001 + peer 10.0.0.2:17001 is OK + c := &Config{ + ClusterEnabled: true, + ClusterNodeID: "n1", + ClusterListen: "0.0.0.0:17000", + ClusterVoiceListen: "0.0.0.0:17001", + ClusterPeers: "n2=10.0.0.2:17000", // voice derives 10.0.0.2:17001 + ClusterPSK: "psk", + UDPListen: "0.0.0.0:50000", + } + if err := c.ValidateCluster(); err != nil { + t.Fatal(err) + } +} + +func TestValidateCluster_DupVoiceAddr(t *testing.T) { + c := &Config{ + ClusterEnabled: true, + ClusterNodeID: "n1", + ClusterListen: "0.0.0.0:17000", + ClusterVoiceListen: "0.0.0.0:17001", + // Same VoiceAddr host:port on two peers (H-20 unique VoiceAddr) + ClusterPeers: "n2=10.0.0.2:17000/17001,n3=10.0.0.2:17010/17001", + ClusterPSK: "psk", + UDPListen: "0.0.0.0:50000", + } + err := c.ValidateCluster() + if err == nil || !strings.Contains(err.Error(), "duplicate peer VoiceAddr") { + t.Fatalf("err=%v", err) + } +} + +func TestParseClusterPeers(t *testing.T) { + fixtures := []struct { + in string + wantID string + wantAddr string + wantVoice string + wantErr string + }{ + {"n2=10.0.0.2:17000", "n2", "10.0.0.2:17000", "10.0.0.2:17001", ""}, + {"n2=10.0.0.2:17000/17100", "n2", "10.0.0.2:17000", "10.0.0.2:17100", ""}, + {"n2=[::1]:17000", "n2", "[::1]:17000", "[::1]:17001", ""}, + {"n2=[::1]:17000/17050", "n2", "[::1]:17000", "[::1]:17050", ""}, + {" n2 = 10.0.0.2:17000/ 17100 ", "n2", "10.0.0.2:17000", "10.0.0.2:17100", ""}, + {"n2=10.0.0.2:65535", "", "", "", "overflow"}, + {"n2=10.0.0.2:17000/", "", "", "", "empty voice"}, + {"n2=10.0.0.2:17000/abc", "", "", "", "non-numeric"}, + {"n2=10.0.0.2:17000/0", "", "", "", "port 0"}, + {"n2=10.0.0.2:0", "", "", "", "port 0"}, + {"n2=::1:17000", "", "", "", "host:port"}, + } + for _, tc := range fixtures { + t.Run(tc.in, func(t *testing.T) { + p, err := parseClusterPeers(tc.in) + if tc.wantErr != "" { + if err == nil || !strings.Contains(err.Error(), tc.wantErr) { + t.Fatalf("err=%v want %q", err, tc.wantErr) + } + return + } + if err != nil || len(p) != 1 { + t.Fatalf("p=%+v err=%v", p, err) + } + if p[0].ID != tc.wantID || p[0].Addr != tc.wantAddr || p[0].VoiceAddr != tc.wantVoice { + t.Fatalf("got %+v want id=%s addr=%s voice=%s", p[0], tc.wantID, tc.wantAddr, tc.wantVoice) + } + }) + } + + // multi + p, err := parseClusterPeers("a=1.2.3.4:5, b=6.7.8.9:10") + if err != nil || len(p) != 2 || p[0].ID != "a" || p[0].VoiceAddr != "1.2.3.4:6" { + t.Fatalf("%+v %v", p, err) + } +} + +func TestNewDefault_ClusterEnabledWiresHybrid(t *testing.T) { dir := t.TempDir() t.Setenv("DATABASE_DRIVER", "sqlite") t.Setenv("DATABASE_SOURCE_NAME", filepath.Join(dir, "x.db")) t.Setenv("DATABASE_AUTO_MIGRATE", "true") t.Setenv("AFV_UDP_ADVERTISE_IPV4", "127.0.0.1:50000") + t.Setenv("AFV_UDP_LISTEN", "127.0.0.1:50000") t.Setenv("AFV_JWT_SECRET", "bootstrap-test-secret-key-material") t.Setenv("AFV_CLUSTER_ENABLED", "true") t.Setenv("AFV_CLUSTER_NODE_ID", "n1") - t.Setenv("AFV_CLUSTER_LISTEN", "127.0.0.1:9100") - t.Setenv("AFV_CLUSTER_PEERS", "n2=127.0.0.1:9101") + t.Setenv("AFV_CLUSTER_LISTEN", "127.0.0.1:19100") + t.Setenv("AFV_CLUSTER_VOICE_LISTEN", "127.0.0.1:19101") + t.Setenv("AFV_CLUSTER_PEERS", "n2=127.0.0.1:19110/19111") t.Setenv("AFV_CLUSTER_PSK", "psk") - _, err := NewDefault(context.Background()) - if err == nil || !strings.Contains(err.Error(), "TCP") { - t.Fatalf("err=%v", err) + s, err := NewDefault(context.Background()) + if err != nil { + t.Fatal(err) + } + if s.Mesh() == nil { + t.Fatal("expected non-nil Mesh") + } + if _, ok := s.Mesh().(*HybridMesh); !ok { + t.Fatalf("want *HybridMesh got %T", s.Mesh()) } } @@ -85,9 +182,19 @@ func TestNewDefault_ClusterIncompleteConfig(t *testing.T) { } } -func TestParseClusterPeers(t *testing.T) { - p, err := parseClusterPeers("a=1.2.3.4:5, b=6.7.8.9:10") - if err != nil || len(p) != 2 || p[0].ID != "a" { - t.Fatalf("%+v %v", p, err) +func TestNewDefault_ClusterDisabledNilMesh(t *testing.T) { + dir := t.TempDir() + t.Setenv("DATABASE_DRIVER", "sqlite") + t.Setenv("DATABASE_SOURCE_NAME", filepath.Join(dir, "z.db")) + t.Setenv("DATABASE_AUTO_MIGRATE", "true") + t.Setenv("AFV_UDP_ADVERTISE_IPV4", "127.0.0.1:50000") + t.Setenv("AFV_JWT_SECRET", "bootstrap-test-secret-key-material") + t.Setenv("AFV_CLUSTER_ENABLED", "false") + s, err := NewDefault(context.Background()) + if err != nil { + t.Fatal(err) + } + if s.Mesh() != nil { + t.Fatal("expected nil mesh when disabled") } } diff --git a/internal/afv/mesh_frame.go b/internal/afv/mesh_frame.go index df96551..47aa276 100644 --- a/internal/afv/mesh_frame.go +++ b/internal/afv/mesh_frame.go @@ -1,16 +1,15 @@ package afv -// AFV multi-node mesh framing (PR-10 / KD-17). +// AFV multi-node mesh framing (PR-10 / PR-10b / KD-17). // // Hello auth (M-3): constant-time compare of shared PSK. Trust root and payload // shape match FSD mesh Hello (nodeID + PSK strings); comparison is CT-upgraded // vs FSD's non-constant-time !=. Do not log PSK contents on failure. // -// Hello first-frame rule (TCP only, PR-10b / M-11): on a TCP mesh connection the -// first frame must be type Hello; any other type closes the conn. MemoryMesh has -// no wire Hello — auth is constructor PSK verify on Start. When mesh_tcp.go -// lands, enforce first-frame type == MeshTypeHello on accept/dial before any -// Snapshot/Interest/AudioRelay. +// Hello first-frame rule (TCP control plane, HybridMesh): on a TCP mesh +// connection the first frame must be type Hello; any other type closes the conn. +// MemoryMesh has no wire Hello — auth is constructor PSK verify on Start. +// AudioRelay (type 20) is forbidden on TCP (H-2); production voice is UDP only. // // Local length-prefix framing — do not import internal/cluster. @@ -103,6 +102,46 @@ func DecodeMeshFrame(r io.Reader) (MeshFrame, error) { return MeshFrame{Type: typ, Payload: payload}, nil } +// EncodeMeshFrameBytes returns the full length-prefixed mesh frame as a byte slice. +func EncodeMeshFrameBytes(typ byte, payload []byte) ([]byte, error) { + if len(payload) > MaxMeshPayload { + return nil, errMeshFrameTooLarge + } + n := uint32(1 + len(payload)) + out := make([]byte, 5+len(payload)) + binary.BigEndian.PutUint32(out[0:4], n) + out[4] = typ + copy(out[5:], payload) + return out, nil +} + +// DecodeMeshFrameExact decodes exactly one frame from b with zero trailing bytes +// (H-3: decoded frame length == len(b)). Used for UDP voice datagrams. +func DecodeMeshFrameExact(b []byte) (MeshFrame, error) { + if len(b) < 5 { + return MeshFrame{}, errMeshShortFrame + } + n := binary.BigEndian.Uint32(b[0:4]) + if n == 0 { + return MeshFrame{}, errMeshShortFrame + } + if n > maxMeshFrameTotal { + return MeshFrame{}, errMeshFrameTooLarge + } + // 4-byte length prefix + n body bytes must consume the whole datagram. + if 4+int(n) != len(b) { + return MeshFrame{}, errMeshBadPayload + } + typ := b[4] + payLen := int(n) - 1 + var payload []byte + if payLen > 0 { + payload = make([]byte, payLen) + copy(payload, b[5:5+payLen]) + } + return MeshFrame{Type: typ, Payload: payload}, nil +} + // --- binary helpers (local copies; no cluster import) --- func meshEncodeString(b []byte, s string) []byte { diff --git a/internal/afv/mesh_hybrid.go b/internal/afv/mesh_hybrid.go new file mode 100644 index 0000000..ec2cbcb --- /dev/null +++ b/internal/afv/mesh_hybrid.go @@ -0,0 +1,752 @@ +package afv + +// HybridMesh is the production AFV multi-node fabric (PR-10b): +// TCP control plane (Hello, HB, Trx*, Leave, Interest) + UDP voice (AudioRelay only). +// MemoryMesh remains tests-only. +// +// Lifecycle: single Start/Stop per instance (not restartable after Stop). + +import ( + "context" + "fmt" + "log/slog" + "net" + "strings" + "sync" + "sync/atomic" + "time" + + "github.com/renorris/openfsd/internal/geo" +) + +// HybridMeshConfig configures production hybrid mesh. +type HybridMeshConfig struct { + NodeID string + ListenTCP string // AFV_CLUSTER_LISTEN + ListenVoice string // AFV_CLUSTER_VOICE_LISTEN + Peers []ClusterPeer + PSK string + + HeartbeatInterval time.Duration // default 2s + PeerDeathAfter time.Duration // default 15s + CtrlQueueDepth int // default 64 + VoiceQueueDepth int // default 256 + Logger *slog.Logger +} + +// HybridMesh implements Mesh over TCP control + UDP AudioRelay voice. +type HybridMesh struct { + cfg HybridMeshConfig + peerDeathGrace time.Duration + hbInterval time.Duration + ctrlDepth int + voiceDepth int + logger *slog.Logger + + // peerID → static config + peerCfg map[string]ClusterPeer + + mu sync.RWMutex + // peerInterest filled only from inbound TCP Interest (H-14) + peerInterest map[string]map[FreqCell]struct{} + peers map[string]*hybridPeer + + // UDP allowlist: canonical key → peerID + allowlist map[string]string + // resolved send targets + sendVoice map[string]*net.UDPAddr + + ln net.Listener + voicePC net.PacketConn + + onAudio func(fromNode string, r AudioRelay) + onPeerDead func(nodeID string) + onDir MeshDirectoryHandler + snapFn func() []MeshSessionBlock + interestFn func() []InterestEntry + + // counters + voiceTx atomic.Uint64 + voiceRx atomic.Uint64 + udpAllowDrops atomic.Uint64 + udpOversizeDrops atomic.Uint64 + inboundRateDrops atomic.Uint64 + voiceQueueDrops atomic.Uint64 + ctrlQueueDrops atomic.Uint64 + udpSendErrs atomic.Uint64 + lastMuteCheck atomic.Int64 // unix for silent-mute debug + + stopCh chan struct{} + stopOnce sync.Once + wg sync.WaitGroup + started atomic.Bool +} + +// hybridPeer is per-remote-peer control + voice state. +type hybridPeer struct { + id string + tcpAddr string + voiceAddr string + + mu sync.Mutex + conn net.Conn + ctrlQ *dropOldestQueue[meshCtrlJob] + voiceQ *dropOldestQueue[voiceJob] + lastHB time.Time + authed bool + stop chan struct{} // closed to stop writer/drainer for this conn gen + + // inbound rate (H-19) + rateMu sync.Mutex + rateWindow time.Time + rateCount int + voiceTxCount atomic.Uint64 + voiceRxCount atomic.Uint64 + // lastVoiceUnixNano: last successful voice tx or rx (mute watch Issue 15) + lastVoiceUnixNano atomic.Int64 +} + +type voiceJob struct { + packet []byte // exclusive copy of full mesh frame +} + +// NewHybridMesh constructs a HybridMesh (not started). +func NewHybridMesh(cfg HybridMeshConfig) (*HybridMesh, error) { + if strings.TrimSpace(cfg.NodeID) == "" { + return nil, fmt.Errorf("afv hybrid mesh: empty NodeID") + } + if strings.TrimSpace(cfg.ListenTCP) == "" || strings.TrimSpace(cfg.ListenVoice) == "" { + return nil, fmt.Errorf("afv hybrid mesh: ListenTCP and ListenVoice required") + } + if strings.TrimSpace(cfg.PSK) == "" { + return nil, fmt.Errorf("afv hybrid mesh: PSK required") + } + if len(cfg.Peers) == 0 { + return nil, fmt.Errorf("afv hybrid mesh: at least one peer required") + } + if len(cfg.Peers) > 4 { + return nil, fmt.Errorf("afv hybrid mesh: max 4 peers") + } + pg := cfg.PeerDeathAfter + if pg <= 0 { + pg = defaultPeerDeathGrace + } + hb := cfg.HeartbeatInterval + if hb <= 0 { + hb = defaultMeshHeartbeat + } + cd := cfg.CtrlQueueDepth + if cd <= 0 { + cd = meshControlQueueDepth + } + vd := cfg.VoiceQueueDepth + if vd <= 0 { + vd = meshVoiceQueueDepth + } + log := cfg.Logger + if log == nil { + log = slog.Default() + } + peerCfg := make(map[string]ClusterPeer, len(cfg.Peers)) + for _, p := range cfg.Peers { + if p.ID == "" || p.ID == cfg.NodeID { + return nil, fmt.Errorf("afv hybrid mesh: invalid peer id %q", p.ID) + } + if _, ok := peerCfg[p.ID]; ok { + return nil, fmt.Errorf("afv hybrid mesh: duplicate peer %q", p.ID) + } + peerCfg[p.ID] = p + } + return &HybridMesh{ + cfg: cfg, + peerDeathGrace: pg, + hbInterval: hb, + ctrlDepth: cd, + voiceDepth: vd, + logger: log, + peerCfg: peerCfg, + peerInterest: make(map[string]map[FreqCell]struct{}), + peers: make(map[string]*hybridPeer), + allowlist: make(map[string]string), + sendVoice: make(map[string]*net.UDPAddr), + stopCh: make(chan struct{}), + }, nil +} + +// hybridConfigFrom builds HybridMeshConfig from AFV Config (bootstrap). +func hybridConfigFrom(cfg *Config) (HybridMeshConfig, error) { + peers, err := parseClusterPeers(cfg.ClusterPeers) + if err != nil { + return HybridMeshConfig{}, err + } + return HybridMeshConfig{ + NodeID: cfg.ClusterNodeID, + ListenTCP: cfg.ClusterListen, + ListenVoice: cfg.ClusterVoiceListen, + Peers: peers, + PSK: cfg.ClusterPSK, + }, nil +} + +func (m *HybridMesh) NodeID() string { + if m == nil { + return "" + } + return m.cfg.NodeID +} + +func (m *HybridMesh) OnAudioRelay(fn func(fromNode string, r AudioRelay)) { + m.mu.Lock() + m.onAudio = fn + m.mu.Unlock() +} + +func (m *HybridMesh) OnPeerDead(fn func(nodeID string)) { + m.mu.Lock() + m.onPeerDead = fn + m.mu.Unlock() +} + +func (m *HybridMesh) OnDirectory(fn MeshDirectoryHandler) { + m.mu.Lock() + m.onDir = fn + m.mu.Unlock() +} + +func (m *HybridMesh) SetSnapshotProvider(fn func() []MeshSessionBlock) { + m.mu.Lock() + m.snapFn = fn + m.mu.Unlock() +} + +func (m *HybridMesh) SetInterestProvider(fn func() []InterestEntry) { + m.mu.Lock() + m.interestFn = fn + m.mu.Unlock() +} + +// PeerWants reports whether peer advertised Interest for freq/cell (inbound TCP only). +func (m *HybridMesh) PeerWants(peerID string, freqHz uint32, cell geo.CellKey) bool { + if m == nil { + return false + } + m.mu.RLock() + defer m.mu.RUnlock() + set := m.peerInterest[peerID] + if len(set) == 0 { + return false + } + _, ok := set[FreqCell{FreqHz: freqHz, Cell: cell}] + return ok +} + +// InterestedPeers returns peer IDs that want any of the given keys (authed only). +func (m *HybridMesh) InterestedPeers(keys []FreqCell) []string { + if m == nil { + return nil + } + // Snapshot under m.mu only — never nest p.mu. + type snap struct { + id string + p *hybridPeer + set map[FreqCell]struct{} + } + m.mu.RLock() + snaps := make([]snap, 0, len(m.peerInterest)) + for pid, set := range m.peerInterest { + if len(set) == 0 { + continue + } + p := m.peers[pid] + if p == nil { + continue + } + // shallow ref to set (read-only after unlock for membership checks) + snaps = append(snaps, snap{id: pid, p: p, set: set}) + } + m.mu.RUnlock() + + var out []string + for _, s := range snaps { + s.p.mu.Lock() + authed := s.p.authed + s.p.mu.Unlock() + if !authed { + continue + } + for _, k := range keys { + if _, ok := s.set[k]; ok { + out = append(out, s.id) + break + } + } + } + return out +} + +// Start binds TCP+UDP, resolves allowlist, accept/dial, heartbeat. +// Single Start/Stop lifecycle — not safe to Start again after Stop. +func (m *HybridMesh) Start(ctx context.Context) error { + if m == nil { + return fmt.Errorf("afv hybrid mesh: nil") + } + if !m.started.CompareAndSwap(false, true) { + return nil // already started (or concurrent Start lost the race after first) + } + if err := m.resolveVoiceAllowlist(); err != nil { + m.started.Store(false) + return err + } + + ln, err := net.Listen("tcp", m.cfg.ListenTCP) + if err != nil { + m.started.Store(false) + return fmt.Errorf("afv hybrid mesh tcp listen: %w", err) + } + m.ln = ln + m.logger.Info("AFV hybrid mesh TCP control listening", "addr", ln.Addr().String()) + + pc, err := net.ListenPacket("udp", m.cfg.ListenVoice) + if err != nil { + _ = ln.Close() + m.started.Store(false) + return fmt.Errorf("afv hybrid mesh udp voice listen: %w", err) + } + m.voicePC = pc + m.logger.Info("AFV hybrid mesh UDP voice listening", "addr", pc.LocalAddr().String()) + + // Pre-create peer shells (queues re-init on Hello) + for id, p := range m.peerCfg { + m.mu.Lock() + m.peers[id] = &hybridPeer{ + id: id, + tcpAddr: p.Addr, + voiceAddr: p.VoiceAddr, + ctrlQ: newDropOldestQueue[meshCtrlJob](m.ctrlDepth), + voiceQ: newDropOldestQueue[voiceJob](m.voiceDepth), + stop: make(chan struct{}), + } + m.peerInterest[id] = make(map[FreqCell]struct{}) + m.mu.Unlock() + } + + m.wg.Add(1) + go func() { + defer m.wg.Done() + m.acceptLoop(ctx) + }() + + for id, p := range m.peerCfg { + if id <= m.cfg.NodeID { + continue // dial only if peer ID > self (H-9) + } + peer := p + m.wg.Add(1) + go func() { + defer m.wg.Done() + m.dialLoop(ctx, peer) + }() + } + + m.wg.Add(1) + go func() { + defer m.wg.Done() + m.heartbeatLoop(ctx) + }() + + m.wg.Add(1) + go func() { + defer m.wg.Done() + m.voiceReadLoop(ctx) + }() + + // Per-peer voice drainers + for id := range m.peerCfg { + pid := id + m.wg.Add(1) + go func() { + defer m.wg.Done() + m.voiceDrainLoop(ctx, pid) + }() + } + + m.wg.Add(1) + go func() { + defer m.wg.Done() + m.muteWatchLoop(ctx) + }() + + return nil +} + +// Stop closes listeners/conns and waits for workers. +func (m *HybridMesh) Stop() error { + if m == nil { + return nil + } + m.stopOnce.Do(func() { + close(m.stopCh) + if m.ln != nil { + _ = m.ln.Close() + } + if m.voicePC != nil { + _ = m.voicePC.Close() + } + m.mu.Lock() + for _, p := range m.peers { + p.mu.Lock() + if p.conn != nil { + _ = p.conn.Close() + } + select { + case <-p.stop: + default: + close(p.stop) + } + p.authed = false + p.mu.Unlock() + } + m.mu.Unlock() + m.started.Store(false) + }) + m.wg.Wait() + return nil +} + +// PublishTrxSnapshot encodes and enqueues control to authed peers. +func (m *HybridMesh) PublishTrxSnapshot() { + if m == nil { + return + } + m.mu.RLock() + snapFn := m.snapFn + m.mu.RUnlock() + var sessions []MeshSessionBlock + if snapFn != nil { + sessions = snapFn() + } + payload := EncodeTrxSnapshot(TrxSnapshotPayload{ + OriginNodeID: m.cfg.NodeID, + Sessions: sessions, + }) + m.broadcastCtrl(MeshTypeTrxSnapshot, payload) +} + +// PublishTrxDelta enqueues a directory delta. +func (m *HybridMesh) PublishTrxDelta(callsign string, isATC bool, trxs []Transceiver) { + if m == nil { + return + } + meshTrxs := make([]MeshTrx, 0, len(trxs)) + for _, t := range trxs { + meshTrxs = append(meshTrxs, MeshTrx{ + ID: t.ID, FreqHz: t.Frequency, + LatDeg: t.LatDeg, LonDeg: t.LonDeg, AltM: t.HeightMslM, + }) + } + payload := EncodeTrxDelta(TrxDeltaPayload{ + OriginNodeID: m.cfg.NodeID, + Callsign: callsign, + IsATC: isATC, + Trxs: meshTrxs, + }) + m.broadcastCtrl(MeshTypeTrxDelta, payload) +} + +// PublishSessionLeave enqueues leave. +func (m *HybridMesh) PublishSessionLeave(callsign string) { + if m == nil { + return + } + payload := EncodeSessionLeave(SessionLeavePayload{ + OriginNodeID: m.cfg.NodeID, + Callsign: callsign, + }) + m.broadcastCtrl(MeshTypeSessionLeave, payload) +} + +// PublishInterest encodes + enqueues control only (does not mutate local peerInterest). +func (m *HybridMesh) PublishInterest(entries []InterestEntry) { + if m == nil { + return + } + payload := EncodeInterest(InterestPayload{NodeID: m.cfg.NodeID, Entries: entries}) + m.broadcastCtrl(MeshTypeInterest, payload) +} + +// EnqueueAudioRelay Interest-filters and enqueues UDP voice frames (MemoryMesh-compatible). +// Lock order: snapshot under m.mu, then p.mu alone (never nested). +func (m *HybridMesh) EnqueueAudioRelay(relay AudioRelay) { + if m == nil { + return + } + if relay.Audio != nil { + relay.Audio = append([]byte(nil), relay.Audio...) + } + if relay.TxRadios != nil { + relay.TxRadios = append([]RelayTxRadio(nil), relay.TxRadios...) + } + if relay.OriginNode == "" { + relay.OriginNode = m.cfg.NodeID + } + if len(relay.TxRadios) > maxMeshVoiceTxRadios { + return + } + + // Snapshot peer pointers under m.mu only. + m.mu.RLock() + type snap struct { + id string + p *hybridPeer + } + snaps := make([]snap, 0, len(m.peers)) + for id, p := range m.peers { + snaps = append(snaps, snap{id: id, p: p}) + } + m.mu.RUnlock() + + // Encode once; per-peer packet copy on enqueue. + var packet []byte + encoded := false + + for _, s := range snaps { + s.p.mu.Lock() + authed := s.p.authed + q := s.p.voiceQ + s.p.mu.Unlock() + if !authed || q == nil { + continue + } + want := false + for _, tx := range relay.TxRadios { + ck := geo.CellKey{ + ILat: geo.CellIndex(tx.LatDeg, geo.DefaultGridCellDeg), + ILon: geo.CellIndex(tx.LonDeg, geo.DefaultGridCellDeg), + } + if m.PeerWants(s.id, tx.FreqHz, ck) { + want = true + break + } + } + if !want { + continue + } + if !encoded { + payload, err := EncodeAudioRelay(relay) + if err != nil { + return + } + pkt, err := EncodeMeshFrameBytes(MeshTypeAudioRelay, payload) + if err != nil || len(pkt) > maxMeshVoiceDatagram { + return + } + packet = pkt + encoded = true + } + job := voiceJob{packet: append([]byte(nil), packet...)} + if q.Enqueue(job) { + m.voiceQueueDrops.Add(1) + } + } +} + +func (m *HybridMesh) broadcastCtrl(typ byte, payload []byte) { + m.mu.RLock() + peers := make([]*hybridPeer, 0, len(m.peers)) + for _, p := range m.peers { + peers = append(peers, p) + } + m.mu.RUnlock() + for _, p := range peers { + p.mu.Lock() + authed := p.authed + q := p.ctrlQ + p.mu.Unlock() + if !authed || q == nil { + continue + } + job := meshCtrlJob{typ: typ, payload: append([]byte(nil), payload...)} + if q.Enqueue(job) { + m.ctrlQueueDrops.Add(1) + m.logger.Debug("AFV hybrid mesh control drop", "peer", p.id, "type", typ) + } + } +} + +// PeerAuthedForTest reports whether peer is control-authed (tests). +func (m *HybridMesh) PeerAuthedForTest(peerID string) bool { + if m == nil { + return false + } + m.mu.RLock() + p := m.peers[peerID] + m.mu.RUnlock() + if p == nil { + return false + } + p.mu.Lock() + defer p.mu.Unlock() + return p.authed +} + +// WaitPeerAuthed waits until peer is authed or timeout. +func (m *HybridMesh) WaitPeerAuthed(peerID string, d time.Duration) bool { + deadline := time.Now().Add(d) + for time.Now().Before(deadline) { + if m.PeerAuthedForTest(peerID) { + return true + } + time.Sleep(10 * time.Millisecond) + } + return false +} + +// ForcePeerDeathForTest closes the peer TCP conn to trigger death path (tests). +func (m *HybridMesh) ForcePeerDeathForTest(peerID string) { + if m == nil { + return + } + m.mu.RLock() + p := m.peers[peerID] + m.mu.RUnlock() + if p == nil { + return + } + p.mu.Lock() + c := p.conn + p.mu.Unlock() + if c != nil { + _ = c.Close() + } +} + +// UDPAllowDrops / InboundRateDrops / VoiceQueueDrops expose counters for tests. +func (m *HybridMesh) UDPAllowDrops() uint64 { return m.udpAllowDrops.Load() } +func (m *HybridMesh) UDPOversizeDrops() uint64 { return m.udpOversizeDrops.Load() } +func (m *HybridMesh) InboundRateDrops() uint64 { return m.inboundRateDrops.Load() } +func (m *HybridMesh) VoiceQueueDrops() uint64 { return m.voiceQueueDrops.Load() } +func (m *HybridMesh) CtrlQueueDrops() uint64 { return m.ctrlQueueDrops.Load() } +func (m *HybridMesh) VoiceTx() uint64 { return m.voiceTx.Load() } +func (m *HybridMesh) VoiceRx() uint64 { return m.voiceRx.Load() } + +// CurrentConnForTest returns the current TCP conn for peer (tests / dual-conn). +func (m *HybridMesh) CurrentConnForTest(peerID string) net.Conn { + if m == nil { + return nil + } + m.mu.RLock() + p := m.peers[peerID] + m.mu.RUnlock() + if p == nil { + return nil + } + p.mu.Lock() + defer p.mu.Unlock() + return p.conn +} + +// ApplyInterestDirect is test helper for Interest maps without TCP. +func (m *HybridMesh) ApplyInterestDirect(fromPeer string, entries []InterestEntry) { + set := make(map[FreqCell]struct{}, len(entries)) + for _, e := range entries { + set[FreqCell{FreqHz: e.FreqHz, Cell: geo.CellKey{ILat: e.ILat, ILon: e.ILon}}] = struct{}{} + } + m.mu.Lock() + m.peerInterest[fromPeer] = set + m.mu.Unlock() +} + +// ForceEnqueueCtrlForTest fills control queue (drop-oldest tests). +func (m *HybridMesh) ForceEnqueueCtrlForTest(peerID string, n int) { + m.mu.RLock() + p := m.peers[peerID] + m.mu.RUnlock() + if p == nil { + return + } + p.mu.Lock() + q := p.ctrlQ + p.mu.Unlock() + if q == nil { + return + } + for i := 0; i < n; i++ { + job := meshCtrlJob{typ: MeshTypeHeartbeat, payload: EncodeHeartbeatPayload(uint64(i))} + if q.Enqueue(job) { + m.ctrlQueueDrops.Add(1) + } + } +} + +// ForceEnqueueVoiceForTest fills voice queue (drop-oldest tests). +func (m *HybridMesh) ForceEnqueueVoiceForTest(peerID string, n int) { + m.mu.RLock() + p := m.peers[peerID] + m.mu.RUnlock() + if p == nil { + return + } + p.mu.Lock() + q := p.voiceQ + p.mu.Unlock() + if q == nil { + return + } + pkt := make([]byte, 32) + for i := 0; i < n; i++ { + job := voiceJob{packet: append([]byte(nil), pkt...)} + if q.Enqueue(job) { + m.voiceQueueDrops.Add(1) + } + } +} + +// PeerCtrlQueueLen / PeerVoiceQueueLen for tests. +func (m *HybridMesh) PeerCtrlQueueLen(peerID string) int { + m.mu.RLock() + p := m.peers[peerID] + m.mu.RUnlock() + if p == nil { + return 0 + } + p.mu.Lock() + q := p.ctrlQ + p.mu.Unlock() + if q == nil { + return 0 + } + return q.Len() +} + +func (m *HybridMesh) PeerVoiceQueueLen(peerID string) int { + m.mu.RLock() + p := m.peers[peerID] + m.mu.RUnlock() + if p == nil { + return 0 + } + p.mu.Lock() + q := p.voiceQ + p.mu.Unlock() + if q == nil { + return 0 + } + return q.Len() +} + +// ListenTCPAddr / ListenVoiceAddr return bound addresses after Start (tests, :0). +func (m *HybridMesh) ListenTCPAddr() string { + if m == nil || m.ln == nil { + return m.cfg.ListenTCP + } + return m.ln.Addr().String() +} + +func (m *HybridMesh) ListenVoiceAddr() string { + if m == nil || m.voicePC == nil { + return m.cfg.ListenVoice + } + return m.voicePC.LocalAddr().String() +} + +var _ Mesh = (*HybridMesh)(nil) diff --git a/internal/afv/mesh_hybrid_e2e_test.go b/internal/afv/mesh_hybrid_e2e_test.go new file mode 100644 index 0000000..e819323 --- /dev/null +++ b/internal/afv/mesh_hybrid_e2e_test.go @@ -0,0 +1,249 @@ +package afv_test + +import ( + "context" + "database/sql" + "net" + "strconv" + "testing" + "time" + + "github.com/renorris/openfsd/internal/afv" + "github.com/renorris/openfsd/internal/db" + "github.com/renorris/openfsd/internal/geo" + "github.com/renorris/openfsd/pkg/afvprotocol" + "github.com/renorris/openfsd/pkg/protocol" +) + +func freePortTCP(t *testing.T) int { + t.Helper() + ln, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + p := ln.Addr().(*net.TCPAddr).Port + _ = ln.Close() + return p +} + +func freePortUDP(t *testing.T) int { + t.Helper() + pc, err := net.ListenPacket("udp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + p := pc.LocalAddr().(*net.UDPAddr).Port + _ = pc.Close() + return p +} + +type hybridNode struct { + srv *afv.Server + mesh *afv.HybridMesh + api string + udp string + cancel context.CancelFunc +} + +func setupHybridPair(t *testing.T) (n1, n2 *hybridNode, cidA, cidB int) { + t.Helper() + tcp1, voice1 := freePortTCP(t), freePortUDP(t) + tcp2, voice2 := freePortTCP(t), freePortUDP(t) + + open := func(name, nodeID, peerID string, tcp, voice, peerTCP, peerVoice int) (*hybridNode, *db.Repositories) { + sqlDB, err := sql.Open("sqlite", "file:hybride2e_"+t.Name()+"_"+name+"?mode=memory&cache=shared") + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = sqlDB.Close() }) + if err := db.Migrate(sqlDB); err != nil { + t.Fatal(err) + } + repos, err := db.NewRepositories(sqlDB) + if err != nil { + t.Fatal(err) + } + hm, err := afv.NewHybridMesh(afv.HybridMeshConfig{ + NodeID: nodeID, + ListenTCP: "127.0.0.1:" + strconv.Itoa(tcp), + ListenVoice: "127.0.0.1:" + strconv.Itoa(voice), + Peers: []afv.ClusterPeer{{ + ID: peerID, Addr: "127.0.0.1:" + strconv.Itoa(peerTCP), + VoiceAddr: "127.0.0.1:" + strconv.Itoa(peerVoice), + }}, + PSK: "hybrid-e2e-psk", + }) + if err != nil { + t.Fatal(err) + } + cfg := &afv.Config{ + APIListen: "127.0.0.1:0", UDPListen: "127.0.0.1:0", + UDPAdvertiseIPv4: "127.0.0.1:0", JWTTTL: time.Hour, + MaxSessions: 100, MaxSessionsPerCID: 5, + AuthFailMax: 50, AuthFailWindow: time.Minute, MaxDatagram: 8192, + RangeDefaultNM: 40, RangeATCNM: 150, RangeUnicomNM: 15, RangeEdgeRatio: 0.1, + HeartbeatTimeout: 30 * time.Second, SessionIdleTimeout: 60 * time.Second, + } + s := afv.New(cfg, repos.UserRepo, repos.ConfigRepo, []byte("hybrid-e2e-jwt-secret-key-32b!!")) + s.SetMesh(hm) + api, udp, cancel := startAFV(t, s) + cfg.UDPAdvertiseIPv4 = udp + return &hybridNode{srv: s, mesh: hm, api: api, udp: udp, cancel: cancel}, repos + } + + var r1, r2 *db.Repositories + n1, r1 = open("n1", "n1", "n2", tcp1, voice1, tcp2, voice2) + n2, r2 = open("n2", "n2", "n1", tcp2, voice2, tcp1, voice1) + t.Cleanup(func() { n1.cancel(); n2.cancel() }) + + // Wait mesh auth both ways + if !n1.mesh.WaitPeerAuthed("n2", 5*time.Second) { + t.Fatal("n1↔n2 auth timeout") + } + if !n2.mesh.WaitPeerAuthed("n1", 5*time.Second) { + t.Fatal("n2↔n1 auth timeout") + } + + uA := &db.User{Password: "pw", NetworkRating: int(protocol.NetworkRatingObserver), PilotRating: 1} + uB := &db.User{Password: "pw", NetworkRating: int(protocol.NetworkRatingObserver), PilotRating: 1} + if err := r1.UserRepo.CreateUser(context.Background(), uA); err != nil { + t.Fatal(err) + } + if err := r2.UserRepo.CreateUser(context.Background(), uB); err != nil { + t.Fatal(err) + } + return n1, n2, uA.CID, uB.CID +} + +func waitHybridPeerWants(t *testing.T, m *afv.HybridMesh, peer string, freq uint32, txLat, txLon float64) { + t.Helper() + ck := geo.CellKey{ + ILat: geo.CellIndex(txLat, geo.DefaultGridCellDeg), + ILon: geo.CellIndex(txLon, geo.DefaultGridCellDeg), + } + deadline := time.Now().Add(5 * time.Second) + for time.Now().Before(deadline) { + if m.PeerWants(peer, freq, ck) { + return + } + time.Sleep(20 * time.Millisecond) + } + t.Fatalf("timeout PeerWants peer=%s", peer) +} + +func TestHybridMeshE2E_A2A(t *testing.T) { + n1, n2, cidA, cidB := setupHybridPair(t) + + tokA := meshAuth(t, n1.api, cidA) + tokB := meshAuth(t, n2.api, cidB) + pcA := meshPostCS(t, n1.api, tokA, cidA, "N1A", n1.udp) + pcB := meshPostCS(t, n2.api, tokB, cidB, "N2B", n2.udp) + + // Same frequency, nearby positions (in range) + freq := uint32(122800000) + latA, lonA := 51.4700, -0.4543 + latB, lonB := 51.4710, -0.4530 + meshPostTrx(t, n1.api, tokA, cidA, "N1A", latA, lonA, freq) + meshPostTrx(t, n2.api, tokB, cidB, "N2B", latB, lonB, freq) + + // Bind UDP + HA first — Interest only includes Bound sessions + srvA, err := net.ResolveUDPAddr("udp", n1.udp) + if err != nil { + t.Fatal(err) + } + srvB, err := net.ResolveUDPAddr("udp", n2.udp) + if err != nil { + t.Fatal(err) + } + cliA, err := net.ListenUDP("udp", &net.UDPAddr{IP: net.IPv4(127, 0, 0, 1), Port: 0}) + if err != nil { + t.Fatal(err) + } + defer cliA.Close() + cliB, err := net.ListenUDP("udp", &net.UDPAddr{IP: net.IPv4(127, 0, 0, 1), Port: 0}) + if err != nil { + t.Fatal(err) + } + defer cliB.Close() + + chA, err := afvprotocol.ClientChannel(pcA.VoiceServer.ChannelConfig.ChannelTag, pcA.VoiceServer.ChannelConfig.AeadReceiveKey, pcA.VoiceServer.ChannelConfig.AeadTransmitKey) + if err != nil { + t.Fatal(err) + } + chB, err := afvprotocol.ClientChannel(pcB.VoiceServer.ChannelConfig.ChannelTag, pcB.VoiceServer.ChannelConfig.AeadReceiveKey, pcB.VoiceServer.ChannelConfig.AeadTransmitKey) + if err != nil { + t.Fatal(err) + } + + meshBindHB(t, chA, cliA, srvA, "N1A", 0) + meshBindHB(t, chB, cliB, srvB, "N2B", 0) + + // Interest after bind; barrier on PeerWants (async TCP apply) + n1.srv.PublishInterestNowForTest() + n2.srv.PublishInterestNowForTest() + waitHybridPeerWants(t, n1.mesh, "n2", freq, latA, lonA) + + // A transmits AT + at := afvprotocol.AudioTx{ + Callsign: "N1A", + SequenceCounter: 1, + Audio: []byte("hybrid-opus"), + LastPacket: false, + Transceivers: []afvprotocol.TxTransceiver{{ID: 0}}, + } + pkt, err := chA.Encapsulate(2, afvprotocol.DTONameAudioTx, at.EncodeMsgpack(), nil) + if err != nil { + t.Fatal(err) + } + if _, err := cliA.WriteTo(pkt, srvA); err != nil { + t.Fatal(err) + } + + ar, ok := readAR(t, chB, cliB, 3*time.Second) + if !ok { + t.Fatal("B did not receive AR via hybrid mesh") + } + if ar.Callsign != "N1A" { + t.Fatalf("callsign %q", ar.Callsign) + } +} + +func TestHybridMeshE2E_WrongPSK(t *testing.T) { + tcp1, voice1 := freePortTCP(t), freePortUDP(t) + tcp2, voice2 := freePortTCP(t), freePortUDP(t) + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + m1, err := afv.NewHybridMesh(afv.HybridMeshConfig{ + NodeID: "n1", ListenTCP: "127.0.0.1:" + strconv.Itoa(tcp1), ListenVoice: "127.0.0.1:" + strconv.Itoa(voice1), + Peers: []afv.ClusterPeer{{ID: "n2", Addr: "127.0.0.1:" + strconv.Itoa(tcp2), VoiceAddr: "127.0.0.1:" + strconv.Itoa(voice2)}}, + PSK: "psk-a", + }) + if err != nil { + t.Fatal(err) + } + m2, err := afv.NewHybridMesh(afv.HybridMeshConfig{ + NodeID: "n2", ListenTCP: "127.0.0.1:" + strconv.Itoa(tcp2), ListenVoice: "127.0.0.1:" + strconv.Itoa(voice2), + Peers: []afv.ClusterPeer{{ID: "n1", Addr: "127.0.0.1:" + strconv.Itoa(tcp1), VoiceAddr: "127.0.0.1:" + strconv.Itoa(voice1)}}, + PSK: "psk-b", + }) + if err != nil { + t.Fatal(err) + } + if err := m1.Start(ctx); err != nil { + t.Fatal(err) + } + if err := m2.Start(ctx); err != nil { + t.Fatal(err) + } + defer m1.Stop() + defer m2.Stop() + // Poll window: must remain unauthenticated under wrong PSK. + deadline := time.Now().Add(1500 * time.Millisecond) + for time.Now().Before(deadline) { + if m1.PeerAuthedForTest("n2") || m2.PeerAuthedForTest("n1") { + t.Fatal("wrong PSK must not auth") + } + time.Sleep(20 * time.Millisecond) + } +} diff --git a/internal/afv/mesh_tcp.go b/internal/afv/mesh_tcp.go new file mode 100644 index 0000000..253aa19 --- /dev/null +++ b/internal/afv/mesh_tcp.go @@ -0,0 +1,493 @@ +package afv + +// HybridMesh TCP control plane: listen/dial, Hello, HB, directory, Interest, peer death. +// AudioRelay (type 20) is forbidden on TCP (H-2). + +import ( + "context" + "errors" + "fmt" + "net" + "strings" + "time" + + "github.com/renorris/openfsd/internal/geo" +) + +func (m *HybridMesh) acceptLoop(ctx context.Context) { + for { + conn, err := m.ln.Accept() + if err != nil { + select { + case <-m.stopCh: + return + case <-ctx.Done(): + return + default: + } + // Permanent close of listener → exit; transient errors → backoff + continue. + if errors.Is(err, net.ErrClosed) { + return + } + m.logger.Debug("AFV hybrid mesh accept error", "err", err) + select { + case <-m.stopCh: + return + case <-ctx.Done(): + return + case <-time.After(100 * time.Millisecond): + } + continue + } + m.wg.Add(1) + go func() { + defer m.wg.Done() + m.serveConn(conn) + }() + } +} + +func (m *HybridMesh) dialLoop(ctx context.Context, peer ClusterPeer) { + backoff := 200 * time.Millisecond + for { + select { + case <-m.stopCh: + return + case <-ctx.Done(): + return + default: + } + // Skip dial if already authed with live conn. + m.mu.RLock() + p := m.peers[peer.ID] + m.mu.RUnlock() + if p != nil { + p.mu.Lock() + live := p.authed && p.conn != nil + p.mu.Unlock() + if live { + select { + case <-m.stopCh: + return + case <-ctx.Done(): + return + case <-time.After(500 * time.Millisecond): + } + continue + } + } + d := net.Dialer{Timeout: 5 * time.Second} + conn, err := d.DialContext(ctx, "tcp", peer.Addr) + if err != nil { + select { + case <-m.stopCh: + return + case <-ctx.Done(): + return + case <-time.After(backoff): + } + if backoff < 5*time.Second { + backoff *= 2 + } + continue + } + backoff = 200 * time.Millisecond + m.serveConn(conn) + // reconnect after disconnect + select { + case <-m.stopCh: + return + case <-ctx.Done(): + return + case <-time.After(200 * time.Millisecond): + } + } +} + +func (m *HybridMesh) serveConn(conn net.Conn) { + defer func() { _ = conn.Close() }() + + // Send Hello immediately + helloPay := EncodeHelloPayload(HelloPayload{NodeID: m.cfg.NodeID, PSK: m.cfg.PSK}) + if err := EncodeMeshFrame(conn, MeshTypeHello, helloPay); err != nil { + return + } + + _ = conn.SetReadDeadline(time.Now().Add(5 * time.Second)) + fr, err := DecodeMeshFrame(conn) + if err != nil || fr.Type != MeshTypeHello { + m.logger.Warn("AFV hybrid mesh non-Hello first frame or read error", "err", err) + return + } + hp, err := DecodeHelloPayload(fr.Payload) + if err != nil { + return + } + if err := VerifyHelloPSK(m.cfg.PSK, hp.PSK); err != nil { + m.logger.Warn("AFV hybrid mesh hello auth failed", "peer", hp.NodeID) + return + } + peerID := strings.TrimSpace(hp.NodeID) + if peerID == "" || peerID == m.cfg.NodeID { + m.logger.Warn("AFV hybrid mesh reject self or empty peer") + return + } + if _, ok := m.peerCfg[peerID]; !ok { + m.logger.Warn("AFV hybrid mesh reject unknown peer", "peer", peerID) + return + } + // H-15: log TCP RemoteAddr host ≠ configured peer TCP host as warn only + if expected, ok := m.peerCfg[peerID]; ok && expected.Addr != "" { + if ra := conn.RemoteAddr(); ra != nil { + host, _, _ := net.SplitHostPort(ra.String()) + expHost, _, _ := net.SplitHostPort(expected.Addr) + if host != "" && expHost != "" && host != expHost && expHost != "0.0.0.0" { + m.logger.Warn("AFV hybrid mesh Hello RemoteAddr host differs from config", + "peer", peerID, "remote", host, "configured", expHost) + } + } + } + _ = conn.SetReadDeadline(time.Time{}) + + // Install peer (H-9 dual-conn replace); capture this generation's stop+queues. + // Soft replace does not run peerDeath / OnPeerDead. + p, stop, ctrlQ := m.installPeerConn(peerID, conn) + if p == nil { + return + } + + // Per-peer TCP writer for this generation only (bound to ctrlQ + stop) + m.wg.Add(1) + go func() { + defer m.wg.Done() + m.peerCtrlWriter(p, conn, ctrlQ, stop) + }() + + // Post-Hello sequence: Snapshot then Interest (M-16) on TCP only + m.sendPostHello(ctrlQ) + + // Read loop + for { + select { + case <-m.stopCh: + return + case <-stop: + // Superseded by dual-conn replace or death of this generation — exit quietly. + return + default: + } + _ = conn.SetReadDeadline(time.Now().Add(m.peerDeathGrace + 5*time.Second)) + fr, err := DecodeMeshFrame(conn) + if err != nil { + // Generation-scoped: only tear down if this conn is still current. + m.peerDeath(peerID, "read", conn) + return + } + // Generation gate (same as lastHB): superseded dual-conn readers must not + // apply Interest/directory or peerDeath after a successful late decode. + p.mu.Lock() + stillMine := p.conn == conn + if stillMine { + p.lastHB = time.Now() // H-8: every post-Hello control frame + } + p.mu.Unlock() + if !stillMine { + return + } + + if !m.handleCtrlFrame(peerID, fr, conn) { + return // closed (type 20, bad, unexpected Hello) + } + } +} + +// installPeerConn replaces any existing conn for peerID (H-9). +// Soft replace: closes previous generation stop+conn without OnPeerDead / Interest wipe. +// Returns the peer, this generation's stop channel, and ctrl queue. +func (m *HybridMesh) installPeerConn(peerID string, conn net.Conn) (*hybridPeer, chan struct{}, *dropOldestQueue[meshCtrlJob]) { + m.mu.RLock() + p, ok := m.peers[peerID] + m.mu.RUnlock() + if !ok { + return nil, nil, nil + } + + p.mu.Lock() + // Close previous live generation (soft replace — no peerDeath) + if p.conn != nil && p.conn != conn { + select { + case <-p.stop: + default: + close(p.stop) + } + _ = p.conn.Close() + } + // Reset queues + stop channel for new generation + ctrlQ := newDropOldestQueue[meshCtrlJob](m.ctrlDepth) + voiceQ := newDropOldestQueue[voiceJob](m.voiceDepth) + stop := make(chan struct{}) + p.ctrlQ = ctrlQ + p.voiceQ = voiceQ + p.stop = stop + p.conn = conn + p.authed = true + p.lastHB = time.Now() + p.mu.Unlock() + + m.logger.Info("AFV hybrid mesh peer authenticated", "peer", peerID) + return p, stop, ctrlQ +} + +func (m *HybridMesh) sendPostHello(ctrlQ *dropOldestQueue[meshCtrlJob]) { + if ctrlQ == nil { + return + } + // Snapshot + m.mu.RLock() + snapFn := m.snapFn + m.mu.RUnlock() + var sessions []MeshSessionBlock + if snapFn != nil { + sessions = snapFn() + } + snap := EncodeTrxSnapshot(TrxSnapshotPayload{ + OriginNodeID: m.cfg.NodeID, + Sessions: sessions, + }) + _ = ctrlQ.Enqueue(meshCtrlJob{typ: MeshTypeTrxSnapshot, payload: snap}) + // Interest + m.mu.RLock() + ifn := m.interestFn + m.mu.RUnlock() + var entries []InterestEntry + if ifn != nil { + entries = ifn() + } + ipay := EncodeInterest(InterestPayload{NodeID: m.cfg.NodeID, Entries: entries}) + _ = ctrlQ.Enqueue(meshCtrlJob{typ: MeshTypeInterest, payload: ipay}) +} + +func (m *HybridMesh) peerCtrlWriter(p *hybridPeer, conn net.Conn, q *dropOldestQueue[meshCtrlJob], stop chan struct{}) { + if q == nil { + return + } + for { + select { + case <-m.stopCh: + return + case <-stop: + return + case job, ok := <-q.Chan(): + if !ok { + return + } + if err := EncodeMeshFrame(conn, job.typ, job.payload); err != nil { + m.peerDeath(p.id, "write", conn) + return + } + } + } +} + +// handleCtrlFrame returns false if the peer should be closed. +// deadConn is the TCP conn of this reader generation (for generation-scoped death/apply). +// Superseded readers (p.conn != deadConn) are no-ops for apply and death. +func (m *HybridMesh) handleCtrlFrame(peerID string, fr MeshFrame, deadConn net.Conn) bool { + // Defense in depth: re-check generation before any side effects. + m.mu.RLock() + p := m.peers[peerID] + m.mu.RUnlock() + if p == nil { + return false + } + p.mu.Lock() + stillMine := p.conn == deadConn + p.mu.Unlock() + if !stillMine { + return true // no apply / no death; caller may already be exiting + } + + switch fr.Type { + case MeshTypeHello: + m.logger.Warn("AFV hybrid mesh unexpected Hello after auth", "peer", peerID) + m.peerDeath(peerID, "unexpected Hello", deadConn) + return false + case MeshTypeHeartbeat: + return true + case MeshTypeTrxSnapshot: + pl, err := DecodeTrxSnapshot(fr.Payload) + if err != nil { + m.peerDeath(peerID, "bad TrxSnapshot", deadConn) + return false + } + if pl.OriginNodeID != peerID { + m.peerDeath(peerID, "snapshot origin mismatch", deadConn) + return false + } + m.mu.RLock() + dir := m.onDir + m.mu.RUnlock() + if dir != nil { + dir.ApplySnapshot(pl.OriginNodeID, meshSessionsToRemote(pl.Sessions)) + } + return true + case MeshTypeTrxDelta: + pl, err := DecodeTrxDelta(fr.Payload) + if err != nil { + m.peerDeath(peerID, "bad TrxDelta", deadConn) + return false + } + if pl.OriginNodeID != peerID { + m.peerDeath(peerID, "delta origin mismatch", deadConn) + return false + } + m.mu.RLock() + dir := m.onDir + m.mu.RUnlock() + if dir != nil { + dir.ApplyDelta(pl.OriginNodeID, pl.Callsign, pl.IsATC, meshTrxToLocal(pl.Trxs)) + } + return true + case MeshTypeSessionLeave: + pl, err := DecodeSessionLeave(fr.Payload) + if err != nil { + m.peerDeath(peerID, "bad SessionLeave", deadConn) + return false + } + if pl.OriginNodeID != peerID { + m.peerDeath(peerID, "leave origin mismatch", deadConn) + return false + } + m.mu.RLock() + dir := m.onDir + m.mu.RUnlock() + if dir != nil { + dir.ApplyLeave(pl.OriginNodeID, pl.Callsign) + } + return true + case MeshTypeAudioRelay: + // H-2: type 20 forbidden on TCP + m.logger.Warn("AFV hybrid mesh type 20 AudioRelay on TCP — closing peer", "peer", peerID) + m.peerDeath(peerID, "audio on TCP", deadConn) + return false + case MeshTypeInterest: + // H-14: full replace for origin + pl, err := DecodeInterest(fr.Payload) + if err != nil { + m.peerDeath(peerID, "bad Interest", deadConn) + return false + } + if pl.NodeID != peerID { + m.peerDeath(peerID, "interest node mismatch", deadConn) + return false + } + set := make(map[FreqCell]struct{}, len(pl.Entries)) + for _, e := range pl.Entries { + set[FreqCell{FreqHz: e.FreqHz, Cell: geo.CellKey{ILat: e.ILat, ILon: e.ILon}}] = struct{}{} + } + m.mu.Lock() + m.peerInterest[peerID] = set + m.mu.Unlock() + return true + default: + m.logger.Warn("AFV hybrid mesh unknown control type", "peer", peerID, "type", fr.Type) + m.peerDeath(peerID, fmt.Sprintf("unknown type %d", fr.Type), deadConn) + return false + } +} + +// peerDeath is generation-scoped death procedure (H-8 / H-9). +// Only tears down if deadConn is still the peer's current conn (or deadConn is nil +// and the peer is authed — used for heartbeat timeout with captured conn). +// Lock order: p.mu alone, then m.mu alone — never nested. +func (m *HybridMesh) peerDeath(peerID, reason string, deadConn net.Conn) { + m.mu.RLock() + p := m.peers[peerID] + m.mu.RUnlock() + if p == nil { + return + } + + p.mu.Lock() + // Generation guard: stale reader/writer from dual-conn replace must not kill successor. + if deadConn != nil && p.conn != deadConn { + p.mu.Unlock() + return + } + if deadConn == nil && (!p.authed || p.conn == nil) { + p.mu.Unlock() + return + } + wasAuthed := p.authed + p.authed = false + if p.conn != nil { + _ = p.conn.Close() + p.conn = nil + } + select { + case <-p.stop: + default: + close(p.stop) + } + // re-init empty queues + p.ctrlQ = newDropOldestQueue[meshCtrlJob](m.ctrlDepth) + p.voiceQ = newDropOldestQueue[voiceJob](m.voiceDepth) + p.mu.Unlock() + + // clear Interest + load callback under m.mu only (no p.mu) + m.mu.Lock() + m.peerInterest[peerID] = make(map[FreqCell]struct{}) + cb := m.onPeerDead + m.mu.Unlock() + + if wasAuthed { + m.logger.Warn("AFV hybrid mesh peer dead", "peer", peerID, "reason", reason) + if cb != nil { + cb(peerID) + } + } +} + +func (m *HybridMesh) heartbeatLoop(ctx context.Context) { + ticker := time.NewTicker(m.hbInterval) + defer ticker.Stop() + for { + select { + case <-m.stopCh: + return + case <-ctx.Done(): + return + case <-ticker.C: + now := time.Now() + ms := uint64(now.UnixMilli()) + pay := EncodeHeartbeatPayload(ms) + + // Snapshot peer pointers under m.mu only — never nest p.mu. + m.mu.RLock() + peers := make([]*hybridPeer, 0, len(m.peers)) + for _, p := range m.peers { + peers = append(peers, p) + } + m.mu.RUnlock() + + for _, p := range peers { + p.mu.Lock() + authed := p.authed + last := p.lastHB + conn := p.conn + q := p.ctrlQ + p.mu.Unlock() + if !authed { + continue + } + if !last.IsZero() && now.Sub(last) > m.peerDeathGrace { + m.peerDeath(p.id, "heartbeat timeout", conn) + continue + } + if q != nil { + _ = q.Enqueue(meshCtrlJob{typ: MeshTypeHeartbeat, payload: pay}) + } + } + } + } +} diff --git a/internal/afv/mesh_tcp_test.go b/internal/afv/mesh_tcp_test.go new file mode 100644 index 0000000..04248f4 --- /dev/null +++ b/internal/afv/mesh_tcp_test.go @@ -0,0 +1,469 @@ +package afv + +import ( + "context" + "io" + "net" + "strconv" + "sync/atomic" + "testing" + "time" + + "github.com/renorris/openfsd/internal/geo" +) + +func freeTCPPort(t *testing.T) int { + t.Helper() + ln, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + defer ln.Close() + return ln.Addr().(*net.TCPAddr).Port +} + +func freeUDPPort(t *testing.T) int { + t.Helper() + pc, err := net.ListenPacket("udp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + defer pc.Close() + return pc.LocalAddr().(*net.UDPAddr).Port +} + +func hybridPairPorts(t *testing.T) (tcp1, voice1, tcp2, voice2 int) { + t.Helper() + tcp1 = freeTCPPort(t) + voice1 = freeUDPPort(t) + tcp2 = freeTCPPort(t) + voice2 = freeUDPPort(t) + return +} + +func startHybridPair(t *testing.T, psk string) (m1, m2 *HybridMesh, cancel context.CancelFunc) { + t.Helper() + tcp1, voice1, tcp2, voice2 := hybridPairPorts(t) + ctx, cancel := context.WithCancel(context.Background()) + t.Cleanup(cancel) + + cfg1 := HybridMeshConfig{ + NodeID: "n1", + ListenTCP: "127.0.0.1:" + itoa(tcp1), + ListenVoice: "127.0.0.1:" + itoa(voice1), + Peers: []ClusterPeer{{ + ID: "n2", Addr: "127.0.0.1:" + itoa(tcp2), VoiceAddr: "127.0.0.1:" + itoa(voice2), + }}, + PSK: psk, + } + cfg2 := HybridMeshConfig{ + NodeID: "n2", + ListenTCP: "127.0.0.1:" + itoa(tcp2), + ListenVoice: "127.0.0.1:" + itoa(voice2), + Peers: []ClusterPeer{{ + ID: "n1", Addr: "127.0.0.1:" + itoa(tcp1), VoiceAddr: "127.0.0.1:" + itoa(voice1), + }}, + PSK: psk, + } + var err error + m1, err = NewHybridMesh(cfg1) + if err != nil { + t.Fatal(err) + } + m2, err = NewHybridMesh(cfg2) + if err != nil { + t.Fatal(err) + } + if err := m1.Start(ctx); err != nil { + t.Fatal(err) + } + if err := m2.Start(ctx); err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = m1.Stop(); _ = m2.Stop() }) + if !m1.WaitPeerAuthed("n2", 3*time.Second) { + t.Fatal("n1 did not auth n2") + } + if !m2.WaitPeerAuthed("n1", 3*time.Second) { + t.Fatal("n2 did not auth n1") + } + return m1, m2, cancel +} + +func itoa(n int) string { + return strconv.Itoa(n) +} + +func geoCell(ilat, ilon int32) geo.CellKey { + return geo.CellKey{ILat: ilat, ILon: ilon} +} + +func TestHybridMesh_HelloWrongPSK(t *testing.T) { + tcp1, voice1, tcp2, voice2 := hybridPairPorts(t) + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + m1, err := NewHybridMesh(HybridMeshConfig{ + NodeID: "n1", ListenTCP: "127.0.0.1:" + itoa(tcp1), ListenVoice: "127.0.0.1:" + itoa(voice1), + Peers: []ClusterPeer{{ID: "n2", Addr: "127.0.0.1:" + itoa(tcp2), VoiceAddr: "127.0.0.1:" + itoa(voice2)}}, + PSK: "secret-a", + }) + if err != nil { + t.Fatal(err) + } + m2, err := NewHybridMesh(HybridMeshConfig{ + NodeID: "n2", ListenTCP: "127.0.0.1:" + itoa(tcp2), ListenVoice: "127.0.0.1:" + itoa(voice2), + Peers: []ClusterPeer{{ID: "n1", Addr: "127.0.0.1:" + itoa(tcp1), VoiceAddr: "127.0.0.1:" + itoa(voice1)}}, + PSK: "secret-b", + }) + if err != nil { + t.Fatal(err) + } + if err := m1.Start(ctx); err != nil { + t.Fatal(err) + } + if err := m2.Start(ctx); err != nil { + t.Fatal(err) + } + defer m1.Stop() + defer m2.Stop() + + // Poll a window: must never become authed under wrong PSK. + deadline := time.Now().Add(1500 * time.Millisecond) + for time.Now().Before(deadline) { + if m1.PeerAuthedForTest("n2") || m2.PeerAuthedForTest("n1") { + t.Fatal("wrong PSK must not authenticate") + } + time.Sleep(20 * time.Millisecond) + } +} + +func TestHybridMesh_NonHelloFirstFrame(t *testing.T) { + tcpPort := freeTCPPort(t) + voicePort := freeUDPPort(t) + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + // Peer that will dial us (higher ID) — but we inject a raw non-Hello first frame + // by connecting as an attacker to the listen port. + m, err := NewHybridMesh(HybridMeshConfig{ + NodeID: "n1", ListenTCP: "127.0.0.1:" + itoa(tcpPort), ListenVoice: "127.0.0.1:" + itoa(voicePort), + Peers: []ClusterPeer{{ID: "n2", Addr: "127.0.0.1:1", VoiceAddr: "127.0.0.1:2"}}, + PSK: "psk", + }) + if err != nil { + t.Fatal(err) + } + if err := m.Start(ctx); err != nil { + t.Fatal(err) + } + defer m.Stop() + + conn, err := net.Dial("tcp", "127.0.0.1:"+itoa(tcpPort)) + if err != nil { + t.Fatal(err) + } + defer conn.Close() + // Send Heartbeat as first frame + if err := EncodeMeshFrame(conn, MeshTypeHeartbeat, EncodeHeartbeatPayload(1)); err != nil { + t.Fatal(err) + } + // Server should close after rejecting non-Hello + _ = conn.SetReadDeadline(time.Now().Add(2 * time.Second)) + buf := make([]byte, 8) + _, err = conn.Read(buf) + if err == nil { + // may get Hello from server first, then close + _, err = io.ReadAll(conn) + } + // peer must not be authed + if m.PeerAuthedForTest("n2") { + t.Fatal("non-Hello first frame must not auth") + } +} + +func TestHybridMesh_Type20OnTCPCloses(t *testing.T) { + m1, m2, _ := startHybridPair(t, "psk") + _ = m2 + + // Inject type 20 on the TCP control conn from m1's peer n2 + m1.mu.RLock() + p := m1.peers["n2"] + m1.mu.RUnlock() + if p == nil { + t.Fatal("no peer") + } + p.mu.Lock() + conn := p.conn + p.mu.Unlock() + if conn == nil { + t.Fatal("no conn") + } + // Write AudioRelay frame on TCP + pay, err := EncodeAudioRelay(AudioRelay{OriginNode: "n1", Callsign: "X", Audio: []byte{1}}) + if err != nil { + t.Fatal(err) + } + if err := EncodeMeshFrame(conn, MeshTypeAudioRelay, pay); err != nil { + t.Fatal(err) + } + // Wait for peer death on m2 (received type 20) + deadline := time.Now().Add(2 * time.Second) + for time.Now().Before(deadline) { + if !m2.PeerAuthedForTest("n1") { + return + } + time.Sleep(20 * time.Millisecond) + } + t.Fatal("type 20 on TCP should close peer on receiver") +} + +func TestHybridMesh_InterestReplace(t *testing.T) { + m1, m2, _ := startHybridPair(t, "psk") + + // m2 publishes Interest → applied on m1's TCP reader + entries := []InterestEntry{{FreqHz: 122800000, ILat: 1, ILon: 2}} + m2.PublishInterest(entries) + deadline := time.Now().Add(2 * time.Second) + for time.Now().Before(deadline) { + if m1.PeerWants("n2", 122800000, geoCell(1, 2)) { + break + } + time.Sleep(10 * time.Millisecond) + } + if !m1.PeerWants("n2", 122800000, geoCell(1, 2)) { + t.Fatal("interest not applied") + } + + // Empty Interest clears (replace, not merge) + m2.PublishInterest(nil) + deadline = time.Now().Add(2 * time.Second) + for time.Now().Before(deadline) { + if !m1.PeerWants("n2", 122800000, geoCell(1, 2)) { + return + } + time.Sleep(10 * time.Millisecond) + } + t.Fatal("empty Interest should clear prior set") +} + +func TestHybridMesh_LastHBAdvancesOnTrxDelta(t *testing.T) { + // Short death grace for test speed — use normal and just verify lastHB moves + m1, m2, _ := startHybridPair(t, "psk") + + m1.mu.RLock() + p := m1.peers["n2"] + m1.mu.RUnlock() + p.mu.Lock() + before := p.lastHB + p.mu.Unlock() + + time.Sleep(20 * time.Millisecond) + m2.PublishTrxDelta("N2PILOT", false, []Transceiver{{ID: 0, Frequency: 1}}) + deadline := time.Now().Add(2 * time.Second) + for time.Now().Before(deadline) { + p.mu.Lock() + after := p.lastHB + p.mu.Unlock() + if after.After(before) { + return + } + time.Sleep(10 * time.Millisecond) + } + t.Fatal("lastHB should advance on TrxDelta") +} + +func TestHybridMesh_PeerDeathReadError(t *testing.T) { + var dead atomic.Bool + m1, m2, _ := startHybridPair(t, "psk") + m1.OnPeerDead(func(id string) { + if id == "n2" { + dead.Store(true) + } + }) + // Arm Interest then kill + m2.PublishInterest([]InterestEntry{{FreqHz: 1, ILat: 0, ILon: 0}}) + deadline := time.Now().Add(2 * time.Second) + for time.Now().Before(deadline) { + if m1.PeerWants("n2", 1, geo.CellKey{}) { + break + } + time.Sleep(10 * time.Millisecond) + } + m1.ForcePeerDeathForTest("n2") + deadline = time.Now().Add(2 * time.Second) + for time.Now().Before(deadline) { + if dead.Load() && !m1.PeerAuthedForTest("n2") && !m1.PeerWants("n2", 1, geo.CellKey{}) { + return + } + time.Sleep(10 * time.Millisecond) + } + t.Fatalf("peer death: dead=%v authed=%v wants=%v", dead.Load(), m1.PeerAuthedForTest("n2"), m1.PeerWants("n2", 1, geo.CellKey{})) +} + +func TestHybridMesh_CtrlQueueDropOldest(t *testing.T) { + // No live peer writer: start only n1 so n2 never Hello — force drops. + tcp1, voice1, tcp2, voice2 := hybridPairPorts(t) + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + m1, err := NewHybridMesh(HybridMeshConfig{ + NodeID: "n1", ListenTCP: "127.0.0.1:" + itoa(tcp1), ListenVoice: "127.0.0.1:" + itoa(voice1), + Peers: []ClusterPeer{{ID: "n2", Addr: "127.0.0.1:" + itoa(tcp2), VoiceAddr: "127.0.0.1:" + itoa(voice2)}}, + PSK: "psk", + }) + if err != nil { + t.Fatal(err) + } + if err := m1.Start(ctx); err != nil { + t.Fatal(err) + } + defer m1.Stop() + + before := m1.CtrlQueueDrops() + m1.ForceEnqueueCtrlForTest("n2", meshControlQueueDepth+50) + if m1.CtrlQueueDrops() <= before { + t.Fatalf("want ctrl drops, got %d→%d", before, m1.CtrlQueueDrops()) + } + if m1.PeerCtrlQueueLen("n2") > meshControlQueueDepth { + t.Fatal("queue over depth") + } +} + +func TestHybridMesh_DualConnReplace(t *testing.T) { + // T10b: second live Hello replaces first without OnPeerDead (soft replace). + m1, m2, _ := startHybridPair(t, "psk") + var deadN1 atomic.Int32 + m2.OnPeerDead(func(id string) { + if id == "n1" { + deadN1.Add(1) + } + }) + + // Seed Interest on m2's view of n1 (m1 publishes → m2 applies). + m1.PublishInterest([]InterestEntry{{FreqHz: 42, ILat: 1, ILon: 2}}) + deadline := time.Now().Add(2 * time.Second) + for time.Now().Before(deadline) { + if m2.PeerWants("n1", 42, geoCell(1, 2)) { + break + } + time.Sleep(10 * time.Millisecond) + } + if !m2.PeerWants("n1", 42, geoCell(1, 2)) { + t.Fatal("interest not applied before replace") + } + + oldConn := m2.CurrentConnForTest("n1") + if oldConn == nil { + t.Fatal("expected live conn") + } + + // Second TCP session claiming n1 → soft replace on m2. + conn, err := net.Dial("tcp", m2.ListenTCPAddr()) + if err != nil { + t.Fatal(err) + } + defer conn.Close() + // m2 sends Hello first; read it, then send ours. + _ = conn.SetReadDeadline(time.Now().Add(2 * time.Second)) + fr, err := DecodeMeshFrame(conn) + if err != nil || fr.Type != MeshTypeHello { + t.Fatalf("expected Hello from m2: %v %+v", err, fr) + } + hello := EncodeHelloPayload(HelloPayload{NodeID: "n1", PSK: "psk"}) + if err := EncodeMeshFrame(conn, MeshTypeHello, hello); err != nil { + t.Fatal(err) + } + + // Wait until current conn is replaced and still authed. + deadline = time.Now().Add(3 * time.Second) + replaced := false + for time.Now().Before(deadline) { + cur := m2.CurrentConnForTest("n1") + if cur != nil && cur != oldConn && m2.PeerAuthedForTest("n1") { + replaced = true + break + } + time.Sleep(10 * time.Millisecond) + } + if !replaced { + t.Fatal("second Hello did not replace peer conn") + } + if deadN1.Load() != 0 { + t.Fatalf("OnPeerDead must not fire on soft replace, got %d", deadN1.Load()) + } + // Soft replace retains Interest (no purge) + if !m2.PeerWants("n1", 42, geoCell(1, 2)) { + t.Fatal("Interest should survive soft replace") + } + + // Stale reader apply gate: Interest clear framed as oldConn must not clobber state. + emptyPay := EncodeInterest(InterestPayload{NodeID: "n1", Entries: nil}) + ok := m2.handleCtrlFrame("n1", MeshFrame{Type: MeshTypeInterest, Payload: emptyPay}, oldConn) + if !ok { + t.Fatal("stale handleCtrlFrame should return true (no death)") + } + if !m2.PeerWants("n1", 42, geoCell(1, 2)) { + t.Fatal("stale reader must not clear Interest after soft replace") + } + // Stale directory delta must also be ignored + deltaPay := EncodeTrxDelta(TrxDeltaPayload{ + OriginNodeID: "n1", Callsign: "STALE", IsATC: false, + }) + _ = m2.handleCtrlFrame("n1", MeshFrame{Type: MeshTypeTrxDelta, Payload: deltaPay}, oldConn) + // live generation apply still works + live := m2.CurrentConnForTest("n1") + if live == nil { + t.Fatal("expected live conn after replace") + } + newPay := EncodeInterest(InterestPayload{NodeID: "n1", Entries: []InterestEntry{{FreqHz: 99, ILat: 3, ILon: 4}}}) + if !m2.handleCtrlFrame("n1", MeshFrame{Type: MeshTypeInterest, Payload: newPay}, live) { + t.Fatal("live handleCtrlFrame failed") + } + if !m2.PeerWants("n1", 99, geoCell(3, 4)) { + t.Fatal("live generation must still apply Interest") + } + + // True death: close the live generation — OnPeerDead must fire, Interest cleared. + m2.ForcePeerDeathForTest("n1") + deadline = time.Now().Add(2 * time.Second) + for time.Now().Before(deadline) { + if deadN1.Load() > 0 && !m2.PeerAuthedForTest("n1") && !m2.PeerWants("n1", 99, geoCell(3, 4)) { + return + } + time.Sleep(10 * time.Millisecond) + } + t.Fatalf("true death: dead=%d authed=%v wants=%v", + deadN1.Load(), m2.PeerAuthedForTest("n1"), m2.PeerWants("n1", 99, geoCell(3, 4))) +} + +func TestHybridMesh_EnqueueDuringDeathNoDeadlock(t *testing.T) { + // Stress: EnqueueAudioRelay while ForcePeerDeath (lock-order regression). + m1, m2, _ := startHybridPair(t, "psk") + m1.ApplyInterestDirect("n2", []InterestEntry{{ + FreqHz: 122800000, + ILat: geo.CellIndex(51.5, geo.DefaultGridCellDeg), + ILon: geo.CellIndex(-0.1, geo.DefaultGridCellDeg), + }}) + + done := make(chan struct{}) + go func() { + defer close(done) + for i := 0; i < 2000; i++ { + m1.EnqueueAudioRelay(AudioRelay{ + OriginNode: "n1", Callsign: "X", SequenceCounter: uint32(i), + Audio: []byte{1, 2, 3}, + TxRadios: []RelayTxRadio{{FreqHz: 122800000, LatDeg: 51.5, LonDeg: -0.1}}, + }) + } + }() + for i := 0; i < 30; i++ { + m1.ForcePeerDeathForTest("n2") + _ = m1.WaitPeerAuthed("n2", 500*time.Millisecond) + } + select { + case <-done: + case <-time.After(10 * time.Second): + t.Fatal("enqueue+death deadlock or hang") + } + _ = m2 +} diff --git a/internal/afv/mesh_udp_voice.go b/internal/afv/mesh_udp_voice.go new file mode 100644 index 0000000..a3d8dce --- /dev/null +++ b/internal/afv/mesh_udp_voice.go @@ -0,0 +1,333 @@ +package afv + +// HybridMesh UDP voice plane: AudioRelay (type 20) only. +// Allowlist H-16, maxMeshVoiceDatagram H-12, inbound rate H-19. + +import ( + "context" + "fmt" + "net" + "strconv" + "time" +) + +// maxMeshVoiceDatagram is the only accepted UDP voice size (bytes). H-12. +// Do NOT use MaxMeshPayload (1 MiB) for UDP reads. +// Var so tests can lower the cap without exceeding OS UDP send limits. +var maxMeshVoiceDatagram = 16 << 10 // 16384 + +// maxMeshVoiceTxRadios caps TxRadios on mesh AudioRelay for voice path. +const maxMeshVoiceTxRadios = 64 + +// meshVoiceInboundRatePerSec soft cap per authed peer (datagrams/s). H-19. +const meshVoiceInboundRatePerSec = 5000 + +// resolveVoiceAllowlist resolves peer VoiceAddrs at Start (H-15 / H-16). +func (m *HybridMesh) resolveVoiceAllowlist() error { + allow := make(map[string]string) + send := make(map[string]*net.UDPAddr) + + for id, p := range m.peerCfg { + host, portStr, err := net.SplitHostPort(p.VoiceAddr) + if err != nil { + return fmt.Errorf("afv hybrid mesh peer %q VoiceAddr: %w", id, err) + } + port, err := strconv.Atoi(portStr) + if err != nil || port <= 0 || port > 65535 { + return fmt.Errorf("afv hybrid mesh peer %q voice port invalid", id) + } + + var ips []net.IP + if ip := net.ParseIP(host); ip != nil { + ips = []net.IP{ip} + } else { + resolved, err := net.LookupIP(host) + if err != nil || len(resolved) == 0 { + return fmt.Errorf("afv hybrid mesh resolve peer %q voice host %q: %w", id, host, err) + } + ips = resolved + } + + // Prefer IPv4 for send target + var sendIP net.IP + for _, ip := range ips { + if v4 := ip.To4(); v4 != nil { + sendIP = v4 + break + } + } + if sendIP == nil { + sendIP = ips[0] + } + send[id] = &net.UDPAddr{IP: sendIP, Port: port} + + for _, ip := range ips { + for _, key := range canonicalUDPKeys(ip, port) { + if other, ok := allow[key]; ok && other != id { + return fmt.Errorf("afv hybrid mesh allowlist collision key %q peers %q and %q", key, other, id) + } + allow[key] = id + } + } + } + m.allowlist = allow + m.sendVoice = send + return nil +} + +// canonicalUDPKeys returns H-16 dual keys for IPv4 / IPv4-mapped IPv6. +func canonicalUDPKeys(ip net.IP, port int) []string { + var keys []string + ip16 := ip.To16() + if ip16 != nil { + keys = append(keys, net.JoinHostPort(ip16.String(), strconv.Itoa(port))) + } + if v4 := ip.To4(); v4 != nil { + keys = append(keys, net.JoinHostPort(v4.String(), strconv.Itoa(port))) + // IPv4-mapped form explicitly + mapped := net.IP{0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0xff, 0xff, v4[0], v4[1], v4[2], v4[3]} + keys = append(keys, net.JoinHostPort(mapped.String(), strconv.Itoa(port))) + } + return keys +} + +func (m *HybridMesh) lookupAllowlist(src net.Addr) string { + ua, ok := src.(*net.UDPAddr) + if !ok { + host, portStr, err := net.SplitHostPort(src.String()) + if err != nil { + return "" + } + ip := net.ParseIP(host) + if ip == nil { + return "" + } + port, _ := strconv.Atoi(portStr) + ua = &net.UDPAddr{IP: ip, Port: port} + } + for _, key := range canonicalUDPKeys(ua.IP, ua.Port) { + if id, ok := m.allowlist[key]; ok { + return id + } + } + // also try raw JoinHostPort of To16 + if ua.IP != nil { + k := net.JoinHostPort(ua.IP.String(), strconv.Itoa(ua.Port)) + if id, ok := m.allowlist[k]; ok { + return id + } + } + return "" +} + +func (m *HybridMesh) voiceReadLoop(ctx context.Context) { + buf := make([]byte, maxMeshVoiceDatagram+1) + for { + select { + case <-m.stopCh: + return + case <-ctx.Done(): + return + default: + } + if m.voicePC == nil { + return + } + _ = m.voicePC.SetReadDeadline(time.Now().Add(1 * time.Second)) + n, src, err := m.voicePC.ReadFrom(buf) + if err != nil { + select { + case <-m.stopCh: + return + case <-ctx.Done(): + return + default: + // timeout or transient + continue + } + } + if n > maxMeshVoiceDatagram { + m.udpOversizeDrops.Add(1) + continue + } + peerID := m.lookupAllowlist(src) + if peerID == "" { + m.udpAllowDrops.Add(1) + continue + } + m.mu.RLock() + p := m.peers[peerID] + m.mu.RUnlock() + if p == nil { + m.udpAllowDrops.Add(1) + continue + } + p.mu.Lock() + authed := p.authed + p.mu.Unlock() + if !authed { + m.udpAllowDrops.Add(1) + continue + } + if !m.allowInboundRate(p) { + m.inboundRateDrops.Add(1) + continue + } + + frame, err := DecodeMeshFrameExact(buf[:n]) + if err != nil || frame.Type != MeshTypeAudioRelay { + continue + } + relay, err := DecodeAudioRelay(frame.Payload) + if err != nil || len(relay.TxRadios) > maxMeshVoiceTxRadios { + continue + } + if relay.OriginNode != peerID { + continue + } + m.voiceRx.Add(1) + p.voiceRxCount.Add(1) + p.lastVoiceUnixNano.Store(time.Now().UnixNano()) + + m.mu.RLock() + fn := m.onAudio + m.mu.RUnlock() + if fn != nil { + fn(peerID, relay) + } + } +} + +func (m *HybridMesh) allowInboundRate(p *hybridPeer) bool { + p.rateMu.Lock() + defer p.rateMu.Unlock() + now := time.Now() + if p.rateWindow.IsZero() || now.Sub(p.rateWindow) >= time.Second { + p.rateWindow = now + p.rateCount = 1 + return true + } + if p.rateCount >= meshVoiceInboundRatePerSec { + return false + } + p.rateCount++ + return true +} + +func (m *HybridMesh) voiceDrainLoop(ctx context.Context, peerID string) { + // TryRecv + short sleep so queue replace on reconnect never blocks forever + // on an abandoned channel (H-9 dual-conn replace). + for { + select { + case <-m.stopCh: + return + case <-ctx.Done(): + return + default: + } + m.mu.RLock() + p := m.peers[peerID] + sendAddr := m.sendVoice[peerID] + pc := m.voicePC + m.mu.RUnlock() + if p == nil || pc == nil { + return + } + p.mu.Lock() + q := p.voiceQ + authed := p.authed + p.mu.Unlock() + if !authed || q == nil { + select { + case <-m.stopCh: + return + case <-ctx.Done(): + return + case <-time.After(20 * time.Millisecond): + } + continue + } + job, ok := q.TryRecv() + if !ok { + select { + case <-m.stopCh: + return + case <-ctx.Done(): + return + case <-time.After(2 * time.Millisecond): + } + continue + } + if sendAddr == nil { + continue + } + _, err := pc.WriteTo(job.packet, sendAddr) + if err != nil { + m.udpSendErrs.Add(1) + m.logger.Debug("AFV hybrid mesh UDP WriteTo", "peer", peerID, "err", err) + // UDP errors never alone mark death (H-8) + continue + } + m.voiceTx.Add(1) + p.voiceTxCount.Add(1) + p.lastVoiceUnixNano.Store(time.Now().UnixNano()) + } +} + +func (m *HybridMesh) muteWatchLoop(ctx context.Context) { + // Design Issue 15: ≤1/min Debug if Interest non-empty and no voice for >30s. + ticker := time.NewTicker(30 * time.Second) + defer ticker.Stop() + // per-peer last mute log time (unix nano) + lastLog := make(map[string]int64) + const muteSilence = 30 * time.Second + const muteLogMin = time.Minute + + for { + select { + case <-m.stopCh: + return + case <-ctx.Done(): + return + case <-ticker.C: + // Snapshot under m.mu only — never nest p.mu. + type snap struct { + id string + p *hybridPeer + has bool + } + m.mu.RLock() + snaps := make([]snap, 0, len(m.peers)) + for id, p := range m.peers { + snaps = append(snaps, snap{id: id, p: p, has: len(m.peerInterest[id]) > 0}) + } + m.mu.RUnlock() + + now := time.Now() + nowN := now.UnixNano() + for _, s := range snaps { + if !s.has { + continue + } + s.p.mu.Lock() + authed := s.p.authed + s.p.mu.Unlock() + if !authed { + continue + } + lastV := s.p.lastVoiceUnixNano.Load() + // No activity yet since connect, or silence > 30s + silent := lastV == 0 || nowN-lastV > int64(muteSilence) + if !silent { + continue + } + if prev, ok := lastLog[s.id]; ok && nowN-prev < int64(muteLogMin) { + continue + } + lastLog[s.id] = nowN + m.logger.Debug("AFV hybrid mesh silent mute watch: Interest non-empty but no voice tx/rx for >30s", + "peer", s.id) + } + } + } +} diff --git a/internal/afv/mesh_udp_voice_test.go b/internal/afv/mesh_udp_voice_test.go new file mode 100644 index 0000000..478398d --- /dev/null +++ b/internal/afv/mesh_udp_voice_test.go @@ -0,0 +1,305 @@ +package afv + +import ( + "bytes" + "context" + "net" + "sync/atomic" + "testing" + "time" + + "github.com/renorris/openfsd/internal/geo" +) + +func TestDecodeMeshFrameExact(t *testing.T) { + pay := []byte("hello") + raw, err := EncodeMeshFrameBytes(MeshTypeHeartbeat, pay) + if err != nil { + t.Fatal(err) + } + fr, err := DecodeMeshFrameExact(raw) + if err != nil || fr.Type != MeshTypeHeartbeat || !bytes.Equal(fr.Payload, pay) { + t.Fatalf("%+v %v", fr, err) + } + // trailing bytes + bad := append(append([]byte{}, raw...), 0x00) + if _, err := DecodeMeshFrameExact(bad); err == nil { + t.Fatal("trailing should fail") + } + // short + if _, err := DecodeMeshFrameExact(raw[:3]); err == nil { + t.Fatal("short should fail") + } +} + +func TestHybridMesh_UDPUnknownSource(t *testing.T) { + m1, m2, _ := startHybridPair(t, "psk") + var got atomic.Int32 + m1.OnAudioRelay(func(from string, r AudioRelay) { + got.Add(1) + }) + // Send spoofed UDP to m1 voice from random port (not peer voice source port) + // Write to m1 voice listen with frame claiming origin n2 + addr, err := net.ResolveUDPAddr("udp", m1.ListenVoiceAddr()) + if err != nil { + t.Fatal(err) + } + cli, err := net.ListenPacket("udp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + defer cli.Close() + relay := AudioRelay{OriginNode: "n2", Callsign: "X", Audio: []byte{1, 2, 3}, + TxRadios: []RelayTxRadio{{FreqHz: 1, LatDeg: 0, LonDeg: 0}}} + pay, _ := EncodeAudioRelay(relay) + pkt, _ := EncodeMeshFrameBytes(MeshTypeAudioRelay, pay) + before := m1.UDPAllowDrops() + _, _ = cli.WriteTo(pkt, addr) + deadline := time.Now().Add(2 * time.Second) + for time.Now().Before(deadline) { + if m1.UDPAllowDrops() > before { + break + } + time.Sleep(10 * time.Millisecond) + } + if got.Load() != 0 { + t.Fatal("OnAudioRelay must not fire for unknown source") + } + if m1.UDPAllowDrops() <= before { + t.Fatalf("want UDPAllowDrops increase, got %d→%d", before, m1.UDPAllowDrops()) + } + _ = m2 +} + +func TestHybridMesh_UDPNotAuthed(t *testing.T) { + tcp1, voice1, tcp2, voice2 := hybridPairPorts(t) + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + + // Only start m1; peer n2 never authenticates + m1, err := NewHybridMesh(HybridMeshConfig{ + NodeID: "n1", ListenTCP: "127.0.0.1:" + itoa(tcp1), ListenVoice: "127.0.0.1:" + itoa(voice1), + Peers: []ClusterPeer{{ID: "n2", Addr: "127.0.0.1:" + itoa(tcp2), VoiceAddr: "127.0.0.1:" + itoa(voice2)}}, + PSK: "psk", + }) + if err != nil { + t.Fatal(err) + } + if err := m1.Start(ctx); err != nil { + t.Fatal(err) + } + defer m1.Stop() + + var got atomic.Int32 + m1.OnAudioRelay(func(from string, r AudioRelay) { got.Add(1) }) + + // Send from the configured peer voice port (allowlisted) but peer not authed + cli, err := net.ListenPacket("udp", "127.0.0.1:"+itoa(voice2)) + if err != nil { + // port may be free since m2 not started + t.Fatal(err) + } + defer cli.Close() + addr, _ := net.ResolveUDPAddr("udp", m1.ListenVoiceAddr()) + pay, _ := EncodeAudioRelay(AudioRelay{OriginNode: "n2", Callsign: "X", Audio: []byte{9}}) + pkt, _ := EncodeMeshFrameBytes(MeshTypeAudioRelay, pay) + _, _ = cli.WriteTo(pkt, addr) + time.Sleep(100 * time.Millisecond) + if got.Load() != 0 { + t.Fatal("not-authed peer must not deliver") + } +} + +func TestHybridMesh_IPv4MappedAllowlist(t *testing.T) { + // Unit: canonicalUDPKeys dual-key + ip := net.ParseIP("127.0.0.1") + keys := canonicalUDPKeys(ip, 17001) + foundV4, foundMapped := false, false + for _, k := range keys { + if k == "127.0.0.1:17001" { + foundV4 = true + } + if k == "[::ffff:127.0.0.1]:17001" || k == "::ffff:127.0.0.1:17001" { + foundMapped = true + } + } + if !foundV4 { + t.Fatalf("missing v4 key in %v", keys) + } + if !foundMapped { + // To16 string form of mapped + mapped := net.IP{0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0xff, 0xff, 127, 0, 0, 1} + mk := net.JoinHostPort(mapped.String(), "17001") + ok := false + for _, k := range keys { + if k == mk { + ok = true + } + } + if !ok { + t.Fatalf("missing mapped key; keys=%v want %s", keys, mk) + } + } +} + +func TestHybridMesh_InterestEmptyNoFlood(t *testing.T) { + m1, m2, _ := startHybridPair(t, "psk") + // Empty Interest: Enqueue should not send UDP + before := m1.VoiceTx() + for i := 0; i < 20; i++ { + m1.EnqueueAudioRelay(AudioRelay{ + OriginNode: "n1", Callsign: "A", SequenceCounter: uint32(i), + Audio: []byte{1, 2, 3}, + TxRadios: []RelayTxRadio{{FreqHz: 122800000, LatDeg: 51.5, LonDeg: -0.1}}, + }) + } + time.Sleep(100 * time.Millisecond) + if m1.VoiceTx() != before { + t.Fatalf("empty Interest must not flood voice tx: %d→%d", before, m1.VoiceTx()) + } + _ = m2 +} + +func TestHybridMesh_VoiceFanoutWithInterest(t *testing.T) { + m1, m2, _ := startHybridPair(t, "psk") + var got atomic.Int32 + m2.OnAudioRelay(func(from string, r AudioRelay) { + if from == "n1" && r.Callsign == "TX1" { + got.Add(1) + } + }) + // m2 advertises Interest for TX cell + lat, lon := 51.5, -0.12 + freq := uint32(122800000) + ck := geo.CellKey{ + ILat: geo.CellIndex(lat, geo.DefaultGridCellDeg), + ILon: geo.CellIndex(lon, geo.DefaultGridCellDeg), + } + m2.PublishInterest([]InterestEntry{{FreqHz: freq, ILat: ck.ILat, ILon: ck.ILon}}) + // Wait until m1 sees PeerWants + deadline := time.Now().Add(2 * time.Second) + for time.Now().Before(deadline) { + if m1.PeerWants("n2", freq, ck) { + break + } + time.Sleep(10 * time.Millisecond) + } + if !m1.PeerWants("n2", freq, ck) { + t.Fatal("PeerWants not ready") + } + m1.EnqueueAudioRelay(AudioRelay{ + OriginNode: "n1", Callsign: "TX1", SequenceCounter: 1, + Audio: []byte("opus"), + TxRadios: []RelayTxRadio{{TxID: 0, FreqHz: freq, LatDeg: lat, LonDeg: lon, HeightM: 100}}, + }) + deadline = time.Now().Add(2 * time.Second) + for time.Now().Before(deadline) { + if got.Load() > 0 { + return + } + time.Sleep(10 * time.Millisecond) + } + t.Fatal("expected AudioRelay delivery over UDP") +} + +func TestHybridMesh_UDPOversizeDrop(t *testing.T) { + // Lower cap so OS can send "oversize" datagrams on localhost. + old := maxMeshVoiceDatagram + maxMeshVoiceDatagram = 256 + t.Cleanup(func() { maxMeshVoiceDatagram = old }) + + m1, m2, _ := startHybridPair(t, "psk") + var got atomic.Int32 + m1.OnAudioRelay(func(from string, r AudioRelay) { got.Add(1) }) + + addr, err := net.ResolveUDPAddr("udp", m1.ListenVoiceAddr()) + if err != nil { + t.Fatal(err) + } + before := m1.UDPOversizeDrops() + // Source from m2's mesh voice socket (allowlisted). + m2.mu.RLock() + pc := m2.voicePC + m2.mu.RUnlock() + if pc == nil { + t.Fatal("m2 voicePC nil") + } + big := make([]byte, maxMeshVoiceDatagram+64) // 320 > 256 + if _, err := pc.WriteTo(big, addr); err != nil { + t.Fatal(err) + } + deadline := time.Now().Add(2 * time.Second) + for time.Now().Before(deadline) { + if m1.UDPOversizeDrops() > before { + break + } + time.Sleep(10 * time.Millisecond) + } + if m1.UDPOversizeDrops() <= before { + t.Fatalf("want oversize drops, got %d→%d", before, m1.UDPOversizeDrops()) + } + if got.Load() != 0 { + t.Fatal("OnAudioRelay must not fire for oversize") + } +} + +func TestHybridMesh_VoiceQueueDropOldest(t *testing.T) { + // No voice drainer consuming: unauthed peer shell only. + tcp1, voice1, tcp2, voice2 := hybridPairPorts(t) + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + m1, err := NewHybridMesh(HybridMeshConfig{ + NodeID: "n1", ListenTCP: "127.0.0.1:" + itoa(tcp1), ListenVoice: "127.0.0.1:" + itoa(voice1), + Peers: []ClusterPeer{{ID: "n2", Addr: "127.0.0.1:" + itoa(tcp2), VoiceAddr: "127.0.0.1:" + itoa(voice2)}}, + PSK: "psk", + }) + if err != nil { + t.Fatal(err) + } + if err := m1.Start(ctx); err != nil { + t.Fatal(err) + } + defer m1.Stop() + // Voice drainers run but skip when !authed — TryRecv only when authed, so queue fills. + // Actually drain loop only TryRecv when authed; unauthed never drains → drops work. + before := m1.VoiceQueueDrops() + m1.ForceEnqueueVoiceForTest("n2", meshVoiceQueueDepth+30) + if m1.VoiceQueueDrops() <= before { + t.Fatalf("want voice drops, got %d→%d", before, m1.VoiceQueueDrops()) + } + if m1.PeerVoiceQueueLen("n2") > meshVoiceQueueDepth { + t.Fatal("voice queue over depth") + } +} + +func TestHybridMesh_InboundRateLimit(t *testing.T) { + // Lower rate isn't configurable; test that under 5000/s delivers and counter type exists. + // Send a burst of 50 — all should pass at 5000/s. + m1, m2, _ := startHybridPair(t, "psk") + var got atomic.Int32 + m1.OnAudioRelay(func(from string, r AudioRelay) { got.Add(1) }) + // Need authed + allowlist source = m2 voice listen + // Use internal path: mark as if from peer by calling handle after allowlist + // Direct unit: allowInboundRate + m1.mu.RLock() + p := m1.peers["n2"] + m1.mu.RUnlock() + // simulate 5000 allows then drop + p.rateMu.Lock() + p.rateWindow = time.Now() + p.rateCount = meshVoiceInboundRatePerSec + p.rateMu.Unlock() + if m1.allowInboundRate(p) { + t.Fatal("should rate-limit when at cap") + } + // reset window + p.rateMu.Lock() + p.rateWindow = time.Time{} + p.rateCount = 0 + p.rateMu.Unlock() + if !m1.allowInboundRate(p) { + t.Fatal("should allow under cap") + } + _ = m2 + _ = got +} diff --git a/wiki/Configuration.md b/wiki/Configuration.md index 16f9ef6..d1002f9 100644 --- a/wiki/Configuration.md +++ b/wiki/Configuration.md @@ -99,13 +99,20 @@ Enabled only when the process is started with **`-afv`**. Shares `DATABASE_*` wi | `AFV_RANGE_EDGE_RATIO` | Range edge ratio for volume falloff | `0.1` | | `AFV_STATIONS_FILE` | Optional station alias file (empty → `[]` aliases) | *(empty)* | | `AFV_CROSS_COUPLE` | Multi-freq ATC cross-coupling | `true` | -| `AFV_CLUSTER_ENABLED` | AFV multi-node mesh | `false` | +| `AFV_CLUSTER_ENABLED` | AFV multi-node mesh (hybrid TCP control + UDP voice) | `false` | | `AFV_CLUSTER_NODE_ID` | AFV mesh node id | *(required when enabled)* | -| `AFV_CLUSTER_LISTEN` | AFV mesh listen | *(required when enabled)* | -| `AFV_CLUSTER_PEERS` | `id=host:port,...` (max 4 remote) | *(required when enabled)* | -| `AFV_CLUSTER_PSK` | AFV mesh shared secret | *(required when enabled)* | +| `AFV_CLUSTER_LISTEN` | **TCP control** listen `host:port` | *(required when enabled)* | +| `AFV_CLUSTER_VOICE_LISTEN` | **UDP mesh voice** listen `host:port` (separate from client `AFV_UDP_LISTEN`) | *(required when enabled)* | +| `AFV_CLUSTER_PEERS` | Remote peers: `id=host:tcpPort[/voicePort],...` (max 4). Default voice port = TCP+1 | *(required when enabled)* | +| `AFV_CLUSTER_PSK` | Shared Hello secret (constant-time compare) | *(required when enabled)* | -**Note:** `AFV_CLUSTER_ENABLED=true` currently **fails closed** without production TCP mesh in the binary (MemoryMesh is tests-only). See [Deployment](Deployment.md#optional-afv-voice). +**Multi-node notes:** + +- Mesh is **inter-node only**. Clients stay sticky to one home node for REST + client UDP (`AFV_UDP_ADVERTISE_IPV4`). +- Open **both** control TCP and voice UDP between node security groups. +- Peer grammar examples: `n2=10.0.0.2:17000` (voice `10.0.0.2:17001`), `n2=10.0.0.2:17000/17100`, `n2=[2001:db8::1]:17000`. +- `AFV_CLUSTER_VOICE_LISTEN` must differ from `AFV_UDP_LISTEN` and from local control port. Recommended multi-host: every node `LISTEN=0.0.0.0:N`, `VOICE_LISTEN=0.0.0.0:N+1`, peers `id=otherHost:N`. +- Design: `docs/design/afv-mesh-pr10b.md`. Ops: [Deployment](Deployment.md#optional-afv-voice). Minimal single-node AFV example: @@ -117,6 +124,32 @@ AFV_UDP_ADVERTISE_IPV4=voice.example.com:50000 DATABASE_SOURCE_NAME=/db/openfsd.db?_pragma=busy_timeout(5000)&_pragma=journal_mode(WAL) ``` +Two-node localhost hybrid mesh example: + +```text +# Node n1 +AFV_CLUSTER_ENABLED=true +AFV_CLUSTER_NODE_ID=n1 +AFV_CLUSTER_LISTEN=127.0.0.1:17000 +AFV_CLUSTER_VOICE_LISTEN=127.0.0.1:17001 +AFV_CLUSTER_PEERS=n2=127.0.0.1:17010/17011 +AFV_CLUSTER_PSK=dev-shared-secret +AFV_UDP_LISTEN=127.0.0.1:50000 +AFV_UDP_ADVERTISE_IPV4=127.0.0.1:50000 +AFV_API_LISTEN=127.0.0.1:8080 + +# Node n2 +AFV_CLUSTER_ENABLED=true +AFV_CLUSTER_NODE_ID=n2 +AFV_CLUSTER_LISTEN=127.0.0.1:17010 +AFV_CLUSTER_VOICE_LISTEN=127.0.0.1:17011 +AFV_CLUSTER_PEERS=n1=127.0.0.1:17000/17001 +AFV_CLUSTER_PSK=dev-shared-secret +AFV_UDP_LISTEN=127.0.0.1:50001 +AFV_UDP_ADVERTISE_IPV4=127.0.0.1:50001 +AFV_API_LISTEN=127.0.0.1:8081 +``` + ### Logging | Name | Description | Default | diff --git a/wiki/Deployment.md b/wiki/Deployment.md index 9c27784..1e7b720 100644 --- a/wiki/Deployment.md +++ b/wiki/Deployment.md @@ -182,6 +182,29 @@ AFV is **opt-in** (`-afv`). It is not started by the default Docker CMD. Share the same `DATABASE_*` as FSD so AFV authenticates against the same certificates. Optional `AFV_REQUIRE_FSD_ONLINE=true` gates voice sessions on the FSD online_users service. -Multi-node AFV mesh (`AFV_CLUSTER_*`) directory + AT relay is implemented for in-process/MemoryMesh testing; **production TCP mesh is not enabled** in the binary yet (`AFV_CLUSTER_ENABLED=true` fails closed until PR-10b). See `docs/design/afv-server.md` and `docs/design/afv-mesh-pr10.md`. +### Multi-node AFV (hybrid mesh) -Full env list: [Configuration](Configuration.md#afv-voice-optional). +Production multi-node voice uses a **hybrid mesh** (PR-10b): **TCP control** (Hello, Heartbeat, directory, Interest) + **UDP AudioRelay** between AFV processes. Clients remain sticky to one home node; mesh is inter-node only. AEAD keys never leave the home node. + +| Plane | Env | Notes | +|-------|-----|--------| +| Control TCP | `AFV_CLUSTER_LISTEN` | Hello first-frame + PSK; dial if peer id > self | +| Voice UDP | `AFV_CLUSTER_VOICE_LISTEN` | Type-20 AudioRelay only; allowlist of peer voice addrs | +| Peers | `AFV_CLUSTER_PEERS` | `id=host:tcpPort[/voicePort]` (default voice = TCP+1); max 4 remote | +| PSK | `AFV_CLUSTER_PSK` | Shared secret; required when enabled | + +**Firewall / security groups:** open **both** control TCP and mesh voice UDP between nodes. Client CryptoDTO UDP (`AFV_UDP_LISTEN`, often 50000) is **independent** of mesh voice (often N+1). + +**Sticky LB:** REST must stay on the node that minted channel keys; advertise per-node `AFV_UDP_ADVERTISE_IPV4`. Mesh does not replace client affinity. + +**Ops checklist:** + +1. Both nodes log control TCP listen + voice UDP listen at Start +2. Hello success both directions +3. Interest non-empty after clients bind +4. Cross-node AR heard (in-range clients on different homes) +5. Kill one process → peer death warn; survivors keep local A2A +6. Wrong PSK → auth fail; no voice +7. If Interest is non-empty but cross-node mute: check UDP firewall / path MTU (large clamps can IP-fragment on hostile paths) + +Full env list + two-node example: [Configuration](Configuration.md#afv-voice-optional). Design: `docs/design/afv-mesh-pr10b.md`.