# DMF Studio — Roadmap

## Next steps (in order)

### 1. WebSocket API in studio-manager
Allow the graph to be changed at runtime without restarting.
- Add a WebSocket server to `studio-manager`
- API: load/reload graph, start/stop individual nodes, query status
- `studio-manager` already has `load_graph()` — the WS layer calls it on demand
  and diffs against the running set (stop removed nodes, fork new ones)
- **Required before**: frontend, live source switching, PiP (otherwise every
  graph change is a full restart)

### 2. Processing nodes — PiP / mixer
First node that takes multiple input flows and produces an output flow.
- Uses `mxlFlowSynchronizationGroup` to align grains from two inputs
- Reference implementation: `nodes/testpattern` (writer) + `nodes/fakesink` (reader)
- Only becomes useful with the WebSocket API (so you can switch sources live)

### 3. Vue.js frontend
Visual graph editor that drives the WebSocket API.

---

## Redundancy

Key constraint: **one writer per MXL flow** — can't run two identical nodes writing
the same flow simultaneously. Redundancy lives at the pipeline level, not the node level.

### Dual pipeline on separate machines

```
Machine 1 (k8s node A)                    Machine 2 (k8s node B)
  decklinkin → [MXL] → ndiout              decklinkin → [MXL] → ndiout
       ↓                                          ↓
  (primary path)                            (backup path)
         \                                       /
          └──────→  [selector node]  ←──────────┘
                          ↓
                       ndiout (final)
```

**Selector node** — reads two input flows, monitors grain validity flags, switches
to backup when primary fails. Fits the existing node model; uses
`mxlFlowSynchronizationGroup` to watch both flows. Key processing node to build
once redundancy becomes a requirement.

MXL shared memory requires all pods in a pipeline to be co-located on the same
physical machine. Redundant pipelines naturally go on *different* machines — which
is exactly right for hardware failure redundancy.

### What k8s gives for free

- Stateless processing nodes (PiP, denoise, format convert): k8s restarts on crash,
  ~1-2 s gap — acceptable for non-critical path
- `PodDisruptionBudget`: ensures critical nodes survive cluster maintenance
- Leader election (k8s lease objects): two studio-managers, one active, one standby;
  automatic failover with no node code changes

---

## Kubernetes integration (mxl-k8s)

Source: `~/codeproj/mxl-k8s` — not official, treat as reference, not truth.

mxl-k8s is a full k8s control plane for MXL flows. Four runtime pieces:

- **Operator** (Deployment): watches `MxlReceiver` CRDs, creates `MxlFlowMirror` per target node
- **Agent** (DaemonSet): watches each node's MXL domain via `fanotify`, publishes `MxlFlow` CRDs with where flows live
- **Gateway** (DaemonSet, `hostNetwork`): drives libmxl-fabrics RDMA/TCP between nodes — zero-copy grain transfer via registered mmap regions
- **Shim** (`libmxl-intent.so`, LD_PRELOAD): intercepts `openat`/`stat`/`access` on `.mxl-flow/` paths in consumer pods; when a flow isn't local, asks the agent's UDS socket (`/run/mxl/agent.sock`) to materialize it via mirror, then retries — transparent to node code

### What changes for our nodes in k8s

**Producer pods** (decklinkin, ndiin, testpattern): **zero code change**.
- Add `hostPath: /run/mxl/domain` volume + `IPC_LOCK`, `SYS_RESOURCE` capabilities
- `NODE_CONFIG` → Pod env var (from ConfigMap)
- `MXL_DOMAIN` → `/run/mxl/domain` (standardized in k8s context)

**Consumer pods, same node**: same as above, no code change.

**Consumer pods, different node**: still no code change.
- Add `initContainer` copying `libmxl-intent.so` from shim image
- Set `LD_PRELOAD=/opt/mxl-intent/libmxl-intent.so`
- Mount `/run/mxl` (whole dir, not just `/domain`) so agent socket is accessible
- Create an `MxlReceiver` CRD pointing at the flow — operator handles the mirror plumbing

### What studio-manager becomes in k8s

Currently: fork/exec child processes. In k8s: apply/delete Pods (or Deployments) with `NODE_CONFIG` env vars. For cross-node flows: create `MxlReceiver` CRDs instead of wiring flows manually.

Same-node pipeline: all pods get `nodeAffinity: requiredDuringScheduling → same host`.
Cross-node: add shim + `MxlReceiver`; mxl-k8s handles the rest.

---

## Architecture decisions

### Separate audio and video threads in nodes

**Decision**: sink nodes (`decklinkout`, `ndiout`) and likely source nodes should
process audio and video on separate threads.

**Why**: audio and video have different timing granularities.
- Video: one grain every ~40 ms (at 25 fps) — coarse, can block
- Audio: must flow continuously at sample granularity — any stall causes dropout

In the current single-thread model, video stalls (e.g. `TOO_EARLY` retries)
pause audio too. In `decklinkout` this is especially bad — DeckLink's timestamped
audio buffer underruns if it isn't fed consistently.

**What it looks like**:
- **Audio thread**: tight loop, continuously drains MXL audio ring buffer and
  pushes to output (DeckLink `ScheduleAudioSamples` / NDI send). No video logic.
- **Video thread**: current main loop, handles grain read → process → output
  at frame rate.
- **Shared state**: only `g_running` and the output handle. No frame data crosses
  the boundary — each thread reads its own MXL flow independently.
  MXL clock keeps them in sync without explicit A/V coordination.

**When**: after the WebSocket API, when running real content and audio quality matters.
Current single-thread model is acceptable for development.