JohannesItten
ca682eaf0a
- NodeBase, Signal, FlowDef, V210 shared headers - testpattern node: SMPTE 75% color bars writer at 25fps - fakesink node: non-blocking MXL reader with per-second stats - studio-manager: FlowGraph data model, graph-driven fork/exec launcher - mxl pinned as submodule at 0ae1dc5 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
512 lines
16 KiB
Markdown
512 lines
16 KiB
Markdown
# DMF Studio — Architecture & Core Concepts
|
||
|
||
A node-based media signal processing system built on the EBU Dynamic Media Facility (DMF) architecture and the open-source MXL SDK.
|
||
|
||
---
|
||
|
||
## 1. What This Project Is
|
||
|
||
A software-defined broadcast processing environment where media workflows are built as graphs of independent processing nodes. Think of it like a virtual patchbay and signal processor combined — but running entirely in software, on standard compute.
|
||
|
||
**Example pipelines:**
|
||
```
|
||
Blackmagic SDI → [ingest node] → MXL → [denoise node] → MXL → [NDI-out node]
|
||
MXL video in → [PiP node] → MXL → [encoder node] → file
|
||
```
|
||
|
||
**Three planned user-facing layers:**
|
||
- **Nodes** — C++ programs that read/write media via MXL
|
||
- **Studio Manager** — orchestrator that launches nodes and manages the graph
|
||
- **Frontend** — Vue.js node editor (not yet built)
|
||
|
||
---
|
||
|
||
## 2. DMF Architecture Context
|
||
|
||
The EBU DMF Reference Architecture (White Paper v2.0, April 2026) defines a 6-layer stack:
|
||
|
||
```
|
||
┌─────────────────────────────┐
|
||
│ Application & UI │ ← Vue.js node editor (future)
|
||
├─────────────────────────────┤
|
||
│ Media Functions │ ← our nodes (testpattern, denoise, etc.)
|
||
├─────────────────────────────┤
|
||
│ Media Exchange (MXL) │ ← shared memory flows between nodes
|
||
├─────────────────────────────┤
|
||
│ Container Platform │ ← Docker / k8s (future)
|
||
├─────────────────────────────┤
|
||
│ Host Platform │ ← OS, drivers
|
||
├─────────────────────────────┤
|
||
│ Infrastructure │ ← physical servers, NICs, GPUs
|
||
└─────────────────────────────┘
|
||
```
|
||
|
||
Cross-cutting concerns (vertical columns): **Orchestration | Control | Monitoring | Security**
|
||
|
||
We are currently building the Media Functions layer and a simplified Orchestration stub (studio-manager). The Container Platform (k8s) comes later.
|
||
|
||
---
|
||
|
||
## 3. MXL — The Media Bus
|
||
|
||
**MXL (Media eXchange Layer)** is the open-source C++ SDK that implements the DMF Media Exchange layer.
|
||
Source: `./mxl/` (Apache 2.0 license)
|
||
Docs: `./mxl/docs/`
|
||
|
||
### Core Model
|
||
|
||
```
|
||
Domain (a directory in tmpfs)
|
||
└── <flow-uuid>.mxl-flow/
|
||
├── data ← ring buffer metadata (mmap'd)
|
||
├── flow_def.json ← NMOS IS-04 flow description
|
||
└── grains/ ← one mmap'd file per ring slot (video)
|
||
channels ← single mmap'd file (audio)
|
||
```
|
||
|
||
| Term | Meaning |
|
||
|------|---------|
|
||
| **Domain** | A directory path (typically `/dev/shm/dmf-studio`). Multiple can coexist. |
|
||
| **Flow** | A ring buffer of media data, identified by a UUID. |
|
||
| **Grain** | One unit of a discrete flow — a single video frame. |
|
||
| **Sample** | One unit of a continuous flow — one audio sample per channel. |
|
||
| **Instance** | A connection from one process to a domain (`mxlCreateInstance`). |
|
||
| **Writer** | One writer per flow (only one process can write). |
|
||
| **Reader** | Multiple readers allowed per flow. |
|
||
|
||
### Flow Types
|
||
|
||
| Type | Media | API |
|
||
|------|-------|-----|
|
||
| Discrete | video/v210, ancillary data | `OpenGrain` / `CommitGrain` / `GetGrain` |
|
||
| Continuous | audio/float32 | `OpenSamples` / `CommitSamples` / `GetSamples` |
|
||
|
||
### Inter-host (Fabrics API)
|
||
|
||
MXL also supports RDMA between hosts via libfabric (TCP/Verbs/EFA). Currently in beta. Addressing stays the same — `{host_id, domain_id, flow_id}` — the SDK handles transport transparently.
|
||
|
||
---
|
||
|
||
## 4. Node — Core Pattern
|
||
|
||
Each node is an **independent OS process**. This maps directly to a k8s Pod when containerized.
|
||
|
||
### What every node does
|
||
|
||
```
|
||
startup:
|
||
read NODE_CONFIG (JSON from env var)
|
||
read MXL_DOMAIN (path from env var)
|
||
mxlCreateInstance(domain)
|
||
create FlowWriter(s) and/or FlowReader(s)
|
||
|
||
loop until SIGTERM:
|
||
[read input grains]
|
||
process
|
||
[write output grains]
|
||
sleep to next frame time
|
||
|
||
shutdown:
|
||
mxlReleaseFlowWriter / mxlReleaseFlowReader
|
||
mxlDestroyInstance
|
||
```
|
||
|
||
### Threading per node
|
||
|
||
```
|
||
main thread
|
||
├── setup & teardown
|
||
└── processing loop (MXL read → process → MXL write)
|
||
|
||
implicit:
|
||
└── signal handler sets g_running = false → loop exits cleanly
|
||
```
|
||
|
||
No separate control thread needed for v1. For future reconfiguration, add a thread that listens on a Unix socket or WebSocket.
|
||
|
||
### NODE_CONFIG format
|
||
|
||
Passed as a JSON string in the `NODE_CONFIG` environment variable.
|
||
|
||
```json
|
||
{
|
||
"node_id": "testpattern",
|
||
"flow_id": "5fbec3b1-1b0f-417d-9059-8b94a47197ed",
|
||
"width": 1920,
|
||
"height": 1080,
|
||
"fps_num": 25,
|
||
"fps_den": 1
|
||
}
|
||
```
|
||
|
||
Nodes never care about who launched them or where other nodes are. They only care about flow UUIDs and domain path. This is what makes k8s migration trivial.
|
||
|
||
---
|
||
|
||
## 5. MXL API Patterns
|
||
|
||
### Writer (output node)
|
||
|
||
```cpp
|
||
// 1. Create instance (one per process)
|
||
mxlInstance inst = mxlCreateInstance("/dev/shm/dmf-studio", nullptr);
|
||
|
||
// 2. Create writer from NMOS IS-04 JSON
|
||
mxlFlowWriter writer{};
|
||
mxlFlowConfigInfo configInfo{};
|
||
bool created = false;
|
||
mxlCreateFlowWriter(inst, flow_def_json, nullptr, &writer, &configInfo, &created);
|
||
|
||
// MXL tells you the actual line stride — never hardcode it
|
||
uint32_t stride = configInfo.discrete.sliceSizes[0]; // bytes per V210 line
|
||
|
||
// 3. Write loop
|
||
mxlRational rate = {25, 1};
|
||
uint64_t index = mxlGetCurrentIndex(&rate);
|
||
|
||
while (running) {
|
||
mxlGrainInfo grain{};
|
||
uint8_t* buf = nullptr;
|
||
mxlFlowWriterOpenGrain(writer, index, &grain, &buf);
|
||
|
||
// buf is mmap'd memory — write directly into it
|
||
fill_frame(buf, width, height, stride);
|
||
|
||
grain.flags = 0; // MXL_GRAIN_FLAG_INVALID if you couldn't fill
|
||
mxlFlowWriterCommitGrain(writer, &grain);
|
||
|
||
// Sleep until next frame's start time
|
||
uint64_t ns = mxlGetNsUntilIndex(index + 1, &rate);
|
||
if (ns > 0 && ns < 2'000'000'000ULL)
|
||
mxlSleepForNs(ns);
|
||
|
||
index++;
|
||
}
|
||
|
||
// 4. Cleanup
|
||
mxlReleaseFlowWriter(inst, writer);
|
||
mxlDestroyInstance(inst);
|
||
```
|
||
|
||
### Reader (input node)
|
||
|
||
```cpp
|
||
mxlInstance inst = mxlCreateInstance("/dev/shm/dmf-studio", nullptr);
|
||
|
||
// Wait for the writer to start
|
||
bool active = false;
|
||
while (!active) {
|
||
mxlIsFlowActive(inst, flow_id, &active);
|
||
if (!active) mxlSleepForNs(100'000'000);
|
||
}
|
||
|
||
mxlFlowReader reader{};
|
||
mxlCreateFlowReader(inst, flow_id, nullptr, &reader);
|
||
|
||
mxlRational rate = {25, 1};
|
||
uint64_t index = mxlGetCurrentIndex(&rate);
|
||
uint64_t timeout_ns = 2'000'000'000ULL / 25; // 2 frame durations
|
||
|
||
while (running) {
|
||
mxlGrainInfo grain{};
|
||
uint8_t* buf = nullptr;
|
||
|
||
mxlStatus st = mxlFlowReaderGetGrain(reader, index, timeout_ns, &grain, &buf);
|
||
|
||
if (st == MXL_STATUS_OK) {
|
||
process(buf, grain.grainSize);
|
||
|
||
} else if (st == MXL_ERR_OUT_OF_RANGE_TOO_LATE) {
|
||
// Fell behind the ring buffer — jump to live head
|
||
index = mxlGetCurrentIndex(&rate);
|
||
continue;
|
||
|
||
} else if (st == MXL_ERR_TIMEOUT) {
|
||
// Writer may have stalled — log and continue
|
||
}
|
||
|
||
index++;
|
||
}
|
||
```
|
||
|
||
### Multi-input node (e.g. PiP — synchronize two flows)
|
||
|
||
```cpp
|
||
mxlFlowSynchronizationGroup group{};
|
||
mxlCreateFlowSynchronizationGroup(inst, &group);
|
||
mxlFlowSynchronizationGroupAddReader(group, reader_a);
|
||
mxlFlowSynchronizationGroupAddReader(group, reader_b);
|
||
|
||
mxlRational rate = {25, 1};
|
||
uint64_t nextIndex = mxlGetCurrentIndex(&rate);
|
||
|
||
while (running) {
|
||
uint64_t ts = mxlIndexToTimestamp(&rate, nextIndex);
|
||
// Blocks until BOTH flows have data at this timestamp
|
||
mxlFlowSynchronizationGroupWaitForDataAt(group, ts, timeout_ns);
|
||
|
||
// Now safe to read both
|
||
mxlFlowReaderGetGrain(reader_a, nextIndex, 0, &grain_a, &buf_a);
|
||
mxlFlowReaderGetGrain(reader_b, nextIndex, 0, &grain_b, &buf_b);
|
||
|
||
composite_pip(buf_a, buf_b, out_buf);
|
||
nextIndex++;
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## 6. V210 Video Format
|
||
|
||
MXL's video format is `video/v210` — 10-bit 4:2:2 YCbCr.
|
||
|
||
### 10-bit limited range values
|
||
|
||
| Signal | Black | White | Neutral (chroma) |
|
||
|--------|-------|-------|-----------------|
|
||
| Y | 64 | 940 | — |
|
||
| Cb/Cr | 64 | 960 | 512 |
|
||
|
||
### Packing: 6 pixels → 16 bytes (4 × 32-bit words)
|
||
|
||
In 4:2:2, pixels are paired — each pair shares one Cb and one Cr sample.
|
||
|
||
```
|
||
Word 0 [bits 9:0][19:10][29:20]: Cb(pair0) | Y(px0) | Cr(pair0)
|
||
Word 1: Y(px1) | Cb(pair1)| Y(px2)
|
||
Word 2: Cr(pair1) | Y(px3) | Cb(pair2)
|
||
Word 3: Y(px4) | Cr(pair2)| Y(px5)
|
||
```
|
||
|
||
### Line size
|
||
|
||
For 1920 pixels: `1920/6 * 16 = 5120 bytes/line`
|
||
For 1920×1080 frame: `5120 * 1080 = 5,529,600 bytes ≈ 5.3 MB/frame`
|
||
|
||
**Always get the stride from MXL** (`configInfo.discrete.sliceSizes[0]`), not hardcoded.
|
||
|
||
### SMPTE 75% color bar values (10-bit)
|
||
|
||
| Bar | Y | Cb | Cr |
|
||
|---------|-----|-----|-----|
|
||
| White | 721 | 512 | 512 |
|
||
| Yellow | 674 | 176 | 543 |
|
||
| Cyan | 581 | 589 | 176 |
|
||
| Green | 534 | 253 | 207 |
|
||
| Magenta | 251 | 771 | 817 |
|
||
| Red | 204 | 435 | 848 |
|
||
| Blue | 111 | 848 | 481 |
|
||
|
||
---
|
||
|
||
## 7. Timing System
|
||
|
||
MXL uses **TAI (International Atomic Time)**, which is ahead of UTC by ~37 seconds. All timing goes through MXL's time API — do not mix with `std::chrono::system_clock` for frame timing.
|
||
|
||
| Function | Purpose |
|
||
|----------|---------|
|
||
| `mxlGetTime()` | Current time in nanoseconds (TAI) |
|
||
| `mxlGetCurrentIndex(&rate)` | Current frame index at system time |
|
||
| `mxlGetNsUntilIndex(index, &rate)` | Nanoseconds until that index starts |
|
||
| `mxlIndexToTimestamp(&rate, index)` | Index → TAI nanoseconds |
|
||
| `mxlTimestampToIndex(&rate, ts)` | TAI nanoseconds → index |
|
||
| `mxlSleepForNs(ns)` | Relative sleep in nanoseconds |
|
||
| `mxlSleepUntil(ts)` | Sleep until a TAI timestamp |
|
||
|
||
The standard write-loop timing pattern:
|
||
```cpp
|
||
// Sleep until next frame time, then advance
|
||
uint64_t ns = mxlGetNsUntilIndex(index + 1, &rate);
|
||
if (ns > 0 && ns < 2'000'000'000ULL) mxlSleepForNs(ns);
|
||
index++;
|
||
```
|
||
|
||
---
|
||
|
||
## 8. NMOS IS-04 Flow Definition
|
||
|
||
MXL requires an NMOS IS-04-compatible JSON to define a flow when creating a writer. This JSON is also stored as `flow_def.json` in the flow directory, making flows self-describing.
|
||
|
||
Minimum required fields for `video/v210`:
|
||
|
||
```json
|
||
{
|
||
"id": "<uuid>",
|
||
"format": "urn:x-nmos:format:video",
|
||
"label": "My Node Video Output",
|
||
"description": "My Node MXL Video Flow",
|
||
"media_type": "video/v210",
|
||
"parents": [],
|
||
"grain_rate": { "numerator": 25, "denominator": 1 },
|
||
"frame_width": 1920,
|
||
"frame_height": 1080,
|
||
"interlace_mode": "progressive",
|
||
"colorspace": "BT709",
|
||
"tags": {
|
||
"urn:x-nmos:tag:grouphint/v1.0": ["My Node:Video"]
|
||
},
|
||
"components": [
|
||
{ "name": "Y", "width": 1920, "height": 1080, "bit_depth": 10 },
|
||
{ "name": "Cb", "width": 960, "height": 1080, "bit_depth": 10 },
|
||
{ "name": "Cr", "width": 960, "height": 1080, "bit_depth": 10 }
|
||
]
|
||
}
|
||
```
|
||
|
||
Generated by `shared/FlowDef.hpp → dmf::make_video_flow_def(...)`.
|
||
|
||
---
|
||
|
||
## 9. Studio Manager
|
||
|
||
The control plane. Owns:
|
||
- Flow UUID generation
|
||
- Node process lifecycle (fork/exec with env vars)
|
||
- Graph state (v1: hardcoded; future: graph.json + WebSocket API)
|
||
|
||
Does **not** touch media. All media flows through MXL between node processes.
|
||
|
||
### Process model
|
||
|
||
```
|
||
studio-manager (pid X)
|
||
├── fork → dmf-node-testpattern (pid A) env: NODE_CONFIG, MXL_DOMAIN
|
||
└── fork → dmf-node-fakesink (pid B) env: NODE_CONFIG, MXL_DOMAIN
|
||
```
|
||
|
||
On SIGINT/SIGTERM: studio-manager sends SIGTERM to children, then waits.
|
||
|
||
### v1 hardcoded graph
|
||
|
||
```
|
||
testpattern → [video flow UUID] → fakesink
|
||
```
|
||
|
||
### Future graph.json
|
||
|
||
```json
|
||
{
|
||
"nodes": [
|
||
{ "id": "tp1", "type": "testpattern", "config": { ... } },
|
||
{ "id": "sink1", "type": "fakesink", "config": { ... } }
|
||
],
|
||
"flows": [
|
||
{ "id": "<uuid>", "from": "tp1", "to": ["sink1"] }
|
||
]
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## 10. Project File Structure
|
||
|
||
```
|
||
dmf-studio-rnd/
|
||
├── mxl/ ← MXL SDK (open-source, Apache 2.0)
|
||
│ ├── lib/include/mxl/ ← public headers (mxl.h, flow.h, time.h, ...)
|
||
│ ├── lib/src/ ← C++ implementation
|
||
│ ├── docs/ ← Architecture.md, Fabrics.md, Usage.md, ...
|
||
│ └── examples/ ← Docker + k8s examples
|
||
│
|
||
├── shared/ ← header-only utilities, all targets link here
|
||
│ ├── Signal.hpp ← g_running atomic + SIGTERM/SIGINT handler
|
||
│ ├── FlowDef.hpp ← NMOS IS-04 JSON generator for video/v210
|
||
│ └── V210.hpp ← SMPTE 75% color bars V210 packer
|
||
│
|
||
├── nodes/
|
||
│ ├── testpattern/main.cpp ← generates color bars, writes MXL video flow
|
||
│ └── fakesink/main.cpp ← reads MXL video flow, logs frame statistics
|
||
│
|
||
├── studio-manager/main.cpp ← launches & monitors node processes
|
||
│
|
||
├── CMakeLists.txt ← root build: links mxl, fetches nlohmann/json
|
||
└── ARCHITECTURE.md ← this file
|
||
```
|
||
|
||
---
|
||
|
||
## 11. Build System
|
||
|
||
CMake 3.24+, C++20.
|
||
|
||
```bash
|
||
# First time
|
||
cmake -B build -DCMAKE_BUILD_TYPE=Debug
|
||
cmake --build build -j$(nproc)
|
||
|
||
# After source changes
|
||
cmake --build build -j$(nproc)
|
||
```
|
||
|
||
MXL is included via `add_subdirectory(mxl)` with tools/tests/docs/utils disabled. The core MXL library has no external dependencies.
|
||
|
||
`nlohmann/json` is fetched automatically by CMake on first configure (requires internet access once).
|
||
|
||
### Binaries produced
|
||
|
||
| Binary | Path |
|
||
|--------|------|
|
||
| Studio manager | `build/studio-manager/dmf-studio-manager` |
|
||
| TestPattern node | `build/nodes/testpattern/dmf-node-testpattern` |
|
||
| FakeSink node | `build/nodes/fakesink/dmf-node-fakesink` |
|
||
|
||
### Running
|
||
|
||
```bash
|
||
# Run everything (studio-manager launches node processes automatically)
|
||
./build/studio-manager/dmf-studio-manager
|
||
|
||
# Or run a node manually for testing
|
||
export MXL_DOMAIN=/dev/shm/dmf-studio
|
||
export NODE_CONFIG='{"node_id":"testpattern","flow_id":"5fbec3b1-1b0f-417d-9059-8b94a47197ed","width":1920,"height":1080,"fps_num":25,"fps_den":1}'
|
||
mkdir -p $MXL_DOMAIN
|
||
./build/nodes/testpattern/dmf-node-testpattern
|
||
```
|
||
|
||
On macOS, `/dev/shm` does not exist — use `/tmp/dmf-studio` instead (no RAM-backing, but works for development).
|
||
|
||
---
|
||
|
||
## 12. What Does Not Exist Yet
|
||
|
||
| Feature | Notes |
|
||
|---------|-------|
|
||
| Audio nodes | MXL continuous flow API — `OpenSamples/CommitSamples/GetSamples` |
|
||
| Real processing nodes | denoise, PiP, format converter, encoder |
|
||
| External device nodes | Blackmagic DeckLink ingest, NDI output, ST 2110 |
|
||
| Vue.js frontend | Node graph editor; talks to studio-manager via WebSocket |
|
||
| graph.json | Persist and load graph state from file |
|
||
| WebSocket API | Studio-manager exposes REST+WebSocket for frontend |
|
||
| Reconfiguration | Nodes currently only configured at startup |
|
||
| Multi-host flows | MXL Fabrics API (libfabric, beta) for RDMA between hosts |
|
||
| k8s deployment | Replace studio-manager fork/exec with Pod specs + ConfigMaps |
|
||
| Security | MXL uses UNIX file permissions; future: TLS 1.3 + OAuth2 |
|
||
|
||
---
|
||
|
||
## 13. K8s Migration Path
|
||
|
||
The architecture is designed so **node code doesn't change** when moving to k8s.
|
||
|
||
| Now | With k8s |
|
||
|-----|----------|
|
||
| `fork/exec` in studio-manager | `kubectl apply` Pod spec |
|
||
| `NODE_CONFIG` env var set by studio-manager | `NODE_CONFIG` in Pod spec `env:` from ConfigMap |
|
||
| `MXL_DOMAIN=/dev/shm/dmf-studio` | `MXL_DOMAIN=/mnt/shm` as `hostPath` volume shared between Pods on same node |
|
||
| Studio-manager monitors children | k8s restarts failed pods |
|
||
| Studio-manager generates flow UUIDs | A lightweight k8s controller or init container |
|
||
|
||
The MXL domain on k8s requires all pods in the same pipeline to run on the same physical node (same shared memory). Use node affinity rules or a DaemonSet pattern.
|
||
|
||
---
|
||
|
||
## 14. Key Dependencies Reference
|
||
|
||
| Dependency | Version | Purpose | How included |
|
||
|------------|---------|---------|--------------|
|
||
| MXL SDK | 1.2.0-dev | Shared-memory media exchange | `add_subdirectory(mxl)` |
|
||
| nlohmann/json | 3.11.3 | JSON config parsing, flow def generation | CMake FetchContent |
|
||
| C++20 stdlib | — | Threading, filesystem, chrono | System |
|
||
|
||
MXL itself uses (only needed if building MXL tools/tests): `stduuid`, `spdlog`, `fmt`, `picojson`, `CLI11`, `catch2` — all via vcpkg. Not needed for our build.
|