DMF-Studio/DMF_STUDIO_PLAN.md

DMF Studio — Implementation Plan

Product Definition

DMF Studio is a node-based visual production platform for the Dynamic Media Facility architecture. It provides a canvas where users create, connect, and control containerized Media Functions that exchange live video via the MXL shared-memory layer.

Technology Stack

Component	Choice
Language	C++20
Build	CMake 3.24+, vcpkg
MXL SDK	Git submodule (github.com/dmf-mxl/mxl)
WebSocket	libwebsockets
JSON	nlohmann/json
Logging	spdlog
GPU compute	Vulkan (headless render + compute shaders)
SDI I/O	Blackmagic DeckLink SDK
Color conversion	libyuv
JPEG encoding	libjpeg-turbo
Testing	Catch2
Frontend (phase 4)	Vue 3 + Vue Flow + Vite + Pinia

Architecture Principles

1. Separate processes per node — crash isolation, true DMF model
2. MXL is the ONLY data path between nodes — zero-copy shared memory
3. Engine owns the graph, nodes own the media — clean separation
4. Engine generates flow IDs, nodes create MXL readers/writers — centralized routing
5. WebSocket control per node — NMOS IS-12 aligned, bidirectional
6. All nodes use libdmf-node skeleton — consistent lifecycle
7. Node binary = 1 C++ class + 1 main.cpp — minimal per-node code
8. Vulkan for GPU, CPU fallback available — GPU-accelerated but not GPU-required
9. Video only for MVP — audio deferred
10. Backend first, frontend last — validate with curl + minimal HTML before building UI

Project Structure

dmf-studio/
├── CMakeLists.txt
├── vcpkg.json
├── vcpkg-configuration.json
│
├── extern/
│   └── mxl/                              # git submodule
│
├── libs/
│   ├── dmf-node/                         # Node skeleton library (static)
│   │   ├── CMakeLists.txt
│   │   ├── include/dmf-node/
│   │   │   ├── node.hpp                  # Abstract Node interface
│   │   │   ├── port.hpp                  # PortDef, PortDirection, MediaType
│   │   │   ├── node_runner.hpp           # CLI, MXL init, control WS, main loop
│   │   │   ├── control_server.hpp        # WS server per-node
│   │   │   └── types.hpp                 # NodeId, PortId, FlowId aliases
│   │   └── src/
│   │       ├── node_runner.cpp
│   │       └── control_server.cpp
│   │
│   └── dmf-engine/                       # Engine library (static)
│       ├── CMakeLists.txt
│       ├── include/dmf-engine/
│       │   ├── graph.hpp                 # Graph model: nodes, edges
│       │   ├── flow_manager.hpp          # Flow UUID gen, NMOS flow defs
│       │   ├── process_manager.hpp       # Spawn/monitor/kill node processes
│       │   ├── api_server.hpp            # REST + WS to browser
│       │   └── types.hpp                 # Graph-level types
│       └── src/
│           ├── graph.cpp
│           ├── flow_manager.cpp
│           ├── process_manager.cpp
│           └── api_server.cpp
│
├── nodes/
│   ├── passthrough/
│   │   ├── CMakeLists.txt
│   │   └── src/
│   │       ├── main.cpp
│   │       ├── passthrough_node.hpp
│   │       └── passthrough_node.cpp
│   │
│   ├── test-source/
│   │   ├── CMakeLists.txt
│   │   └── src/
│   │       ├── main.cpp
│   │       ├── test_source_node.hpp
│   │       ├── test_source_node.cpp
│   │       ├── vulkan_context.hpp
│   │       ├── vulkan_context.cpp
│   │       ├── vulkan_renderer.hpp
│   │       ├── vulkan_renderer.cpp
│   │       └── shaders/
│   │           ├── pattern.vert
│   │           ├── pattern_bars.frag
│   │           ├── pattern_ramp.frag
│   │           ├── pattern_solid.frag
│   │           └── rgb_to_v210.comp
│   │
│   ├── decklink-in/
│   │   ├── CMakeLists.txt
│   │   └── src/
│   │       ├── main.cpp
│   │       ├── decklink_in_node.hpp
│   │       └── decklink_in_node.cpp
│   │
│   ├── decklink-out/
│   │   ├── CMakeLists.txt
│   │   └── src/
│   │       ├── main.cpp
│   │       ├── decklink_out_node.hpp
│   │       └── decklink_out_node.cpp
│   │
│   ├── preview/
│   │   ├── CMakeLists.txt
│   │   └── src/
│   │       ├── main.cpp
│   │       ├── preview_node.hpp
│   │       └── preview_node.cpp
│   │
│   └── mixer/
│       ├── CMakeLists.txt
│       └── src/
│           ├── main.cpp
│           ├── mixer_node.hpp
│           └── mixer_node.cpp
│
├── engine/
│   ├── CMakeLists.txt
│   └── src/
│       └── main.cpp
│
├── web/                                   # Phase 4 — Vue 3 frontend
│   ├── package.json
│   ├── vite.config.ts
│   ├── tsconfig.json
│   └── src/
│       ├── App.vue
│       ├── main.ts
│       ├── api/
│       │   ├── rest.ts
│       │   └── websocket.ts
│       ├── stores/
│       │   ├── graph.ts
│       │   └── connection.ts
│       ├── canvas/
│       │   ├── StudioCanvas.vue
│       │   ├── nodes/
│       │   │   ├── SourceNode.vue
│       │   │   ├── ProcessorNode.vue
│       │   │   ├── OutputNode.vue
│       │   │   └── NodeThumbnail.vue
│       │   └── edges/
│       │       └── FlowEdge.vue
│       ├── panels/
│       │   ├── NodeProperties.vue
│       │   ├── Toolbar.vue
│       │   └── StatusBar.vue
│       └── templates/
│           └── templateLoader.ts
│
├── templates/
│   ├── loopback.json
│   ├── sdi_monitor.json
│   ├── sdi_pass.json
│   └── simple_switcher.json
│
├── docker/
│   ├── Dockerfile.engine
│   ├── Dockerfile.node.test-source
│   ├── Dockerfile.node.decklink-in
│   ├── Dockerfile.node.decklink-out
│   ├── Dockerfile.node.preview
│   ├── Dockerfile.node.passthrough
│   └── docker-compose.yaml
│
└── tests/
    ├── CMakeLists.txt
    ├── test_graph.cpp
    ├── test_flow_manager.cpp
    └── test_control_protocol.cpp

Node Interface

class Node {
public:
    virtual ~Node() = default;
    virtual std::string type() const = 0;
    virtual std::vector<PortDef> input_ports() const = 0;
    virtual std::vector<PortDef> output_ports() const = 0;
    virtual void configure(const nlohmann::json& params) = 0;
    virtual void on_add_writer(const std::string& port_id, mxlFlowWriter writer) = 0;
    virtual void on_add_reader(const std::string& port_id, mxlFlowReader reader) = 0;
    virtual void on_remove_writer(const std::string& port_id) = 0;
    virtual void on_remove_reader(const std::string& port_id) = 0;
    virtual void process() = 0;
    virtual nlohmann::json status() const = 0;
};

Control Protocol (Engine ↔ Node via WebSocket)

Engine → Node:

{ "cmd": "add_writer", "flow_id": "uuid", "flow_def": { ... } }
{ "cmd": "add_reader", "flow_id": "uuid" }
{ "cmd": "remove_writer", "port_id": "out_0" }
{ "cmd": "remove_reader", "port_id": "in_0" }
{ "cmd": "configure", "params": { ... } }
{ "cmd": "shutdown" }

Node → Engine:

{ "event": "ready", "type": "test_source", "ports": { ... } }
{ "event": "flow_status", "port_id": "out_0", "writing": true, "fps": 50 }
{ "event": "error", "message": "..." }
{ "event": "grain_stats", "port_id": "out_0", "grains_written": 1234, "latency_us": 120 }

Flow Connection Sequence

User draws edge in UI (or REST API call):
  1. Engine generates flow UUID + NMOS flow definition
  2. Engine → Node A: { "cmd": "add_writer", "flow_id": ..., "flow_def": ... }
  3. Node A creates mxlFlowWriter, starts writing grains
  4. Node A → Engine: { "event": "flow_status", "writing": true }
  5. Engine → Node B: { "cmd": "add_reader", "flow_id": ... }
  6. Node B creates mxlFlowReader, starts reading grains
  7. Node B → Engine: { "event": "flow_status", "reading": true }

User removes edge:
  1. Engine → Node B: { "cmd": "remove_reader", "port_id": ... }
  2. Node B destroys mxlFlowReader
  3. Engine → Node A: { "cmd": "remove_writer", "port_id": ... }
  4. Node A destroys mxlFlowWriter

Engine REST API

GET    /api/graph                       # Current node graph
POST   /api/graph/nodes                 # Add node { type, id? }
DELETE /api/graph/nodes/:id             # Remove node
POST   /api/graph/edges                 # Connect { from_node:from_port, to_node:to_port }
DELETE /api/graph/edges/:id             # Disconnect
PUT    /api/graph/nodes/:id             # Configure node
POST   /api/graph/start                 # Start all node processes + create flows
POST   /api/graph/stop                  # Stop all
GET    /api/templates                   # List templates
POST   /api/templates/:name/load        # Load template as graph
GET    /api/nodes/types                 # List available node types

Node Designs

Test Source (Vulkan)

• Headless Vulkan: render 2 fullscreen triangles with pattern fragment shader → RGBA8 image
• Compute shader rgb_to_v210.comp: RGBA8 → BT.709 YCbCr → 4:2:2 downsample → V210 packing
• vkMapMemory → memcpy to MXL grain → commit
• Patterns: SMPTE bars, luma ramp, solid color
• Frame pacing to target 50fps
• 1 output port: video_out (V210)

Passthrough

• Read 1 MXL grain → memcpy → write 1 MXL grain
• 1 input: video_in, 1 output: video_out
• Hello-world node for validating the pipeline

Preview

• Read V210 MXL grain → libyuv convert to RGBA → downsample to 480×270 → libjpeg-turbo → JPEG
• Push JPEG frames to subscribed WS clients (throttled to ~15fps)
• 1 input: video_in

DeckLink In

• IDeckLinkInputCallback::VideoInputFrameArrived → GetBytes (V210) → memcpy to MXL grain → commit
• V210 native match = straight copy, no conversion
• 1 output: video_out
• Config: device index, connection type

DeckLink Out

• IDeckLinkOutputCallback::ScheduledFrameCompleted → read MXL grain → memcpy to DeckLink buffer → schedule frame
• 1 input: video_in
• Config: device index, connection type

Mixer

• 2 inputs: video_a, video_b (V210) → 1 output: program (V210)
• Cut: select active input
• Dissolve: blend A×(1-t) + B×t over N frames
• V210 blending: unpack 10-bit → linear blend → repack
• Config: active_input, transition_type, transition_duration_frames

Implementation Phases

Phase 1: Framework + Passthrough (Week 1-2)

Goal: Two processes exchanging video via MXL, controlled by engine

Step	Task	Test
1.1	Repo init, CMake top-level, vcpkg, MXL submodule	Build succeeds
1.2	Verify MXL SDK: mxl-gst-testsrc + mxl-gst-sink work	Video displays
1.3	Build libdmf-node: node.hpp, port.hpp, control_server, node_runner	Unit tests pass
1.4	Build passthrough node binary	Starts, connects to MXL
1.5	Build libdmf-engine: graph, flow_manager, process_manager, api_server	Unit tests pass
1.6	Build engine binary with minimal REST	curl works
1.7	Integration: engine starts 2 passthrough processes, creates flow between them	Frames flow A→B

Testing method: curl + mxl-gst-testsrc/sink as external I/O

Phase 2: Test Source + Preview (Week 3-4)

Goal: GPU-generated color bars visible in browser

Step	Task	Test
2.1	Vulkan headless context (VkDevice, queues, command pool)	Device created
2.2	Pattern fragment shaders (bars, ramp, solid) + fullscreen vert	Render to RGBA8
2.3	rgb_to_v210.comp compute shader	V210 output verified
2.4	test-source node: render → compute → MXL write	MXL flow active
2.5	preview node: MXL read → libyuv → JPEG → WS	Browser sees frames
2.6	Engine integration: test-source → preview	curl starts graph, browser shows bars

Testing method: curl to start graph, open browser to preview WS endpoint, see live color bars

Phase 3: DeckLink I/O (Week 5-6)

Goal: Real SDI signal flowing through DMF

Step	Task	Test
3.1	decklink-in node: DeckLink callback → MXL writer	SDI camera → MXL flow
3.2	decklink-out node: MXL reader → DeckLink playback	MXL flow → SDI monitor
3.3	Templates: sdi_monitor, sdi_loopback, test_output	Load and run via curl
3.4	Full pipeline: SDI in → preview → browser	Camera visible in browser

Testing method: curl to start graph, real SDI camera + monitor

Phase 4: Frontend (Week 7-8)

Goal: Visual graph editor on DMF TV

Step	Task	Test
4.1	Vue 3 + Vite project setup, Vue Flow integration	Dev server runs
4.2	Pinia stores: graph model, WS connection	State management works
4.3	StudioCanvas: drag-add nodes, draw edges, delete	Graph editing works
4.4	Custom node components: type icon, name, status, live thumbnail	Nodes render
4.5	FlowEdge: color-coded by flow status	Edges render
4.6	NodeProperties panel: dynamic config from node schema	Config changes work
4.7	Toolbar: New/Load/Save/Run/Stop + template menu	Full workflow works
4.8	StatusBar: node count, flow count, running state	Status updates live
4.9	Test on DMF TV browser	Everything renders and works

Testing method: Full end-to-end — build graph in UI, run, see live video

Phase 5: Mixer Node (Week 9-10)

Goal: DMF Studio is a production tool

Step	Task	Test
5.1	mixer node: 2-in, 1-out, cut + dissolve	Cut/dissolve work via curl
5.2	Mixer control panel in Vue: PGM/PVW buses, CUT, AUTO TRANS, T-bar	UI controls work
5.3	Template: Simple Switcher (2 sources → mixer → preview + SDI out)	Full production workflow
5.4	Polish, bug fixes, edge cases	Stable demo

vcpkg.json Dependencies

{
  "name": "dmf-studio",
  "version": "0.1.0",
  "dependencies": [
    "libwebsockets",
    "nlohmann-json",
    "spdlog",
    "fmt",
    "libyuv",
    "libjpeg-turbo",
    "catch2"
  ]
}

Vulkan: find_package(Vulkan) — system install, not vcpkg. DeckLink SDK: vendor headers linked manually, not in vcpkg.

Key Risks and Mitigations

Risk	Mitigation
MXL SDK API changes	Pin submodule to v1.0.1 tag, test before upgrading
V210 compute shader correctness	Validate against known reference frames, compare with libyuv output
DeckLink SDK availability on dev machine	Develop without hardware using test-source; DeckLink nodes only need testing on machines with cards
Single-process engine becomes bottleneck	Engine never touches media data; it only manages metadata and process lifecycle
WebSocket control latency	All control is localhost; sub-ms latency. Remote control is future work
Vulkan driver compatibility	Start with standard Vulkan 1.2; no extensions required for headless compute