JohannesItten
139 lines
4.9 KiB
Markdown
139 lines
4.9 KiB
Markdown
# AV Combiner Node — Implementation Steps
|
||
|
||
Takes video from one upstream node and audio from another, outputs both as new MXL flows.
|
||
Does NOT need a sync group — audio and video run at different rates and are handled independently.
|
||
|
||
---
|
||
|
||
## Step 1 — Frontend (`dmf-studio-ui/src/nodeTypes.ts`)
|
||
|
||
Add the node definition. Port IDs use `video_in-in` / `video_out-out` pattern so that
|
||
`handleToPort()` produces distinct keys (`video_in_flow_id` vs `video_out_flow_id`).
|
||
|
||
```typescript
|
||
avcombiner: {
|
||
type: 'avcombiner',
|
||
label: 'AV Combiner',
|
||
ports: [
|
||
{ id: 'video_in-in', kind: 'video', direction: 'in' },
|
||
{ id: 'audio_in-in', kind: 'audio', direction: 'in' },
|
||
{ id: 'video_out-out', kind: 'video', direction: 'out' },
|
||
{ id: 'audio_out-out', kind: 'audio', direction: 'out' },
|
||
],
|
||
params: [],
|
||
},
|
||
```
|
||
|
||
Config keys the node receives:
|
||
- `video_in_flow_id.id` — input video flow UUID
|
||
- `audio_in_flow_id.id` — input audio flow UUID
|
||
- `video_out_flow_id.id` — output video flow UUID
|
||
- `audio_out_flow_id.id` — output audio flow UUID
|
||
|
||
---
|
||
|
||
## Step 2 — CMake
|
||
|
||
Create `nodes/avcombiner/CMakeLists.txt`:
|
||
|
||
```cmake
|
||
add_executable(dmf-node-avcombiner main.cpp)
|
||
target_compile_features(dmf-node-avcombiner PRIVATE cxx_std_20)
|
||
target_link_libraries(dmf-node-avcombiner PRIVATE dmf-shared)
|
||
install(TARGETS dmf-node-avcombiner RUNTIME DESTINATION bin)
|
||
```
|
||
|
||
In the root `CMakeLists.txt`, add alongside the other nodes:
|
||
|
||
```cmake
|
||
add_subdirectory(nodes/avcombiner)
|
||
```
|
||
|
||
---
|
||
|
||
## Step 3 — `nodes/avcombiner/main.cpp`
|
||
|
||
Structure (follow the same patterns as `ndiout`):
|
||
|
||
```
|
||
1. Check video_in_flow_id and video_out_flow_id both present — exit if not
|
||
2. Check audio_in_flow_id and audio_out_flow_id (optional — audio is optional)
|
||
|
||
3. Wait for video input flow active (mxlIsFlowActive loop, 100ms sleep)
|
||
4. Create video input reader (mxlCreateFlowReader)
|
||
5. Get video config info (mxlFlowReaderGetConfigInfo → video_stride)
|
||
6. Read video format from flow_def (dmf::read_video_flow_info(domain(), flow_id))
|
||
→ width, height, fps_num, fps_den
|
||
|
||
7. If has_audio:
|
||
Wait for audio input flow active
|
||
Create audio input reader
|
||
Read audio format from flow_def (dmf::read_audio_flow_info(domain(), flow_id))
|
||
→ sample_rate, channels, samples_per_grain
|
||
|
||
8. Create video output writer (mxlCreateFlowWriter with make_video_flow_def)
|
||
→ video_out_stride from configInfo.discrete.sliceSizes[0]
|
||
|
||
9. If has_audio:
|
||
Create audio output writer (mxlCreateFlowWriter with make_audio_flow_def)
|
||
|
||
10. Init clocks:
|
||
video_index = mxlGetCurrentIndex(&video_rate)
|
||
audio_index = mxlGetCurrentIndex(&audio_rate) // if has_audio
|
||
|
||
11. Main loop (same pattern as ndiout):
|
||
|
||
// Audio — non-blocking
|
||
if (has_audio) {
|
||
mxlFlowReaderGetSamplesNonBlocking(audio_in_reader, audio_index, samples_per_grain, &in_slice)
|
||
if OK:
|
||
mxlFlowWriterOpenSamples(audio_out_writer, audio_index, samples_per_grain, &out_slice)
|
||
memcpy each channel fragment (frag0, frag1 wrap pattern)
|
||
mxlFlowWriterCommitSamples
|
||
audio_index += samples_per_grain
|
||
if TOO_LATE: jump to headIndex
|
||
}
|
||
|
||
// Video — non-blocking
|
||
mxlFlowReaderGetGrainNonBlocking(video_in_reader, video_index, &grain, &in_buf)
|
||
if OK:
|
||
mxlFlowWriterOpenGrain(video_out_writer, video_index, &out_grain, &out_buf)
|
||
memcpy(out_buf, in_buf, video_out_stride * height)
|
||
mxlFlowWriterCommitGrain
|
||
video_index++
|
||
if TOO_EARLY: mxlSleepForNs(1ms)
|
||
if TOO_LATE: jump to headIndex
|
||
```
|
||
|
||
### Audio memcpy pattern (wrapped ring buffer)
|
||
|
||
MXL audio slices can wrap around the ring buffer — always copy both fragments:
|
||
|
||
```cpp
|
||
const size_t frag0 = in_slice.base.fragments[0].size / sizeof(float);
|
||
const size_t frag1 = in_slice.base.fragments[1].size / sizeof(float);
|
||
for (int c = 0; c < channels; ++c) {
|
||
const auto* src0 = reinterpret_cast<const float*>(
|
||
static_cast<const uint8_t*>(in_slice.base.fragments[0].pointer) + c * in_slice.stride);
|
||
auto* dst0 = reinterpret_cast<float*>(
|
||
static_cast<uint8_t*>(out_slice.base.fragments[0].pointer) + c * out_slice.stride);
|
||
std::memcpy(dst0, src0, frag0 * sizeof(float));
|
||
if (frag1 > 0) {
|
||
const auto* src1 = reinterpret_cast<const float*>(
|
||
static_cast<const uint8_t*>(in_slice.base.fragments[1].pointer) + c * in_slice.stride);
|
||
auto* dst1 = reinterpret_cast<float*>(
|
||
static_cast<uint8_t*>(out_slice.base.fragments[1].pointer) + c * out_slice.stride);
|
||
std::memcpy(dst1, src1, frag1 * sizeof(float));
|
||
}
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## Reference nodes
|
||
|
||
- Input flow setup (wait + reader + flow_def read): `nodes/ndiout/main.cpp`, `nodes/decklinkout/main.cpp`
|
||
- Output flow setup (writer creation): `nodes/ndiin/main.cpp`, `nodes/decklinkin/main.cpp`
|
||
- Audio memcpy pattern: `nodes/ndiout/main.cpp` lines 172–190
|
||
- `make_video_flow_def` / `make_audio_flow_def` / `read_video_flow_info` / `read_audio_flow_info`: `shared/FlowDef.hpp`
|