stable version of 2110-20 RX with simple SDP parser

2026-07-16 19:37:52 +03:00
parent 1caeeee17e
commit 0feab360f5
4 changed files with 643 additions and 91 deletions
@@ -1,81 +1,317 @@
-# ST 2110-20 receiver with Intel MTL
+# ST 2110-20 Receiver Runbook

-The MTL pattern is structurally identical to `DeckLinkReceiver` — callback-based, MTL manages frame buffers, you drain them into MXL.
+This project uses Intel Media Transport Library (MTL) for the `2110in` node.
+The current receiver path is:

-## MTL initialization (once per process)
-
-```cpp
-mtl_init_params p{};
-p.ports[MTL_PORT_P] = "0000:31:00.0"; // PCI address or netdev for AF_XDP
-p.num_ports = 1;
-p.flags = MTL_FLAG_BIND_NUMA;
-// For AF_XDP mode (no DPDK hugepages needed):
-p.transport = MTL_TRANSPORT_AF_XDP;
-
-mtl_handle dev = mtl_init(&p);
+```text
+DeckLink ST 2110-20 -> MTL kernel backend -> MXL video/v210 flow
 ```

-## Create RX session
+The first validated stream was:

-```cpp
-st20_rx_ops ops{};
-ops.port.num_port = 1;
-memcpy(ops.port.sip_addr[MTL_PORT_P], multicast_ip, 4); // join this group
-ops.port.udp_port[MTL_PORT_P] = 20000;
-ops.width  = 1280;
-ops.height = 720;
-ops.fps    = ST_FPS_P25;
-ops.fmt    = ST20_FMT_YUV_422_10BIT; // RFC 4175 packed — see note below
-ops.framebuff_cnt = 3;
-ops.notify_frame_ready = on_frame_ready; // your callback
-ops.priv = this;
-
-st20_rx_handle rx = st20_rx_create(dev, &ops);
+```text
+1920x1080p50
+YCbCr 4:2:2 10-bit
+RTP payload type 96
+multicast 239.255.197.181:16388
+source 192.168.0.2
+receiver interface eno1np0
 ```

-## Callback — replaces `wait_for_frame()`
+## Runtime Setup

-```cpp
-static int on_frame_ready(void* priv, void* frame, st20_rx_frame_meta* meta) {
-    auto* self = static_cast<St2110InNode*>(priv);
-    // frame points to a complete assembled frame — MTL did the reassembly
-    const uint64_t index = mxlGetCurrentIndex(&self->video_rate);
-    mxlGrainInfo grain{};
-    uint8_t* buf = nullptr;
-    if (mxlFlowWriterOpenGrain(self->writer, index, &grain, &buf) == MXL_STATUS_OK) {
-        memcpy(buf, frame, self->frame_size); // ← see format note
-        grain.validSlices = grain.totalSlices;
-        mxlFlowWriterCommitGrain(self->writer, &grain);
-    }
-    st20_rx_put_framebuff(self->rx, frame); // return buffer to MTL pool
-    return 0;
+Run these after boot before starting the receiver.
+
+### Hugepages
+
+MTL initializes DPDK EAL even when using the kernel socket backend, so hugepages
+must exist.
+
+```bash
+sudo mkdir -p /mnt/huge
+sudo mount -t hugetlbfs nodev /mnt/huge
+echo 1024 | sudo tee /proc/sys/vm/nr_hugepages
+grep Huge /proc/meminfo
+```
+
+What it does:
+
+- `hugetlbfs` provides the hugepage filesystem DPDK expects.
+- `nr_hugepages=1024` reserves about 2 GB with 2 MB pages.
+- `HugePages_Free` should be greater than zero before running the node.
+
+### RX Ring Size
+
+The Mellanox interface defaulted to RX ring `1024`, which caused
+`rx_out_of_buffer` increments and RTP timestamp gaps. Increase it to the card
+maximum.
+
+```bash
+sudo ethtool -g eno1np0
+sudo ethtool -G eno1np0 rx 8192
+sudo ethtool -g eno1np0
+```
+
+What it does:
+
+- Increases the NIC receive descriptor ring.
+- Gives the driver more buffers to absorb ST 2110 burstiness and scheduler jitter.
+- Prevents drops reported as `rx_out_of_buffer`.
+
+Expected result:
+
+```text
+Current hardware settings:
+RX: 8192
+```
+
+### Kernel Receive Buffers
+
+Increase kernel receive buffering for the kernel socket backend.
+
+```bash
+sudo sysctl -w net.core.rmem_max=268435456
+sudo sysctl -w net.core.rmem_default=268435456
+sudo sysctl -w net.core.netdev_max_backlog=250000
+```
+
+What each setting does:
+
+- `net.core.rmem_max`: maximum receive socket buffer size. Needed so high-rate
+  UDP receivers can request/use large buffers.
+- `net.core.rmem_default`: default receive socket buffer size for sockets that do
+  not explicitly set a larger one.
+- `net.core.netdev_max_backlog`: maximum packets queued in the kernel networking
+  backlog when the kernel cannot immediately process all received packets.
+
+These settings are especially relevant while using `kernel:<interface>` MTL
+ports. DPDK or AF_XDP paths reduce dependence on this kernel socket buffering.
+
+## Verification During A Run
+
+Start with a clean baseline:
+
+```bash
+ethtool -S eno1np0 | grep rx_out_of_buffer
+```
+
+Watch NIC drop-related counters while `2110in` is running:
+
+```bash
+watch -n1 "ethtool -S eno1np0 | grep -E 'rx_out_of_buffer|rx_discards_phy|rx_crc_errors_phy'"
+```
+
+Expected:
+
+```text
+rx_out_of_buffer does not increase
+rx_discards_phy remains 0
+rx_crc_errors_phy does not increase
+```
+
+Watch node stats:
+
+```text
+incomplete=0
+bad_fmt=0
+mxl_open_fail=0
+rtp_gap=0
+rtp_dup=0
+skipped=0
+```
+
+Meaning:
+
+- `incomplete`: MTL delivered incomplete frames. Should stay zero.
+- `bad_fmt`: MTL output format did not match the expected SDP-derived format.
+- `mxl_open_fail`: MXL writer could not open the target grain.
+- `rtp_gap`: RTP timestamp skipped one or more frame positions. Usually packet
+  loss, sender frame drops, or receiver drops.
+- `rtp_dup`: duplicate/backwards RTP timestamp.
+- `skipped`: MXL indices skipped by timestamp mapping. Should stay zero in a
+  clean run.
+
+## Persistent Setup
+
+### Persistent sysctl
+
+Create `/etc/sysctl.d/99-st2110.conf`:
+
+```bash
+sudo tee /etc/sysctl.d/99-st2110.conf >/dev/null <<'EOF'
+net.core.rmem_max=268435456
+net.core.rmem_default=268435456
+net.core.netdev_max_backlog=250000
+EOF
+```
+
+Apply without reboot:
+
+```bash
+sudo sysctl --system
+```
+
+### Persistent Hugepages
+
+Create `/etc/sysctl.d/98-hugepages.conf`:
+
+```bash
+sudo tee /etc/sysctl.d/98-hugepages.conf >/dev/null <<'EOF'
+vm.nr_hugepages=1024
+EOF
+```
+
+Ensure `hugetlbfs` is mounted at boot by adding this line to `/etc/fstab`:
+
+```text
+nodev /mnt/huge hugetlbfs defaults 0 0
+```
+
+Create the mount point and test:
+
+```bash
+sudo mkdir -p /mnt/huge
+sudo mount /mnt/huge
+mount | grep hugetlbfs
+```
+
+### Persistent RX Ring With systemd
+
+`ethtool -G` is not persistent by itself. Use a systemd oneshot service.
+
+Create `/etc/systemd/system/st2110-nic-tuning.service`:
+
+```ini
+[Unit]
+Description=ST 2110 NIC tuning
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+ExecStart=/usr/sbin/ethtool -G eno1np0 rx 8192
+RemainAfterExit=yes
+
+[Install]
+WantedBy=multi-user.target
+```
+
+Enable and start:
+
+```bash
+sudo systemctl daemon-reload
+sudo systemctl enable --now st2110-nic-tuning.service
+sudo systemctl status st2110-nic-tuning.service
+```
+
+Verify after reboot:
+
+```bash
+sudo ethtool -g eno1np0
+```
+
+Expected:
+
+```text
+Current hardware settings:
+RX: 8192
+```
+
+## Receiver Config
+
+The receiver config should carry local NIC settings plus SDP:
+
+```json
+{
+  "interface": "eno1np0",
+  "local_ip": "192.168.0.3",
+  "sdp": "v=0\nm=video 16388 RTP/AVP 96\nc=IN IP4 239.255.197.181/255\n..."
 }
 ```

-## Format conversion: RFC 4175 vs V210
+The SDP parser currently supports:

-ST 2110-20 wire format is RFC 4175 packed 10-bit — **not V210**. They encode the same YCbCr 4:2:2 10-bit data differently:
- RFC 4175: 5 bytes per 2 pixels, big-endian packed
- V210: 4 bytes per 3 luma + 2 chroma, little-endian with padding bits
-
-MXL in this project uses V210. Check MTL's `output_fmt` option — newer MTL versions support `ST_FRAME_FMT_V210` as the output format, which means MTL does the conversion internally. If your version doesn't have it, you'll need a small RFC4175→V210 conversion step before the `memcpy`.
-
-Check `st_frame_fmt` enum in MTL headers for what's available.
-
-## Teardown
-
-```cpp
-st20_rx_free(rx);
-mtl_uninit(dev);
+```text
+m=video
+c=IN IP4
+a=source-filter
+a=fmtp width/height/depth/sampling/exactframerate
 ```

-## Main loop
+Supported video formats:

-The callback is called from MTL's internal thread (like DeckLink's), so the MXL write happens inside the callback rather than in the main loop. The main loop just blocks on `g_running`:
-
-```cpp
-while (dmf::g_running.load(std::memory_order_relaxed))
-    mxlSleepForNs(10'000'000);
+```text
+YCbCr-4:2:2 depth=8  -> MTL UYVY output -> local UYVY to v210 conversion
+YCbCr-4:2:2 depth=10 -> MTL V210 output -> direct copy to MXL
 ```

-MTL drives the pacing, exactly like DeckLink hardware does.
+## Indexing Mode
+
+Default:
+
+```json
+"mxl_index_mode": "rtp"
+```
+
+RTP mode maps `frame->rtp_timestamp` to the MXL grain index. This preserves sender
+media cadence and exposes real RTP timestamp gaps.
+
+Alternative:
+
+```json
+"mxl_index_mode": "live"
+```
+
+Live mode publishes near `mxlGetCurrentIndex() + mxl_latency_frames`. It keeps
+sinks close to the local MXL clock but may skip indices if the source clock and
+local MXL clock drift.
+
+Keep RTP mode for normal ST 2110 ingest.
+
+## Known Failure Signatures
+
+### RTP gaps with `rx_out_of_buffer` increasing
+
+Cause:
+
+```text
+Receiver-side NIC/kernel buffering loss.
+```
+
+Fix:
+
+```text
+Increase RX ring and kernel receive buffers.
+```
+
+### `mxl-gst-sink` reports TOO_EARLY after long run
+
+Cause:
+
+```text
+Writer fell behind the MXL reader clock, usually from clock-domain drift or an
+indexing policy that does not follow source timestamps.
+```
+
+Fix:
+
+```text
+Use mxl_index_mode=rtp and verify rtp_gap=0.
+```
+
+### Clean NIC counters but `rtp_gap` increases
+
+Likely causes:
+
+```text
+Sender/source frame drops, sender media-clock discontinuity, or loss before the
+receiver NIC.
+```
+
+Next debug step:
+
+```bash
+sudo tcpdump -i eno1np0 -nn -s 128 udp port 16388 -w st2110-gap.pcap
+```
+
+Inspect RTP sequence numbers and timestamps around the gap.