dmf-studio-rnd/mtl.md

ST 2110-20 Receiver Runbook

This project uses Intel Media Transport Library (MTL) for the 2110in node. The current receiver path is:

DeckLink ST 2110-20 -> MTL kernel backend -> MXL video/v210 flow

The first validated stream was:

1920x1080p50
YCbCr 4:2:2 10-bit
RTP payload type 96
multicast 239.255.197.181:16388
source 192.168.0.2
receiver interface eno1np0

Runtime Setup

Run these after boot before starting the receiver.

Hugepages

MTL initializes DPDK EAL even when using the kernel socket backend, so hugepages must exist.

sudo mkdir -p /mnt/huge
sudo mount -t hugetlbfs nodev /mnt/huge
echo 1024 | sudo tee /proc/sys/vm/nr_hugepages
grep Huge /proc/meminfo

What it does:

• hugetlbfs provides the hugepage filesystem DPDK expects.
• nr_hugepages=1024 reserves about 2 GB with 2 MB pages.
• HugePages_Free should be greater than zero before running the node.

RX Ring Size

The Mellanox interface defaulted to RX ring 1024, which caused rx_out_of_buffer increments and RTP timestamp gaps. Increase it to the card maximum.

sudo ethtool -g eno1np0
sudo ethtool -G eno1np0 rx 8192
sudo ethtool -g eno1np0

What it does:

• Increases the NIC receive descriptor ring.
• Gives the driver more buffers to absorb ST 2110 burstiness and scheduler jitter.
• Prevents drops reported as rx_out_of_buffer.

Expected result:

Current hardware settings:
RX: 8192

Kernel Receive Buffers

Increase kernel receive buffering for the kernel socket backend.

sudo sysctl -w net.core.rmem_max=268435456
sudo sysctl -w net.core.rmem_default=268435456
sudo sysctl -w net.core.netdev_max_backlog=250000

What each setting does:

• net.core.rmem_max: maximum receive socket buffer size. Needed so high-rate UDP receivers can request/use large buffers.
• net.core.rmem_default: default receive socket buffer size for sockets that do not explicitly set a larger one.
• net.core.netdev_max_backlog: maximum packets queued in the kernel networking backlog when the kernel cannot immediately process all received packets.

These settings are especially relevant while using kernel:<interface> MTL ports. DPDK or AF_XDP paths reduce dependence on this kernel socket buffering.

Verification During A Run

Start with a clean baseline:

ethtool -S eno1np0 | grep rx_out_of_buffer

Watch NIC drop-related counters while 2110in is running:

watch -n1 "ethtool -S eno1np0 | grep -E 'rx_out_of_buffer|rx_discards_phy|rx_crc_errors_phy'"

Expected:

rx_out_of_buffer does not increase
rx_discards_phy remains 0
rx_crc_errors_phy does not increase

Watch node stats:

incomplete=0
bad_fmt=0
mxl_open_fail=0
rtp_gap=0
rtp_dup=0
skipped=0

Meaning:

• incomplete: MTL delivered incomplete frames. Should stay zero.
• bad_fmt: MTL output format did not match the expected SDP-derived format.
• mxl_open_fail: MXL writer could not open the target grain.
• rtp_gap: RTP timestamp skipped one or more frame positions. Usually packet loss, sender frame drops, or receiver drops.
• rtp_dup: duplicate/backwards RTP timestamp.
• skipped: MXL indices skipped by timestamp mapping. Should stay zero in a clean run.

Persistent Setup

Persistent sysctl

Create /etc/sysctl.d/99-st2110.conf:

sudo tee /etc/sysctl.d/99-st2110.conf >/dev/null <<'EOF'
net.core.rmem_max=268435456
net.core.rmem_default=268435456
net.core.netdev_max_backlog=250000
EOF

Apply without reboot:

sudo sysctl --system

Persistent Hugepages

Create /etc/sysctl.d/98-hugepages.conf:

sudo tee /etc/sysctl.d/98-hugepages.conf >/dev/null <<'EOF'
vm.nr_hugepages=1024
EOF

Ensure hugetlbfs is mounted at boot by adding this line to /etc/fstab:

nodev /mnt/huge hugetlbfs defaults 0 0

Create the mount point and test:

sudo mkdir -p /mnt/huge
sudo mount /mnt/huge
mount | grep hugetlbfs

Persistent RX Ring With systemd

ethtool -G is not persistent by itself. Use a systemd oneshot service.

Create /etc/systemd/system/st2110-nic-tuning.service:

[Unit]
Description=ST 2110 NIC tuning
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/sbin/ethtool -G eno1np0 rx 8192
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable --now st2110-nic-tuning.service
sudo systemctl status st2110-nic-tuning.service

Verify after reboot:

sudo ethtool -g eno1np0

Expected:

Current hardware settings:
RX: 8192

Receiver Config

The receiver config should carry local NIC settings plus SDP:

{
  "interface": "eno1np0",
  "local_ip": "192.168.0.3",
  "sdp": "v=0\nm=video 16388 RTP/AVP 96\nc=IN IP4 239.255.197.181/255\n..."
}

The SDP parser currently supports:

m=video
c=IN IP4
a=source-filter
a=fmtp width/height/depth/sampling/exactframerate

Supported video formats:

YCbCr-4:2:2 depth=8  -> MTL UYVY output -> local UYVY to v210 conversion
YCbCr-4:2:2 depth=10 -> MTL V210 output -> direct copy to MXL

Indexing Mode

Default:

"mxl_index_mode": "rtp"

RTP mode maps frame->rtp_timestamp to the MXL grain index. This preserves sender media cadence and exposes real RTP timestamp gaps.

Alternative:

"mxl_index_mode": "live"

Live mode publishes near mxlGetCurrentIndex() + mxl_latency_frames. It keeps sinks close to the local MXL clock but may skip indices if the source clock and local MXL clock drift.

Keep RTP mode for normal ST 2110 ingest.

Known Failure Signatures

RTP gaps with rx_out_of_buffer increasing

Cause:

Receiver-side NIC/kernel buffering loss.

Fix:

Increase RX ring and kernel receive buffers.

mxl-gst-sink reports TOO_EARLY after long run

Cause:

Writer fell behind the MXL reader clock, usually from clock-domain drift or an
indexing policy that does not follow source timestamps.

Fix:

Use mxl_index_mode=rtp and verify rtp_gap=0.

Clean NIC counters but rtp_gap increases

Likely causes:

Sender/source frame drops, sender media-clock discontinuity, or loss before the
receiver NIC.

Next debug step:

sudo tcpdump -i eno1np0 -nn -s 128 udp port 16388 -w st2110-gap.pcap

Inspect RTP sequence numbers and timestamps around the gap.