dmf-studio-rnd/mtl.md

# ST 2110-20 Receiver Runbook

This project uses Intel Media Transport Library (MTL) for the `2110in` node.
The current receiver path is:

```text
DeckLink ST 2110-20 -> MTL kernel backend -> MXL video/v210 flow
```

The first validated stream was:

```text
1920x1080p50
YCbCr 4:2:2 10-bit
RTP payload type 96
multicast 239.255.197.181:16388
source 192.168.0.2
receiver interface eno1np0
```

## Runtime Setup

Run these after boot before starting the receiver.

### Hugepages

MTL initializes DPDK EAL even when using the kernel socket backend, so hugepages
must exist.

```bash
sudo mkdir -p /mnt/huge
sudo mount -t hugetlbfs nodev /mnt/huge
echo 1024 | sudo tee /proc/sys/vm/nr_hugepages
grep Huge /proc/meminfo
```

What it does:

- `hugetlbfs` provides the hugepage filesystem DPDK expects.
- `nr_hugepages=1024` reserves about 2 GB with 2 MB pages.
- `HugePages_Free` should be greater than zero before running the node.

### RX Ring Size

The Mellanox interface defaulted to RX ring `1024`, which caused
`rx_out_of_buffer` increments and RTP timestamp gaps. Increase it to the card
maximum.

```bash
sudo ethtool -g eno1np0
sudo ethtool -G eno1np0 rx 8192
sudo ethtool -g eno1np0
```

What it does:

- Increases the NIC receive descriptor ring.
- Gives the driver more buffers to absorb ST 2110 burstiness and scheduler jitter.
- Prevents drops reported as `rx_out_of_buffer`.

Expected result:

```text
Current hardware settings:
RX: 8192
```

### Kernel Receive Buffers

Increase kernel receive buffering for the kernel socket backend.

```bash
sudo sysctl -w net.core.rmem_max=268435456
sudo sysctl -w net.core.rmem_default=268435456
sudo sysctl -w net.core.netdev_max_backlog=250000
```

What each setting does:

- `net.core.rmem_max`: maximum receive socket buffer size. Needed so high-rate
  UDP receivers can request/use large buffers.
- `net.core.rmem_default`: default receive socket buffer size for sockets that do
  not explicitly set a larger one.
- `net.core.netdev_max_backlog`: maximum packets queued in the kernel networking
  backlog when the kernel cannot immediately process all received packets.

These settings are especially relevant while using `kernel:<interface>` MTL
ports. DPDK or AF_XDP paths reduce dependence on this kernel socket buffering.

## Verification During A Run

Start with a clean baseline:

```bash
ethtool -S eno1np0 | grep rx_out_of_buffer
```

Watch NIC drop-related counters while `2110in` is running:

```bash
watch -n1 "ethtool -S eno1np0 | grep -E 'rx_out_of_buffer|rx_discards_phy|rx_crc_errors_phy'"
```

Expected:

```text
rx_out_of_buffer does not increase
rx_discards_phy remains 0
rx_crc_errors_phy does not increase
```

Watch node stats:

```text
incomplete=0
bad_fmt=0
mxl_open_fail=0
rtp_gap=0
rtp_dup=0
skipped=0
```

Meaning:

- `incomplete`: MTL delivered incomplete frames. Should stay zero.
- `bad_fmt`: MTL output format did not match the expected SDP-derived format.
- `mxl_open_fail`: MXL writer could not open the target grain.
- `rtp_gap`: RTP timestamp skipped one or more frame positions. Usually packet
  loss, sender frame drops, or receiver drops.
- `rtp_dup`: duplicate/backwards RTP timestamp.
- `skipped`: MXL indices skipped by timestamp mapping. Should stay zero in a
  clean run.

## Persistent Setup

### Persistent sysctl

Create `/etc/sysctl.d/99-st2110.conf`:

```bash
sudo tee /etc/sysctl.d/99-st2110.conf >/dev/null <<'EOF'
net.core.rmem_max=268435456
net.core.rmem_default=268435456
net.core.netdev_max_backlog=250000
EOF
```

Apply without reboot:

```bash
sudo sysctl --system
```

### Persistent Hugepages

Create `/etc/sysctl.d/98-hugepages.conf`:

```bash
sudo tee /etc/sysctl.d/98-hugepages.conf >/dev/null <<'EOF'
vm.nr_hugepages=1024
EOF
```

Ensure `hugetlbfs` is mounted at boot by adding this line to `/etc/fstab`:

```text
nodev /mnt/huge hugetlbfs defaults 0 0
```

Create the mount point and test:

```bash
sudo mkdir -p /mnt/huge
sudo mount /mnt/huge
mount | grep hugetlbfs
```

### Persistent RX Ring With systemd

`ethtool -G` is not persistent by itself. Use a systemd oneshot service.

Create `/etc/systemd/system/st2110-nic-tuning.service`:

```ini
[Unit]
Description=ST 2110 NIC tuning
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/sbin/ethtool -G eno1np0 rx 8192
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
```

Enable and start:

```bash
sudo systemctl daemon-reload
sudo systemctl enable --now st2110-nic-tuning.service
sudo systemctl status st2110-nic-tuning.service
```

Verify after reboot:

```bash
sudo ethtool -g eno1np0
```

Expected:

```text
Current hardware settings:
RX: 8192
```

## Receiver Config

The receiver config should carry local NIC settings plus SDP:

```json
{
  "interface": "eno1np0",
  "local_ip": "192.168.0.3",
  "sdp": "v=0\nm=video 16388 RTP/AVP 96\nc=IN IP4 239.255.197.181/255\n..."
}
```

The SDP parser currently supports:

```text
m=video
c=IN IP4
a=source-filter
a=fmtp width/height/depth/sampling/exactframerate
```

Supported video formats:

```text
YCbCr-4:2:2 depth=8  -> MTL UYVY output -> local UYVY to v210 conversion
YCbCr-4:2:2 depth=10 -> MTL V210 output -> direct copy to MXL
```

## Indexing Mode

Default:

```json
"mxl_index_mode": "rtp"
```

RTP mode maps `frame->rtp_timestamp` to the MXL grain index. This preserves sender
media cadence and exposes real RTP timestamp gaps.

Alternative:

```json
"mxl_index_mode": "live"
```

Live mode publishes near `mxlGetCurrentIndex() + mxl_latency_frames`. It keeps
sinks close to the local MXL clock but may skip indices if the source clock and
local MXL clock drift.

Keep RTP mode for normal ST 2110 ingest.

## Known Failure Signatures

### RTP gaps with `rx_out_of_buffer` increasing

Cause:

```text
Receiver-side NIC/kernel buffering loss.
```

Fix:

```text
Increase RX ring and kernel receive buffers.
```

### `mxl-gst-sink` reports TOO_EARLY after long run

Cause:

```text
Writer fell behind the MXL reader clock, usually from clock-domain drift or an
indexing policy that does not follow source timestamps.
```

Fix:

```text
Use mxl_index_mode=rtp and verify rtp_gap=0.
```

### Clean NIC counters but `rtp_gap` increases

Likely causes:

```text
Sender/source frame drops, sender media-clock discontinuity, or loss before the
receiver NIC.
```

Next debug step:

```bash
sudo tcpdump -i eno1np0 -nn -s 128 udp port 16388 -w st2110-gap.pcap
```

Inspect RTP sequence numbers and timestamps around the gap.