# M9 — GUI (Dear ImGui) ## Context The player already works (M0–M8): SDL3 window (purego), Vulkan renderer (`internal/renderer`, `christerso/vulkan-go/vk` wrapper), MXL source (`internal/source`). All rendering happens inside one render pass, one command buffer, single-flight (fence-gated). Dear ImGui is a header-only C++ library. The Go binding `cimgui-go` compiles its core at build time via cgo. cimgui-go ships SDL2 + Vulkan backends in C++, but we use SDL3 via purego, so we write custom Go backends that bridge to our existing SDL3 and Vulkan code. ImGui's core (no backend) has no SDL/Vulkan dependency — it just produces draw data (vertices, indices, commands). Our backends feed it input and render its draw data. Architecture: ``` cmd/mxl-player/main.go | +-- internal/imgui/ new package | imgui.go context, NewFrame/EndFrame, draw data | input_sdl3.go SDL3 events -> ImGui IO | backend_vulkan.go Vulkan pipeline + font atlas + draw | +-- internal/renderer/ existing, unchanged +-- internal/sdl/ existing, add a few input helpers +-- internal/source/ existing, unchanged ``` The ImGui Vulkan backend renders *inside* the existing render pass, after the video `Draw(3)`, before `EndRenderPass`. It has its own pipeline, vertex/index buffers, font texture, and descriptor set — all owned by `internal/imgui`, not `internal/renderer`. ## Steps ### M9.1 — ImGui core init + SDL3 input backend Goal: ImGui context exists, receives SDL3 input, produces draw data (not yet rendered). 1. `go get github.com/AllenDang/cimgui-go@v1.5.0` 2. Create `internal/imgui/imgui.go`: - `type Context struct { ctx *imgui.Context; io *imgui.IO }` - `func New() *Context` — creates context + IO, sets display size from `sdl.GetWindowSizeInPixels`, sets delta time each frame. - `func (c *Context) BeginFrame(dt time.Duration, winW, winH int32)` — updates IO (display size, delta time), calls `imgui.NewFrame()`. - `func (c *Context) EndFrame() *imgui.DrawData` — calls `imgui.EndFrame()` / `imgui.Render()`, returns draw data for the Vulkan backend to consume. - `func (c *Context) Destroy()` 3. Create `internal/imgui/input_sdl3.go`: - `func (c *Context) ProcessEvent(event *[128]byte)` — called from the main loop for every SDL3 event, mutates `c.io`: - `SDL_EVENT_KEY_DOWN` / `SDL_EVENT_KEY_UP` → set key in `io.AddInputCharactersUTF8` for text, set key index. - `SDL_EVENT_MOUSE_BUTTON_DOWN` / `UP` → set mouse button. - `SDL_EVENT_MOUSE_MOTION` → set mouse position. - `SDL_EVENT_MOUSE_WHEEL` → set wheel delta. - `SDL_EVENT_WINDOW_RESIZED` → set display size. - Key mapping: SDL3 scancode → ImGui key enum (a static lookup table or switch). - Mouse: ImGui wants float coordinates; SDL3 gives int32. 4. Wire into `main.go`: - After window creation: `imguiCtx := imgui.New()` - `defer imguiCtx.Destroy()` - In the event poll loop: `imguiCtx.ProcessEvent(&event)` - Before any widget code: `imguiCtx.BeginFrame(dt, winW, winH)` 5. SDL3 text input: call `SDL_StartTextInput` on window creation so `SDL_EVENT_TEXT_INPUT` events fire (add to `internal/sdl`). - SDL3 text input event: `SDL_EVENT_TEXT_INPUT = 0x303`, data is a UTF-8 string at a fixed offset in the event struct. Verify: app runs, no crash, window shows video as before. ImGui is initialized but invisible (no widgets yet, no render backend). ### M9.2 — ImGui Vulkan render backend Goal: ImGui draw data appears on screen inside the existing render pass. 1. Create `internal/imgui/backend_vulkan.go`: - `type VulkanBackend struct { ... }` - Holds: pipeline, pipeline layout, descriptor set layout/pool/set, font image + view + sampler, vertex buffer, index buffer (all `vk.*` types from the wrapper). 2. `func NewVulkanBackend(pd vk.PhysicalDevice, dev vk.Device, queue vk.Queue, cmdPool vk.CommandPool, rp vk.RenderPass, format vk.Format) (*VulkanBackend, error)`: - **Font atlas**: `imgui.GetIO().Fonts.Build()` then `GetTexDataAsRGBA32(&pixels, &w, &h)` → create `vk.CreateImage2D(w, h, Sampled|TransferDst)` → `vk.CreateImageView` → upload pixels via staging buffer + `CopyBufferToImage` → `vk.CreateSampler`. - **Descriptor set**: layout with one `DescriptorCombinedImageSampler` binding (binding 0, fragment stage) → pool → set → `vk.UpdateImageDescriptor(set, 0, fontView, fontSampler)`. - **Pipeline layout**: one set layout, push constants (ImGui's `ImDrawVert`-based push constant, 16 bytes for the scale/translate vec2s, vertex+fragment stages). - **Graphics pipeline**: `vk.CreateGraphicsPipeline` with `Blend: true`, vertex shader + fragment shader (embed ImGui's SPIR-V or compile from `imgui_impl_vulkan`'s GLSL). Vertex attributes: position (vec2), UV (vec2), color (vec4) — matches `ImDrawVert` layout. - **Vertex/index buffers**: created with a max size (e.g. 1 MB vertex, 256 KB index); recreated (larger) if draw data exceeds capacity. 3. Embed ImGui shaders: - Compile `imgui/shaders/imgui.vert` and `imgui/shaders/imgui.frag` (from Dear ImGui's repo) to SPIR-V with `glslc`. - `//go:embed` in `backend_vulkan.go`. 4. `func (b *VulkanBackend) RecordDraw(cmd vk.CommandBuffer, drawData *imgui.DrawData, frameIndex uint32)`: - Called inside the existing render pass (between video `Draw(3)` and `EndRenderPass`). - Map/`CopyToMapped` vertex + index data from `drawData` into the staging buffers (or use `vk.Map`/`Unmap` on a host-visible buffer). - `cmd.BindPipeline(b.pipeline)` - `cmd.BindDescriptorSet(b.layout, 0, b.set)` - `cmd.BindVertexBuffers(...)` - `cmd.BindIndexBuffer(...)` - `cmd.SetViewport(...)` (full window) - Push constants (scale/translate from drawData). - For each draw list, for each command: `cmd.DrawIndexed(...)`. 5. Wire into `main.go` (or `renderer.DrawFrame`): - After `cmd.Draw(3, 1, 0, 0)` (video), before `EndRenderPass`: `imguiBackend.RecordDraw(cmd, drawData, imageIndex)`. Verify: add a test widget `imgui.Text("hello")` in `BeginFrame`/ `EndFrame`. Run — you should see "hello" overlaid on the video. ### M9.3 — The GUI: config panel + stats overlay Goal: usable GUI for entering connection params and showing stats. 1. In `main.go`, between `imguiCtx.BeginFrame(...)` and `imguiCtx.EndFrame()`, build the GUI: - **Config panel** (window, shown by default): ```go imgui.Begin("Connection") imgui.InputText("Domain", &domainBuf, 256) imgui.InputText("Video UUID", &videoBuf, 256) imgui.InputText("Audio UUID", &audioBuf, 256) if imgui.Button("Connect") { // trigger source.Open / OpenSync with the entered values } imgui.End() ``` - **Stats overlay** (window, toggled by F1, no title bar, no background, top-left): ```go if showStats { imgui.SetNextWindowPos(...) imgui.Begin("Stats", &showStats, imgui.WindowFlagsNoTitleBar | imgui.WindowFlagsNoBackground) imgui.Text(fmt.Sprintf("FPS: %.1f", fps)) imgui.Text(fmt.Sprintf("Dropped: %d", dropped)) imgui.Text(fmt.Sprintf("Index: %d", shownIndex)) imgui.Text(fmt.Sprintf("Frame: %dx%d", videoW, videoH)) imgui.Text(fmt.Sprintf("Format: v210 10-bit")) imgui.End() } ``` - **Dummy controls** for future M10 flow discovery: ```go imgui.Button("List Flows") // no-op yet imgui.Button("Refresh") // no-op yet ``` 2. State: `domainBuf`, `videoBuf`, `audioBuf` are `[256]byte` buffers (ImGui's `InputText` needs a fixed buffer + capacity). Convert to Go string on "Connect". 3. "Connect" button: - Close existing source if any. - Call `source.Open` / `source.OpenSync` with the buffer values. - On error, display `imgui.Text` in red below the button. 4. F1 toggle: handle in the keydown event switch, flip `showStats`. Verify: type a domain + UUID, click Connect, video appears. Toggle stats with F1. Resize window — GUI stays usable. ### M9.4 — Wire GUI to engine + polish Goal: GUI controls the engine, not just displays. 1. **Connection lifecycle**: "Connect" button triggers source open + renderer buffer creation (if resolution changed). "Disconnect" button closes source, video freezes on last frame (or clears to black). 2. **Stats read from engine**: expose `Stats` struct from the loop (fps, dropped, index, frameTime, resolution, format). GUI reads it each frame. 3. **Freeze control** (placeholder for M10): `imgui.Checkbox("Freeze", &frozen)`. When frozen, stop calling `DrawFrame` (keep last image on screen, keep polling events + GUI). 4. **Input focus**: when ImGui wants keyboard input (text field focused), don't pass key events to the app (e.g., don't toggle fullscreen on 'F' while typing). Check `imgui.GetIO().WantCaptureKeyboard`. 5. **Mouse capture**: when ImGui wants mouse, don't let the app process mouse events. Check `imgui.GetIO().WantCaptureMouse`. 6. **DPI awareness**: scale ImGui font + style by the window's `SDL_GetDisplayContentScale` (or hardcode 1.0 for now; polish later). Verify: full workflow — launch app, enter params, connect, see video + stats, toggle stats, disconnect, reconnect. All via GUI, no CLI flags needed (though flags still work for headless/automation). ## Notes - ImGui's vertex layout (`ImDrawVert`): `pos [2]float32, uv [2]float32, col uint32` = 20 bytes. The vertex shader applies a scale/translate push constant to convert from ImGui's screen coordinates to clip space. - ImGui's fragment shader samples the font texture (and any user textures) using the UV from the vertex. Color is the vertex color, multiplied by the texture sample. - The backend pipeline uses alpha blending: `src=SRC_ALPHA, dst=ONE_MINUS_SRC_ALPHA, op=ADD`. - ImGui produces draw data *after* `EndFrame`/`Render`. The flow is: `BeginFrame` → build widgets → `EndFrame`/`Render` → get draw data → record Vulkan commands from draw data → submit. - Font atlas upload is one-time, during `NewVulkanBackend`. Vertex/index buffers are updated every frame from ImGui's draw data (host-visible, persistently mapped). - The ImGui pipeline is separate from the decode pipeline. Both render into the same render pass / framebuffer / color attachment.