@@ -0,0 +1,233 @@
# M9 — GUI (Dear ImGui)
## Context
The player already works (M0–
(`internal/renderer` , `christerso/vulkan-go/vk` wrapper), MXL source
(`internal/source` ). All rendering happens inside one render pass, one
command buffer, single-flight (fence-gated).
Dear ImGui is a header-only C++ library. The Go binding `cimgui-go`
compiles its core at build time via cgo. cimgui-go ships SDL2 + Vulkan
backends in C++, but we use SDL3 via purego, so we write custom Go
backends that bridge to our existing SDL3 and Vulkan code. ImGui's core
(no backend) has no SDL/Vulkan dependency — it just produces draw data
(vertices, indices, commands). Our backends feed it input and render its
draw data.
Architecture:
```
cmd/mxl-player/main.go
|
+-- internal/imgui/ new package
| imgui.go context, NewFrame/EndFrame, draw data
| input_sdl3.go SDL3 events -> ImGui IO
| backend_vulkan.go Vulkan pipeline + font atlas + draw
|
+-- internal/renderer/ existing, unchanged
+-- internal/sdl/ existing, add a few input helpers
+-- internal/source/ existing, unchanged
```
The ImGui Vulkan backend renders * inside * the existing render pass,
after the video `Draw(3)` , before `EndRenderPass` . It has its own
pipeline, vertex/index buffers, font texture, and descriptor set —
all owned by `internal/imgui` , not `internal/renderer` .
## Steps
### M9.1 — ImGui core init + SDL3 input backend
Goal: ImGui context exists, receives SDL3 input, produces draw data
(not yet rendered).
1. `go get github.com/AllenDang/cimgui-go@v1.5.0`
2. Create `internal/imgui/imgui.go` :
- `type Context struct { ctx *imgui.Context; io *imgui.IO }`
- `func New() *Context` — creates context + IO, sets display size
from `sdl.GetWindowSizeInPixels` , sets delta time each frame.
- `func (c *Context) BeginFrame(dt time.Duration, winW, winH int32)`
— updates IO (display size, delta time), calls `imgui.NewFrame()` .
- `func (c *Context) EndFrame() *imgui.DrawData` — calls
`imgui.EndFrame()` / `imgui.Render()` , returns draw data for the
Vulkan backend to consume.
- `func (c *Context) Destroy()`
3. Create `internal/imgui/input_sdl3.go` :
- `func (c *Context) ProcessEvent(event *[128]byte)` — called from
the main loop for every SDL3 event, mutates `c.io` :
- `SDL_EVENT_KEY_DOWN` / `SDL_EVENT_KEY_UP` → set key in
`io.AddInputCharactersUTF8` for text, set key index.
- `SDL_EVENT_MOUSE_BUTTON_DOWN` / `UP` → set mouse button.
- `SDL_EVENT_MOUSE_MOTION` → set mouse position.
- `SDL_EVENT_MOUSE_WHEEL` → set wheel delta.
- `SDL_EVENT_WINDOW_RESIZED` → set display size.
- Key mapping: SDL3 scancode → ImGui key enum (a static lookup table
or switch).
- Mouse: ImGui wants float coordinates; SDL3 gives int32.
4. Wire into `main.go` :
- After window creation: `imguiCtx := imgui.New()`
- `defer imguiCtx.Destroy()`
- In the event poll loop: `imguiCtx.ProcessEvent(&event)`
- Before any widget code: `imguiCtx.BeginFrame(dt, winW, winH)`
5. SDL3 text input: call `SDL_StartTextInput` on window creation so
`SDL_EVENT_TEXT_INPUT` events fire (add to `internal/sdl` ).
- SDL3 text input event: `SDL_EVENT_TEXT_INPUT = 0x303` , data is a
UTF-8 string at a fixed offset in the event struct.
Verify: app runs, no crash, window shows video as before. ImGui is
initialized but invisible (no widgets yet, no render backend).
### M9.2 — ImGui Vulkan render backend
Goal: ImGui draw data appears on screen inside the existing render
pass.
1. Create `internal/imgui/backend_vulkan.go` :
- `type VulkanBackend struct { ... }`
- Holds: pipeline, pipeline layout, descriptor set layout/pool/set,
font image + view + sampler, vertex buffer, index buffer (all
`vk.*` types from the wrapper).
2. `func NewVulkanBackend(pd vk.PhysicalDevice, dev vk.Device, queue
vk.Queue, cmdPool vk.CommandPool, rp vk.RenderPass, format
vk.Format) (*VulkanBackend, error)`:
- **Font atlas**: ` imgui.GetIO().Fonts.Build()` then
` GetTexDataAsRGBA32(&pixels, &w, &h)` → create
` vk.CreateImage2D(w, h, Sampled|TransferDst)` →
` vk.CreateImageView` → upload pixels via staging buffer +
` CopyBufferToImage` → ` vk.CreateSampler`.
- **Descriptor set**: layout with one
` DescriptorCombinedImageSampler` binding (binding 0, fragment
stage) → pool → set →
` vk.UpdateImageDescriptor(set, 0, fontView, fontSampler)`.
- **Pipeline layout**: one set layout, push constants
(ImGui's ` ImDrawVert`-based push constant, 16 bytes for the
scale/translate vec2s, vertex+fragment stages).
- **Graphics pipeline**: ` vk.CreateGraphicsPipeline` with
` Blend: true`, vertex shader + fragment shader (embed ImGui's
SPIR-V or compile from ` imgui_impl_vulkan`'s GLSL). Vertex
attributes: position (vec2), UV (vec2), color (vec4) — matches
` ImDrawVert` layout.
- **Vertex/index buffers**: created with a max size (e.g. 1 MB
vertex, 256 KB index); recreated (larger) if draw data exceeds
capacity.
3. Embed ImGui shaders:
- Compile ` imgui/shaders/imgui.vert` and ` imgui/shaders/imgui.frag`
(from Dear ImGui's repo) to SPIR-V with ` glslc`.
- ` //go:embed` in ` backend_vulkan.go`.
4. ` func (b *VulkanBackend) RecordDraw(cmd vk.CommandBuffer, drawData
*imgui.DrawData, frameIndex uint32)`:
- Called inside the existing render pass (between video ` Draw(3)`
and ` EndRenderPass`).
- Map/` CopyToMapped` vertex + index data from ` drawData` into the
staging buffers (or use ` vk.Map`/` Unmap` on a host-visible
buffer).
- ` cmd.BindPipeline(b.pipeline)`
- ` cmd.BindDescriptorSet(b.layout, 0, b.set)`
- ` cmd.BindVertexBuffers(...)`
- ` cmd.BindIndexBuffer(...)`
- ` cmd.SetViewport(...)` (full window)
- Push constants (scale/translate from drawData).
- For each draw list, for each command: ` cmd.DrawIndexed(...)`.
5. Wire into ` main.go` (or ` renderer.DrawFrame`):
- After ` cmd.Draw(3, 1, 0, 0)` (video), before ` EndRenderPass`:
` imguiBackend.RecordDraw(cmd, drawData, imageIndex)`.
Verify: add a test widget ` imgui.Text("hello")` in ` BeginFrame`/
` EndFrame`. Run — you should see "hello" overlaid on the video.
### M9.3 — The GUI: config panel + stats overlay
Goal: usable GUI for entering connection params and showing stats.
1. In ` main.go`, between ` imguiCtx.BeginFrame(...)` and
` imguiCtx.EndFrame()`, build the GUI:
- **Config panel** (window, shown by default):
` ``go
imgui.Begin("Connection")
imgui.InputText("Domain", &domainBuf, 256)
imgui.InputText("Video UUID", &videoBuf, 256)
imgui.InputText("Audio UUID", &audioBuf, 256)
if imgui.Button("Connect") {
// trigger source.Open / OpenSync with the entered values
}
imgui.End()
` ``
- **Stats overlay** (window, toggled by F1, no title bar, no
background, top-left):
` ``go
if showStats {
imgui.SetNextWindowPos(...)
imgui.Begin("Stats", &showStats, imgui.WindowFlagsNoTitleBar | imgui.WindowFlagsNoBackground)
imgui.Text(fmt.Sprintf("FPS: %.1f", fps))
imgui.Text(fmt.Sprintf("Dropped: %d", dropped))
imgui.Text(fmt.Sprintf("Index: %d", shownIndex))
imgui.Text(fmt.Sprintf("Frame: %dx%d", videoW, videoH))
imgui.Text(fmt.Sprintf("Format: v210 10-bit"))
imgui.End()
}
` ``
- **Dummy controls** for future M10 flow discovery:
` ``go
imgui.Button("List Flows") // no-op yet
imgui.Button("Refresh") // no-op yet
` ``
2. State: ` domainBuf`, ` videoBuf`, ` audioBuf` are ` [256]byte` buffers
(ImGui's ` InputText` needs a fixed buffer + capacity). Convert to
Go string on "Connect".
3. "Connect" button:
- Close existing source if any.
- Call ` source.Open` / ` source.OpenSync` with the buffer values.
- On error, display ` imgui.Text` in red below the button.
4. F1 toggle: handle in the keydown event switch, flip ` showStats`.
Verify: type a domain + UUID, click Connect, video appears. Toggle
stats with F1. Resize window — GUI stays usable.
### M9.4 — Wire GUI to engine + polish
Goal: GUI controls the engine, not just displays.
1. **Connection lifecycle**: "Connect" button triggers source open +
renderer buffer creation (if resolution changed). "Disconnect"
button closes source, video freezes on last frame (or clears to
black).
2. **Stats read from engine**: expose ` Stats` struct from the loop
(fps, dropped, index, frameTime, resolution, format). GUI reads it
each frame.
3. **Freeze control** (placeholder for M10): ` imgui.Checkbox("Freeze",
&frozen)`. When frozen, stop calling ` DrawFrame` (keep last image
on screen, keep polling events + GUI).
4. **Input focus**: when ImGui wants keyboard input (text field
focused), don't pass key events to the app (e.g., don't toggle
fullscreen on 'F' while typing). Check
` imgui.GetIO().WantCaptureKeyboard`.
5. **Mouse capture**: when ImGui wants mouse, don't let the app
process mouse events. Check ` imgui.GetIO().WantCaptureMouse`.
6. **DPI awareness**: scale ImGui font + style by the window's
` SDL_GetDisplayContentScale` (or hardcode 1.0 for now; polish
later).
Verify: full workflow — launch app, enter params, connect, see video +
stats, toggle stats, disconnect, reconnect. All via GUI, no CLI flags
needed (though flags still work for headless/automation).
## Notes
- ImGui's vertex layout (` ImDrawVert`): ` pos [2]float32, uv [2]float32,
col uint32` = 20 bytes. The vertex shader applies a scale/translate
push constant to convert from ImGui's screen coordinates to clip
space.
- ImGui's fragment shader samples the font texture (and any user
textures) using the UV from the vertex. Color is the vertex color,
multiplied by the texture sample.
- The backend pipeline uses alpha blending:
` src=SRC_ALPHA, dst=ONE_MINUS_SRC_ALPHA, op=ADD`.
- ImGui produces draw data *after* ` EndFrame`/` Render`. The flow is:
` BeginFrame` → build widgets → ` EndFrame`/` Render` → get draw data →
record Vulkan commands from draw data → submit.
- Font atlas upload is one-time, during ` NewVulkanBackend`. Vertex/index
buffers are updated every frame from ImGui's draw data (host-visible,
persistently mapped).
- The ImGui pipeline is separate from the decode pipeline. Both render
into the same render pass / framebuffer / color attachment.
Dmitry Sergeev