mxl-multiviewer/conv.md

ChatGPT

Yes — those lines go into your cleanup section before destroying the Vulkan device.

Find this cleanup area near the end:

vkDestroyPipeline( device, graphicsPipeline, nullptr );

vkDestroyPipelineLayout( device, pipelineLayout, nullptr );

Add the vertex buffer cleanup BEFORE destroying the device:

vkDestroyBuffer( device,

[1/2] Building CXX object CMakeFiles/mxl_multiviewer.dir/main.cpp.o FAILED: [code=1] CMakeFiles/mxl_multiviewer.dir/main.cpp.o /usr/bin/clang++ -I/usr/include/fribidi -I/usr/include/libdrm -I/usr/include/libdecor-0 -std=gnu++20 -Wall -Wextra -Wpedantic -MD -MT CMakeFiles/mxl_multiviewer.dir/main.cpp.o -MF CMakeFiles/mxl_multiviewer.dir/main.cpp.o.d -o CMakeFiles/mxl_multiviewer.dir/main.cpp.o -c /home/itten/mxl-multiviewer/main.cpp /home/itten/mxl-multiviewer/main.cpp:910:54: error: implicit instantiation of undefined template 'std::array<VkVertexInputAttributeDescription, 2>' 910 | std::array<VkVertexInputAttributeDescription, 2> attributeDescriptions{}; | ^ /usr/lib/gcc/x86_64-linux-gnu/15/../../../../include/c++/15/bits/stl_pair.h:99:12: note: template is declared here 99 | struct array; | ^ /home/itten/mxl-multiviewer/main.cpp:1252:33: error: redefinition of 'allocInfo' with a different type: 'VkCommandBufferAllocateInfo' vs 'VkMemoryAllocateInfo' 1252 | VkCommandBufferAllocateInfo allocInfo{}; | ^ /home/itten/mxl-multiviewer/main.cpp:1121:26: note: previous definition is here 1121 | VkMemoryAllocateInfo allocInfo{}; | ^ /home/itten/mxl-multiviewer/main.cpp:1257:15: error: no member named 'commandPool' in 'VkMemoryAllocateInfo' 1257 | allocInfo.commandPool = | ~~~~~~~~~ ^ /home/itten/mxl-multiviewer/main.cpp:1260:15: error: no member named 'level' in 'VkMemoryAllocateInfo' 1260 | allocInfo.level = | ~~~~~~~~~ ^ /home/itten/mxl-multiviewer/main.cpp:1263:15: error: no member named 'commandBufferCount' in 'VkMemoryAllocateInfo' 1263 | allocInfo.commandBufferCount = | ~~~~~~~~~ ^ /home/itten/mxl-multiviewer/main.cpp:1268:9: error: no matching function for call to 'vkAllocateCommandBuffers' 1268 | if (vkAllocateCommandBuffers( | ^~~~~~~~~~~~~~~~~~~~~~~~ /usr/include/vulkan/vulkan_core.h:4862:32: note: candidate function not viable: no known conversion from 'VkMemoryAllocateInfo *' to 'const VkCommandBufferAllocateInfo ' for 2nd argument 4862 | VKAPI_ATTR VkResult VKAPI_CALL vkAllocateCommandBuffers( | ^ 4863 | VkDevice device, 4864 | const VkCommandBufferAllocateInfo pAllocateInfo, | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /home/itten/mxl-multiviewer/main.cpp:1609:54: error: expected ';' after expression 1609 | vkFreeMemory(device, vertexBufferMemory, nullptr) | ^ | ; 7 errors generated. ninja: build stopped: subcommand failed.

You have 3 simple issues.

1. Missing include

Add this at the top:

#include

That fixes:

undefined template 'std::array'

2. allocInfo name conflict

You already used this name here:

VkMemoryAllocateInfo allocInfo{};

Later you also have:

VkCommandBufferAllocateInfo allocInfo{};

Rename the vertex buffer one.

Change this:

VkMemoryAllocateInfo allocInfo{}; allocInfo.sType = VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO; allocInfo.allocationSize = memRequirements.size; allocInfo.memoryTypeIndex = findMemoryType( physicalDevice, memRequirements.memoryTypeBits, VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | VK_MEMORY_PROPERTY_HOST_COHERENT_BIT );

if (vkAllocateMemory(device, &allocInfo, nullptr, &vertexBufferMemory) != VK_SUCCESS)

to:

VkMemoryAllocateInfo vertexAllocInfo{}; vertexAllocInfo.sType = VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO; vertexAllocInfo.allocationSize = memRequirements.size; vertexAllocInfo.memoryTypeIndex = findMemoryType( physicalDevice, memRequirements.memoryTypeBits, VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | VK_MEMORY_PROPERTY_HOST_COHERENT_BIT );

if (vkAllocateMemory(device, &vertexAllocInfo, nullptr, &vertexBufferMemory) != VK_SUCCESS)

3. Missing semicolon

Change:

vkFreeMemory(device, vertexBufferMemory, nullptr)

to:

vkFreeMemory(device, vertexBufferMemory, nullptr);

Then rebuild:

ninja ./mxl_multiviewer

itten@nettop:~/mxl-multiviewer/build$ ./mxl_multiviewer Selected GPU: AMD Radeon Graphics (RADV RENOIR) Graphics queue: 0 Present queue: 0 Logical device created Swapchain created Swapchain image count: 4 Image views created Render pass created Shaders loaded Graphics pipeline created Framebuffers created Command pool created Command buffers allocated Segmentation fault (core dumped) ./mxl_multiviewer

looks like it works. I have a rectangle with gradient

Great. That means Step 7.1 is complete.

You now have:

vertex buffer + UV attributes + quad rendering + fragment shader using UVs

This is very close to video rendering.

Next step:

CPU generated image → Vulkan texture → sampler → descriptor set → draw textured quad

That is the real foundation for MXL frames.

so go next step

Have a rectangle with checkerboard

Excellent — this is a major milestone.

You now have:

CPU pixels → staging buffer → GPU image → image view → sampler → descriptor set → textured quad

That is the core of video rendering.

For MXL later, this part changes only here:

std::vector<uint32_t> pixels(...)

Instead of generated checkerboard pixels, you will copy bytes from the latest MXL frame.

Next step should be:

Step 7.3 update texture every frame

That simulates live video and prepares the exact mechanism needed for MXL feeds.

go

now checkers constatly moves

Perfect. That means you now have a working live texture upload loop:

CPU pixels change every frame → staging buffer → GPU texture → shader samples latest texture → rendered quad updates live

This is basically the first version of your future MXL path.

Next logical step:

Step 7.4 replace generated pixels with fake “feed frame” abstraction

Instead of writing pixels directly in the render loop, create something like:

struct VideoFrame { uint32_t width; uint32_t height; std::vector<uint32_t> pixels; };

Then later:

fake frame generator → MXL frame reader

So the renderer does not care where frames come from.

go next

Now we’ll separate “video source” from “renderer”. This is important because later MXL should plug into the same interface without changing Vulkan rendering code.

Step 7.4: create a fake feed abstraction.

1. Add struct above main()

struct VideoFrame { uint32_t width = 0; uint32_t height = 0; std::vector<uint32_t> pixels; };

Add fake feed generator:

VideoFrame generateFakeFrame( uint32_t width, uint32_t height, uint32_t frameCounter) { VideoFrame frame; frame.width = width; frame.height = height; frame.pixels.resize(width * height);

for (uint32_t y = 0; y < height; ++y)
{
    for (uint32_t x = 0; x < width; ++x)
    {
        const uint32_t movingX =
            (x + frameCounter) % width;

        const bool checker =
            ((movingX / 32) % 2) ^ ((y / 32) % 2);

        uint8_t r = checker ? 255 : 40;
        uint8_t g = checker ? 80 : 180;
        uint8_t b = checker ? 40 : 255;
        uint8_t a = 255;

        frame.pixels[y * width + x] =
            (a << 24) |
            (b << 16) |
            (g << 8) |
            r;
    }
}

return frame;

}

2. Remove direct pixel generation from the render loop

Replace the moving-checker code inside your main loop with this:

VideoFrame frame = generateFakeFrame( TEXTURE_WIDTH, TEXTURE_HEIGHT, frameCounter );

void* frameTextureData = nullptr;

vkMapMemory( device, stagingBufferMemory, 0, imageSize, 0, &frameTextureData );

std::memcpy( frameTextureData, frame.pixels.data(), static_cast<size_t>(imageSize) );

vkUnmapMemory(device, stagingBufferMemory);

Keep the existing Vulkan upload part after it:

transitionImageLayout(...); copyBufferToImage(...); transitionImageLayout(...);

3. Remove old pixels dependency from loop

You can still keep initial texture creation using pixels, but better replace the first texture fill with:

VideoFrame initialFrame = generateFakeFrame( TEXTURE_WIDTH, TEXTURE_HEIGHT, 0 );

Then upload:

std::memcpy( textureData, initialFrame.pixels.data(), static_cast<size_t>(imageSize) );

After that, you can delete the old manual std::vector<uint32_t> pixels(...) checkerboard generation block.

Expected result should look the same: moving checkerboard.

But architecturally you now have:

fake feed generator ↓ VideoFrame ↓ renderer upload path ↓ GPU texture

Later we replace only this:

VideoFrame frame = generateFakeFrame(...);

with something like:

VideoFrame frame = mxlFeed.getLatestFrame();

That is the correct separation.

3. Remove old pixels dependency from loop

describe it

Great — Step 7.4 works.

You now have:

fake feed source → VideoFrame → staging buffer → GPU texture → textured quad

Next step should be multiple fake feeds:

Feed 1 → texture 1 → tile 1 Feed 2 → texture 2 → tile 2 Feed 3 → texture 3 → tile 3 Feed 4 → texture 4 → tile 4

That is the first real multiview prototype.

go

Now we’ll move from “video renderer” to the first real multiview prototype: several independent fake feeds, each with its own texture, rendered into different tiles.

Step 8: multiple fake feeds / multiple textured quads.

Do it in 2 small parts.

1. Replace single quad vertices with 4 tiles

Replace your quadVertices with this:

std::vector makeQuad( float x0, float y0, float x1, float y1) { return { {{x0, y0}, {0.0f, 1.0f}}, {{x1, y0}, {1.0f, 1.0f}}, {{x1, y1}, {1.0f, 0.0f}},

    {{x0, y0}, {0.0f, 1.0f}},
    {{x1, y1}, {1.0f, 0.0f}},
    {{x0, y1}, {0.0f, 0.0f}},
};

}

Then create 4 quads:

std::vector quadVertices;

auto q1 = makeQuad(-1.0f, -1.0f, 0.0f, 0.0f); auto q2 = makeQuad( 0.0f, -1.0f, 1.0f, 0.0f); auto q3 = makeQuad(-1.0f, 0.0f, 0.0f, 1.0f); auto q4 = makeQuad( 0.0f, 0.0f, 1.0f, 1.0f);

quadVertices.insert(quadVertices.end(), q1.begin(), q1.end()); quadVertices.insert(quadVertices.end(), q2.begin(), q2.end()); quadVertices.insert(quadVertices.end(), q3.begin(), q3.end()); quadVertices.insert(quadVertices.end(), q4.begin(), q4.end());

Important: because this is no longer const, create it inside main() before vertex buffer creation, not globally.

So remove the old global:

const std::vector quadVertices = ...

2. For now, draw same texture 4 times

Your existing command recording probably has:

vkCmdDraw( commandBuffers[i], static_cast<uint32_t>(quadVertices.size()), 1, 0, 0 );

Keep it as-is.

Expected result:

same moving checkerboard appears in 4 tiles

This proves your renderer can draw multiple tiles.

After that works, next step is:

4 fake feeds 4 textures 4 descriptor sets draw each tile with its own texture

That will be the first true multiview.

[1/2] Building CXX object CMakeFiles/mxl_multiviewer.dir/main.cpp.o FAILED: [code=1] CMakeFiles/mxl_multiviewer.dir/main.cpp.o /usr/bin/clang++ -I/usr/include/fribidi -I/usr/include/libdrm -I/usr/include/libdecor-0 -std=gnu++20 -Wall -Wextra -Wpedantic -MD -MT CMakeFiles/mxl_multiviewer.dir/main.cpp.o -MF CMakeFiles/mxl_multiviewer.dir/main.cpp.o.d -o CMakeFiles/mxl_multiviewer.dir/main.cpp.o -c /home/itten/mxl-multiviewer/main.cpp /home/itten/mxl-multiviewer/main.cpp:244:1: error: unknown type name 'quadVertices' 244 | quadVertices.insert(quadVertices.end(), q1.begin(), q1.end()); | ^ /home/itten/mxl-multiviewer/main.cpp:244:13: error: cannot use dot operator on a type 244 | quadVertices.insert(quadVertices.end(), q1.begin(), q1.end()); | ^ /home/itten/mxl-multiviewer/main.cpp:245:1: error: unknown type name 'quadVertices' 245 | quadVertices.insert(quadVertices.end(), q2.begin(), q2.end()); | ^ /home/itten/mxl-multiviewer/main.cpp:245:13: error: cannot use dot operator on a type 245 | quadVertices.insert(quadVertices.end(), q2.begin(), q2.end()); | ^ /home/itten/mxl-multiviewer/main.cpp:246:1: error: unknown type name 'quadVertices' 246 | quadVertices.insert(quadVertices.end(), q3.begin(), q3.end()); | ^ /home/itten/mxl-multiviewer/main.cpp:246:13: error: cannot use dot operator on a type 246 | quadVertices.insert(quadVertices.end(), q3.begin(), q3.end()); | ^ /home/itten/mxl-multiviewer/main.cpp:247:1: error: unknown type name 'quadVertices' 247 | quadVertices.insert(quadVertices.end(), q4.begin(), q4.end()); | ^ /home/itten/mxl-multiviewer/main.cpp:247:13: error: cannot use dot operator on a type 247 | quadVertices.insert(quadVertices.end(), q4.begin(), q4.end()); | ^ 8 errors generated.

So now it running checkerboard, with smaller tiles

Great — Step 8.1 works.

You now have:

one texture → one vertex buffer with 4 quads → 4 rendered tiles

This is the first visual multiview layout.

Next step:

4 fake feeds 4 different textures 4 descriptor sets draw each tile separately

Then it becomes a real multiview prototype instead of one texture repeated four times.

go

Now we’ll make it a true multiview: four independent fake feeds, each with its own GPU texture and descriptor set, drawn into its own tile.

Step 8.2: 4 textures, 4 fake feeds, 4 tiles.

Right now you draw all 4 quads in one vkCmdDraw() using one descriptor set.

Now change to:

bind descriptor set 1 → draw quad 1 bind descriptor set 2 → draw quad 2 bind descriptor set 3 → draw quad 3 bind descriptor set 4 → draw quad 4

Each quad has 6 vertices.

1. Add feed count

Near your constants:

const uint32_t FEED_COUNT = 4; const uint32_t VERTICES_PER_QUAD = 6;

2. Make fake frames look different per feed

Replace your generator with this version:

VideoFrame generateFakeFrame( uint32_t width, uint32_t height, uint32_t frameCounter, uint32_t feedIndex) { VideoFrame frame; frame.width = width; frame.height = height; frame.pixels.resize(width * height);

for (uint32_t y = 0; y < height; ++y)
{
    for (uint32_t x = 0; x < width; ++x)
    {
        const uint32_t movingX =
            (x + frameCounter * (feedIndex + 1)) % width;

        const bool checker =
            ((movingX / 32) % 2) ^ ((y / 32) % 2);

        uint8_t r = 0;
        uint8_t g = 0;
        uint8_t b = 0;
        uint8_t a = 255;

        if (feedIndex == 0)
        {
            r = checker ? 255 : 80;
            g = checker ? 40 : 20;
            b = checker ? 40 : 20;
        }
        else if (feedIndex == 1)
        {
            r = checker ? 40 : 20;
            g = checker ? 255 : 80;
            b = checker ? 40 : 20;
        }
        else if (feedIndex == 2)
        {
            r = checker ? 40 : 20;
            g = checker ? 40 : 20;
            b = checker ? 255 : 80;
        }
        else
        {
            r = checker ? 255 : 80;
            g = checker ? 255 : 80;
            b = checker ? 40 : 20;
        }

        frame.pixels[y * width + x] =
            (a << 24) |
            (b << 16) |
            (g << 8) |
            r;
    }
}

return frame;

}

3. Replace single texture variables with arrays

Find your single variables:

VkBuffer stagingBuffer; VkDeviceMemory stagingBufferMemory;

VkImage textureImage; VkDeviceMemory textureImageMemory;

VkImageView textureImageView; VkSampler textureSampler;

VkDescriptorSet descriptorSet;

Replace with:

std::array<VkBuffer, FEED_COUNT> stagingBuffers{}; std::array<VkDeviceMemory, FEED_COUNT> stagingBufferMemories{};

std::array<VkImage, FEED_COUNT> textureImages{}; std::array<VkDeviceMemory, FEED_COUNT> textureImageMemories{};

std::array<VkImageView, FEED_COUNT> textureImageViews{}; std::array<VkSampler, FEED_COUNT> textureSamplers{};

std::array<VkDescriptorSet, FEED_COUNT> descriptorSets{};

Make sure you have:

#include

4. Change descriptor pool count

Find:

poolSize.descriptorCount = 1; ... poolInfoDesc.maxSets = 1;

Change to:

poolSize.descriptorCount = FEED_COUNT; poolInfoDesc.maxSets = FEED_COUNT;

5. Allocate 4 descriptor sets

Replace this:

descriptorAllocInfo.descriptorSetCount = 1; descriptorAllocInfo.pSetLayouts = &descriptorSetLayout;

with:

std::array<VkDescriptorSetLayout, FEED_COUNT> descriptorSetLayouts{};

for (uint32_t i = 0; i < FEED_COUNT; ++i) { descriptorSetLayouts[i] = descriptorSetLayout; }

descriptorAllocInfo.descriptorSetCount = FEED_COUNT; descriptorAllocInfo.pSetLayouts = descriptorSetLayouts.data();

And replace:

vkAllocateDescriptorSets( device, &descriptorAllocInfo, &descriptorSet )

with:

vkAllocateDescriptorSets( device, &descriptorAllocInfo, descriptorSets.data() )

6. Create textures in a loop

Replace your current single texture creation block with this loop:

VkDeviceSize imageSize = TEXTURE_WIDTH * TEXTURE_HEIGHT * 4;

for (uint32_t feed = 0; feed < FEED_COUNT; ++feed) { VideoFrame initialFrame = generateFakeFrame( TEXTURE_WIDTH, TEXTURE_HEIGHT, 0, feed );

createBuffer(
    device,
    physicalDevice,
    imageSize,
    VK_BUFFER_USAGE_TRANSFER_SRC_BIT,
    VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT |
    VK_MEMORY_PROPERTY_HOST_COHERENT_BIT,
    stagingBuffers[feed],
    stagingBufferMemories[feed]
);

void* textureData = nullptr;

vkMapMemory(
    device,
    stagingBufferMemories[feed],
    0,
    imageSize,
    0,
    &textureData
);

std::memcpy(
    textureData,
    initialFrame.pixels.data(),
    static_cast<size_t>(imageSize)
);

vkUnmapMemory(device, stagingBufferMemories[feed]);

createImage(
    device,
    physicalDevice,
    TEXTURE_WIDTH,
    TEXTURE_HEIGHT,
    VK_FORMAT_R8G8B8A8_UNORM,
    VK_IMAGE_TILING_OPTIMAL,
    VK_IMAGE_USAGE_TRANSFER_DST_BIT |
    VK_IMAGE_USAGE_SAMPLED_BIT,
    VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT,
    textureImages[feed],
    textureImageMemories[feed]
);

transitionImageLayout(
    device,
    commandPool,
    graphicsQueue,
    textureImages[feed],
    VK_FORMAT_R8G8B8A8_UNORM,
    VK_IMAGE_LAYOUT_UNDEFINED,
    VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL
);

copyBufferToImage(
    device,
    commandPool,
    graphicsQueue,
    stagingBuffers[feed],
    textureImages[feed],
    TEXTURE_WIDTH,
    TEXTURE_HEIGHT
);

transitionImageLayout(
    device,
    commandPool,
    graphicsQueue,
    textureImages[feed],
    VK_FORMAT_R8G8B8A8_UNORM,
    VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL,
    VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL
);

VkImageViewCreateInfo textureViewInfo{};
textureViewInfo.sType = VK_STRUCTURE_TYPE_IMAGE_VIEW_CREATE_INFO;
textureViewInfo.image = textureImages[feed];
textureViewInfo.viewType = VK_IMAGE_VIEW_TYPE_2D;
textureViewInfo.format = VK_FORMAT_R8G8B8A8_UNORM;
textureViewInfo.subresourceRange.aspectMask = VK_IMAGE_ASPECT_COLOR_BIT;
textureViewInfo.subresourceRange.baseMipLevel = 0;
textureViewInfo.subresourceRange.levelCount = 1;
textureViewInfo.subresourceRange.baseArrayLayer = 0;
textureViewInfo.subresourceRange.layerCount = 1;

if (vkCreateImageView(
        device,
        &textureViewInfo,
        nullptr,
        &textureImageViews[feed]) != VK_SUCCESS)
{
    throw std::runtime_error("Failed to create texture image view");
}

VkSamplerCreateInfo samplerInfo{};
samplerInfo.sType = VK_STRUCTURE_TYPE_SAMPLER_CREATE_INFO;
samplerInfo.magFilter = VK_FILTER_LINEAR;
samplerInfo.minFilter = VK_FILTER_LINEAR;
samplerInfo.addressModeU = VK_SAMPLER_ADDRESS_MODE_REPEAT;
samplerInfo.addressModeV = VK_SAMPLER_ADDRESS_MODE_REPEAT;
samplerInfo.addressModeW = VK_SAMPLER_ADDRESS_MODE_REPEAT;
samplerInfo.anisotropyEnable = VK_FALSE;
samplerInfo.borderColor = VK_BORDER_COLOR_INT_OPAQUE_BLACK;
samplerInfo.unnormalizedCoordinates = VK_FALSE;
samplerInfo.compareEnable = VK_FALSE;
samplerInfo.mipmapMode = VK_SAMPLER_MIPMAP_MODE_LINEAR;

if (vkCreateSampler(
        device,
        &samplerInfo,
        nullptr,
        &textureSamplers[feed]) != VK_SUCCESS)
{
    throw std::runtime_error("Failed to create texture sampler");
}

}

std::cout << "Feed textures created" << std::endl;

7. Update descriptor sets in a loop

After allocating descriptor sets, replace the single descriptor update with:

for (uint32_t feed = 0; feed < FEED_COUNT; ++feed) { VkDescriptorImageInfo imageInfo{}; imageInfo.imageLayout = VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL; imageInfo.imageView = textureImageViews[feed]; imageInfo.sampler = textureSamplers[feed];

VkWriteDescriptorSet descriptorWrite{};
descriptorWrite.sType = VK_STRUCTURE_TYPE_WRITE_DESCRIPTOR_SET;
descriptorWrite.dstSet = descriptorSets[feed];
descriptorWrite.dstBinding = 0;
descriptorWrite.dstArrayElement = 0;
descriptorWrite.descriptorType =
    VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER;
descriptorWrite.descriptorCount = 1;
descriptorWrite.pImageInfo = &imageInfo;

vkUpdateDescriptorSets(
    device,
    1,
    &descriptorWrite,
    0,
    nullptr
);

}

std::cout << "Descriptor sets created" << std::endl;

Important: descriptor update must happen after texture image views and samplers are created. 8. Update all textures every frame

Replace single-frame update in main loop with:

++frameCounter;

for (uint32_t feed = 0; feed < FEED_COUNT; ++feed) { VideoFrame frame = generateFakeFrame( TEXTURE_WIDTH, TEXTURE_HEIGHT, frameCounter, feed );

void* frameTextureData = nullptr;

vkMapMemory(
    device,
    stagingBufferMemories[feed],
    0,
    imageSize,
    0,
    &frameTextureData
);

std::memcpy(
    frameTextureData,
    frame.pixels.data(),
    static_cast<size_t>(imageSize)
);

vkUnmapMemory(device, stagingBufferMemories[feed]);

transitionImageLayout(
    device,
    commandPool,
    graphicsQueue,
    textureImages[feed],
    VK_FORMAT_R8G8B8A8_UNORM,
    VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL,
    VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL
);

copyBufferToImage(
    device,
    commandPool,
    graphicsQueue,
    stagingBuffers[feed],
    textureImages[feed],
    TEXTURE_WIDTH,
    TEXTURE_HEIGHT
);

transitionImageLayout(
    device,
    commandPool,
    graphicsQueue,
    textureImages[feed],
    VK_FORMAT_R8G8B8A8_UNORM,
    VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL,
    VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL
);

}

9. Record draw commands per feed

Replace your single descriptor bind + draw with this:

VkBuffer vertexBuffers[] = { vertexBuffer }; VkDeviceSize offsets[] = { 0 };

vkCmdBindVertexBuffers( commandBuffers[i], 0, 1, vertexBuffers, offsets );

for (uint32_t feed = 0; feed < FEED_COUNT; ++feed) { vkCmdBindDescriptorSets( commandBuffers[i], VK_PIPELINE_BIND_POINT_GRAPHICS, pipelineLayout, 0, 1, &descriptorSets[feed], 0, nullptr );

vkCmdDraw(
    commandBuffers[i],
    VERTICES_PER_QUAD,
    1,
    feed * VERTICES_PER_QUAD,
    0
);

}

This means:

feed 0 uses vertices 0–5 feed 1 uses vertices 6–11 feed 2 uses vertices 12–17 feed 3 uses vertices 18–23

10. Cleanup arrays

Replace single cleanup with:

for (uint32_t feed = 0; feed < FEED_COUNT; ++feed) { vkDestroySampler(device, textureSamplers[feed], nullptr);

vkDestroyImageView(device, textureImageViews[feed], nullptr);

vkDestroyImage(device, textureImages[feed], nullptr);
vkFreeMemory(device, textureImageMemories[feed], nullptr);

vkDestroyBuffer(device, stagingBuffers[feed], nullptr);
vkFreeMemory(device, stagingBufferMemories[feed], nullptr);

}

Keep:

vkDestroyDescriptorPool(device, descriptorPool, nullptr); vkDestroyDescriptorSetLayout(device, descriptorSetLayout, nullptr);

Expected result:

4 tiles 4 different moving checkerboards different colors / movement speed

That is your first real software multiview prototype.

Describe what and where change from step 6

From Step 6, you had:

one triangle / one pipeline / one draw call

Now you are changing it into:

4 quads 4 textures 4 descriptor sets 4 draw calls

Here is the clean map of changes.

1. Top of file: add constants

Near texture constants:

const uint32_t TEXTURE_WIDTH = 256; const uint32_t TEXTURE_HEIGHT = 256;

const uint32_t FEED_COUNT = 4; const uint32_t VERTICES_PER_QUAD = 6;

2. Above main(): add frame + fake feed

struct VideoFrame { uint32_t width = 0; uint32_t height = 0; std::vector<uint32_t> pixels; };

And:

VideoFrame generateFakeFrame( uint32_t width, uint32_t height, uint32_t frameCounter, uint32_t feedIndex) { // generates colored moving checkerboard }

3. Above main(): replace triangle vertices with quad helper

Keep Vertex:

struct Vertex { float pos[2]; float uv[2]; };

Add:

std::vector makeQuad( float x0, float y0, float x1, float y1) { return { {{x0, y0}, {0.0f, 1.0f}}, {{x1, y0}, {1.0f, 1.0f}}, {{x1, y1}, {1.0f, 0.0f}},

    {{x0, y0}, {0.0f, 1.0f}},
    {{x1, y1}, {1.0f, 0.0f}},
    {{x0, y1}, {0.0f, 0.0f}},
};

}

4. Inside main(): create 4 quads before vertex buffer

Place this before vertex buffer creation:

std::vector quadVertices;

auto q1 = makeQuad(-1.0f, -1.0f, 0.0f, 0.0f); auto q2 = makeQuad( 0.0f, -1.0f, 1.0f, 0.0f); auto q3 = makeQuad(-1.0f, 0.0f, 0.0f, 1.0f); auto q4 = makeQuad( 0.0f, 0.0f, 1.0f, 1.0f);

quadVertices.insert(quadVertices.end(), q1.begin(), q1.end()); quadVertices.insert(quadVertices.end(), q2.begin(), q2.end()); quadVertices.insert(quadVertices.end(), q3.begin(), q3.end()); quadVertices.insert(quadVertices.end(), q4.begin(), q4.end());

5. Pipeline setup: replace empty vertex input

Step 6 had empty vertex input because the triangle was generated in shader.

Now use:

VkVertexInputBindingDescription bindingDescription{}; bindingDescription.binding = 0; bindingDescription.stride = sizeof(Vertex); bindingDescription.inputRate = VK_VERTEX_INPUT_RATE_VERTEX;

with two attributes:

location 0 = position location 1 = uv

6. Pipeline layout: add descriptor set layout

Before pipeline layout creation, add:

VkDescriptorSetLayout descriptorSetLayout;

with binding:

binding = 0 type = VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER stage = VK_SHADER_STAGE_FRAGMENT_BIT

Then pipeline layout must include it:

pipelineLayoutInfo.setLayoutCount = 1; pipelineLayoutInfo.pSetLayouts = &descriptorSetLayout;

7. Shaders: use textured shaders instead of triangle shaders

Instead of:

triangle.vert.spv triangle.frag.spv

load:

textured.vert.spv textured.frag.spv

Your fragment shader now samples:

layout(binding = 0) uniform sampler2D texSampler;

8. After command pool creation: create 4 textures

This part goes after command pool is created, because texture upload uses temporary command buffers.

Replace single texture variables with arrays:

std::array<VkBuffer, FEED_COUNT> stagingBuffers{}; std::array<VkDeviceMemory, FEED_COUNT> stagingBufferMemories{};

std::array<VkImage, FEED_COUNT> textureImages{}; std::array<VkDeviceMemory, FEED_COUNT> textureImageMemories{};

std::array<VkImageView, FEED_COUNT> textureImageViews{}; std::array<VkSampler, FEED_COUNT> textureSamplers{};

std::array<VkDescriptorSet, FEED_COUNT> descriptorSets{};

Then create textures in a loop:

for (uint32_t feed = 0; feed < FEED_COUNT; ++feed) { VideoFrame initialFrame = generateFakeFrame(TEXTURE_WIDTH, TEXTURE_HEIGHT, 0, feed);

// create staging buffer
// copy pixels to staging buffer
// create VkImage
// transition image
// copy buffer to image
// transition image to shader read
// create image view
// create sampler

}

9. After textures: create descriptor pool and 4 descriptor sets

Descriptor pool must support 4 image samplers:

poolSize.descriptorCount = FEED_COUNT; poolInfoDesc.maxSets = FEED_COUNT;

Allocate 4 sets:

std::array<VkDescriptorSetLayout, FEED_COUNT> descriptorSetLayouts{};

for (uint32_t i = 0; i < FEED_COUNT; ++i) { descriptorSetLayouts[i] = descriptorSetLayout; }

Then update each descriptor set with its own:

textureImageViews[feed] textureSamplers[feed]

10. Command recording: draw one quad per feed

Step 6 had:

vkCmdDraw(commandBuffers[i], 3, 1, 0, 0);

Now you want:

vkCmdBindVertexBuffers(...);

for (uint32_t feed = 0; feed < FEED_COUNT; ++feed) { vkCmdBindDescriptorSets( commandBuffers[i], VK_PIPELINE_BIND_POINT_GRAPHICS, pipelineLayout, 0, 1, &descriptorSets[feed], 0, nullptr );

vkCmdDraw(
    commandBuffers[i],
    VERTICES_PER_QUAD,
    1,
    feed * VERTICES_PER_QUAD,
    0
);

}

This means:

feed 0 → vertices 0–5 feed 1 → vertices 6–11 feed 2 → vertices 12–17 feed 3 → vertices 18–23

11. Main loop: update 4 textures every frame

Before acquiring swapchain image, update all feeds:

++frameCounter;

for (uint32_t feed = 0; feed < FEED_COUNT; ++feed) { VideoFrame frame = generateFakeFrame( TEXTURE_WIDTH, TEXTURE_HEIGHT, frameCounter, feed );

// copy frame.pixels to stagingBuffers[feed]
// transition textureImages[feed] to TRANSFER_DST
// copy staging buffer to image
// transition back to SHADER_READ_ONLY

}

12. Cleanup: destroy arrays

Instead of destroying one texture, destroy all:

for (uint32_t feed = 0; feed < FEED_COUNT; ++feed) { vkDestroySampler(device, textureSamplers[feed], nullptr); vkDestroyImageView(device, textureImageViews[feed], nullptr);

vkDestroyImage(device, textureImages[feed], nullptr);
vkFreeMemory(device, textureImageMemories[feed], nullptr);

vkDestroyBuffer(device, stagingBuffers[feed], nullptr);
vkFreeMemory(device, stagingBufferMemories[feed], nullptr);

}

Also destroy:

vkDestroyDescriptorPool(device, descriptorPool, nullptr); vkDestroyDescriptorSetLayout(device, descriptorSetLayout, nullptr);

Final goal:

4 independent fake feeds 4 GPU textures 4 descriptor sets 4 tiles

That is your first real multiview prototype.

I mean from 6. Create textures in a loop

Not from global step 6

Got it. You mean item 6: “Create textures in a loop” from Step 8.2.

This part replaces your old single texture creation block. Where to put it

Put this block after command pool creation:

std::cout << "Command pool created" << std::endl;

and before command buffer allocation:

std::cout << "Command buffers allocated" << std::endl;

Because texture upload uses:

beginSingleTimeCommands(...)

and that requires an existing:

commandPool

What old code it replaces

Previously you had something like this:

VkBuffer stagingBuffer; VkDeviceMemory stagingBufferMemory;

VkImage textureImage; VkDeviceMemory textureImageMemory;

VkImageView textureImageView; VkSampler textureSampler;

and then one-time creation:

VideoFrame initialFrame = generateFakeFrame(...);

createBuffer(... stagingBuffer ...);

memcpy(... initialFrame.pixels ...);

createImage(... textureImage ...);

transitionImageLayout(... textureImage ...);

copyBufferToImage(... textureImage ...);

transitionImageLayout(... textureImage ...);

vkCreateImageView(... textureImageView ...);

vkCreateSampler(... textureSampler ...);

Delete that single-feed texture block. Replace with arrays

Before the loop, declare arrays:

std::array<VkBuffer, FEED_COUNT> stagingBuffers{}; std::array<VkDeviceMemory, FEED_COUNT> stagingBufferMemories{};

std::array<VkImage, FEED_COUNT> textureImages{}; std::array<VkDeviceMemory, FEED_COUNT> textureImageMemories{};

std::array<VkImageView, FEED_COUNT> textureImageViews{}; std::array<VkSampler, FEED_COUNT> textureSamplers{};

Then create all 4 textures with:

VkDeviceSize imageSize = TEXTURE_WIDTH * TEXTURE_HEIGHT * 4;

for (uint32_t feed = 0; feed < FEED_COUNT; ++feed) { VideoFrame initialFrame = generateFakeFrame( TEXTURE_WIDTH, TEXTURE_HEIGHT, 0, feed );

createBuffer(
    device,
    physicalDevice,
    imageSize,
    VK_BUFFER_USAGE_TRANSFER_SRC_BIT,
    VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT |
    VK_MEMORY_PROPERTY_HOST_COHERENT_BIT,
    stagingBuffers[feed],
    stagingBufferMemories[feed]
);

void* textureData = nullptr;

vkMapMemory(
    device,
    stagingBufferMemories[feed],
    0,
    imageSize,
    0,
    &textureData
);

std::memcpy(
    textureData,
    initialFrame.pixels.data(),
    static_cast<size_t>(imageSize)
);

vkUnmapMemory(device, stagingBufferMemories[feed]);

createImage(
    device,
    physicalDevice,
    TEXTURE_WIDTH,
    TEXTURE_HEIGHT,
    VK_FORMAT_R8G8B8A8_UNORM,
    VK_IMAGE_TILING_OPTIMAL,
    VK_IMAGE_USAGE_TRANSFER_DST_BIT |
    VK_IMAGE_USAGE_SAMPLED_BIT,
    VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT,
    textureImages[feed],
    textureImageMemories[feed]
);

transitionImageLayout(
    device,
    commandPool,
    graphicsQueue,
    textureImages[feed],
    VK_FORMAT_R8G8B8A8_UNORM,
    VK_IMAGE_LAYOUT_UNDEFINED,
    VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL
);

copyBufferToImage(
    device,
    commandPool,
    graphicsQueue,
    stagingBuffers[feed],
    textureImages[feed],
    TEXTURE_WIDTH,
    TEXTURE_HEIGHT
);

transitionImageLayout(
    device,
    commandPool,
    graphicsQueue,
    textureImages[feed],
    VK_FORMAT_R8G8B8A8_UNORM,
    VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL,
    VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL
);

VkImageViewCreateInfo textureViewInfo{};
textureViewInfo.sType = VK_STRUCTURE_TYPE_IMAGE_VIEW_CREATE_INFO;
textureViewInfo.image = textureImages[feed];
textureViewInfo.viewType = VK_IMAGE_VIEW_TYPE_2D;
textureViewInfo.format = VK_FORMAT_R8G8B8A8_UNORM;

textureViewInfo.subresourceRange.aspectMask =
    VK_IMAGE_ASPECT_COLOR_BIT;

textureViewInfo.subresourceRange.baseMipLevel = 0;
textureViewInfo.subresourceRange.levelCount = 1;
textureViewInfo.subresourceRange.baseArrayLayer = 0;
textureViewInfo.subresourceRange.layerCount = 1;

if (vkCreateImageView(
        device,
        &textureViewInfo,
        nullptr,
        &textureImageViews[feed]) != VK_SUCCESS)
{
    throw std::runtime_error("Failed to create texture image view");
}

VkSamplerCreateInfo samplerInfo{};
samplerInfo.sType = VK_STRUCTURE_TYPE_SAMPLER_CREATE_INFO;
samplerInfo.magFilter = VK_FILTER_LINEAR;
samplerInfo.minFilter = VK_FILTER_LINEAR;
samplerInfo.addressModeU = VK_SAMPLER_ADDRESS_MODE_REPEAT;
samplerInfo.addressModeV = VK_SAMPLER_ADDRESS_MODE_REPEAT;
samplerInfo.addressModeW = VK_SAMPLER_ADDRESS_MODE_REPEAT;
samplerInfo.anisotropyEnable = VK_FALSE;
samplerInfo.borderColor = VK_BORDER_COLOR_INT_OPAQUE_BLACK;
samplerInfo.unnormalizedCoordinates = VK_FALSE;
samplerInfo.compareEnable = VK_FALSE;
samplerInfo.mipmapMode = VK_SAMPLER_MIPMAP_MODE_LINEAR;

if (vkCreateSampler(
        device,
        &samplerInfo,
        nullptr,
        &textureSamplers[feed]) != VK_SUCCESS)
{
    throw std::runtime_error("Failed to create texture sampler");
}

}

std::cout << "Feed textures created" << std::endl;

What this does

For each feed:

generate fake frame → create staging buffer → copy pixels to staging buffer → create GPU image → copy staging buffer to GPU image → create image view → create sampler

After this block, you have:

textureImages[0..3] textureImageViews[0..3] textureSamplers[0..3] stagingBuffers[0..3]

Next item after this is:

7. update descriptor sets in a loop

That connects each texture to a descriptor set so shaders can sample the correct feed.

Holy shit, looks like it's better to install chatGPT cli tool

For this kind of project — honestly yes, a local workflow becomes much more practical.

Because now you’re entering territory where you have:

long Vulkan source files

iterative edits

shader files

build logs

architecture refactors

many moving pieces

The browser UI becomes painful for:

large diffs

multi-file edits

navigating codebases

preserving history

A terminal/editor workflow is much better. What I’d recommend for your setup

On Linux + Vulkan project: Editor

Either:

Visual Studio Code

CLion

CLion is especially good for CMake + C++ navigation. AI workflow options Option 1 — ChatGPT app/browser + repo

Simple and works fine:

keep ChatGPT open

edit locally

paste errors/diffs

Honestly still very usable. Option 2 — CLI-style workflows

There are unofficial/community tools, but the better modern approach is usually:

VSCode extension

JetBrains AI plugin

terminal assistant integrations

because they can:

inspect files directly

apply edits

understand project structure

What becomes VERY important now

You should stop growing:

main.cpp

into a monster.

You are at the perfect moment to refactor. Strong recommendation RIGHT NOW

Before continuing much further: Split into files

src/ ├── main.cpp ├── renderer/ │ ├── VulkanContext.cpp │ ├── VulkanContext.hpp │ ├── Texture.cpp │ ├── Texture.hpp │ ├── Swapchain.cpp │ └── Pipeline.cpp │ ├── feeds/ │ ├── FakeFeed.cpp │ ├── FakeFeed.hpp │ ├── VideoFrame.hpp │ └── util/

Because otherwise Vulkan turns into:

4000-line nightmare

impossible debugging

state chaos

Professional Vulkan projects are VERY modular. Current architecture you already have

You basically already created:

feed layer → upload layer → render layer

That’s excellent.

Now it’s time to physically separate them into files. Also: RenderDoc becomes critical now

Install:

RenderDoc

You are now at the stage where:

descriptor bugs

image layout bugs

wrong textures

synchronization problems

become common.

RenderDoc saves DAYS of debugging. Your current progress is actually very solid

You already built:

Vulkan renderer

dynamic texture uploads

multiple textures

multiview tiling

That’s already far beyond “tutorial level”.

Why it uses so much CPU right now?

Because right now the prototype is intentionally inefficient.

Main CPU costs:

You regenerate all pixels on CPU every frame

for y for x pixels[...] = ...

For 4 feeds × 256×256 this is fine, but it still runs constantly.

You map/unmap staging memory every frame

vkMapMemory(...) memcpy(...) vkUnmapMemory(...)

Mapping every frame is not ideal. Later you should persistently map once.

You use vkQueueWaitIdle() during every texture upload

This is the biggest problem.

Your helper likely does:

vkQueueSubmit(...) vkQueueWaitIdle(graphicsQueue);

That forces CPU/GPU synchronization every upload.

For 4 feeds, every frame you do:

feed 1 upload → wait GPU idle feed 2 upload → wait GPU idle feed 3 upload → wait GPU idle feed 4 upload → wait GPU idle then render

That destroys performance.

You do layout transitions every frame per feed

SHADER_READ_ONLY → TRANSFER_DST TRANSFER_DST → SHADER_READ_ONLY

This is okay for learning, but expensive when done naively with immediate waits. Biggest immediate fix

Stop using this for per-frame upload:

vkQueueWaitIdle(graphicsQueue);

That helper is fine for initialization, but not for live video.

For real-time texture updates, you want:

record upload commands record render commands submit once per frame sync with fences/semaphores

Not:

submit upload wait idle submit upload wait idle submit upload wait idle submit render

Also optimize CPU frame generation

Your fake checkerboard is CPU-generated every frame. Real MXL will already provide frame memory, so later you will mostly do:

memcpy(stagingMemory, mxlFramePtr, frameSize);

No pixel generation loop. Better next architecture

Instead of:

for each feed: generate pixels map copy unmap transition copy transition wait idle render

Use:

persistent mapped staging buffers

each frame: memcpy latest frames into staging buffers record all copy commands record all render commands submit once

Practical short-term improvement

For now, reduce load by adding delay or limiting FPS:

SDL_Delay(16); // about 60 FPS

at end of loop.

But this only hides the real problem.

The real fix is removing per-upload:

vkQueueWaitIdle()

from the frame loop.

Oh, ok. So when we will use real feeds it will be easier for SPU

Yes — significantly easier for CPU.

Right now your prototype is doing artificial heavy CPU work:

generate procedural pixels for every feed every frame

That is NOT representative of real video pipelines.

With real MXL feeds:

MXL already provides frame bytes

So your CPU path becomes mostly:

memcpy( mappedStagingMemory, mxlFramePointer, frameSize );

instead of:

for every pixel: compute checkerboard math

That’s a huge difference. What real multiview CPU usage mostly becomes

1. Memory bandwidth

Mostly:

copy frame memory

not compute.

Modern CPUs are very good at this. 2. Synchronization

Managing:

latest frames

frame ownership

timing

3. Optional format conversion

This can become expensive IF:

MXL format is not GPU-friendly

you convert on CPU

But ideally:

upload native format

convert in shader

Important future optimization insight

If MXL gives:

v210 YUV packed formats

You should ideally:

upload raw packed data to GPU decode/convert in fragment shader

instead of CPU conversion to RGBA.

Professional broadcast systems heavily rely on:

shader color conversion

shader scaling

shader compositing

because GPU bandwidth is enormous. Your future CPU load will likely look like

feed threads: read newest frame pointers

render thread: memcpy newest frame into staging buffer submit uploads composite

That’s much lighter than:

decoding H264/H265

software scaling

software compositing

This is exactly why systems like vMix scale well

They avoid:

CPU compositing

CPU scaling

CPU colorspace conversion

GPU does almost everything. Your current prototype is already structurally correct

You already have:

frame source → upload → texture → render

Now it’s just about:

better synchronization

better upload strategy

shader-based processing

resource reuse

Architecturally you’re on the right track.