🔒 Locked

Vulkan SSBO scalar alignment - array of larger structs causes large offsets

Started by Lewa May 13 at 12:10 AM 4 replies 1.6k views

Original Post

May 13 at 12:10 AM

I ran into an issue with the alignment/stride of SSBOs in compute shaders which are arrays of larger structs. I'm using a scalar alignment to make the mapping 1:1 as it is in C++.

Specifically in my case the struct has a nested array of integers that looks like this:

#version 450
#extension GL_EXT_scalar_block_layout : enable

struct Cluster
{
    vec4 minPoint;
    vec4 maxPoint;
    uint count;
    uint lightIndices[1]; //<<< 1 element
};

layout(scalar, set = 0, binding = 2) restrict buffer clusterSSBO
{
    Cluster clusters[];
} clusterData;

In Nsight the byte offsets of the array elements look like this:

Screenshot: Nvidia Nsight 1 element

Indexing into that array worked fine in the compute shader.

However, if we increase the "lightIndices" array to a larger size:

struct Cluster
{
    vec4 minPoint;
    vec4 maxPoint;
    uint count;
    uint lightIndices[100]; //<<< 100 elements
};

The offsets between the array elements becomes very large (we have a stride of exactly 4000 bytes?) Which causes issues with indexing into the array.

Screenshot: Nvidia Nsight 100 elements

You can see in the second screenshot in the "offset" column how it jumps from byte 432 to 4000. (And the next struct is then at offset 8000)

Am i missing something with regards to this struct configuration? My assumption was that the scalar layout should result it a tight/packed mapping as it's the case in C++.

Any kind of input/direction would be appreciated.

JoeJ

May 13 at 5:44 AM

Ha, that's bad. Reminds me on the endless frustration i've had when trying to use structs with OpenGL compute shaders.

When i moved to Vulkan, i never used structs again, solving such frustration at the cost of some more manual setup work. I'll explain how i organize my data, maybe it helps.

I use one large buffer of vec4 data, and another for uint data.

Then i use an include file with #defines of offsets to address various data in those large buffers. Some complicated example:

#define TREE_DIR_ROOTS_STRIDE				2 // 1 * static + 1 * dynamic
#define TREE_ROOT							(MEM_MAX_SAMPLES - 1) // root node
#define TREE_1ST_DIR_PARENT					(TREE_ROOT - 8) // each one of 8 dir parents has 1 static and 1 dynamic branch as children 
#define TREE_1ST_STATIC_DIR_ROOT			(TREE_1ST_DIR_PARENT - 8 * TREE_DIR_ROOTS_STRIDE) // each of 8 parents a static subtree for its directioon
#define TREE_1ST_DYNAMIC_DIR_ROOT			(TREE_1ST_STATIC_DIR_ROOT + 1) // each of 8 parents a dynamic subtree for its direction
#define TREE_ROOT_BRANCH_RESERVED			20 // reserved space for the above nodes

#define TREE_STATIC_LOOSE_START				(MEM_MAX_SAMPLES - (TREE_ROOT_BRANCH_RESERVED + MEM_MAX_DYNAMIC_LOOSE_SAMPLES + MEM_MAX_STATIC_LOOSE_SAMPLES))
#define TREE_STATIC_LOOSE_END				(TREE_STATIC_LOOSE_START + MEM_MAX_STATIC_LOOSE_SAMPLES)
#define TREE_DYNAMIC_LOOSE_START			(TREE_STATIC_LOOSE_END)
#define TREE_DYNAMIC_LOOSE_END				(TREE_DYNAMIC_LOOSE_START + MEM_MAX_DYNAMIC_LOOSE_SAMPLES)

You see often the address of the next data block depends on the former block in memory. So i need to be careful when removing a block or adding another in between. This process is a bit error prone and annoying, but i come along.

Then i address some data like that for example: bigUintBuffer[TREE_DYNAMIC_LOOSE_START + someIndex]

It's really not an elegant way, but i never have those stupid alignment issues again and have my data where i want it.

There's also a performance advantage. Say you process your struct like so:

vec4 minPoint = clusters[gl_LocalInvocationID.x].minPoint;

Then your parallel reads are 4000 bytes apart, and performance is likely very bad.

In my case it would look like:

vec4 minPoint = v4Buffer[OFFSET_MIN_POINTS + gl_LocalInvocationID.x];

So i read the data in a nicely packed pattern, and performance is ideal.

Due to this argument i do not really miss using structs at all.

RmbRT

May 13 at 4:44 PM

Why 4000? I could have understood 4096, being a power of two, allowing for shifts. But if it has to multiply anyway, then it could have just chosen the proper size. The spec authors must have had a huge brainfart when writing that up.

Also, I would go as far and interleave the lights by cluster count. So that in memory, you have cluster[0].light[0], cluster[1].light[0], …, cluster[N].light[0], cluster[0].light[1], etc.. That way, assuming most clusters have few lights, all the memory accesses are tightly packed, and you don't waste cachelines that are only partially filled with useful data. And multiple clusters fetching their first light would then both fetch the same cacheline, which should reduce memory bandwidth (for 64 byte lines, 16 uints should fit into one line, opposed to having 16 separate cachelines). That is, if per cluster, on average, you have way less than 64 active lights. If most clusters have 0-1 lights, then you can fit the lights of 16 clusters into one cacheline, instead of forcing one cacheline per cluster.

frob

May 13 at 5:05 PM

4000 does seem weird, but I don't think it is problematic.

Normally things have an alignment or offset of 16 (which 4000 does) or 32 (which 4000 does), sometimes by 64 (which 4000 does not) or other powers of 2.

Is this an actual problem, or is it just that it bothers your sense of rightness of what you think it should be? Be mindful that the memory is virtual, you've got up to 4GB per descriptor but the actual memory location and location used on the card are virtual, it is mapped onto the hardware wherever the drivers decide to put things.

I can see it being an actual problem if you're running out of space on the card or you're blowing out the bounds of your buffers, but I don't think either of those are the case here. It looks like the entire array is still less than the size of a single typically-sized texture.

Lewa

May 13 at 6:35 PM

After a bit more debugging, i'm not quite sure if this padding or an Nsight bug:

Nsight Screenshot: Buffer content

Managed to get the computeshader to work (somewhat) correctly (although i have to double check as i'm second quessing myself right now) and inspected the buffer content of the SSBO in Nsight.

The window on the left shows that "clusterdata.clusters[1] starts at an offset of 5200.

I then inspected the actual buffer content via the resource inspector (window on the right) and there it starts at the address 0x1c0 which is an offset of 448 bytes. The last primitive of the previous array entry (clusters[0]) has an offset of 444. So this is more in line with expectations.

I even added a variable "debugCounter" to the cluster struct which I set to it's coresponding index in the array (to see if Nsight can pick it up...)

struct Cluster
{
    vec4 minPoint;
    vec4 maxPoint;
    uint count;
    vec2 debug;
    uint debugCounter; // is set to it's index in the array in a compute shader
    uint lightIndices[100];
};

... which seems to work fine in the window on the right (see the "clusterData.clusters.debugCounter" column)

So my suspicion is that this is a Bug in Nsight? (No idea why the offset is so high though.)

I'll look if i can run this through RenderDoc to see what's reported there. Other than that i can try to transfer the Buffer back to system RAM and inspect the padding with a regular CPU/memory debugger.

JoeJ wrote:
When i moved to Vulkan, i never used structs again, solving such frustration at the cost of some more manual setup work. I'll explain how i organize my data, maybe it helps.

Yeah, i was thinking about moving the lightindices array into a seperately allocated uint buffer which stores all lightindices in sequence to avoid this issue (at least for this usecase).

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.