Multiple AppendStructuredBuffers

Append Buffers have no special hardware behind them. It's a typical Microsoft invention to overspecify simple things. Other APIs don't have them, also no 'structured' buffers or other confusing differences - it is all just plain memory for me as a Khronos guy.

So what i do is this: Allocate one large buffer with space for all 15 lists i generate in one shader, the first 15 numbers in the buffer being the counters, use some defines to ease up indexing, and finally increment the counters with atomic add and use the result to write the data to the buffer. Of course you can use the buffer for other, unrelated int data as well.

So you don't need to worry and can do the same (notice that for all lists and counters the shader only needs one parameter to adress it all, so there might be an advantage eventually).

Related: It is often a big win if you write your lists in short sequences first to LDS (using atomics to LDS as well), and when the sequence is full write it to global memory in one batch. This minimizes slow atomics and avoids scattered writes to global memory. I can make an example if you don't know what i mean...

We should also talk about using structs as in C, and why you mostly avoid this on GPU and prefer SoA, if you don't know...

42 minutes ago, NikiTo said:

Is emulating the push/pop functionality with one single byte address buffer worth it? Wouldn't it be much slower than using AppendStructuredBuffer?

Probably it would be much slower, because you would need to read the data before you write, and even create some lock to guarantee preserving the other bytes. But you can do this stuff in LDS, as hinted above.

I was thinking to do the very same with byte address buffers in case AppendStructuredBuffer option fails.

It looks very slow, but the next step should compensate it. If I don't use lists, the next shader will have to read through the whole buffer, testing if each offset has valid data.

Plus, making my own push/pop logic is giving me more freedom. I could create a single list apart for lists shared data. It is just that I expected AppendStructuredBuffer to offer me hardware acceleration over my own push/pop-ing code.

40 minutes ago, JoeJ said:

Related: It is often a big win if you write your lists in short sequences first to LDS (using atomics to LDS as well),

Good to know!! I will use this approach for my next shader pass. In this one the LSD is full.

23 minutes ago, JoeJ said:

I can make an example if you don't know what i mean...

It is not needed, I understand it.

26 minutes ago, JoeJ said:

We should also talk about using structs as in C, and why you mostly avoid this on GPU and prefer SoA, if you don't know...

I googled it. My whole struct is 16 bytes. It is 4 floats each and I need to read them all together, so I think there will be not a problem.

Thanks!

16 minutes ago, NikiTo said:

In this one the LSD is full.

I still have connections to Albert Hofmann in case you run out of it, just PM me... ;P

18 minutes ago, NikiTo said:

I was thinking to do the very same with byte address buffers in case AppendStructuredBuffer option fails.

19 minutes ago, NikiTo said:

I expected AppendStructuredBuffer to offer me hardware acceleration over my own push/pop-ing code.

GPUs can't address bytes so writes are very expensive under the hood, and there is no hardware backing for Append Buffers.

(I repeat this so anybody can correct me if i'm wrong.)

in D3D11 it was possible for Append Buffers to have their counter stored in special memory (e.g. GDS) if the hardware had it. Performing atomic memory operations on GDS is much faster than having the count stored in main memory and be atomically incremented/decremented there.

In D3D12 where it's all "just memory" and the counter is a separate resource in main memory this optimisation no longer applies.

If AppendBuffers weren't codified as being in some way 'special', I can imagine it would have been more difficult to allow the IHVs to handle the counters in whatever magic ways they might have wanted.

1 hour ago, ajmiles said:

in D3D11 it was possible for Append Buffers to have their counter stored in special memory (e.g. GDS) if the hardware had it. Performing atomic memory operations on GDS is much faster than having the count stored in main memory and be atomically incremented/decremented there.

In D3D12 where it's all "just memory" and the counter is a separate resource in main memory this optimisation no longer applies.

If AppendBuffers weren't codified as being in some way 'special', I can imagine it would have been more difficult to allow the IHVs to handle the counters in whatever magic ways they might have wanted.

On AMD, I've measured a *massive* performance difference in D3D11 between manually implementing append buffers (putting the counter in a normal UAV and incrementing it with atomic shader instructions), and using D3D's magic counter abstraction. On NVidia I didn't notice a difference.

To me, this hints that (on the two specific devices that I compared):

* the NV one was very capable of dealing with highly contended atomic increments to memory, whole the AMD one struggled here.

* the counter abstraction allows AMD to utilise their GDS to completely make up for this deficiency.

On the other hand, D3D12 brings with it Shader Model 6 and the new 'Wave Intrinsics' which allow for wave-level reductions in the number of atomic operations (32x less on NVIDIA, 64x less on AMD). By balloting the wave on the number of threads that wish to increment a value you can have a single thread perform a single InterlockedAdd on behalf of all threads. In my experience so far (on AMD hardware) this 64-fold reduction in the number of atomic operations to main memory more than makes up for the fact that counters can no longer live in GDS, so we're in a better place than we were on D3D11!

10 hours ago, ajmiles said:

In D3D12 where it's all "just memory" and the counter is a separate resource in main memory this optimisation no longer applies.

.... On the other hand, D3D12 brings with it Shader Model 6 and the new 'Wave Intrinsics' 

Because I'm not doing any fancy SM6 stuff yet, this got me interested because the difference was so pronounced in D3D11, so I finally got around to testing it in D3D12 for the first time
From the looks of it, AMD are still doing something tricky, like copying that bit of memory to GDS before the draw/dispatch and copying it back afterwards.

I tested my pixel-linked-list OIT shader, which has a highly contended atomic counter to allocate fragment storage in the pixel shader. I recorded these timings on an AMD R9 Nano, but they should taken with a large grain of salt as my demo scene has some slight animation in it, and it's timed using D3D GPU timestamps rather than a precise profiler:
D3D12 native append counter: 866µs
D3D12 manual atomic counter in HLSL: 3848µs
D3D12 manual atomic hierarchical counter in HLSL: 1127µs (algorithm to reduce memory contention)
D3D11 native append counter: 1041µs
D3D11 manual atomic counter in HLSL: 4119µs
D3D11 manual atomic hierarchical counter in HLSL: 1349µs (algorithm to reduce memory contention)

Oops again - sorry for spreading wrong things

2 hours ago, Hodgman said:

I tested my pixel-linked-list OIT shader

Could you share some more details about this?

I assume you use 2 passes, first incrementing a single counter to allocate storage, second writing fragments using per pixel counters?

I wonder how you can use intrinsics in a pixel shader here, but assume you can do it only in the first pass. So i assume your timings are just the first pass?

I would be interested in doing this for all geometry, to improve SS Reflections / Shadows, SSAO or SSGI etc as well. Assuming Nano is close to next gen consoles this might start to make sense looking at your numbers.

I suspected Microsoft had to add things to DX other APIs lack.

14 hours ago, NikiTo said:

Is declaring 30+ AppendStructuredBuffers going to overkill the shader?

I assume the answer of this is obvious then

The number of Append Buffers isn't really a problem, it's more a question of what the code looks like when you Append to them.

If, for example, you had a load of code that calculated a number between 0-29 and then proceeded to do this:

switch(value)
{
case 0: appendBuffer0.Append(thing); break;
case 1: appendBuffer1.Append(thing); break;
....
case 29: appendBuffer29.Append(thing); break;
}

Then I'd be very concerned about the resulting performance. However, if whatever you're implementing naturally has 30 separate lists and the appending of items to those lists isn't naturally divergent or conditional then I would be less concerned about having 30 Append Buffers.

Can you give us any more insight into what you're trying to implement and perhaps a little bit of code to demonstrate the pattern for how you add to those 30 append buffers?

My code would be using switch case too. Using my own push/pop implementation would be more elegant looking in code.

I was concerned because some documentation on internet says that buffers descriptors waste SGPRs:

Quote

Running out of SGPRs is also possible, but unlikely. The most common way to exceed your SGPR budget is by using an excessive number of textures and samplers. Texture descriptors consume eight SGPRs each, while samplers consume four SGPRs each. DirectX 11 allows using a single sampler with multiple textures. Usually, a single sampler is enough. Buffers descriptors only consume four SGPRs. Buffer and texture load instructions don’t need samplers, and should be used when filtering is not required.

After reading this, I am not sure at all if declaring multiple append buffers will give me problems.

1 hour ago, ajmiles said:

Then I'd be very concerned about the resulting performance. However, if whatever you're implementing naturally has 30 separate lists and the appending of items to those lists isn't naturally divergent or conditional then I would be less concerned about having 30 Append Buffers.

I will try to add some trees there.

if (target < 8 ) { // separated shader?
switch(target)
{
case 0: appendBuffer0.Append(thing); break;
case 1: appendBuffer1.Append(thing); break;
....
case 7: appendBuffer29.Append(thing); break;
}
}
else {
if (target < 16 ) { // separated shader?
switch(target)
{
case 8: appendBuffer0.Append(thing); break;
.....

Anyways, you are right, maybe I should improve the implementation. The measures @Hodgman shows seem worthy the effort of trying to use append buffers over my own push/pop. I could create a separate shader for each tree of switch cases. I hope loading large shaders into the CU is not too expensive, because few more switches could be hidden by the memory accesses. Even more in the case of using append buffers, I guess.
If nothing, using multiple shaders will reduce the number of append buffers to declare.

Looks like you could calculate the buffer index directly from target variable, which makes my initial suggestion more attractive, as all the cases could be replaced by index math. (In this case experiment with non-power-of-2 list sizes !!!)

If all the shader does is appending, splitting it to multiple shaders will likely just generate more overhead, while still equal amounts of threads go through the same number of cases, if i get this right.

In general i suggest you try multiple options, at least for performance critical things. It's usually worth the effort, and often results are different than expected.

A question regarding Append buffers in regards to SIMT execution. If you have a shader in which all lanes access the same append buffer does the hardware accelerate this some way? Or it it serialized atomic adds equal to the number of lanes?

1 hour ago, Infinisearch said:

If you have a shader in which all lanes access the same append buffer does the hardware accelerate this some way?

I don't know, but e.g. using atomicMax over LDS is faster than a scan algorithm even within a single wavefront (so without barriers). So this serialization is not as bad as i thought.

@JoeJ sure, index math would look a lot nicer. I would try many variants, but it is just the API and shaders are so explicit, it takes a lot of work to change anything....

@Infinisearch I think even if all lanes append at the same time, the act of push/pop is a lineal task, and no matter what, all lines writing to a separated offsets in a regular buffer(no lists) should be faster than appending, because it is a parallel task. In my code, I hope for every lane appending to a different buffer. Divergence here is good as it parallelizes the lineal push/pop task. If physically there is only a single piece of silicon carrying out all the appends, parallelization would matter much less.

35 minutes ago, NikiTo said:

I think even if all lanes append at the same time, the act of push/pop is a lineal task, and no matter what, all lines writing to a separated offsets in a regular buffer(no lists) should be faster than appending, because it is a parallel task. In my code, I hope for every lane appending to a different buffer. Divergence here is good as it parallelizes the lineal push/pop task. If physically there is only a single piece of silicon carrying out all the appends, parallelization would matter much less.

Divergence whether branch or data wouldn't parallelize anything, unless I'm missing what you're trying to say. IIRC in AMD GCN the L1D cache isn't banked while the LDS is banked. If your atomic counters per append list are in LDS there is an opportunity to paralleize the the increment the write pointers (I think). Then I think (i'm not sure) either memory coalescing on access to the L1D cache would take care of parallelizing what can be parallelized in the actual appending or all appends are serialized. But my question had to do with contention on the same atomic counter... I was wondering if the coalescing hardware did some kind of trick by realizing its the same atomic counter and some how parallelize multiple increments.

2 hours ago, NikiTo said:

@JoeJ sure, index math would look a lot nicer. I would try many variants, but it is just the API and shaders are so explicit, it takes a lot of work to change anything....

Be sure you use timestamps or a profiling tool, so you can profile each shader on its own. The results on GPU are much more accurate than on CPU, so timings are fine even for very short durations (but they vary from frame to frame). Not doing any of this is working blindfolded - not knowing register usage etc. as well.

Then you can change a single shader and profile it while disabling other shaders depending on its results. Later you can update them if you made a change.

Care must be taken because the GPU may do some async compute without you knowing, and then you may draw wrong conclusions from timings. With DX12 this is easier to know: Any shaders with no barriers between them will potentially execute in parallel. With DX11, where i assume barriers will be created automatically for you by lookig at data dependencies, this may be more difficult to figure out.

2 hours ago, Infinisearch said:

Divergence whether branch or data wouldn't parallelize anything, unless I'm missing what you're trying to say.

I just say that many pushing to one place is a lineal task. And many pushing to many places is faster than this(as the divergence in the switch cases above). Even if the driver attempts to do some magic, it will be slower than the parallel task. It is an ideological problem, and lineal can not be ideologically faster than parallel. One of the reasons for me to want to use multiple append buffers was to speed up things a little bit.
Maybe you are trying to focus on the strategies the driver takes in order to mitigate the bottleneck. In parallel there should not be a bottleneck at all.
Unless the hardware uses one single piece of silicon for doing the appends, because it would mean the same bottleneck.

1 hour ago, JoeJ said:

Then you can change a single shader and profile it while disabling other shaders depending on its results.

It could sound silly, but I am still painting the pipeline in Photoshop. And I am still drafting the compute shader. It could be silly, but I am still codding the shader.... I have 1000 lines of a single shader code so far and still have to code much more. One single change in one place, requires a lot of changes in many other places. I am codding alone a huge app. I try to save human work. In my own experience it is much less stressful to overthink the things, rather than going back changing the first pass shader, and then having to recode all the next steps from scratch. Overthinking pays it. The DX API is pretty explicit. It is not as easy to make changes like changing the injected dependency in C#. If I had to code various variants and measure all of them, i would never end literally. Maybe I am just a slow coder....