Safe resource deallocation in Vulkan

First thing is to boil it down a bit. Reference counting is out of the scope of this - it's dealing with what happens when you do decide to delete something that is the issue at hand.

Consider a simplistic rendering loop that renders, presents, and then waits for the device to be idle each frame. After the device is idle, but before anything is rendered, you can be sure that destroying a resource will be safe, as long as you don't try to use it again. So you can either defer all logic that would ever delete a resource until that time, or allow a deletion to be requested at any time but put it in a buffer to be "played back" in the safe window.

Fairly trivially, this can be extended to a pipeline that is multiple frames deep. Let's go with 3, for the sake of example. This means you might have rendering commands on the fly up to 2 frames "behind", and therefore you have to take that into account whenever you want to delete a resource. The easiest thing to do in this case IMO would be to have a growable array for each frame (e.g. 3); they keep track of deletion requests, each corresponding to requests issued when the CPU is processing that frame. Each frame would also have a fence associated with it. Whenever a new frame is started, it waits on its fence, and then before rendering anything, it deletes the appropriate resources and clears the vector for new requests to be added. When submitting the commands for that frame, you say that the fence should be signaled when the submission is complete. The fence ensures that the CPU will never get too far ahead of the GPU and cause the renderer to trip over itself, ensuring that you won't ever try to delete a resource that's being used by the GPU.

This concept of shifting by the number of frames in your pipeline applies to destructive updates, such as rendering to a framebuffer, as well. For N frames, you will need N copies of each attachment, and cycle through to determine which copy you are writing to.

Great, thanks for the answer. If I understand correctly, I think what you're suggesting is to basically assume that all resources are used by all command lists.

I think that this leads to. A pretty straight forward implementation... Expanding upon your description, I would probably just keep a list of objects pending destruction. When a vkEndCommandBuffer occurs, I would record the position in this list. Now, when we know for sure that the GPU has finished with a frame's command buffer, we can just track through the list, destroying as we go...

Of course, there are some complications related to secondary command buffers (particularly those build in background threads), but it could be handled within the same basic design.

So, here we're making 2 important assumptions:
* command buffers we're creating are going to be used once (within the current frame) and then discarded
* resources are used in the same frame that they are destroyed (or, at least some recent frame)

It feels to me that these assumptions are probably reasonable in the majority of cases. So it makes sense to use this simple design for the majority of cases...

But there are cases where we may want to break these rules. So, I'm thinking about building a "destruction queue" concept. This would sit alongside the VkQueue, and would be customisable for special cases. Each reource would end up tied to a single destruction queue. So the majority of resource could follow the simple mechanism described here. But other cases could also be handled (such as recording a command buffer to be replayed many times).

I'm also thinking about streaming world cases. In this situation, we may have a finite amount of device memory that is used as a window on a very large set of streamed resources.

In this case, we will frequently want to evict one resource, and then reallocate (or overwrite). Usually the evicted resource will be the least recently used resource... Probably something offscreen, and probably something that isn't being used by the gpu.

Often, it will be safe to overwrite immediately. So we will want to take advantage of that.

But maybe there's still a simple solution -- just by recording that last used frame as part of the LRU cache. Perhaps all we really need is some special case handling for the VkDeviceMemory objects in these cases. Maybe that wouldn't really be too hard, and it would avoid complicating the more general case with extra machinery.

Yeah I keep this as simple as possible. Many moons ago, we inherited an engine that used fences (write to label) every time you issued a draw call that used a newly bound resource, so that you could query to find out when it was safe to deallocate/reallocate/defrag as soon as possible. This kind of fine-grained management is not really ever required IMHO.

Instead we simply just enforced a maximum latency between CPU and GPU -- e.g. CPU is only ever 2 frames behind the GPU -- at which point you can simply create a "deferred deallocation queue". When deallocating a resource, you free up the API handle right away so that the user can pretend that resource has been free'ed, but really you grab the current frame ID, add 2 to it, and put the allocation pointer and this future frame ID into a queue. At the start of each frame (after the current frame ID has been incremented), you pop "deallocation jobs" from this queue until it's empty or you find a job with an ID that's still in the future.

This way, there's no overhead at all when binding resources, and at worst, they're freed up a frame or two later than optimally possible.

When doing memory intensive operations like switching between levels, the loading system could query the GPU device to check if this deallocation queue was empty, so that it would wait those extra two frames before starting to try and allocate all the GPU memory again -- this was important on PS3 where you've only got 256MiB, but probably not even required any more.

We do this on PC too, from D3D9 onwards! Even in D3D9, you can create large vertex buffers and sub-allocate smaller objects within them with, updated using the D3DLOCK_NOOVERWRITE flag to promise to handle the CPU/GPU sync issues yourself, like you do in Vulkan/D3D12/consoles... D3D9 ring buffer management worked the same way -- simply track used regions of the ring buffer at whole-frame granularity, and combine that data with the knowledge of your max CPU/GPU latency to safely overwrite old data without using any fences (except the single fence that ensures your max CPU/GPU latency is enforced).