Are hemicubes necessary for Radiosity?

To accomplish Radiosity, is it really required to rasterize the scene to a hemicube or hemisphere at all?

No. It's just one way to compute visibility and form factors between patches, and it happens to be pretty easy to implement on a GPU.

Is there a problem with running N^2-ish patch to patch bounces, where the receiving patch runs through all of the patches visible to it and adds their point contributions?

Other than performance? Not really.

If it was properly optimized couldn't it be faster? I just hate the idea of rasterizing an entire scene for every patch. I was hoping some kind of clever face->face visibility determination could eliminate huge numbers of patches that needed to be checked against others, and to set it up in such a way that you could pretty much just throw a list of patches that needed to be radiated at a compute shader.

If it was properly optimized couldn't it be faster? I just hate the idea of rasterizing an entire scene for every patch. I was hoping some kind of clever face->face visibility determination could eliminate huge numbers of patches that needed to be checked against others, and to set it up in such a way that you could pretty much just throw a list of patches that needed to be radiated at a compute shader.

The original radiosity algorithm just worked that way, looking at all pairs. There are many papers on how to accelerate it. One advantage you get from the hemicube is that you immediately get the correct form factor (without the costheta), whereas in patch-to-patch interaction you need some analytical expression for it. There are a number of them, some based on contour integrals, and some based on simpler approximations (disks etc). Otoh you get aliasing artifacts when doing hemicube, so there's a tradeoff.

If it was properly optimized couldn't it be faster? I just hate the idea of rasterizing an entire scene for every patch. I was hoping some kind of clever face->face visibility determination could eliminate huge numbers of patches that needed to be checked against others, and to set it up in such a way that you could pretty much just throw a list of patches that needed to be radiated at a compute shader.

Visibility isn't a simple problem, and unfortunately there aren't any simple solutions. You can make a lot of optimizations if you assume the scene is static, but if the scene is static then you might as well pre-compute visibility and form factors.