How can I make a voxel mask destruction system look solid inside?

There's a lot of approaches, but yes, you can't just subdivide the existing mesh and expect it to work well. You'll need to create new meshes that include what the inside is supposed to look like, or just swap it out with a collection of pre-generated, artist-created meshes.

The naive approach is to just have a collection of models already generated. Replace the models, and you're done. Easy to implement, artists can control exactly how they want it to look, it's straightforward.

If you're trying to compute it there are research papers with a variety of fracturing methods, the hard part is making fractures that are realistic for the model. A granite boulder fractures differently from glass, which fractures differently from a tree trunk. There's also a difference between pre-generating them during design versus at runtime with fracturing calculated based on actual physics impacts. Pre-generated is easier but tends to not match what the player did, like glass that always fractures from the same spot, or logs where they're hit on the side but they splinter in the middle. Physics-driven fracturing generated at runtime has the potential to look nicer in a game but takes more work both from the system swapping it out and for design of how the fracturing system generates meshes and assigns textures.

If you're looking for a real-world example, Unreal Engine has the Chaos destruction library, with source code included. There are plenty of videos, guides, and written tutorials on how to use it.

Urban42 wrote:

But there is a problem which I've been struggling to solve. Meshes are made of flat polygons. The exposed interiors of the shards and partially destroyed objects look hollow and empty instead of having convincing internal volume.

Yeah, and i can elaborate on details you may not yet be aware of.
I assume your idea is: 'Let's make a realtime destruction system, with no need to precompute shards.
And we get there by using voxels to keep it simple, eventually accepting some blocky cracks which is fine.'
But as far as i can see, this is not possible with typical game asset meshes. Some additional data is required, and precomputing it is probably the only meaningful way for now, thinking of some larger game worlds.

The primary problem with meshes is: They only describe the surface, but tell us nothing about volume. There is no trivial test to tell us if a point is 'inside a mesh' and in solid matter, or if it is 'outside the mesh' and in empty space.
Thus, generating a 'solid voxelization' of a mesh is already difficult. I wonder how you do this?
Usually, actually referring to Voxel Cone Tracing, the typical method of realtime mesh voxelization is rasterizing the triangles and setting bits in a volume. But notice this only voxelizes the surface. The insides remain empty and hollow. So you would face a problem already here, before you get to any realtime destruction of the voxels.

A way to make the problem easier is restricting your meshes to manifolds, meaning the mesh must describe a solid volume correctly.
With such meshes solid voxelization easy. We can just trace rays and alternate solid / empty voxels along the hit points. Realtime is possible.
But a manifold requirement is a big deal, because game assets usually spare geometry which is never visible. They are not made to define surface of a solid volume. They are only made to represent the visible surface. So they rarely ever have a manifold property.

Personally i've had this problem, and i needed a way of automatic and robust voxelization of non manifold meshes.
My solution isn't realtime. Instead every asset needs to be voxelized once, then a DAG compressed voxel volume is stored on disk. I do not need to ship this data with the game, though. But i think this would be practical if needed.

Here is how my voxelization algorithm works:
First we need a robust inside / outside test. The reference solution would be: Trace many rays in all directions, and count how many front and back faces we hit. If we see more back faces, we are inside solid matter. If we see more front faces, we are outside in empty space.
This method is robust and gives us smooth surfaces even in areas where the mesh lacks a good description of solid volume. But it also still gives us false positives, which i call 'bubbles'. E.g. if a column without a bottom cap stands on a flat ground triangle, we see more front facing surface area if we are inside the column, but closely above the ground. So we gat a small bubble of empty space inside the column. To fix this, i generate a surface mesh from the voxelization, then dividing the mesh into individual parts which are not connected, then fro each part trace rays from it's vertices along normals to the outside. Then i sort the parts by percentage of rays which did not hit anything but escaped the scene. In automated mode, only the single part which is visible from the outside is kept. The rest is marked solid and filled with voxels. A room which has no window to the outside will be removed. So in failure cases a user can manually fix this by selecting which parts to keep.

Another issue here: The raytracing method would be much to slow for my desired resolution of voxels 1 or 2 centimeters large.
But there is a better way. Instead tracing many rays, we can integrate the surface to the point. There are two papers, iirc from Alec Jacobsen, using the concept of 'winding number'. Which can be simplified, and i can post a code snippet if you're interested.
Still, even the 'fast' method proposed in second paper was much too slow for my needs. But i could get much better time complexity using hierarchical optimaztions. That's ok now. Iirc, voxelizing Sponza to 4K^3 takes two minutes.

There also is a newer idea about such problems, in a paper by Keenan Crane, something like 'Monte Carlo geometry processing', working with sphere tracing random paths. This method should work here too, but no idea about performance.

That's a lot of talking, but you can see it's challenging even for offline preprocessing. So likely you want something different - some hack which avoids a need for a truly solid voxelization, and maybe you already have.

Now, regarding your question about visual representation on the intersection of voxel and mesh surfaces, one idea might be to add alpha masked texture blocks to the face of a voxel. So you could 'erase' voxel area which peeks out of the intended surface.
But then you still need to solve the inside/outside problem on each texel of a texture block near to the mesh surface. This could be done by integrating only nearby mesh surface. It should work, and processing the destruction parts in a background thread accepting some latency is possible too, i guess.
But it's hard. Real destruction in realtime is very hard, and i'm happy at least that is not on my personal todo list.
Still, somebody should try, and i wish you luck! \:D/