GaussAnything: Semantic Intent-Driven Refinement of Evolving Gaussian Scenes for Standalone VR

Standalone VR has a hard budget problem: reconstructed environments need to look rich, but mobile headsets can’t afford to keep everything equally detailed. GaussAnything tackles that by making refinement depend on user intent, not just what is visible. The system runs natively on OpenXR and works with evolving semantic Gaussian+SDF scenes, so a class- or instance-level query can lock onto a persistent object and shift detail toward it.

The core idea is to reallocate a fixed resident Gaussian budget toward the selected object while keeping the rest of the scene intact enough to preserve context. To keep updates stable as the scene evolves, the pipeline coordinates Gaussian and TSDF-derived mesh changes through a source-epoch mechanism. That matters for VR because popping, identity drift, and inconsistent updates are exactly the kind of artifacts that break immersion when users start poking at objects up close.

Across eight scenes, the system concentrates 88-90% of the fixed client budget on the queried object without enlarging it, which is the kind of efficiency result that makes this interesting for real headset deployment. On-device rendering stays close to the host render, with a reported margin of up to 36.7 dB, and the standalone client renders each stereo frame at roughly 10 ms on the GPU in steady state. That puts it in the range needed for standard standalone panels.

For developers, the practical takeaway is that semantic intent can be a better driver for detail allocation than classic visibility-only LOD heuristics, especially in VR inspection workflows. If...