🚀 NeuralGS: Bridging Neural Fields and 3D Gaussian Splatting for Compact 3D Representation

Anonymous CVPR submission

Paper ID 801

We propose NeuralGS, a novel framework that effectively adopts the neural field representation to encode the attributes of 3D Gaussians with multiple tiny MLPs, only requiring a small model size even for a large-scale scene.

📌 Abstract


Recently, 3D Gaussian Splatting (3DGS) has gained popularity, demonstrating superior quality and rendering speed in novel view synthesis. However, 3DGS requires millions of 3D Gaussians, each with extensive associated attributes, resulting in significant storage and transmission costs. In contrast, neural fields like NeRF can represent complex 3D scenes with Multi-Layer Perceptron (MLP) neural networks using only a few megabytes. In this paper, we present a novel Gaussian compression method NeuralGS that effectively adopts the neural field representation to encode the attributes of 3D Gaussians with MLPs, requiring a small storage size even for a large-scale scene.However, naively fitting the Gaussian attributes with an MLP network leads to severely degenerated quality. To address this, we adopt a clustering strategy and fit the Gaussians with multiple tiny MLPs for different clusters, based on importance scores of Gaussians as fitting weights.We validate our approach on multiple datasets, achieving a 38x average model size reduction without harming the visual quality.

💡 Compression Method

We propose a simple yet effective framework NeuralGS, which adopts the neural field representation to encode the attributes of 3D Gaussians with MLPs to enable a compact 3D representation. Specifically, as shown in Figure 1, we use a designed criterion to assess the importance of each Gaussian, allowing us to prune Gaussians that have minimal impact on renderings. To reduce variations among Gaussians, we cluster these 3D Gaussians based on their attributes, ensuring similarity within each cluster. Then, each cluster is then assigned a tiny MLP that fits the attributes of its 3D Gaussians. Given the varying contributions of each Gaussian to the renderings, we apply Gaussian’s importances as the fitting weights in the MLP fitting. We also incorporate a fine-tuning stage along with frequency loss to restore quality and preserve high-frequency details. We address the storage and rendering issue of 3D Gaussian Splatting (3DGS) by compressing the reconstructed scene parameters and rendering the compressed representation via GPU rasterization. To compress the scenes, we first analyze its components and observe that the Spherical Harmonics (SH) coefficients and the multivariate Gaussian parameters take up the majority of storage space and are highly redundant. Our compression pipeline consists of three steps:


Compression Pipeline
Figure 1: Overview of NeuralGS.

🔗 Results

We evaluate our method on four datasets for comparison on NVIDIA A100 GPUs.


Qualitative Comparisons

Compression Pipeline
Figure 2: Qualitative comprison on the mentioned four datasets.

Quantitative Comparisons

Table 1. Quantitative results evaluated on Mip-NeRF 360, Tanks&Temples, and Deep Blending datasets. We highlight the best-performing results in red and the second-best results in yellow for all compression methods
Compression Pipeline

Table 2. Quantitative results of the proposed method evaluated on the NeRF-Synthetic dataset. We highlight the best-performing results in red and the second-best results in yellow for all compression methods.
Table 2

🙋‍♂️ Conclusion

In this paper, we introduce NeuralGS, a novel and effective post-compression approach for 3D Gaussian splatting. The core of our approach lies in leveraging compact neural fields to encode the attributes of 3D Gaussians with MLPs, significantly reducing the memory requirements of 3DGS. Thus, we design multiple neural fields based on clusters and incorporate importance scores as fitting weights to enhance the fittting quality of Gaussian attributes. Additionally, we introduce frequency loss during the fine-tuning stage to further preserve high-frequency details. Extensive experiments demonstrate that our method achieves comparable or even superior performance to existing compression methods while utilizing less model size. Overall, NeuralGS significantly alleviates the storage challenges of 3DGS, paving the way for its broader application in large-scale scenes.