Dr. Splat: Directly Referring 3D Gaussian Splatting via Direct Language Embedding Registration

Kim Jun-Seong1, GeonU Kim2, Kim Yu-Ji2, Yu-Chiang Frank Wang3, Jaesung Choe3†, Tae-Hyun Oh4†
1Dept. of Electrical Engineering, POSTECH, 2Grad. School of AI, POSTECH, 3NVIDIA, Taiwan, 4School of Computing, KAIST
†denotes corresponding authors
Arxiv 2025
arXiv GitHub

Abstract

We introduce Dr. Splat, a novel approach for open vocabulary 3D scene understanding leveraging 3D Gaussian Splatting. Unlike existing language-embedded 3DGS methods, which rely on a rendering process, our method directly associates language-aligned CLIP embeddings with 3D Gaussians for holistic 3D scene understanding. The key of our method is a language feature registration technique where CLIP embeddings are assigned to the dominant Gaussians intersected by each pixel-ray. Moreover, we integrate Product Quantization (PQ) trained on general large scale image data to compactly represent embeddings without per-scene optimization. Experiments demonstrate that our approach significantly outperforms existing approaches in 3D perception benchmarks, such as open-vocabulary 3D semantic segmentation, 3D object localization, and 3D object selection tasks.

Dr.Splat

Dr. Splat directly associates CLIP embeddings with 3D Gaussians for open-vocabulary 3D scene understanding, achieving state-of-the-art 3D perception without rendering.

3D object selection

3D object localization

3D semantic segmentation

Large scene results

BibTeX


    @inproceedings{Drsplats,
        title={Dr. Splat: Directly Referring 3D Gaussian Splatting via Direct Language Embedding Registration},
        author={Jun-Seong, Kim and Kim GeonU and Yu-Ji, Kim and Yu-Chiang Frank Wang and Jaesung Choe and Oh, Tae-Hyun},
        booktitle={Arxiv},
        year={2025}
    }