GST: Precise 3D Human Body from a Single Image with Gaussian Splatting Transformers

We base our work on 3D Gaussian Splatting (3DGS), a scene representation composed of a mixture of Gaussians. Predicting such mixtures for a human from a single input image is challenging, as it is a non-uniform density (with a many-to-one relationship with input pixels) with strict physical constraints. At the same time, it needs to be flexible to accommodate a variety of clothes and poses. Our key observation is that the vertices of standardized human meshes (such as SMPL) can provide an adequate density and approximate initial position for Gaussians. We can then train a transformer model to jointly predict comparatively small adjustments to these positions, as well as the other Gaussians' attributes and the SMPL parameters. We show empirically that this combination (using only multi-view supervision) can achieve fast inference of 3D human models from a single image without test-time optimization, expensive diffusion models, or 3D points supervision. We also show that it can improve 3D pose estimation by better fitting human models that account for clothes and other variations.

Given a single input image, GST uses a Vision Transformer (ViT) to predict both a 3D human pose (in the form of SMPL parameters) and a refined full-color 3D model (in the form of 3D Gaussian Splats). Additional input tokens facilitate the output of individual Gaussians’ color c, opacity α, scale, rotation, and position offset δ. Each Gaussian’s position μ is relative to one vertex of the SMPL model v by the offset δ, and so this model can be considered a refinement or residual over the interpretable SMPL mesh, facilitating multi-view rendering with higher visual fidelity.