Skip to content
Published on

Two Paths to 3D — Reconstruction (NeRF & Gaussian Splatting) and Generation (TRELLIS & Hunyuan3D)

🇰🇷KO🇺🇸EN🇯🇵JA
Split View필사 모드
Share
Authors

Introduction — "Making 3D" Is Two Different Problems

Inside the single sentence "make me a 3D model," two completely different requests are actually mixed together.

  • Reconstruction: Bringing an object or space that actually exists back to life in 3D, from several photos (or a video). Heritage scanning, real-estate 3D tours, and a robot's spatial awareness all belong here. The ground truth is in the real world, and the goal is to reproduce it faithfully.
  • Generation: Conjuring something that does not exist, from a single line of text or one image. Game assets, product concepts, and avatars belong here. There is no ground truth, and the goal is to create something plausible and usable.

These two differ in the techniques they use and in how they are evaluated. This article follows each lineage in turn and gives you the coordinates for "where we stand right now." If you are curious about the fundamentals of image and vision models, it pairs well with the Neural Network Architecture Explorer.

Part 1 — Reconstruction: From Photos to 3D

Lineage: SfM → NeRF → Gaussian Splatting

Structure-from-Motion (SfM)   Recover camera poses + a sparse point cloud from many photos (COLMAP)
  │                            — Classical geometry. Still the essential preprocessing for pose estimation
  ▼
NeRF (2020)                    Store the scene "implicitly" in a neural network (MLP) —
  │                            a function that takes coordinates and emits color + density. View synthesis via volume rendering
  ▼
3D Gaussian Splatting (2023)   Store the scene "explicitly" as millions of "Gaussian blobs" —
                               real-time rendering via rasterization. The turning point that changed NeRF's game

SfM (Structure-from-Motion) is a classic that predates deep learning. It matches feature points across overlapping photos to recover where the camera was and a sparse 3D point cloud. COLMAP is the de facto standard, and the NeRF and Gaussian Splatting methods below mostly run SfM first as preprocessing to estimate camera poses.

NeRF (Neural Radiance Fields, 2020) was the idea of cramming an entire scene into a single neural network. For each scene, you train a small MLP that answers "if you look at this coordinate in space from this direction, what color is it and how opaque is it," and you integrate along rays (volume rendering) to produce images from new viewpoints. The quality was astonishing, but it came at a cost — training took hours to days, and rendering was slow too.

Why 3D Gaussian Splatting Changed the Game

3D Gaussian Splatting (3DGS, 2023) flipped the representation. Instead of putting the scene implicitly into a neural network, it represents the scene explicitly as millions of Gaussians (3D ellipsoidal blobs) with position, size, color, and opacity, then projects (splats) these blobs onto the screen and draws them via rasterization. The direction of the idea is reversed too — if NeRF traces 3D backward from 2D images, 3DGS starts from 3D points and produces 2D images.

The results are dramatic: 60+ FPS real-time rendering, training measured in minutes rather than hours, and fewer artifacts as well. Libraries like gsplat underpin this speed. That said, 3DGS has not completely replaced NeRF — where precise geometry is needed, SDF-based methods remain strong, and for view-dependent effects like reflection and transparency, NeRF-based methods still hold their own. Variants such as EGGS (improved geometric accuracy) and EVolSplat4D (dynamic urban scenes) keep appearing, and in practice the work is usually a hybrid.

Feedforward Reconstruction — Not Training Per Scene

Both NeRF and 3DGS fundamentally optimize once per scene. The trend in 2026 is to eliminate this "per-scene training." Meta's MapAnything (1B parameters, January 2026, accepted at 3DV 2026) handles more than 12 different 3D reconstruction tasks with a single unified transformer in one feedforward pass. Feed it photos and 3D comes out instantly without an optimization loop — this is the direction of the "real-time spatial awareness" that robotics and AR have wanted.

Part 2 — Generation: From Text and Images to 3D

Lineage: SDS Distillation → Native 3D Diffusion

DreamFusion (2022)         "Distill" (SDS) the knowledge of 2D diffusion (image generators) into 3D —
  │                        text→3D without 3D data. Slow, with blurry results
  ▼
Native 3D diffusion        Train diffusion directly on a 3D representation (voxels, triplanes, structured latents)
  │                        — trained on 3D datasets. Fast and consistent
  ▼
TRELLIS · Hunyuan3D (2025~) rectified flow + 2 stages (geometry→texture) —
                           game-ready PBR assets in tens of seconds

Early on, DreamFusion (2022) was a clever detour. At the time there was no large-scale 3D data, so it used SDS (Score Distillation Sampling) to distill the knowledge of an already-powerful 2D image diffusion model into 3D. It was a magical approach that pulled 3D from text without any 3D data, but optimization was slow, the results were blurry, and it suffered from the "Janus problem" (the error where a front face appears from every angle).

Today's mainstream is native 3D diffusion — training diffusion directly on top of 3D representations such as voxels, triplanes, and structured latents. Because it is trained on 3D datasets, it is fast and has good multi-view consistency. The common recipe of the latest models is two-stage generation: first build the geometry (shape), then apply the appearance (texture) on top of it.

The 2026 Landscape

Representative models that pull 3D from images and text (per HuggingFace):

Model                Strength                      Notes
──────────────────  ───────────────────────────  ──────────────────────────────
TRELLIS.2 (4B)       Production-grade PBR assets   rectified flow DiT, 1536³ resolution,
                                                    4K textures, complex topology & transparent materials
Hunyuan3D 3.5        High-quality textures         flow-based, under 60s, up to 8K PBR
Hi3DGen              Best geometry quality         shape-accuracy focused
Stable Fast 3D       Speed (under 1s)              suited to real-time previews & bulk generation

TRELLIS.2 (4B-parameter DiT) uses a two-stage pipeline that first captures a sparse structure with rectified flow and then decodes it into a structured latent, producing production-grade assets at 1536³ resolution and 4K textures. Tencent's Hunyuan3D pulled PBR, game-ready assets in about 10 seconds with 2.1, and climbed to under 60 seconds and up to 8K PBR textures with 3.5. Hi3DGen targets geometric accuracy and Stable Fast 3D targets sub-1-second speed, each attacking a different axis. The selection criterion is simple — for games/products, go for PBR quality (TRELLIS.2 & Hunyuan3D); for bulk previews, go for speed (Stable Fast 3D).

Part 3 — Representations and Output Formats

For 3D, "what you represent it with" determines what it can be used for.

Representation   What it is                     Where it's used
─────────────  ─────────────────────────────  ──────────────────────────────
Point cloud      Points with position + color   SfM output, LiDAR, intermediate artifacts
Mesh             Vertices + faces (triangles)   Games & film — DCC-tool standard. UV + PBR textures
Gaussians        Position + covariance + color + opacity   Real-time view synthesis (3DGS). Not a mesh yet
Implicit field   Coordinate→value function (SDF/NeRF)   Precise geometry & smooth surfaces

The realistic goal of a pipeline is usually a mesh with PBR textures applied — because it is the standard that game engines (Unity, Unreal) and 3D authoring tools (Blender) consume. PBR (Physically Based Rendering) describes the physics of a material with albedo, roughness, metallic, and normal maps so that it looks realistic under any lighting. That is why the sales point of the latest generative models is "game-ready PBR" — Gaussians and NeRF may look great, but they are not yet meshes, so putting them into a game requires an extra step of converting them into meshes.

Part 4 — Real-World Pipelines

Reconstruction (turning my own object into 3D):

① Capture      Shoot the object from many overlapping angles (video works too). Watch for blur & focus
② Pose est.    Camera poses + a sparse point cloud with COLMAP (SfM)
③ Training     Pick gsplat (3DGS) or a NeRF method in Nerfstudio — minutes to hours
④ Render/extract   Check in the real-time viewer, extract to a mesh if needed

Nerfstudio is the most widely used open-source framework supporting both NeRF and 3DGS, letting you experiment with many variants through a single CLI. As a starting point, there is nothing quite like it.

Generation (from nothing to 3D): Run HuggingFace's open models (TRELLIS, Hunyuan3D) yourself, or throw a single image at a commercial API (Meshy, Rodin, 3DAI Studio, and the like) and a mesh + PBR comes out in tens of seconds. The sense for data preparation carries over from the LLM Data Preprocessing article — 3D generative models too are ultimately governed by the curation quality of large-scale 3D datasets like Objaverse.

Part 5 — An Engineer's On-Ramp

The notion that "3D is the domain of graphics majors only" is outdated. For VLA robots (Robotics Companies Map) to understand space they need reconstruction, and content pipelines want generation — both are short of people right now. A realistic three steps:

  1. Gaussian Splatting once — Shoot an object all the way around with your smartphone and run 3DGS through Nerfstudio. The experience of "an object on my desk spinning in 3D" gives you the whole picture.
  2. A generative model once — Feed a single image into TRELLIS or Hunyuan3D to pull a mesh and open it in Blender. The feel for output formats and PBR sticks to your hands.
  3. Two papers — the original 3D Gaussian Splatting paper (reconstruction) and TRELLIS (generation). Those two give you a coordinate system for today's landscape.

For an existing ML engineer, 3D is less a "new field" than one more generation/reconstruction problem whose output is space rather than an image — the grammar of diffusion, transformers, and data curation stays the same.

Closing

There are two paths to making 3D. Reconstruction, which revives reality, has advanced from SfM through NeRF to real-time Gaussian Splatting, and on toward feedforward that does not train per scene. Generation, which builds from nothing, has matured from 2D distillation (DreamFusion) to native 3D diffusion, and on to TRELLIS and Hunyuan3D that pull game-ready PBR in tens of seconds. Both paths ultimately sit on the same engines of diffusion and transformers — so if you are an engineer who has worked with LLMs and image models, your skills are already halfway to being 3D skills.

References

Discuss on Twitter • View on GitHub