Perturb-and-Revise: Flexible 3D Editing with Generative Trajectories
Abstract
Recent advancements in text-based diffusion models have accelerated progress in 3D reconstruction and text-based 3D editing. Although existing 3D editing methods excel at modifying color, texture, and style, they struggle with extensive geometric or appearance changes, thus limiting their applications. To this end, we propose Perturb-and-Revise, which makes possible a variety of NeRF editing. First, we perturb the NeRF parameters with random initializations to create a versatile initialization. The level of perturbation is determined automatically through analysis of the local loss landscape. Then, we revise the edited NeRF via generative trajectories. Combined with the generative process, we impose identity-preserving gradients to refine the edited NeRF. Extensive experiments demonstrate that Perturb-and-Revise facilitates flexible, effective, and consistent editing of color, appearance, and geometry in 3D.
3D Object Editing
3D Scene Editing (🐻 ➡ 🐻❄️)
Method Overview
Perturb-and-Revise takes a source NeRF and an edit prompt as input and produces the edited result through: (1) versatile initialization via parameter perturbation, (2) multi-view consistent score distillation, and (3) refinement with the identity-preserving gradient.
Parameter Perturbation
Effect of parameter perturbation. In this example, we aim to make a NeRF model of a standing person sit down using just the word "sitting." The scene converges quickly even with large perturbations (η = 0.6), while complete regeneration yields blurry rendering results given the same number of optimization steps.
Objaverse Object Editing
Fashion Object Editing
Real Scene Editing
Citation
@article{hong2024perturb,
title={Perturb-and-Revise: Flexible 3D Editing with Generative Trajectories},
author={Hong, Susung and Karras, Johanna and Martin-Brualla, Ricardo and Kemelmacher-Shlizerman, Ira},
journal={arXiv preprint arXiv:2408.00760},
year={2024}
}