Empowering Neural Rendering Methods with Physically-Based Capabilities

Introduction

While long restricted to an elite of expert digital artists, 3D content creation has recently been greatly simplified by deep learning. Neural representations of 3D objects have revolutionized real-world capture from photos, while generative models are starting to enable 3D object synthesis from text prompts.

Limitations of Current Methods

These methods use differentiable neural rendering that allows efficient optimization of the powerful and expressive "soft" neural representations, but ignores physically-based principles. Thus, it has no guarantees on accuracy, severely limiting the utility of the resulting content.

Differentiable physically-based rendering, on the other hand, can produce 3D assets with physics-based parameters. However, it depends on rigid traditional "hard" graphics representations required for light-transport computation, which makes optimization much harder and is also costly, limiting applicability.

NERPHYS Approach

In NERPHYS, we will combine the strengths of both neural and physically-based rendering, lifting their respective limitations by introducing polymorphic 3D representations. These representations will be capable of morphing between different states to accommodate both efficient gradient-based optimization and physically-based light transport.

By augmenting these representations with corresponding polymorphic differentiable renderers, our methodology will unleash the potential of neural rendering to produce physically-based 3D assets with guarantees on accuracy.

Impact of NERPHYS

NERPHYS will have a ground-breaking impact on 3D content creation, moving beyond today's simplistic plausible imagery to full physically-based rendering with guarantees on error. This will enable the use of powerful neural rendering methods in any application requiring accuracy.

Our polymorphic approach will fundamentally change how we reason about scene representations for geometry and appearance, while our rendering algorithms will provide a new methodology for image synthesis, e.g., for training data generation or visual effects.