Path-space motion estimation and decomposition for robust animation filtering

1Disney Research Zürich 2ETH Zürich 3Walt Disney Animation Studios

In Computer Graphics Forum (Proceedings of EGSR), 2015

Teaser
Starting from noisy, low-resolution frames generated with a path tracer (red borders), our method improves quality and reduces computational cost by computing spatially and temporally upsampled and denoised frames (green and blue borders) while properly preserving view-dependent shading effects like the reflections in the picture frame and on the robot.
Abstract Downloads Videos Cite Related

Abstract

Renderings of animation sequences with physics-based Monte Carlo light transport simulations are exceedingly costly to generate frame-by-frame, yet much of this computation is highly redundant due to the strong coherence in space, time and among samples. A promising approach pursued in prior work entails subsampling the sequence in space, time, and number of samples, followed by image-based spatio-temporal upsampling and denoising.

These methods can provide significant performance gains, though major issues remain: firstly, in a multiple scattering simulation, the final pixel color is the composite of many different light transport phenomena, and this conflicting information causes artifacts in image-based methods. Secondly, motion vectors are needed to establish correspondence between the pixels in different frames, but it is unclear how to obtain them for most kinds of light paths (e.g. an object seen through a curved glass panel).

To reduce these ambiguities, we propose a general decomposition framework, where the final pixel color is separated into components corresponding to disjoint subsets of the space of light paths. Each component is accompanied by motion vectors and other auxiliary features such as reflectance and surface normals. The motion vectors of specular paths are computed using a temporal extension of manifold exploration and the remaining components use a specialized variant of optical flow. Our experiments show that this decomposition leads to significant improvements in three image-based applications: denoising, spatial upsampling, and temporal interpolation.

Downloads

Videos

Video (full-quality)

Cite

Henning Zimmer, Fabrice Rousselle, Wenzel Jakob, Oliver Wang, David Adler, Wojciech Jarosz, Olga Sorkine-Hornung, Alexander Sorkine-Hornung. Path-space motion estimation and decomposition for robust animation filtering. Computer Graphics Forum (Proceedings of EGSR), 34(4), June 2015. 
@article{zimmer15path,
    author  = {Zimmer, Henning and Rousselle, Fabrice and Jakob, Wenzel and Wang, Oliver and Adler, David and Jarosz,
               Wojciech and Sorkine-Hornung, Olga and Sorkine-Hornung, Alexander},
    title   = {Path-space Motion Estimation and Decomposition for Robust Animation Filtering},
    journal = {Computer Graphics Forum (Proceedings of EGSR)},
    volume  = {34},
    number  = {4},
    month   = jun,
    year    = {2015},
    doi     = {10/f7mb34}
}
© The Author(s). This is the author's version of the work. It is posted here by permission of The Eurographics Association for your personal use. Not for redistribution. The definitive version is available at diglib.eg.org.