TRANSPR: Transparency Ray-Accumulating Neural 3D Scene Point Renderer

3DV 2020


* indicates equal contribution

TL;DR: We extend the Neural Point-Based Graphics (NPBG) by the ability of rendering transparent scenes, both synthetic and captured in-the-wild. Our neural pipeline is capable of performing photorealistic view synthesis of 3d point clouds of objects with semi-transparent parts and complex geometry. Several scenes can be rendered in conjunction, opacity of objects can be edited, and the non-transparent objects can be combined with the introduced transparency.

Abstract

We propose and evaluate a neural point-based graphics method that can model semi-transparent scene parts. Similarly to its predecessor pipeline, ours uses point clouds to model proxy geometry, and augments each point with a neural descriptor. Additionally, a learnable transparency value is introduced in our approach for each point.

Our neural rendering procedure consists of two steps. Firstly, the point cloud is rasterized using ray grouping into a multi-channel image. This is followed by the neural rendering step that "translates" the rasterized image into an RGB output using a learnable convolutional network. New scenes can be modeled using gradient-based optimization of neural descriptors and of the rendering network.

We show that novel views of semi-transparent point cloud scenes can be generated after training with our approach. Our experiments demonstrate the benefit of introducing semi-transparency into the neural point-based modeling for a range of scenes with semi-transparent parts.

Synthetic data

We leverage synthetic RGBA data to demonstrate the ability of TRANSPR to learn physically-based transparency with 4-channel supervision and compare results with the state-of-the-art methods: NPBG, Neural Volumes, and NeRF.

Setting the highest quality is recommended. Original video: aquarium.mp4.


Setting the highest quality is recommended. Original video: static_smoke.mp4.


Dynamic rendering

We show how TRANSPR can model the dynamic behavior of smoke by interpolating descriptors for every second animation frame.

Each animation frame was treated like a separate scene. TRANSPR linearly interpolates the descriptors, and Neural Volumes employes a view-conditioning strategy based on 3 nearest train cameras. The inferred NeRF images for the trained frames were linearly interpolated. The following demo showcases a single frame rendering, as well as the comparison with the ground truth.

Setting the highest quality is recommended. Original video: dyn_smoke.mp4.


Transparent photogrammetry

We present the comparative performance of TRANSPR on an in-the-wild semi-transparent scene trained using only RGB supervision.

Transparent glass is an extremely challenging surface for photogrammetric reconstruction, so a special scenario with two video sequences was considered for the scene to obtain a complex geometry. First, a 180° sequence of flowers in a transparent vase was captured as is. Afterwards, the vase was wrapped into a paper with checkerboard pattern. Finally, point clouds of each sequence were reconstructed and geometrically aligned.


Sequence without wrapping
Sequence with wrapping



Setting the highest quality is recommended. Original video: flowers.mp4.


Transparency manipulation

We demonstrate the possibility of TRANSPR to alter the learned transparency of the objects on Chiffon shirt and Scarf scenes.

Setting the highest quality is recommended. Original video: scarf_table.mp4.


Setting the highest quality is recommended. Original video: scarf_and_shirt.mp4.


Scene composition

We show how TRANSPR extends the scene editing scenario originally proposed in NPBG with added or altered transparency allowing to jointly render synthetic and real-world assets.

Setting the highest quality is recommended. Original video: aloe_smoke.mp4.



Setting the highest quality is recommended. Original video: jewelry_smoke.mp4.



Setting the highest quality is recommended. Original video: church.mp4.



Citation
BibTeX:
@misc{kolos2020transpr,
      title={TRANSPR: Transparency Ray-Accumulating Neural 3D Scene Point Renderer},
      author={Maria Kolos and Artem Sevastopolsky and Victor Lempitsky},
      year={2020},
      eprint={2009.02819},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
    }