MELON: 未知の姿勢の画像から3Dオブジェクトを再構築

<span class="byline-author">Posted by Mark Matthews, Senior Software Engineer, and Dmitry Lagun, Research Scientist, Google Research</span>

<img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh8LjCbKjfNXVUyCpGiZysx_pNF5BK8p5VBCJXXPaz_Bb75CW-33weoMh0YaNcn4AdmGN-Pufd_XlsRzo2MWZLQxqgtri7Nip9tXoGX0CritvRKF-63StOWxp_gVaY-MTnOk9IvJdVt_CczVR6Ip_R8Yv32MHTw2-FckCTF4UOFrgMyq3PCPCkZaZ-nyMcE/s320/MELON%20HERO.jpg" style="display: none;" />

<p>

A person's prior experience and understanding of the world generally enables them to easily infer what an object looks like in whole, even if only looking at a few 2D pictures of it. Yet the capacity for a computer to reconstruct the shape of an object in 3D given only a few images has remained a difficult algorithmic problem for years. This fundamental computer vision task has applications ranging from the creation of e-commerce 3D models to autonomous vehicle navigation.

</p>

<a name='more'></a>

<p>

A key part of the problem is how to determine the exact positions from which images were taken, known as <em>pose inference</em>. If camera poses are known, a range of successful techniques — such as <a href="https://www.matthewtancik.com/nerf">neural radiance fields</a> (NeRF) or <a href="https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/">3D Gaussian Splatting</a> — can reconstruct an object in 3D. But if these poses are not available, then we face a difficult “chicken and egg” problem where we could determine the poses if we knew the 3D object, but we can’t reconstruct the 3D object until we know the camera poses. The problem is made harder by pseudo-symmetries — i.e., many objects look similar when viewed from different angles. For example, square objects like a chair tend to look similar every 90° rotation. Pseudo-symmetries of an object can be revealed by rendering it on a turntable from various angles and plotting its photometric <a href="https://en.wikipedia.org/wiki/Self-similarity">self-similarity</a> map.

</p>

<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjt0nP5M8f5UodttSIPoY5t0JRXEuLosGgock3B0lyOzIn4icGF5jwVuxgX0PiRqc0kBbJ36CLiGA3KPrmaQbjKElGeHrsSRmkpDppU9abE84nuYu9MquqE3gULDzz_INDutmL2i1Wv3_tUpTh5U9UwSck9YRUeVyg-md2GByg3EQYYy7Vs_aeTEk5akpSo/s1764/image5.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="923" data-original-width="1764" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjt0nP5M8f5UodttSIPoY5t0JRXEuLosGgock3B0lyOzIn4icGF5jwVuxgX0PiRqc0kBbJ36CLiGA3KPrmaQbjKElGeHrsSRmkpDppU9abE84nuYu9MquqE3gULDzz_INDutmL2i1Wv3_tUpTh5U9UwSck9YRUeVyg-md2GByg3EQYYy7Vs_aeTEk5akpSo/s16000/image5.png" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Self-Similarity map of a toy truck model. <strong>Left:</strong> The model is rendered on a turntable from various <a href="https://en.wikipedia.org/wiki/Azimuth">azimuthal angles</a>, θ. <strong>Right:</strong> The average <a href="https://en.wikipedia.org/wiki/Norm_(mathematics)#Euclidean_norm">L2</a> RGB similarity of a rendering from θ with that of θ*. The pseudo-similarities are indicated by the dashed red lines.</td></tr></tbody></table>

<p>

The diagram above only visualizes one dimension of rotation. It becomes even more complex (and difficult to visualize) when introducing more degrees of freedom. Pseudo-symmetries make the problem <em>ill-posed</em>, with naïve approaches often converging to local minima. In practice, such an approach might mistake the back view as the front view of an object, because they share a similar silhouette. Previous techniques (such as <a href="https://chenhsuanlin.bitbucket.io/bundle-adjusting-NeRF/">BARF</a> or <a href="https://arxiv.org/abs/2205.15768">SAMURAI</a>) side-step this problem by relying on an initial pose estimate that starts close to the global minima. But how can we approach this if those aren’t available?

</p>

<p>

Methods, such as <a href="https://openaccess.thecvf.com/content/ICCV2021/papers/Meng_GNeRF_GAN-Based_Neural_Radiance_Field_Without_Posed_Camera_ICCV_2021_paper.pdf">GNeRF</a> and <a href="https://dl.acm.org/doi/10.1145/3503161.3548078">VMRF</a> leverage <a href="https://en.wikipedia.org/wiki/Generative_adversarial_network">generative adversarial networks</a> (GANs) to overcome the problem. These techniques have the ability to artificially “amplify” a limited number of training views, aiding reconstruction. GAN techniques, however, often have complex, so