Abstract
Today, most methods for image understanding tasks rely on feed-forward neural networks. While this approach has allowed for empirical accuracy, efficiency, and task adaptation via fine-tuning, it also comes with fundamental disadvantages. Existing networks often struggle to generalize across different datasets, even on the same task. By design, these networks ultimately reason about high-dimensional scene features, which are challenging to analyze. This is true especially when attempting to predict 3D information based on 2D images. We propose to recast 3D multi-object tracking from RGB cameras as an \emph{Inverse Rendering (IR)} problem, by optimizing via a differentiable rendering pipeline over the latent space of pre-trained 3D object representations and retrieve the latents that best represent object instances in a given input image. To this end, we optimize an image loss over generative latent spaces that inherently disentangle shape and appearance properties. We investigate not only an alternate take on tracking but our method also enables examining the generated objects, reasoning about failure situations, and resolving ambiguous cases. We validate the generalization and scaling capabilities of our method by learning the generative prior exclusively from synthetic data and assessing camera-based 3D tracking on the nuScenes and Waymo datasets. Both these datasets are completely unseen to our method and do not require fine-tuning. Videos and code are available at this https URL.
Abstract (translated)
today，大多数图像理解任务的方法都依赖于前馈神经网络。虽然这种方法通过微调获得了 empirical 的准确性和效率，但同时也存在一些基本缺陷。现有的网络往往难以在不同的数据集上泛化，即使是相同任务。通过设计，这些网络最终在预训练的3D对象表示的潜在空间中进行推理，这是具有挑战性的。尤其是在试图根据2D图像预测3D信息时，这更是如此。我们提出将从RGB相机中的3D多对象跟踪重新建模为同义词{反向渲染（IR）问题，通过优化通过不同的渲染管道在预训练3D对象表示的潜在空间中进行优化，并检索在给定输入图像中最好地表示物体实例的潜在。为此，我们优化了一个在生成性潜在空间上进行的图像损失。我们研究了不仅是对跟踪的另一种看法，而且我们的方法还允许我们检查生成的物体，推理失败情况，并解决模糊情况。我们通过仅从合成数据中学习生成先验来评估我们的方法的泛化能力和扩展能力。我们在 nuScenes 和 Waymo 数据集上对相机基于3D跟踪的性能进行了评估。这两个数据集完全未见对我们的方法，也不需要微调。视频和代码可在此处 https:// URL 下载。
URL
https://arxiv.org/abs/2404.12359
