Semantic_Instance_Segmentation
-
FreePoint: Unsupervised Point Cloud Instance Segmentation
2023-05-11 16:56:26Zhikai Zhang, Jian Ding, Li Jiang, Dengxin Dai, Gui-Song XiaAbstract
Instance segmentation of point clouds is a crucial task in 3D field with numerous applications that involve localizing and segmenting objects in a scene. However, achieving satisfactory results requires a large number of manual annotations, which is a time-consuming and expensive process. To alleviate dependency on annotations, we propose a method, called FreePoint, for underexplored unsupervised class-agnostic instance segmentation on point clouds. In detail, we represent the point features by combining coordinates, colors, normals, and self-supervised deep features. Based on the point features, we perform a multicut algorithm to segment point clouds into coarse instance masks as pseudo labels, which are used to train a point cloud instance segmentation model. To alleviate the inaccuracy of coarse masks during training, we propose a weakly-supervised training strategy and corresponding loss. Our work can also serve as an unsupervised pre-training pretext for supervised semantic instance segmentation with limited annotations. For class-agnostic instance segmentation on point clouds, FreePoint largely fills the gap with its fully-supervised counterpart based on the state-of-the-art instance segmentation model Mask3D and even surpasses some previous fully-supervised methods. When serving as a pretext task and fine-tuning on S3DIS, FreePoint outperforms training from scratch by 5.8% AP with only 10% mask annotations.
Abstract (translated)
点云实例分割是三维领域中一个重要的任务，有许多应用涉及在场景中定位和分割物体。然而，要实现满意的结果需要大量的手动标注，这是一个耗时且昂贵的过程。为了减轻依赖标注的情况，我们提出了一种方法，称为FreePoint，用于未受重视的点云 unsupervised 类特异性实例分割。具体来说，我们使用坐标、颜色、正弦和自监督的深度特征来代表点特征。基于点特征，我们执行多切算法将点云分割成粗实例 masks 作为伪标签，用于训练点云实例分割模型。为了减轻训练期间粗标签的不准确情况，我们提出了一种较弱的supervised 训练策略和相应的损失。我们的工作还可以作为 unsupervised 的前序训练目标，以有限标注的 supervised 语义实例分割。对于点云的类特异性实例分割，FreePoint 在很大程度上填补基于先进的实例分割模型Mask3D的未受重视的 gap 并与一些以前的完全监督方法相比。当作为前序训练目标和在S3DIS上进行 fine-tuning时，FreePoint 通过仅10%的 mask 标注比从零开始训练取得了5.8%的AP优势。
URL
https://arxiv.org/abs/2305.06973
PDF
-
Mask3D for 3D Semantic Instance Segmentation
2022-10-06 17:55:09Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, Bastian LeibeAbstract
Modern 3D semantic instance segmentation approaches predominantly rely on specialized voting mechanisms followed by carefully designed geometric clustering techniques. Building on the successes of recent Transformer-based methods for object detection and image segmentation, we propose the first Transformer-based approach for 3D semantic instance segmentation. We show that we can leverage generic Transformer building blocks to directly predict instance masks from 3D point clouds. In our model called Mask3D each object instance is represented as an instance query. Using Transformer decoders, the instance queries are learned by iteratively attending to point cloud features at multiple scales. Combined with point features, the instance queries directly yield all instance masks in parallel. Mask3D has several advantages over current state-of-the-art approaches, since it neither relies on (1) voting schemes which require hand-selected geometric properties (such as centers) nor (2) geometric grouping mechanisms requiring manually-tuned hyper-parameters (e.g. radii) and (3) enables a loss that directly optimizes instance masks. Mask3D sets a new state-of-the-art on ScanNet test (+6.2 mAP), S3DIS 6-fold (+10.1 mAP), STPLS3D (+11.2 mAP) and ScanNet200 test (+12.4 mAP).
Abstract (translated)
URL
https://arxiv.org/abs/2210.03105
PDF
-
Semantic Instance Segmentation of 3D Scenes Through Weak Bounding Box Supervision
2022-06-02 17:59:57Julian Chibane, Francis Engelmann, Tuan Anh Tran, Gerard Pons-MollAbstract
Current 3D segmentation methods heavily rely on large-scale point-cloud datasets, which are notoriously laborious to annotate. Few attempts have been made to circumvent the need for dense per-point annotations. In this work, we look at weakly-supervised 3D instance semantic segmentation. The key idea is to leverage 3D bounding box labels which are easier and faster to annotate. Indeed, we show that it is possible to train dense segmentation models using only weak bounding box labels. At the core of our method, Box2Mask, lies a deep model, inspired by classical Hough voting, that directly votes for bounding box parameters, and a clustering method specifically tailored to bounding box votes. This goes beyond commonly used center votes, which would not fully exploit the bounding box annotations. On ScanNet test, our weakly supervised model attains leading performance among other weakly supervised approaches (+18 mAP50). Remarkably, it also achieves 97% of the performance of fully supervised models. To prove the practicality of our approach, we show segmentation results on the recently released ARKitScenes dataset which is annotated with 3D bounding boxes only, and obtain, for the first time, compelling 3D instance segmentation results.
Abstract (translated)
URL
https://arxiv.org/abs/2206.01203
PDF
-
PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoning
2021-12-09 18:59:34Yining Hong, Li Yi, Joshua B. Tenenbaum, Antonio Torralba, Chuang GanAbstract
A critical aspect of human visual perception is the ability to parse visual scenes into individual objects and further into object parts, forming part-whole hierarchies. Such composite structures could induce a rich set of semantic concepts and relations, thus playing an important role in the interpretation and organization of visual signals as well as for the generalization of visual perception and reasoning. However, existing visual reasoning benchmarks mostly focus on objects rather than parts. Visual reasoning based on the full part-whole hierarchy is much more challenging than object-centric reasoning due to finer-grained concepts, richer geometry relations, and more complex physics. Therefore, to better serve for part-based conceptual, relational and physical reasoning, we introduce a new large-scale diagnostic visual reasoning dataset named PTR. PTR contains around 70k RGBD synthetic images with ground truth object and part level annotations regarding semantic instance segmentation, color attributes, spatial and geometric relationships, and certain physical properties such as stability. These images are paired with 700k machine-generated questions covering various types of reasoning types, making them a good testbed for visual reasoning models. We examine several state-of-the-art visual reasoning models on this dataset and observe that they still make many surprising mistakes in situations where humans can easily infer the correct answer. We believe this dataset will open up new opportunities for part-based reasoning.
Abstract (translated)
URL
https://arxiv.org/abs/2112.05136
PDF
-
DSP-SLAM: Object Oriented SLAM with Deep Shape Priors
2021-08-21 10:00:12Jingwen Wang, Martin Rünz, Lourdes AgapitoAbstract
We propose DSP-SLAM, an object-oriented SLAM system that builds a rich and accurate joint map of dense 3D models for foreground objects, and sparse landmark points to represent the background. DSP-SLAM takes as input the 3D point cloud reconstructed by a feature-based SLAM system and equips it with the ability to enhance its sparse map with dense reconstructions of detected objects. Objects are detected via semantic instance segmentation, and their shape and pose is estimated using category-specific deep shape embeddings as priors, via a novel second order optimization. Our object-aware bundle adjustment builds a pose-graph to jointly optimize camera poses, object locations and feature points. DSP-SLAM can operate at 10 frames per second on 3 different input modalities: monocular, stereo, or stereo+LiDAR. We demonstrate DSP-SLAM operating at almost frame rate on monocular-RGB sequences from the Friburg and Redwood-OS datasets, and on stereo+LiDAR sequences on the KITTI odometry dataset showing that it achieves high-quality full object reconstructions, even from partial observations, while maintaining a consistent global map. Our evaluation shows improvements in object pose and shape reconstruction with respect to recent deep prior-based reconstruction methods and reductions in camera tracking drift on the KITTI dataset.
Abstract (translated)
URL
https://arxiv.org/abs/2108.09481
PDF
-
HIDA: Towards Holistic Indoor Understanding for the Visually Impaired via Semantic Instance Segmentation with a Wearable Solid-State LiDAR Sensor
2021-07-07 12:23:53Huayao Liu, Ruiping Liu, Kailun Yang, Jiaming Zhang, Kunyu Peng, Rainer StiefelhagenAbstract
Independently exploring unknown spaces or finding objects in an indoor environment is a daily but challenging task for visually impaired people. However, common 2D assistive systems lack depth relationships between various objects, resulting in difficulty to obtain accurate spatial layout and relative positions of objects. To tackle these issues, we propose HIDA, a lightweight assistive system based on 3D point cloud instance segmentation with a solid-state LiDAR sensor, for holistic indoor detection and avoidance. Our entire system consists of three hardware components, two interactive functions~(obstacle avoidance and object finding) and a voice user interface. Based on voice guidance, the point cloud from the most recent state of the changing indoor environment is captured through an on-site scanning performed by the user. In addition, we design a point cloud segmentation model with dual lightweight decoders for semantic and offset predictions, which satisfies the efficiency of the whole system. After the 3D instance segmentation, we post-process the segmented point cloud by removing outliers and projecting all points onto a top-view 2D map representation. The system integrates the information above and interacts with users intuitively by acoustic feedback. The proposed 3D instance segmentation model has achieved state-of-the-art performance on ScanNet v2 dataset. Comprehensive field tests with various tasks in a user study verify the usability and effectiveness of our system for assisting visually impaired people in holistic indoor understanding, obstacle avoidance and object search.
Abstract (translated)
URL
https://arxiv.org/abs/2107.03180
PDF
-
Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding
2020-11-04 20:12:07Mike Roberts, Nathan PaczanAbstract
For many fundamental scene understanding tasks, it is difficult or impossible to obtain per-pixel ground truth labels from real images. We address this challenge by introducing Hypersim, a photorealistic synthetic dataset for holistic indoor scene understanding. To create our dataset, we leverage a large repository of synthetic scenes created by professional artists, and we generate 77,400 images of 461 indoor scenes with detailed per-pixel labels and corresponding ground truth geometry. Our dataset: (1) relies exclusively on publicly available 3D assets; (2) includes complete scene geometry, material information, and lighting information for every scene; (3) includes dense per-pixel semantic instance segmentations for every image; and (4) factors every image into diffuse reflectance, diffuse illumination, and a non-diffuse residual term that captures view-dependent lighting effects. Together, these features make our dataset well-suited for geometric learning problems that require direct 3D supervision, multi-task learning problems that require reasoning jointly over multiple input and output modalities, and inverse rendering problems. We analyze our dataset at the level of scenes, objects, and pixels, and we analyze costs in terms of money, annotation effort, and computation time. Remarkably, we find that it is possible to generate our entire dataset from scratch, for roughly half the cost of training a state-of-the-art natural language processing model. All the code we used to generate our dataset will be made available online.
Abstract (translated)
URL
https://arxiv.org/abs/2011.02523
PDF
-
3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation
2020-03-30 23:28:50Francis Engelmann, Martin Bokeloh, Alireza Fathi, Bastian Leibe, Matthias NießnerAbstract
We present 3D-MPA, a method for instance segmentation on 3D point clouds. Given an input point cloud, we propose an object-centric approach where each point votes for its object center. We sample object proposals from the predicted object centers. Then, we learn proposal features from grouped point features that voted for the same object center. A graph convolutional network introduces inter-proposal relations, providing higher-level feature learning in addition to the lower-level point features. Each proposal comprises a semantic label, a set of associated points over which we define a foreground-background mask, an objectness score and aggregation features. Previous works usually perform non-maximum-suppression (NMS) over proposals to obtain the final object detections or semantic instances. However, NMS can discard potentially correct predictions. Instead, our approach keeps all proposals and groups them together based on the learned aggregation features. We show that grouping proposals improves over NMS and outperforms previous state-of-the-art methods on the tasks of 3D object detection and semantic instance segmentation on the ScanNetV2 benchmark and the S3DIS dataset.
Abstract (translated)
URL
https://arxiv.org/abs/2003.13867
PDF
-
Layered Embeddings for Amodal Instance Segmentation
2020-02-14 22:00:45Yanfeng Liu, Eric Psota, Lance PérezAbstract
The proposed method extends upon the representational output of semantic instance segmentation by explicitly including both visible and occluded parts. A fully convolutional network is trained to produce consistent pixel-level embedding across two layers such that, when clustered, the results convey the full spatial extent and depth ordering of each instance. Results demonstrate that the network can accurately estimate complete masks in the presence of occlusion and outperform leading top-down bounding-box approaches. Source code available at this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2002.06264
PDF
-
Rescan: Inductive Instance Segmentation for Indoor RGBD Scans
2019-09-25 03:20:42Maciej Halber, Yifei Shi, Kai Xu, Thomas FunkhouserAbstract
In depth-sensing applications ranging from home robotics to AR/VR, it will be common to acquire 3D scans of interior spaces repeatedly at sparse time intervals (e.g., as part of regular daily use). We propose an algorithm that analyzes these "rescans" to infer a temporal model of a scene with semantic instance information. Our algorithm operates inductively by using the temporal model resulting from past observations to infer an instance segmentation of a new scan, which is then used to update the temporal model. The model contains object instance associations across time and thus can be used to track individual objects, even though there are only sparse observations. During experiments with a new benchmark for the new task, our algorithm outperforms alternate approaches based on state-of-the-art networks for semantic instance segmentation.
Abstract (translated)
URL
https://arxiv.org/abs/1909.11268
PDF
-
BERTgrid: Contextualized Embedding for 2D Document Representation and Understanding
2019-09-11 09:51:02Timo I. Denk, Christian ReisswigAbstract
For understanding generic documents, information like font sizes, column layout, and generally the positioning of words may carry semantic information that is crucial for solving a downstream document intelligence task. Our novel BERTgrid, which is based on Chargrid by Katti et al. (2018), represents a document as a grid of contextualized word piece embedding vectors, thereby making its spatial structure and semantics accessible to the processing neural network. The contextualized embedding vectors are retrieved from a BERT language model. We use BERTgrid in combination with a fully convolutional network on a semantic instance segmentation task for extracting fields from invoices. We demonstrate its performance on tabulated line item and document header field extraction.
Abstract (translated)
URL
https://arxiv.org/abs/1909.04948
PDF
-
U4D: Unsupervised 4D Dynamic Scene Understanding
2019-07-23 14:25:27Armin Mustafa, Chris Russell, Adrian HiltonAbstract
We introduce the first approach to solve the challenging problem of unsupervised 4D visual scene understanding for complex dynamic scenes with multiple interacting people from multi-view video. Our approach simultaneously estimates a detailed model that includes a per-pixel semantically and temporally coherent reconstruction, together with instance-level segmentation exploiting photo-consistency, semantic and motion information. We further leverage recent advances in 3D pose estimation to constrain the joint semantic instance segmentation and 4D temporally coherent reconstruction. This enables per person semantic instance segmentation of multiple interacting people in complex dynamic scenes. Extensive evaluation of the joint visual scene understanding framework against state-of-the-art methods on challenging indoor and outdoor sequences demonstrates a significant (approx 40%) improvement in semantic segmentation, reconstruction and scene flow accuracy.
Abstract (translated)
URL
https://arxiv.org/abs/1907.09905
PDF
-
DetectFusion: Detecting and Segmenting Both Known and Unknown Dynamic Objects in Real-time SLAM
2019-07-22 04:29:32Ryo Hachiuma, Christian Pirchheim, Dieter Schmalstieg, Hideo SaitoAbstract
We present DetectFusion, an RGB-D SLAM system that runs in real-time and can robustly handle semantically known and unknown objects that can move dynamically in the scene. Our system detects, segments and assigns semantic class labels to known objects in the scene, while tracking and reconstructing them even when they move independently in front of the monocular camera. In contrast to related work, we achieve real-time computational performance on semantic instance segmentation with a novel method combining 2D object detection and 3D geometric segmentation. In addition, we propose a method for detecting and segmenting the motion of semantically unknown objects, thus further improving the accuracy of camera tracking and map reconstruction. We show that our method performs on par or better than previous work in terms of localization and object reconstruction accuracy, while achieving about 20 FPS even if the objects are segmented in each frame.
Abstract (translated)
URL
https://arxiv.org/abs/1907.09127
PDF
-
3D-SIC: 3D Semantic Instance Completion for RGB-D Scans
2019-04-26 18:33:34Ji Hou, Angela Dai, Matthias NießnerAbstract
This paper focuses on the task of semantic instance completion: from an incomplete, RGB-D scan of a scene, we aim to detect the individual object instances comprising the scene and jointly infer their complete object geometry. This enables a semantically meaningful decomposition of a scanned scene into individual, complete 3D objects. This semantic instance completion of a 3D scene opens up many new possibilities in enabling meaningful interactions with a scene, for instance for virtual or robotic agents. Rather than considering 3D semantic instance segmentation and scan completion separately, we propose 3D-SIC, a new end-to-end 3D convolutional neural network which jointly learns to detect object instances and predict their complete geometry -- achieving significantly better performance than treating these tasks independently. 3D-SIC leverages joint color-geometry feature learning and a fully-convolutional 3D network to effectively infer semantic instance completion for 3D scans at scale. Our method runs at interactive rates, taking several seconds inference time on scenes of $30$m $\times$ $25$m spatial extent. For the task of semantic instance completion, we additionally introduce a new semantic instance completion benchmark on real scan data, where we outperform alternative approaches by over 15 in mAP@0.5.
Abstract (translated)
本文主要研究语义实例完成的任务：从一个场景的不完全的RGB-D扫描，我们旨在检测包含该场景的单个对象实例，并共同推断它们的完整对象几何。这使得一个有语义意义的扫描场景分解成单个，完整的三维对象。3D场景的语义实例完成为与场景进行有意义的交互提供了许多新的可能性，例如虚拟或机器人代理。我们没有单独考虑3D语义实例分割和扫描完成，而是提出了一种新的端到端3D卷积神经网络3d-sic，它共同学习检测对象实例并预测其完整的几何结构——比独立处理这些任务显著提高性能。3d-sic利用联合颜色几何特征学习和完全卷积的3d网络有效地推断三维扫描的语义实例完成。我们的方法以交互速率运行，在30$M$ imes$25$M空间范围的场景上花费几秒钟的推理时间。对于语义实例完成的任务，我们还引入了一个新的基于真实扫描数据的语义实例完成基准，在map@0.5中，我们的性能优于其他方法超过15个。
URL
https://arxiv.org/abs/1904.12012
PDF
-
3D-BEVIS: Birds-Eye-View Instance Segmentation
2019-04-03 18:51:53Cathrin Elich, Francis Engelmann, Jonas Schult, Theodora Kontogianni, Bastian LeibeAbstract
Recent deep learning models achieve impressive results on 3D scene analysis tasks by operating directly on unstructured point clouds. A lot of progress was made in the field of object classification and semantic segmentation. However, the task of instance segmentation is less explored. In this work, we present 3D-BEVIS, a deep learning framework for 3D semantic instance segmentation on point clouds. Following the idea of previous proposal-free instance segmentation approaches, our model learns a feature embedding and groups the obtained feature space into semantic instances. Current point-based methods scale linearly with the number of points by processing local sub-parts of a scene individually. However, to perform instance segmentation by clustering, globally consistent features are required. Therefore, we propose to combine local point geometry with global context information from an intermediate bird's-eye view representation.
Abstract (translated)
最近的深度学习模型通过直接在非结构化点云上操作，在三维场景分析任务上取得了令人印象深刻的结果。在对象分类和语义分割方面取得了很大的进展。然而，对实例分割的任务探索较少。在这项工作中，我们提出了3D-BEVIS，一个深度学习框架，用于点云上的3D语义实例分割。该模型借鉴了以往的无建议实例分割方法，学习了特征嵌入，并将获得的特征空间分组为语义实例。当前基于点的方法通过单独处理场景的局部子部分与点的数量成线性比例。但是，要通过集群进行实例分割，需要全局一致的特性。因此，我们建议将局部点几何与来自中间鸟瞰图表示的全局上下文信息相结合。
URL
https://arxiv.org/abs/1904.02199
PDF
-
3D Graph Embedding Learning with a Structure-aware Loss Function for Point Cloud Semantic Instance Segmentation
2019-02-14 07:29:40Zhidong Liang, Ming Yang, Chunxiang WangAbstract
This paper introduces a novel approach for 3D semantic instance segmentation on point clouds. A 3D convolutional neural network called submanifold sparse convolutional network is used to generate semantic predictions and instance embeddings simultaneously. To obtain discriminative embeddings for each 3D instance, a structure-aware loss function is proposed which considers both the structure information and the embedding information. To get more consistent embeddings for each 3D instance, attention-based k nearest neighbour (KNN) is proposed to assign different weights for different neighbours. Based on the attention-based KNN, we add a graph convolutional network after the sparse convolutional network to get refined embeddings. Our network can be trained end-to-end. A simple mean-shift algorithm is utilized to cluster refined embeddings to get final instance predictions. As a result, our framework can output both the semantic prediction and the instance prediction. Experiments show that our approach outperforms all state-of-art methods on ScanNet benchmark and NYUv2 dataset.
Abstract (translated)
URL
https://arxiv.org/abs/1902.05247
PDF
-
3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans
2018-12-17 19:04:31Ji Hou, Angela Dai, Matthias NießnerAbstract
We introduce 3D-SIS, a novel neural network architecture for 3D semantic instance segmentation in commodity RGB-D scans. The core idea of our method is to jointly learn from both geometric and color signal, thus enabling accurate instance predictions. Rather than operate solely on 2D frames, we observe that most computer vision applications have multi-view RGB-D input available, which we leverage to construct an approach for 3D instance segmentation that effectively fuses together these multi-modal inputs. Our network leverages high-resolution RGB input by associating 2D images with the volumetric grid based on the pose alignment of the 3D reconstruction. For each image, we first extract 2D features for each pixel with a series of 2D convolutions; we then backproject the resulting feature vector to the associated voxel in the 3D grid. This combination of 2D and 3D feature learning allows significantly higher accuracy object detection and instance segmentation than state-of-the-art alternatives. We show results on both synthetic and real-world public benchmarks, achieving an improvement in mAP of over 13 on real-world data.
Abstract (translated)
URL
https://arxiv.org/abs/1812.07003
PDF
-
Deep Semantic Instance Segmentation of Tree-like Structures Using Synthetic Data
2018-11-08 01:23:45Kerry Halupka, Rahil Garnavi, Stephen MooreAbstract
Tree-like structures, such as blood vessels, often express complexity at very fine scales, requiring high-resolution grids to adequately describe their shape. Such sparse morphology can alternately be represented by locations of centreline points, but learning from this type of data with deep learning is challenging due to it being unordered, and permutation invariant. In this work, we propose a deep neural network that directly consumes unordered points along the centreline of a branching structure, to identify the topology of the represented structure in a single-shot. Key to our approach is the use of a novel multi-task loss function, enabling instance segmentation of arbitrarily complex branching structures. We train the network solely using synthetically generated data, utilizing domain randomization to facilitate the transfer to real 2D and 3D data. Results show that our network can reliably extract meaningful information about branch locations, bifurcations and endpoints, and sets a new benchmark for semantic instance segmentation in branching structures.
Abstract (translated)
URL
https://arxiv.org/abs/1811.03208
PDF
-
Recurrent Neural Networks for Semantic Instance Segmentation
2018-09-03 15:07:52Amaia Salvador, Miriam Bellver, Victor Campos, Manel Baradad, Ferran Marques, Jordi Torres, Xavier Giro-i-NietoAbstract
We present a recurrent model for semantic instance segmentation that sequentially generates binary masks and their associated class probabilities for every object in an image. Our proposed system is trainable end-to-end from an input image to a sequence of labeled masks and, compared to methods relying on object proposals, does not require post-processing steps on its output. We study the suitability of our recurrent model on three different instance segmentation benchmarks, namely Pascal VOC 2012, CVPPP Plant Leaf Segmentation and Cityscapes. Further, we analyze the object sorting patterns generated by our model and observe that it learns to follow a consistent pattern, which correlates with the activations learned in the encoder part of our network. Source code and models are available at https://imatge-upc.github.io/rsis/
Abstract (translated)
我们提出了一种语义实例分割的循环模型，它为图像中的每个对象顺序生成二进制掩码及其相关的类概率。我们提出的系统可以从输入图像到标记掩码序列进行端到端的训练，并且与依赖于对象提议的方法相比，不需要对其输出进行后处理步骤。我们研究了我们的重复模型在三种不同的实例分割基准上的适用性，即Pascal VOC 2012，CVPPP Plant Leaf Segmentation和Cityscapes。此外，我们分析了我们的模型生成的对象排序模式，并观察它学会遵循一致的模式，这与我们网络的编码器部分中学习的激活相关。源代码和模型可在https://imatge-upc.github.io/rsis/获得。
URL
https://arxiv.org/abs/1712.00617
PDF
-
Bayesian Semantic Instance Segmentation in Open Set World
2018-07-30 02:38:12Trung Pham, Vijay Kumar B G, Thanh-Toan Do, Gustavo Carneiro, Ian ReidAbstract
This paper addresses the semantic instance segmentation task in the open-set conditions, where input images can contain known and unknown object classes. The training process of existing semantic instance segmentation methods requires annotation masks for all object instances, which is expensive to acquire or even infeasible in some realistic scenarios, where the number of categories may increase boundlessly. In this paper, we present a novel open-set semantic instance segmentation approach capable of segmenting all known and unknown object classes in images, based on the output of an object detector trained on known object classes. We formulate the problem using a Bayesian framework, where the posterior distribution is approximated with a simulated annealing optimization equipped with an efficient image partition sampler. We show empirically that our method is competitive with state-of-the-art supervised methods on known classes, but also performs well on unknown classes when compared with unsupervised methods.
Abstract (translated)
本文讨论了开放条件下的语义实例分割任务，其中输入图像可以包含已知和未知的对象类。现有语义实例分割方法的训练过程需要所有对象实例的注释掩模，这在某些现实场景中获取或甚至是不可行的，其中类别的数量可能无限增加。在本文中，我们提出了一种新颖的开放式语义实例分割方法，该方法能够基于在已知对象类上训练的对象检测器的输出来分割图像中的所有已知和未知对象类。我们使用贝叶斯框架来制定问题，其中后验分布用配备有效图像分区采样器的模拟退火优化来近似。我们凭经验证明，我们的方法与已知类的最先进的监督方法相比具有竞争力，但与无监督方法相比，它在未知类上也表现良好。
URL
https://arxiv.org/abs/1806.00911
PDF
