Scene_Parsing
-
Treasure What You Have: Exploiting Similarity in Deep Neural Networks for Efficient Video Processing
2023-05-10 23:18:47Hadjer Benmeziane, Halima Bouzidi, Hamza Ouarnoughi, Ozcan Ozturk, Smail NiarAbstract
Deep learning has enabled various Internet of Things (IoT) applications. Still, designing models with high accuracy and computational efficiency remains a significant challenge, especially in real-time video processing applications. Such applications exhibit high inter- and intra-frame redundancy, allowing further improvement. This paper proposes a similarity-aware training methodology that exploits data redundancy in video frames for efficient processing. Our approach introduces a per-layer regularization that enhances computation reuse by increasing the similarity of weights during training. We validate our methodology on two critical real-time applications, lane detection and scene parsing. We observe an average compression ratio of approximately 50% and a speedup of \sim 1.5x for different models while maintaining the same accuracy.
Abstract (translated)
深度学习已经使各种物联网(IoT)应用得以实现。然而，设计高精度和高计算效率的模型仍然是一个 significant 挑战，特别是实时视频处理应用中。这些应用表现出内外帧的冗余，从而允许进一步改进。本文提出了一种相似性 aware 的训练方法，利用视频帧中的数据冗余以高效处理。我们的方法引入了每层 Regularization，在训练期间通过增加权重之间的相似性来提高计算重用。我们对两个关键实时应用，车道检测和场景解析进行了验证。我们观察到平均压缩比例约为 50%，不同模型的速度up 达到了 \sim 1.5x，同时保持相同的精度。
URL
https://arxiv.org/abs/2305.06492
PDF
-
RT-K-Net: Revisiting K-Net for Real-Time Panoptic Segmentation
2023-05-02 08:36:02Markus Schön, Michael Buchholz, Klaus DietmayerAbstract
Panoptic segmentation is one of the most challenging scene parsing tasks, combining the tasks of semantic segmentation and instance segmentation. While much progress has been made, few works focus on the real-time application of panoptic segmentation methods. In this paper, we revisit the recently introduced K-Net architecture. We propose vital changes to the architecture, training, and inference procedure, which massively decrease latency and improve performance. Our resulting RT-K-Net sets a new state-of-the-art performance for real-time panoptic segmentation methods on the Cityscapes dataset and shows promising results on the challenging Mapillary Vistas dataset. On Cityscapes, RT-K-Net reaches 60.2 % PQ with an average inference time of 32 ms for full resolution 1024x2048 pixel images on a single Titan RTX GPU. On Mapillary Vistas, RT-K-Net reaches 33.2 % PQ with an average inference time of 69 ms. Source code is available at this https URL.
Abstract (translated)
Panoptic segmentation 是处理场景解析任务中最具挑战性的之一，将语义分割和实例分割任务结合在一起。尽管已经取得了很多进展，但只有少数工作关注实时 Panoptic segmentation 方法的应用。在本文中，我们重新审视了最近引入的 K-Net 架构。我们提出了关键的变化，修改了架构、训练和推理程序，极大地减少了延迟并提高了性能。我们得到的 RT-K-Net 在 Cityscapes 数据集上实现了实时 Panoptic segmentation 方法的最新前沿技术性能，并在挑战性的 Mapillary Vistas 数据集上取得了令人期望的结果。在 Cityscapes 中，RT-K-Net 的 PQ 准确率达到了 60.2%，平均推理时间为 32 毫秒，对于一张 1024x2048 像素的全分辨率图像，可以在单个 Titan RTX 显卡上执行。在 Mapillary Vistas 中，RT-K-Net 的 PQ 准确率达到了 33.2%，平均推理时间为 69 毫秒。源代码可在 this https URL 获取。
URL
https://arxiv.org/abs/2305.01255
PDF
-
Local and Global Contextual Features Fusion for Pedestrian Intention Prediction
2023-05-01 22:37:31Mohsen Azarmi, Mahdi Rezaei, Tanveer Hussain, Chenghao QianAbstract
Autonomous vehicles (AVs) are becoming an indispensable part of future transportation. However, safety challenges and lack of reliability limit their real-world deployment. Towards boosting the appearance of AVs on the roads, the interaction of AVs with pedestrians including "prediction of the pedestrian crossing intention" deserves extensive research. This is a highly challenging task as involves multiple non-linear parameters. In this direction, we extract and analyse spatio-temporal visual features of both pedestrian and traffic contexts. The pedestrian features include body pose and local context features that represent the pedestrian's behaviour. Additionally, to understand the global context, we utilise location, motion, and environmental information using scene parsing technology that represents the pedestrian's surroundings, and may affect the pedestrian's intention. Finally, these multi-modality features are intelligently fused for effective intention prediction learning. The experimental results of the proposed model on the JAAD dataset show a superior result on the combined AUC and F1-score compared to the state-of-the-art.
Abstract (translated)
自动驾驶车辆(AVs)已成为未来交通运输不可或缺的部分。然而，安全性挑战和可靠性不足限制了其现实世界的部署。为了增加道路上自动驾驶车辆的出现率，包括“预测行人横穿马路的意图”在内的自动驾驶车辆与行人的互动值得深入研究。这是一个极具挑战性的任务，因为它涉及多个非线性参数。在这方面，我们提取和分析行人和交通场景的时间和空间视觉特征。行人特征包括身体姿势和局部场景特征，代表行人的行为。此外，为了理解全局背景，我们利用场景解析技术利用位置、运动和环境信息，代表行人的周围环境，并且可能影响行人的意图。最后，这些多模态特征进行智能融合，以有效预测意图学习。在JAAD数据集上，提议模型的实验结果显示，与当前最先进的模型相比，其综合AUC和F1得分有更好的表现。
URL
https://arxiv.org/abs/2305.01111
PDF
-
DPF: Learning Dense Prediction Fields with Weak Supervision
2023-03-29 17:58:33Xiaoxue Chen, Yuhang Zheng, Yupeng Zheng, Qiang Zhou, Hao Zhao, Guyue Zhou, Ya-Qin ZhangAbstract
Nowadays, many visual scene understanding problems are addressed by dense prediction networks. But pixel-wise dense annotations are very expensive (e.g., for scene parsing) or impossible (e.g., for intrinsic image decomposition), motivating us to leverage cheap point-level weak supervision. However, existing pointly-supervised methods still use the same architecture designed for full supervision. In stark contrast to them, we propose a new paradigm that makes predictions for point coordinate queries, as inspired by the recent success of implicit representations, like distance or radiance fields. As such, the method is named as dense prediction fields (DPFs). DPFs generate expressive intermediate features for continuous sub-pixel locations, thus allowing outputs of an arbitrary resolution. DPFs are naturally compatible with point-level supervision. We showcase the effectiveness of DPFs using two substantially different tasks: high-level semantic parsing and low-level intrinsic image decomposition. In these two cases, supervision comes in the form of single-point semantic category and two-point relative reflectance, respectively. As benchmarked by three large-scale public datasets PASCALContext, ADE20K and IIW, DPFs set new state-of-the-art performance on all of them with significant margins. Code can be accessed at this https URL.
Abstract (translated)
Nowadays, many visual scene understanding problems are addressed by dense prediction networks. But pixel-wise dense annotations are very expensive (e.g., for scene parsing) or impossible (e.g., for intrinsic image decomposition), motivating us to leverage cheap point-level weak supervision. However, existing pointly-supervised methods still use the same architecture designed for full supervision. In stark contrast to them, we propose a new paradigm that makes predictions for point coordinate queries, as inspired by the recent success of implicit representations, like distance or radiance fields. As such, the method is named as dense prediction fields (DPFs). DPFs generate expressive intermediate features for continuous sub-pixel locations, thus allowing outputs of an arbitrary resolution. DPFs are naturally compatible with point-level supervision. We showcase the effectiveness of DPFs using two substantially different tasks: high-level semanticParsing and low-level intrinsic image decomposition. In these two cases, supervision comes in the form of single-point semantic category and two-point relative reflectance, respectively. As benchmarked by three large-scale public datasets PASCALContext, ADE20K and IIW, DPFs set new state-of-the-art performance on all of them with significant margins. Code can be accessed at this https URL.
URL
https://arxiv.org/abs/2303.16890
PDF
-
Traffic Scene Parsing through the TSP6K Dataset
2023-03-06 02:05:14Peng-Tao Jiang, Yuqi Yang, Yang Cao, Qibin Hou, Ming-Ming Cheng, Chunhua ShenAbstract
Traffic scene parsing is one of the most important tasks to achieve intelligent cities. So far, little effort has been spent on constructing datasets specifically for the task of traffic scene parsing. To fill this gap, here we introduce the TSP6K dataset, containing 6,000 urban traffic images and spanning hundreds of street scenes under various weather conditions. In contrast to most previous traffic scene datasets collected from a driving platform, the images in our dataset are from the shooting platform high-hanging on the street. Such traffic images can capture more crowded street scenes with several times more traffic participants than the driving scenes. Each image in the TSP6K dataset is provided with high-quality pixel-level and instance-level annotations. We perform a detailed analysis for the dataset and comprehensively evaluate the state-of-the-art scene parsing methods. Considering the vast difference in instance sizes, we propose a detail refining decoder, which recovers the details of different semantic regions in traffic scenes. Experiments have shown its effectiveness in parsing high-hanging traffic scenes. Code and dataset will be made publicly available.
Abstract (translated)
交通场景解析是实现智慧城市的最重要任务之一。迄今为止, little effort has been spent on constructing datasets specifically for the task of traffic scene parsing. 为填补这一差距,我们介绍了TSP6K dataset,其中包括6,000幅城市交通图像,涵盖数百个街道场景,在各种天气条件下。与大多数从驾驶平台收集的交通场景数据集不同,我们的数据集中的图像是从街道的高架上拍摄的。这些交通图像能够捕捉到比驾驶场景更加拥挤的街道场景,并有数倍于驾驶场景的交通参与者。每个图像在TSP6K dataset中都有高质量的像素级和实例级注释。我们对dataset进行了详细的分析,并全面评估了交通场景解析的最新方法。考虑到实例大小的巨大差异,我们提出了一种细节 refine Decoder,该算法可以恢复交通场景不同语义区域的详细信息。实验表明,它在解析高架上的交通场景方面非常有效。代码和数据集将公开可用。
URL
https://arxiv.org/abs/2303.02835
PDF
-
Multi-Sem Fusion: Multimodal Semantic Fusion for 3D Object Detection
2022-12-10 10:54:41Shaoqing Xu, Dingfu Zhou, Jin Fang, Pengcheng Wang, Liangjun ZhangAbstract
LiDAR-based 3D Object detectors have achieved impressive performances in many benchmarks, however, multisensors fusion-based techniques are promising to further improve the results. PointPainting, as a recently proposed framework, can add the semantic information from the 2D image into the 3D LiDAR point by the painting operation to boost the detection performance. However, due to the limited resolution of 2D feature maps, severe boundary-blurring effect happens during re-projection of 2D semantic segmentation into the 3D point clouds. To well handle this limitation, a general multimodal fusion framework MSF has been proposed to fuse the semantic information from both the 2D image and 3D points scene parsing results. Specifically, MSF includes three main modules. First, SOTA off-the-shelf 2D/3D semantic segmentation approaches are employed to generate the parsing results for 2D images and 3D point clouds. The 2D semantic information is further re-projected into the 3D point clouds with calibrated parameters. To handle the misalignment between the 2D and 3D parsing results, an AAF module is proposed to fuse them by learning an adaptive fusion score. Then the point cloud with the fused semantic label is sent to the following 3D object detectors. Furthermore, we propose a DFF module to aggregate deep features in different levels to boost the final detection performance. The effectiveness of the framework has been verified on two public large-scale 3D object detection benchmarks by comparing with different baselines. The experimental results show that the proposed fusion strategies can significantly improve the detection performance compared to the methods using only point clouds and the methods using only 2D semantic information. Most importantly, the proposed approach significantly outperforms other approaches and sets new SOTA results on the nuScenes testing benchmark.
Abstract (translated)
URL
https://arxiv.org/abs/2212.05265
PDF
-
OneFormer: One Transformer to Rule Universal Image Segmentation
2022-11-10 18:56:04Jitesh Jain, Jiachen Li, MangTik Chiu, Ali Hassani, Nikita Orlov, Humphrey ShiAbstract
Universal Image Segmentation is not a new concept. Past attempts to unify image segmentation in the last decades include scene parsing, panoptic segmentation, and, more recently, new panoptic architectures. However, such panoptic architectures do not truly unify image segmentation because they need to be trained individually on the semantic, instance, or panoptic segmentation to achieve the best performance. Ideally, a truly universal framework should be trained only once and achieve SOTA performance across all three image segmentation tasks. To that end, we propose OneFormer, a universal image segmentation framework that unifies segmentation with a multi-task train-once design. We first propose a task-conditioned joint training strategy that enables training on ground truths of each domain (semantic, instance, and panoptic segmentation) within a single multi-task training process. Secondly, we introduce a task token to condition our model on the task at hand, making our model task-dynamic to support multi-task training and inference. Thirdly, we propose using a query-text contrastive loss during training to establish better inter-task and inter-class distinctions. Notably, our single OneFormer model outperforms specialized Mask2Former models across all three segmentation tasks on ADE20k, CityScapes, and COCO, despite the latter being trained on each of the three tasks individually with three times the resources. With new ConvNeXt and DiNAT backbones, we observe even more performance improvement. We believe OneFormer is a significant step towards making image segmentation more universal and accessible. To support further research, we open-source our code and models at this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2211.06220
PDF
-
VIBUS: Data-efficient 3D Scene Parsing with VIewpoint Bottleneck and Uncertainty-Spectrum Modeling
2022-10-20 17:59:57Beiwen Tian, Liyi Luo, Hao Zhao, Guyue ZhouAbstract
Recently, 3D scenes parsing with deep learning approaches has been a heating topic. However, current methods with fully-supervised models require manually annotated point-wise supervision which is extremely user-unfriendly and time-consuming to obtain. As such, training 3D scene parsing models with sparse supervision is an intriguing alternative. We term this task as data-efficient 3D scene parsing and propose an effective two-stage framework named VIBUS to resolve it by exploiting the enormous unlabeled points. In the first stage, we perform self-supervised representation learning on unlabeled points with the proposed Viewpoint Bottleneck loss function. The loss function is derived from an information bottleneck objective imposed on scenes under different viewpoints, making the process of representation learning free of degradation and sampling. In the second stage, pseudo labels are harvested from the sparse labels based on uncertainty-spectrum modeling. By combining data-driven uncertainty measures and 3D mesh spectrum measures (derived from normal directions and geodesic distances), a robust local affinity metric is obtained. Finite gamma/beta mixture models are used to decompose category-wise distributions of these measures, leading to automatic selection of thresholds. We evaluate VIBUS on the public benchmark ScanNet and achieve state-of-the-art results on both validation set and online test server. Ablation studies show that both Viewpoint Bottleneck and uncertainty-spectrum modeling bring significant improvements. Codes and models are publicly available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2210.11472
PDF
-
Boosting Night-time Scene Parsing with Learnable Frequency
2022-08-30 13:09:59Zhifeng Xie, Sen Wang, Ke Xu, Zhizhong Zhang, Xin Tan, Yuan Xie, Lizhuang MaAbstract
Night-Time Scene Parsing (NTSP) is essential to many vision applications, especially for autonomous driving. Most of the existing methods are proposed for day-time scene parsing. They rely on modeling pixel intensity-based spatial contextual cues under even illumination. Hence, these methods do not perform well in night-time scenes as such spatial contextual cues are buried in the over-/under-exposed regions in night-time scenes. In this paper, we first conduct an image frequency-based statistical experiment to interpret the day-time and night-time scene discrepancies. We find that image frequency distributions differ significantly between day-time and night-time scenes, and understanding such frequency distributions is critical to NTSP problem. Based on this, we propose to exploit the image frequency distributions for night-time scene parsing. First, we propose a Learnable Frequency Encoder (LFE) to model the relationship between different frequency coefficients to measure all frequency components dynamically. Second, we propose a Spatial Frequency Fusion module (SFF) that fuses both spatial and frequency information to guide the extraction of spatial context features. Extensive experiments show that our method performs favorably against the state-of-the-art methods on the NightCity, NightCity+ and BDD100K-night datasets. In addition, we demonstrate that our method can be applied to existing day-time scene parsing methods and boost their performance on night-time scenes.
Abstract (translated)
URL
https://arxiv.org/abs/2208.14241
PDF
-
A Dense Material Segmentation Dataset for Indoor and Outdoor Scene Parsing
Abstract
A key algorithm for understanding the world is material segmentation, which assigns a label (metal, glass, etc.) to each pixel. We find that a model trained on existing data underperforms in some settings and propose to address this with a large-scale dataset of 3.2 million dense segments on 44,560 indoor and outdoor images, which is 23x more segments than existing data. Our data covers a more diverse set of scenes, objects, viewpoints and materials, and contains a more fair distribution of skin types. We show that a model trained on our data outperforms a state-of-the-art model across datasets and viewpoints. We propose a large-scale scene parsing benchmark and baseline of 0.729 per-pixel accuracy, 0.585 mean class accuracy and 0.420 mean IoU across 46 materials.
Abstract (translated)
URL
https://arxiv.org/abs/2207.10614
PDF
-
FLOAT: Factorized Learning of Object Attributes for Improved Multi-object Multi-part Scene Parsing
2022-03-30 09:46:10Rishubh Singh, Pranav Gupta, Pradeep Shenoy, Ravikiran SarvadevabhatlaAbstract
Multi-object multi-part scene parsing is a challenging task which requires detecting multiple object classes in a scene and segmenting the semantic parts within each object. In this paper, we propose FLOAT, a factorized label space framework for scalable multi-object multi-part parsing. Our framework involves independent dense prediction of object category and part attributes which increases scalability and reduces task complexity compared to the monolithic label space counterpart. In addition, we propose an inference-time 'zoom' refinement technique which significantly improves segmentation quality, especially for smaller objects/parts. Compared to state of the art, FLOAT obtains an absolute improvement of 2.0% for mean IOU (mIOU) and 4.8% for segmentation quality IOU (sqIOU) on the Pascal-Part-58 dataset. For the larger Pascal-Part-108 dataset, the improvements are 2.1% for mIOU and 3.9% for sqIOU. We incorporate previously excluded part attributes and other minor parts of the Pascal-Part dataset to create the most comprehensive and challenging version which we dub Pascal-Part-201. FLOAT obtains improvements of 8.6% for mIOU and 7.5% for sqIOU on the new dataset, demonstrating its parsing effectiveness across a challenging diversity of objects and parts. The code and datasets are available at this http URL.
Abstract (translated)
URL
https://arxiv.org/abs/2203.16168
PDF
-
TO-Scene: A Large-scale Dataset for Understanding 3D Tabletop Scenes
2022-03-17 17:00:55Mutian Xu, Pei Chen, Haolin Liu, Xiaoguang HanAbstract
Many basic indoor activities such as eating or writing are always conducted upon different tabletops (e.g., coffee tables, writing desks). It is indispensable to understanding tabletop scenes in 3D indoor scene parsing applications. Unfortunately, it is hard to meet this demand by directly deploying data-driven algorithms, since 3D tabletop scenes are rarely available in current datasets. To remedy this defect, we introduce TO-Scene, a large-scale dataset focusing on tabletop scenes, which contains 20,740 scenes with three variants. To acquire the data, we design an efficient and scalable framework, where a crowdsourcing UI is developed to transfer CAD objects onto tables from ScanNet. Then the output tabletop scenes are simulated into real scans and annotated automatically. Further, we propose a tabletop-aware learning strategy for better perceiving the small-sized tabletop instances. Notably, we also provide a real scanned test set TO-Real to verify the practical value of TO-Scene. Experiments show that the algorithms trained on TO-Scene indeed work on the realistic test data, and our proposed tabletop-aware learning strategy greatly improves the state-of-the-art results on both 3D semantic segmentation and object detection tasks. TO-Scene and TO-Real, plus Web UI, will all be publicly available.
Abstract (translated)
URL
https://arxiv.org/abs/2203.09440
PDF
-
Boundary Corrected Multi-scale Fusion Network for Real-time Semantic Segmentation
2022-03-01 13:31:01Tianjiao Jiang, Yi Jin, Tengfei Liang, Xu Wang, Yidong LiAbstract
Image semantic segmentation aims at the pixel-level classification of images, which has requirements for both accuracy and speed in practical application. Existing semantic segmentation methods mainly rely on the high-resolution input to achieve high accuracy and do not meet the requirements of inference time. Although some methods focus on high-speed scene parsing with lightweight architectures, they can not fully mine semantic features under low computation with relatively low performance. To realize the real-time and high-precision segmentation, we propose a new method named Boundary Corrected Multi-scale Fusion Network, which uses the designed Low-resolution Multi-scale Fusion Module to extract semantic information. Moreover, to deal with boundary errors caused by low-resolution feature map fusion, we further design an additional Boundary Corrected Loss to constrain overly smooth features. Extensive experiments show that our method achieves a state-of-the-art balance of accuracy and speed for the real-time semantic segmentation.
Abstract (translated)
URL
https://arxiv.org/abs/2203.00436
PDF
-
Aerial Scene Parsing: From Tile-level Scene Classification to Pixel-wise Semantic Labeling
2022-01-06 07:40:47Yang Long, Gui-Song Xia, Liangpei Zhang, Gong Cheng, Deren LiAbstract
Given an aerial image, aerial scene parsing (ASP) targets to interpret the semantic structure of the image content, e.g., by assigning a semantic label to every pixel of the image. With the popularization of data-driven methods, the past decades have witnessed promising progress on ASP by approaching the problem with the schemes of tile-level scene classification or segmentation-based image analysis, when using high-resolution aerial images. However, the former scheme often produces results with tile-wise boundaries, while the latter one needs to handle the complex modeling process from pixels to semantics, which often requires large-scale and well-annotated image samples with pixel-wise semantic labels. In this paper, we address these issues in ASP, with perspectives from tile-level scene classification to pixel-wise semantic labeling. Specifically, we first revisit aerial image interpretation by a literature review. We then present a large-scale scene classification dataset that contains one million aerial images termed Million-AID. With the presented dataset, we also report benchmarking experiments using classical convolutional neural networks (CNNs). Finally, we perform ASP by unifying the tile-level scene classification and object-based image analysis to achieve pixel-wise semantic labeling. Intensive experiments show that Million-AID is a challenging yet useful dataset, which can serve as a benchmark for evaluating newly developed algorithms. When transferring knowledge from Million-AID, fine-tuning CNN models pretrained on Million-AID perform consistently better than those pretrained ImageNet for aerial scene classification. Moreover, our designed hierarchical multi-task learning method achieves the state-of-the-art pixel-wise classification on the challenging GID, bridging the tile-level scene classification toward pixel-wise semantic labeling for aerial image interpretation.
Abstract (translated)
URL
https://arxiv.org/abs/2201.01953
PDF
-
Edge-aware Guidance Fusion Network for RGB Thermal Scene Parsing
2021-12-09 01:12:47Wujie Zhou, Shaohua Dong, Caie Xu, Yaguan QianAbstract
RGB thermal scene parsing has recently attracted increasing research interest in the field of computer vision. However, most existing methods fail to perform good boundary extraction for prediction maps and cannot fully use high level features. In addition, these methods simply fuse the features from RGB and thermal modalities but are unable to obtain comprehensive fused features. To address these problems, we propose an edge-aware guidance fusion network (EGFNet) for RGB thermal scene parsing. First, we introduce a prior edge map generated using the RGB and thermal images to capture detailed information in the prediction map and then embed the prior edge information in the feature maps. To effectively fuse the RGB and thermal information, we propose a multimodal fusion module that guarantees adequate cross-modal fusion. Considering the importance of high level semantic information, we propose a global information module and a semantic information module to extract rich semantic information from the high-level features. For decoding, we use simple elementwise addition for cascaded feature fusion. Finally, to improve the parsing accuracy, we apply multitask deep supervision to the semantic and boundary maps. Extensive experiments were performed on benchmark datasets to demonstrate the effectiveness of the proposed EGFNet and its superior performance compared with state of the art methods. The code and results can be found at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2112.05144
PDF
-
Geometric Feature Learning for 3D Meshes
2021-12-03 09:16:49Huan Lei, Naveed Akhtar, Mubarak Shah, Ajmal MianAbstract
Geometric feature learning for 3D meshes is central to computer graphics and highly important for numerous vision applications. However, deep learning currently lags in hierarchical modeling of heterogeneous 3D meshes due to the lack of required operations and/or their efficient implementations. In this paper, we propose a series of modular operations for effective geometric deep learning over heterogeneous 3D meshes. These operations include mesh convolutions, (un)pooling and efficient mesh decimation. We provide open source implementation of these operations, collectively termed \textit{Picasso}. The mesh decimation module of Picasso is GPU-accelerated, which can process a batch of meshes on-the-fly for deep learning. Our (un)pooling operations compute features for newly-created neurons across network layers of varying resolution. Our mesh convolutions include facet2vertex, vertex2facet, and facet2facet convolutions that exploit vMF mixture and Barycentric interpolation to incorporate fuzzy modelling. Leveraging the modular operations of Picasso, we contribute a novel hierarchical neural network, PicassoNet-II, to learn highly discriminative features from 3D meshes. PicassoNet-II accepts primitive geometrics and fine textures of mesh facets as input features, while processing full scene meshes. Our network achieves highly competitive performance for shape analysis and scene parsing on a variety of benchmarks. We release Picasso and PicassoNet-II on Github this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2112.01801
PDF
-
MSP : Refine Boundary Segmentation via Multiscale Superpixel
2021-12-03 06:59:39Jie Zhu, Huabin Huang, Banghuai Li, Yong Liu, Leye WangAbstract
In this paper, we propose a simple but effective message passing method to improve the boundary quality for the semantic segmentation result. Inspired by the generated sharp edges of superpixel blocks, we employ superpixel to guide the information passing within feature map. Simultaneously, the sharp boundaries of the blocks also restrict the message passing scope. Specifically, we average features that the superpixel block covers within feature map, and add the result back to each feature vector. Further, to obtain sharper edges and farther spatial dependence, we develop a multiscale superpixel module (MSP) by a cascade of different scales superpixel blocks. Our method can be served as a plug-and-play module and easily inserted into any segmentation network without introducing new parameters. Extensive experiments are conducted on three strong baselines, namely PSPNet, DeeplabV3, and DeepLabV3+, and four challenging scene parsing datasets including ADE20K, Cityscapes, PASCAL VOC, and PASCAL Context. The experimental results verify its effectiveness and generalizability.
Abstract (translated)
URL
https://arxiv.org/abs/2112.01746
PDF
-
TBN-ViT: Temporal Bilateral Network with Vision Transformer for Video Scene Parsing
2021-12-02 07:53:36Bo Yan, Leilei Cao, Hongbin WangAbstract
Video scene parsing in the wild with diverse scenarios is a challenging and great significance task, especially with the rapid development of automatic driving technique. The dataset Video Scene Parsing in the Wild(VSPW) contains well-trimmed long-temporal, dense annotation and high resolution clips. Based on VSPW, we design a Temporal Bilateral Network with Vision Transformer. We first design a spatial path with convolutions to generate low level features which can preserve the spatial information. Meanwhile, a context path with vision transformer is employed to obtain sufficient context information. Furthermore, a temporal context module is designed to harness the inter-frames contextual information. Finally, the proposed method can achieve the mean intersection over union(mIoU) of 49.85\% for the VSPW2021 Challenge test dataset.
Abstract (translated)
URL
https://arxiv.org/abs/2112.01033
PDF
-
Pointly-supervised 3D Scene Parsing with Viewpoint Bottleneck
2021-09-17 13:54:20Liyi Luo, Beiwen Tian, Hao Zhao, Guyue ZhouAbstract
Semantic understanding of 3D point clouds is important for various robotics applications. Given that point-wise semantic annotation is expensive, in this paper, we address the challenge of learning models with extremely sparse labels. The core problem is how to leverage numerous unlabeled points. To this end, we propose a self-supervised 3D representation learning framework named viewpoint bottleneck. It optimizes a mutual-information based objective, which is applied on point clouds under different viewpoints. A principled analysis shows that viewpoint bottleneck leads to an elegant surrogate loss function that is suitable for large-scale point cloud data. Compared with former arts based upon contrastive learning, viewpoint bottleneck operates on the feature dimension instead of the sample dimension. This paradigm shift has several advantages: It is easy to implement and tune, does not need negative samples and performs better on our goal down-streaming task. We evaluate our method on the public benchmark ScanNet, under the pointly-supervised setting. We achieve the best quantitative results among comparable solutions. Meanwhile we provide an extensive qualitative inspection on various challenging scenes. They demonstrate that our models can produce fairly good scene parsing results for robotics applications. Our code, data and models will be made public.
Abstract (translated)
URL
https://arxiv.org/abs/2109.08553
PDF
-
Exploiting Spatial-Temporal Semantic Consistency for Video Scene Parsing
2021-09-06 08:24:38Xingjian He, Weining Wang, Zhiyong Xu, Hao Wang, Jie Jiang, Jing LiuAbstract
Compared with image scene parsing, video scene parsing introduces temporal information, which can effectively improve the consistency and accuracy of prediction. In this paper, we propose a Spatial-Temporal Semantic Consistency method to capture class-exclusive context information. Specifically, we design a spatial-temporal consistency loss to constrain the semantic consistency in spatial and temporal dimensions. In addition, we adopt an pseudo-labeling strategy to enrich the training dataset. We obtain the scores of 59.84% and 58.85% mIoU on development (test part 1) and testing set of VSPW, respectively. And our method wins the 1st place on VSPW challenge at ICCV2021.
Abstract (translated)
URL
https://arxiv.org/abs/2109.02281
PDF
