Paper Reading AI Learner

Log-barrier constrained CNNs

2019-04-08 17:25:46
Hoel Kervadec, Jose Dolz, Jing Yuan, Christian Desrosiers, Eric Granger, Ismail Ben Ayed

Abstract

This study investigates imposing inequality constraints on the outputs of CNNs for weakly supervised segmentation. In the general context of deep networks, constraints are commonly handled with penalty approaches for their simplicity, and despite their well-known limitations. Lagrangian optimization has well-established theoretical and practical advantages over penalty methods, but has been largely avoided for deep CNNs, mainly due to computational complexity and stability/convergence issues caused by alternating stochastic optimization and dual updates. Several recent studies showed that, in the context of deep CNNs, the theoretical advantages of Lagrangian optimization over simple penalties do not materialize in practice, with performances that are, surprisingly, worse. We leverage well-established concepts in interior-point methods, which approximate Lagrangian optimization with a sequence of unconstrained problems, while completely avoiding dual steps/projections. Specifically, we propose a sequence of unconstrained {\em log-barrier-extension} losses for approximating inequality-constrained CNN problems. The proposed extension has a duality-gap bound, which yields sub-optimality certificates for feasible solutions in the case of convex losses. While sub-optimality is not guaranteed for non-convex problems, the result shows that log-barrier extensions are a principled way to approximate Lagrangian optimization for constrained CNNs. Our approach addresses the well-known limitations of penalty methods and, at the same time, removes the explicit dual steps of Lagrangian optimization. We report comprehensive experiments showing that our formulation outperforms a recent penalty-based constrained CNN method, both in terms of accuracy and training stability.

Abstract (translated)

本研究探讨弱监督分割对CNN输出的不平等限制。在深层网络的一般情况下,约束通常是用惩罚方法处理的,因为它们很简单,尽管它们有众所周知的局限性。拉格朗日优化与罚方法相比,具有良好的理论和实践优势,但深CNN在很大程度上被避免,这主要是由于交替随机优化和双重更新导致的计算复杂性和稳定性/收敛问题。最近的几项研究表明,在深度CNN的背景下,拉格朗日优化相对于简单惩罚的理论优势并没有在实践中体现出来,其性能令人惊讶地更差。我们在内点法中利用了成熟的概念,这些概念用一系列无约束问题近似拉格朗日优化,同时完全避免了双重步骤/投影。具体地说,我们提出了一个无约束em对数势垒扩展损失序列,用于近似不平等约束CNN问题。该推广具有对偶间隙界,在凸损失的情况下,给出了可行解的次最优性证明。当非凸问题不能保证次最优性时,结果表明对数势垒扩展是约束CNN近似拉格朗日优化的一种原则性方法。我们的方法解决了惩罚方法的已知局限性,同时消除了拉格朗日优化的显式对偶步骤。我们报告的综合实验表明,我们的公式优于最近基于惩罚的约束CNN方法,无论是在准确性和训练稳定性方面。

URL

https://arxiv.org/abs/1904.04205

PDF

https://arxiv.org/pdf/1904.04205.pdf


Tags
3D Action Action_Localization Action_Recognition Activity Adversarial Agent Attention Autonomous Bert Boundary_Detection Caption Chat Classification CNN Compressive_Sensing Contour Contrastive_Learning Deep_Learning Denoising Detection Dialog Diffusion Drone Dynamic_Memory_Network Edge_Detection Embedding Embodied Emotion Enhancement Face Face_Detection Face_Recognition Facial_Landmark Few-Shot Gait_Recognition GAN Gaze_Estimation Gesture Gradient_Descent Handwriting Human_Parsing Image_Caption Image_Classification Image_Compression Image_Enhancement Image_Generation Image_Matting Image_Retrieval Inference Inpainting Intelligent_Chip Knowledge Knowledge_Graph Language_Model Matching Medical Memory_Networks Multi_Modal Multi_Task NAS NMT Object_Detection Object_Tracking OCR Ontology Optical_Character Optical_Flow Optimization Person_Re-identification Point_Cloud Portrait_Generation Pose Pose_Estimation Prediction QA Quantitative Quantitative_Finance Quantization Re-identification Recognition Recommendation Reconstruction Regularization Reinforcement_Learning Relation Relation_Extraction Represenation Represenation_Learning Restoration Review RNN Salient Scene_Classification Scene_Generation Scene_Parsing Scene_Text Segmentation Self-Supervised Semantic_Instance_Segmentation Semantic_Segmentation Semi_Global Semi_Supervised Sence_graph Sentiment Sentiment_Classification Sketch SLAM Sparse Speech Speech_Recognition Style_Transfer Summarization Super_Resolution Surveillance Survey Text_Classification Text_Generation Tracking Transfer_Learning Transformer Unsupervised Video_Caption Video_Classification Video_Indexing Video_Prediction Video_Retrieval Visual_Relation VQA Weakly_Supervised Zero-Shot