Paper Reading AI Learner

Diff3DS: Generating View-Consistent 3D Sketch via Differentiable Curve Rendering

2024-05-24 07:48:14
Yibo Zhang, Lihong Wang, Changqing Zou, Tieru Wu, Rui Ma

Abstract

3D sketches are widely used for visually representing the 3D shape and structure of objects or scenes. However, the creation of 3D sketch often requires users to possess professional artistic skills. Existing research efforts primarily focus on enhancing the ability of interactive sketch generation in 3D virtual systems. In this work, we propose Diff3DS, a novel differentiable rendering framework for generating view-consistent 3D sketch by optimizing 3D parametric curves under various supervisions. Specifically, we perform perspective projection to render the 3D rational Bézier curves into 2D curves, which are subsequently converted to a 2D raster image via our customized differentiable rasterizer. Our framework bridges the domains of 3D sketch and raster image, achieving end-toend optimization of 3D sketch through gradients computed in the 2D image domain. Our Diff3DS can enable a series of novel 3D sketch generation tasks, including textto-3D sketch and image-to-3D sketch, supported by the popular distillation-based supervision, such as Score Distillation Sampling (SDS). Extensive experiments have yielded promising results and demonstrated the potential of our framework.

Abstract (translated)

3D草图广泛用于表示物体或场景的3D形状和结构。然而,创建3D草图通常需要用户具备专业艺术技能。现有的研究主要集中在通过在3D虚拟系统上增强交互式草图生成的能力。在这项工作中,我们提出了Diff3DS,一种通过在各种监督下优化3D参数曲线来生成视差一致3D草图的新颖可导渲染框架。具体来说,我们对3D理性Bézier曲线进行透视投影,将其转换为2D曲线,然后通过我们自定义的差分纹理映射器将其转换为2D位图。我们的框架将3D草图和位图领域联系起来,通过在2D图像域计算出的梯度实现端到端优化3D草图。我们的Diff3DS可以实现一系列新颖的3D草图生成任务,包括文本到3D草图和图像到3D草图,通过流行的基于蒸馏的监督,如Score Distillation Sampling(SDS)。大量的实验结果表明,我们的框架具有很好的前景和潜力。

URL

https://arxiv.org/abs/2405.15305

PDF

https://arxiv.org/pdf/2405.15305.pdf


Tags
3D Action Action_Localization Action_Recognition Activity Adversarial Agent Attention Autonomous Bert Boundary_Detection Caption Chat Classification CNN Compressive_Sensing Contour Contrastive_Learning Deep_Learning Denoising Detection Dialog Diffusion Drone Dynamic_Memory_Network Edge_Detection Embedding Embodied Emotion Enhancement Face Face_Detection Face_Recognition Facial_Landmark Few-Shot Gait_Recognition GAN Gaze_Estimation Gesture Gradient_Descent Handwriting Human_Parsing Image_Caption Image_Classification Image_Compression Image_Enhancement Image_Generation Image_Matting Image_Retrieval Inference Inpainting Intelligent_Chip Knowledge Knowledge_Graph Language_Model LLM Matching Medical Memory_Networks Multi_Modal Multi_Task NAS NMT Object_Detection Object_Tracking OCR Ontology Optical_Character Optical_Flow Optimization Person_Re-identification Point_Cloud Portrait_Generation Pose Pose_Estimation Prediction QA Quantitative Quantitative_Finance Quantization Re-identification Recognition Recommendation Reconstruction Regularization Reinforcement_Learning Relation Relation_Extraction Represenation Represenation_Learning Restoration Review RNN Robot Salient Scene_Classification Scene_Generation Scene_Parsing Scene_Text Segmentation Self-Supervised Semantic_Instance_Segmentation Semantic_Segmentation Semi_Global Semi_Supervised Sence_graph Sentiment Sentiment_Classification Sketch SLAM Sparse Speech Speech_Recognition Style_Transfer Summarization Super_Resolution Surveillance Survey Text_Classification Text_Generation Tracking Transfer_Learning Transformer Unsupervised Video_Caption Video_Classification Video_Indexing Video_Prediction Video_Retrieval Visual_Relation VQA Weakly_Supervised Zero-Shot