The Latest Papers about AI
-
Simplifying Semantic Annotations of SMCalFlow
Abstract
SMCalFlow is a large corpus of semantically detailed annotations of task-oriented natural dialogues. The annotations use a dataflow approach, in which the annotations are programs which represent user requests. Despite the availability, size and richness of this annotated corpus, it has seen only very limited use in dialogue systems research work, at least in part due to the difficulty in understanding and using the annotations. To address these difficulties, this paper suggests a simplification of the SMCalFlow annotations, as well as releases code needed to inspect the execution of the annotated dataflow programs, which should allow researchers of dialogue systems an easy entry point to experiment with various dataflow based implementations and annotations.
Abstract (translated)
URL
https://arxiv.org/abs/2206.13425
PDF
-
Generalized Two Color Map Theorem -- Complete Theorem of Robust Gait Plan for a Tilt-rotor
Abstract
Gait plan is a procedure that is typically applied on the ground robots, e.g., quadrupedal robots; the tilt-rotor, a novel type of quadrotor with eight inputs, is not one of them. While controlling the tilt-rotor relying on feedback linearization, the tilting angles (inputs) are expected to change over-intensively, which may not be expected in the application. To help suppress the intensive change in the tilting angles, a gait plan procedure is introduced to the tilt-rotor before feedback linearization. The tilting angles are specified with time in advance by users rather than given by the control rule. However, based on this scenario, the decoupling matrix in feedback linearization can be singular for some attitudes, combinations of roll angle and pitch angle. It hinders the further application of the feedback linearization. With this concern, Two Color Map Theorem is established to maximize the acceptable attitude region, where the combinations of roll and pitch will give an invertible decoupling matrix. That theorem, however, over-restricts the choice of the tilting angles, which can rule out some feasible robust gaits. This paper gives the generalized Two Color Map Theorem; all the robust gaits can be found based on this generalized theorem. The robustness of three gaits that satisfy this generalized Two Color Map Theorem (while violating Two Color Map Theorem) are analyzed. The results show that Generalized Two Color Map Theorem completes the search for the robust gaits for a tilt-rotor.
Abstract (translated)
URL
https://arxiv.org/abs/2206.13422
PDF
-
DeStripe: A Self2Self Spatio-Spectral Graph Neural Network with Unfolded Hessian for Stripe Artifact Removal in Light-sheet Microscopy
2022-06-27 16:13:57Yu Liu, Kurt Weiss, Nassir Navab, Carsten Marr, Jan Huisken, Tingying PengAbstract
Light-sheet fluorescence microscopy (LSFM) is a cutting-edge volumetric imaging technique that allows for three-dimensional imaging of mesoscopic samples with decoupled illumination and detection paths. Although the selective excitation scheme of such a microscope provides intrinsic optical sectioning that minimizes out-of-focus fluorescence background and sample photodamage, it is prone to light absorption and scattering effects, which results in uneven illumination and striping artifacts in the images adversely. To tackle this issue, in this paper, we propose a blind stripe artifact removal algorithm in LSFM, called DeStripe, which combines a self-supervised spatio-spectral graph neural network with unfolded Hessian prior. Specifically, inspired by the desirable properties of Fourier transform in condensing striping information into isolated values in the frequency domain, DeStripe firstly localizes the potentially corrupted Fourier coefficients by exploiting the structural difference between unidirectional stripe artifacts and more isotropic foreground images. Affected Fourier coefficients can then be fed into a graph neural network for recovery, with a Hessian regularization unrolled to further ensure structures in the standard image space are well preserved. Since in realistic, stripe-free LSFM barely exists with a standard image acquisition protocol, DeStripe is equipped with a Self2Self denoising loss term, enabling artifact elimination without access to stripe-free ground truth images. Competitive experimental results demonstrate the efficacy of DeStripe in recovering corrupted biomarkers in LSFM with both synthetic and real stripe artifacts.
Abstract (translated)
URL
https://arxiv.org/abs/2206.13419
PDF
-
Is the Language Familiarity Effect gradual? A computational modelling approach
2022-06-27 16:08:42Maureen de Seyssel, Guillaume Wisniewski, Emmanuel DupouxAbstract
According to the Language Familiarity Effect (LFE), people are better at discriminating between speakers of their native language. Although this cognitive effect was largely studied in the literature, experiments have only been conducted on a limited number of language pairs and their results only show the presence of the effect without yielding a gradual measure that may vary across language pairs. In this work, we show that the computational model of LFE introduced by Thorburn, Feldmand and Schatz (2019) can address these two limitations. In a first experiment, we attest to this model's capacity to obtain a gradual measure of the LFE by replicating behavioural findings on native and accented speech. In a second experiment, we evaluate LFE on a large number of language pairs, including many which have never been tested on humans. We show that the effect is replicated across a wide array of languages, providing further evidence of its universality. Building on the gradual measure of LFE, we also show that languages belonging to the same family yield smaller scores, supporting the idea of an effect of language distance on LFE.
Abstract (translated)
URL
https://arxiv.org/abs/2206.13415
PDF
-
RES: A Robust Framework for Guiding Visual Explanation
2022-06-27 16:06:27Yuyang Gao, Tong Steven Sun, Guangji Bai, Siyi Gu, Sungsoo Ray Hong, Liang ZhaoAbstract
Despite the fast progress of explanation techniques in modern Deep Neural Networks (DNNs) where the main focus is handling "how to generate the explanations", advanced research questions that examine the quality of the explanation itself (e.g., "whether the explanations are accurate") and improve the explanation quality (e.g., "how to adjust the model to generate more accurate explanations when explanations are inaccurate") are still relatively under-explored. To guide the model toward better explanations, techniques in explanation supervision - which add supervision signals on the model explanation - have started to show promising effects on improving both the generalizability as and intrinsic interpretability of Deep Neural Networks. However, the research on supervising explanations, especially in vision-based applications represented through saliency maps, is in its early stage due to several inherent challenges: 1) inaccuracy of the human explanation annotation boundary, 2) incompleteness of the human explanation annotation region, and 3) inconsistency of the data distribution between human annotation and model explanation maps. To address the challenges, we propose a generic RES framework for guiding visual explanation by developing a novel objective that handles inaccurate boundary, incomplete region, and inconsistent distribution of human annotations, with a theoretical justification on model generalizability. Extensive experiments on two real-world image datasets demonstrate the effectiveness of the proposed framework on enhancing both the reasonability of the explanation and the performance of the backbone DNNs model.
Abstract (translated)
URL
https://arxiv.org/abs/2206.13413
PDF
-
Audio Similarity is Unreliable as a Proxy for Audio Quality
Abstract
Many audio processing tasks require perceptual assessment. However, the time and expense of obtaining ``gold standard'' human judgments limit the availability of such data. Most applications incorporate full reference or other similarity-based metrics (e.g. PESQ) that depend on a clean reference. Researchers have relied on such metrics to evaluate and compare various proposed methods, often concluding that small, measured differences imply one is more effective than another. This paper demonstrates several practical scenarios where similarity metrics fail to agree with human perception, because they: (1) vary with clean references; (2) rely on attributes that humans factor out when considering quality, and (3) are sensitive to imperceptible signal level differences. In those scenarios, we show that no-reference metrics do not suffer from such shortcomings and correlate better with human perception. We conclude therefore that similarity serves as an unreliable proxy for audio quality.
Abstract (translated)
URL
https://arxiv.org/abs/2206.13411
PDF
-
Explicitly incorporating spatial information to recurrent networks for agriculture
2022-06-27 15:57:42Claus Smitt, Michael Halstead, Alireza Ahmadi, Chris McCoolAbstract
In agriculture, the majority of vision systems perform still image classification. Yet, recent work has highlighted the potential of spatial and temporal cues as a rich source of information to improve the classification performance. In this paper, we propose novel approaches to explicitly capture both spatial and temporal information to improve the classification of deep convolutional neural networks. We leverage available RGB-D images and robot odometry to perform inter-frame feature map spatial registration. This information is then fused within recurrent deep learnt models, to improve their accuracy and robustness. We demonstrate that this can considerably improve the classification performance with our best performing spatial-temporal model (ST-Atte) achieving absolute performance improvements for intersection-over-union (IoU[%]) of 4.7 for crop-weed segmentation and 2.6 for fruit (sweet pepper) segmentation. Furthermore, we show that these approaches are robust to variable framerates and odometry errors, which are frequently observed in real-world applications.
Abstract (translated)
URL
https://arxiv.org/abs/2206.13406
PDF
-
Avocodo: Generative Adversarial Network for Artifact-free Vocoder
2022-06-27 15:54:41Taejun Bak, Junmo Lee, Hanbin Bae, Jinhyeok Yang, Jae-Sung Bae, Young-Sun JooAbstract
Neural vocoders based on the generative adversarial neural network (GAN) have been widely used due to their fast inference speed and lightweight networks while generating high-quality speech waveforms. Since the perceptually important speech components are primarily concentrated in the low-frequency band, most of the GAN-based neural vocoders perform multi-scale analysis that evaluates downsampled speech waveforms. This multi-scale analysis helps the generator improve speech intelligibility. However, in preliminary experiments, we observed that the multi-scale analysis which focuses on the low-frequency band causes unintended artifacts, e.g., aliasing and imaging artifacts, and these artifacts degrade the synthesized speech waveform quality. Therefore, in this paper, we investigate the relationship between these artifacts and GAN-based neural vocoders and propose a GAN-based neural vocoder, called Avocodo, that allows the synthesis of high-fidelity speech with reduced artifacts. We introduce two kinds of discriminators to evaluate waveforms in various perspectives: a collaborative multi-band discriminator and a sub-band discriminator. We also utilize a pseudo quadrature mirror filter bank to obtain downsampled multi-band waveforms while avoiding aliasing. The experimental results show that Avocodo outperforms conventional GAN-based neural vocoders in both speech and singing voice synthesis tasks and can synthesize artifact-free speech. Especially, Avocodo is even capable to reproduce high-quality waveforms of unseen speakers.
Abstract (translated)
URL
https://arxiv.org/abs/2206.13404
PDF
-
Transfer Learning via Test-Time Neural Networks Aggregation
2022-06-27 15:46:05Bruno Casella, Alessio Barbaro Chisari, Sebastiano Battiato, Mario Valerio GiuffridaAbstract
It has been demonstrated that deep neural networks outperform traditional machine learning. However, deep networks lack generalisability, that is, they will not perform as good as in a new (testing) set drawn from a different distribution due to the domain shift. In order to tackle this known issue, several transfer learning approaches have been proposed, where the knowledge of a trained model is transferred into another to improve performance with different data. However, most of these approaches require additional training steps, or they suffer from catastrophic forgetting that occurs when a trained model has overwritten previously learnt knowledge. We address both problems with a novel transfer learning approach that uses network aggregation. We train dataset-specific networks together with an aggregation network in a unified framework. The loss function includes two main components: a task-specific loss (such as cross-entropy) and an aggregation loss. The proposed aggregation loss allows our model to learn how trained deep network parameters can be aggregated with an aggregation operator. We demonstrate that the proposed approach learns model aggregation at test time without any further training step, reducing the burden of transfer learning to a simple arithmetical operation. The proposed approach achieves comparable performance w.r.t. the baseline. Besides, if the aggregation operator has an inverse, we will show that our model also inherently allows for selective forgetting, i.e., the aggregated model can forget one of the datasets it was trained on, retaining information on the others.
Abstract (translated)
URL
https://arxiv.org/abs/2206.13399
PDF
-
Mushroom image recognition and distance generation based on attention-mechanism model and genetic information
2022-06-27 15:43:03Wenbin Liao, Jiewen Xiao, Chengbo Zhao, Yonggong Han, ZhiJie Geng, Jianxin Wang, Yihua YangAbstract
The species identification of Macrofungi, i.e. mushrooms, has always been a challenging task. There are still a large number of poisonous mushrooms that have not been found, which poses a risk to people's life. However, the traditional identification method requires a large number of experts with knowledge in the field of taxonomy for manual identification, it is not only inefficient but also consumes a lot of manpower and capital costs. In this paper, we propose a new model based on attention-mechanism, MushroomNet, which applies the lightweight network MobileNetV3 as the backbone model, combined with the attention structure proposed by us, and has achieved excellent performance in the mushroom recognition task. On the public dataset, the test accuracy of the MushroomNet model has reached 83.9%, and on the local dataset, the test accuracy has reached 77.4%. The proposed attention mechanisms well focused attention on the bodies of mushroom image for mixed channel attention and the attention heat maps visualized by Grad-CAM. Further, in this study, genetic distance was added to the mushroom image recognition task, the genetic distance was used as the representation space, and the genetic distance between each pair of mushroom species in the dataset was used as the embedding of the genetic distance representation space, so as to predict the image distance and species. identify. We found that using the MES activation function can predict the genetic distance of mushrooms very well, but the accuracy is lower than that of SoftMax. The proposed MushroomNet was demonstrated it shows great potential for automatic and online mushroom image and the proposed automatic procedure would assist and be a reference to traditional mushroom classification.
Abstract (translated)
URL
https://arxiv.org/abs/2206.13383
PDF
-
TextDCT: Arbitrary-Shaped Text Detection via Discrete Cosine Transform Mask
2022-06-27 15:42:25Yuchen Su, Zhiwen Shao, Yong Zhou, Fanrong Meng, Hancheng Zhu, Bing Liu, Rui YaoAbstract
Arbitrary-shaped scene text detection is a challenging task due to the variety of text changes in font, size, color, and orientation. Most existing regression based methods resort to regress the masks or contour points of text regions to model the text instances. However, regressing the complete masks requires high training complexity, and contour points are not sufficient to capture the details of highly curved texts. To tackle the above limitations, we propose a novel light-weight anchor-free text detection framework called TextDCT, which adopts the discrete cosine transform (DCT) to encode the text masks as compact vectors. Further, considering the imbalanced number of training samples among pyramid layers, we only employ a single-level head for top-down prediction. To model the multi-scale texts in a single-level head, we introduce a novel positive sampling strategy by treating the shrunk text region as positive samples, and design a feature awareness module (FAM) for spatial-awareness and scale-awareness by fusing rich contextual information and focusing on more significant features. Moreover, we propose a segmented non-maximum suppression (S-NMS) method that can filter low-quality mask regressions. Extensive experiments are conducted on four challenging datasets, which demonstrate our TextDCT obtains competitive performance on both accuracy and efficiency. Specifically, TextDCT achieves F-measure of 85.1 at 17.2 frames per second (FPS) and F-measure of 84.9 at 15.1 FPS for CTW1500 and Total-Text datasets, respectively.
Abstract (translated)
URL
https://arxiv.org/abs/2206.13381
PDF
-
Interpretable Acoustic Representation Learning on Breathing and Speech Signals for COVID-19 Detection
2022-06-27 15:20:51Debottam Dutta, Debarpan Bhattacharya, Sriram Ganapathy, Amir H. Poorjam, Deepak Mittal, Maneesh SinghAbstract
In this paper, we describe an approach for representation learning of audio signals for the task of COVID-19 detection. The raw audio samples are processed with a bank of 1-D convolutional filters that are parameterized as cosine modulated Gaussian functions. The choice of these kernels allows the interpretation of the filterbanks as smooth band-pass filters. The filtered outputs are pooled, log-compressed and used in a self-attention based relevance weighting mechanism. The relevance weighting emphasizes the key regions of the time-frequency decomposition that are important for the downstream task. The subsequent layers of the model consist of a recurrent architecture and the models are trained for a COVID-19 detection task. In our experiments on the Coswara data set, we show that the proposed model achieves significant performance improvements over the baseline system as well as other representation learning approaches. Further, the approach proposed is shown to be uniformly applicable for speech and breathing signals and for transfer learning from a larger data set.
Abstract (translated)
URL
https://arxiv.org/abs/2206.13365
PDF
-
A Lightweight Force-Controllable Wearable Arm Based on Magnetorheological-Hydrostatic Actuators
2022-06-27 15:14:07Catherine Véronneau, Jeff Denis, Louis-Philippe Lebel, Marc Denninger, Jean-Sébastien Plante, Alexandre GirardAbstract
Supernumerary Robotic Limbs (SRLs) are wearable robots augmenting human capabilities by acting as a co-worker, reaching objects, support human arms, etc. However, existing SRLs lack the mechanical backdrivability and bandwidth required for tasks where the interaction forces must be controllable such as painting, manipulating fragile objects, etc. Being highly backdrivable with a high bandwidth while minimizing weight presents a major technological challenge imposed by the limited performances of conventional electromagnetic actuators. This paper studies the feasibility of using magnetorheological (MR) clutches coupled to a low-friction hydrostatic transmission to provide a highly capable, but yet lightweight, force-controllable SRL. A 2.7 kg 2-DOFs wearable robotic arm is designed and built. Shoulder and elbow joints are designed to deliver 39 and 25 Nm, with 115 and 180° of range of motion. Experimental studies conducted on a one-DOF test bench and validated analytically demonstrate a high force bandwidth (>25 Hz) and a good ability to control interaction forces even when interacting with an external impedance. Furthermore, three force-control approaches are studied and demonstrated experimentally: open-loop, closed-loop on force, and closed-loop on pressure. All three methods are shown to be effective. Overall, the proposed MR-Hydrostatic actuation system is well-suited for a lightweight SRL interacting with both human and environment that add unpredictable disturbances.
Abstract (translated)
URL
https://arxiv.org/abs/2206.13361
PDF
-
iExam: A Novel Online Exam Monitoring and Analysis System Based on Face Detection and Recognition
2022-06-27 15:03:25Xu Yang, Daoyuan Wu, Xiao Yi, Jimmy H. M. Lee, Tan LeeAbstract
Online exams via video conference software like Zoom have been adopted in many schools due to COVID-19. While it is convenient, it is challenging for teachers to supervise online exams from simultaneously displayed student Zoom windows. In this paper, we propose iExam, an intelligent online exam monitoring and analysis system that can not only use face detection to assist invigilators in real-time student identification, but also be able to detect common abnormal behaviors (including face disappearing, rotating faces, and replacing with a different person during the exams) via a face recognition-based post-exam video analysis. To build such a novel system in its first kind, we overcome three challenges. First, we discover a lightweight approach to capturing exam video streams and analyzing them in real time. Second, we utilize the left-corner names that are displayed on each student's Zoom window and propose an improved OCR (optical character recognition) technique to automatically gather the ground truth for the student faces with dynamic positions. Third, we perform several experimental comparisons and optimizations to efficiently shorten the training and testing time required on teachers' PC. Our evaluation shows that iExam achieves high accuracy, 90.4% for real-time face detection and 98.4% for post-exam face recognition, while maintaining acceptable runtime performance. We have made iExam's source code available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2206.13356
PDF
-
A Novel Unified Self-alignment Method of SINS Based on FGO
Abstract
The self-alignment process can provide an accurate initial attitude of SINS. The conventional two-procedure method usually includes coarse and fine alignment processes. Coarse alignment is usually based on the OBA (optimization-based alignment) method, batch estimates the constant initial attitude at the beginning of self-alignment. OBA converges rapidly, however, the accuracy is low because the method doesn't consider IMU's bias errors. The fine alignment applies a recursive Bayesian filter which makes the system error estimation of the IMU more accurate, but at the same time, the attitude error converges slowly with a large heading misalignment angle. Researchers have proposed the unified self-alignment to achieve self-alignment in one procedure, but when the misalignment angle is large, the existing methods based on recursive Bayesian filter are still slow to converge. In this paper, a unified method based on batch estimator FGO (factor graph optimization) is raised. To the best as the author known, this is the first batch method capable of estimating all the systematic errors of IMU and the constant initial attitude simultaneously, with fast convergence and high accuracy. The effectiveness of this method is verified by simulation and physical experiments on a rotation SINS.
Abstract (translated)
URL
https://arxiv.org/abs/2206.13348
PDF
-
Distributional Gaussian Processes Layers for Out-of-Distribution Detection
2022-06-27 14:49:48Sebastian G. Popescu, David J. Sharp, James H. Cole, Konstantinos Kamnitsas, Ben GlockerAbstract
Machine learning models deployed on medical imaging tasks must be equipped with out-of-distribution detection capabilities in order to avoid erroneous predictions. It is unsure whether out-of-distribution detection models reliant on deep neural networks are suitable for detecting domain shifts in medical imaging. Gaussian Processes can reliably separate in-distribution data points from out-of-distribution data points via their mathematical construction. Hence, we propose a parameter efficient Bayesian layer for hierarchical convolutional Gaussian Processes that incorporates Gaussian Processes operating in Wasserstein-2 space to reliably propagate uncertainty. This directly replaces convolving Gaussian Processes with a distance-preserving affine operator on distributions. Our experiments on brain tissue-segmentation show that the resulting architecture approaches the performance of well-established deterministic segmentation algorithms (U-Net), which has not been achieved with previous hierarchical Gaussian Processes. Moreover, by applying the same segmentation model to out-of-distribution data (i.e., images with pathology such as brain tumors), we show that our uncertainty estimates result in out-of-distribution detection that outperforms the capabilities of previous Bayesian networks and reconstruction-based approaches that learn normative distributions. To facilitate future work our code is publicly available.
Abstract (translated)
URL
https://arxiv.org/abs/2206.13346
PDF
-
Prior-Guided One-shot Neural Architecture Search
2022-06-27 14:19:56Peijie Dong, Xin Niu, Lujun Li, Linzhen Xie, Wenbin Zou, Tian Ye, Zimian Wei, Hengyue PanAbstract
Neural architecture search methods seek optimal candidates with efficient weight-sharing supernet training. However, recent studies indicate poor ranking consistency about the performance between stand-alone architectures and shared-weight networks. In this paper, we present Prior-Guided One-shot NAS (PGONAS) to strengthen the ranking correlation of supernets. Specifically, we first explore the effect of activation functions and propose a balanced sampling strategy based on the Sandwich Rule to alleviate weight coupling in the supernet. Then, FLOPs and Zen-Score are adopted to guide the training of supernet with ranking correlation loss. Our PGONAS ranks 3rd place in the supernet Track Track of CVPR2022 Second lightweight NAS challenge. Code is available in this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2206.13329
PDF
-
Key-frame Guided Network for Thyroid Nodule Recognition using Ultrasound Videos
2022-06-27 14:03:26Yuchen Wang, Zhongyu Li, Xiangxiang Cui, Liangliang Zhang, Xiang Luo, Meng Yang, Shi ChangAbstract
Ultrasound examination is widely used in the clinical diagnosis of thyroid nodules (benign/malignant). However, the accuracy relies heavily on radiologist experience. Although deep learning techniques have been investigated for thyroid nodules recognition. Current solutions are mainly based on static ultrasound images, with limited temporal information used and inconsistent with clinical diagnosis. This paper proposes a novel method for the automated recognition of thyroid nodules through an exhaustive exploration of ultrasound videos and key-frames. We first propose a detection-localization framework to automatically identify the clinical key-frames with typical nodules in each ultrasound video. Based on the localized key-frames, we develop a key-frame guided video classification model for thyroid nodule recognition. Besides, we introduce motion attention module to help network focus on significant frames in an ultrasound video, which is consistent with clinical diagnosis. The proposed thyroid nodule recognition framework is validated on clinically collected ultrasound videos, demonstrating superior performance compared with other state-of-the-art methods.
Abstract (translated)
URL
https://arxiv.org/abs/2206.13318
PDF
-
Automatic identification of segmentation errors for radiotherapy using geometric learning
2022-06-27 14:01:52Edward G. A. Henderson, Andrew F. Green, Marcel van Herk, Eliana M. Vasquez OsorioAbstract
Automatic segmentation of organs-at-risk (OARs) in CT scans using convolutional neural networks (CNNs) is being introduced into the radiotherapy workflow. However, these segmentations still require manual editing and approval by clinicians prior to clinical use, which can be time consuming. The aim of this work was to develop a tool to automatically identify errors in 3D OAR segmentations without a ground truth. Our tool uses a novel architecture combining a CNN and graph neural network (GNN) to leverage the segmentation's appearance and shape. The proposed model is trained using self-supervised learning using a synthetically-generated dataset of segmentations of the parotid and with realistic contouring errors. The effectiveness of our model is assessed with ablation tests, evaluating the efficacy of different portions of the architecture as well as the use of transfer learning from an unsupervised pretext task. Our best performing model predicted errors on the parotid gland with a precision of 85.0% & 89.7% for internal and external errors respectively, and recall of 66.5% & 68.6%. This offline QA tool could be used in the clinical pathway, potentially decreasing the time clinicians spend correcting contours by detecting regions which require their attention. All our code is publicly available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2206.13317
PDF
-
Insights into Deep Non-linear Filters for Improved Multi-channel Speech Enhancement
Abstract
The key advantage of using multiple microphones for speech enhancement is that spatial filtering can be used to complement the tempo-spectral processing. In a traditional setting, linear spatial filtering (beamforming) and single-channel post-filtering are commonly performed separately. In contrast, there is a trend towards employing deep neural networks (DNNs) to learn a joint spatial and tempo-spectral non-linear filter, which means that the restriction of a linear processing model and that of a separate processing of spatial and tempo-spectral information can potentially be overcome. However, the internal mechanisms that lead to good performance of such data-driven filters for multi-channel speech enhancement are not well understood. Therefore, in this work, we analyse the properties of a non-linear spatial filter realized by a DNN as well as its interdependency with temporal and spectral processing by carefully controlling the information sources (spatial, spectral, and temporal) available to the network. We confirm the superiority of a non-linear spatial processing model, which outperforms an oracle linear spatial filter in a challenging speaker extraction scenario for a low number of microphones by 0.24 POLQA score. Our analyses reveal that in particular spectral information should be processed jointly with spatial information as this increases the spatial selectivity of the filter. Our systematic evaluation then leads to a simple network architecture, that outperforms state-of-the-art network architectures on a speaker extraction task by 0.22 POLQA score and by 0.32 POLQA score on the CHiME3 data.
Abstract (translated)
URL
https://arxiv.org/abs/2206.13310
PDF
