今天跟大家分享一篇语音相关的论文合集：cs.SD语音7篇，eess.AS音频处理9篇。
cs.SD语音
【1】 Pure Data and INScore: Animated notation for new music
标题：纯数据和INScore：新音乐的动画乐谱
链接：https://arxiv.org/abs/2208.04877
摘要：新的音乐是用计算机制作的，利用它的图形显示而不是它的音频算法。纯数据可以用来创作它们。本文将展示一个案例研究，使用纯数据，与INScore连接，制作一种新的类型的乐谱，使用动画记谱法或动态音乐谱法与表演者一起制作音乐。这个样本是由文本的作者，并且它将展示一些可以使用软件组合完成的记谱可能性。这将伴随着一个简单的预测，一个音乐家可以用它来演奏。摘要：New music is made with computers, taking advantage of its graphics displays rather than its audio algorithms. Pure Data can be used to compose them. This essay will show a case study that uses Pure Data, in connection with INScore, for making a new type of score that uses animated notation or dynamic musicography for making music with performers. This sample was made by the author of the text, and it will show a number of notation possibilities that can be done using the combination of software. This will be accompanied by a simple prediction of what a musician could perform with it.
【2】 Thai Wav2Vec2.0 with CommonVoice V8
标题：泰文Wav2Vec2.0，带有通用语音V8
链接：https://arxiv.org/abs/2208.04799
作者：Wannaphong Phatthiyaphaibun,Chompakorn Chaksangchaichot,Peerat Limkonchotiwat,Ekapol Chuangsuwanich,Sarana Nutanong摘要：最近，自动语音识别（ASR），一个将音频转换为文本的系统，在机器学习界引起了很大的关注，因此，在HuggingFace中发布了很多公开可用的模型，但这些ASR模型大多是英文的;只有一小部分模型有泰语版本。此外，大多数泰语ASR模型都是封闭源代码的，现有的开放源代码模型的性能缺乏鲁棒性。为了解决这个问题，我们使用泰国CommonVoice语料库V8在预先训练的XLSR-Wav2Vec模型上训练新的ASR模型，并且训练三元组语言模型以提高我们的ASR模型的性能。我们希望我们的模式将有利于个人和ASR社区在泰国。摘要：Recently, Automatic Speech Recognition (ASR), a system that converts audio into text, has caught a lot of attention in the machine learning community. Thus, a lot of publicly available models were released in HuggingFace. However, most of these ASR models are available in English; only a minority of the models are available in Thai. Additionally, most of the Thai ASR models are closed-sourced, and the performance of existing open-sourced models lacks robustness. To address this problem, we train a new ASR model on a pre-trained XLSR-Wav2Vec model with the Thai CommonVoice corpus V8 and train a trigram language model to boost the performance of our ASR model. We hope that our models will be beneficial to individuals and the ASR community in Thailand.
【3】 DDSP-based Singing Vocoders: A New Subtractive-based Synthesizer and A Comprehensive Evaluation
标题：基于DDSP的歌唱声码器：一种新的减法合成器及其综合评价
链接：https://arxiv.org/abs/2208.04756
作者：Da-Yi Wu1,Wen-Yi Hsiao,Fu-Rong Yang,Oscar Friedman,Warren Jackson,Scott Bruzenak,Yi-Wen Liu,Yi-Hsuan Yang摘要：声码器是一种有条件的音频生成模型，可将梅尔声谱图等声学特征转换为波形。（DDSP）、本文提出了一种新的歌唱声码器—-SawSing，它利用线性时变有限冲激响应滤波器对锯齿波源信号进行滤波，从而合成歌唱声的谐波部分，该滤波器的系数由输入信号的谐波分量估计得到。SawSing利用神经网络来产生语音信号，由于该方法增强了语音信号的相位连续性，因此，实验结果表明，SawSing算法收敛速度快，性能优于现有的生成式对抗网络和扩散网络.在资源有限的情况下仅使用3个训练记录和3小时的训练时间。摘要：A vocoder is a conditional audio generation model that converts acoustic features such as mel-spectrograms into waveforms. Taking inspiration from Differentiable Digital Signal Processing (DDSP), we propose a new vocoder named SawSing for singing voices. SawSing synthesizes the harmonic part of singing voices by filtering a sawtooth source signal with a linear time-variant finite impulse response filter whose coefficients are estimated from the input mel-spectrogram by a neural network. As this approach enforces phase continuity, SawSing can generate singing voices without the phase-discontinuity glitch of many existing vocoders. Moreover, the source-filter assumption provides an inductive bias that allows SawSing to be trained on a small amount of data. Our experiments show that SawSing converges much faster and outperforms state-of-the-art generative adversarial network and diffusion-based vocoders in a resource-limited scenario with only 3 training recordings and a 3-hour training time.
【4】 Denoising Induction Motor Sounds Using an Autoencoder
标题：基于自动编码器的异步电机声音降噪
链接：https://arxiv.org/abs/2208.04462
作者：Thanh Tran,Sebastian Bader,Jan Lundgren备注：9 pages, 10 figures, conference摘要：去噪是从声音信号中去除噪声，同时提高声音信号的质量和充分性的过程。去噪声音在语音处理、声音事件分类和机器故障检测系统中有许多应用。本文描述了一种创建自动编码器的方法，将有噪声的机器声音映射为干净的声音，以达到去噪的目的。声音中有几种类型的噪声，例如，环境噪声和信号处理方法产生的频率相关噪声。环境活动产生的噪声是环境噪声。在工厂，环境噪声可由车辆、钻井、勘测区工作或交谈的人员、风和流水产生。这些噪声在声音记录中表现为尖峰。在本文的范围内，以感应电机声音中的水龙头噪声为例，说明了高斯分布噪声和环境噪声的去除方法，并对机械故障数据库中的49个正常功能声音和197个水平错位故障声音进行了训练和验证（MAFAULDA）.均方误差以均方误差（MSE）为评价标准，评价了该自动编码器降噪后的语音与测试集原始语音的相似性，均方误差小于或等于0。对正常功能类的15个测试音进行两种类型的噪声去噪后，均方误差均为14，对水平错位故障类的60个测试音进行去噪后，均方误差均小于等于0.15。低均方误差表明，使用所提出的训练后的自动编码器，生成的高斯噪声和环境噪声几乎都从原始声音中去除。摘要：Denoising is the process of removing noise from sound signals while improving the quality and adequacy of the sound signals. Denoising sound has many applications in speech processing, sound events classification, and machine failure detection systems. This paper describes a method for creating an autoencoder to map noisy machine sounds to clean sounds for denoising purposes. There are several types of noise in sounds, for example, environmental noise and generated frequency-dependent noise from signal processing methods. Noise generated by environmental activities is environmental noise. In the factory, environmental noise can be created by vehicles, drilling, people working or talking in the survey area, wind, and flowing water. Those noises appear as spikes in the sound record. In the scope of this paper, we demonstrate the removal of generated noise with Gaussian distribution and the environmental noise with a specific example of the water sink faucet noise from the induction motor sounds. The proposed method was trained and verified on 49 normal function sounds and 197 horizontal misalignment fault sounds from the Machinery Fault Database (MAFAULDA). The mean square error (MSE) was used as the assessment criteria to evaluate the similarity between denoised sounds using the proposed autoencoder and the original sounds in the test set. The MSE is below or equal to 0.14 when denoise both types of noises on 15 testing sounds of the normal function category. The MSE is below or equal to 0.15 when denoising 60 testing sounds on the horizontal misalignment fault category. The low MSE shows that both the generated Gaussian noise and the environmental noise were almost removed from the original sounds with the proposed trained autoencoder.
【5】 Extending GCC-PHAT using Shift Equivariant Neural Networks
标题：用移位等变神经网络扩展GCC-PHAT
链接：https://arxiv.org/abs/2208.04654
作者：Axel Berg,Mark O'Connor,Kalle Åström,Magnus Oskarsson备注：Proceedings of INTERSPEECH摘要：利用麦克风阵列进行说话人定位依赖于精确的时延估计技术，几十年来，基于广义互相关和相位变换的方法（GCC-PHAT）已被广泛地用于此目的，最近，GCC-PHAT还被用于向神经网络提供输入特征，以便去除噪声和混响的影响，但代价是失去了无噪声条件下的理论保证.我们提出了一种新的方法来扩展GCC-PHAT，其中使用保留包含在信号中的定时信息的移位等变神经网络对接收到的信号进行滤波。通过大量的实验表明，该模型在保证精确时延恢复的前提下，能够有效地降低GCC-PHAT算法在恶劣环境下的误差。摘要：Speaker localization using microphone arrays depends on accurate time delay estimation techniques. For decades, methods based on the generalized cross correlation with phase transform (GCC-PHAT) have been widely adopted for this purpose. Recently, the GCC-PHAT has also been used to provide input features to neural networks in order to remove the effects of noise and reverberation, but at the cost of losing theoretical guarantees in noise-free conditions. We propose a novel approach to extending the GCC-PHAT, where the received signals are filtered using a shift equivariant neural network that preserves the timing information contained in the signals. By extensive experiments we show that our model consistently reduces the error of the GCC-PHAT in adverse environments, with guarantees of exact time delay recovery in ideal conditions.
【6】 Recycling an anechoic pre-trained speech separation deep neural network for binaural dereverberation of a single source
标题：用于单源双耳去混响的再循环无回声预训练语音分离深度神经网络
链接：https://arxiv.org/abs/2208.04626
作者：Sania Gul,Muhammad Salman Khan,Syed Waqar Shah,Ata Ur-Rehman摘要：混响会降低正常人和听力受损者的可懂度.本文提出了一种新的心理声学去混响方法，该方法通过重复使用预先训练好的双耳无回声语音分离神经网络来实现.在训练深度神经网络时，首先利用训练好的双耳无回声语音分离神经网络对单个语音源进行去混响处理，然后再利用训练好的双耳无回声语音分离神经网络对混响信号进行去混响处理.实验结果表明，该方法具有较好的去混响效果.（DNN）是一个耗时且计算量大的过程，使用预先训练的分离网络来进行去混响的优点在于网络不需要重新训练，将混响源的耳间线索提供给该预先训练的神经网络以区分直接路径信号和混响语音。结果表明，信号可懂度平均提高1.3%，SRMR平均提高0.83dB（信号与混响能量比）和0.16分的语音质量感知评价（PESQ），清晰度为14%，质量为0.35分，优于采用谱减法的正交匹配追踪（OSS），一种基于机器学习的去混响算法。摘要：Reverberation results in reduced intelligibility for both normal and hearing-impaired listeners. This paper presents a novel psychoacoustic approach of dereverberation of a single speech source by recycling a pre-trained binaural anechoic speech separation neural network. As training the deep neural network (DNN) is a lengthy and computationally expensive process, the advantage of using a pre-trained separation network for dereverberation is that the network does not need to be retrained, saving both time and computational resources. The interaural cues of a reverberant source are given to this pretrained neural network to discriminate between the direct path signal and the reverberant speech. The results show an average improvement of 1.3% in signal intelligibility, 0.83 dB in SRMR (signal to reverberation energy ratio) and 0.16 points in perceptual evaluation of speech quality (PESQ) over other state-of-the-art signal processing dereverberation algorithms and 14% in intelligibility and 0.35 points in quality over orthogonal matching pursuit with spectral subtraction (OSS), a machine learning based dereverberation algorithm.
【7】 An Anchor-Free Detector for Continuous Speech Keyword Spotting
标题：一种无锚点的连续语音关键词检测器
链接：https://arxiv.org/abs/2208.04622
作者：Zhiyuan Zhao,Chuanxin Tang,Chengdong Yao,Chong Luo备注：Accepted by Interspeech 2022摘要：连续语音关键词识别（CSKWS）是一个在连续语音中检测预定义关键词的任务.本文将CSKWS看作一维对象检测任务，提出了一种新的无锚点检测器AF-KWS. AF-KWS通过一个单级深度神经网络直接回归关键词的中心位置和长度，AF-KWS是专门为这一语音任务量身定制的，我们引入了一个辅助的未知类来排除非语音或无声背景中的其他单词，并建立了两个基准数据集LibriTop-20和连续会议分析关键词在这两个数据集上的评估表明，我们提出的AF-KWS的性能大大优于参考方案，因此为未来的研究提供了一个不错的基线。摘要：Continuous Speech Keyword Spotting (CSKWS) is a task to detect predefined keywords in a continuous speech. In this paper, we regard CSKWS as a one-dimensional object detection task and propose a novel anchor-free detector, named AF-KWS, to solve the problem. AF-KWS directly regresses the center locations and lengths of the keywords through a single-stage deep neural network. In particular, AF-KWS is tailored for this speech task as we introduce an auxiliary unknown class to exclude other words from non-speech or silent background. We have built two benchmark datasets named LibriTop-20 and continuous meeting analysis keywords (CMAK) dataset for CSKWS. Evaluations on these two datasets show that our proposed AF-KWS outperforms reference schemes by a large margin, and therefore provides a decent baseline for future research.
【1】 Extending GCC-PHAT using Shift Equivariant Neural Networks
标题：用移位等变神经网络扩展GCC-PHAT
链接：https://arxiv.org/abs/2208.04654
* 与cs.SD语音【5】为同一篇
作者：Axel Berg,Mark O'Connor,Kalle Åström,Magnus Oskarsson备注：Proceedings of INTERSPEECH摘要：利用麦克风阵列进行说话人定位依赖于精确的时延估计技术，几十年来，基于广义互相关和相位变换的方法（GCC-PHAT）已被广泛地用于此目的，最近，GCC-PHAT还被用于向神经网络提供输入特征，以便去除噪声和混响的影响，但代价是失去了无噪声条件下的理论保证.我们提出了一种新的方法来扩展GCC-PHAT，其中使用保留包含在信号中的定时信息的移位等变神经网络对接收到的信号进行滤波。通过大量的实验表明，该模型在保证精确时延恢复的前提下，能够有效地降低GCC-PHAT算法在恶劣环境下的误差。摘要：Speaker localization using microphone arrays depends on accurate time delay estimation techniques. For decades, methods based on the generalized cross correlation with phase transform (GCC-PHAT) have been widely adopted for this purpose. Recently, the GCC-PHAT has also been used to provide input features to neural networks in order to remove the effects of noise and reverberation, but at the cost of losing theoretical guarantees in noise-free conditions. We propose a novel approach to extending the GCC-PHAT, where the received signals are filtered using a shift equivariant neural network that preserves the timing information contained in the signals. By extensive experiments we show that our model consistently reduces the error of the GCC-PHAT in adverse environments, with guarantees of exact time delay recovery in ideal conditions.
【2】 Recycling an anechoic pre-trained speech separation deep neural network for binaural dereverberation of a single source
标题：用于单源双耳去混响的再循环无回声预训练语音分离深度神经网络
链接：https://arxiv.org/abs/2208.04626
* 与cs.SD语音【6】为同一篇
作者：Sania Gul,Muhammad Salman Khan,Syed Waqar Shah,Ata Ur-Rehman摘要：混响会降低正常人和听力受损者的可懂度.本文提出了一种新的心理声学去混响方法，该方法通过重复使用预先训练好的双耳无回声语音分离神经网络来实现.在训练深度神经网络时，首先利用训练好的双耳无回声语音分离神经网络对单个语音源进行去混响处理，然后再利用训练好的双耳无回声语音分离神经网络对混响信号进行去混响处理.实验结果表明，该方法具有较好的去混响效果.（DNN）是一个耗时且计算量大的过程，使用预先训练的分离网络来进行去混响的优点在于网络不需要重新训练，将混响源的耳间线索提供给该预先训练的神经网络以区分直接路径信号和混响语音。结果表明，信号可懂度平均提高1.3%，SRMR平均提高0.83dB（信号与混响能量比）和0.16分的语音质量感知评价（PESQ），清晰度为14%，质量为0.35分，优于采用谱减法的正交匹配追踪（OSS），一种基于机器学习的去混响算法。摘要：Reverberation results in reduced intelligibility for both normal and hearing-impaired listeners. This paper presents a novel psychoacoustic approach of dereverberation of a single speech source by recycling a pre-trained binaural anechoic speech separation neural network. As training the deep neural network (DNN) is a lengthy and computationally expensive process, the advantage of using a pre-trained separation network for dereverberation is that the network does not need to be retrained, saving both time and computational resources. The interaural cues of a reverberant source are given to this pretrained neural network to discriminate between the direct path signal and the reverberant speech. The results show an average improvement of 1.3% in signal intelligibility, 0.83 dB in SRMR (signal to reverberation energy ratio) and 0.16 points in perceptual evaluation of speech quality (PESQ) over other state-of-the-art signal processing dereverberation algorithms and 14% in intelligibility and 0.35 points in quality over orthogonal matching pursuit with spectral subtraction (OSS), a machine learning based dereverberation algorithm.
【3】 An Anchor-Free Detector for Continuous Speech Keyword Spotting
标题：一种无锚点的连续语音关键词检测器
链接：https://arxiv.org/abs/2208.04622
* 与cs.SD语音【7】为同一篇
作者：Zhiyuan Zhao,Chuanxin Tang,Chengdong Yao,Chong Luo备注：Accepted by Interspeech 2022摘要：连续语音关键词识别（CSKWS）是一个在连续语音中检测预定义关键词的任务.本文将CSKWS看作一维对象检测任务，提出了一种新的无锚点检测器AF-KWS. AF-KWS通过一个单级深度神经网络直接回归关键词的中心位置和长度，AF-KWS是专门为这一语音任务量身定制的，我们引入了一个辅助的未知类来排除非语音或无声背景中的其他单词，并建立了两个基准数据集LibriTop-20和连续会议分析关键词在这两个数据集上的评估表明，我们提出的AF-KWS的性能大大优于参考方案，因此为未来的研究提供了一个不错的基线。摘要：Continuous Speech Keyword Spotting (CSKWS) is a task to detect predefined keywords in a continuous speech. In this paper, we regard CSKWS as a one-dimensional object detection task and propose a novel anchor-free detector, named AF-KWS, to solve the problem. AF-KWS directly regresses the center locations and lengths of the keywords through a single-stage deep neural network. In particular, AF-KWS is tailored for this speech task as we introduce an auxiliary unknown class to exclude other words from non-speech or silent background. We have built two benchmark datasets named LibriTop-20 and continuous meeting analysis keywords (CMAK) dataset for CSKWS. Evaluations on these two datasets show that our proposed AF-KWS outperforms reference schemes by a large margin, and therefore provides a decent baseline for future research.
【4】 Pure Data and INScore: Animated notation for new music
标题：纯数据和INScore：新音乐的动画乐谱
链接：https://arxiv.org/abs/2208.04877
* 与cs.SD语音【1】为同一篇
摘要：新的音乐是用计算机制作的，利用它的图形显示而不是它的音频算法。纯数据可以用来创作它们。本文将展示一个案例研究，使用纯数据，与INScore连接，制作一种新的类型的乐谱，使用动画记谱法或动态音乐谱法与表演者一起制作音乐。这个样本是由文本的作者，并且它将展示一些可以使用软件组合完成的记谱可能性。这将伴随着一个简单的预测，一个音乐家可以用它来演奏。摘要：New music is made with computers, taking advantage of its graphics displays rather than its audio algorithms. Pure Data can be used to compose them. This essay will show a case study that uses Pure Data, in connection with INScore, for making a new type of score that uses animated notation or dynamic musicography for making music with performers. This sample was made by the author of the text, and it will show a number of notation possibilities that can be done using the combination of software. This will be accompanied by a simple prediction of what a musician could perform with it.
【5】 Thai Wav2Vec2.0 with CommonVoice V8
标题：泰文Wav2Vec2.0，带有通用语音V8
链接：https://arxiv.org/abs/2208.04799
* 与cs.SD语音【2】为同一篇
作者：Wannaphong Phatthiyaphaibun,Chompakorn Chaksangchaichot,Peerat Limkonchotiwat,Ekapol Chuangsuwanich,Sarana Nutanong摘要：最近，自动语音识别（ASR），一个将音频转换为文本的系统，在机器学习界引起了很大的关注，因此，在HuggingFace中发布了很多公开可用的模型，但这些ASR模型大多是英文的;只有一小部分模型有泰语版本。此外，大多数泰语ASR模型都是封闭源代码的，现有的开放源代码模型的性能缺乏鲁棒性。为了解决这个问题，我们使用泰国CommonVoice语料库V8在预先训练的XLSR-Wav2Vec模型上训练新的ASR模型，并且训练三元组语言模型以提高我们的ASR模型的性能。我们希望我们的模式将有利于个人和ASR社区在泰国。摘要：Recently, Automatic Speech Recognition (ASR), a system that converts audio into text, has caught a lot of attention in the machine learning community. Thus, a lot of publicly available models were released in HuggingFace. However, most of these ASR models are available in English; only a minority of the models are available in Thai. Additionally, most of the Thai ASR models are closed-sourced, and the performance of existing open-sourced models lacks robustness. To address this problem, we train a new ASR model on a pre-trained XLSR-Wav2Vec model with the Thai CommonVoice corpus V8 and train a trigram language model to boost the performance of our ASR model. We hope that our models will be beneficial to individuals and the ASR community in Thailand.
【6】 DDSP-based Singing Vocoders: A New Subtractive-based Synthesizer and A Comprehensive Evaluation
标题：基于DDSP的歌唱声码器：一种新的减法合成器及其综合评价
链接：https://arxiv.org/abs/2208.04756
* 与cs.SD语音【3】为同一篇
作者：Da-Yi Wu1,Wen-Yi Hsiao,Fu-Rong Yang,Oscar Friedman,Warren Jackson,Scott Bruzenak,Yi-Wen Liu,Yi-Hsuan Yang摘要：声码器是一种有条件的音频生成模型，可将梅尔声谱图等声学特征转换为波形。（DDSP）、本文提出了一种新的歌唱声码器—-SawSing，它利用线性时变有限冲激响应滤波器对锯齿波源信号进行滤波，从而合成歌唱声的谐波部分，该滤波器的系数由输入信号的谐波分量估计得到。SawSing利用神经网络来产生语音信号，由于该方法增强了语音信号的相位连续性，因此，实验结果表明，SawSing算法收敛速度快，性能优于现有的生成式对抗网络和扩散网络.在资源有限的情况下仅使用3个训练记录和3小时的训练时间。摘要：A vocoder is a conditional audio generation model that converts acoustic features such as mel-spectrograms into waveforms. Taking inspiration from Differentiable Digital Signal Processing (DDSP), we propose a new vocoder named SawSing for singing voices. SawSing synthesizes the harmonic part of singing voices by filtering a sawtooth source signal with a linear time-variant finite impulse response filter whose coefficients are estimated from the input mel-spectrogram by a neural network. As this approach enforces phase continuity, SawSing can generate singing voices without the phase-discontinuity glitch of many existing vocoders. Moreover, the source-filter assumption provides an inductive bias that allows SawSing to be trained on a small amount of data. Our experiments show that SawSing converges much faster and outperforms state-of-the-art generative adversarial network and diffusion-based vocoders in a resource-limited scenario with only 3 training recordings and a 3-hour training time.
【7】 ASR Error Correction with Constrained Decoding on Operation Prediction
标题：基于操作预测约束译码的ASR纠错
链接：https://arxiv.org/abs/2208.04641
作者：Jingyuan Yang,Rongjun Li,Wei Peng备注：Accepted in Interspeech 2022摘要：纠错技术对改进自动语音识别的输出仍然有效现有的基于编解码器结构的端到端纠错方法在解码阶段处理所有令牌，产生了不期望的延迟.本文提出了一种利用纠错操作预测的ASR纠错方法.更具体地说，我们在编码器和解码器之间构建预测器以学习是否应该保留令牌（“K”），已删除（“D”）、或已更改（“C”）将解码限制为仅部分输入序列嵌入在三个公共数据集上的实验表明，该方法能有效地减少ASR纠错解码过程的延迟，推理速度至少提高了3倍（3.4倍和5.7倍），同时我们提出的两种模型与可靠的编解码器基准相比保持了相同的准确性水平（WER分别降低了0.53%和1.69%）。同时，我们制作并发布了一个基准数据集，为ASR纠错社区提供帮助，以促进这方面的研究。摘要：Error correction techniques remain effective to refine outputs from automatic speech recognition (ASR) models. Existing end-to-end error correction methods based on an encoder-decoder architecture process all tokens in the decoding phase, creating undesirable latency. In this paper, we propose an ASR error correction method utilizing the predictions of correction operations. More specifically, we construct a predictor between the encoder and the decoder to learn if a token should be kept ("K"), deleted ("D"), or changed ("C") to restrict decoding to only part of the input sequence embeddings (the "C" tokens) for fast inference. Experiments on three public datasets demonstrate the effectiveness of the proposed approach in reducing the latency of the decoding process in ASR correction. It enhances the inference speed by at least three times (3.4 and 5.7 times) while maintaining the same level of accuracy (with WER reductions of 0.53% and 1.69% respectively) for our two proposed models compared to a solid encoder-decoder baseline. In the meantime, we produce and release a benchmark dataset contributing to the ASR error correction community to foster research along this line.
【8】 Speaker-adaptive Lip Reading with User-dependent Padding
标题：具有用户相关填充的说话者自适应唇读
链接：https://arxiv.org/abs/2208.04498
作者：Minsu Kim,Hyunjun Kim,Yong Man Ro摘要：唇读主要是根据嘴唇的运动来预测说话人的语音.由于唇读是基于视觉信息来建模的，其性能对人的嘴唇外观和运动具有固有的敏感性.这使得唇读模型在应用于未看过的说话人时，由于训练和测试条件的不匹配而表现出较差的性能.说话人自适应技术旨在减少训练和测试说话人之间的不匹配，从而引导训练好的模型专注于对语音内容的建模，而不受说话人变化的干扰.与基于音频的语音识别相比，在唇读中说话人自适应方法的研究还不多见.本文针对唇读模型在未知说话人情况下性能下降的问题，提出了一种说话人自适应的唇读方法，即用户相关的填充。用户相关的填充是特定于说话人的输入，其可以参与预先训练的唇读模型的视觉特征提取阶段。因此，在视觉特征编码期间，可以自适应地针对各个说话人考虑不同说话人的嘴唇外观和运动信息。此外，该方法不需要1）任何额外的层，2）修改预训练模型的学习权值，3）预训练过程中使用的训练数据的说话人标记.它可以通过只学习依赖于用户的填充，以有监督或无监督的方式直接适应未见过的说话人.最后，为了解决公共唇读数据库中说话人信息不足的问题，我们将一个著名的视听数据库中的说话人标注为LRW，并设计了一个未知说话人唇读场景LRW-ID。摘要：Lip reading aims to predict speech based on lip movements alone. As it focuses on visual information to model the speech, its performance is inherently sensitive to personal lip appearances and movements. This makes the lip reading models show degraded performance when they are applied to unseen speakers due to the mismatch between training and testing conditions. Speaker adaptation technique aims to reduce this mismatch between train and test speakers, thus guiding a trained model to focus on modeling the speech content without being intervened by the speaker variations. In contrast to the efforts made in audio-based speech recognition for decades, the speaker adaptation methods have not well been studied in lip reading. In this paper, to remedy the performance degradation of lip reading model on unseen speakers, we propose a speaker-adaptive lip reading method, namely user-dependent padding. The user-dependent padding is a speaker-specific input that can participate in the visual feature extraction stage of a pre-trained lip reading model. Therefore, the lip appearances and movements information of different speakers can be considered during the visual feature encoding, adaptively for individual speakers. Moreover, the proposed method does not need 1) any additional layers, 2) to modify the learned weights of the pre-trained model, and 3) the speaker label of train data used during pre-train. It can directly adapt to unseen speakers by learning the user-dependent padding only, in a supervised or unsupervised manner. Finally, to alleviate the speaker information insufficiency in public lip reading databases, we label the speaker of a well-known audio-visual database, LRW, and design an unseen-speaker lip reading scenario named LRW-ID.
【9】 Denoising Induction Motor Sounds Using an Autoencoder
标题：基于自动编码器的异步电机声音降噪
链接：https://arxiv.org/abs/2208.04462
* 与cs.SD语音【4】为同一篇
作者：Thanh Tran,Sebastian Bader,Jan Lundgren备注：9 pages, 10 figures, conference摘要：去噪是从声音信号中去除噪声，同时提高声音信号的质量和充分性的过程。去噪声音在语音处理、声音事件分类和机器故障检测系统中有许多应用。本文描述了一种创建自动编码器的方法，将有噪声的机器声音映射为干净的声音，以达到去噪的目的。声音中有几种类型的噪声，例如，环境噪声和信号处理方法产生的频率相关噪声。环境活动产生的噪声是环境噪声。在工厂，环境噪声可由车辆、钻井、勘测区工作或交谈的人员、风和流水产生。这些噪声在声音记录中表现为尖峰。在本文的范围内，以感应电机声音中的水龙头噪声为例，说明了高斯分布噪声和环境噪声的去除方法，并对机械故障数据库中的49个正常功能声音和197个水平错位故障声音进行了训练和验证（MAFAULDA）.均方误差以均方误差（MSE）为评价标准，评价了该自动编码器降噪后的语音与测试集原始语音的相似性，均方误差小于或等于0。对正常功能类的15个测试音进行两种类型的噪声去噪后，均方误差均为14，对水平错位故障类的60个测试音进行去噪后，均方误差均小于等于0.15。低均方误差表明，使用所提出的训练后的自动编码器，生成的高斯噪声和环境噪声几乎都从原始声音中去除。摘要：Denoising is the process of removing noise from sound signals while improving the quality and adequacy of the sound signals. Denoising sound has many applications in speech processing, sound events classification, and machine failure detection systems. This paper describes a method for creating an autoencoder to map noisy machine sounds to clean sounds for denoising purposes. There are several types of noise in sounds, for example, environmental noise and generated frequency-dependent noise from signal processing methods. Noise generated by environmental activities is environmental noise. In the factory, environmental noise can be created by vehicles, drilling, people working or talking in the survey area, wind, and flowing water. Those noises appear as spikes in the sound record. In the scope of this paper, we demonstrate the removal of generated noise with Gaussian distribution and the environmental noise with a specific example of the water sink faucet noise from the induction motor sounds. The proposed method was trained and verified on 49 normal function sounds and 197 horizontal misalignment fault sounds from the Machinery Fault Database (MAFAULDA). The mean square error (MSE) was used as the assessment criteria to evaluate the similarity between denoised sounds using the proposed autoencoder and the original sounds in the test set. The MSE is below or equal to 0.14 when denoise both types of noises on 15 testing sounds of the normal function category. The MSE is below or equal to 0.15 when denoising 60 testing sounds on the horizontal misalignment fault category. The low MSE shows that both the generated Gaussian noise and the environmental noise were almost removed from the original sounds with the proposed trained autoencoder.
机器翻译，仅供参考
每日语音速递