【1】 State of the Art of Audio- and Video-Based Solutions for AA
标题：基于音频和视频的AA解决方案的最新发展
链接：https://arxiv.org/abs/2207.01487
作者：Slavisa Aleksic,Michael Atanasov,Jean Calleja Agius,Kenneth Camilleri,Anto Cartolovni,Pau Climent-Peerez,Sara Colantonio,Stefania Cristina,Vladimir Despotovic,Hazim Kemal Ekenel,Ekrem Erakin,Francisco Florez-Revuelta,Danila Germanese,Nicole Grech,Steinunn Gróa Sigurðardóttir,Murat Emirzeoglu,Ivo Iliev,Mladjan Jovanovic,Martin Kampel,William Kearns,Andrzej Klimczuk,Lambros Lambrinos,Jennifer Lumetzberger,Wiktor Mucha,Sophie Noiret,Zada Pajalic,Rodrigo Rodriguez Peerez,Galidiya Petrova,Sintija Petrovica,Peter Pocta,Angelica Poli,Mara Pudane,Susanna Spinsante,Albert Ali Salah,Maria Jose Santofimia,Anna Sigridur Islind,Lacramioara Stoicu-Tivadar,Hilda Tellioglu,Andrej Zgank摘要：该报告阐述了基于音频和视频数据的最成功AAL应用和功能的最先进水平，即（i）生命记录和自我监测，（ii）生命体征的远程监测，（iii）情绪状态识别，（iv）食物摄入监测、活动和行为识别，（v）活动和个人协助，（vi）手势识别，（vii）坠落检测和预防，（viii）移动性评估和虚弱识别，以及（ix）认知和运动康复。对于这些应用场景，报告从科学进步、可用产品和研究项目方面阐述了发挥作用的状态。还强调了开放性挑战。摘要：The report illustrates the state of the art of the most successful AAL applications and functions based on audio and video data, namely (i) lifelogging and self-monitoring, (ii) remote monitoring of vital signs, (iii) emotional state recognition, (iv) food intake monitoring, activity and behaviour recognition, (v) activity and personal assistance, (vi) gesture recognition, (vii) fall detection and prevention, (viii) mobility assessment and frailty recognition, and (ix) cognitive and motor rehabilitation. For these application scenarios, the report illustrates the state of play in terms of scientific advances, available products and research project. The open challenges are also highlighted.
【2】 CaTT-KWS: A Multi-stage Customized Keyword Spotting Framework based on Cascaded Transducer-Transformer
标题：CATT-KWS：一种基于级联Transformer的多级定制关键词检测框架
链接：https://arxiv.org/abs/2207.01267
作者：Zhanheng Yang,Sining Sun,Jin Li,Xiaoming Zhang,Xiong Wang,Long Ma,Lei Xie机构：Audio, Speech and Language Processing Group, School of Computer Science, Northwestern, Polytechnical University, Xi’an, China, Tencent Technology Co., Ltd, Beijing, China备注：Accepted by Interspeech 2022摘要：定制关键字识别（KWS）在边缘设备上部署具有很大潜力，可以实现免提用户体验。然而，在实际应用中，虚警（FA）对于发现数十个甚至数百个关键词来说是一个严重的问题，这严重影响了用户体验。为了解决这个问题，在本文中，我们利用传感器和基于Transformer的声学模型的最新进展，提出了一种新的多级定制KWS框架，称为级联传感器-TransformerKWS（CaTT-KWS），该框架包括基于传感器的关键字检测器、基于帧级电话预测器的力对齐模块和基于Transformer的解码器。具体来说，流传感器模块用于在音频流中发现候选关键字。然后，使用电话预测器预测的电话后验值来实现力对齐，以完成第一阶段的关键字验证，并细化关键字的时间边界。最后，transformer解码器进一步验证触发的关键字。我们提出的CaTT-KWS框架在不明显影响关键字识别准确性的情况下有效地降低了FA率。具体来说，在具有挑战性的数据集上，我们可以每小时获得令人印象深刻的0.13 FA，与基于传感器的检测模型相比，FA相对减少了90%以上，而关键字识别准确率仅下降不到2%。摘要：Customized keyword spotting (KWS) has great potential to be deployed on edge devices to achieve hands-free user experience. However, in real applications, false alarm (FA) would be a serious problem for spotting dozens or even hundreds of keywords, which drastically affects user experience. To solve this problem, in this paper, we leverage the recent advances in transducer and transformer based acoustic models and propose a new multi-stage customized KWS framework named Cascaded Transducer-Transformer KWS (CaTT-KWS), which includes a transducer based keyword detector, a frame-level phone predictor based force alignment module and a transformer based decoder. Specifically, the streaming transducer module is used to spot keyword candidates in audio stream. Then force alignment is implemented using the phone posteriors predicted by the phone predictor to finish the first stage keyword verification and refine the time boundaries of keyword. Finally, the transformer decoder further verifies the triggered keyword. Our proposed CaTT-KWS framework reduces FA rate effectively without obviously hurting keyword recognition accuracy. Specifically, we can get impressively 0.13 FA per hour on a challenging dataset, with over 90% relative reduction on FA comparing to the transducer based detection model, while keyword recognition accuracy only drops less than 2%.
【3】 Minimizing Sequential Confusion Error in Speech Command Recognition
标题：语音命令识别中序列混淆误差的最小化
链接：https://arxiv.org/abs/2207.01261
作者：Zhanheng Yang,Hang Lv,Xiong Wang,Ao Zhang,Lei Xie机构：School of Computer Science, Northwestern Polytechnical University, Xi’an, China备注：Accepted by Interspeech 2022摘要：语音命令识别（SCR）通常用于资源受限的设备，以实现免提用户体验。然而，在实际应用中，由于边缘设备上部署的小模型容量有限，发音相似的命令之间经常会发生混淆，这严重影响了用户体验。在本文中，受语音识别中判别训练进展的启发，我们提出了一种新的最小化顺序混淆误差（MSCE）训练准则，特别是针对SCR的训练准则，旨在缓解命令混淆问题。具体来说，我们旨在提高基于MCE判别准则的目标命令与其他命令的判别能力。我们通过连接时序分类（CTC）定义不同命令的可能性。在训练过程中，我们提出了几种策略，使用先验知识为相似的发声命令创建混淆序列集，而不是创建整个非目标命令集，这样可以更好地节省训练资源，有效地减少命令混淆错误。具体来说，我们设计并比较了三种不同的混淆集构造策略。通过使用我们提出的方法，在0.01的虚警率下，我们可以相对减少33.7%的误拒绝率，在我们收集的语音命令集上，混淆错误减少18.28%。摘要：Speech command recognition (SCR) has been commonly used on resource constrained devices to achieve hands-free user experience. However, in real applications, confusion among commands with similar pronunciations often happens due to the limited capacity of small models deployed on edge devices, which drastically affects the user experience. In this paper, inspired by the advances of discriminative training in speech recognition, we propose a novel minimize sequential confusion error (MSCE) training criterion particularly for SCR, aiming to alleviate the command confusion problem. Specifically, we aim to improve the ability of discriminating the target command from other commands on the basis of MCE discriminative criteria. We define the likelihood of different commands through connectionist temporal classification (CTC). During training, we propose several strategies to use prior knowledge creating a confusing sequence set for similar-sounding command instead of creating the whole non-target command set, which can better save the training resources and effectively reduce command confusion errors. Specifically, we design and compare three different strategies for confusing set construction. By using our proposed method, we can relatively reduce the False Reject Rate~(FRR) by 33.7% at 0.01 False Alarm Rate~(FAR) and confusion errors by 18.28% on our collected speech command set.
【4】 TMGAN-PLC: Audio Packet Loss Concealment using Temporal Memory Generative Adversarial Network
标题：TMGAN-PLC：基于时间记忆生成对抗网络的音频丢包隐藏
链接：https://arxiv.org/abs/2207.01255
作者：Yuansheng Guan,Guochen Yu,Andong Li,Chengshi Zheng,Jie Wang机构：School of Electronics and Communication Engineering, Guangzhou University, Guangzhou, China, Key Laboratory of Noise and Vibration Research, Institute of Acoustics, Chinese Academy of, Sciences, Beijing, China, Communication University of China, Beijing, China备注：accepted by INTERSPEECH 2022摘要：分组交换网络中的实时通信已广泛应用于日常通信中，但在受限的实时条件下，它们不可避免地遭受网络延迟和数据丢失。为了解决这些问题，人们开发了音频包丢失隐藏（PLC）算法，通过重建丢失的信息来缓解语音传输故障。受传输延迟和设备内存的限制，PLC仍然难以使用相对较小的数据包缓冲区来实现高质量的语音重建。在本文中，我们提出了一种用于音频PLC的时间记忆生成对抗网络，称为TMGAN-PLC，它由一个新型嵌套的UNet生成器和时域/频域鉴别器组成。具体来说，在生成器中精心设计了嵌套的UNet和时间特征线性调制的组合，以精细调整帧内信息并建立帧间时间依赖关系。为了补充由于长时间丢失突发而导致的语音内容缺失，我们采用多级选通矢量量化器来捕获正确的内容并重建接近真实的平滑音频。在PLC挑战数据集上的大量实验表明，该方法在语音质量、可懂度和PLCMOS方面具有良好的性能。摘要：Real-time communications in packet-switched networks have become widely used in daily communication, while they inevitably suffer from network delays and data losses in constrained real-time conditions. To solve these problems, audio packet loss concealment (PLC) algorithms have been developed to mitigate voice transmission failures by reconstructing the lost information. Limited by the transmission latency and device memory, it is still intractable for PLC to accomplish high-quality voice reconstruction using a relatively small packet buffer. In this paper, we propose a temporal memory generative adversarial network for audio PLC, dubbed TMGAN-PLC, which is comprised of a novel nested-UNet generator and the time-domain/frequency-domain discriminators. Specifically, a combination of the nested-UNet and temporal feature-wise linear modulation is elaborately devised in the generator to finely adjust the intra-frame information and establish inter-frame temporal dependencies. To complement the missing speech content caused by longer loss bursts, we employ multi-stage gated vector quantizers to capture the correct content and reconstruct the near-real smooth audio. Extensive experiments on the PLC Challenge dataset demonstrate that the proposed method yields promising performance in terms of speech quality, intelligibility, and PLCMOS.
【5】 Cross-speaker Emotion Transfer Based On Prosody Compensation for End-to-End Speech Synthesis
标题：端到端语音合成中基于韵律补偿的跨说话人情感传递
链接：https://arxiv.org/abs/2207.01198
作者：Tao Li,Xinsheng Wang,Qicong Xie,Zhichao Wang,Mingqi Jiang,Lei Xie机构：Northwestern Polytechnical University, Xi’an, China, School of Software Engineering, Xi’an Jiaotong University, Xi’an, China, Mobvoi AI Lab, Suzhou, China摘要：跨说话人情感转移语音合成旨在通过从另一个（源）说话人记录的参考语音中转移情感来合成目标说话人的情感语音。在这项任务中，从参考语音中提取与说话人无关的情感嵌入起着重要作用。然而，在挤出源说话人音色信息的过程中，这种情感嵌入所传递的情感信息往往会被削弱。针对这一问题，本文提出了一种韵律补偿模块（PCM）来补偿情感信息的丢失。具体来说，PCM试图从预先训练的ASR模型的中间特征中获取与说话人无关的情感信息。为此，引入了一种具有全局上下文（GC）块的韵律补偿编码器，以从ASR模型的中间特征中获取全局情感信息。实验表明，该方法可以有效地补偿情感嵌入对情感信息丢失的影响，同时保持目标说话人的音色。与现有模型的比较表明，该方法在跨说话人情绪传递任务中具有明显的优势。摘要：Cross-speaker emotion transfer speech synthesis aims to synthesize emotional speech for a target speaker by transferring the emotion from reference speech recorded by another (source) speaker. In this task, extracting speaker-independent emotion embedding from reference speech plays an important role. However, the emotional information conveyed by such emotion embedding tends to be weakened in the process to squeeze out the source speaker's timbre information. In response to this problem, a prosody compensation module (PCM) is proposed in this paper to compensate for the emotional information loss. Specifically, the PCM tries to obtain speaker-independent emotional information from the intermediate feature of a pre-trained ASR model. To this end, a prosody compensation encoder with global context (GC) blocks is introduced to obtain global emotional information from the ASR model's intermediate feature. Experiments demonstrate that the proposed PCM can effectively compensate the emotion embedding for the emotional information loss, and meanwhile maintain the timbre of the target speaker. Comparisons with state-of-the-art models show that our proposed method presents obvious superiority on the cross-speaker emotion transfer task.
【6】 Multi-Modal Multi-Correlation Learning for Audio-Visual Speech Separation
标题：用于视听语音分离的多模多相关学习
链接：https://arxiv.org/abs/2207.01197
作者：Xiaoyu Wang,Xiangyu Kong,Xiulian Peng,Yan Lu机构：Xi’an Jiaotong University, Microsoft Research Asia, Beijing, China备注：5 pages, accepted by interspeech2022摘要：在本文中，我们提出了一个多模态多相关学习框架，针对视听语音分离任务。尽管之前的工作已经广泛地将音频和视频模式结合起来，但其中大多数只采用音频和视频特征的直接连接。为了利用这两种模式背后真正有用的信息，我们定义了两个关键相关性：（1）身份相关性（音色和面部属性之间）；（2） 语音相关性（音素和嘴唇运动之间）。这两种相关性共同构成了完整的信息，在分离目标说话人的声音方面显示出一定的优势，尤其是在一些困难的情况下，例如相同性别或类似内容。为了实现这一点，我们采用了对比学习或对抗式训练方法来最大限度地提高这两种相关性。这两种方法都很有效，而对抗性训练则通过避免对比学习的一些局限性而显示出其优势。与以前的研究相比，我们的解决方案在不增加复杂性的情况下对实验指标有了明显的改进。进一步的分析揭示了该架构的有效性及其良好的未来扩展潜力。摘要：In this paper we propose a multi-modal multi-correlation learning framework targeting at the task of audio-visual speech separation. Although previous efforts have been extensively put on combining audio and visual modalities, most of them solely adopt a straightforward concatenation of audio and visual features. To exploit the real useful information behind these two modalities, we define two key correlations which are: (1) identity correlation (between timbre and facial attributes); (2) phonetic correlation (between phoneme and lip motion). These two correlations together comprise the complete information, which shows a certain superiority in separating target speaker's voice especially in some hard cases, such as the same gender or similar content. For implementation, contrastive learning or adversarial training approach is applied to maximize these two correlations. Both of them work well, while adversarial training shows its advantage by avoiding some limitations of contrastive learning. Compared with previous research, our solution demonstrates clear improvement on experimental metrics without additional complexity. Further analysis reveals the validity of the proposed architecture and its good potential for future extension.
【7】 ARAUS: A Large-Scale Dataset and Baseline Models of Affective Responses to Augmented Urban Soundscapes
标题：ARAUS：对增强的城市声音景观的情感反应的大规模数据集和基线模型
链接：https://arxiv.org/abs/2207.01078
作者：Kenneth Ooi,Zhen-Ting Ong,Karn N. Watcharasupat,Bhan Lam,Joo Young Hong,Woon-Seng Gan备注：25 pages, 11 figures. Submitted to IEEE Transactions on Affective Computing摘要：为现有声景选择最佳掩蔽体以通过声景增强实现所需的感知变化是非常重要的，因为掩蔽体种类繁多，并且缺乏用于比较和开发声景增强模型的基准数据集。为了解决这个问题，我们公开了ARAUS（增强城市声景的情感反应）数据集，该数据集包括五重交叉验证集和独立测试集，共25440个对作为视听刺激呈现的增强声景的独特主观感知反应。每个增强的声景都是通过在城市声景记录中以固定的声景与遮罩比率数字添加“遮罩”（鸟、水、风、交通、建筑或沉默）来制作的。然后，根据ISO 12913-2:2018，通过要求参与者对每个增强音景的愉快、恼人、多事、平静、充满活力、单调、混乱、平静和适当程度进行评分来收集回答。参与者还提供了相关的人口统计信息，并完成了标准的心理问卷调查。我们对获得的回答进行探索性和统计分析，以验证内部一致性以及与文献中已知结果的一致性。最后，我们通过训练和比较城市声景观愉悦度的四个基线模型：低参数回归模型、高参数卷积神经网络和文献中的两个基于注意力的网络，证明了数据集的基准能力。摘要：Choosing optimal maskers for existing soundscapes to effect a desired perceptual change via soundscape augmentation is non-trivial due to extensive varieties of maskers and a dearth of benchmark datasets with which to compare and develop soundscape augmentation models. To address this problem, we make publicly available the ARAUS (Affective Responses to Augmented Urban Soundscapes) dataset, which comprises a five-fold cross-validation set and independent test set totaling 25,440 unique subjective perceptual responses to augmented soundscapes presented as audio-visual stimuli. Each augmented soundscape is made by digitally adding "maskers" (bird, water, wind, traffic, construction, or silence) to urban soundscape recordings at fixed soundscape-to-masker ratios. Responses were then collected by asking participants to rate how pleasant, annoying, eventful, uneventful, vibrant, monotonous, chaotic, calm, and appropriate each augmented soundscape was, in accordance with ISO 12913-2:2018. Participants also provided relevant demographic information and completed standard psychological questionnaires. We perform exploratory and statistical analysis of the responses obtained to verify internal consistency and agreement with known results in the literature. Finally, we demonstrate the benchmarking capability of the dataset by training and comparing four baseline models for urban soundscape pleasantness: a low-parameter regression model, a high-parameter convolutional neural network, and two attention-based networks in the literature.
【8】 Generating gender-ambiguous voices for privacy-preserving speech recognition
标题：为保护隐私的语音识别生成性别模糊的声音
链接：https://arxiv.org/abs/2207.01052
作者：Dimitrios Stoidis,Andrea Cavallaro机构：Centre for Intelligent Sensing, Queen Mary University of London, UK备注：5 pages, 4 figures, submitted to INTERSPEECH摘要：我们的语音编码了一个唯一可识别的模式，该模式可用于推断个人在使用语音识别服务时可能不愿透露的私人属性，如性别或身份。为了在语音识别任务中防止属性推理攻击，我们提出了一种生成对抗网络GenGAN，该网络合成隐藏说话人性别或身份的声音。所提出的网络包括一个具有U-Net架构的生成器，该生成器学习愚弄鉴别器。我们仅根据性别信息对生成器进行调节，并在信号失真和隐私保护之间使用对抗性损失。我们表明，与将性别信息视为需要保护的敏感属性的隐私保护表示学习方法相比，GenGAN改进了隐私和效用之间的权衡。摘要：Our voice encodes a uniquely identifiable pattern which can be used to infer private attributes, such as gender or identity, that an individual might wish not to reveal when using a speech recognition service. To prevent attribute inference attacks alongside speech recognition tasks, we present a generative adversarial network, GenGAN, that synthesises voices that conceal the gender or identity of a speaker. The proposed network includes a generator with a U-Net architecture that learns to fool a discriminator. We condition the generator only on gender information and use an adversarial loss between signal distortion and privacy preservation. We show that GenGAN improves the trade-off between privacy and utility compared to privacy-preserving representation learning methods that consider gender information as a sensitive attribute to protect.
【9】 Towards Error-Resilient Neural Speech Coding
标题：面向容错的神经语音编码
链接：https://arxiv.org/abs/2207.00993
作者：Huaying Xue,Xiulian Peng,Xue Jiang,Yan Lu机构：Microsoft Research Asia, Beijing, China,Communication University of China, Beijing, China备注：5 pages, Interspeech 2022(Accepted)摘要：神经音频编码最近在文献中显示出非常有前景的结果，在很大程度上优于传统的编解码器，但对其抗误码能力的关注有限。仅考虑信源编码训练的神经编解码器往往对信道噪声极为敏感，尤其是在误码率高的无线信道中。在本文中，我们研究了如何提高神经音频编解码器对实时通信过程中经常发生的数据包丢失的容错能力。我们提出了一种用于实时神经语音编码的特征域丢包隐藏算法（FD-PLC）。具体来说，我们介绍了一种基于接收到的潜在特征的自注意力模块，用于在解码器之前恢复特征域中丢失的帧。采用混合段级和帧级频域鉴别器引导网络关注丢失帧的生成质量和与相邻帧的连续性。在几种误差模式上的实验结果表明，与相应的无误差和容错基线相比，该方案具有更好的鲁棒性。我们还表明，作为后处理，特征域隐藏优于波形域对应。摘要：Neural audio coding has shown very promising results recently in the literature to largely outperform traditional codecs but limited attention has been paid on its error resilience. Neural codecs trained considering only source coding tend to be extremely sensitive to channel noises, especially in wireless channels with high error rate. In this paper, we investigate how to elevate the error resilience of neural audio codecs for packet losses that often occur during real-time communications. We propose a feature-domain packet loss concealment algorithm (FD-PLC) for real-time neural speech coding. Specifically, we introduce a self-attention-based module on the received latent features to recover lost frames in the feature domain before the decoder. A hybrid segment-level and frame-level frequency-domain discriminator is employed to guide the network to focus on both the generative quality of lost frames and the continuity with neighbouring frames. Experimental results on several error patterns show that the proposed scheme can achieve better robustness compared with the corresponding error-free and error-resilient baselines. We also show that feature-domain concealment is superior to waveform-domain counterpart as post-processing.
【10】 M-Adapter: Modality Adaptation for End-to-End Speech-to-Text Translation
标题：M-Adapter：端到端语音到文本翻译的通道适配
链接：https://arxiv.org/abs/2207.00952
作者：Jinming Zhao,Hao Yang,Ehsan Shareghi,Gholamreza Haffari机构：Department of Data Science & AI, Monash University摘要：端到端语音到文本翻译模型通常使用预训练的语音编码器和预训练的文本解码器初始化。这导致预训练和微调之间存在显著的训练差距，这主要是由于编码器的语音输出和解码器的文本输入之间存在模态差异。在这项工作中，我们旨在弥合语音和文本之间的情态差异，以提高翻译质量。我们提出了一种新的基于Transformer的M-Adapter模块，用于将语音表示适应文本。在缩小语音序列的同时，M-Adapter通过建模语音序列的全局和局部相关性来生成语音到文本翻译所需的特征。我们的实验结果表明，在Must-C En$\ rightarrow$DE数据集上，我们的模型优于强基线，BLEU分数高达1。\脚注{我们的代码可在https://github.com/mingzi151/w2v2-st.}摘要：End-to-end speech-to-text translation models are often initialized with pre-trained speech encoder and pre-trained text decoder. This leads to a significant training gap between pre-training and fine-tuning, largely due to the modality differences between speech outputs from the encoder and text inputs to the decoder. In this work, we aim to bridge the modality gap between speech and text to improve translation quality. We propose M-Adapter, a novel Transformer-based module, to adapt speech representations to text. While shrinking the speech sequence, M-Adapter produces features desired for speech-to-text translation via modelling global and local dependencies of a speech sequence. Our experimental results show that our model outperforms a strong baseline by up to 1 BLEU score on the Must-C En$\rightarrow$DE dataset.\footnote{Our code is available at https://github.com/mingzi151/w2v2-st.}
【11】 Improving Transformer-based Conversational ASR by Inter-Sentential Attention Mechanism
标题：利用感知间注意机制改进基于Transformer的对话式ASR
链接：https://arxiv.org/abs/2207.00883
作者：Kun Wei,Pengcheng Guo,Ning Jiang机构：Northwestern Polytechnical University, Xian, China, Mashang Consumer Finance Co., Ltd.备注：Accepted by Interspeech2022摘要：基于Transformer的模型已证明其在自动语音识别（ASR）任务中的有效性，甚至显示出优于传统混合框架的性能。Transformers的主要思想是通过自我注意力层捕捉话语中的长期全球语境。然而，对于会话语音等场景，这种话语级建模将忽略跨话语的语境依赖性。在本文中，我们提出在基于转换器的端到端架构中显式建模句子间信息，用于会话语音识别。具体来说，对于编码器网络，我们捕获以前语音的上下文，并通过上下文感知的剩余注意力机制将这些历史信息合并到当前输入中。对于译码器，当前话语的预测也通过条件译码器框架以历史语言信息为条件。我们在几个开源的对话语料库上证明了我们提出的方法的有效性，并且该方法不断提高了基于话语级转换的自动语音识别模型的性能。摘要：Transformer-based models have demonstrated their effectiveness in automatic speech recognition (ASR) tasks and even shown superior performance over the conventional hybrid framework. The main idea of Transformers is to capture the long-range global context within an utterance by self-attention layers. However, for scenarios like conversational speech, such utterance-level modeling will neglect contextual dependencies that span across utterances. In this paper, we propose to explicitly model the inter-sentential information in a Transformer based end-to-end architecture for conversational speech recognition. Specifically, for the encoder network, we capture the contexts of previous speech and incorporate such historic information into current input by a context-aware residual attention mechanism. For the decoder, the prediction of current utterance is also conditioned on the historic linguistic information through a conditional decoder framework. We show the effectiveness of our proposed method on several open-source dialogue corpora and the proposed method consistently improved the performance from the utterance-level Transformer-based ASR models.
【12】 Tree-constrained Pointer Generator with Graph Neural Network Encodings for Contextual Speech Recognition
标题：用于上下文语音识别的基于图神经网络编码的树约束指针生成器
链接：https://arxiv.org/abs/2207.00857
作者：Guangzhi Sun,Chao Zhang,Philip C. Woodland机构：Cambridge University Engineering Dept., Trumpington St., Cambridge, CB,PZ U.K.备注：To appear in Interspeech 2022. arXiv admin note: text overlap with arXiv:2205.09058摘要：对于许多自动语音识别（ASR）应用来说，将获得的偏差词作为上下文知识进行合并是至关重要的。本文提出了在树约束指针生成器（TCPGen）组件中使用图神经网络（GNN）编码来实现端到端上下文ASR。通过使用基于树的GNN对前缀树中的偏置词进行编码，在每个树节点上通过合并来自它的树分支上的所有词条的信息来实现端到端ASR解码中对未来词条的展望，这允许更准确地预测偏置词的生成概率。使用模拟偏差任务在Librispeech语料库上评估系统，并通过提出一种新的基于视觉的上下文ASR管道在AMI语料库上评估系统，该管道从每次会议的幻灯片中提取偏差词。结果表明，与原始TCPGen相比，采用GNN编码的TCPGen在偏置字上实现了约15%的相对功耗降低，而解码的计算成本几乎没有增加。摘要：Incorporating biasing words obtained as contextual knowledge is critical for many automatic speech recognition (ASR) applications. This paper proposes the use of graph neural network (GNN) encodings in a tree-constrained pointer generator (TCPGen) component for end-to-end contextual ASR. By encoding the biasing words in the prefix-tree with a tree-based GNN, lookahead for future wordpieces in end-to-end ASR decoding is achieved at each tree node by incorporating information about all wordpieces on the tree branches rooted from it, which allows a more accurate prediction of the generation probability of the biasing words. Systems were evaluated on the Librispeech corpus using simulated biasing tasks, and on the AMI corpus by proposing a novel visual-grounded contextual ASR pipeline that extracts biasing words from slides alongside each meeting. Results showed that TCPGen with GNN encodings achieved about a further 15% relative WER reduction on the biasing words compared to the original TCPGen, with a negligible increase in the computation cost for decoding.
【13】 Unsupervised Symbolic Music Segmentation using Ensemble Temporal Prediction Errors
标题：基于集成时间预测误差的无监督符号音乐分割
链接：https://arxiv.org/abs/2207.00760
作者：Shahaf Bassan,Yossi Adi,Jeffrey S. Rosenschein机构：Hebrew University of Jerusalem摘要：符号音乐分割是将符号旋律划分为更小的有意义组的过程，例如旋律短语。我们提出了一种无监督的符号音乐分割方法。该模型基于时间预测误差模型的集成。在训练期间，每个模型预测下一个标记，以识别音乐短语的变化。在测试时，我们执行峰值检测算法来选择候选段。最后，我们汇总参与集成的每个模型的预测，以预测最终分割。结果表明，在无监督设置下，当考虑F分数和R值时，该方法在埃森民歌数据集上达到了最先进的性能。我们还提供了一项消融研究，以更好地评估每个模型组件对最终结果的贡献。正如预期的那样，所提出的方法不如监督设置，这为在未来的研究中缩小无监督和监督方法之间的差距留下了改进的空间。摘要：Symbolic music segmentation is the process of dividing symbolic melodies into smaller meaningful groups, such as melodic phrases. We proposed an unsupervised method for segmenting symbolic music. The proposed model is based on an ensemble of temporal prediction error models. During training, each model predicts the next token to identify musical phrase changes. While at test time, we perform a peak detection algorithm to select segment candidates. Finally, we aggregate the predictions of each of the models participating in the ensemble to predict the final segmentation. Results suggest the proposed method reaches state-of-the-art performance on the Essen Folksong dataset under the unsupervised setting when considering F-Score and R-value. We additionally provide an ablation study to better assess the contribution of each of the model components to the final results. As expected, the proposed method is inferior to the supervised setting, which leaves room for improvement in future research considering closing the gap between unsupervised and supervised methods.
【14】 Learning Noise-independent Speech Representation for High-quality Voice Conversion for Noisy Target Speakers
标题：学习噪声无关的语音表示法实现含噪目标说话人的高质量语音转换
链接：https://arxiv.org/abs/2207.00756
作者：Liumeng Xue,Shan Yang,Na Hu,Dan Su,Lei Xie机构：Northwestern Polytechnical University, Xi’an, China, Tencent AI Lab, China备注：Accepted by INTERSPEECH 2022摘要：由于在模型训练中使用受污染的语音会明显降低转换性能，因此为有噪声的目标说话人（如提供噪声样本或互联网数据的用户）构建语音转换系统是一项具有挑战性的任务。在本文中，我们利用我们最近提出的Glow-WaveGAN的进展，提出了一种与噪声无关的语音表示学习方法，用于噪声目标扬声器的高质量语音转换。具体来说，我们学习了一个潜在特征空间，在该空间中，我们确保由转换模型建模的目标分布完全来自波形发生器的建模分布。在此前提下，我们进一步设法使潜在特征具有噪声不变性。具体来说，我们介绍了一种噪声可控的WaveGAN，它通过编码器直接从波形中学习与噪声无关的声学表示，并通过解码器中的胶片模块在隐藏空间中进行噪声控制。对于转换模型，重要的是，我们使用基于流的模型从音素后验图中学习与噪声无关但与说话人相关的潜在特征的分布。实验结果表明，该模型在噪声目标说话人的语音转换中实现了较高的语音质量和说话人相似性。摘要：Building a voice conversion system for noisy target speakers, such as users providing noisy samples or Internet found data, is a challenging task since the use of contaminated speech in model training will apparently degrade the conversion performance. In this paper, we leverage the advances of our recently proposed Glow-WaveGAN and propose a noise-independent speech representation learning approach for high-quality voice conversion for noisy target speakers. Specifically, we learn a latent feature space where we ensure that the target distribution modeled by the conversion model is exactly from the modeled distribution of the waveform generator. With this premise, we further manage to make the latent feature to be noise-invariant. Specifically, we introduce a noise-controllable WaveGAN, which directly learns the noise-independent acoustic representation from waveform by the encoder and conducts noise control in the hidden space through a FiLM module in the decoder. As for the conversion model, importantly, we use a flow-based model to learn the distribution of noise-independent but speaker-related latent features from phoneme posteriorgrams. Experimental results demonstrate that the proposed model achieves high speech quality and speaker similarity in the voice conversion for noisy target speakers.
【15】 Building African Voices
标题：打造非洲声音
链接：https://arxiv.org/abs/2207.00688
作者：Perez Ogayo,Graham Neubig,Alan W Black机构：†Language Technologies Institute, Carnegie Mellon University, ‡Inspired Cognition, Pittsburgh, PA, USA摘要：现代语音合成技术可以在提供足够高质量数据和计算资源的情况下产生自然发音的语音。然而，许多语言并没有现成的此类数据。本文的重点是低资源非洲语言的语音合成，从语料库创建到共享和部署文本到语音（TTS）系统。我们首先创建一组通用说明，以最少的技术资源和主题专业知识构建语音合成系统。接下来，我们创建新的数据集，并通过参与式方法从“发现”数据（现有记录）中整理数据集，同时考虑可访问性、质量和广度。我们证明，我们可以开发合成器，用25分钟的创建语音生成可理解的语音，即使在不理想的环境中录制。最后，我们发布了12种非洲语言的语音数据、代码和训练语音，以支持研究人员和开发人员。摘要：Modern speech synthesis techniques can produce natural-sounding speech given sufficient high-quality data and compute resources. However, such data is not readily available for many languages. This paper focuses on speech synthesis for low-resourced African languages, from corpus creation to sharing and deploying the Text-to-Speech (TTS) systems. We first create a set of general-purpose instructions on building speech synthesis systems with minimum technological resources and subject-matter expertise. Next, we create new datasets and curate datasets from "found" data (existing recordings) through a participatory approach while considering accessibility, quality, and breadth. We demonstrate that we can develop synthesizers that generate intelligible speech with 25 minutes of created speech, even when recorded in suboptimal environments. Finally, we release the speech data, code, and trained voices for 12 African languages to support researchers and developers.
【16】 Semi-blind source separation using convolutive transfer function for nonlinear acoustic echo cancellation
标题：基于卷积传递函数的非线性回波对消半盲源分离
链接：https://arxiv.org/abs/2207.01556
作者：Guoliang Cheng,Lele Liao,Kai Chen,Yuxiang Hu,Changbao Zhu,Jing Lu机构：Key Laboratory of Modern Acoustics, Institute of Acoustics, Nanjing University, Nanjing, NJU-Horizon Intelligent Audio Lab, Horizon Robotics, Nanjing , China., Running title: SBSS using CTF for NAEC摘要：最近提出的用于非线性声回波抵消（NAEC）的半盲源分离（SBSS）方法在衰减非线性声回波方面优于自适应NAEC。然而，乘法传递函数（MTF）近似使其不适合实时应用，尤其是在高混响环境中，并且自然梯度使其难以在快速收敛速度和稳定性之间很好地平衡。在本文中，我们提出了两种更有效的基于辅助函数的独立向量分析（AuxIVA）和独立低秩矩阵分析（ILRMA）的SBSS方法。使用卷积传递函数（CTF）近似代替MTF，因此可以在短延迟下模拟长冲击响应。AuxIVA和ILRMA中使用的优化方案根据NAEC的约束分解矩阵仔细正则化。实验结果验证了所提出的方法具有显著更好的回声消除性能。摘要：The recently proposed semi-blind source separation (SBSS) method for nonlinear acoustic echo cancellation (NAEC) outperforms adaptive NAEC in attenuating the nonlinear acoustic echo. However, the multiplicative transfer function (MTF) approximation makes it unsuitable for real-time applications especially in highly reverberant environments, and the natural gradient makes it hard to balance well between fast convergence speed and stability. In this paper, we propose two more effective SBSS methods based on auxiliary-function-based independent vector analysis (AuxIVA) and independent low-rank matrix analysis (ILRMA). The convolutive transfer function (CTF) approximation is used instead of MTF so that a long impulse response can be modeled with a short latency. The optimization schemes used in AuxIVA and ILRMA are carefully regularized according to the constrained demixing matrix of NAEC. Experimental results validate significantly better echo cancellation performance of the proposed methods.
【17】 j-Wave: An open-source differentiable wave simulator
标题：J-Wave：一个开源的可微分波模拟器
链接：https://arxiv.org/abs/2207.01499
作者：Antonio Stanziola,Simon R. Arridge,Ben T. Cox,Bradley E. Treeby机构：Dept. of Medical Physics and Biomedical Engineering, University College London, London WC,E ,BT, UK, Department of Computer Science, University College of London摘要：我们提出了一种开源的可微声学模拟器j-Wave，它可以解决时变和时谐声学问题。它支持自动微分，这是一种程序转换技术，有许多应用，特别是在机器学习和科学计算中。j-Wave由易于定制和重用的模块化组件组成。同时，它与一些最流行的机器学习库兼容，例如JAX和TensorFlow。根据广泛使用的k波工具箱和一系列声学模拟软件，评估了已知配置的模拟结果的准确性。j波可从https://github.com/ucl-bug/jwave.摘要：We present an open-source differentiable acoustic simulator, j-Wave, which can solve time-varying and time-harmonic acoustic problems. It supports automatic differentiation, which is a program transformation technique that has many applications, especially in machine learning and scientific computing. j-Wave is composed of modular components that can be easily customized and reused. At the same time, it is compatible with some of the most popular machine learning libraries, such as JAX and TensorFlow. The accuracy of the simulation results for known configurations is evaluated against the widely used k-Wave toolbox and a cohort of acoustic simulation software. j-Wave is available from https://github.com/ucl-bug/jwave.
【18】 Transfer functions of FXLMS-based Multi-channel Multi-tone Active Noise Equalizers
标题：基于FXLMS的多通道多音有源噪声均衡器的传递函数
链接：https://arxiv.org/abs/2207.01102
作者：Miguel Ferrer,María de Diego,Gema Piñero,Amin Hassani,Marc Moonen,Alberto González机构：Institute of Telecommunications and Multimedia Applications (iTEAM), Universitat Politécnica de Valéncia (UPV), Valencia, Spain, STADIUS center for dynamical systems, signal processing and data analytics, KU Leuven, Leuven, Belgium摘要：即使在不同的空间位置，多通道多音调有源噪声均衡器也可以实现不同的用户选择噪声频谱分布。它们可以在每个噪声频率分量和每个控制点应用不同的均衡因子。理论上，在噪声信号具有能量的频率处的传递函数值由均衡器配置确定。在这项工作中，我们展示了如何以双重目标计算这些传递函数：验证在感兴趣的频率处获得了均衡器设置施加的值，并表征了这些传递函数在其余频谱中的行为，以及获得预测算法收敛行为的线索。由于这些传递函数提供的信息是对收敛性进行繁琐统计分析的一种实用替代方法，其结果通常没有实际用途。摘要：Multi-channel Multi-tone Active Noise Equalizers can achieve different user-selected noise spectrum profiles even at different space positions. They can apply a different equalization factor at each noise frequency component and each control point. Theoretically, the value of the transfer function at the frequencies where the noise signal has energy is determined by the equalizer configuration. In this work, we show how to calculate these transfer functions with a double aim: to verify that at the frequencies of interest the values imposed by the equalizer settings are obtained, and to characterize the behavior of these transfer functions in the rest of the spectrum, as well as to get clues to predict the convergence behaviour of the algorithm. The information provided thanks to these transfer functions serves as a practical alternative to the cumbersome statistical analysis of convergence, whose results are often of no practical use.
【19】 Leveraging Acoustic Contextual Representation by Audio-textual Cross-modal Learning for Conversational ASR
标题：利用音频-文本跨模式学习实现会话ASR的声学语境表征
链接：https://arxiv.org/abs/2207.01039
作者：Kun Wei,Yike Zhang,Sining Sun,Lei Xie,Long Ma机构：Northwestern Polytechnical University, Xian, China, Cloud Xiaowei, Tencent, Beijing, China备注：Accepted by Interspeech2022摘要：利用上下文信息可以提高会话自动语音识别（ASR）的性能。以往的研究通常采用历史话语的公认假设作为前置语境，由于不可避免的历史认知错误，这可能会使当前公认的假设产生偏差。为了避免这个问题，我们提出了一种音频-文本跨模态表示提取器，直接从前面的语音中学习上下文表示。具体来说，它由两个模态相关编码器和一个跨模态编码器组成，前者从语音和相应文本中提取高级潜在特征，后者旨在学习语音和文本之间的相关性。我们随机屏蔽每个模态的一些输入令牌和输入序列。然后在跨模式编码器上执行令牌缺失或模式缺失预测，并具有模式级连接时序分类丢失。因此，该模型不仅捕获了特定模式中的双向上下文依存关系，还捕获了不同模式之间的关系。然后，在会话自动语音识别系统的训练过程中，提取器将被冻结以提取前一语音的文本表示，而这种表示被用作通过注意机制反馈给自动语音识别解码器的上下文。在多个汉语会话语料库上验证了该方法的有效性，在MagicData数据集上实现了最高的字符错误率降低16%。摘要：Leveraging context information is an intuitive idea to improve performance on conversational automatic speech recognition(ASR). Previous works usually adopt recognized hypotheses of historical utterances as preceding context, which may bias the current recognized hypothesis due to the inevitable historicalrecognition errors. To avoid this problem, we propose an audio-textual cross-modal representation extractor to learn contextual representations directly from preceding speech. Specifically, it consists of two modal-related encoders, extracting high-level latent features from speech and the corresponding text, and a cross-modal encoder, which aims to learn the correlation between speech and text. We randomly mask some input tokens and input sequences of each modality. Then a token-missing or modal-missing prediction with a modal-level CTC loss on the cross-modal encoder is performed. Thus, the model captures not only the bi-directional context dependencies in a specific modality but also relationships between different modalities. Then, during the training of the conversational ASR system, the extractor will be frozen to extract the textual representation of preceding speech, while such representation is used as context fed to the ASR decoder through attention mechanism. The effectiveness of the proposed approach is validated on several Mandarin conversation corpora and the highest character error rate (CER) reduction up to 16% is achieved on the MagicData dataset.
【20】 A Graph Isomorphism Network with Weighted Multiple Aggregators for Speech Emotion Recognition
标题：一种用于语音情感识别的加权多聚集器图同构网络
链接：https://arxiv.org/abs/2207.00940
作者：Ying Hu,Yuwu Tang,Hao Huang,Liang He机构：Key Laboratory of signal detection and processing in Xinjiang, China, School of Information Science and Engineering, Xinjiang University, Urumqi, China, Department of Electronic Engineering, Tsinghua University, China备注：Accepted by Interspeech 2022摘要：语音情感识别是人机交互的重要组成部分。在本文中，我们提出了一种基于带加权多聚合器的图同构网络（WMA-GIN）的SER网络，该网络可以有效地处理在GIN结构中将相邻节点的特征聚合在一起时的信息混淆问题。此外，采用全邻接层来缓解所有图神经网络结构（包括GIN）中存在的过度挤压问题。此外，采用多阶段注意机制和多损失训练策略，避免在堆叠的WMA-GIN层中丢失有用的情感信息。我们在流行的IEMOCAP数据集上评估了我们提出的WMA-GIN的性能。实验结果表明，WMA-GIN的加权精度（WA）为72.48%，未加权精度（UA）为67.72%，优于其他基于GNN的方法，与一些先进的非基于图的方法相当。摘要：Speech emotion recognition (SER) is an essential part of human-computer interaction. In this paper, we propose an SER network based on a Graph Isomorphism Network with Weighted Multiple Aggregators (WMA-GIN), which can effectively handle the problem of information confusion when neighbour nodes' features are aggregated together in GIN structure. Moreover, a Full-Adjacent (FA) layer is adopted for alleviating the over-squashing problem, which is existed in all Graph Neural Network (GNN) structures, including GIN. Furthermore, a multi-phase attention mechanism and multi-loss training strategy are employed to avoid missing the useful emotional information in the stacked WMA-GIN layers. We evaluated the performance of our proposed WMA-GIN on the popular IEMOCAP dataset. The experimental results show that WMA-GIN outperforms other GNN-based methods and is comparable to some advanced non-graph-based methods by achieving 72.48% of weighted accuracy (WA) and 67.72% of unweighted accuracy (UA).
【21】 Speaker Diarization and Identification from Single-Channel Classroom Audio Recording Using Virtual Microphones
标题：基于虚拟麦克风的单声道课堂录音说话人二值化与识别
链接：https://arxiv.org/abs/2207.00660
备注：Total of 170 pages, 59 figures. This work is part of the partial fulfillment of the requirements for the degree of PhD in engineering at the University of New Mexico摘要：在嘈杂的录音中，特别是在协作学习环境中，说话人识别可能极具挑战性。有必要将小组讨论的个别学生与同时讨论的其他学生区分开来。为了解决这个问题，我们假设每个学生组使用一个麦克风，而不需要访问以前的大型数据集进行训练。本文提出了一种以物理麦克风为中心，利用与虚拟麦克风阵列相关联的互相关模式进行说话人识别的方法。通过使用从视频记录中观察到的近似扬声器几何形状来模拟虚拟麦克风。模式是基于每个虚拟麦克风的房间冲击响应估计构建的。然后使用相关模式识别说话人。通过课堂音频验证了该方法，结果表明其性能大大优于谷歌云和亚马逊AWS提供的日记服务。摘要：Speaker identification in noisy audio recordings, specifically those from collaborative learning environments, can be extremely challenging. There is a need to identify individual students talking in small groups from other students talking at the same time. To solve the problem, we assume the use of a single microphone per student group without any access to previous large datasets for training. This dissertation proposes a method of speaker identification using cross-correlation patterns associated to an array of virtual microphones, centered around the physical microphone. The virtual microphones are simulated by using approximate speaker geometry observed from a video recording. The patterns are constructed based on estimates of the room impulse responses for each virtual microphone. The correlation patterns are then used to identify the speakers. The proposed method is validated with classroom audios and shown to substantially outperform diarization services provided by Google Cloud and Amazon AWS.
【1】 Semi-blind source separation using convolutive transfer function for nonlinear acoustic echo cancellation
标题：基于卷积传递函数的非线性回波对消半盲源分离
链接：https://arxiv.org/abs/2207.01556
* 与cs.SD语音【16】为同一篇
作者：Guoliang Cheng,Lele Liao,Kai Chen,Yuxiang Hu,Changbao Zhu,Jing Lu机构：Key Laboratory of Modern Acoustics, Institute of Acoustics, Nanjing University, Nanjing, NJU-Horizon Intelligent Audio Lab, Horizon Robotics, Nanjing , China., Running title: SBSS using CTF for NAEC摘要：最近提出的用于非线性声回波抵消（NAEC）的半盲源分离（SBSS）方法在衰减非线性声回波方面优于自适应NAEC。然而，乘法传递函数（MTF）近似使其不适合实时应用，尤其是在高混响环境中，并且自然梯度使其难以在快速收敛速度和稳定性之间很好地平衡。在本文中，我们提出了两种更有效的基于辅助函数的独立向量分析（AuxIVA）和独立低秩矩阵分析（ILRMA）的SBSS方法。使用卷积传递函数（CTF）近似代替MTF，因此可以在短延迟下模拟长冲击响应。AuxIVA和ILRMA中使用的优化方案根据NAEC的约束分解矩阵仔细正则化。实验结果验证了所提出的方法具有显著更好的回声消除性能。摘要：The recently proposed semi-blind source separation (SBSS) method for nonlinear acoustic echo cancellation (NAEC) outperforms adaptive NAEC in attenuating the nonlinear acoustic echo. However, the multiplicative transfer function (MTF) approximation makes it unsuitable for real-time applications especially in highly reverberant environments, and the natural gradient makes it hard to balance well between fast convergence speed and stability. In this paper, we propose two more effective SBSS methods based on auxiliary-function-based independent vector analysis (AuxIVA) and independent low-rank matrix analysis (ILRMA). The convolutive transfer function (CTF) approximation is used instead of MTF so that a long impulse response can be modeled with a short latency. The optimization schemes used in AuxIVA and ILRMA are carefully regularized according to the constrained demixing matrix of NAEC. Experimental results validate significantly better echo cancellation performance of the proposed methods.
【2】 Unify and Conquer: How Phonetic Feature Representation Affects Polyglot Text-To-Speech (TTS)
标题：统一与征服：语音特征表示如何影响多语种文本到语音(TTS)
链接：https://arxiv.org/abs/2207.01547
作者：Ariadna Sanchez,Alessio Falai,Ziyao Zhang,Orazio Angelini,Kayoko Yanagisawa摘要：多语言神经文语转换（NTTS）系统的一个基本设计决策是如何在模型内表示输入语言特征。纵观文献中的各种方法，出现了两种主要范式，即统一表示法和单独表示法。前者使用跨语言的一组共享语音标记，而后者为每种语言使用唯一的语音标记。在本文中，我们对使用这两种表示法训练的多语言NTTS系统模型进行了全面的比较研究。我们的结果表明，统一方法在自然度和重音方面始终实现了更好的跨语言合成。单独表示往往比统一表示多一个数量级的令牌，这可能会影响模型容量。因此，我们进行了一项消融研究，以了解表征类型与令牌嵌入大小的相互作用。我们发现，这两种范式之间的差异仅出现在某个阈值嵌入大小以上。这项研究提供了强有力的证据表明，在构建多语言NTTS系统时，统一表示应该是首选的范例。摘要：An essential design decision for multilingual Neural Text-To-Speech (NTTS) systems is how to represent input linguistic features within the model. Looking at the wide variety of approaches in the literature, two main paradigms emerge, unified and separate representations. The former uses a shared set of phonetic tokens across languages, whereas the latter uses unique phonetic tokens for each language. In this paper, we conduct a comprehensive study comparing multilingual NTTS systems models trained with both representations. Our results reveal that the unified approach consistently achieves better cross-lingual synthesis with respect to both naturalness and accent. Separate representations tend to have an order of magnitude more tokens than unified ones, which may affect model capacity. For this reason, we carry out an ablation study to understand the interaction of the representation type with the size of the token embedding. We find that the difference between the two paradigms only emerges above a certain threshold embedding size. This study provides strong evidence that unified representations should be the preferred paradigm when building multilingual NTTS systems.
【3】 Mix and Match: An Empirical Study on Training Corpus Composition for Polyglot Text-To-Speech (TTS)
标题：混合匹配：多语种文语转换训练语料库合成的实证研究
链接：https://arxiv.org/abs/2207.01507
作者：Ziyao Zhang,Alessio Falai,Ariadna Sanchez,Orazio Angelini,Kayoko Yanagisawa备注：Accepted to be published in the Proceedings of InterSpeech 2022摘要：仅使用单语语料库训练多语言神经文语转换（NTTS）模型已成为构建基于语音克隆的多语言NTTS系统的流行方法。为了训练这些模型，必须了解训练语料库的组成如何影响多语言语音合成的质量。在这种情况下，经常会听到这样的问题：“考虑到两种语言的相近性，包括更多的西班牙语数据是否有助于我的意大利语合成？”。不幸的是，我们发现现有文献在这方面缺乏完整性。在目前的工作中，我们进行了广泛的消融研究，旨在了解训练语料库的各种因素，如语言家庭归属、性别组成和说话人数量，是如何影响多语言合成的质量的。我们的发现包括观察到，在大多数场景中，女性说话人数据是首选的，并且在训练语料库中有更多来自目标语言变体的说话人并不总是有益的。本文的研究结果为数据获取和语料库构建过程提供了信息。摘要：Training multilingual Neural Text-To-Speech (NTTS) models using only monolingual corpora has emerged as a popular way for building voice cloning based Polyglot NTTS systems. In order to train these models, it is essential to understand how the composition of the training corpora affects the quality of multilingual speech synthesis. In this context, it is common to hear questions such as "Would including more Spanish data help my Italian synthesis, given the closeness of both languages?". Unfortunately, we found existing literature on the topic lacking in completeness in this regard. In the present work, we conduct an extensive ablation study aimed at understanding how various factors of the training corpora, such as language family affiliation, gender composition, and the number of speakers, contribute to the quality of Polyglot synthesis. Our findings include the observation that female speaker data are preferred in most scenarios, and that it is not always beneficial to have more speakers from the target language variant in the training corpus. The findings herein are informative for the process of data procurement and corpora building.
【4】 GlowVC: Mel-spectrogram space disentangling model for language-independent text-free voice conversion
标题：GlowVC：语言无关无文本语音转换的MEL-语谱图空间解缠模型
链接：https://arxiv.org/abs/2207.01454
作者：Magdalena Proszewska,Grzegorz Beringer,Daniel Sáez-Trigueros,Thomas Merritt,Abdelhamid Ezzerg,Roberto Barra-Chicote机构：Jagiellonian University, Poland, Alexa AI备注：Accepted at Interspeech 2022摘要：在本文中，我们提出了GlowVC：一个基于多语种多说话人流的模型，用于与语言无关的文本自由语音转换。我们构建了Glow TTS，它提供了一种架构，可以在训练期间使用语言特征，而无需将其用于VC推理。我们考虑了模型的两个版本：GlowVC条件和GlowVC显式。GlowVC条件建模具有说话人条件流的mel谱图分布，并将mel谱图空间分解为与内容和基音相关的维度，而GlowVC显式建模具有非条件流的显式分布，并将所述空间分解为与内容、基音和说话人相关的维度。我们从可懂度、说话人相似性和自然度等方面评估了我们的模型，以实现可见和不可见语言的语内和跨语言转换。GlowVC模型在可懂度方面大大优于AutoVC基线，同时在语言内VC中实现了与说话人相同的高相似度，而在跨语言环境中则稍差。此外，我们还证明了GlowVC显式在自然度方面优于GlowVC条件和AutoVC。摘要：In this paper, we propose GlowVC: a multilingual multi-speaker flow-based model for language-independent text-free voice conversion. We build on Glow-TTS, which provides an architecture that enables use of linguistic features during training without the necessity of using them for VC inference. We consider two versions of our model: GlowVC-conditional and GlowVC-explicit. GlowVC-conditional models the distribution of mel-spectrograms with speaker-conditioned flow and disentangles the mel-spectrogram space into content- and pitch-relevant dimensions, while GlowVC-explicit models the explicit distribution with unconditioned flow and disentangles said space into content-, pitch- and speaker-relevant dimensions. We evaluate our models in terms of intelligibility, speaker similarity and naturalness for intra- and cross-lingual conversion in seen and unseen languages. GlowVC models greatly outperform AutoVC baseline in terms of intelligibility, while achieving just as high speaker similarity in intra-lingual VC, and slightly worse in the cross-lingual setting. Moreover, we demonstrate that GlowVC-explicit surpasses both GlowVC-conditional and AutoVC in terms of naturalness.
【5】 Transfer functions of FXLMS-based Multi-channel Multi-tone Active Noise Equalizers
标题：基于FXLMS的多通道多音有源噪声均衡器的传递函数
链接：https://arxiv.org/abs/2207.01102
* 与cs.SD语音【18】为同一篇
作者：Miguel Ferrer,María de Diego,Gema Piñero,Amin Hassani,Marc Moonen,Alberto González机构：Institute of Telecommunications and Multimedia Applications (iTEAM), Universitat Politécnica de Valéncia (UPV), Valencia, Spain, STADIUS center for dynamical systems, signal processing and data analytics, KU Leuven, Leuven, Belgium摘要：即使在不同的空间位置，多通道多音调有源噪声均衡器也可以实现不同的用户选择噪声频谱分布。它们可以在每个噪声频率分量和每个控制点应用不同的均衡因子。理论上，在噪声信号具有能量的频率处的传递函数值由均衡器配置确定。在这项工作中，我们展示了如何以双重目标计算这些传递函数：验证在感兴趣的频率处获得了均衡器设置施加的值，并表征了这些传递函数在其余频谱中的行为，以及获得预测算法收敛行为的线索。由于这些传递函数提供的信息是对收敛性进行繁琐统计分析的一种实用替代方法，其结果通常没有实际用途。摘要：Multi-channel Multi-tone Active Noise Equalizers can achieve different user-selected noise spectrum profiles even at different space positions. They can apply a different equalization factor at each noise frequency component and each control point. Theoretically, the value of the transfer function at the frequencies where the noise signal has energy is determined by the equalizer configuration. In this work, we show how to calculate these transfer functions with a double aim: to verify that at the frequencies of interest the values imposed by the equalizer settings are obtained, and to characterize the behavior of these transfer functions in the rest of the spectrum, as well as to get clues to predict the convergence behaviour of the algorithm. The information provided thanks to these transfer functions serves as a practical alternative to the cumbersome statistical analysis of convergence, whose results are often of no practical use.
【6】 DailyTalk: Spoken Dialogue Dataset for Conversational Text-to-Speech
标题：DailyTalk：对话式文本到语音的口语对话数据集
链接：https://arxiv.org/abs/2207.01063
作者：Keon Lee,Kyumin Park,Daeyoung Kim机构：School of Computing, KAIST, Korea, KRAFTON Inc., Korea备注：10 pages, 3 figures, 5 tables. Submitted to NeurIPS 2022 Datasets and Benchmarks摘要：当前大多数TTS数据集是单个话语的集合，在风格和元数据方面都很少包含会话方面。在本文中，我们介绍了DailyTalk，一个为文本到语音设计的高质量对话语音数据集。我们从开放域对话数据集DailyDialog中采样、修改和记录了2541个对话，这些对话足够长，可以表示每个对话的上下文。在数据构建步骤中，我们维护了DailyDialog中最初注释的属性分布，以支持DailyTalk中的多样化对话。在我们的数据集之上，我们扩展了以前的工作作为基线，其中非自回归TTS以对话框中的历史信息为条件。我们收集元数据，以便TTS模型可以学习历史对话信息，这是生成上下文感知语音的关键。从基线实验结果来看，我们表明DailyTalk可以用于训练神经文本到语音模型，并且我们的基线可以表示上下文信息。DailyTalk数据集和基线代码可通过CC-BY-SA 4.0许可证免费用于学术用途。摘要：The majority of current TTS datasets, which are collections of individual utterances, contain few conversational aspects in terms of both style and metadata. In this paper, we introduce DailyTalk, a high-quality conversational speech dataset designed for Text-to-Speech. We sampled, modified, and recorded 2,541 dialogues from the open-domain dialogue dataset DailyDialog which are adequately long to represent context of each dialogue. During the data construction step, we maintained attributes distribution originally annotated in DailyDialog to support diverse dialogue in DailyTalk. On top of our dataset, we extend prior work as our baseline, where a non-autoregressive TTS is conditioned on historical information in a dialog. We gather metadata so that a TTS model can learn historical dialog information, the key to generating context-aware speech. From the baseline experiment results, we show that DailyTalk can be used to train neural text-to-speech models, and our baseline can represent contextual information. The DailyTalk dataset and baseline code are freely available for academic use with CC-BY-SA 4.0 license.
【7】 Leveraging Acoustic Contextual Representation by Audio-textual Cross-modal Learning for Conversational ASR
标题：利用音频-文本跨模式学习实现会话ASR的声学语境表征
链接：https://arxiv.org/abs/2207.01039
* 与cs.SD语音【19】为同一篇
作者：Kun Wei,Yike Zhang,Sining Sun,Lei Xie,Long Ma机构：Northwestern Polytechnical University, Xian, China, Cloud Xiaowei, Tencent, Beijing, China备注：Accepted by Interspeech2022摘要：利用上下文信息可以提高会话自动语音识别（ASR）的性能。以往的研究通常采用历史话语的公认假设作为前置语境，由于不可避免的历史认知错误，这可能会使当前公认的假设产生偏差。为了避免这个问题，我们提出了一种音频-文本跨模态表示提取器，直接从前面的语音中学习上下文表示。具体来说，它由两个模态相关编码器和一个跨模态编码器组成，前者从语音和相应文本中提取高级潜在特征，后者旨在学习语音和文本之间的相关性。我们随机屏蔽每个模态的一些输入令牌和输入序列。然后在跨模式编码器上执行令牌缺失或模式缺失预测，并具有模式级连接时序分类丢失。因此，该模型不仅捕获了特定模式中的双向上下文依存关系，还捕获了不同模式之间的关系。然后，在会话自动语音识别系统的训练过程中，提取器将被冻结以提取前一语音的文本表示，而这种表示被用作通过注意机制反馈给自动语音识别解码器的上下文。在多个汉语会话语料库上验证了该方法的有效性，在MagicData数据集上实现了最高的字符错误率降低16%。摘要：Leveraging context information is an intuitive idea to improve performance on conversational automatic speech recognition(ASR). Previous works usually adopt recognized hypotheses of historical utterances as preceding context, which may bias the current recognized hypothesis due to the inevitable historicalrecognition errors. To avoid this problem, we propose an audio-textual cross-modal representation extractor to learn contextual representations directly from preceding speech. Specifically, it consists of two modal-related encoders, extracting high-level latent features from speech and the corresponding text, and a cross-modal encoder, which aims to learn the correlation between speech and text. We randomly mask some input tokens and input sequences of each modality. Then a token-missing or modal-missing prediction with a modal-level CTC loss on the cross-modal encoder is performed. Thus, the model captures not only the bi-directional context dependencies in a specific modality but also relationships between different modalities. Then, during the training of the conversational ASR system, the extractor will be frozen to extract the textual representation of preceding speech, while such representation is used as context fed to the ASR decoder through attention mechanism. The effectiveness of the proposed approach is validated on several Mandarin conversation corpora and the highest character error rate (CER) reduction up to 16% is achieved on the MagicData dataset.
【8】 A Graph Isomorphism Network with Weighted Multiple Aggregators for Speech Emotion Recognition
标题：一种用于语音情感识别的加权多聚集器图同构网络
链接：https://arxiv.org/abs/2207.00940
* 与cs.SD语音【20】为同一篇
作者：Ying Hu,Yuwu Tang,Hao Huang,Liang He机构：Key Laboratory of signal detection and processing in Xinjiang, China, School of Information Science and Engineering, Xinjiang University, Urumqi, China, Department of Electronic Engineering, Tsinghua University, China备注：Accepted by Interspeech 2022摘要：语音情感识别是人机交互的重要组成部分。在本文中，我们提出了一种基于带加权多聚合器的图同构网络（WMA-GIN）的SER网络，该网络可以有效地处理在GIN结构中将相邻节点的特征聚合在一起时的信息混淆问题。此外，采用全邻接层来缓解所有图神经网络结构（包括GIN）中存在的过度挤压问题。此外，采用多阶段注意机制和多损失训练策略，避免在堆叠的WMA-GIN层中丢失有用的情感信息。我们在流行的IEMOCAP数据集上评估了我们提出的WMA-GIN的性能。实验结果表明，WMA-GIN的加权精度（WA）为72.48%，未加权精度（UA）为67.72%，优于其他基于GNN的方法，与一些先进的非基于图的方法相当。摘要：Speech emotion recognition (SER) is an essential part of human-computer interaction. In this paper, we propose an SER network based on a Graph Isomorphism Network with Weighted Multiple Aggregators (WMA-GIN), which can effectively handle the problem of information confusion when neighbour nodes' features are aggregated together in GIN structure. Moreover, a Full-Adjacent (FA) layer is adopted for alleviating the over-squashing problem, which is existed in all Graph Neural Network (GNN) structures, including GIN. Furthermore, a multi-phase attention mechanism and multi-loss training strategy are employed to avoid missing the useful emotional information in the stacked WMA-GIN layers. We evaluated the performance of our proposed WMA-GIN on the popular IEMOCAP dataset. The experimental results show that WMA-GIN outperforms other GNN-based methods and is comparable to some advanced non-graph-based methods by achieving 72.48% of weighted accuracy (WA) and 67.72% of unweighted accuracy (UA).
【9】 Computer-assisted Pronunciation Training -- Speech synthesis is almost all you need
标题：计算机辅助发音训练--语音合成几乎就是你所需要的
链接：https://arxiv.org/abs/2207.00774
作者：Daniel Korzekwa,Jaime Lorenzo-Trueba,Thomas Drugman,Bozena Kostek机构：Amazon Speech Research, Gdansk University of Technology, Poland, ARTICLE HISTORY, Go to the article published in Speech Communication Journal备注：Published in Speech Communication Journal摘要：长期以来，研究界一直在研究非母语语音中的计算机辅助发音训练方法。研究人员专注于研究各种模型架构，例如贝叶斯网络和深度学习方法，以及分析语音信号的不同表示。尽管近年来取得了重大进展，但现有的CAPT方法无法高精度地检测发音错误（在40%-80%的召回率下仅60%的准确率）。关键问题之一是发音错误检测模型可靠训练所需的发音错误语音的可用性低。如果我们有一个生成模型，可以模拟非母语语音并生成任意数量的训练数据，那么检测发音错误的任务就会容易得多。我们提出了三种基于音素到音素（P2P）、文本到语音（T2S）和语音到语音（S2S）转换的创新技术，以生成正确发音和发音错误的合成语音。我们表明，这些技术不仅提高了三种机器学习模型检测发音错误的准确性，而且有助于建立该领域的新技术。早期的研究使用了简单的语音生成技术，如P2P转换，但只是作为一种额外的机制来提高发音错误检测的准确性。另一方面，我们认为语音生成是检测发音错误的一流方法。在检测发音和词汇重音错误的任务中，评估了这些技术的有效性。评估中使用了德语、意大利语和波兰语非母语英语语音语料库。与最先进的方法相比，最佳提议的S2S技术将AUC度量中的发音错误检测精度提高了41%，从0.528提高到0.749。摘要：The research community has long studied computer-assisted pronunciation training (CAPT) methods in non-native speech. Researchers focused on studying various model architectures, such as Bayesian networks and deep learning methods, as well as on the analysis of different representations of the speech signal. Despite significant progress in recent years, existing CAPT methods are not able to detect pronunciation errors with high accuracy (only 60\% precision at 40\%-80\% recall). One of the key problems is the low availability of mispronounced speech that is needed for the reliable training of pronunciation error detection models. If we had a generative model that could mimic non-native speech and produce any amount of training data, then the task of detecting pronunciation errors would be much easier. We present three innovative techniques based on phoneme-to-phoneme (P2P), text-to-speech (T2S), and speech-to-speech (S2S) conversion to generate correctly pronounced and mispronounced synthetic speech. We show that these techniques not only improve the accuracy of three machine learning models for detecting pronunciation errors but also help establish a new state-of-the-art in the field. Earlier studies have used simple speech generation techniques such as P2P conversion, but only as an additional mechanism to improve the accuracy of pronunciation error detection. We, on the other hand, consider speech generation to be the first-class method of detecting pronunciation errors. The effectiveness of these techniques is assessed in the tasks of detecting pronunciation and lexical stress errors. Non-native English speech corpora of German, Italian, and Polish speakers are used in the evaluations. The best proposed S2S technique improves the accuracy of detecting pronunciation errors in AUC metric by 41\% from 0.528 to 0.749 compared to the state-of-the-art approach.
【10】 UserLibri: A Dataset for ASR Personalization Using Only Text
标题：UserLibri：仅使用文本的ASR个性化数据集
链接：https://arxiv.org/abs/2207.00706
作者：Theresa Breiner,Swaroop Ramaswamy,Ehsan Variani,Shefali Garg,Rajiv Mathews,Khe Chai Sim,Kilol Gupta,Mingqing Chen,Lara McConnaughey备注：Accepted for publication in Interspeech 2022. 9 total pages with appendix, 9 total tables, 5 total figures摘要：移动设备上语音模型的个性化（设备上个性化）是一个活跃的研究领域，但通常情况下，移动设备的纯文本数据多于成对的音频文本数据。我们探索在纯文本数据上训练个性化语言模型，在推理过程中用于提高该用户的语音识别性能。我们在用户聚集的LibriSpeech语料库上进行了实验，并为古腾堡项目的每个用户补充了个性化的纯文本数据。我们发布了这个特定于用户的LibriSpeech（UserLibri）数据集，以帮助未来的个性化研究。LibriSpeech音频转录对分为55个来自测试清理数据集的用户和52个来自测试其他的用户。在流媒体和非流媒体模型中，我们能够降低这两个集合中每个用户的平均文字错误率，包括在流媒体时测试其他用户的较难集合的2.5的改进。摘要：Personalization of speech models on mobile devices (on-device personalization) is an active area of research, but more often than not, mobile devices have more text-only data than paired audio-text data. We explore training a personalized language model on text-only data, used during inference to improve speech recognition performance for that user. We experiment on a user-clustered LibriSpeech corpus, supplemented with personalized text-only data for each user from Project Gutenberg. We release this User-Specific LibriSpeech (UserLibri) dataset to aid future personalization research. LibriSpeech audio-transcript pairs are grouped into 55 users from the test-clean dataset and 52 users from test-other. We are able to lower the average word error rate per user across both sets in streaming and nonstreaming models, including an improvement of 2.5 for the harder set of test-other users when streaming.
【11】 Speaker Diarization and Identification from Single-Channel Classroom Audio Recording Using Virtual Microphones
标题：基于虚拟麦克风的单声道课堂录音说话人二值化与识别
链接：https://arxiv.org/abs/2207.00660
* 与cs.SD语音【21】为同一篇
备注：Total of 170 pages, 59 figures. This work is part of the partial fulfillment of the requirements for the degree of PhD in engineering at the University of New Mexico摘要：在嘈杂的录音中，特别是在协作学习环境中，说话人识别可能极具挑战性。有必要将小组讨论的个别学生与同时讨论的其他学生区分开来。为了解决这个问题，我们假设每个学生组使用一个麦克风，而不需要访问以前的大型数据集进行训练。本文提出了一种以物理麦克风为中心，利用与虚拟麦克风阵列相关联的互相关模式进行说话人识别的方法。通过使用从视频记录中观察到的近似扬声器几何形状来模拟虚拟麦克风。模式是基于每个虚拟麦克风的房间冲击响应估计构建的。然后使用相关模式识别说话人。通过课堂音频验证了该方法，结果表明其性能大大优于谷歌云和亚马逊AWS提供的日记服务。摘要：Speaker identification in noisy audio recordings, specifically those from collaborative learning environments, can be extremely challenging. There is a need to identify individual students talking in small groups from other students talking at the same time. To solve the problem, we assume the use of a single microphone per student group without any access to previous large datasets for training. This dissertation proposes a method of speaker identification using cross-correlation patterns associated to an array of virtual microphones, centered around the physical microphone. The virtual microphones are simulated by using approximate speaker geometry observed from a video recording. The patterns are constructed based on estimates of the room impulse responses for each virtual microphone. The correlation patterns are then used to identify the speakers. The proposed method is validated with classroom audios and shown to substantially outperform diarization services provided by Google Cloud and Amazon AWS.
【12】 j-Wave: An open-source differentiable wave simulator
标题：J-Wave：一个开源的可微分波模拟器
链接：https://arxiv.org/abs/2207.01499
* 与cs.SD语音【17】为同一篇
作者：Antonio Stanziola,Simon R. Arridge,Ben T. Cox,Bradley E. Treeby机构：Dept. of Medical Physics and Biomedical Engineering, University College London, London WC,E ,BT, UK, Department of Computer Science, University College of London摘要：我们提出了一种开源的可微声学模拟器j-Wave，它可以解决时变和时谐声学问题。它支持自动微分，这是一种程序转换技术，有许多应用，特别是在机器学习和科学计算中。j-Wave由易于定制和重用的模块化组件组成。同时，它与一些最流行的机器学习库兼容，例如JAX和TensorFlow。根据广泛使用的k波工具箱和一系列声学模拟软件，评估了已知配置的模拟结果的准确性。j波可从https://github.com/ucl-bug/jwave.摘要：We present an open-source differentiable acoustic simulator, j-Wave, which can solve time-varying and time-harmonic acoustic problems. It supports automatic differentiation, which is a program transformation technique that has many applications, especially in machine learning and scientific computing. j-Wave is composed of modular components that can be easily customized and reused. At the same time, it is compatible with some of the most popular machine learning libraries, such as JAX and TensorFlow. The accuracy of the simulation results for known configurations is evaluated against the widely used k-Wave toolbox and a cohort of acoustic simulation software. j-Wave is available from https://github.com/ucl-bug/jwave.
【13】 State of the Art of Audio- and Video-Based Solutions for AA
标题：基于音频和视频的AA解决方案的最新发展
链接：https://arxiv.org/abs/2207.01487
* 与cs.SD语音【1】为同一篇
作者：Slavisa Aleksic,Michael Atanasov,Jean Calleja Agius,Kenneth Camilleri,Anto Cartolovni,Pau Climent-Peerez,Sara Colantonio,Stefania Cristina,Vladimir Despotovic,Hazim Kemal Ekenel,Ekrem Erakin,Francisco Florez-Revuelta,Danila Germanese,Nicole Grech,Steinunn Gróa Sigurðardóttir,Murat Emirzeoglu,Ivo Iliev,Mladjan Jovanovic,Martin Kampel,William Kearns,Andrzej Klimczuk,Lambros Lambrinos,Jennifer Lumetzberger,Wiktor Mucha,Sophie Noiret,Zada Pajalic,Rodrigo Rodriguez Peerez,Galidiya Petrova,Sintija Petrovica,Peter Pocta,Angelica Poli,Mara Pudane,Susanna Spinsante,Albert Ali Salah,Maria Jose Santofimia,Anna Sigridur Islind,Lacramioara Stoicu-Tivadar,Hilda Tellioglu,Andrej Zgank摘要：该报告阐述了基于音频和视频数据的最成功AAL应用和功能的最先进水平，即（i）生命记录和自我监测，（ii）生命体征的远程监测，（iii）情绪状态识别，（iv）食物摄入监测、活动和行为识别，（v）活动和个人协助，（vi）手势识别，（vii）坠落检测和预防，（viii）移动性评估和虚弱识别，以及（ix）认知和运动康复。对于这些应用场景，报告从科学进步、可用产品和研究项目方面阐述了发挥作用的状态。还强调了开放性挑战。摘要：The report illustrates the state of the art of the most successful AAL applications and functions based on audio and video data, namely (i) lifelogging and self-monitoring, (ii) remote monitoring of vital signs, (iii) emotional state recognition, (iv) food intake monitoring, activity and behaviour recognition, (v) activity and personal assistance, (vi) gesture recognition, (vii) fall detection and prevention, (viii) mobility assessment and frailty recognition, and (ix) cognitive and motor rehabilitation. For these application scenarios, the report illustrates the state of play in terms of scientific advances, available products and research project. The open challenges are also highlighted.
【14】 CaTT-KWS: A Multi-stage Customized Keyword Spotting Framework based on Cascaded Transducer-Transformer
标题：CATT-KWS：一种基于级联Transformer的多级定制关键词检测框架
链接：https://arxiv.org/abs/2207.01267
* 与cs.SD语音【2】为同一篇
作者：Zhanheng Yang,Sining Sun,Jin Li,Xiaoming Zhang,Xiong Wang,Long Ma,Lei Xie机构：Audio, Speech and Language Processing Group, School of Computer Science, Northwestern, Polytechnical University, Xi’an, China, Tencent Technology Co., Ltd, Beijing, China备注：Accepted by Interspeech 2022摘要：定制关键字识别（KWS）在边缘设备上部署具有很大潜力，可以实现免提用户体验。然而，在实际应用中，虚警（FA）对于发现数十个甚至数百个关键词来说是一个严重的问题，这严重影响了用户体验。为了解决这个问题，在本文中，我们利用传感器和基于Transformer的声学模型的最新进展，提出了一种新的多级定制KWS框架，称为级联传感器-TransformerKWS（CaTT-KWS），该框架包括基于传感器的关键字检测器、基于帧级电话预测器的力对齐模块和基于Transformer的解码器。具体来说，流传感器模块用于在音频流中发现候选关键字。然后，使用电话预测器预测的电话后验值来实现力对齐，以完成第一阶段的关键字验证，并细化关键字的时间边界。最后，transformer解码器进一步验证触发的关键字。我们提出的CaTT-KWS框架在不明显影响关键字识别准确性的情况下有效地降低了FA率。具体来说，在具有挑战性的数据集上，我们可以每小时获得令人印象深刻的0.13 FA，与基于传感器的检测模型相比，FA相对减少了90%以上，而关键字识别准确率仅下降不到2%。摘要：Customized keyword spotting (KWS) has great potential to be deployed on edge devices to achieve hands-free user experience. However, in real applications, false alarm (FA) would be a serious problem for spotting dozens or even hundreds of keywords, which drastically affects user experience. To solve this problem, in this paper, we leverage the recent advances in transducer and transformer based acoustic models and propose a new multi-stage customized KWS framework named Cascaded Transducer-Transformer KWS (CaTT-KWS), which includes a transducer based keyword detector, a frame-level phone predictor based force alignment module and a transformer based decoder. Specifically, the streaming transducer module is used to spot keyword candidates in audio stream. Then force alignment is implemented using the phone posteriors predicted by the phone predictor to finish the first stage keyword verification and refine the time boundaries of keyword. Finally, the transformer decoder further verifies the triggered keyword. Our proposed CaTT-KWS framework reduces FA rate effectively without obviously hurting keyword recognition accuracy. Specifically, we can get impressively 0.13 FA per hour on a challenging dataset, with over 90% relative reduction on FA comparing to the transducer based detection model, while keyword recognition accuracy only drops less than 2%.
【15】 Minimizing Sequential Confusion Error in Speech Command Recognition
标题：语音命令识别中序列混淆误差的最小化
链接：https://arxiv.org/abs/2207.01261
* 与cs.SD语音【3】为同一篇
作者：Zhanheng Yang,Hang Lv,Xiong Wang,Ao Zhang,Lei Xie机构：School of Computer Science, Northwestern Polytechnical University, Xi’an, China备注：Accepted by Interspeech 2022摘要：语音命令识别（SCR）通常用于资源受限的设备，以实现免提用户体验。然而，在实际应用中，由于边缘设备上部署的小模型容量有限，发音相似的命令之间经常会发生混淆，这严重影响了用户体验。在本文中，受语音识别中判别训练进展的启发，我们提出了一种新的最小化顺序混淆误差（MSCE）训练准则，特别是针对SCR的训练准则，旨在缓解命令混淆问题。具体来说，我们旨在提高基于MCE判别准则的目标命令与其他命令的判别能力。我们通过连接时序分类（CTC）定义不同命令的可能性。在训练过程中，我们提出了几种策略，使用先验知识为相似的发声命令创建混淆序列集，而不是创建整个非目标命令集，这样可以更好地节省训练资源，有效地减少命令混淆错误。具体来说，我们设计并比较了三种不同的混淆集构造策略。通过使用我们提出的方法，在0.01的虚警率下，我们可以相对减少33.7%的误拒绝率，在我们收集的语音命令集上，混淆错误减少18.28%。摘要：Speech command recognition (SCR) has been commonly used on resource constrained devices to achieve hands-free user experience. However, in real applications, confusion among commands with similar pronunciations often happens due to the limited capacity of small models deployed on edge devices, which drastically affects the user experience. In this paper, inspired by the advances of discriminative training in speech recognition, we propose a novel minimize sequential confusion error (MSCE) training criterion particularly for SCR, aiming to alleviate the command confusion problem. Specifically, we aim to improve the ability of discriminating the target command from other commands on the basis of MCE discriminative criteria. We define the likelihood of different commands through connectionist temporal classification (CTC). During training, we propose several strategies to use prior knowledge creating a confusing sequence set for similar-sounding command instead of creating the whole non-target command set, which can better save the training resources and effectively reduce command confusion errors. Specifically, we design and compare three different strategies for confusing set construction. By using our proposed method, we can relatively reduce the False Reject Rate~(FRR) by 33.7% at 0.01 False Alarm Rate~(FAR) and confusion errors by 18.28% on our collected speech command set.
【16】 TMGAN-PLC: Audio Packet Loss Concealment using Temporal Memory Generative Adversarial Network
标题：TMGAN-PLC：基于时间记忆生成对抗网络的音频丢包隐藏
链接：https://arxiv.org/abs/2207.01255
* 与cs.SD语音【4】为同一篇
作者：Yuansheng Guan,Guochen Yu,Andong Li,Chengshi Zheng,Jie Wang机构：School of Electronics and Communication Engineering, Guangzhou University, Guangzhou, China, Key Laboratory of Noise and Vibration Research, Institute of Acoustics, Chinese Academy of, Sciences, Beijing, China, Communication University of China, Beijing, China备注：accepted by INTERSPEECH 2022摘要：分组交换网络中的实时通信已广泛应用于日常通信中，但在受限的实时条件下，它们不可避免地遭受网络延迟和数据丢失。为了解决这些问题，人们开发了音频包丢失隐藏（PLC）算法，通过重建丢失的信息来缓解语音传输故障。受传输延迟和设备内存的限制，PLC仍然难以使用相对较小的数据包缓冲区来实现高质量的语音重建。在本文中，我们提出了一种用于音频PLC的时间记忆生成对抗网络，称为TMGAN-PLC，它由一个新型嵌套的UNet生成器和时域/频域鉴别器组成。具体来说，在生成器中精心设计了嵌套的UNet和时间特征线性调制的组合，以精细调整帧内信息并建立帧间时间依赖关系。为了补充由于长时间丢失突发而导致的语音内容缺失，我们采用多级选通矢量量化器来捕获正确的内容并重建接近真实的平滑音频。在PLC挑战数据集上的大量实验表明，该方法在语音质量、可懂度和PLCMOS方面具有良好的性能。摘要：Real-time communications in packet-switched networks have become widely used in daily communication, while they inevitably suffer from network delays and data losses in constrained real-time conditions. To solve these problems, audio packet loss concealment (PLC) algorithms have been developed to mitigate voice transmission failures by reconstructing the lost information. Limited by the transmission latency and device memory, it is still intractable for PLC to accomplish high-quality voice reconstruction using a relatively small packet buffer. In this paper, we propose a temporal memory generative adversarial network for audio PLC, dubbed TMGAN-PLC, which is comprised of a novel nested-UNet generator and the time-domain/frequency-domain discriminators. Specifically, a combination of the nested-UNet and temporal feature-wise linear modulation is elaborately devised in the generator to finely adjust the intra-frame information and establish inter-frame temporal dependencies. To complement the missing speech content caused by longer loss bursts, we employ multi-stage gated vector quantizers to capture the correct content and reconstruct the near-real smooth audio. Extensive experiments on the PLC Challenge dataset demonstrate that the proposed method yields promising performance in terms of speech quality, intelligibility, and PLCMOS.
【17】 Cross-speaker Emotion Transfer Based On Prosody Compensation for End-to-End Speech Synthesis
标题：端到端语音合成中基于韵律补偿的跨说话人情感传递
链接：https://arxiv.org/abs/2207.01198
* 与cs.SD语音【5】为同一篇
作者：Tao Li,Xinsheng Wang,Qicong Xie,Zhichao Wang,Mingqi Jiang,Lei Xie机构：Northwestern Polytechnical University, Xi’an, China, School of Software Engineering, Xi’an Jiaotong University, Xi’an, China, Mobvoi AI Lab, Suzhou, China摘要：跨说话人情感转移语音合成旨在通过从另一个（源）说话人记录的参考语音中转移情感来合成目标说话人的情感语音。在这项任务中，从参考语音中提取与说话人无关的情感嵌入起着重要作用。然而，在挤出源说话人音色信息的过程中，这种情感嵌入所传递的情感信息往往会被削弱。针对这一问题，本文提出了一种韵律补偿模块（PCM）来补偿情感信息的丢失。具体来说，PCM试图从预先训练的ASR模型的中间特征中获取与说话人无关的情感信息。为此，引入了一种具有全局上下文（GC）块的韵律补偿编码器，以从ASR模型的中间特征中获取全局情感信息。实验表明，该方法可以有效地补偿情感嵌入对情感信息丢失的影响，同时保持目标说话人的音色。与现有模型的比较表明，该方法在跨说话人情绪传递任务中具有明显的优势。摘要：Cross-speaker emotion transfer speech synthesis aims to synthesize emotional speech for a target speaker by transferring the emotion from reference speech recorded by another (source) speaker. In this task, extracting speaker-independent emotion embedding from reference speech plays an important role. However, the emotional information conveyed by such emotion embedding tends to be weakened in the process to squeeze out the source speaker's timbre information. In response to this problem, a prosody compensation module (PCM) is proposed in this paper to compensate for the emotional information loss. Specifically, the PCM tries to obtain speaker-independent emotional information from the intermediate feature of a pre-trained ASR model. To this end, a prosody compensation encoder with global context (GC) blocks is introduced to obtain global emotional information from the ASR model's intermediate feature. Experiments demonstrate that the proposed PCM can effectively compensate the emotion embedding for the emotional information loss, and meanwhile maintain the timbre of the target speaker. Comparisons with state-of-the-art models show that our proposed method presents obvious superiority on the cross-speaker emotion transfer task.
【18】 Multi-Modal Multi-Correlation Learning for Audio-Visual Speech Separation
标题：用于视听语音分离的多模多相关学习
链接：https://arxiv.org/abs/2207.01197
* 与cs.SD语音【6】为同一篇
作者：Xiaoyu Wang,Xiangyu Kong,Xiulian Peng,Yan Lu机构：Xi’an Jiaotong University, Microsoft Research Asia, Beijing, China备注：5 pages, accepted by interspeech2022摘要：在本文中，我们提出了一个多模态多相关学习框架，针对视听语音分离任务。尽管之前的工作已经广泛地将音频和视频模式结合起来，但其中大多数只采用音频和视频特征的直接连接。为了利用这两种模式背后真正有用的信息，我们定义了两个关键相关性：（1）身份相关性（音色和面部属性之间）；（2） 语音相关性（音素和嘴唇运动之间）。这两种相关性共同构成了完整的信息，在分离目标说话人的声音方面显示出一定的优势，尤其是在一些困难的情况下，例如相同性别或类似内容。为了实现这一点，我们采用了对比学习或对抗式训练方法来最大限度地提高这两种相关性。这两种方法都很有效，而对抗性训练则通过避免对比学习的一些局限性而显示出其优势。与以前的研究相比，我们的解决方案在不增加复杂性的情况下对实验指标有了明显的改进。进一步的分析揭示了该架构的有效性及其良好的未来扩展潜力。摘要：In this paper we propose a multi-modal multi-correlation learning framework targeting at the task of audio-visual speech separation. Although previous efforts have been extensively put on combining audio and visual modalities, most of them solely adopt a straightforward concatenation of audio and visual features. To exploit the real useful information behind these two modalities, we define two key correlations which are: (1) identity correlation (between timbre and facial attributes); (2) phonetic correlation (between phoneme and lip motion). These two correlations together comprise the complete information, which shows a certain superiority in separating target speaker's voice especially in some hard cases, such as the same gender or similar content. For implementation, contrastive learning or adversarial training approach is applied to maximize these two correlations. Both of them work well, while adversarial training shows its advantage by avoiding some limitations of contrastive learning. Compared with previous research, our solution demonstrates clear improvement on experimental metrics without additional complexity. Further analysis reveals the validity of the proposed architecture and its good potential for future extension.
【19】 ARAUS: A Large-Scale Dataset and Baseline Models of Affective Responses to Augmented Urban Soundscapes
标题：ARAUS：对增强的城市声音景观的情感反应的大规模数据集和基线模型
链接：https://arxiv.org/abs/2207.01078
* 与cs.SD语音【7】为同一篇
作者：Kenneth Ooi,Zhen-Ting Ong,Karn N. Watcharasupat,Bhan Lam,Joo Young Hong,Woon-Seng Gan备注：25 pages, 11 figures. Submitted to IEEE Transactions on Affective Computing摘要：为现有声景选择最佳掩蔽体以通过声景增强实现所需的感知变化是非常重要的，因为掩蔽体种类繁多，并且缺乏用于比较和开发声景增强模型的基准数据集。为了解决这个问题，我们公开了ARAUS（增强城市声景的情感反应）数据集，该数据集包括五重交叉验证集和独立测试集，共25440个对作为视听刺激呈现的增强声景的独特主观感知反应。每个增强的声景都是通过在城市声景记录中以固定的声景与遮罩比率数字添加“遮罩”（鸟、水、风、交通、建筑或沉默）来制作的。然后，根据ISO 12913-2:2018，通过要求参与者对每个增强音景的愉快、恼人、多事、平静、充满活力、单调、混乱、平静和适当程度进行评分来收集回答。参与者还提供了相关的人口统计信息，并完成了标准的心理问卷调查。我们对获得的回答进行探索性和统计分析，以验证内部一致性以及与文献中已知结果的一致性。最后，我们通过训练和比较城市声景观愉悦度的四个基线模型：低参数回归模型、高参数卷积神经网络和文献中的两个基于注意力的网络，证明了数据集的基准能力。摘要：Choosing optimal maskers for existing soundscapes to effect a desired perceptual change via soundscape augmentation is non-trivial due to extensive varieties of maskers and a dearth of benchmark datasets with which to compare and develop soundscape augmentation models. To address this problem, we make publicly available the ARAUS (Affective Responses to Augmented Urban Soundscapes) dataset, which comprises a five-fold cross-validation set and independent test set totaling 25,440 unique subjective perceptual responses to augmented soundscapes presented as audio-visual stimuli. Each augmented soundscape is made by digitally adding "maskers" (bird, water, wind, traffic, construction, or silence) to urban soundscape recordings at fixed soundscape-to-masker ratios. Responses were then collected by asking participants to rate how pleasant, annoying, eventful, uneventful, vibrant, monotonous, chaotic, calm, and appropriate each augmented soundscape was, in accordance with ISO 12913-2:2018. Participants also provided relevant demographic information and completed standard psychological questionnaires. We perform exploratory and statistical analysis of the responses obtained to verify internal consistency and agreement with known results in the literature. Finally, we demonstrate the benchmarking capability of the dataset by training and comparing four baseline models for urban soundscape pleasantness: a low-parameter regression model, a high-parameter convolutional neural network, and two attention-based networks in the literature.
【20】 Generating gender-ambiguous voices for privacy-preserving speech recognition
标题：为保护隐私的语音识别生成性别模糊的声音
链接：https://arxiv.org/abs/2207.01052
* 与cs.SD语音【8】为同一篇
作者：Dimitrios Stoidis,Andrea Cavallaro机构：Centre for Intelligent Sensing, Queen Mary University of London, UK备注：5 pages, 4 figures, submitted to INTERSPEECH摘要：我们的语音编码了一个唯一可识别的模式，该模式可用于推断个人在使用语音识别服务时可能不愿透露的私人属性，如性别或身份。为了在语音识别任务中防止属性推理攻击，我们提出了一种生成对抗网络GenGAN，该网络合成隐藏说话人性别或身份的声音。所提出的网络包括一个具有U-Net架构的生成器，该生成器学习愚弄鉴别器。我们仅根据性别信息对生成器进行调节，并在信号失真和隐私保护之间使用对抗性损失。我们表明，与将性别信息视为需要保护的敏感属性的隐私保护表示学习方法相比，GenGAN改进了隐私和效用之间的权衡。摘要：Our voice encodes a uniquely identifiable pattern which can be used to infer private attributes, such as gender or identity, that an individual might wish not to reveal when using a speech recognition service. To prevent attribute inference attacks alongside speech recognition tasks, we present a generative adversarial network, GenGAN, that synthesises voices that conceal the gender or identity of a speaker. The proposed network includes a generator with a U-Net architecture that learns to fool a discriminator. We condition the generator only on gender information and use an adversarial loss between signal distortion and privacy preservation. We show that GenGAN improves the trade-off between privacy and utility compared to privacy-preserving representation learning methods that consider gender information as a sensitive attribute to protect.
【21】 Towards Error-Resilient Neural Speech Coding
标题：面向容错的神经语音编码
链接：https://arxiv.org/abs/2207.00993
* 与cs.SD语音【9】为同一篇
作者：Huaying Xue,Xiulian Peng,Xue Jiang,Yan Lu机构：Microsoft Research Asia, Beijing, China,Communication University of China, Beijing, China备注：5 pages, Interspeech 2022(Accepted)摘要：神经音频编码最近在文献中显示出非常有前景的结果，在很大程度上优于传统的编解码器，但对其抗误码能力的关注有限。仅考虑信源编码训练的神经编解码器往往对信道噪声极为敏感，尤其是在误码率高的无线信道中。在本文中，我们研究了如何提高神经音频编解码器对实时通信过程中经常发生的数据包丢失的容错能力。我们提出了一种用于实时神经语音编码的特征域丢包隐藏算法（FD-PLC）。具体来说，我们介绍了一种基于接收到的潜在特征的自注意力模块，用于在解码器之前恢复特征域中丢失的帧。采用混合段级和帧级频域鉴别器引导网络关注丢失帧的生成质量和与相邻帧的连续性。在几种误差模式上的实验结果表明，与相应的无误差和容错基线相比，该方案具有更好的鲁棒性。我们还表明，作为后处理，特征域隐藏优于波形域对应。摘要：Neural audio coding has shown very promising results recently in the literature to largely outperform traditional codecs but limited attention has been paid on its error resilience. Neural codecs trained considering only source coding tend to be extremely sensitive to channel noises, especially in wireless channels with high error rate. In this paper, we investigate how to elevate the error resilience of neural audio codecs for packet losses that often occur during real-time communications. We propose a feature-domain packet loss concealment algorithm (FD-PLC) for real-time neural speech coding. Specifically, we introduce a self-attention-based module on the received latent features to recover lost frames in the feature domain before the decoder. A hybrid segment-level and frame-level frequency-domain discriminator is employed to guide the network to focus on both the generative quality of lost frames and the continuity with neighbouring frames. Experimental results on several error patterns show that the proposed scheme can achieve better robustness compared with the corresponding error-free and error-resilient baselines. We also show that feature-domain concealment is superior to waveform-domain counterpart as post-processing.
【22】 M-Adapter: Modality Adaptation for End-to-End Speech-to-Text Translation
标题：M-Adapter：端到端语音到文本翻译的通道适配
链接：https://arxiv.org/abs/2207.00952
* 与cs.SD语音【10】为同一篇
作者：Jinming Zhao,Hao Yang,Ehsan Shareghi,Gholamreza Haffari机构：Department of Data Science & AI, Monash University摘要：端到端语音到文本翻译模型通常使用预训练的语音编码器和预训练的文本解码器初始化。这导致预训练和微调之间存在显著的训练差距，这主要是由于编码器的语音输出和解码器的文本输入之间存在模态差异。在这项工作中，我们旨在弥合语音和文本之间的情态差异，以提高翻译质量。我们提出了一种新的基于Transformer的M-Adapter模块，用于将语音表示适应文本。在缩小语音序列的同时，M-Adapter通过建模语音序列的全局和局部相关性来生成语音到文本翻译所需的特征。我们的实验结果表明，在Must-C En$\ rightarrow$DE数据集上，我们的模型优于强基线，BLEU分数高达1。\脚注{我们的代码可在https://github.com/mingzi151/w2v2-st.}摘要：End-to-end speech-to-text translation models are often initialized with pre-trained speech encoder and pre-trained text decoder. This leads to a significant training gap between pre-training and fine-tuning, largely due to the modality differences between speech outputs from the encoder and text inputs to the decoder. In this work, we aim to bridge the modality gap between speech and text to improve translation quality. We propose M-Adapter, a novel Transformer-based module, to adapt speech representations to text. While shrinking the speech sequence, M-Adapter produces features desired for speech-to-text translation via modelling global and local dependencies of a speech sequence. Our experimental results show that our model outperforms a strong baseline by up to 1 BLEU score on the Must-C En$\rightarrow$DE dataset.\footnote{Our code is available at https://github.com/mingzi151/w2v2-st.}
【23】 Improving Transformer-based Conversational ASR by Inter-Sentential Attention Mechanism
标题：利用感知间注意机制改进基于Transformer的对话式ASR
链接：https://arxiv.org/abs/2207.00883
* 与cs.SD语音【11】为同一篇
作者：Kun Wei,Pengcheng Guo,Ning Jiang机构：Northwestern Polytechnical University, Xian, China, Mashang Consumer Finance Co., Ltd.备注：Accepted by Interspeech2022摘要：基于Transformer的模型已证明其在自动语音识别（ASR）任务中的有效性，甚至显示出优于传统混合框架的性能。Transformers的主要思想是通过自我注意力层捕捉话语中的长期全球语境。然而，对于会话语音等场景，这种话语级建模将忽略跨话语的语境依赖性。在本文中，我们提出在基于转换器的端到端架构中显式建模句子间信息，用于会话语音识别。具体来说，对于编码器网络，我们捕获以前语音的上下文，并通过上下文感知的剩余注意力机制将这些历史信息合并到当前输入中。对于译码器，当前话语的预测也通过条件译码器框架以历史语言信息为条件。我们在几个开源的对话语料库上证明了我们提出的方法的有效性，并且该方法不断提高了基于话语级转换的自动语音识别模型的性能。摘要：Transformer-based models have demonstrated their effectiveness in automatic speech recognition (ASR) tasks and even shown superior performance over the conventional hybrid framework. The main idea of Transformers is to capture the long-range global context within an utterance by self-attention layers. However, for scenarios like conversational speech, such utterance-level modeling will neglect contextual dependencies that span across utterances. In this paper, we propose to explicitly model the inter-sentential information in a Transformer based end-to-end architecture for conversational speech recognition. Specifically, for the encoder network, we capture the contexts of previous speech and incorporate such historic information into current input by a context-aware residual attention mechanism. For the decoder, the prediction of current utterance is also conditioned on the historic linguistic information through a conditional decoder framework. We show the effectiveness of our proposed method on several open-source dialogue corpora and the proposed method consistently improved the performance from the utterance-level Transformer-based ASR models.
【24】 Tree-constrained Pointer Generator with Graph Neural Network Encodings for Contextual Speech Recognition
标题：用于上下文语音识别的基于图神经网络编码的树约束指针生成器
链接：https://arxiv.org/abs/2207.00857
* 与cs.SD语音【12】为同一篇
作者：Guangzhi Sun,Chao Zhang,Philip C. Woodland机构：Cambridge University Engineering Dept., Trumpington St., Cambridge, CB,PZ U.K.备注：To appear in Interspeech 2022. arXiv admin note: text overlap with arXiv:2205.09058摘要：对于许多自动语音识别（ASR）应用来说，将获得的偏差词作为上下文知识进行合并是至关重要的。本文提出了在树约束指针生成器（TCPGen）组件中使用图神经网络（GNN）编码来实现端到端上下文ASR。通过使用基于树的GNN对前缀树中的偏置词进行编码，在每个树节点上通过合并来自它的树分支上的所有词条的信息来实现端到端ASR解码中对未来词条的展望，这允许更准确地预测偏置词的生成概率。使用模拟偏差任务在Librispeech语料库上评估系统，并通过提出一种新的基于视觉的上下文ASR管道在AMI语料库上评估系统，该管道从每次会议的幻灯片中提取偏差词。结果表明，与原始TCPGen相比，采用GNN编码的TCPGen在偏置字上实现了约15%的相对功耗降低，而解码的计算成本几乎没有增加。摘要：Incorporating biasing words obtained as contextual knowledge is critical for many automatic speech recognition (ASR) applications. This paper proposes the use of graph neural network (GNN) encodings in a tree-constrained pointer generator (TCPGen) component for end-to-end contextual ASR. By encoding the biasing words in the prefix-tree with a tree-based GNN, lookahead for future wordpieces in end-to-end ASR decoding is achieved at each tree node by incorporating information about all wordpieces on the tree branches rooted from it, which allows a more accurate prediction of the generation probability of the biasing words. Systems were evaluated on the Librispeech corpus using simulated biasing tasks, and on the AMI corpus by proposing a novel visual-grounded contextual ASR pipeline that extracts biasing words from slides alongside each meeting. Results showed that TCPGen with GNN encodings achieved about a further 15% relative WER reduction on the biasing words compared to the original TCPGen, with a negligible increase in the computation cost for decoding.
【25】 Unsupervised Symbolic Music Segmentation using Ensemble Temporal Prediction Errors
标题：基于集成时间预测误差的无监督符号音乐分割
链接：https://arxiv.org/abs/2207.00760
* 与cs.SD语音【13】为同一篇
作者：Shahaf Bassan,Yossi Adi,Jeffrey S. Rosenschein机构：Hebrew University of Jerusalem摘要：符号音乐分割是将符号旋律划分为更小的有意义组的过程，例如旋律短语。我们提出了一种无监督的符号音乐分割方法。该模型基于时间预测误差模型的集成。在训练期间，每个模型预测下一个标记，以识别音乐短语的变化。在测试时，我们执行峰值检测算法来选择候选段。最后，我们汇总参与集成的每个模型的预测，以预测最终分割。结果表明，在无监督设置下，当考虑F分数和R值时，该方法在埃森民歌数据集上达到了最先进的性能。我们还提供了一项消融研究，以更好地评估每个模型组件对最终结果的贡献。正如预期的那样，所提出的方法不如监督设置，这为在未来的研究中缩小无监督和监督方法之间的差距留下了改进的空间。摘要：Symbolic music segmentation is the process of dividing symbolic melodies into smaller meaningful groups, such as melodic phrases. We proposed an unsupervised method for segmenting symbolic music. The proposed model is based on an ensemble of temporal prediction error models. During training, each model predicts the next token to identify musical phrase changes. While at test time, we perform a peak detection algorithm to select segment candidates. Finally, we aggregate the predictions of each of the models participating in the ensemble to predict the final segmentation. Results suggest the proposed method reaches state-of-the-art performance on the Essen Folksong dataset under the unsupervised setting when considering F-Score and R-value. We additionally provide an ablation study to better assess the contribution of each of the model components to the final results. As expected, the proposed method is inferior to the supervised setting, which leaves room for improvement in future research considering closing the gap between unsupervised and supervised methods.
【26】 Learning Noise-independent Speech Representation for High-quality Voice Conversion for Noisy Target Speakers
标题：学习噪声无关的语音表示法实现含噪目标说话人的高质量语音转换
链接：https://arxiv.org/abs/2207.00756
* 与cs.SD语音【14】为同一篇
作者：Liumeng Xue,Shan Yang,Na Hu,Dan Su,Lei Xie机构：Northwestern Polytechnical University, Xi’an, China, Tencent AI Lab, China备注：Accepted by INTERSPEECH 2022摘要：由于在模型训练中使用受污染的语音会明显降低转换性能，因此为有噪声的目标说话人（如提供噪声样本或互联网数据的用户）构建语音转换系统是一项具有挑战性的任务。在本文中，我们利用我们最近提出的Glow-WaveGAN的进展，提出了一种与噪声无关的语音表示学习方法，用于噪声目标扬声器的高质量语音转换。具体来说，我们学习了一个潜在特征空间，在该空间中，我们确保由转换模型建模的目标分布完全来自波形发生器的建模分布。在此前提下，我们进一步设法使潜在特征具有噪声不变性。具体来说，我们介绍了一种噪声可控的WaveGAN，它通过编码器直接从波形中学习与噪声无关的声学表示，并通过解码器中的胶片模块在隐藏空间中进行噪声控制。对于转换模型，重要的是，我们使用基于流的模型从音素后验图中学习与噪声无关但与说话人相关的潜在特征的分布。实验结果表明，该模型在噪声目标说话人的语音转换中实现了较高的语音质量和说话人相似性。摘要：Building a voice conversion system for noisy target speakers, such as users providing noisy samples or Internet found data, is a challenging task since the use of contaminated speech in model training will apparently degrade the conversion performance. In this paper, we leverage the advances of our recently proposed Glow-WaveGAN and propose a noise-independent speech representation learning approach for high-quality voice conversion for noisy target speakers. Specifically, we learn a latent feature space where we ensure that the target distribution modeled by the conversion model is exactly from the modeled distribution of the waveform generator. With this premise, we further manage to make the latent feature to be noise-invariant. Specifically, we introduce a noise-controllable WaveGAN, which directly learns the noise-independent acoustic representation from waveform by the encoder and conducts noise control in the hidden space through a FiLM module in the decoder. As for the conversion model, importantly, we use a flow-based model to learn the distribution of noise-independent but speaker-related latent features from phoneme posteriorgrams. Experimental results demonstrate that the proposed model achieves high speech quality and speaker similarity in the voice conversion for noisy target speakers.
【27】 Building African Voices
标题：打造非洲声音
链接：https://arxiv.org/abs/2207.00688
* 与cs.SD语音【15】为同一篇
作者：Perez Ogayo,Graham Neubig,Alan W Black机构：†Language Technologies Institute, Carnegie Mellon University, ‡Inspired Cognition, Pittsburgh, PA, USA摘要：现代语音合成技术可以在提供足够高质量数据和计算资源的情况下产生自然发音的语音。然而，许多语言并没有现成的此类数据。本文的重点是低资源非洲语言的语音合成，从语料库创建到共享和部署文本到语音（TTS）系统。我们首先创建一组通用说明，以最少的技术资源和主题专业知识构建语音合成系统。接下来，我们创建新的数据集，并通过参与式方法从“发现”数据（现有记录）中整理数据集，同时考虑可访问性、质量和广度。我们证明，我们可以开发合成器，用25分钟的创建语音生成可理解的语音，即使在不理想的环境中录制。最后，我们发布了12种非洲语言的语音数据、代码和训练语音，以支持研究人员和开发人员。摘要：Modern speech synthesis techniques can produce natural-sounding speech given sufficient high-quality data and compute resources. However, such data is not readily available for many languages. This paper focuses on speech synthesis for low-resourced African languages, from corpus creation to sharing and deploying the Text-to-Speech (TTS) systems. We first create a set of general-purpose instructions on building speech synthesis systems with minimum technological resources and subject-matter expertise. Next, we create new datasets and curate datasets from "found" data (existing recordings) through a participatory approach while considering accessibility, quality, and breadth. We demonstrate that we can develop synthesizers that generate intelligible speech with 25 minutes of created speech, even when recorded in suboptimal environments. Finally, we release the speech data, code, and trained voices for 12 African languages to support researchers and developers.
机器翻译，仅供参考
每日语音速递