今日论文合集：cs.SD语音3篇，eess.AS音频处理2篇。
【1】 Chat2Map: Efficient Scene Mapping from Multi-Ego Conversations
标题：Chat2Map：多自我对话的高效场景映射
链接：https://arxiv.org/abs/2301.02184
作者：Sagnik Majumder,Hao Jiang,Pierre Moulon,Ethan Henderson,Paul Calamia,Kristen Grauman,Vamsi Krishna Ithapu机构：UT Austin, Reality Labs Research, Meta, FAIR摘要：从多个自我中心的视角捕捉的对话视频能以一种经济高效的方式揭示场景的地图吗？我们试图通过提出一个新问题来回答这个问题：通过利用自然对话中的参与者的自我中心视听观察中的共享信息来高效地构建先前不可见的3D环境的地图。我们的假设是，当多个人（“自我”）在一个场景中移动并相互交谈时，他们会收到丰富的视听线索，这些线索可以帮助发现场景中看不见的区域。考虑到连续处理以自我为中心的视觉流的高成本，我们进一步探索如何主动协调视觉信息的采样，以最小化冗余并降低功耗。为此，我们提出了一种视听深度强化学习方法，该方法与我们共享的场景映射器一起工作，选择性地打开摄像头，以高效地绘制出空间。我们使用最先进的3D场景和真实世界视频的视听模拟器来评估该方法。我们的模型优于以往最先进的映射方法，并实现了良好的成本-精度折衷。项目：http://vision.cs.utexas.edu/projects/chat2map。摘要：Can conversational videos captured from multiple egocentric viewpoints reveal the map of a scene in a cost-efficient way? We seek to answer this question by proposing a new problem: efficiently building the map of a previously unseen 3D environment by exploiting shared information in the egocentric audio-visual observations of participants in a natural conversation. Our hypothesis is that as multiple people ("egos") move in a scene and talk among themselves, they receive rich audio-visual cues that can help uncover the unseen areas of the scene. Given the high cost of continuously processing egocentric visual streams, we further explore how to actively coordinate the sampling of visual information, so as to minimize redundancy and reduce power use. To that end, we present an audio-visual deep reinforcement learning approach that works with our shared scene mapper to selectively turn on the camera to efficiently chart out the space. We evaluate the approach using a state-of-the-art audio-visual simulator for 3D scenes as well as real-world video. Our model outperforms previous state-of-the-art mapping methods, and achieves an excellent cost-accuracy tradeoff. Project: http://vision.cs.utexas.edu/projects/chat2map.
【2】 Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
标题：神经编解码器语言模型是零命中文本到语音合成器
链接：https://arxiv.org/abs/2301.02111
作者：Chengyi Wang,Sanyuan Chen,Yu Wu,Ziqiang Zhang,Long Zhou,Shujie Liu,Zhuo Chen,Yanqing Liu,Huaming Wang,Jinyu Li,Lei He,Sheng Zhao,Furu Wei摘要：本文介绍了一种用于文语合成的语言建模方法。具体而言，我们使用从现成的神经音频编解码器模型导出的离散代码来训练神经编解码器语言模型（称为Vall-E），并将TTS视为条件语言建模任务，而不是如先前工作中的连续信号回归。在预训练阶段，我们将TTS训练数据放大到60 K小时的英语语音，这比现有系统大数百倍。Vall-E具有上下文学习能力，可以用来合成高质量的个性化语音，只需3秒的未见过的说话者的注册录音作为声音提示。实验结果表明，Vall-E在语音自然度和说话人相似度方面明显优于现有的zero-shot TTS系统。此外，我们发现Vall-E在合成时能够保持说话人的情感和声音提示的声学环境。请访问https://aka.ms/valle了解我们工作的演示。摘要：We introduce a language modeling approach for text to speech synthesis (TTS). Specifically, we train a neural codec language model (called Vall-E) using discrete codes derived from an off-the-shelf neural audio codec model, and regard TTS as a conditional language modeling task rather than continuous signal regression as in previous work. During the pre-training stage, we scale up the TTS training data to 60K hours of English speech which is hundreds of times larger than existing systems. Vall-E emerges in-context learning capabilities and can be used to synthesize high-quality personalized speech with only a 3-second enrolled recording of an unseen speaker as an acoustic prompt. Experiment results show that Vall-E significantly outperforms the state-of-the-art zero-shot TTS system in terms of speech naturalness and speaker similarity. In addition, we find Vall-E could preserve the speaker's emotion and acoustic environment of the acoustic prompt in synthesis. See https://aka.ms/valle for demos of our work.
【3】 Automatic Sound Event Detection and Classification of Great Ape Calls Using Neural Networks
标题：基于神经网络的大猩猩叫声事件自动检测与分类
链接：https://arxiv.org/abs/2301.02214
作者：Zifan Jiang,Adrian Soldati,Isaac Schamberg,Adriano R. Lameira,Steven Moran机构：University of Neuchâtel,University of Zurich,University of Warwick摘要：我们提出了一种新的方法来自动检测和分类类人猿呼叫从连续原始录音收集在实地研究。我们的方法利用了深度预训练和序列神经网络，包括wav2vec 2.0和LSTM，并在来自三种不同类人猿谱系（猩猩、黑猩猩和倭黑猩猩）的三个数据集上进行了验证。记录由不同的研究人员收集，包括不同的注释方案，我们的管道以统一的方式预处理和训练这些方案。我们的呼叫检测和分类结果达到了很高的准确率。我们的方法旨在推广到其他动物物种，并且更一般地，推广到声音事件检测任务。为了促进未来的研究，我们公开了我们的研究管道和方法。摘要：We present a novel approach to automatically detect and classify great ape calls from continuous raw audio recordings collected during field research. Our method leverages deep pretrained and sequential neural networks, including wav2vec 2.0 and LSTM, and is validated on three data sets from three different great ape lineages (orangutans, chimpanzees, and bonobos). The recordings were collected by different researchers and include different annotation schemes, which our pipeline preprocesses and trains in a uniform fashion. Our results for call detection and classification attain high accuracy. Our method is aimed to be generalizable to other animal species, and more generally, sound event detection tasks. To foster future research, we make our pipeline and methods publicly available.【1】 Automatic Sound Event Detection and Classification of Great Ape Calls Using Neural Networks
标题：基于神经网络的大猩猩叫声事件自动检测与分类
链接：https://arxiv.org/abs/2301.02214
*与cs.SD语音【3】为同一篇
作者：Zifan Jiang,Adrian Soldati,Isaac Schamberg,Adriano R. Lameira,Steven Moran机构：University of Neuchâtel,University of Zurich,University of Warwick摘要：我们提出了一种新的方法来自动检测和分类类人猿呼叫从连续原始录音收集在实地研究。我们的方法利用了深度预训练和序列神经网络，包括wav2vec 2.0和LSTM，并在来自三种不同类人猿谱系（猩猩、黑猩猩和倭黑猩猩）的三个数据集上进行了验证。记录由不同的研究人员收集，包括不同的注释方案，我们的管道以统一的方式预处理和训练这些方案。我们的呼叫检测和分类结果达到了很高的准确率。我们的方法旨在推广到其他动物物种，并且更一般地，推广到声音事件检测任务。为了促进未来的研究，我们公开了我们的研究管道和方法。摘要：We present a novel approach to automatically detect and classify great ape calls from continuous raw audio recordings collected during field research. Our method leverages deep pretrained and sequential neural networks, including wav2vec 2.0 and LSTM, and is validated on three data sets from three different great ape lineages (orangutans, chimpanzees, and bonobos). The recordings were collected by different researchers and include different annotation schemes, which our pipeline preprocesses and trains in a uniform fashion. Our results for call detection and classification attain high accuracy. Our method is aimed to be generalizable to other animal species, and more generally, sound event detection tasks. To foster future research, we make our pipeline and methods publicly available.
【2】 Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
标题：神经编解码器语言模型是零命中文本到语音合成器
链接：https://arxiv.org/abs/2301.02111
*与cs.SD语音【2】为同一篇
作者：Chengyi Wang,Sanyuan Chen,Yu Wu,Ziqiang Zhang,Long Zhou,Shujie Liu,Zhuo Chen,Yanqing Liu,Huaming Wang,Jinyu Li,Lei He,Sheng Zhao,Furu Wei摘要：本文介绍了一种用于文语合成的语言建模方法。具体而言，我们使用从现成的神经音频编解码器模型导出的离散代码来训练神经编解码器语言模型（称为Vall-E），并将TTS视为条件语言建模任务，而不是如先前工作中的连续信号回归。在预训练阶段，我们将TTS训练数据放大到60 K小时的英语语音，这比现有系统大数百倍。Vall-E具有上下文学习能力，可以用来合成高质量的个性化语音，只需3秒的未见过的说话者的注册录音作为声音提示。实验结果表明，Vall-E在语音自然度和说话人相似度方面明显优于现有的zero-shot TTS系统。此外，我们发现Vall-E在合成时能够保持说话人的情感和声音提示的声学环境。请访问https://aka.ms/valle了解我们工作的演示。摘要：We introduce a language modeling approach for text to speech synthesis (TTS). Specifically, we train a neural codec language model (called Vall-E) using discrete codes derived from an off-the-shelf neural audio codec model, and regard TTS as a conditional language modeling task rather than continuous signal regression as in previous work. During the pre-training stage, we scale up the TTS training data to 60K hours of English speech which is hundreds of times larger than existing systems. Vall-E emerges in-context learning capabilities and can be used to synthesize high-quality personalized speech with only a 3-second enrolled recording of an unseen speaker as an acoustic prompt. Experiment results show that Vall-E significantly outperforms the state-of-the-art zero-shot TTS system in terms of speech naturalness and speaker similarity. In addition, we find Vall-E could preserve the speaker's emotion and acoustic environment of the acoustic prompt in synthesis. See https://aka.ms/valle for demos of our work.
机器翻译，仅供参考
每日语音速递