| Title: |
It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognition |
| Authors: |
Chen, Chen, Li, Ruizhe, Hu, Yuchen, Siniscalchi, Sabato Marco, Chen, Pin-Yu, Chng, Ensiong, Yang, Chao-Han Huck |
| Publication Year: |
2024 |
| Collection: |
Computer Science |
| Subject Terms: |
Computer Science - Computation and Language, Computer Science - Artificial Intelligence, Computer Science - Multimedia, Computer Science - Sound, Electrical Engineering and Systems Science - Audio and Speech Processing |
| More Details: |
Recent studies have successfully shown that large language models (LLMs) can be successfully used for generative error correction (GER) on top of the automatic speech recognition (ASR) output. Specifically, an LLM is utilized to carry out a direct mapping from the N-best hypotheses list generated by an ASR system to the predicted output transcription. However, despite its effectiveness, GER introduces extra data uncertainty since the LLM is trained without taking into account acoustic information available in the speech signal. In this work, we aim to overcome such a limitation by infusing acoustic information before generating the predicted transcription through a novel late fusion solution termed Uncertainty-Aware Dynamic Fusion (UADF). UADF is a multimodal fusion approach implemented into an auto-regressive decoding process and works in two stages: (i) It first analyzes and calibrates the token-level LLM decision, and (ii) it then dynamically assimilates the information from the acoustic modality. Experimental evidence collected from various ASR tasks shows that UADF surpasses existing fusion mechanisms in several ways. It yields significant improvements in word error rate (WER) while mitigating data uncertainty issues in LLM and addressing the poor generalization relied with sole modality during fusion. We also demonstrate that UADF seamlessly adapts to audio-visual speech recognition.
Comment: Accepted to ICLR 2024, 17 pages. This work will be open sourced under MIT license |
| Document Type: |
Working Paper |
| Access URL: |
http://arxiv.org/abs/2402.05457 |
| Accession Number: |
edsarx.2402.05457 |
| Database: |
arXiv |
