Simultaneous Adaptation of Acoustic and Language Models for Emotional Speech Recognition Using Tweet Data

Tetsuo KOSAKA; Kazuya SAEKI; Yoshitaka AIZAWA; Masaharu KATO; Takashi NOSE

doi:10.1587/transinf.2023HCP0010

IEICE TRANSACTIONS on Information

Simultaneous Adaptation of Acoustic and Language Models for Emotional Speech Recognition Using Tweet Data

Tetsuo KOSAKA, Kazuya SAEKI, Yoshitaka AIZAWA, Masaharu KATO, Takashi NOSE

Full Text Views

0

Cite this

Summary :

Emotional speech recognition is generally considered more difficult than non-emotional speech recognition. The acoustic characteristics of emotional speech differ from those of non-emotional speech. Additionally, acoustic characteristics vary significantly depending on the type and intensity of emotions. Regarding linguistic features, emotional and colloquial expressions are also observed in their utterances. To solve these problems, we aim to improve recognition performance by adapting acoustic and language models to emotional speech. We used Japanese Twitter-based Emotional Speech (JTES) as an emotional speech corpus. This corpus consisted of tweets and had an emotional label assigned to each utterance. Corpus adaptation is possible using the utterances contained in this corpus. However, regarding the language model, the amount of adaptation data is insufficient. To solve this problem, we propose an adaptation of the language model by using online tweet data downloaded from the internet. The sentences used for adaptation were extracted from the tweet data based on certain rules. We extracted the data of 25.86 M words and used them for adaptation. In the recognition experiments, the baseline word error rate was 36.11%, whereas that with the acoustic and language model adaptation was 17.77%. The results demonstrated the effectiveness of the proposed method.

Publication: IEICE TRANSACTIONS on Information Vol.E107-D No.3 pp.363-373

Publication Date: 2024/03/01

Publicized: 2023/12/05

Online ISSN: 1745-1361

DOI: 10.1587/transinf.2023HCP0010

Type of Manuscript: Special Section PAPER (Special Section on Human Communication V)

Category

Authors

Tetsuo KOSAKA
  Yamagata University
Kazuya SAEKI
  Yamagata University
Yoshitaka AIZAWA
  Yamagata University
Masaharu KATO
  Yamagata University
Takashi NOSE
  Tohoku University

Keyword

speech recognition, emotional speech, acoustic model adaptation, language model adaptation, deep neural networks

Cite this

Copy

Tetsuo KOSAKA, Kazuya SAEKI, Yoshitaka AIZAWA, Masaharu KATO, Takashi NOSE, "Simultaneous Adaptation of Acoustic and Language Models for Emotional Speech Recognition Using Tweet Data" in IEICE TRANSACTIONS on Information, vol. E107-D, no. 3, pp. 363-373, March 2024, doi: 10.1587/transinf.2023HCP0010.
Abstract: Emotional speech recognition is generally considered more difficult than non-emotional speech recognition. The acoustic characteristics of emotional speech differ from those of non-emotional speech. Additionally, acoustic characteristics vary significantly depending on the type and intensity of emotions. Regarding linguistic features, emotional and colloquial expressions are also observed in their utterances. To solve these problems, we aim to improve recognition performance by adapting acoustic and language models to emotional speech. We used Japanese Twitter-based Emotional Speech (JTES) as an emotional speech corpus. This corpus consisted of tweets and had an emotional label assigned to each utterance. Corpus adaptation is possible using the utterances contained in this corpus. However, regarding the language model, the amount of adaptation data is insufficient. To solve this problem, we propose an adaptation of the language model by using online tweet data downloaded from the internet. The sentences used for adaptation were extracted from the tweet data based on certain rules. We extracted the data of 25.86 M words and used them for adaptation. In the recognition experiments, the baseline word error rate was 36.11%, whereas that with the acoustic and language model adaptation was 17.77%. The results demonstrated the effectiveness of the proposed method.
URL: https://global.ieice.org/en_transactions/information/10.1587/transinf.2023HCP0010/_p

Copy

@ARTICLE{e107-d_3_363,
author={Tetsuo KOSAKA, Kazuya SAEKI, Yoshitaka AIZAWA, Masaharu KATO, Takashi NOSE, },
journal={IEICE TRANSACTIONS on Information},
title={Simultaneous Adaptation of Acoustic and Language Models for Emotional Speech Recognition Using Tweet Data},
year={2024},
volume={E107-D},
number={3},
pages={363-373},
abstract={Emotional speech recognition is generally considered more difficult than non-emotional speech recognition. The acoustic characteristics of emotional speech differ from those of non-emotional speech. Additionally, acoustic characteristics vary significantly depending on the type and intensity of emotions. Regarding linguistic features, emotional and colloquial expressions are also observed in their utterances. To solve these problems, we aim to improve recognition performance by adapting acoustic and language models to emotional speech. We used Japanese Twitter-based Emotional Speech (JTES) as an emotional speech corpus. This corpus consisted of tweets and had an emotional label assigned to each utterance. Corpus adaptation is possible using the utterances contained in this corpus. However, regarding the language model, the amount of adaptation data is insufficient. To solve this problem, we propose an adaptation of the language model by using online tweet data downloaded from the internet. The sentences used for adaptation were extracted from the tweet data based on certain rules. We extracted the data of 25.86 M words and used them for adaptation. In the recognition experiments, the baseline word error rate was 36.11%, whereas that with the acoustic and language model adaptation was 17.77%. The results demonstrated the effectiveness of the proposed method.},
keywords={},
doi={10.1587/transinf.2023HCP0010},
ISSN={1745-1361},
month={March},}

Copy

TY - JOUR
TI - Simultaneous Adaptation of Acoustic and Language Models for Emotional Speech Recognition Using Tweet Data
T2 - IEICE TRANSACTIONS on Information
SP - 363
EP - 373
AU - Tetsuo KOSAKA
AU - Kazuya SAEKI
AU - Yoshitaka AIZAWA
AU - Masaharu KATO
AU - Takashi NOSE
PY - 2024
DO - 10.1587/transinf.2023HCP0010
JO - IEICE TRANSACTIONS on Information
SN - 1745-1361
VL - E107-D
IS - 3
JA - IEICE TRANSACTIONS on Information
Y1 - March 2024
AB - Emotional speech recognition is generally considered more difficult than non-emotional speech recognition. The acoustic characteristics of emotional speech differ from those of non-emotional speech. Additionally, acoustic characteristics vary significantly depending on the type and intensity of emotions. Regarding linguistic features, emotional and colloquial expressions are also observed in their utterances. To solve these problems, we aim to improve recognition performance by adapting acoustic and language models to emotional speech. We used Japanese Twitter-based Emotional Speech (JTES) as an emotional speech corpus. This corpus consisted of tweets and had an emotional label assigned to each utterance. Corpus adaptation is possible using the utterances contained in this corpus. However, regarding the language model, the amount of adaptation data is insufficient. To solve this problem, we propose an adaptation of the language model by using online tweet data downloaded from the internet. The sentences used for adaptation were extracted from the tweet data based on certain rules. We extracted the data of 25.86 M words and used them for adaptation. In the recognition experiments, the baseline word error rate was 36.11%, whereas that with the acoustic and language model adaptation was 17.77%. The results demonstrated the effectiveness of the proposed method.
ER -

IEICE TRANSACTIONS on Information