The search functionality is under construction.
The search functionality is under construction.

Analysis on Norms of Word Embedding and Hidden Vectors in Neural Conversational Model Based on Encoder-Decoder RNN

Manaya TOMIOKA, Tsuneo KATO, Akihiro TAMURA

  • Full Text Views

    0

  • Cite this

Summary :

A neural conversational model (NCM) based on an encoder-decoder recurrent neural network (RNN) with an attention mechanism learns different sequence-to-sequence mappings from what neural machine translation (NMT) learns even when based on the same technique. In the NCM, we confirmed that target-word-to-source-word mappings captured by the attention mechanism are not as clear and stationary as those for NMT. Considering that vector norms indicate a magnitude of information in the processing, we analyzed the inner workings of an encoder-decoder GRU-based NCM focusing on the norms of word embedding vectors and hidden vectors. First, we conducted correlation analyses on the norms of word embedding vectors with frequencies in the training set and with conditional entropies of a bi-gram language model to understand what is correlated with the norms in the encoder and decoder. Second, we conducted correlation analyses on norms of change in the hidden vector of the recurrent layer with their input vectors for the encoder and decoder, respectively. These analyses were done to understand how the magnitude of information propagates through the network. The analytical results suggested that the norms of the word embedding vectors are associated with their semantic information in the encoder, while those are associated with the predictability as a language model in the decoder. The analytical results further revealed how the norms propagate through the recurrent layer in the encoder and decoder.

Publication
IEICE TRANSACTIONS on Information Vol.E105-D No.10 pp.1780-1789
Publication Date
2022/10/01
Publicized
2022/06/30
Online ISSN
1745-1361
DOI
10.1587/transinf.2021EDP7227
Type of Manuscript
PAPER
Category
Natural Language Processing

Authors

Manaya TOMIOKA
  Doshisha University
Tsuneo KATO
  Doshisha University
Akihiro TAMURA
  Doshisha University

Keyword