The search functionality is under construction.
The search functionality is under construction.

New Word Detection Using BiLSTM+CRF Model with Features

Jianyong DUAN, Zheng TAN, Mei ZHANG, Hao WANG

  • Full Text Views

    0

  • Cite this

Summary :

With the widespread popularity of a large number of social platforms, an increasing number of new words gradually appear. However, such new words have made some NLP tasks like word segmentation more challenging. Therefore, new word detection is always an important and tough task in NLP. This paper aims to extract new words using the BiLSTM+CRF model which added some features selected by us. These features include word length, part of speech (POS), contextual entropy and degree of word coagulation. Comparing to the traditional new word detection methods, our method can use both the features extracted by the model and the features we select to find new words. Experimental results demonstrate that our model can perform better compared to the benchmark models.

Publication
IEICE TRANSACTIONS on Information Vol.E103-D No.10 pp.2228-2236
Publication Date
2020/10/01
Publicized
2020/07/14
Online ISSN
1745-1361
DOI
10.1587/transinf.2019EDP7330
Type of Manuscript
PAPER
Category
Natural Language Processing

Authors

Jianyong DUAN
  North China University of Technology,CNONIX National Standard Application and Promotion Lab
Zheng TAN
  North China University of Technology,CNONIX National Standard Application and Promotion Lab
Mei ZHANG
  North China University of Technology
Hao WANG
  North China University of Technology,CNONIX National Standard Application and Promotion Lab

Keyword