The recent development of neural architecture search (NAS) has enabled us to automatically discover architectures of neural networks with high performance within a few days. Convolutional neural networks extract fruitful features by repeatedly applying standard operations (convolutions and poolings). However, these operations also extract useless or even disturbing features. Attention mechanisms enable neural networks to discard information of no interest, having achieved the state-of-the-art performance. While a variety of attentions for CNNs have been proposed, current NAS methods have paid a little attention to them. In this study, we propose a novel NAS method that searches attentions as well as operations. We examined several patterns to arrange attentions and operations, and found that attentions work better when they have their own search space and follow operations. We demonstrate the superior performance of our method in experiments on CIFAR-10, CIFAR-100, and ImageNet datasets. The found architecture achieved lower classification error rates and required fewer parameters compared to those found by current NAS methods.
Kohei NAKAI
Kobe University
Takashi MATSUBARA
Osaka University
Kuniaki UEHARA
Osaka Gakuin University
The copyright of the original papers published on this site belongs to IEICE. Unauthorized use of the original or translated papers is prohibited. See IEICE Provisions on Copyright for details.
Copy
Kohei NAKAI, Takashi MATSUBARA, Kuniaki UEHARA, "Neural Architecture Search for Convolutional Neural Networks with Attention" in IEICE TRANSACTIONS on Information,
vol. E104-D, no. 2, pp. 312-321, February 2021, doi: 10.1587/transinf.2020EDP7111.
Abstract: The recent development of neural architecture search (NAS) has enabled us to automatically discover architectures of neural networks with high performance within a few days. Convolutional neural networks extract fruitful features by repeatedly applying standard operations (convolutions and poolings). However, these operations also extract useless or even disturbing features. Attention mechanisms enable neural networks to discard information of no interest, having achieved the state-of-the-art performance. While a variety of attentions for CNNs have been proposed, current NAS methods have paid a little attention to them. In this study, we propose a novel NAS method that searches attentions as well as operations. We examined several patterns to arrange attentions and operations, and found that attentions work better when they have their own search space and follow operations. We demonstrate the superior performance of our method in experiments on CIFAR-10, CIFAR-100, and ImageNet datasets. The found architecture achieved lower classification error rates and required fewer parameters compared to those found by current NAS methods.
URL: https://global.ieice.org/en_transactions/information/10.1587/transinf.2020EDP7111/_p
Copy
@ARTICLE{e104-d_2_312,
author={Kohei NAKAI, Takashi MATSUBARA, Kuniaki UEHARA, },
journal={IEICE TRANSACTIONS on Information},
title={Neural Architecture Search for Convolutional Neural Networks with Attention},
year={2021},
volume={E104-D},
number={2},
pages={312-321},
abstract={The recent development of neural architecture search (NAS) has enabled us to automatically discover architectures of neural networks with high performance within a few days. Convolutional neural networks extract fruitful features by repeatedly applying standard operations (convolutions and poolings). However, these operations also extract useless or even disturbing features. Attention mechanisms enable neural networks to discard information of no interest, having achieved the state-of-the-art performance. While a variety of attentions for CNNs have been proposed, current NAS methods have paid a little attention to them. In this study, we propose a novel NAS method that searches attentions as well as operations. We examined several patterns to arrange attentions and operations, and found that attentions work better when they have their own search space and follow operations. We demonstrate the superior performance of our method in experiments on CIFAR-10, CIFAR-100, and ImageNet datasets. The found architecture achieved lower classification error rates and required fewer parameters compared to those found by current NAS methods.},
keywords={},
doi={10.1587/transinf.2020EDP7111},
ISSN={1745-1361},
month={February},}
Copy
TY - JOUR
TI - Neural Architecture Search for Convolutional Neural Networks with Attention
T2 - IEICE TRANSACTIONS on Information
SP - 312
EP - 321
AU - Kohei NAKAI
AU - Takashi MATSUBARA
AU - Kuniaki UEHARA
PY - 2021
DO - 10.1587/transinf.2020EDP7111
JO - IEICE TRANSACTIONS on Information
SN - 1745-1361
VL - E104-D
IS - 2
JA - IEICE TRANSACTIONS on Information
Y1 - February 2021
AB - The recent development of neural architecture search (NAS) has enabled us to automatically discover architectures of neural networks with high performance within a few days. Convolutional neural networks extract fruitful features by repeatedly applying standard operations (convolutions and poolings). However, these operations also extract useless or even disturbing features. Attention mechanisms enable neural networks to discard information of no interest, having achieved the state-of-the-art performance. While a variety of attentions for CNNs have been proposed, current NAS methods have paid a little attention to them. In this study, we propose a novel NAS method that searches attentions as well as operations. We examined several patterns to arrange attentions and operations, and found that attentions work better when they have their own search space and follow operations. We demonstrate the superior performance of our method in experiments on CIFAR-10, CIFAR-100, and ImageNet datasets. The found architecture achieved lower classification error rates and required fewer parameters compared to those found by current NAS methods.
ER -