Neural Behavior-Based Approach for Neural Network Pruning

Koji KAMMA; Yuki ISODA; Sarimu INOUE; Toshikazu WADA

doi:10.1587/transinf.2019EDP7177

Neural Behavior-Based Approach for Neural Network Pruning

Koji KAMMA, Yuki ISODA, Sarimu INOUE, Toshikazu WADA

Full Text Views

0

Cite this

Summary :

This paper presents a method for reducing the redundancy in both fully connected layers and convolutional layers of trained neural network models. The proposed method consists of two steps, 1) Neuro-Coding: to encode the behavior of each neuron by a vector composed of its outputs corresponding to actual inputs and 2) Neuro-Unification: to unify the neurons having the similar behavioral vectors. Instead of just pruning one of the similar neurons, the proposed method let the remaining neuron emulate the behavior of the pruned one. Therefore, the proposed method can reduce the number of neurons with small sacrifice of accuracy without retraining. Our method can be applied for compressing convolutional layers as well. In the convolutional layers, the behavior of each channel is encoded by its output feature maps, and channels whose behaviors can be well emulated by other channels are pruned and update the remaining weights. Through several experiments, we comfirmed that the proposed method performs better than the existing methods.

Publication: IEICE TRANSACTIONS on Information Vol.E103-D No.5 pp.1135-1143

Publication Date: 2020/05/01

Publicized: 2020/01/23

Online ISSN: 1745-1361

DOI: 10.1587/transinf.2019EDP7177

Type of Manuscript: PAPER

Category: Biocybernetics, Neurocomputing

Authors

Koji KAMMA
  Wakayama University
Yuki ISODA
  Wakayama University
Sarimu INOUE
  Wakayama University
Toshikazu WADA
  Wakayama University

Keyword

neuro-coding, neuro-unification

Cite this

Copy

Koji KAMMA, Yuki ISODA, Sarimu INOUE, Toshikazu WADA, "Neural Behavior-Based Approach for Neural Network Pruning" in IEICE TRANSACTIONS on Information, vol. E103-D, no. 5, pp. 1135-1143, May 2020, doi: 10.1587/transinf.2019EDP7177.
Abstract: This paper presents a method for reducing the redundancy in both fully connected layers and convolutional layers of trained neural network models. The proposed method consists of two steps, 1) Neuro-Coding: to encode the behavior of each neuron by a vector composed of its outputs corresponding to actual inputs and 2) Neuro-Unification: to unify the neurons having the similar behavioral vectors. Instead of just pruning one of the similar neurons, the proposed method let the remaining neuron emulate the behavior of the pruned one. Therefore, the proposed method can reduce the number of neurons with small sacrifice of accuracy without retraining. Our method can be applied for compressing convolutional layers as well. In the convolutional layers, the behavior of each channel is encoded by its output feature maps, and channels whose behaviors can be well emulated by other channels are pruned and update the remaining weights. Through several experiments, we comfirmed that the proposed method performs better than the existing methods.
URL: https://global.ieice.org/en_transactions/information/10.1587/transinf.2019EDP7177/_p

Copy

@ARTICLE{e103-d_5_1135,
author={Koji KAMMA, Yuki ISODA, Sarimu INOUE, Toshikazu WADA, },
journal={IEICE TRANSACTIONS on Information},
title={Neural Behavior-Based Approach for Neural Network Pruning},
year={2020},
volume={E103-D},
number={5},
pages={1135-1143},
abstract={This paper presents a method for reducing the redundancy in both fully connected layers and convolutional layers of trained neural network models. The proposed method consists of two steps, 1) Neuro-Coding: to encode the behavior of each neuron by a vector composed of its outputs corresponding to actual inputs and 2) Neuro-Unification: to unify the neurons having the similar behavioral vectors. Instead of just pruning one of the similar neurons, the proposed method let the remaining neuron emulate the behavior of the pruned one. Therefore, the proposed method can reduce the number of neurons with small sacrifice of accuracy without retraining. Our method can be applied for compressing convolutional layers as well. In the convolutional layers, the behavior of each channel is encoded by its output feature maps, and channels whose behaviors can be well emulated by other channels are pruned and update the remaining weights. Through several experiments, we comfirmed that the proposed method performs better than the existing methods.},
keywords={},
doi={10.1587/transinf.2019EDP7177},
ISSN={1745-1361},
month={May},}

Copy

TY - JOUR
TI - Neural Behavior-Based Approach for Neural Network Pruning
T2 - IEICE TRANSACTIONS on Information
SP - 1135
EP - 1143
AU - Koji KAMMA
AU - Yuki ISODA
AU - Sarimu INOUE
AU - Toshikazu WADA
PY - 2020
DO - 10.1587/transinf.2019EDP7177
JO - IEICE TRANSACTIONS on Information
SN - 1745-1361
VL - E103-D
IS - 5
JA - IEICE TRANSACTIONS on Information
Y1 - May 2020
AB - This paper presents a method for reducing the redundancy in both fully connected layers and convolutional layers of trained neural network models. The proposed method consists of two steps, 1) Neuro-Coding: to encode the behavior of each neuron by a vector composed of its outputs corresponding to actual inputs and 2) Neuro-Unification: to unify the neurons having the similar behavioral vectors. Instead of just pruning one of the similar neurons, the proposed method let the remaining neuron emulate the behavior of the pruned one. Therefore, the proposed method can reduce the number of neurons with small sacrifice of accuracy without retraining. Our method can be applied for compressing convolutional layers as well. In the convolutional layers, the behavior of each channel is encoded by its output feature maps, and channels whose behaviors can be well emulated by other channels are pruned and update the remaining weights. Through several experiments, we comfirmed that the proposed method performs better than the existing methods.
ER -