The search functionality is under construction.

The search functionality is under construction.

The natural gradient descent is an optimization method for real-valued neural networks that was proposed from the viewpoint of information geometry. Here, we present an extension of the natural gradient descent to complex-valued neural networks. Our idea is to use the Hermitian extension of the Fisher information matrix. Moreover, we generalize the projected natural gradient (PRONG), which is a fast natural gradient descent algorithm, to complex-valued neural networks. We also consider the advantage of complex-valued neural networks over real-valued neural networks. A useful property of complex numbers in the complex plane is that the rotation is simply expressed by the multiplication. By focusing on this property, we construct the output function of complex-valued neural networks, which is invariant even if the input is changed to its rotated value. Then, our complex-valued neural network can learn rotated data without data augmentation. Finally, through simulation of online character recognition, we demonstrate the effectiveness of the proposed approach.

- Publication
- IEICE TRANSACTIONS on Fundamentals Vol.E102-A No.12 pp.1988-1996

- Publication Date
- 2019/12/01

- Publicized

- Online ISSN
- 1745-1337

- DOI
- 10.1587/transfun.E102.A.1988

- Type of Manuscript
- PAPER

- Category
- Neural Networks and Bioengineering

Jun-ichi MUKUNO

Kogakuin University

Hajime MATSUI

Toyota Technological Institute

The copyright of the original papers published on this site belongs to IEICE. Unauthorized use of the original or translated papers is prohibited. See IEICE Provisions on Copyright for details.

Copy

Jun-ichi MUKUNO, Hajime MATSUI, "Natural Gradient Descent of Complex-Valued Neural Networks Invariant under Rotations" in IEICE TRANSACTIONS on Fundamentals,
vol. E102-A, no. 12, pp. 1988-1996, December 2019, doi: 10.1587/transfun.E102.A.1988.

Abstract: The natural gradient descent is an optimization method for real-valued neural networks that was proposed from the viewpoint of information geometry. Here, we present an extension of the natural gradient descent to complex-valued neural networks. Our idea is to use the Hermitian extension of the Fisher information matrix. Moreover, we generalize the projected natural gradient (PRONG), which is a fast natural gradient descent algorithm, to complex-valued neural networks. We also consider the advantage of complex-valued neural networks over real-valued neural networks. A useful property of complex numbers in the complex plane is that the rotation is simply expressed by the multiplication. By focusing on this property, we construct the output function of complex-valued neural networks, which is invariant even if the input is changed to its rotated value. Then, our complex-valued neural network can learn rotated data without data augmentation. Finally, through simulation of online character recognition, we demonstrate the effectiveness of the proposed approach.

URL: https://global.ieice.org/en_transactions/fundamentals/10.1587/transfun.E102.A.1988/_p

Copy

@ARTICLE{e102-a_12_1988,

author={Jun-ichi MUKUNO, Hajime MATSUI, },

journal={IEICE TRANSACTIONS on Fundamentals},

title={Natural Gradient Descent of Complex-Valued Neural Networks Invariant under Rotations},

year={2019},

volume={E102-A},

number={12},

pages={1988-1996},

abstract={The natural gradient descent is an optimization method for real-valued neural networks that was proposed from the viewpoint of information geometry. Here, we present an extension of the natural gradient descent to complex-valued neural networks. Our idea is to use the Hermitian extension of the Fisher information matrix. Moreover, we generalize the projected natural gradient (PRONG), which is a fast natural gradient descent algorithm, to complex-valued neural networks. We also consider the advantage of complex-valued neural networks over real-valued neural networks. A useful property of complex numbers in the complex plane is that the rotation is simply expressed by the multiplication. By focusing on this property, we construct the output function of complex-valued neural networks, which is invariant even if the input is changed to its rotated value. Then, our complex-valued neural network can learn rotated data without data augmentation. Finally, through simulation of online character recognition, we demonstrate the effectiveness of the proposed approach.},

keywords={},

doi={10.1587/transfun.E102.A.1988},

ISSN={1745-1337},

month={December},}

Copy

TY - JOUR

TI - Natural Gradient Descent of Complex-Valued Neural Networks Invariant under Rotations

T2 - IEICE TRANSACTIONS on Fundamentals

SP - 1988

EP - 1996

AU - Jun-ichi MUKUNO

AU - Hajime MATSUI

PY - 2019

DO - 10.1587/transfun.E102.A.1988

JO - IEICE TRANSACTIONS on Fundamentals

SN - 1745-1337

VL - E102-A

IS - 12

JA - IEICE TRANSACTIONS on Fundamentals

Y1 - December 2019

AB - The natural gradient descent is an optimization method for real-valued neural networks that was proposed from the viewpoint of information geometry. Here, we present an extension of the natural gradient descent to complex-valued neural networks. Our idea is to use the Hermitian extension of the Fisher information matrix. Moreover, we generalize the projected natural gradient (PRONG), which is a fast natural gradient descent algorithm, to complex-valued neural networks. We also consider the advantage of complex-valued neural networks over real-valued neural networks. A useful property of complex numbers in the complex plane is that the rotation is simply expressed by the multiplication. By focusing on this property, we construct the output function of complex-valued neural networks, which is invariant even if the input is changed to its rotated value. Then, our complex-valued neural network can learn rotated data without data augmentation. Finally, through simulation of online character recognition, we demonstrate the effectiveness of the proposed approach.

ER -