The search functionality is under construction.

The search functionality is under construction.

In this paper we analytically investigate the generalization performance of learning using correlated inputs in the framework of on-line learning with a statistical mechanical method. We consider a model composed of linear perceptrons with Gaussian noise. First, we analyze the case of the gradient method. We analytically clarify that the larger the correlation among inputs is or the larger the number of inputs is, the stricter the condition the learning rate should satisfy is, and the slower the learning speed is. Second, we treat the block orthogonal projection learning as an alternative learning rule and derive the theory. In a noiseless case, the learning speed does not depend on the correlation and is proportional to the number of inputs used in an update. The learning speed is identical to that of the gradient method with uncorrelated inputs. On the other hand, when there is noise, the larger the correlation among inputs is, the slower the learning speed is and the larger the residual generalization error is.

- Publication
- IEICE TRANSACTIONS on Fundamentals Vol.E91-A No.9 pp.2663-2670

- Publication Date
- 2008/09/01

- Publicized

- Online ISSN
- 1745-1337

- DOI
- 10.1093/ietfec/e91-a.9.2663

- Type of Manuscript
- PAPER

- Category
- Neural Networks and Bioengineering

The copyright of the original papers published on this site belongs to IEICE. Unauthorized use of the original or translated papers is prohibited. See IEICE Provisions on Copyright for details.

Copy

Chihiro SEKI, Shingo SAKURAI, Masafumi MATSUNO, Seiji MIYOSHI, "A Theoretical Analysis of On-Line Learning Using Correlated Examples" in IEICE TRANSACTIONS on Fundamentals,
vol. E91-A, no. 9, pp. 2663-2670, September 2008, doi: 10.1093/ietfec/e91-a.9.2663.

Abstract: In this paper we analytically investigate the generalization performance of learning using correlated inputs in the framework of on-line learning with a statistical mechanical method. We consider a model composed of linear perceptrons with Gaussian noise. First, we analyze the case of the gradient method. We analytically clarify that the larger the correlation among inputs is or the larger the number of inputs is, the stricter the condition the learning rate should satisfy is, and the slower the learning speed is. Second, we treat the block orthogonal projection learning as an alternative learning rule and derive the theory. In a noiseless case, the learning speed does not depend on the correlation and is proportional to the number of inputs used in an update. The learning speed is identical to that of the gradient method with uncorrelated inputs. On the other hand, when there is noise, the larger the correlation among inputs is, the slower the learning speed is and the larger the residual generalization error is.

URL: https://global.ieice.org/en_transactions/fundamentals/10.1093/ietfec/e91-a.9.2663/_p

Copy

@ARTICLE{e91-a_9_2663,

author={Chihiro SEKI, Shingo SAKURAI, Masafumi MATSUNO, Seiji MIYOSHI, },

journal={IEICE TRANSACTIONS on Fundamentals},

title={A Theoretical Analysis of On-Line Learning Using Correlated Examples},

year={2008},

volume={E91-A},

number={9},

pages={2663-2670},

abstract={In this paper we analytically investigate the generalization performance of learning using correlated inputs in the framework of on-line learning with a statistical mechanical method. We consider a model composed of linear perceptrons with Gaussian noise. First, we analyze the case of the gradient method. We analytically clarify that the larger the correlation among inputs is or the larger the number of inputs is, the stricter the condition the learning rate should satisfy is, and the slower the learning speed is. Second, we treat the block orthogonal projection learning as an alternative learning rule and derive the theory. In a noiseless case, the learning speed does not depend on the correlation and is proportional to the number of inputs used in an update. The learning speed is identical to that of the gradient method with uncorrelated inputs. On the other hand, when there is noise, the larger the correlation among inputs is, the slower the learning speed is and the larger the residual generalization error is.},

keywords={},

doi={10.1093/ietfec/e91-a.9.2663},

ISSN={1745-1337},

month={September},}

Copy

TY - JOUR

TI - A Theoretical Analysis of On-Line Learning Using Correlated Examples

T2 - IEICE TRANSACTIONS on Fundamentals

SP - 2663

EP - 2670

AU - Chihiro SEKI

AU - Shingo SAKURAI

AU - Masafumi MATSUNO

AU - Seiji MIYOSHI

PY - 2008

DO - 10.1093/ietfec/e91-a.9.2663

JO - IEICE TRANSACTIONS on Fundamentals

SN - 1745-1337

VL - E91-A

IS - 9

JA - IEICE TRANSACTIONS on Fundamentals

Y1 - September 2008

AB - In this paper we analytically investigate the generalization performance of learning using correlated inputs in the framework of on-line learning with a statistical mechanical method. We consider a model composed of linear perceptrons with Gaussian noise. First, we analyze the case of the gradient method. We analytically clarify that the larger the correlation among inputs is or the larger the number of inputs is, the stricter the condition the learning rate should satisfy is, and the slower the learning speed is. Second, we treat the block orthogonal projection learning as an alternative learning rule and derive the theory. In a noiseless case, the learning speed does not depend on the correlation and is proportional to the number of inputs used in an update. The learning speed is identical to that of the gradient method with uncorrelated inputs. On the other hand, when there is noise, the larger the correlation among inputs is, the slower the learning speed is and the larger the residual generalization error is.

ER -