In the present paper, we attempt to show that the information about input patterns must be as small as possible for improving the generalization performance under the condition that the network can produce targets with appropriate accuracy. The information is defined with respect to the hidden unit activity and we suppose that the hidden unit has a crucial role to store the information content about input patterns. The information is defined by the difference between uncertainty of the hidden unit at the initial stage of the learning and the uncertainty of the hidden unit at the final stage of the learning. After having formulated an update rule for the information minimization, we applied the method to a problem of language acquisition: the inference of the past tense forms of regular and irregular verbs. Experimental results confirmed that by our method, the information was significantly decreased and the generalization performance was greatly improved.

In this paper, we propose a method, called Kernel Hidden Unit Analysis, to reduce the network size. The kernel hidden unit analysis in composed of two principal components: T-component and S-component. The T-component transforms original networks into the networks which can easily be simplified. The S-component is used to select kernel units in the networks and construct kernel networks with kernel units. For the T-component, an entropy function is used, which is defined with respect to the state of the hidden units. In a process of entropy minimization, multiple strongly inhibitory connections are to be generated, which tend to turn off as many units as possible. Thus, some major hidden units can easily be extracted. Concerning the S-component, we use the relevance and the variance of input-hidden connections and detect the kernel hidden units for constructing the kernel network. Applying the kernel hidden unit analysis to the symmetry problem and autoencoders, we perfectly succeeded in obtaining kernel networks with small entropy, that is, small number of hidden units.

The purpose of this paper is to show that a new type of information-theoretic learning method called “potential learning” can be used to detect and extract important tweets among a great number of redundant ones. In the experiment, we used a dataset of 10,000 tweets, among which there existed only a few important ones. The experimental results showed that the new method improved overall classification accuracy by correctly identifying the important tweets.