In D.O.A. estimation, identification of the signal and the noise subspaces plays an essential role. This identification process was traditionally achieved by the eigenvalue decomposition (EVD) of the spatial correlation matrix of observations or the generalized eigenvalue decomposition (GEVD) of the spatial correlation matrix of observations with respect to that of an observation noise. The framework based on the GEVD is not always an extension of that based on the EVD, since the GEVD is not applicable to the noise-free case which can be resolved by the framework based on the EVD. Moreover, they are not applicable to the case in which the spatial correlation matrix of the noise is singular. Recently, a quotient-singular-value-decomposition-based framework, that can be applied to problems with singular noise correlation matrices, is introduced for noise reduction. However, this framework also can not treat the noise-free case. Thus, we do not have a unified framework of the identification of these subspaces. In this paper, we show that a unified framework of the identification of these subspaces is realized by the concept of proper and improper eigenspaces of the spatial correlation matrix of the noise with respect to that of observations.

The solution of the standard 2-norm-based multiple kernel regression problem and the theoretical limit of the considered model space are discussed in this paper. We prove that 1) The solution of the 2-norm-based multiple kernel regressor constructed by a given training data set does not generally attain the theoretical limit of the considered model space in terms of the generalization errors, even if the training data set is noise-free, 2) The solution of the 2-norm-based multiple kernel regressor is identical to the solution of the single kernel regressor under a noise free setting, in which the adopted single kernel is the sum of the same kernels used in the multiple kernel regressor; and it is also true for a noisy setting with the 2-norm-based regularizer. The first result motivates us to develop a novel framework for the multiple kernel regression problems which yields a better solution close to the theoretical limit, and the second result implies that it is enough to use the single kernel regressors with the sum of given multiple kernels instead of the multiple kernel regressors as long as the 2-norm based criterion is used.

For the last few decades, learning with multiple kernels, represented by the ensemble kernel regressor and the multiple kernel regressor, has attracted much attention in the field of kernel-based machine learning. Although their efficacy was investigated numerically in many works, their theoretical ground is not investigated sufficiently, since we do not have a theoretical framework to evaluate them. In this paper, we introduce a unified framework for evaluating kernel regressors with multiple kernels. On the basis of the framework, we analyze the generalization errors of the ensemble kernel regressor and the multiple kernel regressor, and give a sufficient condition for the ensemble kernel regressor to outperform the multiple kernel regressor in terms of the generalization error in noise-free case. We also show that each kernel regressor can be better than the other without the sufficient condition by giving examples, which supports the importance of the sufficient condition.

A fast cross-validation algorithm for model selection in kernel ridge regression problems is proposed, which is aiming to further reduce the computational cost of the algorithm proposed by An et al. by eigenvalue decomposition of a Gram matrix.

In order to produce precise enclosures from a multi-dimensional interval vector, we introduce a sharp interval sub-dividing condition for optimization algorithms. By utilizing interval inclusion properties, we also enhance the sampling of an upper bound for effective use in the interval quadratic method. This has resulted in an improvement in the algorithm for the unconstrained optimization problem by Hansen in 1992.

Practical image restoration filters usually include a parameter that controls regularizability, trade-off between fidelity of a restored image and smoothness of it, and so on. Many criteria for choosing such a parameter have been proposed. However, the relation between these criteria and the squared error of a restored image, which is usually used to evaluate the restoration performance, has not been theoretically substantiated. Sugiyama and Ogawa proposed the subspace information criterion (SIC) for model selection of supervised learning problems and showed that the SIC is an unbiased estimator of the expected squared error between the unknown model function and an estimated one. They also applied it to restoration of images. However, we need an unbiased estimator of the unknown original image to construct the criterion, so it can not be used for general situations. In this paper, we present a modified version of the SIC as a new criterion for choosing a parameter of image restoration filters. Some numerical examples are also shown to verify the efficacy of the proposed criterion.

We consider the situation that plural degraded images are obtained. When no prior knowledge about original images are known, these images are individually restored by an optimum restoration filter, for example, by Wiener Filter or by Projection Filter. If correlations between original images are obtained, some restoration filters based on Wiener Filter or Projection Filter are proposed. In this paper, we deal with the case that some pixels or some parts of original images overlap, and propose a restoration method using a formulae for overlapping. The method is based on Partial Projection Filter. Moreover, we confirm an efficacy of the proposed method by numerical examples.

We consider a property about a result of non-negative matrix factorization under a parallel moving of data points. The shape of a cloud of original data points and that of data points moving parallel to a vector are identical. Thus it is sometimes required that the coefficients to basis vectors of both data points are also identical from the viewpoint of classification. We show a necessary and sufficient condition for such an invariance property under a translation of the data points.

The knowledge of a good enclosure of the range of a function over small interval regions allows us to avoid convergence of optimization algorithms to a non-global point(s). We used interval slopes f[X,x] to check for monotonicity and integrated their derivative forms g[X,x], x X by quadratic and Newton methods to obtain narrow enclosures. In order to include boundary points in the search for the optimum point(s), we expanded the initial box by a small width on each dimension. These procedures resulted in an improvement in the algorithm proposed by Hansen.

It is widely known that the family of projection filters includes the generalized inverse filter, and that the family of parametric projection filters includes parametric generalized projection filters. However, relations between the family of parametric projection filters and constrained least squares filters are not sufficiently clarified. In this paper, we consider relations between parameter estimation and image restoration by these families. As a result, we show that the restored image by the family of parametric projection filters is a maximum penalized likelihood estimator, and that it agrees with the restored image by constrained least squares filter under some suitable conditions.

Optimum filters for an image restoration are formed by a degradation operator, a covariance operator of original images, and one of noise. However, in a practical image restoration problem, the degradation operator and the covariance operators are estimated on the basis of empirical knowledge. Thus, it appears that they differ from the true ones. When we restore a degraded image by an optimum filter belonging to the family of Projection Filters and Parametric Projection Filters, it is shown that small deviations in the degradation operator and the covariance matrix can cause a large deviation in a restored image. In this paper, we propose new optimum filters based on the regularization method called the family of Regularized Projection Filters, and show that they are stable to deviations in operators. Moreover, some numerical examples follow to confirm that our description is valid.

In terms of the formulation of the optimality, image restoration filters can be divided into two streams. One is formulated as an optimization problem in which the fidelity of a restored image is indirectly evaluated, and the other is formulated as an optimization problem based on a direct evaluation. Originally, the formulation of the optimality and the solutions derived from the formulation are identical each other. However in many studies adopting the former stream, an arbitrary choice of a solution without a mathematical ground passes unremarked. In this paper, we discuss the relation between the formulation of the optimality and the solution derived from the formulation from a mathematical point of view, and investigate the relation between a direct style formulation and an indirect one. Through these analyses, we show that the both formulations yield the identical filter in practical situations.

In signal restoration problems, we expect to improve the restoration performance with a priori information about unknown target signals. In this paper, the parametric Wiener filter with linear constraints for unknown target signals is discussed. Since the parametric Wiener filter is usually defined as the minimizer of the criterion not for the unknown target signal but for the filter, it is difficult to impose constraints for the unknown target signal in the criterion. To overcome this difficulty, we introduce a criterion for the parametric Wiener filter defined for the unknown target signal whose minimizer is equivalent to the solution obtained by the original formulation. On the basis of the newly obtained criterion, we derive a closed-form solution for the parametric Wiener filter with linear constraints.

In this paper, we introduce a self-constructive Normalized Gaussian Network (NGnet) for online learning tasks. In online tasks, data samples are received sequentially, and domain knowledge is often limited. Then, we need to employ learning methods to the NGnet that possess robust performance and dynamically select an accurate model size. We revise a previously proposed localized forgetting approach for the NGnet and adapt some unit manipulation mechanisms to it for dynamic model selection. The mechanisms are improved for more robustness in negative interference prone environments, and a new merge manipulation is considered to deal with model redundancies. The effectiveness of the proposed method is compared with the previous localized forgetting approach and an established learning method for the NGnet. Several experiments are conducted for a function approximation and chaotic time series forecasting task. The proposed approach possesses robust and favorable performance in different learning situations over all testbeds.

A lot of optimum filters have been proposed for an image restoration problem. Parametric filter, such as Parametric Wiener Filter, Parametric Projection Filter, or Parametric Partial Projection Filter, is often used because it requires to calculate a generalized inverse of one operator. These optimum filters are formed by a degradation operator, a covariance operator of noise, and one of original images. In practice, these operators are estimated based on empirical knowledge. Unfortunately, it happens that such operators differ from the true ones. In this paper, we show the unified formulae of inducing them to clarify their common properties. Moreover, we investigate their properties for perturbation of a degradation operator, a covariance operator of noise, and one of original images. Some numerical examples follow to confirm that our description is valid.

The solution of the ordinary kernel ridge regression, based on the squared loss function and the squared norm-based regularizer, can be easily interpreted as a stochastic linear estimator by considering the autocorrelation prior for an unknown true function. As is well known, a stochastic affine estimator is one of the simplest extensions of the stochastic linear estimator. However, its corresponding kernel regression problem is not revealed so far. In this paper, we give a formulation of the kernel regression problem, whose solution is reduced to a stochastic affine estimator, and also give interpretations of the formulation.