Compressed sensing is a rapidly growing research field in signal and image processing, machine learning, statistics, and systems control. In this survey paper, we provide a review of the theoretical foundations of compressed sensing and present state-of-the-art algorithms for solving the corresponding optimization problems. Additionally, we discuss several practical applications of compressed sensing, such as group testing, sparse system identification, and sparse feedback gain design, and demonstrate their effectiveness through Python programs. This survey paper aims to contribute to the advancement of compressed sensing research and its practical applications in various scientific disciplines.

In this paper, we propose a method which enables us to control the variance of the coefficients of the LMS-type adaptive filters. In the method, each coefficient of the adaptive filter is modeled as an random variable with a Gaussian distribution, and its value is estimated as the mean value of the distribution. Besides, at each time, we check if the updated value exists within the predefined range of distribution. The update of a coefficient will be canceled when its updated value exceeds the range. We propose an implementation method which has similar formula as the Gaussian mixture model (GMM) widely used in signal processing and machine learning. The effectiveness of the proposed method is evaluated by the computer simulations.

Adaptive Volterra filters (AVFs) are usually used to identify nonlinear systems, such as loudspeaker systems, and ordinary adaptive algorithms can be used to update the filter coefficients of AVFs. However, AVFs require huge computational complexity even if the order of the AVF is constrained to the second order. Improving calculation efficiency is therefore an important issue for the real-time implementation of AVFs. In this paper, we propose a novel sub-band AVF with high calculation efficiency for second-order AVFs. The proposed sub-band AVF consists of four parts: input signal transformation for a single sub-band AVF, tap length determination to improve calculation efficiency, switching the number of sub-bands while maintaining the estimation accuracy, and an automatic search for an appropriate number of sub-bands. The proposed sub-band AVF can improve calculation efficiency for which the dominant nonlinear components are concentrated in any frequency band, such as loudspeakers. A simulation result demonstrates that the proposed sub-band AVF can realize higher estimation accuracy than conventional efficient AVFs.

The least mean squares (LMS) algorithm has been widely used for adaptive filtering because of easily implementing at a computational complexity of O(2N) where N is the number of taps. The drawback of the LMS algorithm is that its performance is sensitive to the scaling of the input. The normalized LMS (NLMS) algorithm solves this problem on the LMS algorithm by normalizing with the sliding-window power of the input; however, this normalization increases the computational cost to O(3N) per iteration. In this work, we derive a new formula to strictly perform the NLMS algorithm at a computational complexity of O(2N), that is referred to as the C-NLMS algorithm. The derivation of the C-NLMS algorithm uses the H∞ framework presented previously by one of the authors for creating a unified view of adaptive filtering algorithms. The validity of the C-NLMS algorithm is verified using simulations.

In this paper, the system identification problem from the high-dimensional input and output is considered. If the relationship between the features extracted from the data is represented as a linear time-invariant dynamical system, the input-output manifold learning method has shown to be a powerful tool for solving such a system identification problem. However, in the previous study, the system is assumed to be initially relaxed because the transfer function model is used for system representation. This assumption may not hold in several tasks. To handle the initially non-relaxed system, we propose the alternative approach of the input-output manifold learning with state space model for the system representation. The effectiveness of our proposed method is confirmed by experiments with synthetic data and motion capture data of human-human conversation.

In this letter, a new subband adaptive filter (SAF) which is robust against impulsive noise in system identification is presented. To address the vulnerability of adaptive filters based on the L2-norm optimization criterion to impulsive noise, the robust SAF (R-SAF) comes from the L1-norm optimization criterion with a constraint on the energy of the weight update. Minimizing L1-norm of the a posteriori error in each subband with a constraint on minimum disturbance gives rise to robustness against impulsive noise and the capable convergence performance. Simulation results clearly demonstrate that the proposal, R-SAF, outperforms the classical adaptive filtering algorithms when impulsive noise as well as background noise exist.

A new type of the affine projection (AP) algorithms which incorporates the sparsity condition of a system is presented. To exploit the sparsity of the system, a weighted l1-norm regularization is imposed on the cost function of the AP algorithm. Minimizing the cost function with a subgradient calculus and choosing two distinct weightings for l1-norm, two stochastic gradient based sparsity regularized AP (SR-AP) algorithms are developed. Experimental results show that the SR-AP algorithms outperform the typical AP counterparts for identifying sparse systems.

This letter deals with a system identification problem with unknown model order, which can be formulated as the matrix rank minimization problem by applying the subspace identification method. A sequential rank minimization algorithm is provided by modifying the null space based alternating optimization (NSAO) algorithm, and a model order identification algorithm is proposed. Numerical examples show that the proposed sequential algorithm can adaptively identify the model order of switched systems whose model order changes.

For an adaptive system identification filter with a stochastic input signal, a coefficient vector updated with an NLMS algorithm converges in the sense of ensemble average and the expected convergence vector has been revealed. When the input signal is periodic, the convergence of the adaptive filter coefficients has also been proved. However, its convergence vector has not been revealed. In this paper, we derive the convergence vector of adaptive filter coefficients updated with the NLMS algorithm in system identification for deterministic sinusoidal inputs. Firstly, we derive the convergence vector when a disturbance does not exist. We show that the derived convergence vector depends only on the initial vector and the sinusoidal frequencies, and it is independent of the step-size for adaptation, sinusoidal amplitudes, and phases. Next, we derive the expected convergence vector when the disturbance exists. Simulation results support the validity of the derived convergence vectors.

This letter proposes a simple heuristic to identify the discrete-time switched autoregressive exogenous (SARX) systems. The goal of the identification is to identify the switching sequence and the system parameters of all submodels simultaneously. In this letter the SARX system identification problem is formulated as the l0 norm minimization problem, and an iterative algorithm is proposed by applying the reweighted least squares technique. Although the proposed algorithm is heuristic, the numerical examples show its efficiency and robustness for noise.

The hyper H∞ filter derived in our previous work provides excellent convergence, tracking, and robust performances for linear time-varying system identification. Additionally, a fast algorithm of the hyper H∞ filter, called the fast H∞ filter, is successfully developed so that identification of linear system with impulse response of length N is performed at a computational complexity of O(N). The gain matrix of the fast filter is recursively calculated through estimating the forward and backward linear prediction coefficients of an input signal. This suggests that the fast H∞ filter may be applicable to linear prediction of the signal. On the other hand, an alternative fast version of the hyper H∞ filter, called the J-fast H∞ filter, is derived using a J-unitary array form, which is amenable to parallel processing. However, the J-fast H∞ filter explicitly includes no linear prediction of input signals in the algorithm. This work reveals that the forward and backward linear prediction coefficients and error powers of the input signal are indeed included in the recursive variables of the J-fast H∞ filter. These findings are verified by computer simulations.

A new compact form of the sliding window recursive least squares (SWRLS) algorithm, the I-SWRLS algorithm, is derived using an indefinite matrix. The resultant algorithm has a form similar to that of the traditional recursive least squares (RLS) algorithm, and is more computationally efficient than the conventional SWRLS algorithm including two Riccati equations. Furthermore, a computationally reduced version of the I-SWRLS algorithm is developed utilizing a shift property of the correlation matrix of input data. The resulting fast algorithm reduces the computational complexity from O(N2) to O(N) per iteration when the filter length (tap number) is N, but retains the same tracking performance as the original algorithm. This fast algorithm is much easier to implement than the existing SWC FTF algorithms.

The fast H∞ filter is developed by one of the authors, and its practical use in industries is expected. This paper derives a linear propagation model of numerical errors in the recursive variables of the fast H∞ filter, and then theoretically analyzes the stability of the filter. Based on the analyzed results, a numerical stabilization method of the fast H∞ filter is proposed with the error feedback control in the backward prediction. Also, the effectiveness of the stabilization method is verified using numerical examples.

The proportionate normalized least mean square algorithm (PNLMS) greatly improves the convergence of the sparse impulse response. It exploits the shape of the impulse response to decide the proportionate step gain for each coefficient. This is not always suitable. Actually, the proportionate step gain should be determined according to the difference between the current estimate of the coefficient and its optimal value. Based on this idea, an approach is proposed to determine the proportionate step gain. The proposed approach can improve the convergence of proportionate adaptive algorithms after a fast initial period. It even behaves well for the non-sparse impulse response. Simulations verify the effectiveness of the proposed approach.

This paper presents a new hierarchical mode segmentation of the observed driving behavioral data based on the multi-level abstraction of the underlying dynamics. By synthesizing the ideas of a feature vector definition revealing the dynamical characteristics and an unsupervised clustering technique, the hierarchical mode segmentation is achieved. The identified mode can be regarded as a kind of symbol in the abstract model of the behavior. Second, the grammatical inference technique is introduced to develop the context-dependent grammar of the behavior, i.e., the symbolic dynamics of the human behavior. In addition, the behavior prediction based on the obtained symbolic model is performed. The proposed framework enables us to make a bridge between the signal space and the symbolic space in the understanding of the human behavior.

We have proposed a noise reduction method based on a noise reconstruction system (NRS). The NRS uses a linear prediction error filter (LPEF) and a noise reconstruction filter (NRF) which estimates background noise by system identification. In case a fixed step size for updating tap coefficients of the NRF is used, it is difficult to reduce background noise while maintaining the high quality of enhanced speech. In order to solve the problem, a variable step size is proposed. It makes use of cross-correlation between an input signal and an enhanced speech signal. In a speech section, a variable step size becomes small so as not to estimate speech, on the other hand, large to track the background noise in a non-speech section.

This paper proposes a new adaptive filter algorithm for system identification by using an independent component analysis (ICA) technique, which separates the signal from noisy observation under the assumption that the signal and noise are independent. We first introduce an augmented state-space expression of the observed signal, representing the problem in terms of ICA. By using a nonparametric Parzen window density estimator and the stochastic information gradient, we derive an adaptive algorithm to separate the noise from the signal. The proposed ICA-based algorithm does not suppress the noise in the least mean square sense but to maximize the independence between the signal part and the noise. The computational complexity of the proposed algorithm is compared with that of the standard NLMS algorithm. The stationary point of the proposed algorithm is analyzed by using an averaging method. We can directly use the new ICA-based algorithm in an acoustic echo canceller without double-talk detector. Some simulation results are carried out to show the superiority of our ICA method to the conventional NLMS algorithm.

A hybrid evolutionary neuro-fuzzy system (HENFS) is proposed in this paper, where the weighted Gaussian function (WGF) is used as the membership function for improved premise construction. With the WGF, different types of the membership functions (MFs) can be accommodated in the rule base of HENFS. A new hybrid algorithm of random optimization (RO) algorithm incorporated with the least square estimation (LSE) is presented. Based on the hybridization of RO-LSE, the proposed soft-computing approach overcomes the disadvantages of other widely used algorithms. The proposed HENFS is applied to chaos time series identification and industrial process modeling to verify its feasibility. Through the illustrations and comparisons the impressive performances for unknown system identification can be observed.

A new system identification based method has been proposed for accurate estimation of vocal tract parameters. An often encountered problem in using the conventional linear prediction analysis is due to the harmonic structure of the excitation source of voiced speech. This harmonic characteristic is coupled with the estimation of autoregressive (AR) coefficients that results in difficulties in estimating the vocal tract filter. This paper models the effective voice source from the residual obtained through the covariance analysis in the first-pass which is then used as input to the second-pass least-square analysis. A better source-filter separation is thus achieved. The formant frequencies and corresponding bandwidths obtained using the proposed method for synthetic vowels are found to be accurate up to a factor of more than three (in percent) compared to the conventional method. Since the source characteristic is taken into account, local variations due to the positioning of analysis window are reduced significantly. The validity of the proposed method is also examined by inspecting the spectra obtained from natural vowel sounds uttered by high-pitched female speaker.

In our previous work, the hyper H∞ filter is developed for tracking of unknown time-varying systems. Additionally, a fast algorithm, called the fast H∞ filter, of the hyper H∞ filter is derived on condition that the observation matrix has a shifting property. This algorithm has a computational complexity of O(N) where N is the dimension of the state vector. However, there still remains a possibility of deriving alternative forms of the hyper H∞ filter. In this work, a fast J-unitary form of the hyper H∞ filter is derived, providing a new H∞ fast algorithm, called the J-fast H∞ filter. The J-fast H∞ filter possesses a computational complexity of O(N), and the resulting algorithm is very amenable to parallel processing. The validity and performance of the derived algorithm are confirmed by computer simulations.