Knowledge distillation approaches can transfer information from a large network (teacher network) to a small network (student network) to compress and accelerate deep neural networks. This paper proposes a novel knowledge distillation approach called multi-knowledge distillation (MKD). MKD consists of two stages. In the first stage, it employs autoencoders to learn compact and precise representations of the feature maps (FM) from the teacher network and the student network, these representations can be treated as the essential of the FM, i.e., EFM. In the second stage, MKD utilizes multiple kinds of knowledge, i.e., the magnitude of individual sample's EFM and the similarity relationships among several samples' EFM to enhance the generalization ability of the student network. Compared with previous approaches that employ FM or the handcrafted features from FM, the EFM learned from autoencoders can be transferred more efficiently and reliably. Furthermore, the rich information provided by the multiple kinds of knowledge guarantees the student network to mimic the teacher network as closely as possible. Experimental results also show that MKD is superior to the-state-of-arts.

This paper proposes a deterministic pilot pattern placement optimization scheme based on the quantum genetic algorithm (QGA) which aims to improve the performance of sparse channel estimation in orthogonal frequency division multiplexing (OFDM) systems. By minimizing the mutual incoherence property (MIP) of the sensing matrix, the pilot pattern placement optimization is modeled as the solution of a combinatorial optimization problem. QGA is used to solve the optimization problem and generate optimized pilot pattern that can effectively avoid local optima traps. The simulation results demonstrate that the proposed method can generate a sensing matrix with a smaller MIP than a random search or the genetic algorithm (GA), and the optimized pilot pattern performs well for sparse channel estimation in OFDM systems.

This Letter proposes a autoencoder model supervised by semantic similarity for zero-shot learning. With the help of semantic similarity vectors of seen and unseen classes and the classification branch, our experimental results on two datasets are 7.3% and 4% better than the state-of-the-art on conventional zero-shot learning in terms of the averaged top-1 accuracy.

Software defect prediction (SDP) plays a vital role in allocating testing resources reasonably and ensuring software quality. When there are not enough labeled historical modules, considerable semi-supervised SDP methods have been proposed, and these methods utilize limited labeled modules and abundant unlabeled modules simultaneously. Nevertheless, most of them make use of traditional features rather than the powerful deep feature representations. Besides, the cost of the misclassification of the defective modules is higher than that of defect-free ones, and the number of the defective modules for training is small. Taking the above issues into account, we propose a cost-sensitive and sparse ladder network (CSLN) for SDP. We firstly introduce the semi-supervised ladder network to extract the deep feature representations. Besides, we introduce the cost-sensitive learning to set different misclassification costs for defective-prone and defect-free-prone instances to alleviate the class imbalance problem. A sparse constraint is added on the hidden nodes in ladder network when the number of hidden nodes is large, which enables the model to find robust structures of the data. Extensive experiments on the AEEEM dataset show that the CSLN outperforms several state-of-the-art semi-supervised SDP methods.

Internal user threats such as information leakage or system destruction can cause significant damage to the organization, however it is very difficult to prevent or detect this attack in advance. In this paper, we propose an anomaly-based insider threat detection method with local features and global statistics over the assumption that a user shows different patterns from regular behaviors during harmful actions. We experimentally show that our detection mechanism can achieve superior performance compared to the state of the art approaches for CMU CERT dataset.

This paper presents a deep learning-based non-intrusive speech intelligibility estimation method using bottleneck features of autoencoder. The conventional standard non-intrusive speech intelligibility estimation method, P.563, lacks intelligibility estimation performance in various noise environments. We propose a more accurate speech intelligibility estimation method based on long-short term memory (LSTM) neural network whose input and output are an autoencoder bottleneck features and a short-time objective intelligence (STOI) score, respectively, where STOI is a standard tool for measuring intrusive speech intelligibility with reference speech signals. We showed that the proposed method has a superior performance by comparing with the conventional standard P.563 and mel-frequency cepstral coefficient (MFCC) feature-based intelligibility estimation methods for speech signals in various noise environments.

The demand response is attracting attention to perform electric power load leveling. In this paper, we consider a power consumption reduction problem with an aggregator that requests electric power consumption reduction to consumers by allocating a part of its profit to them as an incentive. We formulate interactions among consumers as a game, where the incentive to each consumer is determined by his/her contribution to the total power consumption reduction, and the consumer determines his/her own reduction amount selfishly to maximize his/her payoff. The uniqueness of best responses of each consumer and an equilibrium condition of the game are also derived. By using numerical simulations, we show relationship among incentive allocation rate, realized total reduction amount through the game, and the aggregator's payoff for the cases with the for-profit and the nonprofit aggregator.

Embedded software developers assume the behavior of the environment when specifications are not available. However, developers may assume the behavior incorrectly, which may result in critical faults in the system. Therefore, it is important to detect the faults caused by incorrect assumptions. In this letter, we propose a log-based testing approach to detect the faults. First, we create a UML behavioral model to represent the assumed behavior of the environment, which is then transformed into a state model. Next, we extract the actual behavior of the environment from a log, which is then incorporated in the state model, resulting in a state model that represents both assumed and actual behaviors. Existing testing techniques based on the state model can be used to generate test cases from our state model to detect faults.

This paper constructs packet-oriented erasure correcting codes and their systematic forms for the distributed storage systems. The proposed codes are encoded by exclusive OR and bit-level shift operation. By the shift operation, the encoded packets are slightly longer than the source packets. This paper evaluates the extra length of the encoded packets, called overhead, and shows that the proposed codes have smaller overheads than the zigzag decodable codes, which are existing codes using bit-level shift operation and exclusive OR.

This paper proposes a deep neural network (DNN) based framework to address the problem of vector quantization (VQ) for high-dimensional data. The main challenge of applying DNN to VQ is how to reduce the binary coding error of the auto-encoder when the distribution of the coding units is far from binary. To address this problem, three fine-tuning methods have been adopted: 1) adding Gaussian noise to the input of the coding layer, 2) forcing the output of the coding layer to be binary, 3) adding a non-binary penalty term to the loss function. These fine-tuning methods have been extensively evaluated on quantizing speech magnitude spectra. The results demonstrated that each of the methods is useful for improving the coding performance. When implemented for quantizing 968-dimensional speech spectra using only 18-bit, the DNN-based VQ framework achieved an averaged PESQ of about 2.09, which is far beyond the capability of conventional VQ methods.

System logs are useful to understand the status of and detect faults in large scale networks. However, due to their diversity and volume of these logs, log analysis requires much time and effort. In this paper, we propose a log event anomaly detection method for large-scale networks without pre-processing and feature extraction. The key idea is to embed a large amount of diverse data into hidden states by using latent variables. We evaluate our method with 12 months of system logs obtained from a nation-wide academic network in Japan. Through comparisons with Kleinberg's univariate burst detection and a traditional multivariate analysis (i.e., PCA), we demonstrate that our proposed method achieves 14.5% higher recall and 3% higher precision than PCA. A case study shows detected anomalies are effective information for troubleshooting of network system faults.

Data-driven untying of a recursive autoencoder (RAE) is proposed for utterance intent classification for spoken dialogue systems. Although an RAE expresses a nonlinear operation on two neighboring child nodes in a parse tree in the application of spoken language understanding (SLU) of spoken dialogue systems, the nonlinear operation is considered to be intrinsically different depending on the types of child nodes. To reduce the gap between the single nonlinear operation of an RAE and intrinsically different operations depending on the node types, a data-driven untying of autoencoders using part-of-speech (PoS) tags at leaf nodes is proposed. When using the proposed method, the experimental results on two corpora: ATIS English data set and Japanese data set of a smartphone-based spoken dialogue system showed improved accuracies compared to when using the tied RAE, as well as a reasonable difference in untying between two languages.

Due to the need to protect personal information and the impracticality of exhaustive data collection, there is increasing need to deal with datasets with various levels of granularity, such as user-individual data and user-group data. In this study, we propose a new method for jointly analyzing multiple datasets with different granularity. The proposed method is a probabilistic model based on nonnegative matrix factorization, which is derived by introducing latent variables that indicate the high-resolution data underlying the low-resolution data. Experiments on purchase logs show that the proposed method has a better performance than the existing methods. Furthermore, by deriving an extension of the proposed method, we show that the proposed method is a new fundamental approach for analyzing datasets with different granularity.

Recently, multi-view dictionary learning technique has attracted lots of research interest. Although several multi-view dictionary learning methods have been addressed, they can be further improved. Most of existing multi-view dictionary learning methods adopt the l0 or l1-norm sparsity constraint on the representation coefficients, which makes the training and testing phases time-consuming. In this paper, we propose a novel multi-view dictionary learning approach named multi-view synthesis and analysis dictionaries learning (MSADL), which jointly learns multiple discriminant dictionary pairs with each corresponding to one view and containing a structured synthesis dictionary and a structured analysis dictionary. MSADL utilizes synthesis dictionaries to achieve class-specific reconstruction and uses analysis dictionaries to generate discriminative code coefficients by linear projection. Furthermore, we design an uncorrelation term for multi-view dictionary learning, such that the redundancy among synthesis dictionaries learned from different views can be reduced. Two widely used datasets are employed as test data. Experimental results demonstrate the efficiency and effectiveness of the proposed approach.

Traffic engineering refers to techniques to accommodate traffic efficiently by dynamically configuring traffic routes so as to adjust to changes in traffic. If traffic changes frequently and drastically, the interval of route reconfiguration should be short. However, with shorter intervals, obtaining traffic information is problematic. To calculate a suitable route, accurate traffic information of the whole network must be gathered. This is difficult in short intervals, owing to the overhead incurred to monitor and collect traffic information. In this paper, we propose a framework for traffic engineering in cases where only partial traffic information can be obtained in each time slot. The proposed framework is inspired by the human brain, and uses conditional probability to make decisions. In this framework, a controller is deployed to (1) obtain a limited amount of traffic information, (2) estimate and predict the probability distribution of the traffic, (3) configure routes considering the probability distribution of future predicted traffic, and (4) select traffic that should be monitored during the next period considering the system performance yielded by route reconfiguration. We evaluate our framework with a simulation. The results demonstrate that our framework improves the efficiency of traffic accommodation even when only partial traffic information is monitored during each time slot.

Conventional target recognition methods usually suffer from information-loss and target-aspect sensitivity when applied to radar high resolution range profile (HRRP) recognition. Thus, Effective establishment of robust and discriminatory feature representation has a significant performance improvement of practical radar applications. In this work, we present a novel feature extraction method, based on modified collaborative auto-encoder, for millimeter-wave radar HRRP recognition. The latent frame-specific weight vector is trained for samples in a frame, which contributes to retaining local information for different targets. Experimental results demonstrate that the proposed algorithm obtains higher target recognition accuracy than conventional target recognition algorithms.

An autoencoder has the potential ability to compress and decompress information. In this work, we consider the process of generating a stego-image from an original image and watermarks as compression, and the process of recovering the original image and watermarks from the stego-image as decompression. We propose embedder and extractor neural networks based on the autoencoder. The embedder network learns mapping from the DCT coefficients of the original image and a watermark to those of the stego-image. The extractor network learns mapping from the DCT coefficients of the stego-image to the watermark. Once the proposed neural network has been trained, the network can embed and extract the watermark into unlearned test images. We investigated the relation between the number of neurons and network performance by computer simulations and found that the trained neural network could provide high-quality stego-images and watermarks with few errors. We also evaluated the robustness against JPEG compression and found that, when suitable parameters were used, the watermarks were extracted with an average BER lower than 0.01 and image quality over 35 dB when the quality factor Q was over 50. We also investigated how to represent the watermarks in the stego-image by our neural network. There are two possibilities: distributed representation and sparse representation. From the results of investigation into the output of the stego layer (3rd layer), we found that the distributed representation emerged at an early learning step and then sparse representation came out at a later step.

Recent studies utilize multiple kernel learning to deal with incomplete-data problem. In this study, we introduce new methods that do not only complete multiple incomplete kernel matrices simultaneously, but also allow control of the flexibility of the model by parameterizing the model matrix. By imposing restrictions on the model covariance, overfitting of the data is avoided. A limitation of kernel matrix estimations done via optimization of an objective function is that the positive definiteness of the result is not guaranteed. In view of this limitation, our proposed methods employ the LogDet divergence, which ensures the positive definiteness of the resulting inferred kernel matrix. We empirically show that our proposed restricted covariance models, employed with LogDet divergence, yield significant improvements in the generalization performance of previous completion methods.

Dai-Yuan (DY) conjugate gradient method is an effective method for solving large-scale unconstrained optimization problems. In this paper, a new DY method, possessing a spectral conjugate parameter βk, is presented. An attractive property of the proposed method is that the search direction generated at each iteration is descent, which is independent of the line search. Global convergence of the proposed method is also established when strong Wolfe conditions are employed. Finally, comparison experiments on impulse noise removal are reported to demonstrate the effectiveness of the proposed method.

Hidden Singer is a television program in Korea. In the show, the original singer and four imitating singers sing a song in hiding behind a screen. The audience and TV viewers attempt to guess who the original singer is by listening to the singing voices. Usually, there are few correct answers from the audience, because the imitators are well trained and highly skilled. We propose a computerized system for distinguishing the original singer from the imitating singers. During the training phase, the system learns only the original singer's song because it is the one the audience has heard before. During the testing phase, the songs of five candidates are provided to the system and the system then determines the original singer. The system uses a 1-class authentication method, in which only a subject model is made. The subject model is used for measuring similarities between the candidate songs. In this problem, unlike other existing studies that require artist identification, we cannot utilize multi-class classifiers and supervised learning because songs of the imitators and the labels are not provided during the training phase. Therefore, we evaluate the performances of several 1-class learning algorithms to choose which one is more efficient in distinguishing an original singer from among highly skilled imitators. The experiment results show that the proposed system using the autoencoder performs better (63.33%) than other 1-class learning algorithms: Gaussian mixture model (GMM) (50%) and one class support vector machines (OCSVM) (26.67%). We also conduct a human contest to compare the performance of the proposed system with human perception. The accuracy of the proposed system is found to be better (63.33%) than the average accuracy of human perception (33.48%).