Recently, neural networks (NNs) have been extensively applied to many signal processing problem due to their robust abilities to form complex decision regions. In particular, neural networks add flexibility to the design of equalizers for digital communication systems. Recurrent neural network (RNN) is a kind of neural network with one or more feedback loops, whereas self-organizing map (SOM) is characterized by the formation of a topographic map of the input patterns in which the spatial locations (i.e., coordinates) of the neurons in the lattice are indicative of intrinsic statistical features contained in the input patterns. In this paper, we propose a novel receiver structure by combining adaptive RNN equalizer with a SOM detector under serious ISI and nonlinear distortion in QAM system. According to the theoretical analysis and computer simulation results, the performance of the proposed scheme is shown to be quite effective in channel equalization under nonlinear distortion.

Detection of nonlinearly distorted signals is an essential problem in telecommunications. Recently, neural network combined conventional equalizer has been used to improve the performance especially in compensating for nonlinear distortions. In this paper, the self-organizing map (SOM) combined with the conventional symbol-by-symbol detector is used as an adaptive detector after the output of the decision feedback equalizer (DFE), which updates the decision levels to follow up the nonlinear distortions. In the proposed scheme, we use the box distance to define the neighborhood of the winning neuron of the SOM algorithm. The error performance has been investigated in both 16 QAM and 64 QAM systems with nonlinear distortions. Simulation results have shown that the system performance is remarkably improved by using SOM detector compared with the conventional DFE scheme.