Backpressure Learning-Based Data Transmission Reliability-Aware Self-Organizing Networking for Power Line Communication in Distribution Network

Zhan SHI

Summary :

Power line communication (PLC) provides a flexible-access, wide-distribution, and low-cost communication solution for distribution network services. However, the PLC self-organizing networking in distribution network faces several challenges such as diversified data transmission requirements guarantee, the contradiction between long-term constraints and short-term optimization, and the uncertainty of global information. To address these challenges, we propose a backpressure learning-based data transmission reliability-aware self-organizing networking algorithm to minimize the weighted sum of node data backlogs under the long-term transmission reliability constraint. Specifically, the minimization problem is transformed by the Lyapunov optimization and backpressure algorithm. Finally, we propose a backpressure and data transmission reliability-aware state-action-reward-state-action (SARSA)-based self-organizing networking strategy to realize the PLC networking optimization. Simulation results demonstrate that the proposed algorithm has superior performances of data backlogs and transmission reliability.

• Publication: IEICE TRANSACTIONS on Fundamentals Vol.E107-A No.8 pp.1076-1084
• Publication Date: 2024/08/01
• Publicized: 2024/01/15
• Online ISSN: 1745-1337
• DOI: 10.1587/transfun.2023EAP1154
• Type of Manuscript: PAPER
• Category: Systems and Control

1.  Introduction

The utilization of power line communication (PLC) technology has emerged as an effective means of addressing the issue of information exchange within the “last kilometer” of distribution networks [1]-[3]. The benefits of PLC include its adaptability, wide distribution, and low construction cost, rendering it a popular choice for distribution network services such as power consumption information collection and distributed energy access [4]-[6]. The energy supply for PLC carrier modules is reliant on the power grid, whereas the communication of electric equipment is dependent on the PLC network. Consequently, distribution networks and PLC networks are interdependent and exhibit a deep coupling [7]-[9]. With the advent of the power internet of things (PIoT), the introduction of various novel distribution devices, and a high proportion of new energy access to the distribution network, there has been a surge in the dynamic switching of electric equipment, leading to increased complexity and dynamism in the scale and structure of the distribution network topology [10], [11]. Furthermore, the various noise generated by the operation of electric equipment can cause changes in channel states, which pose a threat to the reliability of PLC data transmission and the dependable operation of the distribution network [12], [13]. Therefore, there is a pressing need to enhance the existing PLC networking technology. Nonetheless, several technical challenges must be overcome to achieve this goal.

Firstly, the current networking optimization strategies do not align with the differentiated data importance requirements of communication nodes within the network, which carry distribution network services with varying data importance levels. The conventional networking approach overlooks the significance of diverse data importance, thereby failing to cater to the data transmission demands of distinct distribution network services [14], [15]. Secondly, short-term optimization based on limited information may result in long-term performance degradation. Specifically, short-term networking decisions aimed at enhancing the short-term performance of data queue backlogs may compromise the long-term transmission reliability performance [16], [17]. Lastly, global information, including PLC state, network congestion, and dynamic changes in network topology, is subject to uncertainty. Consequently, optimizing networking decisions under such uncertain global information remains an open issue [18], [19].

To support the provision of wide-area multi-scale new distribution network services, the optimization of self-organizing networking in PLC has made incremental advancements. In [20], Yu et al. proposed a multiple-input and multiple-output (MIMO) PLC networking scheme to improve transmission reliability and minimize networking costs. In [21], Yan et al. proposed a particle swarm optimization-based networking coordination algorithm for a large complex distribution communication network, the objective of which is to minimize the power wasting of distribution network devices. However, these works did not account for the differentiated importance requirements of data during the networking optimization process. In [22], Sung et al. investigated an opportunistic routing-based networking method for PLC-access networks, which minimizes the packet transmission delay. However, the above work ignores the contradiction between long-term constraints and short-term performance optimization. Mauro et al. [23] devised a Geo-routing-based algorithm to optimize networking decisions within the constraints of PLC path energy and delay. However, the above work relies on certain global information, rendering it less suitable for accommodating the dynamic changes in PLC channel state and distribution network topology.

In this article, we propose a backpressure learning-based data transmission reliability-aware self-organizing networking algorithm. Initially, we establish a comprehensive PLC self-organizing networking model to serve as the foundation for our optimization framework. The optimization objective is to minimize the weighted sum of node data backlog while adhering to the long-term constraint of ensuring reliable data transmission. To achieve this, we leverage the backpressure algorithm and Lyapunov optimization to transform the optimization problem. The networking decision is optimized by backpressure and data transmission reliability-aware state-action-reward-state-action (SARSA)-based self-organizing networking strategy. Finally, the performance of the proposed algorithm is verified by simulation. The main contributions are summarized as follows.

\(\bullet\) Guarantee of data transmission performance with differentiated data importance: We consider the different levels of data importance and reliability constraints for different nodes, and minimize the node data backlog using a weighted sum objective, where the weights correspond to the data importance levels. This allows nodes with higher data importance to enhance their performance in terms of data backlog reduction, data transmission volume, and transmission reliability, fulfilling their data transmission demands.

\(\bullet\) Data transmission reliability awareness: We use Lyapunov optimization to convert the long-term transmission reliability constraint into a virtual queue stability problem. Data transmission reliability awareness is achieved by dynamically adapting the networking decisions based on the virtual queue backlog of the long-term transmission reliability constraint.

\(\bullet\) Backpressure learning-based self-organizing networking: The proposed algorithm calculates the node queue backlog difference based on the backpressure algorithm to assess the congestion level of the next-hop node. Then, the node queue backlog difference is incorporated in the penalty value of SARSA to enhance its convergence and learning capabilities for the self-organizing networking strategy.

2.  System Model

Figure 1 illustrates the PLC self-organizing networking model. The PLC network and the power grid are deeply coupled. On one hand, the PLC channel shares the power transmission channel of the distribution network, which makes it susceptible to the electromagnetic interference from electric equipment. Moreover, the grid connection and islanded operation of electric equipment cause a topology change in the communication network. On the other hand, the PLC network collects and transmits service data such as operation state information of electric equipment, which influences the stability of the distribution network.

Fig. 1  PLC self-organizing networking model.

Define the topology of PLC network as \({\mathcal{G}}=\{{\mathcal{V}},{\mathcal{\varepsilon}}\}\), where \({\mathcal{V}}=\{1,v_i,\cdots,v_I\}\) indicates the set of PLC communication nodes and \(\boldsymbol{\mathcal{\varepsilon}}=\{E_{i,j}|v_i,v_j\in\mathcal{V}\}\) indicates the set of PLC communication links. A quasi-static time slot model is adopted in this paper. The overall networking optimization time is divided into \(T\) time slots with the same length \(\tau\), the set of which is \({\mathcal{T}}=\{1,\cdots,t,\cdots,T\}\). The network state remains constant within a time slot and changes dynamically across different time slots. Define the networking decision indicator variables as \(x_{i,j}(t)\in\{0,1\}\), where \(x_{i,j}(t)=1\) indicates the node \(v_i\) selects node \(v_j\) as the next-hop transmission node, otherwise \(x_{i,j}(t)=0\). At the same time, there must be a link between \(v_i\) and \(v_j\), i.e., \(E_{i,j}\in\varepsilon\), and \(v_i\) can only select one next-hop node for data transmission at each time slot due to the PLC communication link characteristics and data reception capability. The constraint is expressed as

\[\begin{align} \sum_{j=1}^Ix_{i,j}(t)\le1. \tag{1} \end{align}\]

The distribution network has distinct multi-level topologies, mainly bus type, star type, and tree type. Each node assigns its networking level based on the shortest distance to the aggregation node. We define the shortest distance between \(v_i\) and the aggregation node as \(\rho_i\) using the Dijkstra algorithm. To enhance the networking efficiency, the next-hop node of the current node should be closer to the aggregation node, i.e.,

\[\begin{align} \rho_i-\rho_j>0. \tag{2} \end{align}\]

2.1  PLC Channel Noise Model

The PLC channel has various noise sources, among which the background noise can be modeled as the Gaussian noise [24], [25], and its power spectral density \(N(f)\) can be written as

\[\begin{align} N(f)=10^{K-3.95f10^{-5}}, \tag{3} \end{align}\]

where \(K\) follows a normal distribution. Hence, the background noise \(N_0\) in the PLC channel is generally modeled as an additive Gaussian white noise with mean \(0\) and variance \(\sigma_{bg}^2\).

Impulse noise is mainly caused by the operation of electrical equipment. We adopt Middleton’s Class A model to describe the impulse noise in the PLC channel, the probability density function (PDF) of which can be expressed as

\[\begin{align} F_M(\delta_m)=\sum_{\lambda=0}^\eta p_\lambda H(\lambda_m;0;\sigma_\lambda^2), \tag{4} \end{align}\]

where \(\lambda\) indicates the noise state and \(\eta\) is the total number of states. \(p_\lambda\) is the generation probability of impulse noise under the \(\lambda\)-th state and satisfies \(p_\lambda=\beta^\lambda e^{-\alpha/\lambda!}\), where \(\beta\le1\) is the impulse noise index that describes the impulse property of noise. \(H(\lambda_m;0;\sigma_\lambda^2)\) is the Gaussian PDF. Define \(\sigma_{im}^2\) as the cumulative impulse noise power for all states. The impulse noise generated by different interference sources follows the Poisson distribution, which is expressed as

\[\begin{align} \sigma_\lambda^2=(\sigma_{bg}^2+\sigma_{im}^2)\frac{\lambda/\beta+\theta}{1+\theta}, \tag{5} \end{align}\]

where \(\theta=\sigma_{bg}^2/\sigma_{im}^2\).

2.2  PLC Channel Model

The PLC channel is modeled using the transmission line theory [26]. In this model, the power line is conceptualized as a distributed circuit comprising a sequence of discrete components. These components include the series resistance \(R'\), inductance \(L'\), parallel conductance \(G'\), and capacitance \(C'\) per unit length \(\Delta l\) of the power line, as illustrated in Fig. 2. The propagation coefficient \(\varpi\) of the power line can be defined as

\[\begin{align} \varpi=\sqrt{(R^{\prime}+2\pi fL^{\prime})(G^{\prime}+2\pi fC^{\prime})}=\epsilon+j\zeta, \tag{6} \end{align}\]

where \(f\) is the frequency of the carrier signal. The real part of the propagation coefficient \(\epsilon\) corresponds to the attenuation constant, which characterizes the amplitude attenuation parameter during the transmission of carrier signals. As stated in [27], within the frequency range of carrier communication, \(\epsilon\) can be expressed as

\[\begin{align} \epsilon=k_{1}\sqrt{f}+k_{2}f. \tag{7} \end{align}\]

Fig. 2  Distribution circuit model of power line.

Considering the sharing and openness of power lines, the signal attenuation constant and the circuit parameters at the line branch will change when the load is connected to the power grid. To reflect the changes in the noise environment, the effective communication distance between nodes is selected to quantify the channel state. The signal attenuation value between node \(v_{i}\) and \(v_{j}\) can be expressed as

\[\begin{align} \delta_{i,j}=(k_{1}\sqrt{f}+k_{2}f)l_{i,j}, \tag{8} \end{align}\]

where \(l_{i,j}\) is the length of the power line between the node \(v_{i}\) and \(v_{j}\).

When the node \(v_{i}\) selects the node \(v_{j}\) for data transmission, the transmission rate between nodes can be expressed as

\[\begin{align} r_{i,j}(t)=\log_{2}(1+\frac{P_{i,j}(t)\delta_{i,j}W_{i,j}(t)}{\Gamma_{i}}), \tag{9} \end{align}\]

where \(P_{i,j}(t)\) represents the transmission power between \(v_{i}\) and \(v_{j}\). Define \(\overline{P}\) as the power spectral density mask. \(P_{i,j}(t)\) needs to satisfy the constraint \(P_{i,j}(t)\leq \overline{P}\). \(W_{i,j}(t)\) represents the signal-to-noise ratio (SNR) gain between \(v_{i}\) and \(v_{j}\), and is expressed as

\[\begin{align} W_{i,j}(t)=\frac{\mid Q_{i,j}(t)\mid^2}{\sigma_{bg}^2+\sigma_{im}^2}, \tag{10} \end{align}\]

where \(Q_{i,j}(t)\) is the channel frequency response of the node \(v_{i}\). \(\Gamma_{i}\) is the SNR gap, which can quantitatively reflect the anti-interference performance of the distribution network communication topology when the SNR deteriorates. \(\Gamma_{i}\) is expressed as

\[\begin{align} \Gamma_{i}\approx\frac{[Y^{-1}(Pe)]^2}{3}, \tag{11} \end{align}\]

where \(Y^{-1}(x)\) represents the inverse function of \(Y(x)=\frac{1}{2\pi}\int_x^\infty e^{-t^2/2}dt\).

2.3  Node Queue Backlog Model

Due to the influence of channel state, the data of the node \(v_{i}\) may not be able to transmit completely within a fixed time slot length \(\tau\). Therefore, a data queue is constructed to store the untransmitted data and the data transmitted from other nodes [28], the backlog of which is updated as

\[\begin{align} Z_{i}(t+1)&= \tag{12} \\ \notag &Z_{i}(t)-\sum_{j=1}^I{x_{i,j}(t)U_{i,j}(t)}+\sum_{k=1}^I{x_{k,i}(t)U_{k,i}(t)}, \end{align}\]

where \(U_{k,i}(t)\) is the amount of data transmitted from the previous node \(v_{k}\), and \(U_{i,j}(t)\) is the amount of data transmitted to the node \(v_{j}\). \(U_{i,j}(t)\) can be expressed as

\[\begin{align} U_{i,j}(t)=\min\{\tau r_{i,j}(t),Z_{i}(t)\}. \tag{13} \end{align}\]

Similarly, the formula of \(U_{k,i}(t)\) can be obtained.

2.4  Transmission Reliability Model

Due to the coupling between the distribution network and the PLC network, and the interference of background noise and impulse noise in the PLC channel, data transmission between nodes may be affected by error codes. Therefore, the cyclic redundancy check (CRC) mechanism is used to verify the data error received during networking. The error probability between \(v_{i}\) and \(v_{j}\) is expressed as

\[\begin{align} p_{i,j}(t)=1-\exp(-\frac{\xi(\sigma_{bg}^2+\sigma_{im}^2)}{P_{i,j}(t)W_{i,j}(t)}), \tag{14} \end{align}\]

where \(\xi\) represents the waterfall threshold. Define \(m_{i,j}(t)\in\{0,1\}\) as the binary data transmission error indicator variable, where \(m_{i,j}(t)=1\) indicates that no errors occurred during data transmission, \(m_{i,j}(t)=0\) otherwise. \(m_{i,j}(t)\) is expressed as

\[\begin{align} m_{i,j}(t)=\Big\{\begin{array}{rcl}1, &\mbox{The probability is $1-p_{i,j}(t)$}\\ 0,&\mbox{The probability is $p_{i,j}(t)$}\end{array}. \tag{15} \end{align}\]

To ensure the long-term reliable transmission of distribution network service data, a long-term transmission reliability constraint is defined as

\[\begin{align} \lim_{T\rightarrow \infty}\frac{1}{T}\sum_{t=1}^T{\sum_{j=1}^I{x_{i,j}(t)m_{i,j}(t)}}\geq\varpi_{i,\rm{min}}, \tag{16} \end{align}\]

where \(\varpi_{i,\rm{min}}\) is the minimum tolerance reliability threshold of \(v_{i}\). This constraint means that for node \(v_{i}\), the total times of no transmission errors between \(v_{i}\) and all other nodes should satisfy a threshold over a long term.

2.5  Networking Optimization Problem Model

In this paper, a PLC self-organizing networking optimization problem is proposed. The objective is to minimize the weighted sum of the node data backlog by optimizing the networking strategy under the long-term transmission reliability constraint. The optimization problem is modeled as

\[\begin{align} \begin{split} &\textbf{P1:}\min_{\{x_{i,j}(t)\}}\sum_{i=1}^I \vartheta_iZ_i(t)\\ \text{s.t.} \quad & C_1: x_{i,j}(t) \in \{0,1\}, \forall v_i,v_j \in \mathcal{V}, \forall t \in \mathcal{T},\\ &C_2: \sum_{j=1}^I x_{i,j}(t)\leq 1, \forall v_i \in \mathcal{V}, \forall t \in \mathcal{T},\\ &C_3: E_{i,j} \in \varepsilon, \forall v_i,v_j \in \mathcal{V},\\ &C_4: \rho_i-\rho_j>0, \forall v_i,v_j \in \mathcal{V},\\ &C_5: \lim_{T\to\infty} \frac{1}{T}\sum_{t=1}^T\sum_{j=1}^I x_{i,j}(t)m_{i,j}(t)\ge\varpi_{i,\rm{min}},\\ \end{split} \tag{17} \end{align}\]

where \(\vartheta_i\) is the data importance weight of \(v_i\). \(C_1\) is the constraint of the networking decision indicator variable. \(C_2\) indicates that each node can select only one next-hop node for data transmission in each time slot. \(C_3\) indicates that the PLC channel must exist between the current node and the next-hop node. \(C_4\) indicates that the next-hop node selected by the current node is closer to the aggregation node. \(C_5\) is the long-term transmission reliability constraint.

3.  Backpressure Learning-Based Data Transmission Reliability-Aware Self-Organizing Networking Algorithm

In this section, we propose a backpressure learning-based data transmission reliability-aware self-organizing networking algorithm to address \(\textbf{P1}\). First, the problem transformation based on the backpressure algorithm and Lyapunov optimization is introduced. Then, a backpressure and data transmission reliability-aware SARSA-based self-organizing networking strategy is given.

3.1  Problem Transformation Based on Backpressure Algorithm and Lyapunov Optimization

To ensure the long-term transmission reliability constraint, a reliability virtual queue is constructed and updated as [29]

\[\begin{align} E_{i}\left(t+1\right)&= \tag{18} \\ \notag \max\Big\{E_{i}\left(t\right)+\varpi_{i,\min}- \sum_{j=1}^{l}x_{i,j}\left(t\right)m_{i,j}\left(t\right),0\Big\}. \end{align}\]

Based on Lyapunov optimization theory [30], [31], \(\textbf{P1}\) is converted to

\[\begin{align} &\textbf{P2:}\min_{\{x_{i,j}(t)\}}\Theta\left(t\right)=\sum_{i=1}^{l}\sum_{j=1}^{l}\bigl[V_{Z}\vartheta_{i}Z_{i}\bigl(t\bigr)-\notag\\ &E_{i}\bigl(t\bigr)x_{i,j}\bigl(t\bigr)m_{i,j}\bigl(t\bigr)\bigr]\notag\\ &\textbf{s.t.}\quad C_{1}\sim C_{4},\notag\\ &~~~~~~ C_{6}:Z_{i}\bigl(t\bigr)\text{ and }E_{i}\bigl(t\bigr)\text{ are mean rate stable,} \tag{19} \end{align}\]

where \(V_Z\) is the weight of the weighted sum of node data backlog.

Further, based on the backpressure algorithm, the problem of minimizing data backlog can be transformed into the problem of maximizing the node data backlog difference. Define \(\Delta Z_{i,j}(t)\) as the backlog difference between node \(v_i\) and \(v_j\), which is given by

\[\begin{align} {\Delta Z_{i,j}\left(t\right)=Z_{i}\left(t\right)-Z_{j}\left(t\right)}. \tag{20} \end{align}\]

When \(\Delta Z_{i, j}(t)\) is larger, the backlog of the next-hop node \(v_j\) is smaller than that of \(v_i\), which helps to smooth the data transmission load to avoid network congestion. Therefore, \(\Theta(t)\) can be converted to \(\hat{\Theta}(t)\), which is given by

\[\begin{align} \hat{\Theta}(t)& =\sum_{i=1}^{I}\sum_{j=1}^{I}\hat{\Theta}_{i,j}\left(t\right)\notag \\ &=-\sum_{i=1}^{I}\sum_{j=1}^{I}\Bigl[V_{z}\vartheta_{i}\Delta Z_{i,j}\bigl(t\bigr)+E_{i}\bigl(t\bigr)x_{i,j}\bigl(t\bigr)m_{i,j}\bigl(t\bigr)\Bigr]. \tag{21} \end{align}\]

3.2  Backpressure and Data Transmission Reliability-Aware SARSA-Based Self-Organizing Networking Strategy

Owing to the intricate nature of PLC channels, which encompasses complex background and pulse noise, as well as the dynamic structure of PLC networks, acquiring real-time global information becomes a challenging task for individual nodes. To address this challenge, reinforcement learning (RL) algorithms enable nodes to continually learn about the network state through real-time interactions between the agent and the environment. By leveraging the value of state-action pairs, i.e., Q value, RL algorithms facilitate online optimization of self-organizing networking decisions within PLC networks, even in the absence of comprehensive global network information.

To incorporate the sequential coupling characteristics of state and action in RL, we transform \(\textbf{P2}\) into a Markov decision process (MDP) that encompasses distinct components such as the state space, action space, and reward. The subsequent sections provide a comprehensive account of these components.

\(\bullet\) State space: The state space includes node, node data backlog, virtual queue backlog of this time slot, networking decision indicator and data transmission error variables of the previous time slot, which are given by

\[\begin{align} &\mathbf{S}_{i}\left(t\right)=\Big\{v_{i},Z_{i}\left(t\right),E_{i}\left(t\right),x_{i,j} \left(t-1\right),m_{i,j}\left(t-1\right)\Big\}. \tag{22} \end{align}\]

\(\bullet\) Action space: The action space is defined as the set of next-hop nodes that can be selected by each node, which is given by

\[\begin{align} {{\mathbf{A}_{i}\left(t\right)=\left\{\nu_{j}\mid\rho_{i}-\rho_{j}>0, E_{i,j} \in \varepsilon\right\}}}. \tag{23} \end{align}\]

\(\bullet\) Penalty: According to the optimization goal after transformation, the penalty for \(v_i\) selecting \(v_j\) as the next-hop node is defined as

\[\begin{align} {\varphi}_{i,j}\left(t\right)=\hat{\Theta}_{i,j}\left(t\right). \tag{24} \end{align}\]

The SARSA algorithm, as a RL technique, incorporates an online optimization mechanism that demonstrates superior robustness compared to the Q-Learning algorithm in decision-making scenarios. By effectively analyzing the dynamic changes in the node data queue and the transmission reliability virtual queue, the SARSA algorithm significantly enhances transmission reliability and alleviates network congestion within PLC networking. Consequently, the SARSA algorithm holds substantial potential for enhancing the transmission performance of PLCs within distribution networks. To further enhance convergence performance and learning ability, we introduce a modification to the traditional SARSA algorithm by incorporating the discrepancy between the converted data transmission reliability virtual queue backlog and the node data queue backlog as a penalty within the networking decision process. The proposed algorithm is shown as Algorithm 1, which includes three stages: initialization, networking decision, and learning. The details are described as follows.

\(\bullet\) Initialization: Initialize the networking decision indicator variables \(x_{i,j}(t)\), action \(\mathbf{A}_i(t)\), and \(Q(\mathbf{S}_i(t),v_j)\).

\(\bullet\) Networking decision: The node \(v_i\) observes the state space \(\mathbf{S}_{i}(t)\) in the \(t\)-th time slot and selects the next-hop node with the maximum Q value, which is given by

\[\begin{align} \nu_{j}*=\arg\max_{\{\nu_{j}\}}Q\left(\mathbf{S}_{i}\left(t\right),v_{j}\right). \tag{25} \end{align}\]

If \(v_i\) selects \(v_j\) for the next-hop transmission, set \(x_{i,j}(t)=1\). Then, \(v_i\) performs the action selected in the previous stage for data transmission.

\(\bullet\) Learning: Each node within the network observes the performance of the data queue backlog and the data transmission reliability virtual queue. Subsequently, each node calculates the penalty using (24). The node then proceeds to update the data queue and transmission reliability virtual queue by employing (12) and (18), respectively. Following this, each node transitions from the current state to the next state, selects the corresponding action, and updates the Q value as

\[\begin{align} Q\left(\mathbf{S}_{i}\left(t\right),v_{j}\right)=&Q\left(\mathbf{S}_{i}\left(t\right),v_{j}\right)+\alpha\Big[\gamma Q\left(\mathbf{S}_{i}\left(t+1\right),v_{j}\right)-\notag \\ &Q\left(\mathbf{S}_{i}\left(t\right),v_{j}\right)+\varphi_{i,j}\left(t\right)\Big]. \tag{26} \end{align}\]

where \(\alpha\) is the learning rate and \(\gamma\) is the attenuation factor of penalty. Repeat Networking decision and Learning until \(t=T\).

4.  Simulation Result

The proposed algorithm is evaluated through simulation. The simulation is performed via MATLAB 2023b and runs over a ThinkStation P520 with Intel Core i7-6900K CPU and 48 GB random access memory (RAM). A PLC network with a topology structure comprising 26 nodes distributed across four levels is employed. Specifically, the network consists of 3 first-level nodes, 5 second-level nodes, 7 third-level nodes, and 11 fourth-level nodes. The first-level nodes possess the highest level of data importance, whereas the fourth-level nodes have the lowest. Distribution network topology has obvious multi-level characteristics, and common topologies are bus, star, tree, etc. Each node decides the grouping level based on the shortest distance to the aggregation gateway. The higher the networking level of the node, the greater the danger of its failure for the safe operation of the distribution network. Therefore, we set the minimum tolerance reliability thresholds as [0.9, 0.8, 0.7, 0.6], respectively. Different power service data have different importance levels, and the backlog of high-importance data has a worse impact. Therefore, we design different data importance levels for different importance levels of power service data to prioritize the transmission of data with high importance levels. The data importance values for the nodes at different levels are set as [1.0, 0.8, 0.6, 0.4]. Additional simulation parameters are shown in Table 1 [32].

Table 1  Simulation parameters.

Two advanced networking algorithms are set for comparison. The first algorithm is the self-organizing network routing algorithm based on Q-learning (SRQ) [33]. It utilizes Q-learning to optimize PLC networking decisions. However, it overlooks the significance of differentiated data importance and fails to address the long-term data transmission reliability constraint. The second one is the particle swarm optimization-based PLC networking (PSO-PLC) algorithm [34]. This algorithm employs the particle swarm optimization technique to search for an optimal PLC networking strategy. It also neglects the differentiated data importance and the long-term data transmission reliability constraint. The optimization objective in both algorithms is defined as the weighted sum of node data backlogs, without considering the problem transformation based on the backpressure algorithm.

Figure 3 shows the weighted sum of the node data backlog versus time slot. The proposed algorithm exhibits the most rapid convergence in comparison to the SRQ and PSO-PLC algorithms. Specifically, when \(t=500\), the proposed algorithm surpasses SRQ and PSO-PLC by \(13.44\%\) and \(19.19\%\), respectively. This significant performance improvement stems from the fact that the proposed algorithm transforms the minimization of node data backlog into the maximization of node data backlog differences. Therefore, network congestion is effectively circumvented, enabling continuous learning of the relationship between network state and networking decisions based on backpressure awareness. Furthermore, the proposed algorithm considers the optimization of node data backlog with differentiated data importance, thereby further enhancing the weighted sum performance. Conversely, SRQ and PSO-PLC algorithms disregard the optimization of node data backlog differences in relation to differentiated data importance, resulting in poorer convergence and node data backlog performance.

Fig. 3  The weighted sum of the node data backlog versus time slot.

The average queue backlogs of different-level nodes are shown in Fig. 4. The proposed algorithm reduces the average queue backlogs of first-level nodes by \(51.46\%\) and \(55.36\%\) compared with SRQ and PSO-PLC, respectively. By taking into account the differentiated data importance, the proposed algorithm optimizes the networking decision and allows the nodes with higher data importance to reduce their data backlogs preferentially, thus alleviating the data load on the PLC network. SRQ and PSO-PLC have similar average queue backlogs for different-level nodes since they neglect the differentiated data importance.

Fig. 4  Average queue backlogs of different-level nodes.

Figure 5 shows the average amount of transmitted data versus time slot. When the time slot reaches \(500\), the proposed algorithm achieves a \(41.19\%\) increase in average transmitted data amount compared to SRQ, and an even more significant enhancement of \(73.37\%\) compared to PSO-PLC. This notable performance gain is attributed to the optimization of networking decisions introduced by the proposed algorithm, which includes the incorporation of node backlog differences into the penalty function, so as to effectively promote the transmission of a greater quantity of data. Conversely, the SRQ and PSO-PLC algorithms do not introduce networked decision optimization and neglect the differentiated data importance.

Fig. 5  The average amount of transmitted data versus time slot.

Figure 6 shows the average transmission reliability of different-level nodes. The result conclusively demonstrates that the proposed algorithm effectively meets the long-term transmission reliability constraints for nodes at different levels. Conversely, the SRQ and PSO-PLC algorithms only partially satisfy these constraints. This superior performance of the proposed algorithm is attributed to its unique approach of transforming the long-term transmission reliability constraint into the optimization of virtual queue stability. Additionally, the proposed algorithm keenly senses and adapts to dynamic changes in virtual queues while considering the differentiated data importance. In contrast, SRQ and PSO-PLC algorithms overlook the importance of the long-term transmission reliability constraint, thereby failing to ensure reliable data transmission for nodes at different levels.

Fig. 6  Average transmission reliability of different-level nodes.

Figure 7 shows the box plots of transmission reliability deficit. The result reveals that the proposed algorithm effectively mitigates the fluctuations in transmission reliability deficit. Specifically, the proposed algorithm achieves reductions of \(65.14\%\) and \(75.39\%\) in the transmission reliability deficit fluctuation compared to SRQ and PSO-PLC, respectively. This performance improvement is attributed to the proposed algorithm’s emphasis on optimizing the stability of the transmission reliability virtual queue. By constantly learning and adapting the networking strategy, the proposed algorithm effectively minimizes penalties and improves the virtual queue deficit. In contrast, SRQ and PSO-PLC algorithms do not incorporate transmission reliability awareness into their decision-making process, resulting in the deteriorating performance of the virtual queue.

Fig. 7  The box plots of transmission reliability deficit.

5.  Conclusion

This paper introduced a backpressure learning-based data transmission reliability-aware self-organizing networking algorithm. The algorithm aims to minimize the weighted sum of node data backlogs. The proposed algorithm first leverages the Lyapunov optimization to transform the optimization problem. Secondly, a backpressure and data transmission reliability-aware state-action-reward-state-action (SARSA)-based self-organizing networking strategy is proposed to realize the PLC networking optimization. Simulation results demonstrate that the proposed algorithm outperforms both the SRQ algorithm and PSO-PLC algorithm in terms of the weighted sum of delay and reliability, achieving reductions of \(13.44\%\) and \(19.19\%\) respectively. Furthermore, the proposed algorithm not only enhances the least transmission reliability deficit but also mitigates network congestion by incorporating transmission reliability and backpressure awareness. In future research, we plan to investigate the joint optimization of self-organizing networking and power control to further enhance data transmission reliability.

Acknowledgments

This work is supported by the Science and Technology Project of China Southern Power Grid Corporation under Grant Number 036000KK52220020 (GDKJXM20220276).

References

[1] J.A. Corchado, J.A. Cortés, F.J. Cañete, A. Arregui, and L. Díez, “Analysis of the spatial correlation of indoor MIMO PLC channels,” IEEE Commun. Lett., vol.2, no.1, pp.40-43, Jan. 2017.

[2] S G. Laguna-Sanchez and L.-G. Miguel, “On the use of alpha-stable distributions in noise modeling for PLC,” IEEE Trans. Power Del., vol.30, no.4, pp.1863-1870, Aug. 2015.

[3] W. Gu, W. Liu, J. Zhu, B. Zhao, Z. Wu, Z. Luo, and J. Yu, “Adaptive decentralized under-frequency load shedding for islanded smart distribution networks,” IEEE Trans. Sustain. Energy, vol.5, no.3, pp.886-895, July 2014.

[4] T.A. Papadopoulos, C.G. Kaloudas, A.I. Chrysochos, and G.K. Papagiannis, “Application of narrowband power-line communication in medium-voltage smart distribution grids,” IEEE Trans. Power Del., vol.28, no.2, pp.981-988, April 2013.

[5] F. Salvadori, C.S. Gehrke, A.C. de Oliveira, M. de Campos, and P.S. Sausen, “Smart grid infrastructure using a hybrid network architecture,” IEEE Trans. Smart Grid, vol.4, no.3, pp.1630-1639, Sept. 2013.

[6] M. Miao, L. Wang, and G. Chen, “Performance and capacity analysis of MDCSK-BICM for impulsive noise in PLC,” IEEE Trans. Power Del., vol.37, no.4, pp.3164-3175, Aug. 2022.

[7] S. Nolan, C.E. Jones, R. Peña-Alzola, P.J. Norman, G. Burt, P. Miller, and M. Husband, “Voltage-based current-compensation converter control for power electronic interfaced distribution networks in future aircraft,” IEEE Trans. Transport. Electrific., vol.6, no.4, pp.1819-1829, Dec. 2020.

[8] A. Mathur, M.R. Bhatnagar, and B.K. Panigrahi, “Performance evaluation of PLC under the combined effect of background and impulsive noises,” IEEE Commun. Lett., vol.19, no.7, pp.1117-1120, May 2015.

[9] B. Chen, N. Yu, B. Chen, C. Tian, Y. Chen, and G. Chen, “Fault location for underground cables in ungrounded MV distribution networks based on ZSC signal injection,” IEEE Trans. Power Del., vol.36, no.5, pp.2965-2977, Oct. 2021.

[10] P. Qin, Y. Fu, X. Zhao, K. Wu, J. Liu, and M. Wang, “Optimal task offloading and resource allocation for C-NOMA heterogeneous air-ground integrated power internet of things networks,” IEEE Trans. Wireless Commun., vol.21, no.11, pp.9276-9292, Nov. 2022.

[11] L. Zhen, A.K. Bashir, K. Yu, Y.D. Al-Otaibi, C.H. Foh, and P. Xiao, “Energy-efficient random access for LEO satellite-assisted 6G internet of remote things,” IEEE Internet Things J., vol.8, no.7, pp.5114-5128, April 2021.

[12] W.E. Sayed, P. Crovetti, N. Moonen, P. Lezynski, R. Smolenski, and F. Leferink, “Electromagnetic interference of spread-spectrum modulated power converters in G3-PLC power line communication systems,” IEEE Lett. Electromagn. Compat. Pract. Appl., vol.3, no.4, pp.118-122, Dec. 2021.

[13] D. Guezgouz, D.E. Chariag, Y. Raingeaud, and J.-C.L. Bunetel, “Modeling of electromagnetic interference and PLC transmission for loads shedding in a microgrid,” IEEE Trans. Power Electron., vol.26, no.3, pp.747-754, March 2011.

[14] Y. Ben-Shimol, S. Greenberg, and K. Danilchenko, “Application-layer approach for efficient smart meter reading in low-voltage PLC networks,” IEEE Trans. Commun., vol.66, no.9, pp.4249-4258, Sept. 2018.

[15] K.M. Rabie, B. Adebisi, H. Gacanin, G. Nauryzbayev, and A. Ikpehai, “Performance evaluation of multi-hop relaying over non-gaussian PLC channels,” J. Commun. Netw., vol.19, no.5, pp.531-538, Oct. 2017.

[16] S.K.E. Awadallah, J.V. Milanović, and P.N. Jarman, “Reliability based framework for cost-effective replacement of power transmission equipment,” IEEE Trans. Power Syst., vol.29, no.5, pp.2549-2557, Oct. 2014.

[17] Z. Zhou, Z. Jia, H. Liao, W. Lu, S. Mumtaz, M. Guizani, and M. Tariq, “Secure and latency-aware digital twin assisted resource scheduling for 5G edge computing-empowered distribution grids,” IEEE Trans. Ind. Informat., vol.18, no.7, pp.4933-4943, Dec. 2022.

[18] I. Rubin, Y.-Y. Lin, and D. Kofman, “Optimal relay configuration for power line communication networks,” IEEE Trans. Commun., vol.64, no.1, pp.130-140, Jan. 2016.

[19] J. Shin, J. Lee, and J. Jeon, “Channel modeling for indoor broadband power-line communications networks with arbitrary topologies by taking adjacent nodes into account,” IEEE Trans. Power Del., vol.26, no.3, pp.1432-1439, July 2011.

[20] Y. Qian, S. Li, L. Shi, J. Li, F. Shu, D.N.K. Jayakody, and J. Yuan, “Cache-enabled MIMO power line communications with precoding design in smart grid,” IEEE Trans. Green Commun. Netw., vol.4, no.1, pp.315-325, March 2020.

[21] Y. Wang, H. Chen, and Z. Miao, “A matching optimization method of distribution power internet of things based on network looseness,” IEEE Access, vol.9, pp.63 676-63 684, April 2021.

[22] S.-G. Yoon, S. Jang, Y.-H. Kim, and S. Bahk, “Opportunistic routing for smart grid with power line communication access networks,” IEEE Trans. Smart Grid, vol.5, no.1, pp.303-311, Jan. 2014.

[23] M. Biagi, S. Greco, and L. Lutz, “Geo-routing algorithms and protocols for power line communications in smart grids,” IEEE Trans. Smart Grid, vol.9, no.2, pp.1472-1481, March 2018.

[24] N. Agrawal and P.K. Sharma, “Capacity analysis of a nb-plc system with background and impulsive noises,” 2017 International Conference on Computer, Communications and Electronics (Comptelix), pp.118-123, 2017.

[25] A. Mathur, M.R. Bhatnagar, and B.K. Panigrahi, “Performance evaluation of plc under the combined effect of background and impulsive noises,” IEEE Commun. Lett., vol.19, no.7, pp.1117-1120, 2015.

[26] L.C. Mathias, L.F. d. Melo, and T. Abrao, “Analysis and modeling of Brazilian indoor PLC channels controlled environment - Testing and validation,” IEEE Latin Am. Trans., vol.13, no.8, pp.2473-2481, 2015.

[27] M. Zimmermann and K. Dostert, “A multipath model for the powerline channel,” IEEE Trans. Commun., vol.50, no.4, pp.553-559, April 2002.

[28] Z. Shi, H. Wei, and J. Zhu, “Edge computing-empowered task offloading in PLC-wireless integrated network based on matching with quota,” Computer Communications, vol.182, pp.110-116, 2022.

[29] Z. Zhou, X. Chen, H. Liao, Z. Gan, F. Xiao, Q. Tu, W. Sun, Y. Liu, S. Mumtaz, and M. Guizani, “Collaborative learning-based network resource scheduling and route management for multi-mode green IoT,” IEEE Trans. Green Commun. Netw., vol.7, no.2, pp.928-939, June 2023.

[30] Z. Zhou, Z. Wang, H. Yu, H. Liao, S. Mumtaz, L. Oliveira, and V. Frascolla, “Learning-based URLLC-aware task offloading for internet of health things,” IEEE J. Sel. Areas Commun., vol.39, no.2, pp.396-410, June 2021.

[31] J.-M. Godhavn and O. Egeland, “A lyapunov approach to exponential stabilization of nonholonomic systems in power form,” IEEE Trans. Autom. Control, vol.42, no.7, pp.1028-1032, July 1997.

[32] H. Liao, Z. Jia, Z. Zhou, N. Liu, F. Wang, Z. Gan, and X. Yao, “Dispatching and control information freshness guaranteed resource optimization in simplified power internet of things,” Journal on Communications, vol.43, no.7, pp.203-214, 2022.

[33] L. Liu, L. Zheng, and Y. Wang, “Q-learning algorithm enabled topology control scheme in power line communication networks,” 2022 4th International Conference on Smart Power and Internet Energy Systems, pp.2229-2232, 2022.

[34] E.W. Akpari, S. Nunoo, F.K. Deynu, and M.W. Apprey, “Efficient low-complexity optimized channel estimating methods for OFDM-based low-voltage broadband power line communication systems,” J. Microw. Optoelectron. Electromagn. Appl., vol.12, no.3, pp.453-480, 2022.

Authors

Zhan SHI
  Power Dispatching and Controlling Center of Guangdong Power Grid Company Limited

Keyword

distribution network,  power line communication,  self-organizing networking,  backpressure,  reinforcement learning