In a multi-agent system, it is important to consider a design method of cooperative actions in order to achieve a common goal. In this paper, we propose two novel multi-agent reinforcement learning methods, where the control specification is described by linear temporal logic formulas, which represent a common goal. First, we propose a simple solution method, which is directly extended from the single-agent case. In this method, there are some technical issues caused by the increase in the number of agents. Next, to overcome these technical issues, we propose a new method in which an aggregator is introduced. Finally, these two methods are compared by numerical simulations, with a surveillance problem as an example.

With the high development of computation requirements in Internet of Things, resource-limited edge servers usually require to cooperate to perform the tasks. Most related studies usually assume a static cooperation approach which might not suit the dynamic environment of edge computing. In this paper, we consider a dynamic cooperation approach by guiding edge servers to form coalitions dynamically. It raises two issues: 1) how to guide them to optimally form coalitions and 2) how to cope with the dynamic feature where server statuses dynamically change as the tasks are performed. The coalitional Markov decision process (CMDP) model proposed in our previous work can handle these issues well. However, its basic solution, coalitional Q-learning, cannot handle the large scale problem when the task number is large in edge computing. Our response is to propose a novel algorithm called deep coalitional Q-learning (DCQL) to solve it. To sum up, we first formulate the dynamic cooperation problem of edge servers as a CMDP: each edge server is regarded as an agent and the dynamic process is modeled as a MDP where the agents observe the current state to formulate several coalitions. Each coalition takes an action to impact the environment which correspondingly transfers to the next state to repeat the above process. Then, we propose DCQL which includes a deep neural network and so can well cope with large scale problem. DCQL can guide the edge servers to form coalitions dynamically with the target of optimizing some goal. Furthermore, we run experiments to verify our proposed algorithm's effectiveness in different settings.

Distributed edge cloud computing is an important computation infrastructure for Internet of Things (IoT) and its task offloading problem has attracted much attention recently. Most existing work on task offloading in distributed edge cloud computing usually assumes that each self-interested user owns one edge server and chooses whether to execute its tasks locally or to offload the tasks to cloud servers. The goal of each edge server is to maximize its own interest like low delay cost, which corresponds to a non-cooperative setting. However, with the strong development of smart IoT communities such as smart hospital and smart factory, all edge and cloud servers can belong to one organization like a technology company. This corresponds to a cooperative setting where the goal of the organization is to maximize the team interest in the overall edge cloud computing system. In this paper, we consider a new problem called cooperative task offloading where all edge servers try to cooperate to make the entire edge cloud computing system achieve good performance such as low delay cost and low energy cost. However, this problem is hard to solve due to two issues: 1) each edge server status dynamically changes and task arrival is uncertain; 2) each edge server can observe only its own status, which makes it hard to optimize team interest as global information is unavailable. For solving these issues, we formulate the problem as a decentralized partially observable Markov decision process (Dec-POMDP) which can well handle the dynamic features under partial observations. Then, we apply a multi-agent reinforcement learning algorithm called value decomposition network (VDN) and propose a VDN-based task offloading algorithm (VDN-TO) to solve the problem. Specifically, the motivation is that we use a team value function to evaluate the team interest, which is then divided into individual value functions for each edge server. Then, each edge server updates its individual value function in the direction that can maximize the team interest. Finally, we choose a part of a real dataset to evaluate our algorithm and the results show the effectiveness of our algorithm in a comparison with some other existing methods.

This paper proposes a switched pinning control method with a multi-rating mechanism for vehicle platoons. The platoons are expressed as multi-agent systems consisting of mass-damper systems in which pinning agents receive target velocities from external devices (ex. intelligent traffic signals). We construct model predictive control (MPC) algorithm that switches pinning agents via mixed-integer quadratic programmings (MIQP) problems. The optimization rate is determined according to the convergence rate to the target velocities and the inter-vehicular distances. This multi-rating mechanism can reduce the computational load caused by iterative calculation. Numerical results demonstrate that our method has a reduction effect on the string instability by selecting the pinning agents to minimize errors of the inter-vehicular distances to the target distances.

This paper considers a distributed subgradient method for online optimization with event-triggered communication over multi-agent networks. At each step, each agent obtains a time-varying private convex cost function. To cooperatively minimize the global cost function, these agents need to communicate each other. The communication with neighbor agents is conducted by the event-triggered method that can reduce the number of communications. We demonstrate that the proposed online algorithm achieves a sublinear regret bound in a dynamic environment with slow dynamics.

This paper considers a velocity control problem for merging and splitting maneuvers of vehicle platoons. In this paper, an external device sends velocity commands to some vehicles in the platoon, and the others adjust their velocities autonomously. The former is pinning control, and the latter is consensus control in multi-agent control. We propose a switched pinning control algorithm. Our algorithm consists of three sub-methods. The first is an optimal switching method of pinning agents based on an MLD (Mixed Logical Dynamical) system model and MPC (Model Predictive Control). The second is a representation method for dynamical platoon formation with merging and splitting maneuver. The platoon formation follows the positional relation between vehicles or the formation demand from the external device. The third is a switching reduction method by setting a cost function that penalizes the switching of the pinning agents in the steady-state. Our proposed algorithm enables us to improve the consensus speed. Moreover, our algorithm can regroup the platoons to the arbitrary platoons and control the velocities of the multiple vehicle platoons to each target value.

In this paper, we propose a distributed subgradient-based method over quantized and event-triggered communication networks for constrained convex optimization. In the proposed method, each agent sends the quantized state to the neighbor agents only at its trigger times through the dynamic encoding and decoding scheme. After the quantized and event-triggered information exchanges, each agent locally updates its state by a consensus-based subgradient algorithm. We show a sufficient condition for convergence under summability conditions of a diminishing step-size.

Pinning control of multi-agent systems is a method that the external control input is added to some agents (pinning nodes), e.g., leaders. By the external control input, consensus to a certain target value and faster consensus are achieved. In this paper, we propose a new method of self-triggered predictive pinning control for the consensus problem. Self-triggered control is a method that both the control input and the next update time are calculated. Using self-triggered control, it is expected that the communication cost can be reduced. First, a new finite-time optimal control problem used in self-triggered control is formulated, and its solution method is derived. Next, as an on-line algorithm, two methods, i.e., the multi-hop communication-based method and the observer-based method are proposed. Finally, numerical examples are presented.

This paper studies the problem of real-time routing in a multi-autonomous robot enhanced network at uncertain and vulnerable tactical edge. Recent network protocols, such as opportunistic mobile network routing protocols, engaged social network in communication network that can increase the interoperability by using social mobility and opportunistic carry and forward routing algorithms. However, in practical harsh environment such as a battlefield, the uncertainty of social mobility and complexity of vulnerable environment due to unpredictable physical and cyber-attacks from enemy, would seriously affect the effectiveness and practicality of these emerging network protocols. This paper presents a GT-SaRE-MANET (Game Theoretic Situation-aware Robot Enhanced Mobile Ad-hoc Network) routing protocol that adopt the online reinforcement learning technique to supervise the mobility of multi-robots as well as handle the uncertainty and potential physical and cyber attack at tactical edge. Firstly, a set of game theoretic mission oriented metrics has been introduced to describe the interrelation among network quality, multi-robot mobility as well as potential attacking activities. Then, a distributed multi-agent game theoretic reinforcement learning algorithm has been developed. It will not only optimize GT-SaRE-MANET routing protocol and the mobility of multi-robots online, but also effectively avoid the physical and/or cyber-attacks from enemy by using the game theoretic mission oriented metrics. The effectiveness of proposed design has been demonstrated through computer aided simulations and hardware experiments.

It is known that policy gradient algorithm can not guarantee the convergence to a Nash equilibrium in mixed policies when it is applied in matrix games. To overcome this problem, we propose a novel multi-agent reinforcement learning (MARL) algorithm called a policy gradient lagging anchor (PGLA) algorithm. And we prove that the agents' policies can converge to a Nash equilibrium in mixed policies by using the PGLA algorithm in two-player two-action matrix games. By simulation, we confirm the convergence and also show that the PGLA algorithm has a better convergence than the LR-I lagging anchor algorithm.

This paper proposes a consensus-based subgradient method under a common constraint set with switching undirected graphs. In the proposed method, each agent has a state and an auxiliary variable as the estimates of an optimal solution and accumulated information of past gradients of neighbor agents. We show that the states of all agents asymptotically converge to one of the optimal solutions of the convex optimization problem. The simulation results show that the proposed consensus-based algorithm with accumulated subgradient information achieves faster convergence than the standard subgradient algorithm.

In this paper, based on the policy of model predictive control, a new method of predictive pinning control is proposed for the consensus problem of multi-agent systems. Pinning control is a method that the external control input is added to some agents (pinning nodes), e.g., leaders. By the external control input, consensus to a certain target value (not the average of the initial states) and faster consensus are achieved. In the proposed method, the external control input is calculated by the controller node connected to only pinning nodes. Since the states of all agents are required in calculation of the external control input, communication delays must be considered. The proposed algorithm includes not only calculation of the external control input but also delay compensation. The effectiveness of the proposed method is presented by a numerical example.

The output feedback consensus problem of nonlinear multi-agent systems under a directed network with a time varying communication delay is studied. In order to deal with this problem, the dynamic output feedback controller with an additional low gain parameter that compensates for the effect of nonlinearity and a communication delay is proposed. Also, it is shown that under some assumptions, the proposed controller can always solve the output feedback consensus problem even in the presence of an arbitrarily large communication delay.

This paper proposes a consensus-based distributed Particle Swarm Optimization (PSO) algorithm with event-triggered communications for a non-convex and non-differentiable optimization problem. We consider a multi-agent system whose local communications among agents are represented by a fixed and connected graph. Each agent has multiple particles as estimated solutions of global optima and updates positions of particles by an average consensus dynamics on an auxiliary variable that accumulates the past information of the own objective function. In contrast to the existing time-triggered approach, the local communications are carried out only when the difference between the current auxiliary variable and the variable at the last communication exceeds a threshold. We show that the global best can be estimated in a distributed way by the proposed event-triggered PSO algorithm under a diminishing condition of the threshold for the trigger condition.

This paper investigates the consensus problem of heterogeneous uncertain multi-agent systems with jointly connected topology, where the considered systems are composed of linear first-order, second-order dynamics and nonlinear Euler-Lagrange (EL) dynamics. The consensus protocol is designed to converge the position and velocity states of the linear and nonlinear heterogeneous multi-agent systems under joint connected topology, and then the adaptive consensus protocol is also proposed for heterogeneous multi-agent systems with unknown parameters in the EL dynamics under jointly connected topology. Stability analysis for piecewise continuous functions induced by the jointly connection is presented based on Lyapunov function and Cauchy's convergence criteria. Finally, some simulation results are provided to verify the effectiveness of the proposed methods.

This letter deals with the consensus problem of multi-agent systems, which are composed of feedforward nonlinear systems under a directed network with a communication time delay. In order to solve this problem, a new consensus protocol with a low gain parameter is proposed. Moreover, it is shown that under some sufficient conditions, the proposed protocol can solve the consensus problem of nonlinear multi-agent systems even in the presence of an arbitrarily large communication delay. An illustrative example is presented to verify the validity of the proposed approach.

This paper addresses a broadcast control problem of multi-agent systems with quantized measurements, where each agent moves based on the common broadcasted signal and tries to minimize a given quadratic performance index. The problem is solved by introducing dither type random movements to the agents' action which reduce the degradation caused by quantized measurements. A broadcast controller is derived and it is proven that the controller approximately achieves given tasks with probability 1. The effectiveness of the proposed controller is demonstrated by numerical simulation.

A consensus problem has been studied in many fundamental and application fields to analyze coordinated behavior in multi-agent systems. In a consensus problem, it is usually assumed that a state of each agent is scalar and all agents have an identical linear consensus protocol. We present a consensus problem of multi-agent systems where each agent has multiple state variables and a performance value evaluated by a nonlinear performance function according to its current state. We derive sufficient conditions for agents to achieve consensus on the performance value using an algebraic graph theory and the mean value theorem. We also consider an application of a performance consensus problem to resource allocation in soft real-time systems so as to achieve a fair QoS (Quality of Service) level.

This paper addresses a discrete-time consensus problem with non-linear performance functions over dynamically changing communication topologies. Each agent has a performance value based on its internal information state and exchanges the performance value with other agents to achieve consensus. We derive sufficient conditions for a global consensus using algebraic graph theory.

Secret sharing is a method for distributing a secret among a party of participants. Each of them is allocated a share of the secret, and the secret can only be reconstructed when the shares are combined together. We have been proposing a secret sharing distributed database system (SSDDB) that uses a secret sharing scheme to improve confidentiality and robustness of distributed database systems. This paper proposes a vertical partitioning algorithm for the SSDDB, and evaluates the algorithm by computational experiments.