The search functionality is under construction.
The search functionality is under construction.

Keyword Search Result

[Keyword] UAV(21hit)

1-20hit(21hit)

  • Communication-Aware Flight Algorithms for UAV-Based Delay-Tolerant Networks

    Hiroyuki ASANO  Hiraku OKADA  Chedlia BEN NAILA  Masaaki KATAYAMA  

     
    PAPER-Network

      Pubricized:
    2023/06/01
      Vol:
    E106-B No:11
      Page(s):
    1122-1132

    In this paper, a wireless communication network that uses unmanned aerial vehicles (UAVs) in the sky to transmit information between ground users is considered. We highlight a delay-tolerant network, where information is relayed in a store-and-forward fashion by establishing two types of intermittent communication links: between a UAV and a user (UAV-to-user) and between UAVs (UAV-to-UAV). Thus, a flight algorithm that controls the movement of the UAVs is crucial in achieving rapid information transmission. Our study proposes new flight algorithms that simultaneously consider the two types of communication links. In UAV-to-UAV links, the direct information transmission between two UAVs and the indirect transmission through other UAVs are considered separately. The movement of the UAVs is controlled by solving an optimization problem at certain time intervals, with a variable consideration ratio of the two types of links. In addition, we investigate not only the case where all UAVs move cooperatively but also the case where each UAV moves autonomously. Simulation results show that the proposed algorithms are effective. Moreover, they indicate the existence of an optimal consideration ratio of the two types of communication and demonstrate that our approach enables the control of frequencies of establishing the communication links. We conclude that increasing the frequency of indirect communication between UAVs improves network performance.

  • Wide-Area and Long-Term Agricultural Sensing System Utilizing UAV and Wireless Technologies

    Hiroshi YAMAMOTO  Shota NISHIURA  Yoshihiro HIGASHIURA  

     
    INVITED PAPER

      Pubricized:
    2023/02/08
      Vol:
    E106-D No:5
      Page(s):
    914-926

    In order to improve crop production and efficiency of farming operations, an IoT (Internet of Things) system for remote monitoring has been attracting a lot of attention. The existing studies have proposed agricultural sensing systems such that environmental information is collected from many sensor nodes installed in farmland through wireless communications (e.g., Wi-Fi, ZigBee). Especially, Low-Power Wide-Area (LPWA) is a focus as a candidate for wireless communication that enables the support of vast farmland for a long time. However, it is difficult to achieve long distance communication even when using the LPWA because a clear line of sight is difficult to keep due to many obstacles such as crops and agricultural machinery in the farmland. In addition, a sensor node cannot run permanently on batteries because the battery capacity is not infinite. On the other hand, an Unmanned Aerial Vehicle (UAV) that can move freely and stably in the sky has been leveraged for agricultural sensor network systems. By utilizing a UAV as the gateway of the sensor network, the gateway can move to the appropriate location to ensure a clear line of sight from the sensor nodes. In addition, the coverage area of the sensor network can be expanded as the UAV travels over a wide area even when short-range and ultra-low-power wireless communication (e.g., Bluetooth Low Energy (BLE)) is adopted. Furthermore, various wireless technologies (e.g., wireless power transfer, wireless positioning) that have the possibility to improve the coverage area and the lifetime of the sensor network have become available. Therefore, in this study, we propose and develop two kinds of new agricultural sensing systems utilizing a UAV and various wireless technologies. The objective of the proposed system is to provide the solution for achieving the wide-area and long-term sensing for the vast farmland. Depending on which problem is in a priority, the proposed system chooses one of two designs. The first design of the system attempts to achieve the wide-area sensing, and so it is based on the LPWA for wireless communication. In the system, to efficiently collect the environmental information, the UAV autonomously travels to search for the locations to maintain the good communication properties of the LPWA to the sensor nodes dispersed over a wide area of farmland. In addition, the second design attempts to achieve the long-term sensing, so it is based on BLE, a typical short-range and ultra-low-power wireless communication technology. In this design, the UAV autonomously flies to the location of sensor nodes and supplies power to them using a wireless power transfer technology for achieving a battery-less sensor node. Through experimental evaluations using a prototype system, it is confirmed that the combination of the UAV and various wireless technologies has the possibility to achieve a wide-area and long-term sensing system for monitoring vast farmland.

  • Optimal Positioning Scheme of Multiple UAVs through DOP Minimization for Location Identification of Unknown Radar

    Jisoo KIM  Seonjoo CHOI  Jaesung LIM  

     
    LETTER-Fundamentals of Information Systems

      Pubricized:
    2022/09/30
      Vol:
    E106-D No:1
      Page(s):
    78-81

    In time difference of arrival-based signal source location estimation, geometrical errors are caused by the location of multiple unmanned aerial vehicles (UAV). Herein, we propose a divide-and-conquer algorithm to determine the optimal location for each UAV. Simulations results confirm that multiple UAVs shifted to an optimal position and the location accuracy improved.

  • Voronoi-Based UAV Flight Method for Non-Uniform User Distribution in Delay-Tolerant Aerial Networks

    Hiroyuki ASANO  Hiraku OKADA  Chedlia BEN NAILA  Masaaki KATAYAMA  

     
    PAPER-Network

      Pubricized:
    2022/05/11
      Vol:
    E105-B No:11
      Page(s):
    1414-1423

    This paper considers an emergency communication system controlling multiple unmanned aerial vehicles (UAVs) in the sky over a large-scale disaster-affected area. This system is based on delay-tolerant networking, and information from ground users is relayed by the UAVs through wireless transmission and the movement of UAVs in a store-and-forward manner. Each UAV moves autonomously according to a predetermined flight method, which uses the positions of other UAVs through communication. In this paper, we propose a new method for UAV flight considering the non-uniformity of user distributions. The method is based on the Voronoi cell using the predicted locations of other UAVs. We evaluate the performance of the proposed method through computer simulations with a non-uniform user distribution generated by a general cluster point process. The simulation results demonstrate the effectiveness of the proposed method.

  • Joint Channel and Power Assignment for UAV Swarm Communication Based on Multi-Agent DRL

    Jie LI  Sai LI  Abdul Hayee SHAIKH  

     
    PAPER-Wireless Communication Technologies

      Pubricized:
    2022/04/13
      Vol:
    E105-B No:10
      Page(s):
    1249-1257

    In this manuscript, we propose a joint channel and power assignment algorithm for an unmanned aerial vehicle (UAV) swarm communication system based on multi-agent deep reinforcement learning (DRL). Regarded as an agent, each UAV to UAV (U2U) link can choose the optimal channel and power according to the current situation after training is successfully completed. Further, a mixing network is introduced based on DRL, where Q values of every single agent are non-linearly mapped, and we call it the QMIX algorithm. As it accesses state information, QMIX can learn to enrich the joint action value function. The proposed method can be used for both unicast and multicast scenarios. Experiments show that each U2U link can be trained to meet the constraints of UAV communication and minimize the interference to the system. For unicast communication, the communication rate is increased up to 15.6% and 8.9% using the proposed DRL method compared with the well-known random and adaptive methods, respectively. For multicast communication, the communication rate is increased up to 6.7% using the proposed QMIX method compared with the DRL method and 13.6% using DRL method compared with adaptive method. Besides, the successful transmission probability can maintain a high level.

  • Model of the LOS Probability for the UAV Channel and Its Application for Environment Awareness

    Chi-Min LI  Yu-Hsuan LEE  Yi-Ting LIAO  Pao-Jen WANG  

     
    PAPER-Terrestrial Wireless Communication/Broadcasting Technologies

      Pubricized:
    2022/02/01
      Vol:
    E105-B No:8
      Page(s):
    975-980

    Currently, unmanned aerial vehicles (UAV) have been widely used in many applications, such as in transportation logistics, public safety, or even in non-terrestrial networks (NTN). In all these scenarios, it is an important issue to model channel behavior between the UAV and the user equipment (UE) on the ground. Among these channel features, a critical parameter that dominates channel behavior is the probability of the line-of-sight (LOS), since the statistical property of the channel fading can be either Ricean or Rayleigh, depending on the existence of LOS. Besides, with knowledge of LOS probability, operators can design approaches or schemes to maximum system performance, such as the serving coverage, received signal to noise ratio (SNR), or the bit error rate (BER) with the limited transmitted power. However, the LOS UAV channel is likely difficult to acquire or derive, as it depends on the deployment scenario, such as an urban or rural area. In this paper, we generated four different scenarios defined by the ITU via the ray tracing simulator. Then, we used the spatial geometric relation and the curve fitting approach to derive the analytic models to predict the probability of the UAV LOS channels for different scenarios. Results show that our proposed relationships yield better prediction results than the methods in the literature. Besides, an example of establishing UAV self-awareness ability for the deployed environment via using proposed models is also provided in this paper.

  • Efficient Task Allocation Protocol for a Hybrid-Hierarchical Spatial-Aerial-Terrestrial Edge-Centric IoT Architecture Open Access

    Abbas JAMALIPOUR  Forough SHIRIN ABKENAR  

     
    INVITED PAPER

      Pubricized:
    2021/08/17
      Vol:
    E105-B No:2
      Page(s):
    116-130

    In this paper, we propose a novel Hybrid-Hierarchical spatial-aerial-Terrestrial Edge-Centric (H2TEC) for the space-air integrated Internet of Things (IoT) networks. (H2TEC) comprises unmanned aerial vehicles (UAVs) that act as mobile fog nodes to provide the required services for terminal nodes (TNs) in cooperation with the satellites. TNs in (H2TEC) offload their generated tasks to the UAVs for further processing. Due to the limited energy budget of TNs, a novel task allocation protocol, named TOP, is proposed to minimize the energy consumption of TNs while guaranteeing the outage probability and network reliability for which the transmission rate of TNs is optimized. TOP also takes advantage of the energy harvesting by which the low earth orbit satellites transfer energy to the UAVs when the remaining energy of the UAVs is below a predefined threshold. To this end, the harvested power of the UAVs is optimized alongside the corresponding harvesting time so that the UAVs can improve the network throughput via processing more bits. Numerical results reveal that TOP outperforms the baseline method in critical situations that more power is required to process the task. It is also found that even in such situations, the energy harvesting mechanism provided in the TOP yields a more efficient network throughput.

  • Distributed UAVs Placement Optimization for Cooperative Communication

    Zhaoyang HOU  Zheng XIANG  Peng REN  Qiang HE  Ling ZHENG  

     
    PAPER-Wireless Communication Technologies

      Pubricized:
    2020/12/08
      Vol:
    E104-B No:6
      Page(s):
    675-685

    In this paper, the distributed cooperative communication of unmanned aerial vehicles (UAVs) is studied, where the condition number (CN) and the inner product (InP) are used to measure the quality of communication links. By optimizing the relative position of UAVs, large channel capacity and stable communication links can be obtained. Using the spherical wave model under the line of sight (LOS) channel, CN expression of the channel matrix is derived when there are Nt transmitters and two receivers in the system. In order to maximize channel capacity, we derive the UAVs position constraint equation (UAVs-PCE), and the constraint between BS elements distance and carrier wavelength is analyzed. The result shows there is an area where no matter how the UAVs' positions are adjusted, the CN is still very large. Then a special scenario is considered where UAVs form a rectangular lattice array, and the optimal constraint between communication distance and UAVs distance is derived. After that, we derive the InP of channel matrix and the gradient expression of InP with respect to UAVs' position. The particle swarm optimization (PSO) algorithm is used to minimize the CN and the gradient descent (GD) algorithm is used to minimize the InP by optimizing UAVs' position iteratively. Both of the two algorithms present great potentials for optimizing the CN and InP respectively. Furthermore, a hybrid algorithm named PSO-GD combining the advantage of the two algorithms is proposed to maximize the communication capacity with lower complexity. Simulations show that PSO-GD is more efficient than PSO and GD. PSO helps GD to break away from local extremum and provides better positions for GD, and GD can converge to an optimal solution quickly by using the gradient information based on the better positions. Simulations also reveal that a better channel can be obtained when those parameters satisfy the UAVs position constraint equation (UAVs-PCE), meanwhile, theory analysis also explains the abnormal phenomena in simulations.

  • An Overview of Aerial Wireless Relay Networks for Emergency Communications during Large-Scale Disasters Open Access

    Hiraku OKADA  

     
    INVITED PAPER

      Pubricized:
    2020/07/01
      Vol:
    E103-B No:12
      Page(s):
    1376-1384

    In emergency communication systems research, aerial wireless relay networks (AWRNs) using multicopter unmanned aerial vehicles (UAVs) have been proposed. The main issue of the AWRNs is how to minimize the delay time of packet transmissions since it is not easy to supply many multicopters to cover a wide area. In this paper, we review the flight schemes and their delay time for the AWRNs. Furthermore, the network has specific issues such as multicopters' drops due to their battery capacity depletion and inclination of moving multicopters. The inclination of multicopters affects the received power, and the communication range changes based on the inclination as well. Therefore, we clarify the effect of these issues on the delay time.

  • A Coil Design and Control Method of Independent Active Shielding System for Leakage Magnetic Field Reduction of Wireless UAV Charger Open Access

    Jedok KIM  Jangyong AHN  Sungryul HUH  Kibeom KIM  Seungyoung AHN  

     
    INVITED PAPER

      Pubricized:
    2020/06/26
      Vol:
    E103-B No:9
      Page(s):
    889-898

    This paper proposes a single coil active shielding method of wireless unmanned aerial vehicle charger for leakage magnetic field reduction. A proposed shielding system eliminates the leakage magnetic field generated from the transmitting and receiving coils by generating the cancelling magnetic field. In order to enhance shielding effectiveness and preserve power transfer efficiency, shielding coil design parameters including radius and turns will analyze. Based on the analysis of coil design, shielding effectiveness and power transfer efficiency will estimate. In addition, shielding current control method corresponding to leakage magnetic field strength and phase will describe. A proposed shielding system has verified by simulations and experiments in terms of the total shielding effectiveness and power transfer efficiency measurements. The simulation and experimental results show that a proposed active shielding system has achieved 68.85% of average leakage magnetic field reduction with 1.92% of power transfer efficiency degradation. The shielding effectiveness and power transfer efficiency variation by coil design has been experimentally verified.

  • Joint Trajectory and Power Design for Secure UAV-Enabled Multicasting

    Ke WANG  Wei HENG  

     
    LETTER-Mobile Information Network and Personal Communications

      Vol:
    E103-A No:6
      Page(s):
    860-864

    This letter studies the physical layer security of an unmanned aerial vehicle (UAV)-enabled multicasting system, where a UAV serves as a mobile transmitter to send a common confidential message to a group of legitimate users under the existence of multiple eavesdroppers. The worst situation in which each eavesdropper can wiretap all legitimate users is considered. We seek to maximize the average secrecy rate by jointly optimizing the UAV's transmit power and trajectory over a given flight period. The resulting optimization problem is nonconvex and intractable to solve. To circumvent the nonconvexity, we propose an iterative algorithm to approximate the solution based on the alternating optimization and successive convex approximation methods. Simulation results validate the convergence and effectiveness of our proposed algorithm.

  • CU-MAC: A MAC Protocol for Centralized UAV Networks with Directional Antennas Open Access

    Aijing LI  Guodong WU  Chao DONG  Lei ZHANG  

     
    PAPER-Network

      Pubricized:
    2019/11/06
      Vol:
    E103-B No:5
      Page(s):
    537-544

    Media Access Control (MAC) is critical to guarantee different Quality of Service (QoS) requirements for Unmanned Aerial Vehicle (UAV) networks, such as high reliability for safety packets and high throughput for service packets. Meanwhile, due to their ability to provide lower delay and higher data rates, more UAVs are using frequently directional antennas. However, it is challenging to support different QoS in UAV networks with directional antennas, because of the high mobility of UAV which causes serious channel resource loss. In this paper, we propose CU-MAC which is a MAC protocol for Centralized UAV networks with directional antennas. First, we design a mobility prediction based time-frame optimization scheme to provide reliable broadcast service for safety packets. Then, a traffic prediction based channel allocation scheme is proposed to guarantee the priority of video packets which are the most common service packets nowadays. Simulation results show that compared with other representative protocols, CU-MAC achieves higher reliability for safety packets and improves the throughput of service packets, especially video packets.

  • Detecting Surface Defects of Wind Tubine Blades Using an Alexnet Deep Learning Algorithm Open Access

    Xiao-Yi ZHAO  Chao-Yi DONG  Peng ZHOU  Mei-Jia ZHU  Jing-Wen REN  Xiao-Yan CHEN  

     
    PAPER-Machine Learning

      Vol:
    E102-A No:12
      Page(s):
    1817-1824

    The paper employed an Alexnet, which is a deep learning framework, to automatically diagnose the damages of wind power generator blade surfaces. The original images of wind power generator blade surfaces were captured by machine visions of a 4-rotor UAV (unmanned aerial vehicle). Firstly, an 8-layer Alexnet, totally including 21 functional sub-layers, is constructed and parameterized. Secondly, the Alexnet was trained with 10000 images and then was tested by 6-turn 350 images. Finally, the statistic of network tests shows that the average accuracy of damage diagnosis by Alexnet is about 99.001%. We also trained and tested a traditional BP (Back Propagation) neural network, which have 20-neuron input layer, 5-neuron hidden layer, and 1-neuron output layer, with the same image data. The average accuracy of damage diagnosis of BP neural network is 19.424% lower than that of Alexnet. The point shows that it is feasible to apply the UAV image acquisition and the deep learning classifier to diagnose the damages of wind turbine blades in service automatically.

  • Decentralized Relay Selection for Large-Scale Dynamic UAVs Networks: A Mood-Driven Approach

    Xijian ZHONG  Yan GUO  Ning LI  Shanling LI  Aihong LU  

     
    LETTER-Communication Theory and Signals

      Vol:
    E102-A No:12
      Page(s):
    2031-2036

    In the large-scale multi-UAV systems, the direct link may be invalid for two remote nodes on account of the constrained power or complex communication environment. Idle UAVs may work as relays between the sources and destinations to enhance communication quality. In this letter, we investigate the opportunistic relay selection for the UAVs dynamic network. On account of the time-varying channel states and the variable numbers of sources and relays, relay selection becomes much more difficult. In addition, information exchange among all nodes may bring much cost and it is difficult to implement in practice. Thus, we propose a decentralized relay selection approach based on mood-driven mechanism to combat the dynamic characteristics, aiming to maximize the total capacity of the network without information exchange. With the proposed approach, the sources can make decisions only according to their own current states and update states according to immediate rewards. Numerical results show that the proposed approach has attractive properties.

  • Fair Deployment of an Unmanned Aerial Vehicle Base Station for Maximal Coverage

    Yancheng CHEN  Ning LI  Xijian ZHONG  Yan GUO  

     
    PAPER

      Pubricized:
    2019/04/26
      Vol:
    E102-B No:10
      Page(s):
    2014-2020

    Unmanned aerial vehicle mounted base stations (UAV-BSs) can provide wireless cellular service to ground users in a variety of scenarios. The efficient deployment of such UAV-BSs while optimizing the coverage area is one of the key challenges. We investigate the deployment of UAV-BS to maximize the coverage of ground users, and further analyzes the impact of the deployment of UAV-BS on the fairness of ground users. In this paper, we first calculated the location of the UAV-BS according to the QoS requirements of the ground users, and then the fairness of ground users is taken into account by calculating three different fairness indexes. The performance of two genetic algorithms, namely Standard Genetic Algorithm (SGA) and Multi-Population Genetic Algorithm (MPGA) are compared to solve the optimization problem of UAV-BS deployment. The simulations are presented showing that the performance of the two algorithms, and the fairness performance of the ground users is also given.

  • Compressed Sensing-Based Multi-Abnormality Self-Detecting and Faults Location Method for UAV Swarms

    Fei XIONG  Hai WANG  Aijing LI  Dongping YU  Guodong WU  

     
    PAPER

      Pubricized:
    2019/04/26
      Vol:
    E102-B No:10
      Page(s):
    1975-1982

    The security of Unmanned Aerial Vehicle (UAV) swarms is threatened by the deployment of anti-UAV systems under complicated environments such as battlefield. Specifically, the faults caused by anti-UAV systems exhibit sparse and compressible characteristics. In this paper, in order to improve the survivability of UAV swarms under complicated environments, we propose a novel multi-abnormality self-detecting and faults location method, which is based on compressed sensing (CS) and takes account of the communication characteristics of UAV swarms. The method can locate the faults when UAV swarms are suffering physical damages or signal attacks. Simulations confirm that the proposed method performs well in terms of abnormalities detecting and faults location when the faults quantity is less than 17% of the quantity of UAVs.

  • A Hybrid CRBP-VMP Cooperative Positioning Algorithm for Distributed Multi-UAVs

    Lu LU  Guangxia LI  Tianwei LIU  Siming LI  Shiwei TIAN  

     
    PAPER

      Pubricized:
    2019/04/26
      Vol:
    E102-B No:10
      Page(s):
    1933-1940

    Positioning information plays a significant role in multi-unmanned aerial vehicles (UAVs) applications. Traditionally, the positioning information is widely provided by Global Navigation Satellite System (GNSS) due to its good performance and global coverage. However, owing to complicated flight environment or signal blockage, jamming and unintentional interference, the UAVs may fail to locate themselves by using GNSS alone. As a new method to resolve these problems, cooperative positioning, by incorporating peer-to-peer range measurements and assisted information, has attracted more and more attentions due to its ability to enhance the accuracy and availability of positioning. However, achieving good performance of cooperative positioning of multi-UAVs is challenging as their mobility, arbitrary nonlinear state-evolution, measurement models and limited computation and communication resources. In this paper, we present a factor graph (FG) representation and message passing methodology to solve cooperative positioning problem among UAVs in 3-dimensional environment where GNSS cannot provide services. Moreover, to deal with the nonlinear state-evolution and measurement models while decreasing the computation complexity and communication cost, we develop a distributed algorithm for dynamic and hybrid UAVs by means of Spherical-Radial Cubature Rules (CR) method with belief propagation (BP) and variational message passing (VMP) methods (CRBP-VMP) on the FG. The proposed CRBP deals with the highly non-linear state-evolution models and non-Gaussian distributions, the VMP method is employed for ranging message, gets the simpler message representation and can reduce communication cost in the joint estimation problem. Simulation results demonstrate that the higher positioning accuracy, the better convergence as well as low computational complexity and communication cost of the proposed CRBP-VMP algorithm, which can be achieved compared with sum-product algorithm over a wireless network (SPAWN) and traditional Cubature Kalman Filters (CKF) method.

  • Delay-Compensated Maximum-Likelihood-Estimation Method and Its Application for Quadrotor UAVs

    Ryosuke ADACHI  Yuh YAMASHITA  

     
    PAPER-Systems and Control

      Vol:
    E101-A No:4
      Page(s):
    678-684

    This study proposes a maximum-likelihood-estimation method for a quadrotor UAV given the existence of sensor delays. The state equation of the UAV is nonlinear, and thus, we propose an approximated method that consists of two steps. The first step estimates the past state based on the delayed output through an extended Kalman filter. The second step involves calculating an estimate of the present state by simulating the original system from the past to the present. It is proven that the proposed method provides an approximated maximum-likelihood-estimation. The effectiveness of the estimator is verified by performing experiments.

  • Towards Practical Store-Carry-Forward Networking: Examples and Issues Open Access

    Masato TSURU  Mineo TAKAI  Shigeru KANEDA   Agussalim  Rabenirina AINA TSIORY  

     
    INVITED PAPER-Network

      Vol:
    E100-B No:1
      Page(s):
    2-10

    In the evolution of wireless networks such as wireless sensor networks, mobile ad-hoc networks, and delay/disruption tolerant networks, the Store-Carry-Forward (SCF) message relaying paradigm has been commonly featured and studied with much attention. SCF networking is essential for offsetting the deficiencies of intermittent and range limited communication environments because it allows moving wireless communication nodes to act as “mobile relay nodes”. Such relay nodes can store/carry/process messages, wait for a better opportunity for transmission, and finally forward the messages to other nodes. This paper starts with a short overview of SCF routing and then examines two SCF networking scenarios. The first one deals with large content delivery across multiple islands using existing infrastructural transportation networks (e.g., cars and ferries) in which mobility is uncontrollable from an SCF viewpoint. Simulations show how a simple coding technique can improve flooding-based SCF. The other scenario looks at a prototype system of unmanned aerial vehicle (UAV) for high-quality video surveillance from the sky in which mobility is partially controllable from an SCF viewpoint. Three requisite techniques in this scenario are highlighted - fast link setup, millimeter wave communications, and use of multiple links. Through these examples, we discuss the benefits and issues of the practical use of SCF networking-based systems.

  • Heuristic Function Negotiation for Markov Decision Process and Its Application in UAV Simulation

    Fengfei ZHAO  Zheng QIN  Zhuo SHAO  

     
    PAPER-Artificial Intelligence, Data Mining

      Vol:
    E97-D No:1
      Page(s):
    89-97

    The traditional reinforcement learning (RL) methods can solve Markov Decision Processes (MDPs) online, but these learning methods cannot effectively use a priori knowledge to guide the learning process. The exploration of the optimal policy is time-consuming and does not employ the information about specific issues. To tackle the problem, this paper proposes heuristic function negotiation (HFN) as an online learning framework. The HFN framework extends MDPs and introduces heuristic functions. HFN changes the state-action dual layer structure of traditional RL to the triple layer structure, in which multiple heuristic functions can be set to meet the needs required to solve the problem. The HFN framework can use different algorithms to let the functions negotiate to determine the appropriate action, and adjust the impact of each function according to the rewards. The HFN framework introduces domain knowledge by setting heuristic functions and thus speeds up the problem solving of MDPs. Furthermore, user preferences can be reflected in the learning process, which improves the flexibility of RL. The experiments show that, by setting reasonable heuristic functions, the learning results of the HFN framework are more efficient than traditional RL. We also apply HFN to the air combat simulation of unmanned aerial vehicles (UAVs), which shows that different function settings lead to different combat behaviors.

1-20hit(21hit)