Search

We study the quadratic cycle cover problem (QCCP), which aims to find a node-disjoint cycle cover in a directed graph with minimum interaction cost between successive arcs. We derive several semidefinite programming (SDP) relaxations and use facial reduction to make these strictly feasible. We investigate a nontrivial relationship between the transformation matrix used in the reduction and the structure of the graph, which is exploited in an efficient algorithm that constructs this matrix for any instance of the problem. To solve our relaxations, we propose an algorithm that incorporates an augmented Lagrangian method into a cutting-plane framework by utilizing Dykstra's projection algorithm. Our algorithm is suitable for solving SDP relaxations with a large number of cutting-planes. Computational results show that our SDP bounds and efficient cutting-plane algorithm outperform other QCCP bounding approaches from the literature. Finally, we provide several SDP-based upper bounding techniques, among which is a sequential Q-learning method that exploits a solution of our SDP relaxation within a reinforcement learning environment. Summary of Contribution: The quadratic cycle cover problem (QCCP) is the problem of finding a set of node-disjoint cycles covering all the nodes in a graph such that the total interaction cost between successive arcs is minimized. The QCCP has applications in many fields, among which are robotics, transportation, energy distribution networks, and automatic inspection. Besides this, the problem has a high theoretical relevance because of its close connection to the quadratic traveling salesman problem (QTSP). The QTSP has several applications, for example, in bioinformatics, and is considered to be among the most difficult combinatorial optimization problems nowadays. After removing the subtour elimination constraints, the QTSP boils down to the QCCP. Hence, an in-depth study of the QCCP also contributes to the construction of strong bounds for the QTSP. In this paper, we study the application of semidefinite programming (SDP) to obtain strong bounds for the QCCP. Our strongest SDP relaxation is very hard to solve by any SDP solver because of the large number of involved cutting-planes. Because of that, we propose a new approach in which an augmented Lagrangian method is incorporated into a cutting-plane framework by utilizing Dykstra's projection algorithm. We emphasize an efficient implementation of the method and perform an extensive computational study. This study shows that our method is able to handle a large number of cuts and that the resulting bounds are currently the best QCCP bounds in the literature. We also introduce several upper bounding techniques, among which is a distributed reinforcement learning algorithm that exploits our SDP relaxations. [ABSTRACT FROM AUTHOR]

In order to improve the effect of modern music education, this paper applies the digital information technology of music resources to the construction of the music teaching system and derives two new types of semi-decision-making process reinforcement learning algorithms based on the Bellman optimality equation base on discrete time. Moreover, this paper uses the comparative research methods to obtain the Q-value learning curve of the incremental value iterative reinforcement learning algorithm based on the semi-Markov decision process and the incremental value iterative reinforcement learning algorithm based on the dichotomy to improve the fusion effect of music teaching resources. Finally, this paper combines the actual needs of modern music education to construct an intelligent music teaching model. [ABSTRACT FROM AUTHOR]

This document is a guest editorial for a special issue on computational methods and artificial intelligence applications in low-carbon energy systems. The editorial highlights the urgent need for advanced computing and artificial intelligence in the clean energy transition to improve system reliability, economics, and sustainability. The special issue includes 19 original research articles covering topics such as energy forecasting, situational awareness, multi-energy system dispatch, and power system operation. The articles present state-of-the-art methods and techniques in these areas, including wind power forecasting, demand-side flexibility, fault diagnosis of photovoltaic strings, and energy management strategies. The authors express their gratitude to the participating authors and anonymous reviewers for their contributions to the special section. [Extracted from the article]

This document is an editorial for a special issue of the journal "Technologies" focused on data science and big data in various fields such as biology, physical science, and engineering. The editorial highlights the importance of analyzing large amounts of data generated by digital technologies and the need for data scientists to use artificial intelligence and machine learning to extract valuable knowledge. The special issue includes 12 papers covering topics such as machine learning techniques for customer churn prediction, agile program management in the U.S. Navy, deep learning for cybersecurity in Industry 5.0, self-directed learning during the COVID-19 era, decision tree-based neural networks for data classification, data-driven governance in technology companies, and more. The papers explore different approaches, models, and tools in the context of data science and big data. [Extracted from the article]

This document discusses the rapid development of Artificial Intelligence (AI) and its bioinspired applications. It highlights the benefits of bioinspired AI, such as increased accuracy in image and speech processing, reduced cost and energy usage through edge devices, and enhanced bio-signal quality. However, it also acknowledges the challenges posed by improper AI utilization, such as the generation of fake news and security issues. The document calls for research papers on bioinspired AI applications to explore its potential and address these challenges. It includes examples of research papers that utilize deep reinforcement learning for robot task sequencing, propose a real-time multi-surveillance pedestrian target detection model, develop an intelligent breast mass classification approach, and introduce a bio-inspired object detection algorithm for remote sensing images. The document concludes by emphasizing the importance of biomimetic artificial intelligence in various fields and promoting further research in this area. [Extracted from the article]

This paper presents a collaborative filtering algorithm based on reinforcement learning theory. Then, the personalized bank financial recommendation system for users is constructed in the massive data environment. Tags mimic different types of user interest points to build a representative personalized data set. The collaborative screening of bank financial products is realized using the simulation results and users' historical access records. The ranking calculation of related financial products is added to the general bank financial product recommendation system. This method can more accurately express the query results for a specific user. It is found that the collaborative filtering algorithm based on enhanced learning theory can improve the efficiency of collaborative screening of bank financial products. The best results can be obtained by combining the two organically. This paper proposes that the recommendation algorithm of reinforcement learning bank financial products based on user preference and collaborative filtering is feasible. [ABSTRACT FROM AUTHOR]

Addressing the formidable challenges posed by multiple jammers jamming multiple radars, which arise from spatial discretization, many degrees of freedom, numerous model input parameters, and the complexity of constraints, along with a multi-peaked objective function, this paper proposes a cooperative jamming resource allocation method, based on evolutionary reinforcement learning, that uses joint multi-domain information. Firstly, an adversarial scenario model is established, characterizing the interaction between multiple jammers and radars based on a multi-beam jammer model and a radar detection model. Subsequently, considering real-world scenarios, this paper analyzes the constraints and objective function involved in cooperative jamming resource allocation by multiple jammers. Finally, accounting for the impact of spatial, frequency, and energy domain information on jamming resource allocation, matrices representing spatial condition constraints, jamming beam allocation, and jamming power allocation are formulated to characterize the cooperative jamming resource allocation problem. Based on this foundation, the joint allocation of the jamming beam and jamming power is optimized under the constraints of jamming resources. Through simulation experiments, it was determined that, compared to the dung beetle optimizer (DBO) algorithm and the particle swarm optimization (PSO) algorithm, the proposed evolutionary reinforcement learning algorithm based on DBO and Q-Learning (DBO-QL) offers 3.03% and 6.25% improvements in terms of jamming benefit and 26.33% and 50.26% improvements in terms of optimization success rate, respectively. In terms of algorithm response time, the proposed hybrid DBO-QL algorithm has a response time of 0.11 s, which is 97.35% and 96.57% lower than the response times of the DBO and PSO algorithms, respectively. The results show that the method proposed in this paper has good convergence, stability, and timeliness. [ABSTRACT FROM AUTHOR]

The low carbon park islanded microgrid faces operational challenges due to the high variability and uncertainty of distributed renewable energy sources. These sources cause severe random disturbances that impair the frequency control performance and increase the regulation cost of the islanded microgrid, jeopardizing its safety and stability. This paper presents a data-driven intelligent load frequency control (DDI-LFC) method to address this problem. The method replaces the conventional LFC controller with an intelligent agent based on a deep reinforcement learning algorithm. To adapt to the complex islanded microgrid environment and achieve adaptive multi-objective optimal frequency control, this paper proposes the quantum-inspired maximum entropy actor-critic (QIS-MEAC) algorithm, which incorporates the quantum-inspired principle and the maximum entropy exploration strategy into the actor-critic algorithm. The algorithm transforms the experience into a quantum state and leverages the quantum features to improve the deep reinforcement learning's experience replay mechanism, enhancing the data efficiency and robustness of the algorithm and thus the quality of DDI-LFC. The validation on the Yongxing Island isolated microgrid model of China Southern Grid (CSG) demonstrates that the proposed method utilizes the frequency regulation potential of distributed generation, and reduces the frequency deviation and generation cost. [ABSTRACT FROM AUTHOR]

This paper considers the confrontation problem for two tank swarms of equal size and capability in a complex urban scenario. Based on the Unity platform (2022.3.20f1c1), the confrontation scenario is constructed featuring multiple crossing roads. Through the analysis of a substantial amount of biological data and wildlife videos regarding animal behavioral strategies during confrontations for hunting or food competition, two strategies are been utilized to design a novel bio-inspired intelligent swarm confrontation algorithm. The first one is the "fire concentration" strategy, which assigns a target for each tank in a way that the isolated opponent will be preferentially attacked with concentrated firepower. The second one is the "back and forth maneuver" strategy, which makes the tank tactically retreat after firing in order to avoid being hit when the shell is reloading. Two state-of-the-art swarm confrontation algorithms, namely the reinforcement learning algorithm and the assign nearest algorithm, are chosen as the opponents for the bio-inspired swarm confrontation algorithm proposed in this paper. Data of comprehensive confrontation tests show that the bio-inspired swarm confrontation algorithm has significant advantages over its opponents from the aspects of both win rate and efficiency. Moreover, we discuss how vital algorithm parameters would influence the performance indices. [ABSTRACT FROM AUTHOR]

The article reports that With the continued growth of IoT devices and their deployment, manually managing and connecting them is impractical and presents multiple challenges. To that end, Zero Touch Networks that rely on software-based modules instead of dedicated propriety hardware become a viable potential solution. The overall aim of zero-touch networks is for machines to learn how to become more autonomous so that we can delegate complex, mundane tasks to them.

In reviewing this Special Issue, various topics have been addressed, predominantly machine learning techniques and heuristic search algorithms. Machine learning and computational intelligence are currently high-profile research areas attracting the attention of many researchers. In the first paper, L. Zhao and H. Jin improved the traditional vector-weighted optimization algorithm (INFO) and designed a promising optimization algorithm (IDEINFO) [[8]]. [Extracted from the article]

Artificial intelligence is changing and influencing our world. As one of the main algorithms in the field of artificial intelligence, convolutional neural networks (CNNs) have developed rapidly in recent years. Especially after the emergence of NASNet, CNNs have gradually pushed the idea of AutoML to the public's attention, and large numbers of new structures designed by automatic searches are appearing. These networks are usually based on reinforcement learning and evolutionary learning algorithms. However, sometimes, the blocks of these networks are complex, and there is no small model for simpler tasks. Therefore, this paper proposes POSS-CNN aiming at target recognition and detection, which employs a multi-branch CNN structure with PSNC and a method of automatic parallel selection for super parameters based on a multi-branch CNN structure. Moreover, POSS-CNN can be broken up. By choosing a single branch or the combination of two branches as the "benchmark", as well as the overall POSS-CNN, we can achieve seven models with different precision and operations. The test accuracy of POSS-CNN for a recognition task tested on a CIFAR10 dataset can reach 86.4%, which is equivalent to AlexNet and VggNet, but the operation and parameters of the whole model in this paper are 45.9% and 45.8% of AlexNet, and 29.5% and 29.4% of VggNet. The mAP of POSS-CNN for a detection task tested on the LSVH dataset is 45.8, inferior to the 62.3 of YOLOv3. However, compared with YOLOv3, the operation and parameters of the model in this paper are reduced by 57.4% and 15.6%, respectively. After being accelerated by WRA, POSS-CNN for a detection task tested on an LSVH dataset can achieve 27 fps, and the energy efficiency is 0.42 J/f, which is 5 times and 96.6 times better than GPU 2080Ti in performance and energy efficiency, respectively. [ABSTRACT FROM AUTHOR]

How to find an effective trading policy is still an open question mainly due to the nonlinear and non-stationary dynamics in a financial market. Deep reinforcement learning, which has recently been used to develop trading strategies by automatically extracting complex features from a large amount of data, is struggling to deal with fastchanging markets due to sample inefficiency. This paper applies the meta-reinforcement learning method to tackle the trading challenges faced by conventional reinforcement learning (RL) approaches in non-stationary markets for the first time. In our work, the history trading data is divided into multiple task data and for each of these data themarket condition is relatively stationary. Then amodel agnosticmeta-learning (MAML)-based tradingmethod involving a meta-learner and a normal learner is proposed. A trading policy is learned by the meta-learner across multiple task data, which is then fine-tuned by the normal learner through a small amount of data from a new market task before trading in it. To improve the adaptability of the MAML-based method, an ordered multiplestep updating mechanism is also proposed to explore the changing dynamic within a task market. The simulation results demonstrate that the proposed MAML-based trading methods can increase the annualized return rate by approximately 180%, 200%, and 160%, increase the Sharpe ratio by 180%, 90%, and 170%, and decrease the maximum drawdown by 30%, 20%, and 40%, compared to the traditional RL approach in three stock index future markets, respectively. [ABSTRACT FROM AUTHOR]

Several deep learning models are presented: Ref. [[4]] designs a network intrusion detection model, DLNID, that combines an attention mechanism, a bidirectional long short-term memory (LSTM), and an adaptive synthetic sampling, and demonstrates improved performance for severely imbalanced data. In addition, there are a couple of papers on HPO: Ref. [[11]] introduces a greedy I k i -fold cross validation method that vastly reduces the average time required to find the best-performing model with or without a computational budget, and experimentally verifies its improved performance over existing methods. Since its inception as a branch of Artificial Intelligence, Machine Learning (ML) has flourished in recent years. [Extracted from the article]

In the paper entitled "Spam Detection Approach for Secure Mobile Message Communication Using Machine Learning Algorithms", L. G. Jun et al. proposed the applications of the machine learning-based spam detection method for accurate detection. The data extraction process of the review showed that (1) latent Dirichlet allocation topic modeling is among the widely used machine learning algorithms in the selected studies and (2) precision and recall are amongst the most utilized evaluation methods for measuring the performance of these machine learning algorithms. Machine learning (ML) and cryptography have many things in common, for instance, the amount of data to be handled and large search spaces. [Extracted from the article]

With the increase in the integration of renewable sources, the home energy management system (HEMS) has become a promising approach to improve grid energy efficiency and relieve network stress. In this context, this paper proposes an optimization dispatching strategy for HEMS to reduce total cost with full consideration of uncertainties, while ensuring the users' comfort. Firstly, a HEMS dispatching model is constructed to reasonably schedule the start/stop time of the dispatchable appliances and energy storage system to minimize the total cost for home users. Besides, this dispatching strategy also controls the switching time of temperature-controlled load such as air conditioning to reduce the energy consumption while maintaining the indoor temperature in a comfortable level. Then, the optimal dispatching problem of HEMS is modeled as a Markov decision process (MDP) and solved by a deep reinforcement learning algorithm called deep deterministic policy gradient. The example results verify the effectiveness and superiority of the proposed method. The energy cost can be effectively reduced by 21.9% at least compared with other benchmarks and the indoor temperature can be well maintained. [ABSTRACT FROM AUTHOR]

Load models have a significant influence on power system simulation. However, current load modeling approaches can hardly satisfy the diversity and time- varying characteristics of loads [including electric vehicles (EVs) and battery storage] in terms of model accuracy and computing efficiency. An online modeling method for composite load models based on measurement information is proposed in this paper. Firstly, the dominant factors in load model output are analyzed based on the active subspace of parameter space. Then the clustering algorithm is applied to cluster the large number of underlying loads based on the characteristics of load daily output curves. Finally, the underlying loads are equivalently aggregated from the low voltage levels to the high voltage levels to construct the composite load model. Simulation results obtained based on PSCAD/EMTDC demonstrate that the load model constructed by the proposed approach can accurately reflect the actual load characteristics of a power system. [ABSTRACT FROM AUTHOR]

As the demand for sensing and monitoring the marine environment increases, the Ocean Mobile Internet of Things (OM-IoT) has gradually attracted the interest of researchers. However, the unreliability of communication links represents a significant challenge to data transmission in the OM-IoT, given the complex and dynamic nature of the marine environment, the mobility of nodes, and other factors. Consequently, it is necessary to enhance the reliability of underwater data transmission. To address this issue, this paper proposes a reinforcement learning-based adaptive network coding (RL-ANC) approach. Firstly, the channel conditions are estimated based on the reception acknowledgment, and a feedback-independent decoding state estimation method is proposed. Secondly, the sliding coding window is dynamically adjusted based on the estimates of the channel erasure probability and decoding probability, and the sliding rule is adaptively determined using a reinforcement learning algorithm and an enhanced greedy strategy. Subsequently, an adaptive optimization method for coding coefficients based on reinforcement learning is proposed to enhance the reliability of the underwater data transmission and underwater network coding while reducing the redundancy in the coding. Finally, the sampling period and time slot table are updated using the enhanced simulated annealing algorithm to optimize the accuracy and timeliness of the channel estimation. Simulation experiments demonstrate that the proposed method effectively enhances the data transmission reliability in unreliable communication links, improves the performance of underwater network coding in terms of the packet delivery rate, retransmission, and redundancy transmission ratios, and accelerates the convergence speed of the decoding probability. [ABSTRACT FROM AUTHOR]

With the increasing popularity of the Internet of Things (IoT), the proliferation of computation-intensive and timing-dependent applications has brought serious load pressure on terrestrial networks. In order to solve the problem of computing resource conflict and long response delay caused by concurrent application service applications from multiple users, this paper proposes an improved edge computing timing-dependent, task-offloading scheme based on Multi-Agent Deep Deterministic Policy Gradient (MADDPG) that aims to shorten the offloading delay and improve the resource utilization rate by means of resource prediction and collaboration among multiple agents to shorten the offloading delay and improve the resource utilization. First, to coordinate the global computing resource, the gated recurrent unit is utilized, which predicts the next computing resource requirements of the timing-dependent tasks according to historical information. Second, the predicted information, the historical offloading decisions and the current state are used as inputs, and the training process of the reinforcement learning algorithm is improved to propose a task-offloading algorithm based on MADDPG. The simulation results show that the algorithm reduces the response latency by 6.7% and improves the resource utilization by 30.6% compared with the suboptimal benchmark algorithm, and it reduces nearly 500 training rounds during the learning process, which effectively improves the timeliness of the offloading strategy. [ABSTRACT FROM AUTHOR]

The advent of Industry 4.0 has significantly promoted the field of intelligent manufacturing, which is facilitated by the development of new technologies are emerging. Robot technology and robot intelligence methods have rapidly developed and been widely applied. Manipulators are widely used in industry, and their control is a crucial research topic. The inverse kinematics solution of manipulators is an important part of manipulator control, which calculates the joint angles required for the end effector to reach a desired position and posture. Traditional inverse kinematics solution algorithms often face the problem of insufficient generalization, and iterative methods have challenges such as large computation and long solution time. This paper proposes a reinforcement learning-based inverse kinematics solution algorithm, called the MAPPO-IK algorithm. The algorithm trains the manipulator agent using the MAPPO algorithm and calculates the difference between the end effector state of the manipulator and the target posture in real-time by designing a reward mechanism, while considering Gaussian distance and cosine distance. Through experimental comparative analysis, the feasibility, computational efficiency, and superiority of this reinforcement learning algorithm are verified. Compared with traditional inverse kinematics solution algorithms, this method has good generalization and supports real-time computation, and the obtained result is a unique solution. Reinforcement learning algorithms have better adaptability to complex environments and can handle different sudden situations in different environments. This algorithm also has the advantages of path planning, intelligent obstacle avoidance, and other advantages in dynamically processing complex environmental scenes. [ABSTRACT FROM AUTHOR]

The study proposes a method for the trajectory tracking control of a fixed-wing unmanned aerial vehicle (UAV) based on the deep deterministic policy gradient (DDPG). First, the problem of controlling the trajectory of a fixed-wing UAV is combined with the reinforcement learning framework and transformed into a Markov decision process, and a DDPG agent is established in the framework of TensorFlow. Second, we conducted simulations to train and optimize the model in a 3D environment of trajectory tracking control and obtained an integrated DDPG-based trajectory tracking controller that can regulate functions ranging from the state of flight of the UAV to rudder control. Third, we constructed a digital simulation system to verify the proposed method while considering the influence of parametric uncertainties, measurement-induced noise, and delays in the response of the control system. The effectiveness and robustness of the proposed DDPG controller were verified by comparing its performance with that of traditional proportional-integral-derivative (PID) control. Unmanned aerial vehicles have gained widespread use across numerous sectors of society. However, the control process for fixed-wing UAVs is more intricate and challenging than that for rotor type UAVs due to the coupled and nonlinear nature of their dynamic models. To address this issue, decoupling and splitting hierarchical control methods are commonly employed. However, these methods can introduce errors and affect flight performance. In contrast, artificial intelligence, particularly deep learning and reinforcement learning, has made significant advancements and found successful applications in areas such as robot control and gaming. To address the challenges posed by complex forms, excessive measurement information, and manual parameter tuning associated with traditional control approaches for fixed-wing UAVs, this paper introduces a deep reinforcement learning algorithm, i.e., the deep deterministic policy gradient, into the control process of fixed-wing UAVs. The paper presents a trajectory tracking control policy for UAVs, which efficiently achieves the task of tracking a desired trajectory without the need for selecting multiple working state nodes or designing linear control laws. Further, through simulation verification, the proposed policy demonstrates adaptability to interference caused by parameter bias, measurement noise, and control system response delays while achieving control accuracy comparable with classical control methods. [ABSTRACT FROM AUTHOR]

The pricing behavior of agricultural processing firms in input markets has large impacts on farmers' and processors' prosperity as well as the overall market structure. Despite analytical approaches to food processors' pricing in agricultural input markets, the need for models to represent complex market features is urgent. Agent-based models (ABMs) serve as computational laboratories to understand complex markets emerging from autonomously interacting agents. Yet, individual agents within ABMs must be equipped with intelligent learning algorithms. In this paper, we propose supervised and unsupervised learning agents to simulate the pricing behavior of firms in agricultural markets' ABMs. Supervised learning firms are pre-trained to accurately best respond to their competitors and are deemed to result in the market Nash equilibria. Unsupervised learning firms play a course of pricing interaction with their competitors without any pre-knowledge but based on deep reinforcement learning. The simulation results show that unsupervised deep learning firms are capable of approximating the pricing equilibria obtained by the supervised firms in different spatial market settings. Optimal discriminatory and uniform delivery pricing emerges in agricultural input markets with the high and intermediary importance placed on space. Free on board pricing emerges in agricultural input markets with small importance placed on space. [ABSTRACT FROM AUTHOR]

The transportation sector remains a major contributor to greenhouse gas emissions. The understanding of energy-efficient driving behaviors and utilization of energy-efficient driving strategies are essential to reduce vehicles' fuel consumption. However, there is no comprehensive investigation into energy-efficient driving behaviors and strategies. Furthermore, many state-of-the-art AI models have been applied for the analysis of eco-friendly driving styles, but no overview is available. To fill the gap, this paper conducts a thorough literature review on ecological driving behaviors and styles, and analyzes the driving factors influencing energy consumption and state-of-the-art methodologies. With a thorough scoping review process, thirty-seven articles with full text were assessed, and the methodological and related data are compared. The results show that the factors that impact driving behaviors can be summarized into eleven features including speed, acceleration, deceleration, pedal, steering, gear, engine, distance, weather, traffic signal, and road parameters. This paper finds that supervised/unsupervised learning algorithms and reinforcement learning frameworks have been popularly used to model the vehicle's energy consumption with multi-dimensional data. Furthermore, the literature shows that the driving data are collected from either simulators or real-world experiments, and the real-world data are mainly stored and transmitted by meters, controller area networks, onboard data services, smartphones, and additional sensors installed in the vehicle. Based on driving behavior factors, driver characteristics, and safety rules, this paper recommends nine energy-efficient driving styles including four guidelines for the drivers' selection and adjustment of the vehicle parameters, three recommendations for the energy-efficient driving styles in different driving scenarios, and two subjective suggestions for different types of drivers and employers. [ABSTRACT FROM AUTHOR]

Chinese Folk Drum music is an excellent traditional cultural resource, it has brilliant historical and cultural heritage and excellent traditional cultural connotation. However, the survey found that the social and cultural values, tourism economic values, and national self-confidence embodied in folk drum music, such as Xi'an drum music, are far from being released, and even its own inheritance and development are facing difficulties. The research focuses on the automatic generation of Xi'an drum music, with the aim of further inheriting, developing, and utilizing this exceptional traditional cultural resource. While Artificial Intelligence (AI) music generation has gained popularity in recent years, most platforms primarily focus on modern music rather than Chinese folk music. To address these issues and the unique challenges faced by Xi'an drum music, this paper proposes a Bi-LSTM network-based deep reinforcement learning model. The model incorporates the distinctive characteristics of ancient Chinese music, such as pitch, chord, and mode, and utilizes the Actor-Critic algorithm in reinforcement learning. During the simulation generation stage, an improved method of generating strategies through reward and punishment scores is introduced. Additionally, the model takes into account abstract concept constraints, such as chord progression and music theory rules, which are translated into computer language. By constructing a chord reward mechanism and a music principle reward mechanism, the model achieves harmony constraints and enables the systematic generation of drum music. Experimental results demonstrate that the proposed model, based on Bi-LSTM deep reinforcement learning, can generate Xi'an drum music with high quality and artistic aesthetics. This research contributes to the preservation, development, and utilization of Xi'an drum music, leveraging advancements in AI music generation technology. [ABSTRACT FROM AUTHOR]

This research paper provides a comprehensive review of the emerging trends in AI-based stock market prediction. The paper highlights the key concepts, approaches, and techniques employed in AI-based stock market prediction and discusses their strengths and limitations. Key topics covered include deep learning, natural language processing, sentiment analysis, and reinforcement learning. This paper also presents case studies and evaluates the performance of different AI-based models in predicting stock market trends. Overall, this research paper provides valuable insights into the latest advancements in AI-based stock market prediction and their potential implications for investors, financial analysts, and policy makers. [ABSTRACT FROM AUTHOR]

There have been a number of approaches to employ neural networks in self-adaptive systems; in many cases, generic neural networks and deep learning are utilized for this purpose. When this approach is to be applied to improve an adaptation process initially driven by logical adaptation rules, the problem is that (1) these rules represent a significant and tested body of domain knowledge, which may be lost if they are replaced by a neural network, and (2) the learning process is inherently demanding given the black-box nature and the number of weights in generic neural networks to be trained. In this paper, we introduce the rule-specific neural network method that makes it possible to transform the guard of an adaptation rule into a rule-specific neural network, the composition of which is driven by the structure of the logical predicates in the guard. Our experiments confirmed that the black box effect is eliminated, the number of weights is significantly reduced, and much faster learning is achieved whilst the accuracy is preserved. This text is an extended version of the paper presented at the ISOLA 2022 conference (Bureš et al. in Proceedings of ISOLA 2022, Rhodes, Greece, pp. 215–230, 2022). [ABSTRACT FROM AUTHOR]

"Multi-agent systems (MAS)" have been extensively studied across various fields, including robotics, economics, biology, and computer science. A distinctive feature of these systems is the ability of multiple agents, each with different characteristics, to perform system-wide tasks through local bottom-up interactions. Furthermore, design and control methods for system networks based on graph theory are being developed. Recent applications of these methods include autonomous driving technology, smart grids, and understanding social systems. This special issue aims to deepen the understanding of MAS, focusing on their control and applications. It features 16 papers, including one review paper. The accepted papers cover a wide range of topics, including reinforcement learning, autonomous mobility systems, and machine learning, presenting the latest research findings on MAS. These studies provide valuable insights into various aspects and potential applications of MAS. We hope that this issue will be beneficial to our readers and contribute to the advancement of future research. [ABSTRACT FROM AUTHOR]

In integrated circuit (IC) design, floorplanning is an important stage in obtaining the floorplan of the circuit to be designed. Floorplanning determines the performance, size, yield, and reliability of very large-scale integration circuit (VLSI) ICs. The results obtained in this step are necessary for the subsequent continuous processes of chip design. From a computational perspective, VLSI floorplanning is an NP-hard problem, making it difficult to be efficiently solved by classical optimization techniques. In this paper, we propose a deep reinforcement learning floorplanning algorithm based on sequence pairs (SP) to address the placement problem. Reinforcement learning utilizes an agent to explore the search space in sequence pairs to find the optimal solution. Experimental results on the international standard test circuit benchmarks, MCNC and GSRC, demonstrate that the proposed deep reinforcement learning floorplanning algorithm based on sequence pairs can produce a superior solution. [ABSTRACT FROM AUTHOR]

To address the problem of low accuracy and efficiency in trajectory planning algorithms for interceptors facing multiple constraints during the midcourse guidance phase, an improved trajectory convex programming method based on the lateral distance domain is proposed. This algorithm can achieve fast trajectory planning, reduce the approximation error of the planned trajectory, and improve the accuracy of trajectory guidance. First, the concept of lateral distance domain is proposed, and the motion model of the midcourse guidance segment in the interceptor is converted from the time domain to the lateral distance domain. Second, the motion model and multiple constraints are convexly and discretely transformed, and the discrete trajectory convex model is established in the lateral distance domain. Third, the deep reinforcement learning algorithm is used to learn and train the initial solution of trajectory convex programming, and a high-quality initial solution trajectory is obtained. Finally, a dynamic adjustment method based on the distribution of approximate solution errors is designed to achieve efficient dynamic adjustment of grid points in iterative solving. The simulation experiments show that the improved trajectory convex programming algorithm proposed in this paper not only improves the accuracy and efficiency of the algorithm but also has good optimization performance. [ABSTRACT FROM AUTHOR]

Today, reinforcement learning is one of the most effective machine learning approaches in the tasks of automatically adapting computer systems to user needs. However, implementing this technology into a digital product requires addressing a key challenge: determining the reward model in the digital environment. This paper proposes a usability reward model in multi-agent reinforcement learning. Well-known mathematical formulas used for measuring usability metrics were analyzed in detail and incorporated into the usability reward model. In the usability reward model, any neural network-based multi-agent reinforcement learning algorithm can be used as the underlying learning algorithm. This paper presents a study using independent and actor-critic reinforcement learning algorithms to investigate their impact on the usability metrics of a mobile user interface. Computational experiments and usability tests were conducted in a specially designed multi-agent environment for mobile user interfaces, enabling the implementation of various usage scenarios and real-time adaptations. [ABSTRACT FROM AUTHOR]

Mean-field games (MFGs) are developed to model the decision-making processes of a large number of interacting agents in multi-agent systems. This paper studies mean-field games on graphs (G -MFGs). The equilibria of G -MFGs, namely, mean-field equilibria (MFE), are challenging to solve for their high-dimensional action space because each agent has to make decisions when they are at junction nodes or on edges. Furthermore, when the initial population state varies on graphs, we have to recompute MFE, which could be computationally challenging and memory-demanding. To improve the scalability and avoid repeatedly solving G -MFGs every time their initial state changes, this paper proposes physics-informed graph neural operators (PIGNO). The PIGNO utilizes a graph neural operator to generate population dynamics, given initial population distributions. To better train the neural operator, it leverages physics knowledge to propagate population state transitions on graphs. A learning algorithm is developed, and its performance is evaluated on autonomous driving games on road networks. Our results demonstrate that the PIGNO is scalable and generalizable when tested under unseen initial conditions. [ABSTRACT FROM AUTHOR]

In data center networks, when facing challenges such as traffic volatility, low resource utilization, and the difficulty of a single traffic scheduling strategy to meet demands, it is necessary to introduce intelligent traffic scheduling mechanisms to improve network resource utilization, optimize network performance, and adapt to the traffic scheduling requirements in a dynamic environment. This paper proposes a fine-grained traffic scheduling scheme based on multi-agent deep reinforcement learning (MAFS). This approach utilizes In-Band Network Telemetry to collect real-time network states on the programmable data plane, establishes the mapping relationship between real-time network state information and the forwarding efficiency on the control plane, and designs a multi-agent deep reinforcement learning algorithm to calculate the optimal routing strategy under the current network state. The experimental results demonstrate that compared to other traffic scheduling methods, MAFS can effectively enhance network throughput. It achieves a 1.2× better average throughput and achieves a 1.4–1.7× lower packet loss rate. [ABSTRACT FROM AUTHOR]

In contemporary warfare, radar countermeasures have become multifunctional and intelligent, rendering the conventional jamming method and platform unsuitable for the modern radar countermeasures battlefield due to their limited efficiency. Reinforcement learning has been proven to be a practical solution for cognitive jamming decision-making in the cognitive electronic warfare. In this paper, we proposed a radar-jamming decision-making algorithm based on an improved Q-Learning algorithm. This improved Q-Learning algorithm ameliorated the problem of overestimating the Q-value that exists in the Q-Learning algorithm by introducing a second Q-table. At the same time, we performed a comprehensive design and implementation based on the classical Q-Learning algorithm, deploying it to a Field Programmable Gate Array (FPGA) hardware. We decomposed the implementation of the reinforcement learning algorithm into individual steps and described each step using a hardware description language. Then, the reinforcement learning algorithm can be computed on FPGA by linking the logic modules with valid signals. Experiments show that the proposed Q-Learning algorithm obtains considerable improvement in performance over the classical Q-Learning algorithm. Additionally, they confirm that the FPGA hardware can achieve great efficiency improvement on the radar-jamming decision-making algorithm implementation. [ABSTRACT FROM AUTHOR]

Fuel cell hybrid electric trucks have become a cutting-edge field in understanding urban traffic emissions due to their enormous potential in low-carbon areas. In order to improve the economy of fuel cell hybrid electric trucks and reduce the decline of fuel cell lifespan, this paper proposes a multi-objective energy management strategy that optimizes weight coefficients. On the basis of establishing a fuel cell battery hybrid system model, three modes of uniform speed, acceleration, and deceleration were identified through clustering analysis of vehicle speed. Reinforcement learning algorithms were used to learn the corresponding weights for different modes, which reduced the decline in fuel cell life while improving the economic efficiency. The simulation results indicate that, under the conditions of no load, half load, and full load, the truck only sacrificed 0.9–5.6%, 1.7–2.6%, and 1.2–1.6% SOC, saving 5.7–6.45%, 5.9–6.67%, and 6.1–6.67% in lifespan loss, and reducing hydrogen consumption by 3.0–7.1%, 2.8–4.4%, and 1.0–3.0%, respectively. [ABSTRACT FROM AUTHOR]

Traditional aerodynamic modeling methods have some difficulties in multiparameter unsteady state-space modeling and control law design, which raise significant challenges in flight control. The rapid development of machine learning provides a new idea for unsteady aerodynamic modeling and control law design, which has significant value for theoretical research and engineering application. In this paper, an unsteady aerodynamic model environment was established based on deep neural network (DNN), and a dynamic computational fluid dynamics (CFD) virtual environment was taken as an approximation of the real environment. A deep reinforcement learning (DRL) longitudinal-axis flight control method was studied on the basis of both environments. The deep deterministic policy gradient (DDPG) algorithm was used to implement flight control, and the effects of different constraints in the reward function on the results of DRL were studied at the same time. The results show that the DDPG agent effectively achieves longitudinal-axis flight control in the model environment. The DDPG agent trained in the model environment was also used for longitudinal-axis flight control in the dynamic CFD virtual environment, and this method provides a reference for the coupling of dynamic CFD and DRL. The control results from the model environment and the dynamic CFD virtual environment are compared, and results show that the DDPG agent has good robustness in both environments. The results of this study suggest that machine learning and deep reinforcement learning can effectively solve complicated multiconstraint, uncertain, and sophisticated control tasks. In this paper, a DNN-based unsteady aerodynamic modeling method was established based on dynamic computational fluid dynamics calculation and machine learning, and the results show that the DNN-based unsteady aerodynamic model has advantages in dealing with highly nonlinear data under unsteady conditions. In addition, the deep deterministic policy gradient algorithm can be well applied in longitudinal-axis flight control, and the agent can be trained in DNN-based unsteady aerodynamic model environment. The agent was also used for longitudinal-axis flight control in the dynamic computational fluid dynamics virtual environment, and the results show that the agent has good robustness in both environments. This provides the possibility for the agent to achieve longitudinal-axis flight control in a real environment. [ABSTRACT FROM AUTHOR]

The objective of portfolio diversification is to reduce risk and potentially enhance returns by spreading investments across different asset classes. Existing portfolio diversification models have traditionally been trained on historical financial time series data. However, several issues arise with historical financial time series data, making it challenging to train models effectively to achieve the portfolio diversification objective: an insufficient amount of training data and the uncertainty deficiency problem, wherein the uncertainty that existed in the past is not visible in the present. Insufficient datasets, characterized by small data size, result in information asymmetry and compromise portfolio performance. This limitation underscores the importance of adopting a pattern-centric data augmentation approach, capable of unveiling hidden patterns and structures within the financial time series data. To address these challenges, this paper introduces the financial time series decomposition-based variational encoder-decoder (FED) method to augment financial time series data, overcoming the limitations of insufficient training data and providing a more realistic and dynamic simulation of the financial market environment. By decomposing the data into distinct components, such as trend, dispersion, and residual, FED leverages pattern-centric data augmentation within the financial time series data. In the environment generated using the FED method, this paper proposes a two-class portfolio diversification, called FED2Port. It integrates stochastic elements into the reward function, enabling a reinforcement learning algorithm to learn from a comprehensive spectrum of financial market uncertainties. The experimental results demonstrate that the proposed model significantly enhances portfolio performance. [ABSTRACT FROM AUTHOR]

The challenge of water level control in steam generators, particularly at low power levels, has always been a critical aspect of nuclear power plant operation. To address this issue, this paper introduces an IHA controller. This controller employs a CPI controller as the primary controller for direct water level control, coupled with an agent-based controller optimized through a DRL algorithm. The agent dynamically optimizes the parameters of the CPI controller in real-time based on the system's state, resulting in improved control performance. Firstly, a new observer information is obtained to get the accurate state of the system, and a new reward function is constructed to evaluate the status of the system and guide the agent's learning process. Secondly, a deep ResNet with good generalization performance is used as the approximator of action value function and policy function. Then, the DDPG algorithm is used to train the agent-based controller, and an advanced controller with good performance is obtained after training. Finally, the popular UTSG model is used to verify the effectiveness of the algorithm. The results demonstrate that the proposed method achieves rise times of 73.9 s, 13.6 s, and 16.4 s at low, medium, and high power levels, respectively. Particularly, at low power levels, the IHA controller can restore the water level to its normal state within 200 s. These performances surpass those of the comparative methods, indicating that the proposed method excels not only in water level tracking but also in anti-interference capabilities. In essence, the IHA controller can autonomously learn the control strategy and reduce its reliance on the expert system, achieving true autonomous control and delivering excellent control performance. [ABSTRACT FROM AUTHOR]

An Overview of Published Articles This Special Issue consists of fifteen practical papers covering key topics in the field of multi-agent systems and complex systems. Using multi-agent simulation, the authors simulated the bounded rational decision-making and interaction process of multiple agents in multiple periods and analyzed the influence of the main parameters. Introduction This Editorial summarizes the content of the Special Issue entitled I Advancements in The Practical Applications of i Agents, Multi-Agent Systems I and Simulating Complex Systems i , published in the "Complex Systems" section of I Systems i (ISSN 2079-8954). Conclusions This Special Issue showcases a variety of research papers on practical approaches to the use of complex systems and complementary agent-based AI models, facilitating the parallel use of data treatment and knowledge processing algorithms. [Extracted from the article]

In recent years, sensor components similar to human sensory functions have been rapidly developed in the hardware field, enabling the acquisition of information at a level beyond that of humans, and in the software field, artificial intelligence technology has been utilized to enable cognitive abilities and decision-making such as prediction, analysis, and judgment. These changes are being utilized in various industries and fields. In particular, new hardware and software technologies are being rapidly applied to robotics products, showing a level of performance and completeness that was previously unimaginable. In this paper, we researched the topic of establishing an optimal path plan for autonomous driving using LiDAR sensors and deep reinforcement learning in a workplace without map and grid coordinates for mobile robots, which are widely used in logistics and manufacturing sites. For this purpose, we reviewed the hardware configuration of mobile robots capable of autonomous driving, checked the characteristics of themain core sensors, and investigated the core technologies of autonomous driving. In addition, we reviewed the appropriate deep reinforcement learning algorithm to realize the autonomous driving of mobile robots, defined a deep neural network for autonomous driving data conversion, and defined a reward function for path planning. The contents investigated in this paper were built into a simulation environment to verify the autonomous path planning through experiment, and an additional reward technique "Velocity Range-based Evaluation Method" was proposed for further improvement of performance indicators required in the real field, and the effectiveness was verified. The simulation environment and detailed results of experiments are described in this paper, and it is expected as guidance and reference research for applying these technologies in the field. [ABSTRACT FROM AUTHOR]

From the perspective of the deep reinforcement learning algorithm, the training effect of the agent will be affected because of the excessive randomness of the ε-greedy method. This paper proposes a novel action decision method to replace the ε-greedy method and avoid excessive randomness. First, a confidence bound span fitting model based on a deep neural network is proposed to fundamentally solve the problem that UCB cannot estimate the confidence bound span of each action in high-dimensional state space. Then, a confidence bound span balance model based on target value in reverse order is proposed. The parameters of the U network are updated after each action decision using the backpropagation of the neural network to balance the confidence bound span. Finally, an exploration-exploitation dynamic balance factor α is introduced to balance exploration and exploitation in the training process. Experiments are conducted using the Nature DQN and Double DQN algorithms, and the results demonstrate that the proposed method achieves higher performance than the ε-greedy method under the basic algorithm and experimental environment of this paper. The method presented in this paper has significance for applying a confidence bound to solve complex reinforcement problems. [ABSTRACT FROM AUTHOR]

In recent years, the Multi-agent Deep Reinforcement Learning Algorithm has been developing rapidly, in which the value method-based algorithm plays an important role (such as Monotonic Value Function Factorisation (QMIX) and Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement learning (QTRAN)). In spite of the fact, the performance of current value-based multi-agent algorithm under complex scene still can be further improved. In value function-based model, a mixing network is usually used to mix the local action value of each agent to get joint action value when the partial observability will cause the problem of misalignment and unsatisfying mixing results. This paper proposes a multi-agent model called Transform Networks that transform the individual local action-value function gotten by agent network to individual global action-value function, which will avoid the problem of misalignment caused by partial observability when the individual action value is mixed, and the joint action value can represent the cooperative conditions of all agents well. Using the StarCraft Multi-Agent Challenge (SMAC) as the experimental platform, the comparison of the performance of algorithms on five different maps proved that the proposed method has better effect than the current most advanced baseline algorithms. [ABSTRACT FROM AUTHOR]

Pulse-coupled neural networks perform well in many fields such as information retrieval, depth estimation and object detection. Based on pulse coupled neural network (PCNN) theory, this paper constructs a visual perception model framework and builds a real image reproduction platform. The model firstly analyzes the structure and generalization ability of neural network multi-class classifier, uses the minimax criterion of feature space as the splitting criterion of visual perception decision node, which solves the generalization problem of neural network learning algorithm. In the simulation process, the initial threshold is optimized by the two-dimensional maximum inter-class variance method, and in order to improve the real-time performance of the algorithm, the fast recurrence formula of neural network is derived and given. The PCNN image segmentation method based on genetic algorithm is analyzed. The genetic algorithm improves the loop termination condition and the adaptive setting of model parameters of PCNN image segmentation algorithm, but the PCNN image segmentation algorithm still has the problem of complexity. In order to solve this problem, this paper proposed an IGA-PCNN image segmentation method combining the improved algorithm and PCNN model. Firstly, it used the improved immune genetic algorithm to adaptively obtain the optimal threshold, and then replaced the dynamic threshold in PCNN model with the optimal threshold, and finally used the pulse coupling characteristics of PCNN model to complete the image segmentation. From the coupling characteristics of PCNN, junction close space of image and gray level characteristics, it determined the local gray mean square error of image connection strength coefficient. The feature extraction and object segmentation properties of PCNN come from the spike frequency of neurons, and the number of neurons in PCNN is equal to the number of pixels in the input image. In addition, the spatial and gray value differences of pixels should be considered comprehensively to determine their connection matrix. Digital experiments show that the multi-scale multi-task pulse coupled neural network model can shorten the total training time by 17 h, improve the comprehensive accuracy of the task test data set by 1.04%, and shorten the detection time of each image by 4.8 s compared with the series network model of multiple single tasks. Compared with the traditional PCNN algorithm, it has the advantages of fast visual perception and clear target contour segmentation, and effectively improves the anti-interference performance of the model. [ABSTRACT FROM AUTHOR]

Reinforcement Learning is one of the many machine learning paradigms. With no labelled data, it is concerned with balancing the exploration and exploitation of an environment with one or more agents present in it. Recently, many breakthroughs have been made in the creation of these agents for video game machine learning development, especially in first-person shooters with platforms such as ViZDoom, DeepMind Lab, and Unity's ML-Agents. In this paper, we review the state-of-the-art of creation of Reinforcement Learning agents for use in multiplayer deathmatch first-person shooters. We selected various platforms, frameworks, and training architectures from various papers and examined each of them, analysing their uses. We compared each platform and training architecture, and then concluded whether machine learning agents can now face off against humans and whether they make for better gameplay than traditional Artificial Intelligence. In the end, we thought about future research and what researchers should keep in mind when exploring and testing this area. [ABSTRACT FROM AUTHOR]

The aim of infrared and visible image fusion is to integrate the complementary information of the two modalities for high-quality fused images. However, many deep learning fusion algorithms have not considered the characteristics of infrared images in low-light scenes, leading to the problems of weak texture details, low contrast of infrared targets and poor visual perception in the existing methods. Therefore, in this paper, we propose a salient compensation-based fusion method that makes sufficient use of the characteristics of infrared and visible images to generate high-quality fused images under low-light conditions. First, we design a multi-scale edge gradient module (MEGB) in the texture mainstream to adequately extract the texture information of the dual input of infrared and visible images; on the other hand, the salient tributary is pre-trained by salient loss to obtain the saliency map based on the salient dense residual module (SRDB) to extract salient features, which is supplemented in the process of overall network training. We propose the spatial bias module (SBM) to fuse global information with local information. Finally, extensive comparison experiments with existing methods show that our method has significant advantages in describing target features and global scenes, the effectiveness of the proposed module is demonstrated by ablation experiments. In addition, we also verify the facilitation of this paper's method for high-level vision on a semantic segmentation task. [ABSTRACT FROM AUTHOR]

Reinforcement learning is one of the promising approaches for algorithmic trading in financial markets. However, in certain situations, buy or sell orders issued by an algorithmic trading program may not be fulfilled entirely. By considering the actual scenarios from the financial markets, in this paper, we propose a novel framework named Supervised Actor-Critic Reinforcement Learning with Action Feedback (SACRL-AF) for solving this problem. The action feedback mechanism of SACRL-AF notifies the actor about the dealt positions and corrects the transitions of the replay buffer. Meanwhile, the dealt positions are used as the labels for the supervised learning. Recent studies have shown that Deep Deterministic Policy Gradient (DDPG) and Twin Delayed Deep Deterministic Policy Gradient (TD3) are more stable and superior to other actor-critic algorithms. Against this background, based on the proposed SACRL-AF framework, two reinforcement learning algorithms henceforth referred to as Supervised Deep Deterministic Policy Gradient with Action Feedback (SDDPG-AF) and Supervised Twin Delayed Deep Deterministic Policy Gradient with Action Feedback (STD3-AF) are proposed in this paper. Experimental results show that SDDPG-AF and STD3-AF achieve the state-of-art performance in profitability. [ABSTRACT FROM AUTHOR]