Search

This document discusses the societal impact of applying artificial intelligence (AI) to traffic and transportation. Sustainable transport is considered crucial for sustainable cities and communities, and AI has the potential to significantly influence daily life in urban areas. The document highlights the interdisciplinary challenges of understanding and coordinating transportation systems, which involve various players with conflicting goals and constraints. It also presents a selection of innovative solutions related to traffic management, including machine learning, reinforcement learning, and optimization approaches. The document concludes by expressing gratitude to the authors, reviewers, and committee members involved in the creation of the special issue. [Extracted from the article]

Considering the load uncertainty and unmodeled dynamics in multicylinder hydraulic systems, this paper proposes a balance control algorithm based on safe reinforcement learning to release the restrictions of classical model-based control methods that depend on fixed gain. In this paper, the hydraulic press is controlled by a trained agent that directly maps the system states to control commands in an end-to-end manner. By introducing an action modifier into the algorithm, the system states are kept within security constraints from the beginning of training, making safe exploration possible. Furthermore, a normalized exponential reward function has been proposed. Compared with a quadratic reward function, the precision is greatly improved under the same training steps. The experiment shows that our algorithm can achieve high precision and fast balance for multicylinder hydraulic presses while being highly robust. To the best of our knowledge, this research is the first to attempt the application of a reinforcement learning algorithm to multi-execution units of hydraulic systems. [ABSTRACT FROM AUTHOR]

Addressing the formidable challenges posed by multiple jammers jamming multiple radars, which arise from spatial discretization, many degrees of freedom, numerous model input parameters, and the complexity of constraints, along with a multi-peaked objective function, this paper proposes a cooperative jamming resource allocation method, based on evolutionary reinforcement learning, that uses joint multi-domain information. Firstly, an adversarial scenario model is established, characterizing the interaction between multiple jammers and radars based on a multi-beam jammer model and a radar detection model. Subsequently, considering real-world scenarios, this paper analyzes the constraints and objective function involved in cooperative jamming resource allocation by multiple jammers. Finally, accounting for the impact of spatial, frequency, and energy domain information on jamming resource allocation, matrices representing spatial condition constraints, jamming beam allocation, and jamming power allocation are formulated to characterize the cooperative jamming resource allocation problem. Based on this foundation, the joint allocation of the jamming beam and jamming power is optimized under the constraints of jamming resources. Through simulation experiments, it was determined that, compared to the dung beetle optimizer (DBO) algorithm and the particle swarm optimization (PSO) algorithm, the proposed evolutionary reinforcement learning algorithm based on DBO and Q-Learning (DBO-QL) offers 3.03% and 6.25% improvements in terms of jamming benefit and 26.33% and 50.26% improvements in terms of optimization success rate, respectively. In terms of algorithm response time, the proposed hybrid DBO-QL algorithm has a response time of 0.11 s, which is 97.35% and 96.57% lower than the response times of the DBO and PSO algorithms, respectively. The results show that the method proposed in this paper has good convergence, stability, and timeliness. [ABSTRACT FROM AUTHOR]

The study proposes a method for the trajectory tracking control of a fixed-wing unmanned aerial vehicle (UAV) based on the deep deterministic policy gradient (DDPG). First, the problem of controlling the trajectory of a fixed-wing UAV is combined with the reinforcement learning framework and transformed into a Markov decision process, and a DDPG agent is established in the framework of TensorFlow. Second, we conducted simulations to train and optimize the model in a 3D environment of trajectory tracking control and obtained an integrated DDPG-based trajectory tracking controller that can regulate functions ranging from the state of flight of the UAV to rudder control. Third, we constructed a digital simulation system to verify the proposed method while considering the influence of parametric uncertainties, measurement-induced noise, and delays in the response of the control system. The effectiveness and robustness of the proposed DDPG controller were verified by comparing its performance with that of traditional proportional-integral-derivative (PID) control. Unmanned aerial vehicles have gained widespread use across numerous sectors of society. However, the control process for fixed-wing UAVs is more intricate and challenging than that for rotor type UAVs due to the coupled and nonlinear nature of their dynamic models. To address this issue, decoupling and splitting hierarchical control methods are commonly employed. However, these methods can introduce errors and affect flight performance. In contrast, artificial intelligence, particularly deep learning and reinforcement learning, has made significant advancements and found successful applications in areas such as robot control and gaming. To address the challenges posed by complex forms, excessive measurement information, and manual parameter tuning associated with traditional control approaches for fixed-wing UAVs, this paper introduces a deep reinforcement learning algorithm, i.e., the deep deterministic policy gradient, into the control process of fixed-wing UAVs. The paper presents a trajectory tracking control policy for UAVs, which efficiently achieves the task of tracking a desired trajectory without the need for selecting multiple working state nodes or designing linear control laws. Further, through simulation verification, the proposed policy demonstrates adaptability to interference caused by parameter bias, measurement noise, and control system response delays while achieving control accuracy comparable with classical control methods. [ABSTRACT FROM AUTHOR]

In this paper, we address the following question: How can multi-robot self-organizing systems be designed so that they show the desired behavior and are able to perform tasks specified by the designers? Multi-robot self-organizing systems, e.g., swarm robots, have great potential for adapting when performing complex tasks in a changing environment. However, such systems are difficult to design due to the stochasticity of system performance and the non-linearity between the local actions/interaction and the desired global behavior. In order to address this, in this paper, we propose a framework for designing self-organizing systems using Multi-Agent Reinforcement Learning (MARL) and the compromise Decision-Support Problem (cDSP) construct. The proposed framework consists of two stages, namely, preliminary design followed by design improvement. In the preliminary design stage, MARL is used to help designers train the robots so that they show stable group behavior for performing the task. In the design improvement stage, the cDSP construct is used to explore the design space and identify satisfactory solutions considering several performance indicators. Surrogate models are used to map the relationship between local parameters and global performance indicators utilizing the data generated in the preliminary design. These surrogate models represent the goals of the cDSP. Our focus in this paper is to describe the framework. A multi-robot box-pushing problem is used as an example to test the framework's efficacy. This framework is general and can be extended to design other multi-robot self-organizing systems. [ABSTRACT FROM AUTHOR]

Mean-field games (MFGs) are developed to model the decision-making processes of a large number of interacting agents in multi-agent systems. This paper studies mean-field games on graphs (G -MFGs). The equilibria of G -MFGs, namely, mean-field equilibria (MFE), are challenging to solve for their high-dimensional action space because each agent has to make decisions when they are at junction nodes or on edges. Furthermore, when the initial population state varies on graphs, we have to recompute MFE, which could be computationally challenging and memory-demanding. To improve the scalability and avoid repeatedly solving G -MFGs every time their initial state changes, this paper proposes physics-informed graph neural operators (PIGNO). The PIGNO utilizes a graph neural operator to generate population dynamics, given initial population distributions. To better train the neural operator, it leverages physics knowledge to propagate population state transitions on graphs. A learning algorithm is developed, and its performance is evaluated on autonomous driving games on road networks. Our results demonstrate that the PIGNO is scalable and generalizable when tested under unseen initial conditions. [ABSTRACT FROM AUTHOR]

Protein structure expectation is a particularly mind boggling issue that it is frequently assaulted and disintegrated using four distinct levels and they are: 1-D forecast of under- lying highlights along the essential succession of amino acids sequences, 2-D forecast of spatial connections between the sequence of amino acids, 3-D forecast of a tertiary structure of protein and quaternary structure of protein. This paper also try to introduce some assessment tools for finding the accuracy of result from applying ML and DL tools. And try to analyses and compare various algorithms based on deep learning methods verses machine learning methods used for sequence prediction. This paper also examines the turn of events and utilization of concealed Markov model, uphold vector machines, Bayesian techniques, and grouping strategies. This investigation will be helpful in creating future strategies to improve the exactness of protein auxiliary structure expectation. In this paper, also introduce and summarize the problem of quantum essential elements of: (1) Variational auto-encoder (2) GAN, generative adversarial network (3) RNN, recurrent neural (4) CNN, convolutional neural networks protein structure prediction. Later on also summarizes the evolution of predictive algorithms for 1-4D structure of protein from Amino Acid Sequences and summarize the deep learning ideas to prediction of structure of protein and learned algorithms of the last decade. [ABSTRACT FROM AUTHOR]

Traditional aerodynamic modeling methods have some difficulties in multiparameter unsteady state-space modeling and control law design, which raise significant challenges in flight control. The rapid development of machine learning provides a new idea for unsteady aerodynamic modeling and control law design, which has significant value for theoretical research and engineering application. In this paper, an unsteady aerodynamic model environment was established based on deep neural network (DNN), and a dynamic computational fluid dynamics (CFD) virtual environment was taken as an approximation of the real environment. A deep reinforcement learning (DRL) longitudinal-axis flight control method was studied on the basis of both environments. The deep deterministic policy gradient (DDPG) algorithm was used to implement flight control, and the effects of different constraints in the reward function on the results of DRL were studied at the same time. The results show that the DDPG agent effectively achieves longitudinal-axis flight control in the model environment. The DDPG agent trained in the model environment was also used for longitudinal-axis flight control in the dynamic CFD virtual environment, and this method provides a reference for the coupling of dynamic CFD and DRL. The control results from the model environment and the dynamic CFD virtual environment are compared, and results show that the DDPG agent has good robustness in both environments. The results of this study suggest that machine learning and deep reinforcement learning can effectively solve complicated multiconstraint, uncertain, and sophisticated control tasks. In this paper, a DNN-based unsteady aerodynamic modeling method was established based on dynamic computational fluid dynamics calculation and machine learning, and the results show that the DNN-based unsteady aerodynamic model has advantages in dealing with highly nonlinear data under unsteady conditions. In addition, the deep deterministic policy gradient algorithm can be well applied in longitudinal-axis flight control, and the agent can be trained in DNN-based unsteady aerodynamic model environment. The agent was also used for longitudinal-axis flight control in the dynamic computational fluid dynamics virtual environment, and the results show that the agent has good robustness in both environments. This provides the possibility for the agent to achieve longitudinal-axis flight control in a real environment. [ABSTRACT FROM AUTHOR]

The objective of portfolio diversification is to reduce risk and potentially enhance returns by spreading investments across different asset classes. Existing portfolio diversification models have traditionally been trained on historical financial time series data. However, several issues arise with historical financial time series data, making it challenging to train models effectively to achieve the portfolio diversification objective: an insufficient amount of training data and the uncertainty deficiency problem, wherein the uncertainty that existed in the past is not visible in the present. Insufficient datasets, characterized by small data size, result in information asymmetry and compromise portfolio performance. This limitation underscores the importance of adopting a pattern-centric data augmentation approach, capable of unveiling hidden patterns and structures within the financial time series data. To address these challenges, this paper introduces the financial time series decomposition-based variational encoder-decoder (FED) method to augment financial time series data, overcoming the limitations of insufficient training data and providing a more realistic and dynamic simulation of the financial market environment. By decomposing the data into distinct components, such as trend, dispersion, and residual, FED leverages pattern-centric data augmentation within the financial time series data. In the environment generated using the FED method, this paper proposes a two-class portfolio diversification, called FED2Port. It integrates stochastic elements into the reward function, enabling a reinforcement learning algorithm to learn from a comprehensive spectrum of financial market uncertainties. The experimental results demonstrate that the proposed model significantly enhances portfolio performance. [ABSTRACT FROM AUTHOR]

OBJECTIVE: Reinforcement learning (RL) is a machine learning technique uniquely effective at sequential decision-making, which makes it potentially relevant to ICU treatment challenges. We set out to systematically review, assess level-of-readiness and meta-analyze the effect of RL on outcomes for critically ill patients. DATA SOURCES: A systematic search was performed in PubMed, Embase.com, Clarivate Analytics/Web of Science Core Collection, Elsevier/SCOPUS and the Institute of Electrical and Electronics Engineers Xplore Digital Library from inception to March 25, 2022, with subsequent citation tracking. DATA EXTRACTION: Journal articles that used an RL technique in an ICU population and reported on patient health-related outcomes were included for full analysis. Conference papers were included for level-of-readiness assessment only. Descriptive statistics, characteristics of the models, outcome compared with clinician's policy and level-of-readiness were collected. RL-health risk of bias and applicability assessment was performed. DATA SYNTHESIS: A total of 1,033 articles were screened, of which 18 journal articles and 18 conference papers, were included. Thirty of those were prototyping or modeling articles and six were validation articles. All articles reported RL algorithms to outperform clinical decision-making by ICU professionals, but only in retrospective data. The modeling techniques for the state-space, action-space, reward function, RL model training, and evaluation varied widely. The risk of bias was high in all articles, mainly due to the evaluation procedure. CONCLUSION: In this first systematic review on the application of RL in intensive care medicine we found no studies that demonstrated improved patient outcomes from RL-based technologies. All studies reported that RL-agent policies outperformed clinician policies, but such assessments were all based on retrospective off-policy evaluation. [ABSTRACT FROM AUTHOR]

This paper addresses the issue of how to endow robots with motion skills, flexibility, and adaptability similar to human arms. It innovatively proposes a hybrid-primitive-frame-based robot skill learning algorithm and utilizes the policy improvement with a path integral algorithm to optimize the parameters of the hybrid primitive framework, enabling robots to possess skills similar to human arms. Firstly, the end of the robot is dynamically modeled using an admittance control model to give the robot flexibility. Secondly, the dynamic movement primitives are employed to model the robot's motion trajectory. Additionally, novel stiffness primitives and damping primitives are introduced to model the stiffness and damping parameters in the impedance model. The combination of the dynamic movement primitives, stiffness primitives, and damping primitives is called the hybrid primitive framework. Simulated experiments are designed to validate the effectiveness of the hybrid-primitive-frame-based robot skill learning algorithm, including point-to-point motion under external force disturbance and trajectory tracking under variable stiffness conditions. [ABSTRACT FROM AUTHOR]

This paper proposes a method for finding the shortest path of a mobile robot using deep reinforcement learning with utilizing Proximal policy optimization algorithm (PPO) enhanced with curriculum learning. By modelling the environment in 3D space using the Webots simulator, we extend the PPO algorithm's capabilities to handle continuous states from 8 IR sensors and control the velocities of two motors of E-puck robot. Our study uniquely integrates curriculum learning into the PPO framework, aiming to improve adaptability and training efficiency in complex environments. A comparative analysis is conducted between the modified PPO, the original PPO, and the deep deterministic policy gradient algorithm, highlighting the strengths of our approach The results demonstrate that our curriculum-augmented PPO algorithm not only accelerates the training process but also shows superior adaptability and generalization in new environments. This work underscores the significant potential of curriculum learning in enhancing the performance of deep reinforcement learning algorithms for robust and efficient robotic navigation. [ABSTRACT FROM AUTHOR]

The increasing demand for mobility in our society poses various challenges to traffic engineering, computer science in general, and artificial intelligence in particular. Increasing the capacity of road networks is not always possible, thus a more efficient use of the available transportation infrastructure is required. Another issue is that many problems in traffic management and control are inherently decentralized and/or require adaptation to the traffic situation. Hence, there is a close relationship to multiagent reinforcement learning. However, using reinforcement learning poses the challenge that the state space is normally large and continuous, thus it is necessary to find appropriate schemes to deal with discretization of the state space. To address these issues, a multiagent system with agents learning independently via a learning algorithm was proposed, which is based on estimating Q-values from k-nearest neighbors. In the present paper, we extend this approach and include transfer of experiences among the agents, especially when an agent does not have a good set of k experiences. We deal with traffic signal control, running experiments on a traffic network in which we vary the traffic situation along time, and compare our approach to two baselines (one involving reinforcement learning and one based on fixed times). Our results show that the extended method pays off when an agent returns to an already experienced traffic situation. [ABSTRACT FROM AUTHOR]

The advent of Industry 4.0 has significantly promoted the field of intelligent manufacturing, which is facilitated by the development of new technologies are emerging. Robot technology and robot intelligence methods have rapidly developed and been widely applied. Manipulators are widely used in industry, and their control is a crucial research topic. The inverse kinematics solution of manipulators is an important part of manipulator control, which calculates the joint angles required for the end effector to reach a desired position and posture. Traditional inverse kinematics solution algorithms often face the problem of insufficient generalization, and iterative methods have challenges such as large computation and long solution time. This paper proposes a reinforcement learning-based inverse kinematics solution algorithm, called the MAPPO-IK algorithm. The algorithm trains the manipulator agent using the MAPPO algorithm and calculates the difference between the end effector state of the manipulator and the target posture in real-time by designing a reward mechanism, while considering Gaussian distance and cosine distance. Through experimental comparative analysis, the feasibility, computational efficiency, and superiority of this reinforcement learning algorithm are verified. Compared with traditional inverse kinematics solution algorithms, this method has good generalization and supports real-time computation, and the obtained result is a unique solution. Reinforcement learning algorithms have better adaptability to complex environments and can handle different sudden situations in different environments. This algorithm also has the advantages of path planning, intelligent obstacle avoidance, and other advantages in dynamically processing complex environmental scenes. [ABSTRACT FROM AUTHOR]

Ground penetrating radar (GPR) systems are effective sensors for discovering various types of objects buried underground, such as military mines, metal objects, and pieces of underground infrastructures. A GPR system can be manually operated by a human or can be an integral part of a host platform. The host platform may be semi- or fully autonomous and may operate in different environments such as land vehicles or more recently air-borne drones. One challenge for the fully or semi-autonomous host platforms in particular is to find an efficient search procedure that would reduce the operation time and optimize resource utilization. Most of the current approaches are based on pre-defined search patterns which, for large and sparse areas, could mean unnecessary waste of time and resources. In this paper, we introduce a method that combines a coarse and therefore relatively low cost initial search pattern with a Reinforcement Learning (RL) driven efficient navigation path for eventual target detection, by exploiting the signal processing pipeline of the onboard GPR. We illustrate the applicability of the method using a well-known, high fidelity GPR simulation environment and a novel RL framework. Our results suggest that combination of a coarse navigation scheme and an RL-based training procedure based on GPR scan returns can lead to a more efficient target discovery procedure for host platforms. [ABSTRACT FROM AUTHOR]

Meta-learning is a pivotal and potentially influential machine learning approach to solve challenging problems in reinforcement learning. However, the costly hyper-parameter tuning for training stability of meta-learning is a known shortcoming and currently a hotspot of research. This paper addresses this shortcoming by introducing an online and easily trainable hyper-parameter optimization approach, called Meta Parameters Learning via Meta-Learning (MPML), to combine online hyper-parameter adjustment scheme into meta-learning algorithm, which reduces the need to tune hyper-parameters. Specifically, a basic learning rate for each training task is put forward. Besides, the proposed algorithm dynamically adapts multiple basic learning rate and a shared meta-learning rate through conducting gradient descent alongside the initial optimization steps. In addition, the sensitivity with respect to hyper-parameter choices in the proposed approach are also discussed compared with model-agnostic meta-learning method. The experimental results on reinforcement learning problems demonstrate MPML algorithm is easy to implement and delivers more highly competitive performance than existing meta-learning methods on a diverse set of challenging control tasks. [ABSTRACT FROM AUTHOR]

Variational quantum circuits are one of the promising ways to exploit the advantages of quantum computing in the noisy intermediate-scale quantum technology era. The design of the quantum circuit architecture might greatly affect the performance capability of the quantum algorithms. The quantum architecture search is the process of automatically designing quantum circuit architecture, aiming at finding the optimal quantum circuit composition architecture by the algorithm for a given task, so that the algorithm can learn to design the circuit architecture. Compared to manual design, quantum architecture search algorithms are more effective in finding quantum circuits with better performance capabilities. In this paper, based on the deep reinforcement learning, we propose an approach for quantum circuit architecture search. The sampling of the circuit architecture is learnt through reinforcement learning-based controller. Layer-based search is also used to accelerate the computational efficiency of the search algorithm. Applying to data classification tasks, we show that the method can search for quantum circuit architectures with better accuracies. Moreover, the circuit has a smaller number of quantum gates and parameters. [ABSTRACT FROM AUTHOR]

An industry-wide paradigm change has been sparked by the growth of Internet of Things (IoT)-based Wireless Sensor Networks (WSNs), which has made reliable and effective routing methods necessary. This thorough analysis looks at how Machine Learning (ML) techniques may be used to solve the problems that come with WSN routing. A summary of standard routing algorithms and an examination of their shortcomings comprise the first portion of the paper. The integration of ML approaches, such as reinforcement learning and supervised and unsupervised learning, is then explored in order to improve WSN routing efficiency. The article examines the difficulties and factors related to ML-based routing, including data quality, energy efficiency, scalability, and security. Applications and case studies show how ML is really used in WSN routing, offering insights into effective tactics and lessons discovered. Evaluation metrics and performance assessments are included in a separate section that uses simulation and experimental data to compare ML-based and conventional techniques. Looking forward, the study describes new breakthroughs in ML for WSNs and points out unresolved issues, providing a guide for future research paths. The important results and their consequences are outlined in the conclusion, which also highlights how ML has the potential to revolutionize WSN routing in the future. [ABSTRACT FROM AUTHOR]

Our increasingly connected world continues to face an ever-growing number of network-based attacks. An Intrusion Detection System (IDS) is an essential security technology used for detecting these attacks. Although numerous Machine Learning-based IDSs have been proposed for the detection of malicious network traffic, the majority have difficulty properly detecting and classifying the more uncommon attack types. In this paper, we implement a novel hybrid technique using synthetic data produced by a Generative Adversarial Network (GAN) to use as input for training a Deep Reinforcement Learning (DRL) model. Our GAN model is trained on the NSL-KDD dataset, a publicly available collection of labeled network traffic data specifically designed to support the evaluation and benchmarking of IDSs. Ultimately, our findings demonstrate that training the DRL model on synthetic datasets generated by specific GAN models can result in better performance in correctly classifying minority classes over training on the true imbalanced dataset. [ABSTRACT FROM AUTHOR]

The transportation sector remains a major contributor to greenhouse gas emissions. The understanding of energy-efficient driving behaviors and utilization of energy-efficient driving strategies are essential to reduce vehicles' fuel consumption. However, there is no comprehensive investigation into energy-efficient driving behaviors and strategies. Furthermore, many state-of-the-art AI models have been applied for the analysis of eco-friendly driving styles, but no overview is available. To fill the gap, this paper conducts a thorough literature review on ecological driving behaviors and styles, and analyzes the driving factors influencing energy consumption and state-of-the-art methodologies. With a thorough scoping review process, thirty-seven articles with full text were assessed, and the methodological and related data are compared. The results show that the factors that impact driving behaviors can be summarized into eleven features including speed, acceleration, deceleration, pedal, steering, gear, engine, distance, weather, traffic signal, and road parameters. This paper finds that supervised/unsupervised learning algorithms and reinforcement learning frameworks have been popularly used to model the vehicle's energy consumption with multi-dimensional data. Furthermore, the literature shows that the driving data are collected from either simulators or real-world experiments, and the real-world data are mainly stored and transmitted by meters, controller area networks, onboard data services, smartphones, and additional sensors installed in the vehicle. Based on driving behavior factors, driver characteristics, and safety rules, this paper recommends nine energy-efficient driving styles including four guidelines for the drivers' selection and adjustment of the vehicle parameters, three recommendations for the energy-efficient driving styles in different driving scenarios, and two subjective suggestions for different types of drivers and employers. [ABSTRACT FROM AUTHOR]

This document discusses the rapid development of Artificial Intelligence (AI) and its bioinspired applications. It highlights the benefits of bioinspired AI, such as increased accuracy in image and speech processing, reduced cost and energy usage through edge devices, and enhanced bio-signal quality. However, it also acknowledges the challenges posed by improper AI utilization, such as the generation of fake news and security issues. The document calls for research papers on bioinspired AI applications to explore its potential and address these challenges. It includes examples of research papers that utilize deep reinforcement learning for robot task sequencing, propose a real-time multi-surveillance pedestrian target detection model, develop an intelligent breast mass classification approach, and introduce a bio-inspired object detection algorithm for remote sensing images. The document concludes by emphasizing the importance of biomimetic artificial intelligence in various fields and promoting further research in this area. [Extracted from the article]

Reinforcement learning is a compelling area of research within machine learning because it enables the improvement of control strategies for future manipulation of dynamic systems, leveraging previous data even without a precise model of the system. It usually makes complex, model-free predictions from data alone, which is actually consistent with the purpose of control in that they both aim to design systems using richly structured perceptions to execute planning and control strategies that adequately adapt to changing environments. The robust trajectory tracking control of intricate mechanical systems presents a challenging problem that necessitates effective control methods. In this paper, we propose a novel nonlinear control strategy based on deep reinforcement learning to solve the trajectory tracking problem of a 3-degree-of-freedom (3-DOF) control moment gyroscope (CMG). First, dynamic modeling of the 3-DOF CMG is used as a policy solver for the reinforcement learning training environment, and transfer learning is employed to bridge the reality gap. Then, the hyperparameters and reward functions of the neural network are optimized using the asynchronous successive halving algorithm. Ultimately, the twin delay depth determination policy gradient algorithm is trained in simulation to yield an agent capable of tracking user-defined trajectory routes as a nonlinear controller for the system. Both simulation and experimental results show that the proposed method works well for both high-frequency and low-frequency varying trajectory tracking control, and that the proposed method has better response speed and robustness than classic linear parameter-varying control methods and the state-of-the-art nonlinear parameter-varying method and the neural network-based feedback linearization adaptive control method. [ABSTRACT FROM AUTHOR]

Stagnation at local optima represents a significant challenge in bio-inspired optimization algorithms, often leading to suboptimal solutions. This paper addresses this issue by proposing a hybrid model that combines the Orca predator algorithm with deep Q-learning. The Orca predator algorithm is an optimization technique that mimics the hunting behavior of orcas. It solves complex optimization problems by exploring and exploiting search spaces efficiently. Deep Q-learning is a reinforcement learning technique that combines Q-learning with deep neural networks. This integration aims to turn the stagnation problem into an opportunity for more focused and effective exploitation, enhancing the optimization technique's performance and accuracy. The proposed hybrid model leverages the biomimetic strengths of the Orca predator algorithm to identify promising regions nearby in the search space, complemented by the fine-tuning capabilities of deep Q-learning to navigate these areas precisely. The practical application of this approach is evaluated using the high-dimensional Heartbeat Categorization Dataset, focusing on the feature selection problem. This dataset, comprising complex electrocardiogram signals, provided a robust platform for testing the feature selection capabilities of our hybrid model. Our experimental results are encouraging, showcasing the hybrid strategy's capability to identify relevant features without significantly compromising the performance metrics of machine learning models. This analysis was performed by comparing the improved method of the Orca predator algorithm against its native version and a set of state-of-the-art algorithms. [ABSTRACT FROM AUTHOR]

Federated learning is a distributed machine learning approach that enables a large number of edge/end devices to perform on-device training for a single machine learning model, without having to share their own raw data. We consider in this paper a federated learning scenario wherein the local training is carried out on IoT devices and the global aggregation is done at the level of an edge server. One essential challenge in this emerging approach is IoT devices selection (also called scheduling), i.e., how to select the IoT devices to participate in the distributed training process. The existing approaches suggest to base the scheduling decision on the resource characteristics of the devices to guarantee that the selected devices would have enough resources to carry out the training. In this work, we argue that trust should be an integral part of the decision-making process and therefore design a trust establishment mechanism between the edge server and IoT devices. The trust mechanism aims to detect those IoT devices that over-utilize or under-utilize their resources during the local training. Thereafter, we introduce DDQN-Trust, a double deep Q learning-based selection algorithm that takes into account the trust scores and energy levels of the IoT devices to make appropriate scheduling decisions. Finally, we integrate our solution into four federated learning aggregation approaches, namely, FedAvg, FedProx, FedShare and FedSGD. Experiments conducted using a real-world dataset show that our DDQN-Trust solution always achieves better performance compared to two main benchmarks: the DQN and random scheduling algorithms. The results also reveal that FedProx outperforms the competitor aggregation models in terms of accuracy when integrated into our DDQN-Trust solution. [ABSTRACT FROM AUTHOR]

At present, autonomous driving decision-making solutions take few elements into account while ignoring the unpredictable nature of driving behavior, which makes it challenging to manage complicated traffic situations. To this end, we present a decision-making architecture in this paper that enhances the existing reinforcement learning methodology by combining the bootstrapped technique and the random prior network (RPN). The RPN can give each learner a neural network with unique weights to avoid the contingency created by the artificially built prior functions, while the Bootstrapped technique can balance out the exploration and exploitation. The ego vehicle was trained by three algorithms and verified in random environments to evaluate the effectiveness of our method. The results show that our algorithm outperformed the current reinforcement learning algorithms. [ABSTRACT FROM AUTHOR]

In this paper, we propose a novel Reinforcement Learning (RL) algorithm for robotic motion control, that is, a constrained Deep Deterministic Policy Gradient (DDPG) deviation learning strategy to assist biped robots in walking safely and accurately. The previous research on this topic highlighted the limitations in the controller's ability to accurately track foot placement on discrete terrains and the lack of consideration for safety concerns. In this study, we address these challenges by focusing on ensuring the overall system's safety. To begin with, we tackle the inverse kinematics problem by introducing constraints to the damping least squares method. This enhancement not only addresses singularity issues but also guarantees safe ranges for joint angles, thus ensuring the stability and reliability of the system. Based on this, we propose the adoption of the constrained DDPG method to correct controller deviations. In constrained DDPG, we incorporate a constraint layer into the Actor network, incorporating joint deviations as state inputs. By conducting offline training within the range of safe angles, it serves as a deviation corrector. Lastly, we validate the effectiveness of our proposed approach by conducting dynamic simulations using the CRANE biped robot. Through comprehensive assessments, including singularity analysis, constraint effectiveness evaluation, and walking experiments on discrete terrains, we demonstrate the superiority and practicality of our approach in enhancing walking performance while ensuring safety. Overall, our research contributes to the advancement of biped robot locomotion by addressing gait optimisation from multiple perspectives, including singularity handling, safety constraints, and deviation learning. [ABSTRACT FROM AUTHOR]

This research paper presents the Buckley-James Q-learning (BJ-Q) algorithm, a cutting-edge method designed to optimize personalized treatment strategies, especially in the presence of right censoring. We critically assess the algorithm's effectiveness in improving patient outcomes and its resilience across various scenarios. Central to our approach is the innovative use of the survival time to impute the reward in Q-learning, employing the Buckley-James method for enhanced accuracy and reliability. Our findings highlight the significant potential of personalized treatment regimens and introduce the BJ-Q learning algorithm as a viable and promising approach. This work marks a substantial advancement in our comprehension of treatment dynamics and offers valuable insights for augmenting patient care in the ever-evolving clinical landscape. [ABSTRACT FROM AUTHOR]

Malware attack is a growing problem on the Android mobile platform due to its popularity and openness. Although numerous malware detection approaches have been proposed, it still remains challenging for malware detection due to a large amount of constantly mutating apps. The opcode, as the most fundamental part of Android app, possesses good resistance against obfuscation and Android version updates. Due to the large number of opcodes, most opcode-based methods employ statistical-based feature selection, which disrupts the correlation and semantic information among opcodes. In this paper, we propose an Android malware detection framework based on sensitive opcodes and deep reinforcement learning. Firstly, we extract sensitive opcode fragments based on sensitive elements and then encode the features using n-gram. Next, we use deep reinforcement learning to select the optimal subset of features. During the process of handling opcodes, we focus on preserving semantic information and the correlation among opcodes. Finally, our experimental results show an accuracy of 0.9670 by using the 25 opcode features we obtained. [ABSTRACT FROM AUTHOR]

In contemporary warfare, radar countermeasures have become multifunctional and intelligent, rendering the conventional jamming method and platform unsuitable for the modern radar countermeasures battlefield due to their limited efficiency. Reinforcement learning has been proven to be a practical solution for cognitive jamming decision-making in the cognitive electronic warfare. In this paper, we proposed a radar-jamming decision-making algorithm based on an improved Q-Learning algorithm. This improved Q-Learning algorithm ameliorated the problem of overestimating the Q-value that exists in the Q-Learning algorithm by introducing a second Q-table. At the same time, we performed a comprehensive design and implementation based on the classical Q-Learning algorithm, deploying it to a Field Programmable Gate Array (FPGA) hardware. We decomposed the implementation of the reinforcement learning algorithm into individual steps and described each step using a hardware description language. Then, the reinforcement learning algorithm can be computed on FPGA by linking the logic modules with valid signals. Experiments show that the proposed Q-Learning algorithm obtains considerable improvement in performance over the classical Q-Learning algorithm. Additionally, they confirm that the FPGA hardware can achieve great efficiency improvement on the radar-jamming decision-making algorithm implementation. [ABSTRACT FROM AUTHOR]

The path planning strategies of multi-objective tasks for autonomous agents are applied more and more widely in the areas of resource exploration, factory transportation, post-disaster relief, etc. These tasks typically require access to all target points within a limited time and can be formulated by signal temporal logic (STL) formulas with multiple logic AND. This paper presents a dual-layer network deep reinforcement learning algorithm to address the decision problem of the multi-objective STL tasks. First, the multi-objective optimization problem is represented by iterable UNTIL operators, and each sub-task is encoded into an one-hot vector. In the control layer, one-hot vector is updated according to the ongoing sub-task. Based on the coupling input of agent states and one-hot vector, the neural network controller is trained to complete the sub-tasks in the given order strictly. In the order layer, robust-based weighted entropy is assigned according to the agent trajectory robust value of each sub-task. By maximizing the entropy, the order priority of access to the target points is updated periodicity and is passed into the control layer. Simulation and comparison with the traditional method show that our proposed dual-layer network algorithm has the reliability for strictly completing sub-tasks of multi-objective task reformed by UNTIL operators and the effectiveness of the access order planning for multi-objective tasks with improved real-time performance. [ABSTRACT FROM AUTHOR]

Recent advances in the field of machine learning have yielded novel research perspectives in behavioural economics and financial markets microstructure studies. In this paper we study the impact of individual trader leaning characteristics on markets using a stock market simulator designed with a multi-agent architecture. Each agent, representing an autonomous investor, trades stocks through reinforcement learning, using a centralized double-auction limit order book. This approach allows us to study the impact of individual trader traits on the whole stock market at the mesoscale in a bottom-up approach. We chose to test three trader trait aspects: agent learning rate increases, herding behaviour and random trading. As hypothesized, we find that larger learning rates significantly increase the number of crashes. We also find that herding behaviour undermines market stability, while random trading tends to preserve it. [ABSTRACT FROM AUTHOR]

With the increasing number of ransomware attacks on critical infrastructures, there is an urgent need to develop effective systems that can detect ransomware early. In order to achieve this objective, many detection solutions rely on machine learning to analyze the features of ransomware samples. However, these solutions often need to execute ransomware to extract sufficient features, which can increase the risk of virus infection. This paper proposes a novel static analysis framework based on the portable executable header, which utilizes deep reinforcement learning for early detection of ransomware. The framework leverages the portable executable header (PE header) from executable files as the key feature to identify ransomware. Moreover, the framework can learn the features of ransomware by using deep reinforcement learning. This is achieved through the interaction between the agent and the environment, and then the samples are segmented into ransomware and benign categories by taking actions. The proposed framework achieves rapid detection speeds without running ransomware samples by employing a lightweight network and the portable executable header from the raw byte files. To the best of our knowledge, we are the first to exploit deep reinforcement learning on the PE header for ransomware early detection. Our experiments on two real-world datasets reveal that the proposed framework can rapidly detect unseen ransomware. [ABSTRACT FROM AUTHOR]

In order to cope with ever-evolving and increasing cyber threats, intrusion detection systems have become a crucial component of cyber security. Compared with signature-based intrusion detection methods, anomaly-based methods typically employ machine learning techniques to train detection models and possess the capability to discover unknown attacks. However, intrusion detection methods face the challenge of low detection rates for minority class attacks due to imbalanced data distributions. Traditional intrusion detection algorithms address this issue by resampling or generating synthetic data. Additionally, reinforcement learning, as a machine learning method that interacts with the environment to obtain feedback and improve performance, is gradually being considered for application in the field of intrusion detection. This paper proposes a reinforcement-learning-based intrusion detection method that innovatively uses adaptive sample distribution dual-experience replay to enhance a reinforcement learning algorithm, aiming to effectively address the issue of imbalanced sample distribution. We have also developed a reinforcement learning environment specifically designed for intrusion detection tasks. Experimental results demonstrate that the proposed model achieves favorable performance on the NSL-KDD, AWID, and CICIoT2023 datasets, effectively dealing with imbalanced data and showing better classification performance in detecting minority attacks. [ABSTRACT FROM AUTHOR]

Proton exchange membrane fuel cells (PEMFCs) constitute nonlinear systems that are challenging to model accurately. Therefore, a controller with robustness and adaptability is imperative for temperature control within the PEMFC stack. This paper introduces a data-driven controller utilizing deep reinforcement learning for stack temperature control. Given the PEMFC system's characteristics, such as nonlinearity, uncertainty, and environmental conditions, we propose a novel deep reinforcement learning algorithm—the deep deterministic policy gradient with priority experience playback and importance sampling method (PEI-DDPG). Algorithm design incorporates technologies such as priority experience playback, importance sampling, and optimized sample data storage structure, enhancing the controller's performance. Simulation results demonstrate the proposed algorithm's superior effectiveness in temperature control for PEMFC, leveraging the PEI-DDPG algorithm's high adaptability and robustness. The proposed algorithm's effectiveness is additionally validated on the RT-LAB experimental platform. The proposed PEI-DDPG algorithm reduces the average adjustment time by 8.3%, 17.13%, and 24.56% and overshoots by 2.12 times, 4.16 times, and 4.32 times compared to the TD3, GA-PID, and PID algorithms, respectively. [ABSTRACT FROM AUTHOR]

Summary: In this paper, we study the task offloading optimization problem in satellite edge computing environments to reduce the whole communication latency and energy consumption so as to enhance the offloading success rate. A three‐tier machine learning framework consisting of collaborative edge devices, edge data centers, and cloud data centers has been proposed to ensure an efficient task execution. To accomplish this goal, we also propose a Q‐learning‐based reinforcement learning offloading strategy in which both the time‐sensitive constraints and data requirements of the computation‐intensive tasks are taken into account. It enables various types of tasks to select the most suitable satellite nodes for the computing deployment. Simulation results show that our algorithm outperforms other baseline algorithms in terms of latency, energy consumption, and successful execution efficiency. [ABSTRACT FROM AUTHOR]

Summary: The cloud computing systems, such as the Internet of Things (IoT), are usually introduced with a three‐layer architecture (IoT‐Fog‐Cloud) for the task offloading that is a solution to compensate for resource constraints in these systems. Offloading at the right location is the most significant challenge in this field. It is more appropriate to offload tasks to fog than to cloud based on power and performance metrics, but its resources are more limited than the resources of the cloud. This paper tries to optimize these factors in the fog by specifying the number of usable servers in the fog. For this purpose, we model a fog computing system using the queueing theory. Furthermore, binary search and reinforcement learning algorithms are proposed to determine the minimum number of servers with the lowest power consumption. We evaluate the cost of the fog in different scenarios. By solving the model, we find that the proposed dispatching policy is very flexible and outperformed the known policies by up to 31% and in no case is it worse than either of them, and the overall offloading cost increases when fog rejects tasks with a high probability. Our offloading method is more effective than running all fog servers simultaneously, based on simulation results. It is evident from the similarities between the simulation results and those derived from the analytical method that the model and results are valid. [ABSTRACT FROM AUTHOR]

This paper addresses distributional offline continuous-time reinforcement learning (DOCTR-L) with stochastic policies for high-dimensional optimal control. A soft distributional version of the classical Hamilton–Jacobi–Bellman (HJB) equation is given by a semilinear partial differential equation (PDE). This 'soft HJB equation' can be learned from offline data without assuming that the latter correspond to a previous optimal or near-optimal policy. A data-driven solution of the soft HJB equation uses methods of Neural PDEs and Physics-Informed Neural Networks developed in the field of Scientific Machine Learning (SciML). The suggested approach, dubbed 'SciPhy RL', thus reduces DOCTR-L to solving neural PDEs from data. Our algorithm called Deep DOCTR-L converts offline high-dimensional data into an optimal policy in one step by reducing it to supervised learning, instead of relying on value iteration or policy iteration methods. The method enables a computable approach to the quality control of obtained policies in terms of both expected returns and uncertainties about their values. [ABSTRACT FROM AUTHOR]

In many real-life scenarios, multiple agents necessitate cooperation to accomplish tasks. Benefiting from the significant success of deep learning, many single-agent deep reinforcement learning algorithms have been extended to multi-agent scenarios. Overestimation in value estimation of Q-learning is a significant issue that has been studied comprehensively in the single-agent domains, but rarely in multi-agent reinforcement learning. In this paper, we first demonstrate that Q-learning-based multi-agent reinforcement learning (MARL) methods generally have notably serious overestimation issues, which cannot be alleviated by current methods. To tackle this problem, we introduce the double critic networks structure and the delayed policy update to Q-learning-based multi-agent MARL methods, which reduce the overestimation and enhance the quality of policy updating. To demonstrate the versatility of our proposed method, we select several Q-learning based MARL methods and evaluate them on several multi-agent tasks on the multi-agent particle environment and SMAC. Experimental results demonstrate that the proposed method can avoid the overestimation problem and significantly improve performance. Besides, application in the Traffic Signal Control verifies the feasibility of applying the proposed method in real-world scenarios. [ABSTRACT FROM AUTHOR]

Optimal power flow (OPF) is a crucial tool in the operation and planning of modern power systems. However, as power system optimization shifts towards larger-scale frameworks, and with the growing integration of distributed generations, the computational time and memory requirements of solving the alternating current (AC) OPF problems can increase exponentially with system size, posing computational challenges. In recent years, machine learning (ML) has demonstrated notable advantages in efficient computation and has been extensively applied to tackle OPF challenges. This paper presents five commonly employed OPF transformation techniques that leverage ML, offering a critical overview of the latest applications of advanced ML in solving OPF problems. The future directions in the application of machine learning to AC OPF are also discussed. [ABSTRACT FROM AUTHOR]

Fuel cell hybrid electric trucks have become a cutting-edge field in understanding urban traffic emissions due to their enormous potential in low-carbon areas. In order to improve the economy of fuel cell hybrid electric trucks and reduce the decline of fuel cell lifespan, this paper proposes a multi-objective energy management strategy that optimizes weight coefficients. On the basis of establishing a fuel cell battery hybrid system model, three modes of uniform speed, acceleration, and deceleration were identified through clustering analysis of vehicle speed. Reinforcement learning algorithms were used to learn the corresponding weights for different modes, which reduced the decline in fuel cell life while improving the economic efficiency. The simulation results indicate that, under the conditions of no load, half load, and full load, the truck only sacrificed 0.9–5.6%, 1.7–2.6%, and 1.2–1.6% SOC, saving 5.7–6.45%, 5.9–6.67%, and 6.1–6.67% in lifespan loss, and reducing hydrogen consumption by 3.0–7.1%, 2.8–4.4%, and 1.0–3.0%, respectively. [ABSTRACT FROM AUTHOR]

Mobile robot path planning has received more and more attention as an important technology in robotics. Based on the D* Lite algorithm, this paper constructs a robot path planning model in a grid map environment, and proposes a deep learning fusion algorithm for path planning under complex large maps. The D* Lite algorithm with excellent performance adopts the enhanced neural network algorithm with environment self-learning ability for local parts. The model introduces the gentle update method of the Q value in the D* Lite algorithm into the optimization target calculation, calculates the loss function and updates the network parameters, thereby solving the overestimation problem of deep reinforcement learning in the application of mobile robot path planning. In the simulation process, the idea of averaging is introduced into the e-greedy strategy, and the value function output by the previous generation parameter network is used to obtain the average result to determine the next action direction of the mobile robot. The experimental results show that in the simple environment and the complex environment, the planned path lengths in the environment are 44.21, 43.63 and 43.61 m, respectively, reducing the number of collisions with obstacles during the training process of the mobile robot, and improving the superiority and effectiveness of the algorithm. [ABSTRACT FROM AUTHOR]

This paper presents a least-mean-square-based (LMS-based) background calibration algorithm with reinforcement learning optimization to calibrate the capacitor mismatch in successive approximation-register (SAR) analog-to-digital converters (ADCs). When calibrating capacitor mismatch, the convergence speed and stability of the conventional LMS-based algorithm cannot be balanced due to the fixed iteration step size. To improve the calibration effect, nine different iteration step sizes are proposed for calibration. An iteration step size selection strategy is obtained by training the Q-learning algorithm with the conventional LMS-based calibration iteration process. Leading by this strategy, the proposed calibration algorithm performs better convergence speed and stability than the conventional LMS-based calibration algorithm. A 12-bit sub-radix-2 redundant SAR ADC is used to test our proposed calibration algorithm for better evaluation. Behavioral simulations show that the proposed calibration algorithm converges at 0.065 M sampling points, and the standard deviation of the effective number of bits (ENOB) in the convergence state is only 0.034-bit. [ABSTRACT FROM AUTHOR]

This paper promotes reinforcement machine learning for route-finding tasks in quantum communication networks, where, due to the non-additivity of quantum errors, classical graph path or tree-finding algorithms cannot be used. We propose using a proximal policy optimization algorithm capable of finding routes in teleportation-based quantum networks. This algorithm is benchmarked against the Monte Carlo search. The topology of our network resembles the proposed 6 G topology and analyzed that quantum errors correspond to typical errors in realistic quantum channels. [ABSTRACT FROM AUTHOR]

This paper presents a range of novel reinforcement learning algorithms derived from the actor–critic approach. These modified algorithms effectively utilize the available information to enhance performance. Our proposed framework introduces several key components to the traditional actor–critic model, including an underlying model learner, selector, tuner, and estimator. The estimator employs an approximate value function and the learned underlying model to estimate the values of all actions at the next state. The selector approximates the optimal action at the next state, which is then utilized by the actor to optimize its policy. In contrast to the conventional actor–critic algorithm where the actor focuses solely on policy optimization and the critic performs value-function approximation and policy evaluation, our selector–actor–critic algorithm employs a selector to approximate the optimal action at the current state, thereby influencing the actor's policy updates. Furthermore, our tuner–actor–critic algorithm incorporates a critic and a model-learner to approximate the action-value function and the dynamics of the underlying environment, respectively. The tuner then utilizes this information to adjust the value of the current state–action pair. In the estimator–selector–actor–critic algorithm, we develop intelligent agents based on the concepts of lookahead and intuition. Lookahead is utilized in estimating the values of available actions at the next state, while intuition guides the maximization of the probability of selecting the approximate optimal action. Through simulation experiments, we evaluate the performance of these algorithms, and the results demonstrate the superiority of the estimator–selector–actor–critic approach over other existing algorithms. [ABSTRACT FROM AUTHOR]

With the extensive utilization of fossil fuels, environmental concerns have been growing. Hydrogen-powered vehicles (HPVs) as well as hydrogen refueling stations (HRSs) are expected to proliferate to alleviate the pollution. In order to promote the economic valibity of HRSs, this paper proposes an optimal scheduling model for distributed HRSs to maximize HRSs' revenue during a day. HRSs serves both the mobility sector with fuel individually and the power system with reserve demand response collaboratively. The elaborated operating of electrolysers and relaxed control of hydrogen storage are involved to enhance practicability and flexibility of the scheduling. Considering that this model has multiple variables, intractable high-dimension nonlinear constraints, a transfer multi-agent reinforcement learning algorithm is developed. The algorithm consists of two parts: the one is the multi-agent optimization method by transforming multi-decision optimization into a multi-agent optimization; the other is to transfer the empirical knowledge of the source task to reinforcement learning according to the similarity between the source task and the target task, making it cut down the blindness of exploration and accelerate convergence. Numerical studies reveal that participating in the reserve services market collaboratively increases overall revenues of HRSs by up to 32.90%, and the computation time is sharply shortened by approximately 34 times compared to the basic reinforcement learning. • A joint operation model of distributed HRSs is proposed to lift revenue by 32.90%. • Relaxed control of hydrogen storage is utilized to minimize service deviation. • An evolutionary transfer reinforcement learning shortens solving time by 34 times. [ABSTRACT FROM AUTHOR]

In this paper, a fractional-order control method based on the twin-delayed deep deterministic policy gradient (TD3) algorithm in reinforcement learning is proposed. A fractional-order disturbance observer is designed to estimate the disturbances, and the radial basis function network is selected to approximate system uncertainties in the system. Then, a fractional-order sliding-mode controller is constructed to control the system, and the parameters of the controller are tuned using the TD3 algorithm, which can optimize the control effect. The results show that the fractional-order control method based on the TD3 algorithm can not only improve the closed-loop system performance under different operating conditions but also enhance the signal tracking capability. [ABSTRACT FROM AUTHOR]

The present study focuses on the implementation of Deep Reinforcement Learning (Deep-RL) techniques for a parallel manipulator robot, specifically the Delta Robot, within a simulated setting. We introduced a simulation framework designed to guide the Delta Robot's end-effector to a designated spatial point accurately. Within this environment, the robotic agent undergoes a learning process grounded in trial and error. It garners positive rewards for successful predictions regarding the next action and faces negative repercussions for inaccuracies. Through this iterative learning mechanism, the robot refines its strategies, thereby establishing improved decision-making rules based on the ever-evolving environment states. Our investigation delved into three distinct Deep-RL algorithms: the Deep Q-Network Algorithm (DQN), the Double Deep Q-Network (DDQN), and the Trust Region Policy Optimization Algorithm (TRPO). All three methodologies were adept at addressing the challenge presented, and a comprehensive discussion of the findings is encapsulated in the subsequent sections of the paper. [ABSTRACT FROM AUTHOR]

Reinforcement learning is an important technique in various fields, particularly in automated machine learning for reinforcement learning (AutoRL). The integration of transfer learning (TL) with AutoRL in combinatorial optimization is an area that requires further research. This paper employs both AutoRL and TL to effectively tackle combinatorial optimization challenges, specifically the asymmetric traveling salesman problem (ATSP) and the sequential ordering problem (SOP). A statistical analysis was conducted to assess the impact of TL on the aforementioned problems. Furthermore, the Auto_TL_RL algorithm was introduced as a novel contribution, combining the AutoRL and TL methodologies. Empirical findings strongly support the effectiveness of this integration, resulting in solutions that were significantly more efficient than conventional techniques, with an 85.7% improvement in the preliminary analysis results. Additionally, the computational time was reduced in 13 instances (i.e., in 92.8% of the simulated problems). The TL-integrated model outperformed the optimal benchmarks, demonstrating its superior convergence. The Auto_TL_RL algorithm design allows for smooth transitions between the ATSP and SOP domains. In a comprehensive evaluation, Auto_TL_RL significantly outperformed traditional methodologies in 78% of the instances analyzed. [ABSTRACT FROM AUTHOR]

Prosthetic technology has witnessed remarkable advancements, yet challenges persist in achieving autonomous grasping control while ensuring the user's experience is not compromised. Current electronic prosthetics often require extensive training for users to gain fine motor control over the prosthetic fingers, hindering their usability and acceptance. To address this challenge and improve the autonomy of prosthetics, this paper proposes an automated method that leverages computer vision-based techniques and machine learning algorithms. In this study, three reinforcement learning algorithms, namely Soft Actor-Critic (SAC), Deep Q-Network (DQN), and Proximal Policy Optimization (PPO), are employed to train agents for automated grasping tasks. The results indicate that the SAC algorithm achieves the highest success rate of 99% among the three algorithms at just under 200,000 timesteps. This research also shows that an object's physical characteristics can affect the agent's ability to learn an optimal policy. Moreover, the findings highlight the potential of the SAC algorithm in developing intelligent prosthetic hands with automatic object-gripping capabilities. [ABSTRACT FROM AUTHOR]

This paper presents the Multiscale Medalist Learning Algorithm (MMLA) as a novel heuristic approach for complex engineering optimization problems. By extending the Medalist Learning Algorithm, MMLA offers enhanced solution efficiency. The algorithm divides the learning process into successive periods with reduced search spaces, enabling focused search efforts in promising areas. Within each period, predefined learning stages are implemented. Top performers, known as medalists, engage in self-improvement operations through neighborhood searches, while common learners either learn from medalists or adapt based on the current state using neighborhood fluctuation. The MMLA balances exploration and exploitation capabilities through a natural growth curve that determines the learning efficiency. The MMLA's effectiveness and robustness are illustrated through the solution of a two-dimensional benchmark optimization problem and the successful resolution of ten well-known engineering design optimization problems. Comparative analysis demonstrates that the MMLA consistently outperforms other algorithms, providing competitive solutions with strict feasibility and minimal variation. [ABSTRACT FROM AUTHOR]