Rethinking Exploration in Reinforcement Learning with Effective Metric-Based Exploration Bonus

Enhancing exploration in reinforcement learning (RL) through the incorporation of intrinsic rewards, specifically by leveraging  state discrepancy measures within various metric spaces as exploration bonuses, has emerged as a prevalent strategy to encourage agents to visit novel states. The critical factor lies in how to quantify the difference between adjacent states as novelty for promoting effective exploration. Nonetheless, existing methods that evaluate state discrepancy in the latent space under  or  norm often depend on count-based episodic terms as scaling factors for exploration bonuses, significantly limiting their scalability. Additionally, methods that utilize the bisimulation metric for evaluating state discrepancies face a theory-practice gap due to improper approximations in metric learning, particularly struggling with hard exploration tasks. To overcome these challenges, we introduce the Effective Metric-based Exploration-bonus (EME). EME critically examines and addresses the inherent limitations and approximation inaccuracies of current metric-based state discrepancy methods for exploration, proposing a robust metric for state discrepancy evaluation backed by comprehensive theoretical analysis. Furthermore, we propose the diversity-enhanced scaling factor integrated into the exploration bonus to be dynamically adjusted by the variance of prediction from an ensemble of reward models, thereby enhancing exploration effectiveness in particularly challenging scenarios. Extensive experiments are conducted on hard exploration tasks within Atari games, Minigrid, Robosuite, and Habitat, which illustrate our method’s scalability to various scenarios, including pixel-based observations, continuous control tasks, and simulations of realistic environments.

Towards Human-like Trajectory Prediction for Autonomous Vehicles: A Behavior-aware Approach

Accurately predicting the trajectories of surrounding vehicles is essential for safe and efficient autonomous driving. This paper introduces a novel behavior-aware trajectory prediction model (BAT) that incorporates insights from traffic psychology, human behavior, and decision-making. BAT seamlessly integrates four modules: behavior-aware, interaction-aware, priority-aware, and position-aware. These modules perceive and understand underlying interactions and account for uncertainty and variability in prediction, enabling higher-level learning and flexibility without rigid categorization of driving behavior. Importantly, BAT eliminates the need for manual labeling in the training process and addresses the challenges of non-continuous behavior labeling and the selection of appropriate time windows. Furthermore, we evaluate BAT’s performance across the Next Generation Simulation (NGSIM), Highway Drone (HighD), Roundabout Drone (RounD), and Macao Connected Autonomous Driving (MoCAD) datasets, showcasing its superiority over prevailing state-of-the-art (SOTA) benchmarks in terms of prediction accuracy and efficiency. Remarkably, even when trained on reduced portions of the training data (25%) and with a much smaller number of parameters, BAT outperforms most of the baselines, demonstrating its robustness and efficiency in challenging traffic scenarios, including highways, roundabouts, campuses, and busy urban locales. This underlines its potential to reduce the amount of data required to train autonomous vehicles, especially in corner cases. In conclusion, the behavior-aware model represents a significant advancement in the development of autonomous vehicles capable of predicting trajectories with the same level of proficiency as human drivers.

Learning 3D Geometry from Visual Consistency for Monocular Vision Tasks in Self-Driving Cars

Monocular vision tasks, like depth estimation and 3D object detection in self-driving cars, face challenges due to the limited 3D information in 2D images. This highlights our recent achievements that leverage visual consistency to enhance monocular vision models. We first introduce the Direction-aware Cumulative Convolution Network (DaCCN) for self-supervised depth estimation. DaCCN addresses direction sensitivity and environmental dependency, improving feature representation and achieving state-of-the-art performance. Next, we present the Rich-resource Prior Depth estimator (RPrDepth), which uses rich-resource data as prior information. RPrDepth estimates depth from a single low-resolution image by referencing pre-extracted features. Finally, we discuss a weakly-supervised approach for monocular 3D object detection that relies only on 2D labels. By using spatial and temporal view consistency, this method achieves results comparable to fully supervised models and enhances performance with minimal labeled data.

An Asymmetric LoRA Architecture for Efficient Fine-Tuning

Adapting Large Language Models (LLMs) to new tasks through fine-tuning has been made more efficient by the introduction of Parameter-Efficient Fine-Tuning (PEFT) techniques, such as LoRA. However, these methods often underperform compared to full fine-tuning, particularly in scenarios involving complex datasets. This issue becomes even more pronounced in complex domains, highlighting the need for improved PEFT approaches that can achieve better performance. Through a series of experiments, we have uncovered two critical insights that shed light on the training and parameter inefficiency of LoRA. Building on these insights, we have developed HydraLoRA, a LoRA framework with an asymmetric structure that eliminates the need for domain expertise. Our experiments demonstrate that HydraLoRA outperforms other PEFT approaches, even those that rely on domain knowledge during the training and inference phases.

Night-Voyager: Consistent and Efficient Nocturnal Vision-Aided State Estimation in Cross-Modal Maps

Accurate and robust state estimation at nighttime is essential for autonomous navigation of mobile robots to achieve nocturnal or round-the-clock tasks. An intuitive and practical question arises: Can low-cost standard cameras be exploited for nocturnal state estimation? Regrettably, most current visual methods tend to fail due to adverse illumination conditions, even with active lighting or image enhancement. A crucial insight, however, is that the streetlights in most urban scenes can provide static and salient visual information at night. This inspires us to design an object-level nocturnal vision-aided state estimation framework, named Night-Voyager, which leverages cross-modal maps and keypoints to enable the versatile all-day localization. Night-Voyager starts with a fast initialization module which solves the global localization problem. With the effective two-stage cross-modal data association approach, the system state can be accurately updated by the map-based observations. Meanwhile, to address the challenge of large uncertainties in visual observations during nighttime, a novel matrix Lie group formulation and a feature-decoupled multi-state invariant filter are proposed for consistent and efficient state estimation. Comprehensive experiments on both simulation and diverse real-world scenarios (about 12.3 km total distance) showcase the effectiveness, robustness, and efficiency of Night-Voyager, filling a critical gap in nocturnal vision-aided state estimation.

Resilient Control for Demand Response in Smart Grid Against Cyber-Attacks

To accommodate power fluctuations caused by renewable energies and maintain the power balance in smart grid, flexible load resources on the demand side have been widely employed to provide flexibility services, a process known as named demand response (DR). To effectively dispatch such distributed flexible load resources, advanced communication, information, and distributed control techniques are developed in the field of DR, transforming DR into a cyber-physical system (CPS). While deep cyber-physical coupling improves the performance of DR, it also introduces cyber-security threats, such as cyber-attacks, which can cause DR system to go out of control, thereby threatening the smart grid’s safe operation. To this end, this talk will focus on analyzing the impact of cyber-attacks on the DR system in smart grid and developing control-based defense schemes to safeguard the flexibility capability of the DR system in harsh cyber environments.

Improving Model Generalization for Short-Term Customer Load Forecasting with Causal Inference

Short-term customer load forecasting is vital for the normal operation of power systems. Unfortunately, conventional machine learning-based forecasting methods are susceptible to generalization issues, manifested in model performance degradation. In recent years, some studies have employed advanced deep learning technology to overcome the aforesaid problems. However, these methods can only alleviate the adverse impacts of generalization, because they are inherently built on unstable relationships. In this talk, we present a causal inference-based method to improve the generalization for customer load forecasting models. Specifically, we first investigate the causal relations in existing methods, and inject the designed load characteristics as an extra model input. Then, we closely inspect the causality in models by using the causal graph, followed by employing the causal intervention with do-calculus to eliminate the spurious correlations. In addition, we present a novel load forecasting framework to realize the causal intervention. Finally, the effectiveness and superiority of our proposed method are validated on a public dataset.

Coordinated Optimization of Power-Communication Coupling Networks for Dispatching Large-Scale Flexible Resources

The growth of renewable energies elevates the significance of maintaining system balance and imposes more demands on regulation resources. Flexible loads have been extensively regarded as prospective regulation resources for providing ancillary services within power networks, including frequency regulation, primary reserve, and synchronized reserve. However, the dispatch of flexible loads presents the challenges of frequent data transmission and explosive data volume, leading to substantial pressure on communication networks. To improve the communication performance for providing effective ancillary services, the work focuses on the coordinated optimization of power-communication coupling networks for dispatching large-scale flexible resources. Firstly, this talk will introduce the comprehensive framework of model-operation-planning-engineering derived from our ongoing and upcoming works. Then, this talk will share the recent progress in the equivalent modeling of communication networks and the coordinated operation of coupling networks considering dynamic communication prices. Finally, this talk will provide a concise overview of the future works.

Hierarchical Coordinated Control Strategies for Flexible Interconnected Power Grids

The integration of renewable energy sources (RESs) into active distribution networks (ADNs) is essential for reducing carbon emissions but presents significant challenges to the traditional power grid’s structure and regulation capability. Employing soft open point for flexible interconnection can enhance power flow regulation capabilty and mitigate the power imbalance among different regions. In modern power grids, power electronic equipment coexists with conventional regulation devices, each with different response times and operational frequencies. Coordinating these heterogeneous controllable devices efficiently, considering their varied characteristics, is a complex challenge. This work addresses these challenges by leveraging the fast response of power electronics to enhance system robustness and stability. This presentation will detail the completed work on hierarchical coordinated volt/var and volt/watt control for ADNs with SOPs considering voltage stability. The future study plan, which emphasizes millisecond-level coordination control considering power electronics dynamics based on a real-time digital system (RTDS), will also be included.

Power Converter’s IGBT Multi-State Reliability Analysis for Low Failure Rate Operation

The reliability of power electronics is a cornerstone of modern technology, ensuring the efficient conversion and control of electrical power in applications ranging from renewable energy systems to electric vehicles and industrial automation. As the demand for reliable power electronics continues to grow, the focus on enhancing the durability and performance of switching devices becomes paramount. Switching devices, the most vulnerable component within converters, underscore the critical need to improve their reliability and prolong the power converter’s lifetime. They are subject to various stresses, including thermal, electrical, and environmental factors, which can lead to premature failure and compromise system performance. This talk will discuss the importance of power electronic reliability, emphasizing the vulnerabilities of IGBTs. Sharing our ongoing work utilizes multistate reliability analysis to assess the reliability of these critical components rigorously. Moreover, this talk will show our proposed model for analyzing IGBT reliability, showcasing how multistate reliability analysis offers insights into their performance under different voltage conditions. Finally, this talk will outline future research directions aimed at further improving the reliability of power electronics.

A Quasi-Z-Source Based Fault-Tolerant PV Micro-Inverter: Design and Control

PV generators are a critical means to decarbonize urban energy systems and alleviate the energy crisis. However, the system reliability is easily compromised by the DC-link capacitor failure and PV module malfunction. To address these problems, a quasi-z-source-based fault-tolerant PV micro-inverter is proposed. The proposed topology features high circuit reliability and fault tolerance with concurrent functions of active power decoupling and differential PV module current manipulation. In this talk, the circuit design will be elaborated. The operation modes of the topology as well as its modulation scheme will be introduced. Based on the circuit models, the control strategy of the proposed converter will be discussed. Finally, simulation results will be analyzed to demonstrate the concurrent functions of APD and fault-resilient operation.

Resilient Control for Demand Response in Smart Grid Against Cyber-Attacks

To accommodate power fluctuations caused by renewable energies and maintain the power balance in smart grid, flexible load resources on the demand side have been widely employed to provide flexibility services, a process known as named demand response (DR). To effectively dispatch such distributed flexible load resources, advanced communication, information, and distributed control techniques are developed in the field of DR, transforming DR into a cyber-physical system (CPS). While deep cyber-physical coupling improves the performance of DR, it also introduces cyber-security threats, such as cyber-attacks, which can cause DR system to go out of control, thereby threatening the smart grid’s safe operation. To this end, this talk will focus on analyzing the impact of cyber-attacks on the DR system in smart grid and developing control-based defense schemes to safeguard the flexibility capability of the DR system in harsh cyber environments.

Improving Model Generalization for Short-Term Customer Load Forecasting with Causal Inference

Short-term customer load forecasting is vital for the normal operation of power systems. Unfortunately, conventional machine learning-based forecasting methods are susceptible to generalization issues, manifested in model performance degradation. In recent years, some studies have employed advanced deep learning technology to overcome the aforesaid problems. However, these methods can only alleviate the adverse impacts of generalization, because they are inherently built on unstable relationships. In this talk, we present a causal inference-based method to improve the generalization for customer load forecasting models. Specifically, we first investigate the causal relations in existing methods, and inject the designed load characteristics as an extra model input. Then, we closely inspect the causality in models by using the causal graph, followed by employing the causal intervention with do-calculus to eliminate the spurious correlations. In addition, we present a novel load forecasting framework to realize the causal intervention. Finally, the effectiveness and superiority of our proposed method are validated on a public dataset.

Coordinated Optimization of Power-Communication Coupling Networks for Dispatching Large-Scale Flexible Resources

The growth of renewable energies elevates the significance of maintaining system balance and imposes more demands on regulation resources. Flexible loads have been extensively regarded as prospective regulation resources for providing ancillary services within power networks, including frequency regulation, primary reserve, and synchronized reserve. However, the dispatch of flexible loads presents the challenges of frequent data transmission and explosive data volume, leading to substantial pressure on communication networks. To improve the communication performance for providing effective ancillary services, the work focuses on the coordinated optimization of power-communication coupling networks for dispatching large-scale flexible resources. Firstly, this talk will introduce the comprehensive framework of model-operation-planning-engineering derived from our ongoing and upcoming works. Then, this talk will share the recent progress in the equivalent modeling of communication networks and the coordinated operation of coupling networks considering dynamic communication prices. Finally, this talk will provide a concise overview of the future works.

Hierarchical Coordinated Control Strategies for Flexible Interconnected Power Grids

The integration of renewable energy sources (RESs) into active distribution networks (ADNs) is essential for reducing carbon emissions but presents significant challenges to the traditional power grid’s structure and regulation capability. Employing soft open point for flexible interconnection can enhance power flow regulation capabilty and mitigate the power imbalance among different regions. In modern power grids, power electronic equipment coexists with conventional regulation devices, each with different response times and operational frequencies. Coordinating these heterogeneous controllable devices efficiently, considering their varied characteristics, is a complex challenge. This work addresses these challenges by leveraging the fast response of power electronics to enhance system robustness and stability. This presentation will detail the completed work on hierarchical coordinated volt/var and volt/watt control for ADNs with SOPs considering voltage stability. The future study plan, which emphasizes millisecond-level coordination control considering power electronics dynamics based on a real-time digital system (RTDS), will also be included.

Power Converter’s IGBT Multi-State Reliability Analysis for Low Failure Rate Operation

The reliability of power electronics is a cornerstone of modern technology, ensuring the efficient conversion and control of electrical power in applications ranging from renewable energy systems to electric vehicles and industrial automation. As the demand for reliable power electronics continues to grow, the focus on enhancing the durability and performance of switching devices becomes paramount. Switching devices, the most vulnerable component within converters, underscore the critical need to improve their reliability and prolong the power converter’s lifetime. They are subject to various stresses, including thermal, electrical, and environmental factors, which can lead to premature failure and compromise system performance. This talk will discuss the importance of power electronic reliability, emphasizing the vulnerabilities of IGBTs. Sharing our ongoing work utilizes multistate reliability analysis to assess the reliability of these critical components rigorously. Moreover, this talk will show our proposed model for analyzing IGBT reliability, showcasing how multistate reliability analysis offers insights into their performance under different voltage conditions. Finally, this talk will outline future research directions aimed at further improving the reliability of power electronics.

A Quasi-Z-Source Based Fault-Tolerant PV Micro-Inverter: Design and Control

PV generators are a critical means to decarbonize urban energy systems and alleviate the energy crisis. However, the system reliability is easily compromised by the DC-link capacitor failure and PV module malfunction. To address these problems, a quasi-z-source-based fault-tolerant PV micro-inverter is proposed. The proposed topology features high circuit reliability and fault tolerance with concurrent functions of active power decoupling and differential PV module current manipulation. In this talk, the circuit design will be elaborated. The operation modes of the topology as well as its modulation scheme will be introduced. Based on the circuit models, the control strategy of the proposed converter will be discussed. Finally, simulation results will be analyzed to demonstrate the concurrent functions of APD and fault-resilient operation.