Expert Knowledge-driven Reinforcement Learning for Autonomous Racing via Trajectory Guidance and Dynamics Constraints

Bo Leng1, Weiqi Zhang1, Zhuoren Li1*, Lu Xiong1, Guizhe Jin1, Ran Yu1, Chen Lv2
1College of Automotive and Energy Engineering, Tongji University, Shanghai, 201804, China 2School of Mechanical and Aerospace Engineering, Nanyang Technological University, 639798, Singapore
* corresponding author
Paper arXiv Code

Abstract

Reinforcement learning has shown significant potential for autonomous racing, but it still faces challenges such as training instability, inefficient exploration, and unsafe action outputs in high-dynamic racing scenarios. This paper proposes a Trajectory guidance and Dynamics constraints Reinforcement Learning (TraD-RL) framework for autonomous racing. The proposed method incorporates expert prior knowledge into policy learning through Minimum Curvature Racing Line (MCRL) guidance, explicit vehicle dynamics constraints, and two-stage curriculum learning. MCRL provides global path and velocity references through observation augmentation and reward shaping, thereby improving exploration efficiency and racing performance. Yaw rate and sideslip angle constraints are introduced to characterize the vehicle dynamic safe operating envelope, and the corresponding stability costs are incorporated into policy optimization through Lagrangian relaxation. Moreover, the two-stage curriculum learning strategy enables a progressive transition from stable trajectory following to highspeed performance exploration. Experiments on two racetracks demonstrate that TraD-RL improves racing performance while maintaining a favorable balance between speed and dynamic stability. Further analyses of ablation, sensitivity, and robustness validate the effectiveness and stability of the proposed framework.

teaser image

The reinforcement learning decision-making and control framework driven by expert prior knowledge. First, to tackle the issues of low exploration efficiency and the difficulty of learning optimal trajectories in complex racing environments, we encode track geometric priors and racing intent into structured guidance information using the racing line. This approach enhances the learning efficiency and stability of the agent within high-dimensional spaces. Second, to reduce the risk of instability inherent in high-speed racing, we explicitly introduce vehicle dynamics stability constraints during policy learning. By comprehensively considering two key stability indicators, yaw rate and sideslip angle, this mechanism mitigates unsafe dynamic behaviors and improves deployment potential. Finally, to fully exploit the racing potential while ensuring training convergence, we develop a two-stage curriculum learning strategy based on dynamic task evolution. This facilitates a smooth transition of the model’s capabilities toward limit-handling dynamic racing.

Contributions

  • We propose an Minimum Curvature Racing Line (MCRL)-guided state representation and reward shaping mechanism for autonomous racing. The pre-computed MCRL is encoded into a reference-augmented observation space and used to construct dense rewards related to trajectory tracking, target-speed following, and heading alignment. By constraining exploration around the reference priors, it alleviates the sparse-reward challenge in high-dimensional continuous action spaces and enables rapid convergence toward a high-performance policy that balances speed and stability.
  • We introduce an explicit dynamics-constrained policy learning method based on vehicle dynamics priors. Yaw-rate and sideslip-angle limits are used to characterize a dynamics-based safe operating envelope, and their CBF-inspired safety margins are transformed into stability cost functions that are incorporated into policy optimization through Lagrangian relaxation. This mechanism regularizes unstable control behaviors during trial-and-error learning, thereby mitigating dynamic instability while maintaining sufficient exploration viability.
  • We propose a progressive two-stage curriculum learning strategy. This strategy partitions the learning process into a trajectory guidance stage and a high-speed exploration stage. By implementing an easy-to-hard training schedule through this progressive reward formulation, we significantly enhance both the learning efficiency and the racing performance of the reinforcement learning agent.

Method Implementation

observation

Observation Construction. Illustration of the ego-centric grid observation space construction, with the MCRL prior and its occupancy-grid encoding. The generated MCRL trajectory is first represented in the global racetrack map and then transformed into the vehicle-aligned local coordinate system, where it is encoded as the MCRL observation in the ego-centric occupancy grid.

train

Two-Stage Curriculum Design and RL Agent Training. SAC-based constrained RL network structure and the pseudocode of proposed TraD-RL Algorithm.

Results Analysis

track

Aerial views of the two racetracks used in the simulation environments.

train results in 2 track

Learning curves of multi-dimensional performance metrics for different algorithms during the training process: (a)time-averaged lap reward; (b) lap time; (c) lap average speed; and (d) lap progress. Solid lines denote the mean values, while the shaded regions represent 95% confidence intervals over five runs.

Test results1 in 2 track

Quantitative comparison of racing performance and safety metrics across different algorithms on different racetracks during the testing process. Values are presented as mean ± standard deviation. Bold values denote the best results.

Test results2 in 2 track

Statistical distributions of control inputs and vehicle dynamic states for different algorithms during testing on the Berlin Racetrack: (a) yaw rate; (b) sideslip angle; (c) longitudinal acceleration; and (d) steering angle.

Test results3 in 2 track

Statistical distributions of control inputs and vehicle dynamic states for different algorithms during testing on the Modena Racetrack: (a) yaw rate; (b) sideslip angle; (c) longitudinal acceleration; and (d) steering angle.

case study

Experimental results comparison in a continuous corner (S-curve) section of the Berlin Tempelhof Airport Street Circuit. Left: Trajectory and speed heat map distributions of TAL and the proposed method (Ours), with the red solid line representing the MCRL. Right: Time-series comparison of vehicle speed, yaw rate, and sideslip angle during the cornering process. Down: G-G diagram comparison of longitudinal and lateral acceleration distributions in the continuous corner (S-curve).

ablation

Quantitative comparison of racing performance and safety metrics across different ablation algorithms during the testing process. Values are presented as mean ± standard deviation.

BibTeX

@article{tradrl2026,
      title={Expert Knowledge-driven Reinforcement Learning for Autonomous Racing via Trajectory Guidance and Dynamics Constraints},
      author={Leng, Bo and Zhang, Weiqi and Li, Zhuoren and Xiong, Lu and Jin, Guizhe and Yu, Ran and Lv, Chen},
      journal={arXiv preprint arXiv:2603.05842},
      year={2026}
}