JetRacer: Spurführung mit künstlicher Intelligenz
| Author: | Evrard Leuteu |
| Type: | Project Work |
| Start Date: | TBD |
| Submission Date: | TBD |
| Supervisor: | Prof. Dr.-Ing. Schneider |
Introduction
The goal of this project was to optimize the JetRacer’s AI for autonomous navigation on the test circuit of the Autonomous Systems Laboratory at HSHL. The JetRacer was required to drive counterclockwise in the right lane as fast and robustly as possible using MATLAB® and a self-developed PD controller.
Tasks included training a neural network, limiting maximum speed to approximately 1 m/s, taking videos during laps, and deploying the system using GPU Coder.
- Familiarization with the existing framework.
- Optimization of the AI for the circuit in the Autonomous Systems Laboratory (speed, robustness).
- Optimization of the controller (e.g., PD controller).
- Use of MATLAB to train the Jetson Nano.
- Drive the JetRacer in the right lane counterclockwise with the gamepad controller. Limit the speed via Software to a maximum (e.g., 1 m/s).
- Take a video while driving a lap with MATLAB® using a MATLAB®-script.
- Load the pretrained NN.
- Train the pretrained NN with MATLAB® with a MATLAB®-App (GUI) by clicking the desired path in the images.
- Option: Use classic lane tracking algorithms to teach the NN automatically.
- Write a PD controller that uses the NN to drive in the right lane. Program this in MATLAB® and let it run on the JetRacer GPU using GPU Coder.
- Goal: the car should drive autonomously several laps in the right lane as fast as possible.
- Documentation according to scientific standards in the HSHL Wiki.
Requirements
The project requires prior knowledge in the following subjects. If you do not meet the requirements, you must acquire these skills during the course of the project using literature or online courses.
- Experience with Artificial Intelligence/Deep Learning
- Programming in C++, Python
- Document versioning with SVN
Requirements for Scientific Work
- Scientific approach (project plan, etc.), useful article: Create a Gantt diagram
- Weekly progress reports (informative), update the Meeting Minutes – live discussion with Prof. Schneider
- Project presentation in the Wiki
- Daily backup of project results in SVN
- Daily documentation of working hours
- Student Projects with Prof. Schneider
- Requirements for Scientific Work
Theoretical Background
The NVIDIA® Jetson Nano™ is a small, powerful computing platform ideal for embedded AI projects. It allows the execution of real-time video analysis, neural network inference, and hardware-accelerated image processing using tools like MATLAB® and GPU Coder.
Classical lane detection algorithms and steering control (e.g., PD controllers) are fundamental in autonomous driving and were combined with neural networks to improve the JetRacer’s decision-making.
Experiment Preparation
Materials and Setup
- Jetson Nano Developer Kit
- USB Gamepad
- MATLAB® 2020B (compatible with CUDA 10.2) [1]
- Wi-Fi connection between Jetson Nano and host PC
Experiment Design and Requirements
- MATLAB® was used to connect to the Jetson Nano via SSH.
- The Gamepad receiver had to be physically connected to the Jetson Nano.
- MATLAB® controlled the Gamepad indirectly using SSH commands to access libraries on the Jetson.
- Data such as images and steering values were collected and transferred via Wi-Fi.
[[File:matlab_ssh_code
.jpg|frame|Fig. 3: Ssh code snipped to connect using MATLAB and JetRacer]]
Experiment Procedure
Step 1: System Connection and Gamepad Control
Since MATLAB® could not directly access the Gamepad receiver, an SSH command method was developed to control the Jetson Nano libraries from the background without delay or signal loss.
Step 2: Data Collection
- Data was collected using the integrated camera at 256×256 pixel resolution.
- Real-time steering commands were logged and synchronized with images.
- To reduce image distortion, camera calibration was performed using a calibration256by256.mat parameter.
[[File:
.png|frame|Fig. 5: Camera setup for image with calibration]]
Step 3: Neural Network Training
- Initially, a pretrained network was used with RGB images, but results were poor. with the following parameters:
- Regression methods were tried but also showed poor performance. with the following parameters:
- The project switched to **classification** using **discrete steering values**. with the following parameters:
- MATLAB® Apps (GUI) were used to manually annotate images by clicking the correct path.
Step 4: System Optimization
- Lane tracking algorithms were integrated to automatically label lane positions.
- Additional features like lane angles were extracted to improve model inputs.
- Grayscale image recordings were used to achieve better generalization and faster processing.
- Still using **classification**. with the following parameters:
Experiment Analysis
Training Results
- Models trained on discrete steering values showed significantly better results than regression-based approaches.
- Grayscale recordings increased model stability.
- Calibrated images helped reduce the steering error caused by lens distortion.
System Performance
- A maximum speed limit (~1 m/s) was successfully enforced via software.
- The JetRacer could autonomously drive several laps counterclockwise in the right lane.
- However, simultaneous recording, steering, and saving caused performance bottlenecks due to the Jetson Nano’s limited resources.
Limitations
- Hardware constraints limited the maximum processing speed.
- Initial data quality was poor due to randomized manual steering input.
- Better results were obtained after switching to discrete, consistent control signals.
Summary and Outlook
This project demonstrated that it is possible to optimize autonomous navigation on the JetRacer using a combination of classical control algorithms and neural networks. Despite hardware limitations, stable autonomous driving behavior was achieved.
Future improvements could include:
- Expanding the dataset with more diverse environmental conditions.
- Upgrading the hardware to a Jetson Xavier NX for better real-time performance.
- Improving automated labeling and refining the PD controller parameters for faster driving without loss of robustness.
SVN Repository
Getting Started
To get started, read the following articles:
- Siddiquy, T.: Automated lane following of a Waveshare JetRacer with artificial intelligence. Bachelor's Thesis
- Kamal, A.: JetRacer: Optimization of Lane Following. Project Work
- Create a Gantt Diagram
- Tips for Writing a Wiki Article
- Introduction to PAP Designer
- Introduction to SVN
Step by Step How to Program Jetson Nano Using Matlab
Useful Articles
- Deep Learning with MATLAB, NVIDIA Jetson, and ROS
- NVidia: JetRacer
- formulaedge.org
- Waveshare Wiki
- FAQ: MATLAB to control JetRacer (Jetson Nano TX1) motor
- Towards Autonomous Driving with Small-Scale Cars: A Survey of Recent Development
- Lane Detection in ROS 2 Using Deep Learning with MATLAB
- Train DQN Agent for Lane Keeping Assist
- Build Neural Networks for Self-Driving Cars with MATLAB
- DLI Training: Deep Learning for Autonomous Vehicles
- Machine Learning with Simulink and NVIDIA Jetson
- Comparative Transfer Learning Models for End-to-End Self-Driving Car
- Multi-task Deep Learning with Optical Flow Features for Self-Driving Cars
- Train DQN Agent for Lane Keeping Assist
- Real-Time End-to-End Self-Driving Car Navigation
- Hello AI World
- Jetson AI Courses and Certifications
- NVIDIA Developer Forums
- ChatGPT Guide for a Lane Following Algorithm
Literature
Schreiber, C.: AI-Supported "Follow-Me" Function Using the JetRacer as an Example. Mittweida, Hochschule Mittweida – University of Applied Sciences, Faculty of Engineering Sciences, Master's Thesis, 2023. URL: [2]
→ back to the main article: Student Projects