JetRacer: Spurführung mit künstlicher Intelligenz

Fig. 1: JetRacer AI Pro from Waveshare

Author:	Evrard Leuteu
Type:	Project Work
Start Date:	TBD
Submission Date:	TBD
Supervisor:	Prof. Dr.-Ing. Schneider

Introduction

The goal of this project was to optimize the JetRacer’s AI for autonomous navigation on the test circuit of the Autonomous Systems Laboratory at HSHL. The JetRacer was required to drive counterclockwise in the right lane as fast and robustly as possible using MATLAB® and a self-developed PD controller.

Tasks included training a neural network, limiting maximum speed to approximately 1 m/s, taking videos during laps, and deploying the system using GPU Coder.

1. Familiarization with the existing framework.
2. Optimization of the AI for the circuit in the Autonomous Systems Laboratory (speed, robustness).
3. Optimization of the controller (e.g., PD controller).
4. Use of MATLAB to train the Jetson Nano.
5. Drive the JetRacer in the right lane counterclockwise with the gamepad controller. Limit the speed via Software to a maximum (e.g., 1 m/s).
6. Take a video while driving a lap with MATLAB^® using a MATLAB^®-script.
7. Load the pretrained NN.
8. Train the pretrained NN with MATLAB^® with a MATLAB^®-App (GUI) by clicking the desired path in the images.
9. Option: Use classic lane tracking algorithms to teach the NN automatically.
10. Write a PD controller that uses the NN to drive in the right lane. Program this in MATLAB^® and let it run on the JetRacer GPU using GPU Coder.
11. Goal: the car should drive autonomously several laps in the right lane as fast as possible.
12. Documentation according to scientific standards in the HSHL Wiki.

Requirements

The project requires prior knowledge in the following subjects. If you do not meet the requirements, you must acquire these skills during the course of the project using literature or online courses.

• Experience with Artificial Intelligence/Deep Learning
• Programming in C++, Python
• Document versioning with SVN

Requirements for Scientific Work

• Scientific approach (project plan, etc.), useful article: Create a Gantt diagram
• Weekly progress reports (informative), update the Meeting Minutes – live discussion with Prof. Schneider
• Project presentation in the Wiki
• Daily backup of project results in SVN
• Daily documentation of working hours
• Student Projects with Prof. Schneider
• Requirements for Scientific Work

Theoretical Background

The NVIDIA® Jetson Nano™ is a small, powerful computing platform ideal for embedded AI projects. It allows the execution of real-time video analysis, neural network inference, and hardware-accelerated image processing using tools like MATLAB® and GPU Coder.

Classical lane detection algorithms and steering control (e.g., PD controllers) are fundamental in autonomous driving and were combined with neural networks to improve the JetRacer’s decision-making.

Experiment Preparation

Materials and Setup

• Jetson Nano Developer Kit
• USB Gamepad
• MATLAB® 2020B (compatible with CUDA 10.2) [1]
• Wi-Fi connection between Jetson Nano and host PC

Datei:Camera-setup.png

Fig. 2: Camera setup for image recording during driving

Experiment Design and Requirements

• MATLAB® was used to connect to the Jetson Nano via SSH.
• The Gamepad receiver had to be physically connected to the Jetson Nano.
• MATLAB® controlled the Gamepad indirectly using SSH commands to access libraries on the Jetson.
• Data such as images and steering values were collected and transferred via Wi-Fi.

[[File:matlab_ssh_code

.jpg|frame|Fig. 3: Ssh code snipped to connect using MATLAB and JetRacer]]

Experiment Procedure

Step 1: System Connection and Gamepad Control

Since MATLAB® could not directly access the Gamepad receiver, an SSH command method was developed to control the Jetson Nano libraries from the background without delay or signal loss.

Step 2: Data Collection

• Data was collected using the integrated camera at 256×256 pixel resolution.
• Real-time steering commands were logged and synchronized with images.
• To reduce image distortion, camera calibration was performed using a calibration256by256.mat parameter.

Fig. 4: Camera setup for image without calibration

[[File:

.png|frame|Fig. 5: Camera setup for image with calibration]]

Step 3: Neural Network Training

• Initially, a pretrained network was used with RGB images, but results were poor. with the following parameters:

Datei:Pretrained NN.png

Fig. 2: Camera setup for image recording during driving

Datei:Pretrained NN.png

Fig. 2: Camera setup for image recording during driving

• Regression methods were tried but also showed poor performance. with the following parameters:

Datei:Regression NN.png

Fig. 2: Camera setup for image recording during driving

Datei:Pretrained NN.png

Fig. 2: Camera setup for image recording during driving

• The project switched to **classification** using **discrete steering values**. with the following parameters:

Datei:Classification NN.png

Fig. 2: Camera setup for image recording during driving

Datei:Pretrained NN.png

Fig. 2: Camera setup for image recording during driving

• MATLAB® Apps (GUI) were used to manually annotate images by clicking the correct path.

Step 4: System Optimization

• Lane tracking algorithms were integrated to automatically label lane positions.
• Additional features like lane angles were extracted to improve model inputs.
• Grayscale image recordings were used to achieve better generalization and faster processing.
• Still using **classification**. with the following parameters:

Datei:Lane tracking.png

Fig. 2: Camera setup for image recording during driving

Datei:Lane tracking.png

Fig. 2: Camera setup for image recording during driving

Experiment Analysis

Training Results

• Models trained on discrete steering values showed significantly better results than regression-based approaches.
• Grayscale recordings increased model stability.
• Calibrated images helped reduce the steering error caused by lens distortion.

System Performance

• A maximum speed limit (~1 m/s) was successfully enforced via software.
• The JetRacer could autonomously drive several laps counterclockwise in the right lane.
• However, simultaneous recording, steering, and saving caused performance bottlenecks due to the Jetson Nano’s limited resources.

Limitations

• Hardware constraints limited the maximum processing speed.
• Initial data quality was poor due to randomized manual steering input.
• Better results were obtained after switching to discrete, consistent control signals.

Summary and Outlook

This project demonstrated that it is possible to optimize autonomous navigation on the JetRacer using a combination of classical control algorithms and neural networks. Despite hardware limitations, stable autonomous driving behavior was achieved.

Future improvements could include:

• Expanding the dataset with more diverse environmental conditions.
• Upgrading the hardware to a Jetson Xavier NX for better real-time performance.
• Improving automated labeling and refining the PD controller parameters for faster driving without loss of robustness.

SVN Repository

• SVN Repository

Getting Started

To get started, read the following articles:

• Siddiquy, T.: Automated lane following of a Waveshare JetRacer with artificial intelligence. Bachelor's Thesis
• Kamal, A.: JetRacer: Optimization of Lane Following. Project Work
• Create a Gantt Diagram
• Tips for Writing a Wiki Article
• Introduction to PAP Designer
• Introduction to SVN

Useful Articles

• Deep Learning with MATLAB, NVIDIA Jetson, and ROS
• NVidia: JetRacer
• formulaedge.org
• Waveshare Wiki
• FAQ: MATLAB to control JetRacer (Jetson Nano TX1) motor
• Towards Autonomous Driving with Small-Scale Cars: A Survey of Recent Development
• Lane Detection in ROS 2 Using Deep Learning with MATLAB
• Train DQN Agent for Lane Keeping Assist
• Build Neural Networks for Self-Driving Cars with MATLAB
• DLI Training: Deep Learning for Autonomous Vehicles
• Machine Learning with Simulink and NVIDIA Jetson
• Comparative Transfer Learning Models for End-to-End Self-Driving Car
• Multi-task Deep Learning with Optical Flow Features for Self-Driving Cars
• Train DQN Agent for Lane Keeping Assist
• Real-Time End-to-End Self-Driving Car Navigation
• Hello AI World
• Jetson AI Courses and Certifications
• NVIDIA Developer Forums
• ChatGPT Guide for a Lane Following Algorithm

Literature

Schreiber, C.: AI-Supported "Follow-Me" Function Using the JetRacer as an Example. Mittweida, Hochschule Mittweida – University of Applied Sciences, Faculty of Engineering Sciences, Master's Thesis, 2023. URL: [2]

---

→ back to the main article: Student Projects