JetRacer: Spurführung mit künstlicher Intelligenz

Fig. 1: JetRacer AI Pro from Waveshare

Author:	Evrard Leuteu
Type:	Project Work
Start Date:	TBD
Submission Date:	TBD
Supervisor:	Prof. Dr.-Ing. Schneider

Introduction

The goal of this project was to optimize the JetRacer’s AI for autonomous navigation on the test circuit of the Autonomous Systems Laboratory at HSHL. The JetRacer was required to drive counterclockwise in the right lane as fast and robustly as possible using MATLAB® and a self-developed PD controller.

Tasks included training a neural network, limiting maximum speed to approximately 1 m/s, taking videos during laps, and deploying the system using GPU Coder.

1. Familiarization with the existing framework.
2. Optimization of the AI for the circuit in the Autonomous Systems Laboratory (speed, robustness).
3. Optimization of the controller (e.g., PD controller).
4. Use of MATLAB to train the Jetson Nano.
5. Drive the JetRacer in the right lane counterclockwise with the gamepad controller. Limit the speed via Software to a maximum (e.g., 1 m/s).
6. Take a video while driving a lap with MATLAB^® using a MATLAB^®-script.
7. Load the pretrained NN.
8. Train the pretrained NN with MATLAB^® with a MATLAB^®-App (GUI) by clicking the desired path in the images.
9. Option: Use classic lane tracking algorithms to teach the NN automatically.
10. Write a PD controller that uses the NN to drive in the right lane. Program this in MATLAB^® and let it run on the JetRacer GPU using GPU Coder.
11. Goal: the car should drive autonomously several laps in the right lane as fast as possible.
12. Documentation according to scientific standards in the HSHL Wiki.

Requirements

The project requires prior knowledge in the following subjects. If you do not meet the requirements, you must acquire these skills during the course of the project using literature or online courses.

• Experience with Artificial Intelligence/Deep Learning
• Programming in C++, Python
• Document versioning with SVN

Requirements for Scientific Work

• Scientific approach (project plan, etc.), useful article: Create a Gantt diagram
• Weekly progress reports (informative), update the Meeting Minutes – live discussion with Prof. Schneider
• Project presentation in the Wiki
• Daily backup of project results in SVN
• Daily documentation of working hours
• Student Projects with Prof. Schneider
• Requirements for Scientific Work

Project Plan

Datei:Project work.png

Fig. 0: Gantt_Diagram

Previous Work and Researched on The Topic

• Apu, M. L.: JetRacer: Spurführung mit künstlicher Intelligenz und MATLAB/Simulink. Projektarbeit
  
• Kamal, A.: JetRacer: Optimierung der Streckenführung. Projektarbeit

Theoretical Background

The NVIDIA® Jetson Nano™ is a small, powerful computing platform ideal for embedded AI projects. It allows the execution of real-time video analysis, neural network inference, and hardware-accelerated image processing using tools like MATLAB® and GPU Coder.

Classical lane detection algorithms and steering control (e.g., PD controllers) are fundamental in autonomous driving and were combined with neural networks to improve the JetRacer’s decision-making.

Convolutional Neural Networks (CNNs) are a core part of this project. They process images in layers—first detecting edges and shapes, then combining them to recognize lanes. Pooling layers reduce data size while preserving important features, helping the model run faster on limited hardware. The final layers (fully connected or “dense” layers) interpret these features and output the steering decision.

In training, the following notion should be understood:

• Epochs: represent full passes through the dataset. More epochs typically improve learning up to a point, but too many can lead to overfitting.
• Root Mean Square Error (RMSE): measures how far off the model’s predictions are—lower RMSE values indicate more accurate steering predictions.
• Accuracy: measures the percentage of correct predictions. In classification models (like predicting discrete steering directions), accuracy helps track how well the model is learning to choose the correct class. Higher accuracy indicates better model performance, especially when validation accuracy is also high.
• Loss: indicates how badly the model is performing. It is a numerical value that shows how far the model’s predictions are from the actual labels. During training, the goal is to minimize the loss function. A low training loss means the model is learning well; a low validation loss means it can generalize to new data.
• Validation: refers to the process of checking how well the model performs on unseen data (not used for training). A separate validation dataset is used to measure generalization performance and avoid overfitting.
• Overfitting: happens when the model learns the training data too well, including noise and outliers, but fails to perform well on new data. Signs of overfitting include high training accuracy but low validation accuracy.
• Underfitting: occurs when the model is too simple or not trained enough to capture the patterns in the data. Both training and validation accuracy remain low.
• Batch: a batch is a small group of training samples processed together before updating the model. Training on batches (instead of the full dataset) helps improve efficiency and generalization. The number of samples in one batch is called the batch size.
• Learning Rate: controls how big the model’s updates are during training. If it’s too high, the model may skip the optimal solution; if too low, training may be very slow or get stuck.
• Training vs. Validation Accuracy/Loss: while training accuracy/loss shows how well the model fits the training data, validation accuracy/loss indicates how well it performs on unseen data. A good model has both high accuracy and low loss on both datasets, with a small gap between them.

Regression approaches output continuous values but were unstable during testing. This project instead uses **classification**, where each image maps to one of a few predefined steering categories (e.g., left, center, right). This gave better performance and more predictable behavior.

Model Architecture

The JetRacer's steering model uses a **Convolutional Neural Network (CNN)**, a type of AI designed to process images and make decisions based on what it “sees”.

The CNN works like this:

• 1. Input Layer: Receives a grayscale image, resized to 224×224 pixels.
• 2. Convolutional Layers: These scan the image to find patterns (like lines or edges). Early layers detect simple shapes; deeper layers recognize more complex patterns like lanes.
• 3. ReLU Activation: This function makes sure only useful data passes through to the next layer.
• 4. Pooling Layers: These reduce the image size while keeping important features, which makes the network faster and less sensitive to noise.
• 5. Flatten Layer: Turns the filtered image into a 1D list of numbers.
• 6. Fully Connected Layers: These make the final decision by combining the information and outputting a single steering command.
• 7. Output Layer: The final result is a classification—e.g., turn left, go straight, or turn right.

This setup is small enough to run efficiently on the Jetson Nano, and accurate enough to follow the lane smoothly.

This model, trained on camera images with corresponding steering labels, helps the JetRacer stay in its lane without human intervention.

SVN Repository

• SVN Repository

Getting Started

To get started, read the following articles:

• Siddiquy, T.: Automated lane following of a Waveshare JetRacer with artificial intelligence. Bachelor's Thesis
• Kamal, A.: JetRacer: Optimization of Lane Following. Project Work
• Create a Gantt Diagram
• Tips for Writing a Wiki Article
• Introduction to PAP Designer
• Introduction to SVN

Useful Articles

• Deep Learning with MATLAB, NVIDIA Jetson, and ROS
• NVidia: JetRacer
• formulaedge.org
• Waveshare Wiki
• FAQ: MATLAB to control JetRacer (Jetson Nano TX1) motor
• Towards Autonomous Driving with Small-Scale Cars: A Survey of Recent Development
• Lane Detection in ROS 2 Using Deep Learning with MATLAB
• Train DQN Agent for Lane Keeping Assist
• Build Neural Networks for Self-Driving Cars with MATLAB
• DLI Training: Deep Learning for Autonomous Vehicles
• Machine Learning with Simulink and NVIDIA Jetson
• Comparative Transfer Learning Models for End-to-End Self-Driving Car
• Multi-task Deep Learning with Optical Flow Features for Self-Driving Cars
• Train DQN Agent for Lane Keeping Assist
• Real-Time End-to-End Self-Driving Car Navigation
• Hello AI World
• Jetson AI Courses and Certifications
• NVIDIA Developer Forums
• ChatGPT Guide for a Lane Following Algorithm

Literature

Schreiber, C.: AI-Supported "Follow-Me" Function Using the JetRacer as an Example. Mittweida, Hochschule Mittweida – University of Applied Sciences, Faculty of Engineering Sciences, Master's Thesis, 2023. URL: [1]

---

→ back to the main article: Student Projects