Controlled Autonomous Driving for a JetRacer

Vid.1 JetRacer ROS AI Driving at 1.44 m/s

Abb. 1: JetRacer ROS AI Roboter von Waveshare

Autor:	Yashodhan Vishvesh Deshpande
Art:	bachelor thesis
Starttermin:	20.01.2026
Abgabetermin:	19.05.2026
Betreuer:	Prof. Dr.-Ing. Schneider

Task List

1. Research of similar projects
2. Evaluation of the solutions using a morphological box (Zwicky box)
3. Choose a software toolchain
4. Design and print a platform for the TopCon-prism.
5. Record the ground truth data
6. Track the JetRacer on a digital map.
7. Control the JetRacer in the right lane without obstacles by camera data.
8. Evaluate the results based on technical features
9. Discussion of the results
10. Testing of the system requirements - proof of functionality
11. Scientific documentation as a report
12. Scientific advertising as a wiki article with a result video.

Thematic Introduction

The JetRacer:_ROS_AI_Kit can be programmed in different ways. The mechatronic student learn to develop C-code, m-code oder model based development with simulink. The task is to develop a "easy to use" framework to get quick results. After choosing the toolchain based on a morphological box an attachment for the reference prism (Referenzmessung_mit_der_Topcon_Robotic_Total_Station) has to be build. By this the ground-truth-data (vehicle pose) can be recorded for further analyzing. The main task is to develop a program that acquires an image frame, transforms it into bird-eye-view to measure the lane's curvature. Based on the lane curvature and actual driving status the car should steer into the right lane. Optional a gyroscope can measure the yaw-rate of the JetRacer.

The goal is to drive in the right lane as fast as possible.

Optional an obstacle detection an avoidance by LiDAR can be added. Future students can compare an AI lanekeeping to this classic image processing approach.

The Bachelor Thesis was carried out in the Autonomous Systems Laboratory at Hochschule Hamm-Lippstadt. The Bachelor Thesis was carried out under the supervision of Prof. Dr. Ulrich Schneider and Prof. Marc Ebmeyer. The Laboratory is designed for Mechatronic and Electronic Engineering Students to realize there Academic Projects.

Bachelor Thesis Plan

The image shown in Fig.2 is the thesis timeline that is illustrated using a Gantt Chart.

Fig.2 Working Timelines for Thesis

Theoretical Background

• Hardware Requirements

The Jetracer ROS Ai kit is used for this thesis as a primary autonomous vehicle. All its system that are integrated in the Jetracer were used for example the LIDAR, Gyroscope, camera, motors, Jetson nano board, Raspberry Pi RP2040 board and the batteries. The autonomous system Laboratory PC was used for the image processing. The operating system for the Jetracer was installed on a external 64 GB micro SD card. To setup the OS a USB supported mouse and keyboard along with HDMI supported monitor was used. The autonomous system laboratory Wifi was used for communication between the Jetracer and the image processing PC. The Topcon Total robotic station PS-103A was used to measure the ground truth of the vehicle by prism tracking. The 3D printing was performed using laboratory facilities of the university. A checkerboard black and white square pattern of 30.83mm length was used to calibrate the camera. Lastly a infrared sensor based speed measuring device was used to measure the real time speed of the vehicle. The hardware requirements are summerised in Tab. 1.

Tab. 1 Hardware Requirements

Component Type	Component Name
Autonomous Vehicle	Jetracer ROS AI Kit
Image processing Unit	Autonomous System Laboratory Windows 10 PC
OS platform	64 GB mircro SD card
Communication setup	Autonomous System Laboratory Wifi
Jetracer OS setup hardware	USB supported keyboard, mouse and HDMI supported monitor
Ground truth data tracking	TopCon Robotic Total Station PS-103A Kit
3D printing	HSHL University 3D printer
Camera calibration setup	Autonomous system laboratory camera calibration checkerboard
Vehicle speed measurement	Infrared based vehicle speed measurement device

• Software Requirements

windows 10 OS PC is required where Matlab 2019b is installed. This is where the image processing occurs. The Matlab should have the necessary extension packages like Matlab Coder and GPU Coder to generate CUDA codes that can be later exported to the Jetracer. A Solidowork 2019 version is also required for CAD designing and 3D printing of the prism holder from the Topcon system. Putty software was used for initial serial communication with the Jetracer. The Jetpack OS was installed on the 64 GB micro SD card that was used by the Jetracer Jetson Nano Board. The installation guide can be found on JetPack_OS_Online_Guide .Knowledge of Matlab m code, Python and bash code is needed as these are the programming languages used in this project. Python and bash programming languages are used for the Jetracer side server programming while the Matlab m code is used Windows PC side programming. The Tab. 2 summerizes the software requirements for this thesis.

Tab. 2 Software Requirements

Software Applications
Matlab 2019b
Solidworks 2019
Jetpack OS
Windows 10 OS
Putty
NVIDIA CUDA Toolkit
MATLAB Coder Support Package for NVIDIA Jetson
GPU Coder

• Lateral Offset Error

Fig.3 Error Estimation

The lateral offset error is the horizontal distance between vehicle position and the target path trajectory. This error is calculated at a specific look ahead distance in the birds eye view image to avoid jitter in the target path polynomial construction. This error can be calculated by the difference between the target path polynomial trajectory and the birds eye view image center at the look ahead distance. The error calculation equation is $e_L=(x_T-B_c).s_x$ where $e_L$ is the lateral error in m. $x_t$ is the target path x-coordinate in the bird’s-eye-view image [pixels]. $B_c$ is the horizontal center coordinate of the bird’s-eye-view image [pixels]. $s_x$ is the Pixel-to-meter scaling factor in x-direction [m/pixel].

• Localized Heading Error

The localized heading error is used to compensate between the angular orientation of the vehicle and target path trajectory direction. This error is estimated by calculating the angular difference between a chord on the target path polynomial and a unit vector that is parallel to the camera optical axis. The heading error calculation equation is $e_h={\tan }^{-1}\left(\frac{x_2-x_1}{y_1-y_2}\right)$ where $e_h$ is the heading error [rad] , $(x_1,y_1)$ is first point defining the target path chord [pixels] and $(x_2,y_2)$ is second point defining the target path chord [pixels] .

• Combined Error

Both the lateral error and the localized heading errors are combined into a single error to give input to the PID controller. The weighted ratio is used to combine these error. The equation to combine these error is $e_c=k_L.e_L+k_h.e_h$ where $e_c$ is the combined error numeric value, $k_L$ is lateral error weighting gain and $k_h$ is heading error weighting gain. The gain $k_L$ and $k_h$ determine the significance of each error on the controller behavior. A large $k_L$ means the controller would aggressively center the vehicle while a larger $k_h$ means the controller will steer the vehicle more smoothly over a curved path.

Fig.3 show the visual illustration of physical origins of the error where the red colored polynomial is fitted on the detected right white solid lane marking. The yellow colored polynomial in center dashed lane marking prediction. The green colored polynomial that is in between the red colored polynomial and the yellow colored predicted polynomial is the lane center target path trajectory that the jetracer has to drive on. The blue dashed line in the center is the camera optical axis. The magenta colored horizontal line between blue dashed line and the green colored target path polynomial is the lateral error distance of the jetracer from the target path. The yellow colored thick chord on the green colored target path polynomial is the lookahead distance for the heading error calculation that is 0.55 m to 0.80 m. The white colored unit vector that starts at 0.55 m is parallel to the camera optical axis. The angle between this white unit vector and the yellow thick chord is the heading error angle that is denoted by $\psi$ .

• PID Controller

A standard PID controller was chosen to calculate the steering commands for the vehicle. A PID controlled steering is stable and robust. The combined error is given as input to the PID controller. The controller equation is $U(t)=K_P.e_c(t)+K_I.\int e_c(t)dt+K_D.\frac{d{e}_c(t)}{dt}$ where $U(t)$ is the PID controller output / steering control command, $K_P$ is proportional gain of PID controller , $K_I$ is integral gain of PID controller and $K_D$ is differential gain of PID controller.

Proposed Solutions

The table Tab. 3 Proposed solution options is used to present a holistic view of the proposed solution for the controlled autonomous driving for a jetracer. This table is created in perspective of limited duration of the bachelor thesis and does not completely explore all possible solutions to the autonomous driving for jetracer as the jetracer can be driven in multiple ways than the solutions proposed here.

Tab. 3 Proposed solution options

Parameter	Option 1	Option 2	Option 3
Software Tool Chain	C Programming	Matlab M code	Simulink Model
Lane Detection Approaches	Hough Transform	Color Thresholding	BEV and Sliding Windows
Vehicle Steering Controller	Pure Pursuit	Stanley Style	PID

• C programming

The C programming based approach enable programming at low level. This gives access to embedded hardware components of the Jetracer directly. It also has lower computational overhead. However debugging and data visualization is significantly complex.

• Matlab M code

Matlab provides a robust prototyping and image processing environment. It has multiple data visualization option from plots to in built tuning application and functions such as Computer Vision Toolbox. However Matlab does not usually give a low level access to embedded hardware directly.

• Simulink Model

The Simulink based implementation allows to graphically develop the implementation. It includes features lie automatic code generation. This is great option for controller design application. However, its complexity increases for image processing applications. Timing issues can also affect the overall model behavior.

• Hough Transform

The Hough transform is simple method that is used for detecting straight lines. It has less computational complexity. However it might fail on curved lane sections.

• Color Thresholding

Color based thresholding have less computational complexity and have faster execution times. The disadvantage of this method is a small change in illumination conditions can disrupt the lane detection algorithms.

• Birds Eye View Transform combined with Sliding Windows

The inverse perspective transform with sliding windows does geometric lane detection. The birds eye view transform removes camera perspective distortion. The sliding windows can efficiently tacked curved section of the track. The combination of this approach allows us to implement classical lane tracking without any kind of model training. The disadvantage is that it requires the camera calibration and world coordinate system transforms.

• Pure Pursuit Controller

Pure pursuit controller is path tracking algorithm that computes steering command at predefined look ahead point on the target trajectory. This approach allows the vehicle to smoothly follow the lane. The disadvantage is that steering command error can arise from noisy lane detection and shifting target trajectories

• Stanley Style Controller

The Stanley style controller takes input of heading error and cross track errors to generate steering commands. This gives us very robust and accurate steering. However the disadvantage is that the tuning parameter for this controller are complex.

• PID Controller

The Proportional Integral Derivative Controller is a simple controller that can give accurate steering command in a feedback error loop system. It has less overhead complexity and can be easily implemented in embedded systems application. It can take incorporate lateral and heading errors.

Solution Selection for implementation

For the software toolchain parameter the Matlab m code based implementation is selected. Compared to the C program option, Matlab offers significantly better algorithm development, visualization and debugging environment. It also has wide range of functions and application available like camera calibration application and computer vision toolbox. This allows us to build a real time robust system. As compared to Simulink, Matlab has greater flexibility for developing image processing pipeline and controller tuning. Additionally Matlab is academically used by several other undergraduate programs like mechatronic students. Therefore, Matlab m code as a software toolchain fulfills all the set selection criteria. The Birds eye view and sliding windows method was chosen for the lane detection method parameter. This is because the bird eye view removes camera perspective distortion allowing us to easily implement classical lane detection algorithms. The sliding window algorithm can effectively track curved lane markings. This makes the application robust and real time as compared to other methods. The PID controller option is best suitable for the vehicle steering and controller parameter. As compared to other pure pursuit and Stanley style controllers the PID controller has less computational workload and is easy to implement. This is because it has fewer tuning parameter. The less workload make it also a suitable candidate for real time system application since it has less latency. Furthermore the PID controller is within the scope of the bachelor program.

Implementation and Results

Implementation This section discusses the algorithms and programs implemented on the Jetracer for it to drive autonomously. The wroking pipeline for various Matlab Toolboxes is shown in Fig. 5.

Fig. 5 Working pipeline in Matlab 2019b

• 3D Printing of Prisma Holder

The task set by the supervisor was to design and 3D print a platform for the Jetracer to mount the prism. The design is implemented on SolidWorks 2019 version as shown in Fig. 4. The result of 3D printing is shown in Fig. 6.

Fig. 4 Prisma holder design SolidWorks

Fig. 6 Prisma holder 3D printing result

• Image Undistortion and Birds eye view

The camera used by the Jetracer for driving is required to be calibrated as the camera lens distortion can cause straight line segments to appear curved. The camera calibration is performed using Matlab Camera Calibration App. Multiple images of the checkerboard pattern in different orientations were given as input to the Matlab application. The application then detected the checkerboard corners and determine the camera intrinsic parameters. The output was a .mat file that contained the information of the intrinsic parameter that can be later used in the implementation. The homography implementation is used to determine the camera height, pitch and the yaw angle at the calibration so that the birds eye view transform can be created. This is done by using transformation and rotation functions. A birds eye illustration on the checkerboard pattern is also created for verification purpose only so that pixel level accuracy achievement is confirmed. The Fig. 7 shows the distorted image of the lane track with white lane markings and black lane background.The camera calibration intrinsic parameters are applied to the distorted image Fig. 7 and a resulting undistorted image is produced. The undistorted image is shown in Fig. 8. It can be observed that the original image has a curvature especially in the peripheral sections of the image as compared to the undistorted image. This curvature is corrected by applying camera intrinsic parameters image undistorted. The undistorted image in Fig. 8 is then transformed into a birds eye view image using camera extrinsic parameters . The result is shown in Fig. 9. The BEV image has a resolution of 220×140 pixels.

Fig. 7 Distorted lane image

Fig. 8 Undistorted lane image

Fig. 9 BEV transformation image

• Region of Interest and Lane detection

The lane detection algorithms and masks are applied on the ROI selection based on previous lane location. The chosen ROI for the figure BEV image is shown in Fig. 10. The red band is the ROI that is dynamically generated from the previous image. This reduces the computational complexity as the ROI is only 13 percent of the whole BEV image. The lane detection masking like the binary masks and the sliding windows processes are performed on ROI of Fig. 10. The result of the lane detection algorithms are shown in Fig. 11. It can be seen that the binary mask has transformed the BEV ROI image into a complete binary black and white image with the lane markings.

Fig. 10 BEV ROI selection

Fig. 11 Detected lane

• Tracking the Jetracer on Digital Map

The Fig. 12 shows a digital map of the track located in the autonomous system laboratory. The track lines are black in color. The red tracked path is the vehicle driving path tracked by the Topcon total station by using the prism. It shows that the car successfully drove for one complete lap in the right lane. The blue tracked path is based on the data from the gyroscope and steering throttle commands by the controller. The yaw rate data from the gyroscope is integrated to get total accumulated yaw of the vehicle. This accumulated yaw gives us the direction of the vehicle. The accumulated yaw is the multiplied by the speed commands of the controller to get the displacement of the vehicle. This displacement of the vehicle based on gyroscope and speed is the blue tracked path curve. The blue curve appears slightly rotated as compared to the lane. This is because the gyroscope based yaw rate data does not contain the lane direction information.

Fig. 12 Digital map with Jetracer tracking

• Accumulated Yaw comparison of Jetracer

The accumulated yaw rate data collected by the gyroscope is integrated over time change using trapezoid rule to get the accumulated total yaw of the car. The Topcon based vehicle displacement ground truth data is differentiated over time to get the yaw of the vehicle. The Fig. 12 provides the illustration for this analysis. The red curve represents the Topcon ground truth based accumulated yaw data while the blue curve represents the Jetracer gyroscope based accumulated yaw data. The vehicle starts moving from time equal to zero seconds hence this should be the starting point for the observation. It can be observed that the curves of accumulated yaw from both the Topcon prism tracking and the Jetracer follow very closely to one another. This points to a successful gyroscope implementation. A drift in the gyroscope accumulate yaw curve is observed over a significant amount of time. This is cause by the mathematical properties of integration which is used to calculate the accumulated yaw data from the raw yaw rate data output of the gyroscope.

Fig. 12 Accumulated yaw gyroscope vs Tocpon prisma

Summary and Outlook

• Summary

All the tasks of this bachelor thesis of steering the autonomous vehicle in the right lane were achieved. The thesis uses classical lane tracking algorithm with sliding windows, birds eye view transform , lane detection implementation, steering using PID controller, gyroscope yaw rate data measurement, LIDAR module implementation for obstacle avoidance and Topcon prism tracking based ground truth data collection. It also includes the Jetson Nano JetPack OS and the python server files that were used for the Jetracer side operations. This thesis showcases an example of academic undergraduate level autonomous lane keeping. Various simulators were created during the implementation phase that can be used for lane detection simulations and PID Controller tuning. These simulators can be used to get a quick overview for the autonomous driving of the Jetracer. Vid. 1 shows the Jetracer following the right lane on the track.

• Outlook

The future students can compare this classical lane tracking algorithm with AI based implementations. This classical algorithm can also be used to build models to train deep learning neural networks. Also students can use various types of controllers like Model Predictive controller and compare them to PID controller. Multiple sensor fusion can be done with the Jetson Nano board to increase the capabilities of the Jetracer for better autonomous driving.The implementations and source codes can be used as a foundation for further development in the Autonomous driving field.

Repository

• Please link your source code repository here : https://svn.hshl.de/svn/HSHL_Projekte/trunk/JetRacer/Jetracer_ROS_AI_KIT/Daten/Controlled%20Autonomous%20Driving%20for%20a%20JetRacer
• Source Code should match the coding guidelines (MATLAB^® | C)
• Add a header and help to your files to make them understandable for the team (MATLAB^® | C).
• Describe what source code does what: Tab. 4 shows the usage and requirements of the source codes.
• What are the preconditions in hardware and software (Autonomous Driving Toolbox, Parallel Computing Toolbox, etc.)?
• Wird eine spezielle Hardware/GPU benötigt?

Tab. 4 Source Code Usage Guideline

File Name	Usage	Software Requirements	Hardware Requirements
jetson_lane_detection_video_simulator.m	Lane detection simulation using a recorded video of track	Matlab 2019b, Computer Vision Toolbox, Image Processing Toolbox, Parallel Computing Toolbox (GPU acceleration), appcameracalibration.mat file, track_recording.avi file	Windows 10 PC
appcameracalibration.mat	camera intrinsic parameters calibration file for Jetracer camera IMX219 - 160 with image resolution of Height = 360px , Width = 480px	Matlab 2019b	Windows 10 PC
track_recording.avi	Raw camera lens distorted video of track with resolution Height = 360px , Width = 480px with camera variables height = 0.126, pitch = 24.673, yawCorrection_deg = 0.459 for my jetson_lane_detection_video_simulator.m application
jetson_rtsp_final.m	Captures and save images from Jetracer camera of checkerboard pattern for camera extrinsic parameters calibration with resolution Height = 360px , Width = 480px	MATLAB Support Package for NVIDIA Jetson and NVIDIA DRIVE Platforms, Image Processing Toolbox , Matlab 2019b , Python server files for Jetracer JetPack OS	Jetracer, Wifi TCP connection, Windows 10 PC
homography_runtime_style.m	Determines camera pitch, height and yaw correction. The computed pitch angle and yaw correction are intended to be used directly in the runtime lane detection pipeline. The sign convention of negative for the pitch displayed must be ignored and a positive pitch value must be used. The camera height caluclated is only a rough estimate and can have serious errors. So a camera height based on Birds eye view region of interest must be used as the lanes can go out of regiuon of interest if the height is too much or too less. If the detected checkerboard coners appear shifted then correct them by shifting them by adding integer values to bevPoints variables that is [u , v]. This helps achieve sub pixel accuracy.	Computer Vision Toolbox, Image Processing Toolbox, Parallel Computing Toolbox (GPU acceleration), appcameracalibration.mat file, checkerboard pattern image with resolution of Height = 360px , Width = 480px
speed_increase.m	This is the main source code file that performs real-time autonomous lane following using Bird's-Eye View lane detection and PID steering control. The JetRacer drives at 1.45 m/s and the lane detection vision pipeline runs at maximum 40 FPS. This is purely vision based steering application for maximum speed and FPS. NO Gyroscope and NO Lidar	MATLAB Support Package for NVIDIA Jetson and NVIDIA DRIVE Platforms, Computer Vision Toolbox, Image Processing Toolbox, Parallel Computing Toolbox (GPU acceleration), appcameracalibration.mat file, python server files on JetRacer JetPack OS, camera extrinsic parameter values.	Jetracer, Wifi TCP connection
sim3.m	This is the PID Controller simulator used for tuning so the JetRacer can drive smoothly. It shows error osillations in technical parameters of offset (m) that is lateral error and psi (deg) that heading error in plots of LOGGED: offset and LOGGED: psi against time (s) respectively.	MATLAB R2019b, runlog.mat file for the logged data of the JetRacer Driving
structure_correction_8_vision_steering_gyro_recording.m	This file is a advanced version of the main source code file of speed_increase.m. It has included gyroscope data logging functionality with vision based steering only. NO gyroscope data used in steering. Maximum FPS achieved for lane detection pipeline is 30 FPS with maximum driving safe speed of 0.5 m/s.	MATLAB Support Package for NVIDIA Jetson and NVIDIA DRIVE Platforms, Computer Vision Toolbox, Image Processing Toolbox, Parallel Computing Toolbox (GPU acceleration), appcameracalibration.mat file, python server files on JetRacer JetPack OS, camera extrinsic parameter values.	Jetracer, Wifi TCP connection
structure_correction_9_gyro_yaw_rate_in_controller.m	This file is a advanced version of the main source code file of speed_increase.m. It has included gyroscope data logging functionality with vision + gyroscope yaw rate data based steering . Gyroscope data is used in steering controller derivative term calculation. Maximum FPS achieved for lane detection pipeline is 30 FPS with maximum driving safe speed of 0.5 m/s.	MATLAB Support Package for NVIDIA Jetson and NVIDIA DRIVE Platforms, Computer Vision Toolbox, Image Processing Toolbox, Parallel Computing Toolbox (GPU acceleration), appcameracalibration.mat file, python server files on JetRacer JetPack OS, camera extrinsic parameter values.	Jetracer, Wifi TCP connection
structure_correction_10_lidar_integrated.m	This file is a advanced version of the main source code file of speed_increase.m. It has vision based steering only with addition of simultaneous obstacle detection based on Lidar. Maximum FPS achieved for lane detection pipeline is 40 FPS with Jetracer driving speed of 1.45 m/s. Obstacle detection and avoidance set at 0.8 m from the car frontier.	MATLAB Support Package for NVIDIA Jetson and NVIDIA DRIVE Platforms, Computer Vision Toolbox, Image Processing Toolbox, Parallel Computing Toolbox (GPU acceleration), appcameracalibration.mat file, python server files on JetRacer JetPack OS, camera extrinsic parameter values.	Jetracer, Wifi TCP connection
structure_correction_11_vision_drive_and_track_recording.m	This file is a advanced version of the main source code file of speed_increase.m. The JetRacer drives at 1.45 m/s and the lane detection vision pipeline runs at maximum 40 FPS. This is purely vision based steering application but has additional functionality of video recording from the Jetracer camera. The recorded video of track can then be used by the jetson_lane_detection_video_simulator.m simulator.	MATLAB Support Package for NVIDIA Jetson and NVIDIA DRIVE Platforms, Computer Vision Toolbox, Image Processing Toolbox, Parallel Computing Toolbox (GPU acceleration), appcameracalibration.mat file, python server files on JetRacer JetPack OS, camera extrinsic parameter values.	Jetracer, Wifi TCP connection
gyro_vs_topcon.m	Comparison of accumulated yaw data from gyroscope vs TopCon Total Station	runlog.mat file from Jetracer gyro data, Messung_*.mat file for Topcon reference measurements, Rundkurs.mat file for Reference track geometry digital map of track, AlphaBot Prism tracking to track prism.	Jetracer, Wifi TCP connection, 2 Windows PC , Topcon Toatl station and Prism
python server files folder	Copy all the python server files from the python server files folder and paste them on Jetracer JetPack OS. The python server files are needed for the JetPack OS to perform tasks such as TCP communication, motor controller actions, gyroscope data processing and sending, Lidar data communication and others. They are responsible for Jetracer Jetson Nano side functionality.	JetPack OS	HDMI Monitor , Keyboard, Mouse, Jetracer Jetson Nano

List of open questions

• The video has images sized 360px x 480px. This does not fit he calibration parameters cameraParams.Intrinsics.ImageSize (480 x 640). This has to match.

Links

• JetRacer ROS Kit
• Waveshare-Wiki: JetRacer ROS AI Kit

---

→ zurück zum Hauptartikel: Studentische Arbeiten