Robotics Nano Degree
The goal of this project were to use perception and decision steps to control a rover in a simulator. Perception occurs via using computer vision techniques to determine navigable terrain and then make decisions to take Action on the rover.
Its the first project of the Robotics Nano Degree program. I ran my simulator in 1600×1200 resolution. Different resolution may impact on the performance of the model in this project.
The first step was to perform some analysis in a jupyter notebook on sample/calibration data.
Run the functions provided in the notebook on test images (first with the test data provided, next on data you have recorded). Add/modify functions to allow for color selection of obstacles and rock samples.
This step involved loading the data
and then doing a perspective transform to get a birds eye view.
color_thresh was provided to do color thresholding (defaulted to RGB channels > 160). It was used as the basis to create an
obstacle_thresh method (which selected the inverse ie RGB color channels <= 160). A
rock_thresh method was created that selected between min and max color channels. The image color channels are converted from RGB to YUV before being used via
process_image() function with the appropriate analysis steps to map pixels identifying navigable terrain, obstacles and rock samples into a worldmap. Run
process_image() on your test data using the
moviepy functions provided to create video output of your result.
1) Define source and destination points for perspective transform
dst_size = 5 bottom_offset = 6 source = np.float32([[14, 140], [301 ,140],[200, 96], [118, 96]]) destination = np.float32([[img.shape/2 - dst_size, img.shape - bottom_offset], [img.shape/2 dst_size, img.shape - bottom_offset], [img.shape/2 dst_size, img.shape - 2*dst_size - bottom_offset], [img.shape/2 - dst_size, img.shape - 2*dst_size - bottom_offset], ])
2) Apply perspective transform
a warped image is created using the source and destination points from above
warped = perspect_transform(img, source, destination)
3) Apply color threshold to identify navigable terrain/obstacles/rock samples
thresh_max values were determined via an interactive cell in the notebook.
threshed = color_thresh(warped) obstacle_threshed = obstacle_thresh(warped) warped_yuv=cv2.cvtColor(warped, cv2.COLOR_RGB2YUV) thresh_min=(0, 38, 153) thresh_max=(145, 148, 170) rock_threshed = rock_thresh(warped_yuv, thresh_min, thresh_max)
4) Convert thresholded image pixel values to rover-centric coords
xpix, ypix = rover_coords(threshed) xpix_obst, ypix_obst = rover_coords(obstacle_threshed) xpix_rock, ypix_rock = rover_coords(rock_threshed)
5) Convert rover-centric pixel values to world coords
world_size = data.worldmap.shape scale = 12 xpos = data.xpos[data.count] ypos = data.ypos[data.count] yaw = data.yaw[data.count] xpix_world, ypix_world = pix_to_world(xpix, ypix, xpos, ypos, yaw, world_size, scale) xpix_world_obst, ypix_world_obst = pix_to_world(xpix_obst, ypix_obst, xpos, ypos, yaw, world_size, scale) xpix_world_rock, ypix_world_rock = pix_to_world(xpix_rock, ypix_rock, xpos, ypos, yaw, world_size, scale)
note: data.count contains the current position in index for the video stream.
6) Update worldmap (to be displayed on right side of screen)
for obstacle_x_world, obstacle_y_world in zip (xpix_world_obst, ypix_world_obst): data.worldmap[obstacle_y_world, obstacle_x_world, 0] = 1 for rock_x_world, rock_y_world in zip (xpix_world_rock, ypix_world_rock): data.worldmap[rock_y_world, rock_x_world, 1] = 1 for navigable_x_world, navigable_y_world, in zip(xpix_world, ypix_world): data.worldmap[navigable_y_world, navigable_x_world, 2] = 1
7) Make a mosaic image
A mosaic image was created showing the rover camera image, warped image, ground truth (with rover location and direction arrow) and another ground truth (showing the current obstacle and navigable mapping)
Test video follows
Autonomous Navigation and Mapping
Fill in the perception_step() (at the bottom of the perception.py script) and decision_step() (in decision.py) functions in the autonomous mapping scripts and an explanation is provided in the writeup of how and why these functions were modified as they were.
This step utilised the efforts from the notebook analysis, described above. The Rover Worldmap was not updated if there was observable pitch or roll (eg > or – 1 degree).
In addition rover polar coordinates were derived and saved against the passed
Rover object for both navigable areas and observed rocks (if no rocks observed set to None).
This is the challenging part of the project.
forward were the two default rover modes supplied. For this project
reverse were added.
forward was modified to have a left hugging biases by adding 65% of the standard deviation of the navigable angles, as long as there had been some travel time either initially or after being stuck.
The rover enters
stuck mode if the rover stays in the same position, whilst not picking up a rock, for 5 seconds. If still stuck after 10 seconds, then
reverse mode is tried. After 15 seconds,
reverse are reset before trying
stuck mode tries rotating if there is an obstruction in front, moving forward if steering not locked full left or right whilst going slow, and breaking if steering is locked full left or right. It will reset to
forward if movement is restored.
reverse mode rotates randomly between 30 and 180 degrees after setting the brakes and reducing velocity to zero. Once its within or – 15 degrees it sets mode to
reverse mode is in-affective to sets it to
If a rock is observed, some false positives are ignored, as well as distant rocks before being placed into
rock mode. Whilst the rock is not close, it tries to navigate closer towards it before breaking or coasting closer. The algorithm still requires more refinement.
Note: All my testing and running in Autonomous mode was done at 1600×1200 resolution.