Double DQN on TurtleBot 2(ROS)

Double DQN on TurtleBot 2(ROS)

This post is a tutorial on how to set up and run a Double DQN(DDQN) algorithm on ROS based robot TurtleBot 2. By the end, you will have a working environment where you can train the TurtleBot 2 robot using the DDQN algorithm. You will be able to visualize the movements of the robot in a 3D simulation tool, Gazebo.

Visualization of operation of the robot in the 3D simulation tool, Gazebo

This post assumes that you are familiar with ROS and ROS packages, and if this is not the case, I highly recommend going through them before starting off(helpful link: ROS-Creating a workspace). However, with basic knowledge of Linux, python and ROS package, you should still be able to follow this tutorial. I will be using my repository for code scripts and I will not be discussing the scripts here as it would increase the length of this tutorial. In the case of doubts, you can get in touch with me over Linkedin.

Here are some important keywords:

1. TurtleBot2: It is the robot that we will simulate and use for training (Link).
2. Openai_ros package: ROS uses packages that contain libraries, configuration files, etc to run ROS projects. opeai_ros package is a package for ROS that helps us use OpenAI architecture based environments in our projects. If you do not know what OpenAI is, I strongly suggest you go through it here and here
3. Gazebo: A 3D simulation tool for ROS robots, Link.

First, you need Ubuntu 18.04, which can be downloaded from here. You can do a dual boot with help of this link.

The next step is to install Anaconda for Linux(Debian), which will be used to create a Conda environment for installing all the python packages needed for running the simulation. To install Anaconda follow these links: File, Installation. Once installed, create a Conda environment, named env1, with python interpreter version 3.6. You will also need to install a bunch of libraries, the list of libraries can be found in the *readme.md* of the repository. You could also skip installing all these libraries for now and install them as the compiler asks(by throwing errors 😅) as you go ahead with setting up the rest of the things.

conda create -n env1 python=3.6
conda activate env1

Next, you need to install the Robot Operating System(ROS). You will need ROS Melodic for this tutorial. To install ROS follow this link.

Now, you need to create a workspace where all ROS projects will be stored. It is called catkin workspace. Open terminal and type:

source /opt/ros/melodic/setup.bash
mkdir -p ~/catkin_ws/src
cd ~/catkin_ws/
catkin_make
source devel/setup.bash

To make sure your workspace is properly overlayed by the setup script, make sure ROS_PACKAGE_PATH environment variable includes the directory of your catkin_ws. In the same terminal, type :

echo $ROS_PACKAGE_PATH

You should see something like below, where youruser will be replaced by your username:

/home/youruser/catkin_ws/src:/opt/ros/melodic/share

Next, you need to set up another workspace, simulation_ws where you will have the openai_ros package and other TurtleBot 2 files. To do so navigate to your home directory and open the terminal and type:

mkdir -p ~/simulation_ws/src
cd ~/simulation_ws/
catkin_make
source devel/setup.bash

Again, to ensure environment variables have the path of this directory, type:

echo $ROS_PACKAGE_PATH

You should see something like this:

/home/youruser/catkin_ws/src:/home/youruser/simulation_ws/src:/opt/ros/melodic/share

Next, download the openai_ros package, by typing:

cd ~/simulation_ws/src
git clone https://bitbucket.org/theconstructcore/openai_ros.git
cd openai_ros
git checkout version2
cd ~/simulation_ws
catkin_make
source devel/setup.bash

Next, you need to create a ROS package for the project. To do so, navigate to catkin_ws and create a package by the name fa_turtlebot. Type in terminal:

cd ~/catkin_ws/src
catkin_create_pkg fa_turtlebot std_msgs rospy roscpp
cd ~/catkin_ws
catkin_make
source ~/catkin_ws/devel/setup.bash

After this, you need to download files for training the robot. Inside fa_turtlebot, navigate to the src folder and clone files from this Github repository. Make sure that the src folder is empty, or you will get an error. Also, add permissions to the script.

cd ~/catkin_ws/src/fa_turtlebot/src
git clone https://github.com/prav2909/DoubleDQN_turtlebot2.git .
chmod +x start_training.py

The robot runs in a simulated world, this world is saved as a .world file. In the cloned repository files, you will see a custom_world_3.world file that you have to move from ~/catkin_ws/src/fa_turtlebot/src folder to /home/youruser/simulation_ws/src/turtlebot/turtlebot_gazebo/worlds. You can do this manually or type the below command in the terminal:

mv ~/catkin_ws/src/fa_turtlebot/src/custom_world_3.world ~/simulation_ws/src/turtlebot/turtlebot_gazebo/worlds

Afterwards, this world file needs to be pointed to in the launch file. You will find a launch file, start_world_maze_loop_brick.launch, among the cloned files. You should move this file to TurtleBot folder /home/youruser/simulation_ws/src/turtlebot/turtlebot_gazebo/launch. You can do this manually, or by typing below commands in the terminal.

mv ~/catkin_ws/src/fa_turtlebot/src/start_world_maze_loop_brick.launch ~/simulation_ws/src/turtlebot/turtlebot_gazebo/launch

EDIT(28/07/2021):

You will also need to move start_training.launch file to the launch folder of your projet i.e. /home/youruser/catkin_ws/src/fa_turtlebot/launch

EDIT(01/08/2021):

You will also need to move fa_turtlebot.yaml file to the launch folder of your projet i.e. /home/youruser/catkin_ws/src/fa_turtlebot/config

Alright, all the setup steps are completed and you are ready to start the training of the robot. To do so, make sure conda environment env1 is activated, which you created with python 3.6 and launch the ROS project. If you get any missing library errors, you can install them now. Most of the libraries can be installed via pip, but if a library is not available via pip then a simple google search should give you an idea about how to install them.

conda activate env1
roslaunch fa_turtlebot start_training.launch

You should see the terminal getting filled with warnings and messages as ROS is initiated. In a few seconds, you should see messages as the robot starts to train. It should look something like this:

Screenshot of how terminal will look like once training starts

You will also see a Gazebo window popup parallelly. It will show a robot roaming in the environment. By default, the script should load a pre-trained model, saved_model.pt. However, if that does not happen, then the script will create a neural network with fresh weights to run the robot. Again, I do not want to discuss the scripts here as it would drag this tutorial. In case a fresh model is created, you will see the robot performing badly in the beginning but it will start doing better after about 50–60 episodes.

Bird’s eye view of the Simulation

That’s all, thank you for reading and if you liked the tutorial, give it a clap 👏. If you have any questions or you run into some trouble while setting up things yourself, post in the comments and I will try to help if I can.

Let’s connect over Linkedin: Praveen Kumar