Understanding KITTI dataset Part 2: 3D labels for objects.

In Part-1 we introduced KITTI dataset briefly and went into details on projection matrices with a hands-on example of transforming point clouds to camera coordinate system.

In this part of the series we will get into the details of 3d object annotations in KITTI dataset used for training and evaluating models for 3d object detection.

The labels for the KITTI object dataset can be downloaded from https://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d just like we downloaded projection matrices , LiDAR and image data in the previous blog.

Each label file is a txtfile wherein each line represents one object instance. The object in annotated in 3D along with some other attributes. The format for object instance can be found in Figure-1.

Figure-1: Annotation format for one 3d object.

To verify this let’s try reading one of the label files-

label_file = Path(data_root) / "label_2/000120.txt"
labels = open(label_file, "r").read()
labels = labels.split("\n")
print(labels)

['Car 0.06 1 2.04 57.05 202.42 394.68 374.00 1.42 1.67 3.80 -3.91 1.82 8.06 1.60',
 'Car 1.00 0 2.56 0.00 210.49 119.82 374.00 1.55 1.71 4.50 -4.09 1.78 2.42 1.57',
 'Car 0.00 0 1.25 814.15 164.79 997.27 262.07 1.50 1.71 4.44 5.18 1.40 13.43 1.61',
 'Car 0.00 1 1.86 346.63 185.82 485.85 278.21 1.54 1.63 3.59 -3.68 1.82 14.17 1.61',
 'Car 0.00 1 1.76 455.26 186.71 532.19 243.46 1.42 1.46 3.63 -3.24 1.84 20.44 1.61',
 'Car 0.00 1 1.69 486.97 186.14 549.32 232.01 1.43 1.66 4.19 -3.19 1.93 25.45 1.57',
 'Car 0.00 0 -1.50 664.25 178.14 721.96 227.83 1.58 1.71 3.75 2.91 1.78 24.95 -1.39',
 'Car 0.00 2 -0.26 785.76 164.26 985.11 229.86 1.49 1.57 4.26 6.55 1.33 17.60 0.09',
 'Car 0.00 1 -1.22 679.78 180.23 745.05 220.33 1.51 1.58 3.31 4.17 1.82 29.24 -1.09',
 'Car 0.00 1 -1.00 716.11 175.23 783.60 212.42 1.61 1.60 3.11 6.40 1.73 33.03 -0.82',
 'DontCare -1 -1 -10 528.31 185.44 570.02 209.44 -1 -1 -1 -1000 -1000 -1000 -10',
 '']

We will parse each object instance, i.e. one line in label file into an object for ease. Let’s write a simple class ObjectLabel3D

class ObjectLabel3D:
    def __init__(self, line: str):
        """
        Kitti Label: [category truncation occlusion alpha x0 y0 x1 y1 h w l x y z yaw]
        """
        self.values = line.strip().split(" ")
        self.category = self.values[0]
        self.truncation = float(self.values[1])
        self.occlusion = float(self.values[2])
        self.alpha = float(self.values[3])
        # Pixel cordinate system
        self.x0 = float(self.values[4])
        self.y0 = float(self.values[5])
        self.x1 = float(self.values[6])
        self.y1 = float(self.values[7])
        # Camera cordinate system
        self.h, self.w, self.l = float(self.values[8]), float(self.values[9]), float(self.values[10])
        self.x, self.y, self.z = float(self.values[11]), float(self.values[12]), float(self.values[13])
        self.yaw = float(self.values[14])

    def get_2d_bbox(self, format="xyxy"):
        if format == "xyxy":
            return np.array([self.x0, self.y0, self.x1, self.y1])
        elif format == "xywh":
            return np.array([self.x0, self.y0, self.x1 - self.x0, self.y1 - self.y0])
        else:
            raise ValueError("Bbox format not supported.")

    def get_3d_bbox(self, coordinate="camera"):
        """
        Retrieve 3d bounding box as 8 corners in 3d space. See Figure-2 for 
        for details on how the corners are labelled and how the rotation matrix is used.
        """
        # In camera coordinate the z-axis is the depth axis and y-axis is the height axis(pointing down).
        # Width is along x-axis
        if coordinate == "camera":
            # Rotate in x-z plane since y is the vertical axis for camera coordinate system.
            # Right handed coordinate system, rotate in counter-clockwise, x->y, y->z, z->x
            # and the angle is measured from x instead of z.
            rot_matrix = np.array(
                                 [
                                    [np.cos(self.yaw), 0, np.sin(self.yaw)],
                                    [0, 1, 0],
                                    [-np.sin(self.yaw), 0, np.cos(self.yaw)]
                                  ]
                                 )
            # coordinates of 8 corners of a bbox at origin(homogenous)
            box_corners_origin = np.array([
                                            [-self.l/2, -self.l/2, self.l/2, self.l/2, -self.l/2, -self.l/2, self.l/2, self.l/2],
                                            [     0,         0,        0,        0,     -self.h,   -self.h,   -self.h,  -self.h],
                                            [self.w/2, -self.w/2, -self.w/2, self.w/2, self.w/2, -self.w/2, -self.w/2, self.w/2],
                                            [     1,         1,        1,        1,        1,         1,         1,        1   ]
                                         ])
            transform = np.concatenate((rot_matrix, np.array([[self.x], [self.y], [self.z]])), axis=1)
            # Rotated box with center now translated to (x,y,z) from (0,0,0)
            box_corners = np.matmul(transform, box_corners_origin)
        elif coordinate == "velodyne":
            pass

        return box_corners

Figure-2: (a) Bounding box in camera coordinate frame with dimension (l, h, w). (b) Top view of xz-plane with yaw angle theta.

The get_3d_bbox method returns rotated box-corners in 3D according the orientation of the bbox from label file. We can then use the coordinates of these 8 corners to create a projection to the image plane and plot these to visual verification and intuition. See Part-1 for details on how this projection works.

The following script show cases an end-to-end example of process.

[embed]

References

[2]: Vision meets Robotics: The KITTI Dataset, Andreas Geiger and Philip Lenz and Christoph Stiller and Raquel Urtasun, https://www.cvlibs.net/publications/Geiger2013IJRR.pdf