Deep Dive into Fundamentals of DeepSORT for Object Tracking

Object tracking aims at estimating bounding boxes and the identities of objects in videos. The main goal is to assign a consistent ID to the object(s) throughout the video, maintaining continuity in its location and appearance, even as the object moves, changes pose, or becomes occluded. It takes in a set of initial object detection, develops a visual model for the objects, and tracks the objects as they move around in a video.

This blog post is intended for readers who are new to object tracking and want to understand concepts as well as code on DeepSORT and understand the different parameters they need to know to make DeepSORT work for their use case.

Section 1: Key Components of SORT

Section 2: Track Representation

Section 3: Understanding a track’s lifecycle

Section 4: Track’s state estimation(Kalman Filter Predict Step)

Section 5: Data association between tracks and detections and track update(Kalman filter update step)

Section 6: Track State Update using Detections

Section 7: Key parameters to tune a tracker model(max_age, Mahalanobis distance, reid vs iou loss weighting, etc.

First, let’s start with some key components of SORT(Simple Online Realtime Tracking) algorithm:

Section 1: Key Components of SORT:

1. Object Detection

• SORT begins with the detection of objects in individual video frames. These detections are typically obtained from an external object detection model (e.g., YOLO, Faster R-CNN). Each detected object is represented by a bounding box with associated position information.

2. State Prediction (Kalman Filter): Kalman Filter Predict Step

• For each existing track, SORT uses a Kalman Filter to predict the next position of the object based on its current state (position and velocity). This prediction provides an estimate of where the object is expected to appear in the next frame, accommodating potential movement between frames.

3. Data Association (Matching Detections to Tracks):

• The next step is to associate the new detections in the current frame with the predicted positions of the existing tracks from the previous frame. IoU (Intersection over Union), Appearence feature similarity and Mahalanobis distance are some of the metrics used to measure the overlap between the predicted track and the detected bounding box. The Hungarian Algorithm is applied to solve the assignment problem, finding the optimal matches between tracks and detections.

4. Track Update

• Once a detection is matched with a predicted track, the Kalman filter updates the object’s state (position, velocity) using the new detection. The tracker keeps updating the object’s state as long as new detections are matched with the track in each frame.

5. Track Management

• New Track Initialization: If a detection does not match any existing tracks, a new track is initialized for that object.
• Track Deletion: If an object is not detected for a specified number of frames (determined by a parameter called max_age), the track is considered lost and removed.
• Track Activation: A track is considered active if it is consistently matched with new detections for a few consecutive frames (probation_age parameter). Tracks that do not reach this threshold are treated as tentative and are deleted if they fail to accumulate enough matches.

6. Output

• The output of SORT is the set of tracked objects across frames, each assigned a unique ID. This allows for tracking objects continuously over time, even as they move, change appearance, or experience minor occlusions.

Summary of Key Components:

1. Detection: Detect objects in each frame.
2. Prediction: Use a Kalman Filter to predict the next position of each object.
3. Association: Match detections with predicted tracks using IoU and the Hungarian Algorithm.
4. Update: Update the tracks with new detections.
5. Track Management: Handle the creation and deletion of object tracks.

Okay, so with a high level summary of SORT out of the way, lets start with defining with what is exactly a track and how is it represented:

Section 2: Track representation:

Each track is associated with a unique ID and consists of information about the object’s location, motion, and possibly appearance.

You can think of a track as a probability distribution over the object’s state. In tracking systems that use filters like the Kalman filter, a track is represented by both a mean vector and a covariance matrix, which together define a multivariate normal distribution over the object’s position, velocity, and potentially other properties (like size or aspect ratio).

A Track representation contains:

1. State Esimate and uncertainty
2. Unique ID
3. Appearance feature vector
4. Track state

import numpy as np

class TrackState:
    """
    Enumeration type for the single target track state. Newly created tracks are
    classified as `tentative` until enough evidence has been collected. Then,
    the track state is changed to `confirmed`. Tracks that are no longer alive
    are classified as `deleted` to mark them for removal from the set of active
    tracks.
    """

    Tentative = 1
    Active = 2
    Deleted = 3

class Track:
    mean : np.ndarray # Mean vector of the initial state distribution.(position, velocity, etc.).
    covariance : np.ndarray # Covariance matrix of the initial state distribution.
    track_id : int # A unique track identifier.
    feature : np.ndarray #Feature vector of the detection this track originates from. If not None this feature is added to the `features` cache.
    state : TrackState # The current track state.

# Credits: https://github.com/ZQPei/deep_sort_pytorch/blob/master/deep_sort/sort/track.py

1. Object State Estimate Vector(Mean) and Uncertainty(Distribution) (Kalman Filter):

• Mean Vector: The mean represents the most likely state of the object (e.g., its position and velocity). This is the expected or predicted value of the object’s state at a given time.

• Covariance Matrix: The covariance matrix is a mathematical representation of the uncertainty or variability in the estimated state of the object. It tells us how confident the tracking algorithm is about its estimate of the object’s state.
• Each element in the matrix indicates how uncertain the estimate is about the relationships between different components of the state vector (e.g., the correlation between the object’s position and velocity).

For example, if you have a covariance matrix for the state vector (x, y, a, h), the matrix might look like:

2. Unique Track ID:

• Each track is assigned a unique ID, which remains consistent across frames. This allows the system to distinguish between different objects in the scene, ensuring that the same object is continuously tracked without confusion.

3. Appearance Features :

• Tracks are augmented with appearance features (e.g., CNN embeddings) that describe the object’s visual characteristics. These features help re-identify the object when it reappears after an occlusion or significant movement.

4. Track Status:

Tracks can have different states, such as:

• Active: The object is being detected and tracked in the current frame.
• Lost/Tentative: The object is temporarily undetected (e.g., due to occlusion), but the track is still being maintained with predictions.
• Terminated: The object is no longer detected for a set number of frames, so the track is closed.

Okay, now that we have represented the track, let’s look into the lifecycle of a given track

Section 3: Track Lifecycle Management

A lifecycle of a track begins with initialization of a new track when the object is first detected, through its confirmed state as a consistently tracked object, to its termination when it’s no longer detected or relevant

Track Lifecycle. Credits: Nvidia Deepstream library

class TrackState:
    """
    Enumeration type for the single target track state. Newly created tracks are
    classified as `tentative` until enough evidence has been collected. Then,
    the track state is changed to `confirmed`. Tracks that are no longer alive
    are classified as `deleted` to mark them for removal from the set of active
    tracks.

    """

    Tentative = 1
    Active = 2
    Deleted = 3

1. Tenative Track State:

When a new object is detected, there’s a chance it could be a false positive. To minimize noise in detections, a newly detected object is monitored for a period before being activated for long-term tracking. Specifically, when a new object is detected, a tracker is created for it, but the object is initially placed in a Tentative mode. This probationary period, defined by a parameter called probation_age, serves as a buffer to validate the detection. During this time, the tracker output is not reported downstream, as the object is not yet confirmed. However, the unreported data (from past frames) is retained by the tracker for future use if the object is validated.

Let’s another variable called track_age to the Track class defined above

class TrackParameters:
    PROBATION_AGE: 100

class Track:
    mean : np.ndarray # Mean vector of the initial state distribution.(position, velocity, etc.).
    covariance : np.ndarray # Covariance matrix of the initial state distribution.
    track_id : int # A unique track identifier.
    feature : np.ndarray #Feature vector of the detection this track originates from. 
    state : TrackState # The current track state.
    """
    New variable to account for track age
    """
    track_age = 0 #Track age


def tentative_to_active_track_state_transition(self, track: Track):
    track.track_age += 1

     if track.state == TrackState.Tentative and track.track_age >= TrackParameters.PROBATION_AGE:
         track.state = TrackState.Active

2. Active Track State:

Once the track receives sufficient detections in consecutive frames, it is upgraded to the Active state. This means the system is confident that the object is being consistently detected and tracked.

Behavior: The track continues to be updated with new detections in each frame. In the active state, the track is more robust, and even if a detection is missed for a few frames (due to occlusion or other factors), the system will continue to predict its position based on a motion model (e.g., a Kalman filter).

Tenative → Active State Transition:

When a newly initialized track is seen for N(probation_age) consecutive number of frames(track_age), the track state transitions from tentative to active:

i.e. track.track_age > probation_age

3. Inactive Track State / Shadow Tracking:

• Description: If no detection is associated with an active track for a certain number of consecutive frames, the track is considered to be "missed." The track's position is predicted, but it is not updated with new detections.
• Behavior: During this phase, the tracker continues predicting the object’s position based on its past state using a motion model(shadow tracking), allowing the system to recover the object if it reappears after occlusion.

Shadow Tracking refers to the process of continuing to track an object that is in an uncertain or temporary state, without actively reporting its position to downstream systems until sufficient evidence has been gathered to confirm its status. In this phase, the object is typically placed in a probationary or tentative state, where it is tracked internally by the system but not yet considered valid or fully confirmed.

This technique helps in:

1. Handling False Positives: It allows the system to suppress potentially noisy detections by not immediately acting on them.
2. Maintaining Continuity: If an object reappears or becomes more certain after some frames, the stored tracking information can be reactivated for reporting.
3. Occlusion Management: It helps maintain a track through brief occlusions or ambiguous detections, ensuring smoother long-term tracking of the object once it is confirmed.

In object tracking, this strategy ensures that the system avoids reporting spurious or unreliable detections to downstream processes until the object’s state is validated over a period of frames.

Let’s update the code to accommodate logic for shadow tracking

class TrackParameters:
    PROBATION_AGE: 100 
    MAX_AGE: 100 
    MIN_TRACK_CONFIDENCE_FOR_DETECTION_MATCH: 100 

class Track:
    mean : np.ndarray # Mean vector of the initial state distribution.(position, velocity, etc.).
    covariance : np.ndarray # Covariance matrix of the initial state distribution.
    track_id : int # A unique track identifier.
    feature : np.ndarray #Feature vector of the detection this track originates from. 
    state : TrackState # The current track state.
    track_age: int #Track age

    """
    New variable to account for track confidence
    """
    track_confidence: int #Track Confidence
    shadow_tracking_age: int #This track's shadow tracking age

def tentative_to_active_track_state_transition(self, track: Track):
    track.track_age += 1

     if track.state == TrackState.Tentative and track.track_age >= TrackParameters.PROBATION_AGE:
         track.state = TrackState.Active

def active_to_inactive_track_state_transition(self, track: Track):
    if track.tracker_confidence < MIN_TRACK_CONFIDENCE_FOR_DETECTION_MATCH:
        track.shadow_tracking_age += 1

Active → Inactive State Transition:

When an existing track either in tentative/active state cannot be matched to any of the existing detections, we increment its shadow tracking age and move it into shadow mode.

Note: you might be thinking why haven’t we explicitly defined an inactive state. Reason in we do need this additional state. It is implicitly modeled by the shadow tracking age. If a given track’s shadow tracking age > 0, it means it is currently inactive

4. Terminated Track State:

• Description: If the track fails to receive a detection for more than the defined number of frames (max_age), it is marked for deletion. This means the object is no longer being tracked, and the track is removed from the active tracks.
• Behavior: Once in the deleted state, the track is no longer updated or predicted, and its ID is retired. In some systems, a new track may be initialized if the object reappears.

Inactive → Terminated State Transition:

When a track is in shadow mode(does not have any associated detections for more than N(MAX_AGE) consecutive frames, it is terminated

def inactive_to_terminated_state_transition(self, track: Track):
    track.shadow_tracking_age > MAX_AGE:
    track.state = TrackState.Deleted

Here is the entire code for Track Lifecycle management:

class TrackState:
    """
    Enumeration type for the single target track state. Newly created tracks are
    classified as `tentative` until enough evidence has been collected. Then,
    the track state is changed to `confirmed`. Tracks that are no longer alive
    are classified as `deleted` to mark them for removal from the set of active
    tracks.

    """

    Tentative = 1
    Active = 2
    Deleted = 3

class TrackParameters:
    PROBATION_AGE: 100 
    MAX_AGE: 100 
    MIN_TRACK_CONFIDENCE_FOR_DETECTION_MATCH: 100 

class Track:
    mean : np.ndarray # Mean vector of the initial state distribution.(position, velocity, etc.).
    covariance : np.ndarray # Covariance matrix of the initial state distribution.
    track_id : int # A unique track identifier.
    feature : np.ndarray #Feature vector of the detection this track originates from. 
    state : TrackState # The current track state.
    track_age: int #Track age

    """
    New variable to account for track confidence
    """
    track_confidence: int #Track Confidence
    shadow_tracking_age: int #This track's shadow tracking age

def tentative_to_active_track_state_transition(self, track: Track):
    track.track_age += 1

     if track.state == TrackState.Tentative and track.track_age >= TrackParameters.PROBATION_AGE:
         track.state = TrackState.Active

def active_to_inactive_track_state_transition(self, track: Track):
    if track.tracker_confidence < MIN_TRACK_CONFIDENCE_FOR_DETECTION_MATCH:
        track.shadow_tracking_age += 1

def inactive_to_terminated_state_transition(self, track: Track):
    track.shadow_tracking_age > MAX_AGE:
    track.state = TrackState.Deleted

Phew! Yeah I know that is a lot of stuff to ingest for someone new to tracking. But, yayy! you’ve made it till here!

Next, we will go over the track’s state estimation and update.

Section 4: State Estimation

State estimation is a critical process in object tracking that involves determining the current state (position, velocity, etc.) of an object based on a series of noisy measurements or detections. This process is essential for tracking systems because sensor data is often imperfect, and objects can move unpredictably or become occluded.

A Kalman filter for tracking bounding boxes in image space is defined as follows. The 8-dimensional state space:

x, y, a, h, vx, vy, va, vh

contains the bounding box center position (x, y), aspect ratio a, height h, and their respective velocities. Object motion follows a constant velocity model. The bounding box location (x, y, a, h) is taken as direct observation of the state space (linear observation model)

Credits: https://www.npmjs.com/package/kalman-filter

Steps involved in state estimation:

1. Initialization

• State Initialization: Set up an initial guess of the object’s state (e.g., position, velocity) based on prior knowledge or a first detection. The state is represented by a state vector (e.g., [x, y, vx, vy]).
• Covariance Matrix Initialization: Initialize the covariance matrix, which reflects the uncertainty in the initial state. Larger values in this matrix indicate greater uncertainty.

2. Prediction Step

In this step, the state of the object is predicted forward in time using a motion model (e.g., constant velocity model).

• State Prediction: Use the state transition model to predict the next state of the object

import numpy as np

class KalmanFilter:
    """
    A simple Kalman Filter implementation for object tracking.

    The Kalman Filter is used to predict the state of a system and update
    the estimate of the state using measurements.

    Attributes:
    -----------
    x : ndarray
        The state estimate vector (position, velocity, etc.).
    P : ndarray
        The covariance matrix representing the uncertainty in the state estimate.
    F : ndarray
        The state transition matrix (describes how the state evolves from one time step to the next).
    Q : ndarray
        The process noise covariance matrix (represents the uncertainty in the system's dynamics).
    H : ndarray
        The measurement matrix (relates the state to the measurement).
    R : ndarray
        The measurement noise covariance matrix (represents the uncertainty in the measurement).
    """

    def __init__(self, state_dim, meas_dim):
        """
        Initialize the Kalman filter with appropriate dimensions.

        Parameters:
        -----------
        state_dim : int
            The dimension of the state vector (e.g., 4 for [x, y, vx, vy]).
        meas_dim : int
            The dimension of the measurement vector (e.g., 2 for [x, y]).
        """
        # State vector (initialized as zeros)
        self.x = np.zeros((state_dim, 1))  # State estimate

        # Covariance matrix (initialized as identity)
        self.P = np.eye(state_dim)  # Uncertainty in the state

        # State transition matrix
        self.F = np.eye(state_dim)  # How state evolves (identity if static)

        # Process noise covariance (uncertainty in the system)
        self.Q = np.eye(state_dim) * 0.01  # Tune this parameter

        # Measurement matrix (maps state space to measurement space)
        self.H = np.zeros((meas_dim, state_dim))  # Will be set based on the problem

        # Measurement noise covariance (uncertainty in the measurements)
        self.R = np.eye(meas_dim) * 0.1  # Tune this parameter

    def estimate_state(self):
        """
        Predict the state for the next time step using the state transition model.
        The prediction step estimates the new state based on the previous state.
        """
        # State prediction: x' = F * x
        self.x = np.dot(self.F, self.x)

        # Covariance prediction: P' = F * P * F^T + Q
        self.P = np.dot(np.dot(self.F, self.P), self.F.T) + self.Q

Section 5: Data Association between Tracks and Detections

Let’s revisit some definitions: Track: A track is a probability distribution over the object’s state Detection: A detection is the output of an object detection model for this given frame.

The aim is to associated the existing tracks (from timestep T-1) to the current detections of this frame(timestep T)

The association step in multi-object tracking, such as in the DeepSORT algorithm, is a critical process where detected objects in a new frame are associated with existing tracks (predicted object states from previous frames). This process ensures that each detected object maintains a consistent identity across frames.

To associated the tracks with detections, we need to perform the follwing 2 steps:

1. Calculate the Cost Matrix:

A cost matrix is generated to quantify the “cost” or “likelihood” of associating each track with each detection. Lower costs indicate better matches. The cost matrix can be computed using multiple metrics:

• Mahalanobis Distance: Measures the distance between the predicted track state (distribution) and the detection, considering the uncertainty (covariance) in the prediction. This helps account for noise and uncertainty in the object’s motion.
• IoU (Intersection over Union): Measures the overlap between the predicted bounding box of the track and the bounding box of the detection.
• Appearance Features: A CNN-based appearance model computes the similarity between the appearance embeddings of the track and the detection. This is often measured using cosine similarity or Euclidean distance.

2. Solving the Assignment Problem:

• Once the cost matrix is built, the assignment problem is solved using the Hungarian algorithm (also known as the Kuhn-Munkres algorithm). This algorithm finds the optimal association between tracks and detections by minimizing the overall cost, ensuring that each detection is assigned to the most likely track (if feasible).
• The Hungarian algorithm ensures a global optimization where the best possible matches are made, minimizing the overall sum of association costs.

Cost Matrix Computation: Mahalanobis Distance

• The Mahalanobis distance measures how far the detection is from the predicted state (mean) of the track, relative to the uncertainty (covariance).
• This distance normalizes the difference between the detection and the predicted state by the covariance (or uncertainty) of the track, making it a better metric for determining if the detection should be associated with the track.

If you recollect, a track is not a bounding box. It is a distribution of bounding box position and velocity with mean value of the distribution representing the current estimate of the track position and velocity and the covairance representing the confidence of the model in this estimate. Hence, to compute distance between a given track and a detection(which is a bounding box), we need to use Mahalanobis distance. The Mahalanobis distance is primarily used to measure the distance between a point and a distribution, taking into account the variance and correlations between variables in the data.

Cost Matrix Computation: IoU Distance

To compute IoU, we can extract the track’s bounding box is obtained from its current state estimation(mean vector):

class Track:
    mean : np.ndarray # Mean vector of the initial state distribution.(position, velocity, etc.).
    covariance : np.ndarray # Covariance matrix of the initial state distribution.
    track_id : int # A unique track identifier.
    feature : np.ndarray #Feature vector of the detection this track originates from. 
    state : TrackState # The current track state.
    track_age: int #Track age

    """
    New variable to account for track confidence
    """
    track_confidence: int #Track Confidence
    shadow_tracking_age: int #This track's shadow tracking age

def get_track_bounding_box(track): 
    ret = track.mean[:4].copy()
    ret[2] *= ret[3]
    ret[:2] -= ret[2:] / 2
    return ret

def iou_distance(tracks, detections, track_indices, detection_indices):
    """Calculate IoU distance between tracks and detections."""
    cost_matrix = np.zeros((len(track_indices), len(detection_indices)), dtype=np.float32)
    for i, track_idx in enumerate(track_indices):
        for j, detection_idx in enumerate(detection_indices):
            cost_matrix[i, j] = 1 - iou(tracks[track_idx].bbox, detections[detection_idx].bbox)
    return cost_matrix

Cost Matrix Computation: Appearance features

Benefits of Appearance Features in DeepSORT

• Improves Robustness: Appearance features add an extra layer of robustness in tracking, particularly in crowded scenes where multiple objects are close to each other.
• Handles Occlusions: Since appearance features can re-identify objects, DeepSORT is better equipped to handle occlusions or re-entries.
• Reduces Identity Switches: By using appearance features, DeepSORT minimizes the chances of switching object identities when similar objects cross paths.

Computing feature vector:

• CNN Model: In the original DeepSORT paper, a ResNet-50 architecture is used, which is pre-trained on ImageNet and then fine-tuned on the Market-1501 dataset for pedestrian re-identification.
• Feature Vector Size: The network outputs a 128-dimensional appearance embedding for each detected object.
• Training: For re-identification, the CNN is trained to minimize the distance between the appearance embeddings of the same object across different frames and maximize the distance between different objects.

Associating tracks with detections:

def associate_tracks_with_detections(tracks, detections, kalman_filter, appearance_model):
    # Step 1: Predict new track positions using the Kalman filter
    predicted_tracks = [kalman_filter.predict(track) for track in tracks]

    # Step 2: Compute cost matrix using Mahalanobis distance, IoU, and appearance features
    cost_matrix = compute_cost_matrix(predicted_tracks, detections, appearance_model)

    # Step 3: Apply gating to eliminate unlikely matches
    gated_cost_matrix = gate_cost_matrix(kalman_filter, cost_matrix, predicted_tracks, detections)

    # Step 4: Solve the assignment problem using the Hungarian algorithm
    matches, unmatched_tracks, unmatched_detections = hungarian_algorithm(gated_cost_matrix)

    # Step 5: Update matched tracks with new detections, handle unmatched
    for match in matches:
        track_id, detection_id = match
        tracks[track_id].update(detections[detection_id])

    handle_unmatched_tracks(unmatched_tracks)
    initialize_new_tracks(unmatched_detections)

    return matches, unmatched_tracks, unmatched_detections

Detailed code on cost matrix computation

import numpy as np
from scipy.spatial.distance import cdist

def compute_cost_matrix(tracks, detections, metric, kf, alpha=1.0, beta=1.0):
    """
    Compute the cost matrix for associating tracks with detections using
    Mahalanobis distance and appearance features.

    Parameters
    ----------
    tracks : List[Track]
        List of active Track objects.
    detections : List[Detection]
        List of Detection objects in the current frame.
    metric : AppearanceMetric
        An instance of AppearanceMetric to compute appearance distances.
    kf : KalmanFilter
        An instance of KalmanFilter used for computing Mahalanobis distances.
    alpha : float, optional
        Weight for the Mahalanobis distance component (default is 1.0).
    beta : float, optional
        Weight for the appearance distance component (default is 1.0).

    Returns
    -------
    cost_matrix : ndarray
        A (N x M) matrix where N is the number of tracks and M is the number of detections.
        Each element (i, j) represents the combined cost of associating track i with detection j.
    """
    num_tracks = len(tracks)
    num_detections = len(detections)

    # Initialize cost matrices
    mahalanobis_cost = np.zeros((num_tracks, num_detections), dtype=np.float32)
    appearance_cost = np.zeros((num_tracks, num_detections), dtype=np.float32)

    # Compute Mahalanobis distance for motion cost
    for i, track in enumerate(tracks):
        for j, det in enumerate(detections):
            # Extract the state and covariance from the Kalman filter
            mean = track.mean[:4].reshape(-1, 1)  # Assuming state is [x, y, a, h, ...]
            covariance = track.covariance[:4, :4]

            # Measurement from detection (e.g., [x, y, a, h])
            measurement = det.to_xyah().reshape(-1, 1)

            # Compute the difference
            diff = measurement - mean

            # Compute Mahalanobis distance
            try:
                inv_covariance = np.linalg.inv(covariance)
                mahalanobis_distance = np.sqrt(np.dot(np.dot(diff.T, inv_covariance), diff)[0, 0])
            except np.linalg.LinAlgError:
                mahalanobis_distance = np.inf  # Assign a large distance if covariance is singular

            mahalanobis_cost[i, j] = mahalanobis_distance
            iou_cost[i, j] = 1 - iou(track.bbox, det.bbox)

    # Normalize Mahalanobis distance
    if mahalanobis_cost.max() > 0:
        mahalanobis_cost = mahalanobis_cost / mahalanobis_cost.max()

    # Compute appearance distance using the provided metric
    if num_tracks > 0 and num_detections > 0:
        track_features = np.array([track.features[-1] for track in tracks])
        detection_features = np.array([det.feature for det in detections])
        # Compute appearance distance matrix (e.g., cosine distance)
        appearance_cost = metric.distance(track_features, detection_features)
        # Normalize appearance distance
        if appearance_cost.max() > 0:
            appearance_cost = appearance_cost / appearance_cost.max()
    else:
        appearance_cost = np.zeros_like(mahalanobis_cost)

    # Combine the costs
    cost_matrix = alpha * mahalanobis_cost + beta * appearance_cost * gamma * iou_cost

    return cost_matrix

# Example Usage:

class AppearanceMetric:
    """
    Dummy AppearanceMetric class.
    Replace this with actual implementation, e.g., using cosine distance.
    """
    def distance(self, track_features, detection_features):
        # Example using cosine distance
        if len(track_features) == 0 or len(detection_features) == 0:
            return np.array([])
        track_norm = track_features / np.linalg.norm(track_features, axis=1, keepdims=True)
        detection_norm = detection_features / np.linalg.norm(detection_features, axis=1, keepdims=True)
        cosine_similarity = np.dot(track_norm, detection_norm.T)
        return 1 - cosine_similarity  # Convert similarity to distance

class Track:
    def __init__(self, mean, covariance, features):
        self.mean = mean
        self.covariance = covariance
        self.features = [features]  # List of feature vectors

class Detection:
    def __init__(self, bbox, feature):
        self.bbox = bbox  # [x1, y1, x2, y2]
        self.feature = feature  # Feature vector

    def to_xyah(self):
        """
        Convert bounding box to [x_center, y_center, aspect_ratio, height]
        """
        x1, y1, x2, y2 = self.bbox
        w = x2 - x1
        h = y2 - y1
        x_center = x1 + w / 2
        y_center = y1 + h / 2
        aspect_ratio = w / h
        return np.array([x_center, y_center, aspect_ratio, h])

kf = KalmanFilter()
metric = AppearanceMetric()

# Create dummy tracks and detections
tracks = [
    Track(mean=np.array([100, 100, 1.0, 50]), covariance=np.eye(4), features=np.random.rand(128)),
    Track(mean=np.array([300, 300, 1.2, 60]), covariance=np.eye(4)*2, features=np.random.rand(128))
]

detections = [
    Detection(bbox=[105, 105, 205, 205], feature=np.random.rand(128)),
    Detection(bbox=[295, 295, 395, 395], feature=np.random.rand(128)),
    Detection(bbox=[500, 500, 600, 600], feature=np.random.rand(128))
]

# Compute the cost matrix
cost_matrix = compute_cost_matrix(tracks, detections, metric, kf, alpha=1.0, beta=1.0)
print("Cost Matrix:\n", cost_matrix)

Section 6: Track State Update using Detections

We have seen how Kalman Filter’s predict(state estimation) works in Section 4:

The Kalman filter operates in two main steps:

Prediction Step: The filter uses the current state to predict the next state based on a motion model.

Update (Correction) Step: When a new measurement is available, the Kalman filter updates the predicted state based on this measurement, correcting the prediction.

Here, we will talk about the Update Step.

• The update method incorporates the new measurement (z) into the state estimate.
• First, it calculates the measurement residual (innovation) by comparing the predicted state to the actual measurement.
• Then, it computes the Kalman gain, which determines how much to trust the measurement relative to the prediction.
• The state and covariance are updated accordingly using the Kalman gain.

matches, unmatched_tracks, unmatched_detections = associate_tracks_with_detections

for track_idx, detection_idx in matches:
    # Update track with the new detection using Kalman filter
    self.tracks[track_idx].update(self.kf, detections[detection_idx])

    # Mark unmatched tracks as missed
    for track_idx in unmatched_tracks:
        # Increment time_since_update for unmatched tracks
        self.tracks[track_idx].mark_missed()

    # Initiate new tracks for unmatched detections
    for detection_idx in unmatched_detections:
        # Create new tracks for detections that were not associated with any track
        self._initiate_track(detections[detection_idx])

    # Remove deleted tracks (those that have timed out)
    self.tracks = [t for t in self.tracks if not t.is_deleted()]

    # Update appearance distance metric with new track features
    active_targets = [t.track_id for t in self.tracks if t.is_confirmed()]
    features, targets = [], []

    # Collect features and track IDs for confirmed tracks
    for track in self.tracks:
        if not track.is_confirmed():
            continue
        # Add all features of the track to the list
        features += track.features
        # Map each feature to the track's unique ID
        targets += [track.track_id for _ in track.features]
        # Clear the track's feature cache
        track.features = []

Section 7: Key parameters to tune a tracker model

Key Hyperparameters to Tune in DeepSORT:

1. Max Age

• Purpose: Determines how many frames a track can remain in memory without receiving new detections before it is considered terminated.

Optimization:

• Increase for scenes with frequent occlusion (so tracks aren’t dropped too quickly).
• Decrease for faster removal of terminated objects (prevents tracking “ghost” objects).

2. Probation Age

• Purpose: The minimum number of frames in which a detection must be consistently matched before the object is confirmed as a valid track.

Optimization:

• Increase for more stable tracking, but slower to initialize new tracks.
• Decrease for faster detection of new objects but with a higher risk of false positives.

3. IoU Threshold

• Purpose: Determines the minimum Intersection over Union (IoU) between the predicted position of the object (from the Kalman filter) and the detected bounding box to consider them a match.

Optimization:

• Increase if you want stricter matching, which will reduce the chance of assigning incorrect detections to a track (but may increase false negatives, i.e., missed matches).
• Decrease to allow more lenient matching, useful when bounding boxes fluctuate more between frames.

4. Appearance Feature Embedding Model (ReID Model)

• Purpose: A deep learning model (usually a CNN) extracts appearance features from detected objects to re-identify them across frames.

Optimization:

• Choose a well-trained model suitable for your dataset (pre-trained models on large datasets like Market1501 can be a good start).
• Fine-tune the model on your specific data to improve feature embedding.
• Feature embedding dimension: Typically 128 or 256-dimensional embeddings work well.
• Goal: Minimize appearance-based ID switches, especially in crowded scenes.

5. Appearance Feature Matching Threshold

• Purpose: This threshold controls how strict the ReID (appearance model) is in associating new detections with existing tracks based on feature similarity.

Optimization

• Increase for more accurate appearance-based matching, which reduces ID switches (but might cause tracks to be missed if appearance changes slightly).
• Decrease for more lenient matching, useful when objects undergo appearance changes or have similar visual features (but this increases the risk of mismatches).

6. Mahalanobis Distance Threshold

• Purpose: Measures the distance between the predicted Kalman filter state and the new detection, factoring in uncertainty.

Optimization:

• Increase to enforce stricter matches, reducing the risk of associating incorrect detections but potentially dropping valid tracks.
• Decrease to allow for more lenient matching, particularly useful if objects move unpredictably or undergo brief occlusions.

7. Kalman Filter Parameters (Process and Measurement Noise)

• Process Noise (Q): Adjusts how much uncertainty is assigned to the object’s state between frames.
• Higher values for fast-moving or erratically moving objects (more uncertainty).
• Lower values for smooth, predictable motion (less uncertainty).
• Measurement Noise (R): Adjusts how much uncertainty is assigned to the measurement (detected bounding box).
• Higher values to accommodate noisy detections.
• Lower values when detection accuracy is high.
• Optimization: Fine-tune these parameters to match the movement characteristics of your objects and the accuracy of your detections.

8. Detection Confidence Threshold

• Purpose: Filters detections based on confidence score before they are passed to the tracker.

Optimization:

• Increase to filter out low-confidence detections, useful for reducing false positives (but may result in missing true detections).
• Decrease to allow more detections through, especially for detecting smaller or occluded objects (but might increase false positives).

9. Non-Maximum Suppression (NMS) Threshold

• Purpose: Filters out overlapping detections, only keeping the highest confidence detection in a given area.

Optimization:

• Increase to keep more overlapping detections (useful in crowded scenes).
• Decrease to eliminate more overlapping detections (helps reduce false positives).

References:

[embed]Object Tracking and Reidentification with FairMOT Explaining and understanding the inner workings of FairMOT Tracker. Checkout the intermediate outputs, and compare the…learnopencv.com

[embed]Understanding Multiple Object Tracking using DeepSORT Surveillance cameras plays an essential role in securing our home or business. These cameras are super affordable. So…learnopencv.com

[embed]

[embed]

[embed]

[embed]GitHub - ZQPei/deep_sort_pytorch: MOT using deepsort and yolov3 with pytorch MOT using deepsort and yolov3 with pytorch . Contribute to ZQPei/deep_sort_pytorch development by creating an account…github.com

[embed]Gst-nvtracker - DeepStream documentation 6.4 documentation The Gst-nvtracker plugin allows the DS pipeline to use a low-level tracker library to track the detected objects over…docs.nvidia.com