Characteristics of MOT Evaluation Metrics

MOT evaluation metrics should exhibit two critical properties:

1. Address Five Types of Errors in MOT: The five error types are False Negatives (FN), False Positives (FP), Fragmentation, Mergers, and Deviation.

   False Negative: Occurs when ground truth exists but is not detected.

   False Positive: Arises when a tracker predicts an object that is not present in the ground truth.

   Merge or ID Switch: Happens when the tracks of two or more objects are confused as they come close together.

   Deviation: Occurs when a tracked object is reinitialized under a different ID.

   Fragmentation: Occurs when tracking suddenly halts even though the ground truth still exists.

2. Monotonicity and Differentiability: The metrics should provide insights into the tracker's performance concerning each of the five error types.

Commonly Used MOT Metrics

• Track-mAP
• Multi-Object Tracking Accuracy (MOTA)
• Multi-Object Tracking Precision (MOTP)
• IDF1
• Higher-Order Tracking Accuracy (HOTA)

Understanding these various metrics is crucial, as the selection of evaluation metrics significantly impacts the insights drawn from the results. Comprehending each metric aids in identifying how different errors affect the overall score, guiding improvements in MOT performance and future research directions.

# Track-mAP

Track-mAP (mean average precision) evaluates predictions against ground truth at the trajectory level. It requires a trajectory similarity score, Str, and a threshold ?tr, where trajectories are matched only if their similarity score exceeds this threshold.

Str and ?tr

Str is calculated by summing the spatial intersections of bounding boxes across all trajectories, divided by the total spatial union of the boxes throughout the trajectories.

A prTraj is matched with a gtTraj based on the highest confidence score among all prTrajs, given that Str exceeds ?tr.

• TPTr: A True Positive trajectory is when a prTraj is matched with a gtTraj.
• FPTr: A False Positive trajectory consists of remaining prTrajs that do not match with any gtTraj.

PrTrajs are ranked by descending confidence scores, enabling the calculation of precision (Prn) and recall (Ren) for each rank.

The precision values are then interpolated (InterpPrn) to ensure a monotonic decrease.

The Track mAP score is derived from the area under the interpolated precision-recall curve plotted with InterpPrn against Ren for all ranks.

Track mAP performs both matching and association at the trajectory level, with a bias towards measuring association. It is non-monotonic in detection.

Challenges with Track mAP

• The interpretation of tracking outputs using Track mAP can be complex and difficult to visualize, as it often results in overlapping outputs, some of which may have low confidence scores. Thus, the final score of each trajectory may be obscured by implicit confidence rankings.
• The threshold of 0.5 for a trajectory to be considered a positive match is relatively high, causing the metric to overlook many improvements in detection, association, and localization. Consequently, even with optimal tracking, over half of the best predictions may be counted as errors in Track-mAP.
• Track-mAP assesses trajectory matches that combine association, detection, and localization in such a way that it becomes challenging to differentiate between error types.

# Multi-Object Tracking Accuracy: MOTA

MOTA stands out as the most representative metric closely aligned with human visual assessment.

In MOTA, matching occurs at the detection level. A one-to-one mapping is formed between prDets (predicted detections) and gtDets (ground truth detections) in each frame based on spatial similarity to calculate True Positives (TP), False Positives (FP), and False Negatives (FN).

MOTA also measures Identity Switch (IDSW), which takes place when a tracker erroneously switches object identities or when a track is lost and reinitialized under a new identity.

MOTA quantifies three tracking error types: False Positive (FP), False Negative (FN), and ID Switch (IDSW).

MOTA does not account for localization errors, and detection performance tends to overshadow association performance.

# Multi-Object Tracking Precision: MOTP

MOTP measures localization accuracy by averaging the overlap between all correctly matched predictions and their corresponding ground truths.

It computes the average similarity score, S, among True Positives (TP) by matching prDets with gtDets that exceed a similarity threshold (S ? ?).

MOTP primarily quantifies localization accuracy and provides limited insights into the tracker's performance.

MOTP and MOTA metrics capture intuitive characteristics of tracking systems, such as precision in localization, object recognition accuracy, threshold configuration, and consistent tracking over time.

# Identification Metrics: IDF1

IDF1 focuses on association accuracy over detection and is employed as a secondary metric on the MOTChallenge benchmark due to its emphasis on measuring association rather than detection.

IDF1 establishes a one-to-one mapping between gtTrajs and prTrajs to identify present trajectories, in contrast to MOTA, which matches at the detection level for temporal association.

IDF1 utilizes IDTPs (Identity True Positives), where prID corresponds with grID when S ? ? of trajectories. IDF1 is calculated as the ratio of correctly identified detections over the average count of ground truth and computed detections. The Hungarian algorithm is utilized to minimize the sum of IDFP and IDFN.

IDF1 combines IDP (ID Precision) and IDR (ID Recall).

IDFN (Identity False Negative) signifies remaining gtID that does not match with prID, while IDFP (Identity False Positive) indicates prID trajectories that lack matches with any gtID.

A high IDF1 score suggests a greater count of unique objects in a scene rather than providing insights into effective detection or association. It also does not evaluate the localization accuracy of trackers.

# High Order Tracking Accuracy: HOTA

HOTA is a unified metric that explicitly evaluates various tracking aspects, including accurate detection, association, and localization.

All evaluation metrics, including MOTA, IDF1, and HOTA, utilize bijective matching between gtDets and prDets using the Jaccard Index or IOU score, which assesses spatial similarity. Any extraneous or missed predictions incur penalties.

A bijective mapping is established between all pairs of gtDet and prDet using the Hungarian algorithm, optimizing the sum of matching scores.

Detection

Detection Accuracy (DetA) signifies the percentage of aligned detections, with a detection classified as TP when Loc-IoU > 0.5 between ground truth detection (grDet) and predicted detection (prDet).

The Hungarian algorithm aids in establishing one-to-one matches, especially when a predicted detection overlaps with multiple ground truths and vice versa.

Association

Association Accuracy (AssA) measures the average alignment across matched trajectories, averaged over all detections.

Localization

Localization Accuracy is the average Loc-IoU calculated over all pairs of matching predicted and ground truth detections across the dataset.

HOTA dissects into a set of sub-metrics, allowing for the independent evaluation of various tracking aspects to provide insights into the specific errors trackers commit. Understanding these error types enhances the ability to fine-tune trackers according to specific use cases.

HOTA categorizes tracking errors into three types:

1. Detection error occurs when a tracker predicts detections that are absent from the ground truth or fails to identify existing ground truth detections. These errors are further divided into detection recall (measured by FNs) and detection precision (measured by FPs).
2. Association error arises when trackers assign the same prID to detections with differing gtIDs or assign distinct prIDs to detections that should share the same gtID. Association errors are categorized into association recall (measured by FNAs) and association precision (measured by FPAs).
3. Localization error happens when prDets do not align spatially with gtDets.

HOTA incorporates localization accuracy into tracking results, which is absent in either MOTA or IDF1.

MOTA emphasizes both matching and association scoring at a local detection level, focusing on detection accuracy, while IDF1 operates at a trajectory level, stressing the significance of association.

Track-mAP is akin to IDF1 in that it performs both matching and association at the trajectory level, with an inclination towards measuring association.

HOTA strikes a balance between detection and association, serving as an explicit amalgamation of detection and association scores by executing matches at the detection level while scoring association comprehensively across trajectories.

Conclusion:

Metrics such as IDF1, MOTA, and MOTP consolidate performance into a single figure for comparison, whereas metrics like HOTA deliver detailed insights into the algorithm's errors, which can be vital for performance enhancement. Selecting the appropriate metrics hinges on the specific use case at hand and will significantly shape future improvement strategies.

References:

HOTA: A Higher-Order Metric for Evaluating Multi-Object Tracking

HOTA (Higher Order Tracking Accuracy) is a novel metric for evaluating multi-object tracking (MOT) performance. It is…

autonomousvision.github.io

MOT16: A Benchmark for Multi-Object Tracking Evaluating Multiple Object Tracking Performance: The CLEAR MOT Metrics Evaluating Multi-Object Tracking An Introduction to Object Tracking ByteTrack: A Simple Yet Effective Multi-Object Tracking Technique