COCO (Object Detection and Segmentation)

COCO (Common Objects in Context) is the central benchmark for object detection and segmentation. Unlike classification, these tasks require locating objects within an image, drawing bounding boxes or pixel masks, and labeling each one. COCO's images show multiple objects in natural, cluttered scenes, making it a realistic and demanding test.

What It Measures

COCO supports several tasks: bounding-box detection, instance segmentation (per-object pixel masks), and keypoint detection for human pose. The dominant metric is mean average precision (mAP), averaged over object categories and over a range of intersection-over-union (IoU) thresholds from 0.5 to 0.95. This averaging rewards both finding objects and localizing them tightly. COCO also breaks down AP by object size, reporting separate numbers for small, medium, and large objects.

Methodology

Models train on a large set of annotated images and are evaluated on held-out validation or test splits. For each image, a model outputs predicted boxes or masks with confidence scores and labels. The evaluator matches predictions to ground-truth annotations using IoU overlap, then computes precision-recall curves per category and averages them. The primary AP averages across IoU thresholds (often written AP or mAP@[.5:.95]); AP50 uses a looser 0.5 threshold and AP75 a stricter 0.75. Standardized evaluation code ensures comparability across methods.

How to Interpret Results

Use the primary mAP across IoU thresholds as the headline; AP50 alone overstates quality because it tolerates loose boxes. Inspect the size breakdown: small-object AP is usually far lower and often the real bottleneck for applications like surveillance or remote sensing. For segmentation, compare mask AP separately from box AP, since tight masks are harder than boxes. Balance accuracy against inference speed, which COCO papers typically report for real-time use.

Limitations

COCO's 80 common categories are narrow relative to open-world detection, so high mAP does not imply broad recognition. Annotation choices, such as how crowds or occluded objects are labeled, affect scores. The strict localization metric penalizes useful detections with slightly loose boxes. As with other mature benchmarks, sustained tuning risks overfitting, and strong COCO numbers may not transfer to domains with different object scales, densities, or imaging conditions.