Evaluate a Pose Estimation Model with COCO Keypoint mAP¶
Pose models are scored on COCO with keypoint mAP. Instead of box IoU, a prediction is matched to a person by Object Keypoint Similarity (OKS): a 0 to 1 score of how close each predicted keypoint lands to the labelled one, relative to the person's size and to how precisely that keypoint can be placed. This cookbook runs RF-DETR Keypoint on a slice of COCO val2017 and scores it with KeyPointMeanAveragePrecision. Along the way it shows two mistakes that can change the score without raising an error: scoring only high-confidence predictions, and mismatched class ids.
A GPU runtime is recommended (Runtime → Change runtime type). The stored outputs were produced on a Mac with Apple silicon (MPS), and the notebook has not been run on Colab. CPU should also work, but it is untested here and the RF-DETR Keypoint checkpoint is large, so 200 images take noticeably longer.
Install required packages¶
supervision[metrics]: keypoint containers, annotators andKeyPointMeanAveragePrecision. Version0.31.7or newer: in earlier versionssv.KeyPoints.empty()holdsNonefields, which breaks the confidence filter used below on frames without poses. Themetricsextra adds pandas forto_pandas.rfdetr: providesRFDETRKeypointPreview, a COCO-pretrained 17-keypoint person model. It is a preview release whose API and weights may change, so the version is pinned.
%pip install -q "supervision[metrics]>=0.31.7" "rfdetr==1.11.2"
import dataclasses
import hashlib
import json
import random
import urllib.request
import zipfile
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
import matplotlib.pyplot as plt
import numpy as np
from PIL import Image
import supervision as sv
from rfdetr import RFDETRKeypointPreview
from supervision.metrics import KeyPointMeanAveragePrecision
# tight layout for every figure, including the ones sv.plot_images_grid creates
plt.rcParams["figure.autolayout"] = True
print(sv.__version__)
0.31.7
Load COCO keypoint annotations¶
We download the official COCO 2017 annotations (about 241 MB) and extract only person_keypoints_val2017.json. To keep the run short, we sample 200 of the 2,693 val2017 images that contain at least one annotated person, with a fixed seed, so every run scores the same images. Because images without people are left out and the slice is small, the score below is a worked example, not a replacement for the full COCO benchmark. The seed fixes only the image sample: model outputs are not seeded, so scores can differ slightly on other devices and library versions. The numbers below come from a Mac with Apple silicon (MPS).
ANNOTATIONS_URL = "http://images.cocodataset.org/annotations/annotations_trainval2017.zip"
# the host's https certificate does not match its name, so verify the plain-http download instead
ANNOTATIONS_SHA256 = "113a836d90195ee1f884e704da6304dfaaecff1f023f49b6ca93c4aaae470268"
ANNOTATIONS_FILE = Path("person_keypoints_val2017.json")
if not ANNOTATIONS_FILE.exists():
archive_path, _ = urllib.request.urlretrieve(ANNOTATIONS_URL) # temporary file
try:
if hashlib.sha256(Path(archive_path).read_bytes()).hexdigest() != ANNOTATIONS_SHA256:
raise ValueError("the downloaded COCO annotations do not match the expected SHA-256")
with zipfile.ZipFile(archive_path) as archive:
# write under a temporary name so an interrupted run never leaves a broken JSON file
partial = ANNOTATIONS_FILE.with_suffix(".part")
partial.write_bytes(archive.read(f"annotations/{ANNOTATIONS_FILE}"))
partial.rename(ANNOTATIONS_FILE)
finally:
Path(archive_path).unlink()
coco = json.loads(ANNOTATIONS_FILE.read_text())
images = {image["id"]: image for image in coco["images"]}
annotations = {}
for annotation in coco["annotations"]:
annotations.setdefault(annotation["image_id"], []).append(annotation)
image_ids = sorted(random.Random(0).sample(sorted(annotations), 200))
people = [a for i in image_ids for a in annotations[i]]
print(f"{len(image_ids)} images, {len(people)} person annotations, "
f"{sum(a['num_keypoints'] == 0 for a in people)} of them without labelled keypoints")
200 images, 802 person annotations, 334 of them without labelled keypoints
IMAGE_DIR = Path("images")
IMAGE_DIR.mkdir(exist_ok=True)
IMAGE_URL_PREFIX = "http://images.cocodataset.org/"
def download(image_id: int) -> None:
"""Download one COCO image into ``IMAGE_DIR``, refusing any other destination or host."""
image = images[image_id]
path = IMAGE_DIR / Path(image["file_name"]).name # keep only the file name from the annotation
inside_image_dir = path.resolve().parent == IMAGE_DIR.resolve()
if not inside_image_dir or not image["coco_url"].startswith(IMAGE_URL_PREFIX):
raise ValueError(f"unexpected image location for {image['file_name']}")
if not path.exists():
# download under a temporary name so an interrupted run never leaves a broken image
partial = path.with_suffix(".part")
urllib.request.urlretrieve(image["coco_url"], partial)
partial.rename(path)
with ThreadPoolExecutor(max_workers=16) as pool:
list(pool.map(download, image_ids))
Turn COCO annotations into sv.KeyPoints¶
Each COCO person carries 17 (x, y, v) keypoints. We keep v > 0 as visible, since COCO evaluation counts occluded (v=1) and visible (v=2) points alike. Three extra fields bring the score in line with pycocotools:
data["area"]: the annotated area, which scales OKS and sets the size buckets (medium: an area of 32² to 96² pixels, large: above 96²).data["iscrowd"]: crowd annotations, where one label covers a group of people. They act as ignore regions: predictions that match them are not counted as false positives.data["xyxy"]: the person box. Many COCO people have no labelled keypoints (counted above). With a box, such a person is an ignore region, as inpycocotools. Without one it is dropped, and predictions on it count as false positives.
See the pose estimation section of the metric docs for the full rules.
The class id is set to 1 because the COCO-pretrained RF-DETR model reports people with the COCO category id 1, not 0. Predictions and targets are grouped by class, so the two must agree.
# COCO person keypoints: 17 points per person, one class (category id 1)
def coco_to_keypoints(person_annotations: list[dict]) -> sv.KeyPoints:
keypoints = np.array([a["keypoints"] for a in person_annotations], dtype=np.float32)
keypoints = keypoints.reshape(-1, 17, 3)
boxes = np.array([a["bbox"] for a in person_annotations], dtype=np.float32)
return sv.KeyPoints(
xy=keypoints[..., :2],
visible=keypoints[..., 2] > 0,
class_id=np.ones(len(keypoints), dtype=int), # must match the model's class id
data={
# float64: float32 rounding can put an area near 32² or 96² into two size buckets
"area": np.array([a["area"] for a in person_annotations], dtype=np.float64),
"iscrowd": np.array([a["iscrowd"] for a in person_annotations], dtype=bool),
"xyxy": sv.xywh_to_xyxy(boxes), # COCO boxes are xywh
},
)
targets = [coco_to_keypoints(annotations[image_id]) for image_id in image_ids]
Run RF-DETR Keypoint¶
model.predict returns an sv.KeyPoints with per-person detection_confidence, per-point keypoint_confidence and the person box in data["xyxy"]. Its threshold drops low-scoring people, and the default 0.5 suits visualization. For mAP we keep almost everything (0.01): the metric ranks predictions by detection_confidence and needs the low-confidence tail to trace the full precision-recall curve. That leaves many candidates per image, but the metric uses only the 20 highest-scoring ones, as COCO does.
When you run it, loading the model prints a few rf-detr INFO and WARNING lines about weights and classes; they are expected for this preview checkpoint.
model = RFDETRKeypointPreview()
def load_image(image_id: int) -> np.ndarray:
return np.array(Image.open(IMAGE_DIR / images[image_id]["file_name"]).convert("RGB"))
predictions = [model.predict(load_image(image_id), threshold=0.01) for image_id in image_ids]
print(f"{sum(len(p) for p in predictions)} predicted people")
17177 predicted people
Here are two of the images, with the ground truth on the left and the confident predictions (detection_confidence > 0.5) on the right. We show the two images with the most people who have at least 12 labelled keypoints, so there are near-complete skeletons to compare:
edge_annotator = sv.EdgeAnnotator(color=sv.Color.GREEN, thickness=3)
vertex_annotator = sv.VertexAnnotator(color=sv.Color.RED, radius=4)
# the two images with the most nearly complete skeletons
showcase = sorted(range(len(image_ids)), key=lambda i: -(targets[i].visible.sum(axis=1) >= 12).sum())[:2]
for index in showcase:
# annotators and plot_images_grid accept PIL images, so no BGR channel handling is needed
image = Image.fromarray(load_image(image_ids[index]))
confident = predictions[index][predictions[index].detection_confidence > 0.5]
labelled = targets[index][targets[index].visible.any(axis=1)]
panels = []
for key_points in (labelled, confident):
panel = edge_annotator.annotate(image.copy(), key_points)
panels.append(vertex_annotator.annotate(panel, key_points))
titles = [f"ground truth: {len(labelled)} people", f"RF-DETR, confidence > 0.5: {len(confident)} people"]
sv.plot_images_grid(panels, grid_size=(1, 2), titles=titles, size=(12, 5))
Compute keypoint mAP¶
KeyPointMeanAveragePrecision follows COCO keypoint evaluation: OKS thresholds from 0.50 to 0.95, up to 20 predictions per image, and medium and large buckets only, since COCO keypoint labels cover only people of at least 32² pixels. With 17 keypoints it uses the COCO sigmas, the per-keypoint tolerances in OKS, by default.
result = KeyPointMeanAveragePrecision().update(predictions, targets).compute()
print(result)
Average Precision (AP) @[ OKS=0.50:0.95 | area= all | maxDets= 20 ] = 0.729 Average Precision (AP) @[ OKS=0.50 | area= all | maxDets= 20 ] = 0.916 Average Precision (AP) @[ OKS=0.75 | area= all | maxDets= 20 ] = 0.800 Average Precision (AP) @[ OKS=0.50:0.95 | area=medium | maxDets= 20 ] = 0.664 Average Precision (AP) @[ OKS=0.50:0.95 | area= large | maxDets= 20 ] = 0.820
The chart compares the overall scores with the medium and large size buckets:
# result.plot() draws a fixed 10x6 inch figure, so plot the same scores at a smaller size
scores = result.to_pandas().iloc[0].rename(lambda name: name.replace("_objects", ""))
ax = scores.plot.bar(figsize=(6, 3.5), ylim=(0, 1), ylabel="mAP", title="Keypoint mAP")
ax.set_axisbelow(True) # draw the grid behind the bars
ax.grid(axis="y")
Large people score higher than medium ones (about 0.82 against 0.66 mAP50:95), because OKS tolerates fewer pixels of error on a small body. The same scores are in the table below.
result.to_pandas()
| mAP@50:95 | mAP@50 | mAP@75 | medium_objects_mAP@50:95 | medium_objects_mAP@50 | medium_objects_mAP@75 | large_objects_mAP@50:95 | large_objects_mAP@50 | large_objects_mAP@75 | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | 0.728627 | 0.916205 | 0.800431 | 0.664166 | 0.881373 | 0.741075 | 0.820041 | 0.960664 | 0.881868 |
Two mistakes that change the score silently¶
Neither mistake below raises a warning: the first returns a lower but plausible score, the second returns 0 for a model that works. Since the predictions are already computed, we can show each one without running the model again.
def keypoint_map(predictions: list[sv.KeyPoints], targets: list[sv.KeyPoints]) -> float:
return KeyPointMeanAveragePrecision().update(predictions, targets).compute().map50_95
# 1. Scoring only the predictions that survive the display threshold
displayed = [p[p.detection_confidence > 0.5] for p in predictions]
print(f"threshold 0.01: mAP50-95 = {result.map50_95:.3f}")
print(f"threshold 0.5: mAP50-95 = {keypoint_map(displayed, targets):.3f}")
# 2. Targets labelled with class 0 while the model reports COCO id 1
shifted = [dataclasses.replace(t, class_id=np.zeros_like(t.class_id)) for t in targets]
print(f"class ids 1 vs 0: mAP50-95 = {keypoint_map(predictions, shifted):.3f}")
threshold 0.01: mAP50-95 = 0.729 threshold 0.5: mAP50-95 = 0.658 class ids 1 vs 0: mAP50-95 = 0.000
If your labels and model disagree on ids, remap one side, or pass class_agnostic=True to KeyPointMeanAveragePrecision when there is a single class.
Using your own data¶
- A different skeleton. For anything other than the 17 COCO keypoints, pass one OKS sigma per keypoint:
KeyPointMeanAveragePrecision(sigmas=...). Predictions and targets must also list keypoints in the same order. - Images without people. Pass
sv.KeyPoints.empty()as the target, so predictions on those images still count as false positives (coco_to_keypointsabove expects at least one person). This cookbook samples only images with people, which is one reason its score differs from the full benchmark. - No annotated area. Without
data["area"]the box spanning a target's visible keypoints is used, so scores may differ from COCO's. - Object size of predictions. A prediction falls into the medium or large bucket by its box in
data["xyxy"]when present, otherwise by the box spanning its keypoints. This matchespycocotoolswith and without a resultbbox. - Other models. Any source that can be turned into
sv.KeyPointsworks the same way, for examplesv.KeyPoints.from_ultralytics(...)for YOLO pose models. Check which class ids each source uses.
Conclusion¶
Keypoint mAP needs only two lists of sv.KeyPoints, but COCO-comparable numbers depend on keeping low-confidence predictions, matching class ids, and passing the annotated area, crowd flags and boxes. For the complete rules, see the KeyPointMeanAveragePrecision docs; for drawing skeletons, see the keypoint annotators.