Keypoint Detection: Pose Estimation with Four Models¶
A pose estimation model finds the joints of every person in an image: nose, shoulders, elbows, wrists, hips, knees and ankles. Each joint is a keypoint, an (x, y) position with a confidence score, and the joints of one person form a skeleton.
supervision stores them in sv.KeyPoints. Every model connector returns the same object, so the drawing and analysis code you write once works for all of them:
sv.KeyPoints.from_ultralyticsfor Ultralytics YOLO pose models.sv.KeyPoints.from_transformersfor Hugging Face Transformers pose models such as ViTPose.sv.KeyPoints.from_inference,from_mediapipe,from_yolo_nasandfrom_detectron2for other frameworks.- RF-DETR needs no connector: its
predictmethod already returnssv.KeyPoints.
This cookbook runs four models on the same photo of a group of children:
- RF-DETR keypoint preview: a one-stage DETR-style model that also predicts how uncertain each joint is, as a covariance that supervision can draw.
- ViTPose-base: a plain vision transformer that estimates joints inside person boxes, so it needs a detector in front of it.
- ViTPose++-large: a larger ViTPose trained on several pose datasets, using the same code as the base model.
- YOLO26m-pose: a one-stage model that finds people and their joints in one pass.
The last section uses the new sv.ColorPalette and sv.ColorLookup support of the keypoint annotators to colour a skeleton by person, by class or by joint.
Install required packages¶
transformers runs ViTPose and the RT-DETR person detector, ultralytics runs YOLO26 and rfdetr runs RF-DETR.
! pip install -q transformers ultralytics rfdetr
Colouring keypoints with sv.ColorLookup.KEYPOINT ships in supervision 0.32.0. Until it is released, install it from the develop branch. This is a separate step with --no-deps because rfdetr and other packages pin their own supervision range.
! pip install -q --no-deps "supervision @ git+https://github.com/roboflow/supervision.git@develop"
Later sections compare models and draw results, so everything is imported in one place.
import urllib.request
import numpy as np
import torch
from PIL import Image
from scipy.optimize import linear_sum_assignment
from rfdetr import RFDETRKeypointPreview
from transformers import (
AutoProcessor,
RTDetrForObjectDetection,
VitPoseForPoseEstimation,
)
from ultralytics import YOLO
import supervision as sv
print(sv.__version__)
0.32.0.dev0
Download the sample image¶
Pose models are easiest to compare on a crowded scene, so this photo shows a group of children on a tennis court. People fill the frame, stand in two rows and overlap, which is where pose models disagree most. It comes from the COCO validation set.
image_url = "http://images.cocodataset.org/val2017/000000001000.jpg"
image_path = "tennis-group.jpg"
urllib.request.urlretrieve(image_url, image_path)
The models take a Pillow image, and supervision annotators take a BGR array, so the photo is kept in both forms.
pil_image = Image.open(image_path).convert("RGB")
image = sv.pillow_to_cv2(pil_image)
height, width = image.shape[:2]
print(f"Image size: {width}x{height}")
Image size: 640x480
The next plot shows the unannotated photo, as a reference for judging the skeletons below.
sv.plot_image(image, size=(3, 2.3))
More than ten children stand in two rows. The front row is fully visible, while the children in the back row are partly hidden behind them. Hidden bodies are the hard case for any pose model.
Shared setup¶
The Hugging Face models run on the device chosen here, and Ultralytics picks its own.
device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"Running on {device}")
Running on cpu
All three models predict the 17 COCO joints, so one edge annotator and one vertex annotator serve them all. EdgeAnnotator picks the COCO skeleton on its own when a person has 17 keypoints. Each person gets a colour by index, so neighbouring skeletons are easy to tell apart.
edge_annotator = sv.EdgeAnnotator(
color=sv.ColorPalette.DEFAULT, thickness=2, color_lookup=sv.ColorLookup.INDEX
)
vertex_annotator = sv.VertexAnnotator(
color=sv.ColorPalette.DEFAULT, radius=3, color_lookup=sv.ColorLookup.INDEX
)
def annotate(scene: np.ndarray, key_points: sv.KeyPoints) -> np.ndarray:
"""Draw the skeleton edges and joints of every person on a copy of the scene."""
annotated = edge_annotator.annotate(scene.copy(), key_points)
return vertex_annotator.annotate(annotated, key_points)
RF-DETR keypoints¶
RF-DETR is a DETR-style detector, and its keypoint preview model finds people and their 17 joints in one pass, so no separate detector is needed. It needs no connector: predict already returns sv.KeyPoints, with the weights of the pretrained COCO model downloaded on first use. The threshold drops detections the model is not confident about.
rfdetr = RFDETRKeypointPreview()
predict takes the Pillow image directly.
rfdetr_key_points = rfdetr.predict(pil_image, threshold=0.5)
KeyPoints is a container of NumPy arrays: xy holds the positions as (people, joints, 2), keypoint_confidence holds one score per joint and class_id holds the class of each person. Printing the shapes shows what the object stores.
print(f"xy: {rfdetr_key_points.xy.shape}")
print(f"keypoint_confidence: {rfdetr_key_points.keypoint_confidence.shape}")
print(f"class_id: {rfdetr_key_points.class_id}")
xy: (11, 17, 2) keypoint_confidence: (11, 17) class_id: [1 1 1 1 1 1 1 1 1 1 1]
The next plot shows whether the skeletons follow the bodies of the children, including those half hidden behind others.
sv.plot_image(annotate(image, rfdetr_key_points), size=(4.5, 3.4))
RF-DETR finds 11 people in one pass and draws the front row cleanly. Its threshold is the confidence of the model itself, a different one from the detector below, so the people counts are not strictly comparable.
Covariance: how sure is each joint?¶
Besides the position, RF-DETR predicts how uncertain each joint is. supervision stores it in key_points.data["covariance"] as one 2x2 matrix per joint in pixels, which describes an ellipse around the joint. A small round ellipse means the model is sure where the joint is, and a large or stretched one means it is not.
covariance = rfdetr_key_points.data["covariance"]
print(f"covariance: {covariance.shape}")
covariance: (11, 17, 2, 2)
The longest semi-axis of each ellipse, in pixels, is one number per joint. The median over the joints of a person ranks the people by how sure the model is about them.
uncertainty = np.sqrt(np.linalg.eigvalsh(covariance)[..., -1])
person_uncertainty = np.median(uncertainty, axis=1)
ranking = np.argsort(person_uncertainty)
most_certain, least_certain = ranking[0], ranking[-1]
print(f"most certain person: {person_uncertainty[most_certain]:.1f}px")
print(f"least certain person: {person_uncertainty[least_certain]:.1f}px")
most certain person: 10.3px least certain person: 81.2px
VertexEllipseOutlineAnnotator draws the ellipses at one and two standard deviations. max_axis caps the size of the rare huge ellipse, so it cannot cover the picture.
ellipse_annotator = sv.VertexEllipseOutlineAnnotator(
sigma=(1.0, 2.0),
color=(sv.Color.YELLOW, sv.Color.RED),
thickness=1,
max_axis=40,
)
The next plot shows the most certain person on the left and the least certain person on the right. The question is where the uncertainty comes from.
panels = []
for person in (most_certain, least_certain):
single = rfdetr_key_points[[person]]
panel = edge_annotator.annotate(image.copy(), single)
panels.append(ellipse_annotator.annotate(panel, single))
sv.plot_images_grid(
images=panels,
grid_size=(1, 2),
titles=["most certain", "least certain"],
size=(9, 3.4),
)
The most certain person is a child on the left of the group, whose ellipses are small and tight around the joints, with a median semi-axis of 10.3px. The least certain person is hidden behind other children in the middle of the group, and its ellipses are wide, with a median of 81.2px. Several are cut at the max_axis of 40px. The model reports a joint it cannot see, and the covariance is what tells that estimate apart from a measured one.
ViTPose¶
ViTPose estimates joints inside a person box, so it is a top-down model: a detector finds the people first, and the pose model then looks at each box on its own. This costs a second model, but the pose network sees every person at the same scale.
The detector is RT-DETR from Transformers. Its checkpoint predicts the 80 COCO classes, so the person class 0 is the only one kept.
detector_name = "PekingU/rtdetr_r50vd_coco_o365"
detector_processor = AutoProcessor.from_pretrained(detector_name)
detector = RTDetrForObjectDetection.from_pretrained(detector_name).to(device)
Gradients are not needed for inference, so disabling them saves memory.
inputs = detector_processor(images=pil_image, return_tensors="pt").to(device)
with torch.no_grad():
outputs = detector(**inputs)
Post-processing turns the raw outputs into boxes in the pixel coordinates of the photo. from_transformers converts them to sv.Detections, the same type the detection cookbooks use. The threshold of 0.5 keeps confident people only.
detector_results = detector_processor.post_process_object_detection(
outputs, target_sizes=torch.tensor([(height, width)]), threshold=0.5
)
detections = sv.Detections.from_transformers(
detector_results[0], id2label=detector.config.id2label
)
people = detections[detections.class_id == 0]
print(f"{len(people)} people, confidence {np.round(people.confidence, 2)}")
13 people, confidence [ 0.95 0.95 0.94 0.94 0.93 0.92 0.91 0.89 0.87 0.87 0.7 0.52 0.51]
ViTPose reads boxes in xywh format, so the xyxy boxes are converted once.
person_boxes = sv.xyxy_to_xywh(people.xyxy)
The base model is the smaller of the two ViTPose checkpoints used here.
vitpose_name = "usyd-community/vitpose-base-simple"
vitpose_processor = AutoProcessor.from_pretrained(vitpose_name)
vitpose = VitPoseForPoseEstimation.from_pretrained(vitpose_name).to(device)
The processor crops and resizes each box, so the model receives one batch entry per person.
inputs = vitpose_processor(pil_image, boxes=[person_boxes], return_tensors="pt").to(
device
)
with torch.no_grad():
outputs = vitpose(**inputs)
Post-processing maps the predicted heatmaps back to positions in the original photo. The result is a list with one entry per image, each holding one dict per person.
vitpose_results = vitpose_processor.post_process_pose_estimation(
outputs, boxes=[person_boxes]
)
from_transformers takes the per-image list. Every person gets class id 0, because the model only reports poses of one object category.
vitpose_key_points = sv.KeyPoints.from_transformers(vitpose_results[0])
print(f"xy: {vitpose_key_points.xy.shape}")
xy: (13, 17, 2)
The next plot shows whether a top-down model draws the same skeletons as the one-stage model above.
sv.plot_image(annotate(image, vitpose_key_points), size=(4.5, 3.4))
The detector found 13 people, so ViTPose draws 13 skeletons. The count comes from the detector: its two lowest confidences are 0.52 and 0.51, so a threshold above 0.52 would drop two children before the pose model sees them.
ViTPose++¶
ViTPose++ is the larger follow-up, trained on several pose datasets at once. It keeps one expert per dataset inside the model, so the forward pass needs to know which expert to use. Index 0 selects the COCO expert, which predicts the same 17 joints as the other models. The processor, post-processing and supervision call do not change.
vitpose_plus_name = "usyd-community/vitpose-plus-large"
vitpose_plus_processor = AutoProcessor.from_pretrained(vitpose_plus_name)
vitpose_plus = VitPoseForPoseEstimation.from_pretrained(vitpose_plus_name).to(device)
dataset_index holds one value per person in the batch.
inputs = vitpose_plus_processor(
pil_image, boxes=[person_boxes], return_tensors="pt"
).to(device)
dataset_index = torch.zeros(len(person_boxes), dtype=torch.long, device=device)
with torch.no_grad():
outputs = vitpose_plus(**inputs, dataset_index=dataset_index)
Post-processing and conversion are identical to the base model.
vitpose_plus_results = vitpose_plus_processor.post_process_pose_estimation(
outputs, boxes=[person_boxes]
)
vitpose_plus_key_points = sv.KeyPoints.from_transformers(vitpose_plus_results[0])
print(f"xy: {vitpose_plus_key_points.xy.shape}")
xy: (13, 17, 2)
The next plot shows what the larger model changes compared with the base one.
sv.plot_image(annotate(image, vitpose_plus_key_points), size=(4.5, 3.4))
The larger model works on the same 13 boxes, so it draws 13 skeletons too, and at this scale it looks like the base model. The differences are a few pixels, which the comparison below measures.
YOLO26 pose¶
Ultralytics ships pose models in its YOLO26 family, in five sizes from n to x. They find the people and their joints in one forward pass, so no separate detector is needed. The medium model is a good size to start with.
yolo = YOLO("yolo26m-pose.pt")
Calling the model returns one result per image. The result holds the boxes, and the joints in keypoints.
yolo_result = yolo(pil_image, verbose=False)[0]
print(yolo_result.keypoints.xy.shape)
torch.Size([12, 17, 2])
from_ultralytics takes the result as it is. Nothing else is needed.
yolo_key_points = sv.KeyPoints.from_ultralytics(yolo_result)
The next plot shows how the Ultralytics model draws the same skeletons.
sv.plot_image(annotate(image, yolo_key_points), size=(4.5, 3.4))
YOLO26m finds 12 people in one pass, without a separate detector. The skeletons of the front row follow the bodies. In the crowded middle the skeletons of neighbouring children cross each other, because their arms and legs overlap in the picture.
Compare the models¶
The four models share one output type, so a side-by-side view takes one list comprehension. The next plot answers where they visibly disagree.
results = {
"RF-DETR keypoint preview": rfdetr_key_points,
"ViTPose-base": vitpose_key_points,
"ViTPose++-large": vitpose_plus_key_points,
"YOLO26m-pose": yolo_key_points,
}
sv.plot_images_grid(
images=[annotate(image, key_points) for key_points in results.values()],
grid_size=(2, 2),
titles=list(results),
size=(8, 6),
)
All four models agree on the front row. They differ in the crowded middle and in the back row, and in how many people they find: 11 for RF-DETR, 13 for both ViTPose models and 12 for YOLO26m.
A picture hides how large the differences are in pixels. To measure it, each person from one model has to be matched to the same person from another model, because the models neither return people in the same order nor find the same number of people. The centre of the confident joints of a person is a stable key, and the Hungarian algorithm picks the pairing with the smallest total distance.
def person_centers(key_points: sv.KeyPoints) -> np.ndarray:
"""Return the mean position of the confident joints of every person."""
weights = (key_points.keypoint_confidence > 0.5)[..., None]
return (key_points.xy * weights).sum(axis=1) / np.maximum(weights.sum(axis=1), 1)
def match_people(reference: sv.KeyPoints, other: sv.KeyPoints) -> tuple:
"""Pair each person of ``other`` with the closest person of ``reference``."""
distances = np.linalg.norm(
person_centers(reference)[:, None, :] - person_centers(other)[None, :, :],
axis=-1,
)
return linear_sum_assignment(distances)
For every matched pair, the distance between the same joint in both models is measured in pixels. Only joints that both models are confident about count, because a model reports a position even for a joint it cannot see.
def joint_errors(reference: sv.KeyPoints, other: sv.KeyPoints) -> np.ndarray:
"""Return the pixel distance of every shared confident joint, NaN elsewhere."""
rows, columns = match_people(reference, other)
distance = np.linalg.norm(reference.xy[rows] - other.xy[columns], axis=-1)
confident = (reference.keypoint_confidence[rows] > 0.5) & (
other.keypoint_confidence[columns] > 0.5
)
return np.where(confident, distance, np.nan)
ViTPose++-large is the reference, as the largest of the four models.
joint_names = [
"nose",
"left eye",
"right eye",
"left ear",
"right ear",
"left shoulder",
"right shoulder",
"left elbow",
"right elbow",
"left wrist",
"right wrist",
"left hip",
"right hip",
"left knee",
"right knee",
"left ankle",
"right ankle",
]
reference = vitpose_plus_key_points
errors = {
name: joint_errors(reference, key_points)
for name, key_points in results.items()
if key_points is not reference
}
assert all(np.isfinite(distance).any() for distance in errors.values())
The printout gives, for each model, how many people it found, how many of them were matched to the reference, the mean error and the three joints with the largest error.
print(f"{'ViTPose++-large (reference)':<26} {len(reference)} people")
for name, distance in errors.items():
valid = np.isfinite(distance)
per_joint = np.nansum(distance, axis=0) / np.maximum(valid.sum(axis=0), 1)
worst = np.argsort(per_joint)[::-1][:3]
print(
f"{name:<26} {len(results[name])} people, {len(distance)} matched, "
f"mean {np.nanmean(distance):.1f}px"
)
for joint in worst:
print(f" {joint_names[joint]:<15} {per_joint[joint]:.1f}px")
ViTPose++-large (reference) 13 people
RF-DETR keypoint preview 11 people, 11 matched, mean 1.8px
right hip 3.9px
left hip 3.4px
left ankle 3.0px
ViTPose-base 13 people, 13 matched, mean 1.7px
left wrist 3.2px
left elbow 2.9px
left ankle 2.9px
YOLO26m-pose 12 people, 12 matched, mean 2.4px
left wrist 4.6px
right ankle 4.2px
left ankle 3.9px
On the joints that both models are confident about, the three models stay within about 2px of the reference on average. The largest errors sit on the arms and feet for YOLO26m and ViTPose-base: the wrists, the elbows and the ankles. RF-DETR differs most on the hips. These numbers measure how much the models agree with the largest of them, not how accurate any of them is, because the photo has no ground-truth annotation. They also ignore the joints a model is not confident about and the people it did not match, so they say nothing about the crowded middle where the models differ most.
Colour keypoints with ColorPalette and ColorLookup¶
So far every skeleton was coloured by person. The vertex, edge and label annotators accept an sv.ColorPalette and a color_lookup that decides which palette entry each keypoint receives:
sv.ColorLookup.INDEX: one colour per person, in the order the model returned them.sv.ColorLookup.CLASS: one colour per class id. All people share class0here, so they share one colour.sv.ColorLookup.KEYPOINT: one colour per joint, so the left wrist has the same colour on every person.
sv.ColorLookup.TRACK is not available for keypoints and raises an error when the annotator is created.
The next plot colours the joints of the ViTPose++ result by index in the first panel, by class in the second and by joint in the third. The question is which view answers which question about a pose.
lookups = {
"INDEX": sv.ColorLookup.INDEX,
"CLASS": sv.ColorLookup.CLASS,
"KEYPOINT": sv.ColorLookup.KEYPOINT,
}
A palette with one entry per joint makes the KEYPOINT lookup readable, so a gradient with 17 colours is built from a Matplotlib colour map.
joint_palette = sv.ColorPalette.from_matplotlib("turbo", len(joint_names))
The edges stay neutral in the third panel, so the joint colours stand out. The other two panels reuse the palette for the edges.
def annotate_lookup(
scene: np.ndarray,
key_points: sv.KeyPoints,
palette: sv.ColorPalette,
color_lookup: sv.ColorLookup,
) -> np.ndarray:
"""Draw skeletons whose edges and joints are coloured by the given lookup."""
edge_color = (
sv.Color.WHITE if color_lookup == sv.ColorLookup.KEYPOINT else palette
)
edges = sv.EdgeAnnotator(color=edge_color, thickness=2, color_lookup=color_lookup)
vertices = sv.VertexAnnotator(color=palette, radius=4, color_lookup=color_lookup)
return vertices.annotate(edges.annotate(scene.copy(), key_points), key_points)
The panels use the default palette for INDEX and CLASS, and the 17-colour gradient for KEYPOINT.
sv.plot_images_grid(
images=[
annotate_lookup(
image,
vitpose_plus_key_points,
joint_palette if lookup == sv.ColorLookup.KEYPOINT else sv.ColorPalette.DEFAULT,
lookup,
)
for lookup in lookups.values()
],
grid_size=(1, 3),
titles=list(lookups),
size=(11, 3.2),
)
Each lookup answers a different question. INDEX gives every person one colour, which keeps skeletons apart when people overlap. CLASS gives all of them the first palette colour, because every person has class 0, so it only helps when a model reports several classes. KEYPOINT gives each joint its own colour on every person, so a swapped left and right limb or a joint pulled onto the wrong body part stands out at once.
VertexLabelAnnotator takes the same arguments and writes a label next to each joint. The label list follows the joint order and each label is coloured by the joint it names. Labelling all 17 joints of every child overlaps, so only three joints of the three most confident people are kept: the nose, the left hip and the left ankle sit far apart on a standing person.
label_joints = [0, 11, 15]
label_key_points = vitpose_plus_key_points[:3][:, label_joints]
labels = [joint_names[joint] for joint in label_joints]
The palette is shortened to match the number of joints kept, so each one keeps a distinct colour.
label_palette = sv.ColorPalette.from_matplotlib("turbo", len(label_joints))
label_annotator = sv.VertexLabelAnnotator(
color=label_palette,
text_color=sv.Color.WHITE,
text_scale=0.4,
border_radius=4,
color_lookup=sv.ColorLookup.KEYPOINT,
)
The next plot shows the three joints of three people, each labelled with its name and coloured by joint.
sv.plot_image(
label_annotator.annotate(image.copy(), label_key_points, labels=labels),
size=(4.5, 3.4),
)
Each label names its joint and takes the colour of that joint: the nose is dark purple, the left hip green and the left ankle dark red. A label is centred on its joint and covers it, so label only the joints you need to read.
What next¶
- Try
sv.VertexEllipseAreaAnnotatorandsv.VertexEllipseHaloAnnotatorfor other looks at the RF-DETR covariance. - Hide uncertain joints:
KeyPointsaccepts a booleanvisiblearray of shape(people, joints), and the annotators skip joints markedFalse. - Send the keypoints through
key_points.as_detections()to get a box around each skeleton and use the usualsv.Detectionsfilters and trackers. - Pass a custom
edgesargument tosv.EdgeAnnotatorfor a model with a different skeleton, such as a hand or a face.