Semantic Segmentation: One Label per Pixel¶
A semantic segmentation model labels every pixel of an image with a class: road, building, sky. It does not separate two people standing side by side, because both pixels are just person. Models return this as one dense map of shape (H, W) holding a class id per pixel.
supervision turns that map into sv.Detections, with one detection per class id holding a mask, a tight box and a class_name. Every annotator, filter and sink then works as usual. Which method to call depends on where the map comes from:
sv.Detections.from_inferencefor Roboflow Inference results, the same call as for detection models. It drops background id0, reads the per-pixel confidence map and can build compact masks.sv.Detections.from_semantic_segmentationfor a raw map, such as the one an Ultralytics YOLO model returns, or to override the class names or keep the background.from_inferenceuses it for semantic results.sv.Detections.from_transformersfor Hugging Face Transformers models. It keeps every class.
This cookbook runs four models, all trained on the 150-class ADE20K scene-parsing dataset, so their outputs are directly comparable:
- RF-DETR: the semantic segmentation models of the RF-DETR family, served by Roboflow Inference.
- SegFormer: a lightweight transformer encoder with a plain MLP decoder.
- Mask2Former: a mask-classification model that predicts a set of masks and classes.
- YOLO26: the semantic segmentation models of Ultralytics, a small convolutional network that predicts the class map in one pass.
Install required packages¶
inference runs RF-DETR and brings torch along, transformers runs SegFormer and Mask2Former, and ultralytics runs YOLO26.
%pip install -q inference transformers ultralytics
Semantic segmentation support ships in supervision 0.32.0. Until it is released, install it from the develop branch. This is a separate step with --no-deps because inference currently pins supervision<0.31, so one pip install cannot resolve both.
%pip install -q --no-deps "supervision @ git+https://github.com/roboflow/supervision.git@develop"
Later sections compare models and draw results, so everything is imported in one place.
%matplotlib inline
import urllib.request
import matplotlib.pyplot as plt
import numpy as np
import torch
from inference import get_model
from PIL import Image
from transformers import (
AutoImageProcessor,
AutoModelForUniversalSegmentation,
SegformerForSemanticSegmentation,
)
from ultralytics import YOLO
import supervision as sv
# Apply tight layout before the plotting helpers display figures.
plt.rcParams["figure.autolayout"] = True
print(sv.__version__)
Matplotlib is building the font cache; this may take a moment.
/private/tmp/supervision-semantic-notebook/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html from .autonotebook import tqdm as notebook_tqdm /private/tmp/supervision-semantic-notebook/lib/python3.12/site-packages/inference/core/env.py:345: InferenceDeprecationWarning: CORE_MODEL_GAZE_ENABLED is True: POST /gaze/gaze_detection is registered as a deprecation stub returning HTTP 410 Gone. The stub and this flag will be removed end of Q2 2026. Set CORE_MODEL_GAZE_ENABLED=False to disable it now. warnings.warn(
/private/tmp/supervision-semantic-notebook/lib/python3.12/site-packages/pydantic/_internal/_fields.py:198: UserWarning: Field name "schema" in "WorkflowsBlocksSchemaDescription" shadows an attribute in parent "BaseModel" warnings.warn(
0.32.0.dev0
Download the sample image¶
Semantic models are easiest to tell apart on a scene with many classes, so this photo has a stone house, a tiled roof, a wall, trees, grass and sky. It is a validation image from the ADE20K scene-parsing dataset, which is released for non-commercial research use.
image_url = "https://huggingface.co/datasets/hf-internal-testing/fixtures_ade20k/resolve/main/ADE_val_00000001.jpg"
image_path = "ade20k-house.jpg"
urllib.request.urlretrieve(image_url, image_path)
('ade20k-house.jpg', <http.client.HTTPMessage at 0x122eeb770>)
The models take a Pillow image, and supervision annotators take a BGR array, so the photo is kept in both forms.
pil_image = Image.open(image_path).convert("RGB")
image = sv.pillow_to_cv2(pil_image)
height, width = image.shape[:2]
print(f"Image size: {width}x{height}")
Image size: 683x512
The next plot shows the unannotated photo, as a reference for judging the masks below.
sv.plot_image(image, size=(12, 12))
The sky and the building take up most of the frame, while the trees, the wall and the small plants on the right are the harder details.
Shared setup¶
The Hugging Face models run on the device chosen here, while Inference and Ultralytics pick their own. Inference is much faster on a GPU, but the small checkpoints here also finish on a CPU.
device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"Running on {device}")
Running on cpu
LabelAnnotator reads class_name from the detections on its own. The sky wraps around the house, so each label is placed at its class's centre of mass rather than at a corner of a frame-wide box.
Annotators pick a colour from the class id. A model that numbers the same classes differently can pass its own ids for the colour lookup, so that a class gets the same colour in every plot.
mask_annotator = sv.MaskAnnotator(opacity=0.6)
label_annotator = sv.LabelAnnotator(
text_position=sv.Position.CENTER_OF_MASS, text_scale=0.5, text_padding=4
)
def annotate(
scene: np.ndarray, detections: sv.Detections, color_ids: np.ndarray | None = None
) -> np.ndarray:
"""Draw one coloured mask per class and write the class name on it."""
annotated = mask_annotator.annotate(
scene.copy(), detections, custom_color_lookup=color_ids
)
return label_annotator.annotate(annotated, detections, custom_color_lookup=color_ids)
RF-DETR with Roboflow Inference¶
RF-DETR semantic segmentation comes in six sizes, rfdetr-sem-nano to rfdetr-sem-2xlarge, pretrained on ADE20K. The smallest one downloads quickly and runs on a CPU.
rfdetr = get_model("rfdetr-sem-nano")
/private/tmp/supervision-semantic-notebook/lib/python3.12/site-packages/rich/live.py:260: UserWarning: install
"ipywidgets" for Jupyter support
warnings.warn('install "ipywidgets" for Jupyter support')
2026-11-13 23:21:50.976760 [W:onnxruntime:, coreml_execution_provider.cc:112 GetCapability] CoreMLExecutionProvider::GetCapability, number of partitions supported by CoreML: 2 number of nodes in the graph: 566 number of nodes supported by CoreML: 565
infer returns one result per image. The result holds the map as a base64-encoded PNG, the class names and a per-pixel confidence map, so nothing needs unpacking by hand.
rfdetr_result = rfdetr.infer(pil_image)[0]
Selected RF-DETR preprocessor implementation=base
from_inference is the same call as for detection and instance segmentation models. It recognises the semantic result and reads all three fields, giving one detection per class. Id 0 is background in this model, so it is left out.
rfdetr_detections = sv.Detections.from_inference(rfdetr_result)
print(list(map(str, rfdetr_detections["class_name"])))
['wall', 'sky', 'tree', 'road', 'grass', 'house']
The confidence map gives each detection the mean confidence of its pixels, which the other three models below do not report through supervision.
for name, confidence in zip(rfdetr_detections["class_name"], rfdetr_detections.confidence):
print(f"{str(name):<8} {confidence:.3f}")
wall 0.862 sky 0.987 tree 0.947 road 0.926 grass 0.975 house 0.963
Because id 0 is background, every ADE20K class sits one id higher here than in the Hugging Face checkpoints below. Shifting the ids back for the colour lookup gives each class the same colour as in the other models' plots.
rfdetr_color_ids = rfdetr_detections.class_id - 1
The next plot shows how the smallest RF-DETR divides the scene.
sv.plot_image(annotate(image, rfdetr_detections, rfdetr_color_ids), size=(12, 12))
The outlines of the house, the wall and the lawn are tight even at this size. RF-DETR names the main structure house rather than building, and it reports no small plant regions. The thin unpainted line along the roof is background, id 0, which was left out. The palette has 21 colours for 150 classes, so house shares its colour with tree.
Any semantic segmentation model deployed on Roboflow goes through the same call, with its own model id. from_inference hands the result to sv.Detections.from_semantic_segmentation, which can also be called directly to load a map alone, override the class names, or keep the background.
SegFormer¶
SegFormer pairs a hierarchical transformer encoder with a small all-MLP decoder, which keeps it fast.
segformer_name = "nvidia/segformer-b2-finetuned-ade-512-512"
segformer_processor = AutoImageProcessor.from_pretrained(segformer_name)
segformer = SegformerForSemanticSegmentation.from_pretrained(segformer_name).to(device)
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
Loading weights: 0%| | 0/380 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 380/380 [00:00<00:00, 56097.27it/s]
Gradients are not needed for inference, so disabling them saves memory.
inputs = segformer_processor(images=pil_image, return_tensors="pt").to(device)
with torch.no_grad():
outputs = segformer(**inputs)
The model predicts logits at a quarter of the input size. post_process_semantic_segmentation upsamples them to the image size and takes the most likely class per pixel, which gives the (H, W) map.
segformer_map = segformer_processor.post_process_semantic_segmentation(
outputs, target_sizes=[(height, width)]
)[0]
print(segformer_map.shape)
torch.Size([512, 683])
from_transformers takes that tensor and the model's id2label. Nothing else is needed.
segformer_detections = sv.Detections.from_transformers(
segformer_map, id2label=segformer.config.id2label
)
print(list(map(str, segformer_detections["class_name"])))
['wall', 'building', 'sky', 'tree', 'road', 'grass', 'plant']
The next plot shows whether the masks follow the outlines of the house, the wall and the lawn.
sv.plot_image(annotate(image, segformer_detections), size=(12, 12))
The large regions follow their outlines closely. The sky label lands on the chimney because the sky surrounds the house, so its centre of mass falls inside the building.
Each detection is an ordinary sv.Detections row, so the area of each mask directly gives the share of the image taken by each class.
coverage = segformer_detections.area / (height * width)
for name, share in sorted(
zip(segformer_detections["class_name"], coverage), key=lambda item: -item[1]
)[:5]:
print(f"{str(name):<12} {share:6.1%}")
building 39.0% sky 38.7% grass 15.3% wall 3.5% tree 2.6%
Mask2Former¶
Mask2Former predicts a fixed set of masks, each with a class score, and combines them into the semantic map. The checkpoint below is the semantic-segmentation variant.
mask2former_name = "facebook/mask2former-swin-tiny-ade-semantic"
mask2former_processor = AutoImageProcessor.from_pretrained(mask2former_name)
mask2former = AutoModelForUniversalSegmentation.from_pretrained(mask2former_name).to(
device
)
Loading weights: 0%| | 0/554 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 554/554 [00:00<00:00, 50639.51it/s]
[transformers] Mask2FormerForUniversalSegmentation LOAD REPORT from: facebook/mask2former-swin-tiny-ade-semantic
Key | Status |
-----------------------------------------------------------------------------------------------------------------------------------+------------+-
model.pixel_level_module.encoder.swin.encoder.layers.{0, 1, 2, 3}.blocks.{0, 1, 2, 3, 4, 5}.attention.self.relative_position_index | UNEXPECTED |
model.pixel_level_module.encoder.swin.layernorm.weight | MISSING |
model.pixel_level_module.encoder.swin.layernorm.bias | MISSING |
Notes:
- UNEXPECTED: can be ignored when loading from different task/architecture; not ok if you expect identical arch.
- MISSING: those params were newly initialized because missing from the checkpoint. Consider training on your downstream task.
The inference call is the same as for SegFormer.
inputs = mask2former_processor(images=pil_image, return_tensors="pt").to(device)
with torch.no_grad():
outputs = mask2former(**inputs)
Post-processing combines the predicted masks into the same (H, W) tensor, so the supervision call does not change.
mask2former_map = mask2former_processor.post_process_semantic_segmentation(
outputs, target_sizes=[(height, width)]
)[0]
mask2former_detections = sv.Detections.from_transformers(
mask2former_map, id2label=mask2former.config.id2label
)
print(list(map(str, mask2former_detections["class_name"])))
['wall', 'building', 'sky', 'tree', 'road', 'grass', 'plant', 'path']
The next plot shows how Mask2Former treats the small regions at the bottom of the frame.
sv.plot_image(annotate(image, mask2former_detections), size=(12, 12))
The large regions match SegFormer, and the paved corner at the bottom right is split into road and path.
YOLO26¶
Ultralytics ships semantic segmentation models in its YOLO26 family. The -sem-ade20k checkpoints are pretrained on ADE20K in five sizes, from yolo26n-sem-ade20k to yolo26x-sem-ade20k. The medium one, yolo26m-sem-ade20k, is about 29 MB and also runs on a CPU. It needs no separate processor, because the model resizes the photo itself.
yolo = YOLO("yolo26m-sem-ade20k.pt")
Calling the model returns one result per image. semantic_mask.data holds the (H, W) class map at the size of the photo.
yolo_result = yolo(pil_image, verbose=False)[0]
yolo_map = yolo_result.semantic_mask.data.cpu()
print(yolo_map.shape)
torch.Size([512, 683])
The map goes through from_semantic_segmentation, with the names from yolo.names. This model has no background class, because ADE20K id 0 is wall, so background_id=None keeps every class.
yolo_detections = sv.Detections.from_semantic_segmentation(
yolo_map, class_map=yolo.names, background_id=None
)
print(list(map(str, yolo_detections["class_name"])))
['wall', 'building', 'sky', 'tree', 'road', 'grass', 'plant', 'house', 'path']
The next plot shows how the medium YOLO26 divides the scene.
sv.plot_image(annotate(image, yolo_detections), size=(12, 12))
The sky, the wall, the lawn and the plants have tight outlines. The house is split between two classes: the roof is building, while the walls and part of the right-hand roof are house. As in the RF-DETR plot, house shares its colour with tree. YOLO26 returns no confidence, so confidence is None.
Compare the models¶
The models share one output type, so a side-by-side view takes one list comprehension. The next plot answers where they visibly differ.
results = {
"RF-DETR-Nano": rfdetr_detections,
"SegFormer-B2": segformer_detections,
"Mask2Former-Swin-T": mask2former_detections,
"YOLO26m": yolo_detections,
}
color_ids = {"RF-DETR-Nano": rfdetr_color_ids}
sv.plot_images_grid(
images=[
annotate(image, detections, color_ids.get(name))
for name, detections in results.items()
],
grid_size=(2, 2),
titles=list(results),
size=(12, 12),
)
Each class now has the same colour in every panel. SegFormer and Mask2Former draw almost the same outlines. RF-DETR calls the whole house house instead of building, and YOLO26m splits it, keeping building for the roof. The corner at the bottom right is path in Mask2Former and YOLO26m, but not in the other two.
A model can spell a class name differently from another one, for example ADE20K's conveyer belt is conveyor belt in YOLO26, so comparing class ids avoids counting one class as two. RF-DETR reserves id 0 for background, which moves each ADE20K class up by one, so its ids are shifted back to line up with the others.
class_ids = {name: set(d.class_id.tolist()) for name, d in results.items()}
class_ids["RF-DETR-Nano"] = set((rfdetr_detections.class_id - 1).tolist())
Classes reported by every model are the safe part of the scene, and classes reported by only one model show where to look closer.
id2label = segformer.config.id2label
shared = set.intersection(*class_ids.values())
print(f"All four report: {sorted(id2label[i] for i in shared)}")
for name, ids in class_ids.items():
others = set.union(*(other for n, other in class_ids.items() if n != name))
print(f"Only {name}: {sorted(id2label[i] for i in ids - others)}")
All four report: ['grass', 'road', 'sky', 'tree', 'wall'] Only RF-DETR-Nano: [] Only SegFormer-B2: [] Only Mask2Former-Swin-T: [] Only YOLO26m: []
Class lists hide how much area a disagreement covers, so the share of pixels with the same class id measures it directly. RF-DETR's map is rebuilt from its masks with the same shift. Its background pixels stay at -1, which counts them as disagreement.
rfdetr_map = np.full((height, width), -1)
for class_id, mask in zip(rfdetr_detections.class_id, rfdetr_detections.mask):
rfdetr_map[mask] = class_id - 1
Each pair of models is compared once.
maps = {
"RF-DETR-Nano": rfdetr_map,
"SegFormer-B2": segformer_map.cpu().numpy(),
"Mask2Former-Swin-T": mask2former_map.cpu().numpy(),
"YOLO26m": yolo_map.numpy(),
}
names = list(maps)
for i, first in enumerate(names):
for second in names[i + 1 :]:
agreement = float((maps[first] == maps[second]).mean())
print(f"{first} vs {second}: {agreement:.1%} of pixels agree")
RF-DETR-Nano vs SegFormer-B2: 59.6% of pixels agree RF-DETR-Nano vs Mask2Former-Swin-T: 59.5% of pixels agree RF-DETR-Nano vs YOLO26m: 89.7% of pixels agree SegFormer-B2 vs Mask2Former-Swin-T: 98.5% of pixels agree SegFormer-B2 vs YOLO26m: 67.4% of pixels agree Mask2Former-Swin-T vs YOLO26m: 67.7% of pixels agree
SegFormer and Mask2Former agree on 98.5% of pixels. RF-DETR and YOLO26m agree with each other on 89.7%, but only on about 60% and 67% with SegFormer and Mask2Former, and one naming difference explains most of it: SegFormer's building covers 39.0% of the image, RF-DETR calls that region house, and YOLO26m calls about 78% of it house.
Large images: compact masks¶
A dense mask is one (H, W) boolean array per class, so a 4K image with 40 classes holds 40 full-size arrays. from_semantic_segmentation also accepts a raw map, and with compact_masks=True it stores each mask as a run-length encoded crop, built one class at a time. background_id=None keeps ADE20K's id 0, which is the class wall rather than background.
compact_detections = sv.Detections.from_semantic_segmentation(
segformer_map,
class_map=segformer.config.id2label,
background_id=None,
compact_masks=True,
)
print(type(compact_detections.mask).__name__)
CompactMask
Compact masks only change the storage, so they must decode to exactly the pixels of the dense masks.
print(np.array_equal(np.asarray(compact_detections.mask), segformer_detections.mask))
True
What next¶
- Filter classes with the usual
sv.Detectionsindexing, for exampledetections[detections["class_name"] == "sky"]. - Try
sv.PolygonAnnotatororsv.HaloAnnotatorfor other looks. - Semantic segmentation cannot count objects, because it does not separate them. For that, use an instance segmentation model with
from_inference,from_ultralyticsorfrom_transformers.