Grounded Vision

Overview

Vision models are useful for interpreting complex or ambiguous images, but they are not always the best tool for simple, repeatable observations. Counting objects, reading a barcode, or locating text can often be handled more consistently by specialized image-processing tools.

Grounded Vision explores a hybrid approach: use dedicated tools to extract measurable information from an image, then provide both those measurements and the original image to a vision model when further interpretation is needed.

The goal is not to replace vision models. It is to give them evidence they can inspect rather than asking them to infer every fact from pixels alone.

The Problem: Counting Cars

Consider a parking-lot camera with a simple requirement: trigger an alert when fewer than three vehicles are present.

There are three ways to approach the task:

Vision model alone. Ask a model to inspect the photograph and count the vehicles. It can interpret complex scenes, but its count may be inaccurate.
Object detector alone. Use a detector to identify vehicles and count the resulting bounding boxes. This produces inspectable evidence but may miss occluded, distant, or unusual objects.
Combined approach. Run a detector first, then give the vision model the original photograph together with the detector's findings. The model can use those findings as evidence while still examining the image itself.

Diagram comparing three ways to answer a counting question: a vision model alone, an object detector alone, and the combined approach that gives the vision model both the photo and the detector's findings. — Three ways to answer: vision alone, detector alone, and combined.

The research focuses on the third approach: combining measurable signals with model-based interpretation.

How the Pipeline Works

Grounded Vision follows four stages.

CaptureObtain the original image from a camera or another source. Preserve it as the evidence the system will interpret.
TriageSelect the most appropriate initial tool for the task. That might be a general object detector, a phrase-based detector, OCR, a barcode reader, or no detector at all.
EnrichRun the selected tool and collect structured findings such as counts, bounding boxes, or recognized text. When the initial result appears weak, the pipeline can invoke a more capable fallback.
DecideProvide the original image and the enrichment findings to a vision model for interpretation when needed. The findings inform the model; they do not automatically override what the image shows.

Flow diagram of the Grounded Vision pipeline: Capture, then Triage, then Enrich, then Decide. — Capture → Triage → Enrich → Decide.

This separation makes the intermediate measurements easier to inspect, compare, and evaluate.

Initial Counting Results

In the car-counting comparison reported in this research, supplying detector measurements alongside the image reduced average absolute counting error:

Method	Average absolute count error
Vision model using the image alone	2.67 vehicles
Vision model using the image plus detector findings	1.33 vehicles

Bar chart of average absolute counting error on labeled car photos: 2.67 vehicles with the photo alone versus 1.33 vehicles with the photo plus detector findings. — Labeled car photos: photo alone versus photo plus detector.

That is approximately a 50% reduction in average counting error across a comparison set of roughly 1,000 labeled photographs.

The reported implementation used YOLOv8s as its general-purpose detector, a confidence threshold of 0.15, an image size of at least 1280, and the original image alongside the detection results. These are experimental configuration choices, not universal defaults for every application.

A Broader Detector Benchmark

The team also tracked detector-layer counting performance across 1,000 labeled photographs:

Image category	Photographs	Mean absolute error	Exact count matches
Ordinary	450	0.82	200
Vehicle	150	1.31	47
Small	150	0.99	53
Crowd	100	4.43	10
Negative	150	0.06	141
Overall	1,000	1.164	451

Chart of the 1,000-photo detector scoreboard by category — ordinary, vehicle, small, crowd, and negative — showing mean absolute error and exact count matches, with crowd scenes the hardest. — 1,000-photo scoreboard by category.

These results highlight both the value and the limitations of the detector. Negative examples performed well, while crowded scenes remained substantially more difficult.

The 1,000-image figures describe the detector layer, not the complete image-plus-model pipeline.

Counting error is also different from alert accuracy. A production system that triggers an alert below a vehicle-count threshold would need separate evaluation of missed alerts and false alarms, particularly on images near that threshold.

Designed for Reuse

The enrichment pipeline is intended to support more than a single camera-monitoring scenario. The initial application described in this work is a monitoring platform, but the same service pattern could support other applications that process images.

An application would submit an image and request one or more enrichment tools. The service would return structured observations that the application can inspect, store, or pass along to a vision model.

This approach aligns with Aon's broader research in Hard Logic and SmartFlow: use deterministic components where possible, preserve a clear execution trail, and reserve flexible AI reasoning for problems that need it.

What Comes Next

The next research questions are practical: when should the pipeline stop after a specialized tool, when should it invoke a more capable detector or vision model, and how should it handle difficult categories such as crowds?

Similar enrichment patterns may also be useful for documents containing complex layouts, tables, and charts.

The larger principle is straightforward: measure what can be measured, preserve the original evidence, and evaluate the decisions that follow.