世界を埋め込む：大規模な航空画像のための多モーダル AI による検索可能化

Turning a library of aerial imagery into a natural-language-searchable knowledge base is a problem that touches every industry that relies on geospatial data — insurance, real estate, government, infrastructure, and agriculture. The traditional path requires either manual tile-by-tile inspection or training a bespoke computer vision model for each new question. Multimodal embeddings, large language model (LLM) captioning, and vector search on AWS offer a faster alternative: index once, then query using natural language.

We worked with Vexcel, an aerial imagery and geospatial data provider that operates one of the largest aerial imagery programs in the world, to evaluate embedding models, fusion strategies, caption integration, and search methods over multi-view aerial imagery. Using its own sensors and a dedicated fleet of aircraft, Vexcel collects high-resolution data across 45+ countries and territories, delivering orthomosaic imagery, oblique imagery from multiple angles, and elevation models. The data exists, and the use cases are numerous, but turning billions of pixels into answers about the real world requires a faster path.

In this post, we walk through the problem space, our architecture on Amazon Bedrock and Amazon OpenSearch Serverless, the evaluation methodology we built on OpenStreetMap ground truth, four experiments that compared embedding models, fusion strategies, captioning, and search methods, and the practical guidance you can apply when building a similar system. You’ll learn which design choices move the needle for geospatial semantic search, including why Amazon Nova Multimodal Embeddings delivered the highest F1 scores across both benchmark queries in our evaluation. The work described here evolved into Vexcel Intelligence, a searchable imagery product.

Searching millions of aerial images without per-feature training

When a customer needs to locate swimming pools in a suburb, identify road networks in a development zone, or count solar panels across a city, someone has to manually look tile-by-tile (inspecting each map tile in turn) across millions of images. The alternative is training a computer vision model for each feature, which requires labeled data, engineering time, and ongoing retraining. When the next customer wants to find warehouses with graffiti on the side (see Figure 1), they repeat the cycle. Semantic search powered by vector embeddings removes this per-feature training step and turns natural-language queries into results in seconds.

*Figure 1. A typical oblique image from Vexcel, providing models rich 360-degree vision of the world*

Vexcel had explored this problem through three prior POCs: an agent-based approach combining imagery with property data, a property embedding system for similarity search, and a tiled multimodal embedding pipeline with captions generated by a large language model (LLM). The third showed promise but raised key questions: which embedding model to use, how to handle multiple views per location, and whether captions actually improve results or just add cost.

The AWS Generative AI Innovation Center (GenAIIC) partnered with Vexcel to answer a focused question: what is the optimal combination of embedding model, fusion strategy, captioning approach, and search method for semantic search over multi-view aerial imagery? Vexcel brought domain expertise and real-world data, while GenAIIC contributed ML architecture, a complete ingestion-to-evaluation pipeline, and AWS service integration. The result is a system Vexcel has since evolved into Vexcel Intelligence, a product now in preview that transforms their imagery library into a searchable, AI-queryable solution.

Why geospatial imagery search is different

Geospatial imagery search is fundamentally different from searching consumer photos. A query for “swimming pool” on Google Images retrieves standalone photographs from a single perspective. Aerial imagery doesn’t work that way.

A single map tile isn’t one image. It’s seven complementary perspectives of the same location.

Each tile includes an orthophoto (the top-down RGB view), four oblique photographs captured at angles from the north, south, east, and west, a Digital Surface Model (DSM) encoding elevation including structures, and a Digital Terrain Model (DTM) representing bare ground height. The following figure shows what these seven perspectives look like for a single tile — each captures different details about the same geographic location.

*Figure 2. Seven complementary views of the same tile (top row: Ortho, North oblique, South oblique, East oblique; bottom row: West oblique, DTM, DSM)*

These perspectives reveal radically different details. In the preceding figure, the building’s front façade (featuring a kiosk-like window) is only visible from the south oblique angle; the orthophoto, the remaining oblique views, and the elevation models miss it entirely. Meanwhile, the DSM captures tree canopy that obscures ground-level features in the RGB views, and the DTM strips vegetation away altogether. An embedding model that sees only one view works with incomplete information. One that sees the seven perspectives needs a strategy for combining them.

The ground truth challenge

Consumer image search has decades of labeled datasets such as ImageNet, COCO, and Open Images. Aerial feature detection at this scale has none. We needed a way to evaluate search quality without a pre-labeled corpus, which led us to OpenStreetMap as an automated ground truth source, a decision that shaped the entire evaluation framework.

Ambiguity in what counts as a correct result

The third challenge is ambiguity, and it runs deeper than view selection. Consider a search for “swimming pools” that returns a tile where the pool is visible only in the ortho photo but not in any oblique view. It is unclear whether that result is correct. The reverse case is equally ambiguous: a tile where the pool is visible from the south oblique but not from above.

Zoom level compounds this. Depending on tile resolution, a single tile might cover a city block or an entire neighborhood. In a dense suburban area, one tile could contain a dozen swimming pools. Should the ground truth require the system to return a tile if it contains *a* matching feature, or should it account for *every* instance within that tile? A tile-level match (at least one pool present) and a feature-level match (every pool accounted for) are fundamentally different evaluation criteria, and they reward different system behaviors. We had to define what “correct” means before we could measure it.

Co-designing the research agenda

Before writing a single line of optimization code, we built the evaluation harness. This was deliberate: measure before you tune. Without a rigorous way to measure search quality, every architectural decision becomes an opinion.

The engagement was structured around the following six questions, each targeting a specific architectural decision that affects search quality:

• Which embedding model best understands aerial imagery? We compared Amazon Nova Multimodal Embeddings, Amazon Titan Multimodal Embeddings G1, and Cohere Embed v4, each available in Amazon Bedrock, a fully managed service that offers a choice of high-performing foundation models (FMs) from leading AI companies through a single API. Amazon Nova, in turn, is a family of foundation models available through Amazon Bedrock.

• How should you handle seven images per geographic location? We tested per-view embeddings, late fusion (average and max-pool), LLM-weighted attention fusion, and Cohere’s native multi-image batch encoding.

• Does LLM-generated captioning improve search accuracy? We designed a custom captioning prompt that instructs the FM to analyze the seven images simultaneously as complementary views of the same geographic location, identifying each image type (ortho, oblique angles, DSM, DTM) and synthesizing a unified description across land use, built environment, infrastructure, natural features, objects, and spatial relationships. The prompt explicitly directs the model to cross-reference perspectives: when something appears unclear in one view, use other angles or elevation data to resolve it. We tested this prompt with Amazon Nova 2 Lite and Anthropic’s Claude in Amazon Bedrock, measuring whether indexing those captions alongside image embeddings improved retrieval.

• Can LLM-extracted metadata improve filtering? We used a second FM pass to extract up to 25 keyword tags from each generated caption (for example, “swimming pool”, “mature trees”, “commercial district”) and stored them in an Amazon OpenSearch Serverless text field alongside the embeddings. At query time, the same extraction runs on the user’s search query. Amazon OpenSearch Serverless k-nearest neighbor (k-NN) filtering then uses these tags as a pre-filter, narrowing the candidate set to documents whose tags match the query terms before running vector similarity. This uses native filtered k-NN to combine structured metadata with semantic search.

• Which search strategy works best for different feature types? We compared basic k-NN, multi-view fusion search, hybrid image+caption scoring, metadata-filtered search, and text-only search.

• Can you build an automated evaluation framework using publicly available ground truth? We sourced ground truth from OpenStreetMap via the Overpass API, enabling repeatable benchmarks without manual labeling.

The evaluation area was Grant Park in Chicago, with two benchmark queries: “swimming pools” (discrete object detection) and “roads” (distributed infrastructure detection). We tested approximately 100 distinct configurations across these dimensions.

Architecture overview

The system follows a five-stage pipeline (Figure 3), each stage independently swappable for A/B experimentation.

*Figure 3. Five-stage pipeline architecture*

Stage 1. Explore Area of Interest (AOI). Users draw a polygon on an interactive map to define their area of interest. The AOI is persisted to Amazon Simple Storage Service (Amazon S3) for reproducibility.

Stage 2. Ingest Imagery. The system fetches tiles from Vexcel’s API for every map tile intersecting the AOI at a configurable zoom level. Each tile yields up to seven images. Rate limiting (100 requests/second) and Amazon S3 caching help prevent redundant API calls. Credentials are managed through AWS Secrets Manager (Figure 4).

*Figure 4. The ingestion interface: selecting imagery types and zoom level*

Stage 3. Embed & Index. Each image passes through the selected Amazon Bedrock embedding model. Optionally, the seven views are sent to a vision LLM (Amazon Nova 2 Lite, or Anthropic’s Claude) to generate a structured text description. Embeddings and captions are then indexed into Amazon OpenSearch Serverless or Amazon S3 Vectors. The interface provides caption generation controls, a real-time dimension impact analysis, embedding model, and fusion strategy selection (Figures 5 and 6).

Stage 4. Search. Natural language queries are embedded using the same model, then matched against the index. The system auto-detects which fields exist in the index (per-view embeddings, fused embeddings, caption text, caption embeddings) and dynamically enables only the search methods that the index supports.

Stage 5. Evaluate. Search results are scored against OpenStreetMap ground truth using precision, recall, and F1 score. The evaluation framework runs the same query across every enabled search method and reports comparative metrics.

The modular design was the key architectural decision. Every component (embedding model, fusion strategy, search method, vector store) connects through a common interface. Swapping Amazon Nova Multimodal Embeddings for Cohere Embed v4 is a configuration change, not a code change. This enabled us to test ~100 configurations in hours rather than weeks.

*Figure 5. The Embed & Index interface – Caption Generation section*

The following figure shows the embedding configuration panel, where users select the embedding model and fusion strategy for their indexing run.

*Figure 6. The Embed & Index interface – Embedding Configuration section*

Choosing the right K

Every vector search returns the top k-NN results. K is a lever with real consequences, and the right value depends on how common the feature is in your dataset.

When a feature is sparse (a few swimming pools scattered across hundreds of tiles), a large K floods results with irrelevant tiles. The system retrieves K results regardless of whether K relevant tiles exist. Set K=50 when only 8 tiles contain pools, and 42 of those results are noise. Precision collapses. F1 follows.

When a feature is abundant (roads appearing in most tiles), a small K artificially caps recall. Set K=5 when 60 tiles contain roads, and you can only find 8% of them. Recall collapses. F1 score follows.

The relationship is mechanical. Precision@K = relevant_found / K. If K exceeds the number of relevant tiles, precision can never reach 1.0. Recall@K = relevant_found / total_relevant. If K is smaller than the total relevant tiles, perfect recall is impossible.

We evaluated at multiple K values simultaneously (K = 3, 5, 7, 10, 15, 20, 25, 30, 50) and tracked how precision, recall, and F1 score shifted across the range. The optimal K consistently landed near the actual count of relevant tiles in the ground truth, a number you won’t know in production. In practice, start with K=10–20 for general queries, observe the precision-recall tradeoff in your evaluation results, and adjust per feature category. The evaluation framework makes this calibration fast; rerunning with different K values costs seconds, not hours.

Two ways to measure: tile-based vs. entity-based evaluation

A single metric can mask important behavior depending on *what you count as a hit*. We built the evaluation framework with two complementary modes that answer different questions.

Tile-based evaluation asks: *did we find the right locations?* A tile is either relevant (contains at least one instance of the feature) or not. If a tile has one swimming pool or twelve, it counts as one relevant tile either way. Precision, recall, and F1 are computed over tiles.

Entity-based evaluation asks: *did we find the most features?* Each individual entity (each pool, each road segment) counts separately. A tile with 5 pools contributes 5 to the relevant total. Finding that tile recovers 5 entities. Missing it loses 5.

The two modes diverge when features are unevenly distributed, and in aerial imagery, they almost always are. Consider this scenario from our evaluation: