Skip to content

Building a Global Multimodal Earth Observation Dataset by Integrating Mapillary Street-Level Imagery with Sentinel-1/2 Satellite Data

Type of project

B.Sc., M.Sc.

Contact

Alkiviadis Koukos (akou@dhigroup.com), Spyros Kondylatos (spko@dhigroup.com)

Topic

Earth observation (EO) has become a fundamental tool for monitoring environmental processes, land use dynamics, and water systems at regional and global scales. Satellites such as the Sentinel-1 and Sentinel-2 provide high-quality data with global coverage and frequent revisit times. Despite their strengths, observations from satellites often lack fine-scale semantic information about objects and structures visible at ground level.

Street-level imagery platforms such as Mapillary provide billions of geotagged images collected worldwide through crowdsourced contributions. These images contain detailed information about roads, drainage infrastructure, coastal protection structures, vegetation, riverbanks, urban waterways, and flood-prone environments. Such ground observations complement satellite data by providing detailed contextual information about landscapes and infrastructure interacting with water systems.

Integrating satellite and street-level imagery therefore presents a unique opportunity to develop multimodal environmental datasets that combine large-scale Earth observation with detailed ground-level perspectives. Despite the availability of large satellite archives and massive collections of geotagged street imagery, there is currently no global dataset systematically linking Sentinel-1, Sentinel-2, and street-level imagery.

This thesis aims to address this gap by developing a global multimodal dataset that integrates Mapillary street-level images with Sentinel-1 and Sentinel-2 satellite observations, with a particular focus on environmental and water-related applications relevant to hydrological and coastal research.

Relevance to DHI

The DHI Earth Observation Center of Excellence frequently applies machine learning methods to EO applications. This project contributes to that work by constructing a global multimodal dataset that links Sentinel-1 and Sentinel-2 satellite imagery with georeferenced street-level images from Mapillary. Such a dataset can support the development of new EO models that combine overhead satellite observations with ground-level visual context, potentially improving the ability of machine learning systems to interpret complex environments.

In addition, many of DHI’s activities relate to water and coastal environments, including flood risk analysis, coastal monitoring, and urban water management. Street-level imagery often captures infrastructure and environmental features relevant to these domains, such as drainage systems, riverbanks, coastal protection structures, and shoreline conditions. Linking these observations with satellite data could enable improved detection and characterization of water-related environments and support the development of EO-based tools that are directly relevant to DHI’s water-focused applications.

What is the student's contribution

The student will develop the pipeline for the generation of the global dataset and generate the dataset itself. An initial estimate of the tasks includes:

  • Retrieve and preprocess data, including downloading, processing, and filtering georeferenced Mapillary street-level images, querying Sentinel-1 and Sentinel-2 data, and ensuring temporal and spatial consistency between the two datasets.
  • Perform spatial alignment and patch extraction, including:
  • matching street-level image coordinates with satellite observations
  • extracting satellite image patches around each ground observation
  • storing paired multimodal samples in a structured dataset.
  • Potentially implement scalable geospatial processing routines capable of handling large volumes of global EO data.
  • Develop and evaluate baseline machine learning models using the generated dataset.
  • Assess the benefits of multimodal inputs, comparing model performance using satellite imagery alone versus combined satellite and street-level imagery for several downstream tasks.