When Enhancement meets Evaluation: superres and image similarity metrics
2021-10-01, 13:00–13:30, Aconcagua

As a general rule, for raster-based analysis, it is often desired to have all bands available at the highest spatial resolution. The question arises whether it is possible to computationally super-resolve the lower-resolution bands, so as to support more detailed and accurate information extraction. Such a high-quality superresolution, beyond naive interpolation or pan-sharpening, is the topic of this talk. We focus specifically on the superresolution of Sentinel-2 images using Convolutional Neural Network (CNN). In addition, we will discuss how we used some well-known image-similarity measures for the evaluation of the model results. As part of the process, we developed an open-source image-similarity-measure python package aiming to make such evaluations easier for other data scientists.

To perform super-resolution on S2 images, we based our efforts on the work of Lanaras et al. (2018). In his paper, the "DSen2" model is introduced. This model is a two-part model, composed of convolutional neural networks that include multiple layers of residual blocks. One model is intended to super-resolve the 20 m resolution imagery to 10 m, while the second one to super-resolved the 60 m resolution bands to 10 m. We took this methodology, the available codebase, and made modifications to train this model with a new data set and a new data type. In addition, we also adapted the current implementation to work with TensorFlow version 2.
Sentinel-2 images are too big to fit into GPU memory for training and testing, and in fact, it is unlikely that long-range context over distances of a kilometer or more plays any significant role for super-resolution at the 10 m level. With this in mind, the network uses small patches of (32×32) and (96×96) pixels for training. This corresponds to a receptive field of several hundred meters on the ground, sufficient to capture the local low-level texture and potentially also small semantic structures such as individual buildings or small water bodies, but not large-scale topographic features. The aim of the model is to achieve global coverage, therefore samples of 60 representative scenes from around the globe, 45 for training and 15 for testing, were used for its training.
The initial implementation of this model supported only level 1 S2 products. These are top of atmosphere (TOA) reflectance products. bottom of atmosphere (BOA) reflectance or level 2 products are the results of atmospheric correction i.e. the impact of atmospheric distortions is removed from the imagery. This is especially important when running any time-series-based analysis, for example, comparing a vegetation index from July 2018 to July 2019. For this reason, we retrained the "DSen2" model with the L2A data set and have achieved a similar level of performance as the original model and paper describe.
As the last step, we wanted to ensure the performance of newly trained models is equal to the performance of the original model and therefore we used image-similarity-measures for evaluation. Image similarity measures play an important role in image processing algorithms and applications, such as duplicate product detection, image clustering, visual search, change detection, quality evaluation, and recommendation tasks. These measures essentially quantify the degree of visual and semantic similarity of a pair of images.
Similarity measure algorithms can be implemented by identifying (and understanding) the mathematical formula from literature and implementing it in your preferred coding language. Having readily available implementations can help reduce the effort and frustration behind understanding every metric's formula. That is why we have developed a Python package with eight image similarity metrics that can be used either for 8-bit (visual) or 12-bit (non-visual) multi-band images.

Authors and Affiliations

Ekhtiari, Nikoo (1), M. Almeida, Rodrigo (2), and U. Müller, Markus (3) (UP42)




Data visualization: spatial analysis, manipulation and visualization


2 - Basic. General basic knowledge is required.

Language of the Presentation


I am a (geospatial) data scientist. Currently, I am mainly working with Satellite imagery. I am also an avid learner of new methods like machine learning and deep learning and applying them to different domains such as computer vision.