Implementing an open data EO platform to enable better environmental outcomes for the UK Government.
2021-09-30, 10:00–10:30, Microsoft AI for Earth

This talk presents an implementation case study for a specialist Earth Observation (EO) data processing pipeline and access portal for the UK Government, using open standards and open source tools. Details of the requirements, the technical solution and lessons learned are presented. The UK Government Department for Environment Food & Rural Affairs (Defra) is responsible for safeguarding the natural environment, supporting the UK food/farming sectors, and promoting rural economy. Defra and its agencies makes use of large volumes of geospatial data and increasingly required EO Analysis Ready Data (ARD) for a variety of applications. Defra estimated that 70% of the cost of using EO data was in the initial, often duplicated, effort of obtaining and processing data into an ARD format. Therefore, Defra adopted a 'Process Once, Use Everywhere' strategy. CGI, in partnership with Defra, delivered a solution to automatically generate and share access to ARDs openly to all Defra users, based on an open source stack. The processing pipeline was based on ARCSI and Snap, whereas the access web portal was built using GeoNode (backed by GeoServer and PostgreSQL), which exposed a variety of the standard OGC-services for downstream applications, all hosted in Microsoft Azure.

The talk illustrates a number of features to support better access that were added to the core components, demonstrating the extensibility of open source software: * Developed additional GeoNode catalogue filters, eg search by user-defined geometry. * Extended the GeoNode api to improve generating cloud-free mosaic imagery by eliminating problematic split sentinel-2 granules. * Developed a python wrapper client for the existing OGC WPS to improve automated processing workflows. * Exposing the GeoServer layergroup type to allow users to combine data easily via the GeoNode portal and via a rest API.

This implementation demonstrates that open source software such as GeoNode provides a robust and reliable choice for spatial content management. The FOSS architecture together with the partnership of CGI and Defra’s EO subject matter expert, JNCC actively contributed to the success of the implementation.

This talk discusses the need for further work by the EO software community to better support sustainability efforts: * Standardisation of an EO ARD data format and provide guidance on data types such as Cloud Optimised GeoTiff and/or the Open Data cube. * Promote the adoption of the STAC catalogue standard by developing translators for legacy EO catalogues to STAC. This would have been beneficial as the project evolved.


This talk presents an implementation case study for a specialist Earth Observation (EO) data processing pipeline and access portal for the UK Government, using open standards and open source tools. Details of the requirements, the technical solution and lessons learned are presented. The UK Government Department for Environment Food & Rural Affairs (Defra) is responsible for safeguarding the natural environment, supporting the UK food/farming sectors, and promoting rural economy. Defra and its agencies makes use of large volumes of geospatial data and increasingly required EO Analysis Ready Data (ARD) for a variety of applications. Defra estimated that 70% of the cost of using EO data was in the initial, often duplicated, effort of obtaining and processing data into an ARD format. Therefore, Defra adopted a 'Process Once, Use Everywhere' strategy. CGI, in partnership with Defra, delivered a solution to automatically generate and share access to ARDs openly to all Defra users, based on an open source stack. The processing pipeline was based on ARCSI and Snap, whereas the access web portal was built using GeoNode (backed by GeoServer and PostgreSQL), which exposed a variety of the standard OGC-services for downstream applications, all hosted in Microsoft Azure.

The talk illustrates a number of features to support better access that were added to the core components, demonstrating the extensibility of open source software: * Developed additional GeoNode catalogue filters, eg search by user-defined geometry. * Extended the GeoNode api to improve generating cloud-free mosaic imagery by eliminating problematic split sentinel-2 granules. * Developed a python wrapper client for the existing OGC WPS to improve automated processing workflows. * Exposing the GeoServer layergroup type to allow users to combine data easily via the GeoNode portal and via a rest API.

This implementation demonstrates that open source software such as GeoNode provides a robust and reliable choice for spatial content management. The FOSS architecture together with the partnership of CGI and Defra’s EO subject matter expert, JNCC actively contributed to the success of the implementation.

This talk discusses the need for further work by the EO software community to better support sustainability efforts: * Standardisation of an EO ARD data format and provide guidance on data types such as Cloud Optimised GeoTiff and/or the Open Data cube. * Promote the adoption of the STAC catalogue standard by developing translators for legacy EO catalogues to STAC. This would have been beneficial as the project evolved.


Authors and Affiliations

Sam Franklin, CGI

Track

Use cases & applications

Topic

Data collection, data sharing, data science, open data, big data, data exploitation platforms

Level

1 - Principiants. No required specific knowledge is needed.

Language of the Presentation

English

See also: slide-deck-from-talk