2.1. Data Inputs: Essential Climate Variables

The Climate Change Initiative data collection consists of Essential Climate Variables derived from satellite observations, covering marine, terrestrial and atmospheric domains. The products are mostly global gridded fields, but also include points or profiles along satellite orbit tracks, Shapefiles¹ (e.g. glacier outlines) and images. The datasets span time periods from a few years to multiple decades. The products have been generated using state of the art scientific algorithms developed by the project teams and are typically updated each year, as algorithms are improved and/or new satellite data are incorporated.

Over 130 discrete datasets have been delivered to date, with some ECVs producing many variants (Hollmann et al., 2013) such as products using different satellite datasets, or different algorithms, or the resulting data projected onto different grids, or different spatial or temporal resolutions. The majority of data products are gridded but with the notable exceptions of glaciers and some of the icesheet data which are vector based. Products are available to a range of the standard earth observation processing levels: Level 2 – geophysical variables derived from measurements but in spatio-temporal scale of the original acquisition; Level 3 – variables mapped to a uniform spatio-temporal scale and Level 4 – datasets created from the analysis of lower level data that result in gridded, gap-free products. Depending on the ECV or product, individual dataset sizes are between a few gigabytes and tens of terabytes. The data are used widely in the Earth Observation and Climate community, alone or in combination with model data to increase understanding of the earth system. Satellite observations of climate variables are increasingly used to evaluate or initialise climate models, as well as provide a global view of key quantities and their spatial and temporal evolution.

2.2. Open Data Portal Context

ESA set the requirements for the Open Data Portal, following consultation with the climate science and earth observation communities, mandating a single repository and point of access for the data working in the context of a pre-existing system, the Earth System Grid Federation (ESGF) but also the adoption of a range of community standards in order to facilitate broad dissemination of the data. One such standard is the Open Geospatial Consortium (OGC) Catalog Services for the Web (CSW, Nebert et al., 2016), which defines a means for catalogue and search of geospatial data records. By providing such a service it would be possible to integrate the Open Data Portal with initiatives such as the Global Earth Observation System of Systems (GEOSS).

ESGF and GEOSS differ markedly: while both provide access to data, the former uses consistent software and a globally federated system of index and data nodes to provide discovery and download services respectively (WIlliams et al., 2011), the latter provides a brokerage service utilising high level discovery before passing users down to other portals which themselves deliver a range of services (Bai et al., 2012). Both are predicated on two key services: discovery and delivery, but the semantics and nature of those services differ substantially.

In ESGF, data discovery utilises faceted search using a customised metadata schema. Discovery metadata is replicated between nodes in the federation. This enables distributed download from multiple sites in the federation. The primary workflow is data discovery and bulk data download via wget² or GridFTP (Allcock et al., 2005), although more sophisticated services such as the Live Access Server (LAS, Hankin et al., 2001) and OPeNDAP (Cornillon et al., 2003) are available for some datasets.

The GEOSS Portal³ leverages the CSW formalism to aggregate discovery metadata from multiple data repositories into a single web-hosted search interface. A brokering approach (Nativi et al., 2013) provides the bridge between heterogeneous underlying technologies, data standards and disciplines, albeit with fundamental limitations in the level of integration (whilst standardisation of discovery metadata allows search across multiple catalogues, the differences in access conditions, semantics and structure for file access limit what can be achieved). As a consequence, it is generally not possible to deliver federated data services from any of the constituent portals. However, most GEOSS portals support one or both of the core OGC protocols: Web Map Server (WMS, Beaujardiere 2006) and Web Coverage Service (WCS, Peter Baumann 2012) for some or all datasets.

In practice the requirement for the Open Data Portal working in the context of both the ESGF and GEOSS has meant the need to support both interfaces: the Open Data Portal needs to integrate into the Earth System Grid Federation and broker into the GEOSS via the OGC-CSW. ESGF conformance also required conforming to the requirements of the Obs4MIPS project (Teixeira et al., 2014).

2.3. Archival and Curation

Delivering a portal to data relies on having data available and conforming to the necessary formats and standards. This involves having an ingest process into a repository which deals with multiple versions of data sourced from the producers, validates formats and prepares any necessary additional metadata.

The Centre for Environmental Data Analysis (CEDA) provides the necessary archival and curation environment – archival provides reliable data availability including multiple copies of data and curation provides active management of the data, including format and metadata migration (if necessary). Systems have been built up over decades (Pepler and Callaghan, 2015) to deliver reliable data services and complex metadata systems are in place (Parton et al., 2015) which exploit a wide range of information categories from archive metadata, to the necessary browse and discovery metadata (Lawrence et al., 2009). These data systems are supported by data scientists who have expertise in the relevant science areas and can actively curate the metadata content to maximise utility as systems and target communities evolve.

2.4. Deployment Environment

The target environment for the system is the JASMIN data analysis facility (Lawrence et al., 2013), a collaborative environment for the environmental sciences community combining access to the archive with user-managed storage, a cloud and a batch processing system. JASMIN uses a tenancy model to allocate its computing resources to different projects and programmes. The CCI Open Data Portal was deployed using cloud and virtualisation services on JASMIN. The CCI data itself was integrated into CEDA’s curated data archive on JASMIN. This consists of a broad range of earth observation and atmospheric science datasets which CEDA provides access to as part of a wider service on behalf of NERC, the Natural Environment Research Council.

4.1. Acquisition

Data is first pulled into a staging area for quality control checks, checking conformance with the CCI data standards (ESA Climate Office, 2015) which had been introduced to make data as consistent as possible. It is then formally incorporated into the CEDA archive, exploiting existing CEDA systems which conform to the OAIS archiving standard for managing migration, ensuring fixity of data etc (Corney et al., 2004). This involved storage allocation and organisation into a directory hierarchy, with each CCI dataset mapping to a directory in a POSIX hierarchy.

4.2. Creation of Metadata Records

Metadata is needed to support the CEDA, ESGF and CSW discovery services, as well as support management and data delivery services.

Dataset-level and dataset collection records are manually created in the CEDA data catalogue by data scientists at the granularity of whole datasets. Data products are organised into datasets divided up by the different characteristics (processing level, spatial resolution, instruments, algorithms or product type) of each ECV.

4.3. Unifying metadata approaches

The ESGF discovery and CSW discovery metadata do not share the same semantics, schema, or API, although they are complementary (see Table 1). The most important distinction arises from both limitations and advantages of the existing ESGF technology: it only supports publishing gridded CF-NetCDF data, but it supports discovery down to the granularity of individual files. By contrast, the CSW is designed for general data, but discovery is at the level of entire datasets and collections.

The limitation on datatypes in ESGF would be problematic for the CCI, were it not for the alternative CSW cataloguing, as the CCI includes different datatypes (e.g. glacier boundaries) and formats (Shapefiles). By supporting both interfaces, it is possible to both construct a complete catalogue of all CCI datasets for exposure using the CSW and ESGF search systems. The ESGF system complemented CSW by providing a more powerful faceted search interface for the gridded CF-NetCDF data. The two interfaces also support broad dissemination of the data via a) the ESGF distributed infrastructure and b) CSW harvesting of discovery records into, for example, the GEOSS Geoportal.

Much of the development effort was concerned with the evolution of a system to reconcile the two different approaches of the ESGF and the OGC-CSW. However, it was possible to exploit a data reference syntax, DRS, to provide a common logical organisation. A DRS consists of a set of controlled vocabularies which enable any given dataset to be uniquely identified and enables the development of faceted search capability. The first use of a DRS in this mode was for CMIP5 (Taylor et al., 2012), but a CCI specific version was developed, including terms such as the ECV name, the processing level, the instrument used for the retrieval and the sensor platform (Petrie et al., 2020).

The utility of DRS approaches is enhanced by exploiting SKOS¹⁰ to link vocabulary terms and serving the vocabularies and linked data definitions from a vocabulary server. The use of a vocabulary service provides a single canonical source for information for use by (multiple) search interfaces and any client applications consuming its content. These developments built on experience with the Climate Information Platform for Copernicus (CLIPC, Mihajlovski et al., 2016) and CHARMe (Blower et al., 2014), both of which had demonstrated the advantages of using independent systems to manage controlled vocabularies.

A data modelling exercise was conducted to generate the necessary content. Data scientists entered vocabulary terms and relationships into spreadsheets. Using a special Python script, the information from the sheet cells was converted into SKOS and OWL¹¹ representations using the Turtle¹² serialisation. The use of SKOS allowed the mapping of concept relationships between geophysical parameters (e.g. sea ice is broader than sea ice concentration, and conversely, sea ice concentration is narrower than sea ice). SKOS concept schemes were developed for the CCI DRS (Figure 4) and the full 50 ECVs defined by the Global Climate Observing System, GCOS.¹³ Where possible pre-existing definitions already available from the NERC Vocabulary Server (Leadbetter, 2012) were used. Once prepared, vocabularies were uploaded to an internal vocabulary server implemented using the open source Apache Jena¹⁴ Fuseki triple store.

4.4. The ESGF Publishing System

The ESGF has its own publishing system which is predicated on gridded CF-NetCDF compliant data providing information for the THREDDS catalogue and the Solr distributed database. After consistency checking, the files can be automatically parsed for metadata extraction; information extracted includes variable detail, the necessary DRS terms and details about spatio-temporal extent. Dataset and file-level information were recorded in both databases along with service endpoint information for access methods such as standard HTTP data download and OPeNDAP.

Given the constraint of gridded CF-NetCDF some datasets were not suitable for publishing through the ESGF system. Such datasets were those which did not conform to a coverage array describing values at a point in space and/or time, such as those describing the spatial extent of a feature (e.g. the Greenland and Antarctic Ice Sheets and Glaciers ECVs).

ESGF publishing was started when the first release of CCI datasets had already been made and the second release was underway. In many cases, datasets did not comply with the data standards. This impacted on ESGF publishing since specific metadata were expected in order to represent the different search facets set by the DRS. The ESGF publisher uses a system of map files to link information from the source files to these facets. In order to deal with the inconsistencies with the data, a special cci-tagger script was developed to extract vocabulary terms from the data and implement workarounds for problems such as inconsistent naming for a term, a term being absent completely or terms being defined in filenames, rather than file metadata. The cci-tagger script makes heavy use of the CCI vocabulary service.

4.5. The OGC-CSW Publishing System

The MOLES system underlying the CEDA catalogue supports the export of dataset records serialised as ISO 19115. This capability was used to populate a dedicated project specific OGC CSW interface with CCI metadata and construct a search interface supporting free text search and results at the granularity of individual datasets. Although project specific, this same interface allowed other systems (e.g. GEOSS) to harvest CCI metadata.

The OGC-CSW records were populated with DRS terms from an extension to the cci-tagger script. The resulting ISO 19115 records for the CSW specified the relevant vocabulary SKOS concept in each case and the URI for the source SKOS collection. Where possible these took advantage of a direct mapping between ISO 19115 records returned from the CSW and the related ESGF dataset search results. In practice, there was not always a one-to-one mapping between ISO records and the ESGF dataset records in all cases, with some ISO records containing multiple DRS dataset identifiers.

5.1. Data Access Services

The first development was to deploy data services. Standard FTP access to CCI data was directly available via the CEDA services, all that was necessary for the CCI was to ensure that FTP endpoints for the datasets were entered into the ISO19115 dataset records served from the OGC-CSW. Each ISO record linked to the FTP service by including an online resource link mapped to the FTP path of the equivalent dataset directory in the data archive.

The THREDDS Data server provided HTTP and OPeNDAP out of the box (for gridded datasets), and WMS and WCS were configured as non-standard options. Although built on organising, finding and delivering files, THREDDS also supports aggregation and the attachment of services at both the file and aggregation level. All the files in a dataset were served individually via HTTP, and as an aggregation – a single contiguous set of data combined along the time axis – via OPeNDAP, WMS and WCS.

The necessary configuration for the aggregations was created by an additional Python script which augmented the THREDDS catalogue and Solr search indexes created by the ESGF publisher by adding the aggregation endpoints using the NetCDF markup language (NcML, Nativi et al., 2005). The NcML specifies the variables to be aggregated and includes the time steps for each file as a measure to optimise load performance. Repeated queries benefit from significant performance improvement by virtue of caching by the HTTP server.

ESA required open access, so no access control was configured (eventually the Web Presence provided the capability for users to register in order to provide a mechanism for them to be kept informed about changes to data and services).

5.2. Portal Dashboard

An early “dashboard” search interface was developed to provide a novel graphical overview of all the ECV data products available and the respective temporal coverage for each (Figure 5). In this interface the CSW content was exposed in such a way that the user could select by ECV to see specific ISO 19115 records. This interface now forms part of the larger Web Presence.

5.3. Web Presence

Following the publication of the first data into ESGF services, the CCI Open Data Portal had dual search interfaces. This served to underline the disparity between the two: the CSW provided complete coverage of all datasets but an only rudimentary interface for file access and manipulation through FTP; ESGF Search provided a comprehensive faceted search interface linking to individual file download via HTTP and OPeNDAP but was limited in its coverage since the ESGF Publisher is only capable of indexing NetCDF files into the search catalogue.

Taking advantage of the controlled vocabularies and vocabulary service however, it was possible to combine ESGF and OGC-CSW to provide a single search interface in the Web Presence which integrated search results from both. (Figure 6).

Facets and the available terms for each facet are populated dynamically from the definitions of concepts and collections in the vocabulary service. It can display context-sensitive help text sourced from the ‘preferred label’ text of the vocabulary terms. When a query selection is made, these are translated into keyword items included in a given CSW query (e.g. Figure 7). Keyword items correspond to the URIs of concepts in the appropriate SKOS ConceptScheme. A drill down approach was implemented for the search logic in the user interface. This takes advantage of the fact that the ISO records returned from the CSW contain links to the corresponding ESGF search records:

1. An initial search query was invoked using the CSW. This returns ISO records containing information about the matching datasets.
2. The ISO records returned are checked for matching ESGF records – stored as DRS identifiers. If found, the ESGF search API is then invoked based on these matching DRS identifiers. The ESGF search query returns the list of files (if any) for that dataset.

5.4. Third party clients

The Open Data portal services can be consumed by third party clients. Figure 8 shows two such examples: a WMS client developed by Plymouth Marine Laboratory which pulls data from the CCI Open Data Portal, and the CCI ToolBox desktop client – Cate – developed as part of a separate ESA project. The latter provides more comprehensive functionality for data analysis and visualisation, using the Open Data Portal search and data services. In particular, it takes advantage of OPeNDAP sub-setting to selectively retrieve portions of data from the portal data archive to the user client’s host computer.

6.1. Information mismatch

The task of integrating a varied set of data into the metadata catalogue and archiving services has demonstrated the intimate relationship between the metadata content and the services, with services dependent on particular information provision. Unfortunately, different service requirements had led to different metadata standards, and reconciling alternative metadata technologies and strategies to deliver a range of services was non-trivial. The development of vocabulary services based on linked data and a common data reference syntax made this possible. Equally important was the quality and extent of the primary content – services and usability benefitted from attention to detail by data providers and extra metadata information added by data scientists during data ingestion.

6.2. Scale Issues

The sheer volume of files per dataset exposed performance limitations in the ESGF Publisher software. The Ocean Colour ECV for example, represents a large data collection which splits into 40 datasets along the lines of the geophysical parameter recorded, the co-ordinate system used and the temporal frequency. Even with this division, the number of files for individual datasets was large; for example, daily data involved in excess of 7000 files over the complete time range 1997–2016.

The ESGF Publisher was originally designed for managing CMIP5 data with a very different structure determined by the experiment design. The initial work-around in the case of Ocean Colour, involved splitting the data and reorganising it into smaller temporal chunks so that rather than publishing the dataset as a single 20-year time span, it was divided into smaller datasets. Similar hacks were initially applied to other ECVs. However, a patch was developed for the Publisher such that it was possible to create datasets spanning the full twenty years. In one publishing update in autumn 2018 over a quarter of a million data files were indexed organised into over 100 datasets.

6.3. Content Issues

For some datasets there were last-mile issues with the publication of the data: though ingestion, checks and metadata catalogue generation processes would succeed, minor formatting inconsistencies in NetCDF header file content (e.g. use of non-standards compliant names for the time dimension) would prevent services from operating correctly (e.g. THREDDS being unable to serve OPeNDAP and WMS queries).

In most cases it was possible to correct these by liaising with the respective ECV data producers to get new versions of the data, but these sorts of issues served to underline the need for rigorous checking and enforcement of data standards both by the data providers, and by the ingestion process.