2.1 Selection of evaluation approaches

We based our selection of tools on the collection of FAIRness evaluation tools prepared by the Research Data Alliance (RDA) FAIR Data Maturity Working Group (WG)⁵ (Bahim, Dekkers & Wyns, 2019). That collection presents twelve FAIR assessment tools having their origins at various institutions around the globe. We find that only two out of the twelve presented tools are actually fit-for-purpose in the context of our study. These are the Checklist for Evaluation of Dataset Fitness for Use (Austin et al., 2019) produced by the Assessment of Data Fitness for Use WG (WDS/RDA)⁶ (cf. Sec. 2.1.1) and the FAIR Maturity evaluation service documented in Wilkinson et al. (2019) (cf. Sec. 2.1.2). The latter is not explicitly listed in Bahim, Dekkers & Wyns (2019), but represents the evolution of a listed tool (Wilkinson et al., 2018a). We did not use the other tools listed in Bahim, Dekkers & Wyns (2019) for a number of reasons (see Table 1).

We further sourced the internet by searching for ‘FAIR data evaluation’. Thereby, we discovered the tool FAIRshake (Clarke et al., 2019) and decided to use it in our ensemble approach (cf. Sec. 2.1.3). We also discovered the ARDC’s FAIR self-assessment tool (Schweitzer et al., 2021), but decided not to use it as it neither provides a download option for test results annotated with sufficient metadata of the evaluated resource nor does it provide a quantitative measure of FAIRness as final output (see Table 1).

Building upon earlier collaboration with the developers of the F-UJI tool (Devaraju & Huber, 2020) (see examples in Devaraju et al., 2021), we also used that tool in its software version v1.1.1 for our assessment ensemble (cf. Sec. 2.1.4). Finally, we build on earlier in-house work to evaluate WDCC’s FAIRness (Peters, Höck & Thiemann, 2020) and by performing a self-assessment using the metric collection presented in Bahim et al. (2020) (cf. Sec. 2.1.5).

We summarise the main characteristics of the five FAIRness evaluation tools in Table 2. The detailed results obtained from applying the FAIRness evaluation approaches are available as supporting data (Peters-von Gehlen 2021; Peters-von Gehlen et al., 2021). All the tools were applied in the time of April and May 2021. The versions of the automated (FMES, F-UJI) and hybrid (FAIRshake) tools correspond to those current at that time.

2.2 Selection of WDCC entries for evaluation

The WDCC is a domain-specific long-term archiving service focusing on ensuring the long-term reusability of datasets relevant for simulation-based climate science. Therefore, the main focus lies on the preservation of datasets stemming from numerical simulations of Earth’s climate. Additionally, datasets originating from observations, for example, satellite data products, aircraft observations and in-situ measurements, are also preserved in WDCC but make up a relatively small fraction of the total data volume. Datasets preserved in the WDCC are required to comply with domain-specific (meta)data standards and file formats and be accompanied by rich and scientifically relevant metadata so as to ensure long-term reusability.

The total volume of datasets preserved in WDCC amounts to ≈3.1 PetaBytes (PB, August 2021).²¹ The largest part is represented by climate model output stemming from globally coordinated model intercomparison efforts like the global Coupled Model Intercomparison Project 5 (CMIP5, Taylor, Stouffer & Meehl, 2012) or regionalisations thereof produced within the Coordinated Regional Climate Downscaling Experiment (CORDEX, Giorgi, Jones & Asrar, 2009). Those datasets are highly standardised, because global intercomparison studies rely on the efficient reusability of produced data across user communities. Indeed, data reuse is high for these datasets, therefore justifying the standardisation effort (Pronk, 2019). Smaller datasets archived in WDCC are comprised of climate modeling or observational projects organised at project or institutional levels (e.g., Heinzeller et al., 2017; Jungclaus & Esch, 2009; Seifert 2020) and research output forming the basis of academic publications (e.g., Klepp et al., 2017; Mülmenstädt et al., 2018).

The degree of data maturity (cf. Höck, Toussaint & Thiemann, 2020, for maturity criteria) required for archival in WDCC depends on whether or not a DOI is to be assigned to the archived data: data have to fulfill higher technical and scientific quality requirements if a DOI is to be assigned in the archival process (cf. Peters, Höck & Thiemann, 2020, and references therein).

Individual WDCC-archived datasets, that is, files, are stored as parts of larger data collections – an approach broadly adopted in simulation-based climate science community (e.g., Evans et al., 2017) and which builds on the OAIS (Open Archival Information System, CCSDS, 2012) framework. In an OAIS, the archived information is organised in Archival Information Packages (AIPs), with two specialised AIP-types being the Archival Information Unit (AIU) and the Archival Information Collection (AIC). Broadly speaking, AICs describe a collection of AIUs which are combined in a meaningful way to enable discoverability. AIUs contain metadata describing the archived actual datasets, whereas AICs contain metadata describing the respective collection of AIUs. For an increased reading experience, we will refer to AIUs as ‘units/datasets’ and AICs as ‘collections’ for the remainder of this paper.

In the WDCC, data collections are comprised of ‘entries’, that is, AIPs, which follow a strictly hierarchical structure:²² the topmost level is the ‘project’, followed by the levels ‘experiment’ (collection), ‘dataset_group’ (collection) and ‘dataset’ (unit/dataset) (WDCC, 2016). Of these, the entry types project and dataset are mandatory, whereas the entry types experiment and dataset_group are used as the organisational backbone of larger collections. At the WDCC, DOIs are assigned at the AIC-level only. This is done to i) keep reference lists in publications using WDCC-archived data clear and concise and ii) display the effort put into the creation of a data collection through a single citation with the aim to elevate the data publication to the level of a paper publication. However, some older data preserved in WDCC also have DOIs assigned at the AIU, that is, the dataset, level (e.g., Stendel et al., 2005).

An evaluation of the entire WDCC-archive is evidently out-of-scope as it contains >1.3M datasets, with a total number of 1126 DOIs assigned at the time of writing (August 2021).²³ We have therefore chosen to evaluate a sample of thirteen WDCC-archived AICs (see Table 3), resulting in a total of 32 evaluated AIPs (thirteen experiments, six dataset_groups, thirteen datasets). In the selection of the sample, we aimed at providing a representative assessment across the entire spectrum of WDCC-archived data collections covering various degrees of data maturity while at the same time providing a representative sample in terms of data volume. We evaluated two AICs for two projects (IPCC-AR5_CMIP5 and CliSAP) because data maturity is heterogeneous in these projects. One AIC was evaluated for the remaining nine chosen projects. The evaluation approach is detailed in the next section.

We consider the evaluated AICs (cf. Table 3) as representative for the data maturity level of the entire WDCC-project they are associated with, allowing us to extrapolate the results of our evaluation. Doing so, the cumulative data volume of the WDCC projects evaluated here amounts to ≈2PB (cf Tab.3). The sample is representative of about 65% of WDCC-archived data. The remaining 35% are represented by a large number of smaller AICs for which testing would have been out-of-scope in the context of this study due to time constraints. The results obtained from the evaluation of our sample thus provide a good indication of overall WDCC-FAIRness. We note here, that some of the evaluated AICs were archived before the advent of the FAIR principles and therefore represent the long-established WDCC-approach to ensure long-term reusability of archived data collections.

2.3 Evaluation approach

The granularity of data collections archived in the WDCC is motivated by providing the most appropriate level of data organisation for accessibility and reuse (see above). The amount and richness of metadata (contacts, references, parameter lists, quality assessment reports, free text summary, etc.) differs starkly between the levels of granularity. Therefore, reporting the FAIRness of WDCC-archived data at the level of individual AIUs would not be informative. Hence, we provide results of our assessment at the AIC level, that is, at the level of a WDCC data collection. Also, this is the only way to do justice to the domain-specific approach of organising climate science related simulation-based and observational datasets in larger collections (Evans et al., 2017; Ganske et al., 2020).

In practice, we assessed all AICs presented in Table 3 at the level of their AIUs and averaged the results at the AIC-level for all assessment approaches for reporting, but for our self-assessment (Sec. 2.1.5). For that approach, we performed the evaluation directly at the AIC-level.

2.4 Achieving comparability among evaluation approaches

The applied FAIRness evaluation tools all show a different number of maturity indicators, which are also differently distributed along the FAIR dimensions. In order to achieve comparability between the assessment approaches, we took a pragmatic approach and simply averaged the results over all maturity indicator tests per approach. We do so, because this approach is automatically applied for the two automatic assessment approaches (F-UJI and FMES). Where necessary, we normalised the results to yield a FAIR-score in the range between 0 and 1, indicating a low- or high-level of FAIRness, respectively.

We acknowledge the fact that this way of comparing the results of different FAIRness evaluation tools somewhat distorts the results, because the results per FAIR dimension are not equally weighted. However, we argue here that our study has the main focus of raising awareness for available FAIRness evaluation tools and highlighting the intricacies associated with applying them. In the end, the results of most tests compare well at the AIC-level (see next section).

3.1 Mean scores of FAIR assessments

We show the calculated scores obtained from the five FAIRness evaluation tools along with some general statistics in Table 4. The calculated level of FAIRness strongly depends on the assessment method and the evaluated AIC. Overall, we obtain an ensemble mean FAIR score for the WDCC of 0.67, with individual results per applied FAIRness evaluation tool ranging from 0.5 to 0.88. The calculation of the mean FAIR score does not account for any weighting by data volume per AIC. Scores are mostly higher for the manual or hybrid approaches compared to the automated ones. This is mostly because the automatic FAIRness evaluation tools include checks on the actual data, which require the evaluated data to be openly accessible by the evaluation tool. Since almost all WDCC-archived data are open and free for use by anyone, but only accessible after authentication, the automatic tests requiring data access fail by design. The manual evaluation tools however allow for an evaluation of WDCC-archived datasets, as these can be accessed through human intervention (wording taken from Bahim et al., 2020). Metadata must be prepared accordingly for automated tools, for example, in the JSON-LD, so that it can also be evaluated. We discuss further aspects behind the differences in FAIRness scores between the applied methods in Section 4.

At the AIC-level (column “∅ per project” in Table 4), the spread around the ensemble mean is slightly smaller, ranging from 0.43 to 0.76. AICs with DOI obtain the highest FAIR scores, with an AIC associated with the project CMIP6_RCM_forcing_MPI-ESM1-2, which has a DOI assigned and is comprised of data produced within the framework of the CMIP6 initiative (Eyring et al., 2016), scoring highest.

Consequently, AICs having no DOI assigned, such as MILLENIUM_COSMOS, score lower. The lowest score is determined for one of the CliSAP AICs (CliSAP, no DOI and no data accessible). While that AIC does provide ample metadata on the corresponding WDCC landing pages (cf. Supplement for details to find the tested AICs), the data is not accessible because the status of the AIC was never set to ‘completely archived’ by WDCC staff. The lack of data accessibility can in this case only be pinpointed using the manual and hybrid approaches – the automatic ones fail to recognise this major shortcoming and therefore cannot be used to capture the actual data curation status. While such curation levels are rather the exception than the rule for the WDCC, we deliberately chose to include an AIC with no accessible data in our evaluation to analyse the entire spectrum of WDCC data curation levels and for checking whether the automated tools recognise this.

Summarising this part of our results, we find that all FAIRness evaluation tools can be used to reliably distinguish between various degrees of (meta)data curation of AICs preserved in the WDCC and that for the most part, AICs preserved in the WDCC satisfy the majority of the FAIR maturity indicators addressed by the applied evaluation approaches.

3.2 Agreement between evaluation approaches

Our ensemble approach to FAIRness evaluation also offers the unique opportunity to analyse the consistency between the assessment approaches at the AIC-level. To illustrate this, we computed the relative standard deviation, defined as the standard deviation of a sample divided by the mean of the sample $(\frac{\sigma }{\varnothing })$M4 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ ({\textstyle{\sigma \over \emptyset }}) \] \end{document} , at the AIC level (rightmost column of Table 4) and the cross-correlations between the tests at the WDCC-level shown in Table 5.

If the applied FAIRness evaluation tools show a small spread in determined FAIRness scores for a particular project, they show agreement and $\frac{\sigma }{\varnothing }$M2 \documentclass[10pt]{article} \usepackage{wasysym} \usepackage[substack]{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage[mathscr]{eucal} \usepackage{mathrsfs} \usepackage{pmc} \usepackage[Euler]{upgreek} \pagestyle{empty} \oddsidemargin -1.0in \begin{document} \[ {\textstyle{\sigma \over \emptyset }} \] \end{document} is small. We find the lowest values for datasets having a DOI assigned and being associated with ample machine-readable relevant metadata, that is, CMIP6_RCM_forcing_MPI-ESM1-2 (Steger et al., 2020) and Storm_Tide_1906_German_Bight (Meyer et al., 2021), or a dataset with a low-level of domain-specific maturity (CARIBIC). At the other end of the spectrum, the FAIRness evaluation tools disagree most for the CliSAP AIC for which no data is accessible – for the reasons we alluded to in the previous paragraph. We provide a more detailed discussion of the differences between test results in Section 4.

The cross-correlations between the applied FAIRness evaluation tools (Table 5) clearly indicate that the level of agreement strongly depends on the applied methodology (manual, hybrid or automated), irrespective of covered FAIR dimensions per approach (see Section 2.1). Generally, the results of manual or hybrid approaches compare better to each other than to the automated ones. Similarly, the two automated approaches (FMES and F-UJI) compare well. However, there is an exception: the results of our Self-Assessment and the F-UJI tool also compare relatively well.

Summarising this part of our results, we find that at the AIC-level, the five evaluation approaches broadly agree on the level of FAIRness (with one notable exception, see above). At the WDCC-level, we find that the scores obtained from FAIRness evaluation tools taking an identical methodology (manual, hybrid or automated) also compare well to each other. Here, manual and hybrid approaches can be seen as applying the same evaluation methodology (‘human expert knowledge’) as compared to the purely automated tests.

4.1 Data granularity

At WDCC, preserved data is organised in data collections following a strict top-down hierarchy (cf. Section 2.3), where each level in the hierarchy is identified by an entry ID and has its own landing page in the WDCC GUI. Initially, we planned to present results for each hierarchy level of an AIC (cf. Table 3), but realised soon in the process that this approach does not reflect the evaluation of domain-specific FAIRness in climate science in general and data curation practice at WDCC in particular. As outlined in Section 2.3, we did in fact test all AIUs of the AICs separately and then computed the average. Because the amount and content of machine-actionable metadata varies starkly between the AIC hierarchy-levels, especially the automated evaluation approaches yielded a range of FAIRness scores for the AIUs of a single AIC. For example, F-UJI computed a scores of 0.54 and 0.7 at the ‘dataset’ and ‘experiment’ levels, respectively, for CMIP6_RCM_forcing_MPI-ESM1-2. In this case, the DOI is assigned at the experiment level, automatically resulting in a higher score. However, both entities must not be considered separately, as on the one hand, the actual data is not available at the experiment level. On the other hand, the dataset level lacks the contextual information required for reuse. These domain-specific particularities of data granularity can at the moment not be captured with automated FAIRness evaluation tools but should be considered if FAIRness evaluation and certification become mandatory (see Section 4.4).

4.2 Comparability of test results

The varying capacities of the different FAIRness evaluation tools became very apparent and transpired early in our analysis. While the automated approaches (FMES and F-UJI) are useful for the evaluation of the machine-actionable aspects of preserved (meta)data, they fail to capture the actual curation status of (meta)data preserved in WDCC. We shortly describe four examples illustrating this point:

• Datasets preserved in WDCC are accessible for free, but only after authentication. The machine actionable metadata (JSON-LD) contain an indicator regarding data accessibility (‘isAccessibleForFree’: true). While this is in full compliance with FAIR principle A1.2, the automated test yield failed tests. While this result is fully explainable (FMES and F-UJI check for dataset URLs which are deliberately not included in the JSON-LDs for security reasons), it does reveal a central shortcoming of the automated evaluation approaches and highlights the intricacies of exactly matching the syntax of machine-actionable content required to pass automated tests.
• In cases when data are actually not available, the information on the availability status of the data is only provided on the landing page and not as part of the machine-readable metadata. Therefore, the automated approaches evaluate these AICs exactly as the other tested WDCC-entries (data is not accessible, test failed), resulting in too high FAIRness scores.
• Contextual information is practically impossible to evaluate using automated approaches. As the main goal behind providing FAIR data is to foster their reuse, providing adequate references, documentation and provenance information is essential. The machine-readable qualifiers (‘subjectOf’) included in the JSON-LDs lead to associated publications or reports. Once such a reference is detected by an automated evaluation approach, the corresponding test is passed. However, the actual content of the linked reference cannot be checked – it could therefore be completely irrelevant in the context of the evaluated (meta)data. In the context of this study, the AIC HDCP2-OBS represents such a case.
• By virtue of their intended application, the automated evaluation approaches do not take any information provided on the human-readable landing pages into account. At the WDCC, these often contain ample information about the data, like dataset size and file format. These parameters are not included in the JSON-LD because schema.org-requirements are vaguely defined.

All of the above points pose no problem to manual or hybrid tools. However, including the ‘human factor’ in the evaluation process may lead to inconsistencies. A further limitation of manual FAIRness evaluation tools is the obvious inability to check for machine-actionability. Since this is an essential component of FAIR data, checking just for the human-readable aspects of preserved (meta)data is just as impeding as only checking for the machine-actionable aspects. Or put in other words, automated FAIRness evaluation tools check for the technical FAIRness – or reusability – whereas manual approaches (can) check for the contextual/scientific reusability.

A further point worth discussing is the comparability of the different test results. As outlined in Section 2.1, the five FAIRness evaluation tools do not cover the four FAIR dimensions in a comparable manner: FMES puts little focus on R (2 of 22), FAIRshake is dominated by R (5 of 9), F-UJI is dominated by F and R (together 17 of 24) and our own self-assessment following Bahim et al. (2020) puts equal emphasis on all FAIR dimensions and is far more comprehensive than the other approaches (45 tests, compared to 20, 22, 9 and 24 for CFU, FMES, FAIRshake and F-UJI, respectively). Since there exist no recommendations regarding the importance of individual FAIR dimensions – apart from F, which is seen as the single most important principle of the FAIR spectrum to enable data reuse (Mons et al., 2017) – and their weighting in an evaluation, we provide simple arithmetic means of the test results. Similar to the ensemble approach applied in simulation based climate science, where the ensemble mean over multiple models is usually a better representation of reality than the simulation of an individual model (Tebaldi & Knutti, 2007), we see an added-value in presenting the mean over all FAIRness evaluation tools as ‘WDCC-FAIRness’ (Table 4) as compared to relying on just a single test. Of course, once FAIRness evaluation becomes standardised and an operational requirement for repositories and archives in order to be regarded as trusted in science, basing a certification on the results of an ensemble of tests is impractical. We therefore hope that the results we present here help the community converge towards standardised, broadly applicable and officially recommended FAIRness evaluation tools.

4.3 Lessons learned

The process of applying five different FAIRness evaluation tools has helped us judge the WDCC preservation practice, critically reflect on our internal workflow, indicate avenues for improving the FAIRness of our (meta)data holdings and develop a sound understanding for domain-specific FAIRness in climate science.

• Machine actionability of archived data need not be the priority for data collections in the climate sciences. The size of datasets archived at WDCC is often O(10²)TB and more. It is simply not practical to include URLs pointing to the actual datasets in the machine readable metadata, as this may incur both security and bandwidth issues. The WDCC is currently implementing a PID-system at the dataset level to increase Findability.
• Some of the automated tests could have been passed, if the information given in the machine-actionable metadata would have been as comprehensive as that supplied on the landing pages of archived datasets. One example would be the specification of the file format. At the moment, we do not provide this information in the JSON-LD, because in some cases, the actual file format is NetCDF, a standard open file format of the climate science community, but the files are packed as .zip or .tar archives for download. Note however, that these issues are rather minor and do not reduce the FAIRness of WDCC data holdings per se – including them would merely increase the FAIR score of the automated evaluation approaches.
• Archiving of climate science related data in data collections characterised by a strict top-down hierarchy which do not have PIDs assigned to every data file is a main characteristic of the discipline-specific standard procedure to make these data available to the community. Evaluating a collection in its entity is essential to fully characterise its FAIRness.
• Reaching out to the developers of the evaluation tools was essential to apply the tools correctly, comprehend the test results and even discover bugs in the tools’ source code. Close communication and collaboration between the tool developers and those wishing to apply them cannot be overrated and we wish to contribute further to their development and testing in the future.
• In the process of defining the sample of AICs to be tested, we discovered several ones in which the data is not available due to shortcomings in the WDCC archival workflow. We are at the moment sieving through the WDCC data holdings to find and amend these AICs and make the data associated with them available to the community.
• Applying the manual evaluation approaches is far less straight forward compared to the automated ones. Even if domain and repository experts perform the evaluation, the results may differ because subjectivity cannot be ruled out. One example would be a maturity indicator demanding the provision of dataset and provenance documentation. While supplying links to a third-party online database containing this information would suffice for one evaluator, this might not be the case for another one. Therefore, evaluation results obtained by one evaluator should always be reviewed. In this context, the list of FAIR maturity indicators compiled by Bahim et al. (2020) helps to reduce the risk of unconscious bias because it provides very specific guidance for testing.
• For some AICs, documentation is provided in terms of README files or reports which are archived along with the data. However, these files are hard to find if a user is not familiar with the WDCC and does not know where to look. WDCC-efforts to improve the user experience in this regard are underway by providing more clear access to associated documents and by working towards a community-acceptance of the EASYDAB (EArth SYstem DAta Branding, Ganske et al., 2021) concept which allows users to clearly identify high-quality archived datasets.

4.4 Recommendations for future FAIRness evaluation tools

In the course of our analysis, it became apparent that none of the five applied FAIRness evaluation approaches was entirely fit-for-purpose to evaluate the WDCC data-holdings (cf. Section 4.2 and 4.3), but all of them have their individual strengths on which to build future FAIRness evaluation tools. We provide an overview table summarising our experiences from applying the five different FAIRNess evaluation approaches in Table 6.

For future FAIRness evaluation tools, we recommend the development of capable hybrid approaches to capture both the technical and contextual reusability of preserved research data.

For the reasons we elaborated on above, automated FAIRness evaluation tools are very good at testing maturity indicators which allow for binary yes/no answers following a standardised protocol. Of the two approaches used here, F-UJI seems to be more mature and capable than FMES, but still fails to capture the actual curation status of WDCC data holdings. At that point, the manual part of a FAIRness evaluation would take over to reliably judge the contextual reusability of the preserved (meta)data. Our recommendation to include domain experts and to not only rely on automated approaches in the evaluation of FAIRness and general (meta)data quality is also in-line with recent work on the same topic following a similar line of argument (Wu et al., 2019; Bugbee et al., 2021; Murphy et al., 2021).

In practice, we envision a hybrid approach similar to that of FAIRshake, but substantially more comprehensive. The tool would also include internal databases specifying domain-specific information, like standards, file formats or essential metadata fields specific to the discipline. In this context, the concepts of FMES and FAIRshake enabling the use of different sets of maturity indicator catalogs is very promising. Nevertheless, even with highly standardised and accepted metrics in place, subjectivity can never be completely ruled out when humans evaluate the contextual reusability of scientific datasets. With the current rapid advances in machine- and deep-learning research applications, it may just be a matter of time until such approaches are mature enough to provide objective assessments of FAIRness, such as by comparing documentation in text form with the associated numeric data.