Usage scenarios

The possibilities and benefits of machine-actionability have not been understood well by the RDM community, yet. This leads to new solutions trying to fine-tune the existing inefficient solutions, instead of re-thinking the bigger picture to improve the DMP review process.

Figure 1 depicts three different scenarios for DMP reviewing. The first one, labelled as AS-IS, depicts the typical setting currently in use by most funders. A researcher uses a DMP Software, such as DMP Tool, DAMAP, Argos, DS-Wizard, etc., to generate a PDF containing answers to questions defined in a funder’s template. Reviewers evaluate the DMP by reading the answers from the text document.

The scenarios labelled as TO-BE-1 and TO-BE-2 depict potential settings that we consider in our work. In TO-BE-1, the researcher still uses the DMP Software to produce a DMP, but, compared to the current practice, the software sends the maDMP to the funder. Funders have a dedicated software for evaluation of maDMP. The software provides structured, human-readable information and metrics to pre-assess the answers in the maDMP. For instance, it displays the level of FAIRness of datasets described by the maDMP, so that the reviewers do not have to check that manually. The reviewer still makes the final judgment, but based on more solid information. This example shows that machine-actionability not only helps in exchanging and structuring information, but that it also enables machines to take actions based on this information, such as calculating the FAIRness level.

The scenario labelled as TO-BE-2 is similar to the TO-BE-1 scenario. The main difference is that the DMP evaluation software is used at the researcher’s side to provide feedback on the quality of the DMP, before it is sent to the funder, either as a traditional DMP or as a maDMP. In case the funders do not want to adopt any changes on their side, then the researcher’s side can still benefit from the automated checks. The symbol of the cloud used in the TO-BE-2 scenario means that what is transferred to the funder is undefined, i.e. whether it is a traditional DMP or a maDMP. This has no impact on this scenario and depends on the specific setting, for example, how the stakeholders want to exchange information. In case the funders do not want to adopt any changes on their side, then the researcher’s side can still benefit from the automated checks.

DMP review goals

The process of writing and reviewing a DMP does not have to be considered only from the perspective of fulfilling funder requirements, since DMPs can also be used or mandated in other settings. For example, writing a DMP can be an internal requirement of universities for their students, or companies can use DMPs as their internal documents to have a better overview of data used in their projects. Despite the fact that the majority of existing resources and use cases focus on checking whether DMPs fill funders’ requirements, we believe that checking compliance with funders’ requirements is not the only goal of the reviewers. The Science Europe Evaluation Rubric states that the work of the reviewer is to ‘assess whether the information provided in the DMP is sufficient to ensure that the research team will manage data as expected’ (Europe 2021). Based on this formulation, our own experience and discussions with fellow reviewers, we broke down this high-level objective into more specific goals:

• G1. Completeness: the coverage of all relevant aspects of research data management in the DMP. It can happen that DMPs miss some sections, e.g. no information on ethical aspects is provided. In other cases, DMP can provide only partial information, e.g. licenses are defined for only a subset of datasets listed in the DMP, and for unknown reasons the rest is undefined.
• G2. Feasibility: the possibility to put all DMP content into practice through concrete actions. This can include the identification of inconsistencies with a DMP. For example, a DMP does not plan sufficient storage for the data that will be produced.
• G3. Quality of actions: the relevance and effectiveness of the actions listed in the DMP and performed according to it. In other words, to assess whether what was described can be or was (depending on the phase) implemented according to community standards. For example, whether the data is FAIR, can be openly accessed, etc.
• G4. Non-ambiguity: clear and non-ambiguous formulation of the DMP. This is especially relevant for the non-machine-actionable parts of DMPs, that is, the parts containing text with verbal explanation, e.g. motivating the use of specific tools, techniques, services, etc.

In the remainder of the paper, we focus on methods dealing with machine-actionable information; language check of nonstructured text is out of scope for our investigations. Hence, methods described in Section 3 address goals G1-G3.

RDF-based validation

The emergence of the RDF serialization of maDMPs (Cardoso et al. 2022) opens up the possibility of utilizing semantic web-based validation of maDMPs. In their paper, Cardoso et al. provide an example maDMP validation using Shape Expression (ShEx) (Boneva et al. 2017), demonstrating the validation capabilities of the semantic web technologies. Recently, W3C has approved recommendation of the Shapes Constraint Language (SHACL)² for RDF data validation, which provides a W3C standard alternative for ShEx.

These RDF-based validation methods facilitate maDMP content validation based on defined constraints. SHACL, in particular, allows users to describe constraints as shapes which contain descriptions of their targets–the nodes that they intend to validate. A target could be all instances of a particular RDF class, subjects or objects of a particular RDF property, or an explicit list of nodes. SHACL supports definitions of various constraint types. We provide the following examples of constraints that are particularly relevant to the maDMPs:

• Class or data types: to check if a property value is compatible with the allowed value types, e.g., ‘the cost value should be of type integer or float’.
• Cardinality of property values: to check whether the number of occurrences for a certain maDMP property is contained between the minimum and the maximum number of values allowed, e.g., ‘the number of title of a DMP should be exactly one’.
• String regular expression matching: to validate a property value against a given regular expression, e.g., ‘The URL of the system hosting should follow the following regex pattern:^\\w+: (\\/){0,2}[^\\s]+$’.
• Controlled vocabularies for property values: to validate a property value against a closed list of allowed values, e.g., ‘the dataset distribution access value should be one of the following string: ‘open’, ‘shared’ or ‘closed’.

To demonstrate such functionality, we have developed the maDMP-Ontology Toolkit (MadPot)³ with two main functionalities: (i) to allow transformation between maDMP JSON serialization to RDF and vice-versa, and (ii) to allow validation of maDMP ontology instances based on the specification provided in the DMP-Common-Standard specification. An excerpt of the SHACL validation shape is shown in Listing 1.

We evaluate MadPot on the examples maDMP JSON files provided from the DMP-Common-Standard GitHub page,⁴ and the result shows that MadPot can detect issues within maDMP files successfully previously mentioned, i.e., class/datatype, cardinality, regular expressions, and controlled vocabulary validations. Further, the use of RDF and SHACL would further custom validation rules to be enforced for certain maDMPs, e.g., for specific institutions of funding agencies.

Identifier analysis

One of the simplest approaches that does not require complex processing of information is identifier analysis. maDMPs contain identifiers that are mostly resolvable links, e.g. URIs that overlap with URLs, or DOIs that are in form of a URL. There are two types of checks for the links used in maDMPs that can help in the assessment:

• Existence of linked objects: to check whether the provided link actually points to an existing resource or whether it is broken. For example, this can be used to check whether a DOI for a dataset exists, i.e. whether the dataset was deposited into a repository.
• Link whitelist/blacklist: to check whether the DMP links to an allowed/forbidden resource.

To implement an evaluator tool for link existence, it is enough to check whether the HTTP response is in the successful range (2xx) or in the error range (4xx, 5xx). If the HTTP response is 3xx, the evaluator should first follow the redirect. If the response is in the error range, the evaluator tool should show a warning that the DMP contains unreachable or erroneous links.

To implement the whitelist for link analysis, one needs to define a list of permitted links for the given field of an maDMP. For example, if an organization allows only a CC-BY license to be assigned to data, then the whitelist must consist of URIs corresponding to different CC-BY licenses. Please note that this is a common requirement to refer to licenses using URIs. For example, the CC-BY license should be linked using https://creativecommons.org/licenses/by/4.0/. The tool must map the whitelist to the corresponding field in the maDMP. In the provided example, it would map to dataset\distribution\license_ref. The blacklist can be implemented analogously.

The link analysis method described in this section checks whether the links are correct and whether they fulfill specific criteria defined by a funder or an organization. It does not fetch information from the resource the link points to. This is in contrast to methods described in consecutive sections that depend on information retrieved from external resources.

Using information from linked resources

Resolving links and fetching additional information from services, registries, and other types of systems to which maDMPs point is crucial in establishing a better context enabling proper assessment of DMPs. Compared to methods in Section 3.2, here we assume that the services in which the linked resources are stored provide custom Application Programming Interfaces (APIs) that we can call to retrieve information on the linked resources. We can distinguish two types of checks:

• Resource validity: to check at the specific linked service whether the given resource is valid, e.g. whether a metadata registry has a metadata standard under a specific identifier.
• Context fetching: to retrieve additional information on the specific resource that allows the reviewer to better understand the context, e.g. whether the specified metadata standard applies to the domain for which the DMP is created.

We developed a prototype of an evaluator⁵ that queries an external resource to get more information on a specific URI used with maDMP. For example, the maDMP specification requires information about the metadata standard that describes the format of the actual metadata of the dataset (metadata_standard_id). Instead of creating a new metadata standard, one should link to the existing one from a curated registry. An example of such a registry is FAIRsharing (Sansone et al. 2019). FAIRsharing record for a metadata standard, such as the DataCite Metadata Schema,⁶ contains information about license of that standard, its readiness, related taxonomies and domains, and additional metadata.

The evaluator tool takes an URI, for example, metadata_standard_id, and checks whether it is registered in FAIRsharing.org. If metadata_standard_id is verified, it gets a high score due to the wide acknowledgment of FAIRsharing. Otherwise, it indicates that this standard is either not suitable/known or that the URI/PID is wrong, and therefore gets penalized with a low score.

In a similar way, a list of suitable/scored URIs or PIDs could be supplied to serve as the evaluation criteria. Furthermore, some of the target metadata about the linked resource could be specified as criteria. For instance, only resources from FAIRsharing that are there marked as ‘ready’ can be used. However, such checks depend on the structure of metadata in the registry and the API provided. Specific constraints can be set by funders or organizations depending on the specific context and requirements.

Using scientific knowledge graphs

Semantic web and specifically Linked Open Data (LOD) (Florian & Martin 2012) allows us to link existing knowledge to help in the interpretation and assessment of DMPs. The semantic web was invented to make machines the primary consumer of information. Similar to Section 3.1, the maDMPs can be serialized using RDF. This in turn enables linking to other concepts from the semantic web and not only makes the information stored in maDMP more precise, but also provides a broader context. For example, for a given dataset, one can fetch additional information on the repository in which it is stored, such as the geographical location or type of the repository. If the repository is located in the United States, then an indicator may be switched reporting a violation of the GDPR regulations for European researchers. The additional information can also reveal that the dataset does not match the domain of the repository, for instance, one should not put biotechnology datasets into a mechanical engineering repository.

One example of an existing Scholarly Knowledge Graph (SKG) (cf. Section 6) is the OpenAIRE Research Graph. We implemented a SPARQL query that fetches additional information on the dataset (see listing 2). The query shows that with a given PID, such as a DOI, we are able to retrieve common metadata as an RDF result set. Many properties are optional in the query. This is necessary since the metadata we wish to retrieve is mostly incomplete. For the example, we see in the first filter statement that we will get metadata like the data source name, subjects, and the title upon execution. However, we will not get any information about their creators or country. Another limitation next to incomplete metadata is the incompleteness of publication records in the OpenAIRE KG. In fact, there is a mismatch between the graphical web browser OpenAIRE Explore and OpenAIRE’s KG. We can find an entry of the publication behind the negative example in our query in OpenAIRE Explore, but not in its KG. However, as this is just a matter of further populating the KG, we see much potential here for future use, especially because RDF is a highly automatable specification and the metadata can therefore be easily linked to maDMPs.

Expected FAIRness assessment

Some parts of a data management plan make statements about planned FAIRness which can be then expected from the related results, e.g., published datasets. There are several tools that support the creation of data management plans including a FAIR metrics evaluation based on users’ answers. We can explore this knowledge in the DMP tools and apply it to the maDMP evaluation to assess expected FAIRness and possibly other metrics relevant to a funder.

For example, in the maDMP specification, there is a field indicating the license for a dataset. This field can contain a known license URL, e.g., CC-BY,⁷ or any other URL, such as a link to a custom PDF. We can define a list of known URLs that match each of the cases. Then, we can easily evaluate the reusability by confronting answers from a maDMP with such metric-assigned lists based on the knowledge extracted from DMP tools. The individual scoring of answers and their mapping to maDMPs again depends on the specific context and requirements of funders or institutions.

An example of such a tool is the Data Stewardship Wizard (Pergl et al. 2019). Researchers answer questions in a so-called ‘smart questionnaire’. FAIR metrics are set for specific answers to indicate whether they are good or bad for a given metric. There is a Common DSW Knowledge Model (capturing the questionnaire structure including metrics configuration) for general data management planning. We can use the FAIR metrics encoded there, and automatically evaluate the FAIRness of certain aspects of maDMP.

With respect to the reusability and license example above, there is the following question about dataset distributions in the model with the answers indicating the Reusability metric:

Under what license will the dataset be made available?

a) They will be freely available for any use (public domain or CC0) (Reusability = 1)

b) They will be freely available with the obligation to quote the source (e.g., CC-BY) (Reusability = 0.9)

c) They will be available under some restrictions (Reusability = undefined)

We can explore further questions in the Common DSW Knowledge Model and map them to maDMP fields to implement evaluation metrics. Furthermore, the assessment can be based on knowledge captured in multiple tools, guidelines, best practices, and other resources.

Achieved FAIRness assessment

Shortly after the FAIR principles (Wilkinson et al. 2016) were introduced, several tools emerged to address automatic checking of compliance with the principles. Such tools are commonly called FAIR evaluation tools or FAIRness evaluators. Usually a PID of a resource (e.g., DOI, Handle, or URL) is accepted as input, a series of checks for each principle is performed, and results are presented.

An maDMP contains dataset identifiers (dmp/dataset/dataset_id/identifier) that can be used as input for the FAIRness assessment. Thus, for each dataset, an automatic check via API of a FAIRness evaluator can be done. The result can be used for indication (show the score) or alternatively a criterion can be set (e.g., minimal score or required checks to pass).

One of the tools that we can use for the evaluation of FAIRness is the F-UJI automated FAIR data assessment tool (Devaraju & Huber 2022). It is an open-source project and can be easily deployed with a well-documented and easily usable REST API. Then, for each dataset (and its identifier), an HTTP POST request to the/evaluate endpoint will be made with the identifier in the payload. The response contains detailed information for 16 tests performed, including summarized scoring, which can be presented to the user who requests a DMP evaluation.

The evaluation of FAIRness using F-UJI is prototyped⁸ as a simple command-line utility in Python. It takes a DMP in JSON as an input, extracts all dataset identifiers, and tries to evaluate them via request to the configured F-UJI API. The prototype demonstrates the possibility of utilization of external FAIRness evaluators and can be easily further extended, for instance, to have configurable criteria or use detailed results.

Other current and future FAIRness evaluators that provide usable APIs can be integrated and used to score various PIDs from maDMPs. We also investigated FAIRshake (Clarke et al. 2019) and the FAIR Maturity Evaluation Service (Wilkinson et al. 2019). The first one is relatively complex, but it is also supplied with a Python client library. It allows the user to specify metrics, projects, digital objects, and assessments; a maDMP would be turned into a project with a list of digital objects assessed using specified criteria/metrics. For the second tool, there is no suitable API documentation; however, it is available as an open-source project.⁹ It could be deployed with its own maturity indicators specified, forming collections, and then evaluate each PID from a maDMP separately.

RDF-based constraint evaluators

Prior to the development of SHACL, several attempts were made to create a constraint validation mechanism using RDF Graph constraint languages or similar methods. Pellet-ICV (Tao et al. 2010) enables users to work within the Closed World Assumption (CWA) and the Weak Unique Name Assumption (Weak-UNA), allowing application developers to combine open world reasoning and closed world constraint validation in a flexible manner.

Another viable option is SPIN. SPIN¹¹ is a SPARQL-based rule and constraint language for the Semantic Web, a programming language that combines concepts from object-oriented languages, query languages, and rule-based systems to describe constraints within RDF graph data. RDFUnit (Kontokostas et al. 2014) is a test-driven data debugging framework for linked data. RDFUnit is not a constraint language in the traditional sense, but it can be used to validate and improve the quality of RDF graph data.

Another high-level RDF vocabulary for specifying the shape of RDF resources is Resource Shapes.¹² Resource Shapes are made up of RDF triples that an RDF graph is expected to have and a set of integrity constraints that the RDF graph must meet. Boneva et al. (2017) defined ShEx as a schema formalism for describing the topology of an RDF graph. It defines constraints on the admissible neighborhood for nodes of a given type using regular bag expressions.

In contrast to other approaches, Semantic Web Rule Language (SWRL)¹³ was designed to be a rule language rather than a constraint language. In practice, however, many users use SWRL to validate constraints.

Scientific knowledge graphs

Persistent identifiers for datasets, funders, grants, metadata, contributors, and also for the DMP itself can point to Scientific Knowledge Graphs (SKG). Generally, a knowledge graph can be defined ‘as a graph of data intended to accumulate and convey knowledge of the real world, whose nodes represent entities of interest and whose edges represent potentially different relations between these entities’ (Hogan et al. 2021). In a scientific context, knowledge graphs specify an area of interest that includes the actors, documents, and other research outputs and knowledge (Manghi et al. 2021). An example can be the OpenAIRE Research Graph Data Model, which uses the European OpenAIRE infrastructure, and is a model used to describe scientific objects (Manghi et al. 2019).

Scientific outputs assessment

In the last decade, there has been a shift in a mindset on how to assess scientific output. There are initiatives like COARA,¹⁴ DORA,¹⁵ or Hicks et al.’s (2015) that promote the introduction of new qualitative indicators. The new indicators should serve as a complement to the peer review. Furthermore, the indicators aim to better evaluate the actual content of the scientific output. It should also foster the reduction of the inappropriate usage of the Journal Impact Factor or h-Index for assessing the quality of the research output. The quality indicators should encourage the sharing of data and results, open collaboration or contributions to the research ecosystem, and knowledge generation.