Environmental Scan

Given the importance of Research Data and Primary Material (RD+PM) management plans, and as we were about to embark on a refresh of the QUT RD+PM policy and procedure, we sought to understand how it was currently being done, what researchers thought of it and how they thought it might be improved. We undertook a review of the QUT approach to managing RDMPs through the evaluation of the existing policies, procedures, related websites and any work instructions or departmental guidance. The QUT library was primarily responsible for managing RDMPs and had constructed a website with an online tool. Whilst the library reported a high level of utilisation, it was revealed that as it was not a structured database, it could not be analysed for content, and there was no record of how the plans were actually being used or monitored.

In addition to our review of our own activity, we examined the various approaches to Research Data Management (RDM) across 20 Australian Universities through an examination of their online policies, procedures, guidance material and forms. We have not presented all of the findings of that review here, as the content rapidly changes, and several have been undertaking their own reviews and enhancement of their processes. However, we found that several of the websites did not provide clear guidance material, and there was practically no consistency between them in terms of an approach.

Undertaking an Internal Consultation Process to Identify the Unmet Need

Like all QUT policies, the RDM policy is subject to cyclical review every three years, and the Office of eResearch was charged with the responsibility of performing a review. We formed the Research Data Management Strategy-Implementation Group (RDMS-IG), which comprised representative key academics across science, biomedical science and the humanities, in addition to the Office of eResearch and the Office of Research Ethics and Integrity. The IG met virtually on a monthly basis to provide guidance to the QUT eResearch team working on revisions to the policy and procedure (P&P). The RDMS-IG identified collectively and through extensive one-on-one interviews with a range of academic and professional staff, that, like many institutional policies and procedures, the DMP was not widely known amongst their colleagues, and they did not refer to it on a regular basis. Instead, they noted that staff and students were required to make themselves aware of and to abide by the relevant P&P, and that a number of internal processes were in place to ensure that this happens. Through this stakeholder engagement, it was agreed that Primary Material management plans should be included.

Whilst the knowledge and application of institutional policies is a requirement of employment compliance, they are regarded as a chore that must be done at the start of employment or enrolment—and can be largely forgotten about in between annual mandatory training requirements. This formal compliance requirement has had an unfortunate effect, moving researchers into a ‘compliance-mentality,’ motivated largely by the need to check off a box, rather than approaching RDM+PM management as an important core component of conducting research itself. It should be noted that this is observed in related areas such as ethical review, where the need to ‘get ethical approval’ has replaced the primary notion of being ethical and making sure that the research project itself is intrinsically ethical (De Peuter & Conix 2022).

However, it was clear that whilst Policies and Procedures (P&Ps) related to Occupational Health and Safety (OHS), and other key compliances had mandatory training through an Learning Management System (LMS), the DMP webpage was not seen as essential, and a review of content indicated that there was a need for a refresh. Indeed, a key finding from the IG activity was the need to tailor the content of a DMP to the type of research being conducted. A DMP for law would necessarily be quite different to one for earth sciences or civil engineering.

Anecdotally, it was revealed to us that a lack of mandatory requirements for DMPs. together with a-less-than-friendly user process meant that DMPs were not taken as seriously as the university would like. We also found that in each institution, there was not one person with responsibility for its oversight, and the majority of senior people, while committed to the ideals and objectives of responsible research practice, felt they were under-resourced to deliver training to their staff and students and monitor how RDM was being conducted in everyday situations.

We responded by developing an institutional framework for RDM+PM management, taking a different co-design approach. Importantly, we adopted the approach to include PM, not just RDM. We sought to explore how we could create policy and guidance material for RDM and PM that fulfils the principles set out in the Australian Code for the Responsible Conduct of Research and in other international consensus policies such as the FAIR principles (Wilkinson et al. 2016).

We recognised that there is a natural alignment, especially for a number of disciplines, and RDM and PM plans can come in different forms. To deliver to this approach, we developed an online RDM+PM checklist to support and direct researchers. This checklist currently asks seven questions, taking into consideration the different needs of disciplines spanning Arts, Law, Business, Humanities, Health, Environment and STEM. In this manuscript, we provide an overview of the checklist tool, its inherent ability to enhance the engagement between researchers, their research students and all key stakeholders across the institution, including: research grants, ethics, facilities management, audit and risk, faculties, schools and research centres. The checklist captures real-time business intelligence (BI) statistics that allows us to: i) understand key blockers for any given researcher need and identify stakeholders to engage with to resolve them; ii) continuous feedback on the policy and guidance material; and iii) the ability to capture real-time BI data that can be used to define a metric to quantify an institution’s maturity in managing research data, which we refer to as its Data Quotient™.

Development Principles

The tool is written as a responsive web application with a React frontend and a stateless API backend. Frontend and backend components are written in JavaScript and leverage cloud native apis to service storage, authentication, authorisation and monitoring requirements. The following principles guided the development of the tool:

• High availability: the tool will be available for use outside of scheduled maintenance periods
• Accessible: the tool is accessible to all university users. Usability is reviewed against university guidelines and audited by external third parties
• Low cost: the tool is low cost to run, and the cost of the provisioning, running, and maintaining the tool will not incur a cost barrier to the tool owners or its users
• Self-service: users are able to perform all required interactions with minimal training, rather than a formal onboarding process

Architectural Design of the RDM+PM Checklist

Figure 2 provides an overview of the architectural design implemented in Amazon Web Services. Each of the architectural components are described in Table 1.

Summary of key findings from consultations

As part of our review to revise the policy and procedures for RDM, we asked standard questions including:

1. Should every research project have institutional project data storage space created by default?
2. What should be the standard storage environment?
3. How do we identify and highlight restricted technology use as part of the Checklist, e.g. Clustered Regularly Interspaced Short Palindromic Repeats (CRISPR) policy, etc.

For the top 20 Australian universities, we examined whether they had a current policy, who is responsible for its implementation and management, what resources are available for help with developing plans, how compliance was monitored and how plans integrated with research ethics and integrity (where applicable). We also examined whether they used any online tools to capture plans and whether these were able to provide a means to find data. The findings are summarised in Table 2. Finally, we explored whether institutions provided support, such as running courses in how to develop RDM or PM plans for undergraduates and staff, and whether this was required as part of policy compliance. Based on our analysis, we then focused our desktop review on each QUT faculty. The findings are summarised in Table 3.

Faculties store research project data on either: institution-provided project space; personal OneDrive folders; or a specialist storage area (on or off site), if it is a requirement of the project.

From this review, we identified there were significant gaps in stakeholder engagement and no consistent approach across faculties. We wanted to ensure that higher degree students and their research supervisor understood the data and primary material needs of their research project; enable a process for ongoing dialogue to address any items that needed extra attention; and be comfortable to ask their supervisor for assistance or contact the responsible university department.

Common themes that emerged during the discussions with stakeholders included:

• Everyone has a commitment to robust data management and all institutions have policies.
• However, what do these data management policies actually deliver in practice?
• How are they used?
• Who monitors them?
• Are they worth the paper they are written on?
• Is there redundancy from requirements elsewhere?
• Who teaches students about the relevance/importance of these?

To develop a research data management plan, it was identified as important to:

• clarify and describe the specific research data needs of their project;
• engage with relevant policies, guidelines, resources, services and contacts;
• share their plan with their supervisor (if they are an HDR student) or collaborators;
• attach an exported copy of their plan as supplementary documentation for grant or ethics applications;
• communicate project data requirements to external and industry partners.

To facilitate this, we implemented an online RDM+PM checklist.

RDM+PM Checklist

The Office of eResearch at QUT has designed a web-based Research Data Management + Primary Materials (RDM+PM) checklist to support and direct researchers, taking into consideration the different needs of disciplines spanning Arts, Law, Business, Humanities, Health, Environment and STEM. The solution conception, design and development has been driven by the end-users, relying on the co-development cohort for iterative feedback and enhancement throughout. The tool utilises QUT Single Sign-On for authentication and tiered user access controls; HDRs, supervisors, and other researchers can all use the same platform for different use cases. The checklist tool will be open sourced.

Once logged in, researchers (HDR or career) can create a checklist specific to their project. The interactive checklist will present the user with simple questions, dynamically updating the question list and directing researchers towards relevant conversations with QUT Subject Matter Experts, depending on the nature and content of the answers they provide. The checklist is shown in Figure 3.

Key functional features in the current version of the Checklist are shown in Table 4.

As part of our extensive stakeholder engagement and piloting of the checklist, we obtained the following feedback from users, as outlined in Table 5.

Researchers can use the tool as a skeleton or framework for the creation of RDM+PM plans, relevant to the evolving requirements of their project. HDR supervisors can use the tool to review student progress in their data management and spark meaningful discussions around data and primary material storage. HDR supervisors are trained with the tool and are expected to ensure all HDR students use the checklist as a mandatory process when initiating projects—complementing the training researchers receive as part of their induction certificate in checklist usage.

The RDM+PM checklist provides researchers and research supervisors with a tool to identify, monitor and report on the specific areas of data and primary materials governance that their project requires. The RDM+PM checklist has been made mandatory for all HDR students to complete from the launch date.

Business intelligence can be obtained in real-time. Since August 2022, the checklist became mandatory for new HDR students. At the time of taking this data snapshot, as shown in Table 5, the compliance rate was very high, which involved a new cohort of 142 HDR students who used the tool. We have developed training on RDM as part of their induction, in addition to access to a self-service training video. Only a small number of students have not answered particular questions, and these can be drilled down to assist particular students and if necessary, revise the guidance material for students. From Table 6, if we simply average the percentage answered across the seven questions, we arrive at 97%. This high percentage not only demonstrates the usefulness of the tool, but also provides a mechanism to ultimately measure the institution’s maturity with regards to RDM and PM management. We refer to this measure as an institution’s Data Quotient, and this will be detailed in a subsequent publication. Within the checklist, there is functionality to enable the analysis of data in real-time (analytics), as shown in Figure 4. A range of graphical and tabular representations are dynamically available.

As data management needs vary widely between projects depending on their discipline and scope, the checklist provides a dynamic solution that adapts to fit the need of the researcher in real-time. Crucially, the checklist offers guidance for both data management and primary material management. It is simple to use and understand and includes links to all relevant Subject Matter Experts (SMEs) to ensure that correct guidance for specific management practices is provided. It will be integrated into the early stages of the researcher lifecycle to ensure information is understood in a timely manner. The checklist aligns to the objectives of FAIR and data sharing programs that aim to minimise waste, allow for the re-purposing of data, facilitate reproducibility tests, and so forth. Importantly, the checklist can also foster collaboration across institutions, where the key seven questions can be customised for individual institutional needs, socialising the opportunity to arrive at a ‘common’ set of RDM and PM questions, while promoting best practice and sharing knowledge.

The approach devised by QUT has been acknowledged nationally through both external research infrastructure funding and by shifting the narrative for institutions to guide researchers in research data management planning, rather than providing a single data management planning tool that is not fit for purpose for many research disciplines (ARDC).

The questions posed by the checklist have been designed to directly reflect the current policy and for the first time, provide a systematic approach to guide researchers. Importantly, statistics are immediately available in real-time to enable managers to begin measuring the institution’s preparedness for RDM. It is envisaged that a score can be devised, which can provide weightings on the answers to specific questions and identify opportunities though ongoing feedback where checklist functionality can be enhanced.

At the time of writing this manuscript, it is mandatory for PhD student projects to use the checklist, and we believe this is a strong representative basis. This was a university-wide decision as an extension of the mandatory induction and onboarding training material for students. The derived benefits include: As PhD projects are diverse in nature, the checklist needs to be ‘fit for purpose’; the checklist promotes RDM+PM skills literacy and training at an early career stage, as well as ensuring supervisors’ responsibility in understanding RDM+PM policy. This is reflected in the high compliance results already captured with the tool. Updates to the tool are underway; the checklist is being scoped to support the development of RDM plans, and feedback is obtained, as the tool is currently undergoing customisation for deployment at another institution. This approach enables an agile iterative refinement of the tool through the sharing of best practice between institutions. Further extensions the tool’s functionality is through the automated provisioning of digital and data storage and analytic tool environments to support specific research projects.