1. Introduction

Scientific data discovery (datasets, documents, facts, visualization, opinions) (e.g., Weikum 2013) often requires users to possess sufficient scientific knowledge to pose useful search questions, along with tools allowing data service providers to be able to correctly understand what users search for and provide usable search results, especially when users search for unfamiliar datasets or information content. On the other hand, data services can also enable self-guided search with capabilities (e.g., spatiotemporal bounding) and abundant dataset-related information (e.g., publications, user forums). Enabling data discovery is listed as one of the challenges (Behnke et al. 2019) for NASA’s Earth Observing System Data and Information System (EOSDIS) (NASA EOSDIS 2022a), which manages 12 discipline-oriented NASA Distributed Active Archive Centers (DAACs) (NASA DAACs 2022). In a recent FAIR (findable, accessible, interoperable, and reusable) (Wilkinson et al. 2016) data assessment for all NASA DAACs (Ramapriyan & Behnke 2020), data findability received the lowest score among all four FAIR categories. In short, data discovery has been a challenge not only for users but also for data producers and data service providers who want their data to be effectively discoverable for maximizing their data distribution.

In this era of rapidly increasing data availability, finding suitable datasets for research, applications, education, and other emerging activities (e.g., water, food, energy nexus) has become increasingly challenging. This is especially true for those who are unfamiliar with scientific disciplines, measurements, or models. At present, datasets are largely archived and disseminated based on data types (e.g., satellite retrievals, field campaigns, models) or disciplines (e.g., each of the 12 NASA DAACs that specialize in certain or multiple disciplines (NASA DAACs 2022)).

Finding data for interdisciplinary activities (involving two or more scientific disciplines; e.g., agriculture and water management) is even more challenging because users often need to visit multiple discipline-oriented data archives and have adequate information to identify suitable datasets. This can be particularly difficult for inexperienced users or users who are not familiar with datasets from other disciplines. The lack of uniform user experiences with different data repositories is another challenge to the search for suitable data products. Currently, most data services or tools are developed for certain groups of scientists or principal investigators in their special disciplinary community. As a result, data service developers often do not have enough knowledge to design data services accommodating users in other disciplines. In addition, different vocabularies used in different communities (e.g., Parsons et al. 2022) can confuse users attempting to both explore and use various data services.

The FAIR data guiding principles (Wilkinson et al. 2016) start with findable, which is (or may be) one of the most challenging tasks for data users, data producers, and service providers worldwide. In the Internet era, data services that facilitate data discovery heavily rely on metadata provided by data producers (e.g., Bugbee et al. 2021; Mathiak et al. 2023). However, many data producers are neither aware of, nor paying attention to, the importance of including sufficient and standardized metadata in their datasets for improving data discovery and usage, mainly because best community data practices and standards have often not been required in their research proposals, such as preparing a data management plan. As a result, data products often do not include sufficient and standardized metadata. This situation has made data service development difficult. For data service developers, without sufficient and standardized metadata from data providers, extra efforts are needed to add additional metadata, which can be a difficult task for many data repositories due to the lack of adequate disciplinary knowledge among staff and the amount of work involved. Furthermore, without proper metadata in a dataset, users will need to seek additional relevant resources (e.g., product documents, research publications), which are often either missing or insufficient.

Despite the reality that most datasets are online with certain heuristic search capabilities available (e.g., filtering), search results often contain many similar datasets that are designed for various research or application purposes by different projects or missions. Without additional information (e.g., publications, usage examples, FAQs), it is often difficult for users to conduct self-guided data discovery (Mathiak et al. 2023). As a result, some users simply ask data support staff for advice.

Lacking unavailable data products that users want is another challenge for both data producers and service providers (e.g., Wu et al. 2019). For example, when a user tries to look for a daily precipitation dataset from a repository but only data with half-hourly or hourly temporal resolution is available, the user may receive a ‘no daily data found’ result. If the daily precipitation dataset is also being provided by the product or service provider as a value-added product, the user will have less difficulty finding the dataset they want or need.

Earth science data users have diverse backgrounds—researchers, application users, educators, students, and ordinary citizens—and possess different knowledge and expertise in handling data and information. Meeting their diverse needs suggests that developing and providing different services (e.g., user interfaces and information contents) is necessary to facilitate data discovery and access. For instance, for a person who simply wants an annual total precipitation map for their region of interest, a traditional workflow for users (e.g., online finding and downloading data followed by processing and visualizing the dataset offline) may not fit this person’s quick need. A more efficient tool, such as Giovanni (NASA Giovanni 2022; Liu and Acker 2017; Acker and Leptoukh 2007), can provide the set of needed procedures (i.e., finding, processing, and visualizing, all online, without having to download data and software) and efficiently (properly and quickly) produce the result (plotting) that fulfills a user’s specific needs (which we would like to provide users with a ‘Window Shopping’ service). In short, meeting users’ needs also plays an imperative role in designing data services that facilitate data and information discovery.

Over the years, there have been several research community activities that have produced recommendations for improving data discovery (e.g., Contaxis et al. 2022). For example, Wu et al. (2019) collected and analyzed 79 data discovery/search scenarios and developed 10 recommendations for data service developers. Another example is the recommendations for earth science data search relevance developed by McGibbney et al. (2019) as a part of NASA’s Earth Science Data System Working Group (ESDSWG) activity. Some discipline-specific websites also post information that guides users on how to select data (e.g., Huffman 2022; NASA Earthdata 2022a, 2022b). As previously mentioned, many existing data services have been developed for a particular dataset or discipline. Therefore, those recommendations may not be suitable for data services for interdisciplinary research and applications in which datasets are involved from several disciplines and archived at different data repositories. The current situation warrants further investigation into this new challenge by reviewing current practices in order to provide practical recommendations.

In this paper, we assess current operational practices in data discovery and publications (e.g., referral papers, reports from working groups). As one of the DAACs managed by NASA’s EOSDIS (NASA EOSDIS 2022a), the NASA Goddard Earth Sciences Data and Information Services Center (GES DISC) (NASA GES DISC 2022a) has archived and distributed multidisciplinary satellite and model data products. Although GES DISC only archives a portion of NASA earth data and users may also need data from other DAACs for their activities, the diverse and interdisciplinary data collection at GES DISC can still serve as an example or use case for this study. Based on the findings of this study, we discuss challenges and opportunities for improving earth science data discoverability and facilitating interdisciplinary research and applications. At the end, we provide practical recommendations. These recommendations may not be limited to GES DISC or NASA.

The structure of the paper is as follows: section 2 overviews existing operational practices, section 3 includes a summary of referral publications and reports from working groups, section 4 discusses challenges and opportunities, and section 5 provides our summary and recommendations.

2. Data Discovery Practices at NASA GES DISC

Established in the mid-1980s, NASA GES DISC (NASA GES DISC 2022a) is in Greenbelt, Maryland. It currently archives a total data volume of 3.4 petabytes consisting of 150 million data files and covering over 3,000 public and restricted multidisciplinary data collections, including atmospheric composition, water & energy cycles, climate variability, carbon cycle & ecosystem from both major NASA satellite missions (e.g., Global Precipitation Measurement (GPM)) and projects (e.g., the Modern-Era Retrospective analysis for Research and Applications, Version 2 (MERRA-2)). MERRA-2 provides NASA’s global atmospheric reanalysis from 1980 onward. Enhancements have been made in MERRA-2, including the use of an upgraded version of the Goddard Earth Observing System Model, Version 5 (GEOS-5), data assimilation system; updates to the model (Molod et al. 2012; Molod et al. 2014) and to the Global Statistical Interpolation (GSI) analysis scheme (Wu et al. 2002); the first global reanalysis to assimilate space-based observations of aerosols and their interactions with other physical processes in the climate system; and a representation of ice sheets over Greenland and Antarctica (Bosilovich et al. 2016). In short, significant steps toward NASA’s Earth system reanalysis goal have been taken in MERRA-2.

The GES DISC provides data services and support to users around the world, including (1) metadata support, documentation, and metrics (Liu et al. 2022) for archived datasets; (2) web-based discovery and access to data products and data download; (3) value-added services on data; (4) user services providing support for data access and use; and (5) community engagement and outreach (e.g., user working groups, workshops, trainings, conferences, webinars).

Over the years, data services at GES DISC have continuously evolved. Guided by these user support activities, best practices, and recommendations from workshops and publications, data service needs have been routinely identified and prioritized based on several criteria, such as available resources, level of difficulty, and user needs. A group consisting of scientists and software developers is formed to formulate implementation details (e.g., service requirements, user interfaces) and acceptance criteria. Metrics (Liu et al. 2022) are routinely collected and evaluated for service improvements.

For example, in the past, users could only download data in the original forms provided by data producers. Since the spatial coverage of most NASA datasets is global, users would have to download the entire global dataset even for local or regional studies or applications if (or when) data subsetting services were not available. This action would increase network congestion and data server loads and cause unnecessary data downloads. Today, GES DISC provides a range of subsetting capabilities, such as spatial subsetting, as well as regridding and reformatting services. With the ongoing cloud evolution (NASA Earthdata 2022a, 2022c; NASA GES DISC 2022b), data services will be significantly improved, especially for handling voluminous global satellite and model datasets that are difficult and inefficient for on-premises services to handle. In addition, users will no longer have to visit multiple DAACs for data services (e.g., download multidisciplinary datasets). In short, cloud environments will provide a wide range of data services on one platform that are currently difficult to provide on-premises.

The GES DISC web portal (Figure 1) (top) provides a Google-like interface for searching data and information (e.g., documents, how-to recipes, FAQs). Prior to this, users had to visit multiple GES DISC websites for needed data services and information, which could be confusing, inconvenient, and time-consuming, as well as difficult for them to remember those websites. The current GES DISC web portal (NASA GES DISC 2022a) has unified these data service websites to provide a one-stop shop for all data-related services and information. In Figure 1 (bottom), each dataset has its own dataset landing page (DLP) including such information as product summary, data access, data citation, and supporting documentation.

Current self-guided search methods are limited to keyword search (NASA GES DISC 2022a). General rules include a single keyword (e.g., product short name, platform short name, measurement, project name), multiple keywords, and simple query string operators (e.g., AND, OR, exclusion, and wildcard) (NASA GES DISC 2022a) for multiple keywords. Advanced search options include spatial and temporal range refinements. Search capabilities are still being developed in terms of relevance and accuracy.

NASA’s Common Metadata Repository (CMR) (NASA Earthdata 2022d, Bugbee et al. 2021) is the backend engine behind GES DISC data search and other NASA data services, such as Earthdata (NASA Earthdata 2022a). CMR is a high-performance, high-quality, and continuously evolving metadata system. With CMR, all data and service metadata records for NASA’s EOSDIS system are cataloged. CMR is also the authoritative management system for all EOSDIS data, including those at GES DISC and other DAACs. Metadata in CMR provide the description of a dataset; therefore, the quantity and quality of metadata records play an imperative role in data search and discovery (Bugbee et al. 2021).

Search results from the GES DISC web portal (NASA GES DISC 2022a) can be refined by subject, measurement, source, processing level, project, and spatiotemporal resolution. However, finding a specific variable can still be a challenge. Most data products at GES DISC are packed as data collections; for example, the data collection of time-averaged two-dimensional monthly means (M2TMNXFLX) in MERRA-2 contains a total of 46 variables. If a user wants one specific variable (e.g., total precipitation) in this collection, the user must find the data collection first and then use the subsetting service or the dataset document to identify the variable, which can be difficult for users who are unfamiliar with the collection name and the DLP information.

Users or individuals with knowledge of dataset names can usually narrow down their search by using the dataset short names. However, only a small percentage of users (e.g., in Figure 2) may be familiar with dataset short names. For example, the Integrated Multi-satellitE Retrievals for GPM (IMERG) is a very popular global precipitation data suite that provides merged multisensor and multisatellite global precipitation estimates ranging from 30 minutes and daily to monthly. When a user searches for ‘IMERG monthly’ for the monthly IMERG dataset, the search results will consist of over 500 datasets. By contrast, a search for the short name, GPM_3IMERGM_06, or ‘IMERG + monthly,’ will only return one result that links to the exact DLP wanted and/or needed. These two keyword searches are, however, not intuitive.

To better understand a user’s search habits, Google Analytics (Liu et al. 2022) is used. Figure 2 (top) shows that the default search type for the Google-like search interface (Figure 1) is the most searched content type. The most popular dataset keyword searches shown in Figure 2 (middle) are associated with satellite precipitation, hydrology, and atmospheric data assimilations (e.g., Global Land Data Assimilation System (GLDAS)), which is not a surprise because GES DISC is home to data from several major NASA precipitation measurement missions (e.g., Tropical Rainfall Measuring Mission (TRMM), GPM) and projects (e.g., Global Precipitation Climatology Project (GPCP), North American Land Data Assimilation System (NLDAS), MERRA-2).

The top searched keyword for information (e.g., FAQs, data documentation) is ‘Giovanni Measurements.’ As previously mentioned, Giovanni is a popular and powerful online tool developed at GES DISC. At present, there are over 2,000 variables served in Giovanni (NASA Giovanni 2022). In the GES DISC web portal, users can only search datasets and not their variable contents. In Giovanni, users or individuals can directly search for data variables (e.g., precipitation, temperature) that are likely more familiar to them, simplifying and expediting data discovery and access. Giovanni can be used for data evaluation, intercomparison, and other activities without downloading data and software (NASA Giovanni 2022; Liu and Acker 2017; Acker and Leptoukh 2007).

Recognizing the limitations (e.g., single dataset-oriented services) in data services and challenges for interdisciplinary data discovery, GES DISC has previously experimented with a novel search relevance method by allowing in-house data specialists to group related and frequently used datasets by research or application subjects (e.g., agriculture, hurricane). A keyword search for these subjects will return the associated datasets properly (NASA GES DISC 2022a, 2022c). However, this approach can be subjective, and the compiled variables may not be suitable for all research activities.

Even with a smaller list of search results, it is still difficult to find a suitable dataset. Previous work and external research publications play an important role in providing additional information (e.g., examples or use cases) (Lafia et al. 2016). Users can use these past investigations available on the DLP to learn how each dataset is utilized in research or applications. An ongoing novel activity at GES DISC is to use AI/ML to harvest science-subject-based use cases from published journal articles (Stoyanova et al. 2021).

Once a dataset is identified in the search results, the user will be directed to its DLP (Figure 1). At present, each DLP (Figure 1) includes key information about a dataset, such as data summary, data access, citation, documentation, and references. The concept of DLP is not novel and is widely used in product-related services like Amazon because it provides a one-stop shop for all dataset-related services and information. The DLP continues to evolve with improvements. More dataset-related information and services need to be added or expanded (e.g., publications). Adding a user forum can also provide useful feedback for product developers to identify product issues. It can help data centers in assisting new users to get help from other experienced users, like services provided on current commercial shopping sites.

A significant number of scientists, including data specialists at GES DISC, are not trained to deal with multidisciplinary science subjects and datasets. Therefore, putting together a list of interdisciplinary datasets can be a challenge, especially if/when it is strongly dependent on the knowledge of data specialists who are often familiar with a few satellite missions or projects but may not be aware of similar datasets from other missions or projects. Ideally, data producers should provide such information (e.g., data usage) in their product metadata for data services, but again, as mentioned, there is no mandate to include such information in the current data management plan for a data producer.

GES DISC is also developing personalized data services in ‘My Dashboard’ for registered users (NASA GES DISC 2022a). With the dashboard, registered users can bookmark their favorite DLPs and information, automatically record site (visit and access) history, and share those links with other colleagues. Users can also manage the dashboard for activities, such as importing and exporting links. With such personalized services, users do not need to repeat the same search and discovery processes each time they visit.

3. Review of Research and Working Group Recommendations

4. Discussion of Challenges and Improvements

Over the years, efforts have been made to improve data discovery by involving data repository practitioners, researchers, and working groups to collaborate on this activity. Recommendations have either been implemented or are being prototyped, as seen in the evolution of GES DISC data services. However, there are still numerous improvements to be made in data discoverability not only for a single dataset but also for multiple datasets. These datasets are often used in interdisciplinary research and applications.

Improving data discoverability involves many factors. Over the years, many rules and recommendations have been developed in previous research and working group reports, presenting different degrees of difficulty in implementation. Some of them (e.g., incomplete metadata and information, lacking standard compliance, federated search) need additional community-level efforts, which may exceed the scope of an individual data repository. For the time being, a more feasible way for data repositories is to further enhance their heuristic capabilities (e.g., providing additional dataset-related information and linking relevant datasets). The following discussion will focus on implementation feasibility.

5. Summary and Recommendations

There are still many challenges in data discovery for interdisciplinary research, applications, and education. Over the years, efforts have been made, but progress has been limited (not as predicted or expected), particularly in the search for knowledge and personalized data services. The main challenges for practices include (1) difficulty in understanding user inquiries; (2) dataset search, other than variable search; (3) limited UI search capabilities (e.g., keywords); (4) missing or incomplete information and data and information not linked in DLP; (5) limited data products that meet users’ needs; (6) missing or incomplete metadata; (7) lack of ontologies and semantic search; (8) difficulties in generating standardized information quality; and (9) lack of adequate interdisciplinary education and training programs. The scope of some of these challenges is beyond the capabilities of one data repository, requiring community efforts to be included in the long-term plan. In the short term, a repository can expand or develop services that allow users to use a hands-on, self-guided, or interactive heuristic approach to discover data and information. Based on the preceding discussion, the following recommendations are made:

1. Provide helpful suggestions in the search box.
2. Improve dataset search to variable search.
3. Enable spatial and temporal ranges and add event search to the search interface.
4. Develop a DLP for an interdisciplinary subject (e.g., air quality, wildfires).
5. Add additional helpful data information services (e.g., FAQs, user forum, data usage and limitations) to enhance heuristic search in DLP.
6. Link relevant datasets and usage information (e.g., publications, applications) in DLP.
7. Provide personalized data services.
8. Have dataset developers provide standardized metadata, including application areas and data quality information.
9. Develop metrics to measure the effectiveness of new improvements.
10. Develop interdisciplinary training and education programs for awareness and workforce development.
11. Develop ontologies and standards for interdisciplinary data services through community efforts.
12. Conduct additional research to develop a better understanding of data quality.
13. Engage a broader community in sharing and discussing best practices.

The first nine recommendations above are not limited to GES DISC and could be considered by other data repositories (e.g., Earthdata) in conjunction with their resources, priorities, and user needs. The last four recommendations require a larger scale of collaboration from scientific communities (e.g., ESIP, RDA). In short, it takes the whole community to improve data discoverability for interdisciplinary research and applications.