1 Introduction

Builders and funders of data infrastructure always bear the risk that subsequent use and impact will fall short of expectations, calling into question what are often substantial up-front investments. In this article we use survey results of geoscientists and cyberinfrastructure experts to assess potential barriers for the U.S. National Science Foundation’s EarthCube initiative, which is advancing the cyberinfrastructure for geoscience. The goal of EarthCube is to serve the geosciences by integrating unique data sets, isolated repositories, separate models, and relevant software through functional tools, thus allowing discovery and reuse of diverse geoscience data. Expanding open sharing and data discovery supports replicability and integrity in science, enables new frontiers of research within and across disciplines, and informs policy decisions. Such integration will likely accelerate progress on grand challenges, such as global climate change, severe weather prediction, natural resource discovery, and, ultimately, understanding the earth as a system. Three potential barriers are assessed: lack of agreement on the need for the infrastructure, lack of support for engaging with the infrastructure, and a lack of balance in engagement of diverse communities.

Because the EarthCube initiative is substantial in scope and vision, a decade or more will be needed for development and operation. EarthCube’s success depends on the engagement of all geoscience fields (we use this term to indicate disciplines as well). The community needs to develop a culture in which physical samples, data on physical samples, streaming data from sensors, visual images, data models, and other aspects of geoscience data are curated, shared, and reused. Indeed, there is public policy support for this sharing and reuse (OSTP Public Access Memo, 2013). EarthCube’s success further depends on expanding workflows to provide appropriate information on data provenance, systems tracking data reuse and providing credit for reuse (comparable to citation counts), universities valuing such behaviors to a much greater degree in promotion and tenure decisions, and appropriate industry partnerships. In addition to the technical infrastructure, data sharing success depends on increasing levels of trust and cooperation across diverse geoscience fields. EarthCube is one of a number of data sharing initiatives emerging to facilitate work across diverse disciplines and public engagement in science. Some of these are long-standing, such as CODATA, and others are newer, such as CyVerse (formerly iPlant), the iSamples initiative, the Materials Genome Initiative (MGI), the National Data Service (NDS), and the Research Data Alliance (RDA).

Although the potential value of EarthCube is clear, success is not assured. In 2012, one year after the formal launch, a second launch conference, termed a “charrette,” engaged over 200 experts in road-mapping exercises. However, the participant mix included many more cyberinfrastructure builders than geoscience end-users. At the conclusion of the session a dozen NSF leaders and session facilitators met and identified seven potential failure modes of EarthCube.¹ One of these was viewed as particularly problematic. It was stated as follows: “The ‘build it and they will come’ mindset – in which users don’t show up, data are not shared.” In this article we present survey data exploring three potential barriers relevant to this possible failure mode. These potential barriers are I. Geoscientists do not perceive the need for change; II. Geoscientists do not have support for engaging in change initiatives; and III. The disciplinary mix of engaged users is not appropriately balanced. Importantly, once addressed, the barriers can become drivers, helping to advance EarthCube. These barriers/drivers are likely common across similar projects in all disciplines involving what is sometimes termed big data.

2 Build it and they will come

The epigram “build it and they will come” or the negative formulation “build it, but they don’t come” has entered common parlance in domains as diverse as information systems design (Markus & Keil, 1994); marketing management (Henrix, 1999); medical services (Tintinalli, 2008), and organizational learning (Yuan, et. al., 2010). In these and other cases, the concern is that target audiences do not come – in contrast to the popularization of the concept in the novel Shoeless Joe (W.P. Kinsella, 1982) and the 1989 movie adaptation, “Field of Dreams,” where faith was rewarded.”²

In the case of EarthCube, the builders include cyberinfrastructure experts, data managers, software engineers, data scientists, and others. The “they” who might or might not come are geoscientists in a broad array of fields. Supporting and conducting research on users as diverse as these is difficult since it involves reference groups that are not easily specified (Lawrence, 2006), though they may be commonly recognizable.³

Divides between technologists and scientists have been documented in a wide range of geoscience settings (Mayernik, Wallis, & Borgman, 2013; Jackson & Buyuktur, 2014; Finholt and Birnholtz, 2006; Ribes and Finholt, 2008). These divides range from challenges in developing effective collaborative structures and aligning incentives, to difficulties in establishing leadership roles and determining who the relevant “community” actually is for a given cyberinfrastructure initiative. For example, Finholt and Birnholtz (2006) note that domain scientists likely will be more successful in leading cyberinfrastructure initiatives than technical experts, since they can marshal participation by their scientific colleagues. In “build it and they will come,” the “build it” part is more certain (and the costs are paid for). Whether “they will come” is more uncertain. At a time when broader impacts are expected of science and of the investments that support science, both parts of the statement are essential. Our large-scale survey was designed to identify key factors in the success of the NSF EarthCube initiative. The idea of barriers and drivers to system change was initially advanced by Kurt Lewin (1939) who noted that barriers, once addressed, often become drivers. He also noted that reducing barriers produced greater leverage in the “force field” than pushing harder on the drivers.

Further underlying the idea of “build it and they will come” is the expectation of value through what are termed “network effects” (Katz and Shapiro, 1994). If people from a sufficiently broad network share and discover data, the system will work. If not, the system is at risk of not having a self-sustaining critical mass of users.

3 Background on EarthCube

The NSF initiated EarthCube in 2009 and charged it with creating the cyberinfrastructure for advancing regular science and addressing grand challenges in the geosciences that involve understanding the earth as a system. In October, 2011, EarthCube conducted its first charrette, which drew over 150 participants. In February, 2012, the NSF commissioned what was termed a “stakeholder-alignment survey” for the second charrette in June, 2012. The survey was designed to indicate points of alignment or misalignment among geo and cyber stakeholders, drawing on methods developed under a prior NSF research grant (NSF-VOSS EAGER 0956472). Although the second charrette drew even more attendees, the high proportion of computer and data experts and the initial survey data both pointed to a need for increased engagement of geoscientists. In response, the NSF made a strategic pivot to solicit proposals for geoscientist-led disciplinary workshops on data sharing. Before each workshop participants were invited to complete the stakeholder alignment survey.

A further impetus for EarthCube emerged on February 22, 2013, when the U.S. Office of Science and Technology Policy (OSTP) directed federal executive offices and agencies to make the direct results of federally funded scientific research available. The directive stated, in part:

The Administration is committed to ensuring that, to the greatest extent and with the fewest constraints possible and consistent with law … the direct results of federally funded scientific research are made available to and useful for the public, industry, and the scientific community. Such results include peer-reviewed publications and digital data…. These policies will accelerate scientific breakthroughs and innovation, promote entrepreneurship, and enhance economic growth and job creation.

This was followed by similar directives on January 1, 2013 from the Australian Research Council and on May 24, 2013, from the UK Research Councils. Most recently, following the end-user workshops and the data collection, EarthCube has developed a system of governance for community engagement and cyberinfrastructure development. Here, we present baseline data collected in advance of the governance and more recent activities.

4 Data Sources

The stakeholder-alignment surveys were designed to enable evidence-based decision making for EarthCube and to provide a baseline for later surveys every 12–18 months. The initial survey was developed in March and April 2012, approved by the University of Illinois Institutional Review Board, and administered via a web-based URL. Questions asked cyberinfrastructure developers and geoscientists for their views of the EarthCube concept and indications of their involvement. Comparing responses across groups indicates the current degree of alignment among builders and potential users and helps identify the barriers to increased alignment that can jeopardize the EarthCube initiative. In this case, scientific fields and professional domains are the stakeholders.

Initially 167 out of 900 website registrants completed the survey. Respondents included a large group of cyberinfrastructure builders. Subsequent outreach using lists from NSF data centers yielded an additional 578 responses, out of approximately 10,000 requests, with a higher proportion of geoscientists (with 1 unspecified).

Beginning in the fall of 2012, throughout 2013, and into 2014, participants at 23 disciplinary workshops also completed the survey. In January 2013, the survey was revised (after the first five workshops). Responses from all workshops were received from 824 of the 1,828 invitees (spanning versions 1 and 2). We used the entire sample of 1,569 respondents in our analysis (167 + 578 + 824). The workshop response rates are indicated in Table 1. The invitees to these workshops were a combination of thought-leaders identified by NSF program officers, individuals contacted through professional associations and societies, and early career scientists. The fields featured at the workshops were based on competitive workshop proposals submitted to the NSF.

We use descriptive statistics, bar charts, a new data visualization format, graphing, and OLS regression modeling to assess potential barriers. In the bar charts and the regression analyses we use the fields and disciplines that respondents selected from a taxonomy included in the survey (different than the workshop titles listed above). Many survey questions use a 16-point scale, which is transposed to a scale from zero to one for ease of interpretation. Both survey instruments included voluntary consent language and received IRB approval at the University of Illinois, Urbana-Champaign. The data is available as DOI: http://dx.doi.org/10.1594/IEDA/100535 (Title: EarthCube Stakeholder Survey Data; Date Available: 2015-04-15; URL: http://dx.doi.org/10.1594/IEDA/100535). In addition, an interactive tool for exploring the data, allowing for two-way charting and other forms of exploratory analysis, is available at: http://maxim.ucsd.edu/suave/survey1544.html.

The stakeholder alignment approach is modeled on previous uses of systematic stakeholder data to inform complex multi-stakeholder initiatives including a 2005 report to the U.S. Congress on aircraft noise and emissions (Waitz, et. al.) and NSF-funded studies of the BioMarkers Consortium, the U.S. Fab Lab Network, and local community green energy (NSF-VOSS EAGER 0956472). Similar stakeholder alignment surveys have been conducted for the National Data Service, the i-Samples initiative, and other complex multi-stakeholder projects. We define stakeholder alignment as “the dynamic process by which interdependent stakeholders orient and connect to advance separate and shared interests.” In this article, scientific fields and related technical domains are all treated as stakeholders to the cyberinfrastructure supporting open data for science. The needed alignment is not a fixed end point, but an ongoing process involving many shared and separate interests.

5 Potential Barrier I: Geoscientists do not perceive the need for change

To assess the need for change, respondents were asked four key questions about both the importance and the ease of finding, accessing, and integrating data, models, and software:

• How IMPORTANT is it for you to find, access, and/or integrate multiple datasets, models, and/or software (e.g. visualization tools, middleware, etc.) in your field or discipline?
• How EASY is it for you to find, access, and/or integrate multiple datasets, models, and/or software (e.g. visualization tools, middleware, etc.) in your field or discipline?
• How IMPORTANT is it for you to find, access, and/or integrate multiple datasets, models, and/or software (e.g. visualization tools, middleware, etc.) that span different fields or disciplines?
• How EASY is it for you to find, access, and/or integrate multiple datasets, models, and/or software (e.g. visualization tools, middleware, etc.) that span different fields or disciplines?

The mean and standard deviation for importance within fields is 0.87 +/– .20, while the result for ease within fields is 0.41 +/– .25. This same gap is found with respect to perceived importance across fields (0.76 +/– .26) as compared with the perceived ease of doing so (0.30 +/– .23).

For the EarthCube participants to better view the data, we used a newly developed format, termed a z-flower. Each respondent is assigned a color-coded hexagon with shades of green for positive responses (darker is more positive), shades of yellow for neutral responses, and shades of red for negative responses, and the white hexagons represent those who didn’t answer the question. The responses are then tiled in a spiral with those close to the mean in the middle (indicating the central tendency) and those farthest from the mean on the outside (the outliers). This heightens the contrast between importance and ease (Figure 1).

The combined responses (Figure 2) are similar and consistent across fields. Overall, the consistent gap between ease and importance, which was greater than 0.5 across nearly all fields, supports the perceived need for the EarthCube initiative. Also the gap across fields is slightly but consistently larger than within them. These data quantify the frustrations of researchers in sharing or obtaining data in their fields and across.

Other responses reinforced the above findings. For example, respondents were primarily neutral or negative, with a few positive “bright spots” to the statements: “There is currently a high degree of sharing of data, models, and software among geoscientists” and “There is currently a high degree of sharing of software, middleware and hardware among those developing and supporting cyberinfrastructure for the geosciences.” Respondents held more negative perceptions to the statements: “There is currently sufficient communication and collaboration between geoscientists and those who develop cyberinfrastructure tools and approaches to advance the geosciences” and “There is currently sufficient geoscience end-user knowledge and training so they can effectively use the present suite of cyberinfrastructure tools and train their students/colleagues in its use” as is illustrated by Figure 3.

Clearly, getting access to others’ data is seen as important, but not easy. Respondents perceive a need for change, which can serve as a driver (rather than a barrier) for EarthCube. This is promising for EarthCube.

6 Potential Barrier II: Geoscientists do not have support for engaging in change initiatives

A second potential barrier to the success of EarthCube arises if geoscientists perceive that their employing organization, such as a university or data center, and professional colleagues do not support engagement with EarthCube. Two questions in the survey addressed these forms of support:

• My employer/organization will most likely value and reward any efforts I make in the shaping and development of EarthCube.
• Any contributions I might make to the shaping and development of EarthCube will likely be recognized and valued by colleagues in my field/discipline.

We combined these two items into a single EarthCube support scale (Cronbach’s alpha = 0.79).

The results of the initial survey suggested that the lack of such support could indeed be a barrier to EarthCube’s success. Thus, two additional questions were added to the survey in January 2013 to see if this was specific to EarthCube or a more general barrier around interdisciplinary science:

• My employer/organization will value and reward efforts I make in bridging across fields and disciplines.
• Efforts that I make to bridge across fields and disciplines will most likely be recognized and highly valued by colleagues in my field/discipline.

We combined these two items into a single interdisciplinary support scale (Cronbach’s alpha = 0.75).

The responses (Figure 4) indicate only moderate perceived support on average for engagement from the respondent’s employing organization (0.48 +/– .31) and colleagues (0.53 +/– .27) across diverse fields. Moreover, the standard deviations are relatively large, indicating considerable variation within the geosciences. Note that these responses do not include the first version of the survey in order to compare the responses with the next question discussed (on support for interdisciplinary science), which was only added in the second survey.

Respondents’ support for engagement in interdisciplinary science is more positive on average at 0.64 +/– .24 than support for engagement with EarthCube. This is consistent with their perceived support from employing organization and 0.64 +/– .25 from colleagues.

In general, driving change is more difficult when there is a high degree of variability (Deming, 1986), suggesting that initial engagement of researchers in EarthCube will be constrained by the lack of strong and consistent support from their home organizations or colleagues. As a baseline assessment of this potential barrier, the results are mixed. Some end-users report support from their home organization and colleagues, but in most cases support is not strong. This suggests that institutional infrastructure will need attention for EarthCube or any cyberinfrastructure initiative to achieve full success. In a separate working paper (Stakeholder Alignment Collaborative, 2015), we focus more deeply on the underlying theory of internal support needed within fields to engage in multi-stakeholder collaboration.

7 Potential Barrier III: The disciplinary mix of engaged users is not appropriately balanced

To assess the degree to which geoscientists are engaged with EarthCube, we use a bivariate figure and multivariate analysis. Respondents’ degree of engagement in the EarthCube initiative, using a 6-point ordinal scale, is the anchor for the bivariate analysis and the dependent variable for the multivariate analysis:

• 1 = I have heard of EarthCube (10.2%)
• 2 = Aware of EarthCube, but no engagement (33.7%)
• 3 = Visited the website (16.5%)
• 4 = Participated in discussions (17.4%)
• 5 = Actively involved (16.7%)
• 6 = Leadership role (5.1%)

Increasing values represent increasing degrees of engagement (0.4% non-responses are excluded). We don’t know the degree to which engagement in EarthCube is representative of other open data or data sharing initiatives. However, increased sharing and reuse of data will involve a mix of individual actions and collective initiatives that involve increasing degrees of engagement. Because EarthCube was still in early stages of formation at the time of data collection, the overall averages indicate that most participants were just at the level of awareness. The bivariate analysis in Figure 5 shows engagement with EarthCube on one axis and support for interdisciplinary science from the respondent’s employing organization or institution on the other axis (a scale from 0–1). Only the relevant portions of each scale are used in this figure and the means are marked as mid-points on each axis. Contour lines are added in the open spaces between clusters of fields and disciplines; they are added for illustrative purposes.

Figure 5 suggests that cyberinfrastructure builders (above the first contour line) have more support and engagement with EarthCube than end-users. This is reasonable: the concept “build it and they will come” states up-front that it will be built. Those who build the infrastructure know they will be paid. Whether “they” – the end-users – will come is an open question. The risk is not in building something. Often it is impossible to tell whether “they will come” until something is built. The risk is that the built artifact will not be what is needed. The second contour points to fields with the greatest initial risk.

Multiple regression was used to test a multivariate model in which the control variables are experience, gender, and international or domestic home organization. To assess the balance of engagement among end-users across fields we focus on a set of traditional geoscience disciplines (atmospheric/space weather science, geology, geophysics, hydrology, and oceanography) and a set of interdisciplinary scientific domains (bio/ecosystem science, climate science, critical zones, physical geographers, and a combined category for other science domains). A set of “builder” domains (computer/ cyberinfrastructure science, data management, software engineering, and a combined category for other expert domains) are also included. Three regression models are estimated, each with one category excluded (the coefficients on the other categories are interpreted relative to the excluded category). This allows for models that are in reference to an established field (geologist), an interdisciplinary domain (bio/ecosystem scientists), and a builders’ domain (cyberinfrastructure experts).

Beginning with the control variables in Table 2, all three models show that more experienced researchers tend to be more engaged with EarthCube at this baseline. This is good news in that EarthCube is attracting experienced individuals at the outset who can provide important leadership. Over time, however, it will be a concern if this isn’t followed by engagement of the next generation (another potential failure mode). Gender is not a factor with respect to engagement in EarthCube, which is a favorable finding. EarthCube is a U.S. based initiative, and researchers affiliated with a U.S. organization or institution are more positively and significantly engaged with EarthCube than respondents outside the U.S. Over time, however, success will depend on broad international participation.

Turning to the fields, Model 1, has geology as the excluded variable. Geology was selected as a well-established field with particular challenges in the sharing and reuse of physical samples. Thus, the regression coefficients should be interpreted relative to geology. We don’t specify a hypothesis since geologists could be expected to be more involved in EarthCube since the need is greater or less, since sharing and reuse is difficult. Among other traditional disciplines atmospheric/space scientists and oceanographers are significantly less likely to be engaged than geologists. Among the more interdisciplinary domains, climate scientists are less likely to be engaged. Interestingly, all three are areas of science that use sensors and modeling, which are challenging to share and reuse. Two of the three “builder” roles (computer/cyberinfrastructure scientists and data managers) are more likely to be involved. In Model 2, bio/ecosystem science is excluded as an illustrative interdisciplinary domain and the results are comparable even though they are in reference this time to bio/ecosystem science.

In Model 3, computer/cyberinfrastructure science is excluded as the lead builder domain. Here all roles other than software engineers are significantly less likely to be engaged in EarthCube than the computer/cyberinfrastructure scientists. Model 4 adds two gap variables (gap between importance and ease) from the first barrier (perceived need for change) and the support variable from the second barrier (support for engaging by colleagues and employing organization).⁴ Holding constant the control variables and the various fields, we see that the interdisciplinary gap between importance and ease is a stronger driver of engagement than the gap within fields. Thus, sharing and reusing data across fields and disciplines is a stronger driver than sharing and reuse within a given field. We also see that internal support for engagement with EarthCube is a positive and significant predictor of engagement. Even though the levels of support (from the employing organizations and from colleagues) are relatively low, the workplaces where there is support do have a positive and significant impact on engagement. Thus, atmospheric/space weather science, oceanography, and climate science are less likely to be involved than other geoscience and interdisciplinary domains while builders of cyberinfrastructure are much more likely to be involved in EarthCube at the baseline – indicating that builders are initially more engaged than potential end-users.

8 Discussion

The gap between importance and ease in data sharing is a potential driver for engagement in EarthCube. The absence of such a gap would be a major barrier for any cyberinfrastructure initiative. This gap is consistent across geoscience fields, as well as among cyberinfrastructure experts, software developers, and data managers. Of course, end-users will only engage in EarthCube if they see it as an effective way to close that gap.

The relatively limited support from employing organizations and colleagues more broadly that is evident in the discussion of barrier two and in regression model 4, is a challenge for EarthCube. The results indicate that the issue is more pronounced with support from the employing organization than with colleagues. If the organizational and institutional infrastructure is not aligned in support of data sharing, then progress will be constrained. Although we did not ask respondents what obtaining such support would require, this was a focus of discussion in the various end-user workshops. Participants pointed to the need for systems to provide credit for reuse of data, similar to citation counts for publications, so that data sharing could be taken into account in tenure and promotion decisions for university faculty and career advancement for professional staff in data facilities. For developers of software, models, and other data products, parallel credit is also important and it will be important to better understand if there are such credit mechanisms in place where people do report strong organizational and institutional support. Participants also highlighted the need for funding, time and other forms of support for attaching metadata to aid in the discovery and reuse of data.

It is not surprising that there would be a gap between ease and importance in the early stages of this initiative. Yet addressing the gap will not be easy since institutional change involves changing institutional leaders’ and scientists’ attitudes and behaviors towards data sharing. A small example of progress along these lines was evident in the end- user workshop for early career scientists. Sixty-eight doctoral students, postdocs, and assistant professors in their first few years participated in the workshop. (They were among the end-user workshop participants surveyed in the stakeholder survey). Afterwards, the participants were offered the chance to have a letter written to their department chair and dean indicating the strategic importance of the workshop and the merits of having been invited to attend. Most of the participants indicated an interest and many reported positive one-on-one conversations with institutional leaders afterwards. Much more is needed for institutions to be seen as valuing data sharing, but this example illustrates that the potential barrier is not insurmountable.

Finally, the evidence clearly indicates that the initial engagement in EarthCube is stronger for cyberinfrastructure experts, software developers, and data managers than it is for geoscientists. There can be carrots and sticks to motivate both the “build it” and the “they will come” aspects of open data initiatives. With sufficient carrots, builders will develop the cyberinfrastructure, which they have illustrated by their early participation in EarthCube. The National Science Foundation responded to the risk of builders getting too far out front by adding end-user workshops. More recently, the EarthCube governance structure has provided explicit avenues for the engagement of scientists – particularly around the identification of “use cases” to guide the development of the cyberinfrastructure. Observers at EarthCube governance sessions, including members of our research team, note that there quickly surfaced widely different definitions of what constitutes a use case. There is emerging recognition that there is value in diverse types of use cases and scholarly recognition for a more systematic and modular approach to use cases (Jacobson, Spence, Kerr, 2016). Beyond seeking greater stakeholder engagement, it is important to advance the enabling mechanisms, such as the utilization of use cases. In further research and future rounds of data collection, it will be important to track the degree to which the engagement of the end-users is expanding and generating actual data sharing behaviors.

9 Conclusion

This analysis of the EarthCube initiative in its early stages reveals there is a substantial gap between the perceived importance of data sharing in the geosciences and the perceived ease of doing so. Rather than being a barrier, this is a potential driver for EarthCube – particularly since the gap is consistent across a wide range of fields and domains. However, while assessment of builders’ and end-users’ needs is necessary, it is not sufficient.

Also important is organizational and institutional support for end-users to engage in the initiative. The baseline data presented here suggest that end-users view organizational and institutional support as moderate. This is important to address because low support may reduce scientists’ interest in participating.

While organizational and institutional change doesn’t happen easily, each step in the direction of lowering barriers not only removes a barrier but produces an additional driver. Given the requirements of the 2013 OSTP data sharing memo and other forces driving increased data sharing, these organizational and institutional considerations pose a crucial set of challenges for data science.

Finally, the data show that in this early stage, builders are more engaged than end-users. This represents an important risk to EarthCube’s success. Builders are essential, but the initiative needs comparable engagement and even leadership from the end-user scientists. Thus, while it seems likely that the data and cyberinfrastructure experts will “build it,” the real concern is that geoscientists “will not come.”

All three potential barriers – perceived need, perceived support, and distribution of engagement – represent key baseline indicators that inform ongoing efforts to avoid a key failure mode in cyberinfrastructure initiatives. The NSF is not blindly building EarthCube, hoping that the geoscientist will come and share data, tools, and models – there is clear evidence of a perceived need. The challenge now is to progressively lower the barriers and increase engagement in the building process, so that as it is built and as it evolves, they will come.