2.1 Defining and citing data

The terms ‘data’ and ‘datasets’ have various meanings depending on their context. Often, what becomes ‘data’ is determined by scientists’ choices as they interact with and record observations. ‘Data,’ then, are a byproduct of interpretation and can be practically understood as ‘referring only to that which is analyzed’ (Coombs 1964). Part of this challenge in defining ‘data’ relates to their ‘unruly’ and ‘poorly bounded’ identities, which makes it difficult for them to function as digital objects that can be readily referenced and retrieved (Wynholds 2011). There is also disagreement on the use of the term ‘dataset’ in technical and scientific literature, which presents challenges for data sharing and preservation (Renear et al. 2010). ‘Data’ are abstract arrangements of symbols that express content; ‘datasets’ are made up of multiple data-bearing entities and may contain additional contextual information about data, such as collection methods (Furner 2016; Wickett et al. 2012). In our analysis, we focus on archived social science datasets that include contextual metadata and documentation, which have been produced and shared by others for research purposes.

Capturing the relationships between datasets and other scholarly works is critical for giving credit to datasets. Data ‘references,’ mentions,’ and ‘citations’ signal importance and enable credit through attribution (Altman M, Borgman C, Crosas M, Matone M 2015). While the terms ‘references’ and ‘citations’ are often used interchangeably, ‘references’ generally indicate that the work is listed in the reference section of a publication (Gilbert and Woolgar 1974). The term ‘citation’ implies the use of a persistent identifier (PID), which carries a more formal connotation in bibliometrics than ‘references’ or ‘mentions’ (Ball and Duke 2015). Citations with PIDs link published works to their usage contexts, enabling the verification and reuse of existing scientific analyses (Buneman et al. 2022; Shotton 2010). While authors are beginning to use digital object identifiers (DOIs) to reference data, best practices for when and why they should do so are not widely followed (Mayo et al. 2016). For example, a recent review of literature citing datasets using their DOIs found that many authors cited data that they had mentioned or described (e.g., data collection methods) but which had not been re-analyzed or used in other ways (Banaeefar et al. 2022). In our work, we use the general term ‘data reference’ to cover informal data mentions (i.e., use of a dataset name only), formal citations (i.e., the inclusion of an APA-style citation for a dataset DOI), and descriptions of data use.

2.2 Citations in scholarly communication

Citation analysis within scientific disciplines reveals information flows and brings together separate strands of information to construct ‘consensus models’ of subjects within science (Garvey and Griffith 1972). Citations reflect influences on authors, and citation patterns trace communication across active research networks (Edge 1979). Citations function differently at the micro and macro levels. At the micro level, citations indicate professional relations and function as rewards, while at the macro level, groups of citations function as concept symbols that codify knowledge in hierarchical social networks (Leydesdorff 1998). When cited, papers can be invoked as symbolic of the ideas expressed in their text (Small 1978). Citations are powerful in that they are persistent and take on a separate identity from the people involved in their creation. They are ‘speech acts,’ which are brief statements that endure in documents and can be inspected over time (Smith 2014). Cited sources often substantiate statements or assumptions, point to further information, acknowledge previous research in the same area, and draw critical comparisons indicating the quality of the research (Spiegel-Rosing 1977).

Studies of research infrastructure rely on citations as metrics for tracing attribution and indicating the impact of scholarly works like datasets or software (Mayernik et al. 2017). Institutions, like journals and data publishers, enforce disciplinary and cultural norms for writing style and citation through publishing guidelines and style manuals. Data citations that use specific identifiers allow readers to identify, retrieve, and give credit to research data. Despite recommendations and best practices, formal data citation is still not commonplace in scholarly writing (Mooney and Newton 2012). Incomplete, informal, or improperly formatted citations present obstacles to tracking data use (Zhao et al. 2018). Vague or implicit references, for example, make it difficult for readers without an intimate knowledge of variables or other data features to understand which data the authors used and how they used them (Moss and Lyle 2018). Thus, focusing exclusively on formal citation practices (e.g., using DOIs) means overlooking many potential data references. We seek a more comprehensive understanding of what authors do rhetorically when they formally and informally refer to data in their papers.

2.3 Meaning and motivations for citation

Citations bestow credit and recognition in science (Cronin 1984). However, there may be a disconnect between authors’ citation practices and the use of citations to evaluate performance and measure research impact. In other words, citations indicate what is cited and how often but do not explain ‘why’ works are cited. Authors’ motivations for citing publications can be classified as scientific or tactical. Scientific citations provide background, identify gaps, and establish bases for comparison. Tactical citations acknowledge subjective norms and advertise published work (Lyu et al. 2021). While it is often assumed that citations indicate high-quality work that has influenced authors’ research, a survey found that authors’ citation decisions were more often motivated by strategic factors rather than their familiarity with the research or perceptions of quality (Teplitskiy et al. 2022).

Behavioral surveys and interviews with authors reveal their judgments about what they are citing and why, which are not reflected in the scholarly record (Liu 1993). In one such study, authors considered the recency of publications, their topical specificity, and ease of use when deciding whether to cite them (White and Wang 1997). Authors also believe that citations reflect the prominence or novelty of a document as a ‘concept marker’ and that citing the concept marker will bolster the authority of one’s work, either through alignment or by critiquing existing work (Case and Higgins 2000). Silvello identified six main motivations for data citations that are shared across scientific fields: data attribution (accountability and merit), data connection (to claims in publications), data discovery (identification and retrieval), data sharing (reputational), data impact (assessing exposure), and reproducibility (validation and procedures) (Silvello 2018). These motivations alone, however, do not explain authors’ data-citing behaviors and why they vary across venues and contexts.

2.4 Analyzing citation content and context

Many computational approaches for citation analysis have been proposed, building on prior insights about authors’ motivations to cite. Common citation categories identified across multiple content analysis studies included background information, theoretical framework, prior empirical or experimental evidence, negative distinction, and explanation of methodology (Ding et al. 2014). Features, such as the sections of publications in which citations appear, can be used along with the semantic content of citations to predict citation intent (Nakov et al. 2004). Various classification schemes have been proposed for labeling authors’ intents in citing published research (Hernández-Alvarez and Gomez 2016). One such scheme accounts for citation purposes (i.e., author intent) and polarity (i.e., author sentiment) by distinguishing and weighting negative, neutral, and positive citations (Abu-Jbara et al. 2013). More granular, rule-based coding schemes differentiate statements of weakness, contrasts, or comparisons with other work, agreement, compatibility with other work, and neutral citations (Teufel et al. 2006). Conversely, less granular schemes support general citation intent classification by distinguishing background information, method, and comparison citations (Cohan et al. 2019). Such schemes can also help distinguish citation framing (e.g., uses, motivation, future, extends, compare or contrast, background) (Jurgens et al. 2018).

Qualitative approaches respond to the difficulty of predicting authors’ intents by focusing on the context and features of references. A review of data citations in academic literature found that they varied along two major dimensions: cited entities and styles (Fear 2013). Data producers (the researchers who created the data) and data providers (the people or the institution from which the data were obtained) were often named in data citations. Another recent study found that researchers tended to use data created by others for comparison (e.g., ground-truthing, calibration, and identifying baseline measures) and integration (e.g., to ask new questions and conduct new studies) (Pasquetto et al. 2019). A large survey found that existing data are often used as the basis for a new study, to prepare for a new project, to generate new ideas, to develop new methods, to verify other data through analysis and sensemaking, and for teaching (Gregory et al. 2020). Given that data reuse fulfills diverse needs, researchers’ purposes for citing data also vary. We build upon established insights into data reuse practices to show how researchers give credit to data.

3.1 Sampling frame

We analyzed a sample of publications referencing one or more datasets available through the Inter-university Consortium for Political and Social Research (ICPSR), a large social science data archive at the University of Michigan. We based our typology on existing citation schemes for academic publications, which we extended and refined through iterative coding. The typology captures structural features and rhetorical functions that authors employ when referencing research data. Prior studies have focused on particular publication styles, such as data papers (Jiao and Darch 2020; Li and Jiao 2022), or publication outlets, such as PLoS One (Zhao et al. 2018). Instead, we drew from multi-disciplinary publications that referenced social science data archived at ICPSR. This approach allowed us to capture a wider variety of data reference contexts. We considered data references as they occurred in the full-length context of research publications. Further, our only selection requirement was that each publication mentioned one or more archived social science datasets.

We analyzed papers retrieved as part of ICPSR’s collection efforts to expand the ICPSR Bibliography of Data-related Literature. The Bibliography includes more than 100,000 publications that use existing social science data available through ICPSR. The review process for the Bibliography involves searching bibliographic databases for references to data available through published ICPSR studies. Staff manually review the metadata and full text of publication search results for evidence of data use. ICPSR maintains strict collection criteria to ensure that publications in the Bibliography reflect data use. Publications are collected if they unambiguously refer to one or more studies available through ICPSR and if it is clear that the authors have accessed and analyzed the data. Publications are rejected from the Bibliography if they fail to demonstrate substantial use of ICPSR data or if the specific studies or series used in the authors’ analyses cannot be determined.

To develop a sampling frame for testing our typology, we first identified five publications from the current ICPSR Bibliography representing the multidisciplinary use of ICPSR data. We closely read these publications to identify data references and develop a provisional typology. We then searched an external index of publication full text provided by the Dimensions bibliometric database (Hook et al. 2018) for additional references to any of ICPSR’s 11,639 study DOIs available as of February 2022. With the support of ICPSR Bibliography staff, we evaluated and classified the 2,546 search results into six categories indicating whether the publications met the collection criteria for the ICPSR Bibliography. These categories were proposed by ICPSR staff (Banaeefar et al. 2022) and are defined in Appendix A (Supplementary File 1). We then randomly selected publications across each category to include in our analysis, resulting in a total of thirty publications. We gathered additional metadata for each publication, such as the field of research categories from Dimensions, to determine the disciplinary coverage of our sample. We report the publication sampling frame and selection criteria in Appendix A (Supplementary File 1).

3.2 Qualitative coding

Our team conducted two phases of coding. The purpose of the first phase was to develop a codebook to describe the diversity of data references. To develop and refine our codes, three annotators from our team read the full-text multiple times for each publication labeled ‘phase I’ listed in Appendix A (Supplementary File 1). The annotators independently proposed refinements to a shared version of the codebook. To achieve qualitative reliability, the annotators drew from many examples and discussed them in weekly meetings (Bauer 2000). The annotators met weekly to review and incorporate proposed changes in interactive coding sessions. Emerging ideas and conflicting opinions created a dialog from which the codebook was created. In addition to codes, annotators also discussed the scope of the data references and the codebook’s focus, purpose, and definitions. Each team member independently applied the updated codes for every iteration of the codebook to identify and annotate all data references in the full text. The team repeated this process until saturation was reached and no new codes were proposed (Charmaz 2006). The first coding phase resulted in a stable codebook, which we report in Appendix B (Supplementary File 2).

We also described the extent to which annotators’ coding aligned. Given that the annotators selected and coded segments from the unstructured, full text of publications, we used Holsti’s Index (1969) as an agreement measure. The final Holsti Index was 67.9%, indicating a relatively high level of agreement, given that annotators selected different text segments, which they coded with multiple codes. A single team member independently applied the typology to the held out set of twenty-five publications labeled ‘phase II’ in the sampling frame reported in Appendix A (Supplementary File 1). This second phase demonstrated the typology in action and captured findings shared in Section 4.

4.1 Features of data references

Features describe the structure, form, and appearance of the Data Reference. The twenty-four features of data references that we identified are organized under five subcodes (Access, Action, Location, Style, and Type). Access codes indicate whether the author describes data sharing or retrieval in the reference. Action codes capture the distance between the author and the data along a continuum covering ‘citing’ (i.e., parenthetically referencing data without further context), ‘mentioning’ (i.e., describing or alluding to data), and ‘using’ (i.e., describing active, hands-on work with data). The Action codes build on the distinction proposed by Pasquetto et al. (2019) between comparative and integrative data reuse. The Location code notes the section of the publication in which the Data Reference occurs, such as the abstract, acknowledgments, captions, figures, tables, footnotes, or methods sections stated in the expanded IMRAD structure (Sollaci and Pereira 2004). The Style code captures how the author specifies data entities through the use of an acronym such as ‘ANES,’ a generic noun such as ‘data’ or ‘study,’ a formal name such as the ‘American National Election Study, 2016,’ or a parenthetical citation using author and year of publication. The Type code captures whether the data entity is derived from existing data, represents a primary source created by the authors, or is a secondary source published for other researchers to use.

4.2 Functions of data references

Functions reflect the purposes of each Data Reference. The ten rhetorical functions of data references we identified are organized into five subcodes (Critique, Describe, Illustrate, Interact, and Legitimize). Definitions and examples for each code and subcode are provided in Appendix B (Supplementary File 2).

The Critique code includes (1) Comparison, which contrasts the author’s work with other work that uses the data or findings from other sources. Authors issue Comparisons to draw a contrast between their work and prior findings or to summarize conclusions drawn from the prior use of data. The Critique code also includes (2) Limitations, which signal authors’ awareness and caution when working with data. This code includes acknowledging quality issues, such as potential errors or sampling biases that should limit how data are used.

The Describe code includes (3) Composition, which explains or discusses knowledge about the data or metadata. Composition data references describe what is in the data (e.g., the sampled population) or the study’s context (e.g., the data collection method). The Describe code also includes (4) Source, in which authors describe the provenance of the data. Source references acknowledge the origin of the data associated with the data producer or provider.

References labeled with the Illustrate code are persuasive. This code includes (5) Context, in which the author provides background, findings, or statistics derived from referenced data. In a Context reference, the data are metonymic, standing in for the point or claim that the authors are making. The Illustrate code also includes (6) Outlook, in which authors speculate on potential applications of data that they did not conduct or review in their work. Outlook references claim the potential utility of data based on their properties.

The Interact code describes hands-on work with data. Interact includes a subcode for (7) Interpretation, where authors make an empirical claim derived from the analysis of referenced data. Interpretation references often follow the description of the authors’ analysis. The Interact code also includes a subcode for (8) Manipulation, where authors describe steps performed while working with data. Manipulation involves selecting variables, preparing or transforming data for analysis, and specific data preparation techniques like sampling, correlating, integrating, and validating analyses.

Finally, the Legitimize code is used for references intended to persuade the reader through value statements made about data. The Legitimize code includes (9) Justification, which draws attention to a feature of data that lends credibility or authority to the authors’ choices. Examples of Justification references include authors’ reasons for why data were selected, discussions about the credibility or representativeness of data, and descriptions of previous data uses that qualify their selection. The Legitimize code also includes a subcode for (10) Transparency, in which authors explain why or how an analysis procedure was applied and signal quality or considerations taken in the analysis. Examples of Transparency references include making methods open or reproducible by including analysis in supplementary materials.

5.1 Limitations and future work

This study proposes a typology that models how authors reference research data. We developed the typology by closely reading papers from the ICPSR bibliography and adding new categories until we reached saturation. The present analysis is not intended to provide quantitative evidence for specific citation trends. We would need to conduct annotation at a larger scale with additional measures in place to verify the agreement of annotators. In addition, we constructed our sampling frame by selecting papers that were first reviewed and classified by experts (i.e., ICPSR Bibliography staff); the sample is balanced across the categories provided in Appendix A (Supplementary File 1). While this sampling strategy is useful for developing and analyzing the ICPSR Bibliography, future uses of the typology for other purposes may require different selection criteria.

We envision applying our typology to study differences in data references across social science disciplines (e.g., sequences or co-occurrences of data reference strategies as markers of scientific disciplines or analytical methods). A recent study of data citation practices at ICPSR observed unexpected uses of dataset DOIs in published literature, which did not indicate data use (Banaeefar et al. 2022). Our typology can be used to study when and why researchers use dataset DOIs and distinguish references that describe data from those that imply data analysis.