Background

Climate and environmental reconstructions from paleo proxy records often emphasize collections of proxy records. Collections expand the spatial and temporal extent of the data, reduce noise, and can enable the reconstruction of multiple climate variables. Following the pioneering multi-proxy temperature reconstruction by Mann et al. (1998), many different reconstructions, based on unique and overlapping collections of proxy records, have been assembled. Most reconstructions have focused on the Northern Hemisphere land surface where observations are concentrated (Mann, et al. 2008; PAGES 2k Consortium 2013; Masson-Delmotte et al. 2013). Recent compilations have expanded networks in the Southern Hemisphere (Neukom et al. 2014), the oceans (Tierney et al. 2015), the Arctic (McKay and Kaufman, 2014), and the globe (PAGES Consortium, 2017). Compilations of moisture-sensitive proxies are being developed using multiple proxies (Smerdon et al. 2017) and tree ring databases (Cook et al. 2009; Cook et al. 2010). Some of these moisture-sensitive chronologies are included here through incorporation of the Breitenmoser et. al. (2014) collection. Previously, the Last Millennium Reanalysis (LMR) (Hakim et al. 2016; Emile-Geay et al. 2018) used the PAGES 2k Consortium (2013) temperature-sensitive multi-proxy database, and subsequently used the PAGES 2k Consortium (2017) update (consisting of 571 records) archived at NOAA (PAGES Consortium (2017a). Additions were sought that would expand the geographic coverage and the sensitivity to different environmental variables. The additional data should be amenable to the development of quantitative proxy system models, including proxy error variance estimates needed for assimilation. Moreover, the data must be structured (i.e. consistently formatted) so that thousands of records, each consisting of tens to hundreds of years of observations, can be parsed by the assimilation algorithm. For these reasons we developed a method (and available code (Table 1)) to quality-control, standardize, and structure a set of records consisting primarily of existing (published) paleo proxy time series.

Our goal is to produce a collection of paleoclimate proxy time series, in their original data units, with minimal processing, add additional metadata about the series characteristics, and provide structured time series in individual files for reanalysis, reconstruction, proxy system model development, and other applications. The additions described here are merged with a collection of 571 records described previously (PAGES 2k Consortium 2017) in a processing step of the LMR code to create a comprehensive, integrated and assimilation-ready proxy database termed LMRdb v1.0.0. The scope of the additions is broad and, in contrast to the PAGES 2k Consortium (2017) data, minimally screened (screening refers to inclusion or rejection of series based on criteria). We added proxies from five archives (tree rings, corals and sclerosponges, ice cores, lake cores, and speleothems (Table 2)). The original series are annually, sub-annually, or super-annually sampled. Short records (less than 30 years length) are included and the maximum length spans the Holocene (10,000 years). All of the records are hypothesized to have some relationship with environmental variables that can be quantified as either an empirical linear univariate or bivariate regression or a physically based proxy system model. The LMR framework includes a data pre-processing step (which can include screening or filtering records based on certain criteria), and a proxy system model step. The model step develops a statistical model relating proxy observations to environmental conditions and quantifies uncertainty in observations required during the assimilation step. The code is designed so that this statistical approach can be replaced by more sophisticated proxy system models. In the final assimilation step the proxy observations are compared with estimates derived from the prior state using the proxy system model. The additional records described here are all-encompassing, from many proxy sources and locations, of varying length and resolution, minimally-screened, for the broadest application. The files are structured and archived as NOAA-formatted files (NOAA Paleoclimatology 2018) to ensure that sufficient metadata accompany the records so that transformations to other formats, such as LiPD (McKay and Emile-Geay 2016), are readily achievable. The original references appear in the header of individual proxy files and should be cited when a record is used.

The selection criteria for the additional records are guided by the needs of the LMR and by the needs of proxy system model development (Dee et al. 2015; Evans et al. 2013). The LMR project aims to make code and data publically available to facilitate use and reuse, thus only publicly accessible or already-archived data are considered. In selecting data to add we targeted data for which forward proxy system models have already been developed, including tree rings (VS-lite; Tolwinski-Ward et al. 2011), corals (Thompson et al. 2011), and ice cores (Dee et al. 2015). In addition to their use in assimilation, availability of proxy data can improve proxy system model development (Dee et al. 2015). Two recent paleo data assimilation efforts (Hakim et al. 2016; Franke et al. 2017) have been designed to use proxy data that are annually or better resolved and that overlap with the instrumental period, over which required characteristics such as the error can be assessed and regression models developed. A third assimilation effort (Steiger et al. 2018) used a draft version of the data described here. The additions focus on proxy time series that overlap the instrumental period. All three assimilation efforts are designed to analyze many of the variables within the climate system that covary with proxy records, encouraging incorporation of proxies sensitive to moisture availability, sunlight, surface pressure, water isotope ratios, and other aspects of the climate.

Data Processing

The workflow consists of 1) adding metadata and information to candidate records needed to meet NOAA archive requirements and structuring the data time series and metadata in the NOAA text format and, and 2) preprocessing the data to identify duplicates and errors, standardize variable names, and average the data to annual values. The workflow is described in the following steps followed by archive-specific notes.

Step 1. Obtain proxy time series and metadata and format in concordance with the standard developed for NOAA text files (NOAA Paleoclimatology Program 2018). Additional metadata and notes were added but the data were not altered. The Breitenmoser et al. (2014) chronologies and metadata were obtained from the authors as Matlab files. The sensitivity of a series to the environment during a specific season was provided by PAGES 2k Consortium (2017) authors, assigned by us for the Breitenmoser et al. (2014) trees (described below), or obtained if it existed from the original metadata.

Step 2. Process individual files. Identify and eliminate duplicates (see notes below): translate the original archive types and variable names to LMR types and names, compute annual January-December means (optionally April–March), optionally gaussianize the time series, make coarse checks on the time variable, standardize the time variable to Year C.E., read metadata (site name, latitude, longitude, elevation, investigators, variable attributes (including variable name and units)), read the numerical data (age and values), check the earliest year and most recent year metadata fields with the numerical data, arrange series from oldest to youngest, and identify data as missing (e.g. data reported as NaN) for years where valid data are not available. The guassianize option is provided because some paleo proxies, notably lake records, have a baseline zero value with excursions from zero which are not successfully modeled by a linear regression. Ranking the data by point value and transforming so the distribution has a mean equal to zero while preserving the rank can improve the linear regression model. Data frames are then built consisting of 1) metadata and 2) numerical data (i.e. the actual proxy time series). The data frames are available from the LMR project documentation web pages (Tardif et al. 2018).

A critical step in compiling heterogeneous paleo data from multiple sources is to determine which variables represent the same quantity, and can be treated identically by proxy system models. The identification of similar variables is listed in Table 2, and accomplished in the processing step, leaving the variable names in the archive unchanged. In cases where suspect data or metadata were found the data series was rejected. No data values were changed. A Python program labeled LMR_proxy_preprocess.py performs this step. The processed data are not archived because many different processing options may be selected. Instead, the raw data are archived and the processing code made available (Hakim et al. 2018). The approach facilitates adding additional raw data that may be processed in the same way. A working version of the processed data is available for use with the LMR code (Tardif et al. 2018). The production version of the code, including the preprocessing code, is available from GitHub (Hakim et al. 2018).

The primary emphasis of the processing step and its description here is to document how the data are transformed, and how heterogenous time conventions and variable names are treated so that disparate data sets produced by different authors can be made amenable to data assimilation, while preserving the original characteristics of the data in the archive. These transformations involve several choices regarding processing that may be changed depending on the application.

Archive-specific Processing

Data Description and Linear Regression Summary

The geographical and temporal distribution of the additional data and the complete database are shown in Figure 1. The archive types are unevenly distributed in time and space. Coral records (27) dominate the tropical ocean, ice core records (88) dominate at latitudes greater than 60 degrees, and the middle northern latitudes are dominated by tree rings. The Breitenmoser et al. (2014) chronologies expand the previous number of records four-fold. The additions are dominated by trees from the Northern Hemisphere mid-latitude continental areas. Going backwards in time the data density decreases rapidly from ca. 1875 to a much smaller number of sites prior to 1600, however some of the sites with data at 1600 also have observations back to 1 CE. The addition of the Breitenmoser et al. (2014) chronologies particularly increases the number of observations in the 20th century. Tree ring chronologies decrease in number in the latter part of the 20th century because many of the chronologies were collected prior to 1980. There are less than ten series within each of the following archive variable categories (Table 3): tree ring isotopes, bivalve δ¹⁸O, ice core accumulation, ice core melt feature, lake core varve, lake core misc, and marine core δ¹⁸O.

Within the current LMR framework, statistical relationships between proxies and key climate variables (e.g. temperature) are used as forward models (i.e. proxy system models) within the data assimilation procedure. To provide a preliminary assessment of the sensitivity of the additional data for this report, linear univariate regressions were calculated for individual proxy time series using annually-averaged calibration data (we used mean annual temperature from GISTEMP v4 (GISTEMP Team, 2018)) available over the instrumental-era (1850 to 2015). The regression parameters are similar to the parameters used by Hakim et al. (2016) on PAGES 2k Consortium (2013) proxy time series. The models are formulated as the relationship (slope (m) and intercept) between the expected proxy value (y_e) and the independent environmental variables (x, in this case temperature) (for example, y_e = m*x + intercept). Regression statistics are included on the ‘dashboard’ graphical data summary pages and the archive summaries. Correlations (R²) relate the expected to the observed values. The program labeled LMR_PSMbuild.py performs this step. The linear univariate annual regressions are intended to be a first step in data exploration, extending the approach used by Hakim et al. (2016) to the the additional data. Other models, particularly models including moisture related variables, can be explored and may be more appropriate for the moisture-sensitive proxies that we have added. These comparisons will be reported elsewhere.

The development of series-specific regression models reduces the number of records that can be assimilated from 2290 to 2244. The linear regression to annual mean temperature used by Hakim et al. (2016) is only one of many possible relationships with the environment, and indeed the purpose of structuring this collection and including moisture-sensitive proxies such as the Breitenmoser et al. (2014) chronologies and ice core oxygen isotopes and coral oxygen isotopes is to support exploration of alternative models, including bivariate regression models (temperature and moisture) and other proxy system models (Dee et al. 2015). Cases where a regression was not produced are due to lack of observations, either proxy observations or calibration observations near the proxy location during the target calibration interval (1850–2015). This reduces the ice core records from 77 to 61, corals from 19 to 11, and speleothem records from 20 to 6 (Table 3). The criteria used here follow Hakim et al. (2016): a linear regression of the annualized proxy values to annual mean temperature of the corresponding grid cell from the GISTEMP v4 temperature product, for the period 1850–2015, with a requirement of annual proxy resolution, and 50% data available over the period 1850–2015. No minimum correlation threshold is set for models. Records with poor fits have less weight in the assimilation.

The regression summary statistics (Table 4) support consideration of all these archive types in assimilation. Statistics based on fewer than 10 records are struck-through and not described here, however we present the R² information on the dashboards for each record. Several measures are useful in assessing how well the regression models fit the observations. The correlation (R²) directly reflects the fit of the regression model and appears for each record in the online supplement. The signal to noise ratio is used in archive summaries to compare different proxies. The coefficient of efficiency used by us (Hakim. et al. 2016) assesses the skill of the assimilation analysis. Here it is applied to the point-based reconstructions. Coral δ¹⁸O (37.82) has the highest signal-to-noise (SNS) ratio, followed by ice core δ¹⁸O. These records also have the highest coefficient of efficiency. The mean SNS decreases from 4.3 to 3.5 and mean CE decreases from 0.033 to 0.030 for chronologies when all trees (irrespective of their sensitivity to temperature or moisture) are considered compared to the chronologies screened for the sensitivity to temperature (not shown). This suggests that alternate regressions, involving moisture in addition to temperature, could provide more accurate proxy modeling. Nevertheless, differences remain small and the similar goodness-of-fit measures compared to the extensively screened PAGES 2k Consortium (2017) tree ring width set (not shown) justifies their consideration in the LMR data assimilation framework, particularly given the enhanced geographic coverage these chronologies provide, for example in the Southern Andes and Tasmania (Figure 1). Breitenmoser et al. (2014) note that 7% of the analyzed land grid cells contain only temperature sensitive series, 5% contain only moisture-limited series, and 37% contain both temperature and moisture-limited chronologies. The record-specific regression equations, along with the R² values, are reported in the online supplement (data dashboards).

Data Records

The principal product is the set of 2290 structured files consisting of metadata and data in the NOAA text format archived at NOAA Paleoclimatology at the National Centers for Environmental Information (Anderson et al. 2018). The file naming convention for the non-tree records is yyxxxxzz.txt, where yy is the last two digits of the publication year, xxxx is a four letter abbreviation of the site name or code, and zz is a two digit sequence number to account for multiple records from the same publication and site. An exception to this convention is the Palmyra coral record, which is named palmyra2013.txt. The site naming convention for the Breitenmoser tree chronologies is a variable-length concatenation of the region, underscore, the ITRDB code, and the letter B (to indicate the source (Breitenmoser et al. 2014)). The file extension is txt. In the NOAA metadata Collection_Name is this filename without the extension. We assigned or wrote metadata for each file according to the NOAA Paleo metadata v1.0 instructions (NOAA Paleoclimatology, 2018). In the notes section of the Breitenmoser et al. (2014) tree chronologies, we added the VS-lite parameters reported by Breitenmoser et al. (2014) coded as key/value pairs following this example using terminology from Breitenmoser et al. (2014): {“VSLite_parameters”: {“T1”: “5.55447702235”, “T2”: “18.6547994501”, “M1”: “0.022191930217”, “M2”: “0.313406788974”}. The expected seasonality for the Tree Rings was added to the metadata notes section according to the following rule. The seasonality is all months [1, ..12] if the site is located in the tropics and assigned Boreal summer [6, 7, 8] if located in the Northern Hemisphere and Austral summer [11, 12, 1] if located in the Southern Hemisphere, as mentioned earlier.

We emphasize that the data records (files) contain the original data series obtained from the source listed in the metadata. We anticipate that users wanting the processed time series will use the Python code LMR_proxy_preprocess.py (with the options, such as making the annual mean, that suits their application).

Usage Notes

1. The original references found in the NOAA text format files should be cited when these data are used.
2. Additional information about the records (metadata) beyond that provided in this archive exists in the files and supplementary information provided by Breitenmoser et al. (2014), and in the original references listed in the metadata.
3. Users wanting the processed data instead of the raw data provided in these files can process the archive using the Python code LMR_proxy_preprocess.py which produces two Pickle files, one containing the annualized time series and one containing the metadata. Code is available on GitHub (Hakim et al. 2018).
4. Users wanting to add or modify a collection for use with LMR code can remove or add NOAA formatted files to their own directory containing a copy of the collection. The code recognizes the original variables listed in Table 2, and the following time units: [‘age’, ‘age_int’, ‘year’,\‘y_ad’, ‘Age_AD’, ‘age_AD’, ‘age_AD_ass’, ‘age_AD_int’, ‘Midpt_year’, ‘AD’,\‘age_yb1950’, ‘yb_1950’, ‘yrb_1950’,\’kyb_1950’,\‘yb_1989’, ‘age_yb1989’,\‘yb_2000’, ‘yr_b2k’, ‘yb_2k’, ‘ky_b2k’, ‘kyb_2000’, ‘kyb_2k’, ‘kab2k’, ‘ka_b2k’, ‘kyr_b2k’,\‘ky_BP’, ‘kyr_BP’, ‘ka_BP’, ‘age_kaBP’, ‘yr_BP’, ‘calyr_BP’, ‘Age(yrBP)’, ‘age_calBP’, ‘cal yr BP’]. Files not using these time and variable names will not be processed.
5. In addition to the code used for data assimilation, LMR code can be used to analyze the collection (Table 2). Proxy system models can be built, and the characteristics of the data and the models can be analyzed using LMR_PSMbuild.py, summarize_proxy_database.py, and summarize_linearPSM.py codes.

Summary and Application

We describe 2290 additions to 571 previously available proxy records, assembled and structured to facilitate reanalysis of the last millennium. The two data sets are merged by processing software to form the Last Millennium Reanalysis dataset version LMRdb v1.0.0. The additions provide a four-fold increase in the number of sites, improve the data density in the Southern Hemisphere, and massively increase the density of Northern Hemisphere tree ring chronologies spanning the last two centuries. The collection consists of observations in their native units and is all-encompassing, minimally-screened, and minimally-processed. Many of the additional records, notably tree ring widths and ice and cave speleothem oxygen isotopes are sensitive to precipitation and other hydrologic variables, and are expected to enable the reanalysis of variables other than temperature, including precipitation, drought indices, and surface pressure. The intended applications include 1) the development and testing of proxy system models, 2) climate reconstruction using various methods such as climate field reconstructions, and 3) reanalysis combining proxy system models and reconstructions derived from model simulations.

Additional Files

The additional files for this article can be found as follows: