2.1.1 History

Today’s connected world crucially depends on the open availability of data on the Internet. The great success of the Protein Data Bank (Berman et al. 2003, 2012) and open sequence databases like UniProt (The UniProt Consortium 2015) demonstrates the power of data sharing when it is unhindered by paywalls and copy restrictions.

The Crystallography Open Database (COD) builds on the experience of such open databases and harnesses the power of the community to build an openly-available chemical crystallography database on the net. The COD ingests data in the standard CIF format maintained by the IUCr (Hall et al. 1991, Bernstein et al. 2016), validates it according to IUCr dictionaries and quality criteria, and offers consolidated data to COD users, again in the standard CIF format. Currently, the COD contains over 376,000 records, spanning the years 1915 to the present.

Numerous people have contributed to the COD since its establishment, and it now contains data from several databases such as the American Mineralogist Crystal Structure Database (Rajan et al. 2006), CrystalEye (Day et al. 2012) and the Predicted Structures Open Database, PCOD (Le Bail 2005). Most structures that have been published in electronic format are represented in the COD, provided they were available on the Internet freely or were donated by authors or their institutions. Some prominent structures published in paper form were also digitized and included.

2.1.2 Scope

The COD stores experimental structures of organic, metal–organic and inorganic compounds and minerals; and collects the result of all types of crystallographic diffraction experiments (X-rays, electrons, or neutrons diffracted from single crystals and powders). In this respect, it is more diverse than most other databases that collect small-unit-cell structures. In recent years, furthermore, quantum mechanics computations using density functional theory and other methods have become powerful enough to yield reliable structural descriptions, either ab initio or in conjunction with experimental crystallographic techniques. Such structures are also collected and are stored in a sister database, the TCOD (Theoretical Crystallography Open Database). Since the COD, TCOD and PCOD all use the same CIF framework for data representation, all databases can be searched and processed using the same software tools.

2.1.3 Features

To keep the COD simple and efficient, several ingredients are crucial. Free/Libre Open Source software (F/LOSS) is at the core of COD development, making it easy to reuse data and algorithms. An open data standard – the Crystallographic Information Framework – and its ontologies maintained by the IUCr are essential for efficient data exchange (Hall and McMahon 2016). The COD has maintained stable REST interfaces since its inception (Gražulis et al. 2009, 2012). From the very beginning each COD record is assigned a persistent identifier (PID) in the form of a seven-digit COD number, and each structure can be identified by a stable URI in the form http://www.crystallography.net/cod/2000000.html. Special care is taken to keep these URIs stable in accordance with the principle that ‘cool URIs do not change’ (Berners-Lee 1998). As a result, the COD is ideally suitable for the 21st century’s open, connected world, providing stable links to crystal structures on the Web. The stable setup allows instantaneous reuse of COD data in various sites, and provides a mechanism for data citation using stable, unique identifiers for every structure.

The COD contains a simple search-and-retrieve Web interface as demonstrated in Figure 1. In addition to web search, it offers download of all data using various protocols (http, svn, rsync) and querying the database directly using a MySQL client. This gives maximum flexibility in cases where the Web interface is not enough, and permits integration of the COD into other websites or standalone programs.

The primary information in the COD is atomic coordinates and crystal descriptions. Moreover, the COD systematically stores experimental data from the diffraction experiment from which the structure was determined (F_obs and powder trace data) when these are available. COD data entries are manually curated as well as automatically checked. From the ingested data, the COD formulates a new CIF file which is guaranteed to be syntactically correct and contains all original data with attached metadata. Should a change of the COD entry be needed for data curation, the COD meticulously stores all changes in a version control database, currently Subversion (Collins-Sussman 2011), thus providing full traceability and data provenance.

The COD enjoys a growing field of applications. Teaching (Gražulis et al. 2015), powder identification (Lutterotti et al. 2015), and source of data for computational material science (First & Floudas 2013, Pizzi et al. 2016) are among the latest areas where the COD has proved useful as a source of data. Structures from the COD were used to produce 3D-printed models that find use in educational work (Moeck et al. 2014, Kaminsky et al. 2014, Stone-Sundberg et al. 2015, Scalfani et al. 2016). Moreover, the open nature of the COD allows structure descriptions to be matched against material properties (Pepponi et al. 2012), yielding the first open collection of material properties and property–structure relations.

In all cases, the COD has benefited from existing crystallographic CIF dictionaries maintained by the IUCr to describe crystallographic entities. However, it has also created domain-specific CIF dictionaries to describe material properties, computational settings and structural metadata. In this way existing CIF dictionaries have been reused without duplication, allowing the application of existing software and database structure for new fields of inquiry.

The COD continuity and its large collection of crystal data allow its use as a data source and management platform for industrial processes. The recently started SOLSA project allows the COD to provide crystal data for mineral identification in mines in real time. It will ingest public data that is obtained during such identification runs, further enriching the COD collection and increasing identification precision.

In the years to come, the COD will proceed to store much larger files that contain diffraction image data. Storing raw diffraction images currently poses a challenge to meet reasonable management and storage costs, along with enough bandwidth for the distribution of image sets. Here again the COD is pursuing a community backed, distributed and open-source solution: it has started to use a Tahoe-LAFS storage engine (Wilcox-O’Hearn & Warner 2008), which is capable of providing affordable redundant storage up to the order of petabytes.

2.2.1 History

In 1965, a small research group based at the University of Cambridge set out to create a compilation of published organic and metal–organic crystal structure data. This group was led by Dr Olga Kennard acting on a vision she shared with the great polymath J D Bernal that ‘collective use of data would lead to the discovery of new knowledge which transcends the results of individual experiments’ (Kennard 1997). In time, this small research group became the Cambridge Crystallographic Data Centre (CCDC) and the initial compilation of data evolved into the Cambridge Structural Database (CSD). Much has changed since those early days: the systems used to maintain the Cambridge Structural Database have had to evolve to accommodate new requirements and many external services have been developed to support the continuing quest for new knowledge.

2.2.2 Scope

The CSD contains the results of over 875,000 crystal structure diffraction experiments and is growing by over 60,000 structures per year (Groom et al. 2016). Structures are typically submitted to the CSD by researchers in CIF format via deposition services provided by the CCDC. These deposition services promote community norms surrounding publication of crystal structure data (Larsen & Kostorz 2011) and make it easy for a researcher to abide by these (Davie & Agbenyega 2015). Many deposited structures end up being associated with a journal article, but increasingly researchers are using the CSD as their primary platform for publication of their crystal structure data (Groom 2016).

A deposited data set is assigned a Digital Object Identifier (DOI), a widely used type of persistent identifier that is actionable and enables interoperability (https://www.doi.org). A CSD DOI will resolve to a landing page where the data can be interactively viewed in a web browser and freely downloaded. The CCDC additionally provides mechanisms that enable linking and discovery of data from journal articles and other resources. Rich metadata is associated with each data set using a combination of automated processing and human validation. This metadata includes a chemical representation of the substance studied, which is vital for enabling the collection of data to be mined for new knowledge. Encoding these chemical representations using the IUPAC International Chemical Identifier or InChI (Heller et al. 2015) enables links between chemical compound records in other resources and crystal structures in the CSD (Day 2014).

2.2.3 Features

The derivation of new knowledge from the data in the CSD is facilitated by a rich set of software applications developed by the CCDC. These enable researchers to search, analyse and visualize the data, and also to readily apply structural knowledge to a range of disciplines including drug discovery (Groom et al. 2012) and materials science (Galek et al. 2016). As well as enabling research, this area of the CCDC’s activity attracts the financial revenue needed to sustain its core community activities from industry and academia without recourse to direct public funding.

The environment in which the CCDC operates has changed significantly since the early beginnings of the CSD in 1965. The rate at which structures are determined has increased rapidly and the data being deposited have become more complex and more diverse. General initiatives around research data management and discovery have resulted in opportunities for new services and greater interoperability between resources.

The CCDC has had to develop its systems over the past half century to adapt to this changing environment whilst remaining sustainable. Internal informatics systems have recently evolved to accommodate increasing throughput and complexity, while external services are developing to ensure that both data and knowledge in the CSD continue to be widely and appropriately available to a broad range of user communities. Although there have been many changes since the early days, one particular feature has been retained: the validation and review undertaken by expert scientists to ensure that data can be reliably used in the discovery of new knowledge today and in the decades to come.

3.1.1 History

The International Centre for Diffraction Data (ICDD®) Powder Diffraction File is a powerful database for materials identification and has been used extensively by the scientific community for 75 years. Over this period, the database has grown exponentially, housing more than 890,000 diffraction patterns and 330,000 crystal structures in Release 2016. Starting with 1000 hand-written file cards in 1941, the Powder Diffraction File (PDF®) has undergone numerous technological developments in the database as well as search/match and data retrieval methods (Faber & Fawcett, 2002).

The PDF started in 1941 as a collection of X-ray powder diffraction patterns of single-phase material providing chemistry, interplanar spacings (d) and relative intensities (I). The amount of data that could be presented was limited in the early days as it was published in printed form. In 1969, it was made available to users in the form of magnetic tapes allowing for some additional crystallographic information to be provided. However, data storage was still an issue on account of expensive disk space and slow computer speeds and data retrieval, rather different from today, when one can store and access gigabytes of data very rapidly using a personal computer. During continued development in data storage technology, the PDF started adding much more information than just d-I lists to the database. In 2000, it was redesigned using a Relational Database (RDB) concept originally proposed by E F Codd (1970).

3.1.2 Scope

Today, the Powder Diffraction File in Relational Database format contains extensive chemical, physical, bibliographic and crystallographic data including atomic coordinates enabling qualitative and quantitative phase analysis (Kabekkodu et al. 2002). In addition to X-ray diffraction patterns, the PDF also contains electron and neutron diffraction patterns enabling scientists to use different diffraction techniques to characterize the material of interest. There have been active recent developments in PDF database capabilities and search/match algorithms.

In 2011, ICDD extended the design of the database to include modulated structures. As these structures need to be described in 3+n dimensions (using a superspace approach), the PDF database has had to be modified in a way that does not change the core format that has been used by several search/match software developers for years.

3.1.3 Features

Over the years Relational Database Management Systems (RDBMS) have evolved exponentially offering much more robust and efficient database platforms. This has facilitated the storage in the PDF of significantly more data including chemical, physical, bibliographic, structural classifications, raw powder patterns, electron and neutron diffraction patterns and crystallographic data including atomic coordinates. All of this additional data content becomes essential when addressing a difficult materials characterization problem. During the past few years ICDD expanded the database scope beyond purely crystalline materials to enable the characterization of amorphous, poorly crystalline and nanomaterials.

The PDF now contains raw diffraction patterns of targeted amorphous, polymer, disordered clays and nanomaterials (and has done so for more than 20 years). Comparing collected diffraction data with reference raw diffraction patterns is essential for these materials as a d-I list is not sufficient to do a comprehensive phase identification. As of release 2016 of the Powder Diffraction File there are 11,284 raw powder diffraction data archived. These experimental raw data are now beginning to be employed by users of the database for their own analysis. Recently the crystal structure of trandolapril was solved by archived powder diffraction raw data in the PDF (Reid et al. 2016). Figure 2 is an example of a PDF entry with archived raw data.

Quality marks, assigned to all PDF database entries, are extremely important when working with a large database with multiple experimental determinations. It is important to know the quality of the crystal structure or diffraction pattern found in the database to screen the search/match result set. With varying quality of published data in the literature, database editorial review processes had to adopt rigorous data evaluation methods to classify diffraction data based on its quality. All PDF data are thoroughly reviewed, edited and standardized. ICDD performs more than 100 quality and error checks on the data prior to publication. ICDD’s editorial group has developed an extensive data validation suite to enrich the editorial process. ICDD corrects all of the resolvable errors found in the original data. An entry gets rejected if the magnitude of an error is large and there is insufficient information to correct the observed error. When the quality mark of a PDF database entry is lowered, those entries are populated with editorial comments describing original data errors and any corrections applied. Data validation, corrections, classification and quality mark assignments are the most important and time consuming step in the editorial process.

4.1.1 History

The Protein Data Bank (PDB) is the single global repository of experimentally determined 3D structures of biological macromolecules and their complexes. It was established in 1971, becoming the first open-access digital data resource in the biological sciences. The PDB serves a growing structural biology community, which generates the incoming data, and provides open access for researchers and educators/students around the world to ~120,000 archival entries plus value added information central to research and education in the basic and applied biological sciences and medicine. The PDB archive also supports research and teaching efforts in the areas of agriculture, animal husbandry, biological energy production, biochemistry, ecology, protein design, and nanotechnology.

The PDB archive is now managed by the Worldwide Protein Data Bank organization (wwPDB) (Berman et al. 2003), which currently includes three founding regional data centres, located in the US (RCSB Protein Data Bank, RCSB PDB: http://rcsb.org), Japan (Protein Data Bank Japan, PDBj: http://pdbj.org) and Europe (Protein Data Bank in Europe, PDBe: http://pdbe.org), plus a global NMR specialist data repository BioMagResBank, composed of deposition sites in the US (BMRB: http://www.bmrb.wisc.edu) and Japan (PDBj-BMRB: http://bmrbdep.pdbj.org). Together, these wwPDB Partners collect, annotate, validate and disseminate standardized PDB data to the public without any limitations on its use.

4.1.2 Scope

Approximately 11,000 new depositions come annually from every inhabited continent, from structural biologists using X-ray crystallography, NMR spectroscopy, and electron microscopy. Worldwide, more than 1 million users come to the PDB every year, as judged by counting unique IP addresses. They download more than 1.5 million structures every day, or more than 500 million per year. They come from 192 of the 195 sovereign nations currently recognized by the United Nations.

4.1.3 Features

The wwPDB Partners are developing the next generation of deposition and annotation tools. A major focus of these tools is validation of submitted atomic coordinates and primary experimental data plus associated metadata. wwPDB biocurators provide depositors with detailed reports that include the results of model and experimental data validation. Validation reports were developed and improved using recommendations from expert Validation Task Forces and Workshops for X-ray (Read et al. 2011), electron microscopy (Henderson et al. 2012), NMR (Montelione et al. 2013), and ligands (Adams et al. 2016). As these wwPDB validation reports provide assessments of structure quality using widely accepted standards and criteria, the wwPDB Partners strongly encourage journal editors and referees to request them from authors as part of the manuscript submission and review process. The reports are date-stamped and display the logo of the wwPDB site where the deposition was curated. They contain the same information, regardless of which wwPDB site processed the incoming data. Provision of wwPDB validation reports is already required by Nature Publishing Group Journals, eLife, the Journal of Biological Chemistry, and IUCr journals as part of their manuscript-submission processes. Once each new PDB entry is released, typically at the time of publication, wwPDB validation reports are made public together with the deposited data.

To analyse how PDB data relate to other publicly available annotations, the RCSB Protein Data Bank (Berman et al. 2000) has developed a novel data integration platform that maps 3D structural information across various data sets. This integration bridges from the human genome across protein sequence to 3D structure space. Users can perform simple searches from the top search bar (e.g. ID, name, sequence, ligand) or build complex combinations of search parameters using Advanced Search. Information from DrugBank (Wishart et al. 2006) is integrated with PDB data to facilitate searches for drugs and drug targets. Other classification systems are used to organize PDB structures in hierarchical trees for browsing and searching (e.g. mpstruc, Gene Ontology, Enzyme Classification).

Protein Feature View provides a graphical mapping of PDB entries onto full-length UniProt sequences. It displays observed and unobserved regions in PDB structures, secondary structure, domain architecture, protein disorder, hydrophobicity profiles, genetic variations, and exon boundaries. Information about available homology models from the Protein Model Portal is also displayed.

Gene View supports visualization of the relationship between genomic and 3D structural data. This tool allows browsing of the human genome with PDB data highlighted onto corresponding genomic ranges. Similar to the Protein Feature View, these data can be correlated with other genomic and protein functional annotations, such as gene structure annotations, DNA repeats, or sequence conservation.

Genome Location to Structure Mapping. A human gene location, e.g. the site of a mutation, can be mapped onto a human gene, and from there to the corresponding protein sequence, and 3D structure. This tool can be used to assess the effect of a single nucleotide variation on the 3D structure.

Validation and Ligand Electron Density Maps. Validation reports and corresponding graphical summary indicators for X-ray, and recently NMR and 3DEM structures are distributed by the wwPDB. These reports assess the quality of each structure and highlight specific concerns. They are provided to depositors as part of the wwPDB Deposition and Annotation System to help identify potential problems that should be addressed prior to PDB deposition and publication. For each structure, the RCSB PDB displays the graphical validation summary and links to the corresponding report. Also at the RCSB PDB website, 3D visualization of the Sigma-weighted 2m|F_o| – d|F_c| electron density ‘mini-maps’ around ligands is available to assess the quality of ligand modeling.

4.1.4 Outreach and Education

PDB archival data are also being used to build educational resources that connect between biomolecular structure and function. RCSB PDB has been publishing Molecule of the Month (Goodsell et al. 2015) articles online continuously for the past 16 years, presenting a selected set of molecular structures in the context of diverse biological and biomedical themes. In 2011, the RCSB PDB established PDB-101 (http://pdb101.rcsb.org) as an outreach and educational resource aimed at training novice PDB users and introducing the power of structural biology and the PDB to diverse audiences. These resources are used by students and educators in primarily high school and undergraduate studies.